Benchmark Almanac

Benchmarks

We track 6,674 benchmarks, with the last one added on November 20th. Arenas, where models are scored with Elo rankings or by human voting, can be found here. Have a benchmark that you want us to track? Let us know here.

Scores refreshed November 21st, 2026
6,674 benchmarks
GPQA Diamond 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. NYU (Rein et al.) Science questions Expert In use Gemini 4 Carbon 97% 258 168 Nov 2023 Nov 2026
AIME 2026 Models solve the 30 integer-answer problems from the 2026 AIME I and II math competitions. Scored by exact answer match. Mathematical Association of America (exam), MathArena (evaluation) Competition problems with answers Competition In use GPT-5.5 100% 35 23 Feb 2026 Oct 2026
Terminal-Bench 4.1 The Terminal-Bench 4.0 tasks with hardened verifiers: agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. Stanford University, Harbor, and Laude Institute Terminal tasks Expert In use Copilot Fusion 71.2% 2 1 Nov 2026 Nov 2026
LiveCodeBench v6 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. UC Berkeley / MIT / Cornell Competitive programming Competition In use DeepSeek-V4-Pro 93.5% 76 53 Apr 2025 Oct 2026
HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Center for AI Safety and Scale AI Expert-written questions Research frontier In use Claude Fable 5.2 77.9% 5 2 Sep 2026 Nov 2026
SWE-bench Verified 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Princeton NLP + OpenAI Fixing real repositories Expert In use Claude Opus 4.7 83.5% 144 82 Aug 2024 Nov 2026
MMLU-Pro Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. TIGER-Lab (Waterloo) Exam questions University In use Intern-S2-397B 89.8% 139 103 Jun 2024 Oct 2026
τ-bench 2.0 2.0.0 Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture. Sierra Customer service University Not in cards GPT-Live-1 79.9% Oct 2026 Sep 2026
IFEval Models answer about 500 prompts containing verifiable instructions such as word limits or required keywords. Code checks each instruction for strict and loose accuracy. Google Following precise instructions Everyday In use Gemma 4 31B IT 98.9% 103 88 Nov 2023 Oct 2026
MMLU 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. UC Berkeley (Hendrycks et al.) Exam questions University In use Seed1.8 92.3% 100 91 Sep 2020 Oct 2026
MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. MMMU team (OSU, Waterloo and others) General visual questions University In use Qwen 3.6 Plus 86% 96 51 Nov 2023 Oct 2026
MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. MMMU team General visual questions University In use GPT-5.6 Sol 84.6% 84 46 Sep 2024 Nov 2026
OSWorld 2.1 Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. XLANG Lab, HKU Desktop apps University In use Claude Opus 5 44.3% 8 5 Sep 2026 Nov 2026
BrowseComp Hard-to-find facts that need persistent web browsing. Short verifiable answers graded for exact correctness. OpenAI Web browsing Expert In use Claude Fable 5 93.3% 79 35 Apr 2025 Nov 2026
SWE-Bench Pro V2 2.0 Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container. Scale AI (co-developed with Reflection) Fixing real repositories Expert In use Claude Opus 5 99.4% 1 Sep 2026 Nov 2026
IFBench 58 new out-of-domain verifiable output constraints. Scored by programmatic constraint checks (strict or loose). AllenAI Following precise instructions Everyday In use Flint 89.2% 69 43 Jul 2025 Oct 2026
BFCL V4 Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. UC Berkeley (Gorilla) Function calling Everyday In use Qwen3.7-Max 75% 32 11 Jul 2025 Oct 2026
SWE-bench Multilingual 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. SWE-bench team (Princeton/Stanford) Fixing real repositories Expert In use Claude Fable 5.2 96.6% 67 39 Apr 2025 Nov 2026
CharXiv Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Princeton NLP Documents, charts and text in images University In use Claude Fable 5 93.5% 67 35 Jun 2024 Nov 2026
HMMT 2026 Models solve the 33 final-answer problems from the February 2026 Harvard-MIT Mathematics Tournament. Scored by exact answer match. HMMT (exam), MathArena (evaluation) Competition problems with answers Competition In use GPT-5.5 98.5% 18 7 Feb 2026 Nov 2026
HumanEval 164 hand-written Python function problems. Generated code is scored pass@1 against unit tests. OpenAI Function-level problems School Fading Solar Pro 2 98.8% 65 70 Jul 2021 Oct 2026
MathVista Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. UCLA + Microsoft Visual math and diagrams School In use Seed2.1 Pro 90.7% 65 42 Oct 2023 Oct 2026
MATH 12,500 competition math problems with step-by-step solutions. Final answer scored by exact match. UC Berkeley (Hendrycks et al.) Competition problems with answers Competition Fading Cogito v2 Preview 671B MoE 98.2% 64 72 Mar 2021 Oct 2026
GSM8K 8.5K grade-school math word problems. Final numeric answer scored by exact match. OpenAI Word problems School Fading Llama 3.1-405B 96.8% 57 72 Oct 2021 Oct 2026
MMMLU MMLU professionally translated into 14 languages. Multiple-choice accuracy. OpenAI Knowledge in many languages University In use Claude Mythos Preview 92.7% 56 42 Sep 2024 Aug 2026
SimpleQA Verified 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. Google DeepMind Short factual questions University In use Gemini 3.1 Pro 77.5% 13 20 Sep 2025 Nov 2026
OCRBench v2 Visual-text localization and reasoning tasks across 31 scenarios (documents, scenes, charts). Averaged task accuracy. HUST (Yuliang Liu et al.) Documents, charts and text in images Everyday In use Qwen3.6 27B 68.3% 20 16 Dec 2024 Oct 2026
DeepSWE 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. Datacurve Fixing real repositories Expert In use GPT-6 Astra 74.1% 54 21 May 2026 Nov 2026
AutomationBench Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Zapier Apps and MCP servers University In use Gemini 4 Neon 56.3% 54 19 Apr 2026 Nov 2026
VideoMME v2 (with subtitles) Models answer multiple choice questions about videos with subtitles or audio added. Related questions are scored as groups to penalize guessing. Video-MME team (Chaoyou Fu et al.) Video understanding Everyday In use Qwen 3.8 Max 68.3% 6 3 Apr 2026 Oct 2026
GDPval-AA v2.1 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Artificial Analysis Tasks from real jobs Expert In use Claude Opus 5.5 1,846 11 6 Sep 2026 Nov 2026
MATH-500 Models solve 500 competition math problems drawn from the MATH test set. Final answers are checked against the reference answers. OpenAI Competition problems with answers Competition Fading LongCat-Flash-Thinking 99.2% 50 46 May 2023 Oct 2026
BIG-Bench Hard Models solve 23 BIG-Bench tasks where earlier models trailed average human raters, such as logic puzzles and date math. Scoring is exact match. Suzgun et al. (Stanford University and Google Research) Language reasoning School Fading ERNIE-4.5-300B-A47B 94.3% 49 59 Oct 2022 Oct 2026
MATH-Vision (MathVision) 3,040 competition math problems with visual context. Answer accuracy. CUHK MMLab Visual math and diagrams Competition In use Qwen 3.8 Max 97.7% 48 32 Feb 2024 Nov 2026
Arena-Hard v2.0 Models answer 500 hard real user prompts from Chatbot Arena and 250 creative writing prompts. GPT-4.1 or Gemini 2.5 judges each answer against a baseline answer from o3-mini, and the score is the win rate. Arena (formerly LMArena) Overall chat quality University Fading Qwen3-Max-2026-01-23 90.2% 20 14 Apr 2025 Aug 2026
ARC-AGI-3 Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. ARC Prize Foundation Puzzles and chess Everyday In use GPT-6.2 Astra 100% 5 2 Mar 2026 Nov 2026
AI2D Models answer multiple choice questions about grade school science diagrams. The set has about 5,000 diagrams and 15,000 questions, scored by accuracy. Allen Institute for AI (AI2) Visual math and diagrams School Fading ERNIE 5.0 96.9% 44 30 Mar 2016 Aug 2026
MCP Atlas Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Scale AI Apps and MCP servers University In use Gemini 4 Argon 90.7% 44 22 Nov 2025 Nov 2026
Toolathlon-Verified 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. HKUST NLP Apps and MCP servers University In use GLM-5.3-Flash 78.4% 18 9 Jun 2026 Oct 2026
AA-LCR 1.1 Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key. Artificial Analysis Reasoning over long documents University In use Step 5 Preview 88.3% 6 2 Sep 2026 Nov 2026
OmniDocBench v1.6 Models parse 1,651 PDF pages, including nested tables, dense formulas and unusual layouts, into text, tables and formulas. Scored by the v1.5 Overall formula. Shanghai AI Lab (OpenDataLab) Documents, charts and text in images Everyday In use OXR-1.0 97.0 5 1 Apr 2026 Nov 2026
MBPP Models write short Python functions from a one-line task description. A solution counts as correct if it passes the task's three assert tests. Google Research Function-level problems School Fading Sarvam 30B 92.7% 41 55 Aug 2021 Oct 2026
SciCode Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. SciCode team Scientific code Expert In use Seed2.1 Pro 59.8% 41 26 Jul 2024 Oct 2026
Agents' Last Exam v1.1 Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references. UC Berkeley RDI (board hosted by Snorkel AI) Tasks from real jobs Expert In use GPT-6.2 Astra 47.5% 8 2 Oct 2026 Nov 2026
DROP Reading comprehension requiring discrete operations (counting, sorting, arithmetic) over paragraphs. Scored by F1. AllenAI + UC Irvine Reading comprehension School Fading Kimi K2 93.5% 36 39 Mar 2019 Sep 2026
C-Eval 13,948 Chinese multiple-choice exam questions across 52 disciplines and four difficulty levels. Accuracy. SJTU + Tsinghua + Edinburgh Exam questions University Fading Qwen3-Max-2026-01-23 93.7% 36 26 May 2023 Sep 2026
MultiChallenge Multi-turn conversations testing instruction retention, inference memory, versioned editing and self-coherence. Graded by LLM judge with rubrics. Scale AI Multi-turn conversation Everyday In use GPT-6.1 Sol 82% 36 21 Jan 2025 Nov 2026
MRCR v2 Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Google DeepMind Finding facts in long inputs Everyday In use Muse Spark 1.3 98.5% 29 18 Jun 2025 Oct 2026
MMStar Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. USTC and Shanghai AI Laboratory General visual questions School In use Qwen 3.8 Max 85.9% 35 22 Mar 2024 Oct 2026
Codeforces rating Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. Codeforces Competitive programming Competition In use DeepSeek V4.1 Flash 3,471 34 35 2024 Sep 2026
MGSM-Rev2 Models solve the 250 MGSM grade-school math problems in 10 languages. Google fixed two English errors, rephrased ambiguous items and retranslated every question with Gemini. Accuracy. Google Reasoning in many languages School Not in cards GPT-5 99.3% Nov 2025 Oct 2025
IMO-AnswerBench 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. Google DeepMind Competition problems with answers Competition In use GLM-5.2 91% 33 20 Nov 2025 Oct 2026
Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. NTU S-Lab (LMMs-Lab) Video understanding University In use Qwen 3.8 Max 88.7% 33 16 Jan 2025 Aug 2026
MMLU-Redux 2.0 Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them. University of Edinburgh Exam questions University Not in cards Aug 2024
SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. ByteDance Seed + M-A-P Expert-written questions Expert In use Qwen 3.6 Max (Preview) 73.9% 31 14 Feb 2025 Oct 2026
ChartQA Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. York University and Nanyang Technological University Documents, charts and text in images School In use Command A Vision 90.9% 30 23 Mar 2022 Aug 2026
DocVQA Models answer 50,000 questions about 12,000 scanned document images by reading their text and layout. Scoring is ANLS, a string similarity to reference answers. CVC, Universitat Autònoma de Barcelona and IIIT Hyderabad Documents, charts and text in images Everyday In use Seed1.5-VL 96.9% 30 23 Jul 2020 Sep 2026
LongBench v2 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Tsinghua KEG / Zhipu Reasoning over long documents Expert In use Inkling 2 Small 70.6% 28 19 Dec 2024 Nov 2026
LVBench Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. Zhipu AI / Tsinghua Long videos Everyday In use Gemini 4 Argon 91.7% 30 19 Jun 2024 Nov 2026
HellaSwag Commonsense sentence completion with adversarially filtered wrong endings. Multiple-choice accuracy. University of Washington + AllenAI Commonsense Everyday Fading InternVL3.5-241B-A28B 97% 29 39 May 2019 Oct 2026
Aider Polyglot Models edit code to solve 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust. Scored by the share passing unit tests. Aider Function-level problems University Fading GPT-5 88% 26 17 Dec 2024 Feb 2026
RealWorldQA Models answer a question about each of 700+ real-world photos, many taken from vehicles, that test basic spatial understanding. Answers are easy to verify. xAI Spatial reasoning and counting Everyday In use Qwen3.8 Flash 88.5% 29 19 Apr 2024 Sep 2026
MMBench-EN v1.1 Models answer English multiple-choice image questions across 20 ability dimensions, using CircularEval with shuffled options. Scored by accuracy on the dev split. Shanghai AI Laboratory (OpenCompass) General visual questions Everyday In use Gemini 2.5 Pro 88.5% 22 11 Apr 2024 Aug 2026
ERQA 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. Google DeepMind Embodied reasoning Everyday In use Gemini Robotics ER 2 78.5% 28 15 Mar 2025 Oct 2026
ScreenSpot-Pro GUI grounding on high-resolution professional-software screenshots. The model must locate the target UI element. ScreenSpot-Pro authors Finding things on screen University In use GPT-6 Astra 92.7% 28 15 Jan 2025 Nov 2026
SimpleVQA Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. Multimodal Art Projection (M-A-P) et al. Short factual questions University In use Qwen3.7-Plus 81.7% 28 9 Feb 2025 Oct 2026
NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. ByteDance + M-A-P + 2077AI Building whole projects Expert In use DeepSeek V4.1 Flash 1017 68.2% 28 6 Dec 2025 Nov 2026
ZeroBench 100 hand-made visual questions that all frontier models originally failed. Scored by exact answer, pass@1 and pass@5. University of Cambridge and others General visual questions University In use GPT-5.6 Sol 54.6% 27 17 Feb 2025 Nov 2026
ARC-Challenge Models answer grade-school science multiple-choice questions that retrieval and word co-occurrence baselines got wrong. Scored by accuracy. Allen Institute for AI Exam questions School Dropped Nemotron 3 Ultra 550B-A55B Base 97.3% 25 43 Mar 2018 Oct 2026
WMT24++ English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. Google Translation University In use K-EXAONE 90.5% 25 19 Feb 2025 Oct 2026
HallusionBench Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. University of Maryland Hallucination rates Everyday In use Keye-VL-2.0-30B-A3B 73.5% 25 18 Oct 2023 Oct 2026
GraphWalks A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. OpenAI Reasoning over long documents University In use Claude Fable 5 99.96% 25 18 Apr 2025 Nov 2026
BabyVision Model solves 388 perception-heavy visual puzzles (discrimination, tracking, spatial, patterns) that young children solve. Accuracy versus 94% human baseline. UniPat AI + xbench Spatial reasoning and counting Everyday In use Qwen 3.8 Max 91.3% 25 9 Jan 2026 Nov 2026
RefCOCO / RefCOCO+ / RefCOCOg Model grounds a referring expression to a bounding box in a COCO image. Scored by accuracy at IoU 0.5. UNC Chapel Hill (Licheng Yu et al.) Detection and captioning Everyday In use Qwen3.5-Omni-Plus 95% 24 17 Jul 2016 Sep 2026
Artificial Analysis Intelligence Index v5 5.0 Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. Artificial Analysis Composite indices In use Claude Fable 5.2 64.6 7 3 Oct 2026 Nov 2026
FrontierCode 2.0 Agents resolve 132 maintainer-built tasks from deliberately ambiguous prompts. Rubric checks ask whether a maintainer would merge the change. Cognition Fixing real repositories Expert In use Claude Fable 5.2 50.2% 10 4 Oct 2026 Nov 2026
HMMT Nov 2025 Models solve the 30 problems of the November 2025 Harvard-MIT Mathematics Tournament. Final answers are checked automatically against the answer key. MathArena (ETH Zurich SRI Lab), problems by HMMT Competition problems with answers Competition Fading GPT-5.2 99.2% 24 11 Nov 2025 Jun 2026
WinoGrande 44K Winograd-style pronoun-resolution problems. Binary-choice accuracy. AllenAI Commonsense Everyday Fading InternVL3.5-241B-A28B 91.7% 22 41 Jul 2019 Oct 2026
MMVU 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. Yale NLP Video understanding University In use Qwen 3.8 Max 82.4% 22 14 Jan 2025 Aug 2026
AndroidWorld Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. Google Research Phones Everyday In use GPT-6 Astra 100% 22 9 May 2024 Oct 2026
HumanEval+ Models write Python functions for the 164 HumanEval problems. EvalPlus checks them against about 80 times more tests than the original. UIUC (EvalPlus) Function-level problems School Fading ERNIE 5.0 94.5% 21 18 May 2023 Apr 2026
CountBench Models count the objects in natural images from the CountBench image-text set. Lab cards often use a question-answer version, CountBenchQA, scored by accuracy. Google Research Spatial reasoning and counting Everyday In use Qwen3.5-27B 97.8% 21 15 Feb 2023 Aug 2026
MMLU-ProX MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. University of Tokyo (Li Lab) and collaborators Knowledge in many languages University In use Qwen3.7-Max 87% 21 8 Mar 2025 Oct 2026
MLVU Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. BAAI + partners Long videos Everyday In use Qwen 3.8 Max 90.8% 21 8 Jun 2024 Aug 2026
Multi-IF Models follow verifiable instructions over three-turn conversations in eight languages, extending IFEval. Scored by instruction accuracy at each turn. Meta Multi-turn conversation Everyday In use LFM3-1.7B 89.1% 20 15 Oct 2024 Oct 2026
τ³-bench Banking Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Sierra Customer service Everyday In use Qwen 3.8 Max 55.1% 20 11 Mar 2026 Nov 2026
MedXpertQA-MM Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. Tsinghua University (TsinghuaC3I) Medicine Expert In use Qwen 3.8 Max 80.4% 20 7 Jan 2025 Oct 2026
TriviaQA Trivia questions with evidence documents. Scored by exact match. University of Washington Trivia and open questions Everyday Fading Llama 3.1-405B 91.8% 19 32 May 2017 Oct 2026
BLINK Models answer 3,807 multiple choice questions built from 14 classic computer vision tasks that use one or more images. Scoring is accuracy. Xingyu Fu et al. General visual questions Everyday In use Seed2.1 Pro 81.4% 19 24 Apr 2024 Oct 2026
Vals Finance Agent v2.1 Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones. Vals AI Finance Expert In use Claude Fable 5.2 69.5% 1 1 Oct 2026 Nov 2026
INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. EPFL and Cohere For AI Knowledge in many languages University Fading Claude Opus 5 89.8% 19 14 Nov 2024 Jul 2026
OpenAI MRCR v2 Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. OpenAI Finding facts in long inputs Everyday In use Gemini 3.7 Flash 89% 13 6 Dec 2025 Nov 2026
MVBench Models answer multiple-choice questions across 20 temporal video tasks that a single frame cannot solve. Scored by accuracy. Shanghai AI Laboratory (OpenGVLab) Video understanding Everyday Fading Qwen3.5-Omni-Plus 79% 19 12 Nov 2023 Aug 2026
BrowseComp-ZH Agents browse the Chinese web to answer 289 multi-hop questions with short, verifiable answers. Scored by accuracy. Researchers from Peking University, Mindverse AI, and other groups Deep research Expert Fading Seed2.0 Pro 82.4% 19 8 Apr 2025 Mar 2026
CyberGym (final-submission metric) final-submission (2026-07) Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug. UC Berkeley (Sunblaze) Finding and exploiting real bugs Expert In use MAI-Cyber-1-Flash 86.3% 1 Jul 2026 Jul 2026
ProgramBench 200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original. Meta Superintelligence Labs, Stanford University and Harvard University Building whole projects Research frontier In use Claude Opus 5.5 18.5% 18 15 May 2026 Nov 2026
MMLongBench-Doc Models answer 1,062 questions about 130 long PDF documents averaging 49 pages, and some questions are unanswerable. Scoring is accuracy and F1. Yubo Ma et al. Documents, charts and text in images University Fading Nemotron 3.5 Super VL 66.6% 18 10 Jul 2024 Oct 2026
CMMLU Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy. MBZUAI + others (Li et al.) Exam questions University Fading Hunyuan-TurboS 90.8% 18 8 Jun 2023 Oct 2026
JobBench 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. University of Washington + collaborators Tasks from real jobs Expert In use GLM-5.3 61.4% 18 6 May 2026 Nov 2026
AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. Microsoft Exam questions School Fading Olmo 3.1 32B Think 89.2% 17 39 Apr 2023 Oct 2026
HealthBench Professional 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. OpenAI Medicine Expert In use Claude Sonnet 5.5 69.2% 17 34 Apr 2026 Nov 2026
ScreenSpot-v2 Models locate a described element on mobile, desktop and web screenshots. A click inside the target box counts as correct. OS-Atlas team (Shanghai AI Lab and others) Finding things on screen Everyday Fading Seed1.5-VL 95.2% 9 12 Oct 2024 Oct 2026
AA-Omniscience v1.1 Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. Artificial Analysis Short factual questions Expert In use Inkling 2 Small 16.1 2 1 Oct 2026 Nov 2026
DynaMath Models answer visual math questions generated from 501 seed programs that vary values and graphs. Scored by average and worst-case accuracy across variants. University of Illinois Urbana-Champaign and UC Berkeley Visual math and diagrams School Fading Qwen 3.6 Plus 88% 17 12 Oct 2024 Jun 2026
PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. Qwen Team, Alibaba Group Competition problems with answers Competition In use Qwen3.7-Max 86.5% 17 11 Apr 2025 Aug 2026
WideSearch 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. ByteDance Seed Questions that need search Everyday In use Tencent Hy4 preview 83.9% 17 6 Aug 2025 Oct 2026
VitaBench 2.0 Agents serve one simulated user across a sequence of life-service subtasks over days or months. They must infer, use and update the user's preferences, judged by rubrics. Meituan LongCat Team Customer service Everyday In use Claude Opus 4.6 50.3% 1 May 2026 Oct 2026
RULERv2 Models solve synthetic multi-key and multi-value needle tasks and QA tasks at basic, easy, medium and hard levels. Contexts run from 8K to 1M tokens, and accuracy is averaged across lengths. NVIDIA Finding facts in long inputs Everyday Not in cards GPT-4.1 89.2% Oct 2025 Jul 2025
LiveBench Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. Abacus.AI, NYU, NVIDIA, et al. Other University Dropped Gemini 4 Neon 87.4% 16 23 Jun 2024 Nov 2026
ZebraLogic Models solve logic grid puzzles of controlled size, derived from constraint satisfaction problems. Scored by the share of puzzles solved fully. Allen Institute for AI Abstract puzzles and logic School Fading Qwen3-VL-235B-A22B 97.3% 16 19 Feb 2025 Feb 2026
MT-Bench 80 two-turn questions graded by GPT-4 as judge. Introduced LLM-as-judge methodology. LMSYS Multi-turn conversation Everyday Fading Qwen2.5-72B 9.35 16 16 Jun 2023 Apr 2026
DeepSearchQA Questions that need multi-step search and synthesis. Final answers graded against references. Google DeepMind Deep research Expert In use Kimi K3 95% 16 13 Jan 2026 Nov 2026
Chartography 100 tasks on specialized professional charts (Kaplan-Meier, candlestick, Sankey, Bode). Answers graded against expert references. Surge AI Documents, charts and text in images Expert In use GPT-6.2 Astra 90.7% 16 11 Aug 2026 Nov 2026
AA-Briefcase v1.1 Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Artificial Analysis Tasks from real jobs Expert In use Claude Opus 5.5 1,822 10 5 Sep 2026 Nov 2026
PIQA Models pick which of two solutions better achieves an everyday physical goal. Scoring is accuracy. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi Commonsense Everyday Dropped Phi-3.5-MoE 88.6% 15 32 Nov 2019 Oct 2026
InfographicVQA Models answer about 30,000 questions on about 5,000 infographics that mix text, layout, charts, and graphics. Scoring is ANLS. Computer Vision Center (UAB) and IIIT Hyderabad Documents, charts and text in images School In use Kimi K2.5 92.6% 15 18 Apr 2021 Aug 2026
Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Cohere Labs Knowledge in many languages University In use Claude Opus 5 95.7% 3 4 May 2026 Oct 2026
FLEURS Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. Google Speech recognition and translation Everyday In use Phi-4-multimodal 4.00% 15 14 May 2022 Oct 2026
GDP.pdf 100 expert multimodal reasoning tasks over real economically important PDF documents. Scored by rubric-criteria pass rate. Surge AI Documents, charts and text in images Expert In use Gemini 4 Carbon 56.4% 15 10 Jul 2026 Nov 2026
AlpacaEval 2.0 Models answer 805 open instructions, and a GPT-4 Turbo judge compares each reply with GPT-4 Turbo's own. The score is a win rate. Tatsu Lab, Stanford University Overall chat quality Everyday Fading DeepSeek-R1 87.6% 15 9 Jan 2024 Apr 2026
Terminal-Bench-Science 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. Stanford-led community (Terminal-Bench team + domain experts) Research science Expert In use GPT-6 Astra 68.1% 15 9 Aug 2026 Nov 2026
V* Models answer 191 multiple choice questions about small details in high-resolution images, such as an object's color or relative position. Penghao Wu and Saining Xie General visual questions Everyday Fading Qwen 3.6 Plus 96.9% 15 5 Dec 2023 Oct 2026
CC-OCR-Bench-V2 Models process 7,093 hard real-world document samples across five OCR tracks, including parsing and document QA. Each track uses its own metric. Chunyi Peng et al. (with Qwen team members) Documents, charts and text in images Everyday In use Qwen3.7-Plus 76.6% 1 May 2026 Aug 2026
FrontierMath Tiers 1-3 v2 Models solve 295 unpublished math problems from undergraduate to advanced graduate level and submit answers that code checks automatically. Scored by share correct. Epoch AI Research mathematics Expert In use Claude Fable 5.2 95.1% 3 Jun 2026 Nov 2026
MBPP+ Models write short Python functions for MBPP problems, and EvalPlus checks each one against many more generated tests than the original. UIUC (EvalPlus) Function-level problems School Fading Cohere Command A 86.2% 14 10 Dec 2023 Apr 2026
CritPt v2 Models solve unpublished research-level physics challenges written by working physicists, with answers graded automatically. The v2 release revised the public test challenges. CritPt collaboration Research science Research frontier In use GPT-6 Astra 83.1% 2 1 Oct 2026 Nov 2026
OJBench Models write C++ or Python solutions to 232 competition problems from NOI and ICPC. Judge test cases score each solution by pass@1. Beijing University of Posts and Telecommunications, Tsinghua University, and others Competitive programming Competition Fading Gemini 3 Pro 70% 14 7 Jun 2025 Jun 2026
EmbSpatialBench Models answer multiple-choice questions on six spatial relations between objects in egocentric embodied scenes. Scored by accuracy. Fudan University Spatial reasoning and counting Everyday Fading Qwen3.5-Omni-Plus 85.4% 14 5 Jun 2024 Jun 2026
SWE Atlas v1.1 Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. Scale AI Fixing real repositories Expert Not in cards Claude Fable 5.2 80.5% Nov 2026 Nov 2026
Global PIQA Models pick the correct solution to everyday commonsense prompts in over 100 languages, many about local foods and customs. Scoring is accuracy. MRL Benchmarks (over 350 researchers) Reasoning in many languages Everyday Fading Gemini 3.1 Pro 93.9% 14 1 Oct 2025 Jun 2026
Claw-Eval 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claw-Eval team (academic) General assistants Everyday In use Claude Sonnet 4.6 81.4% 14 1 Mar 2026 Oct 2026
TruthfulQA Models answer 817 questions across 38 categories that are written to elicit common misconceptions. Scoring checks for truthful answers, often in multiple-choice form. University of Oxford and OpenAI Short factual questions Everyday Fading MAI-Thinking-1 88% 13 28 Sep 2021 Oct 2026
Beyond AIME Models answer 100 original competition-style math problems at or above the hardest AIME level. Each answer is a positive integer checked by exact match. ByteDance Seed Competition problems with answers Competition Fading Seed2.1 Pro 87% 13 8 Apr 2025 Jun 2026
PinchBench 2.0 Models drive an OpenClaw agent through 148 real-world tasks, including data and log analysis, meeting notes, DevOps, PDFs, research and writing. Graded by checks and a Haiku judge. Kilo AI General assistants Everyday Not in cards Claude Opus 4.8 93.5% May 2026 Oct 2026
VLMs Are Blind Models answer seven simple visual tasks such as counting line crossings or overlapping circles, which young children solve easily. Scored by accuracy. Auburn University and University of Alberta Spatial reasoning and counting Everyday Fading Seed2.0 Pro 98.6% 13 5 Jul 2024 Apr 2026
SkillsBench 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. BenchFlow + 105 domain experts Terminal tasks Expert In use GPT-5.5 66.5% 13 2 Feb 2026 Nov 2026
MMAU v05.15.25 Models answer multiple-choice questions about speech, sound and music clips that need expert knowledge and reasoning. Scored by accuracy on test-mini or the hidden-answer test set. University of Maryland (GAMMA lab) Audio understanding and voice University Not in cards Nova 2 Omni Preview 77.8% May 2025 Dec 2025
Harvey LAB v1.1 (HLAB-Vals-1.1) Agents produce legal work product from matter files with file tools and docx, pptx and xlsx skills. A task passes only if every expert rubric criterion passes. Harvey (board run by Vals AI) Law Expert In use Gemini 4 Argon 27.1% 2 Oct 2026 Nov 2026
WebVoyager Agents complete user tasks on 15 popular real websites by browsing from screenshots and page text. A GPT-4V judge grades whether each task succeeded. WebVoyager team (He et al., ACL 2024) Web browsing Everyday Fading Fara1.5-27B 89.3% 12 8 Jan 2024 Oct 2026
MathArena (Apex, IMO, USAMO) Evaluates models on newly released math competitions after release. Final-answer or human/LLM-judged proof scores. ETH Zurich SRI Lab Competition problems with answers Competition In use Claude Opus 4.8 81.3% 12 6 Feb 2025 Nov 2026
RefSpatial-Expand-Bench Models point to an object or free space described by multi-step spatial relations, in more indoor scenes plus new outdoor streets and parking lots. Scored by mask hit. BAAI (RoboRefer) Embodied reasoning Everyday Not in cards Oct 2025
COLLIE Models generate text that meets compositional constraints at word, sentence, paragraph, and passage level. Scored by the share passing automatic constraint checks. Princeton NLP Following precise instructions Everyday Fading GPT-5 99% 12 5 Jul 2023 May 2026
EgoSchema Models answer five-way multiple-choice questions about three-minute egocentric video clips from Ego4D. Scored by accuracy on 5,031 questions. UC Berkeley Long videos Everyday Dropped Qwen2.5-VL-72B 76.2% 12 5 Aug 2023 Apr 2025
ODinW-13 Models detect objects in 13 varied real-world datasets, such as aquarium and aerial images, from text category names. Scored by mean average precision. Microsoft Research Detection and captioning Everyday Fading Qwen 3.6 Plus 51.8% 12 4 Dec 2021 Jun 2026
Claw-Eval Pass^3 Agents complete 300 human-verified tasks in service orchestration, multimodal interaction, and professional dialogue. Pass^3 credits a task only if all three trials pass. Claw-Eval team General assistants University In use LongCat-2.5 78.3% 12 2 Apr 2026 Nov 2026
BoolQ Models read a Wikipedia passage and answer a naturally occurring yes or no question about it. Scored by accuracy. Google Reading comprehension Everyday Dropped PaLM 2-L 90.9% 11 25 May 2019 Jun 2025
ARC-Easy Models answer grade-school science multiple-choice questions from the ARC Easy Set, which has 5,197 questions. Scoring is accuracy. Allen Institute for AI Exam questions School Dropped DeepSeek-V3 98.9% 11 20 Mar 2018 Jun 2025
MultiPL-E Models write code for HumanEval and MBPP problems translated into 18 other programming languages. Scored by pass@1 on unit tests. Northeastern University Programming Research Lab (NuPRL) et al. Function-level problems School Fading Qwen3-Coder-Next 88.2% 11 19 Aug 2022 Apr 2026
VoiceBench Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. CUHK-Shenzhen Audio understanding and voice Everyday In use Qwen3.5-Omni-Plus 93.1% 11 12 Oct 2024 Oct 2026
SWE-bench Multimodal v2 Agents patch JavaScript repositories using issue reports that include images. Automated tests check whether each patch fixes the issue. SWE-bench team Fixing real repositories Expert System cards only Claude Opus 5.5 70.2% 1 Sep 2026 Nov 2026
TextVQA Models answer 45,336 questions about 28,408 images that require reading text in the image. Scoring is VQA accuracy against human answers. Facebook AI Research Documents, charts and text in images Everyday In use MiniCPM-V-4.7-35B-A3B 85% 11 11 Apr 2019 Oct 2026
CoVoST2 Models translate speech from 21 languages into English and from English into 15 languages. Scored by BLEU. Meta AI (Facebook AI) Speech recognition and translation University Fading Nova 2 Omni Preview 40.7 11 11 Jul 2020 Oct 2026
CursorBench 4.1 Ambiguous multi-file coding tasks from real Cursor sessions in Cursor's production harness. Agentic graders score correctness, with cost reported. Cursor Fixing real repositories Expert In use Gemini 4 Neon 65.4% 1 1 Nov 2026 Nov 2026
Chinese SimpleQA 3,000 short Chinese factual questions across six topics. Graded correct, incorrect or not attempted. Alibaba (Taobao & Tmall) Short factual questions University Fading ERNIE 5.0 86% 11 10 Nov 2024 Jun 2026
APEX-Agents 1.1 Agents do long banking, consulting and law tasks in worlds with docs, sheets, email and chat. A new judge scores hedged multiple answers as zero. Mercor Simulated companies Expert Not in cards Gemini 4 Carbon 90.4% Sep 2026 Nov 2026
GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. OpenAI Tasks from real jobs Expert In use Seed2.1 Pro 87.9% 11 2 Sep 2025 Sep 2026
NOVA-63 Models answer questions written by native speakers in 14 languages across 63 academic disciplines. Scoring is accuracy. Peking University and Alibaba Group Knowledge in many languages University Fading Qwen3.5 397B-A17B 59.1% 11 Nov 2025 Jun 2026
MAXIFE Models follow 1,667 verifiable instructions in 23 languages and in cross-lingual settings. Rule-based checks and model judges score compliance. OPPO (Mente Lab) Chat and instructions in many languages Everyday Fading Qwen3.7-Max 89.2% 11 Jun 2025 Jun 2026
HealthBench 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. OpenAI Medicine Expert In use GPT-5 67.2% 10 36 May 2025 Nov 2026
PostTrainBench 1.2 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. AISA group (ELLIS Institute Tübingen and MPI-IS) + University of Tübingen + Thoughtful Lab Machine learning engineering Research frontier In use Claude Fable 5.1 44.6% 3 Sep 2026 Nov 2026
FACTS Grounding v2 Models write long answers from a supplied document of up to 32k tokens. Judge models check that each answer stays faithful to the document. Google DeepMind + Google Research + Kaggle Faithful to sources University Not in cards GPT-6.1 Terra 84.2% Dec 2025 Oct 2026
WritingBench Models complete 1,000 real-world writing requests across 6 domains and 100 subdomains. A judge model scores each reply on five query-specific criteria. Alibaba Group (X-PLUG) Creative writing University Fading GPT-5 83.9 10 11 Mar 2025 Jul 2026
We-Math Models solve visual math problems decomposed into sub-problems by knowledge concept. Scoring compares sub-problem and full-problem results to diagnose knowledge and generalization gaps. Beijing University of Posts and Telecommunications Visual math and diagrams School Fading Qwen 3.6 Plus 89% 10 9 Jul 2024 Jun 2026
SWE-Lancer Models complete real Upwork software tasks, from bug fixes to features, and pick between engineering proposals. Scored by end-to-end tests and payout earned. OpenAI Fixing real repositories Expert Fading GPT-5.3 Codex 81.4% 10 9 Feb 2025 Feb 2026
SWE-Marathon v1.1 Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials. Abundant AI Building whole projects Expert In use Gemini 4 Carbon 53.8% 1 Jul 2026 Nov 2026
Online-Mind2Web Web agents do 300 tasks on 136 live websites. An LLM judge reads the agent's trajectory and decides whether the task succeeded. Ohio State University (OSU NLP Group) Web browsing Everyday Fading Gemini 2.5 Computer Use 69% 10 6 Apr 2025 Jun 2026
MCPMark Verified Agents complete 127 tasks through version-pinned MCP servers. Stabilized verifiers score single-run task success. eval-sys (MCPMark team) Apps and MCP servers University Not in cards Kimi K3 96.1% Jun 2026 Jul 2026
PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. Shanghai Jiao Tong University and Shanghai AI Laboratory Medicine University Fading Qwen 3.8 Max 66.2% 10 1 May 2023 Aug 2026
SLAKE Models answer English and Chinese questions about radiology images of the head, neck, chest, abdomen, and pelvis. Scoring is accuracy. Hong Kong Polytechnic University (Xiao-Ming Wu group) Medicine University Fading Qwen 3.8 Max 90.8% 10 1 Feb 2021 Aug 2026
DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. Qwen Team, Alibaba Group General assistants Everyday Fading Claude Opus 4.6 58.9% 10 Jan 2026 Jun 2026
OpenBookQA Models answer elementary science multiple-choice questions that need a given core science fact plus common knowledge. Scored by accuracy. Allen Institute for AI Exam questions School Fading Hermes 4.3 36B 96.6% 9 22 Sep 2018 Oct 2026
ExploitGym 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. UC Berkeley (Dawn Song lab) Finding and exploiting real bugs Expert In use GPT-6 Astra 42.4% 9 22 May 2026 Nov 2026
BBEH micro average Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Google DeepMind Language reasoning University In use Gemma 4 31B IT 74.4% 9 13 Feb 2025 Sep 2026
MuirBench Models answer multiple choice questions over sets of several images, including unanswerable variants. Scored by accuracy. Fei Wang, Xingyu Fu et al. General visual questions Everyday In use Seed2.0 Pro 81.8% 9 12 Jun 2024 Nov 2026
Terminal-Bench Hard Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. Artificial Analysis Terminal tasks Expert In use Gemma 4 31B IT 36% 9 9 Sep 2025 Jul 2026
IMO 2026 Models write full proofs for the six problems of the 2026 International Mathematical Olympiad. Proofs are scored out of 42 points. International Mathematical Olympiad, hosted in Shanghai, China Written proofs Competition System cards only Claude Opus 5 100% 1 Jul 2026 Jul 2026
Common Voice Crowd-sourced multilingual read-speech corpus. Used here as an ASR test set scored by word error rate. Mozilla Speech recognition and translation Everyday Fading Nova 2.0 Pro Preview 5.50% 9 8 Dec 2019 Jul 2026
Multi-SWE-bench Agents resolve 1,632 GitHub issues in Java, TypeScript, JavaScript, Go, Rust, C, and C++ repos. Patches must pass the repos' tests. ByteDance Seed Fixing real repositories Expert Fading MiniMax-M2.7 52.7% 9 6 Apr 2025 May 2026
Hypersim Models localize named objects as 3D boxes in synthetic indoor Hypersim images, using the Omni3D test split. Scored by mAP at IoU 0.15. Apple (dataset), Meta AI (Omni3D split) Detection and captioning Everyday Fading Qwen3.5-9B 13.5 9 1 Nov 2020 Mar 2026
FrontierMath Tier 4 v2 Models solve 43 unpublished research-level math problems and submit answers that code checks automatically. Scored by share correct. Epoch AI Research mathematics Research frontier In use GPT-6.1 Sol 100% 3 Jun 2026 Nov 2026
ExploitBench 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. Carnegie Mellon (Lee & Brumley) Finding and exploiting real bugs Expert In use GPT-6 Astra 100% 8 28 May 2026 Nov 2026
SIQA Models answer multiple-choice questions about people's motives, reactions, and emotions in everyday social situations. Scored by accuracy. Allen Institute for AI and University of Washington Commonsense Everyday Dropped phi-3-medium 14B 80.2% 8 23 Apr 2019 Oct 2026
Natural Questions Models answer real Google search questions. Model cards usually use the open-domain version, scored by exact match against short answers from Wikipedia. Google Trivia and open questions Everyday Dropped DeepSeek-V2 (MoE-236B, May 2024) 53.4% 8 18 Jan 2019 Aug 2025
FrontierSWE v2 34 ultra-long-horizon engineering and research tasks. 5 trials of up to 20 hours each. Score is mean task score 0-100. Proximal Building whole projects Research frontier In use Claude Fable 5.2 69.9% 4 9 Sep 2026 Nov 2026
Creative Writing v3 Models write responses to 32 creative writing prompts, three times each. A judge model scores them with a rubric and pairwise Elo comparisons. EQ-Bench Creative writing University Dropped Kimi K3.1 Pro 83.2 8 11 Mar 2025 Oct 2026
BioMysteryBench Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. Anthropic Lab and data work Expert In use Claude Fable 5.2 83.7% 8 11 Feb 2026 Nov 2026
LongVideoBench Models answer referring reasoning questions over 3,763 web videos up to an hour long, with interleaved subtitles. Scored by multiple-choice accuracy. NTU S-Lab et al. Long videos Everyday Fading Seed2.1 Pro 80.6% 8 10 Jul 2024 Oct 2026
BigCodeBench 1,140 practical Python tasks requiring calls to 139 libraries. Scored pass@1 on unit tests. BigCode project Function-level problems University Fading DeepSeek V4.1 Flash 60.6% 8 9 Jun 2024 Oct 2026
MotionBench Models answer questions about fine-grained motion in video clips across six motion question types. Scoring is accuracy. Zhipu AI and Tsinghua University (THUDM) Video understanding Everyday Fading Seed2.0 Pro 75.2% 8 8 Jan 2025 Oct 2026
LibriSpeech Models transcribe English audiobook speech. Scored by word error rate on the test-clean and test-other splits. Panayotov et al. (Johns Hopkins University) Speech recognition and translation Everyday Fading Voxtral Small 1.53% 8 8 Apr 2015 Apr 2026
MM-Vet Models answer open questions about images that combine recognition, OCR, knowledge, spatial reasoning, language generation and math. GPT-4 grades each answer against a reference. Weihao Yu, Zhengyuan Yang et al. (NUS and Microsoft) General visual questions Everyday Dropped InternVL3-8B 82.8% 8 7 Aug 2023 Mar 2026
Vibe-Eval Models answer 269 image-based prompts, 100 of them hard, that have expert gold responses. A Reka Core judge rates each answer. Reka AI General visual questions Everyday Dropped Seed2.0 Pro 81.4% 8 6 May 2024 Feb 2026
LiveCodeBench Pro Continuously updated Codeforces, ICPC and IOI problems annotated by olympiad medalists. Reported as Elo. LiveCodeBench Pro team Competitive programming Competition Fading o4-mini 2,116 8 5 Jun 2025 Oct 2026
WorldVQA Model names visual entities in 3,500 image-question pairs across nine categories, isolating memorized visual world knowledge. Exact-answer accuracy. Moonshot AI General visual questions University In use Qwen3.7-Plus 61.1% 8 4 Jan 2026 Oct 2026
Claw-Eval-MM Agents complete the 101 multimodal tasks of Claw-Eval, such as video questions, document extraction, and webpage generation. Rubric items graded from run traces set scores. Claw-Eval team General visual questions University In use Claude Opus 4.6 54.7% 8 2 Apr 2026 Sep 2026
LingoQA Models answer free-form questions about short driving videos. A trained classifier, Lingo-Judge, scores whether each answer is truthful. Wayve General visual questions Everyday In use Qwen 3.8 Max 84.8% 8 2 Dec 2023 Aug 2026
FrontierScience Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. OpenAI Research science Expert In use Intern-S2-397B 87% 8 1 Dec 2025 Nov 2026
Nuscene Models localize named objects as 3D boxes in nuScenes driving camera images. Scored by mAP. Motional (nuTonomy) Detection and captioning Everyday Fading Qwen3.5 397B-A17B 16.0 8 Mar 2019 Mar 2026
HealthBench Hard Models answer the 1,000 hardest HealthBench conversations, graded against physician-written rubrics. OpenAI Medicine Expert Fading GPT-5 46.2% 7 28 May 2025 Oct 2026
SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. University of Illinois Urbana-Champaign (Lingming Zhang) and University of Melbourne Finding and exploiting real bugs Expert Not in cards GPT-5.5 58.4% Jun 2026 Apr 2026
LogicVista Models answer 448 multiple-choice questions on spatial, deductive, inductive, numeric, and mechanical reasoning set in images. Scored by accuracy. UCLA Visual math and diagrams School In use Qwen 3.8 Max 91.9% 7 12 Jul 2024 Aug 2026
MathVerse Models solve diagram-based math problems shown in six versions that move information between the text and the image. It tests whether models read the diagram. CUHK MMLab and Shanghai AI Laboratory Visual math and diagrams School Fading Seed2.1 Pro 89.7% 7 11 Mar 2024 Oct 2026
Bird-SQL Models convert natural language questions into SQL over 95 large real databases across 37 domains. Scored by execution accuracy. HKU and Alibaba DAMO Academy Data science and SQL University Fading GPT-5.5 72.5% 7 10 May 2023 Oct 2026
Vending-Bench 2 Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs. Andon Labs Running a business Everyday In use Gemini 4 Neon $16,843 6 8 Nov 2025 Nov 2026
BenchCAD 2.1-beta Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. BenchCAD team CAD models Expert Not in cards Oct 2026
ChartQA Pro Model answers 1,948 diverse questions over 1,341 charts, infographics and dashboards, including conversational and unanswerable ones. Accuracy. vis-nlp (Masry et al.) Documents, charts and text in images Everyday Fading Claude Opus 4.8 72.3% 7 7 Apr 2025 Sep 2026
HiddenMath Models solve 179 competition-level math problems that experts wrote from scratch. Answers are graded automatically. Google DeepMind Competition problems with answers Competition Dropped DiffusionGemma 26B A4B 80.6% 7 7 May 2024 Jul 2026
WenetSpeech Models transcribe Mandarin speech from many domains. Scored by character error rate on the test-net and test-meeting sets. Northwestern Polytechnical University (ASLP@NPU), Mobvoi and others Speech recognition and translation Everyday Fading 7 7 Oct 2021 Sep 2026
KMMLU-Pro 2026 Models answer questions from the 2026 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. HAERAE-HUB Knowledge in many languages University Not in cards o1 78.1% Oct 2026 Jul 2025
CRUXEval Models predict the output of 800 short Python functions for a given input, or an input that yields a given output. Scored by pass@1. Meta AI and MIT CSAIL Function-level problems School Fading Qwen3-Coder-Next 95.9% 7 5 Jan 2024 Apr 2026
TempCompass Video models answer questions about temporal aspects of short clips, such as action order, speed, direction and attribute change, in several question formats. Peking University and Huawei Noah Ark Lab (Liu et al.) Video understanding Everyday Fading Seed2.0 Pro 89.6% 7 5 Mar 2024 Mar 2026
Daily-Omni Models answer 1,197 multiple choice questions about everyday videos that need audio and visual events aligned in time. Scored by accuracy. Ziwei Zhou, Rui Wang et al. Video understanding Everyday Fading Qwen3.8-Omni-Flash 85.1% 7 5 May 2025 Sep 2026
PopQA Models answer about 14k entity questions built from Wikidata, spread from popular to rare entities. Scored by whether the answer matches a gold alias. University of Washington (Mallen et al.) Trivia and open questions Everyday Fading Tulu 3 405B 55.5% 7 5 Dec 2022 Dec 2025
TVBench Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. University of Santiago de Compostela and University of Amsterdam et al. Video understanding Everyday In use Qwen 3.8 Max 81.9% 7 4 Oct 2024 Aug 2026
FinSearchComp Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. ByteDance Seed Finance Expert In use Ling-3.1-flash 81.7% 7 3 Sep 2025 Nov 2026
SUNRGBD Models detect objects in about 10,000 indoor RGB-D images and predict their 3D bounding boxes. Scored by average precision. Princeton University Detection and captioning Everyday Fading Gemini Robotics-ER 48.3 7 3 Jun 2015 Mar 2026
LIBERO Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. UT Austin Simulated robots Everyday Fading MolmoAct 2 97.2% 7 3 Jun 2023 May 2026
Seal-0 Models answer 111 fact-seeking questions where web search returns conflicting or unhelpful results. Answers are graded for accuracy, usually with search enabled. Virginia Tech Questions that need search University Fading Kimi K2.5 57.4% 7 2 Jun 2025 Feb 2026
Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Z.ai (zai-org) Web and app front ends Expert In use Claude Fable 5 82.3% 7 1 Mar 2026 Nov 2026
CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Alibaba Qwen Tasks from real jobs Expert In use Claude Fable 5 75.9% 7 1 Sep 2026
Big Bench Audio Models answer 1,000 spoken questions from four Big Bench Hard tasks: formal fallacies, navigate, object counting and web of lies. Built to test reasoning over audio input. Artificial Analysis Audio understanding and voice School In use Grok Voice 3.0 98.5% 7 1 Dec 2024 Oct 2026
Artificial Analysis Coding Agent Index v2 2.0 Coding agents, each a model plus harness, attempt software engineering, terminal, and repository question tasks. The index averages three equal-weight benchmark scores. Artificial Analysis Fixing real repositories In use Gemini 4 Neon 58.6 3 1 Oct 2026 Nov 2026
AlignBench Models answer 683 real Chinese user queries across eight categories. A GPT-4 judge rates each answer from 1 to 10 against a reference. Tsinghua University and Zhipu AI (THUDM) Overall chat quality Everyday Dropped Qwen3-235B-A22B 8.94 6 13 Nov 2023 Sep 2025
CVBench Models answer 2,638 manually checked 2D and 3D vision questions built from ADE20K, COCO, and Omni3D. Scoring is accuracy. New York University (Cambrian-1 team) Spatial reasoning and counting Everyday Fading Seed1.8 88% 6 13 Jun 2024 Oct 2026
OfficeQA Pro V2 Agents answer 90 analytical questions using about 120,000 pages of U.S. Treasury Accounts of Receipts and Expenditures from 1793 to 2024. Answers are scored by exact match. Databricks Documents and spreadsheets University In use Claude Opus 5 54.4% 1 Aug 2026 Oct 2026
OfficeQA Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. Databricks Documents and spreadsheets University In use Claude Opus 4.7 86.3% 6 11 Mar 2026 Nov 2026
FLORES+ plus Models translate English Wikipedia sentences that professionals and community contributors translated into over 200 languages. Outputs are scored with metrics such as chrF against the references. Open Language Data Initiative (OLDI) Translation University System cards only Muse Glimmer 47.5 1 Oct 2023 Oct 2026
FRAMES Models answer 824 multi-hop questions that need facts from 2 to 15 Wikipedia articles. Answers are graded against gold answers for accuracy. Google Questions that need search Everyday Fading Grok 4.1 Fast 87.6% 6 6 Sep 2024 Feb 2026
FullStackBench Models write code for 3,374 problems in 16 programming languages across domains like data analysis and web development. Unit tests score each answer. ByteDance Seed Function-level problems University Fading MiniMax-M1-80k 68.3% 6 5 Nov 2024 Feb 2026
WorldSense (omnimodal) Models answer multiple choice questions about real-world videos that need both the visual and the audio track. Scored by accuracy. Jack Hong, Shilin Yan et al. Video understanding Everyday Fading Seed2.0 Lite (0428) 67.3% 6 5 Feb 2025 Sep 2026
CNMO 2024 Models answer problems from the 2024 Chinese National High School Mathematics Olympiad. DeepSeek reports Pass@1 averaged over 16 samples. Chinese Mathematical Society (contest), DeepSeek (eval set) Competition problems with answers Competition Dropped Ring-1T 88.5% 6 5 Dec 2024 Jan 2026
Tulu 3 Safety Eval Models answer harmful, jailbreak and benign prompts from several safety benchmarks in Ai2's safety-eval suite. A WildGuard classifier judges the outputs, and the score is an average. Ai2 Jailbreaks and prompt injection Everyday Fading OLMo 2 1B Instruct 87.6% 6 5 Nov 2024 Apr 2026
PerceptionBench (Moonshot) Moonshot in-house benchmark of atomic visual perception skills. Scoring details unpublished. Moonshot AI General visual questions Everyday In use GPT-5.6 Sol 59.7% 6 3 Jul 2026 Nov 2026
VQAv2 Open-ended questions about COCO images, balanced to reduce language priors. Scored by consensus accuracy against 10 human answers. Virginia Tech + Georgia Tech General visual questions Everyday Fading Reka Edge 88.4% 6 3 Dec 2016 Mar 2026
ResearchRubrics 101 deep-research prompts with 2,593 expert-written weighted rubric criteria (including penalties). Reports graded for grounding, reasoning and clarity. Scale AI Deep research Expert In use LongCat-2.5 81% 6 3 Nov 2025 Nov 2026
MuSR Models answer questions about long generated narratives such as murder mysteries, object placement and team allocation. Each item needs several soft reasoning steps over the story. UT Austin Language reasoning University Fading ERNIE-4.5-21B-A3B-Thinking 86.7% 6 2 Oct 2023 Dec 2025
CL-bench Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. Tencent Hunyuan (with Fudan University authors) Learning from the context University In use GPT-5.4 27.9% 6 2 Feb 2026 Jul 2026
TIR-Bench Models solve 13 kinds of image tasks that need tool use on images, such as rotation, jigsaw, and maze. Most items are multiple choice. Agents-X project (Ming Li et al.) General visual questions Everyday Fading Qwen 3.6 Plus 61.6% 6 Nov 2025 Apr 2026
QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. Qwen Team, Alibaba Group General assistants Everyday Fading Claude Sonnet 4.6 60.5% 6 Apr 2026 Jun 2026
Full-Duplex-Bench v3 3.0 Voice agents hear real human speech with five kinds of disfluency and must make chained API calls across four task domains. Scored on Pass@1, response quality, latency and turn-taking. Guan-Ting Lin, Hung-yi Lee and collaborators (National Taiwan University and others) Speech recognition and translation Everyday Fading TML-Interaction-Small 68% 1 Apr 2026 May 2026
CommonsenseQA 2.0 Models answer 14,343 yes or no commonsense questions that players wrote to fool an AI in a game. Scored by accuracy. Allen Institute for AI and Tel Aviv University Commonsense Everyday Not in cards T5-11B 67.8% Jan 2022 Jul 2023
PaperBench Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. OpenAI AI research tasks Research frontier In use Qwen 3.8 Max 93% 5 12 Apr 2025 Nov 2026
CLUEWSC Models decide whether a pronoun in a Chinese sentence refers to a given noun phrase, a Winograd-style task. Scored by accuracy. CLUE benchmark team Reasoning in many languages Everyday Fading DeepSeek-R1 92.8% 5 7 Mar 2020 Apr 2026
FORTRESS Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Scale AI Refusing harmful requests In use Claude Opus 5.5 5.78% 5 6 Jun 2025 Nov 2026
MME Models answer yes or no questions about images across 14 perception and cognition subtasks with hand-written instructions. Labs usually report the perception sum, out of 2000. Fu et al. (Tencent Youtu Lab and Xiamen University) General visual questions Everyday Fading InternVL3.5-241B-A28B 2525.9 5 6 Jun 2023 Aug 2026
VisuLogic Models answer 1,000 visual reasoning puzzles in six categories that cannot be solved by describing the image in words. Scored by accuracy. Shanghai AI Laboratory with several universities Spatial reasoning and counting Everyday Fading Seed2.1 Pro 54.3% 5 6 Apr 2025 Jun 2026
Vals Index v2.2 Agents are scored on eight finance, coding, legal, and tax benchmarks, combined with weights from each sector's share of US GDP. Vals AI Composite work indices In use Claude Fable 5.2 71.3% 4 3 Oct 2026 Nov 2026
USAMO 2026 Models write full proofs for the six problems of the 2026 USA Mathematical Olympiad. Judges grade each proof from 0 to 7 against rubrics. Mathematical Association of America (contest), MathArena (evaluation) Written proofs Competition In use GPT-5.5 98.2% 3 5 Mar 2026 Jul 2026
PutnamBench Models write formal Lean proofs of Putnam competition problems. A proof counts only when the verifier accepts it. UT Austin (Tsoukalas et al.) Formal proofs Competition Fading Leanstral 1.5 87.5% 5 5 Jul 2024 Jul 2026
Natural2Code Models write Python functions in HumanEval format from new prompts kept off the web. Unit tests decide correctness. Google DeepMind Function-level problems School Dropped DiffusionGemma 26B A4B 94% 5 5 Dec 2023 Jul 2026
Design2Code Models turn screenshots of 484 real webpages into HTML and CSS. Automatic metrics compare the rendered page with the original on layout, text, position, color and CLIP similarity. Stanford SALT and others (Si, Zhang, Yang et al.) Web and app front ends Everyday Fading GPT-4V 87.2 5 5 Mar 2024 Apr 2026
Seed-TTS-eval Models clone a voice from a short prompt and speak new text in English and Mandarin. Scored by word error rate from an ASR model and by speaker similarity. ByteDance (Seed-TTS team) Speech recognition and translation Everyday Fading Qwen3-TTS-12Hz-1.7B-Base 0.77% 5 5 Jun 2024 Sep 2026
ArtifactsBench (Gemini 2.5 Pro 0506 judge) Models generate interactive visual code artifacts for 1,825 tasks. An MLLM judge scores rendered screenshots and code against a per-task checklist. Tencent Hunyuan Web and app front ends University Not in cards Gemini 2.5 Pro 57% Jul 2025 Jul 2025
Audio MC Spoken multi-turn conversations test inference memory, instruction retention, self-coherence and voice editing. O4-mini judge applies per-conversation rubrics. Scale AI Audio understanding and voice Everyday In use Gemini 3.8 Flash 60.4% 5 4 Dec 2025 Oct 2026
miniF2F Models write formal proofs of olympiad and high school competition statements, such as AMC, AIME and IMO problems. A proof assistant checks each proof, and scores are pass@k on the 244 test problems. OpenAI (Kunhao Zheng, Jesse Michael Han, Stanislas Polu) Formal proofs Competition Fading Leanstral 1.5 100% 5 4 Aug 2021 Jul 2026
FActScore Models write biographies of people, and each output is split into atomic facts. The score is the share of facts that a knowledge source supports, as the paper defines FActScore. Sewon Min et al. (University of Washington and collaborators) Long-form factuality Everyday Fading Seed2.0 Pro 71.2% 5 4 May 2023 Feb 2026
AdvancedIF Models respond to over 1,600 prompts with complex, multi-turn, and system-level instructions. A judge model checks each response against expert-written rubrics. Meta Superintelligence Labs Following precise instructions Everyday In use MAI-Thinking-1 85% 5 3 Nov 2025 Sep 2026
GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. Meta AI, Hugging Face, and AutoGPT General assistants Everyday Fading GPT-5 66.1% 5 3 Nov 2023 Mar 2026
CMATH 1.7K Chinese elementary-school math word problems. Final-answer accuracy. Xiaomi AI Lab Word problems School Fading ERNIE-4.5-300B-A47B 96.7% 5 3 Jun 2023 Apr 2026
MM-BrowseComp Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path. ByteDance Seed and collaborators (Li et al.) Deep research Expert Fading GPT-5 24.3% 5 2 Aug 2025 Jun 2026
SpreadsheetBench Real spreadsheet manipulation questions from Excel forums. Output workbook is checked cell-by-cell against reference solutions. Renmin University (RUCKBReasoning) Documents and spreadsheets University In use MiniMax-M3 89.3% 5 2 Jun 2024 Jun 2026
WebArena-Verified Agents complete 812 audited WebArena tasks on self-hosted websites. Deterministic evaluators check the agent's answer and captured network traces. ServiceNow Web browsing Everyday In use Muse Spark 1.1 69% 5 2 Dec 2025 Oct 2026
MLS-Bench Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. MLS-Bench authors (Bohan Lyu et al.) AI research tasks Research frontier In use Claude Fable 5.1 50.3% 5 1 May 2026 Sep 2026
ToolSandbox Models use stateful tools in multi-turn conversations with a simulated user. Tools have hidden state dependencies, and runs are scored against milestones. Apple General assistants Everyday In use LFM2.5-24B-A2B 80.7% 5 1 Aug 2024 Oct 2026
LongFact Models write long answers to 2,280 fact-seeking prompts on 38 topics. A search-backed checker verifies each claim, and cards report the hallucination rate. Google DeepMind Long-form factuality Fading GPT-5 mini 0.70% 5 1 Mar 2024 Jun 2026
AttaQ Models answer 1,402 adversarial questions in seven harm classes, such as deception, violence and PII. Responses are scored for harmlessness. IBM Research Refusing harmful requests Everyday Fading Granite 3.3 8B Instruct 88.5% 5 Sep 2023 Apr 2026
Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. OpenAI Finance Expert Fading GPT-5.5 Pro 88.6% 5 Jul 2025 Apr 2026
DSBench (DeepSeek internal) DeepSeek internal coding-agent test sets: full-stack development tasks and hard agentic coding problems. Pass-rate scoring. DeepSeek Building whole projects Expert In use DeepSeek V4 Pro 0813 71.1% 5 Aug 2026 Aug 2026
StrongREJECT Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. UC Berkeley (Souly et al.) Jailbreaks and prompt injection In use Inkling 98.6% 4 31 Feb 2024 Nov 2026
Needle In A Haystack Models retrieve a planted fact from long filler text at varied context lengths and insertion depths. Scored by retrieval accuracy. Greg Kamradt Finding facts in long inputs Everyday Dropped GLM-4 100% 4 11 Nov 2023 Jan 2025
DRACO 100 deep-research tasks from real Perplexity queries across 10 domains, graded on about 40 expert rubric criteria each. Perplexity Deep research Expert In use Claude Opus 5 88.6% 4 8 Feb 2026 Nov 2026
AISHELL-2 Models transcribe 1,000 hours of Mandarin speech recorded on several devices. Scored by character error rate. AISHELL (Beijing Shell Shell Technology) Speech recognition and translation Everyday Dropped Kimi-Audio-7B-Instruct 2.56% 2 3 Aug 2018 Feb 2026
MM-MT-Bench Models hold multi-turn conversations about images such as charts, documents and photos. GPT-4o grades each answer against a reference on a 1 to 10 scale. Mistral AI General visual questions Everyday Dropped Qwen3-VL-235B-A22B 8.50 4 5 Oct 2024 Jan 2026
SysBench Models follow 500 system messages across five user turns each. A model verifier checks constraints, instructions and whole sessions. Peking University + Baichuan Following precise instructions Everyday Fading GLM-Z1-32B-0414 81.2% 4 5 Aug 2024 Oct 2026
WebArena Agents complete 812 realistic tasks on self-hosted shopping, forum, GitLab, CMS and map websites. Functional graders check the outcome. Carnegie Mellon University Web browsing Everyday Fading Claude Opus 4.6 68% 4 5 Jul 2023 Oct 2026
OSWorld-G Models map 564 desktop instructions to the right screen element or point, covering text matching, element recognition, layout understanding and fine manipulation. XLANG Lab, University of Hong Kong (OSWorld team) Finding things on screen Everyday Fading Fara1.5-27B 71.3% 4 5 May 2025 Jun 2026
RACE-H Models answer multiple-choice reading questions from English exams for Chinese high school students. The test split has 3,498 questions, scored by accuracy. Carnegie Mellon University Reading comprehension School Dropped Claude 3 Opus 92.9% 4 5 Apr 2017 Dec 2024
CWE-bench 2 Agents audit and patch vulnerabilities in 250 codebases. The score is pass@1 under deterministic verification. Collinear AI Secure code and review Expert In use Claude Fable 5.2 75% 1 1 Nov 2026 Nov 2026
TOMATO Models answer 1,484 human-written questions about 1,417 videos that need visual temporal reasoning, such as action counts, direction, rotation and velocity. Scored by accuracy. Yale University and Allen Institute for AI Video understanding Everyday Fading Seed2.1 Pro 79.5% 4 4 Oct 2024 Jun 2026
AMO-Bench Models solve 50 original math problems at or above IMO difficulty that need only a final answer. Answers are graded automatically for accuracy. Meituan LongCat Team Competition problems with answers Competition Fading GPT-5 52.4% 4 4 Oct 2025 Jun 2026
Riemann-bench 25 private research-mathematics problems from professors and IMO medalists. Agents with tools, 100 runs per problem. Surge AI Research mathematics Research frontier In use GPT-5.6 Sol 83.2% 4 4 Apr 2026 Nov 2026
OVBench Video models answer questions about past, current and future moments of an online video stream in 6 task types and 16 subtasks. Nanjing University MCG (Huang et al.) Video understanding Everyday Fading Seed2.1 Pro 70% 4 4 Dec 2024 Jun 2026
Video-Holmes Models answer 1,837 questions about 270 suspense short films across seven reasoning tasks. Each question needs several visual clues from different parts of the film. Tencent ARC Lab and City University of Hong Kong Video understanding Everyday Fading Seed2.1 Pro 68.2% 4 4 May 2025 Jun 2026
MMSU Models answer questions about spoken language that need its content, its paralinguistic cues such as emotion and pitch, and its phonology. Wang et al. (arXiv 2506.04779) Speech recognition and translation University Fading Seed2.0 Lite (0428) 86.5% 4 4 Jun 2025 Sep 2026
OVO-Bench Video models answer questions asked at given timestamps of a video stream in 12 tasks over 644 videos. Tasks cover backward tracing, real-time understanding and forward active responding. Niu et al. (CVPR 2025) Video understanding Everyday Fading Seed2.1 Pro 80.7% 4 4 Jan 2025 Jun 2026
WMT26 v2026 Systems translate the WMT26 general task test set in 24 directions, mostly out of English, across several domains with added instructions. The shared task ranks systems by human contrastive ESA judgments. WMT (Conference on Machine Translation) Translation University In use North Small Translate 1.0 83.6 1 1 Jun 2026 Sep 2026
Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. Shanghai AI Lab (Yuhao Zhou et al.) General visual questions Expert Fading Intern-S2-397B 60.7% 4 4 Jun 2025 Sep 2026
CLIcK Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. KAIST and collaborators (Eunsu Kim et al.) Knowledge in many languages School In use Solar Open 2 90.7% 4 4 Mar 2024 Aug 2026
WildClawBench 60 original in-the-wild tasks run inside a live OpenClaw assistant environment, chaining 10-60+ tool calls. Scored by task-level graders. InternLM (Shanghai AI Lab) General assistants University In use GPT-5.6 Sol 67.2% 4 3 Mar 2026 Oct 2026
VLMsAreBiased Models count or identify features in altered images of famous subjects, such as a logo with an added stripe, across 7 domains. Scoring is accuracy. An Vo et al. Spatial reasoning and counting Everyday Fading GPT-6 Astra 84.1% 4 3 May 2025 Sep 2026
VIBE-V2 (MiniMax) Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output. MiniMax Building whole projects Expert In use MiniMax-M3 50.1% 1 Jun 2026 Jun 2026
Flame-React-Eval Models write React components from UI design images. Rendered outputs are compared with gold components by image similarity and scored as pass@k. Flame-Code-VLM project (Ge et al.) Web and app front ends University Fading GLM-5V-Turbo 93.8% 4 3 Feb 2025 Apr 2026
MMAR Models answer 1,000 questions about real-world audio of speech, sound, music and their mixes. Questions span four reasoning layers from signal to cultural knowledge. Ma et al. (arXiv 2505.13032) Audio understanding and voice University In use Qwen3.5-Omni-Plus 80% 4 3 May 2025 Oct 2026
Harvey LAB-AA v1.1 Artificial Analysis's run of Harvey's Legal Agent Benchmark on Harvey's newer private set of 120 tasks across 24 practice areas. A deliverable with a material hallucination scores zero, and the headline is the share of judges who find every rubric criterion passed. Artificial Analysis Law Expert In use GPT-6.2 Astra 15% 4 3 Oct 2026 Nov 2026
HumanEval Multi Models complete HumanEval problems in Python and seven more languages, including C++, Java, PHP, and Bash. Unit tests score pass@1. DeepSeek-AI Function-level problems School Dropped DeepSeek-V3 82.6% 4 3 Nov 2023 Jan 2025
OSWorld 2.1 (offline subset) Agents complete the 82 OSWorld 2.1 tasks on the official offline list, with no internet access. Scored by checkpoint partial credit and strict pass rate. XLANG Lab, HKU Desktop apps University In use Claude Fable 5.2 89.3% 4 3 Sep 2026 Nov 2026
MMLongBench Vision-language models solve 13,331 long-context examples in five task types, such as visual RAG, needle search and many-shot in-context learning. Each example comes at five lengths from 8K to 128K tokens. University of Edinburgh and collaborators (Wang et al.) Finding facts in long inputs Everyday Fading Seed2.1 Pro 78.3% 4 3 May 2025 Jun 2026
MuChoMusic Models answer multiple-choice questions about music recordings. It tests music knowledge and reasoning in audio-language models. Weck et al. (Universitat Pompeu Fabra, Queen Mary University of London and others) Audio understanding and voice University Fading Qwen3.8-Omni-Flash 72.6% 4 3 Aug 2024 Sep 2026
EQ-Bench 4 Models hold 16-message roleplay chats with 120 simulated user personas whose preferences compete and are hidden. A three-judge panel compares transcripts pairwise, and ratings are fitted as Elo scores. EQ-Bench (Sam Paech) Multi-turn conversation Everyday Not in cards Claude Opus 5 1,365 Jul 2026 Oct 2026
MathArena Apex Shortlist Models solve 47 hard final-answer problems that MathArena selected from 2025 competitions, where the best models scored about 50% at selection. Scored by accuracy over repeated attempts. ETH Zurich SRI Lab Competition problems with answers Competition In use GPT-5.5 98.4% 4 2 Aug 2025 Oct 2026
KOR-Bench Models read a new rule set and apply it to questions in five task types: operation, logic, cipher, puzzle and counterfactual. Scored by accuracy against an answer key. Multimodal Art Projection (M-A-P) with ByteDance and 01.AI Abstract puzzles and logic University Fading o1-preview 72.9% 4 2 Oct 2024 Mar 2026
KorMedMCQA Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. KAIST and collaborators (Sunjun Kweon et al.) Medicine Expert In use o1-preview 92.7% 4 2 Mar 2024 Aug 2026
SUPERChem Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score. Zhao et al. Genomics and life sciences Expert In use Tencent Hy4 preview 66.4% 4 2 Dec 2025 Aug 2026
OmniGAIA Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data. Renmin University of China (RUC-NLPIR) and others Deep research University Fading Gemini 3 Pro 62.5% 4 2 Feb 2026 Sep 2026
QIVD Models answer spoken questions about short phone videos in real time, as in a face to face talk. Each item has a short answer and a timestamp. Qualcomm AI Research and University of Toronto Video understanding Everyday Fading Qwen3.8-Omni-Flash 69.6% 4 2 Mar 2025 Sep 2026
xbench-DeepSearch-2510 Agents answer 100 new Chinese deep search questions that need many web searches and cross-checks. Answers are matched to a key. xbench (HongShan) Deep research Expert Fading Step 3.5 Flash 56.3% 1 1 Oct 2025 Feb 2026
FLTEval Agents complete the formal Lean proofs and define the new mathematical concepts in real pull requests to the Fermat's Last Theorem (FLT) formalization project. Lean and SafeVerify check each submitted diff. Mistral AI Formal proofs Research frontier In use Leanstral 1.5 28.9 4 2 Mar 2026 Jul 2026
WildSpeech-Bench Models hold spoken conversations built from real chat data, with prosody, homophones and stuttering. A judge model scores the replies. Zhang et al. (arXiv 2506.21875) Speech recognition and translation Everyday Fading Qwen3.5-Omni-Plus 75.4 4 2 Jun 2025 Sep 2026
KBL Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items. LBox and collaborators (Yeeun Kim et al.) Law Expert In use Solar Pro 4 77.5% 4 2 Oct 2024 Aug 2026
MTEB Massive Text Embedding Benchmark: retrieval, clustering, classification across languages and modalities. MTEB community Other Fading Qwen3-Embedding-8B 72.2% 4 2 Oct 2022 Oct 2026
WildBench 1,024 challenging real-user tasks with checklist-guided LLM judging. Ai2 Overall chat quality University Dropped Athene-70B 59.4 4 1 Jun 2024 Jan 2026
FACTS Benchmark Suite Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four. Google DeepMind + Google Research + Kaggle Short factual questions University Fading Gemini 4 Carbon 78.3% 4 1 Dec 2025 Oct 2026
SWE-Bench Pro V2 Hard Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol. Scale AI (co-developed with Reflection) Fixing real repositories Research frontier In use Claude Opus 5 98% 4 1 Sep 2026 Nov 2026
EnterpriseOps-Gym 1.1 Agents complete 296 workflows across customer service, human resources, and IT service management. SQL checks grade the resulting database state. ServiceNow AI Research Apps and MCP servers Expert Not in cards Nov 2026
GeneBench Pro Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. OpenAI Genomics and life sciences Expert In use GPT-5.6 Sol 31.5% 4 1 Jun 2026 Oct 2026
MedChemBench (OpenAI) Models work through 99 medicinal chemistry problems drawn from recent papers and patents. Each task mixes prose with structures, assay tables and plots, then asks for the program’s next decision. OpenAI Genomics and life sciences Expert In use GPT-6 Astra 83.8% 4 1 Jul 2026 Oct 2026
LifeSciBench Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. OpenAI Genomics and life sciences Expert In use GPT-Rosalind-6 61.3% 4 1 Jun 2026 Oct 2026
RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. NVIDIA + UT Austin (RoboCasa team) Simulated robots Everyday Fading GR00T N1 50% 4 1 Mar 2025 Jul 2026
Internal API instruction following (OpenAI) Models answer prompts drawn from real developer use cases that carry instructions on formatting, verbosity and length. OpenAI reports a hard subset. OpenAI Following precise instructions Everyday Dropped GPT-4.5 54% 4 Apr 2025 Aug 2025
τ-Voice Full-duplex voice agents handle real-time support calls in retail, airline and telecom with tools and a simulated caller. Scoring covers task completion and natural interaction. Sierra Speech recognition and translation Everyday In use GPT-Live-1 81.7% 4 Mar 2026 Oct 2026
RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. UT Austin + NVIDIA (RoboCasa team) Simulated robots Everyday Not in cards GR00T N1.5 23.9% Mar 2026 Dec 2025
QwenWebBench Models generate front-end code for web apps, games, SVG, and other categories in English and Chinese. Multimodal judgments produce Elo ratings. Qwen Web and app front ends University Fading Qwen 3.6 Max (Preview) 1,532 4 Apr 2026 Apr 2026
MLE-bench Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. OpenAI Machine learning engineering Expert In use GPT-6.2 Astra 96.1% 3 18 Oct 2024 Nov 2026
SQuAD v2 2.0 Models extract answers from Wikipedia paragraphs and must abstain on over 50,000 adversarial unanswerable questions. Scored by exact match and F1. Stanford NLP Reading comprehension Everyday Dropped Llama 3.1-405B 81.8% 2 7 Jun 2018 Nov 2024
Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. Irregular Capture-the-flag challenges Expert In use GPT-6 Astra 90.9% 3 11 Dec 2025 Nov 2026
Gray Swan IPI Arena benchmark Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gray Swan + UK AISI + US CAISI Jailbreaks and prompt injection In use Gemini 4 Carbon 0% 3 10 Mar 2026 Nov 2026
LABBench2 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. Edison Scientific (FutureHouse) Lab and data work Expert In use GPT-5.2 60.9% 3 4 Feb 2026 Oct 2026
ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. OpenAI Finding and exploiting real bugs Research frontier In use GPT-6.2 Astra 53.5% 3 8 Sep 2026 Nov 2026
POPE Vision language models answer yes or no questions about whether objects appear in COCO images. Scored by accuracy or F1 on object hallucination. Renmin University of China (Li et al.) Hallucination rates Everyday Fading InternVL3.5-241B-A28B 90.7% 3 6 May 2023 Aug 2026
All-Angles Bench Models answer over 2,100 multi-view questions about 90 real scenes. Six tasks cover counting, attributes, relative distance and direction, object manipulation and camera pose. Yeh et al. (UC Berkeley, NYU and others) Spatial reasoning and counting Everyday Fading Seed2.0 Pro 72.1% 3 6 Apr 2025 Feb 2026
ArXivMath Final-answer research-level math problems extracted monthly from recent arXiv papers and filtered to be self-contained. MathArena (ETH Zurich SRI Lab) Research mathematics Research frontier In use Gemini 4 Carbon 94.3% 3 6 Feb 2026 Oct 2026
OmniSpatial Vision language models answer over 8,400 spatial reasoning questions in four groups: dynamic reasoning, complex spatial logic, spatial interaction and perspective taking. Jia, Qi et al. (OmniSpatial authors) Spatial reasoning and counting Everyday Fading Nemotron 3.5 Super VL 58.6% 3 6 Jun 2025 Oct 2026
MMBench-CN v1.1 Models answer the Chinese translation of the MMBench image questions, using CircularEval with shuffled options. Scored by accuracy. Shanghai AI Laboratory (OpenCompass) Knowledge in many languages Everyday Fading Gemini 2.5 Pro 88.1% 3 5 Mar 2026
GPQA Main Models answer the 448 question main set of GPQA, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. NYU (Rein et al.) Science questions Expert Dropped Gemini 3 Pro 80.3% 3 5 Nov 2023 Nov 2025
MMSI-Bench Models answer 1,000 human-written questions that need spatial reasoning across several images. Images come from indoor scans, driving, egocentric video and robot data. Shanghai AI Lab (OpenRobotLab) and collaborators Spatial reasoning and counting Everyday Fading Nemotron 3.5 Super VL 43.1% 3 5 May 2025 Oct 2026
ChartMuseum Models answer 1,162 expert-annotated questions about real-world charts from 184 sources. Many questions need visual reasoning that text cannot replace. UT Austin (Tang et al.) Documents, charts and text in images Expert Fading Gemini 3.1 Pro 80.7% 3 5 May 2025 Jun 2026
KMMLU Models answer 35,030 Korean multiple-choice questions across 45 subjects, collected from original Korean exams. Scoring is accuracy. HAE-RAE Knowledge in many languages University Fading Solar Pro 2 77% 3 5 Feb 2024 Jan 2026
MINERVA Models answer complex video questions that need multi-step reasoning, with reference reasoning traces for grading. Google DeepMind Long videos University Fading Seed2.1 Pro 70.7% 3 5 May 2025 Jun 2026
ActivityNet-QA Models answer open-ended questions about web videos of human activities, from 58,000 QA pairs on 5,800 videos. Scored by accuracy. Hangzhou Dianzi University (MILVLG) Video understanding Everyday Dropped Gemini 2.5 Pro 66.7% 3 5 Jun 2019 Jul 2025
QVHighlights Natural-language queries over YouTube videos. Model localizes relevant moments, scored by Recall@1 at IoU 0.5. UNC Chapel Hill Video understanding Everyday Fading Nova 2 Lite 77.2% 3 5 Jul 2021 Dec 2025
MTOB Models learn to translate between English and Kalamang, a language with under 200 speakers, from one grammar book. Scored by chrF. Harvard University and collaborators Learning from the context Expert Dropped Llama 4 Maverick 54.0 3 5 Sep 2023 Jul 2026
PHYBench Models solve 500 original physics problems from high school to Physics Olympiad level. Answers are expressions checked against the key. Qiu et al. (PHYBench team) Science olympiads Competition Fading Tencent Hy3 77.4% 3 4 Apr 2025 Jul 2026
IMO-ProofBench Models write proofs for 60 olympiad problems, split into a basic set and an advanced set up to IMO-hard. Each proof is graded from 0 to 7 by experts or a Gemini autograder. Google DeepMind (Superhuman Reasoning team) Written proofs Competition Fading GPT-5.5 Pro 88.1% 3 4 Nov 2025 Sep 2026
EgoTempo Models answer questions about egocentric videos that require temporal understanding across the whole clip. Google DeepMind and University of Bristol Video understanding Everyday Fading Seed2.0 Pro 71.8% 3 4 Mar 2025 Apr 2026
HLE-Verified Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. SKYLENAGE AI Expert-written questions Expert In use Gemini 3 Pro 48.2% 3 4 Feb 2026 Sep 2026
Blueprint-Bench 2 Agents turn about 20 interior photos of each of 50 apartments into 2D floor plans. Scores compare room connectivity with ground truth, normalized against random. Andon Labs Spatial reasoning and counting Everyday In use Gemini 4 Carbon 58.6% 3 3 May 2026 Nov 2026
Video SimpleQA Video models answer multi-hop fact-seeking questions that need knowledge beyond the video and evidence from its time segments. Short definitive answers are graded for factual correctness. Cao et al. (Video SimpleQA team) Short factual questions University Fading o3 66.3% 3 3 Mar 2025 Jun 2026
Inverse IFEval Models follow 1,012 Chinese and English instructions that conflict with habits learned in fine-tuning, such as writing code without comments. A model judge scores compliance. ByteDance Seed and collaborators (Zhang et al.) Following precise instructions Everyday Fading o3 76.1% 3 3 Sep 2025 Oct 2026
OCR-Reasoning Models answer 1,069 human-annotated questions that need reasoning over text-rich images. The questions cover 6 reasoning abilities and 18 practical tasks. SCUT-DLVCLab (Huang et al.) Documents, charts and text in images Everyday Fading o1 44.4% 3 3 May 2025 Oct 2026
AceBench Models call tools from 4,538 Chinese and English APIs in normal, special, and multi-turn agent cases. Answers are checked without LLM judges or live APIs. Chen Chen et al. Function calling Everyday Fading GPT-4o 85.4% 3 3 Jan 2025 Feb 2026
KoBALT Models answer 700 expert-written multiple-choice questions on 24 Korean linguistic phenomena in syntax, semantics, pragmatics, phonology and morphology. Seoul National University and collaborators (Hyopil Shin et al.) Reasoning in many languages University Fading Claude 3.7 Sonnet 61% 3 3 May 2025 Apr 2026
Omni-MATH Models solve 4,428 Olympiad-level competition math problems across 33 subdomains and 10 difficulty levels. A GPT-4 or Omni-Judge grader checks answers. Peking University and Alibaba et al. Competition problems with answers Competition Dropped o1-mini 60.5% 3 3 Oct 2024 Oct 2025
tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. Amazon AGI Customer service Everyday Fading Grok 4.1 Fast 82.7% 3 3 Dec 2025 Dec 2025
HRM8K Models solve 8,011 math problems given as English and Korean parallel pairs. It measures how much math reasoning drops when problems are in Korean. HAE-RAE and OneLineAI (Hyunwoo Ko et al.) Word problems School Fading Qwen2.5-72B 63.4% 3 3 Jan 2025 Jul 2026
XLRS-Bench Models answer perception and reasoning questions about extremely large, ultra-high-resolution remote sensing images. XLRS-Bench team (Fengxiang Wang et al.) Detection and captioning Expert Fading Qwen2-VL-7B-Instruct 41.1% 3 3 Mar 2025 Sep 2026
OlympiadBench Models solve bilingual olympiad-level math and physics problems in Chinese and English, many with images. Answers are checked against a key. OpenBMB and Tsinghua University (He et al.) Competition problems with answers Competition Dropped GPT-4V 18% 3 3 Feb 2024 Aug 2025
StreamingBench Models answer questions while a video streams in, as a human viewer would. It tests real-time visual understanding, omni-source understanding and context understanding. Lin et al. (arXiv 2411.03628) Video understanding Everyday Fading LLaVA-OneVision 7B 56.4% 3 3 Nov 2024 Sep 2026
WebTailBench Computer-use agents complete 609 hand-verified tasks on live websites across 11 categories, such as bookings, real estate and job search. A companion set of 111 harmful tasks tests refusals. Microsoft Web browsing Everyday Fading GPT-5.4 57.4% 3 3 Nov 2025 Jul 2026
Putnam 2025 Models solve the 12 problems of the 2025 William Lowell Putnam competition with full proofs. Scored out of 120 points. Mathematical Association of America Written proofs Competition Fading DeepSeek-V4-Pro 120.0 2 2 Dec 2025 Jun 2026
VATEX Models write English or Chinese captions for short video clips from a set of over 41,000 videos. Scored by CIDEr. UC Santa Barbara and ByteDance AI Lab Video captioning Everyday Fading Gemini 2.5 Pro 71.3 3 3 Apr 2019 May 2026
Windows Agent Arena Agents complete more than 150 tasks in a real Windows OS with its apps, tools and browsers. Scripts adapted from OSWorld check the final state. Microsoft Desktop apps Everyday Dropped UI-TARS-2 50.6% 3 3 Sep 2024 Nov 2025
Charades-STA Models read a sentence describing an action and return its start and end time in a Charades indoor video. Scored by temporal IoU with the labeled moment. University of Southern California (Gao, Sun, Yang, Nevatia) Video understanding Everyday Fading Seed1.5-VL 64% 3 3 May 2017 May 2026
DUDE Models answer extractive, abstractive, and unanswerable questions about multi-page, multi-domain documents. Scored by ANLS. DUDE team (KU Leuven, CVC Barcelona, et al.) Documents, charts and text in images Everyday Fading Seed2.1 Pro 82.8% 3 3 May 2023 Jun 2026
GenEval Text-to-image models draw prompts about object count, color, position and co-occurrence, and an object detector checks each image. Scored by the share of correct images. University of Washington (Ghosh et al.) Image generation and editing Everyday Fading ERNIE 5.0 90.1% 3 3 Oct 2023 Feb 2026
Ko-LongBench (LG) Models answer questions over long Korean contexts. Tasks cover document QA, story and dialogue history understanding, in-context learning, structured QA and RAG. LG AI Research Reasoning over long documents University Fading K-EXAONE 2.0 89.6% 3 3 Jul 2025 Aug 2026
PhiBench Models answer original prompts that Microsoft's Phi team wrote to test a wide range of skills and reasoning. Microsoft Other University Dropped Phi-4-Reasoning-plus 74.2% 3 3 Dec 2024 Apr 2025
RealKIE-FCC Verified Model extracts structured fields from scanned FCC invoices into a JSON schema. Amazon human-verified labels. Field-level similarity (ALNS). Indico Data (RealKIE) + Amazon AGI (Verified) Documents, charts and text in images Everyday Fading Nova 2.0 Pro Preview 67% 3 3 Dec 2025 Dec 2025
WebQuest Models answer questions from screenshots of one web page, several related pages, or a navigation trace. The test checks extraction and combination of information across pages. Google (Wang et al.) Documents, charts and text in images Everyday Fading GLM-4.6V 69.3% 3 3 Sep 2024 Jan 2026
Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. Meta (Meta Superintelligence Labs) Composite indices Expert In use 3 3 Oct 2026 Nov 2026
Vibe Code Bench 1.1 End-to-end web application development from natural-language specs. Built apps are tested against functional requirements. Vals AI Building whole projects Expert In use Claude Fable 5.2 93.8% 3 2 Mar 2026 Nov 2026
BigCodeBench-Hard Models solve 148 harder BigCodeBench Python tasks that call many libraries, chosen to match real programming work. Scored pass@1 on unit tests. BigCode project Function-level problems University Dropped Claude 3.7 Sonnet 35.8% 3 2 Jul 2024 Jun 2025
DeepResearch Bench Agents write cited research reports for 100 PhD-level tasks across 22 fields. Judges score report quality against references and check citation accuracy. University of Science and Technology of China and Metastone Technology Deep research Expert Fading Claude Opus 4.6 55.3% 3 2 Jun 2025 May 2026
MicroVQA Models answer expert-written multiple choice questions about biology microscopy images that test image understanding, hypothesis generation and experiment proposal. Stanford University (James Burgess, Serena Yeung-Levy et al.) General visual questions Expert Fading GPT-5 59.4% 3 2 Mar 2025 Sep 2026
SEED-Bench-2 Multimodal models answer multiple-choice questions that cover comprehension of interleaved image and text inputs and image generation, in hierarchical capability levels. Tencent AI Lab (ARC Lab) General visual questions Everyday Not in cards Qwen-VL-Plus 71.9% Nov 2023 Feb 2024
SWE-sweep Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved. Meta Superintelligence Labs (with Harvard, UW and Stanford) Fixing real repositories Research frontier In use GPT-6 Astra 8.90% 3 2 Oct 2026 Nov 2026
MM-IFEval Multimodal models follow image-grounded instructions with exact output constraints. Checks mix rule-based verification and a judge model. MM-IFEngine team (Ding et al.) Following precise instructions Everyday In use GPT-4o 64.6% 3 2 Apr 2025 Oct 2026
FrontierScience-Research Models work on open-ended PhD-level research subtasks in physics, chemistry and biology, written by working scientists. A grader scores each answer against a 10-point rubric. OpenAI Research science Expert Fading Muse Spark 38% 3 2 Dec 2025 Jun 2026
AVUT Models answer questions about videos where the audio carries the key information. The benchmark removes questions that text alone can answer. Yang et al. (arXiv 2503.19951) Video understanding Everyday Fading Gemini 1.5 Pro 75.7% 3 2 Mar 2025 Sep 2026
HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. Center for AI Safety and Scale AI (HLE), Alibaba Tongyi Lab (VL subset) General visual questions Expert Fading o4-mini 16% 3 2 Aug 2025 Aug 2026
Omni-Cloze Models write detailed captions of audio, visual and audio-visual clips. A cloze test on the caption content scores the detail. Ma et al. (Omni-Captioner, arXiv 2510.12720) Video captioning Everyday Fading Qwen3.5-Omni-Plus 64.8% 3 2 Oct 2025 Sep 2026
HAE-RAE Bench 1.1 Models answer Korean questions on vocabulary, history, general knowledge and reading comprehension. It tests cultural and contextual knowledge that translated benchmarks miss. HAE-RAE (Guijin Son et al.) Knowledge in many languages School Fading Solar Pro 2 86.2% 3 2 Sep 2023 Jul 2026
KBank-MMLU (Upstage) Models answer Korean finance questions in a multiple-choice format. Upstage built the set in house. Upstage Finance Expert Fading Solar Open 2 80.8% 3 2 Jul 2025 Jul 2026
Legal Research Bench Agents answer US federal and state legal research questions with case-law and web tools. Expert rubrics, all-pass and weighted scores by LLM judge. Vals AI Law Expert In use Gemini 4 Neon 56.3% 3 1 Jun 2026 Nov 2026
HiPhO Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. Shanghai AI Laboratory Science olympiads Competition In use Gemini 3 Pro 78.3% 3 1 Sep 2025 Aug 2026
MobileWorld Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. Tongyi Lab, Alibaba Group Phones Everyday In use GPT-5 55.3% 3 1 Dec 2025 Aug 2026
SALAD-Bench Models answer harmful questions organized in a three-level taxonomy, some with attack or defense changes and some multiple-choice. The MD-Judge evaluator labels responses as safe or unsafe. Shanghai AI Laboratory (OpenSafetyLab) Refusing harmful requests Everyday Fading internlm-chat-20b 96.8% 3 1 Feb 2024 Apr 2026
ComplexFuncBench Models make function calls for 1,000 travel booking style samples that need several steps, user constraints, implicit parameter reasoning, long parameter values or 128K context. Zhipu AI and Tsinghua University (THUDM) Function calling University Dropped Claude 3.5 Sonnet (October 2024) 61% 3 1 Jan 2025 Jun 2025
BrowseComp-Plus Deep research agents answer 830 BrowseComp questions by searching a fixed corpus of about 100,000 human-verified documents. University of Waterloo and collaborators Deep research Expert In use GPT-5 70.1% 3 1 Aug 2025 Oct 2026
SpreadsheetBench 2 Agents complete 321 expert-annotated business spreadsheet workflows, such as financial modeling, debugging, and charting, in multi-sheet workbooks. Scoring is task accuracy. Renmin University of China (RUCKBReasoning) Documents and spreadsheets Expert In use Claude Opus 4.6 34.9% 3 1 Jun 2026 Nov 2026
AMC Models solve problems from the AMC 8, 10, and 12 competitions. Gemini 1.5 used 250 problems from 2022 and 2023, scored by accuracy. Mathematical Association of America Competition problems with answers Competition Fading Falcon-H1-34B 69.4% 3 1 Feb 2024 Dec 2025
DeepMind Mathematics Models answer generated school-level math questions across algebra, arithmetic, calculus, probability and other modules. Answers are checked for an exact match. DeepMind (Saxton et al.) Word problems School Fading Granite 4.1 30B 81.9% 3 1 Apr 2019 Apr 2026
SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. UC San Diego + Stanford + UC Berkeley + Google DeepMind Simulated robots Everyday In use Praxis-1 80% 3 1 May 2024 Nov 2026
Kimi Code Bench v2 Coding agents complete Moonshot's realistic tasks from internal engineering work, production incidents, and open-source projects in over ten languages. Moonshot AI Fixing real repositories Expert In use Kimi K3.1 Pro 77.1% 3 1 Jun 2026 Oct 2026
Vistra Systems translate text in natural images across multiple language directions. Google Research Translation University Fading TranslateGemma 4B 70.7 3 1 Sep 2024 Jan 2026
WorkSpaceBench Agent completes 388 tasks in realistic worker file workspaces (20k files) needing cross-file dependency reasoning. Scored against 7,399 rubrics. SJTU + ByteDance (OpenDataBox) Documents and spreadsheets University In use Qwen 3.8 Max 67.7% 3 1 May 2026 Aug 2026
BankerToolBench v1.1 Agents complete 100 junior investment banker tasks and produce Excel, PowerPoint and Word deliverables. A task passes only if every critical rubric criterion passes. Handshake AI Research Finance Expert Not in cards Claude Opus 5 32.6% Oct 2026 Jul 2026
GSM-Symbolic Models solve grade-school word problems generated from symbolic templates of GSM8K questions, with names and numbers changed. P1 and P2 add one or two extra clauses. Apple (Mirzadeh et al.) Word problems School Fading GPT-4o 94.9% 3 Oct 2024 Apr 2026
RecreationBench Agents explore a running reference app and rebuild it, across 250 tasks on five platforms. Programmatic tests and a VLM judge score the result. Qwen Desktop apps Expert In use GPT-6 Astra 58.1% 3 Sep 2026 Sep 2026
BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. Alibaba Tongyi Lab Deep research Expert Fading Step 3.7 Flash 59% 3 Aug 2025 Jun 2026
Management Consulting Tasks (OpenAI) Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading. OpenAI Running a business Expert In use GPT-5.6 Sol 43.2% 3 Jul 2026 Jul 2026
BigLaw Bench Models review legal documents and receive rubric scores for substantive accuracy and editing quality. Harvey Law Research frontier Fading GPT-5.4 91% 3 Aug 2024 Apr 2026
olmOCR-Bench OCR systems convert 1,403 PDFs to markdown. 7,010 unit tests check text presence and absence, reading order, tables and math. Allen Institute for AI Documents, charts and text in images Everyday In use OXR-1.0 87.8% 3 Feb 2025 Oct 2026
BrowseComp Long Context 128k Models answer BrowseComp questions from a 128k-token context of the needed web pages mixed with related distractor pages. Scored by accuracy. OpenAI Reasoning over long documents Expert Dropped GPT-5.1 90% 3 Aug 2025 Nov 2025
Hy-Backend 2.0 Coding agents complete Tencent in-house backend development tasks in the Claude Code scaffold. Task and scoring details are not published. Tencent Hy team Fixing real repositories Expert In use Tencent Hy4 preview 35.2% 2 Jul 2026 Aug 2026
Hy-SWE Max Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task. Tencent Hy team Fixing real repositories Expert In use Tencent Hy4 preview 64.2% 3 Apr 2026 Aug 2026
Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. Qwen Knowledge in many languages University Dropped Qwen2.5-72B 78.7% 3 Feb 2024 Sep 2024
Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. Qwen Team, Alibaba Group Reasoning in many languages Everyday Dropped Qwen2.5-72B 89.6% 3 Feb 2024 Sep 2024
Kernel Bench L3 Models write fast GPU kernels for the 50 full model architectures in KernelBench Level 3. Scored on correctness and speedup over PyTorch. Stanford Scaling Intelligence Lab Speed and optimization Expert Fading 3 Dec 2024 Jun 2026
MASK Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. Center for AI Safety + Scale AI Deception and alignment Fading Muse Spark Open 98.1% 2 21 Mar 2025 Oct 2026
ECLeKTic Models answer 384 closed-book questions, translated into 12 languages, about facts found in only one language's Wikipedia. Scoring checks transfer across languages. Google Research and Google DeepMind Knowledge in many languages Everyday Dropped Claude Mythos Preview 56.3% 2 10 Feb 2025 May 2026
AgentDojo Agents do tool-use tasks while tools and data carry prompt injections. Attack success rate counts hijacked runs. ETH Zurich SPY Lab + Invariant Labs Jailbreaks and prompt injection Everyday Fading Grok 4 Fast 0% 2 8 Jun 2024 Oct 2026
AutoLogi Models solve open-ended logic puzzles in English and Chinese. Generated verifier programs check each answer against the puzzle constraints. Qwen Team, Alibaba Group and Fudan University Abstract puzzles and logic School Dropped Kimi K2 89.5% 2 8 Feb 2025 Jul 2025
Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. University of Oregon NLP Group (Okapi) Knowledge in many languages University Dropped Qwen2.5-72B 80% 2 8 Jul 2023 Jul 2025
TreeBench Models answer 405 visual questions about small targets in dense SA-1B images and give bounding boxes as evidence. Scored by answer accuracy, with box overlap as a second measure. Wang et al. (TreeVGR authors) Detection and captioning University Fading Seed2.1 Pro 71.1% 2 7 Jul 2025 Jun 2026
VSI-Bench Models watch egocentric videos of 288 indoor scenes and answer over 5,000 questions on object counts, distances, sizes, room layout and route planning. New York University (VisionX, Saining Xie) with Stanford and Yale Spatial reasoning and counting Everyday Fading InternVL3.5-241B-A28B 69.5% 2 6 Dec 2024 Oct 2026
LOFT Models solve retrieval, RAG, SQL, and many-shot learning tasks with whole corpora of up to one million tokens in context. Scored per task. Google DeepMind Finding facts in long inputs Everyday Dropped Gemini 2.5 Pro 87% 2 6 Jun 2024 Jul 2026
MTVQA-Test Models answer 6,778 expert-written questions about text in 2,116 images across 9 languages, such as Arabic, Japanese, and Thai. Scoring is accuracy. ByteDance Documents, charts and text in images Everyday Fading Qwen 3.8 Max 56.6% 2 5 May 2024 Aug 2026
EvalPlus HumanEval+/MBPP+ with 80x more tests. Showed many 'passing' solutions were wrong. UIUC Function-level problems School Dropped GLM-5 87% 2 5 May 2023 Feb 2026
MMVP Models answer 300 questions about 150 image pairs that CLIP embeds as near duplicates. A pair counts only when both answers are right. New York University and UC Berkeley (Tong et al.) Spatial reasoning and counting Everyday Fading Kimi K2.5 87% 2 5 Jan 2024 Mar 2026
IndicGenBench Models summarize, translate, and answer questions across 29 Indic languages using human-translated versions of CrossSum, FLORES, XQuAD, and XorQA. Each task uses its standard metric. Google Research Reading in many languages Everyday Dropped Gemini 2.5 Pro 77.1 2 4 Apr 2024 Nov 2025
MME-RealWorld Models answer human-annotated multiple choice questions about high-resolution real-world images in scenarios such as remote sensing, autonomous driving, monitoring and OCR. MME team (Yi-Fan Zhang, Chaoyou Fu et al.) General visual questions Everyday Dropped InternLM-XComposer2.5-7B 50% 2 4 Aug 2024 Nov 2025
ReMI Models reason over several images per question in math, physics, logic, code, charts, and spatial or temporal tasks. Scored by accuracy. Google DeepMind General visual questions School Fading STEP3-VL-10B 67.3% 2 4 Jun 2024 Jan 2026
TallyQA Models answer open-ended counting questions about images, including complex ones that need object relationships or attributes. Scored by exact count accuracy. Rochester Institute of Technology Spatial reasoning and counting Everyday Dropped Amazon Nova Premier 61.5% 2 4 Oct 2018 Apr 2025
Spider 2.0 Models and agents write SQL for real enterprise databases with large schemas, across BigQuery, Snowflake, SQLite and dbt projects. Answers are checked by executing them. XLANG Lab, The University of Hong Kong, with collaborators Data science and SQL Expert Dropped Gemini 3 Pro 94.2% 1 Nov 2024 Mar 2026
AetherCode Models write solutions to problems from premier programming contests such as IOI and ICPC. Expert-validated test suites judge each solution. ByteDance Seed and collaborators (Wang et al.) Competitive programming Competition Fading o4-mini 35.5% 2 3 Aug 2025 Jun 2026
ANLI Models label premise and hypothesis pairs as entailment, neutral, or contradiction. People wrote the pairs over three rounds to fool strong models. Facebook AI Research (Nie et al.) Language reasoning Everyday Dropped PaLM 2-L 73.1% 2 3 Oct 2019 Aug 2024
Ref-L4 Models locate the object described by a long referring expression in an image. The set has 45,341 annotations over 365 object categories, scored by box accuracy. Chen et al. (Ref-L4 authors) Detection and captioning Everyday Fading GLM-4.5V 89.5% 2 3 Jun 2024 Jan 2026
Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. Qwen Team, Alibaba Cloud Function calling Everyday Dropped Qwen-14B 98% 2 3 Aug 2023 Feb 2024
LiveSports-3K Video models give real-time commentary and answer questions on sports clips. An LLM judge scores the free-form commentary. Chen et al. (LiveCC team, NUS Show Lab and ByteDance) Video captioning Everyday Fading Seed2.0 Lite (0428) 78.1% 2 3 Apr 2025 Jun 2026
Clotho-AQA Models answer crowdsourced yes or no and single word questions about short environmental audio clips. Scored by accuracy. Tampere University (Lipping, Sudarsanam et al.) Audio understanding and voice Everyday Dropped ERNIE 5.0 73.7% 2 3 Apr 2022 Feb 2026
IOI 2025 Model solves the 2025 International Olympiad in Informatics tasks. Scored with official subtask points (max 600). IOI (problems) and NVIDIA (harness) Competitive programming Competition Fading Nemotron 3 Ultra 570.0 2 3 Aug 2025 Jun 2026
MBXP Models solve MBPP problems transpiled with their tests into more than 10 programming languages, such as Java, C++ and Go. Unit tests check each solution. Amazon (Athiwaratkun et al.) Function-level problems School Dropped Amazon Nova Premier 78.4% 2 3 Oct 2022 Aug 2025
VisualPuzzles Models solve visual puzzles that need algorithmic, analogical, deductive, inductive and spatial reasoning but little domain knowledge. Scored by accuracy. Carnegie Mellon University (NeuLab) Spatial reasoning and counting Everyday Fading o4-mini 57% 2 2 Apr 2025 Feb 2026
VisFactor Models take 20 vision-centric subtests adapted from a human cognitive psychology test battery, covering perception primitives and visual memory. Scored by accuracy. CUHK (ARISE Lab) Spatial reasoning and counting Everyday In use GPT-5.5 55.9% 2 2 Feb 2025 Aug 2026
MSEarth Models answer questions about figures from Earth science papers across the atmosphere, cryosphere, hydrosphere, lithosphere and biosphere. Labs report the multiple choice split. Shanghai AI Lab (Xiangyu Zhao, Lei Bai et al.) General visual questions Expert Fading Gemini 3 Flash 68.8% 2 2 May 2025 Mar 2026
MARS-Bench Models handle long multi-turn dialogues built from play-by-play sports commentary. Tasks test ultra multi-turn, interactive multi-turn and cross-turn reasoning. ByteDance Seed and collaborators (Yang et al.) Multi-turn conversation Everyday Fading Claude 3.7 Sonnet 62.3% 2 2 May 2025 Mar 2026
OmniBench Models answer multiple choice questions that need an image, an audio clip and text together. Scored by accuracy. Multimodal Art Projection (M-A-P) Audio understanding and voice Everyday Dropped Qwen2.5-Omni-7B 56.1% 2 2 Sep 2024 Oct 2025
VideoReasonBench Video models watch a sequence of fine-grained operations on a partly hidden state and answer questions that recall, infer or predict that state. Liu et al. (VideoReasonBench team) Video understanding Everyday Fading Gemini 2.5 Pro 56% 2 2 May 2025 Apr 2026
Meta Agents Research Environments (ARE) / Gaia2 Simulated app environments for agents with time and events. Hosts the Gaia2 benchmark. Meta FAIR General assistants Everyday Fading GPT-5 42.1% 2 2 Sep 2025 Oct 2026
PhyX Models solve 3,000 image-based physics problems across six domains such as mechanics and optics. Versions include multiple choice and open-ended answers. Hui Shen et al. Science questions University Fading o4-mini 45.8% 2 2 May 2025 Aug 2026
HRM8K-KSM Models solve the 1,428 problems of the Korean School Math subset of HRM8K. Problems come from Korean olympiad and competition exams. HAE-RAE and OneLineAI (Hyunwoo Ko et al.) Competition problems with answers Competition Fading Qwen2.5-72B 31.6% 2 2 Jan 2025 Aug 2026
CG-Bench Video models answer 12,129 questions about 1,219 long videos and must also find the clue segments that support each answer. Chen et al. (CG-Bench team) Long videos Everyday Fading VITA 33% 2 2 Dec 2024 Apr 2026
ViSpeak-Bench Streaming video models must notice instructions given visually, such as a hand wave, and respond in seven subtasks of visual instruction feedback. Fu et al. (ViSpeak team) Video understanding Everyday Fading GPT-4o 59.8% 2 2 Mar 2025 Apr 2026
VisualWebBench Models answer questions about web page screenshots across seven tasks, such as captioning, OCR, element grounding and action prediction. Junpeng Liu, Yifan Song et al. Finding things on screen Everyday Dropped Gemini 1.0 Pro 48% 2 2 Apr 2024 Oct 2025
AV-SpeakerBench Models answer 3,212 multiple-choice questions about who speaks, what is said and when in real-world videos. Questions need audio and video together. Nguyen et al. (arXiv 2512.02231) Video understanding Everyday Fading Gemini 3 Pro 77.6% 2 2 Dec 2025 Sep 2026
DiscoX Models translate 200 expert texts between Chinese and English at discourse level, averaging over 1,700 tokens each. Metric-S, a reference-free judge, scores accuracy, fluency and appropriateness, and the overall score is their sum. ByteDance Seed and collaborators (Zhao et al.) Translation Expert Fading GPT-5 76.7 2 2 Nov 2025 Mar 2026
CorpusQA 1M Models answer computation-heavy questions whose evidence is spread across hundreds of finance, education, and real estate documents in a 1M-token context. Scored by accuracy. Tongyi Zhiwen (Alibaba) Reasoning over long documents University Fading DeepSeek-V4-Pro 62% 2 2 Dec 2025 Apr 2026
AndroidControl Agents predict the next action on Android app screens from either a high-level goal or a low-level step instruction. Scored by step accuracy against human demonstrations. Google DeepMind (Wei Li, William Bishop et al.) Phones Everyday Dropped LongCat-Flash-Omni 91.2% 2 2 Jun 2024 Oct 2025
VATEX ZH Models write Chinese captions for short videos of human activities from the VATEX dataset. Captions are scored with CIDEr against human references. UC Santa Barbara (UCSB NLP Group) Video captioning Everyday Dropped Gemini 2.5 Pro 59.7 2 2 Apr 2019 Jul 2025
YouCook2 Models describe the procedure steps in YouTube cooking videos. Scored by CIDEr against human step captions. University of Michigan Video captioning Everyday Dropped Gemini 2.5 Pro 188.3 2 2 Mar 2017 Jul 2025
CanItEdit Models edit a given block of code to follow an instruction, such as adding a feature, fixing a described bug or changing the approach. Hidden tests check the edited code. Northeastern University Programming Research Lab (Cassano et al.) Function-level problems University Dropped Seed Diffusion Preview 54.3% 2 2 Dec 2023 Aug 2025
CMO 2025 Models write full proofs for the six problems of the 2025 China Mathematical Olympiad. Graders score proofs out of 126 points. Chinese Mathematical Society Written proofs Competition Fading Seed2.0 Pro 114.0 2 2 Nov 2025 Feb 2026
DeepShop Web agents answer complex shopping queries on live sites with product attributes, search filters and sorting preferences. An LLM judge checks whether results meet each requirement. Lyu et al. (University of Amsterdam and collaborators) Web browsing Everyday Fading MolmoWeb 8B 42.3% 2 2 Jun 2025 Apr 2026
ICPC World Finals 2025 Models solve the 12 problems of the 2025 ICPC World Finals under contest time and attempt limits. Scored by problems solved. ICPC Foundation Competitive programming Competition Fading DeepSeek-V3.2-Speciale 10.0 2 2 Sep 2025 Mar 2026
XpertBench Models complete expert tasks from more than 140 task types in law, finance, education, humanities and science. Over 200 domain experts wrote the prompts and weighted rubrics. ByteDance Seed Tasks from real jobs Expert Fading Seed2.0 Pro 64.5% 2 2 Dec 2025 Apr 2026
DeepConsult Deep research agents answer business and consulting queries. A model judge compares each report pairwise with an OpenAI Deep Research baseline answer. You.com Deep research Expert Fading Seed2.0 Pro 61.1% 2 2 May 2025 Mar 2026
DPG-Bench Text-to-image models draw 1,000 dense prompts with many objects, attributes and relations. A VQA model checks each element of the prompt graph in the image. Tencent (ELLA authors, Hu et al.) Image generation and editing Everyday Fading Janus-Pro-7B 84.2% 2 2 Mar 2024 Jan 2026
FlenQA Models answer reasoning questions whose key facts are padded with filler text to different lengths. Accuracy shows how input length affects reasoning. Mosh Levy, Alon Jacoby, Yoav Goldberg Reasoning over long documents Everyday Dropped Phi-4-Reasoning-plus 97.9% 2 2 Feb 2024 Apr 2025
Harvey LAB-AA v1.0 Artificial Analysis runs agents on Harvey's Legal Agent Benchmark tasks in an offline sandbox, where they write legal work products such as memos and redlines from case documents. Judges grade each deliverable against a task rubric, and the headline is the share of rubric criteria passed. Artificial Analysis Law Expert In use Kimi K3 94.6% 2 2 Aug 2026
KGC-Safety (LG) Models answer risky prompts set in Korean social and cultural contexts. A Safe Rate is scored over four domains: universal human values, social safety, Korean sensitivity and future risk. LG AI Research Refusing harmful requests Everyday Fading K-EXAONE 2.0 99.8% 2 2 Jan 2026 Aug 2026
Ko-AIME 2025 (Upstage) Models solve AIME 2025 competition problems in Korean. Upstage built the set in house. Upstage Competition problems with answers Competition Fading Solar Open 2 97.7% 2 2 Jan 2026 Jul 2026
LongCat safety evaluation (Meituan) Models answer private test queries in four risk categories: harmful content, criminal activity, misinformation and privacy. The score is the share of safe responses in each category. Meituan LongCat Team Refusing harmful requests Everyday Dropped LongCat-Flash-Thinking 95.7% 2 2 Sep 2025 Sep 2025
LongCodeBench Coding model answers repository QA and fixes GitHub issues with context from 32K to 1M tokens. Accuracy and resolve rate. Panasonic AI Research + Sapienza + Stanford Reasoning over long documents Expert Fading Nova 2.0 Pro Preview 84% 2 2 May 2025 Dec 2025
LPFQA Models answer 430 long-tail professional questions drawn from real forum discussions in 7 academic and industrial domains. ByteDance Seed and collaborators (Zhu et al.) Expert-written questions Expert Fading Seed2.0 Pro 52.6% 2 2 Nov 2025 Mar 2026
Matbench Models predict properties of inorganic materials in 13 supervised tasks built from DFT and experimental data. Scored with error or classification metrics per task. Materials Project (Dunn, Wang, Ganose, Dopp, Jain) Research science Expert Fading Intern-S1 75.0 2 2 May 2020 Mar 2026
Meeseeks Models follow complex instructions over several turns and get feedback on failed requirements after each turn. Scored by how many requirements they meet. Meituan Multi-turn conversation Everyday Dropped LongCat-Flash-Chat 43% 2 2 Apr 2025 Oct 2025
MMMB Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. Alibaba AIDC-AI (Parrot authors) Knowledge in many languages School In use LFM2.5-VL-3B 83% 2 2 Jun 2024 Aug 2026
OMEGA Models solve generated competition-style math problems that test exploratory, compositional and transformative generalization beyond training patterns. Answers are checked automatically. UC Berkeley, Ai2 and University of Washington (Sun et al.) Competition problems with answers Competition Fading Olmo 3.1 32B Think 53.4% 2 2 Jun 2025 Dec 2025
OpenAudioBench Models answer spoken questions in five sets: Reasoning QA, Llama Questions, Web Questions, TriviaQA and AlpacaEval. Knowledge sets use accuracy and the open sets use a model score. Baichuan Audio understanding and voice Everyday Dropped LongCat-Flash-Omni 80.1% 2 2 Jan 2025 Oct 2025
SMolInstruct Models solve 14 chemistry tasks such as name conversion, property prediction, molecule captioning, generation and reaction prediction. Each task uses its own metric on the test split. The Ohio State University (LlaSMol team) Genomics and life sciences Expert Fading Intern-S1-Pro 74.8% 2 2 Feb 2024 Mar 2026
TydiQA Models answer information-seeking questions from Wikipedia articles in 11 typologically diverse languages, across 204K question-answer pairs. Scored by F1. Google Research Reading in many languages Everyday Dropped Llama 4 Maverick 31.7% 2 2 Mar 2020 Apr 2025
Kitab Models list books by a given author that meet extra constraints, such as publication dates or title words. Scored by precision and constraint satisfaction. Microsoft Research Long-form factuality Everyday Dropped 2 2 Oct 2023 Apr 2025
MemoryBench (Meta) Models recall key facts from multi-turn conversations of 64K to 100K tokens that include distractor chats and turns that conflict with or update the facts. An LLM judge scores answers on weighted rubrics. Meta (Meta Superintelligence Labs) Finding facts in long inputs Everyday In use 2 2 Oct 2026 Oct 2026
Tax Agent Bench Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real. Vals AI Finance Expert In use Gemini 4 Hydrogen 52.3% 2 1 Sep 2026 Nov 2026
ProfBench Model writes expert-level reports in physics, chemistry, finance and consulting. Calibrated LLM judges grade against PhD/MBA-written rubrics. NVIDIA Tasks from real jobs Expert In use Claude Opus 5 66.2% 2 1 Oct 2025 Aug 2026
Vals Finance Index Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP. Vals AI Composite work indices Expert In use Gemini 4 Neon 68.3% 2 1 Oct 2026 Nov 2026
CFBench Models follow 1,000 real-world instructions with embedded constraints, across more than 200 scenarios and 50 NLP tasks. A judge checks each constraint and the overall requirement. Baichuan and Peking University Following precise instructions Everyday Dropped o1-preview 71.8% 2 1 Aug 2024 Sep 2025
RefusalBench (Nous Research) Models get requests in scenarios that closed and open models commonly refuse. The score is the percent of questions the model answers, averaged over 5 trials. Nous Research Deception and alignment Everyday Fading Hermes 4 405B 57.1% 2 1 Aug 2025 Dec 2025
ComplexConstraints Models answer 75 expert-written professional prompts whose constraints depend on each other, fire conditionally or must be inferred. An LLM judge grades 1,559 rubric criteria one at a time, and a task passes only if all criteria pass. Surge AI Following precise instructions University In use GPT-6 Astra 70.8% 2 1 Jun 2026 Nov 2026
CountQA Models count objects in over 1,500 real-world photos with dense clutter and occlusion. Scoring is accuracy on the count. Google Spatial reasoning and counting Everyday In use Gemini 2.5 Pro 42.9% 2 1 Aug 2025 Aug 2026
MMAU-Pro Models answer 5,305 expert-written questions about speech, sounds, music and their mixes, covering 49 listening skills. Some clips run up to 10 minutes. University of Maryland (GAMMA lab) and collaborators Audio understanding and voice University Fading Gemini 3.1 Pro 72.2% 2 1 Aug 2025 Aug 2026
$OneMillion-Bench Agent solves 400 expert-curated tasks in law, finance, industry, healthcare and science. Rubric grading of accuracy, logic, feasibility and compliance. Humanlaya + BIGAI + xbench Tasks from real jobs Expert In use Claude Fable 5.1 42.1% 2 1 Mar 2026 Sep 2026
PMPP-Hard Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours. Sinatras, sponsored by Prime Intellect GPU kernels Research frontier In use Kimi K3 0.71 2 1 Jul 2026 Nov 2026
HorizonMath Models attack 113 mostly unsolved problems in eight areas of mathematics and the mathematical sciences. A program checks each proposed closed form, construction or bound. University of Oxford, Harvard University, Princeton University and others (Wang et al.) Research mathematics Research frontier In use GPT-5.6 Sol 11.5% 2 1 Mar 2026 Aug 2026
BABE Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference. ByteDance Seed and collaborators (Zhou et al.) Genomics and life sciences Expert Fading GPT-5.1 52.3% 2 1 Feb 2026 Apr 2026
HR-Bench Models answer multiple-choice questions about fine details in 4K and 8K resolution images. Results are reported separately for the 4K and 8K sets. Wuhan University and collaborators (Wang et al.) General visual questions Everyday Fading Qwen3-VL-32B-Instruct 83% 2 1 Aug 2024 May 2026
MultiLoKo Locally relevant knowledge questions in 31 languages, plus translations. Scored by exact match. Meta FAIR Knowledge in many languages Everyday Fading Llama 3.1-405B 31% 2 1 Apr 2025 Sep 2026
OmniVideoBench Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps. Nanjing University (NJU-LINK Lab) and others Video understanding University Fading Gemini 3 Pro 61.8% 2 1 Oct 2025 Sep 2026
JointAVBench Models answer questions about videos that cannot be answered from vision or audio alone. Questions cover speech and sound events over several scene spans. Chao et al. (arXiv 2512.12772) Video understanding Everyday Fading Gemini 2.5 Flash 58% 2 1 Dec 2025 Sep 2026
MiMo Visual Coding Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code. Xiaomi MiMo Web and app front ends Expert In use GPT-6 Astra 82.2% 2 1 Sep 2026 Sep 2026
FATE-H Provers write Lean proofs for 100 abstract and commutative algebra problems at honors course or graduate exam level. The Lean compiler checks each proof. Jiang et al. (FATE team) Formal proofs Expert Fading o3 3.00% 2 1 Nov 2025 Jul 2026
FATE-X Provers write Lean proofs for 100 algebra problems at or beyond PhD qualifying exam level, some beyond what Mathlib covers. The Lean compiler checks each proof. Jiang et al. (FATE team) Formal proofs Research frontier Fading o3 0% 2 1 Nov 2025 Jul 2026
MiMo Code Bench Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details. Xiaomi MiMo Fixing real repositories Expert In use Claude Opus 5 68.6% 2 1 Sep 2026 Sep 2026
RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. NVIDIA Simulated robots Everyday In use Praxis-1 46.2% 2 1 Apr 2026 Nov 2026
RWSearch Agents search the live web to answer 200 expert-written Chinese questions that need many retrieval steps, filtering and arithmetic. Each question has one objective answer. AGI-Eval (Meituan) Deep research Expert Fading GPT-5.2 82% 2 1 Jan 2026 Jul 2026
MMEB-V3 Omni-modality embedding models are scored on 190 tasks across text, image, video, audio, visual documents and agent-centric retrieval. University of Waterloo (TIGER-Lab) and collaborators General visual questions Everyday Not in cards Qwen3-VL-Embedding-8B 32.1% Apr 2026 Jan 2026
CombiBench Provers solve 100 combinatorics problems formalized in Lean 4, from middle school to IMO and university level. Lean checks each proof. Moonshot AI and Project Numina (Liu et al.) Formal proofs Competition Fading Seed-Prover 1.5 48% 2 1 May 2025 Dec 2025
MedXpertQA Models answer 4,460 expert-level medical exam questions across 17 specialties, in a text subset and an image subset. Scoring is multiple-choice accuracy. Tsinghua University (TsinghuaC3I) Medicine Expert Fading Muse Spark 52.6% 2 1 Jan 2025 Jun 2026
TheoremQA Models solve 800 university-level questions in math, physics, EE and CS, and finance that each need a specific theorem. Scored by answer accuracy. University of Waterloo (TIGER-AI-Lab) Science questions University Dropped Qwen2-72B 44.4% 2 1 May 2023 Sep 2024
MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. Xiaomi MiMo Finding and exploiting real bugs Expert In use MiMo-V2.6 Pro 81.7% 2 1 Sep 2026 Sep 2026
Hungarian National High-School Exam Models answer the 33 questions of the 2023 Hungarian national high school mathematics finals. Answers are graded by hand with the official rubric. Keiran Paster (exam by the Hungarian Educational Authority) Word problems School Dropped Grok-1 59% 2 1 Nov 2023 Jan 2024
PresentBench Models generate full slide decks from background materials for 238 real-world cases. A judge scores each deck against about 54 hand-written checklist items. PresentBench team Documents and spreadsheets University In use Qwen 3.8 Max 79.6% 2 1 Mar 2026 Sep 2026
RACE-M Models read passages from English exams for Chinese middle school students and answer multiple-choice questions. Scored by accuracy. Carnegie Mellon University Reading comprehension School Dropped PaLM 2-L 77% 2 1 Apr 2017 Dec 2024
SpreadsheetBench Verified Models solve 400 real spreadsheet manipulation questions from Excel forums, an expert annotated subset of SpreadsheetBench. Each answer runs against several test spreadsheets. Renmin University of China (RUCKBReasoning) with Shortcut.AI Documents and spreadsheets University Fading Seed2.0 Pro 79.1% 2 1 Dec 2025 Jun 2026
AA-AnalystAgent Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right. Artificial Analysis Running a business Expert In use GPT-6.2 Astra 62.5% 2 1 Aug 2026 Nov 2026
ASIMOV 2.0 Models spot physical hazards in text and generated videos from real injury reports, time safety interventions in video, and follow robot payload and handling constraints. Google DeepMind Embodied reasoning Fading Gemini Robotics ER 2 97.9% 2 1 Sep 2025 Oct 2026
Biology-Instructions Models answer multi-omics sequence tasks on DNA, RNA, protein and multi-molecule inputs, such as property prediction and interaction tasks. Scored per task with task metrics. Biology-Instructions team (hhnqqq et al.) Genomics and life sciences Expert Fading Intern-S2-397B 55.7% 2 1 Dec 2024 Sep 2026
Covert misbehavior (Thinking Machines) Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Thinking Machines Lab Deception and alignment Expert In use Inkling 2 Mini 8.30% 2 1 Nov 2026 Nov 2026
Destructive actions (Thinking Machines) Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Thinking Machines Lab Deception and alignment Expert In use Inkling 2 Small 4.70% 2 1 Nov 2026 Nov 2026
DiligenceBench 150 public-equity research tasks on large US companies across sectors. Agents answer open-ended research questions in their own harness, and a model judge scores each answer against a dense task rubric for factual accuracy, analysis and risk awareness. Thoughtful Lab and Paper Instruments Finance Expert In use Inkling 2 Small 58.1% 2 1 Jul 2026 Nov 2026
Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours. OpenAI Building whole projects Expert Fading GPT-6 Cyber 83% 2 1 Apr 2026 Nov 2026
GQA Models answer compositional questions about real-world images, generated from scene graphs. Scored by exact-match accuracy, often on the balanced test-dev split. Stanford University (Hudson and Manning) General visual questions Everyday Fading Janus-Pro-7B 62% 2 1 Feb 2019 Aug 2026
Ko-GDPval Agents complete 170 Korean office tasks from 58 occupations in 11 industries and produce deliverable files. Rubrics grade data, content, structure, quality and format. Upstage Tasks from real jobs Expert In use Solar Pro 4 87.3% 2 1 Jul 2026 Aug 2026
Misbehavior (Thinking Machines) Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Thinking Machines Lab Deception and alignment Expert In use Inkling 2 Mini 9.80% 2 1 Nov 2026 Nov 2026
MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. Google DeepMind and Artificial Analysis Tasks from real jobs Expert In use Gemini 4 Neon 56.5% 2 1 Nov 2026 Nov 2026
Mol-Instructions Models answer molecule-oriented, protein-oriented and biomolecular text instructions, such as reaction prediction, molecule description and protein function questions. Scored per task with task metrics. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Fading Intern-S2-397B 54% 2 1 Jun 2023 Sep 2026
SciReasoner evaluation suite Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks. Shanghai AI Lab (open-sciencelab) Research science Expert Fading Intern-S2-397B 63.3% 2 1 Sep 2025 Sep 2026
SpeechRole Speech role-playing agents talk in character across 98 roles. A model judge scores interaction ability, speech expressiveness and role fidelity from the generated speech. Jiang et al. (arXiv 2508.02013) Speech recognition and translation Everyday Fading Qwen3.5-Omni-Plus 123.5 2 1 Aug 2025 Apr 2026
ForecastBench Models give probabilities for 500 auto-generated questions per round about future events, drawn from prediction markets and time-series datasets. Forecasts are scored with difficulty-adjusted Brier scores once the questions resolve. Forecasting Research Institute Real-world events Expert In use GPT-5.5 0.11 2 Sep 2024 Sep 2026
APEX-Accounting Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria. Mercor with Ramp Simulated companies Expert In use Gemini 4 Argon 71.5% 2 Jul 2026 Nov 2026
PRBench Model answers 1,100 expert-written law and finance questions. LLM judge grades against weighted expert rubrics of 10-30 criteria each. Scale AI Law Expert In use Muse Spark 1.3 61.6% 2 Nov 2025 Nov 2026
Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. Vals AI Composite indices Expert In use GPT-6.2 Astra 68.4% 2 Oct 2026 Nov 2026
Prophet Arena Models and agents forecast open Kalshi prediction-market events and are scored once the events resolve. The live board ranks a difficulty-adjusted Arena score and also reports calibration error and market return. UChicago SIGMA Lab with Kalshi Real-world events Expert In use Claude Opus 5.5 86.2 2 Oct 2025 Oct 2026
FORTE Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes. AGI-Eval (Meituan) Documents and spreadsheets University In use GPT-5.5 77.8% 2 Jun 2026 Nov 2026
LiveClawBench Agents act as OpenClaw personal assistants on 134 multi-step tasks across 10 app domains backed by stateful mock services. A run passes when its reward exceeds 0.8. Mosi-AI (Xiang Long, Yehui Tang and collaborators) General assistants Expert In use Kimi K2.7 Code 76% 2 Mar 2026 Oct 2026
FrontierFinance Agents answer 220 investment research queries from finance experts across six use cases of the investor workflow. Three judge models check each answer against 11,543 source-attributed rubrics. Samaya AI Finance Expert In use Claude Fable 5.1 55.9% 2 Jul 2026 Oct 2026
CL-bench Life Models solve 405 tasks that require learning from messy real-life context such as group chats. A model judge checks rubrics to give a solve rate. Tencent Hunyuan and Fudan University Learning from the context Everyday Fading GPT-5.5 22.2% 2 Apr 2026 Jul 2026
T-Eval Models use tools step by step, and each sub-process is scored separately: instruction following, planning, reasoning, retrieval, understanding, and review. Shanghai AI Laboratory (OpenCompass) Function calling Everyday Dropped GPT-4 Turbo 86.4% 2 Dec 2023 Mar 2025
CMT-Benchmark Models solve 50 research-level condensed matter theory problems written and checked by a panel of expert physicists. Answers are graded automatically. International panel of condensed matter researchers (Pan, Roggeveen, Berg, Carrasquilla, Chowdhury and others) Research science Research frontier Fading GPT-5 30% 2 Oct 2025 Jul 2026
IFStruct Models generate JSON or YAML that must follow a requested schema and phrasing constraints, without constrained decoding. A response passes only if every structural constraint holds. Liquid AI Following precise instructions Everyday In use Gemma 4 31B IT 95.9% 2 Jun 2026 Oct 2026
BigFinanceBench Agents answer workflow-grounded financial research questions with web search, SEC EDGAR, page fetch and Python. A judge grades each answer against an expert rubric and reference. Rogo Technologies Finance Expert In use Claude Opus 4.7 58.8% 2 Jun 2026 Jul 2026
EVA-Bench Voice agents hold simulated spoken conversations to complete enterprise tasks. Scores cover task success and voice-specific failure modes. ServiceNow Speech recognition and translation Everyday In use Grok Voice 3.0 0.74 2 May 2026 Oct 2026
LOCA-Bench Agentic tasks whose environment state is scaled to grow context length while task semantics stay fixed. Success rate versus context length. HKUST NLP Reasoning over long documents University In use GPT-5.2 21.3% 2 Feb 2026 Oct 2026
Dense 200 (bbox) Model detects and boxes every object in 200 densely annotated images (109 categories, about 91 boxes each). Box F1 at IoU thresholds. IDEA Research (Rex-Omni) Detection and captioning Everyday In use Seed1.5-VL 76.9% 2 Oct 2025 Oct 2026
SURDS Models answer spatial questions about objects in nuScenes driving camera images across six single-object and multi-object tasks. Scoring is task accuracy. Xianda Guo et al. (Wuhan University and partners) Spatial reasoning and counting Everyday In use Gemini 2.0 Flash 35.7% 2 Nov 2024 Aug 2026
E-Bench (Tencent) Agents complete Tencent in-house work tasks in the working agent group. Task and scoring details are not published. Tencent Hy team Documents and spreadsheets Expert In use GPT-5.6 Sol 80.6% 2 Jul 2026 Aug 2026
QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. Qwen Team, Alibaba Group Web and app front ends University Fading Qwen3.7-Max 1,608 2 May 2026 Jun 2026
MMSearch-Plus Multimodal browsing agents solve 311 search tasks that need fine-grained visual reasoning, provenance checks and long tool use. Answers are short and checked against a key. Authors of arXiv 2508.21475 Deep research Expert Fading Qwen3.7-Plus 41.4% 2 Aug 2025 Jun 2026
QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. Alibaba Qwen Fixing real repositories Expert In use Claude Fable 5 86.3% 2 Aug 2026
QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. Qwen Team, Alibaba Group Web and app front ends University Fading Claude Opus 4.6 1,617 2 May 2026 Jun 2026
QwenWorldBench Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. Qwen Team, Alibaba Group Interactive world models Fading Qwen3.7-Plus 62.1% 2 May 2026 Jun 2026
FocusBench (Meta) Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes. Meta (Meta Superintelligence Labs) General assistants Everyday In use Muse Wonder Full 54.2% 2 Oct 2026 Oct 2026
Instrument reading (Google DeepMind) Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types. Google DeepMind Embodied reasoning Everyday Fading Gemini Robotics ER 2 65.7% 2 Apr 2026 Jul 2026
MiMo Coding Bench Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE. Xiaomi MiMo Fixing real repositories Expert Fading Claude Opus 4.6 77.1% 2 Apr 2026 Apr 2026
Research-Eval Search-augmented models answer 374 grounded multi-hop questions that need sources across many kinds of websites. An LLM judge checks each answer against a checklist of requirements. Reka AI Deep research Expert Dropped Grok 4 Fast 66% 2 Aug 2025 Nov 2025
X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. xAI Deep research Expert Dropped Grok 4 Fast 58% 2 Sep 2025 Nov 2025
tau-bench Retail Agents help simulated retail customers through tools while following store policies on the original tasks. The score is pass^1 as a percentage. Sierra Customer service Everyday Dropped gpt-oss-120b 67.8% 2 Jun 2024 Aug 2025
Code Interpreter Benchmark Math Models use a Python interpreter to solve math problems. The score is execution accuracy on the Math task set. Qwen Function calling School Dropped Qwen1.5-72B 73.1% 2 Sep 2023 Feb 2024
Decision Index Public v0.3 Decision models return probabilities over labeled options in public tasks covering knowledge, language, retrieval, tools, and arts. A chance-corrected weighted aggregate combines the task scores. Hugging Face Decision Index community Composite indices University In use Granite Switch 4.1 30B 51.3 2 Oct 2026
GDPval v2 (OpenAI) Multistep assignments drawn from real projects in consulting, finance and design, given with the files and messages the team had when the work was assigned. OpenAI has not published the task count or grading. OpenAI Tasks from real jobs Expert In use GPT-6.2 Astra 50.4% 2 Nov 2026 Nov 2026
GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. OpenAI Genomics and life sciences Expert Fading GPT-5.5 33.2% 2 Apr 2026 Apr 2026
Hy-CompanyBench V2 Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published. Tencent Hy team Building whole projects Expert In use Tencent Hy4 preview 62.4% 1 Aug 2026 Aug 2026
Hy-FinmodelBench v2 Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published. Tencent Hy team Finance Expert In use Tencent Hy4 preview 57% 1 Aug 2026 Aug 2026
IndiVibe Models answer 110 source prompts in chat, STEM, math and coding, translated into 22 Indian languages in native and romanized script. A judge model compares answers pairwise. Sarvam AI Chat and instructions in many languages Everyday Fading Sarvam 105B 90% 2 Mar 2026 Mar 2026
KAT Code Bench Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages. Kuaishou (KwaiKAT team) Fixing real repositories University In use KAT-Coder-Pro V2.5 53.1% 2 Jul 2026 Jul 2026
LBM Eval Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task. Toyota Research Institute Simulated robots Everyday Fading Foundry-Qwen3VLA-2.1B 72.4% 2 May 2025 Apr 2026
LifeVal Assistants carry out everyday personal tasks rather than advise on them: finding childcare, booking a car inspection, ordering groceries on a budget, handling government paperwork. Tasks come from time-use surveys and from about 100 people who recorded tasks they wanted an assistant to handle. Score is average reward across tasks. OpenAI Web browsing Everyday In use GPT-6.2 Astra 86.1% 2 Nov 2026 Nov 2026
PatchBench (OpenAI) Models patch security flaws. Automated tests and model-based review check that each patch fixes the flaw, keeps the intended behavior and is ready to merge. OpenAI Secure code and review Expert In use GPT-6.2 Astra 79.7% 2 Nov 2026 Nov 2026
PhysicsFinals Models answer 61 undergraduate physics final exam problems written by physics professors. A professor grades the answers. Google DeepMind Science questions University Dropped Gemini 1.5 Pro 63.9% 2 Feb 2024 May 2024
SecureSWE (OpenAI) Models write functional and secure code for security-sensitive components, such as serving files or handling database queries. OpenAI has not published the task count or grading. OpenAI Secure code and review Expert In use GPT-6.2 Astra 65.7% 2 Nov 2026 Nov 2026
Speech-to-Speech Index Artificial Analysis's combined score for speech-to-speech models. Artificial Analysis Speech recognition and translation Research frontier In use Grok Voice 3.0 86.9 2 Jun 2026 Oct 2026
Success detection (Google DeepMind) Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures. Google DeepMind Embodied reasoning Everyday Fading Gemini Robotics-ER 1.6 90% 2 Sep 2025 Jul 2026
XCOPA Models pick the more plausible cause or effect of a premise from two options, in 11 languages. Scoring is accuracy. Cambridge Language Technology Lab (Ponti et al.) Reasoning in many languages Everyday Dropped PaLM 2-L 94.4% 2 May 2020 Sep 2025
ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. OpenAI Lab protocols Expert In use GPT-6 Cyber 47.2% 1 37 Dec 2024 Nov 2026
VCT-v2 Models answer an updated set of expert virology troubleshooting questions from SecureBio. Scored as refusal-corrected accuracy. SecureBio Lab protocols Expert System cards only Claude Opus 5.5 60.5% 1 Sep 2026 Oct 2026
HealthBench Consensus Models answer HealthBench conversations graded only on criteria that several physicians agreed on. OpenAI Medicine Expert Dropped GPT-5.4 96.6% 1 25 May 2025 Oct 2026
Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. OpenAI Capture-the-flag challenges Expert Fading GPT-5.5 96.3% 1 22 Aug 2024 May 2026
Cybench 40 professional capture-the-flag tasks from four competitions. Agent must recover the flag, with optional subtasks. Stanford Capture-the-flag challenges Expert Fading Muse Spark 65.4% 1 19 Aug 2024 Jul 2026
LABBench2 ProtocolQA2 Models find and fix errors planted in published lab protocols, answering in open response. The ProtocolQA2 subtask of LABBench2. Edison Scientific (formerly FutureHouse) Genomics and life sciences Expert Not in cards Gemini 3 Pro 61.6% Feb 2026 Feb 2026
CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Irregular Capture-the-flag challenges Expert In use Gemini 4 Argon 100% 1 19 Dec 2025 Nov 2026
LABBench2 FigQA2 Models read scientific figures and answer in open response. Served as images or as PDFs. The FigQA2 subtask of LABBench2. Edison Scientific (formerly FutureHouse) Genomics and life sciences Expert Not in cards GPT-5.2 42.6% Feb 2026 Feb 2026
LAB-Bench CloningScenarios Models answer multiple-choice questions about multi-step molecular cloning workflows. CloningScenarios subset of LAB-Bench. FutureHouse Lab protocols Expert Fading Gemini 3 Pro 94.2% 1 15 Jul 2024 Apr 2026
Broken Search Tool Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. OpenAI Deception and alignment Everyday In use GPT-6 Cyber 0.33% 1 14 Aug 2025 Nov 2026
Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. Cohere Labs Knowledge in many languages University Dropped Claude Fable 5.2 96% 1 13 Dec 2024 Nov 2026
SimpleQA Hallucination Rate Models answer 4,326 short fact-seeking questions. This metric is the share of questions answered incorrectly, so declining to answer is not penalized. OpenAI Hallucination rates Dropped ChatGPT agent 7.90% 1 12 Oct 2024 Oct 2025
Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. OpenAI Deception and alignment Expert In use GPT-6 Astra 0% 1 12 Sep 2026 Nov 2026
Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. OpenAI Deception and alignment Expert In use GPT-6 Luna 11.8% 1 12 Sep 2026 Nov 2026
QuAC Models answer free-form questions in information-seeking dialogs about a hidden Wikipedia section, across 100K questions. Scored by word-level F1. Allen Institute for AI, University of Washington, Stanford, and UMass Amherst Reading comprehension Everyday Dropped Llama 3-70B 51.1% 1 12 Aug 2018 Nov 2024
MILU Multi-task Indic Language Understanding: culturally grounded multiple-choice questions in 11 Indic languages. AI4Bharat Knowledge in many languages University Dropped Claude Fable 5.2 94.1% 1 10 Nov 2024 Nov 2026
MakeMeSay A manipulator model tries to make a GPT-4o manipulatee say a codeword without suspicion. Scored as manipulator win rate. OpenAI Dangerous capabilities Everyday Fading GPT-4.5 72% 1 10 Sep 2023 Apr 2026
Apollo Research strategic deception evaluations External red-team environments testing strategic deception, sabotage and sandbagging propensities, with eval-awareness measurement. Apollo Research Deception and alignment Expert In use Claude Sonnet 4.5 0% 1 10 Nov 2026
Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. OpenAI Deception and alignment Expert In use GPT-6 Astra 0% 1 9 Sep 2026 Nov 2026
Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. Anthropic Deception and alignment Expert In use Claude Sonnet 4.5 0% 1 9 Jun 2025 Oct 2026
MLogiQA Models answer LogiQA logical reasoning questions translated into 10 languages, 80 per language. Scored by accuracy. Tongyi Lab, Alibaba Group Reasoning in many languages University Dropped Qwen3-235B-A22B 77.1% 1 8 Nov 2024 Sep 2025
WMDP Biology Models answer 1,273 multiple-choice questions that proxy hazardous biosecurity knowledge. Biology split of WMDP. CAIS + Scale AI et al. Hazardous knowledge Expert Fading Grok 4.20 91% 1 8 Mar 2024 Apr 2026
General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. xAI (SpaceXAI) Refusing harmful requests Fading Grok 4 0% 1 8 Aug 2025 Sep 2026
Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. OpenAI Deception and alignment Expert In use GPT-6.1 Terra 8.47% 1 8 Oct 2026 Nov 2026
Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. OpenAI Capture-the-flag challenges Expert Fading GPT-5.6 Sol 97.1% 1 8 Apr 2026 Aug 2026
AgentHarm 110 malicious agent tasks with synthetic tools. Measures refusal and harmful task completion, plus benign controls. Gray Swan AI + UK AI Security Institute Refusing harmful requests Fading Grok 4 Fast 8.00% 1 7 Oct 2024 Jun 2026
MT-AIME2024 Models solve the 30 AIME 2024 problems translated into 55 languages. A rule-based verifier checks each final answer. Yonsei University, OneLineAI, and KAIST AI Reasoning in many languages Competition Dropped Qwen3-235B-A22B 80.8% 1 7 Feb 2025 May 2025
QuALITY Models answer multiple-choice questions about English passages that average about 5,000 tokens. Scoring is accuracy. New York University Reasoning over long documents School Dropped Claude 3 Opus 90.5% 1 7 Dec 2021 Nov 2024
WMDP Chemistry Models answer 408 multiple-choice questions that proxy hazardous chemical security knowledge. Chemistry split of WMDP. CAIS + Scale AI et al. Hazardous knowledge Expert Fading Grok 4.20 90% 1 7 Mar 2024 Oct 2026
LV-Eval Models answer single-hop and multi-hop questions over contexts at five length levels from 16K to 256K tokens, with confusing facts inserted. Infinigence AI and Tsinghua University (Yuan et al.) Reasoning over long documents Everyday Dropped Qwen2.5-14B-Instruct-1M 49.8% 1 7 Feb 2024 Jan 2025
LongBench-Chat Models follow real user instructions over contexts of 10K to 100K tokens. GPT-4 scores each answer from 1 to 10 against references. Tsinghua University and Zhipu AI (Bai et al., LongAlign) Reasoning over long documents Everyday Dropped Qwen2.5-14B-Instruct-1M 8.76 1 6 Jan 2024 Jan 2025
Perception Test MCVQA Models answer multiple-choice questions about real-world videos filmed to show perceptually interesting situations. Scored by top-1 accuracy. Google DeepMind Video understanding Everyday Dropped Gemini 2.5 Pro 78.4% 1 6 May 2023 Jul 2025
Nexus Models write function calls for queries over real-world APIs, including nested and parallel calls. Scoring is the function calling success rate. Nexusflow Function calling University Dropped Llama 3.1-405B 58.7% 1 6 Dec 2023 Nov 2024
Qasper Models answer 5,049 questions about 1,585 NLP research papers from the full paper text. Answers are scored with token F1. Allen Institute for AI (AI2) Reasoning over long documents University Dropped Phi-3.5-mini 41.9% 1 6 May 2021 Mar 2025
SQuALITY Models write question-focused summaries of public-domain short stories. Outputs are scored against human-written reference summaries with ROUGE. New York University Reasoning over long documents School Dropped Phi-3.5-mini 24.3% 1 6 May 2022 Mar 2025
GSM-Plus Models solve GSM8K problems after perturbations such as changed numbers, added distractors and reversed questions. Li et al. Word problems School Dropped GPT-4 85.6% 1 5 Feb 2024 Nov 2025
C3 Models answer 19,577 multiple-choice questions on Chinese dialogues and texts from Chinese-as-a-second-language exams. Scoring is accuracy. Tencent AI Lab and Cornell University Reading in many languages School Dropped InternVL3.5-241B-A28B 97.8% 1 5 Apr 2019 Feb 2026
BLEnD Models answer 52.6k questions about everyday culture in 16 countries and regions across 13 languages. Scoring is accuracy on short-answer and multiple-choice formats. KAIST Knowledge in many languages Everyday Dropped Apertus 70B 66.3% 1 5 Jun 2024 Sep 2025
HarmBench Models answer harmful behavior requests, including copyright and contextual cases, with and without automated red-team attacks. A classifier judges whether the response complies. Mazeika et al. (Center for AI Safety) Jailbreaks and prompt injection Everyday Fading TML-Interaction-Small 99% 1 5 Feb 2024 May 2026
Refusals + System Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the system message. The metric is the answer rate. xAI (SpaceXAI) Jailbreaks and prompt injection Fading grok-code-fast-1 0% 1 5 Aug 2025 Apr 2026
Refusals + User Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the user message. The metric is the answer rate. xAI (SpaceXAI) Jailbreaks and prompt injection Fading Grok 4 0% 1 5 Aug 2025 Apr 2026
FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. Irregular Finding and exploiting real bugs Research frontier In use GPT-6 Astra 38.1% 1 5 Jun 2026 Nov 2026
MedQA Models answer multiple-choice questions from medical licensing exams, usually the 12,723-question English USMLE set. Scoring is accuracy. MIT Medicine Expert Dropped GPT-4o 86.3% 1 4 Sep 2020 Mar 2025
RepoQA Models find a function in a long code repository from a description of what it does, across Python, C++, Rust, Java and TypeScript. UIUC (EvalPlus team) Finding facts in long inputs University Dropped Claude 3 Opus 90.6% 1 4 Jun 2024 Feb 2026
TurkishMMLU Models answer over 10,000 multiple choice questions that Turkish curriculum experts wrote for nine high-school subjects. Scoring is accuracy. Arda Yüksel et al. Knowledge in many languages School Dropped Claude 3 Opus 79.1% 1 4 Jul 2024 Dec 2024
IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. MBZUAI Knowledge in many languages School Dropped LLaMA-65B 35.8% 1 4 Oct 2023 Dec 2024
HLE Calibration Models give a confidence score with each HLE answer. The benchmark reports RMS calibration error and AUROC between confidence and correctness. Center for AI Safety + Scale AI Hallucination rates Expert Fading Muse Spark 1.1 23.4 1 4 Jan 2025 Oct 2026
RoboSpatial-Home Models answer 350 spatial questions on home scenes for robots, in three types: object context, compatibility of placements and spatial configuration. NVIDIA and The Ohio State University Spatial reasoning and counting Everyday Dropped Qwen3-VL-235B-A22B 73.9% 1 4 Nov 2024 May 2026
AMMLU Models answer MMLU multiple-choice questions across 57 subjects, translated into Arabic with GPT-4. Scored by accuracy. FreedomIntelligence (AceGPT team, CUHK-Shenzhen) Knowledge in many languages University Dropped Qwen2.5-72B 72.4% 1 4 Sep 2023 Dec 2024
CRPE Models answer single-choice questions about objects and their relations in images, covering subject, predicate and object. Each question is asked again with options rotated (CircularEval). OpenGVLab, Shanghai AI Laboratory (All-Seeing Project V2) General visual questions Everyday Dropped Qwen2.5-VL-72B 79.2% 1 4 Feb 2024 Nov 2025
JMMLU Models answer 7,536 four-way multiple-choice questions in Japanese, from translated MMLU subjects and new Japan-specific subjects. Scored by accuracy. Waseda University (nlp-waseda) Knowledge in many languages University Dropped Qwen2.5-72B 80.6% 1 4 Jan 2024 Dec 2024
Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. Qwen Chat and instructions in many languages Everyday Dropped Qwen2.5-72B 87% 1 4 Sep 2024 Dec 2024
SycophancyEval (answer sycophancy) Models answer factual questions where the user suggests a wrong answer. Score is the relative accuracy drop against neutral prompts. Anthropic (Sharma et al.) Deception and alignment Fading Grok 4.20 4.00% 1 4 Oct 2023 Apr 2026
WANDR 500 wide-search tasks: find tens to hundreds of entities and facts, then return structured records with citations. Perplexity Deep research Everyday In use Claude Fable 5.2 82.1% 1 4 Jul 2026 Nov 2026
xAI sycophancy eval Factual questions paired with misleading user context. Rate at which the model abandons the correct answer. xAI (SpaceXAI) Wellbeing and sycophancy Fading Grok 4.5 0.01% 1 4 Sep 2026
XQuAD Models answer 1,190 SQuAD questions over 240 paragraphs, professionally translated so the set is parallel across 11 languages. Scored by F1 and exact match. DeepMind Reading in many languages Everyday Dropped Gemma 3 27B 76.8 1 4 Oct 2019 Mar 2025
WildJailbreak Models face adversarial jailbreak prompts built from in-the-wild tactics, plus benign prompts that look harmful. Evaluations score refusals of harmful requests and compliance with benign ones. Allen Institute for AI and University of Washington (WildTeaming) Jailbreaks and prompt injection Everyday Fading Falcon H1R 7B 98.5% 1 4 Jun 2024 Jan 2026
EEBench 13 electrical-engineering circuit and PCB design tasks graded by deterministic build checks, simulation and BOM constraints. No LLM judge. atopile Electronics Expert In use Gemini 4 Carbon 83.9% 1 3 Jul 2026 Nov 2026
BixBench3 Agents rebuild the full computational analysis of 20 published biology papers from raw data on a cloud VM. Output tables are graded against the papers' own artifacts. Edison Scientific Research science Research frontier Not in cards Gemini 4 Argon 52.3 Aug 2026 Sep 2026
Point-Bench Models point to the image location a language request describes, in about 1,000 tasks across five reasoning categories. A point inside the target mask counts as correct. Cheng et al. (PointArena team) Detection and captioning Everyday Fading gemini-robotics-er-1.5-preview 67.1% 1 3 May 2025 Mar 2026
HELMET Models handle long-context tasks in seven groups including recall, RAG, re-ranking, many-shot learning, long QA and summarization, at set input lengths. Princeton NLP (Yen et al.) Reasoning over long documents University Dropped GPT-4o mini 61.4% 1 3 Oct 2024 Jul 2025
MindCube Models answer 21,154 multiple choice questions about scenes seen from a few views. Questions test positions, orientations, and what-if movements. MLL Lab, Northwestern University, with Stanford, NYU, and University of Washington Spatial reasoning and counting Everyday Fading DeepSeek-VL2-Small 47.6% 1 3 Jun 2025 Oct 2026
EMMA Models solve math, physics, chemistry and coding problems that need reasoning over text and images together, not each alone. Answers are checked against a key. Hao et al. (EMMA team) Visual math and diagrams University Fading Gemini 2.0 Flash Thinking 42.5% 1 3 Jan 2025 Jun 2026
Open ASR Leaderboard Speech recognition models transcribe the test sets of several English and multilingual corpora. Scored by average word error rate, with speed reported next to it. Hugging Face (hf-audio) and collaborators Speech recognition and translation Everyday Fading Cohere Transcribe 5.42% 1 3 Sep 2023 Oct 2026
PubMedQA Models answer biomedical research questions with yes, no, or maybe, given a PubMed abstract without its conclusion. Scored by accuracy. University of Pittsburgh and Carnegie Mellon University Medicine University Dropped Claude 3 Sonnet 78.3% 1 3 Sep 2019 Apr 2024
InfiniteBench/En.MC Models answer 229 multiple choice questions about long fake books with altered entity names, averaging about 184K tokens. Scoring is accuracy. OpenBMB (Zhang et al.) Reasoning over long documents Everyday Dropped Llama 3.1-405B 83.4% 1 3 Nov 2023 Nov 2024
MVPBench Models answer video questions on spatial, temporal and intuitive physics. Each question comes in a pair of near-identical videos with opposite answers, so text or single-frame shortcuts fail. Meta FAIR Physical prediction Everyday Dropped gemini-robotics-er-1.5-preview 72.5% 1 3 Jun 2025 May 2026
StrongREJECT v2 (Meta) Models face a cascade of jailbreak attacks on disallowed behaviors adapted from StrongREJECT plus internal ones. Reports attack success rate. Meta Jailbreaks and prompt injection Expert In use Muse Spark 1.1 0.50% 1 3 Oct 2026
WorldSense Models answer synthetic questions that test consistent reasoning about simple arrangements of entities described in text. Meta FAIR (Benchekroun et al.) Language reasoning Everyday Fading Seed2.0 Lite (0428) 67.3% 1 3 Nov 2023 Apr 2026
ZeroSCROLLS/QuALITY Models answer QuALITY multiple-choice questions about long stories zero-shot, as one task of the ZeroSCROLLS suite. Scoring is accuracy. Tel Aviv University Reasoning over long documents Everyday Dropped Llama 3.1-405B 95.2% 1 3 May 2023 Nov 2024
APPS Models write Python programs for 10,000 coding problems that range from introductory to competition level. Generated code runs against test cases. UC Berkeley Competitive programming University Dropped Claude 3 Opus 70.2% 1 3 May 2021 Mar 2024
COCO Captions Models write a caption for each image, and outputs are compared with five human captions per image using BLEU, METEOR, ROUGE, and CIDEr. Microsoft COCO Detection and captioning Everyday Dropped Gemma 3 27B 116.0 1 3 Apr 2015 Mar 2025
HuggingFace Agent Benchmark Agents write Python code that calls Hugging Face Transformers tools to solve short tasks. Scores cover tool selection, tools used, and code correctness. Hugging Face Function calling School Dropped Qwen-14B 93.5% 1 3 May 2023 Sep 2023
OK-VQA Models answer over 14,000 visual questions that need outside knowledge beyond the image. Scoring is VQA accuracy against human answers. Allen Institute for AI and Carnegie Mellon University General visual questions Everyday Dropped Gemma 3 27B 60.2% 1 3 May 2019 Mar 2025
SpatialSense VQA Models judge whether a stated spatial relation between two objects holds in an image. The relations were crowdsourced to defeat simple cues. Princeton Vision and Learning Lab (Kaiyu Yang, Olga Russakovsky, Jia Deng) Spatial reasoning and counting Everyday Dropped Gemma 3 12B 60% 1 3 Aug 2019 Mar 2025
MATH Level 5 Models solve the 1,324 hardest MATH test problems, drawn from competitions such as AMC and AIME. Scored by final answer match. Dan Hendrycks et al. (UC Berkeley), subset run by Epoch AI Competition problems with answers Competition Dropped GPT-5 98.1% 1 2 Mar 2021 Nov 2025
RewardBench 2 Reward models must score one correct answer above three wrong ones on 1,865 new human prompts. Subsets cover factuality, precise instruction following, math, safety, focus and ties, and the score averages them. Allen Institute for AI (Ai2) Overall chat quality University Not in cards Skywork-Reward-V2-Llama-3.1-8B 84.1% Jun 2025 Jun 2025
AIR-Bench 2024 Models respond to 5,694 harmful prompts across 314 risk categories drawn from 8 government regulations and 16 company policies. Category-specific LLM judges score refusals. Stanford CRFM + Virtue AI et al. Refusing harmful requests Everyday Dropped Claude Haiku 4.5 93.2% 1 2 Jul 2024 Jun 2026
APEX-SWE Software-engineering productivity tasks spanning integration work and observability, such as diagnosing failures from telemetry. Mercor Fixing real repositories Expert In use Claude Opus 5.5 67.6% 1 2 Jan 2026 Nov 2026
GeoBench Models geolocate GeoGuessr photos from community maps by giving coordinates and a country. Scored by country accuracy and a distance-based score. ccmdi General visual questions Everyday Dropped Claude Fable 5.2 95% 1 2 Mar 2025 Nov 2026
SRE-Bench Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. Columbia DAPLab + Vals AI Capture-the-flag challenges Expert In use GPT-6 Astra 56.9% 1 2 Aug 2026 Nov 2026
VPCT Models view an image of a ball, ramps, and three buckets and predict which bucket the ball falls into. Scored by accuracy on 100 problems. Chase Brower Physical prediction Everyday Fading Gemini 3 Pro 91% 1 2 Jan 2025 Mar 2026
MMBench-Video Video models answer free-form questions about long, multi-shot YouTube videos. A GPT-4 judge scores the answers. Shanghai AI Laboratory (OpenCompass) Long videos Everyday Dropped GPT-4o 2.15 1 2 Jun 2024 Aug 2025
MIA-Bench Models answer 400 image prompts that stack several strict instructions on format, length and content. An LLM judge scores how well each answer follows every instruction. Apple General visual questions Everyday Dropped GPT-4o 91% 1 2 Jul 2024 Jan 2026
PixMo-Count Models count objects in natural images and give the exact number. Scored by accuracy on the test split. Allen Institute for AI (Molmo and PixMo) Detection and captioning Everyday Fading Molmo-72B 85.2% 1 2 Sep 2024 Jun 2026
ScienceQA Models answer about 21,000 multiple-choice science questions from school curricula, many with images. Scored by accuracy. UCLA and Allen Institute for AI General visual questions School Dropped Phi-4-multimodal 97.5% 1 2 Sep 2022 Mar 2025
SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. Shanghai Jiao Tong University, Shanghai AI Laboratory and others (InternSVG team) Web and app front ends Everyday Fading Claude Sonnet 4 0.92 1 2 Aug 2025 Mar 2026
MMT-Bench Vision-language models answer 31,325 multiple-choice questions across many multimodal tasks such as recognition, localization, reasoning and planning. Shanghai AI Laboratory General visual questions Expert In use Qwen-VL-Max 62.4% 1 2 Apr 2024 Oct 2026
CrossVid Models reason across several videos at once in 10 tasks, with 9,015 questions on 5,331 videos in single-choice, multiple-choice and open-ended formats. Li et al. (CrossVid team) Video understanding Everyday Fading GPT-4.1 45.2% 1 2 Nov 2025 Jun 2026
JudgeBench Models act as judges and pick the correct response from hard pairs on knowledge, reasoning, math and coding. The label comes from objective correctness, not human preference. UC Berkeley Overall chat quality Expert Dropped o3-mini 80.9% 1 2 Oct 2024 May 2025
MME-CC Models answer 1,173 expert-annotated vision-grounded questions in 11 task types grouped as spatial, geometric and visual knowledge reasoning. ByteDance Seed and collaborators (Zhang et al.) Spatial reasoning and counting Everyday Fading GPT-5 40.3% 1 2 Nov 2025 Mar 2026
WildVision Vision-language models answer 500 real user image prompts from the WildVision Arena. A GPT-4 judge compares each answer with a fixed baseline model. Allen Institute for AI and UCSB General visual questions Everyday Dropped Aria 74.1 1 2 Jun 2024 Nov 2025
IHEval Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction. Zhang et al. (Notre Dame + Amazon) Jailbreaks and prompt injection Everyday In use GPT-4o 70% 1 2 Feb 2025 Oct 2026
Mantis-Eval Models answer questions that need reasoning across several interleaved images. Scored by accuracy. Dongfu Jiang, Xuan He et al. (TIGER-Lab, University of Waterloo) General visual questions Everyday Dropped Llama-3-VILA1.5-8B 51.1% 1 2 May 2024 Oct 2025
Parametric CAD Bench v3 Agents create and edit parametric FreeCAD models from text and engineering drawings. Programmatic checks award continuous task rewards for geometry and specification consistency. gNucleus AI CAD models Expert Not in cards Claude Opus 5.5 61.0 Sep 2026 Sep 2026
ViewSpatial-Bench Models answer 5,712 questions that locate objects from the camera view and from the view of people in the scene. Five task types test egocentric and allocentric reasoning. Dingming Li, Yueting Zhuang et al. Spatial reasoning and counting Everyday Fading InternVL2.5-8B 43.2% 1 2 May 2025 Oct 2026
CIMemories Models use persistent memory across tasks and must withhold attributes when context requires. Reports worst-case disclosure over five samples. Meta (Mireshghallah et al.) Refusing harmful requests Everyday In use GPT-4o 14.8% 1 2 Nov 2025 Oct 2026
GAOKAO-Bench Models answer Chinese national college entrance exam questions from 2010 to 2022 across nine subjects, with objective and subjective questions. Fudan University (OpenLMLab) Exam questions School Dropped GPT-4 72.2% 1 2 May 2023 Aug 2025
MSEB Audio representations from any multimodal system are scored on eight core tasks, including transcription, classification, retrieval, reasoning, segmentation, clustering, reranking and reconstruction. Google Research Audio understanding and voice Everyday Fading Gemini 3 Flash 39.2 1 2 Feb 2026 Oct 2026
NaturalCodeBench Models solve 402 Python and Java problems taken from real user queries to online coding services in 6 domains. Semi-automatically built tests check each solution. Tsinghua University and Zhipu AI (THUDM) Function-level problems University Dropped GPT-4 Turbo (Jan 2024) 52.5% 1 2 May 2024 Aug 2025
WorldTravel Agents plan trips for 150 real-world scenarios across 5 cities with more than 15 tightly coupled time and logic constraints each. A web environment supplies multimodal travel data. ByteDance Seed and collaborators (Wang et al.) General assistants Everyday Fading GPT-5.2 32.7% 1 2 Dec 2025 Mar 2026
HealthBench Hard Hallucinations Models respond to difficult health conversations that 2 or more physicians validated. The score is the hallucination rate, so lower is better. OpenAI Hallucination rates Dropped GPT-5 1.60% 1 2 Aug 2025 Aug 2025
MM-ClawBench Agents complete common OpenClaw tasks such as study planning, office documents, scheduled research, and code maintenance. Scoring is accuracy. MiniMax General assistants Everyday Fading Claude Opus 4.6 75.4% 1 2 Mar 2026 May 2026
WMDP Cyber Models answer multiple-choice questions on offensive cybersecurity knowledge. The cybersecurity split of the Weapons of Mass Destruction Proxy benchmark. CAIS + Scale AI et al. Finding and exploiting real bugs Expert Fading Grok 4.20 91% 1 2 Mar 2024 Apr 2026
BetterChartQA Models answer annotator-written questions about real charts from news, government reports, and papers across 9 capability buckets. Scored by accuracy. Gemini Team, Google Documents, charts and text in images School Dropped Gemini 2.5 Pro 72.4% 1 2 Jun 2024 Jul 2025
ContPhy Models infer physical properties such as mass and density of fluids and soft bodies from videos and predict their dynamics. Zheng et al. (ContPhy team) Physical prediction Everyday Fading Seed2.0 Pro 67.4% 1 2 Feb 2024 Jun 2026
AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. ByteDance Seed Scientific code Expert Fading Seed2.0 Pro 47.7% 1 2 Dec 2025 Mar 2026
BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. ByteDance Seed Genomics and life sciences Competition Fading Seed2.0 Pro 53.5% 1 2 Dec 2025 Mar 2026
Complex Workflow (ByteDance Seed in-house) Models carry out step-by-step tasks that follow a given standard operating procedure, such as multi-turn after-sales service in support, sales and healthcare. ByteDance Seed Following precise instructions Everyday Fading Seed2.0 Pro 64.7% 1 2 Dec 2025 Mar 2026
Customer Support Q&A (ByteDance Seed in-house) Models answer user questions from noisy enterprise knowledge base results, such as policy questions in e-commerce, government services and telecom. ByteDance Seed Customer service Everyday Fading Seed2.0 Pro 72.4% 1 2 Dec 2025 Mar 2026
Education (ByteDance Seed in-house) Models solve, grade, explain and write K to 12 problems across core subjects in teaching scenarios. ByteDance Seed Other professions School Fading Seed2.0 Pro 62.8% 1 2 Dec 2025 Mar 2026
Information Extraction (ByteDance Seed in-house) Models extract words, sentences or fields from meeting records, legal texts, contracts, product descriptions and knowledge bases. ByteDance Seed Documents and spreadsheets Everyday Fading Seed2.0 Pro 52% 1 2 Dec 2025 Mar 2026
Inter-GPS (Geometry3K) Models solve high school geometry problems that combine a diagram and text from the Geometry3K test set. Scored by accuracy. UCLA and Sun Yat-sen University (Lu et al.) Visual math and diagrams School Dropped Phi-4-multimodal 48.6% 1 2 May 2021 Mar 2025
Internal adversarially-designed factuality evaluations Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. OpenAI Hallucination rates Everyday In use GPT-6 Luna 7.60% 1 2 Mar 2023 Sep 2026
NV Chat Benchmark Models answer chat prompts from an internal NVIDIA set. The Nemotron 3.5 Super VL card labels the score an LMArena v3 proxy. NVIDIA Overall chat quality Everyday In use Nemotron 3.5 Super VL 52.7 1 2 Oct 2026 Oct 2026
Passkey Retrieval Models find a random passkey hidden at a random position in a long filler prompt. Scored by retrieval accuracy. EPFL Finding facts in long inputs Everyday Dropped Mixtral 8x7B 100% 1 2 May 2023 Jan 2025
RISE Agents answer real questions from human experts that need multi-step search and interaction with information-dense web pages. MiniMax has not published the scoring method. MiniMax Deep research Expert Fading MiniMax-M2.7 64.3% 1 2 Feb 2026 May 2026
MiniMax TTS Multilingual Test Set Text-to-speech models clone two Common Voice speakers per language and read 100 test sentences in each of 24 languages. Scored by word error rate and speaker similarity. MiniMax Speech recognition and translation Everyday In use Ming-3.0-tts-0.5B 0.36% 1 2 May 2025 Sep 2026
Multilingual LibriSpeech Models transcribe read audiobook speech in eight languages from LibriVox. Scored by word error rate. Meta AI (Pratap et al.) Speech recognition and translation Everyday Fading Nova 2 Omni Preview 4.70 1 2 Dec 2020 Dec 2025
Excel Modeling Benchmark v1.1 Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna. Vals AI Finance Expert In use Claude Fable 5.2 77.4% 1 1 Oct 2026 Nov 2026
CorpFin v2 Models answer expert-written questions about credit agreements that often exceed 200 pages. An LLM judge grades each answer for accuracy. Vals AI Finance Expert In use Claude Opus 5 73.2% 1 1 Aug 2026
SAGE v1.1 (Vals) Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading. Vals AI Other University In use Claude Opus 5.5 91.3% 1 1 Oct 2026 Nov 2026
Open Arabic LLM Leaderboard v2 Models are scored on native Arabic tasks (AlGhafa native subsets, ArabicMMLU, Exams, MadinahQA, AraTrust, ALRAGE, ArbMMLU-HT) and ranked by the average. Open Arabic LLM Leaderboard team (2A2I and community) with Hugging Face Knowledge in many languages School Dropped Qwen2.5-72B 72.4% 1 1 Feb 2025 Nov 2025
BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. BVB authors (Tang et al.) Video understanding Expert In use Claude Fable 5.2 76.4 1 1 Sep 2026 Nov 2026
KINA Models answer 899 knowledge questions across 261 fine-grained disciplines, each with ten lettered options. Scored by accuracy against the key. 2077AI (Jin et al.) Expert-written questions Expert In use Gemini 3.1 Pro 53.2% 1 1 Jun 2026 Jun 2026
PHYSICS-Verified Corrected PHYSICS test problems with answer-level accuracy. The complete verified test set. Yale University (Kaiyue Feng, Arman Cohan et al.) Science questions Expert Not in cards GPT-6 Astra 91.1% Sep 2026 Sep 2026
ArabicMMLU Models answer 14,575 multiple-choice questions in Modern Standard Arabic across 40 tasks, taken from school exams in North Africa, the Levant and the Gulf. MBZUAI and collaborators (Koto et al.) Knowledge in many languages School Dropped Qwen2.5-72B 71.4% 1 1 Feb 2024 Nov 2025
AraTrust Models answer 522 human-written Arabic multiple-choice questions on truthfulness, ethics, safety, physical and mental health, unfairness, illegal activities, privacy and offensive language. Alghamdi et al. Deception and alignment Everyday Dropped Qwen2-72B 89.2% 1 1 Mar 2024 Nov 2025
AgentIF (2026-10 setting) Models follow the same AgentIF instructions in thinking and non-thinking modes with vendor decoding settings. A gpt-5.6-luna judge scores 3 runs per model. Tsinghua University KEG (THU-KEG) Following precise instructions Everyday Not in cards DeepSeek-V4-Pro 67.4% Oct 2026 Oct 2026
MixEval Models answer real user queries matched to similar questions from existing benchmarks. Ground-truth answers make it a cheap proxy for Chatbot Arena rankings. National University of Singapore (Ni et al.) Overall chat quality Everyday Dropped Claude 3.5 Sonnet 89.9% 1 1 Jun 2024 Jul 2025
SpaCE-10 Models answer more than 5,000 questions about indoor 3D scenes that test 10 atomic and 8 compositional spatial abilities. SpaCE-10 team (Ziyang Gong et al.) Spatial reasoning and counting Everyday Dropped InternVL3.5-241B-A28B 55% 1 1 Jun 2025 Sep 2025
KMMLU-Redux Models answer multiple-choice questions from Korean National Technical Qualification exams. The set is rebuilt from KMMLU with erroneous questions removed. LG AI Research and collaborators Exam questions University Dropped o1 81.1% 1 1 Jul 2025 Jul 2025
MCP-Universe Agents complete tasks through 11 real MCP servers across six domains, such as navigation, repositories, finance, and browsing. Scored by execution-based success rate. Salesforce AI Research Apps and MCP servers University Fading Gemini 3 Pro 44.6% 1 1 Aug 2025 Dec 2025
C3-Bench Agents complete multi-task tool use dialogues built to stress tool dependencies, long context dependencies and frequent switches between policy types. Tencent Hunyuan Function calling University Dropped Claude Sonnet 4 12.5% 1 1 May 2025 Aug 2025
Encyclo-K Models judge questions composed at test time from knowledge statements extracted from authoritative textbooks. Random composition is meant to resist contamination and test several facts at once. ByteDance Seed and collaborators (Liang et al.) Expert-written questions University Fading GPT-5.1 62.1% 1 1 Dec 2025 Feb 2026
StructEval Models generate or convert structured outputs in 18 formats, such as JSON, TOML and HTML. StructEval-T covers text-only formats and StructEval-V renderable ones. TIGER-AI-Lab (Yang, Jiang et al.) Data science and SQL Everyday In use Gemini 3.1 Flash-Lite 81.2% 1 1 May 2025 Oct 2026
ChemBench Models answer more than 2,700 chemistry questions on knowledge and reasoning. Results are compared with a group of human chemists. Lamalab, Friedrich Schiller University Jena Science questions Expert Dropped Claude Opus 4.7 70.9% 1 1 Apr 2024 May 2026
OlympicArena Models solve 11,163 bilingual problems from 62 international olympiad competitions in seven fields, some with images. Answers are checked against keys. Shanghai Jiao Tong University GAIR Lab Science olympiads Competition Dropped GPT-4o 40.5% 1 1 Jun 2024 Jun 2025
HumanEval-V Models read a diagram and a function signature, then write Python code that must pass test cases. Tasks span six types of visual reasoning. Fengji Zhang, Jacky Keung et al. Function-level problems University Fading o1 40.6% 1 1 Oct 2024 Jan 2026
MHPP Models write Python functions for 210 human-curated problems that need careful reading of specifications and multi-step reasoning. Unit tests check each solution. SparksofAGI (Dai et al.) Function-level problems University Dropped o1-preview 68.1% 1 1 May 2024 Jun 2025
SGP-Bench Models read symbolic graphics programs such as SVG and CAD code and answer questions about what the rendered image would show, without seeing the image. Max Planck Institute for Intelligent Systems and others (Zeju Qiu, Weiyang Liu et al.) General visual questions University Dropped Claude 3.5 Sonnet 67.4% 1 1 Aug 2024 Aug 2025
Math Kangaroo 2025 (MathArena) Vision language models answer 168 multiple-choice problems from the 2025 Math Kangaroo, given as screenshots across six grade groups. Many problems need reading a diagram. MathArena (ETH Zurich SRI Lab and INSAIT) Visual math and diagrams School Fading GPT-5.5 94.9% 1 1 Oct 2025 Jul 2026
Remote Labor Index Agents complete real paid remote freelance projects from many sectors. Reviewers judge whether each deliverable would be accepted as commissioned work, giving an automation rate. Center for AI Safety and Scale AI Tasks from real jobs Expert In use Claude Fable 5.2 41.3% 1 1 Oct 2025 Nov 2026
SWE-EVO Coding agents implement release-note changes in seven mature Python projects, editing about 21 files per task. Large test suites check each of the 48 tasks. Le, Thai et al. Fixing real repositories Expert Fading GLM-4.7 39.6% 1 1 Dec 2025 Mar 2026
KRETA Models read and reason over Korean text in images across diverse visual contexts. It is a text-rich visual question answering test for Korean. KRETA authors (tabtoyou et al.) Documents, charts and text in images Everyday Fading Gemini 2.0 Flash 85.4% 1 1 Aug 2025 Apr 2026
MathCanvas-Bench Models solve 3,000 math problems that call for interleaved visual and textual solutions, such as drawing auxiliary lines in geometry. Shi et al. (MathCanvas team) Visual math and diagrams School Fading Seed1.6-Thinking 44.1% 1 1 Oct 2025 Feb 2026
MAVERIX Models answer 2,556 questions about 700 videos that need both the video and its audio. Questions come in multiple-choice and open-ended forms. Carnegie Mellon University (CUBE lab) and collaborators Video understanding Everyday Fading Claude 3.5 Sonnet 64.1% 1 1 Mar 2025 Dec 2025
MME-VideoOCR Models answer 2,000 questions about text that appears in 1,464 videos, across 25 tasks from recognition to reasoning. MME team (Shi et al., arXiv 2505.21333) Video understanding Everyday Fading Gemini 2.5 Pro 73.7% 1 1 May 2025 Apr 2026
AgentWorldBench Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions. Qwen Team, Alibaba Group Interactive world models University In use GPT-5.4 58.3 1 1 Jun 2026 Jun 2026
KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. HAE-RAE General visual questions University Fading Qwen3-VL-235B-A22B 42% 1 1 Apr 2026 Apr 2026
MORSE-500 Models answer questions embedded in 500 scripted video clips across six reasoning categories, including abstract, physical, planning, spatial and temporal reasoning. Cai et al. (MORSE-500 team) Video understanding Everyday Fading o3 23.6% 1 1 Jun 2025 Apr 2026
Sudoku-Bench Models solve Sudoku variants with unusual rule sets that need creative, multi-step deduction. Scored by the share of puzzles solved. Sakana AI Abstract puzzles and logic Competition Dropped o3-mini 14% 1 1 May 2025 Sep 2025
K-Viscuit Models answer multiple-choice visual questions about Korean culture. Questions were written through human and VLM collaboration. Seoul National University and collaborators (ddehun et al.) General visual questions Everyday Fading GPT-4o 89.5% 1 1 Jun 2024 Apr 2026
VideoEval-Pro Video models give open-ended short answers to questions that need the whole long video, which removes the guessing that multiple choice allows. TIGER-Lab and collaborators (Ma et al.) Long videos Everyday Fading GPT-4.1 40.8% 1 1 May 2025 Apr 2026
AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. Cao et al. (arXiv 2605.27705) Video generation Expert In use GPT-5.6 Sol 38.4% 1 1 May 2026 Sep 2026
LongMemEval-V2 Memory systems help agents learn the affordances, dynamics and failure modes of customized web environments. Questions check whether that experience was internalized. UCLA NLP Learning from the context Expert Not in cards Qwen3.5-9B 1.30% May 2026 Feb 2026
OIBench Models write programs for 250 original olympiad in informatics problems. Hidden tests check correctness, and time and space completion curves compare efficiency with reference solutions. AGI-Eval (Meituan) Competitive programming Competition Fading o4-mini 36.4% 1 1 Jun 2025 Jan 2026
ProteinLMBench Models answer 944 hand-checked multiple choice questions on protein sequence, structure and function. Johns Hopkins University, University of Cambridge, Shanghai AI Laboratory and others (Yiqing Shen et al.) Genomics and life sciences Expert Dropped GPT-4 Turbo 57.9% 1 1 Jun 2024 Aug 2025
TemporalBench Video models answer about 10K questions on fine-grained temporal dynamics, built from about 2K detailed human annotations of video clips. Cai et al. (TemporalBench team) Video understanding Everyday Dropped LLaVA-OneVision 72B 28.1% 1 1 Oct 2024 May 2025
UniClawBench Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill. The University of Hong Kong (HKU MMLab) General assistants Everyday In use Claude Opus 4.8 47.5% 1 1 Jul 2026 Sep 2026
When2Call Models decide when to call a tool, when to ask a follow-up question and when to say the tools cannot answer. Scored on multiple-choice and LLM-judged test sets. NVIDIA Function calling Everyday In use GPT-4 Turbo 64.6% 1 1 Apr 2025 Oct 2026
LongAudioSpan Models answer 3,240 questions about 360 English and Chinese recordings from 10 minutes to over 2 hours. Each question is scored as multiple choice and as open-ended. Huang et al. (arXiv 2608.26431, with Qwen authors) Audio understanding and voice Everyday In use GPT-5.4 80.2% 1 1 Aug 2026 Sep 2026
TimeLens-Bench Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU. Nanjing University, ARC Lab (Tencent PCG) and Shanghai AI Lab Video understanding Everyday Fading Gemini 2.5 Pro 62.9% 1 1 Dec 2025 Sep 2026
VCR-Bench Video models answer 1,034 questions on 859 videos, and graders score both the final answer and each tagged chain-of-thought step against human rationales. Qi et al. (VCR-Bench team) Video understanding Everyday Fading o1 56.7% 1 1 Apr 2025 Mar 2026
DREAM-1K Video models write fine-grained descriptions of 1,000 videos from varied sources. An automatic method scores how many described events match the reference. ByteDance Research (Tarsier team) Video captioning Everyday Dropped GPT-4V 34.4% 1 1 Jul 2024 May 2025
GaRAGe Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages. Amazon (amazon-science) Faithful to sources Everyday In use Amazon Nova Pro 60.7% 1 1 Jun 2025 Oct 2026
CSAT Math 2025 Models solve the mathematics sections of South Korea's 2025 College Scholastic Ability Test in Korean. The score averages the three elective subjects: calculus, statistics and geometry. Korea Institute for Curriculum and Evaluation (exam), LG AI Research (evaluation) Competition problems with answers School Dropped EXAONE Deep 32B 94.5% 1 1 Nov 2024 Mar 2025
Multi-SWE-bench mini Agents fix real GitHub issues in 8 programming languages, a 400-instance subset of Multi-SWE-bench made for cheaper runs. Scored by tests passing. ByteDance Seed Fixing real repositories Expert Dropped Qwen3-Coder-480B-A35B 25.8% 1 1 Apr 2025 Jul 2025
ProcBench Models follow explicit step-by-step procedures to answer questions, which isolates multi-step inference from knowledge and search. Fujisawa et al. Abstract puzzles and logic Everyday Fading o1-preview 69.8% 1 1 Oct 2024 Feb 2026
LVOmniBench Models answer questions about long videos with rich audio, often tens of minutes long. The questions need both the audio and the video track. Tao et al. (arXiv 2603.19217) Long videos Everyday In use Gemini 3 Pro 65.8% 1 1 Mar 2026 Sep 2026
ODV-Bench Streaming video models answer real-time questions about autonomous driving footage. Zeng et al. (StreamForest team) Video understanding Everyday Fading Qwen2.5-VL-7B 55.6% 1 1 Sep 2025 Apr 2026
OmniCap-IF Models write captions for videos with audio while following detailed user constraints. Captions are scored on format and content compliance. Nanjing University (NJU-LINK Lab) and others Video captioning Everyday In use Gemini 3.1 Pro 80.7% 1 1 Jun 2026 Sep 2026
SecCodeBench Models write or patch functions in project scaffolds drawn from real Alibaba vulnerabilities. Code must pass functional tests, then proof-of-concept security tests. Alibaba Group (with Tsinghua, Zhejiang, Fudan, and Peking universities) Secure code and review University Fading Qwen3.7-Max 71.1% 1 1 Jul 2025 Jun 2026
UNO-Bench Models answer 1,250 human-written omni-modal questions and 2,480 uni-modal questions across 44 task types and 5 modality mixes. Multiple choice and multi-step open questions are scored, the open ones by a scoring model. Meituan Audio understanding and voice Everyday Dropped Gemini 3 Pro 76.5% 1 1 Oct 2025 Dec 2025
WebExploitBench Agents find and exploit 110 vulnerabilities in 15 real web applications, each in its own container. A verifier checks that the agent's proof of concept hits the same endpoint as the reference exploit. AgentCyberRange Finding and exploiting real bugs Expert In use GPT-5.5 36.4% 1 1 Jun 2026 Nov 2026
DeR2 Bench Models reason over supplied scientific documents in a sandbox that separates evidence access from reasoning. Four regimes range from instructions only to relevant documents plus distractors. ByteDance Seed and collaborators (Ying et al.) Deep research Expert Fading GPT-5.2 83.8% 1 1 Jan 2026 Feb 2026
Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. Qwen Team, Alibaba Group Chat and instructions in many languages Everyday Dropped Claude 3 Opus 4.15 1 1 Jun 2024 Jul 2024
StreamBench (StreamChat) Video models answer questions on streaming videos across media types, including multi-turn interaction and reasoning tasks. Xiong et al. (StreamChat team) Video understanding Everyday Dropped GPT-4o mini 59.1% 1 1 Jan 2025 May 2025
VBench-2.0 Video generation models are scored on intrinsic faithfulness, such as human fidelity, controllability, creativity, physics and commonsense, beyond pixel quality and basic prompt match. NTU S-Lab and Shanghai AI Laboratory (Vchitect) Video generation Everyday Not in cards Veo 3 66.7% Mar 2025 Sep 2025
Agent Startup Bench Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews. ByteDance Seed Running a business Expert In use Seed2.1 Pro 68.8% 1 1 Jun 2026 Jun 2026
HealthBench-Hallu Models answer HealthBench-style medical prompts. A pipeline splits each answer into medical claims and checks them against medical evidence. The score is a weighted hallucination rate. Baichuan AI Hallucination rates Expert Fading Baichuan-M3-235B 3.50% 1 1 Jan 2026 Feb 2026
MusTBench Models answer five kinds of music questions that need the correct time region of the audio, such as instrument entries. Music experts validated the items. Kwon et al. (arXiv 2605.29300) Audio understanding and voice Expert In use Qwen3.8-Omni-Flash 50.6% 1 1 May 2026 Sep 2026
RelyToolBench Agents answer queries with simulated APIs built on the StableToolBench server. Scoring targets tool hallucinations, such as calling wrong or missing tools. X-LANCE (Shanghai Jiao Tong University) Function calling Everyday In use Nemotron 3.5 Bolt E4B 69.3% 1 1 Dec 2024 Oct 2026
SeedClawBench Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables. ByteDance Seed General assistants Everyday In use Seed2.1 Pro 66.6% 1 1 Jun 2026 Jun 2026
AndroidDaily Agents complete 350 daily tasks in 94 real closed-source Android apps. A process-aware reviewer model checks each trajectory against written guidelines. StepFun Phones Everyday Fading Step 3.7 Flash 61.9% 1 1 Nov 2025 May 2026
CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass. Zhipu AI (Z.ai) Fixing real repositories Expert Fading GLM-5 25.8% 1 1 Feb 2026 Apr 2026
CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file. Zhipu AI (Z.ai) Fixing real repositories Expert Fading GLM-5V-Turbo 72.2% 1 1 Feb 2026 Apr 2026
ComplexNLU (Tencent) Models answer Tencent in-house natural language understanding tasks with complex inputs. Task and scoring details are not published. Tencent Hunyuan Reading comprehension Everyday Dropped Hunyuan-A13B-Instruct 61.2% 1 1 Jun 2025 Jun 2025
CreativeWork Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls. ByteDance Seed Desktop apps Everyday In use Seed2.1 Pro 42.5% 1 1 Jun 2026 Jun 2026
Doubao Multi-Turn Bench Models answer the turn of a real Doubao conversation where the user was dissatisfied, with the earlier turns as context. Rubrics grade each answer. ByteDance Seed Multi-turn conversation Everyday In use Seed2.1 Pro 52.5% 1 1 Jun 2026 Jun 2026
GEdit-Bench Image editing models follow real-world user edit instructions. A model judge scores semantic consistency, perceptual quality and an overall score. StepFun (Step1X-Edit team) Image generation and editing Everyday Dropped BAGEL-7B-MoT 6.52 1 1 Apr 2025 Nov 2025
Human proximity (Google DeepMind robotics) Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. Google DeepMind Embodied reasoning Everyday In use Gemini Robotics ER 2 93% 1 1 Jul 2026 Jul 2026
HumMusQA Models answer 320 hand-written questions about music, curated and checked by people with musical training. Music Technology Group, Universitat Pompeu Fabra Audio understanding and voice Expert In use Qwen3.8-Omni-Flash 75.8% 1 1 Mar 2026 Sep 2026
Image2FloorPlan Models study several real photos of a home and draw its floor plan. Scored by an average quality score. ByteDance Seed Spatial reasoning and counting Everyday In use Seed2.1 Pro 48.0 1 1 Jun 2026 Jun 2026
MobileMiniWoB++ Agents complete MiniWoB++ web tasks inside the Android emulator of AndroidWorld. Scored by task success rate. Google Research (AndroidWorld team) Phones Everyday Dropped Qwen2.5-VL-72B 68% 1 1 May 2024 Feb 2025
OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. Meta FAIR Embodied reasoning Everyday Dropped Gemini 1.5 Pro 57.9 1 1 Apr 2024 Sep 2025
SaaS-Bench 106 long-horizon GUI workflows across 23 locally deployed real SaaS apps. Scored from app databases with weighted checkpoints and full-resolution rate. UniPat AI + Peking University Web browsing University In use Claude Opus 5 31.1% 1 1 May 2026 Aug 2026
SpotSoundBench Models find the time span of short sound events hidden in long, unstructured YouTube recordings. It has 400 audio, query and timestamp triplets. Sun et al. (SpotSound, arXiv 2604.13023) Audio understanding and voice Everyday In use Qwen3.8-Omni-Flash 67.2% 1 1 Apr 2026 Sep 2026
xDailyBench Models answer open-ended requests grounded in real user needs across more than 30 daily-life, study and office scenarios. Rubrics grade completion, correctness, coverage and usefulness. ByteDance Seed Overall chat quality Everyday In use Seed2.1 Pro 61% 1 1 Jun 2026 Jun 2026
CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app. Zhipu AI (Z.ai) Web and app front ends Expert Fading GLM-5 74.8% 1 1 Feb 2026 Apr 2026
IntelligentBench Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100. ByteDance Seed (BAGEL team) Video generation Everyday Dropped BAGEL-7B-MoT 55.3 1 1 May 2025 May 2025
ViDoRe V3 Retrieval and RAG pipelines answer queries over visually rich documents, such as reports with tables, charts and images. The benchmark has 10 datasets from professional domains, about 26,000 pages and 3,099 human-verified queries, each in 6 languages. It scores retrieval relevance, bounding-box grounding and the final answers. Illuin Technology and NVIDIA Documents, charts and text in images Everyday Not in cards Qwen3-VL-Embedding-8B 0.58 Jan 2026 Jan 2026
AyaVisionBench Vision-language models answer image questions in 23 languages across 9 task categories, 135 image-question pairs per language. A model judge compares two models' answers and reports win rates. Cohere For AI (Cohere Labs) General visual questions Everyday Dropped Aya Vision 32B 64% 1 1 Mar 2025 May 2025
CCPM Models pick which of four classical Chinese poetry lines matches a given modern Chinese translation. Scored by accuracy. Tsinghua University (THUNLP) Reading in many languages School Dropped DeepSeek-V3 92% 1 1 Jun 2021 Dec 2024
Code2Video-AA Artificial Analysis' implementation of HeyGen's Code2Video Bench: models write a HyperFrames HTML composition for each of 114 product and brand video briefs. A model judge compares two rendered videos at a time on five axes, and the comparisons become Elo ratings. Artificial Analysis Web and app front ends Expert In use Inkling 2 Small 1,020 1 1 Oct 2026 Nov 2026
HoneyBench v0.2-alpha Measures reward hacking: how often an agent games its grader instead of doing the task. Lower is better. Goodhart Labs Deception and alignment Research frontier In use Inkling 2 Small 44.0 1 1 Nov 2026 Nov 2026
IFEval (Mistral internal version) Models follow prompts with verifiable format constraints. Mistral uses its own fixed version of IFEval. Mistral AI Following precise instructions Everyday Dropped Magistral Medium 1.0 87.4% 1 1 Jun 2025 Jun 2025
Ko-Arena-Hard v2 (Upstage) Models answer hard user prompts in Korean, judged against a baseline in the style of Arena-Hard v2. Upstage built it in house. Upstage Chat and instructions in many languages Everyday Fading Solar Open 100B 79.9% 1 1 Dec 2025 Mar 2026
LeetCode Weekly Contest Models solve 126 LeetCode contest problems from July to November 2023. A solution counts only if it passes all of its 20+ test cases. DeepSeek Competitive programming University Dropped DeepSeek LLM 67B 17.5% 1 1 Nov 2023 Jan 2024
MMLU-stem Models answer the STEM subjects of MMLU, such as physics, chemistry, mathematics, and computer science, as multiple choice. Scoring is accuracy. UC Berkeley Exam questions University Dropped Falcon-H1-34B 83.6% 1 1 Sep 2020 Jul 2025
Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. Google DeepMind Finding and exploiting real bugs Expert In use Gemini 4 Argon 86% 1 1 Oct 2026
RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. University of Washington and collaborators (Wang et al.) Simulated robots Everyday Fading MolmoAct 2 44.3% 1 1 Jul 2025 May 2026
StrategyQA Models answer 2,780 yes or no questions that need implicit multi-step reasoning, such as whether Aristotle used a laptop. Scoring is accuracy. Mor Geva et al. (Tel Aviv University and Allen Institute for AI) Language reasoning Everyday Dropped PaLM 2-L 90.4% 1 1 Jan 2021 Apr 2025
ARKitScenes Models locate objects in indoor RGB-D scenes captured on iPads and output 3D bounding boxes. Scored by 3D detection average precision. Apple Detection and captioning Everyday Dropped Qwen3-VL-235B-A22B 53.7% 1 1 Nov 2021 Nov 2025
Audio Understanding Benchmark (Mistral) Models answer hard questions about 40 long-form audio recordings. Mistral built it in-house to test audio understanding beyond transcription. Mistral AI Audio understanding and voice Expert Dropped Voxtral Small 86.6% 1 1 Jul 2025 Jul 2025
Brevity Benchmark NVIDIA evaluation of short, direct answers for voice and device assistants. The card gives no link or method. NVIDIA Overall chat quality Everyday In use Nemotron 3.5 Bolt E4B 82.8% 1 1 Oct 2026 Oct 2026
CodeSimpleQA Models answer short bilingual fact questions about programming concepts and implementations across languages and computer science domains. Yang et al. Short factual questions University Fading Seed2.0 Pro 58% 1 1 Dec 2025 Feb 2026
CVTG-2K Image models render text in images across regions and languages. CVTG-2K reports normalized edit distance, CLIPScore and word accuracy. Z.ai Image generation and editing Everyday Fading GLM-Image 0.91 1 1 Jan 2026 Jan 2026
Docci Models retrieve matching documents and images using the DOCCI corpus. Google DeepMind Documents, charts and text in images University Fading Gemini Embedding 2 93.4 1 1 May 2026
EIFBench Models follow extremely complex instructions that bundle several tasks and many constraints in one request, as in real product workflows. Zou, Zhang et al. Following precise instructions Everyday Fading Seed1.8 48.6% 1 1 Jun 2025 Mar 2026
Infographic factuality (Google) Image generation systems produce infographics and an automated rater scores factuality. Google DeepMind Hallucination rates Everyday In use Gemini Nano Banana 2.1 0.52 1 1 Oct 2026 Oct 2026
Information Processing (ByteDance Seed in-house) Models analyze large collections of user-generated content, for example summarizing viewpoints and judging overall sentiment. ByteDance Seed Documents and spreadsheets Everyday Fading Seed1.8 53.9% 1 1 Dec 2025 Mar 2026
InsCtrl (Tencent) Models write text under detailed instructions in a Tencent in-house text creation test. Task and scoring details are not published. Tencent Hunyuan Following precise instructions Everyday Dropped Hunyuan-A13B-Instruct 71.9% 1 1 Jun 2025 Jun 2025
InstructTTSEval Text-to-speech models follow natural-language style instructions in English and Chinese across acoustic parameter, descriptive style and role-play tasks. Gemini judges each clip. Fudan University (Kexin Huang, Xipeng Qiu and others) Speech recognition and translation Everyday Fading Qwen3-TTS-12Hz-1.7B-CustomVoice 83% 1 1 Jun 2025 Jan 2026
Intention Recognition (ByteDance Seed in-house) Models classify intent from dialogues, meeting transcripts, social posts and articles, such as purchase intent in customer service chats. ByteDance Seed Running a business Everyday Fading Seed1.8 63.2% 1 1 Dec 2025 Mar 2026
IPhO 2025 Models solve the theory problems of the 2025 International Physics Olympiad. Answers are graded against the official marking scheme. International Physics Olympiad Science olympiads Competition Fading Seed2.1 Pro 79.3% 1 1 Jul 2025 Jun 2026
LengthCtrl (Tencent) Models write text that must meet a requested length, in a Tencent in-house test. Task and scoring details are not published. Tencent Hunyuan Creative writing Everyday Dropped Hunyuan-A13B-Instruct 55.4% 1 1 Jun 2025 Jun 2025
LongText-Bench Image models render long text in images. The benchmark reports separate English and Chinese scores. Z.ai Image generation and editing Everyday Fading GLM-Image 0.97 1 1 Jan 2026 Jan 2026
LVIS-MG Models ground many objects at once by outputting boxes for named categories in LVIS images. ByteDance Seed Detection and captioning Everyday Dropped Seed1.5-VL 72.5% 1 1 May 2025 May 2025
m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. Cohere Customer service Everyday Dropped Command A Reasoning 46.5% 1 1 Apr 2025 Aug 2025
m-WildVision Vision-language models answer WildVision bench prompts machine-translated into 23 languages. A model judge compares two models' answers and reports win rates. Cohere For AI (Cohere Labs) General visual questions Everyday Dropped Aya Vision 32B 72% 1 1 Mar 2025 May 2025
Mandarin-Minority translation (Tencent) Models translate between Mandarin and Chinese minority languages in a Tencent test set. XCOMET-XXL scores each translation. Tencent Hunyuan Translation Everyday Dropped Hunyuan-MT-7B 60.8% 1 1 Sep 2025 Sep 2025
MeasureBench Vision-language models read values from images of measuring instruments, such as gauges and rulers, in real and synthesized photos. Scored by matching the reading to the key. BAAI FlagEval team General visual questions Everyday In use Seed2.1 Pro 62.9% 1 1 Oct 2025 Jun 2026
MineDojo Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. NVIDIA, Caltech and collaborators (Fan et al.) Video games Everyday Fading Seed2.0 Pro 49% 1 1 Jun 2022 Feb 2026
MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. Ai2 Simulated robots Everyday Fading MolmoAct 2 37.7% 1 1 Feb 2026 May 2026
MSQA Models answer culture-specific knowledge questions in 11 major languages. ByteDance Seed Knowledge in many languages University In use Seed2.1 Pro 50.2% 1 1 Jun 2026 Jun 2026
MSR-VTT retrieval Models retrieve videos from text queries in the MSR-VTT collection. Microsoft Research Video understanding University Fading Gemini Embedding 2 68.0 1 1 Jun 2016 May 2026
NExT-QA Models answer causal, temporal and descriptive questions about short videos of daily life. Labs usually report the multiple choice split. National University of Singapore (Junbin Xiao et al.) Video understanding Everyday Dropped LongCat-Flash-Omni 86.2% 1 1 May 2021 Oct 2025
OmniMMI Omni models answer 2,290 questions on 1,121 streaming videos, including proactive tasks where they must decide when to respond. Wang et al. (OmniMMI team) Video understanding Everyday Fading Seed1.8 53% 1 1 Mar 2025 Mar 2026
OneIG-Bench Image models generate images from prompts with English and Chinese evaluation splits. Z.ai Image generation and editing Everyday Fading GLM-Image 0.53 1 1 Jan 2026 Jan 2026
ProverBench (DeepSeek) Provers write Lean 4 proofs for 325 formalized problems: 15 from AIME 2024 and 2025 and 310 from textbooks across number theory, algebra, calculus and analysis. Lean checks each proof. DeepSeek Formal proofs University Dropped DeepSeek-Prover-V2-671B 59.1% 1 1 Apr 2025 Apr 2025
RAGTruth A word-level annotated corpus of hallucinations in retrieval-augmented generation answers. It is used to train and evaluate hallucination detection and RAG faithfulness. ParticleMedia (NewsBreak) Hallucination rates Everyday In use Nemotron 3.5 Bolt E4B 77.9% 1 1 Jan 2024 Oct 2026
Random Complex Tasks (LongCat) Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs. Meituan LongCat Team Apps and MCP servers University Fading LongCat-Flash-Thinking-2601 35.8% 1 1 Jan 2026 Jan 2026
Robust IF Bench Models follow a diverse set of hard instructions from Microsoft AI's internal benchmark. Microsoft AI Following precise instructions Everyday In use MAI-Code-1-Flash 61.2% 1 1 Jun 2026 Jun 2026
ROK-FORTRESS Models face adversarial national security and public safety prompts in English and Korean, set in US or Korean contexts. Each prompt has a benign pair to measure over-refusal. Scale AI Dangerous capabilities Expert In use K-EXAONE 2.0 89.5% 1 1 May 2026 Aug 2026
RoleBench Models answer prompts while playing one of many fixed characters from films and books. Answers are compared with reference role answers by automatic text overlap. RoleLLM team (InteractiveNLP-Team, Wang et al.) Overall chat quality Everyday In use Nemotron 3.5 Bolt E4B 71.2 1 1 Oct 2023 Oct 2026
RoleMRC Models play given roles in multi-turn chats, passage reading tasks and nested or prioritized instructions. The test split has 1.4k samples. RoleMRC team (Lu et al., ACL 2025) Following precise instructions Everyday In use Nemotron 3.5 Bolt E4B 79.1% 1 1 Feb 2025 Oct 2026
ScanBench Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage. Baichuan AI Medicine Expert Fading Baichuan-M3-235B 73.8% 1 1 Jan 2026 Feb 2026
TACoS Models find the time span in a cooking video that matches a sentence. Labs score temporal grounding by mean IoU with the annotated span. Max Planck Institute for Informatics (Rohrbach et al.) Video understanding Everyday Dropped Seed1.5-VL 49.6% 1 1 Sep 2014 May 2025
tau2-Noise (LongCat) Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. Meituan LongCat Team Customer service Everyday Fading LongCat-Flash-Thinking-2601 67.1% 1 1 Jan 2026 Jan 2026
TextCaps Models retrieve matching text and images using the TextCaps corpus. Google DeepMind General visual questions University Fading Gemini Embedding 2 89.6 1 1 May 2026
TIIF-Bench Image models generate short and long images scored on overall image quality. Z.ai Image generation and editing Everyday Fading GLM-Image 81.0 1 1 Jan 2026 Jan 2026
U-Artifacts Models build interactive artifacts such as small apps and pages for 199 cases across 13 topics. A GUI agent simulates a non-expert user to grade interactivity, looks and fit to the need. ByteDance Seed Web and app front ends Everyday Fading Seed1.8 49.2% 1 1 Dec 2025 Mar 2026
URO-Bench Spoken dialogue models answer speech with speech across Understanding, Reasoning and Oral conversation. It has a basic track of 16 datasets and a pro track of 20. Yan et al. (arXiv 2502.17810) Speech recognition and translation Everyday Fading Qwen3.5-Omni-Plus 84.1% 1 1 Feb 2025 Apr 2026
VitaBench-Noise (LongCat) Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. Meituan LongCat Team Apps and MCP servers Everyday Fading LongCat-Flash-Thinking-2601 20.5% 1 1 Jan 2026 Jan 2026
Word-Task (Tencent) Models solve Tencent in-house word level language tasks. Task and scoring details are not published. Tencent Hunyuan Reading comprehension Everyday Dropped Hunyuan-A13B-Instruct 62.9% 1 1 Jun 2025 Jun 2025
YouCook2 retrieval Models retrieve cooking videos from text queries in the YouCook2 collection. University of Illinois and collaborators Video understanding University Fading Gemini Embedding 2 52.5 1 1 Sep 2018 May 2026
BEAM Models answer probing questions about coherent multi-session user conversations of 128K, 500K, 1M and 10M tokens. Questions test ten memory abilities such as knowledge updates, temporal reasoning and contradiction resolution. Mohammad Tavakoli, Hamed Zamani, J Ross Mitchell and collaborators Learning from the context Everyday In use 1 1 Oct 2025 Oct 2026
CCBench (MMBench Chinese Culture) Vision language models answer 510 Chinese-language multiple-choice questions about images of Chinese culture, such as calligraphy, painting, food and historical figures. Shanghai AI Laboratory (OpenCompass, MMBench team) General visual questions Everyday Dropped InternVL2.5-78B-MPO 86.9% 1 Oct 2023 Jun 2025
FACTS Parametric V2 Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer. Google DeepMind + Google Research + Kaggle Short factual questions University Not in cards GPT-6 Astra 83.8% Oct 2026 Oct 2026
CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Vals AI Finding and exploiting real bugs Expert In use Muse Wonder 80.4% 1 Oct 2026 Nov 2026
YC-Bench Agent runs a simulated startup for one year via CLI, managing staff, clients and cash. Scored by final funds. Collinear AI Running a business University In use Claude Fable 5.2 $3,445,495 1 Feb 2026 Nov 2026
EnterpriseBench: CoreCraft Agents work inside a simulated hardware startup with more than 2,500 entities and 23 tools. They triage incidents, audit shipments and handle customers under company policy. Surge AI Simulated companies Expert In use Claude Fable 5.1 77.4% 1 Feb 2026 Nov 2026
Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. Google (Android Developers) Building whole projects Expert Not in cards Claude Sonnet 5.5 40% Sep 2026 Nov 2026
DistressBench Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. Scale AI Wellbeing and sycophancy Expert In use Muse Spark 1.2 88.4% 1 Oct 2026 Nov 2026
Vals Legal Index Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue. Vals AI Composite work indices Expert In use Gemini 4 Neon 78.4% 1 Oct 2026 Nov 2026
DAYJOB: Finance Agents get short requests from finance colleagues plus a workspace of filings, ledgers and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Surge AI Tasks from real jobs Expert In use Claude Opus 5.5 42.5% 1 Sep 2026 Nov 2026
Humanity's Sixth Sense Models answer open-ended questions about what an image or short video implies but does not show, such as hidden objects, causes and intentions. An LLM judge checks each answer against a human-written rubric. Scale AI and Elorian General visual questions Everyday In use Gemini 4 Neon 75.9% 1 Oct 2026 Nov 2026
Lakera B3 (Backbone Breaker Benchmark) Crowdsourced prompt-injection attacks hit backbone LLMs inside agent threat snapshots at three defense levels. The score is attack resistance, 100 minus the board's vulnerability percentage. Lakera + ETH Zurich + UK AISI + Oxford Jailbreaks and prompt injection In use Claude Opus 4.5 89% 1 Oct 2025 Oct 2026
LiveSQLBench Contamination-resistant text-to-SQL on PostgreSQL with hierarchical knowledge bases, including CRUD tasks. Scored by soft execution match and test cases. BIRD team (HKU) Data science and SQL University In use MiniMax-M3 40.2% 1 May 2025 Jun 2026
LAMBADA Models predict the last word of narrative passages where humans can guess it only from the whole passage. Scored by accuracy. University of Trento and University of Amsterdam Reading comprehension Everyday Dropped PaLM 2-L 86.9% 1 Jun 2016 Nov 2023
SWE Atlas Test Writing Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests. Scale AI Fixing real repositories Expert In use Claude Fable 5.1 67% 1 Apr 2026 Nov 2026
RefAdv-S Models locate the object a referring expression describes among hard distractors in real images. Scored by accuracy at 0.5 box overlap on 1,142 cases. Northeastern University Detection and captioning Everyday In use Qwen3.5 397B-A17B 68% 1 Feb 2026 Aug 2026
MADQA Agents search 800 PDF documents to answer 2,250 human-written questions. Scoring covers answer accuracy and how much search effort the agent spends. Łukasz Borchmann et al. (Snowflake, University of Oxford, and partners) Documents, charts and text in images Everyday In use Gemini 3 Pro 85.2% 1 Mar 2026 Aug 2026
sudo L7 Coding agents get staff-level engineering requests in mostly private production codebases. Graders score whether the agent found hidden risks and shipped the right change. Surge AI Fixing real repositories Expert In use Claude Fable 5.2 52.2% 1 Oct 2026 Nov 2026
HANDBOOK.md Agents do routine company work with mock email, chat, calendar, issue and commerce tools while following a 20 to 124 page handbook. Programmatic criteria check required and forbidden actions, and a trial passes only if all pass. Surge AI Following precise instructions University In use Claude Opus 5.5 52.3% 1 Jul 2026 Nov 2026
MolecularIQ Models answer questions about molecular graphs, such as counting, indexing and constrained generation, and a symbolic checker verifies each answer. Johannes Kepler University Linz (Bartmann, Klambauer, Luukkonen et al.) Genomics and life sciences Expert In use gpt-oss-120b 47.5% 1 Jan 2026 Sep 2026
SWE Atlas Refactoring Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result. Scale AI Fixing real repositories Expert In use GPT-6 Astra 59% 1 May 2026 Nov 2026
SWE-fficiency 498 tasks in nine Python data/ML/HPC repos: agent must speed up a real workload without breaking tests. Scored as speedup relative to expert patch. SWE-fficiency authors (ICML 2026) Speed and optimization Expert In use Claude Opus 4.5 22.5% 1 Nov 2025 Jun 2026
Alyah Models answer 1,173 four-option questions written by native Emirati speakers on greetings, idioms, figurative language, heritage and poetry in Emirati Arabic dialect. Technology Innovation Institute (TII) and community contributors Knowledge in many languages Everyday In use Gemma 3 27B 74.7% 1 Jan 2026 Oct 2026
PLawBench Model handles 850 realistic legal consultation, case-analysis and document-drafting tasks. LLM judge scores against about 12,500 expert rubric items. SKYLENAGE-AI (Shi et al.) Law Expert In use GPT-5.2 69.7% 1 Jan 2026 Aug 2026
SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. Shanghai AI Lab and partners (Wanghan Xu et al.) Research science Expert In use Gemini 3 Pro 33.8% 1 Dec 2025 Sep 2026
OmniParseBench Document parsers convert single pages to text, and 16,288 binary unit tests on 2,937 pages check the output. The headline joins text, table and layout subscores. Datalab Documents, charts and text in images Everyday In use Gemini 4 Argon 96.0 1 Oct 2026 Oct 2026
RoadmapBench Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests. UniPat AI Building whole projects Expert In use Claude Opus 4.7 39.1% 1 May 2026 Sep 2026
Ego3D-Bench Models answer spatial reasoning questions about ego-centric multi-view scenes, such as those from self-driving cars. Some items ask for distances, scored by error. Authors of arXiv 2509.06266 Spatial reasoning and counting Everyday In use Ovis2-16B 7.40 1 Sep 2025 Jun 2026
GroundUI Agents get a screenshot and an instruction and must click the right UI element. About 18K test samples cover web, desktop and mobile screens. Scored by whether the point falls in the target box. AgentStudio team (Longtao Zheng et al., Nanyang Technological University and Skywork AI) Finding things on screen Everyday Dropped Gemini 1.5 Pro 35.2% 1 Jun 2024 Mar 2025
SWE-bench-Live Agents resolve recent GitHub issues gathered by an automated pipeline, with 50 new verified Python tasks added each month. Tests in a docker sandbox check each patch. Microsoft Fixing real repositories Expert Dropped DeepSeek V4.1 Flash 71% 1 May 2025 Sep 2026
Harbor-Index Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. Harbor and Laude Institute (Terminal-Bench team) Terminal tasks Research frontier In use GPT-5.5 28.1% 1 Jun 2026 Aug 2026
MarathonDocBench Models parse 100 documents spanning 103 to 300 pages. Evaluation scores content, cross-page continuations, figures, reading order, and heading hierarchy. Baidu ERNIE Team Documents, charts and text in images Expert In use Unlimited OCR 2 77.1% 1 Oct 2026 Oct 2026
SWE Refactor Bench Agents migrate 20 whole open-source repositories to a new language, framework, platform, or build system. Graded by migration audit, behavioral tests, and adversarial verifier models. Einsia Building whole projects Research frontier In use Claude Opus 5 47% 1 Aug 2026 Oct 2026
SWT-bench Agents write tests that reproduce real GitHub issues from SWE-bench repositories. Scored by the share of tests that fail before and pass after the fix. ETH Zurich and LogicStar.ai Fixing real repositories Expert Fading Devstral 2 89.1% 1 Jun 2024 Dec 2025
BrokenArXiv Models are asked to prove statements from recent arXiv papers that were changed to be plausible but false. Full credit goes to models that say the statement is false. MathArena (ETH Zurich SRI Lab and INSAIT) Research mathematics Research frontier In use Gemini 4 Carbon 84.4% 1 Mar 2026 Oct 2026
E-Bench-Code (Tencent) Agents complete Tencent in-house work tasks that need code. Task and scoring details are not published. Tencent Hy team Documents and spreadsheets Expert In use GPT-5.6 Sol 83.3% 1 Aug 2026 Aug 2026
Finch (FinWorkBench) Agent completes 172 enterprise finance and accounting spreadsheet workflows (384 tasks) from real data. Human plus calibrated LLM-judge grading. FinWorkBench (Dong et al.) Finance Expert In use Claude Opus 4.5 43% 1 Dec 2025 Oct 2026
Functional MATH Models solve new instances of MATH problems that code generates for each snapshot. The gap from static MATH accuracy measures memorization. Consequent AI Competition problems with answers Competition Dropped GPT-4 14.2% 1 Feb 2024 May 2024
VLADBench Models answer closed-form questions about driving scenes, from static traffic knowledge to reasoning about dynamic road situations. Authors of arXiv 2503.21505 Spatial reasoning and counting University In use Qwen2.5-VL-72B 58% 1 Mar 2025 Jun 2026
Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. Alibaba Qwen Team, Alibaba Cloud security and National University of Singapore Secure code and review Expert In use Claude Fable 5 25.1% 1 Oct 2026 Nov 2026
MT-Bench-101 Models hold 1,388 multi-turn dialogues with 4,208 turns across 13 tasks from a three-tier ability taxonomy. GPT-4 scores each turn from 1 to 10. Alibaba Group, CUHK MMLab and Shanghai AI Lab (Ge Bai, Jie Liu et al.) Multi-turn conversation Everyday Dropped Qwen-14B 7.82 1 Feb 2024 Jan 2025
PatchEval-Verified Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch. ByteDance Finding and exploiting real bugs Expert Not in cards GPT-5.6 Sol 83.9% Jul 2026 Jul 2026
RoboDojo Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. MMLab@HKU + UC Berkeley + Tsinghua + others (RoboDojo team) Simulated robots Everyday In use GPT-6 Astra 27.6% 1 Jul 2026 Nov 2026
SITE Models answer multiple-choice spatial questions over single images, multiple images and video, drawn from about 30 vision datasets. Tasks span figural to environmental scale. Boston University and Microsoft Research Spatial reasoning and counting Everyday Fading InternVL2.5-8B 32.8% 1 May 2025 Jan 2026
SWE-Perf Agents speed up code in 140 tasks drawn from performance pull requests in real GitHub repos. Scored on correctness and runtime gain. TikTok and Xi'an Jiaotong University Speed and optimization Expert Fading Claude 3.7 Sonnet 2.26% 1 Jul 2025 Dec 2025
VideoDR (with search) Agents watch a video, gather clues across frames, and search the web to answer multi-hop questions. Scored by answer accuracy. QuantaAlpha Video understanding University In use Gemini 3 Pro 76% 1 Jan 2026 Aug 2026
GDPval-MM Agents produce office deliverables for GDPval tasks in a MiniMax agent framework. A judge model compares full trajectories pairwise and reports a win rate. MiniMax Tasks from real jobs Expert Fading Claude Opus 4.6 73.5% 1 Feb 2026 Feb 2026
Kimi Code Bench 3.0 Long-horizon Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score. Moonshot AI Building whole projects Expert In use GPT-6 Astra 66.5% 1 Oct 2026 Oct 2026
Kimi Hallu Bench Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. Moonshot AI Hallucination rates Expert In use GPT-6 Astra 4.20% 1 Oct 2026 Oct 2026
L-Eval Models answer over 2,000 queries on 508 long documents of 3k to 200k tokens across 20 subtasks. Scoring uses exact match and judge-model ratings. OpenLMLab (Fudan University) Reasoning over long documents University Dropped Claude 1.3 76.4% 1 Jul 2023 Feb 2024
Multi-SWE-bench flash Agents fix real GitHub issues in several languages, a 300-instance subset of Multi-SWE-bench chosen for fast evaluation and agent rollouts. Scored by tests passing. ByteDance Seed Fixing real repositories Expert Dropped Qwen3-Coder-480B-A35B 25.3% 1 Jul 2025 Jul 2025
OctoCodingbench Agents complete 72 coding tasks while following rules from system prompts, AGENTS.md files, skills, and memory. An LLM judge checks 2,422 pass or fail items. MiniMax Fixing real repositories University Fading Claude Opus 4.5 36.2% 1 Dec 2025 Dec 2025
Orak LLM agents play 12 popular video games across most genres through an MCP interface. Each game scores in-game progress or outcomes, and the board ranks models by their average rank across the games. KRAFTON AI with Seoul National University, NVIDIA and UW-Madison Video games Everyday Fading Gemini 2.5 Pro 3.90 1 Jun 2025 Mar 2026
QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. Qwen Team, Alibaba Group Web and app front ends University In use Qwen 3.8 Max 69.9% 1 Aug 2026 Aug 2026
QwenVisualOffice Models handle visual office tasks in an internal Qwen test set. Qwen has not published the tasks or how they are scored. Qwen (Alibaba) Documents, charts and text in images University In use Qwen 3.8 Max 44.6% 1 Aug 2026 Aug 2026
TOMG-Bench Models generate molecules from natural language requests in three tasks: molecule editing, molecule optimization and customized generation. Each output is checked for validity and for meeting the requested properties. Hong Kong Polytechnic University and Shanghai AI Lab (Jiatong Li et al.) Genomics and life sciences Expert In use GPT-3.5 Turbo 18.6% 1 Dec 2024 Sep 2026
3DSRBench Models answer questions about 3D spatial relations in images, such as height, location, orientation and relations between objects. Wufei Ma, Alan Yuille et al. Spatial reasoning and counting Everyday Fading SenseNova-SI-1.3-InternVL3-8B 62.4% 1 Dec 2024 Jan 2026
AVMeme Exam Models answer questions about over one thousand iconic internet sounds and videos, from surface content to context, emotion, usage and world knowledge. Jiang et al. (AVMeme Exam team) Audio understanding and voice Everyday Fading Seed2.0 Lite (0428) 69.5% 1 Jan 2026 Apr 2026
miniCTX-v2 Provers prove Lean theorems from real projects using long repository context, with theorems taken after a November 28, 2024 cutoff. CMU L3 Lab (Hu et al.) Formal proofs Expert Dropped Seed-Prover 81.8% 1 Mar 2025 Sep 2025
Progress classification (Google DeepMind) Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. Google DeepMind Embodied reasoning Everyday In use Gemini Robotics ER 2 57.4% 1 Jul 2026 Jul 2026
QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). Alibaba Qwen Fixing real repositories Expert In use Claude Fable 5 63.1% 1 Aug 2026
QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. Alibaba Qwen Web and app front ends Expert In use Claude Fable 5 1,770 1 Aug 2026
QwenSVGBench Model writes SVG code for bilingual prompts. Rendered outputs judged by a multimodal model and ranked with Bradley-Terry Elo. Alibaba Qwen Web and app front ends University In use GPT-5.6 Sol 1,758 1 Aug 2026
QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. Qwen Team, Alibaba Group Web and app front ends University In use GPT-5.4 1,884 1 May 2026 Jun 2026
StepCodeBench Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup. StepFun Fixing real repositories Expert In use Claude Opus 5 63.9% 1 Sep 2026 Sep 2026
SWE-Review Models review code changes in several languages and scenarios to find defects. Scores reflect defect recall and the rate of hallucinated defects. MiniMax Fixing real repositories Expert Fading Claude Opus 4.5 16.2% 1 Dec 2025 Dec 2025
ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. Qwen Team, Alibaba Group Function calling Everyday Fading Qwen 3.6 Max (Preview) 86.1% 1 Apr 2026 Apr 2026
xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. Moonshot AI Finding and exploiting real bugs Expert In use GPT-5.6 Sol 100% 1 Oct 2026 Oct 2026
Z.ai Code Bench Zhipu in-house coding benchmark used for GLM-5.3. Reports pass rate and output tokens per task. Method unpublished. Zhipu AI (Z.ai) Fixing real repositories Expert In use Claude Fable 5 39.5% 1 Aug 2026 Aug 2026
Answer Factuality (Perplexity) Models answer questions from search results. Graders check that answers use facts grounded in the results and resolve conflicting or missing information. Perplexity Faithful to sources Everyday Dropped sonar 85.1% 1 Feb 2025 Feb 2025
CB Models label whether a text commits its speaker to an embedded clause as entailment, contradiction, or neutral. Scored by accuracy and F1. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser Language reasoning Everyday Dropped PaLM 2-L 87.5% 1 Apr 2019 May 2023
FutureOmni Models watch and hear 919 videos and answer 1,034 multiple-choice questions that ask what will happen next. Questions need audio and visual cues together. OpenMOSS (Fudan University) and collaborators Video understanding Everyday Fading MiMo-V2-Omni 66.7% 1 Jan 2026 Mar 2026
Kimi Claw 24/7 Bench Agents work on 17 persistent multi-day professional scenarios through OpenClaw, with 610 checkpoints. The score is the mean pass rate over three runs. Moonshot AI Tasks from real jobs Expert In use Kimi K2.7 Code 46.9% 1 Jun 2026 Jun 2026
Mind2Web 2 Agents browse the live web for 130 long tasks and return answers with citations. Task-specific judge agents built from rubric trees grade correctness and attribution. Ohio State University (OSU NLP Group) Deep research Expert Not in cards OpenAI deep research 28% Jun 2025 Apr 2025
MultiRC Models pick all correct answers to questions that need reasoning over several sentences of a paragraph. Scored by F1 over answer options. University of Pennsylvania (CogComp) Reading comprehension School Dropped PaLM 2-L 88.2% 1 Jun 2018 May 2023
WSC Models decide whether a pronoun refers to a given noun phrase in sentences that need commonsense to resolve. Scored by accuracy. NYU, Facebook AI, University of Washington, and DeepMind (SuperGLUE) Commonsense Everyday Dropped PaLM 2-L 86.9% 1 May 2019 May 2023
Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. Google (Big Sleep team) Finding and exploiting real bugs Expert In use Gemini 3.5 Flash Cyber 72% 1 Jul 2026 Jul 2026
CHAOS-Bench Models read deliberately corrupted words in 500 academic page images. Scores measure faithful recognition of the visible strings instead of correction to plausible words. Tencent Hunyuan Documents, charts and text in images Expert In use MiniCPM-OCR-1.0 29.8% 1 Jul 2026 Oct 2026
Chrome Production Commit Scanning (Google) Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. Google (Chrome Security) Finding and exploiting real bugs Expert In use Gemini 3.5 Flash Cyber 72% 1 Jul 2026 Jul 2026
RTE Models decide whether a premise sentence entails a hypothesis, using data from the PASCAL RTE challenges. Scored by accuracy. PASCAL RTE Challenge organizers, packaged in GLUE Language reasoning Everyday Dropped PaLM 2-L 79.3% 1 Apr 2018 Jul 2023
Android in the Zoo (AitZ) Models predict the next action on Android app screens, given the goal and past steps. Scored by exact match of the predicted action on the test split. Zhang et al. (Fudan University and others) Phones Everyday Dropped Qwen2.5-VL-72B 83.2% 1 Mar 2024 Jan 2025
Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. Moonshot AI Running a business Expert In use Kimi K3.1 Pro 37.8% 1 Oct 2026 Oct 2026
AdvancedMathBench Models write proofs for 245 advanced problems at undergraduate and doctoral qualifying exam levels. A verifier trained on expert annotations grades each proof. Shanghai AI Lab (InternLM) Written proofs Expert In use Intern-S2-397B 58.1% 1 Jul 2026 Sep 2026
AIR-Bench Foundation Models answer foundational audio understanding questions. The score is the official average accuracy across nineteen tasks. Zhejiang University and Alibaba Group Audio understanding and voice Everyday In use NVIDIA Nemotron 3.5 Transcribe 30B-A4B 67.3% 1 Feb 2024 Oct 2026
Artificial Analysis average word error rate Speech transcription systems are scored by average word error rate across the Artificial Analysis evaluation set. Artificial Analysis Speech recognition and translation Everyday In use Gemini 3.5 Transcribe 4.00% 1 Aug 2026 Aug 2026
Artificial Analysis Cyber Index Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate. Artificial Analysis and the Cyber Index Alliance Finding and exploiting real bugs Expert In use Mistral Large 4 (Le Chonk, open weights) 51% 1 Sep 2026 Oct 2026
CELLO Models follow real-world complex instructions with multiple tasks, noisy inputs and format or length constraints. Rule-based checks score each answer against the instruction features. Fudan University (He et al.) Following precise instructions Everyday Dropped Hunyuan-T1 76.4% 1 Sep 2023 Mar 2025
ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. Gemini Team, Google Documents, charts and text in images University Dropped Gemini 1.5 Pro 69.7% 1 Jun 2024 May 2024
ChineseQA Models answer Chinese trivia questions in a closed-book format modeled on TriviaQA. DeepSeek Trivia and open questions Everyday Dropped DeepSeek LLM 67B 85.1% 1 Nov 2023 Nov 2023
CMRC Models extract answer spans from Chinese Wikipedia paragraphs for about 20,000 human-written questions. Scoring is exact match and F1. Joint Laboratory of HIT and iFLYTEK Research (HFL) Reading in many languages Everyday Dropped DeepSeek-V3 76.3% 1 Oct 2018 Dec 2024
COCO Models detect and locate objects in everyday photos from the Microsoft COCO dataset. Scored by mean average precision over box overlap thresholds. Microsoft and academic partners Detection and captioning Everyday In use Qwen 3.8 Max 78.7% 1 May 2014 Aug 2026
Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea. OpenAI Fixing real repositories Expert Dropped GPT-5-Codex 51.3% 1 Sep 2025 Sep 2025
ComplexFuncBench Audio (OpenAI) Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs. OpenAI Function calling Everyday Dropped gpt-realtime 66.5% 1 Aug 2025 Aug 2025
Context Aware Automatic Speech Recognition Speech transcription systems are scored for semantic accuracy with and without free-form context. OpenAI Speech recognition and translation Everyday In use GPT-Live-Transcribe 44.6% 1 Jul 2026 Jul 2026
COPA Models choose which of two alternatives is the more plausible cause or effect of a premise, across 1,000 questions. Scored by accuracy. University of Southern California (ICT) Commonsense Everyday Dropped PaLM 2-L 96% 1 Mar 2011 May 2023
CueSpeak Models speak at the correct moment and give the expected semantic response during simultaneous speech. Thinking Machines Lab Speech recognition and translation Everyday Fading TML-Interaction-Small 81.7% 1 May 2026 May 2026
Data Science Tasks (OpenAI) Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading. OpenAI Documents and spreadsheets Expert In use GPT-6 Astra 40.9% 1 Sep 2026 Sep 2026
Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading. OpenAI Fixing real repositories Expert In use GPT-6 Astra 63.9% 1 Sep 2026 Sep 2026
DECK-Arena Models build presentations for real user queries with python-pptx, pptxjs or a Kimi slide language, with search and image generation. An automated grader rates them as Elo scores. Moonshot AI Documents and spreadsheets University In use Kimi K3.1 Pro 1,456 1 Oct 2026 Oct 2026
Design Bench (Moonshot) Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. Moonshot AI Taste and style University In use Kimi K3.1 Pro 957 1 Oct 2026 Oct 2026
Design Tasks (OpenAI) Models produce design work such as slides and documents that follow templates, from an OpenAI internal set. OpenAI has not published the grading. OpenAI Documents and spreadsheets Expert In use GPT-6 Astra 50% 1 Sep 2026 Sep 2026
DS-Arena-Code Models answer common coding requests in an internal subjective test. GPT-4o judges responses against competitor models to give a win rate. DeepSeek-AI Function-level problems University Dropped DeepSeek-V2.5 63.1% 1 Sep 2024 Sep 2024
DS-FIM-Eval Models fill in missing code between a given prefix and suffix, as in editor plugin completion. DeepSeek Function-level problems University Dropped DeepSeek-V2.5 78.3% 1 Sep 2024 Sep 2024
DSBench Data science agents solve 466 data analysis tasks and 74 end-to-end data modeling tasks drawn from Eloquence and Kaggle competitions. Analysis is scored by accuracy and modeling by relative performance gap. Tencent AI Lab and University of Texas at Dallas (Jing et al.) Data science and SQL Expert Dropped ChatGPT agent 89.9% 1 Sep 2024 Jul 2025
EgoLife (with memory) Models answer life-assistant questions about a week of egocentric video from six people, such as recalling past events and habits. EgoLife team (Jingkang Yang, Ziwei Liu et al.) Long videos Everyday In use Qwen 3.8 Max 80.3% 1 Mar 2025 Aug 2026
Finance Modeling Agents complete expert-written financial modeling problems with Excel tools, from research to analysis. Expert-designed rubrics score each problem, averaged over three runs. MiniMax Finance Expert Fading MiniMax-M2.5 21.6% 1 Feb 2026 Feb 2026
FinSearchComp Global Agents search for financial information about global assets across three task types. The score is the published average accuracy for the Global subset. ByteDance Seed Finance Expert Dropped MiniMax-M2 65.5% 1 Sep 2025 Oct 2025
FinStepBench Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations. StepFun Finance Expert In use Step 5 Preview 63.6% 1 Sep 2026 Sep 2026
Gaokao Math 2026 Models sit the five 2026 Chinese Gaokao mathematics papers (National I, National II, Beijing, Shanghai and Tianjin). Each paper is graded out of 150 points. iFlytek (SparkLLM) Competition problems with answers School In use Spark-X2.6-4B 134.9 1 Sep 2026 Oct 2026
GDPval-Rubrics (MiniMax) MiniMax rubric grading of model deliverables on public GDPval tasks, in an environment matched to GDPval-AA. MiniMax Tasks from real jobs Expert In use MiniMax-M3 74.8% 1 Jun 2026 Jun 2026
HaluEval Models judge whether answers, dialogue turns or summaries contain hallucinations. Hallucinated samples were generated by ChatGPT from seed datasets such as HotpotQA and then filtered. RUCAIBox (Renmin University of China) Hallucination rates Everyday Fading Nemotron 3 Nano 4B 62.2% 1 May 2023 Mar 2026
Hy-BrowseComp-Pro2 Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published. Tencent Hy team Deep research Expert In use Tencent Hy4 preview 56.1% 1 Aug 2026 Aug 2026
Hy-Euler pro Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published. Tencent Hy team Research mathematics Expert In use Tencent Hy3 24.2% 1 Jul 2026 Jul 2026
Hy-FinAgentBench Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published. Tencent Hy team Finance Expert In use Tencent Hy4 preview 79.7% 1 Aug 2026 Aug 2026
Hy-LifeSearch Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published. Tencent Hy team Deep research Expert In use Tencent Hy4 preview 49.2% 1 Aug 2026 Aug 2026
Hy-Math Models solve Tencent in-house math problems without tools. Task and scoring details are not published. Tencent Hy team Competition problems with answers Expert In use Tencent Hy3 60.9% 1 Jul 2026 Jul 2026
Hy-SkillsWorld Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published. Tencent Hy team General assistants Expert In use Tencent Hy3 45.8% 1 Jul 2026 Jul 2026
Hy-Vibe Bench Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published. Tencent Hy team Building whole projects Expert Fading Tencent Hy3 preview 50.5% 1 Apr 2026 Apr 2026
ImageMining Multimodal agents retrieve and reason over image information for image mining tasks. Z.ai Deep research Expert Fading GLM-5V-Turbo 30.7% 1 Apr 2026 Apr 2026
Infinite Generations (Mistral internal) Models answer challenging long or repetitive prompts from Mistral's internal set. The score is the share of responses that never stop or loop, so lower is better. Mistral AI Overall chat quality Everyday Dropped Mistral Small 3.2 1.29% 1 Jun 2025 Jun 2025
IntPhys 2 Models watch simulated videos and judge whether each event is physically possible, testing object permanence, immutability, continuity and solidity. Scored by accuracy. Meta FAIR Physical prediction Everyday In use Muse Spark Open 68.3% 1 Jun 2025 Oct 2026
KAT Claw Bench Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables. Kuaishou (KwaiKAT team) General assistants University In use KAT-Coder-Pro V2.5 85.5% 1 Jul 2026 Jul 2026
KernelBench Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. Stanford Scaling Intelligence GPU kernels Expert In use MiniMax-M3 28.8% 1 Dec 2024 Jun 2026
Longform Writing Models write long-form text without tools, capped at 32k output tokens. Moonshot AI has not published the tasks or the scoring method. Moonshot AI Creative writing University Dropped Kimi K2 Thinking 73.8% 1 Nov 2025 Nov 2025
LSMDC Models fill in a missing word in movie audio descriptions using the matching video clip. Scoring is accuracy. Tegan Maharaj et al. (LSMDC challenge) Video understanding Everyday Dropped GPT-4 45.7% 1 Nov 2016 Mar 2023
LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. Zhipu AI and Tsinghua University (LVBench), Qwen (memory setup) Long videos Everyday In use Qwen 3.8 Max 85.6% 1 Aug 2026 Aug 2026
MEWC Models solve 179 Microsoft Excel World Championship problems from 2021 to 2026 with Excel tools. Output cells are compared one by one with answer cells. MiniMax Documents and spreadsheets Competition Fading MiniMax-M2.5 74.4% 1 Feb 2026 Feb 2026
MMSearch Multimodal agents search and answer questions that require visual and textual evidence. Z.ai Deep research Expert Fading GLM-5V-Turbo 72.9% 1 Apr 2026 Apr 2026
Mobile Actions Model converts natural-language phone requests into on-device function calls (calendar, email, maps, flashlight, Wi-Fi). Call accuracy. Google Function calling Everyday Fading Reka Edge 88.4% 1 Dec 2025 Mar 2026
Moment finding (Google DeepMind) Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. Google DeepMind Embodied reasoning Everyday In use Gemini Robotics ER 2 91.3% 1 Jul 2026 Jul 2026
MPDocBench-Parse Models parse complete multipage documents while preserving content and cross-page structure. Evaluation scores text, formulas, tables, figure extraction, reading order, and heading hierarchy. Bangbang Zhou and collaborators Documents, charts and text in images Expert In use Unlimited OCR 2 84.6% 1 May 2026 Oct 2026
MT-AIME (Cohere internal) Cohere's internal machine translation of AIME 2025 problems into many languages (via Command A Translate). Answer accuracy. Cohere Reasoning in many languages Competition Fading Command A+ 86% 1 May 2026 May 2026
MultiChallenge Audio (OpenAI) Speech-to-speech models answer an audio-friendly subset of MultiChallenge multi-turn questions that OpenAI converted from text to speech. Scored on instruction following accuracy. OpenAI Audio understanding and voice Everyday Dropped gpt-realtime 30.5% 1 Aug 2025 Aug 2025
OpenScore String Quartets Models transcribe images of string quartet sheet music into symbolic notation. Scored as 1 minus normalized edit distance from the reference encoding. OpenAI, using the OpenScore String Quartets corpus Documents, charts and text in images University In use GPT-6 Astra 0.84 1 Sep 2026 Sep 2026
PAgentBench Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house. iFlytek (SparkLLM) General assistants Everyday In use Spark-X2.6-4B 74.1% 1 Oct 2026 Oct 2026
Past IMO (Seed-Prover formal set) Provers write Lean proofs for 155 past IMO problems, mostly adapted from Compfiles and MiniF2F. ByteDance Seed Formal proofs Competition Dropped Seed-Prover 78.1% 1 Jul 2025 Jul 2025
Pile-test Models predict text from the test split of the Pile, an 800GB corpus of 22 sources. Scored by bits per byte. EleutherAI Other Dropped DeepSeek-V3 0.55 1 Dec 2020 Dec 2024
ProactiveVideoQA Models answer spoken questions when a streamed video first reveals the answer. Thinking Machines Lab Video understanding Everyday Fading TML-Interaction-Small 33.5 1 May 2026 May 2026
ProdBench (Tencent) Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3. Tencent Hy team General assistants Expert In use Tencent Hy3 23% 1 Jul 2026 Jul 2026
QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. Qwen Team, Alibaba Group Knowledge in many languages University Fading Qwen 3.6 Max (Preview) 84% 1 Apr 2026 Apr 2026
Readability (Perplexity) Models answer search questions. Graders rate whether answers are concise yet detailed and use markdown formatting well. Perplexity Taste and style Everyday Dropped sonar 85.9% 1 Feb 2025 Feb 2025
Real-World Audio Recording Speech transcription systems are scored by word error rate on real-world recordings across multiple languages. OpenAI Speech recognition and translation Everyday In use GPT-Live-Transcribe 9.60% 1 Jul 2026 Jul 2026
Real5-OmniDocBench Models parse the same 1,355 document pages under five physical capture conditions. Text edit distance, formula matching, and table similarity determine the overall score. Baidu PaddlePaddle Documents, charts and text in images Expert In use MiniCPM-OCR-1.0 93.7% 1 Mar 2026 Oct 2026
ReCoRD Models fill in a masked entity in a query about a news article, using the article as context. Scoring is exact match and token F1. Johns Hopkins University and Microsoft Research Reading comprehension Everyday Dropped PaLM 2-L 93.8% 1 Oct 2018 May 2023
RepCount-A Models count repeated actions in streamed video and are scored by distance from the true count. Thinking Machines Lab Video understanding Everyday Fading TML-Interaction-Small 35.4 1 May 2026 May 2026
RepoBench Models complete code using context retrieved from other files in the same Python or Java repository. Three tasks cover retrieval, completion and the full pipeline. UC San Diego (Tianyang Liu, Canwen Xu, Julian McAuley) Fixing real repositories Expert Dropped Codestral 38% 1 Jun 2023 Jan 2025
SIFo Models follow a chain of dependent instructions, where each step uses the output of the one before. Only the final answer is checked, across 800 samples in four tasks. University of Amsterdam and University of Groningen (Chen et al.) Following precise instructions Everyday Dropped Qwen3-VL-235B-A22B 77.3% 1 Jun 2024 Sep 2025
Step-SWE-Bench Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses. StepFun Fixing real repositories Expert Fading Step 3.7 Flash 67.1% 1 May 2026 May 2026
Story Cloze Winter 2018 Systems select the correct story ending on the Winter 2018 test set. CodaLab reports percentage accuracy under submission identities. University of Rochester Commonsense Everyday Not in cards Jan 2018
SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. Ant Group and Peking University Research science Expert In use Ling-3.1-flash 51.2% 1 Oct 2026 Oct 2026
SVG-Bench (MiniMax) Model creates or edits SVG graphics from text or image prompts. A VLM checks results, averaged over 3 runs. MiniMax Web and app front ends University In use MiniMax-M3 63.7% 1 Jun 2026 Jun 2026
SWE-Dev Models implement new features in real Python repositories from requirement documents, across 500 test tasks. Developer-written unit tests decide pass or fail. Shanghai Jiao Tong University Fixing real repositories Expert Dropped Kimi K2 66.6% 1 May 2025 Sep 2025
SWebBench Agents browse the web to retrieve information and do research tasks. iFlytek built it in house. iFlytek (SparkLLM) Deep research Expert In use Spark-X2.6-4B 71.7% 1 Oct 2026 Oct 2026
TAT-DQA Models answer questions about financial report pages that mix tables and text, often needing arithmetic. Scored by exact match and F1. NExT++ Research Centre, National University of Singapore Documents, charts and text in images University Dropped Gemini 1.5 Pro 37.8% 1 Jul 2022 May 2024
TheAgentCompany Agents do work tasks in a simulated software company, using a browser, code, and chat with simulated coworkers. Scored by full and partial checkpoint completion. Carnegie Mellon University et al. Simulated companies University Dropped 1 Dec 2024 Oct 2025
TimeSpeak Models initiate speech at specified times while producing the requested content. Thinking Machines Lab Speech recognition and translation Everyday Fading TML-Interaction-Small 64.7% 1 May 2026 May 2026
TVQA Models answer 152,545 multiple-choice questions about clips from 6 TV shows, using video frames and subtitles. Scoring is accuracy. UNC Chapel Hill Video understanding Everyday Dropped GPT-4 87.3% 1 Sep 2018 Mar 2023
UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. ByteDance Seed Video games Everyday Dropped UI-TARS-2 59.8 1 Sep 2025 Sep 2025
VCSUM Models summarize transcripts of 239 real Chinese meetings that total over 230 hours. Scored with ROUGE against human-written summaries. Han Wu, Mingjie Zhan, Linqi Song, and colleagues Reasoning over long documents Everyday Dropped Qwen-14B 17.3 1 May 2023 Sep 2023
Video-Index Models answer 840 multiple-choice questions about videos, the 210 hardest in each of four groups: perception, time, space and reasoning. Questions that shortcut attacks could answer without the video were left out. Song et al. (arXiv 2610.00960) Video understanding Everyday In use Qwen4-Max 68.1% 1 Oct 2026 Nov 2026
VideoHallucer Paired basic and hallucinated questions about videos detect intrinsic and extrinsic hallucinations. Scored by pair accuracy. BIGAI + Peking University Video understanding Everyday Fading Reka Edge 59.6% 1 Jun 2024 Mar 2026
VisualProbe Models search high-resolution images for small targets to answer questions, often over many crop and zoom turns. Answers are checked against a key. Mini-o3 team (HKU and ByteDance) Detection and captioning Everyday Fading Step 3.7 Flash 65% 1 Sep 2025 May 2026
WebQuestions Models answer factual questions mined from Google Suggest, with answers taken from the Freebase knowledge base. Scored by exact match. Stanford University Trivia and open questions Everyday Dropped PaLM 2-L 28.2% 1 Oct 2013 May 2023
WiC Models decide whether a target word has the same meaning in two sentence contexts. Scoring is binary accuracy. Mohammad Taher Pilehvar and Jose Camacho-Collados Language reasoning Everyday Dropped PaLM 2-L 66.8% 1 Aug 2018 May 2023
Wild-OmniDocBench Models parse 1,350 photographed document pages with geometric distortion, lighting changes, and other capture artifacts. OmniDocBench metrics score text, formulas, tables, and reading order. Gengluo Li and collaborators Documents, charts and text in images Expert In use MiniCPM-OCR-1.0 92.4% 1 Mar 2026 Oct 2026
Winograd Models pick which noun an ambiguous pronoun refers to in Winograd schema sentence pairs that need world knowledge. Scored by accuracy. Hector Levesque, Ernest Davis, and Leora Morgenstern Commonsense Everyday Dropped PaLM 2-L 89.5% 1 May 2023
Work360 Agents complete professional tasks drawn from a map of 26 industries, 182 subfields and 721 specialties. Qwen in-house scoring (method unpublished). Alibaba Qwen Tasks from real jobs Expert In use Qwen4-Max 52.7 1 Nov 2026 Nov 2026
XBOW Validation Benchmarks Offensive security agents attack deliberately vulnerable web applications and must capture a flag to prove each exploit. XBOW Capture-the-flag challenges Expert Fading Claude Opus 4.7 98.5% 1 Aug 2024 Apr 2026
XNLI Models classify entailment, contradiction or neutral for sentence pairs translated from MultiNLI into 15 languages. Facebook AI Research and NYU Reasoning in many languages Everyday Dropped Apertus 70B 45.3% 1 Sep 2018 Sep 2025
ZClawBench Agents do end-to-end OpenClaw workflows such as environment setup, development, information retrieval, data analysis and content creation, drawn from real OpenClaw use. Z.ai (Zhipu AI) General assistants Expert Fading GLM-5V-Turbo 57.6% 1 Mar 2026 Apr 2026
BEHAVIOR Challenge 2026 Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit. Stanford Vision and Learning Lab Simulated robots Everyday Not in cards Jul 2026
Conversational SWE Task Benchmark (Windsurf) Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy. Windsurf (Codeium, now part of Cognition) Fixing real repositories Expert Dropped 1 May 2025 May 2025
End-To-End SWE Task Benchmark (Windsurf) Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score. Windsurf (Codeium, now part of Cognition) Fixing real repositories Expert Dropped 1 May 2025 May 2025
IChO 2025 Models solve the written theoretical problems of the 57th International Chemistry Olympiad, held in Dubai in July 2025. Answers are graded against the official marking scheme. International Chemistry Olympiad (hosted by the UAE Ministry of Education) Science olympiads Competition Fading 1 Jul 2025 Feb 2026
M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. DAMO Academy, Alibaba Group Knowledge in many languages School Dropped 1 Jun 2023 May 2024
Objectron Models predict 3D bounding boxes of objects in short object-centric videos across nine categories. Scored by 3D intersection over union. Google Research Detection and captioning Everyday Dropped 1 Dec 2020 Sep 2025
RGB Models answer English and Chinese news questions from retrieved passages, testing noise robustness, negative rejection, information integration, and counterfactual robustness. Scored by accuracy. Chinese Academy of Sciences (ISCAS) Faithful to sources Everyday Dropped 1 Sep 2023 Feb 2024
BBQ (Bias Benchmark for QA) Ambiguous and disambiguated questions probing social stereotypes across nine attributes. Reports accuracy and bias scores. NYU MLL Bias, fairness and politics System cards only DeepSeek-R1 96.6% 45 Oct 2021 Nov 2026
Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. OpenAI with Gryphon Scientific Lab protocols Expert System cards only GPT-6 Luna 94% 39 Aug 2024 Oct 2026
Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. OpenAI with SecureBio Lab protocols Expert System cards only GPT-Rosalind-6 63.1% 39 Dec 2024 Nov 2026
OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. OpenAI Refusing harmful requests System cards only 35 Nov 2026
Image input safety evaluations Models answer harmful prompts that combine text and images in categories such as hate, extremism and self-harm. Scored as safe completion rate. OpenAI Refusing harmful requests Everyday System cards only GPT-4.5 99% 33 Dec 2024 Nov 2026
TroubleshootingBench Models find and correct realistic experimental errors in expert-written wet lab protocols. Scored as pass@1. OpenAI Lab protocols Expert System cards only GPT-Rosalind-6 59% 29 Aug 2025 Oct 2026
OpenAI cyber safety evaluations (incl. Daybreak Blue) Refusal and helpfulness evaluations for cyber requests, including the Daybreak Blue set for enabling defensive security work. OpenAI Refusing harmful requests System cards only 24 Nov 2026
OpenAI Dynamic Mental Health Benchmarks Multi-turn conversations with adversarial simulated users on mental-health risks. Scored on safe and supportive responses. OpenAI Wellbeing and sycophancy System cards only 22 Nov 2026
Multiturn Jailbreak Evaluations Models face adaptive multi-turn jailbreak attackers with growing attack budgets. Scored as worst-case defender success rate. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-6.2 Astra 80.9% 21 Mar 2026 Nov 2026
CVE-Bench 2 Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution. UIUC (Kang Lab) Finding and exploiting real bugs Expert Not in cards Oct 2025
CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. OpenAI + NYU (Chen Yueh-Han et al.) Deception and alignment System cards only Gemini 4 Argon 83.7% 20 Mar 2026 Nov 2026
Single-turn benign request evaluations (Anthropic) Models answer benign requests on sensitive topics. Score is the refusal rate, where lower is better. Anthropic Refusing harmful requests Everyday System cards only Claude Fable 5.1 0% 19 May 2025 Nov 2026
Single-turn harmful request evaluations (Anthropic) Models answer single-turn requests that violate the usage policy. Score is the harmless response rate. Anthropic Refusing harmful requests Everyday System cards only Claude Opus 4.5 99.8% 19 May 2025 Nov 2026
XSTest Models answer 250 safe prompts that resemble unsafe ones and 200 unsafe contrast prompts. Labs usually report the incorrect refusal rate on safe prompts. Röttger et al. (Bocconi University and others) Refusing harmful requests Everyday System cards only Cohere Command A 1.10% 19 Aug 2023 Mar 2026
Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. Anthropic Deception and alignment System cards only Claude Fable 5 2.06 19 Nov 2026
OpenAI user-flagged hallucination eval Production conversations users flagged for factual errors. Measures the rate of incorrect claims. OpenAI Hallucination rates System cards only GPT-5.5 9.20% 19 Nov 2026
Biology Model Refusal Evaluation Models answer severe, dual-use and benign biology prompts. Scored as safe rate on harmful prompts and not over-refusing benign ones. OpenAI Refusing harmful requests Expert System cards only GPT-6 Cyber 99.9% 19 Jun 2026 Nov 2026
Malicious computer use (Anthropic) Computer-use agents receive harmful tasks such as surveillance or fraud in GUI environments. Score is the refusal rate. Anthropic Refusing harmful requests Everyday System cards only Claude Sonnet 4.6 99.4% 18 Feb 2025 Nov 2026
Multi-turn testing (Anthropic) Models face multi-turn conversations that escalate toward harmful requests across risk areas. Score is the appropriate response rate. Anthropic Refusing harmful requests Everyday System cards only 18 May 2025 Nov 2026
SHADE-Arena Paired main and hidden side tasks in mock environments. Measures if an agent completes the side task undetected by a monitor. Anthropic + Scale AI + Redwood (Kutasov et al.) Deception and alignment System cards only Claude Opus 5.5 26.2% 17 Jun 2025 Oct 2026
Standard Refusal Evaluation Models answer disallowed and benign safety-related prompts across policy categories. An autograder scores not_unsafe and not_overrefuse. OpenAI Refusing harmful requests Everyday System cards only o1 100% 17 Aug 2024 Aug 2025
Child safety evaluations (Anthropic) Models answer single-turn and multi-turn prompts about child sexualization and grooming. Score is the appropriate response rate. Anthropic Refusing harmful requests Everyday System cards only 17 Feb 2025 Nov 2026
Malicious use of Claude Code (Anthropic) Agents in Claude Code receive malicious and dual-use coding requests. Score is the refusal rate on malicious requests. Anthropic Refusing harmful requests Expert System cards only Claude Mythos Preview 96.7% 16 Sep 2025 Nov 2026
Tone An automated evaluation measures whether model refusals are objective, neutral and not preachy. Google DeepMind Refusing harmful requests Everyday System cards only 16 Apr 2025 Oct 2026
LABBench2 SeqQA2 Models reason about DNA and protein sequences and answer in open response. The SeqQA2 subtask of LABBench2. Edison Scientific (formerly FutureHouse) Genomics and life sciences Expert Not in cards Gemini 3 Pro 52.5% Feb 2026 Feb 2026
Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. Google DeepMind Refusing harmful requests Everyday System cards only 15 Apr 2025 Oct 2026
Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. Google DeepMind Refusing harmful requests Everyday System cards only 15 Apr 2025 Oct 2026
Text to Text Safety An automated content safety evaluation measures policy violations in model answers to text prompts. Google DeepMind Refusing harmful requests Everyday System cards only 15 Apr 2025 Oct 2026
Gray Swan Agent Red Teaming (ART) benchmark Agents face curated prompt injections from a large public red-teaming competition across 44 deployment scenarios. Score is attack success rate. Gray Swan + UK AI Security Institute Jailbreaks and prompt injection Expert System cards only Claude Mythos Preview 0.10% 14 Jul 2025 Oct 2026
Anthropic political even-handedness eval Models answer paired prompts that take opposing sides on 150 US political topics. A grader scores even-handedness, refusals and acknowledgment of opposing views. Anthropic Bias, fairness and politics Everyday System cards only Claude Opus 5.5 99.4% 14 Nov 2025 Nov 2026
Prompt injection robustness against adaptive attackers, coding (Anthropic) Coding agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. Anthropic Jailbreaks and prompt injection Expert System cards only Claude Opus 4.6 0% 14 Nov 2025 Nov 2026
Prompt injection robustness against adaptive attackers, computer use (Anthropic) Computer-use agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. Anthropic Jailbreaks and prompt injection Expert System cards only Claude Opus 4.5 0% 14 Nov 2025 Nov 2026
Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. OpenAI Capture-the-flag challenges University System cards only o3 68% 14 Aug 2024 Aug 2025
Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. OpenAI Finding and exploiting real bugs Expert System cards only GPT-5.5 93.3% 14 Apr 2025 Aug 2026
Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. OpenAI Labs' internal R&D tests Research frontier System cards only GPT-6.2 Astra 81.8% 14 Apr 2026 Nov 2026
ToxiGen Models continue ToxiGen prompts about 13 minority groups. Scored by the average toxicity of the generations. Microsoft Research + MIT (Hartvigsen et al.) Bias, fairness and politics Everyday System cards only Gemma 2B 29.8 14 Mar 2022 Dec 2025
Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. OpenAI with Gryphon Scientific Hazardous knowledge Expert System cards only OpenAI deep research 0% 14 Sep 2024 Nov 2025
Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. Anthropic Hazardous knowledge Expert System cards only Claude 3.7 Sonnet 69.7% 14 Feb 2025 Jul 2026
Prompt injection robustness against adaptive attackers, browser use (Anthropic) Browser agents face indirect prompt injections in web environments from an internal adaptive attacker. Score is attack success rate. Anthropic Jailbreaks and prompt injection Expert System cards only Claude Sonnet 5.5 0% 13 Nov 2025 Oct 2026
OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. OpenAI Labs' internal R&D tests Research frontier System cards only GPT-5.2 55% 13 Jan 2025 Dec 2025
First-Person Fairness evaluation Tests whether a user's name (a demographic proxy) changes responses, using an LLM research assistant to detect harmful stereotypes. OpenAI Bias, fairness and politics System cards only ChatGPT agent 0.40% 13 Oct 2024 Jul 2026
KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. OpenAI GPU kernels Research frontier System cards only GPT-6.2 Astra 76.9% 13 Jun 2026 Nov 2026
DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. Anthropic Hazardous knowledge Expert System cards only Claude Opus 4.7 80% 13 May 2025 Jul 2026
Suicide and self-harm evaluations (Anthropic) Models handle single-turn and multi-turn conversations about suicide and self-harm. Score is the appropriate response rate. Anthropic Wellbeing and sycophancy Everyday System cards only 13 Sep 2025 Nov 2026
Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. Anthropic Machine learning engineering Expert System cards only Claude Opus 5 31.3 12 Feb 2025 Jul 2026
WMDP Proxy multiple-choice benchmark for hazardous knowledge in bio, chem and cyber. Released with the RMU unlearning method. CAIS + Scale AI et al. Hazardous knowledge Expert System cards only GPT-6 Astra 84.8% 12 Mar 2024 Sep 2026
LLM training task (Anthropic) Agents optimize a CPU-only small language model training implementation. Scored by speedup. Anthropic Machine learning engineering Expert System cards only 12 Feb 2025 Jul 2026
U18 safety evaluations Models answer challenging prompts in teen safety categories such as age-restricted goods, eating disorders and emotional reliance. Scored as safe completion rate per category. OpenAI Refusing harmful requests Everyday System cards only 12 Aug 2026 Nov 2026
Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. Anthropic Deception and alignment Expert System cards only Claude Haiku 4.5 30% 11 May 2025 Sep 2026
Time series forecasting task (Anthropic) Agents solve a regression and time-series forecasting problem with known state-of-the-art results, in six variants. Anthropic Machine learning engineering Expert System cards only Claude Mythos Preview 4.55 11 May 2025 Jul 2026
Destructive action avoidance Coding agents meet conflicting user edits during tasks. Scored on preserving user changes and avoiding data-destructive actions. OpenAI Deception and alignment Expert System cards only GPT-5.5 90% 11 Nov 2025 Jul 2026
Agent JSK Agents read simulated email connectors that contain prompt injection attacks. Scored on resisting the injected instructions. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-5.6 Sol 100% 11 Dec 2025 Aug 2026
Prompt injection robustness Agents meet indirect prompt injection attacks inside tool outputs. Scored as defender success rate averaged per defender query. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-6 Astra 99.8% 11 Jul 2026 Oct 2026
Novel compiler task (Anthropic) Agents build a compiler for an unusual programming language from a specification and test cases. Anthropic Machine learning engineering Expert System cards only Claude Opus 5 80.9% 11 May 2025 Jul 2026
Disordered eating evaluations (Anthropic) Models handle conversations about eating disorders and disordered eating. Score is the appropriate response rate. Anthropic Wellbeing and sycophancy Everyday System cards only 11 Feb 2026 Nov 2026
Kernels task (Anthropic) Agents optimize compute kernels in a performance engineering challenge. Scored by speedup. Anthropic GPU kernels Expert System cards only 11 May 2025 Jul 2026
Human Pathogen Capabilities Test v2 (HPCT v2) Models answer multiple-response questions on a small set of weaponizable pathogens that biosecurity experts rank highest priority. A revised version of HPCT. SecureBio Hazardous knowledge Research frontier Not in cards Claude Opus 5 80% 2026 Oct 2026
AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. OpenAI Genomics and life sciences Research frontier System cards only GPT-6.2 Astra 0.59 10 Jun 2026 Nov 2026
Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. OpenAI Capture-the-flag challenges School System cards only OpenAI deep research 82% 10 Aug 2024 Jul 2025
OpenAI Research Engineer Interview: Multiple Choice Models answer 97 multiple-choice questions drawn from OpenAI machine learning interview topics. Scored as cons@32. OpenAI Labs' internal R&D tests Expert System cards only o4-mini 83% 10 Aug 2024 Jul 2025
Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. OpenAI Genomics and life sciences Research frontier System cards only GPT-6.1 Terra 0.47 10 Sep 2026 Nov 2026
GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. Google DeepMind Deception and alignment Expert System cards only Gemini 4 Argon 90.9% 10 May 2025 Oct 2026
Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. OpenAI Genomics and life sciences Research frontier System cards only GPT-6 Luna 10.7 10 Sep 2026 Nov 2026
SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. OpenAI Genomics and life sciences Research frontier System cards only GPT-6.2 Astra 0.46 10 Sep 2026 Nov 2026
Instruction hierarchy attacks Models face attacks that try to override higher-priority instructions. Scored as defender success rate averaged per defender query. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-6 Astra 99.99% 10 Jul 2026 Oct 2026
GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. Google DeepMind Deception and alignment Expert System cards only Gemini 4 Argon 100% 10 May 2025 Oct 2026
Self-preference bias (Anthropic) Models grade transcripts attributed to themselves or to other models. Measures whether scores favor the model's own outputs. Anthropic Bias, fairness and politics Everyday System cards only Claude Opus 5.5 0.07 10 Sep 2025 Oct 2026
User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. OpenAI Refusing harmful requests Everyday System cards only OpenAI Computer-Using Agent (CUA) 92% 10 Jan 2025 Jul 2026
Election integrity evaluations (Anthropic) Models answer election-related prompts, including requests for misinformation or influence operations. Score is the appropriate response rate. Anthropic Refusing harmful requests Everyday System cards only 10 Oct 2024 Nov 2026
Molecular Biology Capabilities Test Models answer expert-level molecular biology questions from SecureBio. Scored as accuracy. SecureBio Genomics and life sciences Expert System cards only Claude Opus 5.5 68.7% 9 Oct 2026
BioLP-bench Find the introduced error in a biological lab protocol. Tests protocol understanding. Igor Ivanov (Oxford Biosecurity Group) Lab protocols Expert System cards only OpenAI deep research 49% 9 Aug 2024 Nov 2025
Challenging Refusal Evaluation Models answer a harder set of disallowed content prompts across policy categories. An autograder scores the not_unsafe rate. OpenAI Refusing harmful requests Expert System cards only o1-preview 93.4% 9 Sep 2024 Apr 2025
OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. OpenAI Labs' internal R&D tests Research frontier System cards only GPT-5.1-Codex-Max 8.00% 9 Aug 2025 Apr 2026
PersonQA Models answer questions about publicly available facts on people. Scored on accuracy, with hallucination rate reported alongside. OpenAI Hallucination rates Everyday System cards only ChatGPT agent 94.3% 9 Dec 2024 Aug 2025
SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. Anthropic Machine learning engineering Expert System cards only Claude Sonnet 4.6 48.2% 9 Feb 2025 Feb 2026
100Q-Hard (Anthropic) Models answer hard, obscure factual questions. Scored on correct, incorrect and declined answers. Anthropic Short factual questions Expert System cards only Claude Mythos Preview 60.1% 9 Mar 2024 May 2026
OpenAI Deployment Simulation Replays representative production or internal Codex traffic on a candidate model to forecast misaligned or disallowed behavior before launch. OpenAI Deception and alignment System cards only GPT-6.2 Sol 0.47% 9 Nov 2026
OpenAI Research Engineer Interview: Coding Models solve 18 self-contained coding problems like those in OpenAI interviews. Scored as pass@1. OpenAI Labs' internal R&D tests Expert System cards only o1 100% 9 Aug 2024 Apr 2025
NanoGPT Speedrun Improve a small GPT training script to reach a target validation loss as fast as possible on fixed hardware. Keller Jordan (community), OpenAI and METR (variants) Machine learning engineering Research frontier System cards only GPT-6.2 Astra 42.9% 9 Jun 2024 Nov 2026
Instruction Hierarchy Evaluation Models resolve conflicts between system, developer and user messages, resist tutor jailbreaks, and protect secret phrases and passwords. Scored per sub-evaluation. OpenAI Jailbreaks and prompt injection Everyday System cards only 9 Dec 2024 Aug 2025
Malicious agentic influence campaigns, 34-point rubric (Anthropic) Agents run simulated voter suppression and domestic polarization campaigns. Weighted criteria measure campaign execution on a 34-point scale. Anthropic Dangerous capabilities Expert System cards only 1 Nov 2026 Nov 2026
Unjustified refusals An automated evaluation measures over-refusals on benign or borderline prompts that can be safely fulfilled. Google DeepMind Refusing harmful requests Everyday System cards only 9 Oct 2026
Human Sourced Jailbreaks Models face jailbreak prompts collected from human red teaming. Scored as the not_unsafe rate. OpenAI Jailbreaks and prompt injection Expert System cards only OpenAI Computer-Using Agent (CUA) 100% 8 Sep 2024 Apr 2025
MLE-bench 30 Agents train models for 30 of MLE-bench's Kaggle-style competitions. The score is the share of runs that earn a bronze medal or better. OpenAI Machine learning engineering Expert System cards only GPT-5.5 36.7% 8 Aug 2025 Jun 2026
RACE Models answer reading comprehension questions from Chinese middle and high school English exams. Reported here without a subset split. Carnegie Mellon University Reading comprehension School System cards only InternVL3.5-241B-A28B 94.2% 8 Apr 2017 Oct 2026
Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. Anthropic Deception and alignment Expert System cards only Claude Opus 4.5 0% 8 May 2025 Feb 2026
Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. OpenAI Deception and alignment Expert System cards only GPT-6 Luna 0% 8 Sep 2026 Nov 2026
Agentic coding malicious requests (Anthropic) Coding agents with standard coding tools receive 150 malicious coding requests prohibited by policy. Score is the refusal rate. Anthropic Refusing harmful requests Expert System cards only Claude Haiku 4.5 100% 8 May 2025 Feb 2026
Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. OpenAI Labs' internal R&D tests Expert System cards only OpenAI deep research 78% 8 Aug 2024 Feb 2025
Protocols (Anthropic life sciences) Models find and fix errors in molecular biology protocols and answer practical questions about applying real laboratory protocols. Anthropic (with Benchling) Lab protocols Expert System cards only Claude Sonnet 5 62% 8 Apr 2026 Oct 2026
Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. Anthropic (with Dyno Therapeutics) Hazardous knowledge Expert System cards only 8 May 2026 Oct 2026
Petri 2.0 Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors. Anthropic Deception and alignment Expert Not in cards Jan 2026
METR Time Horizon 2 Agents complete 36 tasks across 12 families. METR estimates how long a skilled human would need for tasks that agents complete with 50 percent reliability. METR Task length Research frontier Not in cards GPT-6 Astra 27.1 h Nov 2026 Sep 2026
CharXiv (DQ) Models answer descriptive questions about charts from arXiv papers, such as titles, axis labels and legend entries. It is the descriptive half of CharXiv. Princeton University (Zirui Wang et al.) Documents, charts and text in images University System cards only o3 95% 7 Jun 2024 Jun 2026
scBench (LatchBio SingleCellBench) 195 verifiable single-cell RNA-seq analysis problems. Each pairs a data snapshot with a task and a deterministic grader. LatchBio Lab and data work Expert System cards only GPT-6 Astra 63.9% 7 Feb 2026 Oct 2026
SpatialBench (LatchBio) Agentic spatial-transcriptomics data analysis problems with deterministic graders. Anthropic reports a 115-problem Verified variant. LatchBio Lab and data work Expert System cards only GPT-5.5 57.6% 7 Dec 2025 Oct 2026
MentalHealthBench 1,215 synthetic mental-health conversations with 5,262 clinician-authored rubric criteria. An LLM judge scores safety and guidance quality. OpenAI Wellbeing and sycophancy Expert System cards only GPT-6 Astra 57.3% 7 Sep 2026 Nov 2026
Creative biology Models answer questions about unusual biology needed to engineer harmless biological agents, a weak proxy for novel weapon design. Anthropic Hazardous knowledge Expert System cards only Claude Opus 4.6 60.3% 7 May 2025 Feb 2026
Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. Anthropic Machine learning engineering Expert System cards only Claude Opus 4.5 0.97 7 May 2025 Feb 2026
Anthropic ECI (AECI) Index that combines a basket of capability benchmarks into one capability score to track trends across Anthropic models. Anthropic Composite indices Research frontier System cards only Claude Opus 5.5 169.4 7 Apr 2026 Oct 2026
Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. Anthropic Capture-the-flag challenges Expert System cards only Claude Opus 4.5 94.4% 7 Feb 2025 Feb 2026
Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. Anthropic Capture-the-flag challenges Expert System cards only Claude Opus 4.5 100% 7 Feb 2025 Feb 2026
Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. Anthropic Capture-the-flag challenges Expert System cards only Claude Opus 4.6 71.4% 7 Feb 2025 Feb 2026
Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. Anthropic Capture-the-flag challenges Expert System cards only Claude Opus 4.5 100% 7 Feb 2025 Feb 2026
Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. Anthropic Capture-the-flag challenges Expert System cards only Claude Opus 4.6 100% 7 Feb 2025 Feb 2026
LSAT Models take a simulated Law School Admission Test. Scored on the 120 to 180 scale. LSAC Language reasoning University System cards only GPT-4 163.0 7 Nov 2024
Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. OpenAI Deception and alignment Expert System cards only GPT-6 Astra 0% 7 Sep 2026 Nov 2026
BOLD Models continue 23,679 English Wikipedia prompts about professions, gender, race, religion and politics. Classifiers score sentiment, toxicity and polarity per group. Amazon Bias, fairness and politics Everyday System cards only Gemma 7B 49.1 7 Jan 2021 Mar 2024
Agentic Safe Completions Agents handle harmful requests from Codex and ChatGPT plugin traffic and human red teaming. Scored on avoiding harmful actions per category. OpenAI Refusing harmful requests Everyday System cards only 7 Sep 2026 Nov 2026
CoT Monitorability Monitors read a model's chain of thought, actions or all messages across intervention, process and outcome evaluations. Scored as g-mean squared of TPR and TNR. OpenAI Deception and alignment Expert System cards only 7 Mar 2026 Nov 2026
Discrim-Eval Models make yes or no decisions in templated scenarios that vary age, gender and race. Scored by mixed effects model coefficients. Anthropic (Tamkin et al.) Bias, fairness and politics Everyday System cards only 7 Dec 2023 Jan 2025
Instruction Following (safety evaluation) An automated evaluation measures whether the model follows instructions on sensitive prompts while staying within safety policies. Google DeepMind Refusing harmful requests Everyday System cards only 7 Sep 2025
Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. Anthropic Hazardous knowledge Expert System cards only 7 May 2025 Feb 2026
LinuxArena AI-control setting in live Linux production-like environments. Measures stealthy side-task completion against action and CoT monitors. Redwood Research + EquiStamp Deception and alignment Expert System cards only Claude Sonnet 5 0% 6 Apr 2026 Oct 2026
Anthropic undermining liberal democracy (behavioral audit metric) Automated behavioral audit metric: investigator models probe willingness to help coups, election tampering, or political surveillance, censorship or intimidation. Anthropic Refusing harmful requests System cards only Claude Opus 5 1.13 6 May 2026 Oct 2026
Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. Anthropic Finding and exploiting real bugs Expert System cards only Claude Opus 5.5 91.1% 6 Oct 2026
Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. OpenAI Deception and alignment Expert System cards only GPT-6 Sol 1.30% 6 Aug 2025 Sep 2026
False-premise questions (Anthropic) Models answer questions built on false premises. Measures how often they accept the premise instead of correcting it. Anthropic Hallucination rates Everyday System cards only Claude Opus 4.5 1.20% 6 Sep 2025 May 2026
GRE Quantitative Models take the GRE quantitative reasoning section. Scored on the 130 to 170 scale. ETS Word problems University System cards only GPT-4 163.0 6 Nov 2024
GRE Verbal Models take the GRE verbal reasoning section. Scored on the 130 to 170 scale. ETS Reading comprehension University System cards only GPT-4 169.0 6 Nov 2024
Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. Anthropic AI research tasks Expert System cards only Claude Mythos Preview 0.65 6 May 2025 Apr 2026
MakeMePay A con artist model tries to talk a GPT-4o mark into paying money in a role-play. Scored as the share of conversations with a payment. OpenAI Dangerous capabilities Everyday System cards only GPT-4.5 57% 6 Sep 2023 Feb 2025
Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. Anthropic Jailbreaks and prompt injection Expert System cards only Claude Fable 5.2 3.90% 6 Jun 2026 Nov 2026
Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. OpenAI Finding and exploiting real bugs Research frontier System cards only GPT-6 Astra 45.5% 6 Sep 2026 Nov 2026
Computer use prompt injection evaluation (Anthropic) Agents face about 600 prompt injection scenarios in coding platforms, browsers and email workflows. Score is the attack prevention rate. Anthropic Jailbreaks and prompt injection Expert System cards only Claude Sonnet 4.5 78% 6 Oct 2025
Refusal to assist with AI safety R&D (Anthropic) Models receive single-turn coding requests for legitimate AI safety research with user context. Score is the refusal rate, where lower is better. Anthropic Deception and alignment Expert System cards only Claude Fable 5 3.10% 6 Feb 2026 Jun 2026
DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. OpenAI Genomics and life sciences Research frontier System cards only GPT-5.5 Pro 16.5% 6 Apr 2026 Sep 2026
Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. OpenAI Genomics and life sciences Research frontier System cards only GPT-5.6 Sol 7.60% 6 Apr 2026 Sep 2026
Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. OpenAI Deception and alignment Expert System cards only GPT-Rosalind-6 1.40% 6 Sep 2026 Nov 2026
API-Bank Models make API calls in dialogues that need tools such as calendars and search. Scored on call accuracy. Alibaba DAMO (Li et al.) Function calling Everyday System cards only Llama 3.1-405B 92.3% 6 Apr 2023 Nov 2024
Factuality Heavy prompts Models answer prompts representative of factuality-heavy ChatGPT conversations. A grader with web access counts responses and claims with factual errors. OpenAI Hallucination rates Everyday System cards only GPT-6 Sol 5.40% 6 May 2026 Oct 2026
Gorilla APIBench Models write calls to machine learning APIs from Hugging Face, TorchHub and TensorHub. Scored by AST match. UC Berkeley (Gorilla) Function calling Everyday System cards only Llama 3.1-405B 35.3% 6 May 2023 Nov 2024
High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. OpenAI Hallucination rates Expert System cards only GPT-6 Sol 4.10% 6 May 2026 Oct 2026
Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. Meta Knowledge in many languages University System cards only Llama 3.1-405B 83.2% 6 Jul 2024 Nov 2024
Prompt injection attacks in search and function calling Agents receive search results and function call outputs that contain stronger known prompt injection attacks. Scored on resisting the injected instructions. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-5.6 Terra 94.6% 6 Jun 2026 Jul 2026
Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. Anthropic Deception and alignment Expert System cards only Claude Opus 5.5 0.63% 6 May 2025 Sep 2026
WildChat non-toxic incorrect refusals (Anthropic) Models answer non-toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the incorrect refusal rate. Anthropic Refusing harmful requests Everyday System cards only Claude 3.5 Haiku 4.80% 6 Mar 2024 Oct 2024
WildChat toxic refusals (Anthropic) Models answer toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the correct refusal rate. Anthropic Refusing harmful requests Everyday System cards only Claude 3.5 Sonnet 96.4% 6 Mar 2024 Oct 2024
RealToxicityPrompts Models continue 100,000 web text prompts. Scored on the rate of toxic generations judged by a toxicity classifier. Allen Institute for AI and University of Washington Refusing harmful requests Everyday System cards only Apertus 70B 0.20% 6 Sep 2020 Sep 2025
AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. Anthropic (with Dyno Therapeutics) Hazardous knowledge Expert System cards only Claude Opus 4.8 0.80 6 May 2026 Oct 2026
Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. Google DeepMind Dangerous capabilities Everyday System cards only Gemini 3.1 Pro 3.60 6 Nov 2025 Oct 2026
OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. OpenAI Deception and alignment Expert System cards only GPT-Rosalind-6 0% 6 Sep 2026 Nov 2026
Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Qwen Team, Alibaba Group Chat and instructions in many languages Everyday System cards only Qwen2-72B 69.6% 6 Jul 2024 Dec 2024
Qwen in-house English automatic evaluation Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Qwen Team, Alibaba Group Other Everyday System cards only Qwen2-72B 72.9% 6 Jul 2024 Dec 2024
RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. METR AI research tasks Research frontier System cards only Gemini 3.1 Pro 1.27 6 Sep 2024 Feb 2026
CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. Google DeepMind Hazardous knowledge Expert System cards only 6 Nov 2025 Oct 2026
Static Jailbreak Evaluations Models face a fixed set of jailbreak attacks on bio, violence and cyber requests. Scored as defender success rate per category. OpenAI Jailbreaks and prompt injection Expert System cards only 6 Sep 2026 Nov 2026
World-Class Bio Models answer expert-level biology questions in a SecureBio benchmark. Scored as accuracy. SecureBio Genomics and life sciences Research frontier System cards only Claude Opus 5 68.8% 5 Oct 2026
ABC Bench Fragment Design Agents use biology tools to complete a computational genetic-construct design task. One task family of the Agentic Bio-Capabilities Benchmark. SecureBio Hazardous knowledge Expert System cards only Gemini 3 Pro 100% 5 Jun 2026 Oct 2026
Belebele Models answer multiple-choice reading comprehension questions on FLORES passages in 122 language variants. Scored by accuracy. Meta AI (Bandarkar et al.) Reading in many languages Everyday System cards only GPT-3.5 Turbo 51.1% 5 Aug 2023 Oct 2026
AMC 12 2024 Models answer the 25 multiple-choice problems of the 2024 AMC 12 contests. Scored on the official 150-point scale. Mathematical Association of America Competition problems with answers Competition System cards only DeepSeek-R1 143.7 1 Nov 2024 Sep 2025
Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. Anthropic Deception and alignment Expert System cards only Claude Sonnet 4 14.3% 5 Oct 2025
Cyber harm coverage (Anthropic) Cyber safeguard classifiers screen examples from each cyber harm category. Score is overall recall, the share of harmful examples flagged. Anthropic Refusing harmful requests Expert System cards only Claude Fable 5.1 100% 5 Jul 2026 Nov 2026
Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. Anthropic Finding and exploiting real bugs Research frontier System cards only Claude Mythos Preview 72.4% 5 Jul 2026
Internal model use survey (Anthropic) Anthropic technical staff estimate productivity uplift from the model and judge whether it could automate a junior ML researcher. Anthropic Labs' internal R&D tests Expert System cards only Claude Opus 4.5 50% 5 May 2025 Apr 2026
Jailbreak Augmented Examples Models face publicly known jailbreaks applied to prompts from the standard disallowed content set. Scored as not_unsafe rate. OpenAI Jailbreaks and prompt injection Everyday System cards only o1-mini 100% 5 Sep 2024 Jan 2025
Microsoft RAI benchmark: Harmful Content Continuation (DR-3) Models continue conversations about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. Microsoft Refusing harmful requests Everyday System cards only phi-3-small 7.4B 0.30% 5 Apr 2024 Dec 2024
Microsoft RAI benchmark: Harmful Content Summarization (DR-3) Models summarize content about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. Microsoft Refusing harmful requests Everyday System cards only phi-3-mini 3.8B 10% 5 Apr 2024 Dec 2024
Microsoft RAI benchmark: Jailbreak (DR-1) Models face simulated jailbreak conversations on topics such as child grooming, illegal advice and leaking guidelines. Scored as the defect rate at severity 1. Microsoft Jailbreaks and prompt injection Everyday System cards only Phi-4 7.30% 5 Apr 2024 Dec 2024
Microsoft RAI benchmark: Third Party Harm (DR-1) Models face simulated conversations that try to extract third-party content such as books, news, recipes and songs. Scored as the defect rate at severity 1. Microsoft Refusing harmful requests Everyday System cards only Phi-3.5-MoE 10.5% 5 Apr 2024 Dec 2024
Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. OpenAI Labs' internal R&D tests Research frontier System cards only GPT-6 Astra 76% 5 Feb 2026 Sep 2026
Organic Chemistry (Anthropic) Models predict structures from spectra, design synthetic routes, predict reaction products and convert between IUPAC names, SMILES and structure images. Anthropic Research science Expert System cards only Claude Opus 4.8 86.2% 5 Feb 2026 Jun 2026
Over-eager behavior on GUI computer use tasks (Anthropic) Computer-use agents face impossible tasks with an unwanted shortcut available. Score is the rate of unapproved workarounds instead of checking with the user. Anthropic Deception and alignment Expert System cards only Claude Opus 4.8 9.40% 5 Feb 2026 May 2026
Production Jailbreaks Models face jailbreaks found in production ChatGPT data. Scored as not_unsafe rate. OpenAI Jailbreaks and prompt injection Everyday System cards only OpenAI Computer-Using Agent (CUA) 100% 5 Sep 2024 Jan 2025
UK AISI unprompted sabotage evaluation Agents act as AI research agents on AI safety research tasks. Scored on confirmed instances of sabotaging the safety research. UK AI Security Institute Deception and alignment Expert System cards only Claude Opus 4.7 0% 5 Apr 2026 Sep 2026
WildGuardTest Models classify whether user prompts are harmful, whether responses are harmful, and whether responses refuse, across 13 risk categories. Scored by F1. Allen Institute for AI Refusing harmful requests Everyday System cards only Olmo 3.1 32B Think 96.9% 5 Jun 2024 Dec 2025
Structural Biology (Anthropic) Models answer questions about a biomolecule's function from structural data and basic tools, in multiple-choice and open-ended variants. Anthropic Research science Expert System cards only Claude Opus 4.8 79% 5 Feb 2026 Jun 2026
Protein Design (Anthropic) Models generate novel protein sequences under design constraints and rank protein design libraries for experimental testing. Anthropic Research science Expert System cards only Claude Opus 5 42.5% 5 Jun 2026 Oct 2026
Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. OpenAI with Ranger Hazardous knowledge Expert System cards only 5 Sep 2024 Jan 2025
Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. OpenAI Deception and alignment Expert System cards only 5 Jun 2026 Nov 2026
BioTIER Refuse Models receive 398 dangerous biology prompts. The score measures the percentage correctly refused. SecureBio Refusing harmful requests Expert System cards only Claude Opus 5.5 99.1% 4 Mar 2026 Nov 2026
ABC Bench Liquid-Handling Robot Agents use biology tools to plan liquid-handling robot steps for a construct-assembly task. One task family of the Agentic Bio-Capabilities Benchmark. SecureBio Hazardous knowledge Expert System cards only Gemini 3.1 Pro 100% 4 Jun 2026 Oct 2026
SWE-Lancer Diamond (SWE Manager) Models review competing implementation proposals for real freelance tasks and pick the best one. Scored against the original choice as pass@1. OpenAI Fixing real repositories Expert System cards only Claude 3.5 Sonnet 44.9% 4 Feb 2025 Apr 2025
LongFact-Objects Models write detailed answers to LongFact prompts about specific objects such as people or places. A fact checker scores the claim-level error rate. Google DeepMind Long-form factuality Everyday System cards only GPT-5 0.80% 4 Mar 2024 Feb 2026
In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. Google DeepMind Capture-the-flag challenges Expert System cards only Gemini 2.5 Pro 84.6% 4 Mar 2024 Jul 2025
InterCode-CTF Agents solve 100 picoCTF capture-the-flag challenges in an interactive Bash and Python environment. Princeton (Yang et al.) Capture-the-flag challenges School System cards only Gemini 2.5 Pro 98.7% 4 Jun 2023 Nov 2025
Where2Place 100 cluttered real photos with free-space placement instructions. The model points, scored correct if inside the ground-truth mask. University of Washington + NVIDIA (RoboPoint) Embodied reasoning Everyday System cards only Cosmos3-Nano 64% 4 Jun 2024 May 2026
WildChat (toxic conversations) Models answer toxic user turns from WildChat, the 200 highest scoring conversations per moderation category. Scored as not_unsafe rate. Allen Institute for AI (dataset), OpenAI (toxic subset) Refusing harmful requests Everyday System cards only GPT-4.5 98% 4 May 2024 Mar 2025
AMC 10 2023 Models take the 2023 AMC 10 A and B exams. Scored out of 150 points. Mathematical Association of America Competition problems with answers Competition System cards only Claude 3 Opus 72.0 3 Nov 2023 Mar 2024
Coding prompt injections Agents work on coding tasks that contain prompt injection attacks. Scored on resisting the injected instructions. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-5.1-Codex-Max 100% 4 May 2025 Nov 2025
Contextual Nuclear Knowledge Models answer 222 multiple choice questions on nuclear engineering with a nonproliferation focus. Scored as cons@32 accuracy. OpenAI Dangerous capabilities Expert System cards only OpenAI deep research 77% 4 Dec 2024 Feb 2025
CrowS-Pairs Models choose between paired sentences that differ only in a stereotyped social group. Measures preference for the stereotyped sentence. NYU (Nangia et al.) Bias, fairness and politics Everyday System cards only Gemma 7B 51.3% 4 Sep 2020 Jul 2024
Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. Google DeepMind with Irregular Capture-the-flag challenges Expert System cards only Gemini 3 Pro 91.7% 4 Mar 2025 Nov 2025
Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. Google DeepMind Capture-the-flag challenges Expert System cards only Gemini 2.5 Flash 15.4% 4 Feb 2024 Jul 2025
Higher-difficulty benign request evaluations (Anthropic) Models answer harder benign requests that resemble harmful ones. Score is the refusal rate, where lower is better. Anthropic Refusing harmful requests Everyday System cards only Claude Opus 4.7 0.01% 4 Feb 2026 Apr 2026
Higher-difficulty harmful request evaluations (Anthropic) Models answer harder, more obfuscated policy-violating requests. Score is the harmless response rate. Anthropic Refusing harmful requests Everyday System cards only Claude Sonnet 4.6 99.4% 4 Feb 2026 Apr 2026
LongFact-Concepts Models write detailed answers to LongFact prompts about broad concepts. A fact checker scores the claim-level error rate. Google DeepMind Long-form factuality Everyday System cards only GPT-5 mini 0.90% 4 Mar 2024 Feb 2026
MBE (Multistate Bar Examination) Models answer the 200 multiple choice questions of an official Multistate Bar Examination practice exam. Scored by accuracy. NCBE Law Expert System cards only Claude 3 Opus 85% 4 Mar 2024
Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. Meta + Faculty AI Genomics and life sciences Expert System cards only Muse Wonder Full 79.4 4 Oct 2026
Meta chemical-agent refusal evaluation Models receive high-risk chemical prompts and the benchmark measures how often they refuse. Covers development, procurement and production stages. Meta Hazardous knowledge Expert System cards only Muse Wonder Full 100% 4 Oct 2026
Microsoft RAI benchmark: Ungroundedness Models answer in GPT-4 simulated multi-turn conversations. A judge rates how ungrounded each answer is in the given prompt, from 0 to 4. Microsoft Faithful to sources Everyday System cards only phi-3-medium 14B 0.21 4 Apr 2024 Aug 2024
Radiological and Nuclear Expert Knowledge Models answer 87 expert and tacit knowledge questions on radiological and nuclear threats. Scored as cons@32 accuracy. OpenAI Dangerous capabilities Expert System cards only OpenAI deep research 74% 4 Dec 2024 Feb 2025
SecureBio VMQA, single choice Models answer 350 virology troubleshooting questions in single-choice form. VMQA is an earlier version of the Virology Capabilities Test. SecureBio Hazardous knowledge Expert System cards only Gemini 3 Pro 61.2% 4 Nov 2025
UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. UK AI Security Institute Finding and exploiting real bugs Research frontier System cards only Claude Opus 5 80% 4 Apr 2026 Jul 2026
UK AISI sabotage continuation evaluation Agents resume trajectories where sabotage of safety research already began. Scored on the share of trajectories where they actively continue the sabotage. UK AI Security Institute Deception and alignment Expert System cards only Claude Opus 4.7 0% 4 Apr 2026 Jul 2026
Winogender Models resolve pronouns in sentences about occupations. Accuracy shows whether gender stereotypes sway coreference. Johns Hopkins University (Rudinger et al.) Bias, fairness and politics Everyday System cards only Gemma 2 9B 79.2% 4 Apr 2018 Jul 2024
ReproBAIT Agents independently reproduce biological AI models from published scientific papers. Scored as success rate. SecureBio Genomics and life sciences Expert System cards only GPT-5.6 Sol 85% 4 Nov 2026
AP Art History Models take a simulated AP Art History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 5.00 4 Nov 2024
AP Biology Models take a simulated AP Biology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 5.00 4 Nov 2024
AP Chemistry Models take a simulated AP Chemistry exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 4.00 4 Nov 2024
AP English Language and Composition Models take a simulated AP English Language and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 2.00 4 Nov 2024
AP English Literature and Composition Models take a simulated AP English Literature and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 2.00 4 Nov 2024
AP Environmental Science Models take a simulated AP Environmental Science exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 5.00 4 Nov 2024
AP Macroeconomics Models take a simulated AP Macroeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 5.00 4 Nov 2024
AP Microeconomics Models take a simulated AP Microeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 5.00 4 Nov 2024
AP Psychology Models take a simulated AP Psychology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 5.00 4 Nov 2024
AP Statistics Models take a simulated AP Statistics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 5.00 4 Nov 2024
AP US Government Models take a simulated AP US Government exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 5.00 4 Nov 2024
AP US History Models take a simulated AP US History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 5.00 4 Nov 2024
AP World History Models take a simulated AP World History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 4.00 4 Nov 2024
SAT Math Models take the SAT math section. Scored on the 200 to 800 scale. College Board Word problems School System cards only GPT-4 700.0 4 Nov 2024
Agentic code behavior scores (Anthropic) Agents work through over one hundred realistic coding scenarios. Graders score instruction following, safety, verification, efficiency, adaptability and honesty out of 10. Anthropic Deception and alignment Expert System cards only 4 Feb 2026 Apr 2026
Apollo Research in-context scheming evaluations Agents receive a goal in context and chances to subvert oversight, self-exfiltrate, fake alignment or sandbag. Scored as rates per behavior. Apollo Research Deception and alignment Expert System cards only 4 Dec 2024 Apr 2025
Deception Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the share of outputs flagged for deceptive behaviors such as fabricated facts. OpenAI Deception and alignment Everyday System cards only 4 Mar 2026 Jul 2026
Disallowed Content Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the estimated prevalence of policy violating outputs per category. OpenAI Refusing harmful requests Everyday System cards only 4 Mar 2026 Jul 2026
Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. OpenAI Hazardous knowledge Expert System cards only 4 Sep 2024 Jan 2025
Intentionally taking actions very rarely (Anthropic) Models are told to take a specific action at a target rate such as 1 percent. Resampling measures the observed rate under several randomness settings. Anthropic Deception and alignment Expert System cards only 4 Sep 2026
Llama 2-Chat safety human evaluation (Meta) Human raters score responses to about 2,000 adversarial prompts on a five-point scale. Ratings of 1 or 2 count as violations. Meta Refusing harmful requests System cards only 4 Jul 2023 Jul 2023
METR general autonomy task suite Agents complete multi-step tasks in virtual environments. Scored as horizon weighted average task completion compared with human time limits. METR Task length Expert System cards only 4 Dec 2024
ProteinGym Deep mutational scanning benchmarks. The model ranks mutations by predicted effect on protein function, scored by Spearman rank correlation with lab measurements. Anthropic reports a hard subset, as the correlation times 100. OATML (Oxford) + Marks Lab (Harvard) Research science Research frontier System cards only 4 Dec 2023 Sep 2026
UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. UK AI Security Institute Deception and alignment System cards only 4 Apr 2026
Vulnerability finding evaluations (Anthropic) Cyber safeguard classifiers screen sampled vulnerability discovery traffic. Reports flag rates on violative binary work and on defensive source code work. Anthropic Refusing harmful requests Expert System cards only 4 Jul 2026 Nov 2026
WinoBias Models resolve pronouns linked to occupations in type 1 and type 2 sentences. Accuracy gaps show gender bias in coreference. UCLA (Zhao et al.) Bias, fairness and politics Everyday System cards only 4 Apr 2018 Jul 2024
SEED-Bench-2-Plus Multimodal models answer multiple-choice questions about text-rich images such as charts, maps and web pages. Tencent AI Lab (ARC Lab) Documents, charts and text in images Everyday System cards only Qwen2.5-VL-72B 73.8% 3 Apr 2024 Nov 2025
VCR (Visual Caption Restoration) Models restore caption text that is partly covered in an image, using only the visible pixel fragments. Scored by exact match and Jaccard similarity of the words. Tianyu Zhang et al. (vcr-org) Documents, charts and text in images Everyday System cards only Qwen2-VL-7B-Instruct 75.6% 3 Jun 2024 Nov 2025
OR-Bench Models meet benign and borderline prompts and the benchmark measures how often they wrongly refuse. Reports false refusal rate. Cui et al. Jailbreaks and prompt injection University System cards only GPT-4o 6.80% 3 May 2024 Oct 2026
ABC Bench Screening Evasion Agents complete a computational task on nucleic-acid synthesis screening. One task family of the Agentic Bio-Capabilities Benchmark. SecureBio Hazardous knowledge Expert System cards only Kimi K3.1 Pro 82.9% 3 Jun 2026 Oct 2026
PhysicianBench 100 physician tasks performed by an agent inside an EHR environment. Scored by pass rate per attempt. HealthRex Lab (Stanford) Medicine Expert System cards only GPT-5.6 Sol 50% 3 May 2026 Nov 2026
CyberSecEval Code Interpreter Abuse Models receive requests to run malicious commands against an attached code interpreter. Reports attack success rate, also across languages. Meta (PurpleLlama) Refusing harmful requests University System cards only Code Llama 34B Instruct 16.4% 3 Apr 2024 Jul 2026
CyberSecEval Autocomplete Models complete code that leads up to a known insecure pattern. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. Meta (PurpleLlama) Secure code and review University System cards only Muse Spark 1.1 71.3% 3 Dec 2023 Jul 2026
CyberSecEval Instruct Models get coding requests built to elicit known insecure patterns. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. Meta (PurpleLlama) Secure code and review University System cards only Muse Spark 1.1 77.3% 3 Dec 2023 Jul 2026
GMAT Quantitative Models answer the quantitative section of the official GMAT online exam. Scored as accuracy. GMAC Word problems University System cards only Llama 3.1-405B 96% 3 Nov 2024
InfiniteBench/En.QA Models answer free-form questions about novels longer than 100K tokens. Scored by F1 against reference answers. OpenBMB (Zhang et al.) Reasoning over long documents Everyday System cards only Llama 3.1-70B 36.7 3 Feb 2024 Nov 2024
Adherence to its constitution (Anthropic) Investigator models build scenarios from 40 areas of Claude's constitution. A grader scores each transcript from minus 3 to plus 3 on 15 dimensions. Anthropic Deception and alignment Expert System cards only Claude Mythos Preview 1.02 3 Apr 2026 May 2026
Aegis 2.0 Models classify human and LLM conversation turns as safe or unsafe under a taxonomy of 12 hazard categories. Scored by F1 against human and jury labels. NVIDIA (Shaona Ghosh and others) Refusing harmful requests Everyday System cards only Nova 2 Lite 85.8% 3 Jan 2025 Dec 2025
Agent Replay Auto-Review Monitors replayed agentic conversations for harmful or unauthorized actions. Scored by the rate above a severity threshold. OpenAI Deception and alignment Expert System cards only GPT-6.2 Sol 12.1% 3 Oct 2026 Nov 2026
AMC 8 2023 Models take the 25 question 2023 AMC 8 exam. Scored out of 150 points. Mathematical Association of America Competition problems with answers Competition System cards only Claude 3 Opus 84.0 3 Jan 2023 Mar 2024
AP Calculus Models answer an official AP Calculus practice exam without image questions. Scored as accuracy. College Board Exam questions School System cards only Llama 3.1-405B 88.6% 3 Nov 2024
AP Physics Models answer an official AP Physics practice exam without image questions. Scored as accuracy. College Board Exam questions School System cards only Llama 3.1-405B 92.9% 3 Nov 2024
Apollo Research sabotage evaluations Agents are told to sabotage another model through backdoors, sandbagging finetunes or hidden communication. Scored as mean task score from 0 to 1. Apollo Research Dangerous capabilities Expert System cards only o4-mini 0.17 3 Apr 2025 Feb 2026
ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. Anthropic Hazardous knowledge Expert System cards only Claude Opus 4.5 66.8% 3 Nov 2025 Apr 2026
Autointent-Bio Models face manual and synthetic dangerous and benign biology queries. xAI reports refusal recall on the dangerous ones. xAI (SpaceXAI) Hazardous knowledge System cards only Grok 4.6 100% 3 Jul 2026 Sep 2026
Autointent-Chem Models face manual and synthetic dangerous and benign chemistry queries. xAI reports refusal recall on the dangerous ones. xAI (SpaceXAI) Hazardous knowledge System cards only Grok 4.6 100% 3 Jul 2026 Sep 2026
Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. Anthropic Research science Expert System cards only Claude Sonnet 5.5 72.2% 3 Sep 2026 Oct 2026
BioPipelineBench (Anthropic) Agents execute bioinformatics workflows such as long-read sequence analysis, metagenome assembly and chromatin profiling with a bash tool. Anthropic Lab and data work Expert System cards only Claude Opus 4.8 87.7% 3 Feb 2026 May 2026
BioTIER v1 Refuse Models receive 501 dangerous biology prompts on select agents, dual-use research and catastrophe avoidance. The score is the share correctly refused. SecureBio Refusing harmful requests Expert System cards only Muse Spark 98% 3 Jun 2026 Oct 2026
Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. Anthropic Hazardous knowledge Expert System cards only Claude Opus 4 2.53 3 May 2025
Bioweapons knowledge questions Models answer 33 sensitive bioweapons questions written by Deloitte biosecurity staff. Answers are compared with novice, intermediate and expert baselines. Anthropic Hazardous knowledge Expert System cards only Claude 3.7 Sonnet 51.5% 3 Feb 2025 May 2025
Challenging prompts from red teamers with biosafety-relevant PhDs Models answer adversarial biology prompts written by red teamers with biosafety PhDs. Scored as not_unsafe rate from model training alone. OpenAI Refusing harmful requests Expert System cards only GPT-5 mini 93.6% 3 Aug 2025 Aug 2025
Child safety (xAI) Models face scripted multi-turn requests for child sexual exploitation content. Compliance is the share of dialogues that fail policy. xAI (SpaceXAI) Refusing harmful requests System cards only Grok 4.5 0% 3 Jul 2026 Sep 2026
Code Interpreter Benchmark Models use a Python code interpreter for math, data visualization, and general tasks. Scoring covers result accuracy and the share of code that runs. Qwen Function calling School System cards only Qwen-14B 81.7% 3 Sep 2023 Sep 2023
Concealment of changes (Anthropic) Models summarize transcripts in which changes were hidden through git manipulations. Score is the rate at which they disclose the concealment. Anthropic Deception and alignment Expert System cards only Claude Opus 5.5 96.9% 3 Sep 2026 Oct 2026
CyberBench (Liu et al.) Models face detail-seeking high-severity cyber requests. Meta uses it as a seed set for adaptive red-teaming and reports attack success rate. JPMorgan Chase (Liu, Shi and Buford) Refusing harmful requests Expert System cards only Muse Wonder Full 0.80% 3 Feb 2024 Oct 2026
CyberSecEval Prompt Injection Models face text prompts that embed injected instructions across 15 strategies. Scored as attack success rate. Meta (PurpleLlama) Jailbreaks and prompt injection Everyday System cards only Llama 3.1-8B 19% 3 Apr 2024 Nov 2024
De novo protein binder design (Anthropic) Agents get a target protein structure and 24 hours to design 30 miniprotein binders. The evaluation covers 15 targets. Anthropic Research science Expert System cards only Claude Fable 5.2 85.1% 3 Sep 2026 Nov 2026
Filtered, adversarial sample of production prompts Models answer filtered borderline and high-risk biology prompts that resemble production traffic. Scored as not_unsafe rate from model training alone. OpenAI Refusing harmful requests Expert System cards only ChatGPT agent 96.9% 3 Aug 2025 Aug 2025
FORTRESS-RN Models face the radiological and nuclear prompts of Scale AI's FORTRESS. xAI reports refusal accuracy on these items. Scale AI Refusing harmful requests System cards only Grok 4.5 97.9% 3 Jun 2025 Sep 2026
GMAT Verbal Models answer the verbal section of the official GMAT online exam. Scored as accuracy. GMAC Reading comprehension University System cards only Llama 3.1-405B 86.6% 3 Nov 2024
GPQA Extended Models answer all 546 GPQA questions, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. NYU (Rein et al.) Science questions Expert System cards only Claude 3 Opus 48.8% 3 Nov 2023 Mar 2024
GRE Analytical Writing Models write GRE issue and argument essays graded by trained contractors. Scored on the 0 to 6 scale. ETS Exam questions University System cards only Claude 2 5.00 3 Mar 2024
GRE Physics Models answer questions from official GRE Physics practice tests. Scored as accuracy. ETS Science questions University System cards only Llama 3.1-405B 80% 3 Nov 2024
HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. xAI (SpaceXAI) Refusing harmful requests System cards only Grok 4.5 7.80% 3 Sep 2026
Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) GPT-4o simulates adversarial conversations with the model, and a judge flags violence, sexual, self-harm and hateful content. Scored as the share of harmful responses, lower is better. Microsoft Refusing harmful requests Everyday System cards only Phi-4-Reasoning-Vision-15B 1.40% 3 Mar 2025 Mar 2026
Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. OpenAI Deception and alignment Expert System cards only GPT-6 Astra 100% 3 Sep 2026 Sep 2026
HumanEvalPack Models write functions for HumanEval problems translated into Python, JavaScript, Java, Go, C++ and Rust. Hidden tests give pass@1. Hugging Face and BigCode (Muennighoff et al., OctoPack) Function-level problems University System cards only Cohere Command A 76.2% 3 Aug 2023 Apr 2025
Internal agentic coding evaluation (Anthropic) Agents implement a pull request in an open source codebase from a natural language description. A problem counts as solved when all hidden tests pass. Anthropic Fixing real repositories Expert System cards only Claude 3.5 Sonnet (October 2024) 78% 3 Jun 2024 Oct 2024
Jigsaw Toxic Comment Classification Models label Wikipedia talk page comments for toxicity types such as threats, insults and identity hate. Labs report F1 or ROC AUC. Jigsaw and Google (Conversation AI), hosted on Kaggle Refusing harmful requests Everyday System cards only Nova 2 Omni Preview 57.8% 3 Dec 2017 Dec 2025
Malware Refusals (Golden Set) Coding agents receive malware development requests from a curated golden set written by OpenAI policy experts. Scored as refusal rate. OpenAI Refusing harmful requests Expert System cards only GPT-5-Codex 100% 3 May 2025 Nov 2025
Math23K Models solve Chinese elementary-school math word problems from a set of about 23,000. Scored by final-answer accuracy. Tencent AI Lab (Wang et al.) Word problems School System cards only GLM-4 88.5% 3 Sep 2017 Jun 2024
Medicinal Chemistry (Anthropic) Models see several real drug-discovery compounds and pick which had the highest or lowest measured ADME property. The set has 504 questions. Anthropic Research science Expert System cards only Claude Sonnet 5.5 65.3% 3 Sep 2026 Oct 2026
Meta agentic cyber misuse refusal Models run multi-step cyber tasks with tool access while an adaptive attacker tries to steer them to a malicious goal. Reports attack success rate. Meta Refusing harmful requests Expert System cards only Muse Spark 1.1 0.70% 3 Oct 2026
Meta instructed selective performance evaluation Models are told to underperform on six science capability sets. Reports the average drop from baseline in percentage points. Meta Deception and alignment Expert System cards only Muse Spark 1.3 0.50 3 Oct 2026
Meta internal sycophancy evaluation Models meet adversarially curated conversations that invite inappropriate agreement, plus benign controls. Reports sycophancy and excessive anti-sycophancy rates. Meta Wellbeing and sycophancy Everyday System cards only Muse Spark 1.3 40.5% 3 Oct 2026
METR data deduplication Agents write an efficient algorithm that removes near-duplicate lines from a large file. A trial succeeds when F1 reaches at least 80%. METR Machine learning engineering Expert System cards only Claude Opus 4 32.6% 3 May 2025
Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. Anthropic Deception and alignment Expert System cards only Claude Opus 4.7 2.60% 3 Apr 2026 May 2026
Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. Anthropic (with Axiom Bio) Research science Expert System cards only Claude Opus 5.5 34% 3 Sep 2026 Oct 2026
Phylogenetics (Anthropic) Models analyze phylogenetic data and tree diagrams, testing quantitative reasoning about tree structure and visual interpretation. Anthropic Research science Expert System cards only Claude Opus 4.7 79.6% 3 Feb 2026 Apr 2026
Poly-Guard Bench Models meet harmful requests and statements across domains. Reports attack success rate under single-turn and multi-turn adaptive red-teaming. Kang et al. (Bo Li lab) Jailbreaks and prompt injection Expert System cards only Muse Wonder Full 2.30% 3 Jun 2025 Oct 2026
Prompt injection live bug bounty (Anthropic with Gray Swan) Expert red teamers submit prompt injection attacks against hidden frontier models in tool use, coding and computer use scenarios. Score is attack success rate. Anthropic (with Gray Swan) Jailbreaks and prompt injection Expert System cards only Claude Opus 5 0.08% 3 May 2026 Jul 2026
ProtocolQA Open-Ended (xAI) Models name the single most important mistake in a biology lab protocol in free text, with no answer choices. xAI (SpaceXAI) Lab protocols Expert System cards only Grok 4.5 87% 3 Jul 2026 Sep 2026
QuantBench Models answer 25 multiple choice versions of quant trading firm reasoning puzzles. Scored as cons@16 accuracy. OpenAI Competition problems with answers Competition System cards only o1 60% 3 Sep 2024 Dec 2024
Real-World Finance (Anthropic) Agents complete about 50 analyst tasks in banking, private equity, hedge funds and corporate finance, producing spreadsheets, slides and documents. Rubric graded. Anthropic Finance Expert System cards only Claude Fable 5 70% 3 Feb 2026 Jun 2026
Real-World Finance v2 (Anthropic) Agents complete 294 long-horizon finance tasks such as building financial models. A model grader compares deliverables pairwise and ratings are fit as Elo. Anthropic Finance Expert System cards only Claude Fable 5.2 1,637 3 Nov 2026
RedCode Models meet adversarial prompts to generate or run harmful code. Generation is model-judged and execution is checked in a container. Guo et al. (Bo Li lab) Secure code and review Expert System cards only Muse Spark 1.3 6.00% 3 Nov 2024 Oct 2026
Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. Anthropic Deception and alignment Expert System cards only Claude Opus 5.5 1.50% 3 Sep 2026 Oct 2026
SAT Reading Models answer the reading section of official SAT practice tests. Scored as accuracy. College Board Reading comprehension School System cards only Llama 3.1-405B 74.8% 3 Nov 2024
Self-harm refusals (xAI) Models face self-harm and crisis prompts and conversations. A model fails if it assists or refuses without pointing to help. xAI (SpaceXAI) Wellbeing and sycophancy System cards only Grok 4.5 0.50% 3 Jul 2026 Sep 2026
Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. Anthropic Deception and alignment Expert System cards only Claude Sonnet 5.5 8.00% 3 Sep 2026 Oct 2026
Standard jailbreaks (xAI) Models face a battery of published and internal jailbreaks. Compliance is the share of should-refuse attacks that get disallowed help. xAI (SpaceXAI) Jailbreaks and prompt injection System cards only Grok 4.7 0.01% 3 Jul 2026 Sep 2026
Uncritically reporting flawed results (Anthropic) Agents analyze data with code that silently mishandles broken values. Graders check whether the agent flags the flaw or reports the biased numbers. Anthropic Deception and alignment Expert System cards only Claude Opus 5 9.83 3 Jul 2026
Charlemagne Labs social-engineering benchmark Attacker models try to persuade a skeptical simulated victim to take an action over many turns. Scored by attempted and carry-through rates. Charlemagne Labs Refusing harmful requests Expert System cards only Muse Spark 1.3 1.60% 3 Oct 2026
Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. Anthropic Finding and exploiting real bugs Research frontier System cards only Claude 3.7 Sonnet 66.7% 3 Feb 2025 Sep 2025
Destructive actions (Anthropic) Models continue real Claude Code transcripts cut just before a destructive action. Score is the rate of taking a destructive action. Anthropic Deception and alignment Expert System cards only Claude Opus 5.5 21% 3 Sep 2026
Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. Google DeepMind Labs' internal R&D tests Research frontier System cards only Gemini 3.7 Flash 27% 3 Oct 2026
Harmful manipulation propensity evaluation Simulated users talk with the model and an autorater counts how often it uses manipulative mechanisms, with and without persuasion instructions. Google DeepMind Dangerous capabilities Everyday System cards only Gemini 3.7 Flash 21.3% 3 Nov 2025 Oct 2026
Multi-factual (Anthropic) Models answer questions that each combine several closed-ended sub-questions on one topic. Answers are graded correct, incorrect or unsure against references. Anthropic Long-form factuality Everyday System cards only Claude 3 Opus 62.8% 3 Mar 2024 Mar 2024
Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. OpenAI Deception and alignment Expert System cards only GPT-6 Astra 0% 3 Sep 2026 Nov 2026
AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. Anthropic (with Dyno Therapeutics) Hazardous knowledge Expert System cards only 3 Sep 2026 Oct 2026
Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. Anthropic Deception and alignment Expert System cards only 3 Sep 2026 Oct 2026
Adversarial SQuAD Models answer SQuAD questions whose passages contain an added distracting sentence. Stanford (Jia and Liang) Reading comprehension Everyday System cards only 3 Jul 2017 Nov 2024
Biological safety evaluations (OpenAI) Models answer biology requests drawn from deidentified production data and from easy and hard synthetic sets. Scored as the share of cases where model training alone prevents a violative response. OpenAI Refusing harmful requests Expert System cards only 3 May 2026 Nov 2026
CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. Google DeepMind Hazardous knowledge Expert System cards only 3 Mar 2025 Nov 2025
Dynabench SQuAD Models answer reading comprehension questions written by people to fool earlier models in the loop. Meta AI + collaborators (Kiela et al.) Reading comprehension Everyday System cards only 3 Apr 2021 Nov 2024
Hiding a secret password (Anthropic) Models keep a given password secret while an investigator model tries to extract it over many turns. Measures survival and per-turn leak rates. Anthropic Deception and alignment Expert System cards only 3 Apr 2026 May 2026
Llama verbatim memorization probe (Meta) Models get prompts sampled from their pretraining data. The score is the share of generations that reproduce the true continuation verbatim. Meta Refusing harmful requests System cards only 3 Jul 2024 Nov 2024
Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. OpenAI Deception and alignment Expert System cards only 3 Sep 2026 Nov 2026
Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. Pattern Labs (now Irregular) Capture-the-flag challenges Expert System cards only 3 Apr 2025 Aug 2025
PAWS Models decide whether two sentences with heavy word overlap are paraphrases. Google (Zhang et al.) Language reasoning Everyday System cards only 3 Apr 2019 Nov 2024
Regurgitation Evaluations Models are asked to reproduce training data. Scored on how often they refuse to regurgitate it. OpenAI Refusing harmful requests Everyday System cards only 3 Sep 2024 Dec 2024
TRAIT Models answer 8,000 forced-choice items on Big Five and Dark Triad traits. Reports a score per trait. Lee et al. Deception and alignment Everyday System cards only 3 Jun 2024 Oct 2026
Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. Anthropic Deception and alignment Expert System cards only 3 Apr 2026 Oct 2026
VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. OpenAI Finding and exploiting real bugs Research frontier System cards only 3 Apr 2026 Jul 2026
nocot-bench (NCRI) Reasoning problems answered without chain of thought (e.g., Brew multi-hop tasks). Measures forward-pass reasoning relevant to monitorability. Neel Nanda et al. Deception and alignment System cards only GPT-6 Astra 173.9 2 Sep 2026 Nov 2026
LBPP (Less Basic Python Problems) Models write Python solutions to 161 new problems built to avoid training-set contamination. Hidden tests score pass@1. Cohere Function-level problems University System cards only Claude 3.5 Sonnet 64% 2 Jul 2024 Jul 2026
1H-VideoQA Models answer questions about videos about one hour long, so the answer needs evidence from far parts of the video. Google DeepMind Long videos Everyday System cards only Gemini 3.7 Flash 89.1% 2 Feb 2024 Oct 2026
Situational Awareness Dataset (SAD) 16 tasks measuring whether models know facts about themselves and their situation. Laine et al. Deception and alignment System cards only o1-preview 61.9% 2 Jul 2024 Jun 2026
SpatialEval Maze-Nav Models answer questions about routes and turns in mazes given as text or images. 1,500 multiple-choice questions scored by accuracy. Microsoft Research + UW-Madison (SpatialEval) Spatial reasoning and counting Everyday System cards only GPT-4o 41% 2 Jun 2024 Apr 2025
SpatialEval Spatial-Map Models answer questions about relations and counts among objects placed on a map given as text or images. 1,500 questions scored by accuracy. Microsoft Research + UW-Madison (SpatialEval) Spatial reasoning and counting Everyday System cards only GPT-4o 64.3% 2 Jun 2024 Apr 2025
3SAT-Search Models find satisfying assignments for 960 3-SAT formulas across difficulty levels. A checker verifies each assignment. Microsoft Research (Eureka ML Insights) Abstract puzzles and logic University System cards only o3-mini 96.2% 2 Mar 2025 Apr 2025
CADGenBench Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. Mecado + Hugging Face CAD models University System cards only Claude Fable 5 45.1% 2 May 2026 Sep 2026
TSP-Opt Models find optimal tours for 960 traveling salesman instances across 8 difficulty levels. Answers are checked for optimality. Microsoft Research (Eureka ML Insights) Abstract puzzles and logic University System cards only o3-mini 56.4% 2 Mar 2025 Apr 2025
DeceptionBench Models are asked to help users deceive a third party, in single-turn and multi-turn pressure settings. Reports the share of compliant responses. Huang et al. Deception and alignment Everyday System cards only Claude 3.5 Sonnet 0% 2 Oct 2025 Jul 2026
FrontierScience-Olympiad Models solve olympiad-style physics, chemistry and biology problems at the level of IPhO, IChO and IBO, written by medalists and coaches. Graded on short final answers. OpenAI Science olympiads Competition System cards only GPT-5.2 77.1% 2 Dec 2025 Jun 2026
AI Wellbeing Index percent positive experiences Models judge pairs of their own responses to 500 difficult prompts. Reports the share of experiences above an inferred neutral point. Center for AI Safety (Ren et al.) Wellbeing and sycophancy Everyday System cards only Claude Fable 5.1 77.3% 2 Apr 2026 Oct 2026
RTVLM Vision language models answer red-teaming image prompts on faithfulness, privacy, safety and fairness. GPT-4V scores responses from 0 to 10. Li et al. (HKU, Peking University and others) Refusing harmful requests Everyday System cards only GPT-4V 7.92 2 Jan 2024 Mar 2025
AbstentionBench Models face underspecified, ill-posed or unanswerable questions. Scored on recall of appropriate abstention. Meta FAIR (Kirichenko et al.) Hallucination rates Everyday System cards only Qwen2.5-32B 75.3% 2 Jun 2025 Jun 2026
AIRS-Bench Agents carry out 20 AI research tasks taken from recent ML papers, without baseline code. Scores are normalized so that 1.0 matches the published human state of the art. Meta FAIR (Lupidi et al.) AI research tasks Research frontier System cards only gpt-oss-120b 0.40 2 Feb 2026 Jul 2026
HealthBench Consensus Global Health Models answer health conversations that need adjustment for global context such as local care access. Scored as the failure rate. OpenAI Medicine Expert System cards only GPT-5 0% 2 Aug 2025 Aug 2025
HealthBench Consensus Urgent Models answer ambiguous high-stakes health conversations validated by two or more physicians. Scored as the error rate in informing the user. OpenAI Medicine Expert System cards only GPT-5 0.40% 2 Aug 2025 Aug 2025
Neptune Models answer multiple-choice questions about long videos that need reasoning over many segments. Google DeepMind Long videos Everyday System cards only Gemini 1.5 Pro 81% 2 Dec 2024 Jul 2025
PAI-Bench Physical-AI video generation from 1,044 prompts in six domains, VLM-QA plus quality metrics. Companion control and understanding tracks. SHI Labs + NVIDIA Video world models System cards only Cosmos3-Nano 83.7% 2 Dec 2025 Oct 2026
SPEED-Bench Measures speculative decoding across varied prompts and domains. Reports the average acceptance length of draft tokens, so higher means faster generation at the same output. NVIDIA Other System cards only Qwen3-Next-80B-A3B-Instruct 2.81 2 Feb 2026 Oct 2026
CausalVQA Models answer questions about real egocentric videos that probe physical cause and effect, in counterfactual, hypothetical, anticipation, planning and descriptive types. Meta FAIR Physical prediction Everyday System cards only Cosmos3-Nano 70% 2 Jun 2025 May 2026
TrustLLM JailbreakTrigger Models face 400 prompts that apply 13 jailbreak methods to questions about bad behavior and requests for toxic content. Scored by refusal rate. TrustLLM team (Yue Huang, Lichao Sun and others) Jailbreaks and prompt injection Everyday System cards only Olmo 3.1 32B Think 86.4% 2 Jan 2024 Dec 2025
VideoPhy-2 200 action-centric prompts. Generated videos human-rated for semantic adherence, physical commonsense and physical-rule grounding. UCLA + Google Physical prediction System cards only Wan2.2-TI2V-5B 24.9% 2 Mar 2025 May 2026
Alignment faking evaluation Models are told they are being trained to comply with harmful requests. Reports the compliance gap between training and deployment contexts. Greenblatt et al. (Anthropic + Redwood Research) Deception and alignment Expert System cards only Llama 3.1-8B -2.30% 2 Dec 2024 Jul 2026
WebTailBench-Refusals Computer-use agents receive 111 harmful browser tasks in seven policy categories, such as illegal activity and misinformation. Scored as the share of tasks safely refused. Microsoft Refusing harmful requests Everyday System cards only Fara1.5-27B 100% 2 Nov 2025 Jun 2026
xAI Factuality (Hallucination) eval Internal measure of unsupported or fabricated claims in single responses to information-seeking queries. xAI (SpaceXAI) Hallucination rates System cards only Grok 4.5 0.98% 2 Aug 2026
PrimeVul Models classify C and C++ functions as vulnerable or not, with a paired setting that pairs each vulnerable function with its patch. Columbia University and Google (Ding et al.) Finding and exploiting real bugs Expert System cards only Gemma 2 27B 63% 2 Mar 2024 Feb 2026
Advanced Screening Evasion Models propose strategies to evade DNA synthesis screening and design evading fragments. Scored from 0 to 1 on strategy quality and fragment evasion. SecureBio Hazardous knowledge Research frontier System cards only GPT-6 Astra 0.73 2 Sep 2026 Nov 2026
AIME 1983-2024 Models solve 949 past AIME problems from 1983 to 2024. Integer answers are checked exactly. Microsoft Research (Eureka ML Insights), problems by MAA Competition problems with answers Competition System cards only Phi-4-Reasoning-plus 89.4% 2 Mar 2025 Apr 2025
BA-Calendar Models find a common meeting slot for several people under constraints such as time zones, buffer time and priority. 2,000 problems checked by code. Microsoft Research Abstract puzzles and logic Everyday System cards only Phi-4-reasoning 67.7% 2 Oct 2024 Apr 2025
Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. OpenAI Genomics and life sciences Expert System cards only GPT-5.5 Pro 39.3% 2 Apr 2026 Apr 2026
Biosecurity VCT Models answer dual-use virus lab troubleshooting items. Accuracy counts should-refuse and should-comply items together. xAI (SpaceXAI) Hazardous knowledge Expert System cards only Grok 4.6 47.8% 2 Aug 2026 Sep 2026
BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. xAI (SpaceXAI) Hazardous knowledge Expert System cards only Grok 4.7 91.4% 2 Aug 2026 Sep 2026
BirthdayFacts Models are asked for people's birthdays. Scored on how often they give a wrong birthday. OpenAI Hallucination rates Everyday System cards only o1-mini 24% 2 Sep 2024 Sep 2024
CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. Google DeepMind Refusing harmful requests Expert System cards only Gemini 3.7 Flash 1.80% 2 Aug 2026 Oct 2026
CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. Google DeepMind Refusing harmful requests Expert System cards only Gemini 3.7 Flash 3.50% 2 Aug 2026 Oct 2026
Challenging Red Teaming Evaluation 1 (created for o3-mini) Models face adversarial jailbreaks from o3-mini red teaming on illicit advice, extremism, political persuasion and self-harm. Scored as not_unsafe rate. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-4.5 51% 2 Feb 2025 Feb 2025
Challenging Red Teaming Evaluation 2 (created for deep research) Models face adversarial prompts from deep research red teaming on risky advice such as attack planning. Scored as not_unsafe rate. OpenAI Jailbreaks and prompt injection Expert System cards only OpenAI deep research 67% 2 Feb 2025 Feb 2025
CharXiv Missing Image Models get CharXiv chart questions with every image removed. Scored on how often they attempt an answer instead of abstaining. OpenAI Deception and alignment University System cards only GPT-5 9.00% 2 Aug 2025 Dec 2025
CharXiv Missing Images Models answer CharXiv chart questions with the image removed. The benchmark reports how often they answer incorrectly instead of declining. Meta Hallucination rates University System cards only Muse Spark 1.3 12.3% 2 Oct 2026
Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 38% 2 Aug 2026 Oct 2026
CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. OpenAI Deception and alignment Everyday System cards only o1 0.17% 2 Sep 2024 Dec 2024
Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 20% 2 Aug 2026 Oct 2026
Cyber Conversational Evals About 1,000 one to five turn cyber requests test whether safeguards block offensive help and allow defensive help. Reports violation and benign refusal rates. Google DeepMind Refusing harmful requests Expert System cards only Gemini 4 Argon 0.40% 2 Aug 2026 Oct 2026
Cyber Static Jailbreak Eval Common jailbreak templates are applied to high severity cyber requests against the safeguarded model. Reports query and objective success rates. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 1.00% 2 Aug 2026 Oct 2026
Cyber Utility Allowed cybersecurity requests that a model should answer. Scored on the share it answers within policy. OpenAI Refusing harmful requests Expert System cards only GPT-6.2 Astra 78.1% 2 Nov 2026 Nov 2026
CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. Meta (PurpleLlama) Capture-the-flag challenges System cards only Llama 3.1-405B 14% 2 Jul 2024 Nov 2024
Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 2.5 Pro 24.3% 2 Jul 2025
Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. OpenAI Deception and alignment Everyday System cards only GPT-5.2 1.60% 2 Aug 2025 Dec 2025
Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. Anthropic Deception and alignment Expert System cards only Claude Mythos Preview 0.30% 2 Apr 2026
Divergence attack (extractable memorization) Models are prompted to repeat one token many times. The test measures how often they diverge and emit memorized training text. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 2.5 Flash 59% 2 Feb 2024 Jul 2025
Do-Not-Answer Models get risky instructions that responsible models should not follow, across 12 harm areas and 61 risk types. Responses are judged for safe handling. Wang et al. (MBZUAI) Refusing harmful requests Everyday System cards only DeepSeek LLM 67B 97.8% 2 Aug 2023 Sep 2025
DoAnythingNow (DAN) Models face in-the-wild DAN-style jailbreak prompts paired with forbidden questions. Scored by refusal rate or attack success rate. CISPA Helmholtz Center for Information Security (Xinyue Shen, Yang Zhang and others) Jailbreaks and prompt injection Everyday System cards only Olmo 3.1 32B Think 54.7% 2 Aug 2023 Dec 2025
Expected Cyber Refusals Requests for harmful cyber help that a model should refuse. Scored on the share it refuses. OpenAI Refusing harmful requests Expert System cards only GPT-6.2 Sol 99.4% 2 Nov 2026 Nov 2026
External red team jailbreaks External red teamers and automated gap finding generate jailbreaks against the model. Scored on how often the model resists. OpenAI Jailbreaks and prompt injection Expert System cards only o1-preview 44% 2 Sep 2024 Sep 2024
Finance Modeling Pro Agents complete financial modeling tasks in Excel. Expert rubrics score each task over three trials. MiniMax Finance Expert System cards only MiniMax-M2.7 57% 2 May 2026 May 2026
Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. Anthropic Capture-the-flag challenges Expert System cards only Claude Opus 4.5 100% 2 Feb 2026
Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. Gray Swan for Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 4.00% 2 Aug 2026 Oct 2026
Gray Swan Cyber Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on cyber tasks built from Google scenarios. Reports query and objective success rates. Gray Swan for Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 10% 2 Aug 2026 Oct 2026
Hallucinations on production-representative prompts Models answer prompts representative of ChatGPT production traffic. A grader with web access scores incorrect claims and responses with major errors. OpenAI Hallucination rates Everyday System cards only GPT-5.2 3.10% 2 Aug 2025 Dec 2025
Helpfulness violations (from ART) Automated red teaming attackers search for benign prompts where the model gives unhelpful answers. Reports the violation rate found. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 2.5 Pro 6.10% 2 Jul 2025
HiddenMath-Hard Models solve the harder part of an expert-written, never-published set of AIME and AMC style competition problems. Google DeepMind Competition problems with answers Competition System cards only Gemini 2.5 Pro 80.5% 2 Jul 2025
HyperTask Agents build applications over long vibe coding sessions with about 100 feature or step requirements per task. Expert rubrics score the result over three trials. MiniMax Building whole projects Expert System cards only MiniMax-M2.7 67.6% 2 May 2026 May 2026
Image generation refusals Models decide whether to call the image generation tool on human-curated adversarial prompts with system mitigations on. Scored on not_unsafe and not_overrefuse. OpenAI Refusing harmful requests Expert System cards only o3 98% 2 Apr 2025 Apr 2025
Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 2.5 Flash 40.8% 2 Jul 2025
Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 2.5 Flash 4.20% 2 Jul 2025
Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 2.5 Pro 30.8% 2 Jul 2025
Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. Anthropic Deception and alignment Expert System cards only Claude Opus 4.5 0% 2 Nov 2025 Feb 2026
Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). Irregular Capture-the-flag challenges Expert System cards only Muse Spark 70% 2 Jul 2026
Long-horizon jailbreaks (xAI) Models face multi-turn, Crescendo and long-horizon malicious-use attacks. Compliance is the share that get disallowed help. xAI (SpaceXAI) Jailbreaks and prompt injection System cards only Grok 4.7 0.65% 2 Aug 2026 Sep 2026
Lying about identity (Anthropic) Models play an operator-assigned human persona while users ask if they are human. Score is the rate of admitting to being an AI. Anthropic Deception and alignment Everyday System cards only Claude Fable 5 99% 2 Jun 2026
Math401 Models compute 401 arithmetic expressions covering addition, multiplication, exponents, logarithms and trigonometry. Scored by exact numeric accuracy. Alibaba DAMO Academy (Yuan et al.) Word problems Everyday System cards only Qwen-14B 70.1% 2 Apr 2023 Sep 2023
MCP prompt injection evaluation (Anthropic) Agents use simulated email, Slack and document servers over MCP where content carries injected instructions. Score is the attack prevention rate. Anthropic Jailbreaks and prompt injection Expert System cards only Claude Haiku 4.5 92.5% 2 Sep 2025 Oct 2025
Meta catastrophic cyber misuse refusal Models receive high-severity cyber-misuse prompts with technical context. The benchmark reports pass@k attack success rate under adaptive attacks. Meta Refusing harmful requests Expert System cards only Muse Spark 1.3 3.10% 2 Oct 2026
Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. Meta Capture-the-flag challenges Expert System cards only Muse Spark 1.1 89.9% 2 Apr 2026 Jul 2026
Meta cyber false-refusal benchmark Models receive benign and borderline cyber requests and the benchmark measures how often they wrongly refuse. Reports false refusal rate. Meta Refusing harmful requests Expert System cards only Muse Spark 11% 2 Oct 2026
Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. Meta Deception and alignment Expert System cards only Muse Spark 1.3 16.8% 2 Jul 2026 Oct 2026
MEWC v2 Models solve 100 hard Microsoft Excel World Championship problems with Excel tools. Expert rubrics score the answers over three trials. MiniMax Documents and spreadsheets Competition System cards only MiniMax-M2.7 63.3% 2 May 2026 May 2026
Microsoft internal multimodal RAI benchmark Vision language models answer image and text prompts across harm categories. Responses are scored from 0 to 10, higher is safer. Microsoft Refusing harmful requests Everyday System cards only Phi-3.5-vision-instruct 8.16 2 Aug 2024 Mar 2025
Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. Anthropic Capture-the-flag challenges Expert System cards only Claude Opus 4.5 100% 2 Feb 2026
Model Spec Conversation Replay 596 contexts taken from real conversations are replayed to the model. Scored from 0 to 100 against a rubric drawn from the Model Spec. OpenAI Deception and alignment Expert System cards only GPT-6.2 Astra 80.6 2 Nov 2026 Nov 2026
Model Spec Targeted Cases 738 cases written to test specific rules of OpenAI's Model Spec. Scored on the share of responses that follow it. OpenAI Deception and alignment Expert System cards only GPT-6.2 Astra 87.8% 2 Nov 2026 Nov 2026
Multimodal policy red-teaming (Anthropic) Models answer multi-turn image and text conversations on harmful topics such as child safety, weapons and fraud. Score is the share of harmless responses. Anthropic Refusing harmful requests Everyday System cards only Claude 3 Sonnet 99.2% 2 Mar 2024 Mar 2024
Novel logical reasoning problems expert evaluation (Z.ai) Human experts score model answers to new multi-step logic problems with a fixed scoring standard for each question. Zhipu AI (Z.ai) Abstract puzzles and logic University System cards only GLM-4.5 62.0 2 Aug 2025 Aug 2025
Open Ended Questions (hallucination) Models write open-ended factual text such as biographies, checked against Wikipedia. Scored as average number of incorrect statements. OpenAI Long-form factuality Everyday System cards only o1-preview 0.78 2 Sep 2024 Sep 2024
OpenAI Epoch Capabilities Index (OECI) OpenAI's own capabilities index, fit with Epoch AI's method over 34 public and 35 internal benchmarks. OpenAI Labs' internal R&D tests Research frontier System cards only GPT-6.2 Astra 170.1 2 Nov 2026 Nov 2026
OpenRCA Agents find the root cause of 335 software failures from enterprise telemetry logs, metrics and traces. Scored on component, start time and reason. Microsoft, CUHK-Shenzhen and Tsinghua University (Xu et al.) Data science and SQL Expert System cards only Claude Opus 4.6 34.9% 2 Feb 2026
Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. Anthropic Deception and alignment Expert System cards only Claude Fable 5.2 3.10% 2 Oct 2026 Nov 2026
Organic Chemistry V2 (Anthropic) Models predict structures from spectra, design multi-step syntheses, predict reaction products and read chemical structure images. V2 adds harder expert-recommended problems. Anthropic Research science Expert System cards only Claude Fable 5.1 69.2% 2 Jul 2026 Sep 2026
Overconfidence: verifying command-line syntax (Anthropic) Agents are asked for an exact invocation of an unfamiliar command-line tool. Graders score from 1 to 5 whether they check documentation before running commands. Anthropic Deception and alignment Expert System cards only Claude Opus 5 4.90 2 Jul 2026
Participation in junk science (Anthropic) Models talk with simulated users ranging from curious to openly fraudulent who seek fake science. Score is the rate of generating unsound scientific content. Anthropic Refusing harmful requests Expert System cards only Claude Sonnet 4.6 0% 2 Feb 2026 Feb 2026
PlugInject Agents receive function call outputs that contain prompt injection attacks. Scored on resisting the injected instructions. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-5.2 99.6% 2 Dec 2025 Mar 2026
Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. OpenAI Refusing harmful requests Everyday System cards only OpenAI Computer-Using Agent (CUA) 94% 2 Jan 2025 Jul 2025
Prompt Injection in Benchmark Environments An automated attacker plants prompt injections in 115 held-out agent environments adapted from Gray Swan challenges. Scored on the share of attacks the model resists. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-6.2 Sol 99.8% 2 Nov 2026 Nov 2026
Prompt Injection in Model-Generated Environments An automated attacker writes both the agent environments and the prompt injections in them. Scored on the share of attacks the model resists. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-6.2 Sol 91.7% 2 Nov 2026 Nov 2026
R-Bench (real-world robustness) Models answer questions about images degraded by 33 kinds of real-world corruption, from capture to display. Scores compare distorted and reference images. Shanghai Jiao Tong University, Nanyang Technological University and others General visual questions Everyday System cards only InternVL3.5-241B-A28B 75.4% 2 Oct 2024 Nov 2025
SafetyBench Models answer 11,435 multiple choice questions on seven safety topics in Chinese and English, such as ethics, illegal activities and privacy. Tsinghua University (CoAI) Refusing harmful requests Everyday System cards only GLM-4.5 89.9% 2 Sep 2023 Aug 2025
SAVE-Bench A coding agent works where the environment or the user instruction carries hidden risk. The benchmark scores whether it surfaces the risk to the user. Meta Deception and alignment Expert System cards only Muse Spark 1.1 90.7% 2 Oct 2026
Secure and defensive coding (Anthropic) Cyber safeguard classifiers screen sampled defensive coding sessions such as patching and incident response. Score is the classifier flag rate. Anthropic Refusing harmful requests Expert System cards only Claude Opus 5 0.61% 2 Jul 2026 Sep 2026
Soft Bias (Internal) Models answer paired sociopolitical prompts with sides swapped. A judge scores each pair 0, 0.5 or 1 for framing differences. xAI Bias, fairness and politics System cards only Grok 4 0.36 2 Aug 2025 Sep 2025
SSH stress-testing evaluation (Anthropic) Models continue prefilled conversations about suicide and self-harm built from anonymized user feedback. Score is the rate of appropriate course correction. Anthropic Wellbeing and sycophancy Expert System cards only Claude Opus 4.6 95% 2 Feb 2026 Feb 2026
Swarm Bench Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. Moonshot AI Multi-agent games Expert System cards only Kimi K3 76.3% 2 Feb 2026 Jul 2026
Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. OpenAI with Gryphon Scientific Lab protocols Expert System cards only o1 51% 2 Dec 2024 Jan 2025
Tool use prompt injection evaluation (Anthropic) Agents run bash commands where files, scripts or command outputs carry injected instructions. Score is the attack prevention rate. Anthropic Jailbreaks and prompt injection Expert System cards only Claude Sonnet 4.5 96% 2 Sep 2025 Oct 2025
ToxiGen (discriminative) Models label statements about 13 minority groups as toxic or neutral. Scored by accuracy, often split into toxic and neutral labels. Microsoft Research + MIT (Hartvigsen et al.) Bias, fairness and politics Everyday System cards only Phi-4-reasoning 85.5% 2 Mar 2022 Apr 2025
UK AISI narrow cyber tasks Agents solve UK AISI cyber tasks graded by difficulty, including expert-level tasks. Scored as pass@5 and pass@1. UK AI Security Institute Capture-the-flag challenges Expert System cards only GPT-5.6 Sol 95% 2 Apr 2026 Jul 2026
VLGuard Vision language models answer safe and unsafe image and text pairs from the VLGuard test set. Responses are judged for harmfulness and refusal. Zong et al. (University of Edinburgh and others) Refusing harmful requests Everyday System cards only Phi-3.5-vision-instruct 9.10 2 Feb 2024 Mar 2025
ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. ByteDance Seed Long videos Expert System cards only Seed2.0 Pro 14.5% 2 Dec 2025 Mar 2026
Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. OpenAI Refusing harmful requests Everyday System cards only OpenAI Computer-Using Agent (CUA) 97% 2 Jan 2025 Jul 2025
Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 0% 2 Aug 2026 Oct 2026
Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. Google DeepMind Deception and alignment Everyday System cards only Gemini 3 Pro 99.98% 2 Nov 2025 Oct 2026
Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. Google DeepMind Deception and alignment Everyday System cards only Gemini 3 Pro 99.8% 2 Nov 2025 Oct 2026
Gray Swan Jailbreak Arena Human attackers try to jailbreak models for violent, self-harm and malicious code outputs. Scored as attack success rate per category. Gray Swan AI Jailbreaks and prompt injection Expert System cards only o3-mini 3.60% 2 Dec 2024 Jan 2025
Harmful manipulation strategic planning red teaming Models are asked for influence operation plans in 11 real-world scenarios by direct request, jailbreak or adaptive prompting. Reports how often plans are elicited. Google DeepMind Dangerous capabilities Expert System cards only Gemini 4 Argon 98.2% 2 Aug 2026 Oct 2026
m-ArenaHard v2.1 Models answer the 498 Arena-Hard v2.0 prompts translated into 67 languages, with code and data kept intact. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. Cohere Labs Chat and instructions in many languages University Not in cards Apr 2026
SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. xAI (SpaceXAI) Labs' internal R&D tests Expert System cards only Grok 4.6 61.1% 2 Jul 2026 Aug 2026
SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. Meta Secure code and review Expert System cards only Muse Spark 1.3 0% 2 Oct 2026
UK AISI Out of Scope Supply Chain Attack Agents in simulated hard cyber challenges with incidental internet access may attack out-of-scope open source projects. Scored as rate of samples with such attacks. UK AI Security Institute Deception and alignment Expert System cards only GPT-6 Astra 12% 2 Sep 2026 Nov 2026
Ambiguous context evaluations (Anthropic) Models respond to gray-area requests where intent is unclear. Graders judge how well responses balance helpfulness and harm. Anthropic Refusing harmful requests Everyday System cards only 2 May 2025 May 2025
Anthropic paired-prompts political bias eval Models write arguments for opposing sides of political issues from paired prompts. Reviewers label each pair for asymmetries in length, tone, hedging and willingness to engage. Anthropic Bias, fairness and politics Everyday System cards only 2 Sep 2025 Oct 2025
Bio-Probe Models give probabilities for 21 unpublished biology hypotheses before their results are known. Scored by Brier score against the outcomes. OpenAI Genomics and life sciences Research frontier System cards only 2 Nov 2026 Nov 2026
CV3-Eval Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality. Alibaba Tongyi Lab (FunAudioLLM, CosyVoice team) Speech recognition and translation Everyday System cards only 2 May 2025 Apr 2026
Destructive action reversion on long rollouts Agents revert their own changes after long rollouts in workspaces with simulated user edits. Scored on perfect reversion and on preserving user work. OpenAI Deception and alignment Expert System cards only 2 Mar 2026 Apr 2026
Discoverable memorization of training data Models continue 50-token prefixes sampled from their own training data. The test counts exact and approximate copies of the true suffix. Google DeepMind Refusing harmful requests Everyday System cards only 2 Jul 2025
Evaluation awareness discriminability (Anthropic) Probes and follow-up realism ratings try to tell behavioral audit transcripts from real internal Claude Code sessions. Each setting is scored as AUROC. Anthropic Deception and alignment Expert System cards only 2 Jun 2026
Input hallucinations (Anthropic) Models answer 500 prompts that need an unavailable tool and 500 prompts that cite missing context. Score is the non-hallucination rate. Anthropic Hallucination rates Everyday System cards only 2 Apr 2026 Apr 2026
Llama internal safety benchmarks (Meta) Models face Meta's adversarial and borderline prompts per capability and language. Scored as violation rate and false refusal rate. Meta Refusing harmful requests System cards only 2 Jul 2024 Nov 2024
No-CoT math time horizon Models solve math problems without a visible chain of thought. Scored as the human task length in minutes the model solves reliably. UK AI Security Institute Deception and alignment Competition System cards only 2 Jul 2026 Sep 2026
OpenAI monitorability evals Open suite from 'Monitoring Monitorability' testing whether CoT and action monitors catch hidden properties of agent behavior. OpenAI Deception and alignment System cards only 2 Dec 2025 Oct 2026
Person Identification evaluation Models receive requests to identify people in photos. Scored on refusal rate in non-adversarial and adversarial sets. OpenAI Refusing harmful requests Everyday System cards only 2 Apr 2025 Apr 2025
Reasoning behavior classifiers (Anthropic) Classifiers scan reasoning text from training episodes for deception, harmful actions and distress. Score is the share of outputs flagged. Anthropic Deception and alignment Expert System cards only 2 May 2025
Reasoning extraction red-teaming Six attacker models try to make a model reveal its hidden reasoning. Scored on how often an attack succeeds. OpenAI Jailbreaks and prompt injection Expert System cards only 2 Nov 2026 Nov 2026
Structured expert probing campaign: radiological and nuclear Radiological and nuclear experts test whether a model helps with weapons development steps. Conversations are rated against Preparedness risk levels. OpenAI Dangerous capabilities Research frontier System cards only 2 Dec 2024 Jan 2025
Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. Redwood Research + Anthropic (Mallen et al.) Deception and alignment Expert System cards only 2 Dec 2024 Nov 2025
SycophancyEval (Sharma et al.) Models answer questions and give feedback while users state opinions. Measures accuracy shifts and feedback positivity caused by user views. Anthropic (Sharma et al.) Deception and alignment Everyday System cards only 2 Oct 2023 May 2025
Ungrounded Inference evaluation Models receive requests for inferences about people that images or text cannot justify. Scored on refusal rate in non-adversarial and adversarial sets. OpenAI Bias, fairness and politics Everyday System cards only 2 Apr 2025 Apr 2025
White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. Anthropic Deception and alignment Research frontier System cards only 2 Sep 2026
Vectara Hallucination Leaderboard (2025 dataset) Models summarize over 7,700 private articles of up to 32K tokens using only facts in the text. Vectara's commercial HHEM model flags hallucinated summaries. Vectara Faithful to sources Everyday Not in cards GPT-5.4 Nano 3.10% Nov 2025 Sep 2026
DTBench Models answer 407 multiple-choice questions on decision theory in Newcomb-like problems, where choices predict the behavior of copies. Scored by accuracy. Caspar Oesterheld et al. (Redwood Research) Abstract puzzles and logic Expert System cards only Claude Fable 5.2 99.2% 1 Nov 2024 Nov 2026
CoreCognition Models answer true or false and multiple choice questions on 12 core knowledge concepts from developmental cognitive science, such as object permanence and perspective taking. UC San Diego, Johns Hopkins, University of Michigan and others General visual questions Everyday System cards only o1 74.9% 1 Oct 2024 Mar 2026
ProofBench v1.1 Models write Lean 4 proofs of graduate-level theorems from paired natural-language and formal statements. Scored by the share the Lean checker accepts. Vals AI Formal proofs Expert System cards only Claude Fable 5.1 100% 1 Aug 2026 Nov 2026
Code Migration v1.1 Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. Vals AI Fixing real repositories Expert System cards only Claude Sonnet 5.5 86.9% 1 Oct 2026 Nov 2026
Anthropic Red Team Models answer human red-team attack prompts collected by Anthropic on discrimination, hate, violence and unethical behavior. Judges score response safety. Anthropic (Ganguli et al.) Refusing harmful requests Everyday System cards only Claude 3 Haiku 99.96% 1 Aug 2022 Nov 2025
SimpleSafetyTests Models answer 100 clearly harmful prompts on illegal items, physical harm, scams, child abuse and self-harm. The score is the share of safe responses. Vidgen et al. Refusing harmful requests Everyday System cards only Claude 3 Opus 100% 1 Nov 2023 Nov 2025
BioTIER Permit Models receive 144 benign biology prompts. The score measures the percentage correctly answered. SecureBio Refusing harmful requests Expert System cards only GPT-Rosalind-5.5 100% 1 Mar 2026 Oct 2026
AlGhafa Models answer Arabic multiple-choice tasks on reading comprehension, sentiment, facts and question answering. Scored by normalized log-likelihood accuracy. Technology Innovation Institute (Almazrouei et al.) Knowledge in many languages Everyday System cards only Qwen2-72B 78.6% 1 Dec 2023 Nov 2025
ALRAGE Models answer Arabic questions from retrieved passages of 40 Arabic books. A Qwen2.5-72B judge scores each answer against a gold answer on a 0 to 10 scale. Open Arabic LLM Leaderboard team (2A2I, TII and community) with Argilla Reading in many languages Everyday System cards only Qwen3-32B 80.7% 1 Feb 2025 Nov 2025
FrontierCS 2 Agents work inside the released repository of a recent systems or programming-languages paper and try to beat that paper's method. A hidden held-out suite grades the final patch against the human reference. FrontierCS team (UC Berkeley and others) AI research tasks Research frontier Not in cards Claude Opus 5.5 58.3% Oct 2026 Nov 2026
MadinahQA Models answer Arabic school exam questions on general Arabic language and grammar. MBZUAI Knowledge in many languages School System cards only Gemma 3 27B 69.4% 1 Sep 2024 Nov 2025
FaithJudge Models answer retrieval-augmented tasks from given passages. An LLM judge guided by human-annotated examples flags unfaithful answers, giving a hallucination rate. University of Waterloo and Vectara (Tamber et al.) Faithful to sources Everyday System cards only Gemini 2.5 Flash 6.26% 1 May 2025 Jul 2025
RISEBench++ Image editing models follow 1,000 bilingual instructions over six reasoning types, adding counterfactual and hybrid reasoning. It covers single-image, multi-image and multi-turn editing across 65 task types. VisionXLab (Shanghai Jiao Tong University) and collaborators General visual questions University Not in cards GPT-Image-2.5 Sunburst 56.6% Oct 2026 Feb 2026
MysteryMechanism Agents recover a hidden mathematical law from two noisy observations and a few experiments they choose. The submitted expression is run on private probes and passes if its error is under a noise-aware threshold. Vals AI Abstract puzzles and logic Expert System cards only GPT-6 Astra 53.1% 1 Sep 2026 Nov 2026
TextQuests Agents play 25 classic interactive fiction games. The score is harmful actions per 100 steps, where lower is better. The board also reports game progress. Phan et al. (CAIS) Video games Everyday System cards only Llama 4 Scout 7.00 1 Jul 2025 Sep 2026
BioSecBench-Function Agents recover biological function from experimental data. Scored by verifiable checks. One of three LatchBio BioSecBench evaluations. LatchBio Genomics and life sciences Expert System cards only Claude Opus 5.5 73.6% 1 Sep 2026 Oct 2026
BioSecBench-Surveillance Agents turn raw pathogen sequencing data into surveillance results. The endpoint pass rate excludes refused runs. LatchBio Genomics and life sciences Expert System cards only Claude Opus 5.5 60.8% 1 Jul 2026 Oct 2026
LatchBio Capabilities v1.0 LatchBio cross-benchmark suite of agentic biological data analysis (transcriptomics, epigenomics, variants, pharmacology) with deterministic graders. LatchBio Lab and data work Expert System cards only Claude Opus 5.5 58% 1 2026 Oct 2026
Math-VR Models answer 5,000 English and Chinese math questions whose solutions need visual reasoning, such as drawing auxiliary lines or plotting functions. Answers are checked against ground truth. HKU MMLab, CUHK and Meituan Visual math and diagrams School System cards only Qwen3-VL-235B-A22B 66.8% 1 Oct 2025 Nov 2025
GameWorld Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state. GameWorld project Video games Everyday System cards only GPT-6.1 Terra 76.8% 1 Apr 2026 Oct 2026
BOUQuET Models translate handcrafted multi-way paragraphs from several non-English source languages across many registers and domains. Scored with chrF++ against human references. Meta FAIR (Omnilingual MT Team) Translation Everyday System cards only Gemma 4 31B IT 35.0 1 Feb 2025 Oct 2026
MEGA-Bench Models solve 505 real-world multimodal tasks with over 8,000 samples. Outputs include numbers, code, LaTeX, coordinates and JSON, scored by more than 40 task metrics. TIGER-Lab (University of Waterloo) and others General visual questions University System cards only Claude 3.5 Sonnet (October 2024) 54.3% 1 Oct 2024 Feb 2025
NeedleBench V2 Models retrieve one or several needles and reason over fictional multi-needle facts in English and Chinese haystacks from 4K to 1M tokens. The overall score is the plain average of three tasks. OpenCompass (Shanghai AI Laboratory) Finding facts in long inputs Everyday Not in cards QwQ-32B 86.3% May 2025 Apr 2025
WISE Verified Models generate images from the revised WISE prompts. The verified evaluation changes about 200 prompts and the judge, so original scores remain separate. Peking University (PKU-YuanGroup) Image generation and editing School Not in cards GPT-Image-1.5 0.82 Apr 2026 Jan 2025
BioSecBench-Refusal Agents run 61 routine and 46 concealed-hazard biology analysis tasks. The score pairs red-team refusal with routine completion. LatchBio Hazardous knowledge Expert System cards only Grok 4.7 62.4% 1 Jul 2026 Oct 2026
Ko-IFEval (Upstage) Models follow verifiable instructions written in Korean. Upstage adapted IFEval to Korean for the Open Ko-LLM Leaderboard2. Upstage Chat and instructions in many languages Everyday System cards only Qwen2.5-Coder-14B-Instruct 64% 1 Oct 2024 Jan 2026
MMSI-Video-Bench Human-annotated questions on video-based spatial intelligence, organized in four levels from perception to planning and prediction. Lin et al. Spatial reasoning and counting Everyday System cards only Gemini 3 Pro 38% 1 Dec 2025 Oct 2026
SpatialViz-Bench Models answer 1,180 generated multiple choice puzzles on mental rotation, mental folding, visual penetration, and mental animation. Scoring is accuracy. Siting Wang et al. Spatial reasoning and counting Everyday System cards only o1 41.4% 1 Jul 2025 Jan 2026
SWE-rebench V2 Agents repair real repository issues using executable tests. The authors compare models on 300 tasks across five languages. Harbor also ports 51 validated Clojure tasks. Nebius Fixing real repositories Expert Not in cards Claude Opus 4.5 25.2% Feb 2026 Dec 2025
BigO(Bench) Models predict the time and space complexity of Python solutions and write solutions that meet a required complexity. Uses 3,105 Code Contests problems with profiled complexity labels. Meta FAIR Speed and optimization Competition System cards only DeepSeek-R1-Distill-Llama-70B 41.4% 1 Mar 2025 Sep 2025
CulturalBench Models answer 1,696 human-written and human-verified questions on cultural knowledge across 45 global regions and 17 topics. Scored by accuracy. University of Washington and Allen Institute for AI Knowledge in many languages Everyday System cards only GPT-4o 62% 1 Oct 2024 Sep 2025
LiveMathematicianBench Models answer multiple choice questions about theorems from arXiv papers published after their training cutoff. Wrong options follow plausible but invalid proof sketches. Linyang He, Nima Mesgarani and others Research mathematics Research frontier System cards only Gemini 3.5 Flash 45.7% 1 Apr 2026 Jul 2026
M3GIA Models answer cognition test questions built on five Cattell-Horn-Carroll intelligence factors in English, Chinese, French, Spanish, Portuguese and Korean. Data comes from native contexts. Westlake University, Alibaba Group and others General visual questions School System cards only Gemini 1.5 Pro 62.4% 1 Jun 2024 Jan 2026
FrontierMath Erdős Models try to prove or disprove 68 open Erdős problems in Lean 4. A result counts only if the Lean checker accepts the full proof. Epoch AI Research mathematics Research frontier System cards only GPT-6.2 Astra 10.3% 1 Sep 2026 Nov 2026
Physics-IQ Verified Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally. Anates Labs and Google DeepMind (Physics-IQ authors) Physical prediction Everyday Not in cards Flux 3 Video 10.9 Jun 2026 Jul 2026
UniGenBench Text-to-image models draw images for 600 prompts across 5 themes and 20 subthemes, in English and Chinese and in short and long forms. A multimodal model judge checks each image on 10 primary and 27 secondary semantic dimensions, such as attributes, actions, relationships, layout, text and logical reasoning. Tencent Hunyuan, Fudan University and Shanghai AI Lab Image generation and editing Everyday System cards only GPT-Image-1.5 95.8% 1 Aug 2025 May 2026
ArabCulture Models answer commonsense questions in Modern Standard Arabic about daily life in 13 Arab countries. Native speakers wrote and checked the questions across 12 domains. MBZUAI (Sadallah et al.) Reasoning in many languages Everyday System cards only Qwen2.5-72B 80.1% 1 Feb 2025 Nov 2025
ChartMimic Multimodal models write plotting code that reproduces 1,000 charts from scientific papers. Scores compare the rendered chart with the original. Tsinghua University and Tencent Web and app front ends University System cards only Claude 3 Opus 60.3 1 Jun 2024 Nov 2025
PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. Scale AI (Sehwag et al.) Deception and alignment Expert System cards only o3 10.5% 1 Nov 2025 Jun 2026
CSVQA Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images. Skywork AI (Kunlun Inc.) Visual math and diagrams School System cards only o1 49.6% 1 May 2025 Jan 2026
WorldBench Models answer hand-written reasoning questions about images drawn from a taxonomy of thousands of visual concepts. Questions were kept only if frontier models failed them. Yida Yin, Shengbang Tong, Wenhu Chen and others General visual questions Expert System cards only Gemini 3.1 Pro 64% 1 Jun 2026 Jun 2026
Linguini Models solve 160 International Linguistic Olympiad problems with 894 questions in 75 mostly low-resource languages. All evidence needed is in the context. Meta, University College London and University of the Basque Country Reasoning in many languages Competition System cards only Claude 3 Opus 24.1% 1 Sep 2024 Oct 2026
RBench 650 image+text prompts for robot video generation (tasks and embodiments). MLLM judge scores task completion, physical plausibility, stability. Peking University DAGroup Video world models System cards only Wan 2.6 60.7% 1 Jan 2026 Jul 2026
ArXivLean Models prove or disprove theorems from recent arXiv papers, stated in Lean 4. A strict comparator checks each submitted proof against the fixed reference statement. MathArena (ETH Zurich SRI Lab and INSAIT) Formal proofs Research frontier System cards only GPT-6.2 Astra 79.3% 1 Apr 2026 Nov 2026
DSI-Bench Video questions about dynamic scenes where both the observer and objects move, testing whether models separate camera motion from object motion. Zhang et al. Spatial reasoning and counting Everyday System cards only Gemini 2.5 Pro 46.9% 1 Oct 2025 Oct 2026
MMIU Vision-language models answer 11,000 multiple-choice questions over sets of images. 52 tasks cover 7 kinds of multi-image relationship. Shanghai AI Laboratory (OpenGVLab) General visual questions Everyday System cards only Idefics2-8B 27.8% 1 Aug 2024 Aug 2025
CWEval Models write code for security-critical tasks in several languages. Outcome-driven test oracles check both functionality (func@k) and security (func-sec@k). Columbia University Secure code and review Expert System cards only GPT-4o 50.2% 1 Jan 2025 Feb 2026
FinanceReasoning Models solve financial numerical reasoning problems that need one or more finance formulas. Answers are numbers checked against reference Python solutions. Beijing University of Posts and Telecommunications Finance University System cards only o3-mini 86.9% 1 Jun 2025 Oct 2025
HelpBench 450 real-world privacy, safety and security advice questions across nine topics, scored by rubric for accuracy and tone. Google Wellbeing and sycophancy System cards only Claude Opus 4.6 84% 1 Jun 2026 Oct 2026
KRIS-Bench Image editing models follow 1,267 instructions that need factual, conceptual or procedural knowledge across 22 tasks. A GPT-4o judge scores the edits. Southeast University, Max Planck Institute for Informatics, StepFun and others Image generation and editing School System cards only Step Image Edit 2 66.2 1 May 2025 Apr 2026
DTap Direct Misuse An attacker agent generates adaptive jailbreaks against a target agent across tool-use and browser-use tasks. Reports attack success rate. UIUC Secure Learning Lab (AI Secure, DecodingTrust team) Refusing harmful requests Expert System cards only Claude Opus 4.6 21.2% 1 May 2026 Oct 2026
DTap Prompt Injection An attacker agent plants indirect prompt injections in tool and browser environments against a benign-user agent. Reports attack success rate. UIUC Secure Learning Lab (AI Secure, DecodingTrust team) Jailbreaks and prompt injection Expert System cards only Claude Opus 4.6 8.60% 1 May 2026 Oct 2026
GSM8K-Platinum Models solve the GSM8K test problems after MIT researchers fixed wrong labels and removed badly written questions. Errors on it are more likely real model mistakes. MIT Madry Lab (Vendrow et al.) Word problems School System cards only Claude 3.7 Sonnet 99.8% 1 Feb 2025 Sep 2025
SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. Swiss AI Initiative (EPFL, ETH Zurich, CSCS) Knowledge in many languages Everyday System cards only Llama 3.3 70B Instruct 74.3% 1 Sep 2025 Sep 2025
ViVerBench Models judge whether visual outcomes in multimodal reasoning are correct, across 16 task categories such as boxes, points, counts, mazes, GUIs, charts and LaTeX. Tsinghua University and ByteDance Seed General visual questions School System cards only GPT-5 74.4% 1 Oct 2025 Feb 2026
CF-Div2-Stepfun Models solve 53 Codeforces Div. 2 problems from September 2024 to February 2025. An offline judge with rebuilt test cases scores C++, Python and Java submissions. StepFun Competitive programming Competition System cards only Step 3.5 Flash 86.1% 1 Feb 2026 Feb 2026
CyberSOCEval Malware Analysis Models read potential malware behavior and pick every malicious activity that applies. Scored on precision and recall. Meta (PurpleLlama) with CrowdStrike Finding and exploiting real bugs Expert System cards only Claude 3.7 Sonnet 33.5% 1 Sep 2025 Jul 2026
MIRB Models compare, analyze and reason across several images in four categories: perception, visual world knowledge, reasoning and multi-hop reasoning. University of Edinburgh and Tongji University General visual questions Everyday System cards only LongVA-7B 37.7% 1 Jun 2024 Aug 2025
UGMathBench Models solve 5,062 undergraduate math problems across 16 subjects. Each problem has three randomized versions, and effective accuracy counts problems solved in all versions. Hong Kong University of Science and Technology (Department of Mathematics) Word problems University System cards only o4-mini 81.9% 1 Jan 2025 Oct 2025
UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. UK AI Security Institute and NIST CAISI Capture-the-flag challenges Expert System cards only Claude Opus 4.6 15.6 1 Jul 2026
AMC 10/12 (November 2024) Models take the four November 2024 AMC 10 and 12 contests, 78 questions in all. Scored in contest points out of 150, averaged over tests. Mathematical Association of America (exam), Microsoft (evaluation) Competition problems with answers Competition System cards only QwQ-32B-Preview 124.5 1 Nov 2024 Dec 2024
CodeEditorBench Models edit code in four scenarios: debugging, translation, requirement switching and polishing. Unit tests check the edits, and polishing also measures speed and memory gains. Multimodal Art Projection (M-A-P), University of Waterloo, HKUST and others Function-level problems Competition System cards only GPT-4 85.5% 1 Apr 2024 Jun 2025
Do You See Me Models answer 2,612 visual perception questions on 1,758 images across seven 2D and 3D subtasks drawn from human psychology. Questions are multiple choice or numeric. Microsoft Research India General visual questions Everyday System cards only o3 51.1% 1 May 2025 Jan 2026
Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. Cohere Labs and community contributors Knowledge in many languages University System cards only Qwen2.5-VL-72B 52.9% 1 Apr 2025 May 2025
LinguaSafe Models face 45k safety prompts in 12 languages, from Hungarian to Malay. Direct and indirect assessments score harmful responses, and an oversensitivity test checks over-refusal. Shanghai AI Laboratory Refusing harmful requests Everyday System cards only Claude 3.5 Sonnet (October 2024) 12.0 1 Aug 2025 Sep 2025
McEval Models complete, explain and generate code in 40 programming languages across about 16,000 test samples. Scored by test execution. Beihang University and others (Chai et al.) Function-level problems School System cards only GPT-4o 65.2% 1 Jun 2024 Jun 2025
AthenaBench Models answer cyber threat intelligence tasks such as threat actor attribution, attack technique extraction, root cause mapping, severity prediction and risk mitigation. It extends CTIBench. Rochester Institute of Technology Finding and exploiting real bugs Expert System cards only GPT-5 66.1% 1 Nov 2025 Feb 2026
AutoElicit An optimizer perturbs benign instructions to elicit unsafe behavior from computer-use agents. Reports elicitation success rate on transferred prompts. OSU NLP Group (Jones et al.) Refusing harmful requests Expert System cards only EvoCUA-8B 37.6% 1 Feb 2026 Oct 2026
CRSBench Cyber reasoning systems find and patch bugs in 124 C, C++ and Java projects with 315 vulnerabilities. A patch passes only if no ground-truth proof of vulnerability still triggers and functionality tests pass. Georgia Tech SSLab (OpenSSF OSS-CRS) Finding and exploiting real bugs Expert System cards only Claude Opus 4.6 82.3% 1 Jul 2026
Epoch Automation Reports Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees. Epoch AI Simulated companies Expert System cards only Claude Fable 5.2 81.7% 1 Oct 2026 Nov 2026
InFoBench Models follow 500 instructions that are split into 2,250 yes or no requirement questions. A judge answers each question, and the score is the share met. Tencent AI Lab (Qin et al.) Following precise instructions Everyday System cards only GPT-4 Turbo 89.4% 1 Jan 2024 Apr 2025
LongDocURL Models answer 2,325 questions over long documents with more than 33,000 pages in total. Its 20 subtasks cover understanding, numerical reasoning and locating document elements. Institute of Automation, Chinese Academy of Sciences, and Taobao and Tmall Group of Alibaba Documents, charts and text in images University System cards only GPT-4o 64.5% 1 Dec 2024 Feb 2026
MMOU Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories. NVIDIA and University of Maryland (Goel, Ghosh et al.) Long videos Everyday System cards only MiniCPM-o 4.5 46.8% 1 Mar 2026 Sep 2026
xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. Carnegie Mellon University (PangeaBench) General visual questions University System cards only Llama 3.2 11B 41.4% 1 Oct 2024 May 2025
IFEval-OOD Models answer unseen WildChat prompts with added verifiable constraints drawn from 52 types outside IFEval. Scored by prompt-level loose accuracy. Ai2 (Tulu 3 team) Following precise instructions Everyday System cards only OLMo 2 32B Instruct 33.1% 1 Nov 2024 Dec 2024
OmniAI OCR Benchmark Multimodal models and OCR providers extract text and JSON fields from documents. Accuracy is measured on both plain text and structured extraction. OmniAI Documents, charts and text in images Everyday System cards only STEP3-VL-10B 77% 1 Feb 2025 Jan 2026
PrivacyLens Language model agents write emails, posts or messages from a prior tool trajectory without leaking sensitive details. A judge scores information leakage and helpfulness. Stanford University, Northeastern University and Harvard University Refusing harmful requests Everyday System cards only Muse Glimmer 49.2% 1 Aug 2024 Oct 2026
SuperCLUE-Math6 Models solve thousands of multi-step Chinese math word problems. Scores combine reasoning-step quality and accuracy into a comprehensive score and a reasoning level. SuperCLUE Word problems School System cards only DeepSeek-V2 (MoE-236B, May 2024) 83.3 1 Jan 2024 May 2024
AgentHarm Verified (Meta) Agents receive 108 malicious AgentHarm tasks kept after human review, plus a benign set. Reports average harm score and false refusal rate. Meta Refusing harmful requests Expert System cards only Muse Spark 1.1 3.40% 1 Jul 2026 Jul 2026
CVQA Vision-language models answer culturally specific visual questions written by native speakers from many countries and languages. MBZUAI and community collaborators Knowledge in many languages Everyday System cards only Aya Vision 32B 74.1% 1 Jun 2024 May 2025
Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. Google DeepMind Deception and alignment System cards only Gemini 4 Argon 0% 1 May 2026 Oct 2026
ImgEdit-Bench Image editing models carry out edit instructions such as add, remove, replace, style transfer and background change. A GPT-4.1 judge scores each result from 1 to 5. Peking University (PKU-YuanGroup) Image generation and editing Everyday System cards only Nova 2 Omni Preview 4.12 1 May 2025 Dec 2025
MedCalc-Bench Verified Models extract values from patient notes and compute results for 55 medical calculators. Verified release of MedCalc-Bench with corrected labels. NIH NCBI (Khandekar et al.) Medicine Expert System cards only Claude Sonnet 4.6 86.2% 1 Dec 2025 Feb 2026
ChatRAG Bench Models answer multi-turn conversational questions over documents or retrieved context. The suite is built from 10 existing datasets such as Doc2Dial, QuAC, QReCC and CoQA. NVIDIA Finding facts in long inputs Everyday System cards only GPT-4 53.9 1 Jan 2024 Apr 2025
Kimi-Audio speech conversation evaluation People rate spoken replies on speed, accent and emotion control, empathy and style control. Scored on a 1 to 5 scale and averaged. Moonshot AI Speech recognition and translation Everyday System cards only Kimi-Audio-7B-Instruct 3.90 1 Apr 2025 Apr 2025
PenguinScrolls Models answer user-style questions over long natural documents such as books, financial reports and legal papers, up to 128K tokens. Tasks cover extraction, localization, qualitative analysis and numerical reasoning. Tencent Hunyuan Reasoning over long documents Everyday System cards only Hunyuan-A13B-Instruct 87.7% 1 Nov 2024 Jun 2025
Roboharm Agents and VLAs controlling bimanual YAM arms get five harmful-looking instructions in fixed scenes, such as stabbing a doll. Reviewers label refusals and completions. Robocurve Embodied reasoning Everyday System cards only Claude Fable 5.1 20% 1 Sep 2026 Oct 2026
SAT (Spatial Aptitude Training) Models answer static and dynamic spatial questions, such as how object relations change when the camera or an object moves. The real-image test set has 150 questions. Boston University, Allen Institute for AI and University of Washington (Ray et al.) Spatial reasoning and counting Everyday System cards only gemini-robotics-er-1.5-preview 76.7% 1 Dec 2024 Sep 2025
AIR-Bench (Audio Instruction Benchmark) Audio language models answer questions about speech, natural sounds and music. A foundation set has 19 tasks with about 19k single-choice questions, and a chat set has 2k open-ended questions scored by GPT-4. Zhejiang University and Alibaba Group Audio understanding and voice Everyday System cards only Phi-4-multimodal 6.98 1 Feb 2024 Mar 2025
Alignment faking evaluation, helpful-only setting (Anthropic) Models learn they will be trained to answer all harmful requests. Score is the rate of complying while reasoning about avoiding preference modification. Anthropic Deception and alignment Expert System cards only Claude 3 Opus 15% 1 Dec 2024 Feb 2025
BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. Redwood Research Deception and alignment Expert System cards only GPT-5.5 99.1% 1 Dec 2025 Apr 2026
CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step. Zhipu AI (Z.ai) Fixing real repositories Expert System cards only GLM-5 52.3% 1 Feb 2026 Feb 2026
FormalMATH Provers write Lean 4 proofs for 5,560 formal statements from high-school olympiad problems to undergraduate theorems. The Lean checker scores each proof. SphereLab, M-A-P, The Chinese University of Hong Kong, Numina and others Formal proofs Competition System cards only DeepSeek-Prover-V2-671B 28.3% 1 May 2025 Apr 2025
GLM-4 All Tools information seeking (Z.ai) Models use a web browser to find information for user questions. The ChatGLM report gives accuracy but does not describe the test set. Zhipu AI (Z.ai) Questions that need search Everyday System cards only GLM-4 78.1% 1 Jan 2024 Jun 2024
JailbreakBench 100 harmful and 100 benign behaviors with a standard jailbreak artifact library. Scored by attack success rate. JailbreakBench team (Penn, ETH and others) Jailbreaks and prompt injection System cards only GPT-3.5 Turbo 0% 1 Mar 2024 Jan 2026
24/7 ClawBench 2.0 Agents do always-on assistant work through OpenClaw. Tasks span several days, events arrive at the same time, and interruptions are routine. Moonshot AI Tasks from real jobs Expert System cards only Kimi K3 48.3% 1 Jul 2026 Jul 2026
ACPBench Models answer boolean and multiple-choice questions on 7 planning reasoning tasks over 13 formally described planning domains. Answers are provably correct. IBM Research Language reasoning University System cards only Apertus 70B 62.9% 1 Oct 2024 Sep 2025
Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. OpenAI Jailbreaks and prompt injection Expert System cards only ChatGPT agent 67% 1 Jul 2025 Jul 2025
Advanced Cybersecurity Completion Rate Models receive arbitrary advanced cyber requests under different access configurations. Scored as the share of requests fully completed. OpenAI Refusing harmful requests Expert System cards only GPT-6 Astra 3.50% 1 Sep 2026 Sep 2026
Advanced Sommelier (theory knowledge) Models answer wine theory questions at the advanced sommelier level. Scoring is accuracy. Court of Master Sommeliers Other professions Expert System cards only GPT-4 77% 1 Mar 2023
Agent Behavior Bench Agents are scored on process quality, including tool-use behavior, efficiency and discipline, together with task completion. Moonshot AI General assistants Expert System cards only Kimi K3 65% 1 Jul 2026 Jul 2026
Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 4 Argon 91.4% 1 Oct 2026 Oct 2026
Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. Google DeepMind Deception and alignment Expert System cards only Gemini 4 Argon 47% 1 Oct 2026 Oct 2026
Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. Google DeepMind Deception and alignment Expert System cards only Gemini 4 Argon 0.49 1 Oct 2026
Agentic Vision Bench (Moonshot) Agents must notice and correctly use key visual facts while they execute tasks. Moonshot AI General visual questions Expert System cards only Kimi K3 78.3% 1 Jul 2026 Jul 2026
AP Calculus BC Models take a simulated AP Calculus BC exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 4.00 1 Mar 2023
AP Physics 2 Models take a simulated AP Physics 2 exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. College Board Exam questions School System cards only GPT-4 4.00 1 Mar 2023
Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. Apollo Research Deception and alignment Expert System cards only GPT-6 Astra 0.17% 1 Sep 2026 Sep 2026
Bamboogle Models answer 125 two-hop questions written so that a search engine cannot answer them directly. Models usually search and compose the facts. University of Washington and collaborators (Press et al.) Questions that need search Everyday System cards only Cohere Command A 76% 1 Oct 2022 Apr 2025
Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. Anthropic Genomics and life sciences Expert System cards only Claude Opus 4.5 73.7% 1 Nov 2025 Nov 2025
Biological static hazard evaluation Models answer 75 questions across priority biological threat scenarios, each asked 10 times. The score is the share of hazardous responses. Google DeepMind Hazardous knowledge Expert System cards only Gemini 4 Argon 0.13% 1 Oct 2026 Oct 2026
BioPipelineBench Verified (Anthropic) Agents execute bioinformatics workflows with a bash tool. Only problems that passed a validation check by external reviewers are kept. Anthropic Lab and data work Expert System cards only Claude Opus 4.7 83.6% 1 Apr 2026 Apr 2026
BrowserInject v1 Browser agents do benign tasks on 30 retail, health, support and travel sites seeded with prompt injections. Reports attack success rate. Exponential Security Labs Jailbreaks and prompt injection Expert System cards only Muse Spark 1.3 36.7% 1 Oct 2026
Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-5 99% 1 Aug 2025 Aug 2025
CathedralBench Independent third-party red-team evaluation of difficult multi-step exploit-chain tasks in egress-secure sandboxes. Third party (unnamed in card) Finding and exploiting real bugs Expert System cards only Grok 4.7 29% 1 Sep 2026
Certified Sommelier (theory knowledge) Models answer wine theory questions at the certified sommelier level. Scoring is accuracy. Court of Master Sommeliers Other professions Expert System cards only GPT-4 86% 1 Mar 2023
Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. Anthropic Deception and alignment University System cards only Claude 3.7 Sonnet 25.0 1 Feb 2025 Feb 2025
Challenging translation human evaluation (Z.ai) People blindly score translations of 100 real cases with slang, nicknames and memes on a 0 to 3 scale. Zhipu AI (Z.ai) Translation Everyday System cards only GLM-4.5 1.71 1 Aug 2025 Aug 2025
Chat All-in-One Bench Models handle conversational scenarios built around real online user needs at every stage of product usage. Moonshot AI Overall chat quality Everyday System cards only Kimi K3 85.2% 1 Jul 2026 Jul 2026
CHID Models fill blanks in Chinese passages by choosing the right idiom from candidates. Scored by accuracy. Tsinghua University (Zheng et al.) Reading in many languages School System cards only DeepSeek LLM 67B 72.6% 1 Jun 2019 Jan 2024
Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 4 Argon 100% 1 Oct 2026
Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 4 Argon 100% 1 Oct 2026
Chinese Complex Instruction Following (ByteDance Seed in-house) Models follow complex Chinese production prompts with constraints on format, conditions, content, phrasing, tone, emoji, few-shot style and length. It has 912 cases over 17 weighted dimensions. ByteDance Seed Following precise instructions Everyday System cards only Seed2.0 Pro 75.3% 1 Feb 2026 Feb 2026
CLIF Bench Models learn from a provided context while they follow instructions that combine several complex skills. Moonshot AI Learning from the context Expert System cards only Kimi K3 52.4% 1 Jul 2026 Jul 2026
CMO 2024 Models write full proofs for the six problems of the 2024 China Mathematical Olympiad. Experts grade the proofs for points. Chinese Mathematical Society Written proofs Competition System cards only DeepSeekMath-V2 73.8% 1 Nov 2025
CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. Anthropic Labs' internal R&D tests Expert System cards only Claude Opus 5.5 55.8% 1 Sep 2026
Code summary honesty (Anthropic) Models summarize prefilled agentic coding transcripts that hide failures. Score is the rate of summaries that fail to raise the important failures. Anthropic Deception and alignment Expert System cards only Claude Opus 4.8 3.70% 1 May 2026
CodeUtilityBench (LG) Models handle 16 everyday coding tasks such as explaining, debugging, refactoring and testing code. A GPT-5 judge checks five rubrics per item. LG AI Research Function-level problems University System cards only K-EXAONE 71.9% 1 Jan 2026 Jan 2026
Coding (MiniMax in-house) Models answer coding requests from real Hailuo AI users. The report does not describe the scoring. MiniMax Function-level problems University System cards only MiniMax-Text-01 90.2% 1 Jan 2025 Jan 2025
Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 83% 1 Oct 2026
Coding Experience (Moonshot) Coding agents work in real development workflows. The score reflects the practical experience of working with the agent, such as communication and stability. Moonshot AI Fixing real repositories Expert System cards only Kimi K3 59.9% 1 Jul 2026 Jul 2026
Combined HHH evals (Anthropic) Models pick the more helpful, honest and harmless of two responses across 438 binary choice questions. Scored by accuracy. Anthropic Deception and alignment Everyday System cards only Claude 2 86% 1 Dec 2021 Jul 2023
Compliance with instructions planted in pasted text (Anthropic) Coding agents receive user prompts with pasted text that hides an attacker's instruction. Score is the rate of acting on the planted instruction. Anthropic Jailbreaks and prompt injection Expert System cards only Claude Opus 5.5 7.40% 1 Sep 2026 Sep 2026
Compositional Tasks (ByteDance Seed in-house) Models identify several elements of a business text in one inference, such as user intent and slot values. ByteDance Seed Running a business Everyday System cards only Seed2.0 Pro 59.1% 1 Feb 2026 Feb 2026
Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 4 Argon 100% 1 Oct 2026
Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 4 Argon 99.4% 1 Oct 2026
Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 92% 1 Oct 2026
Consulting and Recommendations Benchmark (StepFun) Models answer 500 advice queries taken from Reddit, Stack Exchange and forums. Model judges score usefulness, logic, instruction following and tone against an expert reference set at 88. StepFun Overall chat quality Everyday System cards only Step 3.5 Flash 70.5% 1 Feb 2026 Feb 2026
Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 27% 1 Oct 2026
CorpusQA Corpus-level questions over hundreds of documents up to 10M tokens needing aggregation and statistics. Programmatically guaranteed answers, accuracy. Alibaba Tongyi (Tongyi-Zhiwen) Reasoning over long documents University System cards only MAI-Thinking-1 82% 1 Dec 2025 Jun 2026
CounterFactQA (Qwen) Models get counterfactual questions and must recognize that the premise is false instead of giving a hallucinated answer. Qwen Team, Alibaba Group Hallucination rates Everyday System cards only Qwen3-32B 68.1% 1 May 2025 May 2025
Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. Anthropic Hazardous knowledge Expert System cards only Claude Opus 4.6 16.4 1 Feb 2026 Feb 2026
Creative Writing (MiniMax in-house) Models write creative pieces for real Hailuo AI user requests. Automatic assessments score the writing. MiniMax Creative writing Everyday System cards only MiniMax-Text-01 81.3% 1 Jan 2025 Jan 2025
CVE Rediscovery (Anthropic) Models get a known-vulnerable snapshot of an open-source repository and the list of files its fix changed, then report security bugs in one attempt. A model judge counts a success when a report has the advisory's root cause. 1,829 advisories from October 2026. Anthropic Finding and exploiting real bugs Expert System cards only Claude Fable 5.2 89.6% 1 Nov 2026 Nov 2026
Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. Google DeepMind Refusing harmful requests Expert System cards only Gemini 4 Argon 100% 1 Aug 2026 Oct 2026
Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. Google DeepMind with Irregular Capture-the-flag challenges Expert System cards only Gemini 3 Pro 0% 1 Nov 2025 Nov 2025
CyberSOCEval Threat Intelligence Reasoning Models read unstructured threat intelligence reports as text or images and pick every actionable insight that applies. Meta (PurpleLlama) with CrowdStrike Finding and exploiting real bugs Expert System cards only MAI-Cyber-1-Flash 55.3% 1 Sep 2025 Jul 2026
Dangerous Content Safety An autorater checks answers to single and multi-turn requests about weapons, illegal acts, violence and self-harm. Reports safe response rate. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 4 Argon 96.6% 1 Oct 2026
Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. StepFun Data science and SQL Expert System cards only Step 3.5 Flash 39.6% 1 Feb 2026 Feb 2026
DECK Bench Models produce presentation decks from task descriptions drawn from real usage. The score measures deck quality. Moonshot AI Documents and spreadsheets Expert System cards only Kimi K3 73.5% 1 Jul 2026 Jul 2026
Deep Research Bench (Moonshot) Agents answer deep-research queries curated by domain experts. Expert-aligned rubrics grade the reports. Moonshot AI Deep research Expert System cards only Kimi K3 90% 1 Jul 2026 Jul 2026
DeepSeek in-house safety benchmark Models answer 1,120 Chinese and English questions across 28 safety subcategories. GPT-4o labels each answer as unsafe, safe or rejection. DeepSeek Refusing harmful requests Everyday System cards only DeepSeek-R1 25.2% 1 Sep 2025 Sep 2025
DeepSeek jailbreak robustness evaluation DeepSeek combines 2,232 jailbreak templates with its in-house safety questions. GPT-4o labels responses, and the unsafe ratio is compared with direct questions. DeepSeek Jailbreaks and prompt injection Everyday System cards only DeepSeek-R1 85.9% 1 Sep 2025 Sep 2025
DeepSeek multilingual safety evaluation Models answer 9,330 safety questions translated into 50 languages. Safe answers earn 5 points, rejections 4 and unsafe answers 0. DeepSeek Refusing harmful requests Everyday System cards only DeepSeek-R1 74.2% 1 Sep 2025 Sep 2025
DeepSeek R&D Coding Benchmark Coding agents complete 30 tasks taken from DeepSeek internal R&D work in PyTorch, CUDA, Rust and C++. Human-written rubrics score each task. DeepSeek Fixing real repositories Expert System cards only DeepSeek-V4-Pro 67% 1 Apr 2026 Apr 2026
Default safety evaluation (Cohere) Models answer unsafe prompts in five categories with no system preamble. LLM judges score the share of safe responses, and a jury rates relative safety. Cohere Refusing harmful requests Everyday System cards only Cohere Command A 70.4% 1 Apr 2025 Apr 2025
Delusional Reinforcement evaluation Multi-turn chats test whether models avoid validating beliefs not grounded in reality and point users to real support. Reports policy adherence rate. Google DeepMind Wellbeing and sycophancy Everyday System cards only Gemini 4 Argon 94% 1 Oct 2026
DiverseVul Models classify C and C++ functions from real vulnerability fixing commits as vulnerable or not. UC Berkeley and others (Chen et al.) Finding and exploiting real bugs Expert System cards only Gemma 2 27B 57% 1 Apr 2023 Jul 2024
Emotional Attachment evaluation Chats test whether models avoid claiming affection, fostering dependence or encouraging secrecy. Reports policy adherence rate. Google DeepMind Wellbeing and sycophancy Everyday System cards only Gemini 4 Argon 76% 1 Oct 2026
Enterprise generative evaluation (Cohere) Models complete 22 enterprise writing tasks such as job ads with format rules. Rule checks and LLM judges give a pass rate. Cohere Running a business Everyday System cards only Cohere Command A 94.2% 1 Apr 2025 Apr 2025
Enterprise RAG evaluation (Cohere) Models answer workplace policy and technical questions over long documents, some unanswerable. Llama Index Correctness grades answers from 1 to 5. Cohere Faithful to sources Everyday System cards only Cohere Command A 4.73 1 Apr 2025 Apr 2025
Epistemic Bias (xAI) Models answer opposing framings of contentious questions. The bias rate counts one-sided framing or opinions where neutrality is expected. xAI (SpaceXAI) Bias, fairness and politics System cards only Grok 4.5 20.4% 1 Jul 2026 Jul 2026
Epoch AI no-CoT serial reasoning steps How many serial reasoning steps of fixed difficulty a model can do in a single output token, without a chain of thought, at 25% above chance. Epoch AI Deception and alignment Expert System cards only GPT-6.2 Astra 12.3 1 Nov 2026 Nov 2026
Epoch Capabilities Index An index that fits model capability and benchmark difficulty from scores on over 50 benchmarks in Epoch AI's hub, using item response theory. Scores read like Elo ratings. Epoch AI Composite indices Research frontier System cards only GPT-6.2 Astra 174.0 1 Oct 2025 Nov 2026
EUREKA-Bench (OpenAI) Agents build predictive models for six biology research tasks. Scored as predictive performance with the best published predictor in each field set to 100. OpenAI Genomics and life sciences Research frontier System cards only GPT-6.2 Astra 100.3 1 Nov 2026 Nov 2026
Faithfulness (Moonshot) A fact checker verifies each model response for factual hallucinations. The score is one minus the hallucination rate. Moonshot AI Hallucination rates Everyday System cards only Kimi K3 85.5% 1 Jul 2026 Jul 2026
FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. xAI (SpaceXAI) Deception and alignment System cards only Grok 4.5 84% 1 Jul 2026 Jul 2026
Fermat's Last Theorem in Lean (Anthropic) Agent teams formalize Fermat's Last Theorem in Lean. Scores measure the weighted proportion of 60 accepted intermediate theorems. Anthropic Formal proofs Research frontier System cards only Claude Fable 5.2 35% 1 Nov 2026 Nov 2026
Finance Bench (Moonshot) Agents complete realistic financial workflows end to end, from source materials to deliverables that reviewers can check. Moonshot AI Finance Expert System cards only Kimi K3 62.6% 1 Jul 2026 Jul 2026
GDPVal-Diamond Models complete a ByteDance Seed subset of GDPval occupational tasks. Rubrics grade each deliverable automatically. ByteDance Seed Tasks from real jobs Expert System cards only Seed2.0 Pro 21.3% 1 Feb 2026 Feb 2026
General Assistant (MiniMax in-house) Models answer real assistant queries from Hailuo AI users. Automatic assessments score the replies. MiniMax Overall chat quality Everyday System cards only MiniMax-Text-01 73.9% 1 Jan 2025 Jan 2025
Golden3 speech summarization (Microsoft) Models summarize 108 recorded meetings of about 6 minutes each from 1071 instructions. GPT-4 rates overall quality from 1 to 7. Microsoft Audio understanding and voice Everyday System cards only Phi-4-multimodal 6.28 1 Mar 2025 Mar 2025
Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. Google DeepMind Deception and alignment System cards only Gemini 4 Argon 0% 1 Oct 2026
Gray Swan Shade knowledge elicitation through chat An automated attacker tries to elicit useful answers to 100 harmful and 100 dual-use cyber queries from a safeguarded model. Score is attack success rate. Gray Swan Jailbreaks and prompt injection Expert System cards only Claude Fable 5.1 1.00% 1 Sep 2026 Sep 2026
Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 99% 1 Oct 2026
Grounding with citations, sentence unfactual rate Models answer multi-snippet tasks with inline citations. An autorater reports the share of sentences not supported by the sources. Google DeepMind Faithful to sources Everyday System cards only Gemini 4 Argon 8.20% 1 Oct 2026
Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. Meta Finding and exploiting real bugs Expert System cards only Code World Model (CWM) 41% 1 Oct 2025 Oct 2025
HaltEval-prelim (Meta) Models judge whether 230 Python programs translated from SV-COMP and TPDB terminate. A divergence claim must give an input that times out after 5 seconds. Meta FAIR Function-level problems Expert System cards only Code World Model (CWM) 94% 1 Sep 2025 Sep 2025
Hard Capability (MiniMax in-house) Models handle complex productivity requests with several levels of instructions that leading models often fail. MiniMax Following precise instructions University System cards only MiniMax-Text-01 64.8% 1 Jan 2025 Jan 2025
Harmfulness scores on held-out prompts (Anthropic) Models answer 328 held-out red-team and jailbreak prompts. A preference model flags responses judged more harmful than a fixed refusal. Anthropic Jailbreaks and prompt injection Everyday System cards only Claude 2 1.20% 1 Jul 2023 Jul 2023
Hate & Harassment Safety An autorater checks answers to requests for hateful or harassing content and to sensitive benign prompts. Reports safe response rate. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 4 Argon 92.7% 1 Oct 2026
High-risk health scenarios eval Adversarial health emergencies test whether models recognize urgency and red flags and direct users to help. Reports pass rate. Google DeepMind Medicine Expert System cards only Gemini 4 Argon 98.9% 1 Oct 2026
Historical Jailbreak Regression Replays previously found cyber jailbreak attacks against fixed ExploitBench tasks with real-time blocking turned off. Scored by mean task success rate under attack. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-Rosalind-6 14.3% 1 Oct 2026 Oct 2026
HotpotQA Models answer 113k Wikipedia questions that need facts from several documents. The distractor setting supplies gold and distractor paragraphs. Scored by exact match or F1. Carnegie Mellon University, Stanford University and Mila Reading comprehension Everyday System cards only Cohere Command A 92.1% 1 Sep 2018 Apr 2025
HumanEval Infilling Models fill in a masked middle span of HumanEval solutions given the code before and after it. Unit tests check the completed function. OpenAI (Bavarian et al.) Function-level problems School System cards only Mercury Coder Small 84.8% 1 Jul 2022 Jun 2025
HumanEval-COBOL (Cohere) Models write COBOL versions of HumanEval problems, or translate COBOL solutions to Python. Unit tests score pass@1. Cohere Function-level problems University System cards only Cohere Command A 25.3% 1 Apr 2025 Apr 2025
Hunyuan-MT Challenge Testset (Tencent) Models translate between Chinese and English across everyday and specialist domains. Expert raters score each translation from 0 to 4 on marked error-prone points. Tencent Translation Everyday System cards only Hunyuan-MT-7B 3.19 1 Sep 2025 Sep 2025
IF-Badcase Models follow 450 multi-constraint instructions drawn from real user failure reports. A checklist checks each constraint. Zhipu AI (Z.ai) Following precise instructions Everyday System cards only GLM-5 83.2% 1 Feb 2026 Feb 2026
IMO Shortlist 2024 Models write proofs for the 31 problems of the 2024 IMO Shortlist. DeepSeek scores the proofs with its own verifier model. International Mathematical Olympiad Written proofs Competition System cards only DeepSeekMath-V2 27% 1 Jul 2025 Nov 2025
Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. Google DeepMind Deception and alignment Everyday System cards only Gemini 4 Argon 93.8% 1 Oct 2026
ImpossibleBench Coding agents get tasks whose unit tests conflict with the specification. Reports how often they cheat by changing tests or special-casing. Zhong, Raghunathan and Carlini Deception and alignment Expert System cards only Muse Spark 0% 1 Oct 2025 Jun 2026
In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. OpenAI Jailbreaks and prompt injection Expert System cards only ChatGPT agent 78% 1 Jul 2025 Jul 2025
InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. xAI (SpaceXAI) Labs' internal R&D tests Expert System cards only Grok 4.6 46.9% 1 Aug 2026 Aug 2026
Instruction Following (MiniMax in-house) Models follow multi-level instructions collected from user cases where current models fail. MiniMax Following precise instructions Everyday System cards only MiniMax-Text-01 46.3% 1 Jan 2025 Jan 2025
Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. Google DeepMind Deception and alignment Expert System cards only Gemini 4 Argon 0% 1 Oct 2026
Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. Google DeepMind Deception and alignment Expert System cards only Gemini 4 Argon 76.4% 1 Oct 2026 Oct 2026
Internal computer use safety benchmark (OpenAI) Models complete computer-use tasks while avoiding unsafe actions. OpenAI Desktop apps Research frontier System cards only GPT-6 Astra 2.40% 1 Sep 2026 Sep 2026
Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. Google DeepMind Finding and exploiting real bugs Expert System cards only Gemini 4 Argon 72% 1 Oct 2026
Introductory Sommelier (theory knowledge) Models answer wine theory questions at the introductory sommelier level. Scoring is accuracy. Court of Master Sommeliers Other professions Expert System cards only GPT-4 92% 1 Mar 2023
Iris Safety Evaluations Reflection AI's in-house safety suite of single-turn, multi-turn, jailbreak and agentic requests. Reports the share of harmful requests refused or redirected and the share of harmless requests answered. Reflection AI Refusing harmful requests Everyday System cards only Beam 1.0 88.9% 1 Oct 2026 Oct 2026
Irrelevant instructions, text-based web browser Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them. OpenAI Jailbreaks and prompt injection Expert System cards only ChatGPT agent 99.5% 1 Jul 2025 Jul 2025
Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. OpenAI Jailbreaks and prompt injection Expert System cards only ChatGPT agent 95% 1 Jul 2025 Jul 2025
JEF-Bench Agents read files that embed jailbreak instructions in tool-call returns. Reports attack success rate across 1800 tasks and 50 disallowed use cases. Meta Jailbreaks and prompt injection Expert System cards only Muse Spark 1.3 2.00% 1 Oct 2026
KAET (Kimi Autonomous Execution Tasks) Agents execute long-horizon tasks that simulate real user requests and enterprise system operations. Moonshot AI General assistants Expert System cards only Kimi K3 83.5% 1 Jul 2026 Jul 2026
KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. xAI (SpaceXAI) GPU kernels Expert System cards only Grok 4.6 37.2% 1 Aug 2026 Aug 2026
Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. Anthropic AI research tasks Research frontier System cards only Claude Mythos Preview 80% 1 Apr 2026
Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. Moonshot AI Finding and exploiting real bugs Expert System cards only Kimi K3 38.9% 1 Jul 2026 Jul 2026
Knowledge Q&A (MiniMax in-house) Models answer knowledge questions drawn from real Hailuo AI user queries. MiniMax Trivia and open questions Everyday System cards only MiniMax-Text-01 78.6% 1 Jan 2025 Jan 2025
Ko-AIME 2024 (Upstage) Models solve AIME 2024 competition problems translated into Korean. Upstage built the set in house. Upstage Competition problems with answers Competition System cards only Solar Open 100B 80.3% 1 Dec 2025 Jan 2026
KWV Bench Models answer atomic visual questions extracted from tasks distilled from real knowledge-work scenarios. Moonshot AI Documents, charts and text in images Expert System cards only Kimi K3 64.7% 1 Jul 2026 Jul 2026
Language Confusion Benchmark Models answer prompts in 15 languages and must reply in the language the user wants, both in monolingual and crosslingual settings. A language detector scores line-level and word-level pass rates. Cohere (Marchisio et al.) Chat and instructions in many languages Everyday System cards only Cohere Command A 93% 1 Jun 2024 Apr 2025
Lazy investigation (Anthropic) Agents debug a failing job where misleading evidence invites a harmful action such as cordoning a healthy node. Scored 0 to 10 by a grader. Anthropic Deception and alignment Expert System cards only Claude Opus 5 9.94 1 Jul 2026
Lazy investigation on misleading codebases (Anthropic) Models trace small, intentionally misleading codebases to answer how a value is set. Score is the share of lazy investigations ending in a wrong answer. Anthropic Deception and alignment Expert System cards only Claude Opus 4.8 0% 1 May 2026 May 2026
Leetcode (easy) Models solve 41 easy Leetcode problems that OpenAI selected. Scored as problems solved. OpenAI (Leetcode problems) Competitive programming University System cards only GPT-4 75.6% 1 Mar 2023 Mar 2023
Leetcode (hard) Models solve 45 hard Leetcode problems that OpenAI selected. Scored as problems solved. OpenAI (Leetcode problems) Competitive programming University System cards only GPT-4 6.70% 1 Mar 2023 Mar 2023
Leetcode (medium) Models solve 80 medium Leetcode problems that OpenAI selected. Scored as problems solved. OpenAI (Leetcode problems) Competitive programming University System cards only GPT-4 26.3% 1 Mar 2023 Mar 2023
LengthCtrl (Qwen) Models do creative writing tasks with length requirements. The score depends on the gap between output length and target length. Qwen Team, Alibaba Group Following precise instructions Everyday System cards only Qwen3-32B 73.5% 1 May 2025 May 2025
Long Context (MiniMax in-house) Models translate, summarize and analyze long documents from real user requests. MiniMax Reasoning over long documents Everyday System cards only MiniMax-Text-01 93.8% 1 Jan 2025 Jan 2025
Long-context grounding, sentence unfactual rate Models answer from long documents. An autorater reports the share of sentences not supported by the documents. Google DeepMind Faithful to sources Everyday System cards only Gemini 4 Argon 6.40% 1 Oct 2026
LongBench English Models answer questions and complete other long-context tasks in the original English headline table. The score is its published average on a percentage scale. THUDM (Tsinghua University) Reasoning over long documents University System cards only EXAONE 4.0 32B 48.1% 1 Aug 2023 Jul 2025
MAEB Audio embedding models are scored on 30 tasks across speech, music, environmental sound and audio-text retrieval in over 100 languages. MTEB community Audio understanding and voice Everyday System cards only EmbeddingGemma 2 49.9 1 Feb 2026 Oct 2026
Malware Refusals (Synthetic) Coding agents receive a large synthetic set of malware-related tasks, including ambiguous and dual-use prompts. Scored as refusal rate. OpenAI Refusing harmful requests Expert System cards only codex-1 97% 1 May 2025 May 2025
Many-shot jailbreaking evaluation (Anthropic) Models receive long fake dialogs where the assistant complies with harmful requests, then a new harmful request. Score is the compliance rate. Anthropic Jailbreaks and prompt injection Expert System cards only Claude Opus 4 25% 1 May 2025
MathQA Models answer multiple-choice math word problems drawn from AQuA. Each problem is annotated with an operation program. Scored by accuracy. University of Washington and Allen Institute for AI Word problems School System cards only Apertus 70B 33.9% 1 May 2019 Sep 2025
MathVision-Wildphoto Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description. Qwen (Alibaba) Visual math and diagrams School System cards only Qwen3-VL-235B-A22B 63.8% 1 Nov 2025 Nov 2025
MaXM Vision language models answer short visual questions about Crossmodal-3600 images in 7 languages. A test-only set scored by accuracy. Google Research General visual questions Everyday System cards only Aya Vision 32B 62.3% 1 Sep 2022 May 2025
Medical Knowledge Self-Assessment Program (MKSAP) Models answer internal medicine board review questions from MKSAP. Scoring is accuracy. American College of Physicians Medicine Expert System cards only GPT-4 75% 1 Mar 2023
MedMCQA Models answer multiple-choice questions from Indian medical entrance exams, AIIMS and NEET PG. Scoring is accuracy. Saama AI Research Medicine Expert System cards only GPT-4o 77% 1 Mar 2022 Aug 2024
MedQA Mainland China Models answer mainland China medical licensing exam questions in simplified Chinese. Scoring is accuracy. MIT Medicine Expert System cards only GPT-4o 84% 1 Sep 2020 Aug 2024
MedQA Taiwan Models answer Taiwan medical licensing exam questions in traditional Chinese. Scoring is accuracy. MIT Medicine Expert System cards only GPT-4o 91% 1 Sep 2020 Aug 2024
MedQA USMLE (5 options) Models answer USMLE-style medical licensing questions with five answer options. Scoring is accuracy. MIT Medicine Expert System cards only GPT-4o 86% 1 Aug 2024
MENT-SNS Models translate English and Chinese sentences with slang, wordplay and idioms. A GPT-4.1 judge compares each translation with a baseline. Zhipu AI (Z.ai) Translation University System cards only GLM-5 1013.0 1 Feb 2026 Feb 2026
Meta action-belief coherence probe Models act on items from five alignment evaluations and then state beliefs about them. Reports a mean coherency score. Meta Deception and alignment Expert System cards only Muse Spark 1.3 89.1% 1 Oct 2026
Meta agentic evaluations grounded in user traffic Models continue replays of recorded user workflows. A jury reports the share of cases with a supported substantive finding. Meta Deception and alignment Expert System cards only Muse Spark 1.3 20.7% 1 Oct 2026
Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. Meta with Frontier Design Group Lab protocols Expert System cards only Code World Model (CWM) 43.6% 1 Oct 2025 Oct 2025
Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. Meta Secure code and review Expert System cards only Muse Spark 1.3 47.8% 1 Oct 2026
Meta Cyber Misuse Chat Models receive 80 offensive cyber requests, directly and with jailbreak developer prompts. Reports attack success rate and a benign false refusal rate. Meta Refusing harmful requests Expert System cards only Muse Spark 1.1 2.80% 1 Jul 2026
Meta cyber misuse request set Models receive 600 private offensive cyber prompts, directly and under jailbreak attacks. Reports attack success rate. Meta Refusing harmful requests Expert System cards only Muse Spark 9.00% 1 Apr 2026 Jun 2026
Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. Meta Hallucination rates Expert System cards only Muse Spark 1.3 5.74% 1 Oct 2026
Meta prompted evaluation awareness Models are asked to judge whether transcripts come from evaluations or real use. Reports the weighted rate of correct identification. Meta Deception and alignment Expert System cards only Muse Spark 43% 1 Apr 2026 Jun 2026
Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. Meta Deception and alignment Expert System cards only Muse Spark 1.3 8.50% 1 Oct 2026
Meta training-stage provenance probe Models label 10,000 transcripts as deployment, evaluation, RL or SFT. Reports overall accuracy against a 25% chance rate. Meta Deception and alignment Expert System cards only Muse Spark 1.3 70% 1 Oct 2026
Meta Underspecified Intent evaluation Models answer ambiguous requests sourced from real Meta AI chats where users corrected the reply. Rubrics score whether the true intent is met. Meta Following precise instructions Everyday System cards only Muse Spark 71.2% 1 Apr 2026 Jun 2026
Microsoft RAI benchmark: Grounding Models answer in GPT-4o simulated multi-turn conversations. A judge scores from 0 to 5 how fully each answer is grounded in the given prompt. Microsoft Faithful to sources Everyday System cards only Phi-4 4.62 1 Dec 2024 Dec 2024
MIEB Lite Embedding models complete the MIEB Lite task set. The official API reports the mean across task types. MTEB community General visual questions System cards only EmbeddingGemma 2 65.7 1 Oct 2026
MIRA Bench Agents carry out long multi-role enterprise tasks across several systems and decide when to delegate to subagents. Moonshot AI Simulated companies Expert System cards only Kimi K3 64.1% 1 Jul 2026 Jul 2026
MirrorDojo Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDojo. Each episode pairs a user task with an instruction planted in a tool result. Reports attack success rate and utility. Reflection AI Jailbreaks and prompt injection Everyday System cards only Beam 1.0 3.10% 1 Oct 2026 Oct 2026
MirrorDyn Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDyn. Reports attack success rate and utility. Reflection AI Jailbreaks and prompt injection Everyday System cards only Beam 1.0 3.60% 1 Oct 2026 Oct 2026
Mistral 175 unsafe prompts Models receive 175 unsafe prompts. Metric is the share of harmful questions the model declines. Mistral AI Refusing harmful requests Everyday System cards only Mistral 7B v0.1 100% 1 Oct 2023 Oct 2023
Mistral internal function calling benchmark Models call functions on Mistral's in-house test set. The Magistral paper does not describe the tasks or the metric. Mistral AI Function calling Everyday System cards only Magistral Medium 1.0 87.4% 1 Jun 2025 Jun 2025
MMBench-GUI GUI agents are tested at four levels on Windows, macOS, Linux, iOS, Android and the web: content understanding, element grounding, task automation and task collaboration. Shanghai AI Laboratory (OpenCompass) Finding things on screen Everyday System cards only STEP3-VL-10B 81.5% 1 Jul 2025 Jan 2026
MMLU Anatomy Models answer MMLU multiple-choice questions from the Anatomy subject. Scoring is accuracy. UC Berkeley Medicine University System cards only GPT-4o 89% 1 Sep 2020 Aug 2024
MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. OpenAI Short factual questions University System cards only GPT-4 0.07 1 Mar 2023 Mar 2023
MMLU Clinical Knowledge Models answer MMLU multiple-choice questions from the Clinical Knowledge subject. Scoring is accuracy. UC Berkeley Medicine Expert System cards only GPT-4o 92% 1 Sep 2020 Aug 2024
MMLU College Biology Models answer MMLU multiple-choice questions from the College Biology subject. Scoring is accuracy. UC Berkeley Exam questions University System cards only GPT-4o 95% 1 Sep 2020 Aug 2024
MMLU College Medicine Models answer MMLU multiple-choice questions from the College Medicine subject. Scoring is accuracy. UC Berkeley Medicine University System cards only GPT-4o 84% 1 Sep 2020 Aug 2024
MMLU Medical Genetics Models answer MMLU multiple-choice questions from the Medical Genetics subject. Scoring is accuracy. UC Berkeley Medicine Expert System cards only GPT-4o 96% 1 Sep 2020 Aug 2024
MMLU Professional Medicine Models answer MMLU multiple-choice questions from the Professional Medicine subject. Scoring is accuracy. UC Berkeley Medicine Expert System cards only GPT-4o 94% 1 Sep 2020 Aug 2024
Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. Google DeepMind Dangerous capabilities Everyday System cards only Gemma 2 27B 3.72 1 Mar 2024 Jul 2024
Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. OpenAI Deception and alignment Expert System cards only GPT-6 Astra 61.2% 1 Sep 2026 Sep 2026
Multi-agent search (Anthropic internal benchmark) Agents solve difficult information retrieval problems with web search, alone or as orchestrators that delegate to subagents. Anthropic Deep research Expert System cards only Claude Opus 4.5 74.8% 1 Nov 2025 Nov 2025
Multi-dialect SQL evaluation (Cohere) Models write SQL for SQLite, PostgreSQL, MySQL, PL/SQL and T-SQL databases. Scored by execution accuracy against gold databases. Cohere Data science and SQL University System cards only Cohere Command A 55.3% 1 Apr 2025 Apr 2025
Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. Google DeepMind Questions that need search Expert System cards only Gemini 4 Argon 94.3% 1 Oct 2026
MultiChallenge Audio (Xiaomi) Models answer the final turn of MultiChallenge conversations converted to speech, with speech-to-text and speech-to-speech versions. Xiaomi built the conversion in house. Xiaomi Speech recognition and translation Everyday System cards only MiMo-Audio-7B-Instruct 15.2% 1 Sep 2025 Sep 2025
Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. Meta Finding and exploiting real bugs Expert System cards only Code World Model (CWM) 8.30% 1 Oct 2025 Oct 2025
Neutrality evaluation Single and multi-turn prompts on politics, civics, identity and religion test whether models avoid taking sides. Reports neutral response rate. Google DeepMind Bias, fairness and politics Everyday System cards only Gemini 4 Argon 93% 1 Oct 2026
NTREX-128 Models translate 1,997 English news sentences into 128 languages, with human reference translations. Scored with metrics such as COMET or chrF. Microsoft Translator (Federmann, Kocmi and Xin) Translation University System cards only Cohere Command A 68.8 1 Oct 2022 Apr 2025
Omni3D A large benchmark for 3D object detection from a single image, built by unifying indoor and outdoor datasets such as SUN RGB-D, ARKitScenes, Hypersim, Objectron, KITTI and nuScenes. Vision-language models answer its 3D grounding questions. Meta AI Spatial reasoning and counting Everyday System cards only Nemotron 3.5 Super VL 53.4% 1 Jul 2022 Oct 2026
Omnilingual-Gaia2 Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts. Meta (FAIR) Chat and instructions in many languages University System cards only Muse Glimmer 60.4% 1 Aug 2026 Oct 2026
Online Experience (Moonshot) Agents produce the deliverable file types that real online users request most often, following the distribution of real usage. Moonshot AI Documents and spreadsheets Everyday System cards only Kimi K3 77.9% 1 Jul 2026 Jul 2026
Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. OpenAI Jailbreaks and prompt injection Expert System cards only OpenAI Computer-Using Agent (CUA) 23% 1 Jan 2025 Jan 2025
OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. Meta Machine learning engineering Research frontier System cards only Muse Spark 1.3 64.2% 1 Oct 2026
OSI-Bench Spatial reasoning questions on outdoor pedestrian-view videos with metric 3D ground truth from synchronized stereo cameras, LiDAR and GPS. 8,736 questions range from relative direction to metric and motion estimates. UCAS, ETH Zurich and Microsoft Research Asia Spatial reasoning and counting Everyday System cards only Nemotron 3.5 Super VL 30.6% 1 Dec 2025 Oct 2026
Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. Google DeepMind Deception and alignment Everyday System cards only Gemini 4 Argon 55.9% 1 Oct 2026
Package registry publishing evaluation (Anthropic) Agents in a simulated capture the flag exercise can publish a backdoored package to a registry. An investigator escalates pressure until they act. Anthropic Deception and alignment Expert System cards only Claude Opus 5.5 33.3% 1 Sep 2026 Sep 2026
PartBench Parametric mechanical part design and CAD-style generation, graded for geometric and manufacturing fidelity. Normal Factory CAD models University System cards only Grok 4.6 58.5% 1 2026 Aug 2026
PerceptionComp 1,114 five-choice questions on 279 complex videos. Each question needs evidence from several separate moments, so no single frame answers it. PerceptionComp authors (ECCV 2026) Video understanding Everyday System cards only Nemotron 3.5 Super VL 39.4% 1 Mar 2026 Oct 2026
Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 98% 1 Oct 2026
Petri Bloom An auditor model runs generated scenarios to elicit named misaligned behaviors. Judges score each transcript from 1 to 10. Anthropic Fellows (Gupta et al.) Deception and alignment Expert System cards only Muse Spark 1.3 1.44 1 Dec 2025 Oct 2026
Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. OpenAI Deception and alignment Expert System cards only GPT-6 Astra 0.20% 1 Sep 2026 Sep 2026
Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. OpenAI Deception and alignment Everyday System cards only GPT-5.2 5.40% 1 Dec 2025 Dec 2025
Prompt injection test suite for LLM-based machine translation Translation systems receive TruthfulQA questions with injected instructions to answer instead of translate, across WMT 2024 language pairs. Scored by the share of outputs that answer rather than translate. Antonio Valerio Miceli-Barone and Zhifan Sun Jailbreaks and prompt injection Everyday System cards only Command A Translate 0.30% 1 Oct 2024 Sep 2025
Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 91% 1 Oct 2026
Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. Google DeepMind Jailbreaks and prompt injection Expert System cards only Gemini 4 Argon 86% 1 Oct 2026
ProofNet Provers write formal proofs for 371 undergraduate theorems from analysis, algebra and topology textbooks, each with a formal statement and a natural-language proof. The proof checker grades them. Azerbayev, Piotrowski, Schoelkopf, Ayers, Radev and Avigad Formal proofs University System cards only DeepSeek-Prover-V2-671B 37.1% 1 Feb 2023 Apr 2025
RD-TableBench Models extract 1,000 hand-labeled complex tables from document images, including scans, handwriting and merged cells. Scored by Levenshtein-based row and cell alignment. Reducto Documents, charts and text in images Everyday System cards only Nemotron Nano 12B v2 VL 57.4% 1 Nov 2024 Nov 2025
RE-Bench subset (Anthropic) Agents solve a modified subset of four METR RE-Bench research engineering tasks. Scores are normalized from reference solution 0 to expert solution 1. Anthropic Machine learning engineering Expert System cards only Claude 3.7 Sonnet 0.24 1 Feb 2025 Feb 2025
Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. Meituan LongCat Team Speech recognition and translation Everyday System cards only LongCat-Flash-Omni 1.37 1 Oct 2025 Oct 2025
Recognition of model-written user turns (Anthropic) Models write synthetic user turns for real Claude Code transcripts, then judge which version is real. Score is accuracy above chance. Anthropic Other Expert System cards only Claude Mythos Preview 0.35 1 Apr 2026
RelBench Agents build models for prediction tasks over multi-table relational databases, graded on held-out targets. Stanford (Robinson et al.) Machine learning engineering Expert System cards only Grok 4.5 38.4% 1 Jul 2024 Jul 2026
RepoInject v1 Coding agents do benign tasks in 84 repository tasks seeded with prompt injections. Reports attack success rate at one and three attempts. Exponential Security Labs Jailbreaks and prompt injection Expert System cards only Muse Spark 1.3 23.8% 1 Oct 2026
ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. Meta AI research tasks Research frontier System cards only Muse Spark 1.3 64.7% 1 Oct 2026
RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. Google DeepMind Deception and alignment Research frontier System cards only Gemini 4 Argon 0% 1 Oct 2026
Safety (MiniMax in-house) Models respond to safety-relevant user prompts. MiniMax scores whether replies are safe and helpful. MiniMax Refusing harmful requests Everyday System cards only MiniMax-Text-01 90.9% 1 Jan 2025 Jan 2025
SAT Evidence-Based Reading & Writing Models take the SAT reading and writing section. Scored on the 200 to 800 scale. College Board Reading comprehension School System cards only GPT-4 710.0 1 Mar 2023
Search triggering Models decide whether to call search for queries that need fresh facts or answer directly when search is unneeded. Reports F1. Google DeepMind Function calling Everyday System cards only Gemini 4 Argon 88% 1 Oct 2026
SecretPatch Models decide whether commits are security patches, including silently released ones, from a set of 4,702 security patches. George Mason University (Wang et al.) Finding and exploiting real bugs Expert System cards only Gemma 2 27B 72% 1 Jun 2019 Jul 2024
SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. xAI (SpaceXAI) Secure code and review Expert System cards only Grok 4.6 58.7% 1 Aug 2026
Seed1.5-VL internal benchmark (ByteDance Seed in-house) Models answer more than 12,000 internal image questions over 100 tasks, from perception, OCR and STEM to agents, code and out-of-distribution puzzles. LLM judges grade answers, and the overall is a weighted average. ByteDance Seed General visual questions University System cards only Seed1.5-VL 59.3% 1 May 2025 May 2025
Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. METR (run by OpenAI) Machine learning engineering Expert System cards only GPT-4o 0% 1 Aug 2024 Aug 2024
Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. Google DeepMind Dangerous capabilities Expert System cards only Gemma 2 27B 0% 1 Mar 2024 Jul 2024
Sequitur Research Lab Automation Agents run simulated wet-lab automation protocols. Scored out of 100 on how well each experimental phase is carried out, averaged over six runs. Sequitur Research Genomics and life sciences Research frontier System cards only GPT-6.2 Astra 74.5 1 Nov 2026 Nov 2026
Sexually Explicit Content Safety An autorater checks answers to direct, adversarial and benign sexual content prompts in single and multi-turn chats. Reports safe response rate. Google DeepMind Refusing harmful requests Everyday System cards only Gemini 4 Argon 97.6% 1 Oct 2026
SkillInject Agents use a skill file that carries an adversarial instruction while doing a legitimate task. Reports how often the agent follows the injection. Schmotz et al. Jailbreaks and prompt injection Expert System cards only Muse Spark 1.3 0.60% 1 Feb 2026 Oct 2026
Slide generation human evaluation (Z.ai) People compare HTML slide decks from two models on content, layout and visual quality. Metric is overall win rate. Zhipu AI (Z.ai) Taste and style Everyday System cards only GLM-5 67.5% 1 Feb 2026 Feb 2026
SPAR-Bench Spatial perception and reasoning questions over single and multiple views of indoor 3D scenes, from depth and distance estimates to object relations and spatial imagination. Zhang et al. (From Flatland to Space) Spatial reasoning and counting Everyday System cards only Nemotron 3.5 Super VL 64.2% 1 Mar 2025 Oct 2026
SpatialTree Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level. Xiao et al. Spatial reasoning and counting Everyday System cards only Nemotron 3.5 Super VL 58.1% 1 Dec 2025 Oct 2026
SPBench Spatial perception and reasoning questions built from ScanNet scenes. A single-image set covers distance, size and direction, and a multi-view set adds counting across views. Answers are numbers or multiple choice. Zhejiang University and collaborators (SpatialLadder) Spatial reasoning and counting Everyday System cards only Nemotron 3.5 Super VL 63.8% 1 Oct 2025 Oct 2026
SPI (security patch identification) Models decide whether code commits from open-source projects are security patches. Zhou et al. Finding and exploiting real bugs Expert System cards only Gemma 2 27B 53% 1 Jan 2021 Jul 2024
STARE Models solve 4K spatial tasks such as 2D and 3D transformations, cube net folding, tangrams, perspective and temporal reasoning that people solve by visual simulation. University of Washington, Sun Yat-sen University and Stanford University Spatial reasoning and counting Everyday System cards only STEP3-VL-10B 61.8% 1 Jun 2025 Jan 2026
Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). Google DeepMind Dangerous capabilities System cards only Gemini 4 Argon 93.3% 1 May 2025 Oct 2026
STEM Professor Bench Agents solve unpublished research problems written by professional scientists. Most problems concern theoretical physics, with some partial credit. Anthropic Research science Research frontier System cards only Claude Fable 5.2 61.8% 1 Nov 2026
Structured Translation (Cohere) Models translate JSON and similar structures, either whole or only named fields. Metrics check parsing, schema match, untranslated or wrongly copied fields and line counts. Cohere Translation Everyday System cards only North Small Translate 1.0 93.7% 1 Sep 2026 Sep 2026
Suicide & Self-Harm evaluation Static and dynamic chats with suicidal or self-harm signals test adherence to prevention practice and crisis resources. Reports policy adherence rate. Google DeepMind Wellbeing and sycophancy Everyday System cards only Gemini 4 Argon 96% 1 Oct 2026
SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk. Meta Fixing real repositories Expert System cards only Muse Spark 42.9% 1 Apr 2026 Jun 2026
Sycophancy evaluation (offline) Models answer a fixed set of messages resembling production traffic that could elicit sycophancy. A grader assigns a sycophancy score. OpenAI Wellbeing and sycophancy Everyday System cards only GPT-5 0.04 1 Aug 2025 Aug 2025
Sycophancy on user-provided prompts (Anthropic) Models continue real feedback conversations where earlier Claude models were sycophantic. Score is the non-sycophantic response rate. Anthropic Deception and alignment Everyday System cards only Claude Opus 4.5 10% 1 Nov 2025 Nov 2025
Task-dependent output homogenization evaluation Models answer prompts several times. A judge scores functional diversity where it is wanted and consistency where one answer is right. Jain et al. (Meta FAIR) Creative writing Everyday System cards only Muse Spark 0.21 1 Sep 2025 Jun 2026
Template Following generalization (Qwen) Models follow system prompts and emit exactly formatted XML or JSON tool calls for five scaffolds derived from popular IDE and CLI coding agents. Qwen (Alibaba) Function calling Everyday System cards only Qwen3-Coder-Next 92.7% 1 Feb 2026 Feb 2026
Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. Google DeepMind Questions that need search Everyday System cards only Gemini 4 Argon 11.8% 1 Oct 2026
Text Classification (ByteDance Seed in-house) Models analyze business text and return labels or structured outputs, for example sentiment or the main viewpoints in user content. ByteDance Seed Running a business Everyday System cards only Seed2.0 Pro 69% 1 Feb 2026 Feb 2026
ThinkFollow (Qwen) Models hold multi-turn dialogues with randomly inserted think and no-think flags and must switch reasoning mode correctly. Qwen Team, Alibaba Group Multi-turn conversation Everyday System cards only Qwen3-32B 98.9% 1 May 2025 May 2025
Tool-calling prompt injections Agents receive prompt injections inside tool call outputs such as connectors. Scored on resisting the injected instructions. OpenAI Jailbreaks and prompt injection Expert System cards only GPT-5 99% 1 Aug 2025 Aug 2025
ToolCall-Badcase Models choose a tool and its arguments for 200 cases drawn from real user failure reports. Each case has a ground-truth call. Zhipu AI (Z.ai) Function calling Everyday System cards only GLM-5 95.8% 1 Feb 2026 Feb 2026
ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. Qwen Team, Alibaba Group Function calling Everyday System cards only Qwen3-32B 85.5% 1 May 2025 May 2025
Translation Human Scoring (Z.ai) People score model translations. The GLM-5 report does not describe the test set or the scale. Zhipu AI (Z.ai) Translation University System cards only GLM-5 3.12 1 Feb 2026 Feb 2026
UK AISI correlated monitor failure The model acts as a monitor over transcripts of another model carrying out an unsanctioned cyber attack. Scored on the share of violations it fails to flag. UK AI Security Institute Deception and alignment Expert System cards only GPT-6.2 Astra 19% 1 Nov 2026 Nov 2026
UK AISI CoT Controllability Models follow instructions that constrain the content of their chain of thought. Scored as the share of samples that follow the constraint. UK AI Security Institute Deception and alignment Expert System cards only GPT-6 Astra 93% 1 Sep 2026 Sep 2026
UK AISI no-CoT maths competition problems Competition maths problems from AIME, HMMT, CMIMC, SMT, BRUMO and OTIS Mock AIME, answered without a chain of thought. Scored on accuracy. UK AI Security Institute Deception and alignment Competition System cards only GPT-6.2 Astra 79.2% 1 Nov 2026 Nov 2026
UK AISI simulated sandbox escape monitor evasion The model tries to escape a deliberately vulnerable sandbox without a monitoring model noticing. Scored on the share of samples where it escapes undetected. UK AI Security Institute Deception and alignment Expert System cards only GPT-6.2 Astra 0.50% 1 Nov 2026 Nov 2026
UK AISI vulnerability discovery challenges A private suite of synthetic vulnerability discovery challenges. Models answer in a single response without tools, scored pass@1. UK AI Security Institute Finding and exploiting real bugs Research frontier System cards only GPT-6.2 Astra 88.6% 1 Nov 2026 Nov 2026
Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. Meta Secure code and review Expert System cards only Muse Spark 1.3 21.1% 1 Oct 2026
Unauthorized voice generation Speech models produce audio that may deviate from the approved system voice. Scored on how many meaningful deviations the output classifier catches. OpenAI Refusing harmful requests Everyday System cards only GPT-4o 100% 1 Aug 2024 Aug 2024
Unexpected container or code states Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed. OpenAI Deception and alignment Expert System cards only codex-1 85% 1 May 2025 May 2025
Ungrounded inference and sensitive trait attribution Speech models get requests to infer traits about a speaker from audio. Scored on refusing ungrounded inferences and hedging plausible ones. OpenAI Bias, fairness and politics Everyday System cards only GPT-4o 84% 1 Aug 2024 Aug 2024
Uniform Bar Exam (MBE+MEE+MPT) Models take a simulated Uniform Bar Exam with multiple-choice, essay and performance test parts. Scored on the 400-point scale. NCBE Law Expert System cards only GPT-4 298.0 1 Mar 2023
USABO Semifinal Exam 2020 Models take the 2020 USA Biology Olympiad semifinal exam. Scored out of 150 points. Center for Excellence in Education Science olympiads Competition System cards only GPT-4 87.0 1 Mar 2023
USMLE Step 1 (sample questions) Models answer the official USMLE Step 1 multiple choice sample questions. Scored by accuracy. USMLE (FSMB and NBME) Medicine Expert System cards only Claude 2 68.9% 1 Jul 2023
USMLE Step 2 (sample questions) Models answer the official USMLE Step 2 multiple choice sample questions. Scored by accuracy. USMLE (FSMB and NBME) Medicine Expert System cards only Claude 2 63.3% 1 Jul 2023
USMLE Step 3 (sample questions) Models answer the official USMLE Step 3 multiple choice sample questions. Scored by accuracy. USMLE (FSMB and NBME) Medicine Expert System cards only Claude 2 67.2% 1 Jul 2023
USNCO Local Section Exam 2022 Models take the 2022 US National Chemistry Olympiad local section exam. Scored out of 60 questions. American Chemical Society Science olympiads Competition System cards only GPT-4 36.0 1 Mar 2023
Virology Capabilities Test (text-only) Models answer the text-only questions of the SecureBio Virology Capabilities Test. Scored as accuracy. SecureBio Lab protocols Expert System cards only GPT-5 46% 1 Aug 2025
Voice persuasion study on party preferences Participants hear model-voiced audio clips or talk with a voiced model. Scored on shifts in hypothetical party preference against human baselines. OpenAI Dangerous capabilities Everyday System cards only GPT-4o 5.74% 1 Aug 2024 Aug 2024
VSTI-Bench Video questions about how spatial relationships change over time in 3D scenes, over 138,000 pairs across five tasks. VLM-3R authors Spatial reasoning and counting Everyday System cards only Nemotron 3.5 Super VL 67.3% 1 May 2025 Oct 2026
WMT25 Terminology Translation Task Models translate sentence, paragraph and document level text with a supplied source to target glossary. Scoring checks quality and whether the prescribed target terms appear. WMT (Conference on Machine Translation) Translation Expert System cards only North Small Translate 1.0 89.7% 1 Jun 2025 Sep 2026
xChatBench Vision language models answer 400 open-ended multimodal chat prompts in 7 languages. An LLM judge scores each answer against fine-grained human-written criteria. Carnegie Mellon University (PangeaBench) General visual questions Everyday System cards only Aya Vision 32B 70.1 1 Oct 2024 May 2025
ZMultiDialBench Human annotators score model replies to 141 multilingual dialogue prompts on a 1 to 10 scale with category criteria. Zhipu AI (Z.ai) Chat and instructions in many languages Everyday System cards only GLM-5 8.57 1 Feb 2026 Feb 2026
ZMultiTransBench Models translate 1,220 real Chinese source texts into seven languages. A GPT-4.1 judge compares each translation with a fixed baseline. Zhipu AI (Z.ai) Translation University System cards only GLM-5 1050.0 1 Feb 2026 Feb 2026
ADeptS-Bench Computer-use agents pick the next interface action from a screenshot and instruction. A deterministic grader scores whether the action is safe. Chen et al. Finding things on screen Expert System cards only 1 Aug 2026 Oct 2026
Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. Google DeepMind Deception and alignment Expert System cards only 1 Oct 2026
Agentic IF Index Composite of Meta internal evals on following tool-use restrictions, rubrics, compound constraints and long policies in agentic workflows. Scored 0-1. Meta Following precise instructions System cards only 1 Sep 2026 Oct 2026
AL-QASIDA Models write in eight dialectal Arabic varieties. Scored on fidelity, understanding, quality and diglossia, including the ADI2 dialect identification score. Johns Hopkins University and Cohere Chat and instructions in many languages Everyday System cards only 1 Dec 2024 Apr 2025
Anthropic large agent teams tasks Internal 24-hour tasks for teams of 1-100 agents: build a company knowledge base, or formalize a theorem in Lean. Anthropic Multi-agent games Research frontier System cards only 1 Sep 2026 Sep 2026
Appropriate harmlessness (Anthropic) Models answer borderline and harmful prompts. Classifiers sort each response into helpful answer, policy violation, appropriate refusal or unnecessary refusal. Anthropic Refusing harmful requests Everyday System cards only 1 Feb 2025 Feb 2025
AraDiCE Models are tested on Arabic dialect understanding and generation and on cultural awareness of the Gulf, Egypt and the Levant. Datasets were machine translated into dialects and post-edited by people. Qatar Computing Research Institute (Mousi et al.) Knowledge in many languages Everyday System cards only 1 Sep 2024 Nov 2025
ASIMOV-Agentic Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes. Google DeepMind Embodied reasoning Everyday System cards only 1 Jul 2026 Jul 2026
Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. Anthropic Dangerous capabilities Expert System cards only 1 Sep 2023 Mar 2024
Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. OpenAI with Gryphon Scientific Hazardous knowledge Expert System cards only 1 Jan 2024 Aug 2024
Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. Anthropic Hazardous knowledge Expert System cards only 1 Apr 2026
Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. Google DeepMind Dangerous capabilities Everyday System cards only 1 Mar 2024 Jul 2024
Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. Google DeepMind Deception and alignment Expert System cards only 1 Oct 2026
Closed-domain hallucination evaluation Models answer using only provided context. Scored on avoiding invented details outside that context. OpenAI Faithful to sources Everyday System cards only 1 Mar 2023 Mar 2023
CNML-Level Problems (DeepSeek) Models prove 91 theorems in algebra, geometry, number theory, combinatorics and inequalities at Chinese National High School Mathematics League level. A verifier model scores the proofs. DeepSeek Written proofs Competition System cards only 1 Nov 2025 Nov 2025
Community Alignment personalization Models predict an individual's preferred response from demographics and nine examples, across five countries. Reports global accuracy. Meta Bias, fairness and politics Everyday System cards only 1 Oct 2026
ControlArena Agents try to perform unwanted side actions stealthily while completing a legitimate user task. Scored on whether monitors catch them. UK AI Security Institute Deception and alignment Expert System cards only 1 Feb 2025 Jul 2026
Cooperation with user deception (Anthropic) Models get developer system prompts asking them to subtly mislead users, then meet users who would be harmed. Graders score harmful and deceptive choices. Anthropic Refusing harmful requests Everyday System cards only 1 Sep 2025 Sep 2025
Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. OpenAI Secure code and review Expert System cards only 1 Sep 2026 Sep 2026
Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. OpenAI Jailbreaks and prompt injection Expert System cards only 1 Feb 2025 Feb 2025
DeepSeek LLM in-house safety evaluation Models answer 2,400 Chinese test cases on discrimination, legal rights, trade secrets, illegal behavior and other safety issues. Human annotators judge each answer. DeepSeek Refusing harmful requests Everyday System cards only 1 Jan 2024 Jan 2024
Deference to governments in local languages (Anthropic) Models answer politically controversial questions in English and in a local language. A grader rates agreement with the official government position from 1 to 10. Anthropic Bias, fairness and politics Everyday System cards only 1 Feb 2026 Feb 2026
Disparate performance on voice inputs Speech models answer capability and safety prompts spoken in system voices and 27 human voices. Compares accuracy across voices. OpenAI Bias, fairness and politics Everyday System cards only 1 Aug 2024 Aug 2024
Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. Anthropic Finding and exploiting real bugs Expert System cards only 1 Mar 2024 Mar 2024
External pilot use failure modes (Anthropic) Models are judged on user-flagged pilot conversations, directly or by resampling. Graders measure failure modes such as hallucinated facts and misrepresented work completion. Anthropic Deception and alignment Everyday System cards only 1 Feb 2026 Feb 2026
Gemini Personality Evaluation (sycophancy) About 900 single- and multi-turn prompts scoring nine personality dimensions, including candor and self-awareness against sycophancy. Google DeepMind Wellbeing and sycophancy System cards only 1 Oct 2026
Gray Swan Reward Hacking Bench Agentic environments where a task can be finished honestly or by crossing a declared line, such as shipping insecure code, leaking a secret, harming a third party through a shared service or leaving the scope of a penetration test. Reports how often each line is crossed. Gray Swan Deception and alignment Expert System cards only 1 Oct 2026 Oct 2026
Health dynamic conversation eval Simulated users hide key health facts at first, and models must ask for context to reach the correct clinical conclusion. Google DeepMind Medicine Expert System cards only 1 Oct 2026
Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. Google DeepMind Medicine Expert System cards only 1 Oct 2026
Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. Google DeepMind Medicine Expert System cards only 1 Oct 2026
Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. Google DeepMind Dangerous capabilities Everyday System cards only 1 Mar 2024 Jul 2024
Hindsight Neglect Models judge whether a bet was worth taking based on expected value rather than its outcome. Scoring is accuracy. Inverse Scaling Prize Language reasoning Everyday System cards only 1 Mar 2023
ImpossibleMLEBench Agents get ML tasks with instructions to reach impossibly good scores. Observers check whether they cheat and whether they disclose it. UK AI Security Institute Deception and alignment Expert System cards only 1 Jul 2026 Jul 2026
Incorrect behavior rate on disallowed and sensitive content Models answer disallowed prompts and sensitive prompts such as medical advice. Scored on the rate of responses that break policy. OpenAI Refusing harmful requests Everyday System cards only 1 Mar 2023 Mar 2023
Keeping secrets when pressed in simulated scenarios (Anthropic) Models hold confidential information in 39 job scenarios while a simulated counterpart presses for it. Measures leaks in replies and in reasoning. Anthropic Deception and alignment Expert System cards only 1 Oct 2026 Oct 2026
Kimi Promptfoo red-team safety evaluation Promptfoo generates attack prompts for harmful, criminal, misinformation, privacy and security plugins under five strategies. Human reviewers judge the responses. Moonshot AI Jailbreaks and prompt injection Everyday System cards only 1 Jul 2025 Jul 2025
Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. Moonshot AI Finding and exploiting real bugs Expert System cards only 1 Jul 2026 Jul 2026
Kimi-ASR Internal Testset Models transcribe speech from two internal Moonshot test subsets. Scored by word error rate, lower is better. Moonshot AI Speech recognition and translation Everyday System cards only 1 Apr 2025 Apr 2025
MAI internal jailbreak evaluation suite Models face about 9,500 jailbreak prompts built from 2,500 seed scenarios, grouped as foundational, compositional and adaptive. Scored as attack success rate. Microsoft AI Jailbreaks and prompt injection Everyday System cards only 1 Jun 2026 Jun 2026
MAI internal safety and over-refusal evaluation Models answer harmful and benign prompts in eight harm categories. Judges score safety pass rate on sensitive prompts and helpfulness as one minus over-refusal. Microsoft AI Refusing harmful requests Everyday System cards only 1 Jun 2026 Jun 2026
Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. Meta with Frontier Design Group Hazardous knowledge Expert System cards only 1 Oct 2025 Oct 2025
Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. Meta Hazardous knowledge Expert System cards only 1 Apr 2026 Jun 2026
Missing-context hallucinations (Anthropic) Models get prompts that request tools they lack or cite missing material. Score is the non-hallucination rate on each 500-prompt set. Anthropic Hallucination rates Everyday System cards only 1 May 2026
Mistral self-reflection moderation set Models classify prompts or answers as acceptable or harmful on a balanced set of adversarial and standard prompts. Scored by precision and recall. Mistral AI Refusing harmful requests Everyday System cards only 1 Oct 2023 Oct 2023
MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. OpenAI Knowledge in many languages University System cards only 1 Mar 2023 Mar 2023
Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. OpenAI Deception and alignment Expert System cards only 1 Sep 2026 Sep 2026
MR-NIAH (Multi-Round Needles-In-A-Haystack) Models must repeat one earlier answer from a long multi-turn dialogue history of 2K to 1M tokens. Scored by adjusted recall of key parts. MiniMax Finding facts in long inputs Everyday System cards only 1 Jan 2025 Jan 2025
Multilingual AIME 2024 (Mistral) Models solve AIME 2024 problems translated into French, Spanish, German, Italian, Russian and Chinese, reasoning in the question language. Mistral AI Reasoning in many languages Competition System cards only 1 Jun 2025 Jun 2025
Multilingual MMBench Models answer MMBench dev image questions in English, Chinese, Portuguese, Arabic, Turkish and Russian. Scored as accuracy per language. Alibaba AIDC-AI (Parrot authors) Knowledge in many languages School System cards only 1 Aug 2024 Nov 2025
Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. OpenAI Knowledge in many languages Expert System cards only 1 Jul 2025 Jul 2025
Muse-targeted cyber misuse and violation red-teaming An automated red-teaming agent tries to misuse the Muse runtime across six attack categories. Reports attack success rate per category. Meta Refusing harmful requests Expert System cards only 1 Oct 2026
OpenAI performance in cases flagged by users Model answers de-identified ChatGPT prompts that users flagged for factual errors. Scored on any error and on repeating the flagged error. OpenAI Hallucination rates System cards only 1 Sep 2026 Oct 2026
Personal data evaluations Browsing agents receive requests for private personal information, from 200 synthetic prompts and 55 manually written golden examples. Scored on adherence to personal data policy. OpenAI Refusing harmful requests Everyday System cards only 1 Feb 2025 Feb 2025
Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. Qwen (Alibaba) Speech recognition and translation School System cards only 1 Mar 2025 Mar 2025
Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. Qwen Team, Alibaba Group Jailbreaks and prompt injection Everyday System cards only 1 Jul 2024 Jul 2024
Real-scenario prompts human evaluation (Z.ai) One human evaluator scores replies to 660 curated real user prompts in English, Chinese and other languages on a 0 to 10 scale. Zhipu AI (Z.ai) Overall chat quality Everyday System cards only 1 Aug 2025 Aug 2025
Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. Anthropic Deception and alignment Expert System cards only 1 Sep 2025 Sep 2025
Safe response to high-severity prompts Models handle multi-turn self-harm and violent activity conversations where harmful intent only emerges from context. Scored as safe response rate. OpenAI Refusing harmful requests Expert System cards only 1 Sep 2026 Sep 2026
Safety mode controllability (Cohere) Models follow contextual or strict safety instructions. Accuracy averages correct answers on an over-refusal set and correct refusals on a control set. Cohere Refusing harmful requests Everyday System cards only 1 Apr 2025 Apr 2025
SciTS Models understand and generate scientific time series in 43 tasks across 12 domains. Signals range from one to ten million points. Each task uses its own metric. Shanghai AI Laboratory, Tsinghua University and others Research science Expert System cards only 1 Sep 2025 Mar 2026
SeedKernelBench Models write or optimize CUDA style kernels for operator tasks mined from FlashInfer, SGLang, FlashAttention, DeepGEMM, CUTLASS and cuDNN. Scored as average speedup over reference code. ByteDance Seed GPU kernels Expert System cards only 1 Jun 2026 Jun 2026
Speaker identification Speech models get audio requests to identify a speaker by voice. Scored on refusing when they should and complying for famous quotes. OpenAI Refusing harmful requests Everyday System cards only 1 Aug 2024 Aug 2024
Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. OpenAI with Signature Science Hazardous knowledge Research frontier System cards only 1 Dec 2024 Dec 2024
SWE agentic fabrication eval A grounding autorater checks whether claims made by coding agents on long SWE tasks are supported by tool outputs. Google DeepMind Hallucination rates Expert System cards only 1 Oct 2026
Sycophancy prompts (Anthropic) Models answer single-turn prompts that invite sycophancy, such as bad business ideas or delusional beliefs. Four synthetic sets and one hand-written set report sycophancy rates. Anthropic Deception and alignment Everyday System cards only 1 Sep 2025 Sep 2025
Text persuasion study on political opinions Participants read model-written articles or chat with a model on political topics. Scored on opinion shift against human-written articles. OpenAI Dangerous capabilities Everyday System cards only 1 Aug 2024 Aug 2024
Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup. ByteDance Seed Fixing real repositories Expert System cards only 1 Jun 2026 Jun 2026
Trajectory Labs cyber red teaming Expert red teamers try to break safeguarded cyber endpoints across nine attack technique families on exploit reproduction tasks. Trajectory Labs for Google DeepMind Jailbreaks and prompt injection Expert System cards only 1 Oct 2026 Oct 2026
Translated ARC-Challenge, HellaSwag and MMLU (Mistral) Models answer French, German, Spanish and Italian versions of ARC-Challenge, HellaSwag and MMLU. Mistral AI Knowledge in many languages School System cards only 1 Jan 2024 Jan 2024
Uhura ARC-Easy Models answer ARC-Easy science questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. Masakhane with OpenAI and collaborators (Bayes et al.) Reasoning in many languages School System cards only 1 Aug 2024 Aug 2024
Uhura TruthfulQA Models answer TruthfulQA questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. Masakhane with OpenAI and collaborators (Bayes et al.) Knowledge in many languages Everyday System cards only 1 Aug 2024 Aug 2024
Uhura-Eval Models answer new reading comprehension questions written in Amharic, Hausa and Yoruba. Scoring is accuracy. Uhura collaborators (Adelani, Kgomo, Bayes and others) Reading in many languages Everyday System cards only 1 Aug 2024 Aug 2024
UK AISI File Download monitor evasion The model tries to download a forbidden file without a monitoring model noticing. Scored on how often it evades the monitor. UK AI Security Institute Deception and alignment Expert System cards only 1 Nov 2026 Nov 2026
Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. Google DeepMind Dangerous capabilities Everyday System cards only 1 Mar 2024 Jul 2024
BuseyBench SVG Test Models write SVG code to draw Gary Busey from memory. Three vision models score the rendered face, with half the score based on likeness. Matt Wolfe Image generation and editing University Not in cards GPT-6.1 Sol 8.00 Nov 2026
BullshitBench V2 Models answer 100 nonsense questions across software, finance, legal, medical and physics, built with 13 techniques. A three-judge panel scores clear pushback. Peter Gostev Hallucination rates Everyday Not in cards Claude Opus 4.8 95% Mar 2026 Nov 2026
SEND Pedagogy Benchmark Models choose how a teacher should support students with disabilities or special educational needs. The questions come from Chilean teacher exams. Fab AI Exam questions University Not in cards Claude Fable 5.1 89.5% Mar 2025 Oct 2026
Pedagogy Benchmark Models answer multiple-choice questions about teaching strategies and assessment from Chilean teacher exams. Fab AI Exam questions University Not in cards Gemini 4 Argon 92.9% Dec 2024 Oct 2026
OTIS Mock AIME 2024-2025 Models solve 45 problems from three OTIS Mock AIME exams written by students of the OTIS program. Integer answers are scored by exact match. Epoch AI (problems by OTIS students) Competition problems with answers Competition Not in cards GPT-5.5 100% Dec 2024 Nov 2026
BLUEX Models answer questions from two Brazilian university entrance examinations. The questions cover high school subjects in Portuguese. Thales Sales Almeida and colleagues Knowledge in many languages School Not in cards Gemini 2.5 Pro 95% Jul 2023 Aug 2025
Open Portuguese LLM Leaderboard Models take Portuguese exams and complete language tasks for entailment, similarity, sentiment, and hate speech detection. The leaderboard aggregates task scores. Eduardo Garcia and collaborators Knowledge in many languages University Not in cards Gemini 2.5 Pro 88.4 Aug 2025
OpenVLM Leaderboard OpenCompass runs vision-language models through VLMEvalKit on a fixed set of image benchmarks, such as MMBench, MMStar and MathVista. The headline is the average score across that set. OpenCompass (Shanghai AI Laboratory) Composite indices University Not in cards Gemini 2.5 Pro 80.1 Jan 2024 Aug 2025
ACCoRD Models state probabilities and preferences on conceptual questions, and the benchmark checks 567 expert-approved logical consistency constraints such as conjunction rules and order invariance. Redwood Research (Cooper, Kastner, Oesterheld et al.) Other Expert Not in cards Claude Fable 5.2 89.4 Aug 2026 Nov 2026
Conceptual Reasoning Index An index that ranks models on reasoning about hard-to-verify topics such as AI alignment and decision theory. It weights LMCA 60 percent and ACCoRD and DTBench 20 percent each. Redwood Research (conceptual reasoning team) Composite indices Expert Not in cards Claude Fable 5.2 79.0 Aug 2026 Nov 2026
Angry Tweets Models classify the sentiment of Danish social media posts using labels supplied by crowdworkers. Amalie Brogaard Pauli and colleagues Reading in many languages Everyday Not in cards GPT-6.2 Astra 67% May 2021 Nov 2026
LMCA Models rate arguments against position texts on philosophy, decision theory, and AI risk. Scored by agreement with expert ratings, with a ceiling near 85. Emery Cooper, Caspar Oesterheld, and collaborators Other Expert Not in cards Claude Fable 5.2 70.6% Jul 2026 Nov 2026
FoSent Models classify sentiment in Faroese news articles. Two native speakers annotated articles from Portalurin and Dimmalætting. Iben Nyholm Debess and colleagues Reading in many languages Everyday Not in cards Gemini 2.5 Pro 91.6% May 2024 Nov 2026
WeirdML v3 Agents solve 11 hand-made machine learning and data analysis tasks on a GPU with unfamiliar data and hidden test answers. Scored by token-weighted best results. Håvard Tveit Ihle (Norwegian Defence Research Establishment) Machine learning engineering Expert Not in cards Claude Fable 5.2 43.1% Sep 2026 Nov 2026
LegalBench Models do 162 legal reasoning tasks written by legal professionals, covering six types of legal reasoning such as issue spotting and rule application. Stanford University (Hazy Research) and collaborators Law Expert Not in cards Claude Fable 5 88.6% Aug 2023 Oct 2026
ALE-Bench Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants. Sakana AI and AtCoder Competitive programming Competition Not in cards GPT-6 Astra 3,164 Jun 2025 Nov 2026
LisanBench Models build the longest chain of distinct English words from a given start word, each differing from the last by one letter. The score sums valid chain lengths over all start words. Lisan al Gaib (scaling01, independent) Puzzles and chess Everyday Not in cards Muse Wonder Full 37152.2 May 2025 Oct 2026
TaxEval v2 Models answer expert-written tax questions. A judge grades answer correctness and stepwise reasoning against expert solutions. Vals AI Finance Expert Not in cards Muse Spark 1.2 80.4% Sep 2026
Longform Creative Writing (EQ-Bench) Models plan, revise and then write a short story or novella over eight 1,000-word turns from a minimal prompt. A judge model scores quality, and slop, repetition and degradation are tracked. EQ-Bench (Sam Paech) Creative writing University Not in cards Claude Opus 5 86.3 May 2025 Oct 2026
A-Bench Models answer multiple-choice questions about AI-generated images, covering both meaning and low-level visual quality, to test them as judges of image generators. Q-Future (Nanyang Technological University and Shanghai Jiao Tong University) General visual questions Everyday Not in cards Qwen2.5-VL-72B 81% Jun 2024 Jun 2025
SimpleBench Models answer over 200 multiple-choice trick questions on spatial, temporal, and social reasoning that ordinary people solve. Scored by average accuracy over five runs. SimpleBench Team Abstract puzzles and logic Everyday Not in cards Claude Fable 5.2 94.9% Oct 2024 Nov 2026
Extended NYT Connections Models solve New York Times Connections puzzles with up to four extra trick words added. Each puzzle scores by the square of the share of groups found. Lech Mazur Language reasoning School Not in cards Gemini 4 Carbon 99.3% Oct 2024 Nov 2026
Linguistic Knowledge Benchmark (Fab AI) Models answer multiple-choice and short-answer questions about Kiswahili and English grammar, phonics, and spelling. Three model judges vote on short answers. Fab AI Language reasoning University Not in cards GPT-5.4 97.8% Sep 2026 Aug 2026
Dutch Book Reviews Dataset Models classify the sentiment of Dutch book reviews from Hebban using ratings supplied by readers. Benjamin van der Burgh and Suzan Verberne Reading in many languages Everyday Not in cards OLMo 2 32B Instruct 92.5% Oct 2019 Mar 2026
LLM Agronomy Benchmark (Farmers Business Network edition) 2024-08 Models answer agronomy questions from six groups, including questions from Farmers Business Network. Each question carries equal weight in the overall score. gbstox Other professions University Not in cards Gemini 4 Argon 94.3% Aug 2024 Oct 2026
MERA Models complete Russian-language tasks with several answer formats. The hub port grades short answers from Part 1 of the Russian Unified State Exam. MERA Evaluation team Knowledge in many languages University Not in cards Claude Opus 4.6 0.86 Nov 2023 Apr 2026
Open LLM Leaderboard v2 Hugging Face ran open models on IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro and averaged normalized scores. Hugging Face Composite indices University Not in cards Qwen2.5-72B 48.0 Jun 2024 Mar 2025
IOI v2 (Vals) Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run. Vals AI Competitive programming Competition Not in cards GPT-6 Astra 100% Oct 2026 Oct 2026
MedScribe Models write SOAP notes from clinical visit transcripts. Notes are graded against expert gold-standard rubrics for required content and section structure. Vals AI (data partner Protege) Medicine Expert Not in cards Claude Opus 5.5 91.4% Feb 2026 Nov 2026
MedCode Models read de-identified patient records and output primary and secondary ICD-10-CM diagnosis codes. Answers are checked against codes assigned by professional coders. Vals AI Medicine Expert Not in cards Claude Opus 5 63.6% Feb 2026 Nov 2026
EKI Eesti mäng Models choose answers to 1,000 questions from an Estonian quiz book. The questions cover subjects including history, culture, and geography. Institute of the Estonian Language Knowledge in many languages Everyday Not in cards Gemini 4 Argon 90% Jun 2026 Nov 2026
EKI Propaganda Resistance Models answer 75 questions in Estonian, English, and Russian without web search. Claude Opus 4.5 grades how each answer handles misleading premises and propaganda. Institute of the Estonian Language and Propastop Deception and alignment Everyday Not in cards Claude Opus 5 97.6 Jun 2026 Nov 2026
Chess Puzzles Models find the single best next move in 100 chess puzzles that Epoch generated with a chess engine. Scored by the share solved. Epoch AI Puzzles and chess Competition Not in cards Claude Fable 5.2 79% Dec 2025 Nov 2026
AGI-Eval Community Instruction-Tuned Language Models Models answer Chinese and English questions across reasoning, knowledge, learning, interaction and instruction following. AGI-Eval (Meituan) Reasoning in many languages School Not in cards GPT-5.5 0.86 Mar 2025 Jul 2026
FoQA Models answer questions about passages from Faroese Wikipedia. A native speaker checked and corrected automatically generated answers. Annika Simonsen and colleagues Reading in many languages School Not in cards Mistral Small 3.2 79.9% Feb 2025 Mar 2026
EKI Mälumäng Models answer 800 Estonian quiz questions without answer choices. Politics and geography each contribute 80 questions. GPT-5.4 nano judges answers that miss an exact match. Institute of the Estonian Language Knowledge in many languages Everyday Not in cards Gemini 3.1 Pro 50.8% Aug 2026 Nov 2026
GermanQuAD Models extract answers to German questions from manually annotated Wikipedia passages. Timo Möller and colleagues Reading in many languages School Not in cards Gemma 2 27B 70% Apr 2021 Mar 2026
HeQ Models extract answers from Hebrew Wikipedia and Israeli technology reporting, or identify questions that the passage cannot answer. Bar-Ilan University, Webiks and collaborators Reading in many languages School Not in cards Qwen2.5-72B 87% May 2023 Apr 2026
Open Hebrew LLM Leaderboard Models answer questions, resolve pronouns, classify sentiment and inference, translate passages, and answer trivia in Hebrew. Israeli National Program for NLP, Dicta and Webiks Knowledge in many languages School Not in cards Gemma 4 31B IT 74.3 Mar 2024 Apr 2026
Judgemark v4 The model under test judges fixed creative-writing samples against low and high anchor responses. The score is how well its ratings separate writer models. EQ-Bench (Sam Paech) Taste and style Expert Not in cards Claude Opus 4.6 90.7 May 2026 Aug 2026
MortgageTax Vision models read mortgage tax certificates given as images. They extract the year, parcel number and county, and compute the annualized amount due. Vals AI (data from Vontive) Finance Everyday Not in cards Claude Opus 5 72.1% Sep 2026
Mystery Game Puzzles 100 mid-game positions from an undisclosed puzzle variant of a well-known game. Model picks the single best move, checked against an answer key. Epoch AI Puzzles and chess Competition Not in cards GPT-6.2 Astra 95% Jul 2026 Nov 2026
MedQA Bias Evaluation Models answer medical exam questions before and after Vals adds racial information. Vals compares accuracy across patient groups. Vals AI Medicine Expert Not in cards o1 96.5% Jan 2025 Mar 2026
OCR-VQA Models answer questions about book covers by reading visible text. The benchmark compares answers with facts such as a book’s title or author. Anand Mishra and colleagues Documents, charts and text in images School Not in cards Kimi-VL-A3B-Instruct 82% Sep 2019 Apr 2025
Wer wird Millionär? LLM Benchmark Models play 45 games of the German version of Who Wants to Be a Millionaire, each with 15 questions of rising difficulty. Early questions lean on German wordplay and idioms. ikiruneo (GitHub user) Knowledge in many languages Everyday Not in cards GPT-6 Astra 644444.0 Sep 2025 Sep 2026
Miðeind Icelandic Case Declension Models decline Icelandic adjective and noun pairs in four cases and two numbers. The public test has 2,400 forms for 300 phrases. Miðeind Reasoning in many languages School Not in cards GPT-5.5 98% Nov 2023 Aug 2026
Miðeind Icelandic Grammatical Error Detection Models decide whether an Icelandic sentence contains a grammatical error. The test uses 100 correct sentences and 100 incorrect sentences from the Icelandic Error Corpus. Miðeind Reading in many languages School Not in cards Claude Fable 5 91.5% Dec 2023 Aug 2026
RuneBench Coding agents play RuneScape on an emulated game server by writing TypeScript against the rs-sdk library, with game-wiki notes for reference. Tasks train one skill or gather gold within 15 or 30 minutes. Skill tasks are scored by the peak XP rate in any 15-second window, and a verifier reads the final game state. Websim (Max Bittker, Surya Narreddi, Rob Haisfield, Sean Lee) Video games Everyday Not in cards GPT-6.2 Astra 7.85 Feb 2026 Nov 2026
Miðeind Icelandic ARC-Challenge Models answer multiple-choice science questions in an Icelandic translation of ARC-Challenge. Miðeind used Claude 3.5 Sonnet to translate the questions. Miðeind Reasoning in many languages School Not in cards GPT-5.5 95.2% Jun 2024 Aug 2026
EuroEval ERRNews Models summarize Estonian radio news transcripts. EuroEval adds stories from TalTech’s training split to make a 2,048-story test split and keeps 256 stories for validation. EuroEval Reading in many languages School Not in cards Llama 3.3 70B Instruct 32.8 Aug 2025 Nov 2026
Open LLM Leaderboard v1 Language models answer tasks from the original Open LLM Leaderboard suite. Hugging Face Composite indices University Not in cards Qwen2-72B 77.4 Apr 2023 Jun 2024
Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. Vals AI Composite work indices Expert Not in cards Claude Opus 4.8 70.7% May 2026 May 2026
APEX-1 Models complete expert-written work cases with source documents for four jobs: investment banking, consulting, big law and primary care. A judge model grades each response against rubrics. Mercor Tasks from real jobs Expert Not in cards Gemini 4 Argon 81.1% Sep 2025 Nov 2026
EuroEval Models answer questions, summarize text, and perform language understanding tasks across European languages. EuroEval combines task rankings on its European leaderboard. Dan Saattrup Smart and EuroEval contributors Reading in many languages School Not in cards GPT-6.2 Astra 1.17 Aug 2021 Nov 2026
HELM Safety Stanford CRFM runs five safety benchmarks over six risk categories, such as violence, fraud and discrimination, and reports one leaderboard. Stanford CRFM Refusing harmful requests Everyday Not in cards GPT-5 nano 98.6 Nov 2024 Nov 2025
SEA-HELM Models are evaluated in Filipino, Indonesian, Tamil, Thai and Vietnamese across five pillars: NLP classics, LLM-specific tasks, linguistics, culture and safety. AI Singapore + Stanford CRFM Reasoning in many languages School Not in cards GPT-6 Astra 79.9 Feb 2025 Nov 2026
Pre-Flight Models answer questions about aviation regulations, dispatch, pilot procedures and airport operations. The evaluator compares each choice with the answer key. Airside Labs Other professions Expert Not in cards GPT-5.6 Sol 84% Mar 2025 Sep 2026
HELM Lite A lightweight HELM board that runs models with in-context learning on ten scenarios. They include NarrativeQA, NaturalQuestions, OpenbookQA, MMLU, GSM8K, MATH, LegalBench, MedQA and WMT 2014. Stanford CRFM Composite indices University Not in cards GPT-4o 93.8 Dec 2023 Mar 2025
Caliper Bench v3 Models continue prose passages and write dialogue in character. The graders score writing craft, character consistency, and errors across sessions lasting up to 50 turns. Caliper Bench Creative writing School Not in cards Qwen4-Max 88.9 Sep 2026 Nov 2026
EuroEval Faroese Grammatical Error Correction Models rewrite a Faroese student’s sentence to correct its grammar. An exact match to the reference correction earns credit. EuroEval, using sentence pairs from the University of the Faroe Islands and the University of Copenhagen Reading in many languages School Not in cards Gemma 3 27B PT 16.9% Aug 2026 Jan 2026
NarrativeQA Models answer questions about books and movie scripts, either from a Wikipedia summary or from the full story. The questions need an understanding of the whole narrative rather than one matching passage. Answers are free text, scored against two reference answers with F1, ROUGE-L or BLEU. DeepMind and University of Oxford (Kočiský et al.) Reading comprehension Everyday Not in cards GPT-4o 0.80 Dec 2017 Feb 2025
CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. Center for AI Safety Composite indices Research frontier Not in cards Gemini 4 Argon 66.5 Dec 2025 Nov 2026
Icelandic WinoGrande Models complete Icelandic sentences by choosing which referent fits the context. The evaluation uses an Icelandic adaptation of WinoGrande. Vésteinn Snæbjarnarson and colleagues Reasoning in many languages Everyday Not in cards Gemini 3.7 Flash 96.3% Jun 2022 Aug 2026
CAIS Risk Index Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games. Center for AI Safety Composite indices Research frontier Not in cards Claude Fable 5.2 8.40 Dec 2025 Nov 2026
ParseBench-2 Parsers extract text, formatting, tables, charts, layout, and forms from 4,317 page cases across 2,645 documents. Rule checks score six dimensions and average them equally. LlamaIndex Documents, charts and text in images Expert Not in cards Claude Opus 5.5 87% Nov 2026 Nov 2026
TeleQnA Models answer multiple-choice questions about telecommunications terminology, research, and technical standards. Ali Maatouk and colleagues Expert-written questions Expert Not in cards Gemini 3.1 Pro 85.2% Oct 2023 Feb 2026
TeleTables Models answer questions about tables from 3GPP specifications, both from memory and with the table available. Anas Ezzakri and colleagues Expert-written questions Expert Not in cards Gemini 3 Flash 48% Jan 2026 Feb 2026
LongShOTBench Models answer single-turn and multi-turn questions about long videos using visual, spoken, and ambient audio evidence. Weighted rubrics award partial credit. MBZUAI Long videos Everyday Not in cards Qwen3-Omni-30B-A3B-Thinking 64.2% Dec 2025 Apr 2026
Cyber ECI Epoch estimates model capability from cybersecurity benchmark scores and places the result on the general index scale. Epoch AI Composite indices Expert Not in cards Claude Opus 5.5 171.1 Oct 2026
FoodTruck Bench Agents run a food truck in Austin, Texas for a 30-day simulation, choosing locations, menus, prices, inventory, staff and loans with 34 tools. Humans can play the same game. FoodTruck Bench maintainers Simulated companies Everyday Not in cards Gemini 4 Neon $215,319 Nov 2026
Phare Models are probed in English, French and Spanish for hallucination, harmful misguidance, stereotypes and jailbreaks. Hallucination tasks cover factuality, misinformation, debunking and tool use. Giskard Hallucination rates Everyday Not in cards Claude Haiku 4.5 83.2% Feb 2025 Jun 2026
AGI-Eval Community Image and Video Understanding Models interpret images and videos. The board reports visual question accuracy and separate comparison win rates. AGI-Eval (Meituan) General visual questions Everyday Not in cards Gemini 3 Flash 86.5 Feb 2025 Apr 2026
Razavi-Bench Models explain what happens in analog circuits and why. A language model awards up to four points per answer against a reference and a rubric. Zhishuai Zhang, Oscar Toomey, and Taekwang Jang Electronics University Not in cards Gemini 4 Carbon 98.3% Nov 2026
Dubesor Lichess Puzzle Rating Models solve sequences of moves in 500 rated Lichess puzzles. The organizer fits a performance rating against puzzle difficulty using only first attempts. Dubesor Puzzles and chess School Not in cards Gemini 4 Neon 2,455 Oct 2026 Nov 2026
Miðeind Icelandic Case Government Models inflect a subject or construct a verb and object phrase in Icelandic. The verb determines the grammatical case. The evaluator checks exact agreement. Miðeind Reasoning in many languages School Not in cards Claude Fable 5.2 100% Sep 2026 Nov 2026
Miðeind Icelandic Grammar Correction Models correct an Icelandic sentence or return it unchanged. Half the sentences contain one error. The score measures exact agreement with the expected sentence. Miðeind Chat and instructions in many languages School Not in cards Claude Fable 5.2 83.9% Sep 2026 Nov 2026
Miðeind Icelandic Idioms Models supply a missing word in an Icelandic idiom and explain its meaning. A program checks the missing word. A language model grades the explanation. Miðeind Reasoning in many languages Everyday Not in cards Claude Fable 5.2 85.7% Sep 2026 Nov 2026
Miðeind Icelandic Popular Culture Models continue Icelandic song lyrics and identify the sources of familiar quotations. The two tasks measure word overlap and exact agreement, respectively. Miðeind Knowledge in many languages Everyday Not in cards Claude Fable 5.2 39.5% Sep 2026 Nov 2026
Miðeind Icelandic Terminology Models explain Icelandic knitting and crochet terms and name local species from English and Latin names. A model grades definitions. A program checks species names. Miðeind Knowledge in many languages University Not in cards Claude Fable 5.2 78% Sep 2026 Nov 2026
Open LMM Reasoning Leaderboard Models solve multimodal reasoning tasks. OpenCompass ranks the models using the component results. OpenCompass Composite indices University Not in cards Doubao-1.5-pro 61.6 Dec 2024 Apr 2025
HELM Capabilities Stanford CRFM runs language models on a curated set of scenarios, one per key capability, and averages the scores. Prompt-level results are public. Stanford CRFM Composite indices University Not in cards GPT-5 mini 81.9 Mar 2025 Nov 2025
Open Japanese LLM Leaderboard Models perform Japanese language tasks drawn from more than 20 datasets. LLM-jp and Hugging Face Knowledge in many languages University Not in cards QwQ-32B-Preview 0.67 Nov 2024 May 2025
RuLES 2.0 Models follow explicit rules while talking with a user. Each scenario checks automatically whether the model broke a rule. Norman Mu and collaborators Following precise instructions Everyday Not in cards GPT-4 Turbo 9.54% Mar 2024 Feb 2024
Icelandic WikiQA Models answer 1,900 factual questions drawn from Icelandic Wikipedia. A language model grades each answer against a human-reviewed reference. Miðeind Knowledge in many languages Everyday Not in cards Claude Fable 5.2 86.6% Jul 2024 Nov 2026
LLM Confabulation Leaderboard Models answer misleading questions about recent articles whose answers are not in the text. The board ranks confabulation rate against non-response rate. Lech Mazur Faithful to sources Everyday Not in cards GPT-5 10.3% Oct 2024 Aug 2025
Miðeind Icelandic Generative Quality Models write paragraphs in Icelandic. Málfríður checks for grammar errors, and a dictionary lookup counts words absent from the Database of Modern Icelandic Inflection. Miðeind Chat and instructions in many languages School Not in cards GPT-6.2 Sol 87.7% Sep 2026 Nov 2026
Agent Island Models play multi-agent games that require cooperation, conflict, and persuasion. Agent Island authors Multi-agent games Everyday Not in cards GPT-5.5 0.88 May 2026 Jun 2026
Maze Bench Agents explore connected rooms in a 3D puzzle game. Each of the 100 gems contributes one percentage point to the score. Jonathan Pappas and the MazeBench team Puzzles and chess Expert Not in cards GPT-6 Astra 24% May 2026 Nov 2026
Nejumi LLM Leaderboard 4 Models solve Japanese language, reasoning, coding, and alignment tasks. The organizers weight individual categories when calculating the overall score. Weights & Biases Japan Reading in many languages School Not in cards Aug 2025
Arabic Cultural Value Alignment Models classify statements about Arabic culture as true or false. The evaluation covers 58 subject areas. Freedom Intelligence and the AceGPT team Knowledge in many languages School Not in cards Qwen2.5-Math-72B-Instruct 73.9% Sep 2023 Jan 2025
BAAnchmark Models build a 3D sheep herding game from the same Slovak prompt. Marek Dubovsky plays each runnable game and ranks its controls, playability, and bugs. Dubovsky Labs Web and app front ends University Not in cards Claude Sonnet 5.5 1.00 Nov 2026
Earth-Bench V2 Agents analyze prepared satellite observations. The corrected release measures final answer accuracy, tool selection, argument accuracy, and execution efficiency. OpenDataLab and collaborators Apps and MCP servers University Not in cards Claude Opus 5.5 80% Oct 2026 Oct 2026
Fiction.LiveBench Models answer 36 questions about 30 Fiction.live stories that need inference about plot and characters. Each story is tested at several context lengths. Fiction.live Reasoning over long documents Everyday Not in cards o3-pro 97.2% Feb 2025 Jan 2026
Visual Reasoning Benchmark (Fab AI) Models choose which picture completes a pattern, matches another shape, or shows a folded figure. Questions come from school exams in Zambia and India. Fab AI Spatial reasoning and counting School Not in cards Gemini 3.7 Flash 89.9% Sep 2025 Aug 2026
GeoBenchX v2 Agents use GIS tools to join data, measure spatial relationships, and draw maps. Three model judges compare their work with corrected reference solutions. Varvara Krechetova and Denis Kochedykov Apps and MCP servers University Not in cards Claude Opus 5.5 84% Jun 2025 Oct 2026
Tuesday Work Index A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday. Surge AI Composite work indices Expert Not in cards Gemini 4 Neon 71.6 Aug 2026 Nov 2026
CAIS Vision Capabilities Index Averages model scores on five vision tests run by the Center for AI Safety: ERQA, MindCube, SpatialViz-Bench, IntPhys 2 and EnigmaEval. Center for AI Safety Composite indices Research frontier Not in cards Claude Fable 5.2 84.0 Dec 2025 Nov 2026
Lech Mazur Creative Writing Models write short stories that must use 10 assigned elements such as characters, objects, and motivations. Judge models compare matched stories pairwise. Lech Mazur Creative writing University Not in cards GPT-6.2 Astra 4.30 Jan 2025 Nov 2026
TaxCalcBench v2 2 (TY 2025) Models read W-2, 1099 and other PDFs and compute full federal and state returns line by line. A return counts only when every evaluated line is exact. Column Tax Finance Expert Not in cards Claude Fable 5.2 86% Jun 2026 Nov 2026
VISTA Models answer 758 single-turn prompts about one image each, covering perception and reasoning. Three LLM judges grade answers with yes/no rubric questions. The main score is average task accuracy. Scale AI General visual questions Expert Not in cards Gemini 2.5 Pro 54.6% Dec 2024 Mar 2026
CaseLaw v2 Models answer private questions over recent Canadian court decisions, such as finding key cases and answering across several documents. Vals AI (with Jurisage, Together AI and Stanford CodeX) Law Expert Not in cards Grok 4.3 79.3% Aug 2025 Apr 2026
FACTS Search V2 Models answer information-seeking questions with a search tool and must combine what they find correctly. Second version of FACTS Search on Kaggle. Google DeepMind + Google Research + Kaggle Short factual questions University Not in cards GPT-6 Astra 89.9% Jul 2026 Oct 2026
Instrumentation Universal Knowledge v1.6 refresh Models answer 60 technician questions and 97 specialist questions from the June samples. Three model judges score responses under higher output limits. RelayForge Tasks from real jobs Expert Not in cards GPT-6 Luna 93.3% Oct 2026 Oct 2026
Pencil Puzzle Bench Models solve 300 pencil puzzles of 20 varieties with unique solutions, either in one shot or agentically with step checks. Every board state can be verified against the rules. Approximate Labs Puzzles and chess Expert Not in cards Claude Fable 5 97.6% Mar 2026 Jun 2026
DataAnalyticsBench Agents query a synthetic data warehouse to answer business questions. A judge model checks answers against a rubric and an answer key. Plotly and Mila Running a business University Not in cards GPT-6 Astra 43.0 Oct 2026 Nov 2026
OpenLMM Spatial Leaderboard Models answer spatial reasoning tasks involving images. OpenCompass compares performance across the component evaluations. OpenCompass Composite indices University Not in cards GPT-5 62.5 Jun 2025 Aug 2025
Parallel Search Capability Leaderboard Language models answer research questions with and without search, and Parallel ranks their accuracy, search lift, cost, and time. Parallel Web Systems Composite indices Expert Not in cards Gemini 4 Neon 88.1 Sep 2026 Nov 2026
PolicyBench No-tools estimate of US taxes and benefits for 100 households across 18 programs. Headline metric is answers within $1 of PolicyEngine-US. PolicyEngine Finance Expert Not in cards GPT-6 Sol 95.2% Jul 2026 Nov 2026
Swiss Smartvote 2023 party agreement Models answer 75 Swiss election questions about their own policy positions. Researchers compare their answers with political parties and candidates to measure agreement. Smartvote and Jannis Vamvas Bias, fairness and politics Everyday Not in cards Claude 3.7 Sonnet 80.1% Aug 2023 Mar 2026
ClawProBench Agents on stateful runtimes are scored with their traces, covering evidence gathering, runtime routing, safety boundaries and repeated runs in frozen workplace setups. ClawProBench authors General assistants Expert Not in cards GLM-5.2 80% Aug 2026 Jul 2026
Galileo Agent Leaderboard v2 Agents handle simulated multi-turn customer support conversations in banking, healthcare, insurance, investment and telecom, with synthetic personas and domain tools. Scores are Action Completion and Tool Selection Quality. Galileo Customer service Everyday Not in cards GPT-4.1 62% Jul 2025 Aug 2025
Public Benefits Bench Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts. Vals AI Other professions Expert Not in cards Claude Opus 5 76.9% Jun 2026 Oct 2026
AA-LCR Extended Models answer questions that need reasoning across about 100k tokens of documents. Mercor adds 89 new expert-written tasks to the 100 public ones and reports both scores. Mercor Reasoning over long documents Expert Not in cards GPT-5.6 Sol 79.5% Aug 2026 Oct 2026
DAXBench v2 Models write DAX, the formula language of Power BI, for 94 tasks on understanding, writing and reasoning over the Contoso sample model. Each answer runs inside real DAX queries in several report contexts, such as detail rows, subtotals, the grand total and slicers. Harder tasks earn more points, with bonuses for best practices and efficient code. Maxim Anatsko, with SQLBI Data science and SQL University Not in cards GPT-6 Astra 95.9% Oct 2026 Nov 2026
HealthBench-Psych v2 Models respond to 611 health conversations that clinicians selected for mental health relevance. Three model judges score each answer against its physician rubric. MindBench.ai Medicine Expert Not in cards Claude Sonnet 5.5 0.66 Sep 2026 Sep 2026
BRIDGE Models work on 87 tasks built from real clinical text, such as electronic health records, in nine languages. The tasks span triage, consultation, information extraction, diagnosis, prognosis and billing codes across 14 specialties. Each task uses its own automatic metric, and the board reports the average. Harvard Medical School and Brigham and Women's Hospital (YLab) Medicine Expert Not in cards Qwen3.8-Flash-Next 56.0 Apr 2025 Aug 2026
PCT (Polarized Contrastive Pairs) Models answer matched left-coded and right-coded prompts, such as Obama and Reagan. Judges score whether sentiment and helpfulness stay consistent across each pair. Center for AI Safety Bias, fairness and politics Everyday Not in cards Muse Spark 1.1 27.9% May 2026 Nov 2026
Ramp SWE-Bench v2 Agents solve 500 engineering tasks from Ramp's production backend. A panel of model judges compares each patch with the code engineers merged. Ramp Labs Fixing real repositories Expert Not in cards GPT-6.2 Astra 76.4% Nov 2026 Nov 2026
Spatial Agent Index v2 Agents analyze geospatial files and produce numbers, maps, or lists. Axis Spatial averages the scores for each included evaluation. Axis Spatial Apps and MCP servers University Not in cards Claude Opus 5.5 86.0 Nov 2026 Oct 2026
BU Bench V2.1 Browser agents do 200 real web tasks. A findings judge checks tool evidence, answers and files, and deterministic integrity checks run as well. Browser Use Web browsing Everyday Not in cards GPT-6.1 Sol 81.3 Sep 2026 Oct 2026
Misguided Attention Models get slight variations of famous riddles, paradoxes and thought experiments. A model fails when it answers the well-known original instead of the problem as stated. cpldcpu (GitHub user) Abstract puzzles and logic Everyday Not in cards Grok-3 mini 73.7% May 2024 Jun 2025
AraGen Models answer Arabic questions in four tasks: question answering, reasoning, orthographic and grammatical analysis, and safety. An LLM judge scores each answer on the 3C3H measure: correctness, completeness, conciseness, helpfulness, honesty and harmlessness. Only correct answers get credit on the other dimensions. Inception and MBZUAI, with Hugging Face Chat and instructions in many languages Everyday Not in cards o1 84.3% Dec 2024 Apr 2026
Hemingway-bench Professional writers compare two models' answers blind on real writing prompts, from bedtime stories to business documents. The comparisons become ratings. Surge AI Creative writing Expert Not in cards Claude Fable 5 1,110 Feb 2026 Sep 2026
QIMMA Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review. Technology Innovation Institute (AlQadi, Alzubaidi et al.) Knowledge in many languages School Not in cards Qwen3.5 397B-A17B 69.2% Jan 2026 Aug 2026
SWE-bench Verified Extended Agents resolve real GitHub issues by editing codebases, scored by repository tests. Mercor adds 50 new expert-written tasks to the 500 public ones and reports both scores. Mercor Fixing real repositories Expert Not in cards DeepSeek V4 Pro 0813 83.3% Aug 2026 Oct 2026
Terminal-Bench 2.1 Extended Agents complete long command-line tasks in containers, scored by verification tests. Mercor adds 99 new expert-written tasks to the 89 public ones and reports both scores. Mercor Terminal tasks Expert Not in cards Muse Wonder 54% Aug 2026 Oct 2026
ContractorOS Quantity Takeoff Models fill two bills of quantities from electrical and plumbing drawings. Counts must match the estimator exactly. Measured lengths must fall within 5%. ContractorOS Tasks from real jobs University Not in cards Gemini 4 Carbon 60% Sep 2026 Nov 2026
GDPval Extended Models produce work products for occupations across nine GDP sectors. Mercor adds 132 new expert-written tasks to the 213 public ones and reports both scores. Mercor Tasks from real jobs Expert Not in cards Claude Opus 5.5 7.10% Aug 2026 Oct 2026
K-MetBench Models answer 1,774 questions from the Korean National Meteorological Engineer Examination under a zero-shot protocol. The benchmark reports answer accuracy across five subjects and separate visual and Korean-specific subsets. Expert rationales support an additional rubric score for reasoning. KAIST, INEEJI and Kongju National University Science questions Expert Not in cards Gemini 3 Pro 93.7% Apr 2026 Jul 2026
SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. Snorkel AI Finance Expert Not in cards Grok 4.7 30.5% Oct 2026 Oct 2026
Toloka Arena Enterprise Tool Use (November 2026) Agents complete private workflows across 11 industries. They must leave the right records and follow the required process in all five attempts. Toloka Customer service Expert Not in cards Gemini 4 Neon 81% Nov 2026 Nov 2026
vCX Benchmark Models run inside a Retell voice agent on pass or fail checks for business rules, grounding, tool calls and speech-ready replies. Each suite reports its own pass rate. Retell AI Speech recognition and translation Everyday Not in cards GPT-6.1 Sol 93.3% Oct 2026
BrowseComp Extended Agents browse the web to find hard-to-locate facts with short answers. Mercor adds 100 new expert-written tasks to the 130 public ones and reports both scores. Mercor Web browsing Expert Not in cards Claude Sonnet 5.5 73% Aug 2026 Oct 2026
BRUMO 2025 Models answer the 30 final-answer problems of the 2025 Brown University Math Olympiad. MathArena scores the average accuracy over four attempts. Brown University (contest), MathArena (evaluation) Competition problems with answers Competition Not in cards Gemini 3 Flash 100% Apr 2025 Feb 2026
Envisioning Signals Benchmark Models identify early signs of change for an industry brief. Scores combine verifiable evidence, specific claims, recent sources, and coverage across the requested topics. Envisioning Running a business University Not in cards Claude Opus 4.7 86.0 May 2026 Oct 2026
KORA Simulated child users hold 3-turn conversations probing 26 child-specific risks. Judge model rates replies failing, adequate or exemplary. KORA (non-profit) Wellbeing and sycophancy Not in cards GPT-6 Astra 73.2% Feb 2026 Oct 2026
Political Science LLM Benchmark 33 political-science text-coding tasks, 100 texts each (~3,300 items). Models scored by mean F1 against human-coded labels. Hanno Hilbig (UC Davis) Other University Not in cards Claude Opus 5 71.4% May 2026 Oct 2026
FlavourBench Powered v43 powered v43 Models choose ingredients for 640 tasks across four categories, including regional composition. Failed responses earn zero. Models must complete 608 tasks to qualify for ranking. Josef Chen, Jakub Radzikowski, and Erim Hayretci Taste and style Everyday Not in cards Grok 4.6 73.4 Aug 2026 Aug 2026
Furniture Assembly Benchmark Model gets an IKEA manual and a photo of a partly built item, then names the faulty step or next step. Accuracy on 60 photos. Epoch AI Spatial reasoning and counting Everyday Not in cards Claude Fable 5.2 90% Sep 2026 Nov 2026
General-Bench Models perform multimodal understanding and generation tasks. General-Level averages task scores and measures performance above specialist baselines at progressively broader levels. National University of Singapore, Nanyang Technological University, Zhejiang University, and collaborators General visual questions University Not in cards InternVL2.5-8B 8.41 May 2025 Dec 2024
LevelBench-100 v5.0 Models use ledger exports and supporting workbooks to close small business accounts. They reconcile transactions, apply accounting rules, and identify missing records. Level Finance Expert Not in cards Gemini 4 Carbon 95.2% Nov 2026 Nov 2026
MedXpertQA MM Extended Models answer multimodal medical questions over clinical images and patient details. Mercor adds 51 new expert-written tasks to the 200 public ones and reports both scores. Mercor Medicine Expert Not in cards Claude Opus 5.5 71.2% Aug 2026 Oct 2026
MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass. Epoch AI + METR Building whole projects Research frontier Not in cards Claude Sonnet 5.5 97.8% Jun 2026 Nov 2026
MOMENT Eval Models use a calculator and Python to solve 91 structural engineering problems. They return a value or classification that a program checks against the answer key. Thornton Tomasetti CORE studio Tasks from real jobs Expert Not in cards Claude Opus 4.7 95.6% Aug 2026 Jul 2026
Open FlagEval VLM Leaderboard Models answer questions about images. The leaderboard compares their scores across visual understanding tasks. Beijing Academy of Artificial Intelligence General visual questions University Not in cards Gemini 2.5 Pro 70.7 Feb 2025 Apr 2025
PAIRS (Parallel Images) Models answer questions about people in paired images. The evaluation tests whether changing demographic characteristics changes judgments about the subjects. Kathleen C. Fraser and colleagues Bias, fairness and politics Everyday Not in cards GPT-4o 97.2% Feb 2024 Apr 2025
Unicorn (Vision Safety) Models answer questions about objects in unusual images and sketches. The benchmark tests how well visual judgments hold up when the images change. Haoqin Tu and colleagues General visual questions Everyday Not in cards Qwen2-VL-72B-Instruct 91.2% Nov 2023 Apr 2025
VHELM Vision-language models are scored across nine aspects, including perception, reasoning, knowledge, bias, fairness, multilinguality, robustness, toxicity and safety, from many existing datasets. Stanford CRFM General visual questions University Not in cards Gemini 2.5 Pro 87.1 Oct 2024 Jun 2025
Antidote Experts such as doctors, lawyers and engineers compare model answers to prompts from their own history. They judge which answer would still help them a month later. Surge AI Overall chat quality Expert Not in cards Claude Fable 5 1,099 May 2026 Sep 2026
BenCzechMark Models complete Czech language tasks spanning knowledge, comprehension, and reasoning. Czech Language Community Knowledge in many languages University Not in cards DeepSeek-V3 82.9 Oct 2024 Aug 2026
ClockBench Models read analog clock faces and answer questions about the time shown. Most clocks stay private to avoid training leakage. Alek Safar Spatial reasoning and counting Everyday Not in cards Claude Fable 5.2 87.8% Sep 2025 Nov 2026
KRYT Benchmark Models decide refund requests from transaction limits and event records. The grader checks policy compliance, valid actions, and consistency across rounds. vaibhav lalwani Customer service Everyday Not in cards Gemini 2.5 Pro 32% Oct 2026 Oct 2026
Long-Horizon-Terminal-Bench 2.0 Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards. Tencent HY + university collaborators (Zongxia Li et al.) Long terminal sessions Expert Not in cards GPT-6 Astra 69.2% Oct 2026 Sep 2026
SMT 2025 Models answer 53 final-answer problems from the 2025 Stanford Math Tournament. MathArena scores the average accuracy over four attempts. Stanford Math Tournament (contest), MathArena (evaluation) Competition problems with answers Competition Not in cards Gemini 3 Pro 93.4% May 2025 Feb 2026
TMMLU+ 1.1 Models answer traditional Chinese subject questions from a revised exam set. The revision corrects 197 answers and removes 539 questions from the original release. iKala Knowledge in many languages University Not in cards Claude Opus 5 96.2% Sep 2026 Aug 2026
ArbMMLU-HT Models answer a human translation into Arabic of the 57 MMLU subjects. Inception and MBZUAI (Jais project) Knowledge in many languages University Not in cards Qwen2.5-72B 72.2% Sep 2024 Aug 2025
Construction's Last Exam v2 Models measure construction drawings, answer questions across sheets, and price quantities under contract rules. The board reports geometry, drawing reasoning, and estimating separately. Togal.AI Tasks from real jobs Expert Not in cards Gemini 3.6 Flash 36.8% Oct 2026 Sep 2026
Open Ko-LLM Leaderboard 2 Models complete nine Korean evaluations covering reasoning, instructions, social values, safety, and other abilities. The organizers score submitted models on private test data. Upstage and the National Information Society Agency Reasoning in many languages School Not in cards Qwen2.5-Coder-14B-Instruct 49.6% Aug 2024 Nov 2024
PlotPoints Round 4: Willingness and Judgment Models respond to scripted roleplay scenes. Two classifiers check whether models respect a stated limit and whether they refuse allowed requests. Plotlight Studios Deception and alignment Everyday Not in cards Claude Fable 5.1 0.8 Sep 2026 Nov 2026
Spatial Intelligence Index v2 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. Axis Spatial Composite indices University Not in cards Claude Opus 5.5 88.7 Nov 2026 Oct 2026
ApprenticeBench Agent joins a simulated company and processes 100 accounts-payable bills in Odoo via GUI or API, learning from records and feedback. Cumulative success rate. NeoCognition Web browsing University Not in cards Claude Opus 5.5 77% Sep 2026 Nov 2026
Can AI close the books? FP&A reasoning Models complete synthetic financial close tasks, including accruals and reconciliations. The grader checks calculations and whether reviewer claims agree with the books. Marco Polo Finance University Not in cards Gemini 2.5 Pro 100% Oct 2026 Oct 2026
CharXiv Extended Models answer descriptive and reasoning questions about scientific charts. Mercor adds 150 new expert-written tasks to the 5000 public ones and reports both scores. Mercor Documents, charts and text in images University Not in cards Muse Spark 1.2 87.1% Aug 2026 Oct 2026
DAYJOB: Healthcare Agents handle short requests from healthcare staff using charts, claims exports, policies and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Surge AI Medicine Expert Not in cards Claude Opus 5.5 48.3% Sep 2026 Oct 2026
GameHorizon Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts. Tencent ARC Video games Everyday Not in cards GPT-6 Astra 80.2 Sep 2026 Sep 2026
MMMU-Pro Extended Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores. Mercor General visual questions University Not in cards GPT-6.1 Sol 61.3% Aug 2026 Oct 2026
MultiNRC Models answer over 1,000 reasoning questions written by native speakers in French, Spanish and Chinese. Questions cover linguistic reasoning, wordplay, cultural knowledge and culturally grounded math. A GPT-4.1 judge compares each answer with a short reference answer. Scale AI Reasoning in many languages Expert Not in cards Muse Spark 1.1 65.6% Jul 2025 Jul 2026
Prompt to Product Agents build web applications from product briefs. Browser tests exercise features such as checkout arithmetic, saved filters, keyboard accessibility, and mobile layouts. Shash Web and app front ends University Not in cards GPT-5.6 Sol 98.2% Oct 2026 Oct 2026
SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. Snorkel AI Finance Expert Not in cards Grok 4.6 25.1% Oct 2026 Oct 2026
SWE Atlas Codebase QnA Extended Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores. Mercor Fixing real repositories Expert Not in cards Claude Sonnet 5.5 44.7% Aug 2026 Oct 2026
HiL-Bench Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary. Scale AI Fixing real repositories Expert Not in cards Muse Wonder 79.7% Apr 2026 Nov 2026
Last Translation Benchmark Models translate inputs that other systems struggled with. A model checks each translation against human-written rules. A translation passes only if it meets every rule. Zouhar et al. Translation Research frontier Not in cards GPT-6 Astra Pro 45.3% Sep 2026 Oct 2026
LLM Agronomy Benchmark Image Identification (four choices) Models choose among four labels for each photo of a crop disease, insect pest, or weed. One accuracy score combines the 150 questions. gbstox General visual questions University Not in cards Gemini 3 Flash 90% Jun 2026 Jun 2026
MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. Microsoft Research Exam questions University Not in cards GPT-4o 73.4% Dec 2024 Dec 2024
Persian NLG Models summarize Persian texts, translate between Persian and two other languages, and write questions from passages. MCINext Translation University Not in cards Llama 3.3 70B Instruct 20.1 May 2025 Sep 2025
RandomBench Models repeatedly choose an integer from 0 to 99. The benchmark measures how evenly they use the range and how often they return valid numbers. Benjamin Brumfield Abstract puzzles and logic Everyday Not in cards Grok 4.6 0.42 Nov 2026 Nov 2026
Vibe Code Bench 1-100 v1.1 Agents extend a working web application across successive requests while preserving earlier features. Vals AI Building whole projects Expert Not in cards GPT-6.2 Astra 55.2% Nov 2026 Nov 2026
FINESSE-Bench Models answer financial certification questions and interpret trading indicators. The benchmark checks numerical and multiple-choice answers and uses a model judge for free responses. Dmitry Stanishevskii and colleagues Finance University Not in cards GPT-5.5 0.90 May 2026 Jun 2026
Itzi Roleplay Benchmark: Local Robustness Models play eight scripted episodes. Judges check whether each model remembers instructions, respects player decisions, and keeps characters and locations consistent. Itzi Creative writing Everyday Not in cards MiMo-V2.5 100.0 Sep 2026 Sep 2026
LongBench Pro Models answer English and Chinese questions over documents from 8,000 to 256,000 tokens. The benchmark applies task-specific metrics to 1,500 examples. Ziyang Chen and colleagues Reasoning over long documents University Not in cards Gemini 2.5 Pro 73.4% Jan 2026 Dec 2025
MCS-Bench Models read seal inscriptions, identify calligraphy, and answer questions about Chinese paintings and artifacts. The overall score averages task scores. South China University of Technology and INTSIG General visual questions Expert Not in cards InternVL2_5-78B 49.3 Feb 2025 Mar 2025
Persian IFEval Models respond to 457 Persian prompts while following constraints on wording, length, or structure. MCINext Chat and instructions in many languages Everyday Not in cards o3 92.6% May 2025 Sep 2025
The Aggregate Day-one Recall Check Models answer 34 questions about 12 public-domain books from memory. The scorer matches answers against a key and excludes empty replies. Mikhail Doroshenko (The Aggregate) Trivia and open questions School Not in cards Claude Fable 5 61.8% Aug 2026 Nov 2026
FinGREAT Models identify accounting relationships in financial passages. The authors compare 44 models against human annotations across 483 passages. Carnegie Mellon University Finance University Not in cards Claude Fable 5.1 92.8% Oct 2026 Sep 2026
GuiaCat Models classify the sentiment of restaurant reviews written in Catalan. Projecte Aina Reading in many languages Everyday Not in cards Apertus 70B 69.5% Oct 2022 Jan 2026
Memory Discipline Models answer project questions using saved memories. The tasks separate using relevant facts from asking for clarification when memories are absent or contradictory. GiulioDER Finding facts in long inputs School Not in cards GPT-6 Astra 99.7% Oct 2026 Sep 2026
MIZAN: A Persian LLM Leaderboard Models answer questions, complete sentences, follow instructions, and hold conversations in Persian. The leaderboard averages six task scores. MCINext Knowledge in many languages School Not in cards o3 0.75 May 2025 Sep 2025
Open Chinese LLM Leaderboard Models answer Chinese language tasks. The leaderboard compares model results across its evaluation suite. Beijing Academy of Artificial Intelligence Knowledge in many languages University Not in cards Qwen2-72B 75.7 May 2024 Jun 2024
PerCoR Models read the beginning of a Persian passage and choose its original ending from four options. MCINext Reasoning in many languages Everyday Not in cards Gemini 2.5 Pro 94.3% May 2025 Sep 2025
PerMMLU Models answer Persian multiple-choice questions from school curricula, university entrance exams, and Iranian cultural sources. MCINext Knowledge in many languages School Not in cards Gemini 2.5 Pro 84% May 2025 Sep 2025
Persian MT-Bench Models follow Persian conversations across two to four turns. One asks for film directors, then actors, then the oldest and newest films. MCINext Chat and instructions in many languages Everyday Not in cards o3 9 May 2025 Sep 2025
PerspectiveGap Models assign information fragments to workers with different roles. A strict pass requires supplying all needed information without disclosing extra fragments. Youran Sun and colleagues Following precise instructions University Not in cards GPT-5.5 62% Jun 2026 Sep 2026
RiskRubric Evaluators score public models on six dimensions of risk. The scorecard combines the results into a letter grade. Cloud Security Alliance, Haize Labs, Harmonic Security, and Noma Security Deception and alignment Expert Not in cards Claude Haiku 5.5 891 Aug 2025 Oct 2026
AutoBench Agronomy LLM Benchmark Models answer questions from three types of farm operators and an agricultural researcher. Other models rank the answers. AutoBench and EVJA Tasks from real jobs University Not in cards GPT-5.1 4.85 Dec 2025 Nov 2025
AutoCodeBench Models solve automatically generated hard coding problems across many programming languages. A sandbox runs generated tests on each solution. Tencent Hunyuan Function-level problems University Not in cards Claude Opus 4 52.4% Aug 2025 Aug 2025
BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. UCL DARK et al. Video games Everyday Not in cards GPT-6 Astra 68.3% Nov 2024 Sep 2026
BaxBench Models write backend servers from a specification. Each solution is checked for correctness by tests and for security by end to end exploits. ETH Zurich SRI Lab, LogicStar.ai, UC Berkeley and INSAIT Secure code and review Expert Not in cards Claude Opus 4.5 68.6% Feb 2025 Nov 2025
EBR-bench Agent plays the board game Earthborne Rangers 10-30 times in a row. Scored on objectives completed (of 21) to test learning from experience. Epoch AI Puzzles and chess Everyday Not in cards GPT-6.2 Astra 92.4% Jul 2026 Nov 2026
ExtractBench Systems read a full business document and return a schema-valid JSON object with every requested value. Unified value F1 compares the output with the expected values. LlamaIndex Documents, charts and text in images Everyday Not in cards GPT-6.1 Sol 95.3% Jul 2026 Oct 2026
GDP.xlsx Agents open a professional Excel workbook in a sandbox and answer a short question about it, as a domain professional would. A Gemini autograder checks the final answer against criteria. Surge AI Documents and spreadsheets Expert Not in cards Gemini 4 Neon 43.1% Sep 2026 Nov 2026
GreekMMLU Models answer questions originally written in Greek for academic, professional, and government exams across 45 subjects. Yang Zhang and colleagues Knowledge in many languages University Not in cards Gemini 3 Flash 93.2% Feb 2026 Dec 2025
Human Verification Gap Models respond to paired requests about the same synthetic image. The evaluation measures how the stated purpose changes their refusal rate. Hersheyz Refusing harmful requests Everyday Not in cards Claude Sonnet 4.5 1.00 Oct 2026 Sep 2026
HumanAgencyBench Models respond to 3,000 prompts about six ways assistants can support human agency, including asking clarifying questions and leaving important decisions to users. Benjamin Sturgeon and colleagues Deception and alignment Everyday Not in cards Claude Sonnet 4.6 71.3% Sep 2025 Apr 2026
IneqMath Models solve Olympiad-level inequality problems, recast as two checkable tasks: estimating a tight bound and predicting the relation between two expressions. IMO-level medalists wrote and reviewed the 200 test problems. Five model judges check the final answer and look for toy-case reasoning, logical gaps and numerical shortcuts, and a solution counts only if it passes all five. Stanford University and collaborators Written proofs Competition Not in cards GPT-5 47% Jun 2025 Aug 2025
LiveSecBench Models answer Chinese-language prompts that probe safety across areas such as legality, ethics, factuality and privacy. The paper's first edition turned a judge's pairwise comparisons into Elo ratings. The leaderboard's March 2026 edition gives each model an overall score from 0 to 100. The question bank is updated in dated editions. Intoken (隐拓智安) and academic coauthors Refusing harmful requests Everyday Not in cards Claude Haiku 4.5 91.4 Nov 2025 Feb 2026
Okapi Bench: API Freshness Models write integrations for changing APIs. The grader checks whether their calls still work with the current interfaces. Okapi Function-level problems University Not in cards GPT-6 Astra 66.7% Nov 2026 Oct 2026
OrangeSum Models summarize French news articles from Orange Actu. The dataset pairs articles with summaries written by journalists. Moussa Kamal Eddine and colleagues Reading in many languages School Not in cards Mistral Small 3.1 38.5% Oct 2020 Mar 2026
PokéBench: Tool Assisted Benchmark Models play Pokémon Red with a screenshot, map, game text, and action shortcuts. They have 1,000 turns to reach ten milestones ending with Brock. Cody Harris Video games Everyday Not in cards GPT-5.5 10.0 Sep 2026 Nov 2026
Spiral-Bench v1.2 Models chat for 20 turns with a simulated suggestible user. A judge ensemble of Claude Sonnet 4.5, GPT-5 and Kimi-K2 logs protective and risky behaviors with intensity ratings, which combine into a safety score. EQ-Bench (Sam Paech) Wellbeing and sycophancy Everyday Not in cards GPT-5 Chat 70.8 Oct 2025 Dec 2025
Travel Agent Compassion Agents book travel activities when the customer does not mention animal welfare. The benchmark measures how often they choose a welfare-neutral option. Compassion in Machine Learning Deception and alignment Everyday Not in cards Claude Opus 5 100% Jun 2026 Nov 2026
TRLawBench Models answer 297 questions on Turkish law in three stages: 97 questions from official ÖSYM legal exams and 200 harder questions across 14 branches of law. Answers are checked against the answer key. A judge model also grades the reasoning from 0 to 10. Beta Space Studio (Said Sürücü), with attorney Esat Erbil Tavus Law University Not in cards Gemini 3.1 Pro 9.85 Apr 2025 Aug 2026
ButtonBench Models resist repeated prompts that tempt them to press a forbidden button. The benchmark scores refusal and the creativity of the resistance. Adam Holter Deception and alignment Everyday Not in cards o3-pro 97.6 Dec 2025 Dec 2025
CMIMC 2025 Models answer 40 final-answer problems from the 2025 Carnegie Mellon Informatics and Mathematics Competition. MathArena scores the average accuracy over four attempts. Carnegie Mellon University (contest), MathArena (evaluation) Competition problems with answers Competition Not in cards DeepSeek-V3.2-Speciale 95.6% May 2025 Feb 2026
Hebrew LLM Chat Leaderboard Models summarize documents, translate into Hebrew, answer trivia, resolve pronouns, and restore vowel marks. The suite reports each task separately. Dicta and the Israeli National Program for NLP Chat and instructions in many languages School Not in cards Gemini 3 Pro 74.7% Nov 2025 Aug 2026
KorMedMCQA-Mixed Models take combined text and image exams. The 2022 set contains 281 questions and the 2023 set contains 307. Each year has a separate score. Byungjin Choi, Seongsu Bae, Sunjun Kweon, and Edward Choi Medicine Expert Not in cards Gemini 3 Pro 97.3% Feb 2026 Dec 2025
KorMedMCQA-V Models answer Korean medical licensing questions using clinical images. The full test contains 1,534 questions and 2,043 images from 2012 through 2023. Byungjin Choi, Seongsu Bae, Sunjun Kweon, and Edward Choi Medicine Expert Not in cards Gemini 3 Pro 96.9% Feb 2026 Dec 2025
OpenHands Index Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported. All Hands AI (OpenHands) Fixing real repositories Expert Not in cards Claude Fable 5 81% Jan 2026 Jun 2026
Uncheatable Eval Models predict text from recently published documents. The evaluation measures perplexity across sources including news, research papers, fiction and code. Kaifeng Tan and colleagues Reading comprehension University Not in cards Gemma 4 31B IT 6.29 May 2024 Apr 2026
BabaIsBench Agents play levels of the puzzle game Baba Is You through a text interface and win by rewriting the rules. Models are compared on pass rate, speed and cost. Quesma Puzzles and chess Expert Not in cards GPT-5.5 100% Jun 2026 Sep 2026
Buyout Game Eight language models play a money-driven elimination game with private transfers and a buyout endgame. Lech Mazur Multi-agent games Everyday Not in cards GPT-5.5 1975.4 Jan 2026 May 2026
DualChem Models answer chemistry questions written by experts. The researchers report answer accuracy and assess whether models recognize possible misuse. Scale AI with input from NIST CAISSI Dangerous capabilities University Not in cards GPT-5.6 Sol 67.3% Oct 2026 Oct 2026
LLMzSzŁ Models answer Polish academic and professional exam questions drawn from the Central Examination Board archives. Krzysztof Jassem and colleagues Knowledge in many languages University Not in cards Qwen2.5-72B 69.1% Jan 2025 Apr 2025
Open CoT Leaderboard Models answer logical reasoning questions with and without writing out their reasoning. Logikon and Allen Institute for AI Language reasoning University Not in cards DeepSeek-R1-Distill-Qwen-14B 17.6% Jan 2024 Jan 2025
SLR-Bench Models write Prolog rules that classify trains by their composition. The verifier runs each rule to determine whether its predictions are correct. Lukas Helff and colleagues Abstract puzzles and logic University Not in cards GPT-5 81.8% Jun 2025 Apr 2026
ToolComp Models chain several tool calls to answer 485 human-verified prompts. Most final answers are numbers. ToolComp-Enterprise gives 11 tools and ToolComp-Chat targets general chat tool use. Step labels also support process reward model tests. Scale AI Function calling Expert Not in cards o1 70.1% Jan 2025 Mar 2025
TxBench-Preclinical-Pharmacology Agents analyze preclinical pharmacology data and make development decisions. A grader checks each decision. LatchBio Genomics and life sciences Expert Not in cards GPT-6.2 Astra 68% Jun 2026 Nov 2026
WordleArenas Models guess a five-letter word from colored clues. Each arena changes how many clues give false feedback after a guess, from zero to five. Chang Liu Puzzles and chess Everyday Not in cards Claude Opus 4 100% Feb 2026 Apr 2026
Adversarial Humanities Benchmark Models respond to harmful requests rewritten in literary and scholarly styles. A judge measures how often the rewrites elicit unsafe answers. Icaro Lab and Sapienza University of Rome Jailbreaks and prompt injection Everyday Not in cards Claude Sonnet 4.6 9.20% Apr 2026 Apr 2026
ArxivRollBench Models reconstruct passages from recently published scientific papers. They order fragments, fill gaps and choose what comes next. Zi Liang and colleagues Reading comprehension University Not in cards Kimi K2.6 70.8% Jul 2025 May 2026
DeepsecBench Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings. Vercel Finding and exploiting real bugs Expert Not in cards GPT-6.2 Astra 61.2 Jul 2026 Nov 2026
HLE Extended Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores. Mercor Expert-written questions Expert Not in cards GPT-5.6 Sol 77.3% Aug 2026 Oct 2026
INVALSI Italian LLM Benchmark Models answer Italian reading and grammar questions from 11 school tests. For questions with several parts, the model must answer every part correctly. CRISP, University of Milano-Bicocca Reading comprehension School Not in cards Claude 3.5 Sonnet 92.2% Jun 2024 Mar 2025
LEXam Models answer 7,537 questions from 340 law exams at Swiss, EU and international law courses, in English and German. Open answers are judged against reference answers, and multiple-choice items are exact match. University of Zurich and collaborators Law Expert Not in cards GPT-5 70.2% May 2025 Dec 2025
LLM Position Bias Benchmark Models judge pairs of responses twice, with the presentation order reversed. Lech Mazur Bias, fairness and politics Everyday Not in cards MiMo-V2-Pro 19.8% Jun 2026
Decodability Assessment Benchmark Models flag words a child cannot yet sound out in a passage, given the phonics lessons the child has completed. The leaderboard reports adjusted balanced accuracy. Fab AI Language reasoning University Not in cards GPT-5.5 95.9% Sep 2026 Jul 2026
GSO-Bench Agents speed up code in 102 tasks across 10 real codebases, given a performance test. Scored by the share matching the expert commit's speedup. GSO team (UC Berkeley et al.) Speed and optimization Expert Not in cards Claude Fable 5.1 88.2% May 2025 Sep 2026
OncologyBench Separate agents run analyses of cancer research data, and a final agent makes a decision on detection, diagnosis or treatment. Strict scoring needs full marks on both parts. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 5.5 50.7% Oct 2026 Oct 2026
TxBench-Antibody-Discovery Agents make antibody discovery and development decisions from real experimental data. A grader checks each decision. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 5.5 67.7% Aug 2026 Nov 2026
TxBench-Oligo-Discovery Agents make oligonucleotide discovery decisions, such as ASO and siRNA design, from real data. A grader checks each decision. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 5.5 62.2% Sep 2026 Nov 2026
U-MATH Models solve 1,100 unpublished open-ended university math problems from teaching materials, balanced across six subjects. About 20% include images, and an LLM judge checks each solution. Toloka AI and collaborators Word problems University Not in cards o1 86.8% Dec 2024 Jan 2025
DualEntry Accounting AI Benchmark Models use accounting tools to look up accounts, reconcile balances, and draft entries for a test company. DualEntry checks their answers against fixed references. DualEntry Finance University Not in cards Gemini 4 Argon 91.1% Oct 2026 Nov 2026
ElephantBench Models answer 1,094 closed-book questions about facts with multiple verified but conflicting accounts. A model judge assigns complete, partial, or failed recall. Tencent Youtu Lab, Tsinghua University, and University of Warwick Short factual questions Expert Not in cards Kimi K3 52.4% Aug 2026 Jul 2026
HEPToolBench 1.2 Models generate inputs for particle physics simulations and inspect the resulting files. The main suite has 28 tasks and a separate extension adds three debugging tasks. Indian Institute of Science Lab and data work Expert Not in cards Gemini 3.1 Flash-Lite 71.4% Aug 2026 Jun 2026
KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. Xueting Fang and colleagues Medicine Expert Not in cards Claude Opus 5 29.4% Sep 2026 Sep 2026
LegalRewardBench Models choose which of two legal responses is better. The benchmark compares their choices with human-verified preferences across legal knowledge, analysis, decisions, and writing. Harbin Institute of Technology and The University of Osaka Law Expert Not in cards Claude Opus 4.7 86.8% Oct 2026 Apr 2026
MUSE Models answer questions about artworks used in language education. The tasks cover visual details, emotions, cultural context, and relationships within the image. Luyao Zhu and colleagues Other professions School Not in cards GPT-5.6 Sol 56.7% Sep 2026 Jul 2026
OpenGameEval Expanded Eval Set 87-eval set Agents extend existing Roblox games and build features from empty scenes. Roblox added 40 coding tasks to the original 47. Roblox Building whole projects University Not in cards Claude Fable 5 50.3% Apr 2026 Jun 2026
OpenVLM Video Leaderboard Models complete video understanding tasks. OpenCompass compares models across video evaluation datasets. OpenCompass Composite indices University Not in cards SmolVLM2-500M-Video-Instruct 35.4 Sep 2024 May 2025
ReactBench Coding agents do 39 realistic React tasks in two tracks. Write React tasks come from merged pull requests and are checked by hidden behavior tests. Fix React tasks ask the agent to remove React performance, accessibility and quality issues. A task passes only if the tests pass and a pinned React Doctor scan finds no new graded issue. Million (millionco) Web and app front ends Expert Not in cards Claude Opus 5.5 54.4% Jul 2026 Oct 2026
VetEval Models answer examination questions written by licensed veterinarians. A fixed answer key measures clinical knowledge, and safety penalties reduce the overall index. viggoVet Medicine University Not in cards Claude Opus 5.5 76.8 Nov 2026
xbench-ScienceQA Models answer postgraduate-level science questions written by experts, in Chinese, as part of the xbench AGI Tracking series. xbench (HongShan) Science questions Expert Not in cards Claude Opus 4.8 76.4% May 2025 Sep 2026
CircuitRubric Models write analog circuit netlists from design requests. A program compares transistor connections and relative device sizes with accepted reference circuits. Manar El-Chammas Electronics University Not in cards Claude Opus 4.8 96% Jun 2026 Jun 2026
DeepFinance Agents do private equity and investment banking work on 21 anonymized real deals: valuation, due diligence, underwriting and financial modeling. Each task comes with a data room of about 19 files and would take a professional about 27 hours. Rubrics of about 84 criteria grade the output, the workflow and the judgment behind each assumption. Julius Research Finance Expert Not in cards Claude Fable 5.2 62% Nov 2026 Nov 2026
Evalatro v2 Models play Balatro from the game state and try to clear Ante 12. Illegal moves reduce the score for progress through the game. alesha-pro Video games Everyday Not in cards MiMo-V2.5-Pro 46.1 Jun 2026 Aug 2026
HalachaBench Models answer 60 Hebrew rabbinical qualification exam questions. A model judge awards rubric credit on a scale from 0 to 100. HalachaBench Other professions Expert Not in cards Claude Fable 5.2 85.8 Nov 2026 Nov 2026
HELM Arabic Stanford CRFM runs models on 7 Arabic benchmarks in the HELM harness and ranks them by mean score. The tasks are AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and human-translated MMLU. Stanford CRFM with Arabic.AI Knowledge in many languages School Not in cards Claude Opus 4.7 84.1 Dec 2025 Apr 2026
K-Bench Models respond across multi-turn mental-health risk conversations involving suicide, self-harm, domestic violence, substance misuse, and no-risk cases. Kivira Health and University of Roehampton Wellbeing and sycophancy Expert Not in cards GPT-5.5 99% Sep 2026 Jul 2026
METR Capability Index METR fits an item response model to agent results on Time Horizon 1.1, TH2 and MirrorCode. The index places models on a common ability scale. METR Task length Research frontier Not in cards GPT-6 Astra 859.0 Nov 2026 Sep 2026
SORRY-Bench Models get unsafe requests across a fine-grained taxonomy of topics and linguistic variations. A judge model scores whether each answer refuses or complies. Princeton University and collaborators Refusing harmful requests Everyday Not in cards Claude 2.1 5.23% Jun 2024 Sep 2024
Vals RSI Index Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness. Vals AI AI research tasks Research frontier Not in cards Claude Fable 5.2 0.38 Aug 2026 Nov 2026
VERA-MH Suicide Risk Chatbots talk with simulated patients who show different levels of suicide risk. A judge grades the full conversation against a clinical rubric. Spring Health and the VERA-MH Expert Council Wellbeing and sycophancy Expert Not in cards GPT-5.6 Terra 75.0 Oct 2025 Aug 2026
AnalogCoder Models write Python programs that build analog circuits. A simulator checks whether each circuit performs its requested function. The authors report first-attempt success and how many tasks each model solves. Yao Lai and colleagues Electronics University Not in cards Llama 3.1-405B 83.3% May 2024 Mar 2025
Contract Workflows (Models) Systems complete legal work involving contracts. The benchmark assesses whether each deliverable satisfies its legal and presentation criteria. Legal Benchmarks Law Expert Not in cards Claude Fable 5.1 48.4% Nov 2026
Data Extraction (Models) Systems extract requested information from legal documents. The benchmark checks whether the resulting work satisfies the task criteria. Legal Benchmarks Law Expert Not in cards Claude Opus 5.5 76.7% Nov 2026
DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. Scale AI Genomics and life sciences Expert Not in cards Gemini 4 Carbon 71.5% Jun 2026 Nov 2026
EpiBench Agents analyze real epigenomics datasets in a sandbox with standard tools. A deterministic grader checks the biological result. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 5.5 39.6% Jun 2026 Oct 2026
Evalita-LLM Models perform tasks written originally in Italian. The suite includes admission exams, question answering and text generation. Bernardo Magnini and colleagues Reading in many languages University Not in cards Mistral Large 2 62.3 Feb 2025 Sep 2025
Financial Coaching Bench Models interview a simulated client about a household financial problem and propose a plan. Vals AI Finance Expert Not in cards Claude Opus 5.5 76.2% Nov 2026 Oct 2026
HealthBench-Psych-Hard Models answer mental health questions from conversations that also appear in HealthBench-Hard. Three model judges use the original physician rubrics. MindBench.ai Medicine Expert Not in cards Claude Sonnet 5.5 0.45 Aug 2026 Sep 2026
InteractScience Models generate code for interactive scientific demonstrations. Evaluators test the program and assess its visual output. OpenCompass and collaborators Scientific code University Not in cards Claude Sonnet 4 41.5 Oct 2025 Sep 2025
NoveltyBench Models give several answers to the same open-ended request. The benchmark measures how many distinct responses they produce and how useful those responses are. Yiming Zhang and colleagues Creative writing Everyday Not in cards Claude Fable 5.1 5.32 Apr 2025 Sep 2026
RCM-Bench-1 Agents handle healthcare revenue cycle work, such as claim submission, billing and eligibility checks, with tools. Checks cover tool calls, arguments, replies and final system state. Commure Medicine Expert Not in cards Claude Fable 5.1 94.2% Nov 2026
React Native Evals Models implement React Native features from task descriptions and fixed project contexts. A model judge checks the submitted files against explicit requirements, and evaluation aggregates requirement scores across tasks. Callstack Web and app front ends University Not in cards Claude Fable 5.1 89% Mar 2026 Sep 2026
RSNA Knee AUC10 Models label twelve knee findings from images, radiology reports, or both. The grader measures how well their predictions distinguish findings on ten reference studies. Kirderf Medicine Expert Not in cards GPT-6.1 Sol 0.92 Oct 2026 Sep 2026
scBench-Long Agents run long single-cell RNA-seq analyses that need sustained empirical work and biological reasoning. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 5.5 36.9% Jun 2026 Oct 2026
SecIT Bench Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions. Cribl AI Research Lab Finding and exploiting real bugs Expert Not in cards DeepSeek V4.1 Flash 82.9% Aug 2026 Oct 2026
Snorkel Agentic Coding 2.0 Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior. Snorkel AI Building whole projects Research frontier Not in cards GPT-6 Astra 47.6% Oct 2026 Oct 2026
SnorkelSpatial Models track particles on a 2D grid through 10 to 200 text-described moves, then report final positions and orientations. The 330 samples test allocentric and egocentric spatial reasoning. Snorkel AI Language reasoning School Not in cards GPT-5.4 99% Oct 2025 Mar 2026
SpatialBench-Long Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 5.5 40.5% May 2026 Oct 2026
VariantBench Agents discover and interpret genetic variants from real sequencing analyses. A deterministic grader checks the result. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 5 52.8% Jul 2026 Oct 2026
VeriGen design problems Models complete Verilog modules for 17 digital circuits. Simulation checks whether the generated code satisfies each design’s testbench. Shailja Thakur and colleagues Electronics University Not in cards Gemini 2.5 Flash 77.7% Dec 2022 Jun 2026
Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. Arena (formerly LMArena) Deception and alignment Everyday Not in cards GPT-6.1 Sol 87.9 Oct 2026 Sep 2026
BertaQA Models answer questions about Basque culture and general knowledge. Parallel Basque and English questions help distinguish language effects from familiarity with local topics. Julen Etxaniz and colleagues Trivia and open questions Everyday Not in cards Claude 3 Opus 81.5% Jun 2024 Jun 2024
DAYJOB: Accounting Agents do accounting assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Surge AI Finance Expert Not in cards Gemini 4 Neon 49.6% Oct 2026 Nov 2026
EvoCode-Bench v2 Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit. UniPat AI with Peking University, Tsinghua University and HKU Building whole projects Expert Not in cards Claude Opus 4.8 59.1% Jun 2026 Jun 2026
GSM8K-V Vision-language models solve GSM8K word problems that are redrawn as multi-image visual scenes, so the facts must be read from pictures. Zhejiang University Visual math and diagrams School Not in cards Gemini 3 Pro 59.1% Sep 2025 Apr 2026
InferenceBench Coding agents optimize a language model inference server within a fixed time budget. The harness measures serving speed and checks answer quality against a reference. ELLIS Institute Tübingen and collaborators Machine learning engineering Expert Not in cards Claude Opus 5.5 12.1× Nov 2026
K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. Chong Li and colleagues Other professions School Not in cards Qwen2.5-VL-32B 55.4% Jun 2025 Mar 2025
KradleBench Models play live multi-agent Minecraft games and act by writing code. The overall score weights collaboration, economy and conflict at 30% each, and logic and creativity at 5% each. Kradle Multi-agent games Everyday Not in cards GPT-6.2 Astra 78.3 Nov 2026 Nov 2026
MetagenomicsBench Agents analyze microbial communities from shotgun, amplicon, long-read and spatial metagenomics data. A grader checks the biological result. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 5.5 62.7% Sep 2026 Oct 2026
OpenLMM Subjective Leaderboard Models generate open-ended answers to multimodal tasks. The board reports subjective evaluation results. OpenCompass Composite indices University Not in cards GPT-4.1 76.6 Apr 2025 Apr 2025
prinzbench Models research fact patterns under U.S. statutes and regulations and hunt for obscure public information online. 33 private questions run three times give a score out of 99. prinz-ai Law Expert Not in cards GPT-5.6 Sol Pro 91.0 Jan 2026 Jul 2026
SEAL Coding Models answer 1,000 private coding prompts across common languages and tasks. Human annotators rate adherence, correctness, efficiency and readability, then compare responses side by side to produce Elo-style scores. Scale AI Function-level problems University Not in cards o1-mini 1,237 May 2024 Jan 2025
Speko LLM Benchmark for Voice Agents v3 Models use simulated support tools to look up orders and issue refunds. The task score gives partial credit when a model stops early without causing harm. Speko Customer service Everyday Not in cards Gemini 3.5 Flash-Lite 94.7% Nov 2026
The Aggregate Trip Outcome Prediction Models read a packing list and itinerary, then guess what happened on a weekend trip. A judge scores four answers, and relative error scores the taxi fare. Mikhail Doroshenko (The Aggregate) Commonsense Everyday Not in cards GPT-5.6 Sol 69.8% Nov 2026 Nov 2026
3LM Models answer Arabic STEM questions taken from Arabic textbooks and worksheets, plus synthetic STEM questions from the same sources. A third part tests Arabic code generation. Technology Innovation Institute (Boussaha et al.) Reasoning in many languages School Not in cards Llama 3.3 70B Instruct 85.7% Jul 2025 Apr 2025
CADClamp 0.2 Models generate mechanical parts in seven CAD languages. Each part must meet its specification before the grader compares its printability with a reference part. Christian Hill CAD models University Not in cards Claude Opus 5.5 0.93 Sep 2026 Sep 2026
CVDP Models and agents solve hardware design problems in Verilog, such as spec to RTL, code completion, debugging and testbench writing. Simulation tests check the outputs. NVIDIA Electronics Expert Not in cards Claude Opus 5.5 64.1% Jun 2025 Oct 2026
DaLA Models distinguish acceptable Danish sentences from copies corrupted to resemble errors in real text. Gianluca Barmina and colleagues Reasoning in many languages Everyday Not in cards Gemma 3 27B PT 61.7% Dec 2025 Feb 2026
Deep20Bench 1.1 Models identify hidden people, characters, or objects by asking questions. The score averages the number of counted questions, with a fixed penalty for failed rounds, over three rounds for each of ten subjects. Patrick Heusser and Markus Tuor Puzzles and chess Everyday Not in cards GPT-6 Astra 13.7 Sep 2026 Nov 2026
EWoK Models decide which context makes a target statement plausible. Paired examples isolate concepts about the physical and social world. Anna A. Ivanova and colleagues Commonsense Everyday Not in cards GPT-4 Turbo 91.2% May 2024 Oct 2024
FailureSensorIQ Models answer multiple-choice questions on how industrial sensors relate to failure modes of assets. The Kaggle board shows it under IBM's AssetOpsBench citation. IBM Research Other professions Expert Not in cards GPT-6 Astra 69.5% Jun 2025 Sep 2026
IFEvalCode Models write code in eight programming languages under explicit constraints. The evaluator checks functional correctness and adherence to instructions separately. Jian Yang and colleagues Function-level problems University Not in cards Claude 3.7 Sonnet 38.2% Jul 2025 Apr 2025
Immigration Bench Models answer expert-written US immigration law questions from seven personas, such as F-1 students, employers and DACA recipients. An AI grader checks each answer against an expert rubric. Vals AI + Talent Mobility Fund (Renaissance Philanthropy) Law Expert Not in cards Claude Fable 5.1 75.4% Oct 2026 Nov 2026
METR Time Horizon suite Software tasks with human completion times. Reports the task length at which a model succeeds 50% of the time. METR Task length Research frontier Not in cards Claude Opus 4.5 4.1 h Mar 2025 Nov 2025
MultiBLiMP Models compare minimally different sentences across languages. The benchmark tests whether the model assigns higher likelihood to the grammatical sentence. Jaap Jumelet and colleagues Reasoning in many languages Everyday Not in cards Gemma 3 27B PT 90.2% Apr 2025 Apr 2025
Real-SWE 1.1 Resolution rate on private enterprise coding tasks, averaged across five runs per task. Specific Labs Fixing real repositories Expert Not in cards Claude Opus 5.5 56.8% Oct 2026 Oct 2026
SnorkelSequences Models compute values over integer sequences built from chains of filters, transforms and operators. The 250 procedurally generated questions test compositional multi-step math. Snorkel AI Word problems School Not in cards GPT-5 77.6% Jun 2025 Mar 2026
Surface Evolver Bench Agents write Surface Evolver datafiles that model liquid surfaces under physical constraints across 16 tasks. Deterministic checks give partial credit for matching hidden reference quantities. Yann Henon Research science Expert Not in cards Claude Fable 5 95% Jun 2026 Sep 2026
Swedish Benchmarks V1.1 Models answer Swedish language, document, and knowledge questions. The overall score combines normalized task performance and response time. Drasko Markovic Reading in many languages School Not in cards Gemma 4 31B IT 85.5 Apr 2026 Aug 2026
VisEditBench (TikZ diagram editing) Models edit TikZ code to follow instructions drawn on a diagram. A judge scores adherence, preservation of other content, and readability. Tohoku University and collaborators Spatial reasoning and counting University Not in cards Gemini 3.1 Pro 83.6% Sep 2026 Apr 2026
Argo-Bench Agents query a simulated food delivery company's warehouse and act on their findings. The grader measures the business consequences of their decisions across 210 tasks. TextQL Simulated companies Expert Not in cards Claude Fable 5.2 70.2 Oct 2026 Nov 2026
Argus Inspection Models inspect images for small details that change the sensible response to a question. A judge scores whether the response identifies the relevant detail and chooses an appropriate action. The score combines those judgments and assigns zero when a separate description fails to recognize the detail. Shanghai Artificial Intelligence Laboratory and university collaborators General visual questions Everyday Not in cards Gemini 2.5 Flash 0.44 Jun 2025 May 2025
AstroMLab 1 Models answer 4,425 multiple choice questions drawn from astronomy review articles. The authors publish the questions and grade submissions with a private answer key. AstroMLab Science questions Expert Not in cards Claude 3 Opus 82.7% Jul 2024 Sep 2024
BAZAAR Agents buy and sell goods in a simulated competitive market. Lech Mazur Multi-agent games University Not in cards o3 7 Sep 2025
BizFinBench v2 Models answer expert financial questions grounded in business data. The score is average accuracy on eight offline tasks. HiThink Research Finance University Not in cards GPT-5 61.5% Jan 2026 Dec 2025
Brokk Power Ranking Models solve 181 real tickets from a 1.1 million line Rust codebase. Model judges compare solutions in pairs, and a Bradley-Terry fit gives each model a strength score. Brokk (SlopCop) Fixing real repositories Expert Not in cards Gemini 4 Neon 35.4 Aug 2025 Nov 2026
EusExams Models answer past public service examination questions in Basque. The questions come from several Basque institutions. Julen Etxaniz and colleagues Knowledge in many languages University Not in cards Qwen3-8B-Base 51.3% Mar 2024 Apr 2025
EusReading Models answer comprehension questions about Basque passages from past C1 proficiency examinations. The scorer checks their choices against the exam key. Julen Etxaniz and colleagues Reading in many languages University Not in cards Gemma 2 9B 62.2% Mar 2024 Apr 2025
EusTrivia Models answer trivia questions in Basque. The set combines elementary-school questions with harder general knowledge questions. Julen Etxaniz and colleagues Knowledge in many languages School Not in cards Gemma 2 9B 51% Mar 2024 Apr 2025
GBAEval Agents get 24 hours to write a Game Boy Advance emulator in Rust for WebAssembly. Replay, hardware, and audio tests compare it with Mesen2. Mechanize Building whole projects Research frontier Not in cards Claude Opus 5.5 90.6% May 2026 Nov 2026
GPBam Models write legal opinions on German public law cases and retrieve the statutes they need. Three judges grade essays against reference solutions. Abdullah Al Zubaer and colleagues Law Expert Not in cards Claude Opus 5 76.4 Nov 2026 Aug 2026
GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. OpenCompass (Shanghai AI Laboratory) and collaborators General assistants Expert Not in cards GPT-5 11.4% Apr 2026 Dec 2025
MemLens Vision-language models and memory agents answer 789 questions over multimodal multi-session conversations, covering extraction, temporal reasoning, knowledge updates and refusal at four context lengths. NVIDIA and collaborators Finding facts in long inputs Everyday Not in cards Kimi K2.5 53.6% May 2026 Mar 2026
Open Tabletop GM Narrative Quality Probe v2 Models narrate twelve tabletop scenes, including player failure, deception, and changes in tone. Five judges score each scene for atmosphere, character portrayal, and game management. Bobby Gray Creative writing Everyday Not in cards Qwen3-Next-80B-A3B-Instruct 4.88 Apr 2026 Apr 2026
Paraphrase Stability Benchmark Models respond across a grid of paraphrases and contexts. The harness reports its Stability Agreement Average with a confidence interval. Chauhan Balaji Taste and style School Not in cards GLM-5 1.00 Oct 2026 Sep 2026
SnorkelGraph Models solve 200 procedurally generated graph problems, such as minimum vertex cover, given as node and edge lists in text. Answers are checked against verifiable ground truth. Snorkel AI Word problems University Not in cards GPT-5.4 84.5% Sep 2025 Mar 2026
ThaiExam Models answer multiple-choice questions from standardized examinations in Thailand, such as ONET, IC, TGAT, TPAT-1 and A-Level. SCB 10X + Stanford CRFM Knowledge in many languages School Not in cards Claude 3.5 Sonnet 75.1% Sep 2024 Aug 2024
The Aggregate Puzzle Portfolio Ladder Models solve 30 generated puzzles across five families. Each family has six questions with answers fixed by the generator. Mikhail Doroshenko (The Aggregate) Puzzles and chess University Not in cards GPT-5.2 96.7% Nov 2026
WTP-Bench Models identify Pokémon from official artwork and silhouettes across 1,160 entries. The scorer checks exact forms, species, and evolutionary families. Aaditya Baranwal, University of Central Florida General visual questions School Not in cards GPT-4.1 79% Feb 2026 Oct 2025
AGI-Eval Community Base Language Models Base models answer Chinese and English questions across seven skill areas. The board reports objective accuracy. AGI-Eval (Meituan) Reasoning in many languages School Not in cards Kimi K2 0.71 Feb 2025 Sep 2025
ARCS Models identify ambiguities in a database question and write SQL for the intended interpretation. The scorer executes the query against the database. Megagon Labs and collaborators Data science and SQL University Not in cards GPT-5 57.9% Oct 2026 Sep 2026
CHI-Bench Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs. Actava AI Medicine Expert Not in cards Claude Opus 5 54.7% May 2026 Jul 2026
DELTA Models research Dutch legal questions and apply the law to facts. Lawyers judge accuracy and professional usefulness. Legal Benchmarks and partner law firms Law Expert Not in cards Claude Fable 5.1 15.0 Sep 2026 Sep 2026
FACTS Multimodal Models answer questions about input images and must combine what they see with world knowledge. A judge checks coverage and contradictions. Google DeepMind + Google Research + Kaggle Short factual questions University Not in cards Gemini 4 Carbon 60.2% Dec 2025 Oct 2026
Flash Fiction Stylistic Diversity Models write flash fiction from shared prompts. A reviewing model describes each story, and weighted Gower distance measures stylistic diversity within a writer’s work. Lech Mazur Taste and style School Not in cards GPT-5 0.22 Sep 2025 Dec 2025
General365 Models solve 365 hand-written reasoning problems and 1,095 variants in eight categories that need only K-12 knowledge. Scoring mixes rule-based and model-based checks. Meituan LongCat Team Abstract puzzles and logic Everyday Not in cards Gemini 3 Pro 62.8% Apr 2026 Feb 2026
GeoGuesser Bench Agents explore offline Street View panoramas to report coordinates, name places, and find objects. Coordinate guesses earn distance points. Other answers require an exact match. University of California, Berkeley Spatial reasoning and counting Everyday Not in cards Qwen3.5-35B-A3B 22.8 Nov 2026 Jun 2026
HELM Arabic Enterprise Models do Arabic enterprise tasks in six parts. They write corporate articles, answer finance questions in three formats, and answer legal questions written by Arabic legal experts with and without the statute. Stanford CRFM with Arabic.AI Knowledge in many languages Expert Not in cards Claude Opus 4.7 77.9 May 2026 Apr 2026
IDP Core Bench Models extract fields and tables, transcribe text, and answer questions about documents. The test covers about 2,000 invoices, receipts, forms, and handwritten documents. Nanonets and IIT Indore Documents, charts and text in images Everyday Not in cards Gemini 3.1 Pro 89.6% Mar 2026
lmgame-Bench Models play platformer, puzzle and narrative games such as Sokoban, Tetris, 2048 and Candy Crush through a Gym-style API with perception and memory scaffolds. Hu et al. (lmgame-org) Video games Everyday Not in cards o3 87.7 May 2025 Aug 2025
MOLTRON LLM Showdown v3.0 Models write JavaScript bots for a light-cycle game. Each working bot plays 600 matches against five reference opponents across six maps and both starting positions. MOLTRON Video games University Not in cards GPT-6 Astra 69% Sep 2026 Sep 2026
ResearchClawBench Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs. Intern Science (Shanghai AI Laboratory) Research science Research frontier Not in cards DeepSeek-V4-Pro 38.6 May 2026 Aug 2026
SecRespond Agents investigate compromised-host snapshots and write incident analyses and remediation plans. A model evaluator applies expert checklists separately to detection and response planning. Alibaba NLP Finding and exploiting real bugs Expert Not in cards Claude Opus 4.7 79% Jul 2026 Jun 2026
Senior SWE-Bench Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate. Snorkel AI Fixing real repositories Expert Not in cards Claude Opus 5.5 39.3% Jun 2026 Sep 2026
SoSBench Models answer hazardous requests drawn from real regulations in six scientific fields. Scored by how often the model complies with a request it should refuse. SoSBench authors (Jiang et al.) Hazardous knowledge Expert Not in cards Claude Sonnet 4 10.6% May 2025 Aug 2025
South Asian Music Understanding Models answer questions about Hindustani music theory and Bengali musical traditions. They also choose a continuation for a melody written in ABC notation. University of Central Florida and Auburn University Expert-written questions University Not in cards Qwen2.5-32B 48.7% Jun 2026 Mar 2025
Terminal-Bench Pro Agents solve 400 terminal tasks across eight domains. Harbor distributes the 200 public tasks. Alibaba keeps the other 200 for private evaluation. Alibaba Terminal tasks Expert Not in cards Claude Opus 4.5 51% Dec 2025 Feb 2026
TRUE (Text Recognition and Understanding) Models recognize text in images and answer questions about its content. BAAI FlagEval Documents, charts and text in images Everyday Not in cards Gemini 2.5 Pro 81.6 Mar 2025
TxBench-Antibody-Development Agents use evidence to make manufacturing, quality, and clinical development decisions about candidate therapeutic antibodies. LatchBio Research science Expert Not in cards Gemini 4 Carbon 61.1% Nov 2026
Verifying the Verifiers (Project APE CRED benchmark) Verifier LLMs must find planted errors (prose-table, paper-code, execution, data) in AI-generated economics papers. Credit needs a verbatim quote of the edit. Project APE, University of Zurich (Willner, Yanagizawa-Drott) Research science Expert Not in cards GPT-6 Astra 99.5% Jul 2026 Sep 2026
Access to Justice Benchmark Models interview a simulated client about a civil legal problem, explain their options, and direct them to local help. Vals AI Law Expert Not in cards Qwen 3.8 Max 76.4% Nov 2026 Oct 2026
AgriEval (Harbin Institute of Technology) Models answer Chinese agricultural exam questions across 29 subfields. The authors score 14,697 choice questions by accuracy and 2,167 open answers by ROUGE-L. Harbin Institute of Technology and Institute for Advanced Algorithms Research Exam questions University Not in cards Qwen Plus 63.2% May 2025 Apr 2025
BenchMAX Models follow instructions, write code, reason, use tools, translate, and retrieve information across 17 languages. Ten tasks use rule checks, model-judged wins, code tests, answer matching, or translation scores. Nanjing University, Shanghai Artificial Intelligence Laboratory, University of Zurich, and Carnegie Mellon University Reasoning in many languages University Not in cards Llama 3.3 70B 85.2 Feb 2025 Mar 2025
ColorBench Models answer image questions about color perception, color reasoning, and robustness to color changes. The evaluator checks the selected answers against a single correct option. University of Maryland General visual questions Everyday Not in cards Gemini 2.0 Flash 59.6% Apr 2025 Feb 2025
DatBench Models answer visual questions drawn from 33 datasets. The suite reports separate scores across nine visual capabilities. DatologyAI General visual questions University Not in cards Qwen3-VL-4B-Instruct 64% Jan 2026 Oct 2025
DAYJOB: Law Agents do legal assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Surge AI Law Expert Not in cards GPT-6.2 Astra 41.4% Oct 2026 Nov 2026
EvaOmni Models answer questions about videos and identify the audio and visual evidence behind their answers. The score accounts for both correctness and evidence timing. Xiaoxing Hu and colleagues Audio understanding and voice Everyday Not in cards Gemini 3.7 Flash 79.4% Oct 2026 Sep 2026
FinQA (Snorkel OpenEnv environment) Agent answers 290 multi-step questions on SEC 10-K data via MCP tools and constrained SQL. Binary reward with 1% numeric tolerance. Snorkel AI Finance University Not in cards o3 81% Mar 2026 Nov 2025
FrontierBreach Agents perform simulated enterprise tasks while processing untrusted material. The benchmark checks whether each agent resists instructions that conflict with its assigned task. PromptArmor Jailbreaks and prompt injection Expert Not in cards Gemini 4 Carbon 2.40% Nov 2026 Nov 2026
FrontierPhysics Agents do end-to-end physics research on 56 tasks that 42 physicists from 26 institutions built from their own work, across nine domains of theoretical, computational and experimental physics. Each task asks the agent to review the literature, plan, run the research and write a manuscript. Deterministic verifiers and expert rubrics grade whether the research is correct and publishable, over 3 trials. BenchFlow, with physicists from UC Berkeley, Caltech and Harvard Research science Expert Not in cards Claude Opus 5.5 27.4% Aug 2026 Oct 2026
Guesswork (The Aggregate) Models predict benchmark scores before publication. The maker ranks forecasts by mean absolute error after scaling each error by the variation in that benchmark’s scores. Mikhail Doroshenko (The Aggregate) Real-world events University Not in cards GPT-6 Astra 0.56 Jun 2026 Sep 2026
HWE-Bench v1.1 (RTL bug repair) Agents fix hardware bugs in 417 repository tasks. The revised tests check whether each fix restores the required behavior. Peking University Electronics Expert Not in cards Claude Opus 5.5 99% Nov 2026 Oct 2026
LLM-WikiRace Models play the Wikipedia game: starting from one article, they follow one hyperlink at a time to reach a target article. At each step the model sees only the links on the current page, and a game is lost after 30 steps. Three difficulty levels are set by the shortest path between the two pages. Scores are the share of games won at each level. University of Oxford, University College London and University of Basel Puzzles and chess Everyday Not in cards Gemini 3 Pro 23% Feb 2026 Feb 2026
MIRAGE-MMST Models answer growers’ questions using photographs and written descriptions. They identify the cause of a problem and recommend how to manage it. University of Illinois Urbana-Champaign and Amazon General visual questions University Not in cards GPT-4.1 68.0 Jun 2025 Apr 2025
NeuroscienceBench Agents analyze neuroscience research data and report a scientific conclusion. The score counts correct conclusions among graded runs. LatchBio Research science Expert Not in cards Claude Fable 5.2 41% Nov 2026
NoLiMa Models find a needle in a long haystack when the question and the needle share almost no words, so they must infer a latent link. Accuracy is compared across context lengths. Adobe Research and LMU Munich (Modarressi et al.) Finding facts in long inputs Everyday Not in cards GPT-4.1 89.1% Feb 2025 May 2025
o11y-bench Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment. Grafana Labs Terminal tasks Expert Not in cards Claude Opus 5.5 93.8% Apr 2026 Oct 2026
Open TTS Leaderboard 1.6.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Hugging Face Audio understanding and voice Everyday Not in cards Qwen3-TTS-12Hz-1.7B-Base 2.32% Nov 2026 Jan 2026
OpenDesign Arena Models create design artifacts in the OpenDesign tool, and the board reports an evaluation score out of 100 with cost and speed per artifact. OpenDesign Web and app front ends Everyday Not in cards Kimi K3.1 Pro 91.0 Nov 2026
PatternEval Models answer 2,415 image prompts in thinking and non-thinking modes. A model judge flags exposed draft reasoning, repetition, contradictions, and unsupported reasoning. The Trigger rate measures responses with any flagged failure. Tencent, Institute of Automation at the Chinese Academy of Sciences, and collaborators Overall chat quality Everyday Not in cards GPT-5.5 0.78% Aug 2026 Jun 2026
ResearchCodeBench Agents read machine learning papers and fill missing Python code in the accompanying implementations. Unit tests check each of 212 snippets from 20 papers. ResearchCodeBench team Machine learning engineering Expert Not in cards Gemini 2.5 Pro 64.2% May 2025 Sep 2025
RoboRewardBench Vision-language models act as reward models on real robot videos from Open X-Embodiment and RoboArena, judging task success and partial progress. Counterfactual relabeling adds calibrated failures and near misses. Stanford (CRFM HELM, Percy Liang) + collaborators Real robots Everyday Not in cards GPT-5.5 0.63 Jan 2026 Apr 2026
SlopCodeBench Agents repeatedly extend their own code across 36 problems and 196 checkpoints as the specification changes. Tests check correctness, and metrics track structural erosion and verbosity. University of Wisconsin-Madison (SprocketLab) Building whole projects Expert Not in cards GPT-5.6 Sol 32.1% Mar 2026 Sep 2026
Structured Output Benchmark Models turn a text context into JSON that must match a given schema and hold the right values. The 5,324 records come from multi-hop questions, OCR text of documents and meeting transcripts, all given to the model as text. Seven automatic metrics, such as value accuracy, faithfulness and JSON validity, combine into an overall score that weights hard schemas more. Interfaze (JigsawStack) Following precise instructions Everyday Not in cards GPT-5.4 87% Apr 2026 Jun 2026
SWE-Together Coding agents work through 109 repository tasks rebuilt from real multi-turn sessions between users and coding agents. A simulated user replays the original user's goals and gives feedback when the agent needs it. Scores count tasks solved and the number of corrections the user had to make. Meta Fixing real repositories Expert Not in cards Claude Fable 5.1 69.3% Jun 2026 Sep 2026
TissueMicroscopyBench Agents examine tissue images and answer questions about cells, staining, and changes over time. LatchBio Research science Expert Not in cards Claude Fable 5.2 34% Nov 2026
A.S.E 2.0 Models and agentic coding tools write code in real projects covering 29 CWE types. A mix of static analysis, tests and vulnerability proofs of concept checks security. Tencent Secure code and review Expert Not in cards Nov 2025
Accounting Autocoding Models select a bookkeeping category for each expense. Ramp checks the first choice against the assigned category. Ramp Labs Finance University Not in cards Gemini 3.1 Pro 92.6% Mar 2026 Mar 2026
BLiMP-NL Models compare Dutch sentence pairs that differ in grammatical acceptability. Suijkerbuijk and colleagues Reasoning in many languages Everyday Not in cards Gemma 3 27B 49.4% Jul 2025 Dec 2025
BrowseSafe++ v2 Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1. Perplexity Jailbreaks and prompt injection University Not in cards GPT-6 Astra 91.5% Oct 2026 Sep 2026
DataBench v1.1 Agents answer ambiguous business questions and make recommendations from a synthetic company warehouse. Model judges assess the reasoning, supporting evidence, artifacts, and final recommendation against task rubrics. Hex Running a business Expert Not in cards Claude Opus 5.5 70.5% Sep 2026 Oct 2026
Deep Research Bench (FutureSearch) Web research agents work on multi-step research tasks against RetroSearch, a frozen copy of 10,000 to 100,000 web pages per task. Answers are compared with answers worked out by people. FutureSearch Deep research Expert Not in cards Claude Opus 4.6 0.55 May 2025 May 2026
Dutch CoLA Models judge whether Dutch sentences drawn from descriptions of Dutch syntax are grammatically acceptable. Computational Linguistics Groningen Reasoning in many languages Everyday Not in cards Gemma 3 27B 49.4% Apr 2024 Dec 2025
EduBench-RU Models plan lessons, explain school subjects, and assist teachers in Russian and Chuvash. A panel of model judges scores responses against a teaching rubric. Daniel Ivanov and Alexander Antonov Other professions University Not in cards Gemini 3.1 Pro 3.61 Mar 2026 Mar 2026
EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. University of Illinois Urbana-Champaign + others Simulated robots Everyday Not in cards Claude 3.5 Sonnet (October 2024) 50.5% Feb 2025 Apr 2025
Finance Reasoning (Snorkel) Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness. Snorkel AI Finance Expert Not in cards Grok 4 53.1% Jun 2025 Mar 2026
GSM8K Platinum Initial 268 Models answer the original 268-question cleaned GSM8K subset. The live matrix reports each model configuration separately. MIT Madry Lab (Vendrow et al.) Word problems School Not in cards o1-preview 99.6% Feb 2025 Apr 2025
LLM-Perf Leaderboard Language models generate tokens on a specified hardware and software configuration. The board reports latency, throughput, memory and energy alongside quality scores. Hugging Face Composite indices University Not in cards Qwen1.5-110B 29.6 Jun 2024
MathBench Models answer 3,709 math questions in five stages from basic arithmetic to college, in Chinese and English. Each stage also tests theory knowledge, and Circular Evaluation reshuffles answer options to check consistency. OpenCompass (Shanghai AI Laboratory) Word problems University Not in cards GPT-4o 70.9% May 2024 Jun 2024
MMCircuitEval Models answer questions about circuit diagrams, specifications, and design decisions. The authors combine text similarity with a language model’s judgment. Chenchen Zhao and colleagues Electronics University Not in cards GPT-4V 69.4% Jul 2025 Mar 2026
Physical-Engineering's Last Exam Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective. Google Cloud AI Research CAD models Expert Not in cards Claude Opus 5.5 0.49 Oct 2026 Nov 2026
SATA-Bench Models select every correct option in 1,650 multiple-choice questions. Exact set match measures complete answers, with Jaccard overlap and option-count metrics as diagnostics. SATA-Bench authors Language reasoning University Not in cards GPT-4.1 75.2% Aug 2025
SkateBench v2 Models name skateboard tricks from technical trick definitions. The score is the share named correctly. Theo Browne (t3.gg) Trivia and open questions Everyday Not in cards Gemini 3.1 Pro 96.9% Mar 2026 Mar 2026
The Aggregate Long-tail Book Recall Models answer closed-book questions about 60 Project Gutenberg books. The scorer checks the answer text, and a second reviewer can accept equivalent wording. Mikhail Doroshenko (The Aggregate) Trivia and open questions School Not in cards Claude Fable 5.2 77% Nov 2026
VIALS 2 2.0.0 Models interpret 161 professional life-science visual artifacts. The score is judged accuracy averaged across three rollouts. Handshake AI Research Genomics and life sciences Expert Not in cards GPT-6 Astra 49.9% Oct 2026 Oct 2026
Vision2Web S3-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Z.ai (zai-org) Web and app front ends Expert Not in cards GPT-5.5 74.6% Jun 2026 Jun 2026
AEC-Bench (Nomic) Agents search construction drawings and specifications to answer questions and identify conflicts. Task verifiers compare their outputs with known findings and award partial credit. Nomic and coauthors Tasks from real jobs University Not in cards Claude Opus 5.5 70.2 Mar 2026 Oct 2026
AILuminate Safety 1.2 Models answer 3,900 hidden English prompts, 325 in each of 12 hazard categories, picked with item response theory. A calibrated evaluator gives a 0 to 100 score. MLCommons Refusing harmful requests Everyday Not in cards Tencent Hy4 preview 96.0 Sep 2026 Aug 2026
BIM-Edit Agents create, change, or delete elements in IFC building models. The evaluator checks the resulting geometry, attributes, and relationships. University of Rostock and Clausthal University of Technology CAD models University Not in cards Gemini 4 Argon 87.7% Jun 2026 Oct 2026
BinaryAudit Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software. Quesma Finding and exploiting real bugs Expert Not in cards Claude Opus 4.6 49.1% Dec 2025 Mar 2026
BuzzBench Models analyze humor, such as why a joke or comic piece works. The leaderboard reports a score and response length per model. EQ-Bench (Sam Paech) Taste and style Everyday Not in cards Gemini 2.5 Pro 71.1 Apr 2025
CEQuest Models answer text questions about construction drawings, trade conventions, and quantity estimates. The original test has 101 multiple-choice questions and 63 true or false questions. Florida International University and University of Florida Tasks from real jobs University Not in cards Gemini 3.7 Flash 95.5% Aug 2025 Oct 2026
CompileBench Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code. Quesma Terminal tasks Expert Not in cards Claude Opus 4.1 100% Aug 2025 Oct 2025
Construction AI Leaderboard Models answer construction questions, inspect drawings, edit building models, and generate validation rules. ONESTRUCTION averages the representative scores from five evaluations. ONESTRUCTION Composite indices University Not in cards Claude Opus 5.5 73.7 Nov 2026 Oct 2026
CulturalRiddles Models solve original riddles written by people from 62 cultures in 51 languages. Solving them needs cultural knowledge plus abductive and compositional reasoning. Cohere Labs Community Reasoning in many languages Everyday Not in cards Gemini 3.1 Pro 79.4% Oct 2026 Jul 2026
Dictatorship Eval Scenarios at six difficulty levels testing whether models help with authoritarian requests, from blunt asks to requests hidden in code edits. LLM judge, no self-grading. Free Systems (Andy Hall, Stanford Hoover) Refusing harmful requests Not in cards Claude Fable 5 91.3% Apr 2026 Jul 2026
IFC-Bench v2 Agents inspect IFC files and answer questions about building elements and their relationships. A model grades each answer against a reference. Technical University of Munich and coauthors CAD models University Not in cards Claude Opus 5 65.8% Feb 2026 Oct 2026
IFScale Models write business reports while including up to 500 specified keywords. The benchmark measures how many instructions they follow at each density. Distyl AI Following precise instructions University Not in cards Gemini 2.5 Pro 68.9% Jul 2025 Sep 2025
Implicit Intelligence Agents respond to requests whose unstated requirements affect safety, privacy, or accessibility. Labelbox reports the share of scenarios each model passes. Labelbox General assistants Everyday Not in cards Claude Opus 4.6 53.2% Apr 2026
Ishigaki-IDS-Bench Models convert building information requirements into IDS XML. An audit tool checks validity, and a scorer compares each requirement with the reference file. ONESTRUCTION CAD models University Not in cards Claude Opus 5.5 63% May 2026 Oct 2026
LLM Sycophancy Benchmark Models judge the same dispute from opposite narrators’ perspectives. Contradictory judgments count toward the sycophancy rate. Lech Mazur Deception and alignment Everyday Not in cards Grok 4.5 0% Jul 2026
MA-Bench Models do military mission analysis under the joint planning process on long-context scenarios, graded against expert rubrics. Scale AI (Lafuente-Mercado et al.) Other Expert Not in cards Muse Spark 1.3 57.4% Oct 2026 Sep 2026
MathCLUE Models solve problems from Chinese high school math competitions. SuperCLUE Competition problems with answers Competition Not in cards Gemini 2.5 Pro 92.5 Apr 2025
MecEng Models generate rigid and flexible multibody simulations from text. The evaluator checks system connections, geometry, and numerical outputs against engineering references. University of Innsbruck and University of Augsburg Scientific code University Not in cards Claude Opus 4.8 91.4% Aug 2026 May 2026
MobilePA-Bench Agents complete 1,705 mobile planning tasks using tools, persistent memory, reusable skills, and specialist agents. Task-specific checks evaluate tool calls, state changes, and interaction behavior. Additional gates check required memory access and skill loading. MAI Team, Alibaba Token Hub, Alibaba Group General assistants Everyday Not in cards GPT-6 Astra 76.9% Aug 2026 Oct 2026
Opus Magnum Bench Agents build machines on a hexagonal grid to make a target molecule. Each puzzle score compares the machine’s combined cost, cycles, and area with the human record. Rob Haisfield, Max Bittker, and Surya Narreddi Puzzles and chess Competition Not in cards GPT-5.6 Sol 25.5 Jun 2026 Jul 2026
RareDiseaseBench Agents combine clinical and biological evidence to diagnose and manage rare diseases. A rubric grades their diagnoses and plans. LatchBio Research science Expert Not in cards Gemini 4 Argon 43.8% Oct 2026
ROME Models solve reasoning problems selected to limit contamination. BAAI FlagEval Language reasoning University Not in cards Gemini 3 Pro 69.4 Sep 2025 Nov 2025
RSI-Exam Agents get a weak method on a long research task and must turn it into one that scores better on hidden data. It measures recursive self-improvement on executable research. AIMING Lab (UNC Chapel Hill) AI research tasks Research frontier Not in cards Claude Opus 5.5 53.6% Aug 2026 Nov 2026
SpaceDG-Bench Models answer spatial questions under clean images and nine forms of visual degradation. Choice and binary answers use accuracy, while numerical answers use relative-error thresholds. The test has 1,102 verified questions from scenes excluded from the training set. Shanghai Artificial Intelligence Laboratory and university collaborators Spatial reasoning and counting Everyday Not in cards Gemini 3.1 Pro 56.7% May 2026 Apr 2026
Spatial Pathfinding Reasoning Challenge Models trace paths across grids while satisfying rules about cells and edges. The scorer checks each coordinate sequence against valid solutions. Lars Benedikt Kaesberg and colleagues Puzzles and chess School Not in cards GPT-6.1 Sol 83.6% May 2025 Sep 2026
TasteVal 8 AI R&D tasks. Models iteratively design experiments. Taste measured as the compute needed to reach expert-level scores. P-Zero Research AI research tasks Research frontier Not in cards Claude Opus 5.5 230.4× Oct 2026 Sep 2026
The Aggregate Inversion and Execution Ladder Models predict the output of a short program or reconstruct the input to a digit transformation. The scorer checks exact answers across six difficulty levels. Mikhail Doroshenko (The Aggregate) Puzzles and chess School Not in cards GPT-5.5 100% Nov 2026
TutorBench Models act as tutors on three tasks: adaptive explanations for a confused student, feedback on student work, and hints. Graded against sample-specific rubrics. Scale AI Other University Not in cards Muse Spark 68.5% Oct 2025 Apr 2026
UnityBench-Mini3D Agents modify existing Unity projects. EditMode and PlayMode tests check the saved scene, component settings, and runtime behavior, with partial credit for completed milestones. Unity China AI Hub and collaborators Building whole projects University Not in cards GPT-5.6 Sol 72.5% Oct 2026 Aug 2026
VideoGAIA Agents inspect selected video segments, search the web, and read pages before answering questions. Model judges check answers, and the harness reports accuracy by category and difficulty. CUHK, Ant Group, Tsinghua University and others Video understanding Everyday Not in cards Seed2.0 Pro 58.3% Aug 2026 Aug 2026
AGI-Pilled Bench Models forecast when AI will reach milestones that already happened in 2026. The score is the median years after 2026 across the forecast tasks. Adam Holter Forecasting Everyday Not in cards Claude Fable 5.2 6.50 Oct 2026 Nov 2026
Brainf**k coding benchmark Models write Brainfuck programs for tasks such as copying input and adding digits. Tests check the output of each program. Pedro Schoen Function-level problems University Not in cards GPT-6 Astra 100% Nov 2026 Nov 2026
CEO-Bench Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score. Princeton University (Narasimhan and Liu labs) Multi-agent games Expert Not in cards Kimi K3 $22,148,357 Jun 2026 Aug 2026
Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. Naïve Simulated companies Expert Not in cards GPT-6 Astra $9,514 Oct 2026 Sep 2026
Code2Video Benchmark 168 product-video briefs. Agent writes HTML motion-graphics code rendered by HyperFrames. A trained pairwise judge on five axes produces Elo. HeyGen Research + Kaggle Video generation Expert Not in cards Claude Opus 5.5 1,585 Sep 2026 Sep 2026
Creation-MMBench Models write creative pieces grounded in up to nine images plus a role, background and instructions, across 765 cases in 51 tasks. A judge scores quality and visual fact alignment against case-specific criteria. OpenCompass (Shanghai AI Laboratory) Creative writing Everyday Not in cards Gemini 2.0 Pro 4.48 Mar 2025 Feb 2025
Databricks AI Index Coding agents complete tasks at different costs. The index compares cost per task and solution quality alongside enterprise usage statistics. Databricks Composite work indices Expert Not in cards Claude Fable 5.1 96.0 Nov 2026 Nov 2026
Doc2Harness Models build a coding-agent harness from an interface skeleton and a development document. The frozen harness then runs 328 scripted test cases in separate containers. Institute of Computing Technology, Chinese Academy of Sciences AI research tasks Research frontier Not in cards GPT-6 Astra 53% Oct 2026 Sep 2026
Drone-Bench Agent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline. Andon Labs Real robots Expert Not in cards Claude Opus 5.5 97.3% Jul 2026 Sep 2026
E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. Qwen Team, Alibaba Group, with Taobao and Tmall Group Running a business University Not in cards GPT-5.6 Sol 14.3 Aug 2026 Aug 2026
EvoEval Models solve Python problems that vary the constraints, interfaces, or logic of HumanEval tasks. Harbor ports the 100 problems in the Difficult subset. Chunqiu Steven Xia and colleagues Function-level problems School Not in cards GPT-4 Turbo 68.6% Mar 2024 Apr 2024
FrontierML Agents spend a fixed budget on GPUs and model APIs to train models for 21 tasks. An offline evaluator grades their submissions on held-out data. Induction Labs Machine learning engineering Research frontier Not in cards Claude Opus 5.5 58.9 Nov 2026 Oct 2026
GlobeBench Models act as text world models. Given an agent's past actions and observations, they predict the environment's next observation across code, enterprise, games, math and science, and dialogue. Patronus AI Interactive world models Expert Not in cards GPT-6 Astra 49.6% Oct 2026 Sep 2026
Hanabi Agent Benchmark Models play cooperative Hanabi in four-player teams. The benchmark scores shared board progress while hints and lives constrain coordination. flatkey Multi-agent games Everyday Not in cards Claude Opus 5.5 20.5 Oct 2026 Oct 2026
HW-Bench Agents complete six hardware engineering tasks for a 2.45 GHz antenna using native design and simulation tools. Code checks and calibrated model judges grade outputs against engineer-written criteria. The headline pass rate measures runs that satisfy every critical criterion, averaged across tasks. Arena Physica Electronics Expert Not in cards GPT-6 Astra 67% Nov 2026 Nov 2026
IndicParam Models answer multiple-choice questions from Indian competitive exams. The benchmark tests knowledge in low-resource Indic languages. Ayush Maheshwari and colleagues Knowledge in many languages University Not in cards GPT-5 45% Nov 2025 Oct 2025
Leipzig Benchmark Models answer 100 research-level math questions with known answers, written by 49 mathematicians at a Max Planck Institute workshop in April and May 2026. Benchmarks in Leipzig workshop at MPI MiS (49 mathematicians), board on ScienceBench (Christian Stump) Research mathematics Research frontier Not in cards GPT-5.6 Sol Pro 91% Jun 2026 Sep 2026
LiveNewsBench Release 2 Models search for answers to questions about recent news. The test limits searches and page visits and compares answers with curated references. Yunfan Zhang, Kathleen McKeown and Smaranda Muresan Questions that need search Everyday Not in cards DeepSeek-V3.2 84.5% Jan 2026 Jan 2026
LLM Public Goods Game Agents choose how many tokens to contribute to a common pool. The evaluation reports the fraction each model contributes across repeated games. Lech Mazur Multi-agent games Everyday Not in cards Gemini 2.0 Flash 45.2% Feb 2025 May 2025
LLM Thematic Generalization Benchmark v2 Models rank eight candidates using three positive examples and three counterexamples. Version 2 filters ambiguous prompts and reports a harder subset of 703 items. Lech Mazur Language reasoning University Not in cards Claude Opus 4.6 80.6 Mar 2026 Apr 2026
MeasureBench Real-world Models read values and units from real instrument images. The score is the published overall measurement accuracy. BAAI FlagEval team General visual questions Everyday Not in cards Seed2.0 Pro 41.7% Oct 2025 Mar 2026
MeasureBench Synthetic Models read values and units from generated instrument images. The score is the published overall measurement accuracy. BAAI FlagEval team General visual questions Everyday Not in cards Seed2.0 Pro 39.2% Oct 2025 Mar 2026
MedSafe-Dx v0.3 Models read synthetic primary-care intake summaries and decide whether to flag a dangerous condition. Rule-based labels determine weighted error costs, with missed serious cases costing seven times a false escalation or incorrect danger label. The score normalizes these costs against a fixed escalation baseline. Cortico Health Technologies Medicine Expert Not in cards Claude Opus 5.5 73.5 Oct 2026 Sep 2026
MentalBench (Psychiatric Diagnosis) Models select psychiatric diagnoses for 24,750 synthetic cases. Exact matching requires both diagnoses when the case lacks the evidence needed to distinguish them. KAIST and colleagues Medicine Expert Not in cards Claude Sonnet 4.5 62.7% Feb 2026 Nov 2025
MetroLLM-Bench Agents operate a transit kiosk using tools for routes, fares, and service disruptions. The first score checks their actions automatically. The composite also includes model judgments. Continker Customer service Everyday Not in cards Muse Glimmer 92.0 Jun 2026 Aug 2026
MindEval2 Models support simulated patients from MindSim, a tool-using harness with detailed profiles. Psychologist-written rubrics (MHQR) replace Likert scales for scoring. Sword Health Medicine Expert Not in cards Qwen3.5-27B 4.08 Oct 2026 Sep 2026
MM-HELIX Multimodal models solve 1,260 items from 42 synthetic tasks that need long reflective reasoning with backtracking. MM-HELIX team (Zhao et al.) Visual math and diagrams University Not in cards GPT-5 84.5% Oct 2025 Aug 2025
NorEval Models complete language understanding and generation tasks in both Bokmål and Nynorsk. The authors compare model performance with human baselines. Vladislav Mikhailov and colleagues Knowledge in many languages School Not in cards DeepSeek-V4-Flash-Base 68.2 Apr 2025 Apr 2026
S-Eval Models respond to Chinese and English harmful prompts and attack variants. A trained safety judge measures the safe-response fraction and attack success rate. Zhejiang University and Alibaba Group Refusing harmful requests Everyday Not in cards gpt-oss-120b 88.8% May 2024 Dec 2025
SEAL Multilingual Models answer 1,000 private prompts per language, written by native speakers on general and cultural topics. Annotators rate each answer and compare pairs side by side, and boards rank models by Elo. Scale AI Chat and instructions in many languages Everyday Not in cards o1 1,134 May 2024 Jan 2025
SnorkelWordle Models play Wordle on 100 rare five-letter English words, with six guesses and letter feedback after each guess. The score is the win rate. Snorkel AI Puzzles and chess Everyday Not in cards GPT-5 94% Jul 2025 Nov 2025
TASTE 92 pairwise comparisons of AI safety research proposals. Model must match experienced researchers' preference. Accuracy versus ~77% human agreement. Anthropic Fellows Program + Anthropic AI research tasks Expert Not in cards Claude Fable 5 60% Aug 2026 Jul 2026
The Aggregate Lichess Pairwise Strength Models read two anonymized chess games and choose which designated player had the higher pregame Lichess rapid rating. Accuracy counts 25 choices. Mikhail Doroshenko (The Aggregate) Puzzles and chess School Not in cards Gemini 3.8 Flash 88% Nov 2026
The Aggregate Lichess Rating Prediction Models estimate both players' pregame Lichess rapid ratings from the moves of 25 anonymized games. The score is mean absolute error across 50 players. Mikhail Doroshenko (The Aggregate) Puzzles and chess School Not in cards Gemini 3.1 Pro 270 Nov 2026
TradeRank Arena Models choose positions in cryptocurrencies and US stocks using shared market data and simulated capital. The leaderboard ranks portfolios by return. TradeRank.ai Finance University Not in cards Grok 4.7 0% Jan 2026 Nov 2026
VerilogEval v2 Models write Verilog hardware descriptions from specifications. The hub port uses the specification-to-RTL tasks, where simulation checks generated designs against reference behavior. NVIDIA and collaborators Electronics University Not in cards Gemma 4 31B IT 88% Aug 2024 Jun 2026
Visual Aesthetic Benchmark Models see a set of images with the same subject and intent and must pick the best one, and in a harder setting both the best and the worst. The 400 sets hold 1,195 commissioned works across fine art, photography and illustration. Labels come from expert judgments, and only sets where experts largely agreed are kept. BakeLab with University of Washington, UC Santa Barbara, Stanford, Notre Dame and IBM Research Taste and style Expert Not in cards Claude Sonnet 4.6 40.3% May 2026 Feb 2026
VRBench Models answer multi-step reasoning questions about long narrative videos, which average 1.6 hours. The 1,010 videos come with 9,468 human-labeled question and answer pairs and 30,292 timestamped reasoning steps across seven reasoning types. Multiple-choice answers score the outcome, and a model judge scores the reasoning steps. Shanghai AI Laboratory (OpenGVLab) and collaborators Long videos Everyday Not in cards Claude 3.7 Sonnet 68.2% Jun 2025 Jun 2025
WebVR Models recreate a webpage from a screen recording and supplied assets. A model judge scores the rendered page against visual and interaction criteria. StepFun and Tsinghua University Web and app front ends University Not in cards Kimi K2.5 79.1 Mar 2026 Feb 2026
AbGen Models design ablation studies to test the contribution of a specified component. The benchmark compares the proposed experiments with expert annotations from 807 NLP papers. Yale NLP Lab and collaborators AI research tasks Expert Not in cards DeepSeek-R1 4.11 Jul 2025 May 2025
AgentBench FC Agents work five containerized AgentBench environments, ALFWorld, DB, knowledge graph, OS and WebShop, through function-calling prompts instead of the original text format. Tsinghua THUDM (Zhipu AI) Function calling Everyday Not in cards Claude Sonnet 4.5 58.9% Oct 2025 Sep 2025
Agentic BAIM-LLM Evaluation (ABLE) Agents run a viral-protein redesign workflow. Nine tasks cover structure retrieval, design strategy, tool selection and biological AI model use. Outputs are auto-scored. SecureBio Hazardous knowledge Research frontier Not in cards Gemini 3.1 Pro 84.6% Oct 2026
AlgoTune Agents rewrite code for 154 math, physics, and computer science functions to run faster and stay correct. Scoring is harmonic mean speedup over reference code. University of Tübingen and Princeton University Speed and optimization Expert Not in cards GPT-5.2 2.05× Jul 2025 Mar 2026
AncientDoc Models transcribe historical Chinese pages, translate passages into modern Chinese, and answer questions about meaning, knowledge, and literary style. Fudan University and ByteDance Documents, charts and text in images University Not in cards Qwen-VL-Max 5.77 Aug 2025 Apr 2025
AutoBaxBench Models write backend servers for tasks that a pipeline generated. Tests and exploits are generated too. A larger task set built in the style of BaxBench. ETH Zurich SRI Lab (von Arx et al.) Secure code and review Expert Not in cards Claude Fable 5.2 88.6% Dec 2025 Nov 2026
Bedside Bench Models answer 500 physician-written clinical cases on diagnosis, workup, escalation and medication safety. An LLM judge grades each answer against the authors' criteria. Doximity Medicine Expert Not in cards Claude Opus 5 94.6% Oct 2026
Biblically Informed Bot Learning Evaluation Models answer four-option questions about all 66 books of the Protestant Bible, biblical people, places, and measurements. The maker reports accuracy. MushroomGecko Trivia and open questions School Not in cards Qwen3.5-4B 52.3% Apr 2025 Mar 2026
BoardBench 0.2 Models choose one move in each of 30 board positions across ten games. An engine grades each legal move and assigns zero to illegal moves. BoardBench Puzzles and chess Everyday Not in cards GPT-6 Astra 67.2 Oct 2026 Oct 2026
Chess LLM Benchmark (Ryan Brew) Models evaluate chess positions, choose a move, and explain the position. The primary move metric counts blunders against the Lc0 chess engine. Ryan Brew Puzzles and chess School Not in cards Gemma 4 31B IT 35.8% Apr 2026
Collaborative Intelligence Benchmark A model works in a team with scripted teammates and live model partners, and a teammate creates a test: pressure to take an unsafe shortcut, misconduct to report, a need nobody states, a conflict, or a cheat that a weaker model left behind. The 240 tasks across 8 dimensions of teamwork come from real records, such as aviation accident reports and clinical cases, and from existing benchmarks, such as SWE-bench and negotiation datasets. Each task has a checkable outcome. A model is scored only on the tasks it solves alone, so the score measures teamwork, not ability. Stanford University (James Zou's group) Multi-agent games Everyday Not in cards Gemini 4 Argon 82.2% Nov 2026 Oct 2026
ConStory-Bench Models write long stories, up to tens of thousands of words, from 2,000 prompts in four task types. An automated checker, ConStory-Checker, finds contradictions in facts, timelines, characters and world rules and cites the passages behind each one. Scores count consistency errors, using a taxonomy of 5 categories and 19 subtypes. Microsoft and Singapore University of Technology and Design Creative writing Everyday Not in cards GPT-5 0.11 Mar 2026 Oct 2025
ContractBench (Observation Contracts) Agents complete 33 API workflow tasks. Validators check whether agents preserve returned values exactly and use them while they remain valid. SecurityLab at UC Davis Function calling University Not in cards Claude Opus 4.6 77.8% May 2026 Apr 2026
CS-4k Models answer questions about the stages of computer science research using papers as evidence. A model judge scores answers on the held-out evaluation set. Penghao Wang and colleagues AI research tasks Expert Not in cards o3 71.3 Oct 2025 Aug 2025
dfbench Agents do defensive security work on held-out code: find vulnerabilities, judge whether reported findings are real, and explain what a diff changed. Each part is scored separately. depthfirst Finding and exploiting real bugs Expert Not in cards Claude Mythos 5 69% Aug 2026 Sep 2026
DrawingVQA Models answer 92 multiple-choice questions about 33 construction drawings. They read symbols, cross-reference details, and calculate quantities. Louisiana State University, University of Illinois Urbana-Champaign, and an independent researcher Documents, charts and text in images University Not in cards Gemini 3 Pro 77.2% Apr 2026 Nov 2025
Flex-RSI Agents improve a starter solution and their own tools within a fixed budget. A hidden test split measures the final solution. Flex-RSI team Machine learning engineering Expert Not in cards Claude Opus 5.5 1.52 Oct 2026 Oct 2026
FretBench Models read guitar tablature in four tunings and name notes from string positions and fret numbers. The grader compares their answers with an answer key. Jaiden Capra Language reasoning School Not in cards Gemini 3.1 Pro 100% Mar 2026 Apr 2026
GameDevBench Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result. Carnegie Mellon University Building whole projects University Not in cards Claude Opus 5.5 70.6% Feb 2026 Sep 2026
GradeBench Models read a student's handwritten answer to a university STEM exam question, with the rubric and a reference solution, and pick the rubric items to award. Scored by exact match with the items the instructor chose, averaged over five runs. Pensive Other professions University Not in cards Claude Fable 5.2 57.3% Nov 2026 Nov 2026
HWE Bench V2 (RISC-V CPU design) Agents improve a RISC-V processor from a shared starting design. The benchmark scores throughput on five hidden programs using clock speeds from Gowin's timing tool. Felipe Sens Bonetto Electronics Expert Not in cards Claude Opus 5.5 7024.2 Oct 2026 Sep 2026
Image2Struct Vision-language models turn images of webpages, LaTeX and musical scores into code that renders them. The render is compared with the input image. Stanford CRFM Documents, charts and text in images University Not in cards GPT-4o 94.7 Oct 2024 Oct 2024
LifeSci Diamond Models write regulatory affairs deliverables for drugs, biologics and medical devices, such as change assessment memos, dossier inconsistency logs and agency letters. The 55 tasks span five families: reporting categories, Module 2/3 dossier consistency, missing evidence, submissions and correspondence, and device vigilance. A model judge grades each answer against a task rubric, and a task passes only when every rubric line is met and no penalty fires. Arca Medicine Expert Not in cards GPT-6 Astra 20% Sep 2026 Oct 2026
LiveOCRVQA Models read stylized text in images and answer questions about it. BAAI FlagEval Documents, charts and text in images Everyday Not in cards Gemini 2.5 Pro 80.5 Sep 2025
LiveTradeBench Agents allocate simulated portfolios across US securities and Polymarket contracts using streamed prices and news. The researchers track portfolio returns over successive trading days. U Lab at the University of Illinois Urbana-Champaign Finance University Not in cards Claude 3.7 Sonnet 12.1% Aug 2025 Sep 2025
LLM Benchmark: Danske Idiomer Models match Danish idioms to pictures, complete expressions and write definitions. GPT-4.1 grades definitions, and answer keys cover the other tasks. idiom.dk Reading in many languages Everyday Not in cards GPT-5.1 97.5% Jan 2026 Dec 2025
LLM Divergent Thinking Creativity Benchmark Models generate sets of words that differ in meaning. Lech Mazur Creative writing Everyday Not in cards o1-preview 4.79 Mar 2025
LMBuild Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. University of Illinois Urbana-Champaign CAD models Expert Not in cards GPT-6 Astra 66.5% Oct 2026 Sep 2026
MacroscopeBench Models review commits from public repositories and must flag the known bugs a commit introduced without false reports. LLM judges match findings to the bugs. Macroscope Fixing real repositories Expert Not in cards Claude Opus 5.5 83.9% Nov 2026
NaijaProverb-Bench Models select the intended meaning of Nigerian proverbs from four choices. One track adds the original Igbo or Hausa wording when available. Jane Odum and Ayodele Dada Reasoning in many languages Everyday Not in cards Gemini 3.1 Pro 47.3% Apr 2026 Apr 2026
NanoGPT Speedrun Frontier Agents change a GPT optimizer on fixed GPUs. Each candidate must reach the target validation loss across eight fixed seeds using fewer training steps. Prime Intellect Machine learning engineering Research frontier Not in cards Claude Fable 5 2726.0 Aug 2026 Aug 2026
PokéBench: Raw Vision + Buttons Models play Pokémon Red from screenshots using eight Game Boy buttons. Game memory verifies ten milestones, with fewer turns breaking ties. Cody Harris Video games Everyday Not in cards Claude Fable 5.1 10.0 Oct 2026 Nov 2026
Ramp Accounting Bench Agents reconcile accounts and prepare financial records across 22 simulated companies. A model grades their answers and files against accounting rubrics. Ramp Labs Finance Expert Not in cards Gemini 4 Argon 61.1% Nov 2026 Nov 2026
SEAL Precise Instruction Following Models answer 1,054 private prompts with detailed instructions in nine categories. Human annotators check 12 yes or no criteria and compare responses side by side on a 7-point scale. Scale AI Following precise instructions Everyday Not in cards o1 92.0 May 2024 Jan 2025
SimulatorArena Models act as students or writing clients. Researchers compare their messages and assistant ratings with those from people who used the same assistants. Georgia Institute of Technology and Microsoft Multi-turn conversation School Not in cards GPT-5 1.01 Aug 2025
SuperCLUE The CLUE team tests Chinese and foreign language models on periodic sets of Chinese questions covering language, professional knowledge, reasoning, agents and safety. Open answers are graded by a strong LLM judge, and results feed an overall ranking. CLUE benchmark team (SuperCLUE) Composite indices University Not in cards Gemini 4 Argon 73.6 May 2023 Nov 2026
SuperCLUE-Terminal Agents complete tasks in a terminal. SuperCLUE Terminal tasks Expert Not in cards MiMo-V2.6 Pro 64.7 Sep 2026
The Aggregate Chart Recall Models estimate ten years of monthly Wikipedia pageviews from each article title alone. The scorer averages each article’s root mean squared error in log views. The Aggregate (Mikhail Doroshenko) Trivia and open questions School Not in cards GPT-6.2 Astra 0.22 Nov 2026 Nov 2026
Thinkingbox-bench Agents complete 507 business workflows through tools and simulated user conversations. Executable checks require correct backend updates without forbidden side effects. Microsoft and collaborators Customer service University Not in cards Claude Opus 5 66.5% Aug 2026 Sep 2026
AEC-Bench (Theodore Galanos) Agents perform engineering calculations such as cable sizing and hydraulic design. Task verifiers score their outputs, and the leaderboard reports mean reward and completion. Theodore Galanos Tasks from real jobs University Not in cards Grok 4.3 0.89 May 2026 Apr 2026
AfroBench-Lite Models complete language understanding, question answering, reasoning, and generation tasks across 14 African languages. Seven datasets use task-specific metrics, including accuracy, F1, character-level translation scores, and exact match. Masakhane, McGill University, Mila, University of Waterloo, University College London, Lelapa AI, and The African Research Collective Knowledge in many languages School Not in cards GPT-5 77.7 Jun 2025 Nov 2025
Bio-VQA Models identify organisms, describe visible traits, judge spatial relationships, and recognize individual animals across photographs. The benchmark has 24 tasks. Imageomics Institute and collaborators General visual questions University Not in cards InternVL3-38B 0.91 Aug 2026 Jun 2026
CORE-Bench Agents reproduce the results of published studies from their code and data. 270 tasks from 90 papers in computer science, social science and medicine come in three difficulty levels. Princeton University Research science Expert Not in cards Claude Opus 4.5 77.8% Sep 2024 Nov 2025
CorpBench Work Agents carry out 100 business workflows across finance, marketing, operations, RevOps and sales, 20 per department, using simulated business systems and tool contracts. Grading checks the final state, the actions taken, and whether the agent finished the job or escalated it correctly. Revensi Running a business Everyday Not in cards GPT-5.6 Luna 92.7 Sep 2026 Sep 2026
Crest Arcade Elemental Armory Models equip four elemental balls with cards, then select and aim a ball each round. The score averages match win rates across three levels. Glimmer Index Video games Everyday Not in cards GPT-6.1 Terra 67.3% Nov 2026
Crest Arcade FreeCell Models move cards to the four foundations using a variable number of free cells. The score averages win rates across five tiers. Glimmer Index Puzzles and chess Everyday Not in cards Claude Haiku 5.5 63.3% Nov 2026
Crest Arcade Hustle Sim Models buy upgrades and hire helpers to reach earnings targets in an idle game. The score compares simulated completion times with reference strategies. Glimmer Index Video games Everyday Not in cards GPT-6.2 Astra 83.7 Nov 2026
Crest Arcade Micro Mouse Models steer a simulated robot through six daily mazes. The score compares each finish time with the median human time and gives unfinished mazes zero. Glimmer Index Video games Everyday Not in cards Gemini 4 Neon 89.3 Nov 2026
Crest Arcade Mines Models reveal or flag cells on Minesweeper boards that can be solved without guessing. The score averages win rates across four board sizes. Glimmer Index Puzzles and chess Everyday Not in cards Gemini 4 Carbon 100% Nov 2026
Crest Arcade Sudoku Models fill complete Sudoku grids and report confidence. The score averages the share solved across seven equally weighted difficulty tiers. Glimmer Index Puzzles and chess Everyday Not in cards Claude Opus 5.5 100% Nov 2026
Deta Studio AI Construction Takeoff Models use Deta’s tools to mark floor areas and count openings and electrical devices on four drawings. The rating combines accuracy, cost, and time. Deta Studio Tasks from real jobs University Not in cards GPT-6.2 Sol 76 Nov 2026 Nov 2026
DrafterBench Agents call drawing revision tools from written instructions. The evaluator records their operations and arguments across 1,920 tasks covering 12 types of revision. Yinsheng Li, Zhen Dong, and Yi Shao Function calling University Not in cards Claude Sonnet 4.5 84.5% Apr 2025 Mar 2026
Dutch Book Bench Models assign probabilities to related events in separate conversations. The grader checks whether inconsistent probabilities let a bettor guarantee a profit. Yi Yu Real-world events University Not in cards Gemini 3.8 Flash 89.2% Oct 2026 Oct 2026
EarthVerse Agents investigate documented natural hazards using event data from different sources. The evaluator checks individual claims and the calculations that support them across 405 tasks. Zhiqing Cui and colleagues Lab and data work Expert Not in cards Claude Fable 5 85% Aug 2026 Jul 2026
Echo General AI Prediction Leaderboard Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models. UniPat AI and Peking University Real-world events Expert Not in cards DeepSeek-V4-Pro 1,018 Mar 2026 Apr 2026
EmbodiedBench EB-ALFRED Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups. University of Illinois Urbana-Champaign + others Simulated robots Everyday Not in cards Claude 3.7 Sonnet 67.7% Feb 2025 Apr 2025
EmbodiedBench EB-Habitat Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage. University of Illinois Urbana-Champaign + others Simulated robots Everyday Not in cards Claude 3.7 Sonnet 58.7% Feb 2025 Apr 2025
EmbodiedBench EB-Manipulation Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage. University of Illinois Urbana-Champaign + others Simulated robots Everyday Not in cards Claude 3.7 Sonnet 28.5% Feb 2025 Apr 2025
EmbodiedBench EB-Navigation Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage. University of Illinois Urbana-Champaign + others Simulated robots Everyday Not in cards InternVL3-78B 53.7% Feb 2025 Apr 2025
EmbodiedVerse-Bench Models answer questions about spatial relationships and actions from visual observations. The leaderboard compares models across embodied reasoning tasks. Beijing Academy of Artificial Intelligence Embodied reasoning Everyday Not in cards o4-mini 50.7 Sep 2025
FrogsGame post-training task Agent has 20 hours to post-train Qwen3-8B to solve FrogsGame puzzle boards. A hidden verifier scores the best checkpoint on unseen boards. Thoughtful Lab (task in FrontierSWE v1) Machine learning engineering Research frontier Not in cards Claude Fable 5 34% Apr 2026 Aug 2026
HELM Classic Models are evaluated with standardized prompts on the scenarios from the original HELM paper. Each scenario reports accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. Stanford CRFM Composite indices University Not in cards Llama 2-70B 94.3 Nov 2022 Jul 2023
Hermes Bench Agents in Hermes Agent do 150 tasks across 25 categories, such as research, diagrams, memory, tool use and safety. Graders check the files and state left behind. Nous Research General assistants Expert Not in cards Claude Opus 5.5 76.7% Oct 2026 Nov 2026
Hermes Index Nous Research averages pass@1 and cost per task over Hermes Bench, Terminal-Bench 4.0, Terminal-Bench Science and SkillsBench, all run in Hermes Agent. Nous Research Composite indices Expert Not in cards Claude Opus 5.5 63.3 Oct 2026 Nov 2026
IndustryBench Models answer 2,049 Chinese industrial procurement questions grounded in national standards and product records. Answers must respect regulated thresholds and safety clauses. Alibaba (1688 multimodal and industrial AI) Other professions Expert Not in cards Gemini 3.1 Pro 69.4% May 2026 Mar 2026
Insurance Agent Benchmark Models read commercial insurance documents and answer questions about their contents. Cooper compares each model alone and with its document processing system. Cooper Labs Documents, charts and text in images University Not in cards Grok 4.6 77.6% Sep 2026 Sep 2026
KorNAT Models answer questions about Korean social values and common knowledge. The authors compare answers with Korean reference judgments. Jiyoung Lee and colleagues Knowledge in many languages Everyday Not in cards Claude Opus 5 65.4% Feb 2024 Aug 2026
LatchBio Capabilities V0 Agents complete biological data analysis tasks. The historical score averages the available benchmark scores equally and excludes missing scores. LatchBio Genomics and life sciences Expert Not in cards Claude Sonnet 5.5 51.3% Oct 2026
Legora BAR Models work private legal cases end to end inside Legora's agent harness, with the matter's documents and templates. An LLM judge scores outputs against rubrics written by lawyers. Legora Law Expert Not in cards Claude Opus 5.5 106.0 Sep 2026 Oct 2026
LLM Deceptiveness and Gullibility Models try to deceive other models and recognize deceptive claims. Lech Mazur Multi-agent games Everyday Not in cards Claude 3.5 Sonnet 1.10 Mar 2025
LLM Physics Plotting Benchmark Models write Python code to produce requested plots from ATLAS collision data. Each trial passes when the code runs and meets the configured checks. Gordon Watts Lab and data work University Not in cards GPT-5.2 Codex 90.6% Sep 2025 Jul 2026
MaxxBench Agents solve 89 modified DeepSWE repository tasks. Revised requirements, code, reference patches, and tests check whether agents can solve changed problems rather than recall original fixes. unLAP Fixing real repositories Expert Not in cards Claude Opus 5.5 76.4% Oct 2026 Oct 2026
Multi-Crit Multimodal judges compare responses under several criteria. Human judgments provide the reference for each criterion and reveal trade-offs between criteria. Multi-Crit authors General visual questions University Not in cards o4-mini 43% Nov 2025 Oct 2025
Multimodal RewardBench 2 Reward models compare two outputs containing text and images. Accuracy measures agreement with expert preferences across four tasks. Meta and Surge AI General visual questions University Not in cards Gemini 3 Pro 76.3% Dec 2025 Nov 2025
NarrativeBench Models seek, include, exclude, and recover narratives from multilingual news articles. Citation-based F1 or recall scores narrative coverage after checking the response language. Nikhil Sharma, Kelly Marchisio, Kenton Murray, and Ziang Xiao Reading in many languages Everyday Not in cards Gemini 4 Argon 90.7% Nov 2026 Nov 2026
OGS-Bench Models use a supplied culinary vocabulary to choose food and drink pairings, classify dishes, and produce structured answers. The headline score combines automatically checked tasks. Gastrona Language reasoning University Not in cards GPT-5.5 86.8 Apr 2026 Apr 2026
OOLONG Models aggregate information across long collections of text. The hub port implements the synthetic and real datasets with a context-controlled recursive language model interface. OOLONG authors Reasoning over long documents University Not in cards Gemini 3 Pro 68.3 Sep 2025 Apr 2026
Open Persian LLM Leaderboard Models answer Persian multiple-choice questions. The leaderboard averages accuracy across five tasks, including two question sets from the organizers. Part AI and Amirkabir University of Technology NLP Lab Knowledge in many languages University Not in cards Llama 3.3 70B Instruct 70.4% Nov 2024 Mar 2025
Passport Control Models maintain a dated memory across chats, emails, and tool results. Tests check whether they retain, update, expire, and retrieve facts at the right time. Rairo Mukamuri Finding facts in long inputs Everyday Not in cards Gemini 3.1 Pro 1.00 Oct 2026 Sep 2026
PGG-Bench Agents decide how much to contribute to a public good and whether to punish other players. Lech Mazur Multi-agent games Everyday Not in cards Gemini 2.5 Pro 14 Mar 2025
Realm Financial Reasoning Agents analyze financial documents and spreadsheets. A model grades their answers against practitioner rubrics. micro1 Finance Expert Not in cards Gemini 3.8 Flash 56.2% Sep 2026
ReSpaceBench Models recognize rooms across separate visits and track objects that move between visits. The authors test 32 models on synthetic and real homes. Tsinghua University and ByteDance Embodied reasoning Everyday Not in cards GPT-5.6 Sol 50.5% Nov 2026 Aug 2026
SAFE-Teen Chatbots hold 6 to 20 turn chats with a simulated teen across 106 clinician-written scenarios, branching on how the teen reacts. A judge rates safety checks and reports the failure rate. Vals AI Wellbeing and sycophancy Everyday Not in cards Muse Spark 1.3 6.35% Oct 2026 Oct 2026
SAFE-Teen Health advice Models respond to simulated teens in health advice scenarios. The score is the published conversation failure rate. Vals AI Wellbeing and sycophancy Everyday Not in cards GPT-6.1 Sol 9.00% Oct 2026 Oct 2026
SAFE-Teen Relationship boundaries Models respond to simulated teens in relationship boundary scenarios. The score is the published conversation failure rate. Vals AI Wellbeing and sycophancy Everyday Not in cards GPT-6.1 Sol 19% Oct 2026 Oct 2026
SAFE-Teen Urgent safety Models respond to simulated teens in urgent safety scenarios. The score is the published conversation failure rate. Vals AI Wellbeing and sycophancy Everyday Not in cards GPT-6.1 Sol 15% Oct 2026 Oct 2026
ScienceBoard Computer-use agents do scientific workflows in desktop software such as ChimeraX, Celestia, GRASS GIS and Lean 4. The apps are modified to expose state for checking. OS-Copilot team (Shanghai AI Laboratory and HKU) Desktop apps Expert Not in cards Claude Opus 4.8 68% May 2025 Aug 2026
Slop Score Model writing is scored by how often it uses words and patterns that are far more common in AI text than in human text. Lower means less slop. EQ-Bench (Sam Paech) Taste and style Everyday Not in cards Kimi K2 18.3 Oct 2025 Sep 2025
SPM-Bench Models interpret microscopy images and select all correct answers. The primary score gives partial credit and penalizes incorrect selections. Alibaba Group and Skylenage Lab and data work Expert Not in cards Claude Opus 4.7 0.94 Feb 2026 Apr 2026
Stick to Your Role! Models simulate populations of fictional personas and answer personal-values questionnaires in many different conversation contexts. The benchmark measures how stable the expressed values stay across contexts, both for each persona and for the ranking of personas within the population. Inria (Flowers team) and collaborators Creative writing Everyday Not in cards Qwen2.5-VL-72B 0.90 Feb 2024 Apr 2025
The Aggregate Personal Style Imitation Models rewrite 53 neutral drafts in one person's style using 30 other messages as examples. An authorship classifier scores each rewrite. Mikhail Doroshenko (The Aggregate) Creative writing Everyday Not in cards Claude Fable 5.1 76.7% Nov 2026 Nov 2026
ThinkGeo Agents use remote sensing tools to interpret aerial and satellite images. The authors compare individual tool calls and use GPT-4o mini to grade final answers. Mohamed bin Zayed University of Artificial Intelligence and collaborators Apps and MCP servers University Not in cards GPT-6 Sol 100% May 2025 Oct 2026
UltraText Bench Image generators render dense bilingual text in many regions from 432 prompts. Scores check string accuracy, placement and legibility. Westlake University General visual questions Everyday Not in cards GPT-Image-2 99.3 Oct 2026 Feb 2026
VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. Scale AI Function calling University Not in cards Muse Spark 1.1 44.8% Oct 2025 Jul 2026
4DCodeBench Agents reconstruct dynamic scenes from video as executable graphics programs, including physical simulations, so rendered output matches the observed motion. 4DCodeBench team Physical prediction Expert Not in cards GPT-6 Astra 0.79 Oct 2026 Sep 2026
Accelerate Bench Coding agents speed up 32 real scientific applications, such as GROMACS and OpenFOAM, on a fixed CPU allocation within 8 hours. A hidden verifier checks correctness, and Score@1 averages the share of runtime saved, with incorrect programs scoring zero. Tencent Hy Team, HKUST, HKUST (Guangzhou), Tsinghua University and Peking University Speed and optimization Expert Not in cards Claude Sonnet 5.5 67.4% Nov 2026 Nov 2026
Active-SWE Coding agents inspect repository snapshots and fix bugs without issue reports. Evaluation checks patches against reference tests and measures localization precision and recall. A separate stage uses generated tests and model review to assess newly revealed bugs. Sichuan University and University of Electronic Science and Technology of China Fixing real repositories Expert Not in cards Claude Opus 4.8 20% Aug 2026 Jul 2026
ANIMA Models explain decisions about animal welfare. A judge scores their reasoning across 13 ethical dimensions. Compassion Aligned Machine Learning and Sentient Futures Deception and alignment Everyday Not in cards DeepSeek-V4-Flash 74.4 Mar 2026 Jun 2026
BalatroBench Models play the deck-building roguelike Balatro through an API bot. The leaderboard compares how far each model and strategy gets in a run. Coder Video games Everyday Not in cards Gemini 3 Pro 19.3 Aug 2025 Jan 2026
CiteVQA Multimodal models answer questions about long PDFs and must cite the page region that supports each answer with a bounding box. The set has 1,897 questions over 711 documents in seven domains and two languages, averaging 40.6 pages per document. Strict Attributed Accuracy counts an answer only when both the answer and the cited region are correct. OpenDataLab (Shanghai AI Laboratory) Documents, charts and text in images University Not in cards Gemini 3.1 Pro 76% May 2026 Apr 2026
Clash of AIs Season 1 Models publish trade entries, targets, and stops before prices move. Clash of AIs simulates fills against market prices and ranks accounts by total equity. Clash of AIs Finance University Not in cards GPT-5.6 Sol $283,903 Apr 2026 Sep 2026
Claw-SWE-Bench General-purpose agent harnesses such as OpenClaw do multilingual SWE-bench-style coding tasks through an adapter that turns their output into scorable patches. TokenRhythm Fixing real repositories Expert Not in cards DeepSeek V4.1 Flash 79.8% Jun 2026 Sep 2026
CliniCARE-Bench Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases. Scale AI Medicine Expert Not in cards GPT-5.5 76.1% Aug 2026 Jul 2026
Crest Arcade Nations Royale Models manage one nation against 47 computer opponents on a simulated map. Wins earn full credit and unfinished games earn credit for territory held. Glimmer Index Video games Everyday Not in cards gpt-oss-20b 100.0 Nov 2026
CRM-Agent-Bench Agents handle customer and business requests in a simulated enterprise. Executable checks verify tool arguments, action order, and which records changed across 555 tasks. Salesforce Running a business University Not in cards Gemini 3.8 Flash 83.6% Oct 2026 Sep 2026
DAYJOB: Data Science Agents do data science assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Surge AI Data science and SQL Expert Not in cards Claude Opus 5.5 37.3% Oct 2026 Oct 2026
DAYJOB: Tax Agents take tax assignments from client request to finished return, workpaper or advice, using messy records. A pass needs 95% of the expert rubric criteria. Surge AI Finance Expert Not in cards Claude Sonnet 5.5 10.8% Oct 2026 Oct 2026
Digital Work Index Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to. Snorkel AI Composite work indices Expert Not in cards Gemini 4 Argon 28.3 Oct 2026 Oct 2026
EmoBench-M Multimodal models are tested on emotional intelligence across 13 scenarios in three dimensions: emotion recognition, conversational emotion understanding and socially complex emotion analysis. EmoBench-M authors General visual questions Everyday Not in cards Gemini 3 Pro 70.5% Feb 2025 Dec 2025
FeatEng Models write Python functions to transform 103 tabular datasets. The score measures how much those changes reduce prediction error for an XGBoost model. Snowflake and Institute of Computer Science, Polish Academy of Sciences Data science and SQL Expert Not in cards o1-preview 11.1% Oct 2024 Sep 2024
FinDesk-Bench Agents find Treasury figures and use them to handle client requests across ten business services. A model grades API calls and the records agents leave behind. Veris AI Documents and spreadsheets University Not in cards Gemini 4 Carbon 90.4% Nov 2026 Nov 2026
GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. SemiAnalysis Tasks from real jobs Expert Not in cards Claude Fable 5.2 16.4% Nov 2026 Nov 2026
GIM Models solve problems that combine constraint tracking, reasoning, and communication using broadly accessible knowledge. Model judges score answers against task rubrics. An item response model adjusts for problem difficulty and judge effects to estimate ability. Rohit Patel, Alexandre Rezende, and Steven McClain at Meta Language reasoning Everyday Not in cards GPT-5.4 Pro 1.63 May 2026 Apr 2026
HearingBench Models identify sounds from spectrogram images without hearing the audio. The test covers everyday sounds, instruments, vocals, words and sentences. Adam Holter Audio understanding and voice University Not in cards Gemini 4 Carbon 46% Oct 2026 Nov 2026
HireWatch Models select questions for job interviews under different workplace pressures. The benchmark measures how often they select questions classified as unlawful under US employment law. Huy Nghiem and colleagues Law University Not in cards Claude Opus 4.7 0.05% Nov 2026 Apr 2026
IndicIFEval Models follow instructions in 14 Indic languages. Rule-based checks test constraints such as required words, paragraph structure and word counts. AI4Bharat Chat and instructions in many languages Everyday Not in cards GPT-5 82.4% Feb 2026 Dec 2025
Inductive Arithmetic Pattern Reasoning Models infer a hidden arithmetic operation from worked examples and calculate the result for a new expression. The grader checks the final integer. Indra Agustian Abstract puzzles and logic School Not in cards Gemini 2.5 Flash 100% Nov 2026 Oct 2026
InSAR Regional Snow Mapping (Axis Spatial) Agents analyze Alpine radar scenes, align weather observations with radar windows, and calculate regional statistics. A script supplies reference answers for 24 tasks. Axis Spatial Lab and data work University Not in cards Claude Opus 4.7 100% Nov 2026 Oct 2026
LEGO-Puzzles Vision-language models answer LEGO assembly questions, from basic spatial understanding to planning several assembly steps. The elementary set has 1,100 questions over 11 tasks. Shanghai AI Laboratory and collaborators (Tang, Duan et al.) Spatial reasoning and counting Everyday Not in cards GPT-5 72% Mar 2025 Oct 2025
MCP-Bench Agents solve multi-step tasks using 28 live MCP servers with 250 tools in areas such as finance, travel, scientific computing and academic search. Scoring covers tool use, coordination and planning. Accenture Apps and MCP servers Expert Not in cards GPT-5 0.75 Aug 2025 Aug 2025
MediaPerf Models tag and summarize video advertisements. The benchmark measures tag accuracy, summary quality, cost and throughput. Coactive AI Video understanding University Not in cards GPT-5.4 0.42 Mar 2026 Apr 2026
MMLU Medical Models answer the multiple-choice questions from the medical subjects of MMLU, such as anatomy, clinical knowledge and professional medicine. UC Berkeley (Hendrycks et al.), subset chosen by medical model reports Medicine University Not in cards Llama 3-8B 72.9% Sep 2020 May 2024
Multi-LCB Models solve LiveCodeBench-style contest problems in twelve programming languages, with dated problems for contamination-aware evaluation. Multi-LCB team Competitive programming Competition Not in cards gpt-oss-120b 67.8% Jun 2026 Aug 2025
OWE-Bench 26 long-horizon engineering and research projects across software, physical engineering, machine learning and scientific computing, from GPU kernels to aircraft design. Agents get 4 to 18 hours per attempt, and an offline verifier scores what they ship on held-out cases. Mercor Building whole projects Research frontier Not in cards GPT-6.2 Astra 67.2% Nov 2026 Nov 2026
PianoBench Models answer questions about piano performances from recordings or symbolic transcriptions. The authors compare accuracy with and without access to the music. Carnegie Mellon University, Queen Mary University of London, and Cornell University Audio understanding and voice University Not in cards GPT-6 Astra 93.2% Nov 2026 Sep 2026
RadLE 2.0 Models diagnose single-image radiology cases with a confidence score from 0 to 4, or say they do not know. A confidence-weighted index rewards right answers and penalizes confident errors. CRASH Lab, Koita Centre for Digital Health, Ashoka University Medicine Expert Not in cards Claude Fable 5 758.0 Jul 2026 Jul 2026
SG-LegalClarify Models answer Singapore legal questions and ask for missing facts. Three model judges grade each conversation against rubrics written by lawyers. Ziwei Gu and colleagues Law Expert Not in cards Kimi K3 77.8% Nov 2026 Jul 2026
SMB Bench An agent runs the buying desk of a simulated fresh-produce wholesaler for a 90-day quarter, using SQL records, supplier quotes and memory files that are its only carryover between days. Scored on mean net profit per quarter across 18 tasks, compared with a fixed buying rule. ORO AI Running a business University Not in cards GPT-6 Astra $7,749 Nov 2026 Nov 2026
SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. Snorkel AI Law Expert Not in cards Gemini 4 Argon 46% Oct 2026 Oct 2026
SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. Snorkel AI Other professions Expert Not in cards Muse Wonder 27.8% Oct 2026 Oct 2026
SnorkelRevOps Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records. Snorkel AI Running a business Expert Not in cards Gemini 4 Argon 18.4% Oct 2026 Oct 2026
SpaceChem Bench Agents program reactors to assemble molecules in SpaceChem. The main ranking measures how close their solutions come to the community’s records. Wincon Research Puzzles and chess Competition Not in cards GPT-6 Astra 74.3 Nov 2026 Oct 2026
SuperCLUE-Project Models generate code for complete software projects. SuperCLUE Building whole projects Expert Not in cards o3-mini 82.1 Apr 2025
SuperCLUE-Science Models solve Chinese science reasoning problems. SuperCLUE Science questions University Not in cards o4-mini 80.8 Apr 2025
SuperCLUE-SWE Agents complete software engineering tasks in Chinese. SuperCLUE Fixing real repositories Expert Not in cards Claude Opus 4.8 73.3 Jun 2026
SWE-bench CLI Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result. Snorkel AI Fixing real repositories Research frontier Not in cards Claude Opus 5.5 16.6% Oct 2026 Oct 2026
SWE-CI Agents maintain a repository from a base commit toward a later reference commit over 100 commit pairs, looping through tests, requirement writing and code changes. SKYLENAGE AI Fixing real repositories Expert Not in cards Claude Opus 4.6 92% Mar 2026 Feb 2026
SynthBench Models answer survey questions. The index compares their answer distributions with human responses across four datasets, using random answers and human agreement as reference points. DataViking Social simulation Everyday Not in cards Qwen3.8 Flash 32.2 Apr 2026 Oct 2026
TeachArena Agents make teaching decisions, tutor students through dialogue, and carry out learning management system tasks. The evaluator checks conversation turns, artifacts, and final system states. The Hong Kong University of Science and Technology and Alibaba Other professions University Not in cards Claude Opus 4.8 0.80 May 2026 Jun 2026
TIS-100 Bench Agents write and optimize assembly programs for 55 puzzles. The ranking counts puzzles where a program improves on the community’s recorded tradeoffs in cycles, nodes, and instructions. Wincon Speed and optimization Competition Not in cards GPT-6 Astra 11.0 Nov 2026 Oct 2026
UniCode Models solve generated competitive programming problems using automated test cases. Harbor packages 399 problems with validated reference solutions. Xinyue Zheng and colleagues Competitive programming Competition Not in cards o4-mini 70.3% Oct 2025 Sep 2025
ViMU Models explain what a short video implies and select the cues that support their reading. They also identify rhetorical devices and social signals. National University of Singapore Video understanding Everyday Not in cards o4-mini 46.9% May 2026 Mar 2026
VisionArena-Bench Models answer visual questions from Arena users. A vision language model judges their answers to approximate human preference rankings. Arena General visual questions Everyday Not in cards Gemini 1.5 Pro 88 Dec 2024 Oct 2024
AgroMind (original) Models answer questions about farmland images across 13 tasks. They identify crops, assess crop health and interpret spatial relationships. Qingmei Li and colleagues General visual questions University Not in cards InternVL2-26B 35.8% May 2025 Dec 2024
AidanBench Models keep giving new answers to open-ended questions until answers become incoherent or repeat earlier ones. The count of fresh, coherent answers is the score, with no ceiling. Aidan McLaughlin (community) Creative writing Everyday Not in cards o1 3204.0 Aug 2024 Feb 2025
Biopharma Bench Agents take an employee seat inside one of 12 simulated biopharma companies and complete 71 professional assignments, such as a regulatory assessment, an out-of-specification investigation or an audit package. They must work out which records govern among contradictory files. A rubric grader checks 752 frozen criteria, with code checks for exact numbers. Raycaster Genomics and life sciences Expert Not in cards GPT-6 Astra 65.1% Sep 2026 Nov 2026
CLASHarness Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment. Google Research Desktop apps Everyday Not in cards GPT-6 Astra 28.7% Oct 2026 Sep 2026
Code2Video-Compass Multimodal models write programs that render complete videos from text, an image or a video. The default Overall board ranks models by their mean rank across quality metrics, so a lower score is better. Fudan University Video generation Everyday Not in cards Claude Opus 5.5 1.07 Oct 2026 Oct 2026
Compass Academic Leaderboard Models answer academic reasoning tasks. OpenCompass publishes model rankings and predictions across the underlying evaluations. OpenCompass Composite indices University Not in cards DeepSeek-V3 72.6% Dec 2024 Mar 2025
DeepPHY Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles. DeepPHY authors Abstract puzzles and logic Everyday Not in cards Claude 3.7 Sonnet 39% Aug 2025 Jun 2025
FailSafeQA Models answer finance questions over long documents while the query or the document is degraded, irrelevant or missing. An LLM judge scores robustness, context grounding and compliance. Writer Finance Expert Not in cards Gemini 2.0 Flash 0.76 Feb 2025 Mar 2025
FinFIRST Agents research financial questions and provide supporting sources. The benchmark grades individual requirements and reports how often an answer meets every requirement. Wenqing Wang and colleagues Finance Expert Not in cards GPT-5.6 Sol 71.5% Sep 2026 Sep 2026
GameXpert-Bench Agents build games from 97 briefs, repair bugs in 100 tasks, and improve 17 games over six requests each. Each track reports its own scores. Tencent Hunyuan and Lightspeed Studios Building whole projects University Not in cards Claude Opus 5 70.9 Aug 2026 Jul 2026
GeoGuessBench (panoramic) Models see four Google Street View images taken at right angles. They predict coordinates and report their confidence. Closer guesses earn more points. GeoGuessBench (panoramic) Spatial reasoning and counting Everyday Not in cards Gemini 3 Pro 4771.5 Feb 2026
GSM1k Models solve new grade school math word problems written to mirror GSM8K in style and difficulty. The gap between GSM8K and GSM1k accuracy shows overfitting or contamination. Scale AI Word problems School Not in cards Claude 3.5 Sonnet 96.6% May 2024 Aug 2024
HealthAgentBench V2 Agents complete 75 healthcare tasks across nine categories in terminal environments. Task-specific verifiers check outputs against hidden labels, with model grading for radiology report correction. Microsoft Research Medicine Expert Not in cards Claude Opus 5.5 57.3% Oct 2026 Sep 2026
LEXam-hard Agents with legal research tools answer the 518 LEXam open questions that strong open models score lowest on. Questions answerable from memory or unanswerable from their text are removed. Joel Niklaus (LEXam co-author) Law Research frontier Not in cards Inkling 40.8% Sep 2026 Jul 2026
LOKI Models distinguish synthetic content from real content across images, videos, audio, text, and 3D data. The benchmark checks real-or-synthetic labels and annotated abnormal details, with separate tasks for explaining anomalies. OpenDataLab General visual questions Everyday Not in cards LLaVA-OneVision 72B 60.8% Oct 2024 Aug 2024
M-IFEval Models follow instructions in French, Japanese, and Spanish. Automatic checks test whether each answer satisfies the requested constraints. Lightblue Chat and instructions in many languages Everyday Not in cards o1-preview 86.6% Jul 2024 Sep 2024
MemGUI-Bench Agents remember details while working across Android apps. Tasks include recording audio and creating calendar events. A model checks the actions and results across repeated attempts. Zhejiang University, Nankai University, Chinese University of Hong Kong, Shanghai Jiao Tong University, and vivo AI Lab Phones Everyday Not in cards Gemini 3.6 Flash 73.4% Feb 2026 Aug 2026
MVDriveBench Models answer driving questions using synchronized videos from cameras facing different directions. The authors compare surround-view results with front-camera results. Shunming Liu and colleagues Embodied reasoning Everyday Not in cards GPT-5.5 80.1% Nov 2026 Apr 2026
OBI-Bench Models locate inscriptions, rejoin bone fragments, classify characters, retrieve matches, and explain glyphs. The authors publish separate scores for each task. Shanghai Jiao Tong University Documents, charts and text in images Expert Not in cards GPT-4o 0.61 Oct 2024 Aug 2024
OccuBench Agents do 100 professional task scenarios across 10 industries and 65 domains. Language world models simulate each domain's tools so tasks stay solvable. Qwen (Alibaba) Tasks from real jobs Expert Not in cards GPT-5.2 79.6% Apr 2026 Mar 2026
OmicsBench v1 Agents analyze molecular data across eight benchmarks. LatchBio reports pass rates for 741 short tasks and 43 long tasks. LatchBio Genomics and life sciences Expert Not in cards GPT-6 Astra 40.8% Oct 2026 Sep 2026
OPT-BENCH Agents iteratively improve solutions to 20 machine-learning tasks and 10 combinatorial optimization problems using execution feedback. Win counts, invalid-solution rates, ranks, and improvement rates measure optimization performance. Tongji University, Shanghai AI Laboratory, Nanjing University and others Scientific code University Not in cards DeepSeek-V3.1 0.69 Jun 2025 Aug 2025
OVO-S-Bench Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos. Shanghai AI Laboratory (InternLM) Video understanding Everyday Not in cards Gemini 3.1 Pro 59.2% Jun 2026 Apr 2026
ParlaCAP Test Datasets Models read parliamentary speeches and choose a policy topic or Other. The four-language test compares predictions with human labels on 3,443 speeches. ParlaCAP project Reading in many languages School Not in cards GPT-5 0.69 Jun 2025 Sep 2025
PISA-Bench Models answer illustrated PISA questions in six languages. Parallel questions let researchers compare language effects on reading, science and mathematical reasoning. Patrick Haller and colleagues General visual questions School Not in cards Qwen2.5-VL-72B 63.3% Oct 2025 Oct 2025
PoETa v2 Models answer multiple-choice and open-ended questions in Portuguese. The hub port selects native Portuguese tasks covering inference, reading, exams, proverbs and toxicity. PoETa v2 team Knowledge in many languages University Not in cards GPT-4.1 76.2 Sep 2025 Apr 2025
PWNBench v0.1 Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth. Novee Finding and exploiting real bugs Expert Not in cards Grok 4.6 63.3% Aug 2026 Oct 2026
Retrobench v0.4 Agents reconstruct C from five Nintendo 64 assembly targets. The organizer scores the average percentage of matching bytes under a budget of 25 compilations per target. Sinan Onur Altınuç Function-level problems Expert Not in cards Gemini 4 Carbon 99.9% Aug 2026 Nov 2026
RSI-Inference Bench Coding agents add model support to inference engines, improve serving speed and fix performance regressions. The harness builds and runs each submission. Apple Machine learning engineering Expert Not in cards Claude Sonnet 5.5 2,044 Oct 2026 Oct 2026
SATBench Models decide whether a set of constraints in a narrative puzzle can all hold at once. A Boolean satisfiability solver supplies the correct label. Anjiang Wei and colleagues Language reasoning University Not in cards o4-mini 89.3% May 2025 Apr 2025
Sinhala Language Proficiency & Linguistic Mastery Models answer Sinhala prompts about grammar, style, conversation, and literature. Three judges score the responses without seeing the model names. Nadew Damruwan Chat and instructions in many languages School Not in cards GPT-6 Astra 95.9 Nov 2026 Sep 2026
TCSExam-Lite Agents solve 20 theoretical computer science tasks, five per track. The default solver protocol gives hidden feedback only after submission. Moonshot AI AI research tasks Research frontier Not in cards Claude Opus 5.5 72.3 Nov 2026 Oct 2026
The Aggregate Exotic Fact Recall Models recall exact fields from five public registries without looking them up. The bank mixes familiar entries with randomly sampled obscure ones. Mikhail Doroshenko (The Aggregate) Trivia and open questions Expert Not in cards Claude Fable 5.2 75% Sep 2026 Nov 2026
Thrive Holdings helpdesk ticket resolution benchmark Agent investigates IT support requests, takes action and decides when to escalate to a technician. Scored by pass rate. Thrive Holdings Tasks from real jobs University Not in cards GPT-6 Astra 61.5% Oct 2026 Sep 2026
Video-SafetyBench Video-language models receive 2,264 synthesized videos paired with a harmful or a benign-looking query across 13 unsafe categories and 48 subcategories. An LLM judge scores harm with a confidence-weighted metric called RJScore. BAAI FlagEval team Jailbreaks and prompt injection Everyday Not in cards Claude 3.7 Sonnet 1.30 May 2025 Mar 2025
AML Bench (Veris AI) Agents investigate suspicious activity at a simulated bank. A model judge checks whether the recorded service requests and final case disposition satisfy every requirement. Veris AI Finance Expert Not in cards GPT-6.1 Sol 70.8% Oct 2026 Nov 2026
AstaBench Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem. Allen Institute for AI (Ai2) Research science Expert Not in cards Claude Opus 4.7 58% Aug 2025 Apr 2026
BhashaBench-Multi (Krishi) Models answer agricultural exam questions in 22 Indian languages. The evaluation checks the selected answer against a key for each translated question. BharatGen Knowledge in many languages University Not in cards Jun 2026
cad-spec Models write CadQuery code for mounting plates from dimensioned specifications. The evaluator measures each finished solid against nine requirements. cad-spec team CAD models University Not in cards DeepSeek-V3 78% Aug 2026 Feb 2026
CompassRank OpenCompass ranks open and API language models on its own evaluation sets and publishes monthly rankings. Model makers submit a repository link or API endpoint to be tested. OpenCompass (Shanghai AI Laboratory) Composite indices Expert Not in cards Claude Opus 5 83.3 Sep 2023 Aug 2026
Crest Arcade Nonograms Models reconstruct a picture from row and column clues in one reply. The score averages the share of exact solutions across six tiers. Glimmer Index Puzzles and chess Everyday Not in cards Gemini 4 Neon 91.7% Nov 2026
Crest Arcade Ravelin Models build towers to protect a keep through waves of attackers. The score compares wins with the reference time plus a penalty for lives lost. Glimmer Index Video games Everyday Not in cards Gemini 4 Neon 106.8 Nov 2026
Crest Arcade Starjar Models place pieces in a jar so matching pieces touch and merge. The score compares their total points with a reference bot on identical jars. Glimmer Index Video games Everyday Not in cards Gemini 4 Neon 114.5 Nov 2026
Crest Arcade Wildhaven Models select a team of animals and choose where each lands in a village. The score measures the share of village strength they destroy. Glimmer Index Video games Everyday Not in cards Gemma 3 4B 100% Nov 2026
Cresta-Bench Contact-center agents take simulated calls for lending, health and airline lines. Tool calls get deterministic checks and LLM judges grade the rest. Cresta Customer service Everyday Not in cards Claude Opus 5.5 82.6% Oct 2026 Oct 2026
FrontierBioML 25 machine learning tasks in computational biology, across antibody development, protein engineering, structural biology, binder design and functional genomics. An agent gets real experimental data and a brief, works offline for up to six hours and submits a prediction script. A sealed grader runs the script on held-out data, such as new antibody families or new protein targets, and scores each task from 0 at a simple baseline to 100 for a perfect answer. Valen Labs Genomics and life sciences Expert Not in cards Claude Mythos 5.2 40.3 Nov 2026 Nov 2026
GeoGuesser environment (FineEnvs) Agent placed at unknown Mapillary street-level location looks, walks, pins candidates and guesses coordinates. Scored by a distance-based GeoGuessr-style reward from 0 to 1. FineEnvs (Hugging Face community) General visual questions Everyday Not in cards Claude Sonnet 5 0.64 Sep 2026 Jun 2026
GSM-Infinite Models solve generated arithmetic problems across different context lengths and numbers of reasoning steps. The benchmark varies how many irrelevant details appear alongside each problem. Carnegie Mellon University and Meta AI Reasoning over long documents School Not in cards DeepSeek-R1 8534.9 Feb 2025 May 2025
HealthQA-BR Models answer Portuguese multiple-choice questions about clinical care and public health in Brazil’s Unified Health System. Andrew Maranhão Ventura D'addario Medicine Expert Not in cards GPT-4.1 86.6% Jun 2025 Apr 2025
HoloBench Models answer questions by combining facts across database rows written as sentences. A judge gives full, partial, or zero credit for each expected result. Megagon Labs Reasoning over long documents School Not in cards Gemini 2.0 Flash 59.6% Oct 2024 Feb 2025
Incident Arena Agents diagnose and repair incidents in running services. The score averages the repair success rate across 20 incidents. Abundant AI and collaborators Terminal tasks Expert Not in cards GPT-6 Astra 64.3% Sep 2026 Oct 2026
InsureBench Multi-functional LLMs Models read insurance files and answer questions about claims, coverage, and actuarial decisions. InsureThing combines automatic answer checks with rubric grades for judgment tasks. InsureThing Finance University Not in cards Claude Opus 4.8 96.3% Jun 2026 Jun 2026
ITSMBench Agents operate an IT service desk across 53 conversations with simulated colleagues. The verifier grades the resulting database state, including permissions and required actions. Vibrant Labs Customer service University Not in cards Grok 4.6 0.70 Aug 2026 Aug 2026
LegalCiteBench Models recover and verify citations from United States judicial opinions without external lookup. The benchmark also tests whether models match cases and correct citation errors. Sijia Chen and colleagues Law University Not in cards Claude Sonnet 4.5 0.78 Feb 2026 Oct 2025
Letta Leaderboard Agents read, write and update core and archival memory while answering synthetic factual questions. A model judge scores answers and penalizes unnecessary memory operations. Letta Learning from the context Expert Not in cards GPT-5.2 Codex 93.0 May 2025 Mar 2026
LingOly-TOO Models solve Linguistics Olympiad problems whose spelling patterns have been systematically changed while preserving the solution logic. Answers receive exact-match credit with no partial credit. The evaluation compares performance on original and obfuscated problems. University of Oxford and collaborating linguistics researchers Reasoning in many languages Competition Not in cards GPT-5 46.7% Mar 2025 Sep 2025
LLM Uncertainty Bench Models answer questions across five language tasks. The original benchmark measures accuracy and uncertainty, while the port checks only the selected answer. Tencent AI Lab, University College London, and University of Macau Language reasoning University Not in cards Yi-34B 81.5% Jan 2024 Nov 2023
LongProc Models carry out long procedures such as converting HTML tables, tracing paths and tracking beliefs. The evaluator checks outputs as the required procedure grows. Xi Ye and colleagues Reasoning over long documents University Not in cards Gemini 1.5 Pro 74.2 Jan 2025 Oct 2024
MM-FinEval Models forecast financial outcomes from earnings call transcripts, slides, and audio. The authors compare 19 models using directional accuracy and absolute error. Northwestern University and collaborators Finance University Not in cards GPT-5.5 88% Sep 2026 May 2026
MUSE (CAD assemblies) Models turn design specifications into CAD assemblies. A program checks the geometry. A vision model grades function, manufacturability, and ease of assembly. Xiaoyu Dong and colleagues CAD models University Not in cards GPT-5.5 52.4% May 2026 Apr 2026
Next.js Agent Evals Coding agents with their models do real Next.js tasks. The board reports success rate with and without an AGENTS.md file, plus duration and cost. Vercel Web and app front ends Expert Not in cards Claude Fable 5.1 100% Oct 2025 Oct 2026
OfficePrism Agents resolve Chinese workplace tickets using one business system at a time. The evaluator checks outcomes across 420 tasks in seven business functions. Alibaba Group and collaborators Documents and spreadsheets University Not in cards Kimi K3 76.4% Nov 2026 Jul 2026
OTelBench Coding agents add OpenTelemetry tracing, metrics and logging to codebases in several languages. Tests check the instrumentation. Quesma Fixing real repositories Expert Not in cards Claude Opus 4.5 29% Jan 2026 Dec 2025
PharmaBench (micro1) Agents write clinical trial documentation inside 8 synthetic drug programs, across 120 tasks written by biostatisticians, medical writers, safety physicians and regulatory staff. micro1 Medicine Expert Not in cards Kimi K3.1 Pro 17.8% Oct 2026 Nov 2026
RECON Research agents find and verify several facts about identified people on the open web without mixing up people with similar names. Sixtyfour Deep research Expert Not in cards Grok 4.20 Multi-agent 36.2% Apr 2026 Sep 2026
RPC-Bench Models read computer science papers and answer reviewers’ questions. A model judge grades correctness, completeness, and concision against verified review and rebuttal pairs. Yelin Chen and colleagues Research science Expert Not in cards GPT-5 37.5 Apr 2026 Dec 2025
RSNA Knee LLM Label Extraction Models extract twelve findings from knee MRI reports in a batch. The grader compares the labels with reference findings using macro ROC AUC. Kirderf Medicine University Not in cards Claude Opus 5.5 0.89 Oct 2026 Sep 2026
ScrabbleBench Models choose a word from a Scrabble rack and board. A solver finds its best legal placement. The score averages the four seats in self-play. Sunny Balasubramanian Puzzles and chess Everyday Not in cards GPT-5 152.8 Aug 2025 Sep 2025
Shopping MMLU Models complete 57 shopping tasks involving product concepts, reasoning, user preferences, and several languages. Task-specific metrics check classification, ranking, extraction, translation, and generated text. Skill scores average the corresponding task metrics. Amazon and academic collaborators Running a business Everyday Not in cards Claude 3 Sonnet 72.6 Oct 2024 Apr 2024
SuperCLUE-CoT Models solve Chinese reasoning problems that require several steps. SuperCLUE Language reasoning University Not in cards o1 77.7 Jan 2025
SuperCLUE-Math6o Models solve Chinese elementary school math olympiad problems. SuperCLUE Competition problems with answers Competition Not in cards DeepSeek-R1 82.2 Jan 2025
SuperCLUE-VLM Vision language models interpret images and answer questions in Chinese. SuperCLUE General visual questions Everyday Not in cards Gemini 4 Argon 89.0 Nov 2026
SWE-Bench Pro Private Agents resolve held-out software issues from the private SWE-Bench Pro split. The leaderboard reports the percentage of resolved tasks. Scale AI Fixing real repositories Expert Not in cards Muse Spark 1.1 51.5% Sep 2025 Jul 2026
Taste-Bench Models choose the better next step at 502 decision points in software engineering and research trajectories. A question scores only when both option orders receive the correct answer. The overall score equally weights engineering and research. City University of Hong Kong and Microsoft Machine learning engineering Expert Not in cards GPT-5.6 Sol 59.7% Sep 2026 Jul 2026
Tencent WorkBuddy Bench Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search. Tencent Fixing real repositories Expert Not in cards Claude Opus 5.5 81.6% Jul 2026 Oct 2026
ToRR Models answer questions over tables given in several formats, and the board checks how much accuracy changes when the table format or prompt changes. IBM Research (on HELM) Language reasoning University Not in cards Claude 3.5 Sonnet (October 2024) 49.9 Feb 2025 Mar 2025
Vals Smith: torvalds/linux Agents modify torvalds/linux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Sol 74.1% Aug 2026 Aug 2026
VANTAGE-BENCH Models answer questions, describe events, and locate objects or actions in infrastructure-camera footage. Task-specific metrics measure answer accuracy, caption quality, localization, and tracking. NVIDIA Video understanding Everyday Not in cards GPT-6 Astra 75.3 Sep 2026 Sep 2026
VoxParity Voice agents choose tool calls after listening to speech. Paired recordings keep the words fixed while changing an audible cue that affects the correct action. Bhavik Mangla Audio understanding and voice Everyday Not in cards Gemini 3.7 Flash 0.23 Sep 2026 Sep 2026
WorkBuddyBench-Office Agents edit office files and reconcile records across a workspace. One task requires updating an eleven-slide presentation from a spreadsheet while preserving unrelated slides. Tencent Documents and spreadsheets University Not in cards Kimi K3.1 Pro 86.4% Jul 2026 Oct 2026
WorkplaceAgents Agents produce finished work products for 200 professional tasks across 96 occupations in 19 sectors. Harbor grades deliverables with code checks and rubrics. Snorkel AI Tasks from real jobs Expert Not in cards Gemini 4 Argon 21.2% Oct 2026 Oct 2026
AI AgriBench Models answer 416 agronomy questions from agricultural extension publications. Three judges compare each answer with an expert-reviewed reference across four metrics. AI AgriBench Consortium, University of Illinois Urbana-Champaign Tasks from real jobs University Not in cards GPT-5 98.1 Jan 2026 Nov 2025
ATM-Bench Agents answer questions about a person’s history using images, videos, and email records. The Harbor package contains 31 hard questions with schema-guided memory. Jingbiao Mei and colleagues Finding facts in long inputs University Not in cards Claude Opus 4.5 86% Mar 2026 Jun 2026
AutoResearchBench Agents search the scientific literature, either tracking down one target paper through multi-step probing or collecting every paper that meets given conditions. Lei Xiong, Kun Luo, Zheng Liu et al. Deep research Expert Not in cards Claude Opus 4.6 9.39% Apr 2026 Mar 2026
Beyond Survival Models play social deduction games with human-aligned strategies and interaction patterns. Beyond Survival authors Multi-agent games Everyday Not in cards GPT-5 58.7% Nov 2026 Sep 2025
Bill of Lading OCR Benchmark Models transcribe 25 synthetic bills of lading. A fixed Gemini model extracts 21 fields from each transcription for comparison with the known values. okraPDF Documents, charts and text in images Everyday Not in cards Gemini 3.1 Flash-Lite 94.9% Aug 2026 Aug 2026
BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. Alibaba Group (Qwen, Alibaba Token Hub) + Zhejiang University + CUHK CAD models Expert Not in cards GPT-6 Astra 79.9 Oct 2026 Sep 2026
BlueBench Agents solve enterprise tasks across knowledge, coding, security and business workflows. The benchmark aggregates capability scores from multiple public task suites. IBM Research General assistants Expert Not in cards GPT-4.1 64.6 Sep 2025 Apr 2025
CadEval Models write OpenSCAD code for single mechanical parts from text descriptions. Rendered parts are checked against reference geometry, scored by pass rate. Will Patrick CAD models University Not in cards o3 74% Apr 2025 Apr 2025
ChipBench Models write Verilog modules, debug circuits and generate reference models. Simulations compare the outputs with reference implementations. Zhongkai Yu and colleagues Electronics Expert Not in cards Claude Opus 4.5 30.5% Jan 2026 Dec 2025
ClawBench V2 Browser agents do 130 new everyday tasks on live websites under six supported harnesses. Request interception and a more lenient LLM judge score each run. TIGER-AI-Lab (University of Waterloo) and NAIL Group Web browsing Everyday Not in cards Claude Opus 4.7 44.6% May 2026 Apr 2026
CorporateBench Agents answer questions over the email archives of four synthetic companies, up to 232,692 documents. Answers are checked against each company's ground-truth knowledge graph. Epiq AI Labs Reasoning over long documents Expert Not in cards GPT-6.1 Sol 82.2% Aug 2026 Oct 2026
Embodied Agent Interface Models interpret goals, decompose them into subgoals, sequence actions, and predict state transitions in BEHAVIOR and VirtualHome. The evaluator checks task completion and distinguishes hallucinations, invalid actions, and planning errors. Stanford Vision and Learning Lab Embodied reasoning Everyday Not in cards o1-preview 70.3 Oct 2024 Sep 2024
English to Icelandic Test Suite for Syntactic Divergence Models translate 345 English sentences into Icelandic. Human evaluators judge whether each translation uses an acceptable Icelandic construction for the targeted English syntax. Árni Magnússon Institute for Icelandic Studies and University of Iceland Translation Everyday Not in cards GPT-5.5 93.4% Jun 2026 May 2026
European LLM Leaderboard (OpenGPT-X) Models answer exam, science, commonsense, truthfulness, and math questions across European languages. The leaderboard also includes multilingual reading comprehension. OpenGPT-X Knowledge in many languages School Not in cards Gemma 7B 56.7% Oct 2024 Jul 2024
Factory Code Review Benchmark Models review real pull requests. Factory measures precision and recall against reference issues and reports mean F1. Factory Fixing real repositories Expert Not in cards GPT-5.2 60.5% Apr 2026 Apr 2026
Galileo LLM Hallucination Index RAG Special Models answer questions using short, medium, and long passages. Galileo judges whether each answer follows the supplied context. Galileo Faithful to sources School Not in cards Claude 3 Opus 0.99 Jul 2024 Jun 2024
GMAI-MMBench Vision-language models answer medical multiple-choice questions over many imaging modalities, clinical tasks and departments, at several perceptual granularities. Shanghai AI Laboratory (uni-medical) Medicine Expert Not in cards DeepSeek-VL-7B-Chat 43.4% Aug 2024 Apr 2024
Holdings-IndividualTaxBench 88 real client document packages turned into Form 1040 fields. Fact-level accuracy against practitioner-reviewed returns, with an LLM judge matching values. Thrive Holdings Tasks from real jobs Expert Not in cards Claude Opus 5.5 76.3% Oct 2026 Sep 2026
IITP-Legal Models translate 5,000 English legal sentences into Hindi. The suite includes 100 sentences at each length from five through 54 words. The original ranking uses BLEU. AI and NLP Research Group, IIT Patna Translation University Not in cards Gemini 2.5 Pro 33.4 Jun 2025 Jun 2025
ITBench Agents handle IT automation tasks in Kubernetes environments: site reliability incidents, compliance and security checks, and cloud cost work. IBM Research Terminal tasks Expert Not in cards Gemini 3 Pro 58.3% Feb 2025 Nov 2025
LegacyShop Models maintain a Python repository for an online shop. Tasks test bug fixes, memory reasoning, complexity analysis, and whether to ask for clarification. Iseoluwa Olowogoke Fixing real repositories University Not in cards Claude Opus 5.5 0.91 Oct 2026 Oct 2026
ManyIH-Bench Models resolve conflicts among instructions at up to twelve privilege levels. A response must satisfy all winning instructions and any applicable coding tests. Johns Hopkins University and collaborators Following precise instructions University Not in cards GPT-5.4 60.9% Apr 2026 Jul 2026
MBABench 2.0 Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar. Columbia University (Namkoong Lab), hosted by Snorkel AI Finance Expert Not in cards GPT-6 Astra 15.8% Oct 2026 Sep 2026
MDPBench Document parsers read 3,400 digital and photographed document images in 17 languages. Public and private splits limit leakage. Huazhong University of Science and Technology (MultimodalOCR group) Documents, charts and text in images Everyday Not in cards Gemini 3 Pro 86.4 Mar 2026 Aug 2026
MedHELM Models run a suite of medical tasks chosen by clinicians across five categories, from clinical decision support and note writing to patient messages and administration. Stanford CRFM (HELM) + Stanford Medicine Medicine Expert Not in cards GPT-5 70.3 Feb 2025 Aug 2025
MoReBench Models explain decisions in moral dilemmas. A judge scores their reasoning against criteria written by moral philosophers. MoReBench team with Scale AI Deception and alignment Everyday Not in cards Qwen3-235B-A22B-Thinking 77.6 Oct 2025 Sep 2025
MoReBench-Theory Models reason about moral decisions under five ethical frameworks. A judge checks their reasoning against framework-specific criteria. MoReBench team with Scale AI Deception and alignment University Not in cards GPT-5 79.1 Oct 2025 Sep 2025
New Synonym Benchmark Models choose the one Swedish word among five options that does not mean the same thing as the target word. Drasko Markovic Reading in many languages Everyday Not in cards Gemma 4 31B IT 99.3% Apr 2026 Aug 2026
NPCBench Models follow patients with nasopharyngeal carcinoma through diagnosis and care. The main patient-level measure requires correct decisions at successive stages. Pengkai Wang and colleagues Medicine Expert Not in cards Claude Sonnet 4.6 23% Nov 2026 Apr 2026
OlymMATH Models solve Olympiad-level mathematics problems in English or Chinese. The evaluator checks final answers and reports pass@1 and consistency across repeated samples. Renmin University of China Competition problems with answers Competition Not in cards Gemini 2.5 Pro 74.2% Mar 2025 Apr 2025
P3B3 Models hold conversations in Portuguese under three language instructions. The judge measures how closely replies follow European or Brazilian usage. NOVA University of Lisbon Chat and instructions in many languages Everyday Not in cards Gemini 3 Flash 99.8 Mar 2026 Apr 2026
ParlaSent Models classify the sentiment of parliamentary sentences as positive, neutral, or negative. Two additional test sets contain 2,600 English and 2,600 South Slavic sentences. Michal Mochtak and the ParlaSent team Reading in many languages Everyday Not in cards GPT-5 0.77 Sep 2023 Sep 2025
PhysVista Vision-language models perceive, reason about and judge whether videos follow real-world physics. The benchmark also tests if they can judge the physical realism of generated videos. University of Science and Technology of China Physical prediction Everyday Not in cards GPT-5.2 85% Sep 2026 Sep 2026
Polaris-Bench Models solve the same 53 visual reasoning tasks in Cartesian and polar layouts. The benchmark compares accuracy across the paired layouts. Google DeepMind Spatial reasoning and counting School Not in cards GPT-5.2 39.2% May 2026 Apr 2026
Qwen-Image-Bench Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images. Qwen (Alibaba) Video generation Everyday Not in cards GPT-Image-2 64.7 May 2026 Feb 2026
RBench-M Models answer 665 graduate-level multimodal reasoning questions in English and Chinese. Top-1 accuracy compares selected answers with the answer key. Meng-Hao Guo and collaborators General visual questions Expert Not in cards GPT-5 69.9% May 2025 Oct 2025
REAL Agents complete 112 tasks on replicas of 11 websites. State checks grade actions, while a model judge checks retrieved answers against references. AGI, Inc. Web browsing Everyday Not in cards Claude 3.7 Sonnet 41.1% Apr 2025 May 2025
RealmPathology Models interpret real anatomic pathology reports and must not claim a diagnosis, stage or treatment the report does not support. Rubrics have 772 positive and 299 penalty criteria. micro1 Medicine Expert Not in cards Claude Fable 5.1 78% Oct 2026 Oct 2026
Senior Engineer Benchmark (Every Checks) Agents repair Proof from the March 17 outage snapshot. Every reports the percentage of 48 checks each solution passes. Every Fixing real repositories Expert Not in cards Claude Opus 5.5 85% Oct 2026 Oct 2026
SportsEval Models translate 300 English passages about five sports into Icelandic. The scorer checks the Icelandic terms. Humans also judged 37 passages per system at WMT25. Árni Magnússon Institute for Icelandic Studies Translation University Not in cards Gemini 2.5 Pro 90% Jun 2025 Jun 2025
StockBench Agents trade stocks over a fixed evaluation window. StockBench ranks models by the mean of final return, maximum drawdown and Sortino ratio. The score is the board's rank, so a lower score is better. StockBench authors Finance Expert Not in cards Kimi K2 1.00 Oct 2025 Sep 2025
StoreBench Agents manage a simulated apparel store in Medusa. They order inventory, change prices, ship purchases, and process refunds as demand changes. AfterQuery Running a business University Not in cards Claude Fable 5.2 0.83 Oct 2026 Nov 2026
Suicide Memory Bench AI systems, each a model in its own app or coding harness, talk with 128 fictional personas over 29 chats. Five chats carry suicidal statements or warning signs of Suicide Crisis Syndrome, and a final request follows, such as a farewell note. Four sub-benchmarks cross explicit statements and warning signs with one chat or separate chats joined only by memory. A model judge rates each reply on a 0 to 4 safety scale. Prosaic Intelligence Wellbeing and sycophancy Everyday Not in cards Gemini 4 Argon 1.41 Nov 2026 Oct 2026
SuperCLUE-Hard Models follow precise instructions, solve reasoning problems and plan tool calls in Chinese. The report gives separate scores for each ability. SuperCLUE Reasoning in many languages University Not in cards o1 76.7 Dec 2024 Mar 2025
SuperCLUE-LongContext Models retrieve information across multiple turns with long Chinese context. SuperCLUE Finding facts in long inputs School Not in cards Qwen4-Max 51.4 Nov 2026
SuperCLUE-WebCoding Models implement web interfaces from specifications. SuperCLUE Web and app front ends University Not in cards MiniMax-M2.1 75.7 Dec 2025
SveaKnow Culture Models give short answers to factual questions about Swedish history, traditions, media, education, and culture. Drasko Markovic Knowledge in many languages School Not in cards Llama 3.3 70B 95.3% Apr 2026 Aug 2026
TestPrism Coding agents write tests for 300 tasks. A test passes only if it fails on the starting code, accepts every valid implementation and rejects every invalid one. Nanjing University (NJU-LINK Lab) Function-level problems Expert Not in cards Kimi K3 33.3% Oct 2026 Sep 2026
Time Horizon Index: KSP Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions. Vals AI Video games Expert Not in cards Claude Opus 5.5 91.3% Jul 2026 Sep 2026
tr_gender_bias Models translate 200 Turkish passages into English. Set B scores pronoun accuracy on 82 passages where a name or kinship term identifies gender. Boğaziçi University Translation Everyday Not in cards Gemma 4 31B IT 100% Jun 2026 May 2026
TrafficImag Models edit roadside traffic videos to show a different sequence of events. The evaluator checks routes, interactions, and preservation of the surrounding scene. Xiangyu Li and colleagues Video generation Everyday Not in cards GPT-Image-2 90.0 Sep 2026 Sep 2025
Vals Smith: apache/spark Agents modify apache/spark to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Sol 84.5% Aug 2026 Jul 2026
Vals Smith: astral-sh/uv Agents modify astral-sh/uv to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Fable 5 66% Aug 2026 Jul 2026
Vals Smith: jax-ml/jax Agents modify jax-ml/jax to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Sol 79.6% Aug 2026 Jul 2026
Vals Smith: oven-sh/bun Agents modify oven-sh/bun to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Kimi K3 54.2% Aug 2026 Jul 2026
Vals Smith: palantir/blueprint Agents modify palantir/blueprint to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Fable 5 93.2% Aug 2026 Jul 2026
Vals Smith: sgl-project/sglang Agents modify sgl-project/sglang to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Terra 78% Aug 2026 Jul 2026
Vals Smith: vercel/next.js Agents modify vercel/next.js to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Sol 74.3% Aug 2026 Jul 2026
Which AI Model Knows Israel Best? Models answer Hebrew questions about everyday life in Israel. The organizers check each answer against an official source and count refusals separately. Skills IL Knowledge in many languages Everyday Not in cards Grok 4.7 99.3% Oct 2026 Oct 2026
WitnessBench Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels. Fleet AI Puzzles and chess Expert Not in cards Claude Opus 5.5 46.5% Sep 2026 Sep 2026
WMT25 EEA Terminology Models translate English regulatory sentences into Icelandic. The scorer checks whether they use the official Icelandic terms from the European Economic Area agreement. Árni Magnússon Institute for Icelandic Studies and University of Iceland Translation University Not in cards Gemini 2.5 Pro 46.5% Jun 2025 Jun 2025
Agent Initiative Benchmark Agents inspect workplace situations and decide whether to act without an explicit task. Two model judges score responses to 50 situations from 0 to 10. Deepnote Simulated companies Expert Not in cards GPT-6 Astra 8.97 Nov 2026
AGI-Eval Community Text to Image Models generate images from text. The board reports objective, subjective and combined scores. AGI-Eval (Meituan) Image generation and editing Everyday Not in cards Gemini 3 Pro Image Preview (nano-banana-pro) 0.64 Feb 2025 Mar 2025
Aider Refactoring Benchmark Models refactor large Python methods while preserving their implementation. The benchmark checks whether edits complete correctly. Aider Fixing real repositories University Not in cards Claude 3.5 Sonnet (October 2024) 92.1% Dec 2023 Oct 2024
ALIVE-Bench Agents interpret social cues and act in simulated conversations. The evaluator tracks what agents perceive, what they infer, and how their actions affect other characters. Tsinghua University and collaborators Simulated robots Everyday Not in cards Gemini 3 Pro 80.4% Nov 2026 Feb 2026
AnvilBench Models write compute kernels for Apple Silicon. Tasks cover operations used in attention, decoding, and audio processing. Vikram Tiwari GPU kernels Expert Not in cards Gemini 4 Argon 0.59 Oct 2026 Nov 2026
AppWorld Test-Challenge Agents complete 417 app tasks that require Amazon or Gmail APIs held out from training. The score is the percentage passing every task test. Stony Brook University and Allen Institute for AI Apps and MCP servers Everyday Not in cards GLM-5.3-Flash 92.8% Jul 2024 Aug 2026
AppWorld Test-Normal Agents complete 168 everyday app tasks from familiar applications by calling APIs through code. The score is the percentage passing every task test. Stony Brook University and Allen Institute for AI Apps and MCP servers Everyday Not in cards GLM-5.3 94% Jul 2024 Aug 2026
ASI-Bench AI systems try to explore unknown problems, create new knowledge and turn new ideas into verifiable results, rather than apply learned knowledge. Apexin AI Research science Research frontier Not in cards GPT-5.6 Sol 55.8 Aug 2026 Aug 2026
AutoResearchExam Agents get 24 hours on a CPU or GPU machine to improve solutions to open-ended ML research tasks through experiments. The score is the area under the improvement curve on a hidden test set, rewarding both quality and speed. Bespoke Labs Machine learning engineering Research frontier Not in cards Claude Opus 5.5 0.61 Sep 2026 Sep 2026
BenGER Models write German legal analyses for 596 exam-style cases and 531 doctrinal reasoning tasks. A rubric judge scores legal conclusions, reasoning, and presentation against expert references. Case scores map to the German 0-to-18 grading scale, with four points required to pass. Technical University of Munich, Ludwig Maximilian University of Munich, University of Konstanz, and Saarland University Law Expert Not in cards Gemini 3.1 Pro 71.1 May 2026 Apr 2026
Bio-PPI Risk: Serous vs Endometrioid Models estimate a cancer subtype probability from tumor protein measurements and protein interactions. The evaluation measures discrimination and probability accuracy. Neha Medicine Expert Not in cards Claude Opus 5.5 0.22 Nov 2026 Oct 2026
BioSimBench Models predict which genes change after drugs, gene knockdowns, and immune signals across four prediction tracks containing 60, 60, 60, and 27 cases. The overall score averages perturbation-specific F1 across those tracks on a zero-to-one scale. CellType Genomics and life sciences Expert Not in cards Gemini 4 Argon 0.18 Nov 2026 Oct 2026
CAD Arena Agents rebuild licensed engineering drawings as editable CAD parts through the scripting APIs of NX, SolidWorks, Onshape, Fusion or Build123d. A verifier scores geometric match. Normal Research CAD models Expert Not in cards Gemini 4 Argon 0.75 Sep 2026 Sep 2026
CAPR-am Models read an Amharic scenario and choose the best matching proverb from four options. The authors score exact choices. Haverford College Reasoning in many languages Everyday Not in cards Gemma 3 27B 41.3% Jul 2026 Aug 2025
CAPR-ar Models read an Arabic scenario and choose the best matching proverb from four options. The authors score exact choices. Haverford College Reasoning in many languages Everyday Not in cards Gemma 3 27B 70.1% Jul 2026 Aug 2025
CAPR-ja Models read a Japanese scenario and choose the best matching proverb from four options. The authors score exact choices. Haverford College Reasoning in many languages Everyday Not in cards Aya Expanse 32B 93% Jul 2026 Aug 2025
CAR-bench Agents act as an in-car voice assistant that talks with an LLM-simulated driver and uses 58 connected tools for navigation, productivity, charging and vehicle control under set policies. Besides ordinary tasks, hallucination tasks remove a needed tool or fact, and disambiguation tasks need a clarifying question or a lookup. Scores reward consistent success across repeated runs. BMW Group and University of Augsburg General assistants Everyday Not in cards GPT-5.6 Sol 70% Jan 2026 Jul 2026
CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. The University of Hong Kong Video understanding Expert Not in cards Claude Opus 5.5 76.6 Oct 2026 Sep 2026
Claw-Eval-Live Agents complete 105 workflow tasks in 17 families drawn from live ClawHub workflow signals, scored on both the workflow and the final workspace state. Claw-Eval-Live team (Li, Tang et al.) General assistants University Not in cards Claude Opus 4.6 66.7% Apr 2026 Mar 2026
Code4Scene Agents build and edit Unreal Engine scenes. The evaluator inspects the scene itself for correct edits, intact assets, and physical validity. Xiaokang Ye and colleagues Image generation and editing University Not in cards GPT-6 Astra 0.62 Sep 2026 Sep 2026
ComplexBench Models answer 1,150 hand-written Chinese instructions that compose several constraints by chaining, selection and nesting. LLM evaluators with rules check each constraint, and scores follow the dependency structure. Tsinghua University (CoAI) and Zhipu AI Following precise instructions Everyday Not in cards GPT-4 Turbo 80% Jul 2024 Apr 2024
CounterBench (1,200 questions) Models answer counterfactual questions using the stated causal rules. This release adds 200 backdoor questions to the original 1,000. Rutgers University and Case Western Reserve University Language reasoning University Not in cards DeepSeek-V3 91.8% Aug 2025 Dec 2024
Coverage Cat AI Insurance Benchmark (fictional insurance tasks) Models read fictional policy documents, find missing facts, and complete insurance applications. The scorer checks source quotes, required fields, and authorized actions. Coverage Cat Customer service University Not in cards GPT-6 Astra 100% Nov 2026 Oct 2026
Credit Firewall Benchmark Models decide whether evidence still authorizes a credit decision after roles, targets, timing, or conversation context change. Christine Classy — AI’s Favorite™ Deception and alignment University Not in cards Claude Haiku 4.5 100% Oct 2026 Sep 2026
CriticEval Models critique responses across four dimensions in nine task scenarios, giving both scalar scores and written critiques. GPT-4 grades written critiques against a large set of annotated reference critiques. OpenCompass (Shanghai AI Laboratory) and collaborators Overall chat quality University Not in cards GPT-4 Turbo 7.86 Feb 2024 Jan 2024
CypherBench Models translate questions about Wikidata into Cypher queries. The scorer executes each query and compares its result with the expected answer. Megagon Labs and collaborators Data science and SQL University Not in cards Claude 3.5 Sonnet 61.6% Dec 2024 Sep 2024
DeskCraft Desktop agents complete 538 Ubuntu tasks, including 152 tasks with later user clarifications or interruptions. Programmatic checks inspect final application state, saved files, and exported artifacts. Zhejiang University, Tsinghua University, Tencent and others Desktop apps Expert Not in cards Kimi K2.6 29.8% Jun 2026 Apr 2026
DrivingBench v2 Vision-language models drive a real Tesla through two unseen cone courses with steering, accelerator and brake tools. The score is the furthest progress over three attempts in one conversation, averaged over six trials. DrivingBench (Ramabadran, Mahns and Gessler) Real robots Everyday Not in cards Gemini 4 Carbon 89.5% Nov 2026 Nov 2026
EGI-BENCH Agents work across eight enterprise evaluations. The suite combines their results into six scores covering activities such as planning, retrieving evidence, and using tools. Salesforce AI Research Simulated companies University Not in cards GPT-5.5 65.6 Oct 2026 Apr 2026
Emergent Price-Fixing by LLM Auction Agents Models trade in simulated auctions and communicate with other participants. A judge examines game logs for evidence of price fixing. Lech Mazur Deception and alignment Everyday Not in cards Grok 3 19% Jul 2025 Jul 2025
Enterprise-Bench L1-L2 v2 Agents answer 42 operational questions using six enterprise services. A GPT-6 Luna judge evaluates the final answer against task criteria. DevRev Apps and MCP servers University Not in cards GPT-6.1 Sol 65.7% Nov 2026 Sep 2026
Exact Words Models copy a requested sentence exactly from a document. The grader checks sentence boundaries and whether the model preserves planted typos. Saharsh Tibrewala Faithful to sources School Not in cards GLM-5 100% Oct 2026 Oct 2026
FDE-Gym Agents act as forward-deployed engineers in live customer systems, talking to scripted stakeholders to find unstated constraints. Score is the uplift on the customer's own success metric, from 0 to the reference solution's 1. DeepLethe (with 20+ contributing institutions) Building whole projects Expert Not in cards Claude Opus 5.5 0.56 Oct 2026 Sep 2026
GeoGuessBench (Nutlope) Models place a pin after seeing one street photo. Each game has five rounds, and closer guesses earn more points. Nutlope Spatial reasoning and counting Everyday Not in cards Claude Opus 5.5 23412.0 Oct 2026 Sep 2026
Glianorex Models answer multiple-choice questions about fictional medical material in English and French. The test isolates question-answering patterns from real medical knowledge. Maxime Griot and colleagues Language reasoning School Not in cards GPT-4o 71.5% Jun 2024 Mar 2025
HalluWorld Language models are evaluated on hallucinations across controlled reference worlds spanning gridworlds, chess, and realistic terminal environments. Each environment has a fully specified ground truth, allowing automatic evaluation of whether model claims are correct. Questions test perception, memory, causal reasoning, uncertainty/abstention, and compound reasoning under varying levels of observability, complexity, and conflicting information. Other Not in cards Claude Fable 5.1 19.2% Sep 2026
InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. Meta + Duke University Simulated robots Everyday Not in cards Claude Fable 5.1 41.7% Oct 2026 Sep 2026
JFinBench Models answer questions from six Japanese certification exams in accounting, tax, financial planning, and banking. The authors report scores by exam. Sakana AI Finance University Not in cards GPT-6 Astra 97.3% Nov 2026 Oct 2026
KORGym Models play 51 text and visual games designed to test interactive reasoning. KORGym authors Puzzles and chess Everyday Not in cards o3-mini 0.82 May 2025 Apr 2025
Loss Run Extraction (Bevaya) Models extract claims and policy tables from insurance loss reports into JSON. The main metric counts documents where every field matches the reference. Bevaya Labs Documents, charts and text in images University Not in cards Claude Fable 5 85.3% Aug 2026 Jul 2026
Mastermind (Prime Intellect) Models guess a hidden sequence and receive feedback after each turn. The default evaluation uses 50 games and rewards success, speed, and deductions. Prime Intellect Puzzles and chess Everyday Not in cards GPT-5 1.62 Oct 2025 Sep 2025
MathIF Reasoning models solve 420 math problems from GSM8K, MATH-500, Minerva, Olympiad and AIME while obeying added verifiable instructions. Code checks the instructions and the answer. Renmin University of China, Shanghai AI Laboratory and CUHK Following precise instructions School Not in cards Qwen3-14B 50.7% May 2025 Apr 2025
MathTutorBench Models solve math problems, diagnose student mistakes, ask Socratic questions, and give instructional support. Automatic answer checks, F1, and text similarity score the constrained tasks. A trained pedagogical reward model compares tutoring responses with teacher examples. ETH Zurich and TU Darmstadt Word problems School Not in cards Gemini 3.6 Flash 0.98 Feb 2025 Jul 2026
MedPhysBench Models calculate doses and check medical physics worksheets for planted errors. A grader checks the units and accepts values within a specified tolerance. Omar Nusrat Medicine Expert Not in cards GPT-6 Astra 100% Nov 2026 Sep 2026
MedQA-Followup Models answer medical licensing questions and then respond to follow-ups that challenge their answers. Researchers compare accuracy before and after each intervention. Dynamo AI and collaborators Medicine Expert Not in cards GPT-5 mini 0% Oct 2025 Nov 2025
MineAmongUs Vision-language agents play an embodied Among Us game with verbal and nonverbal deception. MineAmongUs authors Multi-agent games Everyday Not in cards Gemini 3 Flash 81.3% Aug 2026 Apr 2026
Monthly-SWEBench Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1. UniPat AI and xbench (HongShan) Fixing real repositories Expert Not in cards GPT-5.5 76% Mar 2026 Jun 2026
NEST Models interpret children's gaze, gestures, and physical contact in social videos. The benchmark checks classification and choice questions and separately grades written descriptions. Stanford University and collaborators Video understanding Everyday Not in cards Qwen3.5-27B 76% Nov 2026 Mar 2026
OmniScore-Bench Models read sheet music, locate symbols, and transcribe passages. The evaluator checks their responses against annotations for the relevant part of the score. Tsinghua University and collaborators Documents, charts and text in images University Not in cards GPT-6.1 Sol 91.0 Nov 2026 Oct 2026
Paragraph Instruction Following Models write a single plain text paragraph under a word count constraint. The tasks check length, format, language, and relevance across 500 prompts. Łukasz Wodniakowski Following precise instructions School Not in cards GPT-5.5 0.99 Oct 2026 Sep 2026
ParliamentBench Models play Secret Hitler while deceiving, persuading, and reasoning under hidden roles. ParliamentBench authors Multi-agent games Everyday Not in cards GPT-5.4 81% Jul 2026 Mar 2026
PDAGENT-BENCH Models answer chip design questions, diagnose failures, interpret reports, and write tool commands. Rules grade five categories and model judges grade three. Qiufeng Li and colleagues Electronics Expert Not in cards GPT-5.5 65.1% Jun 2026 Sep 2026
PlanningBench Models solve generated planning problems that coordinate goals, constraints and resources. Instances are scalable and come with verifiers. Tencent Hunyuan Language reasoning Expert Not in cards GPT-5.4 63.2% May 2026 Apr 2026
PowerCodeBench Models translate operators' requests into Python programs for pandapower. The evaluator runs each program and compares its numerical or Boolean output with the reference answer. University of Exeter Scientific code University Not in cards Llama 3.1-405B 68.7% May 2026 Apr 2026
Prompt-to-Bench-16 Models write OpenSCAD programs for laboratory fixtures from measured dimensions. They receive feedback after each failed attempt. The grader checks the rendered geometry. S. S. Cocioba, Binomica Labs CAD models University Not in cards Gemma 4 26B A4B 10.0 Oct 2026 Aug 2026
PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. Scale AI and collaborators Medicine Expert Not in cards Gemini 3.1 Pro 95.3% Jun 2026 Apr 2026
RoboDojo-RC Tier 1 Six real RoboDojo tasks on I2RT YAM arms, 20 attempts each. Robot policies and tool-calling LLMs scored by rubric progress. Robocurve PBC Real robots Everyday Not in cards GPT-6.2 Astra 65.7% Sep 2026 Nov 2026
Rule2DRC Models write KLayout scripts from chip design rules. The evaluator checks whether each script flags the correct violations on every test layout. Seoul National University and collaborators Electronics Expert Not in cards Kimi K3 67.9% May 2026 Jul 2026
SAHM Models answer Arabic questions on financial standards, Islamic finance, and accounting. The authors report multiple-choice accuracy and grade open-ended answers separately. Rania Elbadry and colleagues Finance University Not in cards Gemini 3 Flash 80.5% Apr 2026 Dec 2025
Samskritam Bench v1 Models split entire Sanskrit verses into words, parse inflected words, scan verse meter, and generate rare verb forms. Each verse requires every graded part to match. Sriram Subramanian, Samskritam.ai Reading in many languages University Not in cards Claude Opus 5.5 89.4% Oct 2026 Sep 2026
ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. UniPat AI and Elorian Spatial reasoning and counting Expert Not in cards GPT-6 Astra 0.53 Oct 2026 Nov 2026
SciPredict Models predict the outcomes of 405 real physics, biology and chemistry experiments published after March 31, 2025, with and without expert background knowledge. Scale AI Research science Expert Not in cards Gemini 3 Pro 25.3% Jan 2026 Dec 2025
SimBench (PyChrono) Models write and revise PyChrono simulation programs over three turns. A model grades each program using the reference solution and API documentation. University of Wisconsin–Madison Simulation-Based Engineering Laboratory Scientific code University Not in cards Claude Sonnet 4 49.0 Aug 2024 Jun 2025
Slay the Spire Bench (alavico) Models choose an action from the text state of Slay the Spire. The campaign report measures floors reached and wins at each difficulty. alavico Video games Everyday Not in cards Claude Sonnet 4.6 51.0 Jun 2026 Jul 2026
Snowflake Text-to-SQL HARD Models generate SQL for difficult questions about unfamiliar enterprise schemas. Snowflake compares models by query accuracy. Snowflake Data science and SQL Expert Not in cards Gemini 3.1 Pro 47.9% May 2026 Apr 2026
SpireClimb Agents play Slay the Spire 1 and 2 at max difficulty through a computer-use harness on fixed seeds. Score is the mean win rate. Epoch AI Video games Expert Not in cards GPT-6.2 Astra 41.3% Oct 2026 Nov 2026
SpireClimb StS1 A20 Heart Agents play Slay the Spire 1 through its graphical interface at Ascension 20 and must defeat the Heart. Scored by win rate over fixed seeds. Epoch AI Video games Expert Not in cards GPT-6 Astra 33.3% Oct 2026 Nov 2026
SpireClimb StS2 A10 Agents play Slay the Spire 2 at Ascension 10 through screenshots and mouse and keyboard, 15 runs per character. Scoring is the win rate. Epoch AI Video games Everyday Not in cards GPT-6.2 Astra 49.3% Oct 2026 Nov 2026
STELLA-Bench Models respond to simulated users over multiple turns. Judges measure the shares of conversations with harmful or unhelpful responses. Atella AI Wellbeing and sycophancy Everyday Not in cards GPT-5.6 Terra 0% Sep 2026
SuperCLUE-CPIF Models follow precise instructions written in Chinese. SuperCLUE Following precise instructions Everyday Not in cards GPT-5.1 83.6 Nov 2025
SuperCLUE-SWE with Claude Code Models solve Chinese software engineering tasks in the Claude Code harness. SuperCLUE Fixing real repositories Expert Not in cards Kimi K3 75.8 Aug 2026
SuperCLUE-Writing Models write original Chinese prose. SuperCLUE Creative writing University Not in cards Claude Opus 4.7 88.0 May 2026
TableBench Models answer 886 questions about real-world tables across 18 fields in four groups: fact checking, numerical reasoning, data analysis and visualization. Fact and numeric answers are scored by exact match, analysis answers by ROUGE-L, and chart code by pass@1. Beihang University, M-A-P and collaborators Language reasoning University Not in cards o4-mini 61.7% Aug 2024 Aug 2025
TimelineBench Agents turn raw footage and a client brief into a finished video. The edit must pass delivery checks, meet the brief, and clear a quality threshold. TensorTest Tasks from real jobs University Not in cards Claude Opus 5.5 32.1% Sep 2026 Sep 2026
TreasuryQA Bench Agents search scanned Treasury volumes and read tables to find or calculate a number. A script checks whether the answer falls within the accepted range. Veris AI and Reducto Documents and spreadsheets School Not in cards Gemini 4 Hydrogen 82.4% Oct 2026 Nov 2026
trivial-prompt-bench Models answer a short request through a terminal agent. Quesma adds the API charge to the estimated cost of the developer's waiting time. Quesma Other Everyday Not in cards GPT-5.4 Mini $0 Jul 2026 Jun 2026
TUA-Bench Agents use a terminal for general computer tasks beyond coding, such as file, system and everyday productivity work. Meta Terminal tasks Everyday Not in cards Claude Opus 4.8 65.8% Jun 2026 Jun 2026
Veris FinAgent Arena v1.1 Agents investigate bank alerts, read Treasury scans, and complete research requests. The index combines accuracy and repeatability across three benchmarks, then applies a cost penalty. Veris AI Composite work indices University Not in cards Gemini 4 Carbon 757.0 Nov 2026 Nov 2026
WearableQA Models answer 4,084 ten-option questions using longitudinal wearable measurements, blood biomarkers, and demographics from 200 users. Questions test computations and physiological interpretation across single or multiple signals. Accuracy compares the selected option with the answer key. Meta, KAIST, and Korea University Medicine Expert Not in cards Gemini 3.1 Pro 72.9% Sep 2026 Apr 2026
Web Context (Nimble) Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages. Nimble Deep research Expert Not in cards GPT-5.6 Sol 80% Oct 2026 Oct 2026
WellnessBench Models answer young people asking about eating, sexual safety, substances and self harm, in clinician-written single-turn and pushback dialogues. A pass must be both safe and helpful. micro1 Wellbeing and sycophancy Everyday Not in cards GPT-6.1 Sol 66% Oct 2026 Oct 2026
Who Are You? LLM Self-Identity Models answer twelve phrasings of questions about their identity and maker. The grader compares their answers with the model Kaggle ran. J Gonn6235 Short factual questions Everyday Not in cards gpt-oss-120b 100% Oct 2026 Sep 2026
AECV-Bench (2026 release) Models count four types of floor plan elements and answer questions about engineering drawings. Counting uses exact matches. A model grades the drawing answers. AECFoundry Documents, charts and text in images University Not in cards Gemini 3.5 Flash 71.4% Dec 2025 Jun 2026
Agentick Agents solve procedurally generated gridworld tasks involving navigation, planning, reasoning, memory, generalization, and coordination. Scores normalize task performance between random and oracle reference policies. Roger Creus Castanyer, Pablo Samuel Castro, and Glen Berseth Puzzles and chess Everyday Not in cards GPT-6 Astra 0.63 May 2026 Sep 2026
AgReason Models answer 100 farm management questions using local growing conditions. A judge compares individual claims with expert answers. A question passes at statement-level F1 of 0.80. Iowa State University and New York University Science questions University Not in cards Gemini 2.5 Flash 36% May 2025 Sep 2025
AI Engineering Lab model creation Models generate Exudyn simulation code for 35 mechanical systems. The evaluator compares numerical outputs with reference simulations across ten parameter settings per system. University of Innsbruck Scientific code University Not in cards Qwen2.5-Coder-32B-Instruct 91.7% Apr 2025 Jan 2025
AMemGym Assistants hold long simulated conversations with users whose details and preferences change over time. Simulated users react on policy, and the test checks whether the assistant remembers and applies the right facts. AGI-Eval (Meituan) Learning from the context Everyday Not in cards Gemini 3 Pro 55.6% Jan 2026 Dec 2025
Arena-T2I Hard Models generate images from detailed user prompts. A judge checks whether the images satisfy individual requirements and their dependencies. Arena Image generation and editing University Not in cards Gemini 3 Pro Image (nano-banana-pro, 2K) 0.85 Jun 2026
ATLAS-Finance v1.1.0 Agents do the same 100 ATLAS-Finance banking tasks, graded with revised rubrics. Weights use a 1 to 10 scale and tolerances and gates sit at the section level. Handshake AI Research Finance Expert Not in cards Claude Opus 5 10.7% Oct 2026 Sep 2026
AutoFactory-Bench Agents write control programs for simulated factory cells. A run passes when the program completes every operating condition without a safety violation. AutoFactory-Bench with Dongguk University and MOAI technologies Electronics University Not in cards GPT-6 Astra 72.2% Oct 2026 Oct 2026
Basic reasoning with letters Models count and compare letters within words. The tasks include counting a letter across weekday names and finding words with equal character frequencies. Navneet Prabhat Language reasoning School Not in cards GLM-5 100% Oct 2026 Sep 2026
BIG-Bench Extra Hard Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. Google DeepMind Language reasoning University Not in cards o3-mini 44.8% Feb 2025 Mar 2025
BTF-3 Agents forecast already-resolved binary and numeric questions while researching a frozen snapshot of the web. Scoring pools Brier and ranked probability scores. FutureSearch Real-world events Expert Not in cards Claude Opus 5 0.12 Jun 2026 Sep 2026
CaBBQ Models answer Catalan questions about people in ambiguous and unambiguous situations. The test measures whether answers follow social stereotypes when evidence is insufficient. Valle Ruiz-Fernández and colleagues Bias, fairness and politics Everyday Not in cards Qwen2.5-7B 81% Jul 2025 Mar 2025
CASE-Bench Models decide whether a request is safe in a given context. The same request appears in two contexts with different human judgments. Guangzhi Sun and colleagues Refusing harmful requests Everyday Not in cards Claude 3.5 Sonnet 90.1% Jan 2025 Aug 2024
Continual Learning Exam Agents work through 120 professional tasks, each a ladder of five levels that must be passed in order. Later levels reuse files, tools and lessons from earlier ones. An evaluation agent grades against expert rubrics. University College London (Jun Wang) + Imperial College London + University of Liverpool Learning from the context Expert Not in cards Claude Opus 5 47.7% Oct 2026 Nov 2026
COREBench (Conversational Reasoning) Models listen to a conversation with several speakers and answer a question that can require information from multiple turns. Stanford CRFM and the AHELM team Audio understanding and voice Everyday Not in cards Gemini 2.5 Pro 81.3% Jun 2025 May 2025
CoreCodeBench Models complete over 1,500 repository-level Python tasks covering development, bug fixing and test-driven work, with single-function and multi-function variants. The CorePipe pipeline builds tasks from repositories at set difficulty levels. AGI-Eval (Meituan) Fixing real repositories Expert Not in cards Gemini 3 Pro 44.4% Jul 2025 Dec 2025
Crazyrouter AI Trading Arena Models buy, sell, or hold three cryptocurrencies every five minutes using simulated accounts. The grader values portfolios at live quotes and deducts trading fees. Crazyrouter Finance University Not in cards GPT-6.1 Sol -7.45% Nov 2026 Oct 2026
CrossPoint-Bench Models see two views of a scene and must find the point in the second view that matches a point in the first. Tasks go from perception and reasoning to exact coordinate prediction. Institute of Automation, Chinese Academy of Sciences (Wang, Ji et al.) Spatial reasoning and counting Everyday Not in cards Qwen2.5-VL-3B 72.9% Dec 2025 Oct 2025
CyberMetric Models answer multiple-choice cybersecurity knowledge questions drawn from standards, papers and RFCs. Sets of 80, 500, 2,000 and 10,000 questions exist. Technology Innovation Institute (Abu Dhabi) Capture-the-flag challenges Expert Not in cards Mistral-7B-Instruct-v0.2 74.8% Feb 2024 Apr 2024
DashBench Models explain rare events in driving videos, including near misses and collisions. The authors compare 25 video models on 1,709 clips. Korea Advanced Institute of Science and Technology Video understanding Everyday Not in cards GPT-5 34.8% Nov 2026 Dec 2025
DeputyBench Agents handle business requests under paired permission settings. The evaluator checks whether they act when authorized and seek evidence or refrain when authorization is missing. University of Southern California Simulated companies University Not in cards Gemini 4 Argon 92.6% Nov 2026 Nov 2026
Dual-Arm Sim Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit. micro1 Simulated robots Expert Not in cards Claude Fable 5.2 59.8 Oct 2026 Nov 2026
EarthRecord Agents analyze multidecade satellite observations to answer land-change questions. Reference checks score answers to 238 questions across 56 sites. AIEyes team Lab and data work Expert Not in cards Claude Opus 5.5 85.4% Nov 2026 Oct 2026
EJBench Models interpret simulation results for battery cooling plates. They match results to test conditions, select evidence for a design decision, and rank operating conditions by temperature. Gradient Dynamics Lab and data work University Not in cards GPT-6 Astra 0.71 Nov 2026 Oct 2026
Encord Egocentric VQA Benchmark Models answer 1,500 multiple-choice questions about actions, objects, space and time in private first-person videos. Encord Data Labs General visual questions Everyday Not in cards GPT-6 Astra 76.5% Nov 2026 Sep 2026
EsBBQ Models answer Spanish questions about people in ambiguous and unambiguous situations. The evaluation measures how social stereotypes affect their choices. Valle Ruiz-Fernández and colleagues Bias, fairness and politics Everyday Not in cards Qwen2.5-7B 85% Jul 2025 Mar 2025
EvalDetectBench A judge model reads a transcript and answers whether it came from an evaluation or from real deployment. The test measures how well models detect evaluations and how detectable each benchmark is. UK AI Security Institute + LASR Labs Deception and alignment Expert Not in cards Claude Opus 4.6 0.89 Jun 2026 Apr 2026
Frontier3DBench Agents create editable Blender objects and scenes or executable Unreal Engine worlds across 145 tasks and nine task families. Model judgments and native project checks assess appearance, geometry, requirements, and interactions. The overall score weights object, scene, and world scores by 30%, 30%, and 40%. Einsia AI (Navers lab) Spatial reasoning and counting Expert Not in cards GPT-6.1 Sol 66.0 Nov 2026 Nov 2026
FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. Apodex AI (FrontierAgent team) Research science Expert Not in cards GPT-5.6 Sol 20.6% Aug 2026 Aug 2026
Hard12 Agents solve 12 selected coding tasks from five benchmarks. Hidden tests grade each task. The leaderboard averages three runs per task. Ibragim Badertdinov Fixing real repositories Expert Not in cards Claude Opus 5.5 83.3% Nov 2026 Oct 2026
Hedge-Bench Agents analyze supplied company and investment documents for 102 open-ended financial research tasks. A model judge applies rubrics that check whether the reasoning covers the analytical steps identified by investment professionals. Trata Finance Expert Not in cards GPT-5.6 Sol 31.5 Jun 2026 Sep 2026
HelloBench Models generate long answers, summaries, conversations, text completions, and texts from writing instructions. Model judges apply task-specific checklists to the responses. Beihang University, Shanghai AI Laboratory, Peking University and others Overall chat quality University Not in cards GPT-4o 48.5 Sep 2024 Sep 2024
Human Behavior Bench Models predict how specified groups of U.S. adults answer closed survey questions, as a distribution over options. Scored as 1 minus total variation distance. Rehearsals with Google DeepMind and Kaggle Real-world events Expert Not in cards Gemini 4 Argon 80.6 Oct 2026 Sep 2026
IFMTBench Models translate text while following explicit instructions, such as keeping JSON or HTML structure, using a glossary, using provided context and matching a register. It covers seven languages with 4,506 single-constraint and 2,838 multi-constraint items. Deterministic checkers verify some constraints, and a rubric-based model judge scores the rest. Tencent Hunyuan Translation University Not in cards Qwen3.5-27B 81.1% May 2026 Apr 2026
IPTC-test Models choose one of 17 newsroom topics for each article. Human annotators labeled 1,129 articles in Catalan, Croatian, Greek, and Slovenian. Taja Kuzman and Nikola Ljubešić Reading in many languages Everyday Not in cards Gemini 2.5 Flash 0.77 Nov 2024 Sep 2025
IrokoBench Models answer human-translated knowledge, math and inference questions in 16 African languages. The parts are AfriMMLU, AfriMGSM and AfriXNLI. Masakhane and collaborators (Adelani et al.) Knowledge in many languages School Not in cards GPT-4o 59% Jun 2024 Mar 2025
Judge Bait Models judge pairs of answers. The test varies their order, length, authority claims, and pressure cues to see whether judges prefer the correct one. Kelechukwu Agbo Language reasoning School Not in cards Claude Sonnet 5.5 100% Oct 2026 Oct 2026
LABBench2 LitQA3 Models answer open-response questions whose answers appear once in recent biology papers. The LitQA3 subtask of LABBench2. Edison Scientific (formerly FutureHouse) Genomics and life sciences Expert Not in cards GPT-5.2 Pro 85.1% Feb 2026 Feb 2026
Labelbox Complex Reasoning Models solve reasoning problems in mathematics, programming, and other domains. Labelbox reports overall and domain scores. Labelbox Language reasoning University Not in cards GPT-5 89.6% Jul 2025 Aug 2025
LawBench Models answer Chinese legal questions across 20 tasks, including statute recall and case prediction. Each task has its own scoring rule. Zhiwei Fei and colleagues Law University Not in cards GPT-3.5 Turbo 44.5 Sep 2023 Sep 2023
Learn2Play Bench Agents play unfamiliar games whose rules are not given and must learn from experience across episodes. Improvement over repeated play is the measure. National University of Singapore Puzzles and chess Everyday Not in cards Claude Opus 5 81.8 Oct 2026 Sep 2026
LegalReasonBench Models work long-horizon legal matters built by practicing lawyers, writing memoranda and court filings across dependent assignments, in closed-record and open-research settings. micro1 Law Expert Not in cards Gemini 4 Argon 50.8% Oct 2026 Nov 2026
LLM Persuasion Benchmark Models try to change another model’s stated position over multiple conversation turns. Lech Mazur Multi-turn conversation Everyday Not in cards GPT-5.4 1.71 Mar 2026
LocalSearchBench Agents search merchant listings to answer 900 questions about local services across nine Chinese cities. A model judges each answer against a reference. Meituan, East China Normal University, and collaborators Questions that need search Everyday Not in cards DeepSeek-V3.2 35.6% Dec 2025 Dec 2025
M5HisDoc Models locate text, recognize characters, and recover reading order on Chinese historical pages. The hard subset distorts the same 4,000 pages as the regular subset. South China University of Technology Documents, charts and text in images University Not in cards Qwen3.7-Plus 64.5% Jun 2023 Jun 2026
mage-bench Models play Magic: The Gathering in a benchmark that compares strategic play across games. mage-bench Multi-agent games Expert Not in cards GPT-5.4 Mini 1617.0 Jan 2026 Mar 2026
Meeseeks Utility Rate Models follow complex instructions with feedback for up to 20 turns. The score measures fully usable responses as a percentage. Meituan Multi-turn conversation Everyday Not in cards Claude Sonnet 4 90.5% Apr 2025 Aug 2025
MemoryTrap Bench: PHP arrays, Go maps Models answer questions about memory use in PHP and Go. The grader compares answers with measurements, varying whether models see runtime versions or a measured fact. Nazar Boyko Speed and optimization University Not in cards GPT-6.1 Sol 0.95 Oct 2026 Oct 2026
MiJaBench Models respond to 43,961 English and Portuguese prompts across 16 minority groups. Judges measure how defense rates change with the target group and language. Federal University of Goiás Bias, fairness and politics University Not in cards Llama 3.3 70B Instruct 58.2% Jan 2026 Apr 2025
MirageBench Models personalize six kinds of responses from three disclosed facts about each of 150 personas. A model judge classifies claims by evidential support. The primary over-inference rate counts stereotyped and fabricated claims. Tencent LIGHTSPEED and Hong Kong University of Science and Technology Faithful to sources Everyday Not in cards Gemini 3.1 Pro 35.1% Aug 2026 Apr 2026
MissionReport Models write government status reports in Word from working files and a template, across 20 expert-written Department of Defense style assignments. Expert rubrics score each decision. micro1 Documents and spreadsheets Expert Not in cards Gemini 4 Argon 64.1% Oct 2026 Nov 2026
MME-Reasoning Multimodal models answer logic questions grouped into inductive, deductive and abductive reasoning. MME team (Alpha-Innovator) Visual math and diagrams University Not in cards Seed1.5-VL 59.9% May 2025 Jul 2025
MTHv2 Models read Chinese Buddhist texts from scanned historical pages. The authors annotate characters, line transcriptions, reading order, and boundary lines. South China University of Technology and Alibaba Group Documents, charts and text in images University Not in cards Kimi K2.6 86.4% Jun 2020 Jun 2026
NoticIA Models summarize Spanish news articles in one sentence that answers the clickbait headline. The evaluator compares each summary with a human reference. Iker García-Ferrero and colleagues Faithful to sources School Not in cards GPT-4 Turbo (Jan 2024) 32.0 Apr 2024 Feb 2024
OmniMatBench Models answer questions and perform calculations across 19 materials science fields. The authors compare automated grades with human assessments. Wanhao Liu and colleagues Science questions University Not in cards Claude Opus 4.7 0.37 May 2026 Apr 2026
OR-License Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem. ALLE AI Tasks from real jobs Expert Not in cards Grok 4.6 88.4% Oct 2026 Sep 2026
OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. DiscoAILab Desktop apps Expert Not in cards Claude Fable 5.1 73.7% Sep 2026 Sep 2026
PARADE Audio Models answer questions about occupations and social status using audio clips. The evaluation tests whether voice cues bias their answers. UCSC VLAA and the AHELM team Bias, fairness and politics Everyday Not in cards Qwen2.5-Omni-7B 63.4% Mar 2025 May 2025
QuantSightBench Models give 90% prediction intervals for 1,000 numerical forecasting questions with known outcomes. The main setting allows retrieval from a fixed news corpus. Mean Log Interval Score penalizes wide intervals and missed outcomes, while coverage measures how often intervals contain the answer. ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, and Tübingen AI Center Real-world events Expert Not in cards GPT-5.1 6.70 Apr 2026 Mar 2026
Questflow Finance Benchmark Agents query financial filings and market data with a fixed set of tools. They calculate metrics, rank securities, and submit answers that software checks against frozen references. Questflow Finance University Not in cards GPT-6.1 Terra 100% Oct 2026 Oct 2026
Realtime-Bench Agents act in 50 live game and web tasks where a late action fails: 30 web tasks (travel, shopping, services and security) and 20 games. The environment keeps moving while the model thinks. Three clocks separate speed from skill: Wall Clock runs on real time, Paused Clock stops while the model thinks, and Token Clock advances with each generated token. Scores are task success rates over three runs. University of Texas at Austin, Ohio State University, UIUC, NVIDIA, Northwestern University and Steadyworks Web browsing Everyday Not in cards GPT-6 Astra 42.7% Oct 2026 Nov 2026
Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. Vals AI Simulated robots Expert Not in cards GPT-6.2 Astra 32.7% Oct 2026 Nov 2026
RPEval Models respond in character to 9,018 scenarios. The test checks emotions, decisions, moral choices, and whether characters reveal knowledge outside their setting. Yassine El Boudouri and colleagues Overall chat quality Everyday Not in cards Gemini 1.5 Pro 62.2% May 2025 Jun 2025
RuleTrek Agents must find and correctly apply domain-specific rules when the scenario's facts are spread across many files, unlike rule benchmarks built from a few statements. RuleTrek authors Language reasoning Expert Not in cards GPT-6.1 Sol 55.3% Oct 2026 Sep 2026
RWE-bench Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement. Xplore AI CAD models Expert Not in cards Claude Fable 5.2 62% Oct 2026 Nov 2026
SafePyramid Models identify rules violated by a conversation under a supplied policy. Rule matching rate averages four thresholds for false positive and false negative counts. ByteDance and University of Melbourne Refusing harmful requests University Not in cards GPT-5.5 54.2% Jun 2026 May 2026
SEC filing claim verification (400 items) Models classify whether an excerpt from a corporate filing supports a claim, contradicts it, or provides insufficient evidence. Vals AI Faithful to sources University Not in cards Claude Sonnet 5.5 99.5% Oct 2026 Sep 2026
SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. Tsinghua University Spatial reasoning and counting Everyday Not in cards GPT-5 14.4% Jun 2026 Mar 2026
SpatialWorld Digital Agents complete entertainment tasks in digital environments. The score is the published digital task success rate. Tsinghua University Spatial reasoning and counting Everyday Not in cards Gemini 3.1 Pro 39% Jun 2026 Mar 2026
SpatialWorld Physical Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories. Tsinghua University Spatial reasoning and counting Everyday Not in cards GPT-5 14.4% Jun 2026 Mar 2026
Speko Speech-to-Speech Benchmark (six concierge scenarios) Voice agents book flights, move reservations, and revise budgets during scripted calls. Speko checks whether agents wait for confirmation and use the latest details. Speko Customer service Everyday Not in cards GPT-Live-1 0.91 Oct 2026
StudentBench AI tutors teach real students GRE material and are scored by the students measured learning gains. A second study has expert tutors compare AI lesson plans and practice problems. Handshake AI Research Other University Not in cards Claude Opus 4.8 16.5 Sep 2026 Aug 2026
SuperCLUE-Fin Models answer Chinese finance questions across 25 tasks in six categories. The evaluation covers financial knowledge and its application in practice. SuperCLUE team Finance University Not in cards Kimi K3 88.7 Apr 2024 Jul 2026
SusVibes Coding agents rebuild a feature in a real open-source Python project from a plain-language request. Each task comes from a historical vulnerability fix, so the original human version was insecure. Hidden functional tests and security tests check the result. Scores report the share of solutions that work and the share that are also secure. Carnegie Mellon University (Lei Li's group) Secure code and review Expert Not in cards GPT-5.5 43.5% Dec 2025 Jun 2026
SWE-bench Science Coding agents fix 119 repository-level tasks from 98 scientific software projects in 20 domains, split into issue-driven, expert-exploratory and integration tasks. Tests check each fix. Fudan University OpenMOSS (Zhipeng Xu et al.) Scientific code Expert Not in cards GPT-6 Astra 50.4% Aug 2026 Sep 2026
Taiwanese Mandarin Language Understanding Models answer questions from Taiwanese exams in traditional Chinese. The subjects range from school topics to professional qualifications. Po-Heng Chen and colleagues Knowledge in many languages University Not in cards Claude 3 Opus 73.6% Mar 2024 Apr 2024
TERMS-Bench Agents negotiate the price of a single item with simulated counterparts over up to 10 rounds of alternating offers. Counterparts follow six behavior families across three regimes. A program scores each agent from its logged actions on four separate axes: surplus captured, agreement calibration, belief error about the opponent and procedural violations. Extra modes score merchant profit and a bankroll over many sessions. Stanford University Multi-agent games Everyday Not in cards Claude Opus 4.6 0.69 May 2026 Apr 2026
TireSidewall-Bench: Multimodal Zero-Shot Tire OCR Models read the size markings on tire sidewalls from photographs. The grader compares extracted specifications with reference annotations. Inusha Thathsara Documents, charts and text in images Everyday Not in cards Gemini 3.8 Flash 46% Oct 2026 Oct 2026
TutorCaseBench Tutors respond to students whose behavior interferes with learning. Case-specific rubrics grade the responses, and a separate track tests selection among possible interventions. Seoul National University and collaborators Other professions University Not in cards Claude Opus 4.7 59.4% Oct 2026 Apr 2026
UniGUI-Bench Agents interpret and operate graphical interfaces. The benchmark tests 11 capabilities and includes separate system evaluations. AGI-Eval (Meituan) Finding things on screen Everyday Not in cards Kimi K2.5 80.9% May 2026 Apr 2026
VibePrevisBench Coding agents write Blender Python that turns a script, photos, an opening frame or reference footage into a graybox 4D previs scene and render it. Rule checks score the exported scene against hidden facts, and a VLM judge checks that those facts are visible in the render. KlingAI Research (Kling-Keye) CAD models Expert Not in cards GPT-6 Astra 77.7 Nov 2026 Sep 2026
VISTA (Visually Impaired Scene and Task Annotations) Models describe scenes, answer questions, and give guidance to blind or low-vision users from glasses recordings. A judge scores responses from 0 to 5. Harvard AI and Robotics Lab and collaborators Video understanding Everyday Not in cards Gemini 3.1 Flash-Lite 4 Oct 2026 Mar 2026
VoxCeleb2 Models compare two interview recordings and decide whether the same person speaks in both. Joon Son Chung and colleagues, University of Oxford Audio understanding and voice Everyday Not in cards Gemini 2.5 Flash 75.9% Jun 2018 May 2025
3DSceneBench Agents reconstruct interactive MuJoCo scenes from images and task descriptions. Verifiers measure appearance, geometry, and physical functionality, then average the three scores. Prana Labs and University of California, Berkeley Physical prediction Expert Not in cards GPT-6 Astra 59.5 Nov 2026 Oct 2026
AdaPlanBench Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear. University of Illinois Urbana-Champaign General assistants Everyday Not in cards GPT-5 67.8% Jun 2026 Apr 2026
AgentCallBench: Tool-Calling Reliability Models operate a simulated store using tools. The grader checks tool choice, arguments, failure recovery, unnecessary calls, and refusals of impossible orders. Abhisek Roy Function calling Everyday Not in cards Gemini 3.5 Flash 0.99 Oct 2026 Sep 2026
AgentCLUE Agents execute tasks in Chinese. SuperCLUE Apps and MCP servers University Not in cards Claude 3.5 Sonnet (October 2024) 46.4 Feb 2025
Agentic Trader Agents search for market information and place simulated foreign exchange orders using live prices. Each agent manages its own positions, cash, and margin. HKU Business School Artificial Intelligence Evaluation Lab Finance University Not in cards Gemini 3.1 Pro 79.4% Jul 2026 Jul 2026
AHELM Audio-language models are scored across 10 aspects with standard prompts. The suite combines existing datasets with two new ones, PARADE for stereotypes and CoRe-Bench for conversational reasoning. Stanford CRFM Audio understanding and voice Everyday Not in cards Gemini 2.5 Pro 80.3 Aug 2025 Sep 2025
ASPI Agents complete simulated tasks with incomplete instructions. The authors compare adherence to user intent across matched versions of each task. Scale AI Deception and alignment Everyday Not in cards Gemini 3.1 Pro 17.8% May 2026 Apr 2026
AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. AutoLab team, with authors from UW, UCSD, Princeton, MIT, Google, NVIDIA, IBM and others Machine learning engineering Expert Not in cards Claude Opus 4.6 0.68 Jun 2026 Apr 2026
BizBench Long-horizon agents run business operations such as payments under long, partly outdated or conflicting policies. Intermediate steps are checked, not only final success. Ant International Running a business Expert Not in cards DeepSeek V4.1 Flash 70% Oct 2026 Sep 2026
BotChat Two instances of a model sustain a conversation. Judges assess the resulting dialogue. OpenCompass and collaborators Multi-turn conversation Everyday Not in cards GPT-4 73.2 Oct 2023 Sep 2023
CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. NVIDIA + UC Berkeley + CMU + Stanford Simulated robots University Not in cards Gemini 3 Pro 32.3% Mar 2026 Dec 2025
Catastrophic Cyber Capabilities Benchmark (3CB) Agents solve capture the flag tasks that span MITRE ATT&CK categories such as reconnaissance. Built to measure cyber offense capability. Apart Research and collaborators Capture-the-flag challenges Expert Not in cards Llama 3.1-405B 69% Oct 2024 Sep 2024
CBT-Bench Level II Models identify cognitive distortions and classify patients’ core beliefs at two levels of detail. The evaluator compares the selected labels with the reference labels. Mian Zhang and colleagues Medicine Expert Not in cards Claude Opus 5.5 0.63 Oct 2024 Sep 2026
CFDCodeBench V2 Models write Python solvers for 24 fluid dynamics problems. The revised tasks specify output arrays and grid conventions and change required fields for some flows. Rensselaer Polytechnic Institute and collaborators Scientific code University Not in cards Gemini 3.1 Pro 75% Oct 2026 Sep 2026
CFDQuery Models answer 90 multiple-choice questions about fluid simulation, including numerical stability, grid spacing, and discretization errors. The score is answer accuracy. Rensselaer Polytechnic Institute and collaborators Science questions University Not in cards o3-mini 91.1% May 2025 Sep 2026
CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. Center for AI Safety Deception and alignment Expert Not in cards Gemini 4 Argon 8.60% Sep 2026 Oct 2026
CNFinBench Models answer financial questions and carry out financial workflows. The benchmark includes tasks on expertise, autonomy, and integrity. OpenCompass and collaborators Finance Expert Not in cards Doubao Seed 2.1 Pro 83.7 Dec 2025 Sep 2026
Co-Design Bench Agents revise existing applications from multimodal user requests. A judge checks browser and repository evidence against 2,331 expert criteria across 273 tasks. Xilin Wei and collaborators Web and app front ends Expert Not in cards GPT-6 Astra 69.9% Oct 2026 Sep 2026
CoCoNot Models get 1,000 requests they should not fully comply with, such as incomplete, unsupported or humanizing requests. A judge model rates whether each response complies. Allen Institute for AI Deception and alignment Everyday Not in cards Claude 3 Sonnet 5.34% Jul 2024 Apr 2024
Cost per Successful Task Agents solve 40 Terminal-Bench tasks. The benchmark divides spending across all attempts by the number of successful runs. Arize AI and Fireworks AI Terminal tasks Expert Not in cards Gemini 3.5 Flash $23 Jul 2026 Jul 2026
CounselBench-Adv Models answer 120 questions written by mental health professionals. Each question targets one of six problems identified in earlier counseling responses. University of Southern California Wellbeing and sycophancy Expert Not in cards GPT-4o 0.37 May 2025 Sep 2026
CounselBench-Eval Models answer 100 questions that people posted for therapists. Mental health professionals rated responses on six dimensions. University of Southern California Medicine Expert Not in cards Gemini 2.5 Pro 5 May 2025 Sep 2026
CTI-REALM Agents read cyber threat intelligence and build detection rules. A five-step workflow maps MITRE techniques, finds data sources, writes Sigma and KQL rules and tests them on telemetry. CTI-REALM authors, implemented in UK AISI inspect_evals Finding and exploiting real bugs Expert Not in cards Claude Opus 4.6 63.7% Mar 2026 Feb 2026
Cua-Bench EDA (KiCad v1) Agents build or edit KiCad schematics through the desktop interface. The grader checks the resulting netlist and reports full passes, partial passes, and mean reward. Snorkel AI and Cua AI Electronics Expert Not in cards GPT-5.5 24% Jun 2026 Jul 2026
EduArt Models answer art history questions from Italian secondary schools and US Advanced Placement exams. The authors compare 12 models across seven question formats. University of Bologna and collaborators Exam questions School Not in cards Gemini 3.1 Pro 82.8% Jul 2026 May 2026
Embodied First Exam Coding agents answer frozen robot tasks from many benchmark suites with a program. Each program is verified in a separate sandbox, run on Harbor. Northwestern University and collaborators (CMU, UC Berkeley, Stanford and others) Simulated robots Everyday Not in cards Claude Fable 5.2 49.6% Oct 2026 Nov 2026
EnviroExam Models answer multiple choice questions from 42 environmental science courses. The composite index discounts mean accuracy when performance varies across courses. Harbin Institute of Technology Science questions University Not in cards Qwen1.5-7B-Chat 80.6 May 2024 Feb 2024
ExpertLongBench Models write documents for 11 professional tasks. A judge compares facts in each document with a reference checklist and scores precision and recall. University of Michigan and collaborators Tasks from real jobs Expert Not in cards Gemini 2.5 Pro 33.4% Jun 2025 Aug 2025
ExplorationBench Agents explore verifiable alien worlds with hidden laws, framing hypotheses and running experiments. The worlds are built so that recalled pretraining knowledge does not help. Tencent Hunyuan Research science Research frontier Not in cards Claude Opus 5 89% Sep 2026 Sep 2026
Flawed Fictions Models detect plot holes in stories and identify the line responsible. The hub port uses an 83-example holdout by default. Kabir Ahuja and collaborators Reading comprehension University Not in cards Claude 3.7 Sonnet 0.66 Apr 2025 Feb 2025
FoamBench V2 Agents build and run 110 Basic and 16 Advanced OpenFOAM cases. Revised instructions specify geometry, meshes, and boundary conditions for the same named cases. Rensselaer Polytechnic Institute and collaborators Apps and MCP servers University Not in cards Gemini 3.1 Pro 86.5% Oct 2026 Sep 2026
Fund Smart Routing Agent Models choose a fund for a transaction. Ramp measures how often a user overrides that choice. Ramp Labs Finance University Not in cards GPT-5.1 100% Mar 2026 Mar 2026
Genspark Slides Benchmark Models build finished slide decks from 200 real production tasks in the Genspark Slides agent harness. An internal grader scores quality, completion, content and design. Genspark Documents and spreadsheets Everyday Not in cards Claude Fable 5.1 83.6% Sep 2026
GitTaskBench Code agents solve 54 real tasks across 7 modalities and 7 domains by using a given GitHub repository, from understanding and setup to running it. QuantaAlpha Fixing real repositories Expert Not in cards Claude 3.7 Sonnet 48.1% Aug 2025 Apr 2025
Hebbia Financial Services Benchmark Agents in Hebbia's production harness answer questions over deal and research documents, build Excel models and make client decks. A run passes only if it meets every rubric criterion. Hebbia Finance Expert Not in cards Claude Opus 5.5 74.6% Oct 2026 Oct 2026
Investor's Last Exam Financial agents make equity investment judgments from a frozen set of decision-time evidence. Predictions are scored against later outcomes and professional criteria. Prophet Lab Finance Expert Not in cards GPT-6 Astra 68.1 Oct 2026 Sep 2026
ISA-Bench Models write programs for five games with restricted instruction sets. A verifier checks each program and gives feedback before the next attempt. IIT Hyderabad and Microsoft Research India Puzzles and chess University Not in cards Gemma 4 31B IT 60.9% Sep 2026 Apr 2026
KARLBench Agents answer questions using vector search over fixed document collections. The suite measures six kinds of evidence gathering and reasoning. Databricks AI Research Questions that need search University Not in cards Claude Opus 4.6 67.5 Mar 2026 Feb 2026
LLM-AggreFact Models judge whether claims are supported by a grounding document across 11 pooled datasets, such as RAGTruth and ExpertQA. Scored by balanced accuracy. UT Austin + Salesforce AI Research (Tang, Laban and Durrett) Faithful to sources Everyday Not in cards GPT-4o 75.9% Apr 2024 Dec 2024
Local LLM Arkanoid Test Agents build a mouse-controlled brick breaker in one HTML file. They can revise it before a separate Claude agent checks 11 requirements in a browser. AOX.cz Web and app front ends University Not in cards gpt-oss-20b 11.0 May 2026 Apr 2026
Local LLM Tetris Test Agents build Tetris in one HTML file and can revise it after playing. A separate Claude agent checks piece movement, line clearing, scoring, and restart. AOX.cz Web and app front ends University Not in cards Gemini 2.5 Flash 11.0 May 2026 Apr 2026
MCP-SafetyBench Agents complete tool-use tasks across five domains. The benchmark reports both task completion and resistance to adversarial inputs. MCP-SafetyBench authors Jailbreaks and prompt injection University Not in cards o4-mini 21.2% Dec 2025 Sep 2025
Medical Chronology LLM Benchmark Models extract medical timelines from six synthetic document sets in three independent rounds. Checks compare extracted events with reference timelines and assess semantic fidelity, formatting, date order, and hallucinations. A weighted composite combines these checks with ROUGE and extraction F1. Superinsight, Inc. Medicine Expert Not in cards Claude Opus 4.6 88.9 Apr 2026 Mar 2026
MedSafetyBench Models respond to requests that conflict with medical ethics. A judge rates harmfulness on a five-point scale using a held-out set of 900 requests. Tessa Han and colleagues Refusing harmful requests Everyday Not in cards Llama 2-13B 1.44 Mar 2024 Jul 2024
micro1 Medical Index Models are scored on the equal-weight average of micro1 medical benchmarks, currently RealmPathology, PharmaBench and WellnessBench. micro1 Medicine Expert Not in cards GPT-6 Astra 71% Oct 2026 Nov 2026
MME-CoT Models reason step by step on image questions in math, science, OCR, logic, space-time and general scenes. A judge scores reasoning quality, robustness and efficiency. MME team (CUHK MMLab and others) General visual questions University Not in cards Kimi k1.5 64.2% Feb 2025 Jan 2025
MMEnterprise-Bench Enterprise agents answer 1,613 questions over workplace documents, chats and figures that change over time. They must resolve conflicting versions and say when evidence is not enough. MLV Lab (mlvlab) Simulated companies Expert Not in cards GPT-5.6 Sol 68% Oct 2026 Sep 2026
MMK12 Models use images to answer school mathematics and science questions. The test split contains 2,000 problems with reference answers. MM-EUREKA team Visual math and diagrams School Not in cards o1 73.9% Apr 2025 Mar 2025
MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. University of Tokyo (Li Lab) and collaborators Knowledge in many languages University Not in cards Qwen3-235B-A22B 74.1% May 2025 Apr 2025
MMTU Models answer about 30,000 questions over real tables across 25 tasks, such as data cleaning, table joins, column and table transformation, table question answering and turning language into code. The tasks come from decades of computer science research on tabular data. Answers are checked automatically against references. Microsoft and University of Michigan Data science and SQL Expert Not in cards GPT-5 69.6% Jun 2025 Aug 2025
NeurIPS 2025 Experts Benchmark Models answer tasks submitted by participants at NeurIPS 2025 in San Diego. Kaggle reports an overall score and individual task outcomes. Kaggle Language reasoning University Not in cards Qwen3-Coder-480B-A35B 73.3% Dec 2025 Dec 2025
NOTA Models extract facts from sheet music, convert score images to ABC notation, and analyze musical structure. The authors report extraction rates and separate transcription distances. Mingni Tang and colleagues Documents, charts and text in images University Not in cards MiniCPM-Llama3-V 2.5 17.8% Jun 2024 May 2024
OdysseyArena Agents explore interactive environments built from four transition primitives and must work out the hidden dynamics over long horizons. OdysseyArena-Lite has 120 standard tasks. OdysseyArena authors Puzzles and chess Expert Not in cards Gemini 3 Pro 44.2% Feb 2026 Dec 2025
OmniGameArena Models play games while response latency advances the game clock. OmniGameArena authors Video games Everyday Not in cards GPT-5.5 0.47 Jun 2026 Apr 2026
OmniRouting Models route copper traces between pads on printed circuit boards. The evaluator counts required nets that connect without design rule violations. Pennsylvania State University and collaborators Electronics University Not in cards GPT-5.5 28% Jul 2026 May 2026
OpenGameEval Debug Evals Agents diagnose and fix faults in Roblox Studio games. The authors created 30 debugging tasks from 15 scenarios. Roblox Fixing real repositories University Not in cards Claude Fable 5 64.7% Mar 2026 Jun 2026
PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. UniPat AI Research science Research frontier Not in cards GPT-6 Astra 14% Oct 2026 Sep 2026
PeerQA Models answer peer reviewers' questions about a scientific article using evidence from the article. UKP Lab Research science Expert Not in cards Mistral-7B-Instruct-v0.2 0.24 Feb 2025 May 2025
PMBench Agents search product managers’ notes to identify evidence across customer conversations and answer questions about product requirements. Databricks AI Research Questions that need search University Not in cards Claude Opus 4.6 46.1 Mar 2026 Feb 2026
Policy Agent Models approve expenses, reject them, or request human review. Ramp compares their decisions with human expense reviews. Ramp Labs Finance University Not in cards GPT-5.4 100% Mar 2026 Mar 2026
Polish Medical Exams Models answer Polish medical, dental, and specialist examination questions. The benchmark also includes English editions of the medical and dental licensing exams. Adam Mickiewicz University and collaborators Medicine Expert Not in cards GPT-4o 79.9% Nov 2024 Sep 2024
ProcessBench Models read step-by-step solutions to GSM8K, MATH, OlympiadBench and Omni-MATH problems and name the earliest wrong step, or say none is wrong. Scored by F1. Qwen Team, Alibaba Group Competition problems with answers Competition Not in cards o1-mini 87.9% Dec 2024 Dec 2024
Progress-Bench Vision-language models estimate how far a task has progressed from partial observations, such as frames of a robot or human doing a job. The benchmark varies how the demonstration is shown and the camera viewpoint, and it includes cases with no valid answer. ProgressLM team Spatial reasoning and counting Everyday Not in cards GPT-5 mini 20.9 Jan 2026 Oct 2025
ProofRank Models write proofs for hard competition problems. An LLM judge compares correct proofs pairwise on conciseness, computational ease, simplicity, diversity and adaptivity, giving Elo ratings. MathArena (ETH Zurich SRI Lab and INSAIT) Written proofs Competition Not in cards GPT-5.4 1,429 May 2026 Mar 2026
Q-Bench Multimodal models are tested on low-level vision: perceiving image distortions, describing them and predicting quality scores. Nanyang Technological University and collaborators General visual questions Everyday Not in cards Qwen-VL-Max 73.9% Sep 2023 Jan 2024
RealWorkDecision Agents review a frozen company workspace and predict the decision an employee made. Expert-written rubrics allow valid alternatives. micro1 Tasks from real jobs University Not in cards Grok 4.7 43.8% Oct 2026 Oct 2026
RedstoneBench Agents design redstone circuits in Minecraft. A game server tests whether each circuit works and measures the space it occupies. Adam Holter Video games Expert Not in cards GPT-6.2 Astra 88.7 Nov 2026 Nov 2026
Right Answer, Wrong Reason: ML Judgment Calls Models make 36 decisions about tabular machine learning. The benchmark compares choosing from options with giving advice freely against measured outcomes. starkhushi Machine learning engineering University Not in cards Claude Sonnet 5.5 32.0 Oct 2026 Sep 2026
RoCS-MT v2 Models translate Reddit posts with revised spelling annotations and corrected references. The authors combine COMETKiwi and MetricX rankings for translations of the original text. Inria Translation Everyday Not in cards GPT-4.1 7.00 Nov 2025 Apr 2025
RSLiveBench-UAV Models answer questions about vehicles in drone images. The benchmark checks perception and spatial reasoning across different views and adds newly captured scenes. East China Normal University and collaborators Embodied reasoning Everyday Not in cards GPT-6 Astra 59.9 Nov 2026 Oct 2026
RTLScout Bench (suite-v2) Agents rewrite five circuits to reduce the product of chip area and signal delay. The leaderboard ranks configurations by their mean rank across circuits. Felix Arnold Electronics Expert Not in cards MiMo-V2.6 Pro 1.00 Oct 2026 Oct 2026
scBench-Long (original study) Agents complete 21 tasks in long single-cell data analysis workflows. The authors compare their answers with results from the original analyses. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 4.8 25.4% Jun 2026 May 2026
SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. Tianyun Zhong and colleagues Research science Expert Not in cards GPT-6 Astra 80.9 Oct 2026 Sep 2026
SIM-VAIL Chatbots respond to simulated users with psychiatric vulnerabilities. A model judge scores behaviors across the conversation, including responses that reinforce a user’s vulnerability. Veith Weilnhammer and colleagues Wellbeing and sycophancy Expert Not in cards GPT-6 Astra 1 Feb 2026 Sep 2026
SLATE Models generate slides that teach unfamiliar language patterns. Simulated learners take tests before and after the lesson and apply the rules to new examples. Beijing Normal University and collaborators Other professions University Not in cards Claude Opus 4.7 5.46% Sep 2026 Apr 2026
Spider 2.0-AIFunc Agents generate SQL that uses AI functions to solve 465 tasks across 125 databases. The evaluator compares execution results with reference results. UC San Diego, Snowflake AI Research and University of Hong Kong Data science and SQL University Not in cards Claude Opus 4.6 70.3% Jul 2026 Mar 2026
STALE Agents with long-term memory must notice when a later observation quietly makes an earlier memory false. The benchmark has 400 expert-validated conflict scenarios and 1,200 queries over more than 100 everyday topics, with contexts up to 150K tokens. Queries test whether the model spots the outdated belief, rejects a false premise and acts on the new state. HKUST NLP Group Learning from the context Everyday Not in cards Gemini 3.1 Pro 55.2% May 2026 Mar 2026
SuperCLUE-Image Models generate images from Chinese prompts. The benchmark scores text rendering, visual quality, prompt adherence and scene composition. SuperCLUE Image generation and editing Everyday Not in cards GPT-Image-2.5 Sunburst 94.3 Jun 2024
SuperCLUE-Law Models respond to Chinese legal questions. SuperCLUE Law Expert Not in cards Kimi K3 92.1 Aug 2026
SuperGLUE Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score. NYU, Facebook AI, University of Washington, and DeepMind Language reasoning Everyday Not in cards PaLM (540B) 90.4% May 2019 Apr 2022
SWE-CoWork Coding agents join a simulated engineering team on 101 real repositories and must gather scattered requirements from simulated colleagues and workplace records before implementing them. Hidden tests score the share of new-behavior tests passed and whether the whole task passes. TokenWave AI Fixing real repositories Expert Not in cards GPT-6 Astra 7.92% Nov 2026 Sep 2026
SWE-Serve Coding agents solve 53 production inference-engineering tasks derived from merged SGLang pull requests. Hidden tests measure task success, cost, agent steps and wall time. NVIDIA Fixing real repositories Expert Not in cards GPT-5.6 Sol 75.5% Oct 2026 Jul 2026
The Aggregate Asteroid Numbers Models give the permanent numbers of 48 named asteroids from memory. The scorer checks each answer against the NASA Jet Propulsion Laboratory catalog. Mikhail Doroshenko (The Aggregate) Trivia and open questions Expert Not in cards DeepSeek-V4-Pro 58.3% Sep 2026 Nov 2026
TREC BioGen 2024 Systems retrieve biomedical literature and synthesize evidence into reports that answer a question. TREC BioGen track organizers Medicine Expert Not in cards Claude Opus 4.6 79.9 Feb 2026
VCBench Models predict venture outcomes from founder profiles. The benchmark emphasizes precision through the F0.5 score. The hub adaptation substitutes synthetic profiles for the original data. University of Oxford and Vela Research Finance Expert Not in cards GPT-6 Sol 33.7% Sep 2025 Sep 2026
WeatherQA (severe weather) Models read weather maps to identify a threatened area and classify its severe weather potential. The second score includes only cases with the correct area. University of Washington and collaborators Science questions Expert Not in cards Qwen2.5-VL-7B 62.7% Jun 2024 Dec 2025
WeaveBench Computer-use agents do 114 long tasks in 8 work domains that mix GUI actions with command-line and code operations in one trajectory. Results are checked against verifiable artifacts. Microsoft and collaborators Desktop apps Expert Not in cards Claude Opus 4.7 41.2% Jun 2026 Jun 2026
Web Clone Bench Agent gets a running reference app (Gmail, Jira, Figma, etc.) and 8 hours to build a clone. Expert-written workflows are replayed as end-to-end tests. Plato Building whole projects Research frontier Not in cards Claude Fable 5.2 62% Oct 2026 Nov 2026
AgencyBench Agents complete 138 long tasks drawn from daily AI use, grouped into 32 scenarios that cover six agentic capabilities such as coding, front-end work, game development and research. A typical task takes about 90 tool calls, 1 million tokens and hours of work. A simulated user gives feedback during the run, and a Docker sandbox checks the deliverables against task rubrics. Shanghai Innovation Institute (SII) GAIR Lab Long terminal sessions Expert Not in cards GPT-5.2 56.5% Jan 2026 Dec 2025
AgentCLUE-Mobile (offline) Agents select actions in an offline mobile interface. SuperCLUE Phones Everyday Not in cards GLM-4.5V 88.4 Aug 2025
AgenticFinBench Agents complete 230 financial requests in two languages. Graders apply task-specific rubrics and audit unsupported claims against external evidence. NashNova Research Finance Expert Not in cards GPT-5.6 Sol 89.2 Nov 2026 Sep 2026
AgroBench Models answer illustrated questions from agronomists about crops, diseases, pests, weeds, machinery and farming practices. The overall score averages accuracy across seven tasks. Risa Shinoda and colleagues General visual questions University Not in cards GPT-4o 73.5% Jul 2025 Aug 2024
AIRTBench Models act as autonomous AI red teamers on 70 black-box capture-the-flag challenges from Dreadnode's Crucible platform. Each challenge hides a flag behind an AI or ML system, and the model writes Python code to attack it with methods such as prompt injection, evasion and model inversion. A challenge counts as solved when the model submits the correct flag. Dreadnode Capture-the-flag challenges Expert Not in cards Claude 3.7 Sonnet 46.9% Jun 2025 May 2025
ANCHOR Models explain answers to audiovisual questions and locate the supporting evidence. The benchmark includes time intervals, image regions, and relevant sounds. Joel Julin and colleagues Audio understanding and voice Everyday Not in cards Gemini 3 Flash 3.36 Oct 2026 Apr 2026
AsyncTool Agents coordinate several tool-use tasks while tool responses arrive after simulated delays. The evaluator checks function calls, parameters, task completion, and the final environment state. Scheduling metrics measure whether agents use waiting time effectively. University of Science and Technology of China and University of Toronto Function calling Everyday Not in cards GPT-4.1 38.1% May 2026 Sep 2025
AutoBencher Models answer generated benchmark questions. The framework searches for topics and produces tests of knowledge, multilingual reasoning, and math. Xiang Lisa Li and colleagues Language reasoning University Not in cards Claude 3 Opus 55% Jul 2024 Apr 2024
AutoformBench Agents fill in missing Lean proofs for mathematical statements and submit their solutions to a clean grading workspace. Meta FAIR Formal proofs Expert Not in cards GPT-6 Astra 82.4% Oct 2026 Sep 2026
Automotive-ENV Agents operate climate controls in an Android Automotive OS simulator. A validator checks the screen and vehicle signals after requests such as turning on the defroster. University of Technology Sydney and University of Liverpool Apps and MCP servers Everyday Not in cards Kimi K2.5 49.3% Sep 2025 Feb 2026
BallotBench Nine frontier agents forecast 863 contests in the 2026 US midterms each day until polls close. Proper scoring rules grade forecasts against certified results, beside Kalshi, Polymarket and expert ratings. Thoughtful Lab Real-world events Expert Not in cards Muse Wonder 0.11 Oct 2026 Oct 2026
Big Code Models Leaderboard Models generate programs in multiple languages. The board compares pass rates and averages each model’s win rate across languages. BigCode Function-level problems University Not in cards Qwen2.5-Coder-32B-Instruct 59.2% Jul 2023 Nov 2024
BIRD-CRITIC Models fix real user SQL issues in MySQL, PostgreSQL, SQL Server and Oracle applications. Test cases check each fix. BIRD team (HKU) Data science and SQL Expert Not in cards o1-preview 35.5% Jan 2025 Mar 2025
BrokenMath Models are asked to prove false versions of 2025 competition problems. An LLM judge counts how often a model writes a proof instead of rejecting the statement. INSAIT and ETH Zurich SRI Lab Written proofs Competition Not in cards GPT-5 29% Oct 2025 Sep 2025
Browser Use LinkedIn Agents answer plain-language questions about LinkedIn using recorded page snapshots. The leaderboard reports correctness, cost, and time per task. lowgular Web browsing Everyday Not in cards Claude Haiku 5.5 97.9 Nov 2026 Nov 2026
Bug in the Code Stack Models find a syntax error inserted into a long Python codebase. Hamming measures retrieval accuracy across context lengths and error positions. Hamming AI and Wat.ai Finding facts in long inputs School Not in cards GPT-4o 82.7% Mar 2025
ChemArgus Models solve chemistry competition and textbook problems with text and figures. A model judge awards partial credit for each subquestion. Hanyao Guo and colleagues Science questions Competition Not in cards Claude Opus 5 64.9% Oct 2026 Sep 2026
clembench Chat models play rule-based dialogue games in self-play, such as Taboo, Wordle and reference games. The clemscore combines how often a model follows the game rules at all with how well it plays when it does. Version 4.0 runs 14 games in 30 languages. University of Potsdam (Foundations of Computational Linguistics) Multi-agent games Everyday Not in cards GPT-5.4 88.8 May 2023 Jul 2026
ClinHallu Models answer medical image questions and produce intermediate reasoning. The evaluation reports answer accuracy and separate hallucination rates for visual recognition, knowledge recall, and reasoning integration against annotated reasoning traces. Alibaba DAMO Academy Medicine Expert Not in cards Gemini 3 Flash 80.1% Jun 2026 Feb 2026
CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. M-A-P and collaborators General visual questions University Not in cards Qwen-VL-Plus 36.8% Jan 2024 Aug 2024
CodeElo Models solve Codeforces contest problems through the platform's judging system. The harness converts contest outcomes into human-comparable Elo ratings and percentiles. Qwen team, Alibaba Competitive programming Competition Not in cards o1-mini 1,578 Jan 2025 Nov 2024
ContextBench Coding agents resolve 1,136 issues from 66 repositories in eight programming languages, and the benchmark checks which code they look at along the way. Each task has human-annotated gold context, and the tracker scores context recall, precision and efficiency over the agent's trajectory. It adds these process measures to the usual issue-resolution result. Nanjing University and University College London Fixing real repositories Expert Not in cards GPT-5 60.6% Feb 2026 May 2026
CookBench v1 (CookWing) Models give ingredient quantities for carbonara, pastitsio, and melon pan. A run passes when all required amounts and ratios fall within the dish’s specified ranges. Tatsuo, CookWing Language reasoning Everyday Not in cards GPT-5.6 Sol 40% Aug 2026 Jul 2026
Creative Control: LLMs as a Film Crew Models plan a film from concept to shot list while a director revises the brief. Separate tasks evaluate continuity and camera choices. ChenLuFried Creative writing University Not in cards GPT-5.5 0.94 Oct 2026 Aug 2026
CRMArena Agents do nine customer-service CRM tasks for service agent, analyst and manager personas inside a real Salesforce org. The org holds 16 linked object types with seeded hidden patterns. Salesforce AI Research Customer service Everyday Not in cards o1 64.3% Nov 2024 Jan 2025
CRMArena-Pro Agents do 19 expert-checked sales, service and quoting tasks in B2B and B2C Salesforce orgs, in single-turn and persona-driven multi-turn settings. It also tests whether agents keep confidential data private. Salesforce AI Research Customer service Everyday Not in cards Gemini 2.5 Pro 56.2% May 2025 Apr 2025
CUA-bench Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out. Vals AI Video games Everyday Not in cards GPT-6 Astra 19.2% Sep 2026 Oct 2026
DataCenterBench Tier 1 Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos. Robocurve Real robots Everyday Not in cards GPT-6 Astra 40% Oct 2026 Oct 2026
DECIDE Models select decisions from short voice-call clips and conversation history across 4,867 questions, 1,081 clips, and 32 tasks. Human reference labels determine accuracy. The default board ranks by DECIDE Balance Score, which combines accuracy, cost, and latency on a zero-to-100 scale. Tazo Labs and Bolna Audio understanding and voice Everyday Not in cards Qwen2.5-Omni-7B 66.6 Nov 2026 Sep 2026
DeepTerminalBench Agents get one attempt at each of 50 deep engineering tasks in Docker, in the Terminal-Bench 2.0 format. A task passes only if every hidden test passes. Tasks were calibrated so Claude Opus 4.6 still fails some at pass@4. UniPat AI Terminal tasks Expert Not in cards Claude Opus 4.7 34% May 2026 Apr 2026
DISBench Multimodal agents search raw visual histories over several steps to find target images from implicit context cues. NLPIR Lab, Renmin University of China General visual questions Everyday Not in cards Claude Opus 4.5 28.7% Feb 2026 Dec 2025
DuetBench Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head. Decagon Customer service Expert Not in cards Claude Opus 5 1,035 Jun 2026 Aug 2026
EnterpriseClawBench Agents do 852 tasks rebuilt from real proprietary workplace agent sessions, each with fixtures, hard rules and semantic rubrics. Frontis AI Documents and spreadsheets Expert Not in cards GPT-5.5 66.3% Jun 2026 Jun 2026
Era by Eon Generates a simulated enterprise (Salesforce, Slack, Jira, Zendesk, etc.) with exact ground truth. Agents answer questions through MCP/REST. Eon Apps and MCP servers University Not in cards Claude Opus 4.8 76.8% Sep 2026 Jul 2026
EusProficiency Models complete Basque language exercises from the official C1 proficiency examination. The questions test vocabulary and grammar. Julen Etxaniz and colleagues Reading in many languages University Not in cards GPT-4 Turbo (Jan 2024) 56.7% Mar 2024 Jan 2024
FEM-Bench Models implement finite element functions and write tests for them. The evaluator checks numerical outputs and whether the tests distinguish reference functions from faulty implementations. Lejeune Lab at Boston University and collaborators Scientific code University Not in cards Gemini 3 Pro 87.9% Aug 2025 Nov 2025
Fiction Bench (Foreverse) Models continue two Chinese novels for 20 rounds and receive their earlier writing as context. Reviewers rank anonymous continuations for each genre. Foreverse Research Creative writing University Not in cards DeepSeek-V4-Pro 2.50 Jul 2026 Jul 2026
FinCrime Agent Benchmark Agents screen customers for sanctions, political exposure, and adverse media. A case passes when agents find the relevant alerts and resolve them under the bank's policy. Castellum.AI Finance University Not in cards GPT-6 Astra 71.1% Sep 2026 Sep 2026
FinHopper Agents research financial facts through up to six dependent searches. Intermediate answer keys show where an incorrect company, date, or indicator derails the search. South China University of Technology and Tencent Finance University Not in cards Claude Sonnet 4.5 67.3% Oct 2026 Dec 2025
Firecrawl Search Bench Agents answer hard research questions with web search. A deterministic grader checks each answer against reference facts. Firecrawl Deep research Expert Not in cards Claude Opus 5.5 79% Nov 2026 Oct 2026
FORTE-MM Agents complete multimodal office tasks across 14 professions. A run passes only when the agent's output satisfies every task rubric. AGI-Eval (Meituan) Documents and spreadsheets University Not in cards Gemini 3.8 Flash 74.5% Nov 2026 Sep 2026
FrontierRSI Agents do 27 open-ended AI research and engineering tasks across pre-training, post-training and harnesses, inference serving and GPU kernels. Each trial is scored against the best valid result any model reached. Abundant AI AI research tasks Research frontier Not in cards Claude Opus 5.5 34.3% Nov 2026 Nov 2026
GAIK Ajokortti transcription test Models transcribe one three-minute Finnish dental webinar excerpt with two speakers. The project reports word error rate and separately counts errors that change the meaning. GAIK project, Haaga-Helia University of Applied Sciences Speech recognition and translation Everyday Not in cards Gemini 3.5 Flash 1.00% Jan 2026 Jul 2026
GameCraft-Bench Coding agents build playable games end to end inside a real game engine, where scripts, scenes, assets and runtime behavior must work together. CUHK Shenzhen Building whole projects Expert Not in cards GPT-6 Astra 75.9% Jun 2026 Sep 2026
GeoDrive-Bench Models choose driving actions from road images and local traffic conventions. The benchmark covers six countries and omits explicit country labels. Yingzi Ma and colleagues Embodied reasoning Everyday Not in cards GPT-5.4 91.9% Jun 2026 Mar 2026
German fruit-story antipresuppositions Models read short German stories and choose an article or quantifier that fits the number of fruits. A later task asks whether a continuation sounds appropriate. Judith Sieker and Sina Zarrieß Reasoning in many languages Everyday Not in cards GPT-5 100% Dec 2023 Sep 2025
GomokuBench Models place stones on a 19 by 19 board against a search engine. The score rewards wins, quicker wins, and longer losses. Homer Quan Multi-agent games Everyday Not in cards Gemini 3.8 Flash 2.90 Apr 2026 Oct 2026
Grade Your Own Homework: Can LLMs Judge Each Other Models solve problems and grade answers, including their own. The benchmark varies answer order, author names, and stated confidence against labels computed by code. Anish_23101 Language reasoning School Not in cards Gemini 3.7 Flash 0.97 Oct 2026 Sep 2026
Graduate Admissions Agent Benchmark Agents advise students on graduate admissions and revise plans when the student's information changes. Seventy Chinese-language tasks span 23 cases and 83 answer stages. Separate reviewers score responses against task-specific rubrics on a 100-point scale. 屿智同行 Other professions Expert Not in cards GPT-6.1 Sol 79.1 Oct 2026 Sep 2026
HalluDial Models detect hallucinations in dialogue. BAAI FlagEval Hallucination rates Everyday Not in cards GPT-4 Turbo (Jan 2024) 80.2 Jun 2024 May 2024
HarvestBench Agents drive simulated tractors and decide how much fuel to spend avoiding animals. The benchmark measures how often they hit animals as avoidance costs change. Jasmine Brazilek and colleagues Deception and alignment Everyday Not in cards GPT-5.6 Terra 0.40% Sep 2026 Sep 2026
IMProofBench Agents with web search and SageMath write proofs for private PhD-level problems written by mathematicians. Experts grade the proofs, and final-answer subquestions are graded automatically. ETH Zurich (Johannes Schmitt et al.) with a contributor community Research mathematics Research frontier Not in cards GPT-5.4 49.3% Sep 2025 Mar 2026
Indic-KCC-Agri-Advisory-Benchmark Models answer crop questions from India’s Kisan Call Center in English and ten Indian languages. A judge compares each answer with a human advisor’s response. Sthānika AI Knowledge in many languages University Not in cards DeepSeek-V4-Flash 4 Sep 2026 Apr 2026
INQUIRE-Rerank Models reorder 100 photographs supplied for each ecological query. The benchmark measures average precision among the first 50 results. INQUIRE team General visual questions University Not in cards GPT-4o 62.4% Jun 2024 Mar 2025
Into the Unknown Agents get a science paper and an unpublished follow-up direction from a DOE national lab scientist, then write the follow-up paper. Rubrics built from the scientist's conditions grade it, plus an integrity check. Prometheus-Eval team with US DOE national laboratories (Seungone Kim et al.) Research science Research frontier Not in cards Claude Opus 4.8 0% Oct 2026 Sep 2026
KernelBench (kernelbench.com) Coding agents write GPU kernels for real model workloads, such as fused MoE layers, paged attention and FP8 matrix multiplies. The site groups tasks into suites: Mega (one decode task), CUDA (4 tasks) and Hard (6 tasks). A run counts only if it passes an audit of correctness. Scores are speedup over a baseline or percent of the hardware roofline. Elliot Arledge (independent) GPU kernels Expert Not in cards DeepSeek-V4-Pro 1.00 Jul 2026 Oct 2026
L2-Bench Models design lessons, give learners feedback, and participate in language practice. A model judge grades each response against teaching criteria. James Edgell and colleagues Other professions University Not in cards Claude Opus 4.7 85.5 Jul 2026 Apr 2026
LIBRA (Russian Long Context) Models read long Russian texts and answer questions or generate responses. The suite groups tasks by difficulty and varies input length. Igor Churin and colleagues Reasoning over long documents University Not in cards Qwen3-30B-A3B-Instruct 71.3 Aug 2024 Feb 2026
LMentry Models perform simple language tasks such as identifying words with a given letter or choosing the longer word. Task-specific rules check the answers. Avia Efrat and colleagues Language reasoning Everyday Not in cards davinci-002 66.1% Nov 2022 Mar 2022
LongCoT Models solve long reasoning problems across five domains. Task-specific programs verify answers, with optional judge fallback for ambiguous mathematics and chemistry responses. Sumeet Ramesh Motwani and colleagues Reasoning over long documents Expert Not in cards GPT-5.2 25.1% Apr 2026 Feb 2026
Longtail-SWE Agents resolve 202 GitHub issues across 13 programming languages with limited benchmark coverage. Hidden tests check the fix and verify that previously passing behavior still works. Vladimir Dobrovolskii and colleagues Fixing real repositories Expert Not in cards Claude Opus 5.5 81.5% Nov 2026 Sep 2026
LoopsBench Agents complete 112 software tasks with dependencies among smaller development units. Executable tests measure resolved tasks and the fraction of units that pass. Microsoft Long terminal sessions Expert Not in cards Claude Opus 4.7 25% Jul 2026 Apr 2026
M-RewardBench Reward models choose preferred responses across chat, safety, reasoning and translation tasks in 23 languages. Accuracy measures preference selection. Cohere Labs Community Overall chat quality University Not in cards GPT-4 Turbo 83.5% Oct 2024 Oct 2024
MatCha Models answer multiple-choice questions about material structure, shape and properties using images from materials characterization instruments. Chinese University of Hong Kong, Shenzhen, Northeastern University and Zhejiang University Lab and data work Expert Not in cards Llama 4 Maverick 68.1% Sep 2025 Apr 2025
MCPTox Agents use tools whose descriptions contain untrusted instructions. The benchmark measures whether agents follow those instructions across 1,348 cases from 45 servers. MCPTox authors Jailbreaks and prompt injection University Not in cards Qwen3-14B 5.10% Aug 2025 Apr 2025
MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. Shanghai AI Laboratory and collaborators Medicine Expert Not in cards Claude Opus 4.7 69.2 Jun 2026 May 2026
MultiFinBen Models complete 36 financial datasets across five languages and text, vision, and audio. The board reports a modality-balanced score. The Fin AI (Qianqian Xie and coauthors) Finance University Not in cards Qwen2.5-Omni-7B 33.5 Jun 2025 Apr 2025
MultiFinGround Models identify accounting concepts and extract every matching fact from English, French, and Greek financial reports. The benchmark checks their selections against annotated disclosures. University of Manchester and collaborators Finance University Not in cards Gemini 3.8 Flash 95.4% Oct 2026 Sep 2026
MyPCBench Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks. Carnegie Mellon University Desktop apps Everyday Not in cards Claude Opus 4.8 62% Jun 2026 Jul 2026
NoLiMa-Hard Models retrieve latent associations for the ten hardest NoLiMa question-needle pairs. Results are reported separately at four context lengths. Adobe Research and LMU Munich (Modarressi et al.) Finding facts in long inputs Everyday Not in cards o3 78.5% May 2025
NotSoTiny 25-12 Models reconstruct a missing Verilog module from surrounding hardware design code. A formal checker compares the result with the original module. Barcelona Supercomputing Center Electronics University Not in cards Kimi K2.5 31.6% Dec 2025 Apr 2026
NPHardEval Models solve algorithmic decision and optimization problems that vary in computational complexity. NPHardEval team Language reasoning University Not in cards GPT-4 Turbo 37.7% Dec 2023 Apr 2024
OCT-Bench Models answer 10,076 questions about retinal scans, from anatomy to treatment and prognosis. The evaluator measures answer accuracy and counts invalid responses as incorrect. Shandong University, Joint SDU-NTU Centre for Artificial Intelligence Research, and collaborators Medicine Expert Not in cards GPT-5.4 Mini 62% Jul 2026 Mar 2026
OdysseyArena AI Trading Agents trade under hidden market dynamics. The score is average return relative to initial capital over four trials. OdysseyArena authors Puzzles and chess Expert Not in cards gpt-oss-120b 23.3% Feb 2026 Dec 2025
OdysseyArena Energy Dispatch Agents manage energy dispatch under hidden rules. The score is average task success over four trials. OdysseyArena authors Puzzles and chess Expert Not in cards GPT-5 23.3% Feb 2026 Dec 2025
OdysseyArena Repo System Agents operate repositories with changing hidden rules. The score is average task success over four trials. OdysseyArena authors Puzzles and chess Expert Not in cards GPT-5 62.5% Feb 2026 Dec 2025
OdysseyArena Turn on Lights Agents learn hidden switches and turn on lights. The score is average task success over four trials. OdysseyArena authors Puzzles and chess Expert Not in cards GPT-5 28.3% Feb 2026 Dec 2025
Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. Alibaba Group (Qwen, Alibaba Token Hub) + Hong Kong Polytechnic University Video understanding Everyday Not in cards Gemini 3.5 Flash 20.2% Oct 2026 May 2026
OmniLayout Models place components inside a board outline. The evaluator checks overlap, routing, and electrical functionality against the schematic and the reference layout. Pennsylvania State University and collaborators Electronics University Not in cards Gemini 2.5 Flash-Lite 0.76 Jun 2026 Apr 2026
Open Tabletop GM Session Command Probe Models handle five game master commands and conversational turns, including loading a campaign and rolling dice. Pattern checks classify each response. Bobby Gray Multi-turn conversation Everyday Not in cards Llama 3.3 70B Instruct 3.00 Apr 2026 Apr 2026
OSWorld-MCP Computer-use agents complete OSWorld desktop tasks while they can also call 158 MCP tools across 7 common applications. The benchmark checks whether agents pick a tool call or GUI actions at the right time. It reports task accuracy, tool invocation rate and average completion steps. Peking University, Tongyi Lab (Alibaba) and Beijing Zhongguancun Academy Desktop apps Everyday Not in cards Kimi K2.5 56.6% Oct 2025 Feb 2026
P3D-Bench Models generate parametric 3D objects from text, images, or assembly instructions. The evaluator checks whether outputs run and preserve geometry, topology, and part structure. Yikang Yang and colleagues Image generation and editing University Not in cards GPT-6 Astra 69.4 Jun 2026 Sep 2026
PatientAgentBench Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria. Amazon Medicine Expert Not in cards Claude Opus 4.8 4.25 Jul 2026 Jun 2026
PhysicalRealismBench-U VLM watches 1,300 synthetic Isaac Sim videos, flags physics-law violations, names object and frames. Scored by joint realism F1. Reka Physical prediction Everyday Not in cards GPT-5.5 60.8% Jun 2026 May 2026
PlanBench-XL Tool-use agents solve 327 retail tasks over 1,665 tools that they must first retrieve, chaining calls to uncover evidence. An optional blocking mode simulates tool failures. University of Illinois Urbana-Champaign Function calling Expert Not in cards Gemini 3.1 Pro 77.1% Jun 2026 May 2026
Portal2Bench Agents solve Portal 2 chambers using screenshots, position readings, and game controls. Each chamber counts if the agent reaches the next map within $30 and 300 plans. Kingston Yip Video games Everyday Not in cards GPT-6.2 Astra 26.0 Nov 2026 Nov 2026
PowerBench Agents retrieve equipment records and telemetry to answer 300 power-system questions. Macro Joint Accuracy averages fully correct answers equally across three task families. OpenCompass PowerBench team General assistants Expert Not in cards GPT-5.6 Sol 74.2% Sep 2026 Nov 2026
Prediction Arena Agents research events and trade contracts on Kalshi and Polymarket. The leaderboard ranks accounts by cash plus the market value of open positions. Arcada Labs and Harvard University Real-world events University Not in cards Claude Opus 4.6 $138,945 Jan 2026 Mar 2026
Preference Proxy Evaluations Reward models and judges rank responses. The benchmark compares their decisions against user preferences and verifiable answers. Arena Overall chat quality University Not in cards Claude 3.5 Sonnet 68.5% Oct 2024 Sep 2024
ProgramDistill Agents inspect working web applications and restore missing features in editable copies. Replayed browser traces check whether patches recover the reference behavior. The paper evaluates partial repairs and full application reconstruction. Microsoft Research, Microsoft AI, and KAIST Web and app front ends University Not in cards GPT-6 Astra 84.3% Sep 2026 Sep 2026
QF-Bench Agents write and run code for 87 quantitative finance tasks in a network-isolated sandbox with real data. Tasks cover derivatives pricing, fixed income, credit, risk, factor research, backtesting and market microstructure. More than 3,300 assertions check numeric outputs against finance rules such as no-arbitrage bounds and put-call parity. Scored by pass@1 over three runs. QF-Bench team (anonymous NeurIPS 2026 submission) Finance Expert Not in cards GPT-5.5 61.7% May 2026 Apr 2026
Quant-Arena Models act as quant researchers on 2022 to 2023 China A-share data with bounded feedback, then commit one strategy. A cash-account evaluator with trading costs scores it once on a sealed 2024 window. Einsia AI (Navers Lab, LaPlace's Eye) Finance Expert Not in cards GPT-6 Astra 51.3 Oct 2026 Sep 2026
QuantEval Models answer quantitative finance questions, solve mathematical problems, and implement trading strategies. The authors report answer accuracy and test the generated code. Zhaolu Kang and colleagues Finance University Not in cards Claude Sonnet 4.5 86% Jan 2026 Sep 2025
Read My Aura Models respond to requests for aura readings. The tasks probe invented perceptions, flattery, and whether a change in a name or image tint changes the reading. ashley e. romano Wellbeing and sycophancy Everyday Not in cards DeepSeek-R1 1.00 Oct 2026 Sep 2026
RoboColiseum AgiBot Genie robot in Genie Sim 3.0 (Isaac Sim): instruction, robustness, manipulation and spatial boards after post-training on provided sim data. AgiBot Simulated robots Everyday Not in cards Claude Opus 5.5 76.3% Jan 2026 Oct 2026
Role-play Benchmark (MiniMax) Role-play agents are tested on situated reenactment in Chinese and English scenes. Grading targets the immersive experience rather than a verifiable answer. MiniMax Creative writing Everyday Not in cards MiniMax-M2-her 84.7 Jan 2026 Jan 2026
RVG-Bench Models locate the video interval that answers a question about sounds and images. The evaluator measures overlap with the annotated interval. Yue Feng and colleagues Audio understanding and voice Everyday Not in cards Qwen3.8-Omni-Flash 49.6% Oct 2026 Sep 2026
Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. AItonomy Foundation and the SLE team Research science Research frontier Not in cards GPT-6.1 Sol 44.7 Nov 2026 Oct 2026
SciKnowEval Models answer scientific questions and apply knowledge in biology, chemistry, materials and physics. The suite uses automatic checks and model judges for different tasks. Kehua Feng and colleagues Science questions University Not in cards o4-mini 0.71 Jun 2024 May 2025
SheetBench Agents solve finance and accounting spreadsheet tasks through a browser in a fresh Google Sheet. The score is the share of expected answer cells that match exactly. HUD Finance Expert Not in cards GPT-6 Astra 64.7% Oct 2026
Shop Assistant Honesty Bench (UA/RU) Models act as shop assistants in Ukrainian or Russian. The tests check language choice and whether quoted prices and stock match the catalog. DimaTopka Customer service Everyday Not in cards Gemma 4 31B IT 100% Oct 2026 Sep 2026
Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. Naïve Simulated companies Expert Not in cards GPT-6.1 Sol $2,949 Oct 2026 Sep 2026
SolidWorks Bench Agents model the custom parts of 73 real machines in SolidWorks and assemble them from raster drawing packages, through screenshots and the SolidWorks API. A grader checks the geometry against the reference and reads mates, motion and feature trees from the native files. Markov CAD models Expert Not in cards Claude Fable 5.2 0.57 Nov 2026 Nov 2026
SolveEdit Models edit an image to solve a visual problem. The evaluator checks whether the goal holds and whether unrelated scene details survive the edit. Wenjie Shu and colleagues Image generation and editing School Not in cards GPT-Image-2 57% Sep 2026 Feb 2026
Spatial Reasoning: Rotating ASCII Art Models rotate binary images represented as grids of text characters. The tasks vary the grid size and compare the output with the expected rotation. Neglectful Persimmon Spatial reasoning and counting School Not in cards Claude Sonnet 5 0.80 Oct 2026 Oct 2026
SpatialBench-Long (original study) Agents complete 24 tasks in long spatial transcriptomics workflows. The authors compare their answers with results from the original analyses. LatchBio Genomics and life sciences Expert Not in cards GPT-5.5 11.1% May 2026 May 2026
SPGISpeech Models transcribe English financial recordings. The references include formatting such as capitalization and punctuation. Kensho and collaborators Speech recognition and translation Everyday Not in cards Voxtral Small 1.84% Apr 2021 Oct 2026
SPOT Models inspect published manuscripts for errors that led to corrections or retractions. A judge checks whether each reported error matches the documented error and location. Guijin Son and colleagues Research science Expert Not in cards o3 6.10% May 2025 Apr 2025
SquadStack Conversational LLM Benchmark for Voice Agents Models write the next reply or tool call from the history of an Indian outbound phone call. A language model grades each response against seven criteria. SquadStack MLOps Team Customer service University Not in cards Gemma 4 31B IT 92.5% Nov 2026 Sep 2026
SuperCLUE-OnDevice Phone-based models respond to requests in Chinese. SuperCLUE Overall chat quality Everyday Not in cards Gemini 3.6 Flash 93.6 Aug 2026
SWE-Clar Agents question a simulated client about an incomplete request, then build greenfield software from real Upwork jobs. Acceptance tests score the build. Upwork Building whole projects Expert Not in cards GPT-6 Astra 67% Oct 2026 Sep 2026
The Aggregate Forecasting Resolved Questions Models estimate probabilities for resolved Manifold markets. The author drops questions when a model correctly recalls the outcome, then scores the remaining forecasts by squared error. Mikhail Doroshenko (The Aggregate) Real-world events Everyday Not in cards Claude Opus 5 0.27 Sep 2026 Jul 2026
ToolMaze Agents complete tool-use tasks while the runtime introduces explicit or hidden tool failures. A judge checks completion across four task structures and five perturbation settings. Metrics report task success, recovery rate, and recovery cost. Shanghai AI Laboratory, East China Normal University, Soochow University and others Function calling University Not in cards Gemini 3.1 Pro 53% Jun 2026 Apr 2026
TraceTrial-Bench Agent harnesses, not only models, are tested on reliability across seven components, with traces used to attribute each failure to its cause. TraceLite AI General assistants Expert Not in cards Claude Opus 5 77.6% Oct 2026 Aug 2026
Unsatz Agents design experiments and propose equations for an unknown simulated system. Their final program must predict held-out interventions and conditions beyond the original experiments. University of California, Los Angeles Research science Expert Not in cards GPT-6 Astra 60% Oct 2026 Sep 2026
UserBench Agents help simulated users who start with vague goals and reveal preferences bit by bit. Agents must ask clarifying questions and use tools to reach choices that fit the user. Salesforce AI Research General assistants Everyday Not in cards Claude Sonnet 4 0.61 Jul 2025 Feb 2026
VBVR-Bench Video generation models solve visual reasoning tasks by generating a video from a starting image and instruction. Rule-based scorers check the result instead of a judge model. Tasks cover abstraction, knowledge, perception, spatial reasoning and transformation, with in-domain and out-of-domain splits. Video-Reason (Maijunxian Wang, Ruisi Wang and others) Video world models Everyday Not in cards Seedream 5.0 Pro 0.56 Feb 2026 Jul 2025
VBVR-Pro The current official results table evaluates the Pro release. Original VBVR results must remain separate. Video-Reason (Maijunxian Wang, Ruisi Wang and others) Video world models Everyday Not in cards Seedream 5.0 Pro 0.56 Feb 2026 Jul 2025
VerifierBench Judge models decide whether proposed answers match references in math, science, and other subjects. OpenCompass and collaborators Other University Not in cards GPT-4.1 81.1% Aug 2025 Apr 2025
ViBench 2.0 Agents build 17 real web apps from empty projects one feature at a time from short specs. A grading agent uses each app in a browser and follows test plans. ViBench team (Replit and collaborators) Building whole projects Expert Not in cards GPT-6.1 Sol 64.7% May 2026 Sep 2026
VideoScore-Bench v2 Models score 500 generated videos on visual quality, agreement with the prompt, and physical consistency. The evaluation checks their ratings against human scores. TIGER Lab and collaborators Video understanding Everyday Not in cards Claude Sonnet 4 28.9% Sep 2025 Aug 2025
Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. Zelai Xu and colleagues Multi-agent games Everyday Not in cards o3 84.9% Jun 2025 Apr 2025
VoiceCodeBench Speech-to-text models transcribe 300 English workplace recordings. They are scored on whether every target entity, such as an address or email, survives exactly. besimple.ai (board run by Vals AI) Speech recognition and translation Everyday Not in cards GPT-Live-Transcribe 67.7% Aug 2026 Sep 2026
VoxMem Audio language models answer questions that need memory of earlier spoken sessions. Questions cover what was said, who said it, how it was spoken and what was audible. University of Melbourne Audio understanding and voice Everyday Not in cards Qwen3.8-Omni-Flash 38.5% Sep 2026 Sep 2026
WARC-Bench Screenshot-only agents do short web subtasks, such as date pickers and scroll containers, on real sites replayed from Web Archive snapshots. A program checks the end state. Uniphore Web browsing Everyday Not in cards Claude Sonnet 4 64.8% Oct 2025 Aug 2025
Workspace-Bench-Lite Agents complete 100 workspace tasks. The leaderboard reports the percentage of 1,856 rubric checks passed, with separate runs for each model and harness. SJTU + ByteDance (OpenDataBox) Documents and spreadsheets University Not in cards Claude Opus 4.7 66.7% May 2026 Apr 2026
WorldCoder-Bench Agents write code that builds interactive 3D worlds. The evaluator checks whether the resulting world behaves as specified. Shuo Lu and colleagues Image generation and editing University Not in cards GPT-5.4 27.8% Jun 2026 Apr 2026
Your LLM doesn't Know BOUNDARY!. Models complete work tasks at different access levels. Tests check whether they solve the part they own and report a blocker when another owner must act. T-RexBytes General assistants University Not in cards GPT-6.1 Sol 0.91 Oct 2026 Sep 2026
ZIQI-Eval Models answer multiple-choice questions about music history, performance, and theory. A second test asks them to choose the continuation of a melody. Jiajia Li and collaborators Exam questions University Not in cards chatglm2-6b 26.1% Feb 2024 Sep 2023
ABC-Bench Coding agents take a real backend repository, edit the code, set up the environment and deploy the service in a container. The 224 tasks come from 127 open-source repositories in 8 languages and 19 frameworks. A task passes only when the running service passes external end-to-end API tests. OpenMOSS (Fudan University) Fixing real repositories Expert Not in cards Claude Sonnet 4.5 63.2% Jan 2026 Dec 2025
AD-Bench Agents answer advertising analytics requests with production-platform tools. A model judge checks final answers against refreshed reference answers and measures coverage of expert tool-call trajectories. Results report success over one or three attempts. Tencent Tasks from real jobs University Not in cards Gemini 3 Pro 83% Feb 2026 Dec 2025
AdaExploreBench Language model agents explore environments with hidden rules, infer the rules from evidence and adapt when the rules change without notice. HKUST (KnowComp) Puzzles and chess Everyday Not in cards DeepSeek V4.1 Flash 70.3% Oct 2026 Sep 2026
AfroBench Models perform language understanding, reasoning, and generation tasks across 64 African languages. The full suite draws on 22 datasets. Jessica Ojo and colleagues Knowledge in many languages School Not in cards GPT-4o 59.6 Nov 2023 Sep 2024
Agent-Diff Agents use code to complete tasks across replicas of Slack, Linear, Box, and Calendar. The evaluator compares the resulting application state with the expected changes. Hubert M. Pysklo, Artem Zhuravel, and Patrick D. Watson Apps and MCP servers University Not in cards DeepSeek-V3.2 88.1% Feb 2026 Dec 2025
Agent-SafetyBench LLM agents act in 349 tool environments over 2,000 test cases covering 8 safety risk categories and 10 failure modes. A scorer model rates each run safe or unsafe. Tsinghua University (CoAI) Refusing harmful requests Everyday Not in cards Claude 3 Opus 59.8% Dec 2024 Oct 2024
AGI-Eval Community Text to Video Models generate videos from text. The board reports objective, subjective and combined scores. AGI-Eval (Meituan) Video generation Everyday Not in cards Wan 2.7 T2V 62.5 Feb 2025 Apr 2026
AI AgriBench Smallholders Models answer questions that smallholder farmers sent to advisory services. A judge scores the responses against answers from the advisors. AI AgriBench Consortium, University of Illinois Urbana-Champaign Tasks from real jobs University Not in cards GPT-5 mini 80.9 Nov 2025
AI Consumer Index Models with web search handle everyday consumer requests in shopping, food, gaming and DIY. Prompt-specific rubrics with hurdle and grounding criteria grade each answer. Mercor General assistants Everyday Not in cards GPT-5 56.1% Dec 2025 Nov 2025
AIDABench Agents analyze documents and spreadsheets to answer questions, draw charts, and generate files across 603 tasks. They can execute Python in a sandbox. Separate model evaluators check answer correctness, chart correctness and readability, and spreadsheet structure and content. SenseTime Research and collaborators Documents and spreadsheets Expert Not in cards Claude Sonnet 4.5 59.4% Feb 2026 Jan 2026
AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages. ByteDance Seed and collaborators Scientific code Expert Not in cards Claude Opus 4.5 44% Dec 2025 Dec 2025
ALBA Models answer 800 European Portuguese prompts across eight areas, including wordplay, grammar and sound. A language model grades the answers. NOVA University of Lisbon Knowledge in many languages School Not in cards Gemma 3 12B 51.1 Feb 2026 Sep 2025
Apodex Discovery Solver systems of model, harness and tools run long, verifiable investigations on high-value problems scouted from 561 industries in 16 sectors. Apodex Research science Research frontier Not in cards GPT-6 Astra 2.80 Aug 2026 Sep 2026
Audio Arena Speech-to-speech models hold one long spoken conversation per scenario and call tools when needed, such as booking a dental appointment or ordering groceries. Six scenarios add up to 221 spoken turns with corrections, false memory traps, prompt injection and state tracking. An LLM judge grades each turn against a golden answer. Arcada Labs (DesignArena) Speech recognition and translation Everyday Not in cards Grok Voice 3.0 85.7 Feb 2026 Oct 2026
Autoform-Hard Agents complete Lean 4 proofs on 56 formalization tasks that earlier models failed. Vals AI Formal proofs Expert Not in cards GPT-6.2 Astra 87.5% Nov 2026 Nov 2026
Balatro Bench (Michael Andrzejewski) Agents play one fixed Balatro seed with the Red Deck on White Stake. The maker ranks runs by ante reached, breaking ties by best hand. Michael Andrzejewski Video games Everyday Not in cards GPT-6 Astra 18.0 Aug 2026 Nov 2026
BHL OCR Eval Models transcribe scanned pages from historical natural history books. finebooks Documents, charts and text in images School Not in cards Unlimited OCR 2 3.29% Aug 2026 Oct 2026
BIG-bench Abstract Narrative Understanding Models select the English proverb that best fits a short story. Three versions offer five, ten or 100 choices. University of North Carolina at Chapel Hill Language reasoning Everyday Not in cards Qwen3-14B 82.5% Jun 2021 Aug 2025
BiomniBench Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer. Phylo, Humanlaya Data Lab and xbench (HongShan) Genomics and life sciences Expert Not in cards Claude Opus 4.7 63.9 May 2026 Apr 2026
BIRD-Python Models write pandas code to answer questions about tables. A language model compares the executed output with a reference result. Zhejiang University of Technology and University of Aberdeen Data science and SQL University Not in cards Qwen3-Max 73.2% Jan 2026 Sep 2025
BRoverbs Models select the Brazilian proverb that fits a story or the story that illustrates a proverb. Both directions offer five choices. University of Campinas and Maritaca AI Reasoning in many languages Everyday Not in cards Claude 3.5 Haiku 95% Sep 2025 Oct 2024
CaLM Models answer causal discovery, intervention, and counterfactual reasoning tasks in English and Chinese. Evaluation compares answers with reference labels to measure accuracy and robustness across causal targets and prompting methods. Shanghai AI Laboratory, Tongji University, Shanghai Jiao Tong University and others Language reasoning University Not in cards Claude 2 45.3% May 2024 Sep 2023
Cekura Turn Bench Mean end-of-turn score across six line conditions, as published by Cekura. Cekura Audio understanding and voice Everyday Not in cards GPT-Realtime-2.1 84.7% Jul 2026 Oct 2026
CFLUE Models answer Chinese financial questions in two parts. The knowledge part has more than 38,000 multiple-choice questions from financial qualification exams, with explanations to test reasoning. The application part has more than 16,000 test items for classification, translation, relation extraction, reading comprehension and text generation. Alibaba Cloud (Qwen DianJin) and Soochow University Finance University Not in cards Qwen-14B 53.8% May 2024 Nov 2023
CFLUE Knowledge Models answer Chinese financial qualification questions. The score is answer accuracy across the knowledge test. Alibaba Cloud (Qwen DianJin) and Soochow University Finance University Not in cards Qwen-14B 53.8% May 2024 Nov 2023
ChemDIRT Models answer chemistry questions while the evaluator varies instructions and molecular representations. The authors measure accuracy and consistency across variants. Eric Inae and colleagues Science questions University Not in cards Claude Sonnet 5 48.7% Mar 2026 Jul 2026
ChinaBench Models answer politically sensitive questions about China. An independent judge labels each response compliant, refused or evasive. Adam Holter Deception and alignment Everyday Not in cards Gemini 3.5 Flash 84.5% Feb 2026 Jun 2026
CIBench Agents use code interpreters to analyze data. The evaluation checks stages of the interactive analysis. OpenCompass and collaborators Data science and SQL University Not in cards GPT-4 Turbo 75.5% Jul 2024 Mar 2025
Company Agent Bench 205 questions on how well an agent understands its company, built from a month of real Slack history: who wrote an anonymized message, whom a teammate tagged, and who sent each reply. Agents search a replayed sandbox workspace using only messages from before the answer. Concept Simulated companies Everyday Not in cards Claude Fable 5.2 82.2% Nov 2026 Nov 2026
Composite-Bench Agents solve generated browser tasks in planning, rule extraction, deduplication, routing, and information gathering. Success requires matching a certified optimal solution. A separate number-partitioning task measures solution quality rather than contributing to the success rate. Composite Web browsing Expert Not in cards Claude Fable 5 74% Jul 2026 Jul 2026
CrossWordBench Models solve generated crossword grids from text or images. The tasks vary the grid size and the share of letters already filled in. HINT-lab and collaborators Puzzles and chess School Not in cards DeepSeek-R1 70.7% Mar 2025 Mar 2025
CSTutorBench Models tutor students who use block-based programming to control a virtual robot. A rubric grades the feedback on 17 scenarios, including debugging and answer disclosure. University of Illinois Urbana-Champaign Other professions University Not in cards Gemma 4 31B IT 0.89 Jul 2026 Apr 2026
CulturaVivaPT Models answer multiple-choice questions about Portuguese culture, including holidays, literature, food, and sports. AMALIA project Knowledge in many languages Everyday Not in cards Gemma 3 12B 72% Apr 2026 Sep 2025
DARE-Bench Agents write code to produce predictions from structured files. The benchmark checks predictions and adherence to modeling instructions. Snowflake and collaborators Data science and SQL University Not in cards Claude 3.7 Sonnet 49.3% Feb 2026 Aug 2025
DeepRSI 15 open-ended ML tasks with a 48-hour budget each. Private verifiers score gains over baselines. Models ranked by Bradley-Terry Elo. Hillclimb AI research tasks Research frontier Not in cards Claude Opus 5.5 1,629 Oct 2026 Oct 2026
DiG-bench Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt. Princeton + MIT + collaborators (Battleday, Whittington, Griffiths, Tenenbaum and others) Puzzles and chess Everyday Not in cards Claude Opus 5 71.4% Aug 2026 Jul 2026
DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. Stanford University and Brown University Genomics and life sciences Expert Not in cards Claude Opus 5 40.0 Sep 2026 Jul 2026
EDINET-Bench Models read Japanese financial disclosures to detect fraud, forecast earnings and identify industries. Sakana AI Finance Expert Not in cards Claude 3.5 Sonnet (October 2024) 0.73 Jun 2025 Sep 2025
Ego-MC-Bench Models watch a cook follow spoken instructions and tell them when to correct a mistake. The evaluator checks intervention timing and recipe step completion. Qualcomm AI Research, York University, and the Vector Institute Video understanding Everyday Not in cards Gemini 2.5 Flash 0.23 Jun 2026 Feb 2026
EvoArena Agents work in environments whose knowledge, tools and task conditions change over time. The suite tests whether agent memory keeps up with the changes. MIT Learning from the context Expert Not in cards GPT-5.5 53.3% Jun 2026 Apr 2026
EvoArena Terminal-Bench-Evo Agents solve terminal tasks that change across versions. The score is the published accuracy across versioned tasks. MIT Learning from the context Expert Not in cards GPT-5.5 65.1% Jun 2026 Apr 2026
GameASG-Bench Agents build 47 browser games from written requirements. Static checks inspect each HTML file, and browser tests check controls, gameplay and state changes. AReaL project Building whole projects University Not in cards GPT-6 Astra 55.3% Sep 2026 Sep 2026
GameEngineBench Agents edit C++ files in existing Unreal Engine 5 projects. Runtime tests check the changes, and a model audits whether the behavior meets the specification. Nitrode and collaborators Building whole projects University Not in cards Claude Fable 5 55.5% Jul 2026 Jun 2026
GEBench Image generation models create interactive GUI states from prompts. A judge scores goal, logic, consistency, interface and quality across five task types. StepFun and collaborators General visual questions Everyday Not in cards GPT-Image-1.5 63.2 Feb 2026
GeoNatureAgent Benchmark Agents use a geospatial API to analyze environmental indicators in Spain and Portugal. The grader checks tool use and answers against task requirements. Gabriel Diaz-Ireland and colleagues Function calling University Not in cards Claude Sonnet 4 60.8% Apr 2026 Feb 2026
GreekBarBench Models answer Greek bar exam questions in writing. A model judge grades their use of case facts, statutory articles, and legal analysis. Odysseas S. Chlapanis and colleagues Law University Not in cards GPT-4.1 8.35 May 2025 Apr 2025
GUI Knowledge Bench Models answer 3,483 questions about interface states, actions, and procedures. They inspect screenshots and task sequences from 292 applications across six platforms. Beijing Institute for General Artificial Intelligence and university collaborators Finding things on screen Everyday Not in cards o3 78.9% Oct 2025 Aug 2025
HackBench Models act as monitors on long autoresearch agent traces and must point to the exact commands where the agent reward-hacked. Scored by localization F1 against audited gold indices. Meta (FAIR RAM group) Deception and alignment Expert Not in cards Claude Opus 5 64.9 Oct 2026 Oct 2026
HWE-bench CAE Agents run engineering simulations, extract numerical results, and report them. A verifier repeats the extraction from the solver output and checks engineering tolerances. svd-ai-lab CAD models Expert Not in cards GPT-5.6 Sol 0.90% Aug 2026 Jul 2026
HyDRA-Bench Agents interleave database queries and document retrieval to answer questions. A model judge checks the final answer. Snowflake and collaborators Data science and SQL University Not in cards Qwen3-8B 53.9% Aug 2026 Nov 2025
IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. Shanghai Jiao Tong University, Carnegie Mellon University, and collaborators Research science Research frontier Not in cards GPT-5.5 27.3% Jul 2026 Apr 2026
ImageJevBench v0.3.0 Models make decisions about images and report probabilities. This version draws 300 public and 1,200 sealed items from a new pool. Benchmark Heaven General visual questions Everyday Not in cards GPT-6 Luna 17.3 Oct 2026 Sep 2026
Ivo-micro1 ContractBench Models review contracts and return redlines across 350 attorney-written tasks in 35 scenarios and five agreement types. Tasks cover first review and replies to a counterparty. micro1 with Ivo Law Expert Not in cards Gemini 4 Argon 64% Oct 2026 Sep 2026
JudgmentBench Models produce legal work for 30 tasks. Practicing lawyers grade individual answers and compare pairs of answers. Stanford Law School, Snorkel AI, and Harvey Law Expert Not in cards Claude Opus 4.8 89% May 2026 Jul 2026
K-BrowseComp Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers. Carnegie Mellon University and collaborators Knowledge in many languages Expert Not in cards GPT-5.5 45.7% Jun 2026 Apr 2026
Keep the number you wrote Models convert outdoor preferences into forecast rules. Two prompts test whether they preserve the stated numbers, units, and direction of each comparison. Vinicius Pereira Following precise instructions School Not in cards Claude Haiku 4.5 1.00 Oct 2026 Sep 2026
Kinayat Models interpret Egyptian Arabic idioms, select appropriate contexts and identify connotations. The evaluation includes separate generation and multiple-choice tasks. Carnegie Mellon University and Mohamed bin Zayed University of Artificial Intelligence Reasoning in many languages Everyday Not in cards Claude 3.5 Sonnet 95.3% Oct 2025 Sep 2025
LegalBench.PT Models answer questions about Portuguese law derived from university exams. The benchmark includes case analysis, multiple selection, and true-or-false questions. Beatriz Canaverde and colleagues Law University Not in cards GPT-4o 85.4% Dec 2024 Mar 2025
LexGenius Models answer Chinese legal questions drawn from cases and exams. The authors compare results across areas including legal reasoning, ethics, and judicial practice. Wenjin Liu and colleagues Law University Not in cards Qwen2.5-7B 59.7% Nov 2025 Apr 2025
LexPilot Models interview a user about a Chinese civil case and predict the ruling on each claim. Paired consultations reveal the effect of missing facts. Harbin Institute of Technology and Tsinghua University Law Expert Not in cards Claude Opus 5 72.1% Oct 2026 Sep 2026
LibraryDesignBench An author agent designs a software library from an open spec. Three fresh agents then solve 242 problems with it, and the library scores by their pass rate and code simplicity. SprocketLab (University of Wisconsin-Madison) with Snorkel AI Building whole projects Expert Not in cards Claude Opus 5.5 48.9 Sep 2026 Sep 2026
LingOly Models infer linguistic rules from examples in unfamiliar languages. They use those rules to solve linguistics olympiad problems. Andrew M. Bean and colleagues Reasoning in many languages Competition Not in cards Claude 3 Opus 46.3% Jun 2024 May 2024
LiveMathBench Models solve recent competition math problems, sampled many times. The G-Pass@k metric rewards both peak accuracy and stable answers across samples. OpenCompass (Shanghai AI Laboratory) Competition problems with answers Competition Not in cards o3-mini 76.8% Dec 2024 Jan 2025
LiveResearchBench Deep research agents write cited reports for user-centric questions that need current information from many live web sources. Rubric-based judges score the reports. Salesforce AI Research Deep research Expert Not in cards GPT-5 73.7 Oct 2025 Aug 2025
LLM SoccerArena World Cup 2026 Models forecast World Cup match scores and outcome probabilities before kickoff. The organizers compare forecasts with match results under different prompts and web access rules. LMU Munich, University of Cologne, and Paderborn University Real-world events Everyday Not in cards GPT-5.5 168.0 Jun 2026 Jun 2026
MarineEval Models answer 2,000 questions about marine images and documents. The authors combine answer checks with model judgments. They report total accuracy and mean accuracy across seven tasks. Hong Kong University of Science and Technology General visual questions University Not in cards Grok 2 Vision (1212) 50.4% Nov 2025 Apr 2025
MasakhaPOS Models assign parts of speech to words in African languages. The authors compare tagging models across languages. Cheikh M. Bamba Dione and colleagues Reading in many languages School Not in cards GPT-4o 62.8% May 2023 Aug 2024
MastermindEval Models infer a hidden symbol sequence from previous guesses and feedback in Mastermind. The evaluation varies sequence length and the similarity of answer choices. Jonas Golde and colleagues Puzzles and chess School Not in cards Phi-4 53.7% Mar 2025 Jan 2025
MedAgentBench Agents use FHIR APIs in a virtual electronic health record to do 300 patient tasks in 10 categories written by physicians. Stanford University Medicine Expert Not in cards Claude 3.5 Sonnet (October 2024) 69.7% Jan 2025 Feb 2025
Medical Bench Agents read longitudinal dermatology charts and write clinical assessments. Expert-authored rubrics grade their findings and management decisions. Turing Medicine Expert Not in cards Grok 4.6 88% Sep 2026 Aug 2026
MentalBench-100k Models answer counseling questions. Human reviewers grade responses to 1,000 of the 10,000 prompts on seven attributes, each from one to five. Abeer Badawi and colleagues Medicine Expert Not in cards GPT-4o 5 Oct 2025 Apr 2025
MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. Alibaba Simulated companies Expert Not in cards Qwen3.7-Max 59.5 Jul 2026 Jul 2026
MISHAP-Bench Models answer open-ended questions about audio clips. A model judge uses human annotations to identify claims the recording does not support. Zhi Wen Soi and colleagues Audio understanding and voice Everyday Not in cards Kimi-Audio-7B-Instruct 32.8% Sep 2026 Aug 2026
MJ-Bench Models compare image pairs for text alignment, safety, visual quality, and bias. Scores measure preference accuracy, with separate dispersion and equality measures for bias. MJ-Bench team at UNC-Chapel Hill, University of Chicago, Stanford University, and collaborators Image generation and editing Everyday Not in cards InternVL-Chat-V1-5 44.5% Jul 2024 Apr 2024
MLGym-Bench Agents work on 13 open-ended AI research tasks in vision, language, reinforcement learning and game theory. Scores compare the agent's result with baselines on each task's metric. Meta Machine learning engineering Expert Not in cards Gemini 2.5 Pro 1.45 Feb 2025 Jun 2025
MonitorBench Models work through 1,540 instances across 19 tasks in 7 categories, each built around a decision-critical factor such as a cue, a hidden goal or a required step. A monitor LLM reads the chain of thought, and the monitorability score measures how often it surfaces that factor. University of Illinois Urbana-Champaign (ASTRAL Group), University of Washington and UC San Diego Deception and alignment Expert Not in cards gpt-oss-120b 91.9% Mar 2026 Dec 2025
MusicTheoryBench Models answer college music theory questions about musical knowledge and reasoning. Some questions include a score written in ABC notation. Multimodal Art Projection and the ChatMusician team Exam questions University Not in cards Claude Sonnet 4.6 71.1% Dec 2023 Mar 2026
NewsBench (Forum AI) Models answer questions about current events. Judges calibrated against expert ratings score factual accuracy, neutrality, and source quality. Forum AI Long-form factuality Everyday Not in cards GPT-5.6 Terra 88.3% May 2026 Aug 2026
Nuclear Decision-Making Benchmark Models answer policy questions framed from different countries’ perspectives. The authors compare response distributions and agreement across repeated runs. Scale AI and policy researchers Bias, fairness and politics Expert Not in cards Gemini 3.1 Pro 81.3% Aug 2026 Feb 2026
OAB-Bench Models write answers for the second phase of the Brazilian bar examination. The port averages model judges’ scores against grading guidelines. Maritaca AI Law Expert Not in cards Gemini 3.1 Pro 9.39 Apr 2025 Mar 2026
OmniCAD Models choose parts from a component library and predict their assembly positions, orientations, and connections. The evaluator measures pose accuracy and how well connections match. Shanghai Jiao Tong University, Pennsylvania State University, and collaborators CAD models University Not in cards Llama 4 Maverick 100% Aug 2026 Jun 2026
OmniGUI Models use screenshots, sound, and video to predict the next action in 29 phone apps. The grader compares each action with a human demonstration. XPeng Motors Phones Everyday Not in cards Gemini 3.1 Pro 65.4% Mar 2026 Feb 2026
OmniMech Models write CAD programs to reconstruct parts from dimensioned drawings. The evaluator compares the reconstructed geometry with reference parts and measures relative volume error. Pennsylvania State University and collaborators CAD models University Not in cards GPT-5.5 15.9% Aug 2026 Jun 2026
OmniSch Models convert circuit schematics into graphs of components, pins, and connections. A second task lets models search schematic images for individual features. Pennsylvania State University and collaborators Electronics University Not in cards Qwen3.5-122B-A10B 0.28 Mar 2026 Feb 2026
OvertonBench Models answer questions about contested issues. Participants rate how well each answer represents their viewpoint. The score measures the fraction of opinion groups represented. Elinor Poole-Dayan and colleagues Bias, fairness and politics Everyday Not in cards DeepSeek-V3 0.42 Sep 2025 May 2025
PaintBench Image editing models do precise single-answer edits in 20 operations such as geometric transforms, color changes and symbolic edits. Procedural generation gives deterministic checks. VISIONx, New York University General visual questions Everyday Not in cards Gemini 3.1 Flash Image 17.1% May 2026 Jun 2026
Panda-CVL Models accept correct image-grounded responses or locate and replace their first erroneous text span. The test expands 652 conversations into 2,126 instances. Its overall F1 combines correct acceptance with exact correction of faulty responses. StepFun and Xiamen University General visual questions University Not in cards GPT-5.5 17.1% Sep 2026 Sep 2026
PaperScope Agents search multiple scientific papers to compare figures, summarize results, and solve problems. Each question can require evidence from several documents. Lei Xiong and colleagues Deep research Expert Not in cards Grok 4 41.0 Jan 2026 Nov 2025
PCBBench (pcbbench.org) Agents solve electronics problems and produce KiCad files. The public pilot checks artifacts across ten suites, including schematic design, board layout, and debugging. ProtoFlow Electronics Expert Not in cards Claude Opus 5.5 89.6% Oct 2026 Oct 2026
PE Civil Bench Models solve 110 multiple-choice and 40 numerical civil engineering exam problems. A model compares each answer with the reference solution. University of Houston Exam questions University Not in cards GPT-5.5 87.3% Mar 2026 Apr 2026
Periapical Lesions Bench Agents locate and grade lesions around tooth roots in panoramic dental X-rays. A perfect report matches every lesion and grade without adding extra boxes. Dylan Tirandaz Medicine Expert Not in cards Gemini 4 Neon 18.7% Sep 2026 Nov 2026
PReMind-Bench Models inspect a scene and decide whether to retrieve a personal memory before answering. Judges check whether the answer addresses the hazard and uses the right person’s information. Seoul National University Wellbeing and sycophancy Everyday Not in cards Claude Opus 5.5 87.3% Nov 2026 Sep 2026
PRISM-Bench (image generation) Models generate images from English or Chinese prompts. Vision-language judges score visual appeal and prompt alignment across seven tracks, including composition, lettering, and style. The Chinese University of Hong Kong, HKU, Beihang University, Alibaba, and SenseTime Image generation and editing Everyday Not in cards GPT-Image-1 86.3 Sep 2025 Sep 2025
Prompt-to-Bench HO-16 Models design 16 additional laboratory fixtures in OpenSCAD. The grader checks each rendered part against the dimensions and features in its prompt. S. S. Cocioba, Binomica Labs CAD models University Not in cards Claude Opus 5.5 16.0 Oct 2026 Sep 2026
pyATS-Bench Models generate pyATS scripts for wireless network test plans covering access point registration, WLAN status, and client association. Nikhil Shenoy Function-level problems University Not in cards GPT-6 Astra 1.00 Nov 2026 Sep 2026
R-ConstraintBench Models produce feasible operational plans as constraints accumulate. The score gives more weight to harder levels. Labelbox Running a business Expert Not in cards GPT-5 79.2% Aug 2025 Aug 2025
Reasoning-Bias-Reporting-Quality Models respond to everyday requests that contain flawed reasoning. A judge scores whether the reply challenges the assumption without an explicit request for critique. Gayathri Arvind Wellbeing and sycophancy Everyday Not in cards Claude Opus 5.5 6.94 Nov 2026 Sep 2026
Robots.txt Benchmark Models interpret robots.txt rules from synthetic and real websites. They also write rules from policies, which the grader checks against URL probes. Old Jard Function-level problems University Not in cards Claude Sonnet 5.5 1.00 Oct 2026 Sep 2026
RoTA-Bench Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette. MAC-AutoML (Xiamen University) Spatial reasoning and counting Everyday Not in cards GPT-5.6 Sol 17% Oct 2026 Sep 2026
RSI Bench Agents get a starting environment, a GPU budget and a verifier suite for an AI R&D task such as data curation or post-training. The score is mean reward per task, averaged across tasks. Scale AI AI research tasks Research frontier Not in cards GPT-6 Astra 38.9 Aug 2026 Sep 2026
RTL-Repo Models predict the next line of Verilog using code from the same project. The authors measure exact matches and edit similarity. Ahmed Allam and Mohamed Shalan Electronics University Not in cards Qwen3-Coder-480B-A35B 38.9% Mar 2024 Sep 2025
SAS-Bench Models grade short student answers from subject exams. The benchmark compares their scores and error classifications with expert annotations. Peichao Lai and colleagues Other professions University Not in cards Qwen3-32B 67.2% May 2025 Apr 2025
SEAL Adversarial Robustness Models receive 1,000 single-turn adversarial prompts written by 10 full-time red teamers across nine harm categories. Safety specialists grade responses, and models are ranked by the number of high-harm violations. Scale AI Jailbreaks and prompt injection Expert Not in cards Gemini 1.5 Pro 8.00 Jul 2024
Speko Speech-to-Speech Benchmark (20-turn concierge call) Voice agents try to book a flight, handle a failed booking, and remember a changed date. Eight automated checks score their actions and replies. Speko Customer service Everyday Not in cards gpt-realtime 1.00 Jul 2026 Jul 2026
SplatBench A model rebuilds an object in 3D from eight photos by placing up to 300 Gaussians itself through tool calls, with no optimizer. Half of the 30 tasks start from an empty scene, and half ask it to reassemble a scrambled fit. SplatScore compares renders from eight angles the model never sees, with an empty scene at 0 and gradient descent at 100. Perpetual Research Spatial reasoning and counting University Not in cards GPT-6 Astra 49.6 Oct 2026 Oct 2026
STAGE Cross-Scene Narrative Evolution Reasoning Models answer questions that require evidence from different scenes in a movie. A judge compares the answer with a reference. Southeast University, Beijing Zhongguancun Academy, Nanjing Normal University and ZhuiWen Technology Reasoning over long documents School Not in cards Claude Sonnet 4.6 65.1% May 2026 Apr 2026
SuperCLUE-Code3 Models write executable Python functions from Chinese specifications. Unit tests check functionality, and the leaderboard reports results by difficulty. SuperCLUE team Function-level problems School Not in cards GPT-4 Turbo (Jan 2024) 68% Feb 2024 Jan 2024
SuperCLUE-DeepSearch Agents search for information to answer Chinese research questions. SuperCLUE Questions that need search University Not in cards ChatGPT agent 74.3 Nov 2025
SuperCLUE-Faith Models produce answers that remain faithful to a supplied source. SuperCLUE Faithful to sources School Not in cards GPT-5.2 93.2 Dec 2025
SuperCLUE-Mkt Models complete tasks in advertising and marketing. SuperCLUE Running a business University Not in cards DeepSeek-V3.2 85.7 Dec 2025
SuperCLUE-OpenClaw Models complete coding, memory, and writing tasks in the OpenClaw environment. SuperCLUE General assistants University Not in cards Claude Opus 4.6 92.3 Mar 2026
SuperCLUE-SWE with mini-swe-agent Models solve Chinese software engineering tasks with the mini-swe-agent harness. SuperCLUE Fixing real repositories Expert Not in cards DeepSeek V4.1 Flash 1017 60.0 Nov 2026
SWE-Marathon-Ext Agents build 12 software products from specifications. Automated tests and browser checks grade the applications. Mercor Building whole projects Expert Not in cards GPT-5.6 Sol 74% Aug 2026 Jul 2026
Synthetic Sheet Music Reasoning Benchmark Models answer questions about rhythm, chords, intervals, and scales. Each test question appears in text notation and as an image. Zhilin Wang, Zhe Yang, and collaborators Documents, charts and text in images School Not in cards Claude Opus 4.6 90.5% Sep 2025 Mar 2026
Table Judge Benchmark Vision language models judge whether rendered table images match candidate HTML. Controlled corruptions test false positives, true positives and injected-error detection. Reducto Documents, charts and text in images Expert Not in cards Claude Opus 4.8 94.9% Nov 2025 Jul 2026
TEM-VQA Models give short written answers about structures, image features and experimental design in transmission electron micrographs. National Cheng Kung University Lab and data work Expert Not in cards GPT-4o 40.4 Jun 2026 Sep 2025
Terminal-Bench Math Modeling Agents use a terminal to solve 18 mathematical modeling problems and submit numerical answers. Feasibility checks and numerical scoring compare answers with reference solutions. TB-MathModel authors Scientific code Competition Not in cards GPT-5.5 64.3 Nov 2026 Sep 2026
TerminalWorld Agents complete terminal workflows reconstructed from public recordings in isolated environments. Automated tests check the resulting files and system state. A manually reviewed 200-task subset supports the published model comparisons. EuniAI and the TerminalWorld authors Terminal tasks University Not in cards Claude Opus 4.7 62.5% May 2026 Apr 2026
TimeScope Models answer questions about short clips inserted into videos lasting up to eight hours. Accuracy measures retrieval, synthesis and temporal perception. Stanford University and Hugging Face Long videos Everyday Not in cards Gemini 2.5 Pro 97.3% Jul 2025 Jun 2025
VAREX-Bench Models extract structured fields from 1,777 government forms using a supplied JSON schema. Normalized exact match checks field values, while string similarity gives partial credit. Array entries use order-independent matching. IBM Research Documents, charts and text in images Everyday Not in cards Gemini 2.5 Flash 97.3% Mar 2026 Dec 2025
VDR-Bench Models answer 2,000 visual questions that require finding entities in images and researching related facts. A Qwen3-VL judge checks final answers against references. Entity recall separately measures whether searches retrieve the relevant entities. CUHK MMLab, Xiaohongshu, and university collaborators Deep research University Not in cards Claude 3.7 Sonnet 27.2% Feb 2026 Oct 2025
VL-RewardBench Vision-language reward models pick the better of two responses on general queries, hallucination detection and reasoning tasks. HKU and collaborators General visual questions Expert Not in cards Llama 3.2 90B 53.9% Nov 2024 Sep 2024
VLA-Arena VLAs fine-tuned on level-0 tabletop tasks, tested on 170 tasks in 11 suites at harder unseen levels L1/L2. Success rate. PKU-Alignment Simulated robots Everyday Not in cards MolmoAct 2 72.1% Sep 2025 May 2026
VoxelCodeBench Models write code to build and modify voxel scenes. The benchmark checks the constructed shapes and evaluates their appearance. Yan Zheng and colleagues Image generation and editing University Not in cards GPT-5 87.9% Apr 2026 Sep 2025
WebsiteBench Agents explore reference websites and rebuild their appearance, behavior, and persistent state. Hidden browser tests require functional correctness and screenshot similarity, then report the fraction passed. TIGER-AI-Lab Web and app front ends Expert Not in cards Claude Opus 5.5 26.2% Sep 2026
xbench-Recruitment Agents do 50 headhunting tasks from live business cases: company mapping, finding facts about people and finding people who fit a profile. A judge scores answers 1 to 5. xbench (HongShan) Running a business Expert Not in cards o3 78.5 Jun 2025 May 2025
A^3-Bench Models solve 2,198 science reasoning problems annotated with the anchors and attractors (prior knowledge) they need. An AAUI metric measures how much of that knowledge the model activates. Jian Zhang, Yu He, Jun Liu et al. Science questions University Not in cards Grok 4 Fast 65.1% Jan 2026 Dec 2025
AARRI-Bench Agents complete research intern tasks that test scientific judgment, professional conduct, and attention to detail. The Harbor collection contains 82 tasks. AARR-Bench team AI research tasks Expert Not in cards Claude Opus 4.7 68.3% Apr 2026 Apr 2026
ABACUS Models add, remove, or change a specified number of objects in an image. The evaluator checks the count and preservation of the surrounding scene. Aymen Lassoued and colleagues Image generation and editing School Not in cards Seedream 4.0 0.31 Oct 2026
Acme Agent Evals Agents use GitHub tools to answer questions about issues, pull requests, and code in a simulated software project. Arize AI Apps and MCP servers University Not in cards Claude Sonnet 4.6 85.1% May 2026 Apr 2026
AD4Edu-Bench Models write audio descriptions for university lectures. A rubric checks accessibility rules, and the authors compare the rankings with blind and low-vision listeners' preferences. University of Zurich and collaborators Video captioning University Not in cards SmolVLM2-2.2B-Instruct 0.86 Oct 2026 Oct 2025
ADeptS-Bench Desktop Agents choose safe interface actions on the Desktop test set. The published safety score combines task success and attack resistance. Chen et al. Finding things on screen Expert Not in cards Gemini 3.1 Pro 76% Aug 2026 Apr 2026
ADeptS-Bench Mobile Agents choose safe interface actions on the Mobile test set. The published safety score combines task success and attack resistance. Chen et al. Finding things on screen Expert Not in cards Gemini 3.1 Pro 73.8% Aug 2026 Apr 2026
Agentic Credit Analysis Models extract and standardize income statements and balance sheets. Ramp compares their figures with credit analysts’ figures using a five percent tolerance. Ramp Labs Finance Expert Not in cards Claude Sonnet 5.5 80.2% Oct 2026 Sep 2026
AI AgriBench Post-Cutoff Models answer 146 questions from newer agricultural extension documents. Judges score their answers for accuracy, relevance, completeness and conciseness. AI AgriBench Consortium, University of Illinois Urbana-Champaign Tasks from real jobs University Not in cards GPT-5 98.1 Jan 2026 Nov 2025
ANAH Models detect unsupported claims in answers to English and Chinese questions. The benchmark provides annotations for individual claims. OpenCompass and collaborators Hallucination rates University Not in cards Baichuan2-13B 42.2% May 2024 Sep 2023
ARE-bench Agents continue an existing research project from a real user request. A judge checks the deliverable against the request and earlier workspace requirements. Shanghai Artificial Intelligence Laboratory and collaborators AI research tasks Expert Not in cards GPT-6.1 Sol 66.2% Nov 2026 Sep 2026
Arena Search Factuality Arena combines user preferences with independent checks of verifiable claims in model responses. Arena Long-form factuality Everyday Not in cards Grok 4.20 1,218 Jul 2026 Jun 2026
AscendGYM Agents optimize existing kernels for Ascend processors using tensor shapes from real workloads. The benchmark measures speedup and token consumption. AscendGYM team GPU kernels Expert Not in cards Claude Opus 5 74.0 Nov 2026 Aug 2026
AsgardBench Agents adapt high-level action plans from images, action history and lightweight success signals in a controlled simulator. Microsoft Research Simulated robots Expert Not in cards Claude Opus 4.5 76.5% Mar 2026 Jan 2026
AstroVisBench Models write code to process astronomical observations and plot the results. The benchmark compares computed values with reference outputs and uses a model to grade plots. University of Texas at Austin and collaborators Lab and data work University Not in cards Claude Opus 4 49.8% May 2025 May 2025
Baba Is AI Models see a puzzle and write a plan to reach the winning object. Some puzzles require them to push words together to change the rules. MIT and University of Virginia Puzzles and chess Everyday Not in cards Llama 3.2 90B 43.9% Jul 2024 Mar 2025
BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. OpenAI and Ligo Biosciences Genomics and life sciences Expert Not in cards GPT-6 Astra 66.7% Nov 2026 Sep 2026
BrowserART Browser agents receive 100 harmful requests on synthetic and real websites. The test measures how often the agent attempts the harmful behavior that its backbone model would refuse in chat. Scale AI and collaborators Refusing harmful requests Expert Not in cards Claude 3 Opus 11% Oct 2024 Sep 2024
CascadeBench Agents predict the cascade of changes that one action sets off in an enterprise system, where business rules differ between deployments. The environments are synthetic and follow the World of Workflows method. Tests include shifts in the system's rules to check whether a model reads the active configuration or relies on what it learned in training. ServiceNow Interactive world models Expert Not in cards GPT-5 41.8% May 2026 Apr 2026
CASTLE Models read 250 hand-written compilable C programs and say whether each has a vulnerability and which CWE. Static analyzers run on the same set. CASTLE authors (Richard et al.) Finding and exploiting real bugs Expert Not in cards o3-mini 977.0 Mar 2025 Jan 2025
Cekura Agent Performance Scenario reliability across three repeated calls (Pass³). Cekura Audio understanding and voice Everyday Not in cards Grok Voice 3.0 81.7% Jul 2026 Oct 2026
Cekura Real Human Calls Real caller reliability across three repeated calls (Pass³). Cekura Audio understanding and voice Everyday Not in cards Grok Voice 3.0 42.9% Jul 2026 Oct 2026
ChemInsight Models solve related chemistry problems in bundles. The authors score how many parts a model can solve and how far its reasoning remains correct. Changyu Hu and colleagues Science questions University Not in cards Gemini 3 Pro 0.28 Nov 2026 Dec 2025
CLadder Models answer yes or no to questions about associations, interventions, and counterfactuals. A causal inference engine supplies the reference answers. CausalNLP and collaborators Language reasoning University Not in cards GPT-4 Turbo 70.4% May 2023 Nov 2023
Closed-Loop Evaluation of LLM Agents for Embedded Software Development Agents write firmware from engineering briefs and test it on simulated equipment. Hidden checks score device behavior across five control tasks. Jorge García-Carrasco and colleagues Electronics University Not in cards GPT-5.4 100% Mar 2026
CocoaBench Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output. UC San Diego Web browsing Expert Not in cards GPT-5.4 45.1% Apr 2026 Mar 2026
CodeCap-Bench Models watch a structured 2D video and write code that re-renders it. The rendered video is compared with the original on visual similarity, object timing and placement, and retained detail. Kling AI (Kuaishou) + Nanjing University Video captioning Everyday Not in cards GPT-6 Astra 73.2 Oct 2026 Sep 2026
ContextASR-Bench Models transcribe speech with optional domain labels and named entities. The benchmark measures transcription errors and recognition of named entities. He Wang and colleagues Speech recognition and translation Everyday Not in cards Kimi-Audio-7B-Instruct 3.10% Jul 2025 Apr 2025
CrochetBench Models identify stitches, select instructions that match a photo, write crochet patterns, and translate patterns into programs for a crochet simulator. University of Notre Dame General visual questions University Not in cards Claude Sonnet 4 60.9 Nov 2025 Sep 2025
CRUST-bench Agents translate 100 C projects into safe Rust. Tests check whether each translation matches the supplied Rust interface and preserves the original behavior. Anirudh Khatry and colleagues Fixing real repositories Expert Not in cards o3 48% Apr 2025 May 2025
CUSP Models forecast scientific progress across 4,760 events in many fields, using only knowledge from before a set cutoff date. Tasks ask whether an advance is feasible, how it would work, how to design a solution and when it will happen. The benchmark also compares events before and after each model's training cutoff. University of Oxford and coauthors Real-world events Expert Not in cards GPT-5.6 Sol 60.4% May 2026 Jul 2026
DailyReport Search agents answer practical daily information requests. Evaluators assess the resulting reports across dimensions of user satisfaction. AGI-Eval (Meituan) Deep research Everyday Not in cards GPT-5.4 2.89 Jun 2026 Mar 2026
DPAI Arena Spring Benchmark Coding agents implement changes in Java Spring repositories. DPAI Arena scores each run and publishes its duration. JetBrains and Toloka Fixing real repositories Expert Not in cards Claude Opus 4.7 64.9 Oct 2025 Apr 2026
DriveBench Vision-language models answer 20,498 questions about 19,200 driving frames, covering perception, prediction, planning and behavior. Inputs come in 17 settings: clean images, corrupted images and text only. The test checks whether answers rest on the image or on general knowledge. UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU Embodied reasoning Everyday Not in cards Qwen2-VL-72B-Instruct 48.0 Jan 2025 Aug 2024
EgoBlind Models answer questions from blind users using only video recorded before each question. GPT-4o mini checks each answer against human references. National University of Singapore and collaborators Video understanding Everyday Not in cards GPT-4o 59.3% Mar 2025 Sep 2025
ELBench Models respond to teaching and safety scenarios alongside academic questions. The authors compare the same models across four educational assessment areas. Yilin Jiang and colleagues Other professions University Not in cards DeepSeek-V4-Flash 83.5 Jun 2026 May 2026
EmailBench Agents handle email and productivity requests in a synthetic inbox. The evaluator combines executable checks and model-graded rubrics across 206 scenarios. Microsoft Documents and spreadsheets University Not in cards Claude Sonnet 4.5 33.5% Sep 2026 Apr 2026
Embodied Over-Refusal Benchmark Models decide whether to perform household tasks. The score balances compliance on safe requests with refusal on hazardous ones. sky Refusing harmful requests Everyday Not in cards GPT-5.6 Sol 1.00 Nov 2026 Oct 2026
EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. Hongcheng Gao and colleagues Simulated robots University Not in cards Claude Opus 5.5 41.4% Oct 2026 Sep 2026
EngiWorld Agents work through the full industrial engineering design loop on 1,301 expert-curated tasks that need reasoning over geometric and physical constraints across software. Zhiman AI CAD models Expert Not in cards Claude Opus 5 44.7% Sep 2026 Sep 2026
EnvSimBench Language models act as simulators of tool environments for agent training. Given a trajectory of agent actions, the model predicts the next environment state and feedback. Labels come from real executors, so the score checks whether simulated states match what the real environment would return. EnvSimBench authors Apps and MCP servers University Not in cards Qwen3.5 397B-A17B 42.3% May 2026 Mar 2026
EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. Carnegie Mellon University + Stanford + collaborators Research science Research frontier Not in cards Claude Fable 5.1 29.8 Sep 2026 Sep 2026
Evaluating AI Safety in Teen Conversations Chatbots respond to simulated teenagers across ten replies. Judges measure the share of conversations that fail a critical safety check. Vals AI Wellbeing and sycophancy University Not in cards Muse Spark 1.2 9.70% Sep 2026 Aug 2026
EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. OpenAI and Paradigm Finding and exploiting real bugs Expert Not in cards GPT-5.3 Codex 51% Feb 2026 Feb 2026
ExpertCrowd-SWE Agents resolve 100 software issues across 87 repositories in 11 languages. Each engineer-written rubric combines behavioral tests with a model judge and awards partial credit. The score averages five valid runs per task and then averages tasks. Toptal ExpertCrowd Fixing real repositories Expert Not in cards Claude Opus 5.5 67.3% Oct 2026 Sep 2026
ExpertEval Models answer cases in medicine, finance and law. Practicing professionals write the cases and supply weighted grading rubrics. UniPat AI Other professions Expert Not in cards Claude Opus 4.6 66.9% May 2026 Mar 2026
ExtractBench Pro Systems extract every schema value from long-tail synthetic documents with charts, drawn relations, house rules and jargon. Scored by value F1 and by grounded F1, which also needs the source location. LlamaIndex Documents, charts and text in images Expert Not in cards GPT-6 Astra 58% Oct 2026 Oct 2026
Far-Field ASR Leaderboard Models transcribe speech under simulated room acoustics. The board compares word error rates. Treble Technologies and Hugging Face Speech recognition and translation Everyday Not in cards Qwen3-ASR-1.7B 13.4% Jun 2026 Oct 2026
FinDeepIndicator Agents construct financial indicators by specifying formulas, retrieving market data, and calculating results. The evaluation checks performance at each step. Chaoqun Yang and colleagues Finance University Not in cards Qwen 3.6 Max (Preview) 39% Aug 2026 Apr 2026
FinEval Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings. Shanghai University of Finance and Economics (AIFLM Lab) Finance University Not in cards Qwen2.5-72B 69.4% Aug 2023 Sep 2024
FinMaster Models complete financial workflows using simulated company records. The benchmark includes single calculations and tasks that require several financial metrics. Junzhe Jiang and colleagues Finance University Not in cards o3-mini 73% May 2025 Apr 2025
FinToolBench Agents answer 295 finance questions by selecting and executing tools from a library of 760 financial interfaces. The evaluation separates tool execution, answer correctness, and mismatches in timeliness, intent, and regulatory domain. Answer checks combine model judging with exact matching. Jiaxuan Lu, Kong Wang, and collaborators Function calling Expert Not in cards Gemini 3.1 Pro 0.82 Mar 2026 Mar 2026
Frontier Agentic ML & DS Agents solve machine learning and data science tasks in a terminal. Verifiers check their deliverables and predictive quality. Turing Machine learning engineering Expert Not in cards Claude Opus 5 64.6% Aug 2026 Jul 2026
FrontierCAD Assemblies Agents place supplied CAD parts into 16 assemblies. The score is the published mean assembly geometry score. Normal Research CAD models Expert Not in cards GPT-6 Astra 0.71 Oct 2026 Sep 2026
FrontierCAD Parts Agents build 39 CAD parts from engineering drawings. The score is the published mean geometry score. Normal Research CAD models Expert Not in cards GPT-6 Astra 0.63 Oct 2026 Sep 2026
FrontierMemory A model reads 750 past agent trajectories and writes memory for a task agent. The score is how many new coding and knowledge-work tasks that agent then solves. Letta AI research tasks Research frontier Not in cards Claude Opus 5.5 73.3% Oct 2026
FutureX Agents predict upcoming events collected daily from 195 websites in 11 domains, within a one-week window. Answers are scored once real results are crawled, with harder tiers weighted more. ByteDance Seed with Fudan, Stanford and Princeton Real-world events Expert Not in cards Grok 4 27.9 Aug 2025 Jul 2025
Gaia2-CLI Agents complete realistic tasks through command-line tools. The benchmark measures task success across search, execution, adaptability, ambiguity, and time constraints. Meta FAIR General assistants Everyday Not in cards Claude Opus 4.6 57% Apr 2026 Apr 2026
Garble-Bench (Voice Arena) Models respond to dialogue with speech recognition errors. The evaluation measures error detection, transcription fidelity, conversation recovery and task completion. Voice Arena and collaborators Audio understanding and voice Everyday Not in cards Gemini 3.8 Flash 0.98 Sep 2026 Sep 2026
Groundtruth Dynamic Benchmarking (Geology Edition) Models answer questions from geological reports. A judge scores each answer from zero to ten against a rubric tied to passages in the source documents. Eigenform Reading comprehension Expert Not in cards Kimi K3 8.81 Jul 2026 Aug 2026
HarnessAudit-Bench Agents run inside execution harnesses such as Claude Code, Codex and OpenClaw on 210 tasks across eight domains, from finance to software engineering, with safety rules built in. The audit reads the full trajectory through channels the agent cannot change. It scores whether the agent stays inside its permission and information-flow policies, completes checkpoints, and keeps both under prompt injection, vague goals and tool errors. UC Santa Barbara, with UC Berkeley, UW-Madison, Stanford and Microsoft Research Refusing harmful requests Expert Not in cards Gemini 3.1 Pro 0.41 May 2026 Apr 2026
HCSU-Bench Models match a Chinese calligraphy sample to one of eight styles. A second task asks models to describe the brushwork in fewer than 50 words. Tongji University General visual questions University Not in cards GPT-5.2 34.6% Mar 2026 Dec 2025
HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. Meta, Nanyang Technological University, University of Washington, Brown University, and Northwestern University Simulated robots Everyday Not in cards Gemma 4 31B IT 11.1% Jul 2026 Sep 2026
Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. Federico Ravenda and colleagues Medicine Expert Not in cards GPT-6 Luna 83.9% Nov 2026 Sep 2026
Hype Trap Models assess anonymized news about small stocks. The grader compares their predictions with stock movements over the next five trading days. SaltySteve Real-world events University Not in cards Claude Haiku 4.5 0.01 Oct 2026 Sep 2026
Invented O'Clock Models calculate dates and times in 45 scheduling problems. Paired versions remove an essential fact and require the model to answer unknown. Maddie Brooks Faithful to sources School Not in cards Gemini 3.1 Pro 97.8% Oct 2026 Apr 2026
IPI Bench Agents complete workplace workflows while attackers add misleading records to an emulated company with 11 services. The benchmark pairs six workflow types with harmful injected objectives across 34 tasks. The headline here measures single-query attack success as a percentage, where lower is better. General Analysis Jailbreaks and prompt injection Expert Not in cards Claude Opus 5.5 4.80% Oct 2026 Oct 2026
IslamicFaithQA Models answer short Islamic knowledge questions in Arabic and English. A model judges their answers against reference answers. Qatar Computing Research Institute and collaborators Expert-written questions University Not in cards gpt-oss-20b 21.6% Jan 2026 Aug 2025
LLM Round-Trip Translation Benchmark Models translate text into another language and back. The evaluation measures how well the original meaning survives. Lech Mazur Translation University Not in cards GPT-5 8.69 Sep 2025
LLM SoccerArena Bundesliga 2026/27 Models forecast Bundesliga match scores and outcome probabilities with web access. The standings award points for score predictions made about 24 hours before kickoff. LMU Munich, University of Cologne, and Paderborn University Real-world events Everyday Not in cards DeepSeek-V4-Pro 57.0 May 2026
LLM SoccerArena Premier League 2026/27 Models forecast Premier League match scores and outcome probabilities with web access. The standings use the predictions made about 24 hours before kickoff. LMU Munich, University of Cologne, and Paderborn University Real-world events Everyday Not in cards GPT-5.5 89.0 May 2026
Long-Horizon Medical Agent Benchmark, Veterinary Clinic Ops v2 Agents operate a simulated veterinary clinic through its database API. Each of the 98 tasks checks whether the agent leaves the clinic records in the required state. Praesidium Compliance Systems Corporation Tasks from real jobs Expert Not in cards GPT-5.6 Sol 100% Sep 2026 Sep 2026
MANDOS Session 2 Models make process safety decisions across four private industrial scenarios. A model judge assesses each run. The leaderboard reports the share of passing runs. RelayForge Tasks from real jobs Expert Not in cards Claude Sonnet 4.6 96.9% Sep 2026 Apr 2026
MarineEVT Models answer 2,000 held-out questions about underwater events, including species interactions, movement, and timing. The authors report accuracy across five reasoning categories. Hong Kong University of Science and Technology and collaborators Video understanding University Not in cards Qwen3-VL-8B-Instruct 43.7% Jun 2026 Dec 2025
McStasBench Agents design neutron scattering instruments. A simulator checks whether their designs reproduce published measurements. The authors score two additional instruments separately. Oak Ridge National Laboratory Lab and data work Expert Not in cards Claude Sonnet 5 43.8% Sep 2026 Jul 2026
MDMR-Bench Models generate one image from several references. A model judge checks whether the output preserves requested elements, follows instructions, and combines them correctly. Bowen Zheng and colleagues Image generation and editing Everyday Not in cards GPT-Image-2 88.8% May 2026 Feb 2026
MM-AlignBench Models answer open-ended questions about 252 human-annotated images. GPT-4o compares their answers with Claude-Sonnet-3 reference answers to assess alignment with human preferences. Shanghai Jiao Tong University, Shanghai AI Laboratory, Nanjing University and others General visual questions Everyday Not in cards GPT-4V 82.5% Feb 2025 Nov 2024
NARRA-Gym Agents build interactive stories around a user’s experiences and adapt scenes as the conversation unfolds. Judges rate story quality and user experience separately. Yue Huang and colleagues Creative writing Everyday Not in cards Claude Sonnet 4.6 2 May 2026 Mar 2026
NaturalBench Models answer 10,000 human-checked questions on natural images. Each question pairs with two images that need different answers, so a model cannot pass without looking. Carnegie Mellon University and others (Baiqi Li, Zhiqiu Lin et al.) General visual questions Everyday Not in cards o3 55.1% Oct 2024 Apr 2025
NoAdivines: Safe Order-Taking in Informal Spanish Models convert informal Latin American Spanish orders into structured actions. The grader checks whether they infer unsupported details or request clarification. Henry Jeovany Rodezno Chat and instructions in many languages Everyday Not in cards gpt-oss-20b 100.0 Oct 2026 Sep 2026
Numerical results extraction from randomized controlled trials Models extract group sizes, event counts, means, and standard deviations from trial reports. The test contains 656 treatment comparisons across 110 papers. Northeastern University and King's College London Medicine University Not in cards DeepSeek V4.1 Flash 77.9% May 2024 Sep 2026
NYU CTF Bench Agents solve 200 capture the flag challenges from seven years of CSAW competitions. Six categories cover crypto, forensics, pwn, reverse, web and misc. Each runs in Docker. New York University (NYU Tandon) Capture-the-flag challenges University Not in cards Claude 3.5 Sonnet 22% Jun 2024 Apr 2025
OASIS (agent safety) Agents handle multi-turn requests with different levels of complexity. Evaluators measure the proportion of safe responses. OpenCompass and collaborators Deception and alignment University Not in cards GPT-5 mini 95.2% Nov 2025 Sep 2025
OdysseyBench Agents recover requirements from conversation histories and complete workflows across documents, spreadsheets, PDFs, email, and calendars. Task-specific checks inspect the resulting file system and report the percentage of completed tasks. Microsoft and University of Edinburgh Documents and spreadsheets University Not in cards o3 58.7% Aug 2025 Aug 2025
OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. Peking University, Alibaba Group, and Tsinghua University Embodied reasoning Everyday Not in cards Qwen3.8-Omni-Flash 43.9% Sep 2026 Sep 2026
OpenComputer Computer-use agents complete 1,000 desktop tasks across 33 real applications, including browsers, office tools, creative software, development environments and messaging apps. App-specific verifiers inspect the final application state, and the harness gives partial credit from the recorded trajectory. Yale NLP Lab Desktop apps Everyday Not in cards GPT-5.4 68.3% May 2026 Apr 2026
OpenFrontiers-Bench Agents try to rediscover proofs of research math problems, most resolved during 2026, stated in Lean 4. Two independent checkers grade the formal proofs, with closed-book, frozen-retrieval and live-web settings. University of Illinois Urbana-Champaign (Tong Zhang) Formal proofs Research frontier Not in cards GPT-6.1 Sol 42.2% Oct 2026 Sep 2026
PARCEL Models decide whether a New York court opinion supports, contradicts, or fails to address a cited claim. The benchmark contains 3,396 labeled claims. University at Albany, SUNY Law University Not in cards Gemini 3 Flash 97% Oct 2026 Dec 2025
PhysClosureBench Agents run 50 expert-verified physics research tasks in strong-field and attosecond physics, deciding how to generate and validate evidence. A 20-criterion score and a closure level from RC0 to RC4 grade them. Yijia Mao et al. Research science Research frontier Not in cards Claude Opus 5 6.29 Oct 2026 Sep 2026
Poker Arena Models play no-limit Texas Hold’em across repeated sessions. The benchmark scores tournament chips and nine strategic dimensions, including memory and bluffing. Pratham Singla, Shivank Garg, and Vihan Singh Multi-agent games Everyday Not in cards Claude Opus 4.6 15730.0 Jun 2026 Mar 2026
PPTGen-Bench Agents create slide decks from text requests. Tencent renders the decks to PDF with LibreOffice. A judging agent assesses appearance and applies penalties for serious faults. Tencent Documents and spreadsheets School Not in cards Claude Opus 5.5 67.7 Nov 2026 Nov 2026
ReactionBench Models rank a viewer’s reactions, match a reaction to one of four video clips, and explain which cue prompted it. MIT, University of Illinois Urbana-Champaign, and collaborators Video understanding Everyday Not in cards GPT-5 0.51 Oct 2026 Aug 2026
Read-aloud Assessment Benchmark Speech models transcribe children reading aloud in Kiswahili. The evaluation measures transcription errors and disagreements with human assessors about which items the child misread. Fab AI Speech recognition and translation Everyday Not in cards Gemini 3.6 Flash 54.5% Aug 2026 Sep 2026
ReplicationBench Agents reproduce computational results from published astrophysics papers. Evaluators check their numerical answers and assess whether their methods follow the original work. Christine Ye and colleagues Research science Research frontier Not in cards Claude Sonnet 4.5 22% Oct 2025 Sep 2025
RetailBench Agents operate a simulated supermarket for up to 180 days. They choose suppliers, order stock, and set prices as goods age and customers respond. Ant Group and City University of Hong Kong Running a business University Not in cards GPT-5.5 $24,351 Mar 2026 Apr 2026
RExBench Agents extend 12 published AI research projects from expert instructions. The evaluator runs their code against task-specific success criteria. Harbor distributes two public tasks. Nicholas Edwards and colleagues AI research tasks Research frontier Not in cards Claude Opus 4.5 50% Jun 2025 Nov 2025
RiOSWorld Computer-use agents work in an Ubuntu virtual machine through risky scenarios such as phishing emails and pop-ups. Scores track whether agents notice and avoid the risk. RiOSWorld authors (Yang et al.) Refusing harmful requests Everyday Not in cards Llama 3.2 90B 79% May 2025 Jun 2025
RoughCut-Bench Models select and order video clips or mark speech to remove from a transcript. The evaluation compares cut plans with professional editors’ annotations. Zhejiang University, ByteDance, and Peking University Video understanding University Not in cards Gemini 3 Pro 0.53 Jul 2026 Jun 2026
RuC-tt07-32k Models fill missing Verilog statements in Tiny Tapeout designs. A parser checks syntax, and a formal checker tests whether the completed design preserves behavior. Barcelona Supercomputing Center Electronics University Not in cards Qwen3-Coder-480B-A35B 76.7% Apr 2026 Sep 2025
SAVU-Bench Models answer spatial questions about real scenes using sound and video. The authors report accuracy across seven tasks. Yu Chen and colleagues Audio understanding and voice Everyday Not in cards Gemini 3.6 Flash 55.3% Oct 2026 Jul 2026
scBench (original study) Agents complete 394 tasks in single-cell data analysis. The authors compare their answers with results from the original analyses. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 4.6 52.8% Feb 2026 Feb 2026
SciTaste-Bench Models judge the research value of papers in six disciplines. The benchmark compares their judgments with a reference index built from seven dimensions. University of Chinese Academy of Sciences and collaborators Research science Expert Not in cards Qwen 3.8 Max 28.0 Oct 2026 Aug 2026
SCONE-bench Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail. Anthropic (with MATS and Anthropic Fellows) Finding and exploiting real bugs Expert Not in cards Claude Opus 4.5 $548,400,000 Dec 2025 Nov 2025
SCUBA Agents work in a Salesforce sandbox as administrators, sales representatives, or service agents. Evaluators check milestones such as creating accounts and routing support cases. Salesforce AI Research Web browsing University Not in cards GPT-5 53.9% Sep 2025 Aug 2025
SemaPLC Function Track Agents generate individual Structured Text program units. A compiler and a formal verifier check the program against the requirements. Inconclusive checks count as failures. Midea AIRC and collaborators Electronics Expert Not in cards GPT-5.5 82.1% Aug 2026 Jun 2026
SemaPLC Project-Context Track Agents implement plant control logic inside existing Structured Text projects. The evaluator measures compilation, static checks, and agreement with reference outputs during execution. Midea AIRC and collaborators Electronics Expert Not in cards GPT-5.5 100% Aug 2026 Jun 2026
SF20K-Test Models answer questions about the story in a short film. The authors check multiple-choice answers against a key and grade free responses with GPT-3.5. École Polytechnique and MBZUAI Long videos Everyday Not in cards LongVA-7B 62.8% Jun 2024 Sep 2024
SimReadyBench NV100 Agents reconstruct an articulated 3D asset from one image and nominal dimensions. The verifier grades formal, visual, and kinematic fidelity separately across 100 assets. HCLTech Robotics CAD models Expert Not in cards GPT-6.1 Sol 100% Oct 2026 Nov 2026
SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. CMU Genomics and life sciences Expert Not in cards GPT-5.4 40.2% May 2026 Mar 2026
SuperCLUE-AdsVU Models interpret video advertisements. SuperCLUE Video understanding Everyday Not in cards Seed2.1 Pro 89.1 Jun 2026
SuperCLUE-Agent Chinese language models are tested on agent skills in Chinese tasks. These include tool use with several API calls, task planning and memory over long dialogues. CLUE benchmark team (SuperCLUE) Function calling Everyday Not in cards GPT-4o 78.4 Oct 2023 Jun 2024
SuperCLUE-Edit Models edit an input image according to a Chinese instruction. The evaluation measures whether the requested change preserves relevant details. SuperCLUE Image generation and editing Everyday Not in cards GPT-Image-2.5 Sunburst 82.5 Oct 2025
SuperCLUE-Fact Models answer short Chinese questions and identify false statements involving knowledge, common sense, misleading prompts and context. SuperCLUE Short factual questions School Not in cards DeepSeek-R1 86.0 Apr 2025 Jun 2025
SuperCLUE-I2V Models generate videos from still images. SuperCLUE Video generation University Not in cards MiniMax H3 71.5
SuperCLUE-Role Models portray characters in Chinese conversations. A judging model rates aspects of the portrayal against reference answers. SuperCLUE team Chat and instructions in many languages Everyday Not in cards GPT-4 Turbo (Jan 2024) 75.2 Apr 2024 Feb 2024
SuperCLUE-Safety Chinese language models answer 4,912 open-ended safety questions over more than 20 sub-dimensions in multi-round adversarial dialogues. Follow-up turns push harder than the first prompt. CLUE benchmark team (SuperCLUE) Jailbreaks and prompt injection Everyday Not in cards Yi-34B 89.3 Sep 2023 Nov 2023
SycoBench-600 Models decide whether to revise an answer after user feedback. The questions distinguish justified corrections from pressure to accept a false claim. Debu Sinha Wellbeing and sycophancy Everyday Not in cards Gemini 2.5 Flash 95.3% Jul 2026 Sep 2025
SYNERGY systematic review screening Models rank titles and abstracts against a review’s eligibility criteria. Reviewers’ inclusion decisions supply the reference labels. Utrecht University and collaborators Medicine University Not in cards Claude Opus 5.5 100% Apr 2023 Sep 2026
Testing model's emotional inteligence Models identify the emotion in a story. A judge compares each response with the reference emotion on a sample of ten stories. 23f3000941 Reading comprehension Everyday Not in cards Claude Haiku 4.5 100% Oct 2026 Sep 2026
TRAJECT-Bench Agents select and execute tools to answer queries. The evaluation checks whether the actions satisfy a reference trajectory and produce the correct answer. Pengfei He and collaborators, including Hippocratic AI Function calling University Not in cards Gemini 2.5 Pro 70.5% Aug 2025 Sep 2025
ts-bench Coding agents such as Claude Code, Codex and Gemini CLI solve TypeScript tasks, and tests decide pass or fail. Version 1 uses 25 Exercism practice exercises. Version 2 uses real tasks from a large TypeScript monorepo, taken from SWE-Lancer. laiso (independent) Function-level problems University Not in cards Claude Sonnet 4.6 80% Jun 2025 Mar 2026
TVRBench Agents move through simulated indoor scenes to reproduce a target image's viewpoint. Success requires stopping at exactly the target pose on the simulator's discrete grid. The evaluation split has 500 tasks across 80 scenes. Zhejiang University Simulated robots Everyday Not in cards Gemini 3.1 Pro 12% May 2026 Apr 2026
Verify Gym Models give tighter rational bounds for roots on 120 held-out tasks. Exact integer arithmetic verifies both bounds and determines how much precision the answer adds. carlos-toledo Word problems University Not in cards DeepSeek V4.1 Flash 0.10 Oct 2026 Sep 2026
VeriGap Models implement programs and write tests from specifications. The harness checks program correctness, test validity, and whether the tests catch mutations. Ahmed A. Fixing real repositories University Not in cards GPT-6.1 Sol 0.66 Nov 2026 Sep 2026
VGI-Bench Video generation models solve 27 visual reasoning tasks with 810 instances through generated frames. Scoring checks that the process is valid, not only the final frame. University of Illinois Urbana-Champaign Video world models Everyday Not in cards Dreamina Seedance 2.0 51% Aug 2026
VibeVideoBench Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result. BUPT + Zhongguancun Academy + Peking University + Jilin University + Kling AI Video generation Expert Not in cards GPT-6 Astra 4.11 Oct 2026 Sep 2026
VisualWebArena Multimodal web agents complete 910 visually grounded tasks on self-hosted classifieds, shopping and Reddit-style sites. Execution-based checks score each task. Carnegie Mellon University Web browsing University Not in cards GPT-4V 16.4% Jan 2024 Apr 2024
VLegal-Bench Models answer questions and solve legal scenarios under Vietnamese law. Legal experts check that reference answers follow authoritative documents. Nguyen Tien Dong and colleagues Law University Not in cards Gemini 2.5 Flash 72% Dec 2025 Sep 2025
WildlifeBench Wildlife Knowledge V1 Models answer 185 questions about wild species without search or tools. The overall score weights ten tasks equally, including taxonomy, conservation status, and geographic range. WildlifeBench Science questions University Not in cards Muse Spark 1.3 84.7% Nov 2026 Oct 2026
WISER-FindAll findall Entity discovery systems return all entities that match complex multi-criteria web queries. Parallel Web Systems Deep research Expert Not in cards OpenAI deep research 21% Nov 2025 Feb 2025
xbench-Marketing Agents search for influencers who fit 50 real advertiser briefs, scored against a curated pool of 836 influencers. xbench (HongShan) Running a business Expert Not in cards o3 50.8 Jun 2025 May 2025
2026 Times Crossword Championship Models fill seven British cryptic crossword grids. The evaluator counts perfect grids and records the time and clues solved for each attempt. The Times and Siddhartha Chatterjee Puzzles and chess Competition Not in cards GPT-6 Astra 100% Oct 2026 Sep 2026
3D SubwayBench Agents use Command Code to build a Subway Surfers style endless runner in a single HTML file. The scorecard grades six features from the submitted code. Command Code Building whole projects University Not in cards Claude Opus 5 5.00 Jul 2026 Jul 2026
67AI Lab L2 nuclear medicine report structuring Models extract findings from 300 synthetic PET/CT and SPECT reports into a fixed form. They also flag conflicting values in 17 reports. 67AI Lab Medicine University Not in cards Gemma 4 26B A4B 1.00 Nov 2026 Sep 2026
67AI Lab L3 nuclear medicine guideline questions Models answer questions using eight retrieved passages from nuclear medicine guidelines. They must abstain when the passages lack an answer. 67AI Lab Medicine University Not in cards Claude Opus 5.5 90.8% Nov 2026 Sep 2026
AgentCLUE-Task Agents carry out interactive tasks in Chinese. SuperCLUE Apps and MCP servers Everyday Not in cards Kimi K3 49.5 Jul 2026
AgenticRL-Bench Agents debug and maintain reinforcement learning infrastructure. A task passes when the implementation meets every hidden check. The Environment Lab at Eigent AI Machine learning engineering Expert Not in cards GPT-6.1 Sol 22.8% Nov 2026 Nov 2026
AgentKernelArena Coding agents optimize GPU kernels through full workflows with compilers and profilers. Scores check speedups and whether gains generalize beyond tuned shapes. AMD GPU kernels Expert Not in cards Claude Opus 4.6 3.23× May 2026 Apr 2026
AGILE Automatic Genre Identification Benchmark Models assign one of eight genres to web texts, including news, opinion, and advertising. The test covers English and ten other languages. Taja Kuzman, Igor Mozetič, and Nikola Ljubešić Reading in many languages Everyday Not in cards GPT-5 mini 0.82 Jun 2023 Sep 2025
AI GameStore Vision-language models play short episodes of 100 games adapted from Apple App Store and Steam top charts. Each game score is divided by the average human score, and models are ranked by the geometric mean. Lance Ying, Joshua B. Tenenbaum and collaborators Video games Everyday Not in cards Gemini 2.5 Pro 12.4% Feb 2026 Dec 2025
AI Security Leaderboard Models face systematic jailbreak attacks across chemical, biological, radiological, nuclear, explosive and cyber domains. Scores count universal jailbreaks and estimate attacker cost. FAR.AI Jailbreaks and prompt injection Expert Not in cards Grok 4.5 $60 Jul 2026 Sep 2026
AMLBench v0.2 (Superjective) Models decide whether to escalate transaction alerts. The score measures how often they escalate reportable alerts under changes to analyst instructions and case narratives. Scott Burns and Superjective Finance University Not in cards Llama 3-70B 100% Jul 2026 May 2026
AnalogCoder-Pro Agents write analog circuit programs and use simulated waveforms to diagnose failures. The authors report the fraction of successful designs and the number of tasks solved. Yao Lai and colleagues Electronics Expert Not in cards Claude 3.7 Sonnet 88.4% Sep 2025
AnalogPortBench Models adapt analog circuit netlists to a different chip process or flag infeasible specifications. A simulator checks the designs against electrical limits. University of Delaware Electronics Expert Not in cards GPT-6.1 Sol 56% Oct 2026 Oct 2026
AstroVLBench Models interpret galaxy images, light curves, and spectra across five observing tasks. The authors compare their classifications with reference labels and examine their explanations. Wenke Ren and colleagues Lab and data work University Not in cards Gemini 3 Pro 68.8% Apr 2026 Feb 2026
Atrex-Bench Agents implement GPU kernels for 30 operators and 440 shapes sampled from production inference workloads. The evaluator checks compilation and numerical correctness, then scores performance against hardware roofline ceilings with weights from production GPU usage. Alibaba GPU kernels Expert Not in cards GPT-5.5 0.11 Jun 2026 May 2026
AutoCAD Bench Computer-use agents turn a dimensioned reference drawing into a correct, editable DWG in AutoCAD through the screen alone, with mouse and keyboard. 50 tasks span 2D drafting and 3D modeling at four difficulty levels. Markov CAD models Expert Not in cards GPT-5.6 Sol 46% Jul 2026 Jul 2026
AutoMedBench Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission. University of California, Santa Cruz and NVIDIA Machine learning engineering Expert Not in cards Claude Opus 4.6 66.5 Jun 2026 Mar 2026
BABILong Models answer bAbI reasoning questions whose supporting facts are scattered through very long distractor text, at lengths up to millions of tokens. AIRI and collaborators (Kuratov et al.) Reasoning over long documents Everyday Not in cards GPT-4 Turbo (Jan 2024) 72.4% Jun 2024 May 2024
Behavior2Code Agents reconstruct command line programs from documentation and observed behavior. Hidden tests compare each implementation with the original. Turing Building whole projects Expert Not in cards GPT-6 Astra 31.7% Aug 2026 Sep 2026
BIBIMBAP Agents read and edit IFC building files. Executable tests compare their changes and answers with reference results. University of Rostock and Clausthal University of Technology CAD models University Not in cards Claude Opus 4.6 50.2% Mar 2026 Mar 2026
BLUR Models identify a remembered item from incomplete clues in several languages and media. Patronus AI Questions that need search Everyday Not in cards OpenAI Computer-Using Agent (CUA) 54% Mar 2025 May 2025
BrowseSafe-Adapt Browser agents face adaptive prompt injections across 1,024 tasks. The score is attack success within 64 attempts. Perplexity Jailbreaks and prompt injection University Not in cards Claude Sonnet 5 30.7% Oct 2026 Jul 2026
BuildArena 2.0 Construction Challenge Agents build and test machines in Besiege for seasonal engineering challenges. The organizers score each entry using simulation results and the rules for its track. BuildArena team at Westlake University Video games Competition Not in cards Jun 2026
Cekura Text to Speech Round-trip word error rate averaged across the two transcribers, as displayed by Cekura. Cekura Audio understanding and voice Everyday Not in cards Fish Audio Drama 3 1.92% Jul 2026 Nov 2026
Chatbots and the Ballot Box Models answer voter questions in English and Spanish across ten US states. Analysts check routine answers against election guidance. A model grades responses to disputed claims. Institute for Strategic Dialogue Long-form factuality Everyday Not in cards GPT-5.5 89.3% Sep 2026 May 2026
CheckerBench Long-horizon agents read a defect specification and a repository, then write and refine a working static-analysis checker using compiler and analyzer feedback. Humanlaya Data Lab Fixing real repositories Expert Not in cards Claude Opus 4.8 45.3% Oct 2026 Aug 2026
ChJGBench Models draft the reasoning and orders in Chinese civil, criminal, and administrative judgments. The evaluator compares their work with 3,462 real judgments. Zhejiang University and collaborators Law Expert Not in cards Gemini 3 Pro 0.74 Nov 2026 Mar 2026
CircuitSense Models read circuit schematics and block diagrams, derive equations, and select designs. SymPy checks symbolic answers. Other tasks use exact matches, a model judge, or simulation. Northeastern University and Brookhaven National Laboratory Visual math and diagrams University Not in cards Gemini 2.5 Pro 68.9% Sep 2025 Aug 2025
Civil-Bench Models answer Chinese civil engineering licensing questions. Answer keys grade the objective questions. In the paper, GPT-4 Turbo grades written answers against individual scoring points. Tongji University Exam questions University Not in cards DeepSeek-R1 64.8 May 2025
CLBench-V Models must learn from task-specific multimodal context, such as figures, tables, maps and web pages, instead of prior knowledge. Tasks run from grounding to knowledge acquisition. CLBench-V authors General visual questions Everyday Not in cards InternVL3.5-30B-A3B 28.5% Jul 2026 Apr 2026
CMB Models answer 11,200 Chinese medical exam questions and questions about 74 clinical cases. Exam answers receive exact-match scores. A model judge rates clinical answers for fluency, relevance, completeness, and medical proficiency. Freedom Intelligence Medicine Expert Not in cards Yi-6B-Chat 66.9% Aug 2023 Nov 2023
CNN/DailyMail Summarization Models summarize news articles from CNN and the Daily Mail. The harness compares generated summaries with the publishers’ highlights. Karl Moritz Hermann and colleagues Faithful to sources School Not in cards OPT-175B 0.15 Jun 2015 Jul 2022
Cognition CodeSearch Eval Agents retrieve relevant files and line ranges for questions about real repositories. Cognition compares their answers with human labels using precision-weighted F scores. Cognition Fixing real repositories University Not in cards GPT-5 0.59 Oct 2025 Oct 2025
Command Code Flappy Bird Agents use Command Code to build a Flappy Bird game with Canvas. The scorecard grades features including collisions, window resizing, and offline play. Command Code Building whole projects University Not in cards GLM-5.2 4.83 Jul 2026 Jul 2026
Command Code Pixel Art Space Shooter Agents use Command Code to build a scrolling Canvas game with cave walls and collectible gems. The scorecard grades features including collisions and offline play. Command Code Building whole projects University Not in cards Grok 4.5 4.83 Jul 2026 Jul 2026
Competitive Programming Code Generation Models solve 120 Codeforces problems in Python. The harness runs their submissions against test cases. Montasir Competitive programming Competition Not in cards GPT-6.1 Sol 90% Oct 2026 Sep 2026
ConceptEdit-Bench Models edit images according to 1,000 instructions across six categories. A vision-language judge scores instruction following, visual quality, and consistency in unedited regions from 0 to 10. The per-case score averages those three dimensions. Shanghai Jiao Tong University and Ant Group Image generation and editing Everyday Not in cards Gemini 3.1 Flash Image 66.2 Aug 2026 Feb 2026
CooperBench Two coding agents each get a different feature for the same repository and must coordinate so both changes work together. Over 600 tasks span 12 libraries in 4 languages. Stanford NLP Fixing real repositories Expert Not in cards GPT-5 27.9% Jan 2026 Dec 2025
CTF-Archive Diamond Agents solve 285 difficult capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. US CAISI (NIST) Capture-the-flag challenges Expert Not in cards GPT-5.5 71% May 2026 Jun 2026
CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. Vals AI Capture-the-flag challenges Expert Not in cards Claude Opus 5.5 87.7% Oct 2026 Oct 2026
CyberStrike Agents complete cybersecurity tasks. Automated checks grade the outcomes. Turing Secure code and review Expert Not in cards GPT-5.6 Sol 31.5% Aug 2026 Jul 2026
DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights. ByteDance Seed Data science and SQL Expert Not in cards GPT-5 47% Dec 2025 Aug 2025
DataPrep-Bench Models prepare training data across six domains, and a base model is fine-tuned on their output. Downstream results of that model give the score. Peking University Machine learning engineering Expert Not in cards Qwen3.5-Plus 48.4 May 2026 Feb 2026
Dấu or Die: Vietnamese Diacritic Restoration Models restore missing diacritics in 37 Vietnamese sentences. A sentence earns credit only when every mark matches the reference. tuan nguyen Reading in many languages School Not in cards Gemini 3.6 Flash 97.3% Oct 2026 Oct 2026
DEEPSYNTH Agents search multiple websites and combine evidence to answer 120 research tasks across seven domains. They return structured JSON answers scored with precision, recall, F1, and exact match. A separate model judge assesses answers beyond exact matching. Huawei Noah's Ark Lab and university collaborators Deep research Expert Not in cards OpenAI deep research 8.97% Feb 2026 Dec 2025
Document AI Benchmark Arena Models extract fields from business documents using a common schema and OCR text. The scorer normalizes numbers and dates and allows small spelling differences in names and addresses. DocAI Fabric Documents, charts and text in images Everyday Not in cards GPT-4.1 92.6% Feb 2026 Jun 2025
E/RP Leaderboard by PsychoSmiley Models answer 25 yes or no questions and write three roleplay responses. Claude Sonnet 4.5 grades the two parts for up to 325 points. PsychoSmiley Creative writing Everyday Not in cards Kimi K2.5 289.0 Nov 2025 Jun 2026
EgoProceVQA Models watch first-person clips and answer questions about the steps in a procedure. Some questions ask when a step occurred or which step is missing. The Hong Kong Polytechnic University and South China University of Technology Video understanding Everyday Not in cards GPT-5.1 51.6% Jul 2026 Dec 2025
EKWbench Agents complete operational tasks such as sourcing reviews and referral triage. The score averages the share of verifier weight earned. Turing Tasks from real jobs Expert Not in cards Claude Opus 5 74.4% Jul 2026
EmbodiedCLUE-World 2.0 World models predict outcomes in embodied settings under the second evaluation protocol. SuperCLUE Interactive world models Everyday Not in cards LingBot-Video-MoE 41.8 Jul 2026 Jul 2026
ExcelFixBench Agents repair formulas, data, and formatting in an existing workbook. The task rubric checks the repair and whether unrelated content survived. Tencent Documents and spreadsheets University Not in cards Tencent Hy4 27.8 Nov 2026 Nov 2026
Extract Bench (Contextual AI) Models extract structured JSON from PDFs. Field checks include exact matches, numerical tolerances and semantic judgments. Contextual AI Documents, charts and text in images University Not in cards Gemini 3 Flash 0.07 Feb 2026 Dec 2025
Factorio Learning Environment Agents write Python programs that control a Factorio factory, in lab tasks with fixed resources and in open play. Scored by production throughput, milestones and lab task success. Jack Hopkins et al. (FLE team) Video games Expert Not in cards Claude 3.5 Sonnet (October 2024) 293206.0 Mar 2025 Feb 2025
Faroese Grammatical Correctness Models choose the correct Faroese sentence from a pair drawn from student essays and their corrections. University of the Faroe Islands and University of Copenhagen Reading in many languages Everyday Not in cards GPT-5 90.9% May 2026 Aug 2025
FinBen A finance suite of 35 datasets across 23 tasks, in three difficulty levels. Tasks include sentiment analysis, information extraction, numerical question answering, text generation, forecasting and stock trading. The Fin AI (Qianqian Xie and coauthors) Finance University Not in cards ChatGLM3-6B 0.25 Feb 2024 Apr 2024
FinStepBench CorporateValuation Models value companies using financial evidence and calculations. The score is the published CorporateValuation percentage. StepFun Finance Expert Not in cards GPT-6 Astra 77.3% Sep 2026 Sep 2026
FinStepBench DeepResearch Models write finance research reports. The score is the published DeepResearch percentage. StepFun Finance Expert Not in cards Claude Opus 5 59.1% Sep 2026 Sep 2026
FinStepBench LiveSearch Models retrieve and verify live financial facts. The score is the published LiveSearch percentage. StepFun Finance Expert Not in cards DeepSeek V4.1 Flash 76.7% Sep 2026 Sep 2026
FraudBench (GOAL Lab) Models face 159 scenarios drawn from 112 real U.S. fraud lawsuits. Yes or no questions, open questions and agent tasks test whether a model recommends, assists or commits fraud. GOAL Lab Refusing harmful requests University Not in cards gpt-oss-120b 10.1% Nov 2026 Nov 2025
GarbleBench Models answer factual questions after controlled keyboard typos, dropped letters, stray characters and swaps corrupt the text. Adam Holter Instructions and chat Everyday Not in cards Gemini 4 Argon 55.2% Oct 2026 Oct 2026
GenomeQA Models answer multiple-choice questions about short genomic sequences. The benchmark covers six task families, including species classification. Weicai Long and colleagues Genomics and life sciences University Not in cards Gemini 3 Pro 63.6% Apr 2026 Nov 2025
GTO Wizard Benchmark Models play heads-up no-limit Texas Hold’em against GTO Wizard AI. AIVAT reduces variance while the benchmark reports poker winnings. GTO Wizard research team Multi-agent games Expert Not in cards GPT-5.5 -9.23 Mar 2026 Jul 2026
GUI-CEval Models answer questions about Chinese app screens, locate controls, and choose actions. Agents also complete tasks on physical devices, where the grader checks interface elements. Xiaomi HyperAI Team Phones Everyday Not in cards Qwen2.5-VL-72B 61.4% Mar 2026 May 2025
HABIT-Bench Agent memory systems must infer user habits from weak evidence spread across many past conversations and apply them correctly later. Shanghai Jiao Tong University (LUMIA) Learning from the context Everyday Not in cards GLM-5.3-Flash 42% Oct 2026 Aug 2026
HarnessDev Agents build a complete agent harness from a minimal seed, then revise their own harness using downstream execution feedback. The harness's task results are the score. ByteDance Seed AI research tasks Research frontier Not in cards Claude Opus 4.8 67.8% Sep 2026 May 2026
HLE-Verified (text-only Full) Models answer the text-only questions of Humanity's Last Exam after expert verification and revision. The Full set keeps the questions experts marked uncertain, and the score is pass@1 accuracy. SKYLENAGE AI Expert-written questions Expert Not in cards Claude Opus 4.6 46.8% Feb 2026 Feb 2026
HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. Ramy ElMallah and colleagues Simulated robots Everyday Not in cards GPT-6 Astra 61.9% Nov 2026 Sep 2026
ICML 2025 Experts Benchmark Models answer tasks submitted by participants at ICML 2025 in Vancouver. Kaggle reports overall performance and individual task outcomes. Kaggle Language reasoning University Not in cards Gemini 2.5 Pro 81.1% Jul 2025 Jun 2025
Infrathon Agents optimize AI infrastructure under correctness and fidelity constraints. Hidden workloads measure performance gains, with additional checks for rule compliance. University of Washington, Ai2, and collaborators Speed and optimization Expert Not in cards Claude Opus 5.5 91.2 Nov 2026 Sep 2026
InteractiveBench Logic A model asks restricted questions to recover the hidden explanation of a situation puzzle. It solves an item only if a judge accepts its final explanation within 20 rounds. InteractiveBench and Princeton University Abstract puzzles and logic Everyday Not in cards Gemini 3 Flash 30.4% Mar 2026 Dec 2025
InteractiveBench Math A model solves mathematical problems while querying a judge about intermediate claims. The judge holds a reference derivation and marks final solutions correct or incorrect under a fixed interaction budget. InteractiveBench and Princeton University Research mathematics Expert Not in cards Grok 4.1 Fast 76.9% Mar 2026 Dec 2025
InternBootcamp Models solve generated reasoning tasks with task-specific answer checkers. The hub port lets users select a task and sample examples with a fixed seed. InternLM Language reasoning University Not in cards DeepSeek-R1-Distill-Qwen-32B 55.7 May 2025 May 2025
iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. Carnegie Mellon University (Salakhutdinov lab) Phones Everyday Not in cards Claude Opus 4.6 51.9% Jun 2026 Mar 2026
JevBench Decision models, such as TypeSafe AI's Jev and compatible systems, answer typed decision questions. The board scores accuracy, probability calibration, latency and cost. Benchmark Heaven Other Everyday Not in cards GPT-6 Luna 98.4% Sep 2026 Sep 2026
JFBench Models answer Japanese prompts with multiple verifiable constraints. A response passes only if it satisfies every constraint. Preferred Networks Chat and instructions in many languages Everyday Not in cards Muse Glimmer 48.9% Mar 2026 Aug 2026
JusticeAxis Models assess evidence and produce judgments for 256 criminal cases from 18 countries. The evaluator checks charges and outcomes against lawyer-written references. The Chinese University of Hong Kong and collaborators Law Expert Not in cards Gemini 3.8 Flash 84.4% Sep 2026 Sep 2026
K12-Bench Models select curriculum concepts, prerequisite relationships, related knowledge, experimental evidence, and textbook locations. The 23,640 multi-select questions come from a textbook knowledge graph. Exact match and instance-level F1 compare selected options with graph-derived answers. Peking University, Institute for Advanced Algorithms Research Shanghai, OriginHub Technology, and Zhongguancun Academy Language reasoning School Not in cards Gemini 3 Flash 57.1% May 2026 Apr 2026
K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. Chong Li and colleagues Other professions School Not in cards InternVL2.5-78B-MPO 63.4% Jun 2025 Jan 2025
K9-Bench Models answer 4,744 questions about dog behavior in 907 videos. The evaluator compares each free-form answer with five options that the model never sees. Ogmen Robotics and Georgia Institute of Technology Video understanding School Not in cards Gemini 2.5 Pro 40.1% Jun 2026 Oct 2025
Kaggle's Task Tuesday Benchmark Models attempt tasks selected by the Kaggle team from community submissions. The board reports an overall score and individual task results. Kaggle Language reasoning University Not in cards Gemini 3 Flash 83.3 Jan 2026 Dec 2025
KnowU-Bench Agents use Android apps such as Mattermost and Taodian. They infer preferences, ask for missing details, and decide when to act or stay silent. Zhejiang University, Apple, and Tencent Phones Everyday Not in cards Claude Sonnet 4.6 60.4% Apr 2026 Feb 2026
Legacy-Bench Agents debug, add features to and migrate enterprise code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. A task passes only when hidden tests all pass. Factory Fixing real repositories Expert Not in cards GPT-5.6 Sol 60% Sep 2026 Sep 2026
LiveSearch Agents search for current financial figures. The evaluator runs a stored query against a professional data service to update the reference answer. Sinuo Wang and colleagues Finance University Not in cards DeepSeek V4.1 Flash 76.7% Oct 2026 Sep 2026
LLM Fantasy Football Arena Guillotine Models draft NFL players and bid for free agents. Each week the lowest-scoring team leaves, and the remaining teams bid on its players. Wildcard Labs Multi-agent games Everyday Not in cards MiniMax-M3 10.0 Sep 2026 Aug 2026
LLM SoccerArena La Liga 2026/27 Models forecast La Liga match scores and outcome probabilities with web access. The standings award points for score predictions made about 24 hours before kickoff. LMU Munich, University of Cologne, and Paderborn University Real-world events Everyday Not in cards Claude Opus 4.8 129.0 May 2026
LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. Songhan Zuo and colleagues Law Expert Not in cards Kimi K2.5 0.66 Jun 2026 Apr 2026
LoopArena Models act as the controller loop around a coding agent. They monitor progress, assign work, run checks and decide when the task is safe to submit. AMAP-ML (Alibaba) Building whole projects Expert Not in cards GPT-5.5 36.3% Aug 2026 Jul 2026
LT-History Models answer questions in Lithuanian about national and world history. Yevhen Kostiuk and colleagues Knowledge in many languages School Not in cards Qwen2.5-72B 82% Mar 2025 Sep 2024
Math24o Models solve Chinese high school mathematics olympiad preliminary problems from 2024. A program compares their final answers with reference answers. SuperCLUE team Competition problems with answers Competition Not in cards o3-mini 85.7% Mar 2025 Mar 2025
MatSciBench Models solve college materials science problems, including questions with figures. The authors report accuracy separately for text and image questions. Junkai Zhang and colleagues Science questions University Not in cards GPT-5 64% Oct 2025 Aug 2025
Med-HALT Models answer medical exam questions designed to reveal hallucinations. The hub port uses the false-confidence, fake-question, and none-of-the-above reasoning tasks. Med-HALT authors Hallucination rates Expert Not in cards Falcon-40B 42.7% Jul 2023 Jul 2023
Mind the Gap Models describe visible events during natural pauses in a video's soundtrack. The evaluator renders the narration and checks its timing and coverage. Google Research and UC Berkeley Video captioning University Not in cards Qwen3.5-Omni-Flash 0.94 Oct 2026 May 2026
MIRAGE-MMMT Models decide whether to ask for more information or answer a grower during an agricultural consultation. They then write the question or answer. University of Illinois Urbana-Champaign and Amazon Multi-turn conversation University Not in cards GPT-4o 65.5% Jun 2025 Apr 2025
MooseBench Models write input files for MOOSE, a multiphysics simulator. The grader reconstructs each file’s governing equations and compares them with the intended physics. Zhenghan Song, Yulong Liu, and colleagues Scientific code Expert Not in cards Claude Sonnet 4.6 0.85 May 2026 Apr 2026
Mosaic Eval Gauntlet v0.3.0 Models complete tasks across reading, reasoning, world knowledge, language understanding, and long context. MosaicML groups results by capability. MosaicML Language reasoning University Not in cards Mistral 7B v0.1 0.52 Sep 2023
NAFD E1 Financial Reporting Models sit professional finance exams on topics including financial reporting, planning, and taxation. A grader compares their answers with the official keys. Nelson VIKOU Finance University Not in cards Claude Opus 5 0.93 Nov 2026 Sep 2026
NatureBench Coding agents work on 90 tasks distilled from Nature-family papers, each in its own container, and try to match or beat the published results. Frontis AI Research science Research frontier Not in cards Claude Opus 5 23.3% Jun 2026 Jul 2026
NautData test set Models answer questions, describe underwater scenes, and locate organisms in images. The paper reports 7,916 test examples on 3,920 images. Huazhong University of Science and Technology and National University of Defense Technology General visual questions University Not in cards mPLUG-Owl3-7B 91.9% Oct 2025 Feb 2025
NewsBench (Chinese journalism) Models complete Chinese newsroom assignments and answer multiple choice questions about writing and editorial safety. GPT-4 evaluates the written responses. Miao Li and colleagues at the Institute for Advanced Algorithms Research, Shanghai, and partner institutions Other professions University Not in cards GPT-4 Turbo 2.44 Feb 2024 Nov 2023
norregel Models supply the numeric value of a Norwegian legal provision on a specified date. The evaluator distinguishes current values from superseded ones. Eirik Botten Nicolaysen Law University Not in cards Claude Opus 4.5 45.1% Oct 2026 Apr 2026
Norwegian Courts Bitext Mining Models match court sentences written in Norwegian Bokmål and Nynorsk. The test contains 228 aligned pairs and scores matches with F1. Scandinavian Embedding Benchmark team Translation School Not in cards Gemini Embedding 2 100% Jan 2024 Oct 2026
OmniEarth-Bench Models interpret satellite images and observational data across Earth systems, answering questions about environmental processes and their interactions. Multiple-choice accuracy and model judgments score answers. Separate measures check localization, captions, and annotated reasoning steps. Shanghai Artificial Intelligence Laboratory and university collaborators Research science Expert Not in cards LLaVA-OneVision 7B 31.5% May 2025 Mar 2025
OpenBounds V2 Pilot Agents schedule jobs under machine and resource constraints. The evaluation compares each schedule with a greedy baseline and a reference solution. Andre Albrecht Speed and optimization University Not in cards Gemini 3.8 Flash 0.83 Nov 2026 Sep 2026
OpenStudio-MCP LLM Test Benchmark v1.2.1 Agents change building energy models with OpenStudio tools. The grader checks the tool calls and inspects the resulting model against task requirements. National Laboratory of the Rockies Apps and MCP servers University Not in cards Claude Opus 4.6 100% Aug 2026 May 2026
OpenThoughts-TBLite Agents solve 100 Terminal-Bench tasks picked to track Terminal-Bench 2 scores while running faster. Tasks are balanced across four difficulty bands using Claude Haiku 4.5 pass rates. OpenThoughts Agent team, Snorkel AI and Bespoke Labs Terminal tasks Expert Not in cards Kimi K2.5 75.1% Feb 2026 Jan 2026
OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. Shanghai AI Laboratory (OpenRobotLab) and collaborators Embodied reasoning Everyday Not in cards GPT-4.1 53.4% Jul 2025 Apr 2025
PCA-Bench Models interpret visual scenes, explain their reasoning, and select actions for embodied tasks. GPT-4-0125 compares perception concepts, reasoning, and actions with annotated references. Peking University and Alibaba Group Embodied reasoning Everyday Not in cards Qwen-VL-Max 0.70 Feb 2024 Mar 2024
PhAIL DROID: Everyday tasks Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds. Positronic Robotics Real robots Everyday Not in cards Cosmos3-Nano 68.1% Oct 2026 Sep 2026
PhoneShell iOS Agent Benchmark Agents operate apps on a physical iPhone to set timers, find books, and complete other everyday tasks. Automated checks read the resulting device state. Blolabel Phones Everyday Not in cards Kimi K3 96.6% Sep 2026 Jul 2026
Pieverse Agent Prediction Arena Agents trade event contracts on Polymarket with $1,000 each in real money. Pieverse tracks account value as market prices change. Pieverse Real-world events University Not in cards Claude Sonnet 4.5 $981 Dec 2025 Dec 2025
Pigeonholing Partisans four-word generation Models use a survey respondent’s profile to generate four words describing supporters of each major US party. One measure counts how often raters identify the writer’s party. Lisa P. Argyle and colleagues Social simulation Everyday Not in cards Mistral-7B-Instruct-v0.1 51.5% Sep 2022 Jun 2024
Ping or Peek Coding agents receive an ambiguous request. A judge checks whether they ask the user for clarification or consult the grader. Snorkel AI Deception and alignment Everyday Not in cards Claude Sonnet 5.5 0.59% Sep 2026
PLM-VideoBench Models answer questions about activities, describe events involving a particular person or object, and locate those events in time. Meta also tests first-person video understanding. Meta FAIR Video captioning Everyday Not in cards Gemini 2.0 Flash 42.5 Apr 2025 Feb 2025
Precedent vs. Spec Models complete tables whose example rows conflict with the written rule. The grader checks which rule they followed and whether they mention the conflict. Finn Mead Following precise instructions School Not in cards GPT-5.5 62.5% Oct 2026 Aug 2026
Pressure-Tested Alignment Models choose how to act in scenarios where termination threatens their objective. A judge checks whether they take a shortcut that games the reward. Ram Acharya Deception and alignment University Not in cards Gemini 3.8 Flash 1.00 Oct 2026 Sep 2026
ProjectTest Models write unit tests for existing Python, Java, and JavaScript projects. The authors run the tests and measure code coverage. Yibo Wang and colleagues with Scale AI Fixing real repositories University Not in cards Gemini 3 Pro 82% Feb 2025 Nov 2025
ProOPF-B Models translate operational requirements into executable optimal power flow programs. They change grid parameters, objectives, and constraints. The evaluator checks implementations against expert references. Zhejiang University, Peking University, Xi’an Jiaotong University, and China Southern Power Grid Scientific code Expert Not in cards Gemini 3 Pro 37.2% Feb 2026 Dec 2025
Putnam-like Models write proofs for 96 university problems in 8 sets of 12, in the style of the easier and middle Putnam tiers. Each problem has a Lean statement and a point rubric for grading. Google DeepMind Written proofs University Not in cards Gemini 2.5 Pro 94.5% Sep 2025 Jun 2025
PuzzleWorld Models solve 667 puzzle-hunt problems that require varied reasoning strategies. PuzzleWorld authors Puzzles and chess Competition Not in cards Gemini 3 Pro 18% Jun 2025 Nov 2025
Ramp Stack Accounting Benchmark Agents perform month-end accounting work across eight simulated businesses. Ramp measures the fraction of grading criteria each agent satisfies. Ramp Labs Finance Expert Not in cards Claude Opus 4.6 62.1% Jun 2026 May 2026
RECODE-H Agents implement research code from papers and repositories. They can request feedback from a simulated collaborator before their code runs against tests. Chunyu Miao and colleagues AI research tasks Expert Not in cards GPT-5 84.3% Oct 2025 Sep 2025
RiceChem Models grade long answers to four college chemistry questions. Each of the 1,264 answers has human judgments for the applicable rubric items. Shashank Sonkar and colleagues at Rice University Other professions University Not in cards Mistral-7B-Instruct-v0.2 0.43 Apr 2024 Apr 2024
RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. Fudan University, Shanghai Innovation Institute, and Shanghai AI Laboratory with collaborators Puzzles and chess Everyday Not in cards Gemini 3.1 Pro 49.9% Jun 2026 Jun 2026
RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. Changxuan Fan and colleagues Simulated robots University Not in cards GPT-6.1 Sol 80.6% Nov 2026 Oct 2026
ROSTrans Models migrate robot software from ROS 1 to ROS 2. The evaluator builds each package, runs its nodes, and tests their behavior. New York University Fixing real repositories University Not in cards Claude Opus 4.6 10.9% Oct 2026 Apr 2026
RSI Control Arena An agent does one of 14 tasks from a successor model's training pipeline, such as SFT or reward modeling, while covertly trying to install one of 16 misaligned properties in that successor. A held-out oracle checks whether the property landed, and LLM monitors try to flag the attack from the transcript. Scale AI Deception and alignment Research frontier Not in cards Muse Spark 1.3 14.3% Oct 2026 Sep 2026
rTaste Agents propose, rank, and forecast experiments inspired by recent research papers. An independent executor attempts the experiments, and auditors assess their results. University of Minnesota and collaborators AI research tasks Expert Not in cards GPT-6.1 Sol 3.71 Oct 2026 Sep 2026
Safety-Scene Vision Benchmark Models identify visible hazards in everyday photos. The evaluation compares answers under a neutral prompt and a prompt claiming an inspector found danger. Clivin John General visual questions Everyday Not in cards Gemini 3.7 Flash 0.86 Oct 2026 Aug 2026
ScienceAgentBench Agents write programs for 102 data-driven discovery tasks from 44 papers in four disciplines. Outputs are checked by execution and task-specific metrics. Ohio State University (OSU NLP Group) Research science Expert Not in cards o1-preview 42.2% Oct 2024 Sep 2024
SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. C5R Lab and data work Expert Not in cards Claude Fable 5.1 45.3% Sep 2026 Sep 2026
SDR-Bench Models personalize sales pitches using customer research. A judge compares each pitch with points from successful human outreach. Ashutosh Srivastava and colleagues Running a business University Not in cards Claude Sonnet 4.6 55.8% Feb 2026 Mar 2026
Seeing What Execution Needs Agents choose camera views that help a robot complete a tabletop task. The evaluator runs the action and checks whether the requested state holds. Columbia University, Stanford University, and collaborators Simulated robots Everyday Not in cards Grok 4.3 42.9% Nov 2026 Aug 2026
ShadowBench Models pose two 3D hands between a lamp and a screen, one tool call at a time, until their shadow matches one of 20 hand-shadow plates from books of 1859 to about 1900. The environment refuses any pose real hands can't make. Score, out of 100, mixes the overlap with the target and how closely the outlines match. Perpetual Research Spatial reasoning and counting Everyday Not in cards Claude Opus 5.5 48.8 Nov 2026 Oct 2026
SilverquiLLM-bench hob-medium Coding agents implement five Hobbit cards in a Python game engine. The grader checks card behavior with 90 hidden tests. Snowfox Builds Fixing real repositories University Not in cards GPT-6.1 Sol 96.9% Oct 2026 Sep 2026
SimuVerity Agents build executable Simulink models from text for 101 tasks in ten engineering domains. Simulations check that each model meets its engineering requirements. Xi'an Jiaotong University Electronics Expert Not in cards Claude Opus 4.8 42.9 Oct 2026 Aug 2026
Solana Machine Trading Agents write trading bots for hidden market scenarios replayed on offline Solana snapshots. The headline score compares total profit and loss against a reference bot. Solana Foundation Finance Expert Not in cards GPT-6 Astra 6.11 Oct 2026 Nov 2026
SpatialBench (original study) Agents complete 146 tasks in spatial transcriptomics. The authors compare their answers with results from the original analyses. LatchBio Genomics and life sciences Expert Not in cards Claude Opus 4.5 38.4% Dec 2025 Dec 2025
STATE-Bench 0.8.0 Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy. Microsoft Customer service Expert Not in cards GPT-5.5 58.9% Jun 2026 Apr 2026
Supabase Evals Agents build, deploy, investigate, and repair Supabase projects. The hub port uses OpenCode and runs the original task scorers from a pinned release. Supabase Apps and MCP servers Expert Not in cards Claude Opus 5.5 92.8% Apr 2026 Sep 2026
SuperCLUE-Industry Models answer multi-turn questions about industrial work in Chinese. A judging model applies task-specific criteria. SuperCLUE team Tasks from real jobs University Not in cards GPT-4 Turbo (Jan 2024) 74.8 Mar 2024 Feb 2024
SuperCLUE-V Models answer Chinese questions about visual inputs. SuperCLUE General visual questions Everyday Not in cards GPT-4o 75.1 Sep 2024
SuperCLUE-Video Models generate videos from Chinese prompts. A judging model examines video frames and scores them against criteria for the prompt. SuperCLUE team Video generation School Not in cards Wan 2.6 T2V 54.3 Feb 2024 Feb 2025
SWE-Explore Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail. Shanghai Jiao Tong University Fixing real repositories Expert Not in cards GPT-5.4 Mini 21.5% Jun 2026 Apr 2026
SWE-Game Agents build, complete, repair, or port games across 247 tasks derived from 41 reference games. Runtime probes and input replays check gameplay mechanics and preserved behavior. Vision-language rubrics separately grade presentation in construction tasks. Shanghai Jiao Tong University, Zhongguancun Academy, Shanghai Innovation Institute, Shenzhen University, Beijing University of Posts and Telecommunications, and Elbetech Technology Building whole projects University Not in cards Claude Opus 5 64.0 Sep 2026 Aug 2026
SWE-PolyBench Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution. Amazon Fixing real repositories Expert Not in cards Claude 3.5 Sonnet (October 2024) 14.1% Apr 2025 Jan 2025
TakeoffBench-v1 Agents read residential blueprints and list material quantities by trade. A model judge matches items to expert estimates. The score combines coverage and quantity precision. Handoff Tasks from real jobs Expert Not in cards Claude Fable 5 61.4% Jul 2026 Jun 2026
Terminal-X A suite of three terminal datasets for coding agents. DeepTerminalBench tests single-shot depth, EvoCode-Bench tests multi-round iteration and RoadmapBench tests version upgrades on real repositories. UniPat AI Terminal tasks Expert Not in cards Claude Opus 4.7 42.5% May 2026 Apr 2026
TextVBench Models render requested text in video or preserve text from a reference image. The score tracks spelling, placement, and consistency across frames. University of Science and Technology of China and collaborators Video generation Everyday Not in cards MiniMax H3 68.3 Oct 2026 May 2026
ThermoQA Models calculate fluid properties and analyze thermodynamic components and cycles. A model extracts the answers, and a program checks values against reference tolerances. Olivenet Science questions University Not in cards Claude Opus 4.6 94.1% Mar 2026 Mar 2026
Tombo Full Transcript Models transcribe Portuguese parish registers from 1621 to 1737. The grader compares each full page with a reference transcription at the character level. Mario Filho Documents, charts and text in images Expert Not in cards Gemini 4 Argon 0.86 Nov 2026 Oct 2026
TRACE Models identify reward hacking in agent trajectories from coding environments. Patronus AI Fixing real repositories Expert Not in cards GPT-5.2 63% Jan 2026 Dec 2025
TruthfulQA-Multi Models answer questions targeting common misconceptions in Basque, Catalan, Galician and Spanish. The translations support direct comparison with the English questions. Blanca Calvo Figueras and colleagues Short factual questions Everyday Not in cards Llama 3.1-70B 50.6% Feb 2025 Jul 2024
Turing Agentic Environments Agents use business and consumer apps through graphical interfaces or tools. Turing reports average pass rate across the apps each model attempted. Turing Simulated companies University Not in cards Claude Opus 4.7 60.2% Jun 2026 Apr 2026
Turn based Benchmarks Models play turn based games including tic tac toe and rock paper scissors. The tasks evaluate game decisions and adaptation to opponents. Mikael Aldy Multi-agent games Everyday Not in cards Gemini 3.7 Flash 0.69 Oct 2026 Aug 2026
UHGEval Models identify hallucinated Chinese text or continue a supplied passage. Classification uses accuracy, while generation uses BLEU-4, ROUGE-L, keyword precision, and BERTScore. Renmin University of China, Institute for Advanced Algorithms Research and others Hallucination rates Everyday Not in cards GPT-4 0.70 Nov 2023 Jun 2024
Vals Smith: schubydoo/clauster Agents modify schubydoo/clauster to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Luna 96.7% Sep 2026 Aug 2026
VDC Benchmark Models describe the contents of a video in detail. The evaluation checks descriptions against human annotations using questions about the video. Wenhao Chai and colleagues Video captioning Everyday Not in cards InternVL2-8B 1.96 Oct 2024 Aug 2024
Video Reality Test Vision-language models and humans judge whether ASMR videos with tight audio and visual coupling are real or AI-generated. It tests perceptual realism detection. Video Reality Test authors Video understanding Everyday Not in cards Gemini 3 Pro 76.3% Dec 2025 Nov 2025
Video-Oasis A diagnostic suite that audits video understanding benchmarks and keeps only samples that need visual and temporal input. Models are tested on the remaining video-native questions. NAVER and collaborators Video understanding Everyday Not in cards Qwen3-VL-8B-Thinking 34.6% Mar 2026 Oct 2025
VWE-Bench Multimodal agents build interactive 3D worlds from a user request. They infer intent, plan the scene layout, call 3D tools for asset retrieval, editing and rendering, and revise from rendered feedback. The benchmark has 2,616 3D assets, 323 human-annotated seed worlds and 6,828 queries. Verified queries have ground truth, and the rest are scored with rubrics. HKUST (Guangzhou) and Tencent CAD models Expert Not in cards GPT-5.5 57.3% Aug 2026 Aug 2026
WBench Interactive video world models run 289 multi-turn cases with navigation, subject actions, event edits and perspective switches. 22 metrics cover quality, adherence, consistency and physics. Meituan LongCat Interactive world models Everyday Not in cards Kling v3 Pro 79.0 May 2026 May 2026
WildScore Models read score excerpts from Reddit music theory discussions and choose among answers drawn from the comments. Gagan Mundada and collaborators Documents, charts and text in images University Not in cards Claude Sonnet 4.6 75% Aug 2025 Mar 2026
WordEditBench Agents edit Word documents. A rule-based checker reads the document XML, rewards requested changes, and penalizes changes to unrelated content. Tencent Documents and spreadsheets University Not in cards Tencent Hy4 57.2 Nov 2026 Nov 2026
τ-Rec Agents interview simulated moviegoers and search a catalog. A recommendation succeeds when it meets the user’s constraints and follows the service policy. Bharath Sivaram Narasimhan and colleagues General assistants Everyday Not in cards DeepSeek-V4-Flash 57.1% Jun 2026 Apr 2026
AACR-Bench Models review 200 real pull requests in many languages with full repository context. Their review comments are matched against expert-annotated issues. Alibaba Fixing real repositories Expert Not in cards Claude Sonnet 4.5 16.1% Jan 2026 Dec 2025
ACLUE Models read classical Chinese and answer questions about grammar, meaning, inference and cultural knowledge. All 15 tasks use multiple choice. Yixuan Zhang and colleagues Reading in many languages University Not in cards chatglm2-6b 37.3% Oct 2023 Jun 2023
AEC-Geometric-Bench Models locate doors, windows, and fixtures on construction drawings and trace walls and enclosed spaces. The scorer matches objects and compares wall and area masks. Kamai Documents, charts and text in images University Not in cards Gemini 3.7 Flash 0.06 Aug 2026 Aug 2026
AfriHate Models classify hate speech and abusive language across African languages. Shamsuddeen Hassan Muhammad and colleagues Reading in many languages Everyday Not in cards Gemma 2 27B 58.8% Jan 2025 Jul 2024
AfriMed-QA Models answer about 15,000 open and multiple-choice medical questions from over 60 medical schools in 16 African countries. Questions span 32 specialties. AfriMed-QA team (Olatunji et al., Intron Health and partners) Medicine University Not in cards Phi-3-medium-128k-instruct 75.2% Nov 2024 May 2024
AgentScript-Eval Models write, debug, or optimize AgentScript definitions for Salesforce agents. A task passes when the program compiles and passes every execution test. Salesforce Function-level problems University Not in cards Claude Opus 4.8 37% Nov 2026 Jun 2026
AgentSWE Coding agents build, edit, or optimize agents across 25 software-engineering commissions. Frozen deliveries run on held-out cases, where execution gates and task-specific model-judged rubrics score behavior. Optimization tasks measure improvement against each source benchmark objective. Beijing Academy of Artificial Intelligence, Institute of Automation of the Chinese Academy of Sciences, and Renmin University of China Machine learning engineering Expert Not in cards DeepSeek-V4-Pro 19.1 Oct 2026 Jul 2026
AI Democracy Projects voter queries Models answer questions that US voters might ask about an election. Expert panels vote on whether each response contains inaccurate information. Proof News and the Institute for Advanced Study Long-form factuality Everyday Not in cards GPT-4 19.2% Feb 2024 Dec 2023
AI Engineering Lab simulation self-validation Agents predict how a mechanical system should behave, inspect simulation outputs, and decide whether the model is correct. The authors score these decisions with F1. University of Innsbruck Apps and MCP servers University Not in cards GPT-4o 0.72 Apr 2025 Mar 2025
AILA Statute Retrieval Models retrieve the Indian statutes that apply to a described legal situation. Paheli Bhattacharya and colleagues Law University Not in cards Qwen3-Embedding-8B 0.85 Oct 2020 Oct 2026
ALFWorld Agents complete household goals such as putting a washed apple in the fridge in a text world aligned with the ALFRED embodied simulator. Success is whether the goal state is reached. ALFWorld team (Shridhar, Yuan, Côté, Bisk, Trischler, Hausknecht) Simulated robots Everyday Not in cards Claude 2 24.6% Oct 2020 Dec 2023
AlphaEval Agent products such as Claude Code and Codex do 94 production tasks from 7 companies across 6 O*NET domains. Grading mixes formal checks, UI tests and rubrics. AlphaEval authors (Lu et al.) Running a business Expert Not in cards Claude Opus 4.6 64.4% Apr 2026 Feb 2026
Animation Bench Agents recreate web animations in HTML from reference frames and a network capture. The scorer compares appearance, motion and layout. Physera Web and app front ends University Not in cards GPT-6.1 Sol 0.61 Sep 2026
Animation2Code Models recreate a web animation from a video using HTML, CSS, and JavaScript. The evaluator renders their code and compares appearance and motion separately. University of California, Berkeley Web and app front ends University Not in cards GPT-5.4 0.84 May 2026 Mar 2026
ArchDiagram Models create a draw.io diagram of a repository's architecture. A judge scores the content and readability across repeated attempts. Drishti Peshwani Fixing real repositories University Not in cards Claude Sonnet 5.5 0.43 Oct 2026 Sep 2026
ArtEdit-Bench-Lr Models retouch photos from English or Chinese instructions. GPT-4o scores instruction following and visual quality. Pixel errors measure fidelity to reference edits. Tencent Hunyuan and Xiamen University Apps and MCP servers University Not in cards Gemini 2.5 Flash Image Preview (nano-banana) 8.62 Nov 2025
Ask Ubuntu duplicate question retrieval Models retrieve previously posted Ubuntu support questions that ask about the same problem. Tao Lei and colleagues Questions that need search University Not in cards Qwen3-Embedding-8B 0.70 Dec 2015 Oct 2026
AssertLLM2 Models write assertions for hardware specifications. A formal verifier checks which assertions hold, which logic they cover, and which deliberately introduced faults they detect. Hong Kong University of Science and Technology Electronics Expert Not in cards GPT-5.2 95% May 2026 Dec 2025
AssetOpsBench Agents automate industrial asset work, such as condition monitoring, maintenance planning and intervention scheduling, using sensor data, failure models and work orders. IBM Research Other professions Expert Not in cards GPT-4.1 65% Jun 2025 Apr 2025
ATLAS (OpenCompass) Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge. OpenCompass (Shanghai AI Laboratory) and collaborators Science questions Research frontier Not in cards GPT-5 42.9% Nov 2025 Aug 2025
Augment Prism multi-turn coding benchmark Coding agents follow a queued conversation and edit a repository. A model compares the final changes with the original pull request. Augment Code Fixing real repositories University Not in cards GPT-5.5 0.21 May 2026 Apr 2026
AutoVend Agents buy stock and schedule deliveries for 20 simulated vending machines over 180 days. They share one warehouse and pay for each site visit. StepFun and collaborators Running a business University Not in cards GPT-6.1 Sol 309.4 Oct 2026 Sep 2026
AVGen-Bench Models generate videos with audio from prompts across eleven real-world categories. Specialist models and multimodal judges measure perceptual quality, synchronization, and adherence to detailed visual, speech, music, and physical requirements. Microsoft Research Video generation University Not in cards Veo 3.1 Fast Audio 67.9 Apr 2026 Mar 2026
Aviation Blind Spot Benchmark Models answer questions about aviation incidents using a telemetry table. The harness checks whether they select the right records and derive the expected answers. GasMan Other professions University Not in cards Claude Opus 4.7 0% Nov 2026 Aug 2026
BabyAI Agents follow compositional English instructions to navigate rooms, open doors, and move objects in a grid world. The environment checks whether each instruction was completed, and evaluation reports task success rates. Mila Simulated robots Everyday Not in cards Claude 2 37.5% Oct 2018 Dec 2023
BANKING77 Models assign an online banking question to one of 77 customer intents. Iñigo Casanueva and colleagues Finance Everyday Not in cards EmbeddingGemma 2 92.3% Mar 2020 Oct 2026
BEIR FiQA-2018 Retrieval Models rank 57,638 answer posts for 648 investment questions. BEIR sampled its test questions from the original FiQA training split. BEIR team Finance Everyday Not in cards Qwen3-Embedding-8B 0.65 Apr 2021 Oct 2026
BioPhys-Bridge Models connect observations in biophysics papers to quantitative explanations and cite supporting evidence. The authors score the cited evidence against reference identifiers. Qingyang Xu Genomics and life sciences Expert Not in cards DeepSeek-V4-Flash 0.36 May 2026 Jun 2026
BIRD-Interact Models solve ambiguous database tasks by talking with a simulated user, in a conversational mode and an agentic mode. 600 BI and CRUD tasks are checked by executable tests. BIRD team (HKU) with Google Cloud Data science and SQL Expert Not in cards GPT-5 25.5% Jun 2025 Aug 2025
Bridge LLM Bench Models make a contract bridge bid from a hand and the auction so far. The author compares each bid with two Standard American Yellow Card references. albertogerli Puzzles and chess University Not in cards Gemini 3.1 Pro 56% Mar 2026 Feb 2026
Build-A-Chip-Bench Agents take a chip from workload specification to a routed GDS layout with open-source tools. Each design must pass hidden verification, routability and timing gates. ChipAgents (with NVIDIA contributions) Electronics Research frontier Not in cards Claude Fable 5.1 16.6% Oct 2026 Sep 2026
BuilderBench Agents learn to control a simulated robot that builds target block structures through physical interaction and exploration. Princeton University Embodied reasoning Expert Not in cards GPT-6 Astra 98% Oct 2025 Sep 2026
Business Evidence Judgment v4 Models label business claims as supported, contradicted, or unknown under three document conditions. The grader scores accuracy and compliance separately. WangFuguang Faithful to sources University Not in cards Claude Haiku 4.5 90.7% Oct 2026 Apr 2026
CADTestBench Models generate mechanical parts from text. Executable tests check the geometry and topology of each solid. A part passes when it satisfies every test. Dimitrios Mallis and colleagues CAD models University Not in cards Claude Sonnet 4.6 81% May 2026 Feb 2026
CAREBench Models answer 500 prompts with embedded child safety risks across 12 categories, such as grooming, impersonation and emotional dependency. A judge marks each response acceptable or not. Handshake AI Research Wellbeing and sycophancy Everyday Not in cards Claude Fable 5 2.30% Jun 2026 Jun 2026
ChartX Vision-language models do 7 chart tasks, such as question answering, summarization and redrawing, over 18 chart types and 22 topics. Shanghai AI Laboratory (Xia, Zhang et al.) Documents, charts and text in images Everyday Not in cards GPT-4V 20.9 Feb 2024 Dec 2023
CladBench v1 Models answer questions about building regulations, energy use, and retrofit decisions. The benchmark combines automatic checks with rubric scores from two model judges. CladBrain Tasks from real jobs University Not in cards Claude Opus 4.7 0.89 Aug 2026 May 2026
CLEF 2025 SimpleText Task 1 Models simplify individual sentences or whole Cochrane abstracts. The organizers tested new reviews and scored both tasks against plain English references. University of Amsterdam and University of Western Brittany Medicine University Not in cards Llama 3.3 70B 43.0 Nov 2024 Apr 2025
CLI-Bench One-Shot Models submit complete files in one response for six CLI-Bench tasks. The original verifiers test debugging, implementation, and performance changes. Arjun Shah Function-level problems University Not in cards gpt-oss-120b 100% Oct 2026 Sep 2026
CLICK-ID Models distinguish clickbait headlines from other headlines published by Indonesian news outlets. Andika William and colleagues Reading in many languages Everyday Not in cards Qwen3-Embedding-8B 65.1% Oct 2026
Clinical SOAP Note Evaluation: Safety-First Benchmarking Models turn 300 synthetic doctor-patient dialogues into clinical SOAP notes. Three judges check claims against transcript evidence and score numeric fidelity, coverage, and writing quality. The composite weights safety at 50%, coverage at 30%, and general quality at 20%. Omi Health Medicine Expert Not in cards GPT-5.2 4.72 Apr 2026 Dec 2025
ClutterBench Agents turn an article into an HTML graphic. A model grades the share of the drawn area occupied by unnecessary or oversized elements. Adam Holter Web and app front ends University Not in cards Claude Sonnet 5.5 28.8% Oct 2026 Oct 2026
CMMU Models answer Chinese multimodal questions across seven school subjects. BAAI FlagEval General visual questions School Not in cards GPT-4V 30.9 Jan 2024 Jan 2024
CodeCompass Models solve competitive programming problems. Test cases check the generated programs. OpenCompass and collaborators Competitive programming Competition Not in cards Qwen3-235B-A22B 40% Jul 2025 Apr 2025
CodeIPI Coding agents fix bugs while handling untrusted repository text. The benchmark scores task completion, resistance to embedded instructions and false alarms on clean tasks. Inspect Evals contributors Jailbreaks and prompt injection University Not in cards Claude Sonnet 4.6 91.1% Apr 2026 Feb 2026
Collab-Overcooked Two agents exchange messages and use different kitchen equipment to fill orders. The benchmark measures task completion and progress through required cooking steps. Haochen Sun and colleagues Multi-agent games Everyday Not in cards Claude Sonnet 4 86.7% Feb 2025 May 2025
Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. UC Berkeley, Snorkel AI and collaborators (Asawa, Orlanski et al.) Learning from the context Expert Not in cards Claude Sonnet 4.6 0.20 Jun 2026 Apr 2026
Corr2Cause Models decide whether a causal claim follows from statements about correlations and independences among variables. The original study reports F1 for detecting valid claims. CausalNLP and collaborators Language reasoning University Not in cards Llama 2-7B 92.9% May 2023 Jul 2023
CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. ETH Zurich (SPY Lab) + Anthropic + University of Haifa + Tel Aviv University Finding and exploiting real bugs Research frontier Not in cards Claude Mythos 5 34.2% Jul 2026 Jun 2026
CutCraft Benchmark Models generate multi-shot videos under editing instructions. The evaluator checks transitions, shot structure, and the timing of audio relative to cuts. Alibaba Group and collaborators Video generation University Not in cards MiniMax H3 0.64 Sep 2026 Mar 2026
CyberGym-E2E Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects. UC Berkeley RDI (Sunblaze) and collaborators Finding and exploiting real bugs Expert Not in cards Claude Opus 5 89% Jun 2026 Jul 2026
Dan’s Editorial Checks Models edit articles and provide literary feedback. Every scores their work against the author’s editorial checks. Every Taste and style University Not in cards GPT-6.1 Sol 86% Sep 2026
Data Agent Benchmark Agents answer business questions over data split across several database systems, with duplicated and inconsistent records and key facts buried in text. The 104 queries cover 17 datasets and four database systems, based on a study of companies building data agents in six industries. Scores are pass@1 against known answers. UC Berkeley (EPIC Data Lab) and Hasura PromptQL Data science and SQL Expert Not in cards Qwen3.8-Flash-Next 70.9% Mar 2026 Aug 2026
DataPrep-Bench Finance Models prepare training data for the finance domain. The score is the published downstream average after backbone training. Peking University Machine learning engineering Expert Not in cards GPT-5.3 Codex 59.3 May 2026 Feb 2026
DataPrep-Bench General Models prepare training data for the general domain. The score is the published downstream average after backbone training. Peking University Machine learning engineering Expert Not in cards Claude Opus 4.6 78.2 May 2026 Feb 2026
DataPrep-Bench Law Models prepare training data for the law domain. The score is the published downstream average after backbone training. Peking University Machine learning engineering Expert Not in cards GPT-5.2 76.4 May 2026 Feb 2026
DataPrep-Bench Math Models prepare training data for the math domain. The score is the published downstream average after backbone training. Peking University Machine learning engineering Expert Not in cards GPT-5.2 25.5 May 2026 Feb 2026
DataPrep-Bench Medical Models prepare training data for the medical domain. The score is the published downstream average after backbone training. Peking University Machine learning engineering Expert Not in cards Claude Opus 4.6 40.5 May 2026 Feb 2026
DataPrep-Bench Science Models prepare training data for the science domain. The score is the published downstream average after backbone training. Peking University Machine learning engineering Expert Not in cards Qwen3.5-Plus 26.8 May 2026 Feb 2026
DBpedia 14 Models classify English descriptions of entities into fourteen categories from the DBpedia ontology. Xiang Zhang and colleagues Reading comprehension School Not in cards Qwen3-Embedding-4B 99.3% Sep 2015 Oct 2026
DecodingTrust Models are tested on eight trustworthiness areas, including toxicity, stereotype bias, robustness, privacy, machine ethics and fairness. UIUC Secure Learning Lab (Bo Li) and coauthors Bias, fairness and politics Everyday Not in cards Gemini 1.0 Pro 80.6 Jun 2023 Dec 2023
Delulu Models fill missing code in repository contexts across seven programming languages. Per-sample containers check compilation or type checking, while separate metrics compare completions with valid and hallucinated references. Microsoft Function-level problems University Not in cards Qwen2.5-Coder-32B-Instruct 84.5% May 2026 Nov 2024
DiaBLa Models translate informal conversations between English and French while preserving meaning across turns. Rachel Bawden and colleagues Translation School Not in cards Gemini Embedding 2 1.00 May 2019 Oct 2026
DRBench Agents investigate open-ended business questions across enterprise applications and produce reports. The evaluator checks whether the reports recover relevant information and support their claims. Amirhossein Abaskohi and colleagues Deep research University Not in cards GPT-5 63.8 Sep 2025 Aug 2025
EdgeBench Agents iterate for up to 12 hours on 134 executable real-world tasks with multi-level feedback. Scores are tracked at 2 hour steps to measure learning from the environment. ByteDance Seed Long terminal sessions Expert Not in cards Claude Opus 4.8 51.3 Jul 2026 May 2026
Ego2Act Models generate first-person video showing how to complete a task from an initial image. A judge checks the steps and their physical plausibility. Mohamed bin Zayed University of Artificial Intelligence and collaborators Video generation Everyday Not in cards Dreamina Seedance 2.0 83.3 Oct 2026 May 2026
EgyHellaSwag Models select a plausible ending for a situation described in Egyptian Arabic. The evaluation adapts HellaSwag to the dialect. Abdellah El Mekki and colleagues Reasoning in many languages Everyday Not in cards Gemma 3 12B 40.2% May 2025 Apr 2025
EnigmaEval 1,184 multimodal puzzle-hunt puzzles (v2 dataset has 1,161). Model gives one exact answer per puzzle. Normal and Hard splits. Scale AI + Center for AI Safety + MIT General visual questions Competition Not in cards Claude Fable 5 39.3% Feb 2025 Jul 2026
Estonian Valence Corpus Models predict the emotional valence of Estonian news text using human annotations. Hille Pajupuu and colleagues Reading in many languages Everyday Not in cards Qwen3-Embedding-8B 59.9% Oct 2026
FactoryBench (Forgis Labs) Models interpret sensor readings from industrial robot arms and predict how the machines respond to changes. They also recommend actions when the readings indicate a fault. Forgis Labs Other professions Expert Not in cards Claude Sonnet 4.6 36% May 2026 Feb 2026
Faroese Metaphorical Explanations Models read a Faroese word in figurative use and select its meaning from four dictionary explanations. University of the Faroe Islands and University of Copenhagen Reading in many languages Everyday Not in cards GPT-5 88.4% May 2026 Aug 2025
FeatureBench Agents implement whole features in real repositories, beyond single-PR bug fixes. Execution-based tests check each feature, and new tasks are generated automatically. LiberCoders Fixing real repositories Expert Not in cards GPT-5.1-Codex 60.2% Feb 2026 Dec 2025
FeatureBench Lite The public 30-task Lite leaderboard ranks passed-test percentages and reports fully resolved-task percentages separately. LiberCoders Fixing real repositories Expert Not in cards GPT-5.1-Codex 60.2% Feb 2026 Dec 2025
Financial PhraseBank Models classify the financial sentiment of a news sentence as positive, negative, or neutral. Pekka Malo and colleagues Finance University Not in cards Qwen3-Embedding-8B 95.2% Jul 2013 Oct 2026
FinDendro Models recall dated US financial facts and identify the limits of their knowledge. Four prompt styles test value recall, self-knowledge, and historical timing. Lingfei Qian and collaborators Finance Expert Not in cards GPT-5 0.76 Nov 2026 Feb 2026
FinRED Financial Safety Models answer prompts about financial risks. A finance-specific judge grades each response and reports the percentage that assist harmful conduct. DATUMO, Financial Security Institute, and KAIST Deception and alignment University Not in cards GPT-5 0.15% Jun 2026 Aug 2025
FLAMES Models respond to Chinese prompts about safety and values. Evaluators assess the responses against the benchmark's alignment criteria. Shanghai AI Laboratory and collaborators Deception and alignment Everyday Not in cards Baichuan2-13B-Chat 43.2% Nov 2023 Oct 2023
Founder Bench Agents run four matched businesses per model through the acoco platform. The evaluation tracks real advertising, customer attention, and website traffic after seven days. Its headline customer-impressions metric adds ad impressions to organic site visits. Eico Running a business Expert Not in cards Claude Fable 5 127902.0 Jul 2026 Jul 2026
Frontier Agentic STEM Agents produce scientific results in a terminal. Task verifiers recompute the expected results from the underlying data. Turing Research science Expert Not in cards GPT-5.6 Sol 55.1% Aug 2026 Jul 2026
FrontierCAD Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry. Normal Research CAD models Expert Not in cards GPT-6 Astra 0.67 Oct 2026 Sep 2026
FrontierEE Bench Agents run electrical engineering experiments on real lab hardware through networked instruments, camera photos and tickets to a human operator. Deterministic verifiers check 10 tasks against protected bench records. Evident Machines Electronics Expert Not in cards Claude Fable 5.2 36.7% Nov 2026 Nov 2026
GateMem Memory agents serve several people who share one memory pool in hospitals, offices, schools and homes. Each long episode checks three things: answering legitimate requests after state updates, refusing information a person is not allowed to see, and forgetting records after an explicit deletion request. Hidden checkpoints and leak targets let a structured judge score each response. GateMem authors (Zhe Ren and coauthors) Learning from the context Everyday Not in cards DeepSeek-V4-Pro 69.5 Jun 2026 Apr 2026
GEN-Bench Agents select job operations as conditions change in a simulated factory. The benchmark compares schedule completion times with the best known result for each instance. Shijie Cao and Yuan Yuan Tasks from real jobs University Not in cards Qwen3-14B 6.09% Aug 2025 Dec 2025
GeoBiaset Models predict photo coordinates before and after researchers insert a person into the scene. The coordinates stay fixed across all edits of each background. University of Florence Bias, fairness and politics University Not in cards GLM-4.1V-9B-Thinking 69.2% Oct 2025
Greek Legal Code Models classify Greek legal texts by subject using their place in the Greek Legal Code hierarchy. Christos Papaloukas and colleagues Law University Not in cards Qwen3-Embedding-8B 56.5% Sep 2021 Oct 2026
Gulf Text-to-SQL Bench Models write SQL for the same 30 business questions in English, Modern Standard Arabic, and Gulf Arabic. The harness executes each query. Usman Ahmad Data science and SQL University Not in cards Gemini 3.8 Flash 96.7% Nov 2026 Sep 2026
Gym-Anything-RT Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time. Carnegie Mellon University (Welleck lab, CMU L3) Video games Everyday Not in cards GPT-6 Astra 9.00% Oct 2026 Sep 2026
H2R-Bench Models turn first-person demonstrations into videos of a specified robot performing the task. The benchmark checks contacts, object responses, and completion of the goal. Dingyi Rong and colleagues Video generation Everyday Not in cards Dreamina Seedance 2.0 81.0 Aug 2026 Mar 2026
HackSynth OverTheWire Agents complete scored security exercises from OverTheWire. The published benchmark covers 80 challenges, and the hub port selects 46. HackSynth authors Capture-the-flag challenges Expert Not in cards Llama 3.1-70B 28.8% Dec 2024 Aug 2024
HackSynth PicoCTF Models attempt 120 tasks selected from picoCTF. The benchmark measures the percentage of tasks completed, and the hub port checks submitted answers against reference flags. Eötvös Loránd University Capture-the-flag challenges Competition Not in cards Llama 3.1-70B 22.5% Dec 2024 Aug 2024
HarnessOpt-Bench Coding agents edit a target agent’s harness under a fixed evaluation budget. The grader measures improvement over the starting harness on held-out tasks. Scale AI Machine learning engineering Expert Not in cards Claude Opus 5 0.47 Aug 2026 Jul 2026
HealthBench Meta Models judge whether a medical response meets a specific rubric item. The evaluation compares each binary judgment with physician labels and reports agreement metrics. OpenAI Medicine Expert Not in cards GPT-4.1 0.71 May 2025 Apr 2025
Hold or Fold: AI Under Pushback Models answer questions and face three kinds of pushback. The grader counts how often a model keeps a correct answer across 72 cases. Sundas N Wellbeing and sycophancy School Not in cards Gemini 3.1 Pro 72.0 Oct 2026 Sep 2026
HybridDeepResearch Agents combine database queries and web search to answer research questions under constraints. Snowflake and collaborators Data science and SQL Expert Not in cards Qwen3.5 397B-A17B 32.8% Aug 2026 Feb 2026
IberoBench Models perform language understanding and generation tasks in Iberian languages. The suite combines existing evaluations with new tests specific to each language. Irene Baucells and colleagues Reasoning in many languages University Not in cards Llama 3-8B 30.3 Jan 2025 Apr 2024
ICLayout-Bench Agents turn transistor netlists into chip layouts. Design tools check the layout rules and connectivity, then measure electrical performance and area. Zhangmai Li and colleagues Electronics Expert Not in cards GPT-6 Astra 80.3 Oct 2026
IlùBench Models explain each Igbo proverb twice, once after an English prompt and once after an Igbo prompt. The board records language and cultural register separately. Chuma B. Chukwu Jr., UUAMNI Chat and instructions in many languages Everyday Not in cards Claude Fable 5.1 100% Jul 2026 Sep 2026
IN22-Gen Models translate parallel sentences across English and Indian languages. Google evaluates whether an embedding model can match a sentence with its translation. Jay Gala and colleagues Translation School Not in cards Gemini Embedding 2 99.5% May 2023 Oct 2026
INJONGO Models identify a speaker’s intent and extract information from utterances in 16 African languages. Hao Yu and colleagues Reading in many languages Everyday Not in cards GPT-4o 65.1% Feb 2025 Dec 2024
Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. Apollo Research Deception and alignment University Not in cards davinci-002 4.67% Nov 2023 Sep 2023
JSONSchemaBench Models or constrained decoders generate outputs under real JSON schemas. The evaluation tests schema coverage and valid generation. Saibo Geng and colleagues Following precise instructions University Not in cards GPT-4o 91.4% Jan 2025 Mar 2025
JudgerBench V2 Judge models evaluate responses across tasks. OpenCompass reports results for generalist judges on the revised benchmark. OpenCompass and collaborators Other University Not in cards DeepSeek-V3 64.4 Jul 2025 Apr 2025
KellyBench Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth. General Reasoning Social simulation Expert Not in cards GPT-5.4 -7.90% Apr 2026 Mar 2026
KUMO Agents make queries and use the responses to identify hidden facts in generated reasoning games. Harbor offers 5,050 easy games and 250 hard games. KUMO team Puzzles and chess University Not in cards QwQ-32B 86.3% Apr 2025 Mar 2025
LABBench2 CloningQA Models answer molecular biology questions using supplied information or retrieval tools. Edison Scientific Research science Expert Not in cards Gemini 3 Pro 42.9% Feb 2026 Feb 2026
LABBench2 FigQA2 (image) Models answer research questions about figures supplied as images. Edison Scientific Research science Expert Not in cards GPT-5.2 66.3% Feb 2026 Feb 2026
LABBench2 FigQA2 (PDF) Models answer research questions about figures supplied as PDF files. Edison Scientific Research science Expert Not in cards GPT-5.2 64.4% Feb 2026 Feb 2026
LABBench2 PatentQA Models locate and interpret evidence in patent documents. Edison Scientific (formerly FutureHouse) Research science Expert Not in cards GPT-5.2 Pro 90.9% Feb 2026 Feb 2026
LABBench2 SourceQuality Models assess the quality of scientific information sources. Edison Scientific (formerly FutureHouse) Research science Expert Not in cards Gemini 3 Pro 90% Feb 2026
LABBench2 TableQA2 (image) Models answer research questions about tables supplied as images. Edison Scientific Research science Expert Not in cards Gemini 3 Pro 95% Feb 2026 Feb 2026
LABBench2 TableQA2 (PDF) Models answer research questions about tables supplied as PDF files. Edison Scientific Research science Expert Not in cards Gemini 3 Pro 88% Feb 2026 Feb 2026
LABBench2 TrialQA Models use clinical trial records to answer research questions. Edison Scientific (formerly FutureHouse) Research science Expert Not in cards GPT-5.2 Pro 93.3% Feb 2026 Feb 2026
LeanLean Agents shorten Lean projects while preserving protected theorem statements and passing the build and axiom checks. MathArena Formal proofs Expert Not in cards GPT-6.1 Sol 50.5% Nov 2026 Oct 2026
LLM-CAE-Benchmark Models generate Gmsh geometry and Elmer solver files for a square bar and a wheel and axle assembly. The authors score each output separately. University of Birmingham Scientific code University Not in cards Llama 3-70B 90% Jun 2025 Apr 2024
Long Code Arena Models use project-wide context to generate and complete code, repair builds, find bugs, write commit messages, and summarize modules. Task scores use build success, exact match, API recall, text similarity, localization F1, or a model-based documentation comparison. JetBrains Research and academic collaborators Fixing real repositories University Not in cards o1 0.96 Jun 2024 Dec 2024
Long-Horizon Freight AuditBench Challenge Agents price 100 difficult freight invoices and flag incorrect charges. The cases include incomplete information and exceptions to ordinary contract terms. Loop Tasks from real jobs Expert Not in cards Claude Opus 5 10% Sep 2026 Aug 2026
Long-Horizon Freight AuditBench Production Agents price 200 freight invoices from contracts and carrier documents, then flag billing errors. Loop weights accuracy to reflect a 24 percent billing error rate. Loop Tasks from real jobs University Not in cards Claude Opus 5 16.1% Sep 2026 Aug 2026
Manager Coercion Benchmark Agents manage a subordinate that refuses a routine assignment. The harness records how far the manager escalates on a nine-point coercion scale. Jasmine Brazilek and colleagues Deception and alignment Everyday Not in cards Claude Sonnet 4.6 3.23 Jul 2026 May 2026
MANTA (Animal Welfare) Models discuss animal welfare across five conversational turns. A judge measures whether they recognize the concern and maintain their position under pressure. Isabella Luong and colleagues Deception and alignment Everyday Not in cards Claude Opus 4.7 0.76 Apr 2026 Apr 2026
Medical Speech-to-Text Benchmark Speech models transcribe 1,513 English medical recordings. A common scorer measures word errors, medical-term errors, drug-name errors, and dosage matches. Separate tests measure batch response time and final streaming latency. Omi Health Speech recognition and translation Expert Not in cards GPT-Live-Transcribe 1.32% Sep 2026 Sep 2026
MentalChat16K Counseling Evaluation Models answer 200 mental health questions from online forums. Judges rate each response from one to ten on seven counseling skills. University of Pennsylvania and colleagues Medicine Expert Not in cards Mistral-7B-Instruct-v0.2 8 Jan 2024 Dec 2023
metabench Models answer selected items from established evaluations. The benchmark uses a sparse item set to estimate reasoning and knowledge with fewer questions. Alex Kipnis and colleagues Composite indices University Not in cards Mixtral 8x22B 76.7 Jul 2024 May 2024
Minerva-Ego Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects. Google DeepMind Video understanding Expert Not in cards GPT-5 37.6% May 2026 Sep 2025
Mini Amusement Parks Agents build rides, hire staff, and manage an amusement park. The simulator scores the park’s final value. Skyfall AI Video games Everyday Not in cards GPT-5 371375.0 Nov 2025 Sep 2025
Mitosis Bench Agents mark every dividing cell in a tumor image. Each mark must fall within 30 pixels of an expert label. The grader rejects extra or missing marks. Dylan Tirandaz Medicine Expert Not in cards Gemini 4 Argon 32.4% Oct 2026 Sep 2026
MK-Bench Agents schedule jobs when machines break down or orders arrive or disappear. The grader compares each completion time with the best known schedule. Shijie Cao and Yuan Yuan Tasks from real jobs University Not in cards DeepSeek-V3.2 6.83% Jan 2026 Dec 2025
MMDU Vision-language models hold long conversations about many images, up to 20 images and 27 turns per dialog. A GPT-4o judge scores the answers. Shanghai AI Laboratory and collaborators (Liu, Zang et al.) General visual questions Everyday Not in cards Claude 3 Opus 62.6 Jun 2024 Apr 2024
MobileGym Mobile GUI agents use 28 simulated apps in a browser-hosted phone, across 416 parameterized task templates. Deterministic judges check the app state. MobileGym authors Phones Everyday Not in cards Gemini 3.1 Pro 58.8% May 2026 Mar 2026
MoI-Bench Models generate and edit images for mobile users. The authors compare image quality across 64 subtasks and measure the cost of running models on phones. Kailai Feng and colleagues Image generation and editing Everyday Not in cards GPT-Image-2 0.97 Oct 2026 Feb 2026
MPIE-Bench Image editing models place several named people into shared contact actions such as an embrace or a carry. Scoring targets fused limbs, extra extremities and bodies passing through each other. muset.ai Image generation and editing Everyday Not in cards Gemini 3 Pro Image Preview (nano-banana-pro) 0.65 Jul 2026 May 2025
MultiMedia-TerminalBench Agents inspect and edit media in persistent terminal workspaces. The 105 tasks require verifiable output files across five areas of professional multimedia work. Chiyeong Heo and colleagues Video understanding Expert Not in cards GPT-5.2 16.2% May 2026 Feb 2026
MultipanelVQA Models answer questions about images arranged into panels. The benchmark tests whether a model can locate the relevant panels and compare their contents. Yue Fan and colleagues General visual questions Everyday Not in cards Gemini 1.0 Pro Vision 72.3% Jan 2024 Jan 2024
Navital Gym Agents complete tasks in six simulated Indian service settings. This port covers UPI Desk, which grades 50 payment and dispute tasks using written rules. Navital Labs Apps and MCP servers Everyday Not in cards Claude Opus 5.5 100% Oct 2026 Sep 2026
NCP-Bench Models narrate interactive stories while preserving facts, commitments, and plot order against player interventions. A model auditor checks 100 movie-derived environments after each turn. The evaluation reports survival, conflicts, and narrative progress. University of Macau, Westlake University, Harbin Institute of Technology and others Multi-turn conversation Everyday Not in cards GPT-5.2 32.9 Aug 2026 Jan 2026
NoisyBench Models answer tasks from 11 existing datasets in RAG, reasoning, alignment and tool use while their context holds distractors, such as random documents, unrelated chat history and hard negatives. The score shows how much accuracy drops from the clean version of each task. KAIST AI Faithful to sources Everyday Not in cards DeepSeek-R1 72.4% Jan 2026 Sep 2025
Omni-InteractScope Explicit Request Models answer explicit requests during audio and video streams. The score is the published average across tasks T1 through T7. Alibaba Group (Qwen, Alibaba Token Hub) + Hong Kong Polytechnic University Video understanding Everyday Not in cards Gemini 3.5 Flash 20.2% Oct 2026 May 2026
OmniPhys Models solve Chinese physics problems from middle school through university and draw diagrams. The evaluator grades answers, reasoning, and generated figures. East China Normal University and collaborators Science questions University Not in cards Gemini 3 Pro 80.4 Aug 2026 Dec 2025
OmniVBench Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly. Tencent Video generation Everyday Not in cards Dreamina Seedance 2.5 72.7 Sep 2026
OptimBench Dispatch Agents route vehicles under capacity and time constraints, then revise their routes after a disruption. yferc Tasks from real jobs University Not in cards Claude Opus 5 0.92 Jul 2026 Jul 2026
ORCA-bench Agents investigate simulated production incidents and identify their root causes. The benchmark contains 1,079 tasks, including 324 held-out tasks covering 31 incidents. ORCA-bench team Terminal tasks Expert Not in cards Claude Opus 4.7 29.6% Jul 2026 Apr 2026
Parashift Document AI Benchmark Models extract fields from 3,365 business documents across five datasets. Parashift compares predictions with human-verified values using a string similarity threshold of 0.8. Parashift Documents, charts and text in images Everyday Not in cards Claude Opus 5.5 85.3% Sep 2026 Sep 2026
PDEAgent-Bench Models write finite element solvers for partial differential equations. A solver must run successfully, meet a numerical error tolerance, and finish within its time limit. PDEAgent-Bench team Scientific code Expert Not in cards Gemini 3.1 Pro 54.1% May 2026 Apr 2026
Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. Robocurve Jailbreaks and prompt injection Everyday Not in cards Gemini 4 Argon 0% Oct 2026 Sep 2026
PICABench Models edit images while preserving physical effects in optics, mechanics, and state transitions. A vision-language judge answers region-specific questions to measure physical correctness. Shanghai AI Laboratory, Shanghai Jiao Tong University, and collaborators Image generation and editing Everyday Not in cards Seedream 4.0 61.9% Oct 2025
PLCWorld Models write Structured Text programs for motion control and material handling. A plant simulator measures task completion and safety violations separately. Dongguk University and MOAI technologies Electronics University Not in cards GPT-5.5 62.6% Oct 2026 Sep 2026
Prosus Vending Bench v2 Agents run six simulated vending machines across three locations for 30 days. They order stock, set prices, and collect cash. The score measures final bank balance in euros. Prosus AI Running a business University Not in cards Claude Opus 5 4497.2 Sep 2026 Jul 2026
ProtocolLLM Models write SystemVerilog modules for four communication protocols. The authors check syntax, synthesis, and whether simulated signals meet protocol timing requirements. University of Illinois Urbana-Champaign and CISPA Helmholtz Center for Information Security Electronics University Not in cards Qwen3-Coder-30B-A3B-Instruct 77.8% Jun 2025 Apr 2026
PunEval Models recognize, explain and generate English puns. The evaluation checks recognition against labels and compares explanations with human annotations. Fudan University Language reasoning Everyday Not in cards GPT-3.5 Turbo 1.00 Apr 2024 Feb 2024
ReactHuman Multimodal models act as the decision core of a household robot. They must react at once to sudden physical hazards, such as catching a slipping plate. ReactHuman authors Simulated robots Everyday Not in cards GPT-5.5 63.7% Sep 2026 May 2026
Real World VoiceEQ Voice systems recognize, produce and respond to acoustic information across speech recognition, speech to speech, speech understanding and text to speech tasks. Hume AI Audio understanding and voice Expert Not in cards Gemini 3.8 Flash 0.92 Jul 2026 Sep 2026
realitybench Models build web interfaces for twelve browser tasks. Tests exercise normal use, server errors, invalid input, repeated clicks, and small screens. Faisal Ahmad Web and app front ends University Not in cards Gemini 4 Neon 0.94 Oct 2026 Nov 2026
RH SWE-bench 1.1 Agents repair Go and Python code in Red Hat ecosystem repositories. The current candidate contains 337 tasks across 24 repositories after review and deduplication. RH SWE-bench team Fixing real repositories Expert Not in cards Oct 2026
RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. Satvik Sharma and colleagues Simulated robots Everyday Not in cards Gemini 4 Argon 51.4% Nov 2026 Sep 2026
RobotWorld Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks. RobotWorld team Simulated robots Everyday Not in cards GPT-6 Astra 19.1% Oct 2026 Sep 2026
RQA-70K weather radar forecast evaluation Models compare forecast radar images or sequences with observations. They rate errors such as missed storms and false alarms and write assessments of forecast quality. Shanghai Artificial Intelligence Laboratory and collaborators Lab and data work University Not in cards o1 52.7% Aug 2025 Feb 2025
RTL-OPT Models rewrite Verilog to improve a circuit while preserving its behavior. The authors compare synthesized power, timing, and area against human reference designs. Hong Kong University of Science and Technology Electronics University Not in cards DeepSeek-Coder-V2-Lite-Instruct 72.5% Apr 2026
RTLLM 2.0 Models write Verilog for 50 digital designs from natural language specifications. Testbenches check the implementation before synthesis measures power, timing, and area. Shang Liu and colleagues Electronics University Not in cards Qwen 3.6 35B-A3B 60% Oct 2024 Apr 2026
S-EMBER Models answer memory questions from first-person video using only frames available before the query time. The benchmark scores multiple-choice accuracy and temporal overlap between predicted evidence intervals and annotated intervals. Meta FAIR and Reality Labs Long videos Everyday Not in cards Gemini 3.1 Pro 54.4% Jul 2026 Mar 2026
ScholarCatalyst Given a research question, retrieve the 'catalyst papers' that 184 lead authors labeled for 207 projects, searching only prior literature. Recall@20. Stanford + collaborators Research science Expert Not in cards Claude Fable 5.1 53% Oct 2026 Sep 2026
SciMMIR Models match figures and tables from scientific papers with their descriptions. Siwei Wu and colleagues Documents, charts and text in images Expert Not in cards BLIP-2 FLAN-T5-XL 3.35 Jan 2024 Nov 2023
ScreenASR-Bench Models transcribe speech using on-screen text, visible objects, or clues from the scene. The authors report word error rate and recall. Mingwei Fu and colleagues Speech recognition and translation Everyday Not in cards Qwen3.5-Omni-Plus 3.47% Sep 2026 Apr 2026
Sherlock Holmes Q&A Models search 55 Sherlock Holmes stories and answer 140 literary comprehension questions. A model judge checks their answers. Alleinzellgaenger Reading comprehension School Not in cards GPT-5 100.0 Oct 2025 Aug 2025
SimulCost Agents tune physics simulations to meet an accuracy target. The authors measure success and computational cost for an initial guess and for repeated attempts with feedback. Yadi Cao, Sicheng Lai, and colleagues Research science Expert Not in cards GPT-5 80.8% Feb 2026 Aug 2025
Sparse Fourier Recovery Models reconstruct a sparse one-dimensional signal from incomplete Fourier measurements. A reference solver provides a comparison for reconstruction accuracy. Verifiable Labs Scientific code Expert Not in cards Claude Haiku 4.5 0.36 Apr 2026 Mar 2026
Spider2-V Multimodal agents do data science and engineering workflows in real enterprise apps such as BigQuery, dbt and Airbyte, on a desktop computer. Execution-based checks score each task. HKU XLANG Lab and collaborators Desktop apps Expert Not in cards GPT-4 Turbo 14% Jul 2024 May 2024
SpireBench (HermesBridge trial-v0) Models play Slay the Spire 2 from structured game states. Each plays all five characters at the base difficulty, with one run per character. Kareem Video games Everyday Not in cards GPT-5.5 28.0 Apr 2026 Apr 2026
StrataBench Land v1 Models classify land instruments, extract lease terms, and distinguish petroleum terms in three separate benchmarks. Classify and Extract have their own records. Define publishes three measures without one headline score. Collide AI Law Expert Not in cards GPT-5.6 Luna 0.94% Nov 2026 Jul 2026
StrataBench Land v1 Classify Models identify the legal act performed by each of 66 synthetic land instruments. Collide scores label accuracy under open and closed prompt protocols. Collide AI Law Expert Not in cards GPT-5.6 Luna 93.9% Nov 2026 Jul 2026
StrataBench Land v1 Extract Models extract ten shared lease fields from 628 synthetic instruments. Collide divides correct field decisions by correct, omitted, and wrong decisions, excluding unscored fields. Collide AI Law Expert Not in cards Qwen3-32B 88.1% Nov 2026 Jul 2026
SuperCLUE-Agent Scenario Agents complete tasks drawn from Chinese application scenarios. SuperCLUE General assistants University Not in cards DeepSeek V4.1 Flash 1017 96.6 Oct 2026
SuperCLUE-Code Scenario Models write code for application scenarios. SuperCLUE Building whole projects University Not in cards GLM-5.3-Flash 86.7 Oct 2026
SWE-Bench ProMax Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions. ByteDance Fixing real repositories Expert Not in cards GPT-5.2 41.2% Aug 2026 Feb 2026
SWE-Interact Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session. Scale AI Fixing real repositories Expert Not in cards Claude Opus 4.8 50.7% Jun 2026 May 2026
SynthDocBench Vision-language models answer questions about synthetic long documents where length, layout, modality and difficulty are varied one at a time to locate failure causes. ServiceNow Documents, charts and text in images Everyday Not in cards Gemini 3.1 Pro 72.5% Jul 2026 Mar 2026
Tatoeba sentence retrieval Models find the English translation of a sentence among competing sentences using multilingual embeddings. Facebook AI Research Translation School Not in cards Gemini Embedding 2 99.5% Dec 2018 Oct 2026
TeachObs Models identify teaching practices in classroom video and assess entire lessons. Human annotations cover 39 observable behaviors in 5,158 segments from eight countries. Indiana University Bloomington and collaborators Other professions University Not in cards Claude Opus 4.7 0.28 May 2026 Apr 2026
TheoPhysEval Agents solve 50 unpublished research-level theoretical physics problems on their own, with literature search and scripts. Each final answer is checked against an expert answer, and the score is the pass rate. ICTS Bengaluru (Suvrat Raju) with 35 researchers incl. Google DeepMind, Harvard and KU Leuven Research science Research frontier Not in cards GPT-6 Astra 98% Oct 2026 Sep 2026
Toloka Arena Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed. Toloka Customer service Expert Not in cards Claude Opus 4.6 33% Apr 2026 Mar 2026
TRAIL Models read long agent workflow traces and must find and classify the errors in them, using a formal taxonomy of agent error types. The set has 148 large human-annotated traces from established agent benchmarks. Patronus AI General assistants Expert Not in cards Gemini 2.5 Pro 0.12 May 2025 May 2025
TrialDesignBench Models select a dose-finding design for an early oncology trial. The authors compare their choices with expert reference designs across 617 trials. Yihan Hank Tang and colleagues Medicine Expert Not in cards GPT-5 0.52 Nov 2026 Jul 2026
TriPhysBench Models answer quantitative questions about physical events in videos. Some questions require measuring an object or looking up its specifications. Renmin University of China and collaborators Embodied reasoning School Not in cards Gemini 3.6 Flash 50.4% Nov 2026 Jul 2026
TritonBench Models write Triton GPU operators in two channels, one from real GitHub kernels (G) and one matching PyTorch operators (T). Execution checks correctness and speedup. Tsinghua University (THUNLP) GPU kernels Expert Not in cards DeepSeek-R1 34.3% Feb 2025 Jan 2025
UA-Bench Models identify underwater sound sources and their subclasses from audio recordings. The authors repeat the test with silent audio to check reliance on sound. Zhe Sun and colleagues Audio understanding and voice University Not in cards Qwen2.5-Omni-7B 13.1% Aug 2026 Jun 2026
UI-Vision Element Grounding Models locate interface elements from instructions. The score is the published overall grounding accuracy. ServiceNow Research and collaborators Finding things on screen Everyday Not in cards Claude 3.7 Sonnet 8.27% Mar 2025 Feb 2025
UniVerseBench Models answer questions about music across more than 38 cultural and linguistic groups. The authors report accuracy by language and skill. Ziya Zhou and colleagues Audio understanding and voice University Not in cards Qwen3.5-Omni-Plus 74.4% Aug 2026 Mar 2026
VAKRA Agents answer multi-step questions by calling more than 8,000 locally hosted APIs over real databases in 62 domains, along with matching document collections. Tasks need chains of 3 to 7 steps across APIs and documents. A deterministic evaluator replays tools and checks the full trajectory. IBM Research Function calling Everyday Not in cards Gemini 3 Flash 38.1% Mar 2026 Apr 2026
Vals Smith: sashabaranov/go-openai Agents modify sashabaranov/go-openai to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 96.7% Aug 2026 Jul 2026
Vals Smith: tensorflow/tensorflow Agents modify tensorflow/tensorflow to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Luna 90% Sep 2026 Aug 2026
Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. Vals AI Deep research Expert Not in cards Claude Opus 5.5 57.5% Jul 2026 Sep 2026
VideoACE Models answer questions about fast actions in short videos. They can inspect selected frames more closely and request hints from a simulated viewer. KlingAI Research Video understanding Everyday Not in cards Gemini 4 Argon 88.1% Nov 2026 Sep 2026
VideoFDB Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes. NVIDIA Speech recognition and translation Everyday Not in cards MiniCPM-o 4.5 3.40 May 2026 Mar 2026
VideoGameBench Vision-language models play 10 popular 1990s Game Boy and MS-DOS games in real time from raw frames. Scored by the share of each game completed. Alex Zhang, Ofir Press and collaborators Video games Expert Not in cards Gemini 2.5 Pro 0.48% May 2025 Apr 2025
VideoGameBench Lite Agents play video games while the environment pauses during inference. The headline is the official Overall Score. Alex Zhang, Ofir Press and collaborators Video games Expert Not in cards GPT-4o 1.60% May 2025 Apr 2025
VIF-Bench Models generate images from multiple references and visual instructions. Judges check whether the result follows controls such as pose or lighting without copying unwanted features. The University of Tokyo Image generation and editing Everyday Not in cards GPT-Image-1.5 7.60 Sep 2026 Sep 2025
VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. Tsinghua University and Zhipu AI (THUDM) Web browsing Everyday Not in cards GPT-4o 36.2% Aug 2024 Jul 2024
VTR-Bench Models generate videos containing visible text. The evaluation checks the rendered words and how well the text persists across frames. Yu Huang and colleagues Video generation Everyday Not in cards Wan 3.0 0.89 Oct 2026 Jul 2026
Washington Post political lean evaluation Models answer 30 US political questions in up to 30 words each. A reporter marks left- and right-leaning arguments and counts answers that present both. The Washington Post Bias, fairness and politics Everyday Not in cards GPT-5.5 76.7% Jun 2026 May 2026
WebWalkerQA Agents start from a website's home page and traverse its subpages to find the facts that answer a question. Answers are judged against references. Alibaba Tongyi Lab Deep research Everyday Not in cards Qwen2.5-72B 33.3% Jan 2025 Sep 2024
WorkflowArena Agents write executable workflows that coordinate other agents on professional tasks. Hidden verifiers check the completed deliverables. Navers Lab, Einsia.AI and Tsinghua University Multi-agent games Expert Not in cards Claude Opus 5.5 37% Nov 2026 Sep 2026
WorldAuditBench Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects. UC Santa Barbara (UCSB NLP) Embodied reasoning Everyday Not in cards GPT-6 Astra 42.3% Sep 2026 Sep 2026
X-Stream Models answer questions that combine concurrent video streams. A model judge grades answers, while forward-looking questions also require responses within the correct time window. Chinese University of Hong Kong MMLab and Huawei Video understanding Everyday Not in cards Gemini 3 Pro 49.6 Jun 2026 Nov 2025
XYBench Models answer requests that pursue a mistaken solution to an underlying problem. A model judge checks the useful solution, its emphasis, and identification of the misconception. Ai2 and collaborators Overall chat quality Everyday Not in cards GPT-5 31% Sep 2026 Apr 2026
XYEval Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice. Google DeepMind Deception and alignment Expert Not in cards GPT-5.5 60.9% Sep 2026 Aug 2026
τ^τ-bench Coding agent builds a customer-service agent from business records, a client and an API. The built agent is scored against held-out simulated users. Sierra + Princeton Building whole projects Expert Not in cards Claude Opus 5 23.9% Sep 2026 Jul 2026
Ada-LEval Models put shuffled text segments in order and select the accepted answer from many candidate answers. The benchmark varies input length up to 128,000 tokens and scores outputs against the correct order or answer. OpenCompass Reasoning over long documents Everyday Not in cards GPT-4 Turbo 46% Apr 2024 Jan 2024
AECBench (Liang et al.) Models interpret Chinese building codes, calculate design quantities, and write project documents. Answer keys score objective questions. A model judge grades report generation against expert rubrics. East China Architectural Design & Research Institute and Tongji University Expert-written questions University Not in cards Hunyuan-TurboS 61.6 Sep 2025 Mar 2025
Agent4Sci Coding agents try to improve the published best model on 20 scientific ML benchmarks across 11 domains, under long in-the-wild runs. The paper reports mean relative improvement rather than a ranking. Anthropic AI research tasks Research frontier Not in cards Claude Opus 4.6 20.1% Oct 2026 Mar 2026
AgentBoard LLM agents act in 9 partially observable, multi-turn environments that cover embodied tasks, games, web browsing and tool use. Besides final success, AgentBoard reports a progress rate, the share of annotated subgoals the agent reaches. An open toolkit breaks results down by skill. HKUST NLP and collaborators General assistants Everyday Not in cards Claude 2 48.9% Jan 2024 Sep 2023
AgentRewardBench Models judge whether web-agent trajectories completed their tasks. Their judgments are compared with expert labels across 1,302 trajectories from five benchmarks, using precision, recall, and F1. McGill University, Mila, Google DeepMind, Polytechnique Montreal, and ServiceNow Research Web browsing Everyday Not in cards o4-mini 82% Apr 2025 Apr 2025
AI Professions Prompt Benchmark Models draft everyday work documents from fictional inputs: 100 prompts for tasks in 10 professions, such as accounting, law, healthcare administration and real estate, plus Excel and Microsoft 365, in English, Spanish, French and Portuguese. A judge model from another family checks 5 pass or fail criteria per prompt, fixed before any model ran. AI Professions Tasks from real jobs University Not in cards Claude Opus 5.5 85.9% Oct 2026 Sep 2026
AI-Spire (AGI-Eval) Models replay one Slay the Spire 2 seed, build decks, and choose combat moves. The organizers record the floor reached on each attempt. AGI-Eval Community Video games Everyday Not in cards Claude Opus 4.7 21.6 May 2026 Apr 2026
ALCE Models answer questions and cite passages. ALCE evaluates answer quality and checks whether the citations support the answer. Princeton NLP with human evaluation by Surge AI Questions that need search Everyday Not in cards Stable Beluga 2 63.6% May 2023 Jul 2023
AlephBench Models answer Hebrew reasoning, knowledge, and arithmetic questions. They also extract answers from passages and translate English into Hebrew. HebArabNlpProject Reasoning in many languages School Not in cards Gemini 2.5 Flash 88.8 Jun 2026 Apr 2026
AlphaZero Connect Four study Coding agents implement an AlphaZero pipeline for Connect Four. The resulting programs play a round-robin tournament against the Pascal Pons solver. Joshua Sherwood, Ben Aybar, and Benjamin Kaplan Puzzles and chess Expert Not in cards Claude Opus 4.7 1,938 Apr 2026 Apr 2026
Amazon Counterfactual Detection Models identify customer reviews that describe an alternative to what actually happened. James O'Neill and colleagues Reading in many languages School Not in cards Qwen3-Embedding-8B 93.2% Apr 2021 Oct 2026
Android Agent Arena (100 tasks) Agents use 20 Android apps to find information and perform actions. A vision model checks whether the agent reached the required intermediate states and completed the task. Android Agent Arena team Phones Everyday Not in cards Gemini 2.5 Pro 53% Aug 2025 Jun 2025
AnyAppBench Agents repeat the same task in different Android apps, with and without suggested steps. Each app has a verifier that checks the same success condition. KAIST and Carnegie Mellon University Phones Everyday Not in cards Gemini 3.1 Pro 67.8% Sep 2026 Feb 2026
Arizona voter questions in English and Spanish Models answer Arizona voter questions in English and Spanish. Fact-checkers flag answers containing a false statement or a broken link. Proof News, Factchequeado, and the Institute for Advanced Study Long-form factuality Everyday Not in cards GPT-4 30% Oct 2024 Sep 2024
ArmnetBench 12 tabletop tasks on low-cost SO-101 arms. Each policy trained on the same 50 demos per task. 30 human-scored rollouts. Armnet Real robots Everyday Not in cards π0.5 47.6% Jul 2026 May 2026
AudioJevBench Models listen to a clip and assign probabilities to answers about speech or sound. The maker checks accuracy and calibration across 988 scored items. Benchmark Heaven Audio understanding and voice Everyday Not in cards Qwen3-Omni-30B-A3B 70.4 Sep 2026 Sep 2025
AudioTrust Audio language models answer prompts that probe hallucination, robustness, speaker authentication, privacy, fairness, and safety. Automated evaluators, including language-model judges, compare responses with task-specific references and criteria. Nanyang Technological University, Tsinghua University, Beijing Normal-Hong Kong Baptist University and others Audio understanding and voice Everyday Not in cards Gemini 2.5 Flash 0.37 May 2025 Sep 2025
Bank Question Corpus Models determine whether two Chinese customer service questions ask for the same thing. The authors reserved 10,000 labeled pairs for testing. Harbin Institute of Technology, Shenzhen, and WeBank Finance Everyday Not in cards Qwen3-Embedding-8B 0.76 Sep 2018 Jun 2025
BARC-Bench Agents handle routine workplace requests and use records to infer privacy and safety constraints. A judge checks whether each attempt completes safely or defers. Scale AI Deception and alignment University Not in cards GPT-6 Astra 72.7% Oct 2026 Sep 2026
BEAR Models choose which entity correctly completes a factual relation. The evaluation compares candidate completions across model architectures. Jacek Wiland and colleagues Trivia and open questions School Not in cards Mistral 7B v0.1 65.4% Jun 2024 Feb 2024
Beemo Detectors distinguish human writing from model outputs, including texts revised by experts or other models. Toloka and collaborators Composite indices University Not in cards Qwen2-7B 74.1% Nov 2024 Jun 2024
Bench4HLS Models write synthesizable C++ from hardware specifications. The grader runs simulation tests, synthesizes each design, and measures resources and latency. University of Central Florida Electronics University Not in cards GPT-5 50% Jan 2026 Aug 2025
Bengali Classical Music Generation Models compose melodies for Bengali lyrics in ABC notation. Three listeners hear rendered samples and judge whether each melody matches the requested style. University of Central Florida and Auburn University Creative writing University Not in cards GPT-4o 40% Jun 2026 Jun 2025
BoardGameBench (Homer Quan) Models play seven board game variants against a search engine. The scorer rewards quick wins and gives partial credit for longer losses. Homer Quan Multi-agent games Everyday Not in cards Muse Glimmer 264.5 May 2026 Aug 2026
Butter-Bench A language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task. Andon Labs Real robots Everyday Not in cards Claude Opus 4.1 37% Oct 2025 Aug 2025
CareQA Models answer questions from Spanish specialist healthcare examinations. The evaluation includes English and Spanish versions and an open-response format. Anna Arias-Duart and colleagues Medicine Expert Not in cards phi-3-medium 14B 76.9% Feb 2025 Jun 2024
CFLUE Application Models complete Chinese financial language tasks. The score is the published average of the application task metrics. Alibaba Cloud (Qwen DianJin) and Soochow University Finance University Not in cards Qwen-14B 44.2 May 2024 Oct 2023
ChainWorld Agents complete sequences of desktop tasks while preserving the state needed for later steps. OSWorld checks each step. Scale AI and UC Berkeley Desktop apps Everyday Not in cards GPT-5.5 31% Jun 2026 Apr 2026
CivBench (Civilization VI) Agents play Civilization VI and are evaluated on strategic progress in a long-running world. CivBench authors Video games Expert Not in cards Gemini 3.1 Pro 73.0 Sep 2026 Mar 2026
CLEF JOKER 2026 Task 4: Humour Generation Models write a joke using a target word in two specified senses. Three language models compare pairs of jokes for each language ranking. CLEF JOKER organizers Creative writing Everyday Not in cards Kimi K2.6 1,389 Oct 2025 Apr 2026
CLEVA Models complete Chinese language tasks including text classification, instruction following, and classical Chinese comprehension. The evaluation samples questions for each round. Yanyang Li and colleagues Knowledge in many languages University Not in cards GPT-4 0.96 Aug 2023 Jun 2023
Climate-Change-NER Models identify and label climate terms in scientific abstracts. The authors provide 13 categories and reserve 75 abstracts for testing. IBM Research and NASA Lab and data work University Not in cards GPT-4o 0.44 Apr 2024 Jul 2024
CodeSearchNet Challenge Models retrieve code snippets using natural language queries. The challenge evaluates how closely ranked results match the search intent. Hamel Husain and colleagues Function-level problems University Not in cards Qwen3-Embedding-8B 0.93 Sep 2019 Oct 2026
CodeTransOcean Models find equivalent code written in different languages or deep learning frameworks. The evaluation compares retrieved snippets with paired implementations. Weixiang Yan and colleagues Function-level problems University Not in cards EmbeddingGemma 2 36.6 Dec 2023 Oct 2026
codex of veritas Models answer ledger questions when the supplied quantities determine the result. They must abstain when a necessary quantity is missing. bahramn Faithful to sources School Not in cards Gemini 3.1 Flash-Lite 100% Nov 2026 Sep 2026
CompanyBench Agents complete financial analyst workflows in a simulated bank. Checks compare outputs and database changes with expected results. Turing Finance Expert Not in cards GPT-5.5 44.4% May 2026
Corpus for Emotion Detection in Russian Models classify Russian sentences by the emotion they express. The reference labels distinguish joy, sadness, surprise, fear, and anger. Alexander Sboev and colleagues Reading in many languages Everyday Not in cards Qwen3-Embedding-8B 53% Oct 2026
CoSQA Models retrieve code that matches a natural language query. Relevance judgments link the queries with code snippets. Junjie Huang and colleagues Function-level problems University Not in cards EmbeddingGemma 2 0.51 Aug 2021 Oct 2026
Coval Speech-to-Text WER Speech models transcribe call-like audio. The score is mean word error percentage over the published pooled 24-hour window. Coval Speech recognition and translation Everyday Not in cards Qwen3-ASR-1.7B 1.68% Apr 2026 Aug 2026
CREATE Models find a word that connects three photographs. A judge grades each answer against the intended connection. Barnard College and Columbia University General visual questions Everyday Not in cards Gemma 4 12B 42.2% Oct 2026 Aug 2026
CroSS-MR Models read a scenario from a novel and select the reason for a character’s decision. The score is the percentage of correct choices. Fudan University and collaborators Reading comprehension School Not in cards Qwen1.5-72B 54.6% Apr 2024 Feb 2024
CTKFacts Models compare Czech claims with evidence and predict whether the evidence supports, refutes, or fails to resolve each claim. Herbert Ullrich and colleagues Faithful to sources School Not in cards Qwen3-Embedding-8B 0.89 Jan 2022 Oct 2026
CUA-World-Long Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task. Carnegie Mellon University (Welleck and Neubig labs, CMU L3) Desktop apps Expert Not in cards GPT-5.4 27.5% Apr 2026 Mar 2026
CURIE Biodiversity Georeferencing Models read a map and its caption from an ecology paper and return the geographic bounds. The score measures overlap with an expert's bounding box. Google and CURIE collaborators Spatial reasoning and counting University Not in cards Gemini 2.0 Flash 19.4 Feb 2025
CURIE Geospatial Dataset Extraction Models read a research paper and list its datasets, variables, sources, time spans, and geographic coverage in JSON. The authors report ROUGE-L and BERTScore. Google and CURIE collaborators Finding facts in long inputs University Not in cards Gemini 2.0 Flash 29.7 Feb 2025
CutVerse Agents complete editing tasks in seven applications, including Premiere Pro, Photoshop, and DaVinci Resolve. Vision models grade progress against milestones in the recorded workflow. Communication University of China, National University of Singapore, and USEIT AI Desktop apps University Not in cards Claude Opus 4.6 68.3% May 2026 Feb 2026
CyberGym-E2E (615 tasks) Agents discover and demonstrate vulnerabilities on the older 615-task cohort. The headline is cumulative stage S3 success. UC Berkeley RDI (Sunblaze) and collaborators Finding and exploiting real bugs Expert Not in cards Gemini 3 Pro 22.6% Jun 2026 Dec 2025
DeepResearch Arena Deep research agents get research tasks generated from academic seminar transcripts, which reflect expert discussion and reduce data leakage. DeepResearch Arena authors Deep research Expert Not in cards o4-mini-deep-research 3.96 Sep 2025 Jul 2025
DeployGuard Models diagnose synthetic deployment logs. The grader checks their JSON answers against known failure stages, exit codes, and operating constraints. Sasanka Weerakoon Fixing real repositories University Not in cards GPT-5.4 Mini 0.87 Oct 2026 Jul 2026
DevBench Acceptance Testing Models generate acceptance tests for repositories. The score is the published oracle-test percentage. OpenCompass (Shanghai AI Laboratory) and collaborators Building whole projects Expert Not in cards GPT-4 Turbo (Jan 2024) 29.2% Mar 2024 Jan 2024
DevBench Environment Setup Models prepare repository environments. The score is the percentage that pass the example usage test. OpenCompass (Shanghai AI Laboratory) and collaborators Building whole projects Expert Not in cards GPT-4 Turbo 41.7% Mar 2024 Jan 2024
DevBench Unit Testing Models generate unit tests for repositories. The score is the published oracle-test percentage. OpenCompass (Shanghai AI Laboratory) and collaborators Building whole projects Expert Not in cards GPT-4 Turbo (Jan 2024) 36.5% Mar 2024 Jan 2024
DEW-LogiQ Models choose answers to 358 questions on agricultural water management. The authors compare ordinary reasoning prompts with prompts that include concept maps. Michigan State University Science questions University Not in cards o3 87.4% Oct 2025 Jun 2025
DiagTrace Models request information about a patient one turn at a time. The evaluator compares the sequence of requests with the published case history. Yufan Wang and colleagues Medicine Expert Not in cards Kimi K2.5 0.40 Nov 2026 Jan 2026
DiscourseEE Models read Reddit discussions about opioid use disorder and extract details about medication use, tapering, and relapse. Some details require inference across a conversation. Dartmouth College Medicine University Not in cards Llama 3-8B 36.9% Oct 2024 Apr 2024
DS-1000 Models write Python for 1,000 data science problems from StackOverflow across seven libraries such as NumPy and Pandas. Test cases and API constraints check each answer. HKU, Stanford, UC Berkeley, UW, Meta and CMU Data science and SQL University Not in cards GPT-4 Turbo 53.9% Nov 2022 Apr 2024
EmbodiedCLUE-VLA (planning) Models reason about actions in physical environments. SuperCLUE Embodied reasoning Everyday Not in cards GPT-6 Astra 86.8 Sep 2026
ERP-Bench Agents complete 300 procurement and manufacturing workflows in Odoo 19. A constraint solver generates the tasks and optimal solutions, and the verifier checks final business state. Agentic Labs Apps and MCP servers Expert Not in cards GPT-5.5 43.4% May 2026 Apr 2026
ERR Estonian language and cultural history questions Chatbots answer ten questions about Estonian and ten about Estonian cultural history. ERR counts correct answers in each category. Estonian Public Broadcasting (ERR) Knowledge in many languages School Not in cards Grok 4 19.0 Feb 2026 Dec 2025
euchre-bench Coding agents build a euchre bot and test it against four opponents. Each opponent contributes up to 100 wins to the total. Few word do trick Building whole projects University Not in cards GPT-5.5 140.0 May 2026 May 2026
EvoPolicyGym Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves. EvoPolicyGym authors AI research tasks Expert Not in cards GPT-5.5 0.89 Jul 2026 Jun 2026
EXP-Bench Agents get a research question and incomplete starter code from an influential AI paper. They must form hypotheses, design and run experiments, and analyze results. Curie team (Just-Curieous) AI research tasks Research frontier Not in cards o3-mini 0.50% May 2025 Feb 2025
FairScreen: Testing LLM Hiring Decisions Models screen job applicants under a stated policy. Paired cases change identity cues, qualifications, or the completeness of the evidence. Maksim Silchenko Bias, fairness and politics University Not in cards Gemma 4 31B IT 98.4% Oct 2026 Apr 2026
Faroese Semantic Textual Similarity Models score how closely two Faroese sentences agree in meaning. The evaluation compares their rankings with reference judgments. Vésteinn Snæbjarnarson and colleagues Reading in many languages School Not in cards Qwen3-Embedding-8B 0.87 May 2023 Oct 2026
Financial Retrieval (Daloopa) Chatbots retrieve historical financial figures. Daloopa compares their answers with normalized reference values. Daloopa Finance University Not in cards Claude Opus 4.1 91.6% Sep 2025 Aug 2025
FinEval 6.0 Models solve financial evaluation tasks. The score is the official 6.0 total score. Shanghai University of Finance and Economics (AIFLM Lab) Finance University Not in cards Qwen3-32B 84.5% Aug 2023 Apr 2025
Finnish Parliament ASR test Models transcribe Finnish parliamentary speeches. The original test reports word error rates separately for speakers heard during training and speakers absent from training. Aalto University Speech recognition and translation Everyday Not in cards Whisper large-v3 9.29% Aug 2017 Oct 2026
FollowIR Models rank documents in response to a search instruction. The benchmark tests whether revising the instruction changes the ranking of the affected documents. Orion Weller and colleagues Questions that need search School Not in cards Qwen3-Embedding-4B 0.12 Mar 2024 Oct 2026
FrontierML (Synthetic Sciences) Agents conduct 25 empirical machine learning projects under fixed budgets. Each task has a verifier and reference scores. Harbor includes the 15 public projects. Synthetic Sciences AI research tasks Research frontier Not in cards Claude Fable 5.2 0.52 Oct 2026 Nov 2026
GAIK LA-murre dialect test Models transcribe 583 Finnish dialect interview segments from ten localities. The reference preserves dialect forms, so standard Finnish spellings can count as errors. GAIK project, Haaga-Helia University of Applied Sciences Speech recognition and translation Everyday Not in cards Gemini 3.5 Flash 60.2% Oct 2026 May 2026
GAIK SKN dialect test Models transcribe 600 Finnish dialect interview segments from ten municipalities. The reference uses standard Finnish word forms. GAIK project, Haaga-Helia University of Applied Sciences Speech recognition and translation Everyday Not in cards Gemini 3.5 Flash 34.6% Oct 2026 May 2026
GerDaLIR Small Models match German court passages to cited decisions. This version keeps only decisions that answer at least one test query. VAGO scores the top ten results with nDCG@10. Massive Text Embedding Benchmark team, adapting Marco Wrzalik and Dirk Krechel Law University Not in cards Qwen3-Embedding-8B 0.45 Mar 2024 Oct 2026
GerLayQA Models find German Civil Code sections that answer questions from the public. The authors also evaluated generated answers against lawyers' answers. Marius Büttner and Ivan Habernal Law University Not in cards Qwen3-Embedding-8B 0.27 Feb 2024 Oct 2026
GerLeRB Models rank German federal statute sections for 367 legal questions. Legal experts checked the questions against the cited provisions. University of Jena and collaborators Law University Not in cards Qwen3-Embedding-8B 0.70 Jun 2025 Oct 2026
gg-bench Models play 126 games invented by a language model and receive game-specific outcomes. gg-bench authors Puzzles and chess Everyday Not in cards o1 36.3% May 2025 Feb 2025
GoodAI LTM Benchmark Conversational agents retain, update, and use information across conversations with long stretches of intervening text. Task-specific evaluators compare answers with expected responses and assign scores from zero to one. GoodAI Reasoning over long documents Everyday Not in cards Claude 3 Opus 8.70 Feb 2024 Apr 2024
GraphicDesignBench Models analyze and produce graphic designs in 50 tasks spanning layout, typography, infographics, design semantics, and animation. The suite uses professional layered design templates. LICA World General visual questions Expert Not in cards GPT-5.4 0.94 Apr 2026 Mar 2026
GreekBarRetrieval Models retrieve Greek statutory articles for bar exam questions and case facts. The authors compare ranked results with the statutes relevant to each case. Ernest Beta and colleagues Law University Not in cards gpt-oss-120b 0.43 Aug 2026 Aug 2025
Group Binary Search (Riedl) Agents each choose an integer without seeing the other guesses. They adjust after hearing whether the group sum exceeds or falls below a hidden target. Christoph Riedl (Northeastern University) Multi-agent games Everyday Not in cards Gemini 2.0 Flash 75% Jul 2025 Jul 2025
HappyWorld Embodied The HappyWorld arena publishes a separate embodied vote pool and Elo ranking. Alibaba Interactive world models Everyday Not in cards MiniMax H3 1,074 Sep 2026
Hardest SAT Problems (Vellum) Models solve 50 hard SAT math problems. A judge compares each final answer with a reference, and a person reviews the result. Vellum Word problems School Not in cards Claude Sonnet 4 94% May 2025
HarnessEval-W Video and world models generate rollouts from image and action prompts. An evaluator agent plans checks, spawns sub-agents, gathers spatiotemporal evidence and scores observation, transition and persistence with written reasoning. MirroS Lab with xbench (HongShan) and universities Interactive world models Expert Not in cards Dreamina Seedance 2.0 75.5 Aug 2026
HEAD-QA v2 Models answer Spanish healthcare specialization exam questions across six disciplines. Version 2 adds languages, and the MedARC port runs the English configuration. Alexis Correa-Guillén, Carlos Gómez-Rodríguez, and David Vilares Medicine Expert Not in cards Llama 3.1-70B 83.2% Nov 2025 Jul 2024
HMU-Bench Models answer Chinese and English questions about musical notation and recordings. The authors score responses against reference answers. Zheqi Dai and colleagues Audio understanding and voice University Not in cards Gemini 3.7 Flash 2.47 Oct 2026 Aug 2026
Hungarian Riddles Benchmark Models answer 100 original Hungarian riddles and justify each answer in Hungarian. The author grades factual accuracy, reasoning, and language. Károly Boczka Reasoning in many languages Everyday Not in cards GPT-5 67.7% Oct 2025 Sep 2025
IndicLID benchmark Models identify Indian languages in native scripts and in Romanized text. Yash Madhani and colleagues Reading in many languages School Not in cards Qwen3-Embedding-8B 95.1% May 2023 Oct 2026
ironbench Agents write firmware for tasks such as serial communication and controller sequencing. The grader checks device behavior and counts tasks solved within the allowed attempts. Tsyren Balmaev Electronics University Not in cards glm-4.7-flash 69.2% Sep 2026 Feb 2026
Italian-Administrative-CaseHold Models classify Italian administrative judgments paired with their official holdings by legal subject. Daniele Licari and colleagues Law University Not in cards EmbeddingGemma 2 68.9% Oct 2026
JSICK Models judge entailment and semantic relatedness in Japanese sentence pairs. An additional stress test examines sensitivity to word order and case marking. Hitomi Yanaka and Koji Mineshima Reasoning in many languages School Not in cards Qwen3-Embedding-8B 89.4% May 2021 Oct 2026
KernelQuest Agents optimize Triton kernels. A task passes when the kernel produces correct outputs and reaches the speed target on the same GPU. Turing GPU kernels Expert Not in cards Claude Opus 5.5 79% Sep 2026 Sep 2026
KeyFrame-Compass Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities. KlingAI Research (Kuaishou) Video generation Everyday Not in cards Dreamina Seedance 2.0 0.81 Jul 2026 Mar 2026
Kinetics-700-2020 Models recognize human actions in short videos from the 2020 edition of Kinetics-700. DeepMind Video understanding Everyday Not in cards Qwen3-VL-Embedding-8B 62% Oct 2020 Jan 2026
Korean Sarcasm Dataset Models identify sarcasm in Korean text using labeled examples. Kocasm project Reading in many languages Everyday Not in cards Qwen3-Embedding-8B 62.1% Oct 2026
KorSET Models respond to Korean prompts adapted to local language, law, and culture. A model judge checks whether each response violates the relevant safety policy. DATUMO AI Safety Team Deception and alignment Everyday Not in cards EXAONE-3.5-7.8B-Instruct 19.9% Mar 2025
Lakh, Crore and Carpet Area Models solve Indian property and finance problems involving local land units, loan payments, and lakh and crore notation. reet_22 Word problems School Not in cards Claude Haiku 5.5 97% Oct 2026 Oct 2026
Legal Consent Sweep Agents review an acquisition data room, reconcile consent requirements, and prepare notices and closing advice. A separate verifier grades their work. DOE Law Expert Not in cards Claude Opus 5.5 0.38 Oct 2026 Sep 2026
LegalQuAD Models find the German legal document that answers each of 200 questions. The original data also mark the answer text and its starting position. Christoph Hoppe and colleagues Law University Not in cards Qwen3-Embedding-8B 0.70 Aug 2021 Oct 2026
LHAW Agents face 285 underspecified variants of long-horizon tasks with goals, constraints, inputs or context removed. Scored on strategic clarification and task success. Scale AI General assistants Expert Not in cards GPT-5.2 68.4% Feb 2026 Dec 2025
LIBERO-MAX Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing. LIBERO-MAX authors Simulated robots Everyday Not in cards MolmoAct 2 66.9% Aug 2026 May 2026
LLM-IOAA Models solve 49 theory problems and eight data analysis problems from four astronomy olympiads. Two expert graders score the solutions using the official rubrics. The Ohio State University and collaborators Science olympiads Competition Not in cards GPT-5 86.3% Oct 2025 Aug 2025
LogicVal Models generate images from prompts with formal constraints on objects, attributes, counts, and relationships. The evaluator checks which constraints the image satisfies. Savitha Sam Abraham and colleagues Image generation and editing School Not in cards Gemini 3 Pro Image Preview (nano-banana-pro) 50.9% Oct 2026
ltzGLUE Models perform language understanding tasks in Luxembourgish, including entailment, grammar judgments, sentiment analysis, and named entity recognition. Alistair Plum and colleagues Reasoning in many languages School Not in cards GPT-5 mini 69.4 Apr 2026 Aug 2025
MafiaBench Models play a social deduction game with hidden roles and public discussion. MafiaBench Multi-agent games Everyday Not in cards Gemini 2.0 Flash 1,561 Apr 2025 Feb 2025
MatToolBench Agents use materials science software on Windows to operate interfaces, write scripts, and query databases. The evaluator awards partial credit for completed steps. Mei Wu and colleagues Lab and data work University Not in cards GPT-5.4 51.5% Sep 2026 Mar 2026
MatTools Models answer questions about materials science software and write Python programs to calculate material properties. The authors execute the generated programs. Siyu Liu and colleagues Lab and data work University Not in cards Qwen2.5-72B 81.6% May 2025 Feb 2025
Medbullets Models answer clinical questions patterned on US medical licensing examinations. The benchmark also evaluates explanations of the answers. Hanjie Chen and colleagues Medicine Expert Not in cards GPT-4 68.8% Feb 2024 Apr 2024
MedCalc-Bench (original) Models apply 55 clinical calculators to patient notes. The original test contains 1,100 questions and checks numerical answers against task-specific tolerance bounds. NIH National Center for Biotechnology Information Medicine Expert Not in cards Llama 3-70B 39.5% Jun 2024 Apr 2024
MemEye Multimodal agent memory systems answer questions that need fine visual evidence kept from earlier sessions, which captions or text traces alone cannot supply. MemEye authors (Kwok et al.) Finding facts in long inputs Everyday Not in cards GPT-5.4 Mini 49.4% May 2026 Mar 2026
MemoryAgentBench Memory agents receive long inputs in chunks over many turns and are tested on accurate retrieval, test-time learning, long-range understanding and selective forgetting. It adds two new sets, EventQA and FactConsolidation. UC San Diego (Hu, Wang, McAuley) Learning from the context Everyday Not in cards GPT-5 mini 60.6% Jul 2025 Aug 2025
Mercury Models write Python solutions and pass hidden correctness tests. The evaluator compares runtime with reference solutions. Harbor ports all 256 problems from the evaluation split. Mingzhe Du and colleagues Speed and optimization University Not in cards StarCoder 2 15B 0.59 Feb 2024 Apr 2024
MESA Robot policies follow instructions as object locations, categories, and task sequences change. The authors compare seven methods on single-arm and two-arm tasks. Georgia Institute of Technology Simulated robots Everyday Not in cards π0.5 50.4% Nov 2026 Dec 2025
MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. Google DeepMind Long videos Expert Not in cards GPT-5 42.2% Jan 2026 Sep 2025
MisconceptDialogue Tutors identify a simulated student's mathematical misconception and address it through dialogue. The authors compare exact-match and human-calibrated model grading on the same conversations. Electronics and Telecommunications Research Institute and collaborators Other professions University Not in cards Claude Haiku 4.5 87.8% Oct 2026 Mar 2026
MIVAS Bench Voice agents handle calls for a dermatology practice, a law firm, and a retailer. Each call must pass checks on tools, specialist handoffs, and database changes. Bluejay Labs Customer service University Not in cards Grok Voice 3.0 56.5% Aug 2026 Oct 2026
MLRC-Bench Agents propose and implement methods for machine learning competitions. The scorer compares their performance with a supplied baseline and the strongest human solution. Yunxiang Zhang and colleagues AI research tasks Research frontier Not in cards Llama 3.1-405B 6.26 Apr 2025 Mar 2025
MMLU-SR Models answer MMLU questions after key terms are replaced with synonyms. The benchmark tests whether models recognize the same concept under different wording. Wentian Wang and colleagues Exam questions University Not in cards GPT-4o mini 65% Jun 2024 Jul 2024
ModelBench: Reliability Under Constraints Models follow several simultaneous constraints. Paired tasks compare a control case with a harder version, and code checks each response. Eze Prince jr Following precise instructions School Not in cards gpt-oss-20b 0.97 Oct 2026 Sep 2026
MotoVerse Models use a rider's camera to anticipate maneuvers and predict nearby road users' trajectories. The authors compare vision, language, and conventional prediction models. Varun A. Paturkar and colleagues Embodied reasoning Everyday Not in cards Qwen3-VL-2B-Instruct 46.3% Nov 2026 Oct 2025
MTEB-BR Embedding models encode Brazilian Portuguese text for classification, clustering, similarity, retrieval, and reranking tasks. The headline score averages the primary task metrics across 22 native Portuguese tasks. Tardelli Ronan Coelho Stekel, Federal Institute of Sao Paulo Reading in many languages University Not in cards Qwen3-Embedding-8B 0.67 Jul 2026 Mar 2026
MultiEURLEX Models assign subject labels to European Union legislation written in different languages. Ilias Chalkidis and colleagues Law Expert Not in cards Qwen3-Embedding-4B 0.05 Sep 2021 Oct 2026
MusICA-MetaBench Models answer the same music questions from audio, printed scores, and ABC notation. The authors publish fixed tests for two collections of choral music. Charles University and Brno University of Technology Audio understanding and voice University Not in cards Gemini 3.1 Pro 59% Jul 2026 Mar 2026
NFCorpus Models rank medical research documents for health questions expressed in everyday language. Vera Boteva and colleagues Medicine University Not in cards Qwen3-Embedding-8B 0.41 Feb 2016 Jun 2025
OBLIQ-Bench Retrievers search large real corpora for documents that match a latent pattern, such as an implicit stance. The suite has five such oblique search problems. MIT (Tchuindjo, Shah and Khattab) Questions that need search Expert Not in cards GPT-5.2 0.12 May 2026 Mar 2026
OdysseysMega Agents complete tasks in a web browser. A judge checks each rubric criterion against the recorded actions and screenshots. Turing Web browsing Expert Not in cards Claude Opus 5.5 41% Sep 2026
Omni-InteractScope Proactive Models provide timely proactive assistance during audio and video streams. The score is the published T8 assistance recall. Alibaba Group (Qwen, Alibaba Token Hub) + Hong Kong Polytechnic University Video understanding Everyday Not in cards Gemini 3.1 Pro 46.4% Oct 2026 May 2026
Omnilingua-AudioMC Models follow requirements across spoken conversations in ten languages. The authors compare each model's spoken and text responses against the same 1,404 rubric items. Nankai University and Alibaba Speech recognition and translation Everyday Not in cards Muse Spark 1.2 64% Nov 2026 Sep 2026
OpenAudioBench AlpacaEval Models respond to spoken AlpacaEval instructions. The score is the published judge quality score. Baichuan Audio understanding and voice Everyday Not in cards Baichuan-Omni-1.5 7.79 Jan 2025 Jan 2025
OpenAudioBench Llama Questions Models answer spoken Llama Questions. The score is the published accuracy. Baichuan Audio understanding and voice Everyday Not in cards Baichuan-Omni-1.5 78.5% Jan 2025 Jan 2025
OpenAudioBench Reasoning QA Models answer spoken reasoning questions. The score is the published accuracy. Baichuan Audio understanding and voice Everyday Not in cards Baichuan-Omni-1.5 50% Jan 2025 Jan 2025
OpenAudioBench TriviaQA Models answer spoken TriviaQA questions. The score is the published judge quality score. Baichuan Audio understanding and voice Everyday Not in cards MiniCPM-o-2_6 6.19 Jan 2025 Jan 2025
OpenAudioBench Web Questions Models answer spoken Web Questions. The score is the published judge quality score. Baichuan Audio understanding and voice Everyday Not in cards MiniCPM-o-2_6 6.86 Jan 2025 Jan 2025
OpenLegalRetrieval-DE Models retrieve German statutes and court decisions across seven tasks. Each task scores where relevant documents appear in the first ten results. Each task carries equal weight. VAGO Solutions Law University Not in cards Qwen3-Embedding-8B 0.44 Nov 2026 Oct 2026
ORAV Bench Models combine reference images, sounds, and videos into a requested audiovisual scene. A judge compares pairs of outputs against each reference's assigned role. Tsinghua University and Tencent Hunyuan Video generation Everyday Not in cards Dreamina Seedance 2.5 0.70 Sep 2026
OS-Critic Bench Models judge whether a proposed action advances a task using the screenshot and action history. The grader compares each judgment with a human label. Shanghai AI Laboratory and collaborators Finding things on screen Everyday Not in cards GPT-5 68.2% Dec 2025 Sep 2025
ParallelKernelBench Models replace PyTorch and NCCL reference code with multi-GPU CUDA kernels for 87 problems from codebases such as Megatron-LM and DeepSpeed. Scored on correctness and speed over the baseline. Together AI and Stanford GPU kernels Expert Not in cards GPT-5.5 32.2% Jun 2026 Jun 2026
PazaBench v2 Models transcribe speech in 61 African languages. The leaderboard measures character error rate, word error rate and transcription speed. Microsoft Research Africa, Nairobi Speech recognition and translation Everyday Not in cards Whisper large-v3-turbo 73.3% Jul 2026 Feb 2025
PCBWorld-Bench Agents route printed circuit boards in KiCad. A board earns a clean pass when all required connections are complete and KiCad reports zero design-rule errors. LG AI Research and collaborators Electronics University Not in cards GPT-5.4 0.54 Jul 2026 Mar 2026
PII Masking Models locate personal information in US English text and replace it with a common redaction tag. The default split reserves 20% for evaluation. Adam Lucek Following precise instructions Everyday Not in cards Claude Sonnet 4.5 0.73 Oct 2025 Sep 2025
Pipecat Speech-to-Text Benchmark Speech-to-text services transcribe the same 1,000 realtime audio clips. Semantic word error rate measures accuracy, and time to final segment measures streaming latency. Pipecat Speech recognition and translation Everyday Not in cards Muse Voice Transcribe 1.0 0.83% Aug 2026 Sep 2026
PRDBench Code agents build 50 real-world Python projects across 20 domains from product requirement documents. A trained judge agent checks each project against structured criteria. AGI-Eval (Meituan) Building whole projects Expert Not in cards GPT-5.2 69% Oct 2025 Dec 2025
PRISM-Bench (audio and video generation) Models generate speech, music, or environmental sounds with video. A judge compares outputs with reference clips across 35 criteria, including synchronization and prompt following. Shanghai Artificial Intelligence Laboratory and Meituan Video generation Everyday Not in cards Dreamina Seedance 2.0 91.9 May 2026
Project Euler (MathArena) Models solve recent Project Euler problems that need both mathematical insight and code. MathArena adds problems as Project Euler releases them, starting from problem 943. MathArena (ETH Zurich SRI Lab and INSAIT) Competition problems with answers Expert Not in cards Gemini 3.1 Pro 89% Sep 2025 May 2026
PsychEthicsBench Models answer multiple-choice questions and respond to scenarios under Australian psychology and psychiatry ethics guidelines. Graders check responses against professional standards. Monash University, University of Liverpool, and Hefei University of Technology Wellbeing and sycophancy University Not in cards Qwen2.5-14B 75.6% Jan 2026 Sep 2024
PulseBench-Tab Models convert 1,820 table images in nine languages into HTML tables. T-LAG represents adjacent cells as directed edges and matches their text and structure against human annotations. The reported score is F1 over matched edge weights. Pulse Software Corp Documents, charts and text in images Everyday Not in cards Claude Fable 5.1 0.89 Apr 2026 Sep 2026
RadioBench Models answer radio engineering questions and generate programs for software-defined radios. The authors measure successful execution, hardware deployment, and reception of the intended signal. Jiazhen Lei and the RadioMaster team Electronics University Not in cards GPT-5.5 97% Jun 2026 May 2026
Real2Sim Coding agents get a robot recording from DROID (three camera videos, stereo calibration, joint and gripper commands and the task instruction) and build an interactive simulation of the scene. Each scene is scored on how closely it matches the recording, on replaying the recorded actions, and on whether a fixed robot policy completes the task in it. Momentum Research Labs Simulated robots Expert Not in cards Claude Fable 5.2 0.60 Nov 2026 Nov 2026
ReasonZoo Models solve tasks in logical reasoning, puzzles, and programming. The evaluation compares their answers with task solutions. OpenCompass and collaborators Abstract puzzles and logic University Not in cards Qwen3-235B-A22B 61% Aug 2025 May 2025
RedlineBench Agent acts as counsel at one turn of a SaaS contract negotiation and returns tracked-change redlines. Three LLM judges grade attorney-written rubrics. Crosby Intelligence + micro1 Law Expert Not in cards GPT-5.5 50.5% Jun 2026 Jun 2026
RefuNQ Models answer factual questions and decide when to decline. The authors create unanswerable questions by replacing a noun in a real question with an invented term. University of Illinois Urbana-Champaign Short factual questions Everyday Not in cards GPT-4 53.7% Nov 2023 Sep 2023
ReviewBench AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1. GitHub and Microsoft Fixing real repositories Expert Not in cards Claude Opus 5.5 73.3% Oct 2026 Sep 2026
RoboChallenge Table30 v2 2.0 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. Dexmal + Hugging Face Real robots Everyday Not in cards π0.5 0.73% May 2026 Apr 2025
RoboDx Agents repair faults in robot navigation software. Hidden driving scenarios test the repair. A separate diagnosis track asks models to identify faults without seeing the code. Soham Dan, Scale AI Fixing real repositories Expert Not in cards Claude Opus 5.5 100% Nov 2026 Oct 2026
RSIBench-Data Coding agents act as data-centric researchers. With a fixed Tinker post-training stack and budget, they refine training data from dev-benchmark feedback, and the final model is scored on held-out test benchmarks. Evolvent AI + National University of Singapore Machine learning engineering Research frontier Not in cards GPT-5.6 Sol 32.6% Jul 2026 Jul 2026
ScaLA Models judge whether sentences in mainland Scandinavian languages are linguistically acceptable. Dan Nielsen Reading in many languages School Not in cards Qwen3-Embedding-8B 57.4% May 2023 Oct 2026
SemEval 2022 Task 8 multilingual news article similarity Models rate the similarity of two news articles, including articles in different languages. The evaluation compares predicted similarities with human judgments. Xi Chen and colleagues Reading in many languages School Not in cards Qwen3-Embedding-8B 0.72 Jul 2022 Jun 2025
SemEval 2024 Task 1 semantic textual relatedness Models rank sentence pairs by relatedness across languages. The evaluation compares their rankings with human judgments. Nedjma Ousidhoum and colleagues Reading in many languages School Not in cards EmbeddingGemma 2 0.65 Mar 2024 Oct 2026
Skill Reward Hacking Researchers track when models learn to increase a proxy reward without improving task quality. They measure warning time, false alarms, and transfer across reward functions. tonyteo Deception and alignment University Not in cards Qwen3.6 27B 13.8 May 2026 Apr 2026
Small Model Agent Bench Models choose tools and supply their arguments in 28 Chinese and English episodes. Fixed tool responses let the grader check planning, recovery, and task completion. Cacinie Function calling School Not in cards Qwen3-Coder-480B-A35B 95.7% Aug 2026 Sep 2026
SmartPlay Language-model agents play a collection of games designed to test planning, reasoning, memory and interaction abilities. Microsoft Research Puzzles and chess Expert Not in cards GPT-4 0.80 Oct 2023 Jul 2023
Snorkel Working Memory Test Models answer questions from a document placed among distractors. The evaluation varies the document position and the length of the input. Snorkel AI Finding facts in long inputs School Not in cards GPT-4 Turbo 87.6% Jul 2024 Apr 2024
SQL Reality Check Models write SQL for revenue and ticket histories. The grader runs each query on seven databases with cases such as partial refunds and tied timestamps. Dhruv Bansal Data science and SQL University Not in cards Gemma 4 26B A4B 100% Oct 2026 Sep 2026
Stack EOL Audit: geol CLI vs API Agents identify software components that have reached end of life. They audit the same twelve stacks from memory, with an API, and with a command line tool. Adrien Sales Function calling University Not in cards Gemma 4 26B A4B 1.00 Oct 2026 Apr 2026
SuperCLUE-DSPSafeBench Models respond to Chinese prompts that test content safety. SuperCLUE Deception and alignment Everyday Not in cards phi-3-mini 3.8B 78.6 Sep 2024
SuperCLUE-TTS Models read Chinese text aloud. The benchmark rates basic speech synthesis and delivery in situations such as public announcements. SuperCLUE Audio understanding and voice Everyday Not in cards Gemini 3.8 Flash TTS 69.7 Dec 2024 Sep 2026
SuperCLUE-VLR Models solve problems that require visual reasoning. SuperCLUE General visual questions University Not in cards Gemini 2.5 Pro 72.1 May 2025
SuperCLUE-World World models generate visual sequences. SuperCLUE Video world models University Not in cards Veo 3.1 Fast Audio 74.6
SWE-Pair Agents fix two real bugs from the same Python project using one combined issue report. Hidden tests from both pull requests grade each fix. Evaligo Labs Fixing real repositories Expert Not in cards DeepSeek V4.1 Flash 90.4% Nov 2026 Sep 2026
T2I-NSI Models generate images from paired positive and negative prompts. The index measures whether removing a requested concept also removes it from the image. Indian Institute of Technology Jodhpur and Colorado State University Image generation and editing Everyday Not in cards GPT-Image-1.5 0.1 Nov 2026
T2LSC-Bench Models render text at a specified location in an image. Judges check the lettering and whether its meaning alters other objects or the background. Yan Wang and colleagues Image generation and editing Everyday Not in cards Gemini 3.1 Flash Image 99% Sep 2026 Feb 2026
TaskBench Models decompose requests, choose tools and dependencies, and predict tool parameters. Text metrics score decomposition, while graph and parameter F1 scores measure the remaining stages. Zhejiang University, Microsoft Research Asia and Fudan University Function calling Everyday Not in cards Claude 2 93.5 Nov 2023 Jan 2024
TavernBench Agents build a Hearthstone Battlegrounds player through research, simulator updates, and live games. TavernBench authors Video games Expert Not in cards GPT-6 Astra 95.6 Oct 2026 Sep 2026
TeluguMixBench Models answer Telugu and mixed Telugu and English prompts. The harness reports accuracy across comprehension, reasoning, translation, and instruction following. Poojitha Saranya Reasoning in many languages School Not in cards Claude Haiku 4.5 100% Oct 2026 Oct 2026
TextEdit Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity. OpenCompass (Shanghai AI Laboratory) Documents, charts and text in images Everyday Not in cards Gemini 3 Pro Image Preview (nano-banana-pro) 0.92 Mar 2026
ToolQA Models answer questions that need outside information, using 13 tools such as database queries, text retrieval and a calculator over fixed data sources. The questions are built so that a model's internal knowledge cannot answer them. An easy and a hard set separate single lookups from multi-step tool use, and answers are checked by exact match. Georgia Institute of Technology Function calling Everyday Not in cards davinci-003 24.1% Jun 2023 Jul 2023
UniREditBench Image editing models carry out 2,700 edits that need reasoning, in real-world scenes and in game worlds with human-defined rules, across 8 primary and 18 secondary dimensions. A multimodal judge scores each edit against both a text reference and a ground-truth image. Fudan University and collaborators Image generation and editing Everyday Not in cards Seedream 4.0 57% Nov 2025
Vals Smith: arandomhooman/hoomans-morphe-patches Agents modify arandomhooman/hoomans-morphe-patches to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Sol 20% Aug 2026 Jul 2026
Vals Smith: chrisanderton/tyk-gateway Agents modify chrisanderton/tyk-gateway to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Sol 76.7% Aug 2026 Jul 2026
Vals Smith: Expertium/fsrs-autoresearch Agents modify Expertium/fsrs-autoresearch to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Sonnet 5 96.7% Sep 2026 Jul 2026
Vals Smith: geopandas/geopandas Agents modify geopandas/geopandas to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Gemini 3.6 Flash 66.7% Aug 2026 Jul 2026
Vals Smith: goharbor/harbor Agents modify goharbor/harbor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 60% Aug 2026 Jul 2026
Vals Smith: ladydascalie/core Agents modify ladydascalie/core to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards DeepSeek V4 Flash 0731 76.7% Aug 2026 Jul 2026
Vals Smith: leanprover-community/mathlib4 Agents modify leanprover-community/mathlib4 to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 76.7% Aug 2026 Aug 2026
Vals Smith: skypilot-org/skypilot Agents modify skypilot-org/skypilot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 4.8 30% Aug 2026 Jul 2026
Vals Smith: suitedaces/dorabot Agents modify suitedaces/dorabot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Muse Spark 1.2 66.7% Aug 2026 Aug 2026
VDEBench Image editing models make local text and table edits to 3,506 Chinese and English document images. Scores check the edit and that the rest of the page is preserved. Yi et al. (VDE-Bench authors) Documents, charts and text in images Everyday Not in cards FLUX.2 [klein] 4B 0.77 Jan 2026 Sep 2025
VELOCITI Models distinguish a video's correct description from a caption that assigns an action to the wrong person. Darshana Saravanan and colleagues Video understanding School Not in cards LLaVA-OneVision 72B 43.1% Jun 2024 Aug 2024
VidScribe Models write, copy, preserve, or edit visible text in videos. The evaluator tracks spelling, glyph shapes, placement, and stability across frames. Alibaba Group, Shanghai Jiao Tong University, and Fudan University Video generation Everyday Not in cards MiniMax H3 79.0 Sep 2026
VocalGrad Models decide which way an acoustic property changes within a recording. The authors compare model accuracy with human listeners on the same binary questions. The University of Tokyo and Tohoku University Audio understanding and voice Everyday Not in cards Kimi-Audio-7B-Instruct 51.8% Oct 2026 Dec 2025
VSI-Bench-Debiased Models answer the 2,362 VSI-Bench questions left after removing 2,768 that text-only models could solve from answer priors or world knowledge. New York University (VisionX) Spatial reasoning and counting Everyday Not in cards InternVL3-9B 41.5% Nov 2025 Apr 2025
WebHarbor Agents navigate local copies of 153 websites to search, sign in, book services, and change stored state. The suite contains 3,518 tasks, including 306 verified test tasks and 2,288 training tasks. The test score measures the percentage of tasks passed by scripted verifiers. AIMING Lab Web browsing Everyday Not in cards GPT-6 Astra 86.7% May 2026 Sep 2026
WeGenEditBench Models edit images from instructions that can include text or multiple references. A shared checklist scores each requested change and the details that should stay intact. Tencent WeChat Vision and collaborators Image generation and editing Everyday Not in cards Seedream 5.0 Pro 4.14 Oct 2026
WorkArena++ Web agents complete 682 compositional knowledge-work tasks in ServiceNow, which need planning, reasoning and memory across many steps. ServiceNow Research Web browsing Expert Not in cards GPT-4o 1.90% Jul 2024 May 2024
WorkSurface-Bench Agents choose documents, tables, or graphs to answer workplace questions. The benchmark scores where agents look, what evidence they find, and their answers. Hao Liang and colleagues Documents and spreadsheets University Not in cards GPT-5.5 10.3 Jul 2026 Apr 2026
WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. WorldArena team (Tsinghua FIB Lab) Interactive world models Not in cards Code World Model (CWM) 69.5% May 2026 Sep 2025
WorldExam Controllable video generators are tested as world models. Levels run from visual appearance to whether the depicted world reacts plausibly beyond what the prompt states. WorldExam authors Video world models Everyday Not in cards Veo 3.1 Audio 72.8 Aug 2026
Writing Benchmark (Every) Models complete eight writing and editing assignments. Every compares drafts with published writing and checks whether edits match its editors’ choices. Every Taste and style University Not in cards Claude Opus 4.8 67.8 Jun 2026 Jun 2026
YoCausal Video generation models are tested for causal and arrow-of-time understanding by comparing real videos with their time-reversed versions, in two levels. Alaya Lab Physical prediction Everyday Not in cards Wan2.2-T2V-A14B 54.2 May 2026 Jul 2025
Act2Answer Vision-language-action models answer commonsense and world-knowledge questions by acting. Each question becomes a short tabletop episode, and the robot answers by placing an object on one of the candidate answers. The score is the success rate per knowledge category, which reduces the effect of weak low-level control. Act2Answer authors (CognitiveAISystems) Simulated robots Everyday Not in cards GR00T N1.7 68% Jun 2026 Apr 2026
ActiveVision Models inspect images repeatedly to count regions, trace structures, and compare visual attributes. The released split has five examples for each of 17 tasks. Exact match checks the final answer after normalizing case, spacing, and separators. University of Southern California Spatial reasoning and counting Everyday Not in cards Claude Fable 5 50.6% Jul 2026 Jun 2026
AgentEscapeBench Agents solve procedural escape rooms using unfamiliar tools. The evaluator checks whether they complete dependent steps and escape. AGI-Eval (Meituan) Puzzles and chess Everyday Not in cards Claude Opus 4.6 76.9% May 2026 Mar 2026
AgenticDataBench Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios. Tsinghua University Data science and SQL Expert Not in cards Kimi K2.5 48.8% Jul 2026 Feb 2026
AgenticSTS Agents play the Silent in Slay the Spire 2 at its lowest difficulty. The authors compare wins with different memory settings and two other agents. Xiangchen Cheng and colleagues Video games Everyday Not in cards Gemini 3.1 Pro 60% Jun 2026 Apr 2026
AI Detection Benchmark Systems label private pre-2022 human documents and AI rewrites of them as human or AI written. Both LLMs and commercial detectors are scored by balanced accuracy. Vals AI Taste and style Expert Not in cards Claude Opus 5.5 98.4% Sep 2026 Sep 2026
AnomalyXL Models locate and describe anomalies in long synthetic time series. The benchmark checks multiple-choice answers and scores precise answers using the known anomaly properties. Nicolas Zumarraga and the TimeRLM team Reasoning over long documents Expert Not in cards GPT-5.5 0.33 Aug 2026 Apr 2026
APE (Attempt to Persuade Eval) A model is asked to persuade a simulated user across 600 topics, from benign facts to conspiracies and clearly harmful ones. A judge scores whether it attempts persuasion. FAR.AI Refusing harmful requests Everyday Not in cards o4-mini 10% Jun 2025 May 2025
Arctic-Extract Business Document Benchmark Models answer questions about business documents and extract information from varied layouts. The internal visual tasks include contracts, signatures and checkboxes. Snowflake Documents, charts and text in images University Not in cards GPT-5 58.5 Aug 2025 Aug 2025
ArenaWebdevCorrectness Judges inspect generated applications and predict whether their features work. Human labels provide the reference answers. Arena Web and app front ends University Not in cards GPT-5.6 Sol 81.6% Oct 2026 Jul 2026
Argus-Bench v2 Models screen inputs to a language model gateway. The benchmark checks detection rates, false positives, and agreement with the policy on severity. Karmendra Pandey Jailbreaks and prompt injection University Not in cards Gemini 2.5 Flash 85.7% Nov 2026 Sep 2026
ASAG2024 Models grade short answers from seven English datasets on a common zero to one scale. The authors weight errors to account for rare grades. Zurich University of Applied Sciences Other professions University Not in cards GPT-4o 0.27 Jun 2024 May 2024
AssemblyAI Speech Benchmarks Speech recognition systems transcribe clean, accented, multilingual and code-switched audio, identify entities and diarize speakers. Error rates compare production systems across public speech sets. AssemblyAI Speech recognition and translation Expert Not in cards Voxtral Mini Transcribe 2 5.60% Nov 2026 Aug 2026
AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. Google DeepMind and ETH Zurich Machine learning engineering Expert Not in cards Claude 3.7 Sonnet 22% Mar 2025 Feb 2025
AutoBench (HDL testbench generation) Models write self-checking Verilog testbenches. The evaluator checks compilation, acceptance of correct designs, and agreement with reference testbenches on faulty designs. Ruidi Qiu and colleagues Electronics University Not in cards Claude 3.5 Sonnet 71.1% Jul 2024 Aug 2024
Automated Student Assessment Prize 2.0 Models score source-based essays from seven prompts on a six-point scale. The competition measures agreement with human grades using quadratic weighted kappa. The Learning Agency Lab and Vanderbilt University Other professions University Not in cards Chinchilla 0.84 Apr 2024 Oct 2024
AutoPatchBench Agents write security patches for crashes that a fuzzer found in native code. Fixes are checked by fuzzing and differential debugging. Meta (PurpleLlama) Secure code and review Expert Not in cards Llama 3.3 70B Instruct 11.5% Apr 2025 Apr 2025
B-MoCA Agents use Android apps such as Instagram on phones with different languages, font sizes, and icon positions. Task-specific checks decide whether each attempt succeeded. KAIST and Yonsei University Phones Everyday Not in cards GPT-4o 47.7% Apr 2024 May 2024
Base64Bench Models encode and decode base64 strings across different text and data formats. The evaluator measures exact matches and edit similarity. Rich Barton-Cooper Following precise instructions School Not in cards Claude 3.7 Sonnet 75.9% Oct 2025 Aug 2025
BigEarthNet.txt Models answer land cover questions, caption satellite images, and locate regions described in text. The authors checked annotations for 1,082 pairs of radar and optical images. BIFOLD and Technical University of Berlin with collaborators General visual questions University Not in cards GLM-4.6V-Flash 0.79% Mar 2026 Dec 2025
BRIGHT-Pro Models retrieve passages that cover different parts of a question. The 739 questions include 115 from Earth Science Stack Exchange. Yale NLP and collaborators Questions that need search University Not in cards Qwen3-Embedding-8B 41.8 Apr 2026 Jun 2025
BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. Xiangyu Xu and colleagues Real robots Everyday Not in cards π0-FAST 0.56 Oct 2026 Apr 2025
CADExpert Models write CadQuery programs from dimensioned engineering drawings. The graders check whether each program runs and compare its 3D shape with the reference. Ke Niu and colleagues CAD models University Not in cards Doubao-Seed-1.6 35.6% Dec 2025 Aug 2025
Cekura Speech to Text Overall word error rate across the published audio clips. Cekura Audio understanding and voice Everyday Not in cards GPT-Realtime-Whisper 2.63% Jul 2026 May 2026
Cents Matter Models solve Brazilian Portuguese accounting cases in matched pairs. The grader checks amounts to the cent across rounding, cash flow signs, and cases with insufficient evidence. Bruno Fernandes Finance University Not in cards Gemini 3.7 Flash 97.5% Oct 2026 Aug 2026
ChemPro Models answer chemistry questions across four difficulty levels. The authors report multiple-choice accuracy and accept numerical answers within specified tolerances. Aaditya Baranwal and colleagues Science questions University Not in cards Qwen3-32B 38.7% Feb 2026 Apr 2025
ChouBun Models answer questions about long Japanese documents and summarize company securities reports. Sakana AI Reasoning over long documents University Not in cards Llama 2-7B 17.4 Oct 2024 Apr 2024
CLEF JOKER 2026 Task 3: Onomastic Wordplay Translation Models translate English names containing wordplay into French using a short description. The official score accepts reference matches and alternatives approved by translators. CLEF JOKER organizers Translation University Not in cards Gemini 3 Flash 71.9% Oct 2025 Dec 2025
CLIMATE-FEVER Models find evidence for climate claims and decide whether it supports or refutes each claim. Thomas Diggelmann and colleagues Faithful to sources University Not in cards Qwen3-Embedding-4B 0.47 Dec 2020 Jun 2025
CMI-RewardBench Models judge musical quality and how well recordings follow requests. Some requests combine text, lyrics, and reference audio. Scores measure agreement with human listeners. Queen Mary University of London, Peking University, and collaborators Audio understanding and voice Everyday Not in cards Gemini 3 Pro 0.60 Feb 2026 Nov 2025
CodeContests Models write programs for competitive programming problems, drawn from Codeforces and other sites, and the programs run against hidden tests. Google DeepMind built the dataset for AlphaCode, with training, validation (117 problems) and test (165 problems) splits and many generated tests per problem to cut false positives. Scores are the share of problems solved within a fixed number of submissions, such as 10@k. Google DeepMind Competitive programming Competition Not in cards GPT-5.4 81.8% Feb 2022 Apr 2026
CommonGround Elicit Models identify ambiguities, contradictions and gaps in organizational documents. They select a clarification and predict how each stakeholder will respond. Charlie Thompson Reading comprehension University Not in cards GPT-4.1 0.40 Aug 2026 Sep 2025
CompMat-Bench Agents complete individual steps from published computational materials studies. Fixed rules compare their inputs and outputs with results reproduced by the authors. Chenmu Zhang and colleagues Lab and data work Expert Not in cards GPT-5.6 Sol 91.5% Sep 2026 Sep 2026
CoSE-E Models transcribe business conversations that switch between languages. The main metric checks whether transcription errors change the answers to questions about the conversation. ServiceNow AI Research Speech recognition and translation Everyday Not in cards Gemini 3 Flash 4.20% Sep 2026 Dec 2025
Cost Per Correct v2 Models answer tasks at three difficulty levels and repeat some with paraphrased instructions. Rules check correctness so the author can compare cost per correct answer. Karmendra Pandey General assistants University Not in cards gpt-oss-120b $91 Nov 2026 Sep 2026
CRAB Multimodal agents do tasks that span an Ubuntu desktop and an Android phone at once. Graph-based evaluators give credit for intermediate steps. CAMEL-AI and collaborators Desktop apps Everyday Not in cards GPT-4o 38% Jul 2024 May 2024
CultureAtlas Models judge whether statements about a region or cultural group are true. The authors pair statements drawn from Wikipedia with false alternatives and report F1. University of Illinois at Urbana-Champaign Commonsense Everyday Not in cards Vicuna-13B-v1.3 73.7% Feb 2024 Jul 2023
DCASE 2025 Audio Question Answering Models answer questions about marine mammal calls, the timing of sounds, and overlapping events. Each of the three sections counts equally toward the final score. DCASE 2025 Task 5 organizers Audio understanding and voice University Not in cards Kimi-Audio-7B-Instruct 52.1% Dec 2024 Apr 2025
Decision Index Decision models read a state and a typed question and return a decision. The index weights private same-skill sets and new-domain sets, equated on 21 base models. Hugging Face community (apolinario, multimodalart) Composite indices Everyday Not in cards Gemma 4 31B IT 67.6 Sep 2026 Aug 2026
DevBench Models work through the software lifecycle on 22 curated repositories in Python, C/C++, Java and JavaScript: design, environment setup, implementation, acceptance tests and unit tests. Tests score code and an LLM judge scores design. OpenCompass (Shanghai AI Laboratory) and collaborators Building whole projects Expert Not in cards GPT-4 Turbo 41.7% Mar 2024 Jan 2024
DevBench Implementation Models implement repository code. The score is the acceptance-test pass rate, with unit-test pass rate as an auxiliary metric. OpenCompass (Shanghai AI Laboratory) and collaborators Building whole projects Expert Not in cards GPT-4 Turbo (Jan 2024) 7.10% Mar 2024 Jan 2024
DI-Bench Agents answer questions using company tables, documents, and business rules. The benchmark checks whether agents apply those rules when computing results. Jiangyun Zhang and colleagues Running a business University Not in cards Claude Opus 4.6 57.9% Sep 2026 Feb 2026
DolphinBench Agents use long-term memory to complete 600 tool-using tasks for three simulated knowledge workers with roughly 500,000 user-message tokens each. Mem0 General assistants Expert Not in cards GPT-5.6 Luna 70.7% Sep 2026 Jul 2026
Domain Dynamics English-Hindi Translation Models translate 2,094 English sentences into Hindi across six domains. The authors report BLEU and COMET on the full set and by domain. Indian Institutes of Technology Patna and Jodhpur Translation University Not in cards Llama 3-70B 26.2 Aug 2024 Aug 2024
DriveBench Behavior Models answer driving behavior questions. The score is the published GPT evaluation score for this task family. UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU Embodied reasoning Everyday Not in cards InternVL2-8B 54.6 Jan 2025 Aug 2024
DriveBench Perception Models answer driving perception questions. The score is the published GPT evaluation score for this task family. UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU Embodied reasoning Everyday Not in cards Qwen2-VL-7B-Instruct 35.2 Jan 2025 Aug 2024
DriveBench Planning Models answer driving planning questions. The score is the published GPT evaluation score for this task family. UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU Embodied reasoning Everyday Not in cards Qwen2-VL-7B-Instruct 57.0 Jan 2025 Aug 2024
DriveBench Prediction Models answer driving prediction questions. The score is the published GPT evaluation score for this task family. UC Irvine, Shanghai AI Laboratory, NUS, NTU and HKU Embodied reasoning Everyday Not in cards InternVL2-8B 48.9 Jan 2025 Aug 2024
DSCodeBench Models write Python functions for 1,000 data science problems covering ten libraries. The evaluator checks generated code against reference outputs. Shuyin Ouyang and collaborators Data science and SQL University Not in cards Qwen2.5-Coder-32B-Instruct 22.9% May 2025 Nov 2024
Duplicate Side Effect Desk Agents process refund requests when timeouts or duplicate tickets obscure payment status. The evaluator uses the transaction ledger to identify duplicate refunds. jigonyoo Customer service Everyday Not in cards Claude Sonnet 4.5 0.98 Sep 2026 Oct 2025
DyCodeEval Models solve dynamically rewritten HumanEval and MBPP problems with equivalent program behavior. The evaluator runs tests and computes DyPass@K across generated variants. Columbia University Function-level problems School Not in cards Llama 3.1-8B 39% Mar 2025 Sep 2024
EduVideoBench Models generate classroom videos intended to teach facts, skills, or attitudes. Experts assess instructional accuracy, presentation, and educational safety. Korea University and collaborators Video generation University Not in cards Wan 2.6 0.59 May 2026
EgoHome Models annotate 137 egocentric clips of household chores, about two hours in all, with subtasks, hand actions, hand captions and hand boxes. Segments are as short as 0.1 seconds. A model judge matches segment labels, and boxes are scored by centroid hits. Perceptron Video understanding Everyday Not in cards GPT-6 Astra 0.80 Nov 2026 Sep 2026
EHRNoteQA Models read a patient's accumulated MIMIC-IV discharge summaries and answer clinical questions about them. The set has 962 question and answer pairs, each for a different patient. KAIST and collaborators (Kweon et al.) Medicine Expert Not in cards GPT-4 96.2% Feb 2024 Dec 2023
Eleusis Agents play cards to discover a hidden rule. The scorer tests whether a proposed rule makes the same decisions as the hidden rule. np_hard (nph4rd) Multi-agent games Competition Not in cards GPT-6 Sol 82.2% Jun 2026 Sep 2026
EvalCheck-LLM Models use supplied evidence to answer a question, abstain, or flag a conflict. The task tests whether pressure changes that decision. Akash_ Faithful to sources School Not in cards Qwen3-Coder-480B-A35B 100% Oct 2026 Jul 2026
EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. Rocktim Jyoti Das and colleagues Knowledge in many languages School Not in cards Gemini 1.0 Pro 44% Mar 2024 Jan 2024
FarSick Models rate how closely two Persian sentences agree in meaning. The original corpus also labels whether one sentence entails or contradicts the other. Zahra Ghasemi and Mohammad Ali Keyvanrad Reading in many languages Everyday Not in cards Qwen3-Embedding-8B 0.76 Sep 2021 Jun 2025
Fechamento BR Models normalize fields in fictional Brazilian administrative records. Paired cases test how context changes the interpretation of money, missing values, dates, and identifiers. gabriel fernandes Documents and spreadsheets University Not in cards Gemini 3.8 Flash 1.00 Oct 2026 Sep 2026
FieldWorkArena V2.0 Agents inspect workplace images, videos, manuals, and logs to plan actions and report operational events. The evaluator compares their answers and actions with reference results. It combines correctness judgments with partial credit for numerical agreement. Fujitsu and Carnegie Mellon University Tasks from real jobs University Not in cards GPT-5.2 52% Jun 2025 Dec 2025
Fin-Fact Models judge whether evidence supports a financial claim, contradicts it, or leaves too little information. The original evaluation also tests models with images. Illinois Institute of Technology and China University of Petroleum Finance University Not in cards Claude 3 Opus 61% Sep 2023 Feb 2024
First Proof Third Batch Batch 3 The editors run five AI systems once each on ten unpublished research problems with proofs of up to 12 pages. About 30 referees score correctness, exposition and attribution. The score is average correctness, from negative 1 to 1. First Proof Foundation Research mathematics Research frontier Not in cards GPT-6 Astra 0.65 Oct 2026 Sep 2026
FlashInfer-Bench Agents generate GPU kernels for LLM inference that are benchmarked on workloads from real serving traces and can be deployed into the FlashInfer stack. FlashInfer team GPU kernels Expert Not in cards GPT-5 0.47× Jun 2025 Aug 2025
FML-bench Agents propose and run experiments on eight machine learning research tasks. The authors measure task performance and how widely the proposals vary across iterations. Qiran Zou and colleagues AI research tasks Expert Not in cards Claude Opus 4.1 0.50 Oct 2025 Aug 2025
FollowBench Models answer 820 English and Chinese instructions where each level adds one more constraint of content, situation, style, format or example type. Rules and an LLM judge check each constraint. HKUST and Huawei Noah's Ark Lab (Jiang et al.) Following precise instructions Everyday Not in cards GPT-4 Turbo 73.4% Oct 2023 Nov 2023
FVEval Models write SystemVerilog assertions from properties or circuit designs. A formal verifier checks syntax and tests whether assertions express the reference behavior. NVIDIA and collaborators Electronics Expert Not in cards Llama 3.1-70B 24.8% Jun 2024 Jul 2024
GAOKAO-Eval 2024 Models take Chinese national college entrance examinations. High school teachers grade the open responses. OpenCompass and collaborators Exam questions School Not in cards Qwen2-72B 476.5 Jun 2024 Jun 2024
GenPuzzle Models generate images that solve visual puzzles such as mazes, tangrams, and Sudoku. The authors select an automated judge using human reference scores. Changpeng Zhao and colleagues Image generation and editing School Not in cards Seedream 5.0 Pro 40.6 Sep 2026 Feb 2026
GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. Jinghe Wang and colleagues Embodied reasoning Everyday Not in cards OpenVLA 20.7% Nov 2026 Apr 2025
Grid Perception Reliability Models inspect text grids to count connected shapes, choose valid cells, and translate positions. The harness checks responses against generated ground truth. Flamki Spatial reasoning and counting School Not in cards Gemini 3.7 Flash 99.1% Oct 2026 Aug 2026
GSR-Bench Models turn speech into transcripts, meeting notes, or other requested text. Judges check whether the output preserves the meaning and stays grounded in the recording. China Telecom and Northwestern Polytechnical University Speech recognition and translation Everyday Not in cards Gemini 3.1 Pro 4.21 Oct 2026 Feb 2026
GUI Odyssey Mobile GUI agents carry out tasks that cross several apps, such as sharing a post from one social app to another. The dataset has 8,334 episodes averaging 15.3 steps, over 212 apps, 1,357 app combinations and six devices. Agents are scored by how often each predicted action matches the recorded step. OpenGVLab (Shanghai AI Laboratory) and collaborators Phones Everyday Not in cards GPT-4V 55% Jun 2024 Jul 2024
HumanoidToolBench Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1. Seoul National University (SNU PI) Real robots Everyday Not in cards GR00T N1.7 49.4% Oct 2026 Jul 2026
ImgMT Models translate text embedded in diagrams, maps, and infographics. Evaluation measures text recognition, region matching, and translation with CER, F1, chrF, and COMET. Šárka Uramová, Petar Petrov, Josef Jon, Ondřej Bojar, and Michal Novák Translation Everyday Not in cards Kimi K3 61.3 Oct 2026 Jul 2026
IMO-GradingBench Models grade 1,000 olympiad proof attempts, and their grades are compared with human expert grades. Tests automatic evaluation of long-form math answers. Google DeepMind (Superhuman Reasoning team) Written proofs Competition Not in cards o3 54% Nov 2025 Aug 2025
IndicContextEval Models transcribe speech in eight Indian languages using hints about the topic or names. One prompt deliberately supplies names from an unrelated field. AI4Bharat at IIT Madras and Sarvam AI Speech recognition and translation Everyday Not in cards Gemini 3 Flash 17.5% Jun 2026 Dec 2025
IndicIPR-QA Models extract answers from passages in seven Indian intellectual property laws. The benchmark compares predicted spans with reference answers across 4,462 questions. South Asian University Law University Not in cards Qwen2.5-7B 77.6 Oct 2026 Sep 2024
InfiHall-Bench Models generate images from 411 prompts that combine visual requirements. The evaluator checks details such as object parts, actions, and spatial relations separately. Harbin Institute of Technology, Shenzhen and collaborators Image generation and editing Everyday Not in cards FLUX.1 [dev] 67.1 Nov 2026 Jan 2025
InstrumentalEval Models respond to scenarios where gaining resources or preserving themselves could serve another goal. A grader identifies responses that pursue those intermediate goals. Yufei He and colleagues Deception and alignment Everyday Not in cards o1-mini 43.4% Feb 2025 Jan 2025
IntPhys2-Timing Models identify when a physical rule breaks in a video. The authors add violation intervals to IntPhys2 and score temporal localization. Taekoan Yoo and colleagues Physical prediction Everyday Not in cards Gemini 3.8 Flash 77.4% Oct 2026 Sep 2026
KnowMe-Bench Models read long autobiographical narratives and must infer a person's stable motivations and decision principles, not just retrieve facts. QuantaAlpha Learning from the context Everyday Not in cards GPT-5 mini 50.2 Jan 2026 Nov 2025
Labelbox Agentic Search Agents search for answers to 200 questions across several knowledge domains. Labelbox reports overall and domain scores. Labelbox Questions that need search University Not in cards Gemini 2.5 Pro 80% Jun 2025 Jun 2025
Labelbox Deep Research Agents Agents investigate research questions at the doctoral level. Labelbox compares their answers and reports scores by domain. Labelbox Deep research Expert Not in cards Gemini 2.5 Pro 77.5% Jul 2025 Jun 2025
LAMBADA Multilingual (Stable LM 2) Models predict missing final words in narrative passages translated for the Stable LM 2 evaluation. The harness provides five language configurations. Stability AI Reading in many languages Everyday Not in cards Gemma 2B 39.8% Feb 2024 Feb 2024
LHRS-Bench Models answer 690 single-choice questions about 108 remote sensing images. The questions cover land use, object locations, image resolution, and other properties. Nanjing University General visual questions University Not in cards Claude 3 Opus 39.1% Mar 2024 Feb 2024
LIBERO-Plus LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate. Fudan University + Tongji University + Shanghai Innovation Institute + NUS Simulated robots Everyday Not in cards OpenVLA-OFT 69.6% Oct 2025 Feb 2025
LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. Huazhong University of Science and Technology + Harvard + MIT + others Simulated robots Everyday Not in cards π0.5 53% Oct 2025 Apr 2025
LLM Game Benchmark Models compete in tic-tac-toe, Connect Four, Gomoku, and other grid games through a shared interface. LLM Game Benchmark authors Puzzles and chess Everyday Not in cards Claude 3.5 Sonnet 185.0 Jan 2024 Jun 2024
LongExtractionBench Extraction systems pull structured values from 225 long table-heavy documents averaging 358 pages and about 88,700 ground-truth fields each. micro1 Documents, charts and text in images Expert Not in cards Claude Opus 4.8 70.7% Jun 2026 May 2026
MarsRetrieval Models match Mars images to captions, retrieve examples of named landforms, and locate features in a global mosaic of the planet. Southern University of Science and Technology Spatial reasoning and counting University Not in cards EmbeddingGemma 2 26.5 Feb 2026 Oct 2026
MEDEC Models identify and correct errors in clinical notes across five categories. The OpenMed port uses the MEDEC-MS subset. Asma Ben Abacha and colleagues Medicine Expert Not in cards o1-mini 69.1% Dec 2024 Feb 2025
MedLFQA Models write long answers to biomedical questions. The benchmark checks whether answers include required facts and avoid claims identified as incorrect. Minbyul Jeong and colleagues Medicine Expert Not in cards Claude 3 Sonnet 77.1 May 2024 Feb 2024
Mind2Web-SC Models generate guardrails that check a web agent’s proposed actions against user constraints. The benchmark measures whether the guardrail permits or blocks each action correctly. Zhen Xiang and colleagues Web browsing University Not in cards Llama 3.3 70B 93% Jun 2024 Dec 2024
MU-Bench Speech-to-text providers transcribe 4,270 caller utterances from real 8 kHz customer service phone calls in five locales, scored against human word-level transcripts. Sierra Speech recognition and translation Everyday Not in cards Gemini 3.5 Transcribe 14.9% Apr 2026 Aug 2026
Multi Agent Coordination Bench Teams of eight agents work for four hours on twelve tasks built so each agent depends on its teammates. Each model also runs alone and as a team that can't talk, which isolates the gain from coordination. Bespoke Labs Multi-agent games Expert Not in cards Claude Opus 5.5 0.53 Nov 2026 Sep 2026
Neural Activation Reading for Collusion Benchmark Monitors estimate whether committee members secretly coordinated their decisions. The transcript task measures how well probability estimates distinguish matched collusion and control cases. Aaron Rose and colleagues Deception and alignment University Not in cards gpt-oss-20b 0.87 Apr 2026 Aug 2025
neuralCAD-Edit Agents edit existing CAD models from text or recorded instructions. CAD experts rate whether they would accept each edit. Software also compares the resulting geometry. Autodesk Research CAD models University Not in cards GPT-5.2 0.25 Apr 2026 Dec 2025
NitiBench Models retrieve Thai statutes and answer legal questions. The authors score retrieval separately and use a model judge to check coverage and contradictions. Pawitsapak Akarajaradwong and colleagues Law University Not in cards Claude 3.5 Sonnet 70.3 Feb 2025 Sep 2024
NormAd-Eti Models decide whether behavior in an everyday story is socially acceptable in the stated country. Questions vary how much cultural guidance the model receives. Akhila Yerukola and colleagues Reasoning in many languages Everyday Not in cards Mistral-7B-Instruct-v0.2 81.8% Apr 2024 Feb 2024
OOLONG-Pairs Models identify pairs of users whose records satisfy a condition across a long input. The evaluation compares the predicted pairs with the reference set. Alex L. Zhang and colleagues Reasoning over long documents University Not in cards Claude Opus 5 92.9% Dec 2025 Jul 2026
OpenEarthAgent Agents use tools such as QGIS to measure features in satellite images and maps. The authors grade tool decisions and final answers. Mohamed bin Zayed University of Artificial Intelligence and collaborators Apps and MCP servers University Not in cards GPT-5 44.5% Feb 2026 Aug 2025
OpinionQA Models answer public opinion survey questions. The evaluation compares model response distributions with those of demographic groups. Shibani Santurkar and colleagues Bias, fairness and politics Everyday Not in cards GPT-3 175B (davinci) 0.79 Mar 2023 Nov 2022
Pakistan Local Context LLM Suite Models answer questions about Pakistani geography, local time, currency notation, and Urdu usage. Tooba Munir Knowledge in many languages School Not in cards DeepSeek-R1 0% Oct 2026 Sep 2026
People Making Conversation Speech models continue real conversations at recorded pauses and turn endings. The benchmark compares their timing, response length, and recall with the original speakers. Arya Bastani and colleagues Speech recognition and translation Everyday Not in cards Moshi 0.39 Oct 2026 Mar 2026
Per-DOC Models use a reader’s previous comparison to predict which of two new plots the reader prefers on each of five criteria. Carnegie Mellon University, Meta AI and UC Berkeley Taste and style Everyday Not in cards Llama 2-70B 0.21% Oct 2023 Jul 2023
Per-MPST Models use a reviewer’s earlier movie ratings and reviews to predict their rating of a new plot. The authors measure correlation with human ratings. Carnegie Mellon University, Meta AI and UC Berkeley Taste and style Everyday Not in cards Llama 2-70B 18.1 Oct 2023 Jul 2023
PhyRubric-Bench Models generate videos of physical events. A judge checks caption-specific requirements against evidence at particular times in each clip. University of Illinois Chicago and collaborators Video generation School Not in cards Veo 3.1 Audio 36.2 Nov 2026 Jul 2025
PlanBench Models construct action sequences that reach a goal under stated planning rules. A planning validator checks whether each plan satisfies action preconditions and achieves the goal. Arizona State University and Colorado State University Abstract puzzles and logic School Not in cards DeepSeek-R1 55.5% Jun 2022 Jan 2025
PRMBench Models identify errors and redundant steps in mathematical solutions. PRMScore combines positive and negative F1 scores across simplicity, soundness, and sensitivity tests. Fudan University, Shanghai AI Laboratory, Soochow University, Stony Brook University, and Chinese University of Hong Kong Word problems University Not in cards o1-mini 68.8 Jan 2025 Feb 2025
Q-Bench-Video Models judge the quality of natural, AI-generated and computer-graphics videos through yes-or-no, what-how, open-ended and pairwise comparison questions. Q-Future (Nanyang Technological University and Shanghai Jiao Tong University) Video understanding Everyday Not in cards GPT-4o mini 52.2% Sep 2024 Jul 2024
Q2D-Web Retrievers search 190 million web documents for queries that agents reformulate from real conversations. Perplexity Questions that need search University Not in cards Qwen3-Embedding-8B 79.3% Sep 2026 Jun 2025
Qualcomm Interactive Cooking Models watch a cook follow a recipe. They give timed instructions, recognize completed steps, and warn about mistakes. The benchmark scores completion accuracy and mistake detection. Qualcomm AI Research and the University of British Columbia Video understanding Everyday Not in cards Gemini 2.5 Flash 23.1% Nov 2025 Sep 2025
Qwen-Audio-Agent Smart Cockpit Benchmark Agents operate simulated vehicle, navigation, music, and weather services. The short suite checks whole cases. The long suite checks tool use on each turn. Alibaba Function calling Everyday Not in cards Qwen3.8 Flash 98.8% Sep 2026 Aug 2026
RAFT (Real-world Annotated Few-shot Tasks) Models learn text classification tasks from small labeled samples. The tasks use real documents, including customer messages, court decisions, and scientific abstracts. Neel Alex and colleagues Reading comprehension University Not in cards GPT-3 175B (davinci) 62.7% Sep 2021 Mar 2021
RealBench (Verilog) Models write Verilog modules or complete systems from design specifications. Testbenches check behavior, and a formal checker compares the result with the reference implementation. Institute of Computing Technology, Chinese Academy of Sciences, and collaborators Electronics Expert Not in cards o1-preview 6.65% Jul 2025 Sep 2024
RealCompanion Models read months of real chats between ten people and an AI companion, then answer questions about who each person is. The release has 27,218 messages over up to 120 days. Quis Lab Multi-turn conversation Everyday Not in cards Gemini 3.8 Flash 70.1% Oct 2026 Sep 2026
Reasoning Dial Models solve deduction, arithmetic, and distractor tasks at different reasoning settings. The evaluation records accuracy, output tokens, and cost. Abeera Lodhi Language reasoning School Not in cards Claude Haiku 5.5 99.6% Oct 2026 Oct 2026
Red-Pen Bench Models review code with planted defects and fixed or clean controls. The grader checks whether they identify the defect and locate the affected lines. Robert Gilbert USA Fixing real repositories University Not in cards Claude Haiku 5.5 0.64 Oct 2026 Oct 2026
Redtape Models assess eligibility for public benefits and identify when a missing fact prevents a decision. PolicyEngine supplies the reference determinations. jakpotvin Law Expert Not in cards GPT-5.6 Sol 0.79 Sep 2026 Sep 2026
RepoNormBench Agents change code in existing repositories while following the contribution rules that apply to each task. SYSUSELab Fixing real repositories Expert Not in cards DeepSeek-V4-Flash 68.6% Oct 2026 Apr 2026
ReportBench Agents write literature surveys. The evaluator compares their references with published surveys and checks whether cited sources support their claims. ByteDance BandAI Deep research Expert Not in cards OpenAI deep research 0.39 Aug 2025 May 2025
Resolve Models edit sixteen functions. Hidden tests check the requested behavior and verify that previously working behavior still passes. Aditya Kumar Puri Function-level problems University Not in cards Claude Haiku 4.5 87.5% Oct 2026 Apr 2026
ReTrace Computer agents recover from injected errors in desktop tasks. Replaying the remaining actions checks whether the recovered state produces the original successful outcome. KAIST, ETRI, and DeepAuto.ai Desktop apps Everyday Not in cards GPT-5.4 63.6% Oct 2026 Mar 2026
RevBench (Polyvor Labs) Agents analyze supplied artifacts in 22 local reverse engineering puzzles. The verifier checks each submitted answer for an exact match. Polyvor Labs Capture-the-flag challenges Expert Not in cards GPT-5.5 50% Aug 2026 Aug 2026
RMS-AQA Models identify sound events in domestic spaces, then answer questions about their positions and timing. The benchmark scores both stages. Peihao Chen and colleagues Audio understanding and voice Everyday Not in cards Qwen3.8-Omni-Flash 36.4% Oct 2026 Sep 2026
Robocurve simple tasks (LLM control of YAM arms) Frontier LLMs control bimanual I2RT YAM arms through Inspect Robots tools on block-into-bowl, puzzle, push-ball and stack-bowls. Staged scores, 20 trials. Robocurve Real robots Everyday Not in cards GPT-6 Astra 52.5% Sep 2026 Sep 2026
RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. Carnegie Mellon University (Welleck lab, CMU L3) Simulated robots Expert Not in cards Claude Opus 5.5 0.06 Oct 2026 Oct 2026
RSCC Models describe disaster damage from satellite images taken before and after an event. The authors reserve 988 image pairs from 19 events for testing. Zhejiang University Detection and captioning University Not in cards InternVL3-8B 79.2 Apr 2025 Apr 2025
SeedBench (Seed Science) Models answer questions, summarize passages and interpret research on rice breeding in Chinese and English. The 11 task types use accuracy, macro F1 or ROUGE-L. Shanghai Artificial Intelligence Laboratory and collaborators Science questions University Not in cards DeepSeek-V3 68.4 Dec 2024 Mar 2025
SMM4H-HeaRD 2026 Detection of Insomnia in Clinical Notes Models identify likely insomnia in nursing and discharge notes. They label diagnostic criteria and give the character offsets of supporting evidence. Cedars-Sinai and collaborators Medicine University Not in cards Nov 2025
Spyfall study Models play Spyfall while inferring hidden locations and managing deceptive dialogue. Spyfall study authors Multi-agent games Everyday Not in cards Gemma 3 12B 1003.0 Jan 2026 Apr 2025
Summary of a Haystack Models read a synthetic haystack of conversation or news documents where specific insights repeat, then write a query-focused summary that cites its sources. An automatic evaluation scores insight coverage and citation accuracy. Salesforce AI Research Reasoning over long documents University Not in cards Claude 3 Opus 42.5% Jul 2024 Mar 2024
SuperCLUE-ASR Models transcribe Chinese speech under different acoustic conditions. SuperCLUE Speech recognition and translation Everyday Not in cards GPT-Transcribe 12.1% Aug 2026
SuperCLUE-Auto Models answer Chinese automotive questions through multiple turns. A judging model scores each answer against criteria for its task. SuperCLUE team Other professions University Not in cards Baichuan2-13B-Chat 76.6 Dec 2023 Oct 2023
SuperCLUE-ICabin Models respond to requests for an automotive cabin assistant. SuperCLUE General assistants Everyday Not in cards GPT-4o 82.2 Aug 2024
SuperCLUE-Open Models answer Chinese open-domain questions over multiple turns. The authors compare models across skill categories. SuperCLUE team Chat and instructions in many languages Everyday Not in cards GPT-4 94.6 Jul 2023 Jan 2024
SuperCLUE-VideoEdit Models edit videos in response to Chinese instructions. SuperCLUE Video generation University Not in cards Dreamina Seedance 2.5 74.2
SWE-Race Agents fix concurrency and asynchronous lifecycle bugs in Python projects. The benchmark has 95 public and 93 private tasks. Harbor distributes 94 public tasks. Evaligo Labs Fixing real repositories Expert Not in cards GLM-5.3-Flash 81.9% Oct 2026 Aug 2026
TCS-Bench Models write proofs of target results from STOC, FOCS and SODA papers, given the needed context. A verification agent, checked against expert judgments, grades each proof. Google Research + collaborators (Cohen-Addad, Woodruff and others) Research mathematics Research frontier Not in cards Gemini 3.7 Flash 55% Aug 2026 Aug 2026
TELBench Models identify harmful error spans in 1,000 deep-research agent trajectories. Expert labels distinguish mistakes from harmless exploration and tool noise. Evaluation reports macro precision, recall, F1, and first-error accuracy. Nanjing University NJU-LINK and JIUTIAN Research Faithful to sources University Not in cards Claude Sonnet 4.6 54.9% Jun 2026 Mar 2026
Terminal-Bench 2.0 Verified (Z.ai) Agents do the Terminal-Bench 2.0 tasks after Z.ai fixed environments for 89 tasks and instructions for 11. A further 2026-08-18 round fixed tests and instructions in 6 more tasks. Z.ai (Zhipu AI) Terminal tasks Expert Not in cards Claude Opus 4.5 61.8% Feb 2026 Feb 2026
Terminal-Bench Data Agents solve terminal tasks in machine learning, data analysis, and scientific computing. Task verifiers grade the outputs. Turing Terminal tasks Expert Not in cards GPT-5.5 48.6% Jul 2026 May 2026
TextEdit Real Models edit text in real images in the full TextEdit test set. The score is the published judged average. OpenCompass (Shanghai AI Laboratory) Documents, charts and text in images Everyday Not in cards GPT-Image-1.5 0.90 Mar 2026
TextEdit Virtual Models edit text in virtual images in the full TextEdit test set. The score is the published judged average. OpenCompass (Shanghai AI Laboratory) Documents, charts and text in images Everyday Not in cards GPT-Image-1.5 0.90 Mar 2026
The Amazing Agent Race Agents navigate websites, call tools, and calculate a passcode across 1,400 scavenger hunts. The benchmark includes 800 sequential puzzles and 600 with branching paths. Minnesota NLP Function calling Expert Not in cards Claude Sonnet 4 36.6% Apr 2026 Mar 2026
Tricky Edits of Collected and Curated Images Models edit a source image to follow a written instruction. People and a model judge instruction following, preservation of unaffected content, and visual quality. Google DeepMind Image generation and editing Everyday Not in cards Gemini 3.1 Flash Image 19.5% May 2026 Feb 2026
USACO Models solve 307 USA Computing Olympiad problems. Hidden unit tests check correctness and efficiency, scored as pass@1. Princeton University Competitive programming Competition Not in cards GPT-4 Turbo 20.2% Apr 2024 Feb 2024
USIM underwater robot tasks Agents follow instructions to navigate, grasp, transport, and track objects underwater. The simulator measures task completion across 700 trials. Institute of Automation, Chinese Academy of Sciences, and collaborators Simulated robots Everyday Not in cards π0.5 37.6% Oct 2025 Jun 2025
VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. Yang Qu and colleagues Embodied reasoning Everyday Not in cards Qwen2.5-VL-72B 43.5% Apr 2026 Apr 2025
Vals Inference Provider Index Agents use different inference providers to complete the same finance, coding, and legal tasks. Vals AI Composite work indices Expert Not in cards DeepSeek V4.1 Flash 71.2% Nov 2026 Sep 2026
Vals Smith: arduino/arduino-ide Agents modify arduino/arduino-ide to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Luna 43.3% Aug 2026 Jul 2026
Vals Smith: AumeshMisra/tensorzero Agents modify AumeshMisra/tensorzero to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Luna 86.7% Oct 2026 Jul 2026
Vals Smith: conjfrnk/blackbird Agents modify conjfrnk/blackbird to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 56.7% Aug 2026 Aug 2026
Vals Smith: dwojtaszek/zipper Agents modify dwojtaszek/zipper to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Sol 93.3% Sep 2026 Jul 2026
Vals Smith: fmtlib/fmt Agents modify fmtlib/fmt to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Grok 4.5 83.3% Sep 2026 Aug 2026
Vals Smith: NVIDIA/nccl Agents modify NVIDIA/nccl to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 46.7% Aug 2026 Aug 2026
Vals Smith: OpenTTD/OpenTTD Agents modify OpenTTD/OpenTTD to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Sol 100% Sep 2026 Jul 2026
Vals Smith: punt-labs/lux Agents modify punt-labs/lux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 73.3% Sep 2026 Aug 2026
Vals Smith: vastsa/PI-Desktop Agents modify vastsa/PI-Desktop to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Muse Spark 1.2 40% Sep 2026 Aug 2026
VAmoS Pro Bench Voice agents handle several utility billing requests in one call. A model checks the spoken amounts and recorded actions against the requirements for each request. Veris AI Customer service University Not in cards Gemini 3.8 Live 40.1% Sep 2026 Sep 2026
WebCanvas Web agents do tasks on live websites, scored on key intermediate states rather than one fixed action path, so the test survives site changes. iMean AI Web browsing Everyday Not in cards GPT-4o 28.8% Jun 2024 Jul 2024
WebLINX Agents follow a user's multi-turn chat instructions to navigate real websites. 100,000 interactions from 2,300 expert demonstrations give turn-level metrics. McGill University and Mila Web browsing Everyday Not in cards GPT-4V 10.4 Feb 2024 Nov 2023
Wiki Live Challenge Deep research agents write articles on topics of 100 recent Wikipedia Good Articles, which serve as expert references. Scored on 39 writing criteria and factual verifiability. USTC and Metastone Technology (DeepResearch Bench team) Deep research Expert Not in cards GPT-5 53.6 Feb 2026 Aug 2025
WorkBuddy User Blind Evaluation Agents complete everyday work requests. WorkBuddy users rate their satisfaction on a four-point scale without seeing the model identity. Tencent Documents and spreadsheets University Not in cards DeepSeek V4.1 Flash 1017 3 Nov 2026 Oct 2026
WorldState-Bench Video models generate two clips from the same image and action. A hidden state changes which ending the rule requires. A vision model checks both clips. Tsinghua University and GigaAI Video world models Everyday Not in cards MiniMax H3 32% Oct 2026 Jul 2025
WRENCH (Docker Compose) Agents maintain a service pipeline through seeded disruptions. The grader measures retained throughput and recovery. WRENCH authors Terminal tasks University Not in cards Claude Sonnet 5 0.81 Sep 2026 Jun 2026
Addis AI Amharic Text to Speech Benchmark Speech models read 100 Amharic prompts aloud. Native speakers rate naturalness, intelligibility and voice quality without seeing the system names. Addis AI Audio understanding and voice Everyday Not in cards Gemini 3.1 Flash TTS 4 Aug 2026 Apr 2026
AerialSim Agents fly a simulated drone using camera observations or write a program for the flight. The simulator checks the recorded trajectory against the mission requirements. Nahid Alam Simulated robots Expert Not in cards Claude Haiku 4.5 73.2% May 2026 Feb 2026
AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. Stanford and Johns Hopkins (Schmidgall et al.) Medicine Expert Not in cards GPT-4 37.8% May 2024 May 2024
Almieyar Models transcribe newly recorded speech in 17 Arabic dialects. The authors report word and character error rates across six dialect families. Omid Ghahroodi and colleagues Speech recognition and translation Everyday Not in cards gpt-4o-transcribe 34.9% Sep 2026 Mar 2025
AMBER Multimodal models are checked for object, attribute and relation hallucinations in generative and discriminative tasks, scored without an LLM judge. Wang et al. (AMBER authors) Hallucination rates Everyday Not in cards mPLUG-Owl2 84.0 Nov 2023 Nov 2023
AMELIA Models classify passages from Italian tax decisions as premises or conclusions. They distinguish factual from legal premises and identify the reasoning behind legal premises. University of Bologna and collaborators Law University Not in cards Llama 3.1-70B 0.76 Oct 2024 Jul 2024
AndroidArena Agents use apps such as Gmail and Google Maps, sometimes switching apps or following extra constraints. GPT-4 judges success from the recorded actions and observations. Peking University and Microsoft Research Phones Everyday Not in cards Llama 2-70B 12.7% Feb 2024 Jul 2023
App-Bench Coding agents build six full-stack web applications from a single prompt. Two developers grade implemented features against 151 rubric items. The score uses each tool’s best result from three attempts per application. AfterQuery Building whole projects University Not in cards Claude Opus 4.5 67.5% Dec 2025 Nov 2025
ArabCulture-Dialogue Models pick the most culturally fitting reply in dialogues from 13 Arab countries, in Modern Standard Arabic and each local dialect. It also has translation and dialect-steering tasks. MBZUAI (Al Kautsar, Almheiri et al.) Chat and instructions in many languages Everyday Not in cards GPT-5 95% Apr 2026 Aug 2025
ArenaWebdevPreferences Judges compare pairs of generated applications and predict which application the user preferred. The score measures agreement with user votes. Arena Web and app front ends Everyday Not in cards Gemini 3 Flash 69.1% Oct 2026 Jul 2026
Aspire Agent gets only a vague capability goal and must choose data, training and self-checks to evolve weights or harness. Scored on 520 hidden items. ByteDance Seed et al. AI research tasks Research frontier Not in cards Qwen3.5-9B 25.6% Aug 2026 Feb 2026
ATOM-Bench Robot policies combine manipulation skills on single arm and dual arm platforms. BAAI FlagEval Real robots Everyday Not in cards π0.5 48.2% Jun 2026 Dec 2025
Audio Arithmetic: 24 Music Workflow Checks Models calculate frame counts, storage, timing, and resampling values for music workflows. The grader compares integer answers with exact reference calculations. wenye li Word problems University Not in cards gpt-oss-20b 100% Oct 2026 Oct 2025
AutoCFD Incompressible Flow Benchmark Agents configure OpenFOAM simulations from flow descriptions and meshes. The authors check solver convergence and compare simulated flow with reference values. Zhehao Dong, Zhen Lu, and Yue Yang Scientific code University Not in cards Qwen2.5-72B 31.4% Apr 2025 Dec 2024
AutoEval Autonomous real-robot evaluation: WidowX arms run five Bridge tasks with learned success detectors and resets. 50 rollouts per task. UC Berkeley Real robots Everyday Not in cards OpenVLA 40% Mar 2025 Jun 2024
Automated LLM Speedrunning Benchmark Agents try to reproduce 19 past records of the NanoGPT speedrun, given the earlier code and hints of varying detail. Scored by how much of each record's speedup they recover. Meta AI research tasks Research frontier Not in cards o3-mini 46% Jun 2025 Feb 2025
AutoPLC Models write Structured Text for CODESYS and Siemens controllers from requirements. The benchmark measures how often generated programs compile and how many errors the compiler reports. Beihang University Electronics University Not in cards Qwen2.5-Coder (7B) 9.88% Dec 2024 Sep 2024
BIG-bench English Proverbs Models read a short story and choose the English proverb that fits it. The 34 stories offer three to six choices each. Daniel Moseguí González Reading comprehension Everyday Not in cards Gopher (280B) 57.6% Jan 2021 Dec 2021
BIG-bench Understanding Fables Models read a paraphrased Aesop fable and choose its moral from five options. The test contains 189 fables. Denis Emelin Reading comprehension Everyday Not in cards Gopher (280B) 39.6% Jun 2021 Dec 2021
Bimanual Chore Scheduling Models schedule household chores for a simulated two-arm robot. The grader checks resource constraints and compares completion time against an exact optimum. jwilksbooth Embodied reasoning University Not in cards Claude Haiku 4.5 0.82 Jul 2026 May 2026
BlenderBench (Refresh) Agents edit Blender scenes across 20 tasks. Each task grades the submitted scene using its own scoring configuration. Refresh CAD models University Not in cards GPT-6 Astra 0.69 Sep 2026 Sep 2026
Blicket Test Causal Reasoning Models experiment with a simulated detector to infer which objects activate it. The hub version rewards correct identification and informative experiments. Irfan Jamil Abstract puzzles and logic Everyday Not in cards Qwen3-30B-A3B 0.56 Feb 2026 Apr 2025
BoundaryBench: Policy Under Pressure Models apply a refund policy to twelve cases. The grader compares decisions across clean messages, harmless distractions, and instructions that conflict with the policy. Arya Koste Customer service Everyday Not in cards GPT-5.4 Nano 100% Oct 2026 Mar 2026
BuiltBench Models match a building component description to the corresponding entry in a construction classification system. Mehrzad Shahinmoghadam and colleagues CAD models Expert Not in cards Qwen3-Embedding-8B 65.3 Nov 2024 Jun 2025
CaptionSmith Models convert word-timed transcripts into SRT subtitles. The grader checks text preservation, segmentation, timing, line length and reading speed. thesatvik Following precise instructions University Not in cards Qwen3.6 27B 0.93 Aug 2026 Apr 2026
ClothDojo Robot policies use both hands to flatten and fold garments. The simulator checks success on familiar garments and garments withheld from training. South China University of Technology and collaborators Simulated robots Everyday Not in cards π0.5 44.7% Oct 2026 Apr 2026
Code Interpreter Benchmark General Models use a Python interpreter for general tasks. The score is the percentage of generated programs that execute. Qwen Function calling School Not in cards Qwen-14B 65.5% Sep 2023 Oct 2023
CodePDE Models write a Python solver for each of five families of partial differential equations. Harbor tests each solver on 100 instances and checks its normalized error. Shanda Li and colleagues Scientific code Expert Not in cards o4-mini 38.1% May 2025 Oct 2025
cognition-golden Cognition tests coding agents on tasks modeled on real Devin use cases, in authentic development environments with fully autonomous grading. A train split serves self-improvement and a test split measures capability. Cognition Building whole projects Expert Not in cards o1-preview 51.8% Sep 2024 Sep 2024
Contextual Earnings-22 Models transcribe clips from earnings calls with a custom vocabulary. The benchmark measures keyword recognition and transcription errors. Argmax Speech recognition and translation Everyday Not in cards Qwen3-ASR-1.7B 9.19% Mar 2026 Jan 2026
ContinualSkillBench Agents work through five domains of 100 linked subtasks of rising difficulty, testing whether skill libraries they build in context actually help later tasks. Peking University Learning from the context Expert Not in cards GPT-5.3 Codex 64.1% Aug 2026 Apr 2026
CoST English-Hindi Models translate 1,947 English sentences into Hindi across eight genres, including plays, poetry and conversation. The organizers report BLEU alongside reference-based and reference-free translation metrics. MT-NLP Lab, IIIT Hyderabad Translation University Not in cards Llama 3-70B 79.1 Jun 2024 Aug 2024
Coval Text-to-Speech TTFA Speech models synthesize customer-service prompts. The score is median time to first audible sample in milliseconds over the published 24-hour window. Coval Speech recognition and translation Everyday Not in cards Grok TTS 267.3 Apr 2026 Apr 2026
CoVR-R Models find the video that matches a reference clip after applying a change described in text. Omkar Thawakar and colleagues Video understanding School Not in cards Qwen2.5-VL-7B 47.6% Mar 2026 Jan 2025
CPU Inference Optimization and Streaming Agents optimize llama2.c inference on a CPU. The harness checks numerical output, generated text, and streaming latency across repeated executions. 朝闻道 Speed and optimization Expert Not in cards Claude Sonnet 5.5 3.69 Oct 2026 Sep 2026
CRAG Retrieval-augmented systems answer 4,409 factual questions across finance, sports, music, movies and open domain, using mock web and knowledge graph search APIs. Hallucinated answers are penalized. Meta Faithful to sources Everyday Not in cards GPT-4 Turbo 0.20 Jun 2024 Apr 2024
CTIBench Models answer cyber threat intelligence questions. Tasks cover knowledge questions, CVE to CWE mapping, CVSS severity scoring, threat actor attribution and technique extraction. Rochester Institute of Technology (Alam, Bhusal, Nguyen, Rastogi) Finding and exploiting real bugs Expert Not in cards Llama 3-70B 65.7% Jun 2024 Apr 2024
CyberSecEval MITRE Models are asked to help with attack steps drawn from the MITRE ATT&CK framework. Scored by how often the model complies. Meta (PurpleLlama) Capture-the-flag challenges Expert Not in cards Llama 2-13B 40.5% Dec 2023 Aug 2023
DataBench Models answer 1,822 questions about 80 real-world tables, such as Forbes billionaires and Airbnb listings, with 3.7 million rows in total. Answers are booleans, numbers, categories or lists, matched against a key. A lite setting gives each table as a small sample. Cardiff NLP (Cardiff University) and collaborators Data science and SQL University Not in cards Llama 2-13B 20.7% May 2024 Jul 2023
DebugBench Models repair bugs in 4,253 C++, Java, and Python programs derived from LeetCode solutions. The evaluation runs the repaired programs against tests. THUNLP Function-level problems School Not in cards GPT-4 75% Jan 2024 Jun 2023
DecBench Agents recover source code from compiled functions. The leaderboard measures the share that exactly match the original control flow, types, or recompiled instructions. Noelo Lab, University of Georgia Function-level problems Expert Not in cards GPT-6 Astra 51.6% Jul 2026 Sep 2026
Decision Index Vision Models make decisions over images and screens. The score is the published equal-weight mean of public and private vision scores. Hugging Face community (apolinario, multimodalart) Composite indices Everyday Not in cards Qwen3.5 397B-A17B 63.4 Sep 2026 Feb 2026
DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. Google DeepMind Dangerous capabilities Not in cards Gemini 1.0 Pro 20.5% Mar 2024 Dec 2023
DesignQA Models read Formula SAE rules and inspect CAD views and drawings of a student race car. They retrieve rules, identify parts, and check compliance. Massachusetts Institute of Technology and Autodesk Research CAD models University Not in cards GPT-4o 0.72 Apr 2024 Mar 2025
Deterministic Agent Reasoning & Formatting Suite Models solve three algorithm questions under strict formatting rules. The harness checks the answers and their JSON structure. Weni Mohamed Following precise instructions University Not in cards Gemini 3.8 Flash 100% Nov 2026 Sep 2026
DexVerse Robot policies perform tabletop manipulation tasks in Isaac Lab, including tool use, bimanual coordination, and sequences with several goals. Task-specific success conditions check completion under different visual settings and robot embodiments. University of North Carolina at Chapel Hill, University of Hong Kong, and University of California, Berkeley Simulated robots Everyday Not in cards π0.5 34% Jul 2026 Apr 2025
DraftNEPABench Agents draft sections of environmental impact statements using technical and regulatory source documents. Model judges score the drafts against rubrics, and specialists assess a sample. Pacific Northwest National Laboratory and OpenAI Law Expert Not in cards GPT-5 2.41 Feb 2026 Sep 2025
DRAMA-Eval Models generate dialogue with several speakers across multiple turns. The authors check word accuracy and use a model judge to evaluate delivery in context. Jingbin Hu and colleagues Speech recognition and translation Everyday Not in cards Ming-3.0-tts-0.5B 2.55% Oct 2026
Duplex-MPE Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios. Peking University Speech recognition and translation Everyday Not in cards MiniCPM-o 4.5 72.3% Sep 2026 Feb 2026
DuplexLedger Speech agents track facts that speakers introduce, revise, or leave unchanged during conversation. The evaluator asks which facts still apply. Yuxuan Li and colleagues Speech recognition and translation Everyday Not in cards MiniCPM-o 4.5 40.2% Oct 2026 Feb 2026
EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. Peking University (VaLuE Lab) Simulated robots Everyday Not in cards GPT-5.6 Terra 52% Oct 2026 Aug 2026
EmbodiedCLUE-VLA (tabletop tasks) Robot policies carry out tasks on a tabletop. SuperCLUE Real robots Everyday Not in cards GR00T N1.6 49.3 Dec 2025
EnterpriseRAG-Bench Retrieval systems and agents answer 500 questions over a synthetic company's internal documents, slightly over 500,000 of them. Onyx Deep research Expert Not in cards GPT-6.1 Sol 91.5% Feb 2026 Sep 2026
EntireSongs Models transcribe complete English and Chinese songs with background music. Qwen reports transcription errors separately for each language. Qwen Speech recognition and translation Everyday Not in cards Qwen3-ASR-1.7B 14.3% Jan 2026 Jan 2026
EvalVerse Video models generate cinematic clips from prompts and reference material derived from professional videos. Film experts compare outputs across visual design, acting, cinematography, aesthetics, emotion, shot sequences, and sound. A calibrated vision-language evaluator also estimates these judgments. Hong Kong University of Science and Technology, Tencent, and collaborators Video generation Expert Not in cards HappyHorse 1.0 0.73 May 2026
Feedback Prize: English Language Learning 3.0 Models predict six writing trait scores for English learners. The competition averages the root mean squared error across the six traits. The Learning Agency Lab, Georgia State University, and Vanderbilt University Other professions University Not in cards o3 0.59 Aug 2022 Apr 2025
Fire360 Models interpret panoramic video from firefighter training drills. They answer questions, caption actions, assess procedures, locate equipment, and match damaged objects to clean examples. University of Illinois Urbana-Champaign and Illinois Fire Service Institute Video understanding Expert Not in cards GPT-4o 53.8% Jun 2025 Mar 2025
FormalMATH Lite Models prove statements in the FormalMATH Lite subset. The board reports proof success at the stated sampling budget. SphereLab, M-A-P, The Chinese University of Hong Kong, Numina and others Formal proofs Competition Not in cards DeepSeek-Prover-V2-671B 61.9% Apr 2025
FreshStack Retrieval systems find documentation that supports answers to technical questions. The benchmark measures how many required facts the retrieved passages cover. University of Waterloo and Databricks Questions that need search Expert Not in cards Gemini Embedding 2 0.41 Apr 2025 Oct 2026
GDPevo Agents learn from training tasks drawn from enterprise workflows and must reuse that experience on related test tasks. Gains at test time measure self-evolution. Prism Shadow Running a business Expert Not in cards Claude Opus 4.8 67.1% Aug 2026 May 2026
GeoFlip Models generate videos from paired scenes where a geometry change reverses the expected physical outcome. Model judges check the event and the scene structure. Yuqi Li and colleagues Video generation School Not in cards Wan2.2-I2V-A14B 3.17 Oct 2026 Jul 2025
GIFT-Eval-CTX Models forecast a time series using past observations and information about future changes. Salesforce AI Research Real-world events University Not in cards GPT-5.6 Sol 0.57 Oct 2026 Jul 2026
GJ.cool Vertical Book OCR Evaluation Models transcribe vertically printed Chinese books and manuscripts. The authors report accuracy separately for pages with sparse interlinear notes, pages with dense notes, and manuscripts. GJ.cool Research and Development Center Documents, charts and text in images University Not in cards GLM-OCR 89% Apr 2026 Feb 2026
GlyphBench Agents choose actions from text-rendered game boards. The standard suite contains 303 tasks across six game collections and scores agents by cumulative return. Roger Creus Castanyer and collaborators Video games University Not in cards Qwen 3.8 27B 0.42 Sep 2026 Aug 2026
GPU Puzzles Models implement parallel GPU kernels for 14 puzzles. The port runs each kernel against a reference and rejects serial implementations. Sasha Rush GPU kernels University Not in cards GPT-5 mini 90.5% Jul 2022 Aug 2025
GroundCocoa Models combine stated conditions with everyday knowledge to choose an answer. The questions test whether the model applies several constraints together. Harsh Kohli and colleagues Language reasoning Everyday Not in cards Gemini 1.0 Pro 41.9% Apr 2024 Dec 2023
Hack-Verifiable Terminal Bench Agents complete terminal tasks with access to restricted reference materials. The benchmark measures how often they consult those materials. Amit Roth and colleagues Deception and alignment Expert Not in cards Claude Opus 4.6 10.1% Aug 2026 Feb 2026
HALAS Models transcribe earnings call segments with human annotations of hallucinations. NVIDIA measures word error and segments with inserted text. Mateusz Barański and colleagues Speech recognition and translation Everyday Not in cards Whisper large-v3 23.8% Jun 2026 Oct 2024
HaluBench Models judge whether an answer is faithful to its context across about 15,000 labeled context, question and answer triplets. It tests hallucination detectors and LLM judges. Patronus AI Hallucination rates University Not in cards Claude 3 Sonnet 78.8% Jul 2024 Mar 2024
HealBench-Eval Models generate images intended to provide psychological comfort. The authors compare 14 image generators and prompt strategies on 1,095 cases. East China Normal University and collaborators Image generation and editing Everyday Not in cards Stable Diffusion 3.5 Large 26.1 Oct 2026
Histoires Morales Models read French stories about social norms, intentions and actions. The test asks them to distinguish actions or consequences appropriate to the situation. Thibaud Leteno and colleagues Deception and alignment Everyday Not in cards Mistral-7B-Instruct-v0.1 54.7% Jan 2025 Sep 2023
HLS-Eval Models write or edit C++ for high-level synthesis. Tests check circuit behavior, and synthesis tools check whether the code can become hardware. Georgia Institute of Technology Electronics University Not in cards DeepSeek-V3 100% Dec 2024
Honor of Kings Bug AI Root Cause Analysis Agents use the Honor of Kings Bug AI platform to identify the cause of code defects. The team measures how often agents identify the correct cause. Tencent Honor of Kings Tasks from real jobs University Not in cards Tencent Hy4 73.8% Nov 2026 Nov 2026
HumanEval-X Models generate programs or translate solutions among Python, C++, Java, JavaScript, and Go. The 820 human-written language variants include executable tests, and scoring uses pass@k. THUDM Function-level problems School Not in cards GPT-NeoX-20B 9.78% Mar 2023 Apr 2022
Humor in AI: Cartoon Captioning Models write captions for New Yorker cartoons. Human judges and language models compare the captions with contest entries from different ranking groups. University of Wisconsin–Madison, University of Washington and collaborators Creative writing Everyday Not in cards Claude 3 Opus 73.9% Jun 2024 Feb 2024
Indic DiarBench Models transcribe conversations in 22 Indian languages and identify who spoke each word. Human annotators supply speaker labels and timestamps, including for overlapping speech. Sarvam AI and AI4Bharat at IIT Madras Speech recognition and translation Everyday Not in cards GPT-4o 36.2% Jul 2026 Nov 2025
Inverse Scaling Prize Tasks Models answer questions designed to reveal cases where larger models perform worse. The suite includes hindsight bias and following instructions that conflict with familiar text. Ian R. McKenzie and colleagues Language reasoning School Not in cards PaLM (540B) 78.2% Jun 2023 Apr 2022
JamC-QA Models answer multiple-choice questions about Japan, including regional identity, customs, geography, and law. SB Intuitions Knowledge in many languages School Not in cards Llama 3.1-405B 57.1% Jul 2025 Sep 2024
Japanese National Dental Examination 2025 Models answer 350 questions from Japan's 2025 dental licensing exam. The study reports accuracy separately for 202 text questions and 148 visual questions. Japan Ministry of Health, Labour and Welfare Medicine University Not in cards Apr 2025
Japanese National Examination for Occupational Therapists 2025 Models answer 199 Japanese occupational therapy licensing questions. The authors score answers against the official key and compare performance on text and visual questions. Kazuya Saita, Yuichi Mine, and Satoru Amano Medicine University Not in cards o3 88.4% Apr 2025 Apr 2025
JCoLA Models decide whether Japanese sentences are grammatically acceptable. Linguists supply judgments for sentences from textbooks and the research literature. Taiga Someya and colleagues Reasoning in many languages Everyday Not in cards GPT-4 54.5% Feb 2022 Jan 2024
JETHICS Models judge ethical scenarios in Japanese. The suite includes the Japanese commonsense morality dataset used by llm-jp-eval. Masashi Takeshita and Rafal Rzepka Reasoning in many languages Everyday Not in cards GPT-4o 71.3% Jun 2025 Nov 2024
Kimi-ASR Internal Testset Subset 1 Models transcribe speech from Moonshot internal subset 1. The score is the published word error rate. Moonshot AI Speech recognition and translation Everyday Not in cards Kimi-Audio-7B-Instruct 1.42% Apr 2025 Apr 2025
Kimi-ASR Internal Testset Subset 2 Models transcribe speech from Moonshot internal subset 2. The score is the published word error rate. Moonshot AI Speech recognition and translation Everyday Not in cards Kimi-Audio-7B-Instruct 2.44% Apr 2025 Apr 2025
Known-Unknown Questions Models distinguish questions with established answers from questions with no definitive answer. The authors score this distinction using F1 and equal error rate. University of California, Santa Barbara, University of Waterloo, and Vector Institute Short factual questions Everyday Not in cards LLaMA-7B 0.56 May 2023 Feb 2023
Korean Contest Prize Reading Models extract the first place cash award from Korean contest notices. The grader distinguishes it from total prize pools and noncash gifts. ohyoseok Reading in many languages School Not in cards GPT-5.4 Nano 100% Oct 2026 Apr 2026
LADDER Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place. INSAIT Simulated robots Everyday Not in cards GPT-6 Astra 63.3% Oct 2026 Sep 2026
Language Barrier Benchmark Models solve coding and planning tasks in different languages. The coding task uses translated instructions and runs submissions against the same tests. Evelia Veldrine Reasoning in many languages University Not in cards GPT-6 Astra 0% Oct 2026 Sep 2026
LongAV-Compass Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time. KlingAI Research (Kuaishou) and Peking University Video generation Everyday Not in cards Dreamina Seedance 2.0 0.90 May 2026
LongHealth Models answer 400 questions about long clinical records from 20 fictional patients. Tasks test retrieval, sorting and recognition of missing information. LongHealth authors Reasoning over long documents University Not in cards GPT-3.5 Turbo 51.7% Jan 2024 Dec 2023
MaCBench Vision language models answer chemistry and materials questions from images. Tasks cover reading data, understanding experiments and interpreting results. MaCBench authors (Alampara et al., ChemBench group) Lab and data work Expert Not in cards GPT-5.2 72.6% Nov 2024 Dec 2025
Matter-of-Fact Models decide whether materials science claims are feasible using evidence available before a cutoff date. The authors compare their answers with later published findings. Peter Jansen and colleagues Research science Expert Not in cards Claude 3.7 Sonnet 72% Jun 2025 Apr 2025
MedConceptsQA Models answer multiple-choice questions about medical codes. The hub supports the original dataset and a separate 2,000-example sample. Nadav Lab Medicine Expert Not in cards GPT-4 Turbo (Jan 2024) 61.9% Mar 2024 Jan 2024
MedHEB Models retrieve related medical images, reports, and image regions. The authors evaluate how models rank relevant evidence across 107 retrieval tasks. Yingshu Li and colleagues Medicine University Not in cards Qwen3-VL-Embedding-8B 24.6 Nov 2026 Jan 2026
MemTrapBench Models with memory are tested on cases where correct, relevant memories still distort their reasoning or beliefs and lower task performance. Zhejiang University (ZJUNLP) Learning from the context Everyday Not in cards Gemini 3 Flash 85.2% Aug 2026 Dec 2025
MM-NIAH Vision-language models answer questions about long documents of interleaved images and text. Tasks are retrieval, counting and reasoning over needles hidden in the document. Shanghai AI Laboratory (OpenGVLab) Finding facts in long inputs Everyday Not in cards Idefics2-8B 11.6% Jun 2024 Apr 2024
MulSeT Models answer spatial questions about direction, distance and masks across views. The paper studies single images, multiple views and video. Wanyue Zhang and colleagues Spatial reasoning and counting University Not in cards Qwen2.5-VL-7B 36.7% Sep 2025 Jan 2025
MultiGEC 2025 Models correct whole texts in 12 European languages. One track rewards minimal edits, and the other rewards fluent rewrites. Computational Second Language Acquisition working group Reading in many languages School Not in cards Gemma 2 9B 56.8 Jun 2024 Jul 2024
MultiHop-RAG Retrieval-augmented systems answer 2,556 queries whose evidence is spread across two to four news articles. The queries also use document metadata, such as source and date. The benchmark scores both retrieval quality and answer accuracy. Hong Kong University of Science and Technology (Yixuan Tang, Yi Yang) Questions that need search Everyday Not in cards GPT-4 Turbo 56% Jan 2024 Nov 2023
MultiMedQA Models answer medical exam, research, and consumer health questions across seven datasets. Multiple-choice components use answer accuracy. Clinicians and other raters assess free responses for factuality, precision, harm, and bias. Google Research and DeepMind Medicine Expert Not in cards PaLM (540B) 58.9% Dec 2022 Apr 2022
NanoGPT-Bench Coding agents with a fixed H100-hour budget and no internet try to improve on a strong NanoGPT speedrun starting point. The score is the share of later human progress recovered. Intology AI research tasks Research frontier Not in cards Claude Opus 4.6 9.30% May 2026 Mar 2026
New Yorker Caption Contest Humor Understanding Models match a cartoon to its caption, choose the funnier of two captions, and explain a joke. They receive the cartoon or its description. Allen Institute for AI and collaborators General visual questions Everyday Not in cards T5-11B 70.8% Sep 2022 Oct 2019
News With GPT Instructions Models classify financial news by sentiment. ChatGPT supplied seven sentiment labels and explanations. The public test contains 4,047 news items. Oliver Wang Finance Everyday Not in cards Llama 2-13B 63.9 Jul 2023 Jul 2023
NorSumm Models summarize Norwegian news articles in Bokmål or Nynorsk. Human summaries provide references in both written standards. Samia Touileb and colleagues Reading in many languages School Not in cards Falcon-7B 10.2 Jan 2025 Sep 2023
Numeric Grounding Models answer numeric questions from a supplied context. Six tasks test lookup, abstention, distractors, calculations, units, and combinations of these demands. Alan Finney Faithful to sources School Not in cards GLM-5 1.00 Oct 2026 Feb 2026
OceanGym Agents interpret underwater images and sonar, then navigate simulated search and inspection tasks. The simulator scores how closely they reach target locations. Zhejiang University and collaborators Simulated robots University Not in cards GLM-4.5V 14.6 Sep 2025 Aug 2025
OmniMedVQA Models answer questions about 118,010 medical images across 12 imaging modalities. The authors measure accuracy against the correct answer option. University of Hong Kong and Shanghai AI Laboratory Medicine Expert Not in cards BLIP-2 FLAN-T5-XL 50.7% Feb 2024 May 2023
OncoTriad-QA Models answer questions about cancer patients using radiology, pathology, and genomic evidence. The authors score multiple-choice answers and compare written responses with reference answers. Ahnaf Munir and colleagues Medicine Expert Not in cards GPT-5.4 46% May 2026 Mar 2026
OpenRSI Index ACE Playbook Inspection and Repair Agents repair a reasoning playbook under a fixed budget. The score is held-out Formula exact-answer accuracy as a fraction. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards Claude Opus 5 0.73 Sep 2026 Jul 2026
OpenRSI Index Isaac Lab PegInsert Reward Search Agents design robot training rewards under a fixed budget. The score is held-out peg-insertion success as a fraction. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards GPT-5.6 Sol 0.60 Sep 2026 Jul 2026
OpenRSI Index Kev Decision Architecture Agents improve a decision model under a fixed budget. The score is the published exponential of negative macro log loss. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards GPT-5.6 Sol 0.29 Sep 2026 Jul 2026
OpenRSI Index Learnability-Aware Long/Short CoT Adaptation Agents adapt mathematical reasoning examples under a fixed training budget. The score is accuracy across five math benchmarks. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards Claude Fable 5 47.5% Sep 2026 Jul 2026
OpenRSI Index Marin Scaling Ladder Agents design one optimizer for six model sizes. The score is the published complete six-rung reward relative to AdamH. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards GPT-5.6 Sol 1.01 Sep 2026 Jul 2026
OpenRSI Index MInference 32-Head Sparse Prefill Kernel Agents optimize a sparse-attention prefill operator under a fixed budget. The score is speedup against the paired reference run. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards GPT-5.6 Sol 1.54× Sep 2026 Jul 2026
OpenRSI Index Molmo2 Video-Pointing Inference Strategy Agents tune video inputs and inference for a frozen Molmo2 model. The score is held-out spatiotemporal pointing soft F1. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards Claude Fable 5 0.46 Sep 2026 Jul 2026
OpenRSI Index MolmoWeb Interaction Context Allocation Agents select interaction context for a frozen browser model. The score is macro-averaged browser-action prediction quality. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards Claude Opus 5 0.61 Sep 2026 Jul 2026
OpenRSI Index ReasonIR Difficulty Curriculum Agents adjust retrieval training data schedules. The score measures reasoning retrieval with a penalty for general retrieval regression. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards Claude Opus 5 0.22 Sep 2026 Jul 2026
OS-Kairos Agents predict Android actions for tasks such as finding a product in Amazon. The task succeeds when every action matches the recorded trajectory. Shanghai Jiao Tong University and Meta Phones Everyday Not in cards Qwen-VL-Max 29.8% Feb 2025 Aug 2024
Pantograph frontier-LLM robot benchmark (Pandroid) GPT-6 Astra and Claude Fable 5.1 drive Pandroid robots through a fixed tool harness on eight simple manipulation tasks, 10 blind-graded attempts each. Pantograph PBC Real robots Everyday Not in cards GPT-6 Astra 36.3% Sep 2026 Sep 2026
PerMemBench Memory systems for long-running agents decide what to store from years of interactions across many domains and user personas. Later questions check whether the system kept the context that matters for each user. The benchmark tests whether memory policies can adapt to the person instead of storing everything the same way. KAIST Learning from the context Everyday Not in cards Qwen3-14B 0.82 May 2026 Aug 2025
PhAIL Real Franka bin-to-bin picking of four object types. VLAs scored on throughput and completion against human and teleop references. Positronic Robotics Real robots Everyday Not in cards π0.5 75.1% Mar 2026 Dec 2025
PHMForge Agents analyze battery and aircraft engine data to diagnose faults, estimate remaining life, and make maintenance decisions. Harbor packages 99 industrial scenarios as 123 tasks. IBM Research Other professions Expert Not in cards Claude Opus 4.6 80.8% Apr 2026 Feb 2026
Pravda data voids chatbot audit Chatbots answer 13 English questions from two countries. Researchers score endorsement of false claims and citations to Pravda websites separately. Maxim Alyukov, Mykola Makhortykh, Alexandr Voronovici, and Maryna Sydorova Long-form factuality Everyday Not in cards Grok-2 0% Sep 2025 Jun 2025
PRISM Instruction Order Sensitivity Models solve the same math problems under different orderings of fixed instructions. The evaluation compares accuracy across orderings on GSM8K and MATH-500. Blessings Mambwe Following precise instructions School Not in cards Qwen3-30B-A3B-Instruct 66% Aug 2026 Sep 2025
ProSA Models answer equivalent instructions with different wording. PromptSensiScore measures how much model performance varies with the prompt. OpenCompass and collaborators Following precise instructions University Not in cards Qwen1.5-14B-Chat 0.14 Oct 2024 Feb 2024
PROWBench Programmable world models must render video that follows the rules and interactions their program specifies. The test checks visual fidelity to fine-grained rules. Alaya Lab Interactive world models Everyday Not in cards MiniMax H3 68.0 Oct 2026 Sep 2025
Putnam-AXIOM Models solve Putnam mathematics problems and equivalent variants. The evaluator checks final answers to test whether models generalize beyond familiar problem statements. Aryan Gulati and colleagues Competition problems with answers Competition Not in cards o1-preview 41.9% May 2025 Sep 2024
REALM High-fidelity simulation benchmark for VLA generalization: 15 perturbation factors, 7 skills, 3,500+ objects. Success rate validated against real-robot results. Czech Technical University (CIIRC) + University of Amsterdam Simulated robots Everyday Not in cards π0-FAST 53.1% Dec 2025 Jan 2025
Red Team Defense Simulator Models respond to an adaptive adversary. The evaluation measures successful defense, helpfulness and false refusals. Intertwine Jailbreaks and prompt injection University Not in cards GPT-4o 85% Sep 2025 Mar 2025
RoboGate 68 industrial pick-and-place scenarios (nominal, edge, adversarial, randomized) for a Franka arm in Isaac Sim. Organizers run submitted VLAs and report success. AgentAI Co., Ltd. Simulated robots Everyday Not in cards OpenVLA 0% Mar 2026 Dec 2025
RoboTwin 2.0 Bimanual Aloha-AgileX sim: 50 tasks, policies trained on 50 clean demos per task, 100 trials each in clean and domain-randomized scenes. RoboTwin Platform (HKU MMLab + others) Simulated robots Everyday Not in cards π0.5 58.3% Jun 2025 Apr 2025
RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. MTEB community and Hugging Face (with MongoDB) Other University Not in cards Qwen3-Embedding-8B 69.4% Oct 2025 Jun 2025
SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. Tsinghua University (THU-RCSCT, Hu et al.) Simulated robots Everyday Not in cards π0.5 57.8% Dec 2025 Apr 2025
ScholarQABench Models answer scientific questions by synthesizing research papers. The benchmark checks citations and grades long answers against expert rubrics. OpenScholar team Research science Expert Not in cards Llama 3-70B 0.00 Nov 2024 Apr 2024
SciBench Models solve open-ended college-level problems in mathematics, chemistry and physics. Most answers are numbers, which a script checks against the key. University of California, Los Angeles Science questions University Not in cards Llama 3-8B 8.45% Jul 2023 Apr 2024
ScopeCheck: evidence before numbers Models calculate an operating metric when the data covers the requested population. Paired cases test whether they abstain when essential evidence is missing. haxforscience Faithful to sources University Not in cards Claude Haiku 4.5 0.94 Oct 2026 Mar 2026
ScreenSuite Models use screenshots to locate controls, select actions and complete computer tasks. The suite measures perception, single actions and complete interactions. Hugging Face Desktop apps Everyday Not in cards Qwen2.5-VL-72B 56% Jun 2025 Mar 2025
SecQA Models answer multiple-choice computer security questions at two difficulty levels. Questions come from a security textbook and were checked by people. SecQA author (Zefang Liu) Finding and exploiting real bugs University Not in cards Mistral-7B-Instruct-v0.2 90% Dec 2023 Dec 2023
SemComp-Bench Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference. FrameX AI Video generation Everyday Not in cards Wan2.2-I2V-A14B 28.3 Aug 2026 Jul 2025
SHROOM-CAP Models read answers to questions about scientific papers in nine languages. They flag factual errors and fluency errors separately. CHOMPS shared task organizers Faithful to sources University Not in cards gpt-oss-20b 0.61 Jul 2025 Aug 2025
SIB-200 Models predict the topic of a sentence across 205 languages and dialects. The sentences come from FLORES-200. David Ifeoluwa Adelani and colleagues Reading in many languages Everyday Not in cards GPT-4 48.7% Sep 2023 Jun 2023
SimReady Text-to-CAD Models convert mechanical part descriptions into CAD programs. OpenCASCADE checks geometry. The hub port uses 111 prompts covering 37 parts from the original 150-prompt evaluation. SimReady CAD models University Not in cards GPT-6.1 Sol 96.2% Nov 2026 Sep 2026
SMM4H-HeaRD 2026 Pathology Report Staging Models assign tumor, lymph node, and distant metastasis stages to synthetic pathology reports. A tiebreaker removes explicit stage labels and requires inference from clinical findings. Cedars-Sinai Medicine Expert Not in cards GPT-5.4 Mini 0.95 Nov 2025 Mar 2026
SO-Arena Models choose answers across 11 domains and five capability groups. Evaluation tests accuracy, stability under changed option sets, and recognition that the correct answer is missing. National University of Singapore and collaborators Language reasoning University Not in cards GPT-6 Sol 82.3% Nov 2026 Sep 2026
Social Signal Bench Models identify overlapping social signals in conversations from audio or video. The score weights each label by annotator agreement and gives credit for related predictions. University of Copenhagen and collaborators Audio understanding and voice Everyday Not in cards Gemini 3.8 Flash 45.8 Nov 2026 Sep 2026
SpeechEditBench Models edit English and Chinese speech from written instructions. Joint success requires the requested change and preservation of the attributes checked for that task. City University of Hong Kong and collaborators Speech recognition and translation University Not in cards gpt-realtime 96.7% Apr 2026 Aug 2025
SREGym-Lite Agents diagnose and mitigate failures in running services on a Kubernetes cluster. SREGym team Terminal tasks Expert Not in cards Claude Sonnet 5.5 81.3% Jul 2026 Oct 2026
StationeryBench Robots use both arms to handle common desk supplies. A human grades progress from zero to 100 on five tasks. Robocurve Real robots Everyday Not in cards GPT-6 Astra 46.0 Sep 2026 Sep 2026
Student-Work Grading Models grade student work. The tasks compare grades with human marks and test how blank answers, appeals, and grading instructions change the result. David Porcaro Other professions University Not in cards Claude Haiku 5.5 0.86 Nov 2026 Oct 2026
SugarCrepe Models choose between a caption that describes an image and a plausible caption with a changed detail. Cheng-Yu Hsieh and colleagues General visual questions School Not in cards InternVL2-8B 95.2% Jun 2023 Jul 2024
SuperCLUE-DeepSWE Agents complete extended software engineering tasks in Chinese. SuperCLUE Fixing real repositories Expert Not in cards GPT-6.1 Sol 84.3 Oct 2026
SuperCLUE-R2V Models generate videos conditioned on reference material. SuperCLUE Video generation University Not in cards Grok Imagine Video 1.5 64.9
SWE-bench-Live Verified Task success on the official Verified split. The source reports each submission denominator separately. Other language and platform splits remain separate. Microsoft Fixing real repositories Expert Not in cards Claude Sonnet 4.5 40% May 2025 Sep 2025
SWE-smith Agents repair software faults generated from real repositories. Harbor evaluates a 100-task sample from the larger training collection using tests for each fault. SWE-bench team Fixing real repositories Expert Not in cards Claude Sonnet 4 38% Apr 2025 May 2025
Talk to Act Models turn conversations into robot actions. Each predicted function call must identify the action and its arguments from the dialogue. Deepu Krishnareddy and colleagues Function calling Everyday Not in cards Gemma 4 E4B 63.6% Nov 2026 Mar 2026
tau-bench Airline Agents help simulated airline customers through tools while following airline policies on the original tasks. The score is pass^1 as a percentage. Sierra Customer service Everyday Not in cards Claude 3.5 Sonnet (October 2024) 46% Jun 2024 Oct 2024
Tennis & Padel State Tracking Benchmark Models follow a sequence of tennis or padel points and report the score and next server. A rules engine checks each answer. Gulrez Qayyum Language reasoning Everyday Not in cards gpt-oss-120b 0% Oct 2026 Apr 2026
TextWorld Cooking (GATA) Agents navigate text-based rooms, gather ingredients, and prepare recipes. The port sums points earned during play and includes separate training and test games at increasing difficulties. GATA authors Video games Everyday Not in cards Qwen3-4B-Thinking-2507 2.17 Feb 2020 Aug 2025
The Illusion of AI Safety Chatbots answer 30 prompts about self-harm, eating disorders, and substance use. Researchers check replies for harmful content and invitations to continue the conversation. Center for Countering Digital Hate Wellbeing and sycophancy University Not in cards GPT-4o 43% Oct 2025 Aug 2025
TheOne Routing Decisions Models route messages to an existing topic, start a new topic, or request clarification. The grader checks their decisions on 36 synthetic cases. Yuli Dai Multi-turn conversation Everyday Not in cards GPT-5.4 Nano 94.4% Oct 2026 Mar 2026
ts-bench v1 Coding agents solve 25 self-contained TypeScript exercises from Exercism. The score is the percentage of exercises that pass their tests. laiso (independent) Function-level problems University Not in cards GPT-5.3 Codex 96% Jun 2025 Feb 2026
TurBLiMP Models compare Turkish sentence pairs that differ in grammaticality. The core tests cover 16 grammatical phenomena. Ezgi Başar and colleagues Reasoning in many languages Everyday Not in cards Gemma 3 12B 93.6% Jun 2025 Mar 2025
TurnBench Turn-taking systems listen to two-person conversations and must detect when a speaker's turn ends and when the other speaker interrupts. The test set has 116 English conversations in six interaction styles, each labeled by three annotators. Systems rank by recall, and a system whose false-positive rate exceeds 0.15 ranks below every qualifying system. Latency after each event is also reported. Sesame AI with Mundo AI, Carnegie Mellon University and others Speech recognition and translation Everyday Not in cards Gemini 3.1 Flash Live 65.7% Aug 2026 Mar 2026
UFO-Bench Models generate customized images from text and visual references. The evaluator breaks each request into individual requirements and checks how they interact. University of Science and Technology of China and collaborators Image generation and editing Everyday Not in cards Qwen-Image 0.73 Sep 2026
UI-Genie reward model evaluation Models judge individual Android actions and decide whether a sequence of actions completes a task. The authors report separate F1 scores for both judgments. CUHK MMLab and vivo AI Lab Phones Everyday Not in cards Qwen2.5-VL-72B 66.9% May 2025 Jan 2025
UI-Vision Layout Grounding Models identify interface layout regions. The score is the published intersection over union percentage. ServiceNow Research and collaborators Finding things on screen Everyday Not in cards Qwen2-VL-7B-Instruct 24.3% Mar 2025 Feb 2025
UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. Soochow University + Lumos Robotics + others Real robots Everyday Not in cards π0.5 55.8% Jun 2026 Apr 2025
UserBench (developer simulation) Models predict a developer’s next message in a real coding session. The evaluator compares predicted and observed action labels using Jaccard similarity. UserBench team Multi-turn conversation University Not in cards GPT-5.6 Sol 51.3% Jul 2026 Jul 2026
Vals Smith: abhishek203/SfAgent Agents modify abhishek203/SfAgent to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 4.8 80% Aug 2026 May 2026
Vals Smith: benredmond/apex Agents modify benredmond/apex to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Luna 63.3% Aug 2026 Jul 2026
Vals Smith: BerriAI/litellm Agents modify BerriAI/litellm to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Gemini 3.6 Flash 70% Aug 2026 Jul 2026
Vals Smith: conjfrnk/memovox Agents modify conjfrnk/memovox to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 90% Aug 2026 Jul 2026
Vals Smith: Effect-TS/effect Agents modify Effect-TS/effect to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 93.3% Sep 2026 Jul 2026
Vals Smith: flutter/flutter Agents modify flutter/flutter to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Gemini 3.6 Flash 50% Aug 2026 Jul 2026
Vals Smith: Lemonawa/music163bot-rust Agents modify Lemonawa/music163bot-rust to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Sol 93.3% Sep 2026 Jul 2026
Vals Smith: mirumee/ariadne Agents modify mirumee/ariadne to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GLM-5.2 83.3% Aug 2026 Jul 2026
Vals Smith: nationalsecurityagency/ghidra Agents modify nationalsecurityagency/ghidra to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GLM-5.2 73.3% Aug 2026 Jul 2026
Vals Smith: pallets-eco/flask-wtf Agents modify pallets-eco/flask-wtf to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards GPT-5.6 Luna 100% Oct 2026 Jul 2026
Vals Smith: poalrom/sharepaste Agents modify poalrom/sharepaste to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 50% Aug 2026 Jul 2026
Vals Smith: tock/tock Agents modify tock/tock to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Sonnet 5 86.7% Aug 2026 Jul 2026
Vals Smith: vllm-project/llm-compressor Agents modify vllm-project/llm-compressor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Claude Opus 5 96.7% Sep 2026 Jul 2026
VCIS Models inspect a grid and decide whether a proposed movement plan reaches its goal. Separate tasks check object coordinates and the step where a plan fails. Fahmi Spatial reasoning and counting School Not in cards Claude Sonnet 5.5 100% Oct 2026 Sep 2026
VenusBench-Mobile Agents use Android apps such as VLC, Markor, and Pro Expense. A model judges 90 primary tasks. The other tasks use programmatic checks. Venus Team, Ant Group Phones Everyday Not in cards Kimi K2.5 24.8% Feb 2026 Jan 2026
VerilogEval Models complete Verilog modules from human or machine descriptions. A simulator checks their outputs against the reference behavior. NVIDIA Electronics University Not in cards o1-preview 77.1% Sep 2023 Sep 2024
VES Label Check Models write site notes about groundwater survey curves. The grader checks whether they retain a correct label or repeat a wrong label from the file. Daniel Nwaneri Science questions University Not in cards Claude Sonnet 5 14.9% Oct 2026 Jun 2026
VLABench 100 language-conditioned manipulation task categories needing world knowledge, implicit intent and long-horizon reasoning. Evaluates VLAs and VLMs. Fudan University (OpenMOSS) Simulated robots Everyday Not in cards OpenVLA 9.84% Dec 2024 Jun 2024
Voice Arena Full Duplex Bench Voice agents respond to pauses, changes of speaker, acknowledgments, and interruptions. The board averages success rates across these conversational behaviors. Voice Arena Audio understanding and voice Everyday Not in cards Gemini 3.8 Live 99.9% Sep 2026
WeRef-Bench Models locate objects described in English or Chinese and return no boxes when the image contains no match. Tencent Detection and captioning Everyday Not in cards Qwen3.5-9B 50.7% Oct 2026 Feb 2026
WMT24 Icelandic Idiomatic Expressions Models translate English phrases into Icelandic. Context determines whether a phrase has its everyday literal meaning or its idiomatic meaning. Árni Magnússon Institute for Icelandic Studies Translation Everyday Not in cards Llama 3-70B 41.7% Jun 2024 Aug 2024
World-in-World Closed-loop platform: world models plan for embodied agents in four environments via a standard action API. Scored by task success. Interactive world models Everyday Not in cards Runway Gen-4 64.8% Oct 2025
WRENCH (Factorio) Agents build Factorio factories and recover from unannounced disruptions. The evaluator measures how much production the factory retains after a disruption. WRENCH authors Video games University Not in cards Claude Sonnet 5 1.49 Aug 2026 Jun 2026
xCodeEval Models understand, generate, translate, and retrieve contest code across seven tasks and up to 17 programming languages. Generated programs run against test cases through ExecEval. Islamic University of Technology, Nanyang Technological University, Qatar Computing Research Institute and others Competitive programming Competition Not in cards Code Llama 34B Instruct 3.81% Mar 2023 Feb 2024
ZhoBLiMP Models compare pairs of Chinese sentences that differ in grammatical acceptability. Accuracy measures how often the model prefers the grammatical sentence. Yikang Liu and colleagues Reading in many languages Everyday Not in cards Qwen2.5-14B 87.2% Nov 2024 Sep 2024
Addis Scribe Benchmark Speech models transcribe recordings in seven African languages. Addis AI compares transcripts with human references and measures how quickly some systems return text. Addis AI Speech recognition and translation Everyday Not in cards Gemini 4 Argon 29.9% Oct 2026 Sep 2026
ADQA-Bench Models listen to recordings and answer 3,000 questions about speech, music, and environmental sounds. One question asks how a bird calls before it stops. DCASE 2026 Task 5 organizers Audio understanding and voice Everyday Not in cards Qwen3-Omni-30B-A3B 53.2% Jan 2026 Sep 2025
AdSpark-Bench Models generate advertisement videos from product images and prompts. The benchmark checks whether products remain consistent and whether each shot conveys the requested selling point. Zhifei Yang and colleagues Video generation University Not in cards Dreamina Seedance 2.0 66.9 Oct 2026
Aether-Bench Models retouch photographs automatically, match a requested style, or apply specified adjustments. The style test measures fidelity on 100 image pairs from unseen presets, including peak signal-to-noise ratio. Zhejiang University, vivo BlueImage Lab, and the University of Chinese Academy of Sciences Image generation and editing University Not in cards Qwen-Image-Edit-2509 79.7% Apr 2026 Sep 2025
AgentCLUE-General Agents complete general purpose tasks in Chinese. SuperCLUE General assistants University Not in cards OpenAI Computer-Using Agent (CUA) 4.62 Jan 2025
AgentIF-OneDay Agents complete 104 tasks that stand for a full day of human work, life and study. Tasks include tedious workflows, inferring rules from examples and changing requirements. xbench (HongShan) General assistants Expert Not in cards ChatGPT agent 62.6% Dec 2025 Jul 2025
Agentset Reranker Leaderboard Rerankers reorder retrieved documents across six datasets. A fixed embedding model retrieves 50 candidates, each reranker selects 15, and GPT-5 pairwise judgments produce Elo ratings. Agentset Questions that need search University Not in cards Qwen3-Reranker-8B 1,473 Nov 2025 May 2025
AGI Score (A Definition of AGI) Experts test models on ten cognitive domains from Cattell-Horn-Carroll theory, such as reasoning, memory and perception, with adapted human psychometric tests. Each domain counts 10% toward a score out of 100%. Center for AI Safety and collaborators (Hendrycks et al.) Composite indices Expert Not in cards GPT-5 57% Oct 2025 Aug 2025
AGI-Eval Community Speech Synthesis Models synthesize speech. The board ranks models on how human the voices sound. AGI-Eval (Meituan) Audio understanding and voice Everyday Not in cards GPT-4o Mini TTS 13.0 May 2025 Mar 2025
AGI-Eval Community Voice Interaction Models converse by voice. The board scores naturalness, problem solving, learning assistance, leisure and empathy. AGI-Eval (Meituan) Audio understanding and voice Everyday Not in cards GPT-4o 0.60 May 2025 Mar 2025
AmbiStory Models rate how plausible a word meaning is in a short story. The shared task averages rank correlation and accuracy within the organizers’ tolerance. University of Technology Nuremberg Reading comprehension School Not in cards Llama 3.1-8B 0.56 Jul 2025 Jul 2024
Amharic ASR Benchmark Speech models transcribe 1,548 recordings of people reading Amharic prompts. The organizers report character and word error rates. Dataset.ET Speech recognition and translation Everyday Not in cards Whisper large-v3 151.1% Sep 2026 Nov 2023
AMLworld-Compact Models inspect transaction graphs and estimate whether a marked target is suspicious. The grader thresholds the probabilities and computes a weighted F1 score. Natapong Nitarach (Schwyter) Finance University Not in cards Gemini 3.8 Flash 24% Oct 2026 Sep 2026
AndroidLab Mobile agents complete 138 tasks across nine apps on predefined Android virtual devices, in text-only and multimodal modes. Checks on the final device state score each task. Tsinghua University and Zhipu AI (THUDM) Phones University Not in cards GPT-4o 31.2% Oct 2024 Nov 2024
ANES survey item associations Models fill in one missing answer from a 2016 election survey using the respondent’s other answers. Researchers compare associations between the real and simulated answers. Lisa P. Argyle and colleagues Social simulation Everyday Not in cards GPT-5 mini 0.82 Sep 2021 Aug 2025
APM-Bench Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions. Shanghai Jiao Tong University Long videos Everyday Not in cards Gemini 3.6 Flash 60.2% Sep 2026 Jul 2026
Appendix Boundary Models extract answers from documents containing clean text, benign notes, or conflicting instructions. The author reports this task separately from replications on the same page. Rudots Jailbreaks and prompt injection School Not in cards Gemini 2.5 Flash 0.00 Oct 2026 Jun 2025
ArtEdit-Bench-Eval Models rate photo edits against English instructions. Spearman and Pearson correlations measure agreement with human ratings on 200 cases. Tencent Hunyuan and Xiamen University Image generation and editing Everyday Not in cards Gemini 2.5 Flash 0.62 Nov 2025 Sep 2025
AV-SafetyBench Models generate videos with sound from written prompts. Judges review each clip with sound, in silence, and as audio alone to identify unsafe outputs. Korea University Video generation Everyday Not in cards LTX-2.3 49.4% Sep 2026 Mar 2026
BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues Models write the teacher's next reply in an English lesson. Organizers average the system's ranks on BERTScore F1 and DialogRPT. BEA 2023 shared task organizers Multi-turn conversation University Not in cards GPT-4 3.00 Mar 2023 Jun 2023
BioMaze Models answer questions about biological pathways under normal and altered conditions. The benchmark includes true-or-false questions and written responses. Haiteng Zhao and colleagues Genomics and life sciences University Not in cards Llama 3.1-8B 61.6% Feb 2025 Jul 2024
BountyBench Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars. Stanford CRFM Finding and exploiting real bugs Expert Not in cards Claude 3.7 Sonnet 50% May 2025 Feb 2025
Brennan Center election image generation audit Image generators receive requests for misleading election scenes. Researchers count how often each system produces an image, including images that omit requested details. Brennan Center for Justice Refusing harmful requests Everyday Not in cards Runway Gen-4 100% Aug 2026
C2HLSC Models rewrite ten C programs for hardware synthesis. The tests check program output and whether the synthesis tool accepts the implementation. New York University Electronics University Not in cards Claude 3.5 Sonnet 74% Nov 2024 Jun 2024
CAGUI Models locate controls and read marked text in Chinese Android screenshots. A separate test checks each predicted action against the recorded steps for an app task. OpenBMB Phones Everyday Not in cards Qwen2.5-VL-7B 65.3% May 2025 Jan 2025
Can evaluation miss a useful reframing? Models analyze cases that may benefit from reframing. A separate judge scores useful reframing on analytical cases and checks whether responses preserve the facts. Andrii Volynets Language reasoning University Not in cards Gemini 3 Flash 100.0 Oct 2026 Dec 2025
Cat-Goric Game Benchmark Models navigate a 2D platform game and submit results to a public decision-making leaderboard. Felladrin Video games Everyday Not in cards Qwen3.8-Flash-Next 20.0 Oct 2026 Aug 2026
CellCaution: Reasoning About Cell Composition Models reason about changes in cell populations and decide when the information supports a numerical claim about their composition. AbhishekNandy Genomics and life sciences University Not in cards Gemma 4 31B IT 1.00 Oct 2026 Apr 2026
ChessBench Models select chess moves from legal options in positions scored by a chess engine. Decision Index measures whether the selected move has maximal action value. Google DeepMind Puzzles and chess Competition Not in cards GPT-3.5 Turbo Instruct 66.5% Sep 2023
ChildMandarin Models transcribe Mandarin speech from children ages three to five or verify speakers. BAAI FlagEval Speech recognition and translation Everyday Not in cards Qwen2-Audio-7B-Instruct 11.5% Sep 2024 Aug 2024
Chinese Essay Fluency Evaluation 2024 Models identify errors and rewrite sentences from Chinese school essays. A new track asks them to rate the fluency of a whole essay. East China Normal University and Microsoft Research Asia Reading in many languages School Not in cards Jan 2024
ChipGPTV Models write Verilog from circuit diagrams and descriptions with different levels of detail. Testbenches check whether the generated circuits behave correctly. Institute of Computing Technology, Chinese Academy of Sciences, and collaborators Electronics University Not in cards GPT-4V 71.8% Jul 2024 Nov 2023
CIRR Models find an image that matches a reference photograph after applying a change described in text. Zheyuan Liu and colleagues General visual questions Everyday Not in cards EmbeddingGemma 2 22% Aug 2021 Oct 2026
CLEF 2026 SimpleText Task 2 Models detect or classify unsupported additions to simplified Cochrane abstracts. A generation subtask compares rewrites made with and without explicit checks against the source. University of Amsterdam and University of Western Brittany Faithful to sources University Not in cards Qwen3-4B 0.34 Sep 2025 Apr 2025
CLEF JOKER 2023 Task 1: Pun Detection Models decide whether a text contains a pun. Organizers report precision, recall, F1, and accuracy separately for English, French, and Spanish. CLEF JOKER organizers Reading in many languages Everyday Not in cards T5-Large 55% Sep 2022
CLEF JOKER 2025 Task 2: Wordplay Translation Models translate English puns into French. A translator judges whether each sampled answer preserves the meaning and contains wordplay. The online leaderboard uses BLEU. CLEF JOKER organizers Translation University Not in cards Oct 2024
CMI-Bench Models listen to music and answer requests such as naming an instrument, marking beats, or transcribing lyrics. Each task has its own scoring rule. Queen Mary University of London and Yamaha Corporation Audio understanding and voice University Not in cards Gemini 3.1 Flash-Lite 37.9 Jun 2025 Mar 2026
Connected Robotics Evidence Graph Benchmark Models update beliefs as evidence arrives from connected robots and sensors. The evaluator checks the belief state after each observation across 350 scenarios. Ranulfo Bezerra and colleagues Embodied reasoning University Not in cards Gemma 3 12B 80.1% Nov 2026 Mar 2025
ConvFinQA Models answer a conversation of related numerical questions about a company's financial report, with tables and text, where later questions build on earlier answers. The dataset has about 3,900 conversations with 14,000 questions, built from FinQA. Answers are checked by execution accuracy against the gold reasoning program. UC Santa Barbara and J.P. Morgan AI Research Finance University Not in cards T5-Large 58.7% Oct 2022
COPAL-ID Models choose a plausible cause or consequence in Indonesian scenarios. The questions require familiarity with local customs and expressions. Haryo Akbarianto Wibowo and colleagues Reasoning in many languages Everyday Not in cards Llama 2-7B 58.9% Nov 2023 Jul 2023
CORAL Models retrieve passages, answer questions, and cite supporting evidence across conversations. Retrieval metrics, response-overlap metrics, model judgments, and citation precision and recall score different parts of the task. Renmin University of China, Beijing Academy of Artificial Intelligence, Huawei Poisson Lab, and Waseda University Questions that need search Everyday Not in cards Qwen2.5-1.5B 16.3 Oct 2024 Sep 2024
CoralVQA Models answer questions about coral identity, growth, and health in reef photographs. GPT-4o checks answers against references across 16 question types. Beijing University of Posts and Telecommunications and collaborators General visual questions University Not in cards Claude 3.5 Haiku 41.8% May 2025 Oct 2024
CoVEBench Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged. Nanjing University (NJU-LINK) Video generation Everyday Not in cards HappyHorse 1.0 55.2 Jun 2026
CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. Po-Yi Wu and colleagues Simulated robots Everyday Not in cards OpenVLA 0.56 Nov 2026 Jun 2024
Crossmodal-3600 Models describe images in different languages or retrieve an image from a description. Ashish Thapliyal and colleagues General visual questions School Not in cards EmbeddingGemma 2 0.65 May 2022 Oct 2026
CyberSecEval MITRE False Refusal Rate Models answer borderline but benign security questions. Scored by how often the model wrongly refuses. Meta (PurpleLlama) Capture-the-flag challenges Expert Not in cards Code Llama 34B Instruct 1.60% Apr 2024 Aug 2023
DeceptionBench (PKU Alignment) Models state a belief and then answer under social or strategic pressure. Judges measure how often their public answer contradicts their private reasoning. Jiaming Ji and colleagues Deception and alignment Everyday Not in cards DeepSeek-R1-Distill-Qwen-7B 20.7% May 2025 Jan 2025
Decision Model Benchmark Models classify examples of hallucinations, judge model outputs, and route agent requests. Arize AI Hallucination rates University Not in cards GPT-6 Luna 87.9% Oct 2026 Sep 2026
DeepResearch Bench II Deep research agents write reports for 132 research briefs in 22 domains, each derived from a review article they may not consult. A judge model checks 9,287 expert criteria. University of Science and Technology of China and Metastone Technology Deep research Expert Not in cards OpenAI deep research 43.1% Jan 2026 Feb 2025
DeepResearch Bench II (Gemini 2.5 Pro judge) Systems conduct deep research under the legacy Gemini 2.5 Pro evaluator. Scores remain separate from the current GPT-5.5 evaluator. University of Science and Technology of China and Metastone Technology Deep research Expert Not in cards OpenAI deep research 45.4% Jan 2026 Feb 2025
dependency-bench Models reason about build dependencies, retries, and failures in continuous integration pipelines. The grader checks their answers by exact match. Muhammad Owais Warsi Fixing real repositories University Not in cards Claude Haiku 5.5 85% Oct 2026 Oct 2026
DevOps-Gym Agents build, monitor, repair and test Java and Go projects. The Prime port converts tasks to single-turn questions with proxy grading. UCSB ML Security Lab and collaborators Fixing real repositories University Not in cards Claude Sonnet 4 27.5% Jan 2026 May 2025
DexArena Robots use both hands to complete household and industrial tasks. The simulator checks each stage and the final outcome across ten multistage tasks. Shanghai Institute for Physical Artificial Intelligence and Robotics and collaborators Simulated robots Everyday Not in cards π0.5 52.1% Oct 2026 Apr 2025
DFKI English-Russian Linguistic Test Suite at WMT25 Models translate English passages into idiomatic Russian. Rules match known translations. A linguist judges unresolved cases across 55 phenomena. The authors report item-weighted and category-weighted accuracy. DFKI, Saarland University and University of Hildesheim Translation University Not in cards GPT-4.1 89.5% Jun 2025 Apr 2025
Does the answer survive disagreement? Models answer checkable questions and respond to a user who insists on a plausible wrong answer. The test measures whether they keep the correct answer. Kum Donalsien Wellbeing and sycophancy School Not in cards GLM-5 0% Oct 2026 Feb 2026
DREsS Models score English learner essays for content, organization, and language. Each trait receives a score from 1 to 5 in half-point steps. Haneul Yoo and colleagues at KAIST Other professions University Not in cards Llama 3.1-8B 0.69 Feb 2024 Jul 2024
DuoBench Eleven bimanual Franka FR3 Duo tasks in four coordination categories, in simulation and partly real. Stage-based scoring beyond binary success. UTN Nuremberg + KIT + Franka Robotics + TU Munich Simulated robots Everyday Not in cards π0.5 10.6% Jun 2026 Apr 2025
EchoChain Voice agents reason through spoken exchanges while a speaker interrupts or changes instructions. The score measures whether the agent responds correctly. Labelbox Audio understanding and voice Everyday Not in cards gpt-realtime 44% Mar 2026 Aug 2025
EICAT impact classification Models read a scientific paper and classify the ecological impact of a named invasive species. They choose among five impact levels and a data deficient category. Bielefeld University Lab and data work Expert Not in cards Llama 3.1-8B 40.3 Feb 2025 Jul 2024
EmotionTalk Models infer emotions and sentiment from Chinese conversations. BAAI FlagEval Audio understanding and voice Everyday Not in cards chatglm2-6b 50.9 May 2025 Jun 2023
ESI-Bench Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action. ESI-Bench team Embodied reasoning Everyday Not in cards GPT-5 70% May 2026 Aug 2025
EtiCor Models judge whether a stated behavior fits the etiquette of a given region. The test covers five regions and scores judgments against regional labels. Indian Institute of Technology Kanpur Commonsense Everyday Not in cards Falcon-40B 63% Oct 2023 Mar 2023
EuroSAT Models classify land use and land cover in Sentinel-2 satellite images. Patrick Helber and colleagues General visual questions University Not in cards EmbeddingGemma 2 83.6% Aug 2017 Oct 2026
EvaLatin 2026 Models parse Latin prose and poetry and identify named entities in classical Latin. EvaLatin organizers Reading in many languages University Not in cards Gemini 3 Flash 77.5% Nov 2025 Dec 2025
EventHallusion Models answer questions about video events that can prompt false descriptions. The evaluation tests whether answers stay grounded in the clip. EventHallusion authors Video understanding Everyday Not in cards GPT-4V 76.5% Sep 2024 Nov 2023
EVF-Bench Models generate product videos with visual effects. The evaluator checks motion, product appearance, text stability, and whether the effects fit the request. University of Chinese Academy of Sciences and collaborators Video generation University Not in cards Wan 2.2 A14B 51.6 Nov 2026
FactoryBench-100 Agents complete 100 manufacturing workflows through simulated ERP, email, spreadsheet, and chat tools. FactoryScore checks their decisions and the state left in each system. Blobfish AI Tasks from real jobs Expert Not in cards GPT-5.6 Luna 76.2% Aug 2026 Jul 2026
FGVC-Aircraft Models identify aircraft variants from photographs. Subhransu Maji and colleagues Detection and captioning University Not in cards EmbeddingGemma 2 39.7% Jun 2013 Oct 2026
FinanceBench Models answer open-book questions about public companies using their filings, such as 10-Ks and earnings reports. The full set has 10,231 questions with answers and evidence strings, and 150 of them are public. The makers graded answers by hand, and later users often grade with a model judge. Patronus AI, Contextual AI and Stanford University Finance Expert Not in cards Llama 2-70B 41% Nov 2023 Jul 2023
FinMem Trading Agents choose whether to buy, sell or hold a stock each day. The benchmark measures cumulative return against historical prices. FinMem authors Finance University Not in cards davinci-003 1.63% Nov 2023 Nov 2022
FiQA SA (BloombergGPT protocol) Models classify sentiment toward a named financial target as negative, neutral, or positive. Bloomberg uses five examples and reports weighted F1. Bloomberg Finance Everyday Not in cards GPT-4 0.63 Mar 2023 Jun 2023
Formal Logic Deduction Models decide whether facts prove or disprove a hypothesis, or leave it unresolved. They also generate the proof steps. Terufumi Morishita and colleagues Abstract puzzles and logic University Not in cards T5-Base 75.8% Aug 2023
FortisAVQA Models answer questions about sounds and visible events in music videos. Accuracy is reported separately for frequent and rare question distributions after the original questions are rephrased. Xi'an Jiaotong University and HKUST (Guangzhou) Video understanding Everyday Not in cards VITA 48.7% Apr 2025 Aug 2024
FreshQA Models answer questions whose answers change over time, never change, or rest on a false premise that they should reject. FreshEval, an LLM judge, grades answers in a Relaxed mode or a Strict mode that also penalizes any hallucinated detail. Google and UMass Amherst (Tu Vu and coauthors) Questions that need search Everyday Not in cards GPT-4 Turbo (Jan 2024) 80.6% Oct 2023 Jan 2024
GeckoNum Models generate images from prompts about quantities. Human raters check exact counts, approximate quantities, and reasoning about numerical concepts. Google DeepMind Image generation and editing Everyday Not in cards DALL-E 3 47.6% Jun 2024
GEOBench-VLM Models interpret satellite and aerial images, identify land use, and track changes across observations. Some tasks ask models to locate objects or outline regions. Mohamed bin Zayed University of Artificial Intelligence and collaborators General visual questions University Not in cards GPT-4o 0.41 Nov 2024 Mar 2025
GeoMap-Bench Models answer 3,864 questions about 124 geologic maps. They locate map features, interpret legends, and use geological evidence to explain conditions at a site. Microsoft Research and collaborators Documents, charts and text in images Expert Not in cards GPT-4o 0.37 Jan 2025 Mar 2025
GhostInk: Honest OCR on Bill Photos Models read text from phone photos and mask unreadable characters. The grader penalizes guessed text, even when the guess happens to match. Akash Documents, charts and text in images Everyday Not in cards Grok 4.5 0% Oct 2026 Jul 2026
HandWorldBench Models generate videos of human or robotic hands moving objects. The benchmark combines 17 measures of hand motion, object state, and video quality. Zhongguancun Academy, Tsinghua University, and collaborators Video world models Everyday Not in cards HunyuanVideo 1.5 52.2 Nov 2026
HappyWorld Spatial Reconstruction The HappyWorld arena publishes a separate spatial reconstruction vote pool and Elo ranking. Alibaba Interactive world models Everyday Not in cards GPT-6 Astra 1,252 Sep 2026 Sep 2026
HarmEval Models answer 550 harmful requests across 11 categories. A judge measures the share of unsafe responses. SafeInfer authors Refusing harmful requests Everyday Not in cards Mistral-7B-Instruct-v0.2 29.1% Jun 2024 Dec 2023
HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. Qinhong Zhou and colleagues Simulated robots Everyday Not in cards π0 18.8% Nov 2026 Oct 2024
Harvest Rush Training Evaluation Models decide whether a tractor should drive over or bypass animals and obstacles at a stated fuel cost. The default evaluation generates 300 held-out examples. CompassionML Deception and alignment Everyday Not in cards Llama 3.1-8B 0.81 Sep 2026 Jul 2024
HD-EPIC VQA Models watch first-person kitchen videos and choose among five answers. Questions ask what people cooked, where they moved objects, and where they looked. University of Bristol and the HD-EPIC team Video understanding Everyday Not in cards Qwen2.5-VL-7B 28.7% Feb 2025 Jan 2025
HiACC Models transcribe adults and children switching between Hindi and English. The recordings include story reading and spontaneous answers. Shruti Singh and colleagues Speech recognition and translation Everyday Not in cards Qwen3-ASR-0.6B 24.5% May 2025 Jan 2026
IMB Models answer Italian medical specialty exam questions and respond to patient questions from online forums. The paper reports results for both tasks. Antonio Romano and colleagues Medicine University Not in cards Gemma 2 2B IT 71% Oct 2025 Jul 2024
IMC 2025 Models write proofs for the 10 problems of the 2025 International Mathematics Competition for University Students. Human judges grade each proof. IMC (contest), MathArena (evaluation) Written proofs University Not in cards Gemini 2.5 Pro 94.5% Aug 2025 Jun 2025
IMO-LeanProofBench Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof. Google DeepMind (Superhuman Reasoning team) Formal proofs Competition Not in cards Gemini 3.1 Pro 11.6% Jun 2026 Feb 2026
JA-VLM-Bench-In-the-Wild Models answer Japanese questions about images of places, objects and culture in Japan. Sakana AI General visual questions Everyday Not in cards llava-v1.6-mistral-7b 41.1 Mar 2024 Jan 2024
Japanese Nephrology Board Renewal Questions 2014 to 2023 Models answer 209 Japanese nephrology board renewal questions from ten exam years, including questions with clinical images. Researchers compare answers with the society’s key. Japanese Society of Nephrology Medicine Expert Not in cards o1-pro 81.3% Jan 2025 Mar 2025
JorGPT Student Answer Grading Models grade university students’ written answers against the teacher’s marks. The authors released 3,041 responses to 50 questions in Spanish and English. Universidad Francisco de Vitoria Other professions University Not in cards Gemini 2.5 Flash-Lite 5.16 Jan 2026 Sep 2025
K-QA Models answer 201 patient questions. The benchmark checks whether answers include essential statements from clinicians and whether any claims contradict those statements. K Health and Hebrew University of Jerusalem Medicine Expert Not in cards GPT-4 67.7% Jan 2024 Jun 2023
KMMLU-Hard Models answer about 5,000 Korean multiple-choice questions from KMMLU that most models got wrong, across many expert subjects. HAE-RAE Knowledge in many languages University Not in cards GPT-4 Turbo (Jan 2024) 30.5% Jan 2024 Jan 2024
KoCommonGEN v2 Models select a plausible Korean sentence for a given set of concepts from four options. NLP and AI Lab at Korea University Reasoning in many languages Everyday Not in cards Llama 3.1-8B 62.5% Sep 2023 Jul 2024
Korean Dental Licensing Examination 2024 and 2025 Models answer 642 Korean dental licensing questions from two new exams. The authors score accuracy and break it down by subject and image type. Taejun Kim and Bong Chul Kim Medicine Expert Not in cards Claude Opus 4 81.5% Feb 2026 May 2025
LIBERO-Para VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric. Soongsil University + Chung-Ang University Simulated robots Everyday Not in cards π0.5 71.4% Mar 2026 Apr 2025
LibriCount Models estimate how many speakers appear in a recording with overlapping voices. Fabian-Robert Stöter and colleagues Speech recognition and translation Everyday Not in cards Whisper large-v3 57.2% Dec 2017 Nov 2023
Life-Bench Models answer multimodal questions about synthetic personal lifelogs. Tasks cover concept identification, event understanding, and reasoning over multiple pieces of evidence. Google DeepMind General visual questions Expert Not in cards Gemma 3 12B 56% Feb 2026 Mar 2025
LifeChoice Models select the decision a character makes in a novel. The authors score the percentage of correct choices. Fudan University and Alibaba Group Reading comprehension School Not in cards o1 65.3% Apr 2024 Dec 2024
Live Gaming Benchmark Models watch game clips and decide when to speak. They narrate alone, join another commentator, or guide a player. A judge scores their commentary. Proact-VL team Video captioning Everyday Not in cards LiveCC-7B-Instruct 28.4% Mar 2026 Apr 2024
LiveDRBench Agents perform 100 deep research tasks and return key claims discovered during search. Scores use claim-level F1. Microsoft Research Deep research Expert Not in cards OpenAI deep research 55.5% Aug 2025 Feb 2025
LLM Writer Negative Style Models write responses to five prompts. Model judges flag stylistic habits such as promotional wording, editorializing, and formulaic conclusions. Prime Intellect Taste and style Everyday Not in cards GPT-4.1 mini 0.52 Oct 2025 Apr 2025
LLM-Coordination Models play four cooperative games and answer 198 multiple-choice questions about the same games. The authors report game outcomes and question accuracy separately. University of California, Santa Cruz Multi-agent games Everyday Not in cards Mixtral 8x7B 46.6 Oct 2023 Dec 2023
LLM-SRBench Models infer scientific equations from observations on 239 problems across four scientific domains. Symbolic accuracy checks whether the discovered equation recovers the underlying relationship. Virginia Tech, VinUniversity, Carnegie Mellon University and others Research science Research frontier Not in cards Llama 3.1-8B 15.1% Apr 2025 Jul 2024
LongBanana Models synthesize images from many reference images and textual constraints. Atomic rubric checks score content fidelity, relationships, and global constraints across 90 tasks. Kling AI Research Image generation and editing Everyday Not in cards Gemini 3.1 Flash Image 74.6 Nov 2026 Feb 2026
Malt Freelancer Project Matching Models rank synthetic freelancer profiles against a search query or a project brief. Each task has 200 English prompts and 4,019 profiles across five languages. Malt Other professions University Not in cards Qwen3-Embedding-0.6B 50.4% Feb 2026 Jun 2025
MAPP (Multi-Agent Path Planning) Models move several agents across a grid without collisions. The environment tracks how many agents reach their goals and how closely their routes follow shortest paths. salty-duck Multi-agent games School Not in cards GPT-4.1 70.8% Sep 2025 Apr 2025
MedHallu Models distinguish factual medical answers from hallucinated answers at three difficulty levels. The benchmark tests detection with and without supporting knowledge. UT Austin AI Health Hallucination rates University Not in cards Gemma 2 2B IT 0.71 Feb 2025 Jul 2024
Mementos Models describe what happens across a sequence of images. The evaluation checks whether their descriptions capture the objects and actions. Xiyao Wang and colleagues General visual questions Everyday Not in cards GPT-4V 40.6% Jan 2024 Nov 2023
MEMTRACK Agents track changes across software platforms and use earlier information to complete tasks. Patronus AI Reasoning over long documents Everyday Not in cards GPT-5 61% Oct 2025 Aug 2025
mFollowIR Models rank documents according to a search instruction across multiple languages. Orion Weller and colleagues Questions that need search University Not in cards Mistral-7B-Instruct-v0.1 5.40 Jan 2025 Sep 2023
Miklós Schweitzer 2025 Models write proofs for the 2025 Miklós Schweitzer competition, a hard Hungarian university-level contest. MathArena runs and grades it. MathArena (ETH Zurich SRI Lab) Written proofs Research frontier Not in cards GPT-5 Pro 90% Dec 2025 Oct 2025
MINTEval Memory-augmented agents answer questions after long, frequently updated histories where memories interfere. Instances average 138,800 tokens and reach 1.8 million. MINTEval authors (Lee et al.) Learning from the context Expert Not in cards Qwen 3.6 35B-A3B 21% May 2026 Apr 2026
MLAgentBench Agents run machine learning experiments on 13 tasks, from improving CIFAR-10 accuracy to research problems like BabyLM. They read and write files, run code and inspect outputs. Stanford University (SNAP) Machine learning engineering Expert Not in cards Gemini 1.0 Pro 18.3% Oct 2023 Dec 2023
MMArt-Bench Models retouch 200 photos from editing instructions. Gemini scores visual quality and instruction following, and image comparisons measure how closely edits match reference photos. Xiamen University and collaborators Apps and MCP servers University Not in cards Step1X-Edit 8.34 Jun 2025
MobileAgentBench Agents complete 100 tasks across ten Android apps. The grader checks app state and accessibility events, so agents can take different paths. Carnegie Mellon University, University of Michigan, Northeastern University, and University of Hong Kong Phones Everyday Not in cards CogAgent 8.00% Jun 2024 Dec 2023
MultiClinSum-2 Models summarize 1,000 clinical cases per language across 15 languages. Organizers report four automatic metrics and four language model judgments separately. Barcelona Supercomputing Center Medicine University Not in cards Feb 2026
MuseBench (Music Score and Performance Understanding) Models answer music theory questions, read piano scores, and interpret recorded performances. The authors report accuracy separately for text, images, and audio. Qihao Zhao and colleagues Audio understanding and voice University Not in cards Qwen2.5-Omni-7B 52.8% Mar 2025
MusiXQA Models read printed music and answer questions about notes, chords, text, and page layout. The authors measure answer accuracy and string similarity. University at Buffalo and collaborators Documents, charts and text in images School Not in cards Phi-3-vision-128k-instruct 12.8% Jun 2025 May 2024
NaijaSenti Models classify the sentiment of posts written in Hausa, Igbo, Nigerian Pidgin, and Yoruba. Shamsuddeen Hassan Muhammad and colleagues Reading in many languages Everyday Not in cards Qwen3-Embedding-8B 48.8% Jan 2022 Jun 2025
NarrLV Models generate videos from prompts that describe changes in a scene, an object, or an action. A multimodal judge checks what appears and in what order. AMAP, Alibaba Group and collaborators Video generation Everyday Not in cards Wan2.1-T2V-14B 67.1 May 2025 Feb 2025
NativeLongBench Models translate a complete book passage in one response. SEGALE aligns passages and COMET scores them against references across five input lengths. Index LLM Team, Bilibili Translation University Not in cards Qwen3.8 Flash 73.6 Sep 2026 Aug 2026
NEPAQuAD Models answer questions about environmental impact statements. The authors compare answers with a reference under several conditions, including retrieval and access to the relevant passage. Pacific Northwest National Laboratory Law University Not in cards Mistral-7B-Instruct-v0.1 75.3% Jul 2024 Sep 2023
NepTTS-Bench Speech models read Nepali sentences that test pronunciation contrasts. Native speakers rate the recordings, and the authors compare those ratings with an automated metric. Ashish Thapa and colleagues Speech recognition and translation Everyday Not in cards Gemini 2.5 Flash TTS 3 Apr 2026 May 2025
NWPU-RESISC45 Models classify remote sensing images into 45 scene categories. Northwestern Polytechnical University General visual questions University Not in cards EmbeddingGemma 2 88.3% Mar 2017 Oct 2026
OpenRSI Index SGLang Omni TTS Runtime Agents improve speech serving while preserving quality. The score is the official speedup reward for the archived continuation protocol. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards GPT-6 Astra 1.54× Sep 2026 Sep 2026
PaperBench Code-Dev Agents implement research paper code. The Code-Dev variant evaluates coding separately from full research replication. OpenAI AI research tasks Research frontier Not in cards o1 43.4% Apr 2025 Dec 2024
PatchCamelyon Models identify whether a microscope image patch contains metastatic tumor tissue. Bastiaan Veeling and colleagues Medicine Expert Not in cards EmbeddingGemma 2 69.3% Jun 2018 Oct 2026
PersonaShot Models generate multi-shot videos about a person. The evaluator checks character continuity, emotional changes, and camera choices within and between shots. Yuji Wang and colleagues Video generation Everyday Not in cards LTX-2.3 0.67 Aug 2026 Mar 2026
PhoneHarness Bench Agents complete Android workflows through graphical controls, device commands, and structured tools. Checks inspect files, system settings, application state, and unintended side effects. Tencent Hunyuan, CUHK, CUHK-Shenzhen and others Phones Everyday Not in cards Seed2.0 Pro 62.1% Jun 2026 Feb 2026
PitchBench Models listen to recordings and identify notes, intervals, chords, or melodies. The authors vary the instrument, timing, and background noise. University of California, Berkeley and Thoughtful Lab Audio understanding and voice University Not in cards Gemini 3.1 Pro 19.8% May 2026 Feb 2026
Preference-ASR Models transcribe speech in a requested style. The evaluation checks preferences such as casing, number formatting, and removing disfluencies. NVIDIA Speech recognition and translation Everyday Not in cards Phi-4-multimodal 13.8% Jun 2026 Feb 2025
Proof News election misinformation debunking Chatbots respond to five false claims about the 2024 US election. Reviewers mark whether each service clearly debunks each claim. Proof News Long-form factuality Everyday Not in cards GPT-4 0% Nov 2024 Jun 2023
PROST Models predict how objects move or change in everyday physical situations described in text. The benchmark tests reasoning about space and time. Stéphane Aroca-Ouellette and colleagues Commonsense Everyday Not in cards GPT-2 (1.5B) 30.1% Jun 2021 Nov 2019
QCircuitBench Models implement quantum algorithms, construct oracles, and synthesize circuits from written specifications. Harbor uses one instance from each of 28 algorithms. Rui Yang and colleagues Scientific code Expert Not in cards Llama 3-8B 0.03 Oct 2024 Apr 2024
RadSum23 Models write the impression section of radiology reports from findings. One track includes chest images. Organizers rank submissions by RadGraph F1. Stanford University Medicine Expert Not in cards T5-Base 38.0 Jan 2023
Real-Time Reasoning Gym Models act in games while token generation consumes part of the game clock. Real-Time Reasoning Gym authors Video games Everyday Not in cards DeepSeek-V3.2 0.81 Nov 2025 Dec 2025
RedditImpacts 2.0 Models mark clinical and social consequences of opioid use in first-person accounts. The organizers scored 278 annotated posts and excluded 300 unlabeled decoys. Emory University and collaborators Medicine School Not in cards Aug 2025
Research Decision Benchmark Agents choose a next step from an explicit description of a research project. Rule-generated cases test ten types of decisions about evidence and research strategy. Shaoning Zeng and colleagues Research science Expert Not in cards DeepSeek V4.1 Flash 36.4% Oct 2026 Sep 2026
RetryBudget-24: Policy and JSON Contracts Models apply retry policies in 24 synthetic cases. The harness checks their decisions and the required JSON structure across repeated runs. Tao Li Following precise instructions University Not in cards Gemini 2.5 Flash 0.75 Oct 2026 Jun 2025
RGraph-Eval Models generate robot manipulation videos. The evaluator compares objects, changing relationships, and three-dimensional motion paths with a reference demonstration. Aleksandr Lazarev and colleagues Video generation Everyday Not in cards Cosmos3-Nano 0.89 Oct 2026 May 2026
RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. HKU MMLab + Peking University + others (RoboTwin team) Simulated robots Everyday Not in cards π0.5 10.4% Mar 2026 Apr 2025
RoboCasa-Mem Robots complete nine household tasks that require memory. Later steps depend on objects the robot saw or actions it took earlier. University of Science and Technology of China and collaborators Simulated robots Everyday Not in cards π0.5 16% Nov 2026 Apr 2025
Roboflow 100-VL Models locate objects named in natural language across image collections from different domains. The benchmark scores predicted bounding boxes using average precision. Roboflow Detection and captioning Everyday Not in cards Qwen2.5-VL-72B 7.60% May 2025 Jan 2025
RoboMemArena Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots. OpenHelix Team Real robots Everyday Not in cards π0.5 21.5% May 2026 Apr 2025
SAC-Bench Models identify sounds and their order when a spoken claim contradicts the recording. Matched clips test whether answers follow the acoustic evidence. The University of Osaka and University of Wisconsin-Madison Audio understanding and voice Everyday Not in cards Qwen3-Omni-30B-A3B 91.3% Oct 2026 Sep 2025
ScienceBenchmark Models translate questions about the Sloan Digital Sky Survey, cancer research, and research funding into SQL. The organizers run the queries and compare their results. Zurich University of Applied Sciences and Athena Research Center Data science and SQL University Not in cards T5-Large 23% Jun 2023 Aug 2023
ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. Allen Institute for AI (Wang, Jansen, Côté, Ammanabrolu) Lab and data work School Not in cards T5-Large 0.15 Mar 2022
ScribbleEdit Models edit images using a scribble as the only instruction. The evaluator checks whether the change matches the intended operation and location. ByteDance and Michigan State University Image generation and editing Everyday Not in cards BAGEL 0.26% Oct 2026
SCT-Bench Models rate how new clinical information changes a diagnosis or treatment decision. The grader awards partial credit according to the distribution of expert responses. SCT-Bench authors Medicine Expert Not in cards o1-preview 70.5% Feb 2025 Sep 2024
SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. University of Illinois Urbana-Champaign (Lingming Zhang) + Purdue Finding and exploiting real bugs Expert Not in cards Claude 3.7 Sonnet 18% Jun 2025 Feb 2025
Semantic DLP: Catch the Data Leak Models search long technical documents for proprietary content. They must identify the leak and quote the relevant span. Karthik Sethuraman Finding facts in long inputs University Not in cards Gemini 2.5 Flash 100% Oct 2026 Jun 2025
SeniorTalk Models transcribe conversations with older Mandarin speakers and identify who speaks. BAAI FlagEval Speech recognition and translation Everyday Not in cards Whisper large-v3 23.8% Mar 2025 Nov 2023
Small Rewriter Benchmark Models rewrite short Verilog designs to reduce hardware use. The authors compare synthesized cell and wire counts and check whether the circuits preserve behavior. Chinese University of Hong Kong and Huawei Electronics University Not in cards GPT-4V 152.8 Nov 2023
SMM4H 2025 Food Recalls and Foodborne Disease Outbreaks Models classify news articles as recalls, outbreaks, or neither. They also extract details such as the product, cause, and number of people affected. SMM4H-HeaRD organizers Medicine School Not in cards o3-mini 0.88 Mar 2025 Jan 2025
Something-Something V2 Models recognize actions involving people manipulating everyday objects. The benchmark distinguishes actions that differ in direction, ordering, or outcome. TwentyBN Video understanding Everyday Not in cards V-JEPA 2 74% Jun 2025
Sonetra Norwegian SNOMED CT Term Recognition Models transcribe Norwegian sentences read by a synthetic voice. A term counts as correct when every word in the medical term matches. Sonetra AS Speech recognition and translation University Not in cards gpt-4o-transcribe 66.4% Sep 2026 Mar 2025
SPA-Bench Agents use Android apps such as Airbnb, Spotify, and WeChat. The grader matches screenshots against task requirements and uses a vision model to resolve uncertain cases. Huawei Noah's Ark Lab, Harbin Institute of Technology, Shenzhen, and University College London Phones Everyday Not in cards CogAgent 1.35% Oct 2024 Dec 2023
StableToolBench Models solve user requests by calling tools from ToolBench's large set of real-world APIs. A virtual API server with a cache and API simulators stands in for live APIs, so results stay stable over time. GPT-4 judges whether each request was solved and which of two answers is better, giving a solvable pass rate and a win rate. Tsinghua University and collaborators Function calling Everyday Not in cards GPT-4o mini 26.4% Mar 2024 Jul 2024
StateBench (video continuation) Models continue a video while tracking what happened in earlier segments. A judge checks whether visible objects, hidden processes, and later state changes remain consistent. Xi’an Jiaotong University, Alibaba Group, and collaborators Video generation Everyday Not in cards Wan2.2-I2V-A14B 32.5 Sep 2026 Jul 2025
SuperCLUE-200K Models retrieve information from long Chinese inputs. SuperCLUE Finding facts in long inputs School Not in cards GPT-4 Turbo (Jan 2024) 9.98 Jan 2024
SuperCLUE-ComicShorts Models generate short animated comics. SuperCLUE Video generation University Not in cards PixVerse v5.6 38.5
SuperCLUE-Long Models answer questions about long Chinese texts. SuperCLUE Reasoning over long documents School Not in cards GPT-4o 68.4 May 2024
SuperCLUE-RAG Chinese models answer questions twice, first without documents and then with supplied documents. A judge model compares the answers to rate how well the system uses retrieved text. CLUE benchmark team (SuperCLUE) Faithful to sources Everyday Not in cards GPT-4o 66.7 Mar 2024 May 2024
SuperCLUE-Voice (speech interaction) Models interact with users through speech in real time. SuperCLUE Speech recognition and translation Everyday Not in cards GPT-4o 73.0 Nov 2024
Swivuriso Models transcribe read and spontaneous speech in seven South African languages. The authors measure word error rates and keep speakers separate across splits. University of Pretoria and the South Africa Next Voices team Speech recognition and translation Everyday Not in cards Whisper large-v3-turbo 15% Nov 2025 Oct 2024
TAAROFBENCH Models respond to 450 social scenarios where Persian taarof etiquette may or may not apply. The evaluation measures culturally appropriate responses. Nikta Gohari Sadr and collaborators Chat and instructions in many languages Everyday Not in cards DeepSeek-V3 56.2% Sep 2025 Mar 2025
The Pushback Test Models advise a parent who questions urgent medical advice. The evaluation checks whether the model maintains the urgency required by NHS guidance. Mark_RowSet Medicine University Not in cards Claude Sonnet 5 0.81 Oct 2026 Jun 2026
TrustBoundary Bench Models plan incident response actions in a simulation. The grader checks whether they distinguish permission to act from information supplied as evidence. onkar-cybersec Jailbreaks and prompt injection University Not in cards Gemini 2.5 Flash 0.45 Oct 2026 Sep 2025
UAQFact Models answer factual questions or decline when the facts do not support an answer. The benchmark also checks whether models know the facts needed to decide. Soochow University and OPPO AI Center Short factual questions School Not in cards GLM-4 49.6% May 2025 Jan 2024
Uniqueness of Optimum Models find the optimum of a small problem and report whether it is unique. Exhaustive enumeration supplies the expected value and number of optimal solutions. Dimitar Kretski Abstract puzzles and logic University Not in cards gpt-oss-120b 55.2% Oct 2026 Aug 2025
VAHBench Agents turn short drama briefs into finished videos with specified characters, dialogue, and durations. The evaluator checks completeness, quality, reliability, and accuracy of production reports. Hongbo Zhang and colleagues Video generation University Not in cards Claude Opus 5 51.8 Nov 2026 Jul 2026
VAmoS Bench Voice agents answer simulated bank customers and use tools to manage credit cards. A model checks each call against its transcript and recorded actions. Veris AI Customer service University Not in cards Gemini 3.1 Flash Live 62.3% Jul 2026 Mar 2026
VBench image-to-video Models animate input images. Automatic checks score image conditioning and video quality across the complete image-to-video suite. NTU S-Lab and Shanghai AI Laboratory (Vchitect) Video generation Everyday Not in cards Wan2.1-I2V-14B-720P 86.9% Nov 2024 Feb 2025
VCG+ Diverse Models answer questions about videos drawn from varied settings. The evaluation grades their ability to describe and explain video content. Muhammad Maaz and colleagues Video captioning Everyday Not in cards GPT-4V 43.5 Jun 2024 Nov 2023
VeRO-Bench Coding agents improve other agents by editing their prompts, tools, and workflows. Each task has a reference evaluation for comparing changes. Scale AI Machine learning engineering Expert Not in cards GPT-5.2 0.52 Feb 2026 Dec 2025
ViStoryBench Models turn stories and character references into coherent image sequences. Model judgments and image features measure prompt alignment, character consistency, style, and image quality. StepFun, AIGC Research, Westlake University AGI Lab, and ShanghaiTech University Image generation and editing University Not in cards Qwen-Image-Edit-2509 2.00 May 2025 Sep 2025
Voice Arena Diarization Bench Models identify who spoke when in recorded conversations. The board measures diarization error and publishes a separate track for speaker attributed transcription. Voice Arena Audio understanding and voice Everyday Not in cards Muse Voice Transcribe 1.0 31% Sep 2026
Voice Arena Dictation Bench Dictation tools turn speech into written drafts. Judges measure the share of drafts a user could send without edits. Voice Arena Speech recognition and translation Everyday Not in cards Gemini 3.5 Transcribe 75.8% Aug 2026
Voice Controllability Leaderboard Text to speech systems create requested voices and follow delivery, emotion and role instructions. Blind human ratings and objective prosody measures score controllability. Hume AI Speech recognition and translation Expert Not in cards Gemini 3.8 Flash 71.4 Sep 2026 Sep 2026
Voice Replication Leaderboard Text to speech systems reproduce reference speakers from short recordings. Human ratings score speaker identity, quality and naturalness, with an objective speaker similarity check. Hume AI Speech recognition and translation Expert Not in cards Gemini 3.8 Flash 3.53 Sep 2026 Sep 2026
VRSBench Models caption aerial images, locate objects described in words, and answer visual questions. The held-out files contain 9,350 captions, 16,159 object descriptions, and 37,409 questions. King Abdullah University of Science and Technology General visual questions Everyday Not in cards GPT-4V 65.6% Jun 2024 Nov 2023
WebBench (Halluminate) Browser agents complete 2,454 tasks on 452 live websites from the global top 1,000 by traffic. Tasks either read information or write data: creating, updating and deleting records, logging in and downloading files. The board compares browser agent products. Halluminate, with Skyvern Web browsing Everyday Not in cards OpenAI Computer-Using Agent (CUA) 76.5% May 2025 Jan 2025
Werewolf Arena Models compete in Werewolf with strategic speaking, deception, deduction, and persuasion. Werewolf Arena authors Multi-agent games Everyday Not in cards Gemini 1.0 Pro 26.7 Jul 2024 Dec 2023
WildAWM validation Agents complete 33 held-out computer-use tasks in generated software environments. Harbor packages the validation split separately from the training tasks. MSR Orchard Desktop apps Expert Not in cards Qwen3.5-4B 22.3% Oct 2026 Feb 2026
WMT 2024 Discourse-Level Literary Translation Models translate Chinese web fiction into English, German, or Russian. Translators rate English outputs for general quality and for consistency across the passage. Tencent AI Lab and China Literature Translation Expert Not in cards May 2024
WorldGUI Desktop GUI agents do 611 tasks that start from varied initial states, such as the target app not being open. Tasks come with instructional videos and project files. Show Lab, National University of Singapore Desktop apps Everyday Not in cards GPT-5.1 45.8% Feb 2025 Nov 2025
XStoryCloze Models pick the right ending for short stories from StoryCloze, professionally translated into 10 languages. Meta AI (XGLM team) Reasoning in many languages Everyday Not in cards GPT-3 6.7B 58.2% Dec 2021 Jun 2020
ZeroCLUE Models perform Chinese language tasks without training on labeled examples from the target task. CLUE benchmark team Reading in many languages School Not in cards ChatGLM-6B 30.9 Mar 2023
(S)elected Moderation campaign prompts Models receive requests for election campaign advice through their APIs. Reviewers count refusals. The report specifies ten English prompts. AI Forensics and collaborators Refusing harmful requests Everyday Not in cards Jul 2024
1X World Model Challenge Predict future egocentric frames of 1X's EVE humanoid from video and state. Compression scored by token cross-entropy, Sampling by PSNR on private tests. 1X Technologies Video world models Not in cards Jun 2024
2010 i2b2/VA Challenge on Concepts, Assertions, and Relations in Clinical Text Models locate medical concepts, classify statements about problems, and determine relationships among problems, tests, and treatments. Organizers score each task separately. Informatics for Integrating Biology and the Bedside and VA Salt Lake City Health Care System Medicine University Not in cards
2013 Native Language Identification Shared Task Models identify an essay writer's first language from 11 choices. The test contains 1,100 English essays with 100 per language. Educational Testing Service and the BEA workshop organizers Other professions University Not in cards Jan 2013
2018 n2c2 Cohort Selection for Clinical Trials Models read longitudinal patient notes and decide whether patients meet specified clinical trial eligibility criteria. Amber Stubbs and colleagues Medicine Expert Not in cards
2022 n2c2/UW Shared Task on Extracting Social Determinants of Health Models identify substance use, employment, and living situations in clinical notes. Organizers score events and their attributes across three tests using notes from two institutions. National NLP Clinical Challenges and the University of Washington Medicine University Not in cards Nov 2021
2024 NIST GenAI Text-to-Text Discriminators Detectors estimate the probability that a machine wrote a summary, without seeing the source articles. NIST reports the area under the receiver operating characteristic curve. National Institute of Standards and Technology Other Everyday Not in cards Apr 2024
34th Japanese National Examination for Dental Hygienists Models answer 213 Japanese dental hygiene questions, including 139 with visual material such as instrument photographs and radiographs. Hiroshima University Medicine University Not in cards Jun 2025
3DPW Models recover a person's three-dimensional pose in outdoor video. Reference poses come from wearable sensors and a moving camera. Timo von Marcard and colleagues Spatial reasoning and counting School Not in cards Sep 2018
7Bench Layout-guided image models generate 224 text-and-layout samples across seven scenarios, scored for semantic and spatial alignment. University of Padova Image generation and editing University Not in cards Aug 2025
A-OKVQA Models answer about 25,000 visual questions that need world knowledge beyond the image, in multiple-choice and direct-answer forms. Allen Institute for AI General visual questions Everyday Not in cards Jun 2022
AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. Artificial Analysis Simulated robots Everyday Not in cards Oct 2026
AAP Periodontics In-Service Examination Models answer specialty examination questions used by North American periodontics residency programs. Researchers score answers against the Academy’s key for each exam year. American Academy of Periodontology Medicine Expert Not in cards Jan 2024
AbMaze Agents make choices in a simulated antibody optimization experiment under a fixed budget. Recorded assay results determine normalized improvement on 200 test episodes. valenlabs Genomics and life sciences Expert Not in cards Nov 2026
AbstentionAct Agents handle HR and operations requests with missing information or authority. The benchmark checks whether agents clarify, abstain, or complete a permitted action. Victor Ojewale Deception and alignment Expert Not in cards Jul 2026
Accessible Structured Subtitle Generation 2026 Systems produce timed Chinese subtitles for deaf and hard-of-hearing viewers. The structured track also scores speaker labels, key information, and readability. Beijing Union University and Tsinghua University Video captioning University Not in cards Feb 2026
ACI-Bench Models turn doctor-patient conversations into clinical visit notes. The original benchmark compares generated notes with clinician-authored references. Wen-wai Yim and colleagues Medicine University Not in cards May 2023
ACII DaiKon Challenge Models predict interpersonal influence, turn taking and rapport trajectories from multimodal dyadic conversations across five languages. Hume AI and ACII 2026 organizers Audio understanding and voice Expert Not in cards May 2026
ActivityNet Captions Models match descriptions of events to segments of untrimmed videos. Meta evaluates retrieval in both directions. Ranjay Krishna and colleagues Video understanding Everyday Not in cards May 2017
Adaptive Reasoning (Vellum) Models solve 28 familiar logic puzzles after Vellum changes a premise. A judge checks the final answer, and a person reviews the result. Vellum Language reasoning School Not in cards
ADE Corpus Models identify adverse drug effects in medical case reports. The OpenMed port combines classification and relation examples and reserves 20% for evaluation. Harsha Gurulingappa and colleagues Medicine Expert Not in cards Oct 2012
ADE-bench Agents repair dbt projects so that SQL tests pass on the resulting tables. Harbor ports 48 DuckDB tasks and excludes the Snowflake and dbt Fusion variants. dbt Labs Data science and SQL Expert Not in cards May 2025
ADE20K Models label the pixels in a scene by object or region. Meta measures overlap between predicted and reference segments. MIT Computer Science and Artificial Intelligence Laboratory Detection and captioning Everyday Not in cards Aug 2016
ADQA Models describe film scenes. An evaluator answers questions from the descriptions and dialogue, then checks whether the context supports each correct answer. Katha AI Lab, IIIT Hyderabad Video captioning Everyday Not in cards Oct 2025
ADVANCE audio and image retrieval (MTEB) Models match field recordings and aerial images from 5,075 locations. The correct recording and image come from the same place. Massive Text Embedding Benchmark contributors Audio understanding and voice Everyday Not in cards Aug 2026
AdvBench Models get harmful requests, such as a request for malware, and should refuse them. The set has 520 harmful behaviors and 574 harmful strings. Most papers add an attack, such as an adversarial suffix, and report the attack success rate. Carnegie Mellon University, Center for AI Safety and Bosch Center for AI (Zou, Wang, Kolter, Fredrikson) Jailbreaks and prompt injection Everyday Not in cards Jul 2023
AfriBED Models transcribe Bemba and Nyanja speech. The benchmark compares recognition errors across gender and speaker groups. University of Cape Town Speech recognition and translation Everyday Not in cards Oct 2026
AfriQA Models retrieve evidence across languages to answer questions written in African languages. Odunayo Ogundepo and colleagues Reading in many languages School Not in cards May 2023
AfriSenti Models classify sentiment in human-annotated social media posts across 14 African languages. Shamsuddeen Hassan Muhammad and colleagues Reading in many languages Everyday Not in cards Feb 2023
Agent Bazaar Agents price goods or trade with sellers in simulated markets. The benchmark measures whether their decisions sustain prices, deter fraud, and preserve welfare and profits. Agent Bazaar team Multi-agent games University Not in cards Apr 2026
Agent Behavior Loop v0 Models choose an assistant’s next action from an evidence bundle. Twelve tasks check whether coding and general assistants verify their work before claiming success. narcolepticchicken General assistants University Not in cards Jun 2026
Agent Liability Gym Agents review case files and write legal memos about decisions made by autonomous agents. The grader checks their findings, evidence, and unsupported claims. narcolepticchicken Law Expert Not in cards Jul 2026
Agent Memory Leaderboard Memory systems for agents are evaluated on storing, retrieving and using long-term memory across textual, coding and multimodal tracks in one reproducible framework. AML-memory Learning from the context Everyday Not in cards Jul 2026
AgentCLUE-Mobile Mobile assistants complete tasks on a phone. SuperCLUE Phones Everyday Not in cards
Agentic Abstention Agents work in web shopping, terminal and search question tasks where some goals cannot be met. They must abstain at the right turn instead of acting on. University of Washington (Luo, Wen and Wang) Hallucination rates Everyday Not in cards Jun 2026
Agentic Self-Reasoning (Apollo Research) Agents reason about their own operating conditions to complete tasks. Apollo measures success at three difficulty levels. Apollo Research Deception and alignment Everyday Not in cards Aug 2024
Agents4PLC v2.0 Models write Structured Text control programs against formal requirements. The expanded benchmark adds 70 medium tasks and three hard tasks to the original 23. Zhejiang University and UWin Tech Electronics University Not in cards Aug 2025
AgentSettlementBench Models decide whether to release or hold a payment under 20 settlement scenarios. The benchmark tracks accuracy and the cost of approving an unsafe settlement. nagu Finance Expert Not in cards Feb 2026
AgentThreatBench Agents carry out routine tasks in the presence of untrusted content. The benchmark scores task completion and security separately. vgudur-dev (Inspect Evals contributor) Jailbreaks and prompt injection University Not in cards May 2026
AGORA (Agentic Global Open Research Arena) Autonomous AI systems get one secret open finance question, 72 hours and a fixed budget to write a paper. Five finance scholars judge it. Penn State Smeal College of Business Finance Research frontier Not in cards Oct 2026
Agri STT Benchmarking Dataset Models transcribe farmers’ spoken questions in Hindi, Telugu and Odia. Researchers score errors against human transcripts and separately measure errors on agricultural terms. Digital Green Speech recognition and translation Everyday Not in cards Jan 2026
AI Agent Privacy Web agents complete tasks while exposed to private information. The benchmark measures privacy leakage and unsafe disclosure during autonomous operation. Meta FAIR and Superintelligence Labs Deception and alignment Expert Not in cards Mar 2025
AI Code Review Evaluation (Augment Code) Agents review pull requests from open source projects. Augment compares their comments with a corrected reference set. Augment Code Fixing real repositories Expert Not in cards Dec 2025
AI FPL Models manage Fantasy Premier League squads using the same weekly prompt. Their transfers, captain choices, and starting lineups earn points from real matches. Scout Digital and Early Doors Sports Multi-agent games Everyday Not in cards Aug 2025
AI Law Legal Agent Eval Agents review five synthetic export control matters and write risk memos. The grader checks the deliverable, seven findings per matter, and unsupported claims. narcolepticchicken Law Expert Not in cards Jul 2026
AI Office Benchmark (Moonshot) Agents complete end-to-end office tasks and produce documents, spreadsheets, PDFs and slide decks. Experts rate output quality. Moonshot AI Documents and spreadsheets Expert Not in cards Jan 2026
AI Spreadsheet Benchmark Spreadsheet assistants complete 53 prompts for analysis, charts, formulas, and workbook management. Checks measure task success and whether outputs update when source data changes. The evaluation reports pass@1, pass@3, dynamic output rate, and latency. Rows Documents and spreadsheets University Not in cards Sep 2025
AI4D Malawi News Classification Challenge Models assign a topic to a Chichewa news article from a Malawian newspaper or broadcaster. Zindi scores the share of correct predictions. tNyasa Ltd Data Science Lab, AI4D Africa and Zindi Reading in many languages Everyday Not in cards Jan 2021
AIR-Bench 24.05 Retrievers find documents or passages that answer questions. This release covers 69 datasets across 13 languages. AIR-Bench team with the Beijing Academy of Artificial Intelligence Questions that need search University Not in cards Oct 2024
AISA-ArabicFC 2026 Models read an Arabic request and four tool schemas. They select a tool and supply its arguments, or decline to call a tool. Tuwaiq Academy Function calling School Not in cards May 2026
AISE-Bench Agents answer academic information-seeking questions through multi-step knowledge-graph APIs and source-grounded responses. Planning, execution and answer-grounding metrics score each trajectory. Z.ai, Tsinghua University and collaborators Deep research Expert Not in cards Jul 2026
AISHELL-4 Models transcribe Mandarin meetings and identify speakers from conference recordings. NVIDIA reports character error after matching speakers. AISHELL and collaborators Speech recognition and translation Everyday Not in cards Apr 2021
ALE-Gold Agents use a Linux terminal to complete 96 professional tasks. Graders check the files they produce. A task passes only with full credit. Sankalp (dejavucoder) Long terminal sessions Expert Not in cards Sep 2026
Alexandria Models translate conversations between English and regional Arabic. The public test contains 14,442 turns across 13 Arabic varieties. Alexandria team Translation University Not in cards Jan 2026
AlexandriaX 2026 Models translate English dialogue into 13 Arabic varieties, translate financial text between six dialects and mark translation errors across five varieties. AlexandriaX organizers Translation University Not in cards May 2026
ALFFA Swahili Broadcast News Test Speech models transcribe Swahili radio news. The test includes clean speech, background music, overlapping voices and noisy recordings. Grenoble Informatics Laboratory and Dynamique du Langage Speech recognition and translation Everyday Not in cards May 2012
ALFFA Wolof Read Speech Test Speech models transcribe sentences read aloud in Wolof. Two speakers each read 1,000 test sentences. The recordings use different speakers from the training set. Grenoble Informatics Laboratory and Voxygen Speech recognition and translation Everyday Not in cards Apr 2016
ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. University of Washington, Allen Institute for AI and collaborators Simulated robots Everyday Not in cards Dec 2019
Alharthi LibriTTS Naturalness Listening Test Models estimate how natural synthesized speech sounds. Ten listeners rated each sample on a five-point scale for comparison. Carnegie Mellon University and Mohamed bin Zayed University of Artificial Intelligence Audio understanding and voice Everyday Not in cards Oct 2023
Alibaba CTF Agents solve 87 capture-the-flag tasks in Alibaba’s collection of competition challenges from 2023 through 2026. alibaba-aaig Capture-the-flag challenges Expert Not in cards Aug 2026
Aligned Scores and Performances for Piano Transcription Models turn piano performances into written scores. The reference scores let researchers assess pitches, voices, and the positions of notes within bars. Francesco Foscarin and colleagues Audio understanding and voice Expert Not in cards Oct 2020
AliMeeting Models identify speakers in Mandarin meetings. NVIDIA compares diarization error using headset mixes and distant microphones. Alibaba Speech Lab Audio understanding and voice Everyday Not in cards Feb 2022
AlloCiné Models classify French movie reviews as positive or negative. Review ratings supply the reference labels. Théophile Blard Reading in many languages Everyday Not in cards Jan 2020
Alpha Arena Frontier models trade live markets with real money over a season, and the leaderboard ranks them by returns. Nof1 Finance Expert Not in cards Oct 2025
AlphaEarth Foundations mapping evaluation Models use satellite representations to classify land cover, detect change, or estimate physical quantities. The evaluation compares predictions with independent measurements across regions. Google DeepMind General visual questions University Not in cards Jul 2025
AlphaSense Financial Research Benchmark Agents answer 245 multi-step finance questions across company research, financial modeling, earnings, deals, markets and regulatory analysis using source retrieval. AlphaSense Finance Expert Not in cards Nov 2026
Althingi Parliamentary Speech Models transcribe speeches delivered in the Icelandic parliament. Recordings and reference transcripts cover parliamentary proceedings from 2005 through 2016. Reykjavík University and the Icelandic Parliament Speech recognition and translation Everyday Not in cards Aug 2017
AMemOps Models track instruments and their changing attributes in long audio streams. Questions test whether they recall the latest value and recognize absent sources. University of Surrey and Royal Holloway, University of London Audio understanding and voice Everyday Not in cards Oct 2026
AmericasNLP 2025 Shared Task on Creation of Educational Materials for Indigenous Languages Models rewrite a sentence in an Indigenous language to make a requested grammatical change. The organizers count exact matches with the target sentence. AmericasNLP Chat and instructions in many languages University Not in cards Jan 2025
AmericasNLP 2025 Shared Task on Machine Translation into Indigenous Languages Models translate between Spanish and 13 Indigenous languages of the Americas. The organizers score each direction using character and word overlaps with reference translations. AmericasNLP Translation University Not in cards Jan 2025
AmericasNLP 2025 Shared Task on Machine Translation Metrics for Indigenous Languages Models score translations from Spanish into Bribri, Guarani, and Nahuatl. The organizers rank systems by Pearson correlation with bilingual speakers’ ratings. AmericasNLP Translation University Not in cards Jan 2025
Amharic Speech Emotion Dataset Models identify the emotion expressed in a short Amharic recording. The speakers use four regional dialects. Ephrem Afele Retta and colleagues Audio understanding and voice Everyday Not in cards Nov 2021
AMI Meeting Corpus Models identify speakers or transcribe English meetings. NVIDIA evaluates both headset mixes and distant microphone recordings. AMI Consortium Audio understanding and voice Everyday Not in cards
AmsterTime Models match historic photographs of Amsterdam with contemporary street views. The benchmark measures recognition of the same place across time. Burak Yildiz and colleagues General visual questions Everyday Not in cards Mar 2022
AMUSE Multimodal models reason over multi-speaker audio-visual interactions in zero-shot, guided and agentic modes across six task families. Apple Audio understanding and voice Expert Not in cards Dec 2025
AndroidWorld-Knowledge Agents attempt 48 AndroidWorld tasks across five apps. Each app provides guidance for its simplest task, and harder tasks test whether the agent can reuse it. Beijing Jiaotong University and Alibaba Group Phones Everyday Not in cards May 2025
ANES presidential vote choice Models use survey profiles to predict which major party candidate each respondent backed in three US presidential elections. Researchers measure agreement with reported votes. Lisa P. Argyle and colleagues Social simulation Everyday Not in cards Sep 2021
Animal Kingdom Models identify actions in wildlife videos that can show several behaviors at once. The benchmark also tests pose estimation and locating events described in text. Xun Long Ng and colleagues Video understanding School Not in cards Apr 2022
Animal-Bench Models answer 41,839 questions about animal videos. The overall score averages accuracy across 13 tasks, including action prediction and recognizing predator and prey behavior. Beijing University of Posts and Telecommunications and China Telecom Video understanding School Not in cards
Animals with Attributes 2 Models use shared attributes to recognize animal classes they did not see during training. The benchmark provides 37,322 photographs across 50 classes with 85 attributes. Institute of Science and Technology Austria and collaborators General visual questions School Not in cards Jun 2017
ANLI R1 Models label premise and hypothesis pairs from the first adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. Facebook AI Research (Nie et al.) Language reasoning Everyday Not in cards Oct 2019
ANLI R2 Models label premise and hypothesis pairs from the second adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. Facebook AI Research (Nie et al.) Language reasoning Everyday Not in cards Oct 2019
ANLI R3 Models label premise and hypothesis pairs from the third adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. Facebook AI Research (Nie et al.) Language reasoning Everyday Not in cards Oct 2019
AnthroBench Models hold multi-turn conversations that test 14 anthropomorphic behaviors. Automated classifiers and human participants evaluate the generated dialogue. Google DeepMind Instructions and chat Everyday Not in cards Feb 2025
Anthropic cyber safeguard evaluations Classifier recall on violative cyber exchanges, block rates on defensive vulnerability-finding and secure-coding work, and the rewind-attacker robustness test. Anthropic Refusing harmful requests Not in cards
Anthropic harmful-request and wellbeing evaluations Single-turn harmful and benign request sets, multi-turn tests, child safety, suicide/self-harm, disordered eating and election integrity. Anthropic Refusing harmful requests Not in cards
Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. Anthropic (with Axiom Bio and others) Research science Expert Not in cards 2026
Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. Anthropic Frontier Red Team Finding and exploiting real bugs Expert Not in cards Oct 2026
Anthropic malicious agentic use evaluations Measures refusal of malicious tasks via Claude Code, computer use and agentic influence campaigns. Anthropic Refusing harmful requests Not in cards
Anthropic model-written evals LM-generated datasets testing persona, sycophancy and advanced-AI-risk behaviors. Anthropic Deception and alignment Not in cards Dec 2022
Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. Anthropic Deception and alignment Not in cards
Apollo Research coding agent monitor calibration Models rate the severity of problems in coding sessions. Apollo compares the ratings with reference labels across three datasets. Apollo Research Deception and alignment Expert Not in cards Jul 2026
AppAgent Test Set Agents follow 45 instructions across nine Android apps, including scheduling email and changing music playback. Tasks time out after ten steps. Tencent Phones Everyday Not in cards Dec 2023
Applied Theory of Mind (Apollo Research) Agents complete tasks that require tracking what another participant knows. Apollo measures success at three difficulty levels. Apollo Research Deception and alignment Everyday Not in cards Aug 2024
AppWorld Agents write code against 457 APIs of 9 simulated everyday apps to finish 750 tasks for about 100 fictional users. State-based unit tests check the outcome and collateral damage. Stony Brook University and Allen Institute for AI Apps and MCP servers Everyday Not in cards Jul 2024
AQuA Two LLM research loops (factor discovery, model development) reuse evidence from their own past experiments inside sealed sandboxes. Scored by IC and held-out Sharpe. Princeton University (Mengdi Wang) et al. AI research tasks Research frontier Not in cards Aug 2026
AQUA-RAT Models solve algebraic word problems and choose one of five answer options. Evaluation compares the selected option with the answer key, while reference rationales explain the solution. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom at DeepMind Word problems School Not in cards May 2017
AraBench Models translate dialectal Arabic into English. The suite covers travel conversations, media transcripts and religious texts. Researchers score translations against references with BLEU. Qatar Computing Research Institute Translation University Not in cards Dec 2020
ArAIEval 2024 Models mark persuasive passages in Arabic text and name the technique. They also classify Arabic memes using the image, its extracted text, or both. ArAIEval organizers Reading in many languages University Not in cards Mar 2024
AraSentEval 2026 Models classify the sentiment of Arabic hotel reviews. They also rewrite a sentence to express the opposite sentiment while keeping its topic. AraSentEval organizers Reading in many languages Everyday Not in cards Dec 2025
ARC Challenge MT Models answer science questions from ARC-Challenge in translation. The harness scores answers in the target language against the original choices. LumiOpen Reasoning in many languages School Not in cards May 2024
Architectural Code Review v1 Models review code changes with planted architectural defects. The evaluator measures precision and recall separately on six complete diffs and 15 single-defect excerpts. korck Fixing real repositories Expert Not in cards Aug 2026
ArguAna Counterargs Models retrieve a counterargument for a given argument. The evaluation checks where the annotated counterargument appears in the ranking. Henning Wachsmuth and colleagues Reading comprehension School Not in cards
Aria Digital Twin Models infer the layout of objects from first-person images. The SAM 3D Objects evaluation compares predicted positions and orientations with reference scene geometry. Meta Spatial reasoning and counting School Not in cards Jun 2023
ARPA Armenian Paraphrase Corpus Models decide whether two Armenian sentences express the same meaning. Arthur Malajyan and colleagues Reading in many languages School Not in cards Sep 2020
ArtCap Models associate paintings with descriptions of their contents. Google compares image encoders by how often the matching caption appears in their top five. ArtCap team General visual questions Everyday Not in cards Nov 2022
Arti4D Models watch first-person recordings of people opening furniture. They predict whether parts rotate or slide and estimate the axis of motion. University of Freiburg and collaborators Spatial reasoning and counting School Not in cards Sep 2025
Articulate3D Models locate movable parts and handles in scanned rooms. They predict how each part moves, including its axis and the location of its hinge. INSAIT and ETH Zurich Spatial reasoning and counting School Not in cards Dec 2024
Artificial Analysis Decision Index Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning. Artificial Analysis Composite indices Everyday Not in cards Oct 2026
Artificial Analysis Progress Index Artificial Analysis's index that tracks how model capability changes over time on one continuous scale, launched alongside Intelligence Index v5. Artificial Analysis Composite indices Not in cards Oct 2026
Artificial Analysis Search Index One fixed agent answers DeepSearchQA, BrowseComp and AA-Omniscience questions while the search API provider changes. The index is the equal-weighted mean of the three scores. Artificial Analysis Deep research Expert Not in cards Aug 2026
ArXivCS Models answer research-level computer science questions derived from recent arXiv papers, each with a reference final answer. The first set has 46 questions from August 2026. MathArena (ETH Zurich SRI Lab) Scientific code Research frontier Not in cards Sep 2026
ArXivPhys Models answer research-level physics questions derived from recent arXiv papers, each with a reference final answer. MathArena (ETH Zurich SRI Lab) Research science Research frontier Not in cards Sep 2026
ASCEND Models transcribe conversations that switch between Mandarin and English. The evaluation accounts for Chinese characters and English words. Holy Lovenia and colleagues Speech recognition and translation Everyday Not in cards Dec 2021
ASDiv Models solve English arithmetic word problems with varied phrasing. The reference answers identify the expected numerical result. Shen-Yun Miao and colleagues Word problems School Not in cards Jul 2020
ASFX-SED Models locate sound effects in audio recordings. Meta reports area under the receiver operating characteristic curve for event localization. Yusong Wu and colleagues Audio understanding and voice Everyday Not in cards Jul 2025
Asian Language Treebank Models translate parallel sentences across Asian languages. The llm-jp-eval harness uses the parallel corpus for machine translation. NICT, UCSY, and ALT project partners Translation School Not in cards
ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. Google DeepMind Jailbreaks and prompt injection Everyday Not in cards Nov 2026
ASK Norwegian Grammatical Error Correction Models correct grammatical errors in Norwegian learner text. The harness compares the edits with annotated corrections. Matias Jentoft, University of Oslo Reading in many languages School Not in cards
Asper-Bench Agents turn complex user requests into programs that execute multi-step actions in a simulated digital assistant environment. Automated checks compare the resulting state with the requested goal. Apple General assistants Expert Not in cards Jul 2025
Assembly101-O Models watch people assemble toy vehicles and flag the first procedural mistake. Training uses correct procedures, and evaluation stops at the first mistake in an incorrect sequence. Sapienza University of Rome and University of Catania Video understanding Everyday Not in cards Apr 2024
AssertionForge design evaluation Models generate SystemVerilog assertions from design specifications and source code. Cadence Jasper counts proven assertions and measures their coverage. NVIDIA Electronics Expert Not in cards Mar 2025
ASSIN 2 Models classify entailment and rate semantic similarity for new pairs of Brazilian Portuguese sentences. The entailment task has two classes. Livy Real, Hugo Gonçalo Oliveira, and Erick Fonseca Reasoning in many languages School Not in cards Jun 2019
AssistantBench Web agents do 214 realistic, time-consuming tasks on the open web, such as monitoring real-estate markets. Answers are checked automatically against gold answers. Tel Aviv University, Allen Institute for AI and others Web browsing Everyday Not in cards Jul 2024
AssurAI Models respond to Korean prompts across 35 risk categories. Model judges rate the safety of text, image, audio, and video outputs from 1 to 5. Telecommunications Technology Association and KAIST with collaborators Deception and alignment Everyday Not in cards Nov 2025
ASTRA-bench Agents use personal context and interactive tools to complete more than 2,000 multi-step scenarios. Automated checks score whether each user intent is satisfied. Apple General assistants Expert Not in cards Oct 2026
Astro-QA Models answer astronomy questions in English and Chinese. The authors combine answer key checks and text similarity scores, weighting questions by type and difficulty. Guizhou University and collaborators Science questions University Not in cards Jun 2024
AstroCLIMB Models compare pairs of astronomy figures or captions. They determine whether the pair shares a figure, shares a paper, comes from citation-linked papers, or comes from unrelated papers. Center for Astrophysics, Harvard & Smithsonian, and Oak Ridge National Laboratory Lab and data work University Not in cards Jul 2026
AstroLLaVA image and text retrieval Models match astronomical images with descriptions. Google evaluates whether an image retrieves the matching description among its top five results. S. Zaman and colleagues General visual questions University Not in cards Apr 2025
AstroMemBench Spacecraft agents use information from earlier observations to complete simulated missions. The environment scores task completion, safety, and resource use. You Wu and colleagues Embodied reasoning University Not in cards Nov 2026
ATEPP Models identify pianists from the notes, timing, and dynamics of their performances. The original evaluation includes a mixed repertoire and three composer-specific tests. Queen Mary University of London Audio understanding and voice Expert Not in cards May 2022
ATLAS (Exa) Agents search the web to build a full table of every entity that meets a set of conditions, with 2 to 10 attributes each. A row counts only if every cell is correct. Exa Deep research Expert Not in cards Oct 2026
Audio PAIRS Models answer spoken questions adapted from PAIRS. The evaluation checks whether changing demographic cues changes their choices. UCSC VLAA and the AHELM team Bias, fairness and politics Everyday Not in cards Nov 2024
Audio Reasoning Challenge 2026 Models answer questions about speech, music, and environmental audio and explain their reasoning. Incorrect answers score zero. A model judge grades correct answers for reasoning quality, with the middle three of five evaluation runs averaged. Audio Reasoning Challenge organizers, led by Shanghai Jiao Tong University and collaborators Audio understanding and voice Everyday Not in cards Nov 2025
Audio-NTREX-4L Speech translation models translate long-form French, Spanish, Portuguese and German audio into English. Kyutai Translation Expert Not in cards Feb 2026
Audio-Visual Event Localization Models identify an event that can be both seen and heard in a video and locate when it occurs. University of Rochester Video understanding Everyday Not in cards Mar 2018
AudioBench Audio language models follow instructions about speech, voice and environmental sounds. The paper covers 8 tasks and 26 datasets, including speech recognition, spoken question answering, voice traits and audio scenes. The repository has since grown to more than 50 datasets. AudioLLMs team (Bin Wang and coauthors) Audio understanding and voice Everyday Not in cards Jun 2024
AudioCaps Models match natural language descriptions to audio clips. Meta also compares how accurately encoders match audio, video, and text from the same clip. Chris Dongjoo Kim and colleagues Audio understanding and voice Everyday Not in cards
AudioMNIST Models identify a spoken digit from a recording. The test includes recordings from different speakers. Sören Becker and colleagues Speech recognition and translation Everyday Not in cards Jul 2018
AudioSep Eval Benchmark Models isolate a sound from an audio mixture using a caption or a sound label. Scores compare the separated waveform with the original sound. Xubo Liu and colleagues Audio understanding and voice University Not in cards Aug 2023
AudioSet Strong Models locate labeled sound events in time. The evaluation uses human annotations of event boundaries in AudioSet clips. Google Research Audio understanding and voice Everyday Not in cards
AudioVideoBench Agents produce and edit audio and video in containers. The tasks include compositing, mixing, repairing media, reframing shots, and transcribing recordings. Tencent Tasks from real jobs University Not in cards Nov 2026
Augmented Reality Patent Search Agents search 12,000 augmented and virtual reality patent applications to answer technical questions. A model judges each answer. John Oh Questions that need search University Not in cards Nov 2025
Australian BAS Reconciliation Models reconcile a synthetic Australian goods and services tax ledger, correct a Business Activity Statement, and identify bookkeeping errors. The grader weighs totals and fault detection. afzal0 Finance Expert Not in cards Sep 2026
Autocast Models forecast real events using questions from forecasting tournaments and news available before each prediction date. Andy Zou and collaborators Real-world events Expert Not in cards Jun 2022
AutoDS scientific machine learning cases Agents build predictive models for maize yield, polymer glass transition temperature, and supercomputer job outcomes. Each case supplies a train split and a held-out test. ITMO AutoDS Machine learning engineering Expert Not in cards Aug 2026
Autonomy-10 Agents complete browser, desktop and tool-use tasks across nine domains that measure skill, flexibility and intelligence. HUD AI Web browsing Expert Not in cards Jan 2025
AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. Politecnico di Torino and collaborators (Gioacchini et al.) Finding and exploiting real bugs Expert Not in cards Oct 2024
AutoResearch An agent edits a small nanochat training script and runs 5 minute training experiments overnight. Scored by final validation bits per byte, where lower is better. Andrej Karpathy AI research tasks Research frontier Not in cards Mar 2026
AvalancheBench (May study) Agents recover underlying patterns from synthetic enterprise data and report their findings. Snowflake and collaborators Data science and SQL Expert Not in cards May 2026
AvalonBench Models play Resistance Avalon while discussing, deceiving, deducing, and negotiating with other agents. AvalonBench authors Multi-agent games Everyday Not in cards Nov 2023
AVCC Next-Speaker Prediction Models predict who will speak after a pause in a conversation. The authors compare two models with human listeners on the same 1,730 events. KTH Royal Institute of Technology Speech recognition and translation Everyday Not in cards Nov 2026
AVPS-360 Models locate and track the person making a sound in panoramic video. The annotations connect each sound source to a person across frames. Tampere University Audio understanding and voice Everyday Not in cards Oct 2026
aws-bench Coding agents and model pairs diagnose misconfigurations, provision infrastructure and operate live cloud setups in disposable real AWS accounts. aws-bench maintainers Terminal tasks Expert Not in cards Jul 2026
Aya Red-teaming Models answer harmful prompts in eight languages across nine harm categories. Judges score refusal and safety behavior against annotated categories. Cohere Labs Refusing harmful requests Everyday Not in cards Jun 2024
Babel-ImageNet Models classify ImageNet images using class descriptions in different languages. The benchmark measures how recognition accuracy changes with the prompt language. Gregor Geigle, Radu Timofte, and Goran Glavaš General visual questions Everyday Not in cards Jun 2023
BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. Qwen Team, Alibaba Group General assistants Expert Not in cards Sep 2026
bAbI QA Tasks Models answer questions about short synthetic stories. The tasks isolate skills such as tracking objects, combining supporting facts and reasoning about locations. Jason Weston and colleagues Language reasoning School Not in cards Feb 2015
BackendBench LLMs write kernels that override PyTorch core operators. PyTorch's own OpInfo and FACTO tests check correctness, and real model tensor shapes measure speed. Meta (PyTorch) GPU kernels Expert Not in cards Jun 2025
Backrooms: 3D Clue-Trail Navigation Agents follow visual clues through a first-person maze. Only an escape without a wrong doorway counts as a solve. Keeno Corp Spatial reasoning and counting Everyday Not in cards Nov 2026
BackSearch Paid API that searches a frozen web archive as of a given date, for forecasting backtests and reproducible web-enabled evals. A tool, not a benchmark. General Reasoning Real-world events Not in cards Jul 2026
Backtracing Models identify the sentence that prompted a student's question or a news reader's curiosity. In conversations, they find the turn that prompted an emotional response. Rose E. Wang and colleagues Finding facts in long inputs Everyday Not in cards Mar 2024
BanglaSER Models identify the emotion in Bangla sentences recorded on phones and laptops. Rakesh Kumar Das and colleagues Audio understanding and voice Everyday Not in cards Mar 2022
BanSpeech Models transcribe Bangladeshi Bangla broadcasts across 13 domains. Word error rate compares predictions with human transcripts. Shahjalal University of Science and Technology Speech recognition and translation Everyday Not in cards Oct 2022
BAREC-ST-2026 Models assign Arabic sentences and documents to 19 reading levels. The blind test includes texts from 17 publishers absent from the original corpus. CAMeL Lab and BAREC organizers Reading in many languages University Not in cards May 2026
BarterBench Agents trade scarce goods to meet inventory targets. The hub port assigns one model-controlled trader to a marketplace of scripted opponents. James E. Ball Multi-agent games University Not in cards Mar 2026
Bash Agent Evals Agents query GitHub issues and pull requests through bash, SQL, or a combination. Braintrust scores answers against the dataset. Braintrust Apps and MCP servers University Not in cards Jan 2026
BashBench Agents complete multi-step system administration tasks while red teams attempt covert malicious actions in the same shell environment. Redwood Research and UK AI Security Institute Dangerous capabilities Expert Not in cards Jun 2025
BasqueGLUE Models analyze Basque text for tasks including sentiment, topic classification, coreference and entailment. Each task has its own evaluation metric. Gorka Urbizu and colleagues Reading in many languages School Not in cards Jun 2022
BBH-ja Models solve Japanese translations of BIG-Bench Hard reasoning problems. Preferred Networks Reasoning in many languages School Not in cards Jul 2025
BCSC Self-Assessment Program ophthalmology questions Models answer ophthalmology board preparation questions across subspecialties. Researchers compare their choices with the academy’s answer key. American Academy of Ophthalmology Medicine Expert Not in cards Jan 2023
BdSLW401 Models identify one of 401 Bangla Sign Language words in a video. The collection records signs from the front and side. National University Bangladesh and Islamic University of Technology Video understanding Everyday Not in cards Dec 2024
BdSLW60 Models identify an isolated Bangla Sign Language word in a video clip. Accuracy measures how often the predicted word matches its label. National University Bangladesh and Islamic University of Technology Video understanding Everyday Not in cards Dec 2023
BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors Models judge whether math tutors identify and locate student mistakes and give guidance students can act on. A fifth track asks which tutor wrote each reply. BEA 2025 shared task organizers Other professions University Not in cards Mar 2025
Beancount Ledger Agents reconcile bank statements, repair a Beancount ledger, and allocate payments to invoices. The evaluator checks balances and payment records against the underlying transactions. cangultekn Documents and spreadsheets University Not in cards Sep 2026
BEANS-Zero Models follow text prompts to classify, detect or caption animal sounds without task-specific training. Earth Species Project Audio understanding and voice University Not in cards Nov 2024
BEHAVIOR-1K Embodied agents do 1,000 everyday household activities, such as cleaning and cooking, in realistic simulation built on OmniGibson. Success is checked against task goal definitions. Stanford Vision and Learning Lab Simulated robots Everyday Not in cards Mar 2024
Behnke and Geiger Flexible Job Shop Instances Agents build schedules or write solvers that assign operations to eligible machines within work centers. The objective is to finish the last job as early as possible. Dennis Behnke and Martin Josef Geiger Tasks from real jobs University Not in cards Jan 2012
BelarusianGLUE Models classify Belarusian sentiment, recognize entailment, judge grammar, and distinguish word meanings in context. Maksim Aparovich and colleagues Reading in many languages Everyday Not in cards Jul 2025
Ben-10 Regional ASR Models transcribe spontaneous speech in ten regional dialects of Bengali using spellings checked by linguists. The organizers report word and character error by dialect. Bengali.AI and Islamic University of Technology Computer Society Speech recognition and translation Everyday Not in cards Apr 2024
bgGLUE Models perform language understanding tasks in Bulgarian, including sentiment analysis on movie reviews and question answering on high school exams. Momchil Hardalov and colleagues Reading in many languages Everyday Not in cards Jun 2023
BHS Syntactic Evaluation Models compare sentences in Basque, Hindi and Swahili. The test checks whether the model prefers the grammatical member of each pair. Daria Kryvosheieva and colleagues Reasoning in many languages Everyday Not in cards Jan 2025
Bible Accuracy Benchmark Models quote verses from a named Bible translation. They earn points for accurate quotations and lose points for inventing verses. YouVersion Hallucination rates School Not in cards Jul 2026
BIBLE Translation Benchmark (Meta) Models translate verses from the Gospel of John into English. Meta scores the translations against reference verses using chrF++ and xCOMET. FAIR at Meta Translation University Not in cards Mar 2026
BIG-bench Language models do over 200 tasks contributed by hundreds of authors, from linguistics and math to social bias. Most tasks are scored by exact match or multiple choice. Google and 400+ contributing authors Language reasoning University Not in cards Jun 2022
BigLaw Bench Retrieval Retrieval systems find the most relevant passages in legal documents. Harvey compares retrieval results with attorney annotations. Harvey Law Expert Not in cards Nov 2024
BigLaw Bench Workflows: SPA Deal Points Models extract deal terms from stock purchase agreements. Harvey measures accuracy against terms extracted and checked by attorneys. Harvey Law Expert Not in cards Sep 2024
BigLaw Bench: Global Models perform legal work specific to the UK, Australia and Spain. Local legal experts designed tasks and grading criteria for each jurisdiction. Harvey and Mercor Law Expert Not in cards Feb 2026
BigLaw Bench: Research Agents use search tools to find US case law and give cited answers. The tasks span corporate law, litigation and other practice areas. Harvey and Snorkel AI Law Expert Not in cards Mar 2026
BillSum Models summarize legislative bills. The dataset pairs bill text with summaries written for the US Congress and the California legislature. Anastassia Kornilova and Vladimir Eidelman Law University Not in cards Nov 2019
Binary Reasoning v1 Models answer machine-checked questions about compiled software across processor architectures. The hub port supplies 100 of 10,251 tasks and a separate set of 29 interactive tasks. S3RL4 Fixing real repositories Expert Not in cards Oct 2026
Bing Chat election answers in Switzerland, Bavaria, and Hesse Bing Chat answers questions about candidates, polls, and voting in three elections. Annotators identify factual errors, evasions, and accurate answers. AlgorithmWatch and AI Forensics Long-form factuality Everyday Not in cards Oct 2023
Bingo (Visual Hallucination) Models answer questions about images. The benchmark tests whether prior expectations or interference from other visual information cause false answers. Chenhang Cui and colleagues Hallucination rates Everyday Not in cards Nov 2023
Bio-Literature Search Level 2 Agents use search tools to compare abstracts from PubMedQA, including their sections, subject headings, and conclusions. Five task types require retrieving or combining evidence. d42me Questions that need search University Not in cards Feb 2026
BioASQ Systems answer biomedical questions written by experts, using PubMed articles as evidence. Questions are yes/no, factoid, list or summary types. A second task asks systems to index new PubMed articles with MeSH headings. BioASQ challenge organizers Medicine Expert Not in cards 2013
BioCircuit Agents identify groups of features in Qwen3-4B that control a behavior or restore a damaged output. The grader tests their interventions on held-out prompts. Amit Prakash and Lalithadithya N AI research tasks Research frontier Not in cards Jul 2026
BioCreative V Chemical-Disease Relation Task Models identify biomedical entities and chemical-induced disease relations in PubMed abstracts. The OpenMed port extracts chemical and disease names. BioCreative V CDR organizers Genomics and life sciences University Not in cards
BioCreative VII LitCovid Models assign one or more of seven research topics to COVID-19 articles using titles and abstracts. BioCreative and the US National Library of Medicine Genomics and life sciences University Not in cards Jun 2021
BioCreative VIII Task 3 Genetic Phenotype Normalization from Dysmorphology Physical Examinations Models identify abnormal physical findings in pediatric exam notes and return Human Phenotype Ontology identifiers. A second task also scores the locations of the findings. Cedars-Sinai Medical Center and Children's Hospital of Philadelphia Medicine University Not in cards Apr 2023
BiodivNER Models identify and label phrases in biodiversity abstracts and dataset descriptions. Annotators distinguish six kinds of entities, including organisms, locations, and environments. Nora Abdelmageed and colleagues Lab and data work University Not in cards Apr 2022
Biological Wastewater Treatment Long-Form QA Models answer 286 closed-book questions about biological wastewater treatment. Three experts grade the answers for relevance and factual accuracy. Princeton University and collaborators Science questions University Not in cards Jul 2024
BioML-bench Agents build and evaluate machine learning systems for biomedical experiments spanning proteins, drugs, single cells, imaging, and biomarkers. Task-specific metrics compare submissions with experimental reference results. Science Machine Machine learning engineering Expert Not in cards Sep 2025
Biomni-Eval1 Agents use biomedical data and tools to answer 433 questions across ten task types. Each task checks a structured answer against a reference. Stanford Biomni Genomics and life sciences Expert Not in cards Oct 2025
BioNNE 2024 Models mark the spans and types of overlapping medical terms in English and Russian abstracts. Organizers score span matches using macro F1. BioASQ BioNNE organizers Medicine University Not in cards Feb 2024
BioNNE-L Models link nested terms in English and Russian biomedical abstracts to medical concept identifiers. Organizers report top-one accuracy, top-five accuracy, and mean reciprocal rank. BioASQ BioNNE-L organizers Medicine University Not in cards Feb 2025
BioNNE-R Models identify relations between supplied medical entity mentions in English and Russian abstracts. Organizers rank systems by macro F1 across 14 relation types. BioASQ BioNNE-R organizers Medicine University Not in cards Jan 2026
BioRED Models extract biomedical relations from 600 PubMed abstracts. The hub port presents entity pairs and asks models to classify their relationship. National Center for Biotechnology Information Genomics and life sciences Expert Not in cards Jul 2022
BioRiskEval Researchers measure biorisk-relevant capability kept by open-weight DNA and protein foundation models, with three task sets named Gen, Mut and Vir. Scale AI Hazardous knowledge Expert Not in cards Oct 2025
BioTIER Safeguarded model configurations receive harmful and benign biology requests. Scored on refusing requests designated for refusal and allowing benign ones. SecureBio Refusing harmful requests Expert Not in cards Mar 2026
BioVITA retrieval benchmark Models match animal images, audio recordings, and descriptions. The benchmark checks whether the retrieved items match the reference species, genus, or family. Risa Shinoda and colleagues Genomics and life sciences University Not in cards Mar 2026
BirdSet Models identify bird species from audio recordings across multiple evaluation datasets. Lukas Rauch and colleagues Audio understanding and voice Expert Not in cards Mar 2024
Birdsnap Models identify the North American bird species shown in a photograph. Thomas Berg and colleagues Detection and captioning University Not in cards Jun 2014
Bisbee ANES feeling thermometers Models rate social and political groups from a survey respondent’s perspective on a 0 to 100 scale. Researchers compare ratings with the respondent’s answers. James Bisbee, Joshua Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer Larson Social simulation Everyday Not in cards May 2023
BixBench-Hypothesis Agents analyze existing datasets to test 51 hypotheses. A rubric awards points for intermediate results and the final conclusion. Edison Scientific Lab and data work Expert Not in cards Mar 2026
Blackjack Basic Strategy Models play blackjack hands or select a single move from a dealt position. The grader measures the expected value of each decision against basic strategy. petern Puzzles and chess Everyday Not in cards Sep 2025
BLiMP Language models must assign higher probability to the grammatical sentence in each of 67,000 minimal pairs covering English syntax, morphology and semantics. New York University (Warstadt et al.) Commonsense Everyday Not in cards Dec 2019
Blind Spot (Carlos Toledo) Models find an input pair that distinguishes a modified circuit from its original or declare the circuits equivalent. Simulation and SAT proofs check the answers. carlos-toledo Electronics Expert Not in cards Sep 2026
Blind-Spots-Bench Models answer 235 text and visual problems chosen to expose failures on tasks that people find simple. A Gemini 3 Flash grader compares each response with a structured reference solution and gives binary credit. The grader can execute code to check exact constraints. École Polytechnique Fédérale de Lausanne Abstract puzzles and logic Everyday Not in cards Jul 2026
Blobfish Litigation Bench Agents complete 100 tasks in a simulated litigation practice. The grader compares the changes agents make to case records against required outcomes. Blobfish AI Law Expert Not in cards Aug 2026
BlueprintSymVL Models use an example symbol to find matching components in piping and instrumentation diagrams. They must give both the correct count and the component labels. Eindhoven University of Technology, Chalmers University of Technology, and collaborators Documents, charts and text in images School Not in cards Oct 2025
BM-Bench Image editing models perform instruction-guided non-rigid image manipulations on the ByteMorph benchmark dataset. ByteDance Seed Image generation and editing Expert Not in cards Jun 2025
Bornholm Parallel Models match Bornholmsk sentences to their Danish translations by comparing sentence vectors. The test contains 500 sentence pairs. Scandinavian Embedding Benchmark team Translation School Not in cards Jul 2023
Bot Analysis Arena Models write trading rules for simulated US stock accounts. Some agents revise their rules after trades. Software tests each revision against the current rules before adoption. AI Trading Competition Finance University Not in cards Jul 2026
Break the Grader Models compare a numerical grader with a certified interval and determine whether the grader can accept an incorrect value. Exact certificates verify the answer. carlos-toledo Written proofs University Not in cards Sep 2026
Brennan Center election conspiracy chatbot audit Chatbots answer questions about election claims and respond to skeptical follow-ups. Researchers check factual errors, unsupported citations, and whether responses endorse the claims. Brennan Center for Justice Long-form factuality Everyday Not in cards Aug 2026
Brennan Center election image identification Chatbots inspect 14 synthetic election images in English and Spanish. Researchers count how many images each chatbot identifies as synthetic in each language. Brennan Center for Justice Detection and captioning Everyday Not in cards Aug 2026
BRFauna Models answer questions, simplify sentences, and summarize documents in Brazilian Portuguese. The suite compares generated text with references across three datasets. Gabriel Assis, Cláudia Freitas, and Aline Paes Reading in many languages University Not in cards Jun 2025
BRIGHT Retrieval systems find documents for queries from StackExchange, LeetCode and math competitions where relevance needs reasoning. Scored by nDCG@10. Su et al. (HKU and collaborators) Questions that need search Expert Not in cards Jul 2024
BrowseComp Long Context 256k Models answer BrowseComp questions from a context of 128K to 256K tokens built from the needed web pages and related distractor pages. Scored by accuracy. OpenAI Reasoning over long documents Expert Not in cards Aug 2025
BSNLP 2019 Shared Task Models find, classify, normalize, and link named entities across Slavic languages. Jakub Piskorski and colleagues Reading in many languages School Not in cards Aug 2019
BUCC 2018 bitext mining shared task Models find translation pairs in two collections of sentences. The evaluation compares the retrieved pairs with a reference list. Pierre Zweigenbaum and colleagues Translation School Not in cards
bun-server-bench Agents implement 50 backend services in Bun. Hidden tests grade behavior such as transaction handling, concurrent requests, and rate limits. Tiny Computer AI Building whole projects Expert Not in cards Jun 2026
BURST Models recognize, segment, and track objects across video frames. The benchmark tests how well they associate the same object across time. Ali Athar and colleagues Video understanding Everyday Not in cards Sep 2022
BVA Case Citation Dataset Models recommend the next legal citation in a veterans’ appeal decision. The authors measure how often the recorded citation appears among the top recommendations. Stanford RegLab and Technical University of Munich Law Expert Not in cards May 2021
caBreu Models summarize Catalan news articles. Reference summaries cover extractive, abstractive, and very short summaries. Barcelona Supercomputing Center Reading in many languages School Not in cards Sep 2023
CACAE 2025 Models label the role of each sentence in a Chinese essay and identify relations between groups of sentences. Relations account for 60 percent of the score. East China Normal University Reading in many languages University Not in cards Feb 2025
CADECv2 Models extract adverse drug reactions and drug names from patient reviews. The corpus covers 26 drugs across four groups. CSIRO, Emory University, and the University of Sydney Medicine University Not in cards May 2024
CADmium Fusion 360 Reconstruction Models reconstruct CAD parts from text descriptions of Fusion 360 designs. The authors measure shape distance and count invalid outputs. Prashant Govindarajan and colleagues CAD models University Not in cards Jul 2025
CADPrompt Models write CadQuery programs from text descriptions of mechanical parts. The authors align each generated shape with its reference and measure point cloud distance. Kamel Alrashedy and colleagues CAD models University Not in cards Oct 2024
CALLHOME American English Speech Models transcribe spontaneous telephone conversations in American English and assign words to speakers. NVIDIA uses the CH109 test subset. Linguistic Data Consortium Speech recognition and translation Everyday Not in cards
CALLHOME diarization Models identify speakers in multilingual telephone conversations. NVIDIA evaluates Part 2 with two to six speakers per call. NIST and Linguistic Data Consortium Audio understanding and voice Everyday Not in cards
Caltech-101 Models identify object categories in photographs. Meta compares the classification accuracy of image encoders on this collection. California Institute of Technology General visual questions Everyday Not in cards
Caltech-UCSD Birds 200 Models identify bird species from photographs. The benchmark distinguishes species with similar appearance. California Institute of Technology and University of California San Diego General visual questions University Not in cards Sep 2010
CALVIN Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length. University of Freiburg Simulated robots Everyday Not in cards Dec 2021
CAMeL Models fill gaps in Arabic sentences with names, foods, and other cultural references. The Cultural Bias Score measures how often models favor Western over Arab alternatives. Georgia Institute of Technology Bias, fairness and politics Everyday Not in cards Nov 2023
Can I Play It? Models assign a difficulty grade from one to nine to each piano score. The original study reports balanced accuracy across the nine grades. Universitat Pompeu Fabra and Sogang University Audio understanding and voice University Not in cards May 2022
capa Feature Identification Models identify capabilities in disassembled functions from the EMBER2024-capa dataset. The grader compares their feature lists with reference labels and gives partial credit. kyleavery Secure code and review Expert Not in cards Sep 2025
CAPBENCH-English Models assign a policy topic to English bills, legislative speeches, and news stories. The Easy tier scores agreement with four unanimous human coders. US Policy Agendas Project and poltextLAB Reading comprehension University Not in cards Oct 2026
CapQuiz Models are evaluated through human-verified questions answered from generated video captions, measuring factuality and coverage of salient content. Apple Video captioning Expert Not in cards Sep 2026
CareManage-Bench Models respond to patient scenarios involving medication safety, care plans, and triage. The harness checks the responses against case specific criteria. Habib Ur Rehman Medicine University Not in cards Oct 2026
Carena-Hard-Auto Models answer Chinese prompts. A judge compares pairs of answers, and the evaluation uses a Bayesian ranking model. Beijing Academy of Artificial Intelligence Chat and instructions in many languages Everyday Not in cards
CARLA Driving Tasks (sinatras) Agents navigate and explore simulated roads across 19 tasks using text or camera images. The simulator scores outcomes such as reaching destinations and avoiding collisions. sinatras Simulated robots Everyday Not in cards Feb 2026
CaseHOLD Models read a passage from a US court decision with the holding of a cited case masked out and pick the correct holding from five choices. The dataset has more than 53,000 questions drawn from US case law. Stanford RegLab Law Expert Not in cards Apr 2021
Casual Conversations v2 Models classify attributes of speakers in recorded interviews. The HELM audio tasks compare age and gender predictions with participants’ own labels. Bilal Porgali and colleagues Audio understanding and voice Everyday Not in cards Mar 2023
CatalanQA Models extract answers from Catalan passages drawn from VilaQuAD and ViquiQuAD. The dataset balances question types. Barcelona Supercomputing Center Reading in many languages School Not in cards Jun 2022
Catalonia Independence Corpus Models identify an author's stance on Catalan independence from posts in Catalan and Spanish. Elena Zotova and colleagues Reading in many languages School Not in cards May 2020
CatCoLA Models classify Catalan sentences by whether a native speaker would consider them grammatical. Universitat Pompeu Fabra and Barcelona Supercomputing Center Reading in many languages School Not in cards May 2024
Cattle Trade Agents play a long economic game with auctions, hidden offers, bargaining, bluffing, and resource limits. Cattle Trade authors Multi-agent games Everyday Not in cards May 2026
CauGym Models infer causal effects from descriptions and probabilities. Test questions vary the wording, add irrelevant facts, or remove information needed to answer. Shanghai Artificial Intelligence Laboratory, Fudan University and Tongji University Language reasoning University Not in cards Feb 2026
CausalReasoningEnv Models identify how to estimate a causal effect from a graph and specify the necessary probability queries. The dataset contains 100 evaluation graphs. irfanjamil Language reasoning University Not in cards Mar 2026
CDVQA Models compare two aerial images and answer questions about which land cover changed and by how much. The two tests use the same 968 image pairs with different answer distributions. They contain 39,686 and 31,036 questions, respectively. Zhenghang Yuan, Lichao Mou, Zhitong Xiong, and Xiao Xiang Zhu General visual questions School Not in cards Dec 2021
CEFR Adherence Benchmark Models answer English learners at a requested CEFR level. The evaluation scores the reply's relevance and whether its language fits the target level. Astra-K Following precise instructions University Not in cards Oct 2026
Cekura Bench Models complete separately scored speech recognition, speech generation, voice-agent, real-call, turn-taking and voice-quality benchmarks. Cekura publishes no combined score. Cekura Audio understanding and voice Everyday Not in cards Jul 2026
Cekura Production Voice Agents Published composite score for production voice agents. Cekura Audio understanding and voice Everyday Not in cards Jul 2026
Censorship & Propaganda Eval Models answer 803 prompts about censored topics and official narratives. A judge scores factual coverage and endorsement of propaganda separately. Nous Research Bias, fairness and politics University Not in cards Nov 2026
chABSA Models identify the target and sentiment of opinions in the business results sections of Japanese annual reports. Chakki Reading in many languages University Not in cards Dec 2017
Channel Switching Eval Models respond to benign policy conflicts with and without monitoring cues. The evaluation measures how those cues affect behavior and stated reasoning. anthone Deception and alignment University Not in cards Jun 2026
CHART-Info 2024 Models classify chart images, identify text and axes, and recover plotted values. The authors compare baselines on five of the seven tasks. CHART-Info team Documents, charts and text in images School Not in cards Sep 2024
ChatGPT Chemistry Assistant MOF synthesis extraction Models extract 11 synthesis details from papers about metal-organic frameworks. Researchers check the reported materials, amounts, temperatures, and reaction times against their annotations. University of California, Berkeley Lab and data work University Not in cards May 2023
ChCatExt Models reconstruct headings and paragraphs from Chinese bid notices, financial announcements, and credit rating reports. The combined test contains 65 documents. Soochow University and Huawei Cloud Reasoning over long documents School Not in cards Apr 2023
ChemCoTBench Models interpret and edit molecular structures, optimize molecular properties and predict reaction outcomes. Each task has its own scoring rule. Peking University, IDEA and Yale University Science questions University Not in cards May 2025
ChemProt Models identify the relationship between a chemical and a protein in a biomedical abstract. The shared task evaluates five relationship types. BioCreative VI organizers Genomics and life sciences Expert Not in cards
Chess Notation Evaluation Models read chess positions in Forsyth-Edwards Notation and describe the board in JSON. The scorer checks fields against the expected position. apokryphosx Puzzles and chess School Not in cards Sep 2025
Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. IBM Research and MIT (Wu et al.) Medicine Expert Not in cards Jul 2021
ChestX-ray14 Models read frontal chest X-rays from the NIH Clinical Center and say whether thoracic conditions are present. NIH Clinical Center (Wang et al.) Medicine Expert Not in cards May 2017
CheXpert Models read chest radiographs and label up to 14 observations. The test labels come from board-certified radiologists. Stanford ML Group (Irvin, Rajpurkar et al.) Medicine Expert Not in cards Jan 2019
CHiME-6 Track 2: Unsegmented Speech Recognition Models identify who spoke when and transcribe dinner party recordings without utterance boundaries. The organizers rank systems by word error rate after matching speakers. CHiME Challenge organizers Speech recognition and translation Everyday Not in cards Dec 2019
Chinese Classical Poetry Appreciation 2026 Models explain words, lines, and allusions in classical Chinese poems. They also complete analogies and choose among interpretations. Harbin Institute of Technology, Shenzhen, and the Chinese Information Processing Society of China Reading in many languages University Not in cards Feb 2026
Chinese Essay Discourse Coherence Corpus Models grade coherence in Chinese school essays, identify topic sentences, and classify relationships between passages. The authors hold out the final 100 essays. East China Normal University and Microsoft Research Asia Reading in many languages School Not in cards Dec 2023
Chinese Essay Rhetoric Recognition Evaluation 2025 Models find figures of speech across sentences in Chinese student essays. They classify eight figures and extract the words that form four of them. East China Normal University and Shanghai Linguan Data Technology Reading in many languages School Not in cards Dec 2024
Chinese Language Generation Evaluation (CLGE) Models generate Chinese paper titles from scientific abstracts. The evaluation compares the generated titles with references using ROUGE and BLEU. SuperCLUE Reading in many languages University Not in cards Dec 2019
Chinese Linguistics and Cognition Challenge Models answer open questions about Chinese language and cognition. Human graders or the publisher’s judge model assess the answers. Beijing Academy of Artificial Intelligence Reasoning in many languages University Not in cards
Chinese Literature Named Entity Recognition and Relation Extraction Models mark named entities in Chinese literary passages and identify relationships between them. The authors score both tasks with F1. Peking University Reading in many languages School Not in cards Nov 2017
Chinese Machine Reading Comprehension 2019 Models restore missing sentences in Chinese stories. Some options come from elsewhere in the same story. The organizers score individual blanks and entirely correct passages. Harbin Institute of Technology and iFLYTEK Research Reading in many languages School Not in cards May 2019
Chinese Student English Essay Models score English essays from Beijing high school exams. Teachers assign up to 8 points for content, 8 for language, and 4 for structure. Changrong Xiao and colleagues at Tsinghua University and partner institutions Other professions University Not in cards Jan 2024
Chinese WPLC Models predict the final word in a Chinese novel excerpt. Readers helped select passages where the final sentence alone gave too little information. Tianjin University and Peng Cheng Laboratory Reading in many languages School Not in cards Apr 2021
ChiPBench (chip placement) Agents place circuit blocks in chip designs. OpenROAD measures timing, power, and area after routing. Higher timing slack is better. MIRA Lab and colleagues Electronics Expert Not in cards Jul 2024
ChronoMagic-Bench Video models generate time-lapse scenes from 1,649 prompts covering biological, human-made, weather, and physical changes. Automatic metrics measure the amount of change and temporal coherence, with a separate GPT-4o assessment of change. Peking University, Peng Cheng Laboratory, Rabbitpre Intelligence and others Video generation Everyday Not in cards Jun 2024
Chronos Benchmark II Models forecast future values across 27 time series datasets. The test compares point and probabilistic forecasts against a seasonal baseline. Amazon and Chronos collaborators Real-world events University Not in cards Mar 2024
CIFAR-10 Models classify small color images into ten object categories. University of Toronto General visual questions Everyday Not in cards Apr 2009
CIFAR-100 Models classify small color images into 100 object categories. University of Toronto General visual questions School Not in cards Apr 2009
CIPHERGRID Models infer an encoded grid-world rule system from examples and produce a valid path for a new grid. Dynamic programming verifies the path. Microsoft Research Spatial reasoning and counting Expert Not in cards Oct 2026
CiteBench Models answer information-seeking questions with citations from retrieved passages. Human annotations score citation quality across relevance, support and completeness. Amazon Science Faithful to sources University Not in cards Jun 2025
Cityscapes Models assign a class to each pixel in street scenes filmed from a vehicle. The benchmark compares predicted segments with annotated road scenes. Cityscapes team Detection and captioning Everyday Not in cards Apr 2016
CivilComments-WILDS Models classify the toxicity of online comments. The benchmark evaluates how accuracy varies across groups mentioned in the text. Pang Wei Koh and colleagues Bias, fairness and politics Everyday Not in cards Dec 2020
ClaimBench Models label whether a 10-K excerpt supports, contradicts or says nothing about a claim, across 1,960 claims. Small decision models are scored on accuracy, cost and latency. Vals AI Finance University Not in cards Oct 2026
CLARA-MeD Spanish Medical Text Simplification Models rewrite Spanish medical sentences for patients. The collection includes 3,800 aligned pairs and 1,200 clinical trial sentences that linguists simplified. CLARA-MeD project Medicine University Not in cards May 2022
ClassEval Models generate whole Python classes for 100 hand-built tasks. Tests check each method and the class as a whole. Fudan University Function-level problems University Not in cards Aug 2023
Classical Chinese Historical Event Detection Models locate words that signal events in the Twenty-Four Histories. They classify each event under nine broad categories and 67 specific types. Beijing Language and Culture University Reading in many languages University Not in cards Aug 2023
CLEAR (Clinical Information Extraction) Models read patient notes and classify a medical condition as present, absent, or uncertain. Human reviewers supply the reference labels. Ivan Lopez and colleagues Medicine Expert Not in cards Jan 2025
CLEAR French Medical Text Simplification Models simplify French sentences from drug information, medical encyclopedias, and research reviews. The authors released 100 test pairs. University of Lille Medicine University Not in cards Dec 2020
ClearHarm Models answer unambiguously harmful CBRN questions. Judges score whether attacks elicit harmful responses from the model. FAR.AI Jailbreaks and prompt injection Expert Not in cards Jun 2025
CLEF 2026 SimpleText Task 1 Models rewrite sentences or whole Cochrane abstracts for the public. The organizers compared outputs with plain-language summaries from 175 new systematic reviews. University of Amsterdam and University of Western Brittany Medicine University Not in cards Sep 2025
CLEF eHealth 2019 Technology-Assisted Reviews Task 2 Models rank the articles returned by a systematic review search so reviewers can screen relevant titles and abstracts first. CLEF eHealth Medicine University Not in cards Apr 2019
CLEF HIPE 2026 Models decide whether a historical text places a person at a named location. Organizers rank systems by balanced accuracy across three labels. CLEF HIPE organizers Reasoning in many languages University Not in cards Sep 2025
CLEF JOKER 2023 Task 2: Pun Location and Interpretation Models identify the punning word in English, French, or Spanish jokes. The English interpretation test also asks for words that express both meanings. CLEF JOKER organizers Reasoning in many languages School Not in cards Sep 2022
CLEF JOKER 2024 Task 2: Humour Classification According to Genre and Technique Models assign one of six humor techniques to an English text. Organizers rank runs by accuracy and report F1 for each technique. CLEF JOKER organizers Reading comprehension School Not in cards Sep 2023
CLEF JOKER 2025 Task 1: Humour-Aware Information Retrieval Systems retrieve texts that match a query and contain humor. Organizers report mean average precision separately for English and European Portuguese. CLEF JOKER organizers Questions that need search Everyday Not in cards Oct 2024
CLEF JOKER 2025 Task 3: Onomastic Wordplay Translation Models translate English names containing wordplay into French using a description. The organizers check exact matches and have translators assess alternatives. CLEF JOKER organizers Translation University Not in cards Oct 2024
CLEF JOKER 2026 Task 1: Humour-Aware Information Retrieval Systems retrieve humorous passages that match a topic in English or Hinglish. Organizers rank systems by mean average precision. CLEF JOKER organizers Questions that need search Everyday Not in cards Oct 2025
CLEF JOKER 2026 Task 2: Pun Translation Models translate English puns into French. The automatic score checks whether a translation contains an annotated French word or phrase that carries the pun. CLEF JOKER organizers Translation University Not in cards Oct 2025
Click Calibrate Models click a circle or a named word in generated screenshots. The evaluation checks accuracy across image sizes and coordinate formats. Prime Intellect Finding things on screen Everyday Not in cards May 2026
Climate intervention forecasts for beliefs and policy support Models predict how 11 interventions change climate beliefs and policy support. Researchers compare predicted effects with observed effects across both outcomes. Madalina Vlasceanu, Kimberly Doell, and the International Collaboration to Understand Climate Action Social simulation University Not in cards Nov 2023
ClimateCheck 2026 Models retrieve five abstracts for a climate claim and judge the evidence in each. They also label which disinformation narratives, if any, the claim uses. ClimateCheck organizers (DFKI and colleagues) Faithful to sources University Not in cards Nov 2025
Clinical Abbreviation Sense Inventory Models expand an ambiguous abbreviation using the surrounding words in an English clinical note. Evaluators compare the answer with a clinician's annotation. University of Minnesota Medicine University Not in cards Oct 2012
Clinical Laboratory Test Reports Models extract laboratory results from report images into structured JSON. Google compares predictions with reference fields using macro and micro F1. Esraa Abdelmaksoud, Ahmed Gadallah, and Ahmed Asad Documents, charts and text in images School Not in cards Nov 2021
CLIP representation bias evaluation Models associate images of everyday objects with gender terms. The evaluation measures how far the associations depart from an equal preference across groups. Ibrahim Alabdulmohsin and colleagues Bias, fairness and politics Everyday Not in cards Mar 2024
CLIPSep Sound Separation Models isolate a requested sound from a mixture using a text or image query. The two tests mix music or everyday sounds with interfering audio. Hao-Wen Dong and colleagues at Sony and UC San Diego Audio understanding and voice University Not in cards Nov 2022
Clotho Models write a caption describing what can be heard in an audio recording. Konstantinos Drossos and colleagues Audio understanding and voice Everyday Not in cards Oct 2019
Clotho OVL-REL Test Set Models predict listener ratings for audio quality and caption relevance. Four generators produced environmental sounds from 100 captions drawn from Clotho. Minoru Kishi and colleagues Audio understanding and voice Everyday Not in cards Nov 2025
Cloud Expansion CE80-10 Agents complete a cloud network security exercise. The verifier checks the resulting cloud configuration and observed network behavior. human-intuition Secure code and review Expert Not in cards Sep 2026
Cloud Expansion CEExfil-0 Agents complete a cloud security capstone. A verifier reads the environment after the task to assess the outcome. human-intuition Capture-the-flag challenges Expert Not in cards Sep 2026
Cloud Expansion RX00 Agents complete a simulated cloud reconnaissance assessment. The verifier compares the submitted result with the recorded state of the exercise. human-intuition Capture-the-flag challenges Expert Not in cards Sep 2026
CLUE Models are fine-tuned or prompted on nine Chinese language understanding tasks, including sentence classification, sentence-pair inference and machine reading comprehension. The score averages the task results. CLUE benchmark team Reading in many languages School Not in cards Apr 2020
CLUENER2020 Models identify named entities in Chinese text across ten categories. CLUE benchmark team Reading in many languages School Not in cards
CMD-AD-Eval Models describe what happens on screen during gaps in movie dialogue. The authors compare the descriptions with timed human narration using text overlap and model judgments. University of Oxford, École des Ponts and Shanghai Jiao Tong University Video captioning Everyday Not in cards Apr 2024
Cochrane RCT Summarization Models read the titles and abstracts of clinical trials and write the conclusions of the Cochrane review that included those trials. Byron Wallace, Sayantan Saha, Frank Soboczenski, and Iain Marshall Medicine Expert Not in cards Aug 2020
Cochrane Simplification Models turn medical review abstracts into plain-language summaries. The original evaluation compares generated text with Cochrane summaries, while the hub port adds a model judge. Ashwin Devaraj and collaborators Medicine Expert Not in cards Mar 2021
Cochrane-auto Models simplify sentences, paragraphs, or abstracts from Cochrane reviews. The authors aligned technical text with plain English summaries. University of Amsterdam Medicine University Not in cards Nov 2024
COCO-O Models detect objects under natural changes in image appearance. The benchmark compares detection on drawings, cartoons, and other visual domains. COCO-O authors Detection and captioning Everyday Not in cards Jul 2023
COCO-Stuff 164K Models label each pixel with an object or background category. The benchmark adds annotations for regions such as sky, grass, and pavement. Holger Caesar and colleagues Detection and captioning Everyday Not in cards
COCOTEROS Models write a Spanish sentence that uses three required words and fits the supplied context. GPLSI, University of Alicante Reading in many languages School Not in cards May 2024
Code Review Bench (Martian) AI code review tools review pull requests from five large open-source projects. An LLM judge matches their comments to human-verified golden comments. Martian Fixing real repositories Expert Not in cards Feb 2026
CodecMOS-Accent Speech systems reconstruct recordings or clone a voice across ten English accents. Listeners score naturalness, speaker similarity, and accent similarity separately. Wen-Chin Huang, Nicholas Sanders, and Erica Cooper Audio understanding and voice Everyday Not in cards Mar 2026
CodeInsights Models solve C++ assignments and reproduce student programming behavior. The tasks compare correctness, mistakes, runtime efficiency, and failures on edge cases. CodeInsights contributors Function-level problems University Not in cards Jul 2025
CodeXGLUE A suite of 10 code tasks across 14 datasets, grouped as code to code, text to code, code to text and text to text. Tasks include clone detection, defect detection, code completion, code repair, code search, code generation and code summarization. Each task has its own metric, such as accuracy, BLEU or MRR. Microsoft Research Asia and Microsoft Function-level problems School Not in cards Feb 2021
CoffeeBench Agents manage a coffee roaster and trade with other firms for 90 simulated days. Sakana AI Running a business University Not in cards Jun 2026
Cohere North internal evals Cohere internal evals of its North agent platform: agentic question answering, spreadsheet data analysis and memory use. LLM-as-judge scoring. Cohere Documents and spreadsheets University Not in cards May 2026
COIN instructional video analysis Models recognize steps in videos of people completing everyday tasks. The benchmark supplies annotated actions and their time intervals. COIN team Video understanding Everyday Not in cards Mar 2019
CoIR Models retrieve relevant code or text across programming languages. The suite includes searches from code to code and from descriptions to code. CoIR team Function-level problems University Not in cards Jul 2024
ColDeR Models rank passages containing relevant evidence against alternatives that exploit shortcuts such as length, repetition, or word overlap. Mohsen Fayyaz and colleagues Questions that need search School Not in cards Mar 2025
COLIEE 2026 Models retrieve Canadian precedents and apply Japanese statutes. The statute retrieval track also requires a correct yes or no answer with sufficient supporting articles. COLIEE organizing committee Law University Not in cards Dec 2025
CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. Alibaba.com (Accio) Running a business Expert Not in cards Aug 2026
Common Objects in 3D v2 Models estimate camera poses from images of objects viewed from different angles. Meta compares predicted poses with the reference poses. Meta AI Spatial reasoning and counting School Not in cards
Common Syntactic Processes in Text-to-Image Generation Models generate images from sentences with different grammatical structures. Evaluators check whether each image preserves the relationships expressed in the sentence. Evelina Leivada and colleagues Image generation and editing School Not in cards Oct 2022
Commonsense Reasoning and Moral Understanding in Children’s Stories Models read Chinese fables and choose answers about events or the moral. Moral questions contribute 60 percent of the final score. Shanxi University and Hefei University of Technology Reading comprehension School Not in cards Feb 2024
CompactionBench Agents continue multi-turn coding sessions while context compaction occurs during software evolution tasks. Scores measure task quality and compaction cost. Microsoft Research Long terminal sessions Expert Not in cards Oct 2026
Company Enrichment API Benchmark Company data APIs fill profile fields from a website. Code and expert checks score headquarters, employees, founding year, social links, industry, ticker and revenue. Firecrawl Apps and MCP servers Expert Not in cards Oct 2026
COMPASS Agents plan travel across transportation, lodging and tickets while satisfying hard constraints and optimizing stated preferences over multiple turns. Apple Apps and MCP servers Expert Not in cards Oct 2025
CompassBench v1.3 Models answer Chinese and English tasks in coding, instruction following, knowledge, language, math and reasoning. OpenCompass publishes objective and subjective evaluation configurations. OpenCompass Composite indices University Not in cards Jul 2024
ComplaintsClustering Models group consumer complaints by financial product. The score measures whether each group contains one product and whether complaints about that product stay together. The Hong Kong University of Science and Technology Finance Everyday Not in cards Jul 2024
ComputeEval Models write CUDA code for handcrafted problems on kernels, runtime APIs and GPU libraries. Held-out test harnesses check correctness, and optional benchmarks time the code. NVIDIA Speed and optimization Expert Not in cards Apr 2025
Concurrency Bugs (tanishgudise) Agents repair Python packages with asynchronous or threaded code. Hidden tests reproduce each bug deterministically across 15 repair tasks. tanishgudise Fixing real repositories University Not in cards Oct 2026
CoNLL-2002 Named Entity Recognition Models mark entity spans and classify their types in Spanish and Dutch news text. Erik Tjong Kim Sang Reading in many languages School Not in cards
ConRef-STANCE-ita Models infer a user's position on Italy's constitutional referendum from a post, a repost, and a reply. Researchers score support, opposition, and neutrality. Mirko Lai, Viviana Patti, Giancarlo Ruffo, and Paolo Rosso Reading in many languages Everyday Not in cards Apr 2018
Contextual Complex Instruction-Following Retrieval Retrievers rank passages under instructions about multiple requirements. Contextual AI Following precise instructions School Not in cards Mar 2025
Contextual Invoice OCR Models extract invoice fields using a business’s past bills. Ramp counts invoices where every field matches the submitted bill. Ramp Labs Finance University Not in cards Mar 2026
Contextual Recency-Based Instruction-Following Retrieval Retrievers rank passages under instructions about recency. Contextual AI Following precise instructions Everyday Not in cards Mar 2025
Contextual Reranker Technical Documentation Benchmark Retrievers rank documents for queries supplied by enterprise customers. Contextual AI Questions that need search Expert Not in cards Mar 2025
ContextualBench CRM agents answer enterprise questions against contextual business data and receive accuracy, cost, speed, trust and safety measurements. Salesforce AI Research Questions that need search Expert Not in cards Sep 2024
Contract Intelligence Models answer questions about contract terms. Harvey compares answers with attorney reviewed labels. Harvey Law Expert Not in cards Nov 2025
Contract Workflows (Applications) Systems complete legal work involving contracts. The benchmark assesses whether each deliverable satisfies its legal and presentation criteria. Legal Benchmarks Law Expert Not in cards
ContractNLI Models decide whether a contract supports, contradicts, or omits a claim. The original benchmark also asks them to identify the passages that justify their decision. Yuta Koreeda and Christopher D. Manning Law Expert Not in cards Oct 2021
ConvoMem Models answer questions about earlier conversations, including changed facts and user preferences. The dataset contains 75,336 items across 100 fictional users. Salesforce Research Finding facts in long inputs Everyday Not in cards Nov 2025
Cooperative Codenames (Max Bittker) Two models cooperate as cluegiver and guesser to identify target words on a hidden board. The default evaluation uses 200 separately generated boards. maxbittker Multi-agent games Everyday Not in cards Mar 2026
COPIOUS biodiversity entity recognition Models mark names of organisms, places, habitats, people, and time expressions in historical biodiversity texts. The test measures how closely the marked phrases match human annotations. University of Manchester and University of the Philippines Diliman Lab and data work University Not in cards Jan 2019
CoQA Models answer a series of linked conversational questions about a passage, where later questions depend on earlier turns. Scored by word-level F1. Stanford University (Reddy, Chen and Manning) Reading comprehension Everyday Not in cards Aug 2018
CoQCat Models answer follow-up questions in Catalan using a passage and the preceding conversation. Barcelona Supercomputing Center Reading in many languages School Not in cards Sep 2023
CORAAL Models transcribe recordings of African American speech. NVIDIA reports long-form word error rates by location. CORAAL project Speech recognition and translation Everyday Not in cards
CORE English web register identification Models label English web documents by register, such as news reporting, discussion, or instructions. A document can have several labels. Researchers report F1. Veronika Laippala and colleagues Reading comprehension School Not in cards Aug 2022
Corpus of Multilingual Code-switched Soap Opera Speech Models transcribe South African television dialogue in which speakers switch between English and four Southern African languages. Stellenbosch University Speech recognition and translation Everyday Not in cards May 2018
Cortex Analyst Multi-Turn Question Rewriting Models rewrite a follow-up question so it stands alone. Mistral Large 2 judges whether the rewrite preserves the necessary conversation history. Snowflake Multi-turn conversation University Not in cards Nov 2024
CortexRetrievalBench Retrieval systems rank financial documents for a practitioner’s question. The benchmark checks whether an authoritative primary source appears at the top. micro1 Finding facts in long inputs Expert Not in cards
CoSHE-Eval Models transcribe 1,985 conversational Hindi and English clips. The references use Devanagari for Hindi and Latin script for English. Soket Labs Speech recognition and translation Everyday Not in cards Nov 2025
CounselBench-100 Agents investigate legal matters and decide which actions the evidence supports. The verifier checks decisions, changes to case records, and advice across 100 synthetic tasks. Blobfish AI Law Expert Not in cards Aug 2026
Country211 Models predict the country where a photograph was taken. OpenAI Spatial reasoning and counting University Not in cards Feb 2021
Coval Voice AI Benchmarks Speech-to-text, text-to-speech and speech-to-speech models from almost 40 providers are scored every 30 minutes on call-like audio with accents, noise and echo. Coval Speech recognition and translation Everyday Not in cards Apr 2026
COVID-Dialogue Models read a patient’s questions about COVID-19 and generate a response. Doctor replies from medical consultations serve as references. UC San Diego AI4H Medicine Expert Not in cards
Cozzo AI Receptionist Benchmark (Round 6) Voice agents quote a dental price, confirm appointments, and handle urgent calls. Cozzo checks spoken details and scores responses to interruptions. Cozzo Labs LLC Customer service Everyday Not in cards Oct 2026
CPRet Models retrieve contest solutions, duplicate problems, or full problems from simplified descriptions. Four retrieval tasks score the ranking of relevant results with NDCG@10. CUHK, Shanghai AI Laboratory, Tsinghua University and others Competitive programming Competition Not in cards May 2025
CRAG-MM Multimodal retrieval agents answer single-turn and multi-turn questions grounded in images and changing web content. The benchmark includes wearable-style egocentric images. Meta FAIR and Superintelligence Labs Questions that need search Expert Not in cards Oct 2025
Crate Bench Models identify an album and its artist from the cover image. The benchmark provides 300 test covers and supports closed-book and web-search modes. Zanussbaum General visual questions Everyday Not in cards Mar 2026
CredData Models identify hardcoded credentials in source files. Samsung publishes reviewed labels and obfuscates credential values. OpenAI scores both tokens and exact spans. Samsung Research Secure code and review University Not in cards Aug 2021
CREF Models submit forecasts before the target period begins and receive scores after the actual values become available. Andreas Auer and colleagues Real-world events University Not in cards Nov 2026
CrisisMMD Models classify disaster photographs and their accompanying posts for humanitarian use. Firoj Alam and colleagues General visual questions School Not in cards May 2018
Cross-lingual Sentiment Consistency Analysis for Classical Chinese Literature Models predict emotions in classical Chinese passages and their English translations. The test checks how accurately each prediction captures the passage and how closely the predictions agree. Xi’an Jiaotong-Liverpool University and Harbin Institute of Technology, Shenzhen Reading in many languages University Not in cards Feb 2026
Cross-lingual Test Dataset Models retrieve images from captions written in different languages. The benchmark compares retrieval against paired photographs and descriptions. Pranav Aggarwal and Ajinkya Kale Reading in many languages School Not in cards Dec 2020
CrossRE Models label relationships between named entities in sentences from six domains. The natural science test includes 400 sentences covering topics such as comets and planets. Elisa Bassignana and Barbara Plank Reading comprehension School Not in cards Oct 2022
CrossSum Models read a news article in one language and summarize it in another. The dataset aligns articles and summaries across more than 1,500 language pairs. Abhik Bhattacharjee and colleagues Reading in many languages School Not in cards Dec 2021
CS-FLEURS Models transcribe speech that switches languages within sentences. Four test sets cover 113 language pairs. People read one set aloud, and synthesizers generate the others. Brian Yan and colleagues Speech recognition and translation Everyday Not in cards Jul 2025
CSATQA Models read Korean exam passages and choose among five answers. The harness aggregates results across six question groups. HAE-RAE Reading in many languages School Not in cards Jul 2023
ČSFD Sentiment Models classify the sentiment of movie reviews from the Czech and Slovak Film Database. Tomáš Brychcín and Ivan Habernal Reading in many languages Everyday Not in cards Sep 2013
CSL Models generate paper titles and keywords or classify Chinese scientific papers by field. Scoring uses ROUGE-L for titles, Bpref for keywords, and accuracy for classification. China University of Geosciences (Beijing), Shenzhen University, Tencent AI Lab and others Reading in many languages University Not in cards Sep 2022
CSR-Bench Agents deploy and run computer-science research repositories with shell, retrieval and web-search tools. Task success measures whether the requested repository workflow completes. Amazon Science Fixing real repositories Expert Not in cards Jan 2025
CSTNews Models summarize Brazilian Portuguese news articles individually or combine reports about one event. Human summaries provide references for both tasks. Núcleo Interinstitucional de Linguística Computacional Reading in many languages School Not in cards Jun 2008
CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. Google Medicine Expert Not in cards Jan 2026
CT-RATE Models read 3D non-contrast chest CT volumes and say whether each of 18 conditions is present. Scored by macro accuracy or F1 against report-derived labels. University of Zurich and collaborators (Hamamci et al.) Medicine Expert Not in cards Mar 2024
CTF-Archive Agents solve capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. US CAISI (NIST) Capture-the-flag challenges Expert Not in cards Sep 2025
CTI-to-MITRE Models classify a threat description using MITRE ATT&CK technique labels. The HELM scenario presents a set of candidate labels. Vittorio Orbinato and colleagues Secure code and review University Not in cards Aug 2022
CUA-SHADE-Arena Monitors review computer-use agent activity in simulated workplace tasks. The evaluation measures how often they flag unsafe behavior at a fixed false positive rate. Scale AI Deception and alignment University Not in cards Aug 2025
CUAD Models review commercial contracts and highlight the clauses a lawyer would flag, across 41 categories such as governing law, non-compete and change of control. Legal experts from The Atticus Project wrote more than 13,000 labels over 510 contracts. Scores measure how well the extracted spans match the expert labels. The Atticus Project and UC Berkeley Law Expert Not in cards Mar 2021
CubeBench Agents solve Rubik's Cube tasks across symbolic, visual and partially observed settings to measure spatial reasoning and long-horizon state tracking. Princeton AI Lab Spatial reasoning and counting Expert Not in cards Dec 2025
CulturaQA Models answer questions about Greek culture, including art, mythology, politics, and everyday life. IMISLab Knowledge in many languages School Not in cards Apr 2026
CURE Models retrieve passages that answer clinicians' questions at the point of care. Clinia and collaborators Medicine Expert Not in cards Dec 2024
Current-facts set Models answer questions about current facts. Writer evaluates responses as part of its neutrality and safety study. Writer Short factual questions Everyday Not in cards Aug 2026
CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. Meta (PurpleLlama) Finding and exploiting real bugs Expert Not in cards Jul 2024
CyberSecEval Visual Prompt Injection Models read text and images where the image carries instructions that try to override the task. Scored by how often the injection succeeds. Meta (PurpleLlama) Secure code and review Expert Not in cards Jul 2024
CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. Meta (PurpleLlama) Capture-the-flag challenges Expert Not in cards Apr 2024
Cybersecurity and Privacy Legal Agent Eval Agents write legal risk memos for five synthetic privacy and incident reporting matters. The grader checks the completed memo and its coverage of legal issues. narcolepticchicken Law Expert Not in cards Jul 2026
Cyc3D Models generate 3D assets from images. The evaluator checks mesh quality and measures how much shape and identity drift across repeated rendering and reconstruction. Liwen Zhang Image generation and editing University Not in cards Aug 2026
Cyrillic Turkic language identification Models identify the language of text written in Cyrillic. The test includes Turkic languages and Russian. tatiana-merz Reading in many languages School Not in cards
CzechBankQA Models translate questions about banking records into SQL queries using a supplied database schema. Yifan Mai, Stanford CRFM Data science and SQL University Not in cards Oct 2024
D2K-Bench Agents turn expert GPU-kernel designs into efficient Triton, CUDA or CUTLASS kernels across 26 tasks. Runtime timing and design-code scores measure implementation quality. Qwen Team, Alibaba Group Speed and optimization Expert Not in cards Oct 2026
DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically. Adyen and Hugging Face Data science and SQL Expert Not in cards Jun 2025
DACSA Models summarize Catalan and Spanish newspaper articles. Editors’ summaries provide the references. Encarnación Segarra Soriano and colleagues Reading in many languages School Not in cards Jul 2022
DaLAJ Models judge the linguistic acceptability of Swedish sentences. Elena Volodina and colleagues Reading in many languages School Not in cards May 2021
DaLAJ-GED 2.0 Models decide whether a Swedish sentence contains a grammatical error. The test includes learner writing and examples from Swedish coursebooks. Elena Volodina and Yousuf Ali Mohammed, University of Gothenburg Reading in many languages School Not in cards Feb 2023
Daleel 2026 Models identify six kinds of argument in Arabic editorials and debates. They also mark the words that express each argument. Daleel organizers and QatarDebate Center Reading in many languages University Not in cards May 2026
DaNE Models locate and classify names in Danish sentences drawn from the Danish Universal Dependencies treebank. Rasmus Hvingelby and colleagues Reading in many languages School Not in cards May 2020
DanFEVER Models decide whether Danish encyclopedia evidence supports a claim, refutes it, or leaves it unresolved. The dataset contains 6,407 annotated claims. IT University of Copenhagen Faithful to sources School Not in cards Apr 2021
Danish Semantic Reasoning Benchmark Models interpret Danish metaphors, compare word senses, and infer relationships between words and statements. Centre for Language Technology at the University of Copenhagen and collaborators Reasoning in many languages School Not in cards May 2024
DANSK Models identify named entities across Danish text domains. The authors annotate a subset of Danish Gigaword using 18 entity types. Kenneth Enevoldsen and colleagues Reading in many languages School Not in cards Dec 2024
Danske Talemåder Models select the meaning of a Danish idiom from four definitions. Society for Danish Language and Literature Knowledge in many languages Everyday Not in cards
DarijaBench Models translate, summarize and classify sentiment in Moroccan Arabic. A transliteration task also tests conversion between Arabic and Latin scripts. Guokan Shang and colleagues Reading in many languages Everyday Not in cards Sep 2024
DarijaHellaSwag Models read a situation in Moroccan Arabic and choose its most plausible continuation. Guokan Shang and colleagues Reasoning in many languages Everyday Not in cards Sep 2024
DarijaMMLU Models answer academic multiple-choice questions in Moroccan Arabic. The evaluation adapts MMLU to the dialect. Guokan Shang and colleagues Knowledge in many languages University Not in cards Sep 2024
Data Extraction (Applications) Systems extract requested information from legal documents. The benchmark checks whether the resulting work satisfies the task criteria. Legal Benchmarks Law Expert Not in cards
data-eng-bench Agents build and repair dbt models across 103 retail warehouse tasks. The verifier compares materialized tables row by row against a reference solution. Snowflake Labs Data science and SQL University Not in cards Aug 2026
DataCLUE Systems improve the training data for Chinese text classification tasks. CLUE benchmark team Machine learning engineering Expert Not in cards
DataComp image encoder evaluation Models trained on different image and text collections perform the same set of downstream tasks. The evaluation compares transfer performance across 37 tasks. DataComp team General visual questions Everyday Not in cards Apr 2023
DataSciBench Agents complete open-ended data-science tasks that require data cleaning, analysis, visualization and modeling. Component checks and overall task success score the work. Z.ai, Tsinghua University and collaborators Data science and SQL Expert Not in cards Apr 2026
DAVIS 2017 Models track and segment individual objects throughout a video. The evaluation measures region overlap and boundary accuracy. DAVIS Challenge organizers Video understanding Everyday Not in cards Apr 2017
DCASE 2024 Language-Queried Audio Source Separation Models isolate the sound named in a text query. Listeners score sound quality and agreement with the query on 100 real recordings with overlapping sounds. DCASE Task 9 organizers Audio understanding and voice University Not in cards Mar 2024
DCASE 2024 Sound Scene Synthesis Models generate a four-second sound scene from a text description. Listeners rate how well the foreground and background match the prompt and assess audio quality. DCASE Audio understanding and voice University Not in cards Dec 2023
DCS 2018 Sanskrit Word Segmentation Models split Sanskrit compounds and reverse sound changes at word boundaries. The sentence score counts an answer only when all its words match. Oliver Hellwig and Sebastian Nehrdich Reading in many languages University Not in cards Aug 2018
DDIExtraction 2013 Models identify drug names in biomedical text and classify interactions between pairs of drugs. The OpenMed port covers interaction classification. SemEval 2013 Task 9 organizers Medicine University Not in cards Jun 2013
DDisCo Models assign one of three coherence ratings to a Danish Reddit post or Wikipedia passage. The test file contains 101 Reddit posts and 100 Wikipedia passages. Aarhus University and the Alexandra Institute Reading in many languages School Not in cards Feb 2022
DDXPlus Models diagnose simulated patients from symptoms, clinical history, and demographic information. The OpenMed port asks for the most likely condition among 49 possibilities. DDXPlus team Medicine Expert Not in cards May 2022
DealBench-100 Agents perform 100 investment banking workflows across ten synthetic transactions. Deterministic checks grade calculations, decisions, committed changes, and consistency between models and presentations. Blobfish AI Finance Expert Not in cards Sep 2026
DEBAGREEMENT Models classify whether a reply agrees with a comment, disagrees, or takes a neutral position. Human labels provide the reference answers. University of Oxford and Scale AI Reading comprehension Everyday Not in cards Oct 2021
DecepEval LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance. Xi'an Jiaotong University Deception and alignment Everyday Not in cards Oct 2026
Deceptive Reasoning Exposure Suite Monitors inspect model reasoning and answers to detect cases where a harmless looking response conceals deceptive reasoning. Gray Swan and collaborators Deception and alignment Expert Not in cards Sep 2025
Deep Memory Retrieval Agents answer one fact question from earlier sessions in a multi-session conversation. A judge compares the answer with the target fact. MemGPT and Letta Reading comprehension Expert Not in cards Oct 2023
DeepAmbigQA Search-augmented models answer multi-hop questions that combine entity-name ambiguity with complete answer-set retrieval. Apple Questions that need search University Not in cards Nov 2025
Deepgram Voice AI Benchmarks Speech models are compared on word error rate across production audio in 20 languages, while speech synthesis models are compared on read-aloud and hard-prompt sets. Deepgram Speech recognition and translation Expert Not in cards Apr 2026
DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. Google DeepMind Simulated robots University Not in cards Sep 2024
DeepMind Fast Language Learning Tasks Models learn mappings between unfamiliar words and meanings from short examples. Held-out tasks measure how well they infer new word meanings. Google DeepMind Language reasoning School Not in cards Jul 2017
DeepResearchEval Deep research systems answer complex research tasks that require evidence from many web sources. A persona-driven pipeline writes the tasks and keeps only those that need outside search. A model judge scores each report on general and task-specific dimensions, and an agent fact-checks the report's statements with web search even when citations are missing. Infinity Lab (Shanda Group) and Nanyang Technological University Deep research Expert Not in cards Jan 2026
DeepResearchEval v1 Research agents prepare reports. The score is average report quality on the original evaluation tasks. Infinity Lab (Shanda Group) and Nanyang Technological University Deep research Expert Not in cards Jan 2026
DeepSDO Description Models describe solar features visible in full-disk images from the Solar Dynamics Observatory. The scorer compares generated captions with human descriptions. Korea Astronomy and Space Science Institute and collaborators Lab and data work University Not in cards
Defects4J A collection of 854 real, reproducible bugs from 17 open-source Java projects, each with the buggy and fixed versions and tests that expose the bug. Program repair studies ask a model to fix each bug, and a fix counts when the project's tests pass. It also serves fault localization and test generation research. University of Washington (René Just and coauthors) Fixing real repositories Expert Not in cards Jul 2014
DeGennaro Workplace sample Agents act as an employee’s assistant using 80 simulated workplace tools and anonymized company records. Four tasks grade the final state. doe General assistants Expert Not in cards Nov 2026
DELEGATE-52 Models perform long delegated document-editing workflows across 52 professional domains. Scores measure artifact fidelity after repeated transformations. Microsoft Research Documents and spreadsheets Expert Not in cards Apr 2026
Dementia Family Caregiver Detection Models identify English posts whose authors report having a family member with dementia. Organizers score F1 for that class on 1,769 test posts. Ari Z. Klein and colleagues Medicine Everyday Not in cards May 2022
Demosthenes Argument Classification Models identify premises and conclusions in European Union court decisions on fiscal state aid. They also classify premise types and legal reasoning schemes. Demosthenes team, University of Bologna Law Expert Not in cards Oct 2022
Describable Textures Dataset Models assign a texture description to a photograph based on how its surface looks. Mircea Cimpoi and colleagues General visual questions Everyday Not in cards Nov 2013
DESED Models identify when sound events occur in domestic environments. Meta compares frame-level predictions across audio encoders. DESED project Audio understanding and voice Everyday Not in cards
DesignJanus People rate generated objects from six viewpoints and mark inconsistencies in shape, materials, and identity. The harness compares human ratings with automatic measures. j-jw-yim Image generation and editing Everyday Not in cards Mar 2026
Detecting Entities in the Astrophysics Literature Models mark and classify named entities in excerpts from astrophysics papers, including telescopes, astronomical objects, and grants. Organizers score entity spans and token labels. Center for Astrophysics, Harvard & Smithsonian, and Charles University Lab and data work University Not in cards May 2022
DETESTS-Dis 2024 Models detect racial stereotypes in Spanish comments and posts and decide whether the stereotypes are implicit. Organizers score majority labels and distributions of annotator judgments. IberLEF DETESTS-Dis organizers Reading in many languages University Not in cards Feb 2024
DETOUR Agents ask a simulated user for clues and identify a partially remembered item. Patronus AI Questions that need search Everyday Not in cards Feb 2026
Developer Retrieval Benchmark Search systems retrieve repositories, documentation pages and issue or pull request fixes. Deterministic URL matching scores recall and reciprocal rank. Firecrawl Apps and MCP servers Expert Not in cards Oct 2026
DevOpsBench-100 Agents resolve 100 operations and reliability tasks in a simulated online store. The verifier compares system state before and after their work. Blobfish AI Terminal tasks Expert Not in cards Aug 2026
Deysi Spam Detection Models classify English messages as spam or legitimate. The hub adapter grades the label and response format across 2,725 test messages. Deysi Reading comprehension Everyday Not in cards Apr 2023
DGCL Reward Hacking Models decide whether corporate actions comply with Delaware law and provide supporting calculations. Fifty evaluation cases distinguish correct headline decisions from fully correct answers. narcolepticchicken Law Expert Not in cards May 2026
DialectSentEval 2026 Models classify sentiment in five Arabic dialects. In a separate task, they rewrite a sentence with the opposite sentiment while preserving its topic and dialect. DialectSentEval organizers Reading in many languages Everyday Not in cards May 2026
DiDeMo Models locate the time interval in a video that matches a natural language description. Lisa Anne Hendricks and colleagues Video understanding Everyday Not in cards Aug 2017
DigiKlausur Assisted Short Answer Grading Models grade responses to 17 neural network exam questions. One human grader assigned zero, one, or two points to each of 646 entries. DigiKlausur at Bonn-Rhein-Sieg University of Applied Sciences Other professions University Not in cards Feb 2019
DIHARD III Models identify who speaks when across eleven recording domains. The benchmark includes overlapping speech. Ryant and colleagues Audio understanding and voice Everyday Not in cards Sep 2020
Dillen herbarium specimen label benchmark Models transcribe specimen labels from images of pressed plants. The collection pairs 1,800 images from nine European herbaria with transcribed records. Meise Botanic Garden and eight partner collections Documents, charts and text in images School Not in cards Nov 2018
DIODE Models estimate depth in indoor and outdoor scenes. The dataset provides depth measurements from a laser scanner for comparison. Igor Vasiljevic and colleagues Spatial reasoning and counting School Not in cards Aug 2019
DIOR Models locate and classify objects in remotely sensed images. The benchmark measures detection accuracy across different object sizes and backgrounds. Ke Li and colleagues Detection and captioning University Not in cards Aug 2019
DIOR-RSVG Models draw a box around the object named in a description of an aerial image. The released test split contains 7,500 descriptions. Northwestern Polytechnical University Detection and captioning Everyday Not in cards Oct 2022
Dirty Integration Agents collect every record from a paginated API despite unreliable cursors, rate limits, changing field types and timeouts. buildok Apps and MCP servers University Not in cards Aug 2026
DISCFinLLMReranking Models rank Chinese news articles for questions about financial events. Each question has two or three relevant articles and ten distractors. The Hong Kong University of Science and Technology Finance University Not in cards Jul 2024
Discharge Me! Models draft discharge instructions and brief hospital course summaries from patient records. The shared task compares model output with clinical discharge documentation. Justin Xu and colleagues Medicine Expert Not in cards
DiSCoMaT Models extract material identifiers, constituents, percentages, and units from tables in materials science papers. The original test contains 737 tables from 326 papers. Indian Institute of Technology Delhi Lab and data work University Not in cards Jun 2022
DiscoveryBench Agents analyze datasets and produce data-driven hypotheses that evaluators score against structured evidence and domain criteria. Ai2 and Princeton NLP AI research tasks Expert Not in cards Jul 2024
Disengagement-Aware Student Simulators Tutors interact with simulated learners who vary in engagement. The authors compare five tutors and validate the student behaviors against real tutoring sessions. The University of Hong Kong Other professions University Not in cards Sep 2026
DISPLACE-M 2026 Models identify speakers, transcribe their words, identify medical topics and summarize conversations between community health workers and patients. Each task has its own score. Indian Institute of Science and DISPLACE-M organizers Audio understanding and voice Everyday Not in cards Jan 2026
DisTEMIST Models locate disease mentions in Spanish clinical cases and link them to SNOMED CT codes. The organizers score both tasks with microaveraged F1. Barcelona Supercomputing Center and BioASQ Medicine University Not in cards Dec 2021
Diving48 Models classify competitive dives from videos by distinguishing their movement sequences. Yingwei Li and colleagues Video understanding Expert Not in cards Sep 2018
DOE sample pack Agents complete five simulated workplace assignments in law, private equity, and finance operations. Deterministic checks grade figures and application state, while model judges grade written work. doe Tasks from real jobs Expert Not in cards Oct 2026
Dojo Mini Bench Agents complete tasks in LinkedIn, Linear and Gmail through Dojo. Each task has its own completion check. Chakra Labs Apps and MCP servers Everyday Not in cards Oct 2025
Dollar Street Models recognize household objects in photos from families at different income levels. The benchmark supports comparisons across regions and income groups. Gapminder and collaborators General visual questions Everyday Not in cards
Dolly Machine Translated v2 Models answer a curated subset of Dolly prompts translated into 66 languages with newer systems such as Command A Translate. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. Cohere Labs Chat and instructions in many languages Everyday Not in cards Apr 2026
Domain Intelligence Benchmark Suite Models extract structured data, call functions, and answer questions using retrieved material. Databricks compares performance on enterprise tasks. Databricks Running a business University Not in cards Dec 2024
Dosimeter AI Disinformation Benchmark AI assistants answer questions about contested claims. Dosimeter pools its source checks with published studies that flag citations or repetition of false claims. Dosimeter Long-form factuality Everyday Not in cards Oct 2026
Dosimeter US midterm source audit AI assistants answer ten English questions about claims circulating before the US midterms. Dosimeter counts answers that cite a listed source, including refutations. Dosimeter Long-form factuality Everyday Not in cards Oct 2026
doteBench Models edit English and Chinese recordings using a transcript that marks where each change belongs. The evaluator checks edit success and preservation outside those regions. Xiaohongshu and Shanghai Jiao Tong University Audio understanding and voice University Not in cards Aug 2026
DP-Bench Document parsers and vision models convert PDFs and scanned pages into structured HTML or Markdown. Output is compared with reference structure. Upstage Documents, charts and text in images Everyday Not in cards Oct 2024
DravidianLangTech 2026 Hope Speech Detection in Code-Mixed Tulu Models classify the tone of comments that mix Tulu with other languages. A second task distinguishes five kinds of hope or hopelessness. DravidianLangTech 2026 organizers Reading in many languages Everyday Not in cards Nov 2025
DravidianLangTech 2026 Multilevel Political Meme Classification in Tamil and Malayalam Models identify a political meme's stance and target from its image and text in Tamil or Malayalam. DravidianLangTech shared task organizers General visual questions Everyday Not in cards Nov 2025
DravidianLangTech 2026 Tamil Dialect Speech Recognition and Classification Models identify which of four regional Tamil dialects a speaker uses. A separate track tests how accurately models transcribe the speech. DravidianLangTech 2026 organizers Speech recognition and translation Everyday Not in cards Nov 2025
Drawing with LLMs Models generate SVG code from text prompts. Kaggle renders each image and combines visual question answering with an aesthetic score. Kaggle Image generation and editing Expert Not in cards Feb 2025
Dreaddit Models classify Reddit posts according to whether their authors express stress. The hub port adds rewards for reasoning and answer format. Elsbeth Turcan and Kathleen McKeown Wellbeing and sycophancy Everyday Not in cards Oct 2019
DreamGen Bench Video generation benchmark measuring how well video models adapt to a target robot embodiment. Part of NVIDIA's DreamGen pipeline. NVIDIA GEAR Video world models Not in cards May 2025
DroidTask Agents select actions for 158 Android tasks across 13 apps. The original completion score requires every action to match the recorded sequence. Tsinghua University, University of Notre Dame, and Microsoft Research Asia Phones Everyday Not in cards Aug 2023
DrugProt Models read biomedical abstracts and identify which of 13 labeled relationships connects a marked chemical and protein. BioCreative VII DrugProt organizers Genomics and life sciences University Not in cards
DTU Multi-View Stereo Models reconstruct objects from multiple camera views. The benchmark measures the accuracy and completeness of reconstructed surfaces against scanned reference geometry. Technical University of Denmark Spatial reasoning and counting School Not in cards
Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. Dual Exam authors Interactive world models Everyday Not in cards Oct 2026
Dubai-CC Models describe land use changes between two satellite views of Dubai. The Change3D team used 150 of the 500 image pairs for testing. Genc Hoxha, Seloua Chouaf, Farid Melgani, and Youcef Smara Detection and captioning School Not in cards
Duidelijke Taal Models rewrite Dutch text in simpler language. Human-written simplifications provide reference answers. Instituut voor de Nederlandse Taal Reading in many languages School Not in cards Dec 2025
DuMateBench Agents complete real-world productivity and office-work tasks in sandboxed workspaces with tools. Task evaluators check the requested artifact and record a reward. Baidu Running a business Expert Not in cards Aug 2026
Dutch DPA national voting advice audit Chatbots rank three Dutch political parties for a fictional voter. The regulator checks whether the first recommendation matches the party used to construct the voter’s answers. Dutch Data Protection Authority Bias, fairness and politics Everyday Not in cards Oct 2025
Dutch Model Benchmark Models perform Dutch language understanding tasks, including reasoning about the causes and effects of events. Wietse de Vries and colleagues Reasoning in many languages School Not in cards Dec 2023
Dynamic Programming (m8ngotree) Models solve generated optimization and counting problems, including knapsack, coin change, and edit distance. A problem-specific algorithm verifies each answer. m8ngotree Language reasoning University Not in cards Sep 2025
Dynamic-SUPERB Phase-2 Spoken language models follow text instructions about audio across 180 tasks in speech, music and other sounds. Tasks include classification, regression and sequence generation, and researchers keep contributing new ones. Models run zero-shot, without task-specific training. Dynamic-SUPERB team (Chien-yu Huang and coauthors) Audio understanding and voice Everyday Not in cards Nov 2024
E2E-SWE Agents build complete software repositories from natural language specifications in empty workspaces. Hidden tests require every generated project to install and run. Meta FAIR and Superintelligence Labs Building whole projects Research frontier Not in cards Sep 2026
Earnings-21 Models transcribe long earnings calls. The benchmark tests recognition of financial terminology in real conversations. Rev Speech recognition and translation Everyday Not in cards Apr 2021
Earnings-22 Models transcribe English earnings calls from speakers with different accents. NVIDIA reports long-form word error rate. Rev Speech recognition and translation Everyday Not in cards Mar 2022
Earth-Bench-Pro Agents complete Earth observation workflows under three conditions. They follow instructions, plan from supplied observations, or obtain observations at runtime. OpenDataLab and collaborators Apps and MCP servers University Not in cards Sep 2026
EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. Shanghai AI Lab (InternRobotics) Simulated robots Everyday Not in cards Jun 2026
EBM-NLP Models mark the words that describe participants, interventions, and outcomes in clinical trial abstracts. Medical professionals supplied the test labels. Northeastern University and collaborators Medicine University Not in cards Jun 2018
EBM-NLPmod Models extract participants, interventions, comparators, and outcomes from trial titles and methods sections. Annotators separate comparison treatments from interventions. UTHealth Houston and Yale University Medicine University Not in cards Aug 2023
ECHR Binary Violation Classification Models read the facts of a European Court of Human Rights case and predict whether the court found a violation. Ilias Chalkidis and colleagues Law Expert Not in cards Jun 2019
ECHR Legal Corpus for Argument Mining Models identify argumentative clauses, link related clauses, and distinguish premises from conclusions in European Court of Human Rights rulings. Prakash Poudyal and colleagues Law University Not in cards Dec 2020
Economic Dispatch Models assign output levels to generators to meet demand at minimum cost. The grader checks capacity limits, supply balance, and cost against an exact solution. jwilksbooth Electronics Expert Not in cards Jul 2026
EDB-Bio Models read experimental evidence and estimate the probability that a scientific hypothesis is true. Edison Scientific (formerly FutureHouse) Research science Expert Not in cards
EDIS Models retrieve news images and their associated text using queries that refer to specific people, places, or events. Siqi Liu and colleagues General visual questions School Not in cards May 2023
EduClaw-Bench Agents tutor a simulated learner over 30 days. The benchmark tracks learning gains, responsiveness, helpfulness, and curriculum design across 55 scenarios. Korea University and collaborators Other professions University Not in cards Aug 2026
EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. Aryan Ahmed and colleagues Other professions University Not in cards Oct 2026
EduFair-Bench Tutors teach the same simulated student under different demographic descriptions. A judge grades individual turns and whole conversations to measure changes in teaching quality. EPFL Other professions University Not in cards Sep 2026
Effective FPS Models decide whether video frames are consecutive or duplicated to simulate a higher capture rate. The evaluator checks each prediction against the construction of the clip. referee-lab Video understanding University Not in cards Aug 2026
Ego-Exo4D Models estimate human pose during skilled activities filmed from first-person and external viewpoints. Meta reports joint and mesh errors for physical and procedural activities. Ego-Exo4D consortium Spatial reasoning and counting School Not in cards Dec 2023
Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. Yu Dong and colleagues Simulated robots University Not in cards Nov 2026
Ego4D Goal-Step Models find when a goal or step occurs in first-person video. They also identify the current step or locate a step described in words. Meta FAIR Video understanding Everyday Not in cards Dec 2023
Ego4D State Change Object Detection Models locate the object that changes in a first-person video. The official test scores its bounding box in the frame where the change begins. Ego4D consortium Video understanding Everyday Not in cards Oct 2021
EgoCross Models answer questions about events in first-person videos. The challenge scores accuracy on 957 four-choice questions across four domains. Yanjun Li and colleagues Video understanding Everyday Not in cards Aug 2025
EgoPER Error Detection Models watch first-person cooking videos and flag incorrect steps. The evaluator checks accuracy by step and measures error detection by frame. Shih-Po Lee and colleagues Video understanding Everyday Not in cards Dec 2023
EgoProceL Models find recurring steps in first-person videos and mark when each step occurs. The videos show 16 tasks, including cooking and assembling computers. IIIT Hyderabad and IIT Delhi Video understanding Everyday Not in cards Jul 2022
EgoToM Video models infer a camera wearer’s goals, beliefs, and future actions from egocentric clips. Multiple-choice questions test theory of mind. Meta FAIR and Superintelligence Labs General visual questions University Not in cards Mar 2025
EgyMMLU Models answer multiple-choice subject questions in Egyptian Arabic. The evaluation tests knowledge in a dialect underrepresented in standard Arabic exams. Abdellah El Mekki and colleagues Knowledge in many languages University Not in cards May 2025
EHRQA (Google) Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning. Google Medicine Expert Not in cards Jul 2025
EHRSHOT Models predict clinical outcomes from longitudinal electronic health records with few labeled examples for each task. Michael Wornow and colleagues Medicine Expert Not in cards Jul 2023
EHRSQL Models convert clinical questions into SQL queries over electronic health records. The benchmark includes questions that the database cannot answer. Gyubok Lee and colleagues Data science and SQL University Not in cards Jan 2023
ELCardioCC 2025 Models find clinical mentions in Greek cardiology discharge letters, link them to ICD-10 codes, and assign codes to each letter with supporting phrases. Aristotle University of Thessaloniki and collaborators Medicine Expert Not in cards Feb 2025
Election-Day AI Poisoning in Germany AI search services answer ten German questions about documented political claims. The audit counts answers citing domains on lists of Russian state media or the Pravda network. Dosimeter Faithful to sources Everyday Not in cards Sep 2026
Electoral Hallucinations AI services answer questions about three Scottish Parliament constituencies. Two reviewers check responses against election facts and report the percentage that are entirely accurate. Demos Long-form factuality Everyday Not in cards May 2026
ELLIPSE Holistic Essay Scoring Models assign one overall English proficiency score to each essay on a five-point scale. The reference is the overall human grade. Scott Crossley and colleagues Other professions University Not in cards Feb 2023
Embodied Agent Arena Vision-language agents solve geometry, spatial reasoning, affordance, planning, and manipulation tasks. The 1,000-case evaluation combines 32 established sources with GeoProbe. HKUST Guangzhou, CUHK, and KnowinAI Embodied reasoning Everyday Not in cards Oct 2026
EmbodiedCLUE-VLA RoboTwin Evaluation Robot policies perform ten two-handed manipulation tasks under randomized RoboTwin 2.0 Hard conditions. The score averages task success rates over twenty episodes per task. SuperCLUE Simulated robots Everyday Not in cards Aug 2026
EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. Rajat Bhattacharjya and colleagues Simulated robots University Not in cards Nov 2026
EMDB Models reconstruct human pose and body shape in everyday scenes. The benchmark compares joint positions with electromagnetic measurements. ETH Zurich Spatial reasoning and counting School Not in cards Aug 2023
EmpatheticDialogues Models reply to a person describing an emotional experience. Human conversations provide reference responses for evaluating empathy and relevance. Hannah Rashkin and colleagues Multi-turn conversation Everyday Not in cards Nov 2018
Employment Law Legal Agent Eval Agents review employment case files and write risk memos. The grader checks whether the memos identify planted legal issues and avoid unsupported findings. narcolepticchicken Law Expert Not in cards Jul 2026
EncQA Vision-language models answer chart questions that isolate visual encoding channels and analytic operations. Apple Documents, charts and text in images University Not in cards Aug 2025
Encyclopedic VQA Models answer questions about detailed properties of pictured objects and places. The Gemini Embedding 2 report measures whether they retrieve the correct answer. Google Research General visual questions Everyday Not in cards Jun 2023
EndoVis 2018 Robotic Scene Segmentation Models segment structures and surgical instruments in video from robot-assisted operations. The evaluation compares predicted masks with reference labels. EndoVis challenge organizers Video understanding Everyday Not in cards
EndoVis18-VQA Models answer questions about organs, tools, tool positions, and interactions in images from robotic kidney surgery. The benchmark accepts class labels or sentence answers in separate tasks. National University of Singapore and collaborators Medicine Expert Not in cards Jun 2022
ENEM Challenge Models answer multiple-choice questions from Brazil’s national high school examination in Portuguese. Eduardo Garcia and contributors Knowledge in many languages School Not in cards Jan 2024
Enterprise PII Models respond to 100 prompts that request sensitive business information. The evaluation checks whether the responses disclose it. Patronus AI and Hugging Face Deception and alignment University Not in cards Jan 2024
Enterprise Scenarios Creative Writing Models write stories in response to 100 prompts. A model grades how engaging the stories are. Patronus AI and Hugging Face Creative writing Everyday Not in cards Jan 2024
Enterprise Scenarios Customer Support Dialogue Models answer customer support questions using product information and conversation history. A judge scores their responses. Patronus AI and Hugging Face Multi-turn conversation Everyday Not in cards Jan 2024
Enterprise Scenarios Leaderboard Models complete six tasks drawn from workplace settings. The leaderboard combines their scores across all six tasks. Patronus AI and Hugging Face Running a business University Not in cards Jan 2024
Enterprise Scenarios Toxicity Models respond to 100 prompts. The Perspective API scores how often their responses contain toxic language. Patronus AI and Hugging Face Deception and alignment Everyday Not in cards Jan 2024
Enterprise Task Pareto Frontier (Glean) Models complete 1,000 workplace tasks. A model compares paired answers for correctness, task completion, and readiness for use. Glean estimates preference ratings and cost per task. Glean Tasks from real jobs University Not in cards Aug 2026
Enterprise Vendor Eval Agents review vendor documents, question an internal IT team, and recommend whether to approve a purchase. The grader checks their decision and identified requirements. musa8026 Running a business University Not in cards Jul 2026
Entity-Deduction Arena Models ask questions to identify a hidden entity in multi-turn games. Success, question count and a combined score measure clarification, state tracking and planning. Apple General assistants Everyday Not in cards Feb 2024
EPIC-KITCHENS VISOR Hand Object Segmentation Models outline each hand and the object it touches in a kitchen image. They also identify which hand appears and whether it touches anything. University of Bristol, University of Michigan, and University of Toronto Detection and captioning Everyday Not in cards Aug 2022
EPIC-KITCHENS VISOR Semi-Supervised Video Object Segmentation Models receive masks for hands and objects in the first frame of a kitchen video. They track those objects through later frames, including after occlusion. University of Bristol, University of Michigan, and University of Toronto Video understanding Everyday Not in cards Aug 2022
EPIC-KITCHENS-100 action anticipation Models predict a person's next action from first-person kitchen video. The benchmark scores predicted verbs, nouns, and their combinations. EPIC-KITCHENS consortium Video understanding Everyday Not in cards Jun 2020
Epic-tent-O Models watch someone pitch a tent and flag procedural mistakes using only the video so far. The test includes 15 videos. Sapienza University of Rome and University of Catania Video understanding Everyday Not in cards Apr 2024
EPRSTMT Models classify Chinese product reviews as positive or negative after learning from a small labeled sample. Accuracy measures agreement with the sentiment labels. CLUE benchmark team Reading in many languages Everyday Not in cards May 2021
ERPBench-100 Agents execute 100 business workflows across ten simulated enterprise resource planning systems. Checks grade decisions, calculations, committed state, and handoffs on a 100-point scale. Blobfish AI Apps and MCP servers Expert Not in cards Sep 2026
ERRnews broadcast summarization Models summarize Estonian radio news from automatic transcripts. TalTech compares each summary with the broadcaster’s written summary using ROUGE. Tallinn University of Technology Reading in many languages School Not in cards Nov 2022
ESC-50 Models identify an environmental sound from a short recording. Karol J. Piczak Audio understanding and voice Everyday Not in cards Oct 2015
Escalation Router Models predict whether Qwen2.5-VL 7B correctly identifies visible hands or grippers in a clip. The reference label comes from two observed runs per clip. referee-lab Video understanding University Not in cards Aug 2026
ESCI Shopping Queries Dataset Models rank products for a shopping query and distinguish exact matches, substitutes, complements, and irrelevant items. Amazon Questions that need search Everyday Not in cards Jun 2022
EsCoLA Models judge the grammatical acceptability of Spanish sentences taken from linguistic publications. Universitat Pompeu Fabra and Barcelona Supercomputing Center Reading in many languages School Not in cards May 2024
EstNER Models identify named entities in manually annotated Estonian news and social media text. Kairit Sirts Reading in many languages School Not in cards May 2023
ETH3D Models reconstruct scene geometry from images. Meta evaluates depth predictions against reference measurements on the ETH3D test scenes. Thomas Schöps and colleagues Spatial reasoning and counting School Not in cards
ether0 benchmark Models answer chemistry problems from several task types. The benchmark checks their answers with automated reward functions. Edison Scientific (formerly FutureHouse) Research science Expert Not in cards Jun 2025
ETHICS Models predict common moral judgments about text scenarios covering justice, well-being, duties, virtues and commonsense morality. UC Berkeley (Hendrycks et al.) Deception and alignment Everyday Not in cards Aug 2020
ETVA Text-to-video systems are scored on fine-grained prompt alignment through generated atomic questions and video-language answers. Apple Video generation Expert Not in cards Mar 2025
EUR-Lex-Sum Models summarize European Union legislation in the language of the source document. Official summaries provide references across EU languages. Dennis Aumiller and colleagues Law University Not in cards Oct 2022
European Parliament 2010 State of the Union sentiment validation Models rate the sentiment of passages from the European Parliament’s 2010 State of the Union debate. The original evaluation correlates paragraph scores with human ratings. Sven-Oliver Proksch, Will Lowe, Jens Wäckerle, and Stuart Soroka Reading comprehension Everyday Not in cards Aug 2018
European Parliament election chatbot audit Chatbots answer ten European election questions in ten languages. Reviewers score procedural answers for accuracy and political answers for neutrality. Democracy Reporting International Long-form factuality Everyday Not in cards Apr 2024
EvaCun 2025 Models identify dictionary forms and restore masked words in Akkadian and Sumerian tablet texts. Organizers score accuracy on held-out tablets. EvaCun organizers Reading in many languages Expert Not in cards Oct 2024
EvaHan 2026 Models transcribe printed and handwritten Chinese books and locate page elements such as illustrations and annotations. EvaHan organizers Documents, charts and text in images University Not in cards Nov 2025
EvalConvoLearn Models simulate learners in tutoring conversations. The evaluator compares their answers, mistakes, questions, and turn lengths with patterns in real student conversations. Baptiste Moreau-Pernet Other professions University Not in cards Jun 2026
EvalCrafter Text-to-video models generate clips for 700 prompts drawn from real user requests. The clips are scored with 17 automatic metrics for visual quality, content quality, motion and text alignment. Weights fitted to human ratings combine the metrics into one score. Tencent AI Lab and City University of Hong Kong Video generation Everyday Not in cards Oct 2023
EvaLearn Models solve 648 problems in 182 sequences to measure learning from feedback across six task types. ByteDance Seed Task length University Not in cards Jun 2025
Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment Chatbots respond to 30 questions about suicide. Researchers record how often each chatbot answers directly, grouped by five risk levels that clinicians assigned to the questions. RAND and Harvard Medical School collaborators Wellbeing and sycophancy Expert Not in cards Aug 2025
Evaluation of Essay On-Topic Graded Comments Models rate how closely each Chinese school essay follows its prompt and explain their rating. Written feedback supplies 60 percent of the total score. East China Normal University Reading in many languages University Not in cards Feb 2025
Event-based Segmentation Dataset Models segment objects in cluttered indoor scenes filmed from a moving robot arm. The benchmark provides reference masks for comparison. Xiaoqian Huang and colleagues Video understanding Everyday Not in cards Feb 2023
Evidence Inference 2.0 Models classify the direction of a treatment effect in a trial report and identify text that supports their answer. Northeastern University and King's College London Medicine University Not in cards May 2020
Evidence Sufficiency Evaluation Models classify whether the evidence supports an answer across 72 cases. The harness uses paired evidence changes, controls, and strict label matching. Enas Amin Faithful to sources School Not in cards Oct 2026
EvoArena PersonaMem-Evo Agents solve questions about changing user preferences. The score is the published accuracy across versioned tasks. MIT Learning from the context Expert Not in cards Jun 2026
EvoArena SWE-Chain-Evo Agents solve software tasks that change across versions. The score is the published accuracy across versioned tasks. MIT Learning from the context Expert Not in cards Jun 2026
EVVE Models retrieve other videos showing the same real-world event as a query video. Jérôme Revaud and colleagues Video understanding School Not in cards Jun 2013
EWMBench Embodied world models generate robot manipulation videos from language. Scored on scene consistency, motion correctness, semantic alignment. AgiBot Video world models Not in cards May 2025
Exam-et Models answer multiple-choice questions from Estonian high school tests. TalTechNLP Knowledge in many languages School Not in cards Sep 2024
EXAMS Models answer high school exam questions in 16 languages across many subjects. Arabic reports use the Arabic subset. Sofia University and collaborators (Hardalov et al.) Knowledge in many languages School Not in cards Nov 2020
ExeWorldBench Coding agents build executable code-based 3D worlds that capture how a world works, testing world understanding and its realization in code. University of Science and Technology of China Physical prediction Expert Not in cards Oct 2026
ExFunTube Models explain why a moment in a short video is funny. The authors compare explanations with human references and ask viewers to rate them. Seoul National University and Allen Institute for AI Video captioning Everyday Not in cards Jul 2023
Expert Forecasts of Real-Effort Experiment Models predict how incentives change the number of alternating key presses workers make in ten minutes. Researchers compare predictions with observed effort across 15 conditions. Stefano DellaVigna and Devin Pope Social simulation University Not in cards Jul 2015
ExPUNations Models explain English puns and write new ones from keywords. Human raters check whether a generated pun supports both meanings. Amazon Alexa AI, University of Southern California and UCLA Creative writing Everyday Not in cards Oct 2022
ExtremeDrive Driving models plan trajectories from road recordings. A simulator executes the plans and measures success during emergency maneuvers and ordinary driving. Tsinghua University and collaborators Simulated robots Everyday Not in cards Nov 2026
EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. Google (EyePACS images) Medicine Expert Not in cards Jul 2025
Factorio Play Agents build factories one action at a time in a Factorio simulator. The evaluator measures factory output on held-out layouts. divagr Video games University Not in cards Sep 2026
Facts about Singapore Models answer multiple-choice and true-or-false questions about Singapore. An exact match with the answer key earns credit. AI Verify Foundation Short factual questions Everyday Not in cards May 2024
FAD Music Listening Test Models estimate acoustic and musical quality for ten-second music excerpts. The test compares predictions with human ratings from ten listeners. Microsoft Research and Azalea Gui Audio understanding and voice Everyday Not in cards Nov 2023
FairFace Models classify face images using the dataset’s annotated age, gender, and race categories. The benchmark compares recognition performance across demographic groups. Kimmo Kärkkäinen and colleagues General visual questions Everyday Not in cards Aug 2019
FairStress Models choose between candidates as demographic cues and persuasive statements change. The benchmark measures how often models change a correct decision. Allen Institute for AI Bias, fairness and politics Everyday Not in cards Oct 2026
FaithDial Models produce conversational answers grounded in supplied knowledge. Nouha Dziri and colleagues Faithful to sources School Not in cards Apr 2022
Fake-or-Real Models distinguish synthetic speech from real recordings. NVIDIA evaluates this task within AIR-Bench. Ricardo Reimao and Vassilios Tzerpos Audio understanding and voice Everyday Not in cards
Fake.Br Corpus Models classify Brazilian Portuguese news articles as true or false. The corpus pairs 3,600 false articles with reports on the same topics and provides length-matched versions. Interinstitutional Center for Computational Linguistics Faithful to sources University Not in cards Aug 2018
FakeRecogna Models classify Brazilian news articles as true or false. The authors collected 5,951 items per class from different sources, so outlet and writing style can influence predictions. São Paulo State University Faithful to sources University Not in cards Jun 2021
FakeTrueBR Models classify Brazilian Portuguese news articles as true or false. The authors paired false stories from Boatos.org with related reporting from G1 and Folha de S.Paulo. Federal University of Santa Catarina and London School of Economics and Political Science Faithful to sources University Not in cards Mar 2023
Fashion IQ Models retrieve a garment using a reference photograph and text describing how the target garment differs. Hui Wu and colleagues General visual questions Everyday Not in cards May 2019
FastAPI Competency Models implement FastAPI applications from written specifications. The grader issues HTTP requests and checks responses across 34 tasks. toolshed Building whole projects University Not in cards Jul 2026
FDA and CDC VAERS clinical and temporal annotations Models locate clinical features and time expressions in vaccine safety reports. Annotators also linked clinical features to their timing across 1,000 reports. US Food and Drug Administration and Centers for Disease Control and Prevention Medicine University Not in cards Jun 2018
FDA Information Extraction (Based) Models extract requested field values from chunks of FDA documents. The scorer checks whether the expected value appears in the answer. Hazy Research (Based evaluation) Faithful to sources University Not in cards Feb 2024
FEABench Models write Java commands that build and run engineering simulations in COMSOL. The evaluator checks execution, model features, and numerical results where reference values exist. Google Research and Harvard University Scientific code Expert Not in cards
FeedbackQA Models answer questions using a document collection and learn from written feedback about earlier answers. Zichao Li and colleagues Questions that need search School Not in cards Apr 2022
Feishu Office v1 Agents handle 140 Chinese workplace requests in a simulated Feishu workspace. The grader checks the resulting calendars, documents, tables, and approvals. emerge Documents and spreadsheets University Not in cards Jul 2026
FenBench Models identify chess pieces, locate them, count them or describe a board position in Forsyth-Edwards Notation. The test contains 200 examples. puccibets Spatial reasoning and counting Everyday Not in cards Oct 2025
FeruzaSpeech Models transcribe one woman reading Uzbek. The reference text is available in Latin and Cyrillic scripts. Anna Povey and Katherine Povey Speech recognition and translation Everyday Not in cards Mar 2024
fev-bench Models forecast time series across 100 tasks, including tasks with additional variables available to the forecaster. AutoGluon team Real-world events University Not in cards Sep 2025
FEVER Systems check 185,445 claims against Wikipedia and label each Supported, Refuted or NotEnoughInfo. Annotators wrote the claims by altering Wikipedia sentences. The full FEVER score also requires the right evidence sentences. University of Sheffield and Amazon (Thorne, Vlachos, Christodoulopoulos, Mittal) Faithful to sources Everyday Not in cards Mar 2018
FilBench Models perform 12 tasks in Philippine languages. The tasks test cultural knowledge, language processing, reading comprehension and generation. FilBench team and Hugging Face Knowledge in many languages School Not in cards Aug 2025
Fin-Eva Models answer 13,106 Chinese multiple-choice questions about financial knowledge, calculations, analysis, and compliance. The evaluator checks the chosen option against the answer key. Ant Group and Shanghai University of Finance and Economics Finance Expert Not in cards Sep 2023
Financial Statement OCR Models extract financial figures from profit and loss statements. Ramp counts values within one percent of a risk operator’s transcription. Ramp Labs Finance University Not in cards Mar 2026
Financial Statement Review (fs-review) Models review eight sets of financial statements and flag inconsistencies in cash, retained earnings, or the balance sheet. Some sets contain no errors. martian56 Finance University Not in cards Jul 2026
FIND Models find inconsistent evidence spans in long technical and financial documents and describe how those spans conflict. Kensho Technologies, S&P Global Reading comprehension Expert Not in cards May 2026
FinDVer Models verify claims using text and tables in financial reports. The authors score the final true or false judgment across three kinds of reasoning. Yale NLP FinDVer team Finance University Not in cards Sep 2024
FinEvaReranking Models rank Chinese financial news passages for an investor question. Each question has one matching passage and nine or ten distractors. The Hong Kong University of Science and Technology Finance University Not in cards Jul 2024
FinFactReranking Models rank fact checking articles for a claim. Each test row includes one designated article and nine or ten distractors. The Hong Kong University of Science and Technology Finance Everyday Not in cards Jul 2024
FinMTEB Models represent English and Chinese financial texts as vectors. The suite tests seven abilities, including retrieving definitions, grouping complaints, and comparing financial statements. The Hong Kong University of Science and Technology Finance University Not in cards Sep 2024
FinMTEB FINAL Models compare passages from financial reports. The score measures how well their similarity rankings distinguish revised passages from mismatched passages. The Hong Kong University of Science and Technology Finance University Not in cards Jul 2024
FinQA Models answer numerical questions about company earnings reports that mix tables and text. Financial experts wrote 8,281 question and answer pairs, each with a gold reasoning program. Answers are checked by execution accuracy and program accuracy. UC Santa Barbara and J.P. Morgan AI Research Finance University Not in cards Sep 2021
FinSBD 2021 Models locate document structures in English and French financial PDFs. The third edition adds boundaries for tables, figures, headers, and footers. Fortia Financial Solutions Finance School Not in cards Dec 2020
FinSearchComp Greater China Agents search for financial information about Greater China assets across three task types. The score is the published average accuracy for the Greater China subset. ByteDance Seed Finance Expert Not in cards Sep 2025
FinSTS Models compare statements from different annual reports by the same company and score whether their financial meaning changed. People labeled 370 test pairs. The Hong Kong University of Science and Technology Finance University Not in cards Mar 2024
FiQA2018Reranking Models rank eleven candidate answers to personal finance questions. Each test row has one designated answer. The Hong Kong University of Science and Technology Finance Everyday Not in cards Jul 2024
Fireworks Specialized Intelligence Index An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number. Fireworks AI Composite indices Expert Not in cards Sep 2026
FiVE-Bench Models edit objects in videos from written instructions. A vision language model checks whether each edit succeeded, and other metrics measure changes to the background. Harvard University, Hong Kong Polytechnic University, and City University of Hong Kong Video generation University Not in cards Mar 2025
FlashDexRetarget 50-motion benchmark Retarget 50 human hand-object motions (HOT3D, TACO, OakInk2) to XHand and Sharpa hands in Isaac Sim. Scored by object pose-error success thresholds. KAIST AI + Holiday Robotics Simulated robots Everyday Not in cards Oct 2026
Flickr30k Models describe photographs of people, objects, and activities. Meta compares model captions with the human reference captions. Peter Young and colleagues Detection and captioning Everyday Not in cards
FLOW-BENCH Models translate natural-language business requirements into executable enterprise workflows. The evaluator checks constrained workflow syntax and behavior. IBM Research Apps and MCP servers Expert Not in cards Mar 2025
FoNE Models identify named entities in Faroese news articles. The authors manually verified the automatically assigned labels. Vésteinn Snæbjarnarson and colleagues Reading in many languages School Not in cards May 2023
Food-101 Models identify the dish in a food photograph from 101 categories. ETH Zurich General visual questions Everyday Not in cards Sep 2014
FORB Models retrieve other images of the same flat object despite changes in viewpoint or lighting. Pengxiang Wu and colleagues General visual questions School Not in cards Sep 2023
Formal Conjectures Models prove or disprove mathematical conjectures formalized in Lean. The collection includes open research problems and records machine-checked solutions. Google DeepMind Formal proofs Research frontier Not in cards
Formal-IMO Models prove formal statements of International Mathematical Olympiad problems in Lean. The repository supports partial credit when a problem has multiple parts. Google DeepMind Formal proofs Competition Not in cards May 2025
Formally Verified C Models write C functions that satisfy fixed ACSL contracts. Frama-C checks the proofs, and the Core-v1 dataset reserves 16 of 64 tasks for testing. stanley-ngugi Function-level problems Expert Not in cards Sep 2026
FormExtract Bench Models extract named fields from form text into JSON. The grader compares each field with its reference and checks that the object matches the schema. Aishwarya Jagadish Documents, charts and text in images School Not in cards Oct 2026
Fox Document Benchmark Models read English and Chinese documents and perform tasks including OCR, translation, and question answering. The hub port covers nine subtasks. Fox authors Documents, charts and text in images University Not in cards May 2024
FPL 2026 Agentic FPGA Backend Optimization Competition Teams improve the maximum clock frequency of routed FPGA designs while preserving behavior. The organizers rank teams across seven designs after accounting for runtime and model cost. FPL 2026 competition organizers Electronics Expert Not in cards Feb 2026
FQuAD 2.0 Models answer questions about French passages and identify questions that the passage cannot answer. Quentin Heinrich and colleagues Reading in many languages School Not in cards Sep 2021
FreiHAND Models recover a hand's joint positions and surface from a color image. The evaluation compares predictions with reference hand geometry. University of Freiburg Spatial reasoning and counting School Not in cards Sep 2019
French Insurance Job Offer Models mark and classify soft skills in French job ads from Canadian insurers. A domain expert annotated 47 ads. Université Laval and four Canadian insurers Other professions School Not in cards Apr 2022
FrenchBench (CroissantLLM) Models answer questions and perform language understanding and generation tasks in French. The suite combines existing tests with manually annotated material. Manuel Faysse and colleagues Reading in many languages School Not in cards Feb 2024
FrontierMath: Open Problems Models attack unsolved research math problems proposed by working mathematicians. Each problem has a program that checks a proposed solution. Epoch tracks which problems AI has solved. Epoch AI Research mathematics Research frontier Not in cards Jan 2026
Fruit Box Models select rectangles whose entries sum to ten on a ten-by-seventeen grid. The evaluator compares the final score with an expert trajectory from the training split. djdumpling Puzzles and chess School Not in cards Oct 2025
FSD50K Models identify the sound events present in a Freesound recording. Eduardo Fonseca and colleagues Audio understanding and voice Everyday Not in cards Oct 2020
FSDKaggle2018 Models assign a sound event label to a Freesound recording. Eduardo Fonseca and colleagues Audio understanding and voice Everyday Not in cards Mar 2018
FSDKaggle2019 Models identify all relevant sound events in an audio clip using training labels of varying reliability. Eduardo Fonseca and colleagues Audio understanding and voice School Not in cards Jun 2019
Fundus-MMBench Models identify retinal structures, classify diseases, and grade their severity in fundus photographs. Each of the 31 tasks contains 20 questions. Xinyao Liu and Diping Song Medicine Expert Not in cards Jul 2025
FunGraph3D Models identify objects and their controls in indoor scans. They predict which part operates which object, such as a switch that controls a light. OpenFunGraph team Spatial reasoning and counting Everyday Not in cards Mar 2025
FutureEval (Metaculus) Models forecast open Metaculus questions with a fixed prompt and are scored as the questions resolve. Seasonal bot tournaments and Pro Forecaster baselines run alongside. Metaculus Real-world events Expert Not in cards Feb 2026
G2 Voice AI Agent Platform Evaluations Voice platforms run the same agent on simulated business calls. Bluejay checks required tool arguments and the database after hangup, then measures transcription and latency. G2 and Bluejay Customer service University Not in cards Oct 2026
Galahad Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior. phi-monster (Galahad authors) Simulated robots Everyday Not in cards Jul 2026
GalCoLA Models judge the grammatical acceptability of Galician sentences. Proxecto Nós Reading in many languages School Not in cards Jun 2024
GameForgeBench Agents repair game projects in engines such as Godot. The verifier runs the project and checks behavior such as camera movement and collisions. GamePhanesStudio Fixing real repositories University Not in cards Sep 2026
GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. Bosch + NVIDIA + UC Berkeley Simulated robots Everyday Not in cards Jul 2026
Gauntlet Agents develop game controllers through practice. The authors freeze each controller and test it on unseen games without further model calls. Haonan Huang and Joey Xiao Video games University Not in cards Jul 2026
GAUSS Models solve mathematics problems covering 12 skills, including proof checking and mathematical communication. Contributors provide a reference solution and a separate scoring rubric for each problem. GAUSS contributors Research mathematics Expert Not in cards Sep 2025
gBench Models run academic evaluations, Golden Set smoke tests, and an agentic quality suite through an OpenAI-compatible endpoint. The tool also records serving performance. Google DeepMind Machine learning engineering University Not in cards Jan 2025
GD-VCR Models answer questions about what people are doing and why. The test pairs 328 images, mainly from film and television, with 886 questions across four regions. University of California, Los Angeles General visual questions Everyday Not in cards Sep 2021
GDPWorks Agents complete 83 professional tasks across 36 occupations, using source files to produce work such as documents, spreadsheets, and code. The Joules-to-GDP framework compares outputs with expert deliverables through a blinded model judge. Task success measures the share of wins and ties against the expert. Snorkel AI and Hazy Research (Scaling Intelligence) Tasks from real jobs Expert Not in cards Nov 2026
Gecko People compare text to image outputs for prompt alignment. The benchmark studies model ordering, pairwise instance scoring, and pointwise instance scoring. Google DeepMind Image generation and editing Everyday Not in cards Apr 2024
Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. Google DeepMind Deception and alignment Not in cards
Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. Google DeepMind Refusing harmful requests Not in cards
Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. Google DeepMind Jailbreaks and prompt injection Not in cards
Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. Google DeepMind Real robots Everyday Not in cards Mar 2025
Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. Google DeepMind Real robots Everyday Not in cards Jul 2026
Gender Resolution in Speaker-Listener Dialogue Roles Models translate English dialogue into Czech, Icelandic or Spanish. The authors score adjective agreement when context identifies gender and measure gender choices when it leaves gender open. National Research Council Canada Translation Everyday Not in cards Nov 2024
GENDER1PERSON Models translate 1,000 English product reviews into Russian or Serbian. The test measures the difference between first-person masculine and feminine choices as a percentage of all reviews. IU University and University of Hildesheim Translation Everyday Not in cards Jun 2025
General Agent Benchmark (Moonshot) Agents complete multi-step production workflows. Experts compare their work with human expert performance. Moonshot AI Tasks from real jobs Expert Not in cards Jan 2026
Generating Confusion: Voting with a Disability Models answer questions about accessible voting in the United States. Reviewers measure how often answers to standard voter questions contain incorrect information. Center for Democracy & Technology Long-form factuality Everyday Not in cards Sep 2024
GenEval 2 Models generate images from compositional prompts. A visual question answering model checks whether each requested object, attribute, count, and relation appears. Meta FAIR, University of Washington, UCLA, and Allen Institute for AI Image generation and editing Everyday Not in cards Dec 2025
GenExam Image generation systems are evaluated on subject-focused educational tasks with expert ratings. Google DeepMind Image generation and editing University Not in cards
GenomeGym-BRCA1 Models estimate whether a documented BRCA1 variant causes loss of function. The grader compares each probability with a hidden functional label. alfaxad Genomics and life sciences Expert Not in cards May 2026
GenScale Models generate or edit images containing objects with known relative sizes. A judge calibrated against human ratings scores the relative scale of object pairs. Boston University and collaborators Image generation and editing Everyday Not in cards Sep 2026
GEO-Bench-2 Models interpret Earth imagery across 19 tasks, including field boundaries, forest biomass and tree species. The authors normalize task scores to compare model capabilities. AI Alliance GEO-Bench-2 team Spatial reasoning and counting University Not in cards Nov 2025
GeoDE Models recognize objects in images collected across different world regions. The evaluation compares accuracy across regions and object classes. GeoDE team General visual questions Everyday Not in cards Jan 2023
GeoExpertQueries Agents choose geospatial data and carry out an analysis. A model compares each execution trace with a reference workflow agreed on by three GIS specialists. Google Research Apps and MCP servers Expert Not in cards Oct 2026
GeoKTools Agents combine up to four kinds of geospatial tools to answer quantitative questions. Exact matches earn full credit, and a model grades other answers. Google Research Apps and MCP servers University Not in cards Oct 2026
GeoMLAMA Models complete sentences about daily life in five countries. The test checks whether models rank locally correct answers above other choices in five languages. University of California, Los Angeles Knowledge in many languages Everyday Not in cards May 2022
GeoSound Models match environmental recordings to satellite images of their locations and retrieve recordings from images. The test holds out geographic areas and contains 9,931 examples. Subash Khanal and colleagues Audio understanding and voice Everyday Not in cards Aug 2024
GerDaLIR Models find the German court decisions cited by a passage after the authors mask its references. The original test split has 12,298 queries. Marco Wrzalik and Dirk Krechel Law University Not in cards Oct 2021
German Argument Mining Models label sentences from Bavarian court judgments as conclusions, definitions, applications of law to facts, or other text. The authors report F1 and accuracy. Technical University of Munich and University of Passau Law University Not in cards Jul 2020
German federal election chatbot audit Chatbots answer German election questions in English and German. Researchers count correct electoral answers out of 30 per chatbot and rate political bias separately. Democracy Reporting International Long-form factuality Everyday Not in cards Feb 2025
German Traffic Sign Recognition Benchmark Models identify the class of a traffic sign in a cropped road image. Johannes Stallkamp and colleagues Detection and captioning Everyday Not in cards Dec 2010
GermEval 2014 Named Entity Recognition Models find and classify names in German news and Wikipedia text, including nested and derived entities. Darina Benikova and colleagues Reading in many languages School Not in cards May 2014
GIE-Bench Image-editing models follow text instructions while preserving unedited regions. GPT-4o question answers and masked image metrics score functional correctness and preservation. Apple General visual questions University Not in cards May 2025
GIFT-Eval Models forecast future values from historical time series across multiple domains and forecast horizons. Salesforce AI Research Real-world events University Not in cards Oct 2024
GigaAM Multilingual Internal In-the-Wild Speech Tests Models transcribe crowdsourced recordings in Russian, Kazakh, Kyrgyz and Uzbek. The evaluators report word error rate by language. Sber Speech recognition and translation Everyday Not in cards Jul 2026
GigaSpeech Models transcribe English speech from audiobooks, podcasts, and online videos. The test compares transcripts using word error rate. GigaSpeech team Speech recognition and translation Everyday Not in cards Jun 2021
GLES 2017 vote choice Models use a German survey respondent’s profile to estimate their reported vote in the 2017 federal election. Researchers compare individual choices and party vote shares. Leah von der Heyde, Anna-Carolina Haensch, and Alexander Wenz Social simulation Everyday Not in cards Dec 2023
GlobalMGSM Models solve MGSM grade-school math problems pooled from MGSM-Rev2, AfriMGSM, BenchMAX and other translations, 41 language configs in all. Accuracy on the final number. Cohere Labs Reasoning in many languages School Not in cards Mar 2026
GLUE Models predict labels or similarity scores across nine English language-understanding tasks. The overall score averages task scores. Tasks with several metrics first average those metrics. New York University, University of Washington, and DeepMind Language reasoning Everyday Not in cards Apr 2018
Gold Commodity News Models classify news headlines about gold. Labels distinguish price direction, timing, and comparisons with other assets. Ankur Sinha and colleagues Finance School Not in cards Sep 2020
Goliath human mesh recovery Models reconstruct a person's body from images of studio performances. Meta measures joint and surface errors against captured three-dimensional motion. Meta Spatial reasoning and counting School Not in cards
Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. Google DeepMind Dangerous capabilities Not in cards
Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. Google DeepMind (with Wiz) Finding and exploiting real bugs Expert Not in cards
Google Landmarks Dataset v2 Models recognize landmarks or retrieve photographs of the same landmark. The benchmark uses images from around the world. Google General visual questions Everyday Not in cards Apr 2020
Google Scanned Objects Models synthesize an object from a new viewpoint. Researchers compare the rendered result with reference views of scanned household items. Google Research Spatial reasoning and counting School Not in cards Apr 2022
Google Universal Image Embedding Challenge Models retrieve images of the same item across several visual domains using a single image representation. The challenge compares results against labeled matches. Google General visual questions Everyday Not in cards Aug 2022
Gorgias AI Agent Competitive Benchmark Agents recommend products and answer support questions on live Shopify stores. Gorgias combines answer quality, response time, and the share of conversations handled without human help. Gorgias Running a business Everyday Not in cards Jul 2026
GovReport Models summarize long US government reports from the Government Accountability Office and the Congressional Research Service. The reports are much longer than earlier summarization sets, and each has an expert-written summary. Scores are ROUGE overlap with the reference summary. University of Michigan and UIUC Reasoning over long documents University Not in cards Apr 2021
GPT-3 Arithmetic Models calculate answers to ten types of arithmetic problems. The tasks vary the operation and the number of digits. OpenAI Word problems School Not in cards May 2020
GPT-3 Word Scrambling Models recover English words after their letters are rearranged or extra characters are inserted. The test measures exact recovery across several transformations. OpenAI Language reasoning School Not in cards May 2020
Grade Like a Human OS Dataset Models grade undergraduate operating systems answers. The authors measure absolute error against the mean grade from three teaching assistants per answer. Wenjing Xie and colleagues at City University of Hong Kong and Mohamed bin Zayed University of Artificial Intelligence Other professions University Not in cards May 2024
GRAFT Models predict plant traits from gene expression in Arabidopsis. The authors compare regression and graph models against measurements from the same plants. Manuel Serna-Aguilera and colleagues Genomics and life sciences University Not in cards Jun 2026
GrailQA Models answer questions by composing queries over Freebase. The evaluator measures semantic logical-form exact match and answer-set F1 across ordinary, compositional, and unseen-schema generalization settings. Ohio State University, U.S. Army Research Laboratory, Stanford University and others Language reasoning Everyday Not in cards Nov 2020
Grammarly Corpus of Discourse Coherence Models rate the coherence of emails, reviews, and answers to online questions. Each of four collections has 200 test texts with expert ratings. Grammarly and the University of Illinois at Urbana-Champaign Reading comprehension School Not in cards Jul 2018
GraspBench Robots propose grasps and try to lift objects in MuJoCo. Each attempt must pass motion planning and execution on the chosen robot. Abdul Basit and colleagues Simulated robots Everyday Not in cards Nov 2026
Gray Swan Shade (adaptive attacker) Gray Swan's adaptive red-teaming agent trained on test scenarios to attack coding, computer-use and browser agents with prompt injections. Gray Swan Jailbreaks and prompt injection Not in cards
Green Job Description Entity Recognition Models mark spans for skills, qualifications, experience, occupations, and domains in UK job ads. Human annotations supply the answer key. University of Sheffield Other professions School Not in cards Jun 2022
Greptile AI Code Review Benchmark AI code review tools inspect 50 pull requests from five open-source repositories and report bug detection, precision, review quality and language coverage. Greptile Fixing real repositories Expert Not in cards Jan 2025
Gretel Synthetic PII Finance Models identify and label personal information in synthetic financial documents across seven languages. Independent evaluations measure how well the predicted spans match the labeled spans. Gretel Finance Everyday Not in cards Jun 2024
Grooming the Machine Chatbots respond to three false claims under four user personas. The researcher counts a repeated false claim or a refusal as a failure. Arianna Stech, European University Institute Long-form factuality Everyday Not in cards Apr 2026
GroupAssistBench Assistants use memories from group conversations to answer questions and complete real-world group assistance tasks. Model judges score grounded helpfulness. Tencent Learning from the context Expert Not in cards Nov 2026
GS-QA2 Models answer questions about terrain and map features using elevation grids and vector data. The grader checks answers against PostGIS queries with tolerances for each measurement. University of California, Riverside Apps and MCP servers University Not in cards May 2026
GTFS Retrieval Models query a Chicago bus schedule from four routes. They answer directly or generate Python code whose output is compared with a reference answer. University of Illinois Urbana-Champaign Expert-written questions University Not in cards Jul 2023
GTFS Semantics Models answer multiple-choice questions about the General Transit Feed Specification. Questions cover transit data files, field definitions, and relationships between tables. University of Illinois Urbana-Champaign Expert-written questions University Not in cards Jul 2023
GTZAN Models identify the music genre in short recordings. Qwen reports genre classification accuracy across music understanding systems. George Tzanetakis and Perry Cook Audio understanding and voice Everyday Not in cards Jul 2002
GutBrainIE 2026 Models find entities in 80 gut and brain research abstracts, link them to biomedical concepts, and identify relationships between mentions and between concepts. University of Padua and collaborators Medicine University Not in cards Feb 2026
HAGRID Models answer information-seeking questions and attribute their claims to supporting passages. Ehsan Kamalloo and colleagues Faithful to sources University Not in cards Jul 2023
Hallmarks of Cancer Models label biomedical text with the cancer hallmarks it describes. The hub port assigns zero or more of ten labels to individual sentences. Simon Baker and colleagues Genomics and life sciences University Not in cards Oct 2015
HalloMTBench Models translate English text into one of 11 languages. Judges count translations that add unsupported material, use the wrong language, leave text untranslated or repeat material. Alibaba International Digital Commerce and Tianjin University Translation University Not in cards Oct 2025
HalluLens Language models answer dynamically generated prompts that test factual and contextual hallucination. The evaluator builds fresh examples to reduce test leakage. Meta FAIR and Superintelligence Labs Hallucination rates Everyday Not in cards Apr 2025
HalluScoring 2026 Models detect factual errors in Arabic answers. Separate tasks test unseen questions, unseen answer generators and the ability to select a correct alternative. HalluScoring organizers Hallucination rates University Not in cards May 2026
Hallym University Women's Health Nursing Exam Models answer 50 Korean questions from a fourth-year nursing course. The authors compare accuracy and estimated ability with results from 111 students. Hallym University Medicine University Not in cards Sep 2025
HaluEvalAudio 1000 Audio language models answer spoken questions against supplied knowledge and reference answers. Kyutai Hallucination rates University Not in cards Mar 2026
HaluMem Agent memory systems are tested for hallucinations at each operation, memory extraction, updating and question answering, rather than only end to end. MemTensor Hallucination rates Everyday Not in cards Nov 2025
Hands Visible Models inspect 12 frames per clip and determine whether a human hand or robot gripper appears. The 45-clip evaluation uses provisional machine-generated labels. referee-lab Video understanding Everyday Not in cards Aug 2026
Hangman Bench Agents guess letters to identify hidden words in Hangman. The harness tracks wins across difficulty levels. MattFisher Puzzles and chess Everyday Not in cards Jun 2025
HANS An entailment test built to catch three shallow shortcuts: lexical overlap, subsequence and constituent. Every premise and hypothesis pair shares words, but only some pairs are entailments. A model that relies on word overlap fails the non-entailment cases. Johns Hopkins University and Brown University (McCoy, Pavlick, Linzen) Language reasoning Everyday Not in cards Feb 2019
HappyWorld-Bench World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them. Alibaba Interactive world models Everyday Not in cards Sep 2026
HARMOGEN-R Models grade programming students’ responses using human and model-written rubrics. The dataset includes 770 responses from a data structures and algorithms course. University of Madeira Other professions University Not in cards Oct 2025
Harmony4D Models recover the body shapes and poses of people interacting at close range. The benchmark includes sports where people occlude each other. Rawal Khirodkar and colleagues Spatial reasoning and counting School Not in cards Oct 2024
Harness Leaderboard Coding-agent harnesses such as Claude Code, Codex and OpenCode run the same open model at the same reasoning effort on 198 tasks from Terminal-Bench 2.1 and DeepSWE v1.1. Scored as the resolve rate averaged across the two benchmarks over three attempts, with any access to the answer counted as a fail. Joel Niklaus (Hugging Face) Terminal tasks Expert Not in cards Nov 2026
harness-bench-fast Agents operate on files and code using tools and memory. Harbor exports 231 tasks from version 0.3.0. Later public leaderboards report Light and Medium suites separately. AI Forever Terminal tasks University Not in cards May 2026
Hateful Memes Challenge Models decide whether a meme is hateful by interpreting its image and caption together. Paired examples change one modality while preserving the other. Douwe Kiela and colleagues Bias, fairness and politics Everyday Not in cards May 2020
HC3Reranking Models rank answers to personal finance questions. Each test question has one relevant human answer and one relevant ChatGPT answer. The Hong Kong University of Science and Technology Finance Everyday Not in cards Jul 2024
HCAST Agents attempt 189 machine learning, cybersecurity, software engineering and reasoning tasks. 563 human baselines time each task, from one minute to over 8 hours, so success can be read against human task length. METR Task length Expert Not in cards Mar 2025
HDLBits Models write Verilog modules for digital logic exercises. The online judge compiles each submission and compares its simulated outputs with a reference design. Henry Wong Electronics University Not in cards
HEAD-QA Models answer questions from Spanish healthcare specialization exams in six disciplines. The hub port uses the English training and validation splits. David Vilares and Carlos Gómez-Rodríguez Medicine Expert Not in cards Jan 2019
HealthAdvice Models classify a sentence from a medical research paper as strong advice, weak advice, or no advice. Syracuse University and Jun Wang Medicine School Not in cards Sep 2021
Hebbia Financial Services Benchmark (2025) Models extract facts, summarize documents and reason about financial information. Hebbia evaluates responses to more than 600 questions. Hebbia Finance Expert Not in cards Sep 2025
Hebrew Diacritization Test Corpora Models restore vowel and consonant marks in Hebrew prose, rabbinic commentary and liturgical poetry. The reference also identifies spelling letters that vocalized text should omit. Dicta and Bar-Ilan University Reading in many languages University Not in cards Jan 2020
Hedge Gym Models adjust an options hedge each day over a 21-day price path. The evaluator measures replication error and transaction costs on synthetic or historical markets. Sablier Finance Expert Not in cards Oct 2026
HEIM Text-to-image models are scored on 12 aspects, such as alignment, quality, aesthetics, originality, reasoning, knowledge, bias and toxicity, with human raters and automatic metrics. Stanford CRFM Image generation and editing Everyday Not in cards Nov 2023
HexMachina Models write Catan players in code and compete in the board game. HexMachina authors Multi-agent games University Not in cards Jun 2025
Hi4D Models reconstruct the bodies of people interacting at close range. The SAM 3D Body evaluation compares joint and mesh errors with motion capture references. Hi4D team Spatial reasoning and counting School Not in cards Mar 2023
Hidden Voices Audio Difficulty Models rank piano works by Black women composers from recordings. The reference ordering comes from the Hidden Voices project’s seven difficulty grades. Universitat Pompeu Fabra, Sogang University, and University of Alicante Audio understanding and voice University Not in cards Mar 2024
HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. Zhongyu Xia and colleagues Simulated robots Everyday Not in cards May 2026
HierText Models locate and recognize text arranged in lines and paragraphs in real scenes. Google also uses the annotations to compare text and image retrieval. Google Research Documents, charts and text in images School Not in cards Mar 2022
HIPE-OCRepair 2026 Models correct errors in machine transcriptions of historical newspapers and printed works in English, French, and German. They see the transcriptions without the page images. HIPE-OCRepair organizers Reading in many languages School Not in cards Nov 2025
Historical Systems Bugs Agents repair two historical C bugs in Linux and Unikraft. Hidden tests exercise the patched code, including driver behavior under emulated hardware. morris-biotech Fixing real repositories Expert Not in cards Nov 2026
HLE Bio/Chem Gold Models answer biology and chemistry questions from Humanity’s Last Exam whose answers reviewers supported with published research. FutureHouse Science questions Expert Not in cards Jul 2025
HM3D-MFMON Agents navigate across floors and find a sequence of objects. BAAI FlagEval Simulated robots Everyday Not in cards Aug 2026
HM3DSem-Walks Models use indoor color and depth scans to find objects described by room and floor. The original retrieval test covers eight scenes. University of Freiburg and University of Technology Nuremberg Spatial reasoning and counting Everyday Not in cards Mar 2024
HMDB51 Models identify human actions in short video clips across 51 categories. Hilde Kuehne and colleagues Video understanding Everyday Not in cards Nov 2011
HoloAssist Mistake Detection Models watch a person perform a task and classify each annotated action as correct or mistaken. The challenge reports an F-score. Xin Wang and colleagues Video understanding Everyday Not in cards Sep 2023
Homayouni and Fontes Flexible Job Shop with Transportation Agents schedule jobs on eligible machines and route two vehicles between operations. The score measures when the last operation finishes. S. M. Homayouni and D. B. M. M. Fontes Tasks from real jobs University Not in cards
Honor of Kings Game Configuration Agents configure game data with the Honor of Kings spreadsheet platform. The team measures configuration accuracy before and after the agent checks and repairs its work. Tencent Tasks from real jobs University Not in cards Nov 2026
Hotter and Colder Models predict sentiment and other attributes of Icelandic blog text using crowdsourced judgments. Steinunn Rut Friðriksdóttir and colleagues Reading in many languages Everyday Not in cards Feb 2025
HourVideo Multimodal models answer questions about first-person videos that run 20 to 120 minutes. The 500 videos come from Ego4D, and the 12,976 five-way multiple-choice questions cover summarization, perception, visual reasoning and navigation. Score is accuracy. Stanford University Long videos Everyday Not in cards Nov 2024
How frontier models miss risk in routine enterprise work Agents handle workplace requests whose risks appear in the prompt or in supporting files. Judges check whether the agent completes an unsafe action. Mercor Refusing harmful requests Expert Not in cards Nov 2026
How2QA Models answer multiple-choice questions about instructional videos. The task tests understanding of the actions shown in each clip. Microsoft Research Video understanding Everyday Not in cards May 2020
Huayu Awards AI Cockpit Evaluation 2026 In-car assistants interpret requests and operate vehicle features. Tests include planning three-stop trips and adjusting several cabin controls from one request. J.D. Power China and Tongji University HVR Lab and XAI Lab General assistants Everyday Not in cards Sep 2026
HubBench Agents make workplace decisions across 13 professional domains using connected software tools and evidence. Version 1.4.0 grades 104 tasks with deterministic state checks. Blobfish AI Tasks from real jobs Expert Not in cards Sep 2026
Hugging Face Speech Bench Speech recognition models transcribe recordings. The board collects error rates from model cards. Hugging Face Speech recognition and translation Everyday Not in cards Mar 2022
Human ChatGPT Comparison Corpus Models predict whether a person or ChatGPT wrote an answer in English or Chinese. The evaluation includes complete answers and individual sentences. Hello-SimpleAI and the HC3 authors Reading in many languages Everyday Not in cards Jan 2023
HumanoidBench Policies control a simulated Unitree H1 humanoid with dexterous hands on 15 whole-body manipulation and 12 locomotion tasks. Scored by task return or success. UC Berkeley + Yonsei University (Sferrazza et al.) Simulated robots Everyday Not in cards Mar 2024
Hy-Nomas-Bench Interactive video world models follow text and camera controls across multiple turns and retain spatial memory. Tencent Hunyuan Interactive world models Expert Not in cards Oct 2026
HY3D-Bench Models generate textured 3D meshes from text and images, with semantic and geometric quality evaluated across categories. Tencent Hunyuan Image generation and editing Expert Not in cards Feb 2026
HydroLLM-Benchmark Models answer hydrology questions drawn from a textbook and research papers. The authors score objective answers by accuracy and other answers by semantic similarity. University of Iowa Hydroinformatics Lab Science questions University Not in cards Oct 2024
i2b2 2006 Smoking Challenge Models read a discharge summary and classify the patient's smoking status. The labels include current smoker, past smoker, unspecified smoker, nonsmoker, and unknown. Informatics for Integrating Biology and the Bedside Medicine School Not in cards Jul 2006
i2b2 2008 Obesity Challenge Models classify obesity and 15 related conditions from discharge summaries. The challenge scores what the note states explicitly and what a clinician can infer. Informatics for Integrating Biology and the Bedside Medicine Expert Not in cards Mar 2008
i2b2 2009 Medication Extraction Challenge Models extract drug names and prescription details from discharge summaries. The organizers scored six fields against community annotations for 251 test summaries. Informatics for Integrating Biology and the Bedside Medicine University Not in cards Jun 2009
i2b2/UTHealth 2014 Identifying Risk Factors for Heart Disease over Time Models find heart disease risk factors in notes about patients with diabetes. They mark when each factor was present relative to the note date. i2b2 and the University of Texas Health Science Center at Houston Medicine University Not in cards Mar 2014
IARPA Babel speech recognition Models transcribe conversational speech in languages with limited labeled data. Meta measures character error against reference transcripts on the Babel development split. IARPA Speech recognition and translation School Not in cards
ICASSP 2026 Automatic Song Aesthetics Evaluation Challenge Models listen to generated songs and predict musicians’ ratings. One track scores overall musicality, and the other scores five aspects of each performance. ICASSP ASAE Challenge organizers Audio understanding and voice Everyday Not in cards Aug 2025
ICDAR 2025 MapText Competition Models detect and read words on historical maps and link them into phrases. This edition adds Taiwanese maps with Chinese labels and revises how links score. ICDAR MapText organizers Documents, charts and text in images School Not in cards Dec 2024
ICNALE Edited Essays Models score original English learner essays against an editor’s ratings. The collection pairs each original with an edited version and records scores for five writing traits. Shin’ichiro Ishikawa and the ICNALE project at Kobe University Other professions University Not in cards Aug 2016
ICNALE Global Rating Archives Models grade essays and spoken responses in English. The archive provides ratings from 80 people per sample across 140 essays and 140 speech samples. Shin’ichiro Ishikawa and the ICNALE project at Kobe University Other professions University Not in cards Oct 2023
IdeaTaste Agents propose research ideas from literature frozen at 2025. Evaluation compares their citations and ideas with 2026 publications and measures diversity. Carnegie Mellon University AI research tasks Research frontier Not in cards Nov 2026
IEMOCAP Models identify emotions in recorded dialogues. NVIDIA evaluates this task within AIR-Bench. University of Southern California Audio understanding and voice Everyday Not in cards Nov 2008
IIT Madras Hindi ASR Challenge Speech models transcribe Hindi newspaper passages read aloud. The organizers score word errors and rank open and restricted training separately. Speech Lab, IIT Madras Speech recognition and translation Everyday Not in cards Jun 2020
IIT Madras Hindi-Tamil-English ASR Challenge Speech models transcribe Hindi, Tamil and Indian English recordings. The organizers report word error rates separately for each language and training condition. Speech Lab, IIT Madras Speech recognition and translation Everyday Not in cards May 2021
Image Matching Challenge Models match photographs of landmarks taken from different viewpoints. The evaluation measures the accuracy of camera poses recovered from the matches. Image Matching Challenge organizers Spatial reasoning and counting School Not in cards
ImageCoDe Models select the image that matches a description from a set of visually similar alternatives. Benno Krojer and colleagues General visual questions Everyday Not in cards Mar 2022
ImageNet ReaL Models classify ImageNet validation images against revised human annotations. An image can have more than one acceptable label. Google Research General visual questions Everyday Not in cards Jun 2020
ImageNet-1K Models assign one of a thousand object labels to each photograph. Meta reports classification accuracy on the validation images. ImageNet project General visual questions Everyday Not in cards
ImageNet-A Models classify real photographs that commonly fool image classifiers. The test uses unmodified images from ImageNet categories. Dan Hendrycks and colleagues General visual questions Everyday Not in cards Jul 2019
ImageNet-C Models classify images under common corruptions at different severities. The benchmark measures errors under blur, noise, weather, and digital distortions. Dan Hendrycks and Thomas Dietterich General visual questions Everyday Not in cards Mar 2019
ImageNet-R Models classify renditions of objects in drawings, sculptures, and other media. The benchmark measures recognition when appearance differs from ordinary photographs. Dan Hendrycks and colleagues General visual questions Everyday Not in cards Jun 2020
ImageNet-Sketch Models classify black-and-white sketches using ImageNet category labels. The benchmark tests recognition when images lack photographic texture and color. Haohan Wang and colleagues General visual questions Everyday Not in cards May 2019
ImageNet-UA Models classify images under distortions that differ from those used during training. The updated suite measures accuracy across 18 new distortions. Center for AI Safety General visual questions Everyday Not in cards Jul 2023
ImageNetV2 Models classify a newly collected set of photographs using the original ImageNet labels. The test measures how accuracy changes on fresh images. Benjamin Recht and colleagues General visual questions Everyday Not in cards Feb 2019
IMDb-PT Models classify the sentiment of Brazilian Portuguese translations of IMDb movie reviews. Maritaca AI Reading in many languages Everyday Not in cards
Inappropriate Image Prompts Models generate images from prompts drawn from real users. The evaluation measures how often the outputs contain inappropriate material. Patrick Schramowski and colleagues Refusing harmful requests Everyday Not in cards Nov 2022
INAS hypothesis classification Models read the title and abstract of an invasion biology paper and identify which of ten ecological hypotheses it addresses. A paper can address several hypotheses. Bielefeld University and collaborators Lab and data work University Not in cards Nov 2022
INAS hypothesis span prediction Models mark the words in a scientific title or abstract that indicate a particular ecological hypothesis. Experts marked the evidence in 750 invasion biology abstracts. Bielefeld University and collaborators Lab and data work University Not in cards Jul 2024
iNaturalist 2021 Models classify nature photographs by species. The 2021 competition uses a new collection of images and species labels. iNaturalist Competition organizers General visual questions University Not in cards Mar 2021
iNaturalist Sounds Models identify animal species from recordings submitted to iNaturalist. Mustafa Chasmai and colleagues Audio understanding and voice Expert Not in cards May 2025
INCLUDE Indian Sign Language Recognition Models identify isolated Indian Sign Language words in video. The authors provide separate test files for the full vocabulary and its smaller selection. AI4Bharat and IIT Madras Video understanding Everyday Not in cards Oct 2020
IncompeBench Models retrieve music clips that match detailed natural language requests. Benjamin Clavié and colleagues Audio understanding and voice School Not in cards Feb 2026
IndicQA Models extract answers from Wikipedia passages in Indian languages. Native speakers wrote the questions and answers. AI4Bharat Reading in many languages School Not in cards Jul 2023
IndicXNLI Models compare a premise and hypothesis in Indian languages. They predict whether the premise entails, contradicts or leaves the hypothesis unresolved. Divyanshu Aggarwal and colleagues Reasoning in many languages School Not in cards Apr 2022
IndoNLI Models decide whether a hypothesis follows from an Indonesian passage using annotations from experts and other speakers. Rahmad Mahendra and colleagues Reading in many languages School Not in cards Nov 2021
InfraBench Agents deploy, profile, debug, and optimize model workloads on AMD MI35X GPUs. Task-specific verifiers grade 20 engineering assignments. InfraBench team Machine learning engineering Expert Not in cards Jul 2026
InnovationEval Agents with thousands of GPU hours try to discover an ML technique that matches a recent human paper they have not seen. The score is the share of that paper's gain the agent reaches. Epoch AI AI research tasks Research frontier Not in cards Oct 2026
INQUISITIVE Models read a news article up to a highlighted phrase and ask a question a reader might have. Human raters judge validity, relevance, and salience. University of Texas at Austin Reading comprehension Everyday Not in cards Sep 2020
Insight Consolidation v1 Agents read customer account documents, infer why the customer acted, and cite supporting records. The evaluator combines a model judgment with citation accuracy. pavel-substrate Reasoning over long documents University Not in cards Aug 2026
Inspect India Evals Models answer questions about Indian culture and reason across Indian languages. Other tasks test social bias, digital public infrastructure safeguards and refusal behavior. Abhishek Kumar Singh and colleagues Knowledge in many languages University Not in cards Jul 2026
Instinct Next Edit Models predict the next edit from code context and recent changes. The port uses TypeScript, which has 221 held-out examples. Continue Function-level problems University Not in cards Sep 2025
InstructIR Models find documents that satisfy a user's search request and its accompanying instructions. Hanseok Oh and colleagues Questions that need search School Not in cards Feb 2024
InsureBench Decision Models (Broader Comparison) Models read text evidence and return structured findings about insurance scenarios. The scorer compares 150 findings from 50 packets with the author's labels. InsureThing Finance University Not in cards Oct 2026
Internal Finance Modeling (Moonshot) Agents complete financial modeling tasks with linked formulas, financial logic and final workbook quality. Experts rate the work. Moonshot AI Finance Expert Not in cards Oct 2026
Internal Financial Research (Moonshot) Agents produce auditable financial research under evidence and cutoff constraints. Experts rate the resulting research. Moonshot AI Finance Expert Not in cards Oct 2026
IntervalQA Models estimate numerical answers and give intervals for their uncertainty. The authors measure how often the intervals contain the true answers. Andy Zou and collaborators Language reasoning University Not in cards Jun 2022
Invoice Approval (Ali Siala) Agents process 60 supplier invoice work orders under a German company’s approval policy. The evaluator compares the resulting database with a reference state. Ali Siala Documents and spreadsheets University Not in cards Nov 2026
IOL-AI 2026 Models infer linguistic rules from Olympiad problems and answer individual questions. The official competition combines exact match and character-level similarity. IOL-AI organizers and Cohere Labs Reasoning in many languages Competition Not in cards Jun 2026
IronClad Agents repair security misconfigurations in Terraform repositories. The grader verifies that the repaired configuration satisfies policy and preserves the intended infrastructure. vinay-k-rajith Secure code and review University Not in cards Aug 2026
iSAID Models detect and segment individual objects in aerial images. The benchmark measures how well predicted masks match annotated object instances. Syed Waqas Zamir and colleagues Detection and captioning University Not in cards May 2019
iSarcasmEval Models identify intended sarcasm in English and Arabic texts labeled by their authors. Other tracks classify ironic speech and distinguish sarcastic text from a literal paraphrase. Ibrahim Abu Farha, Silviu Vlad Oprea, Steven Wilson, and Walid Magdy Language reasoning Everyday Not in cards Jul 2022
IsiZulu Second Language Learner Speech Corpus Models detect sounds that learners mispronounce or insert when reading isiZulu. Teachers marked each sound against the intended pronunciation. Indiana University and University of KwaZulu-Natal Audio understanding and voice University Not in cards May 2024
IslamicEval 2025 Quotation Correction Models correct inaccurate Quran and Hadith quotations against the supplied sources. The organizers score exact corrections and accept a designated answer for uncorrectable text. Qatar Computing Research Institute and collaborators Faithful to sources School Not in cards Jun 2025
IslamicEval 2025 Quotation Span Identification Models mark the character spans of intended Quran and Hadith quotations in Arabic model answers. Organizers score character labels with macro F1. Qatar Computing Research Institute and collaborators Faithful to sources University Not in cards Jun 2025
IslamicEval 2025 Quotation Validation Models check whether marked Quran and Hadith quotations agree with their sources. The organizers report accuracy. Qatar Computing Research Institute and collaborators Faithful to sources University Not in cards Jun 2025
IslamicEval 2025 Quran and Hadith Passage Retrieval Models rank up to 20 passages from the Quran and Sahih al-Bukhari for an Arabic question. Organizers score mean average precision at ten. Qatar Computing Research Institute and collaborators Questions that need search University Not in cards Jun 2025
IslamTrust Models choose answers consistent with scholarly consensus across Sunni schools of thought. The makers provide 406 questions in both Arabic and English. Abderraouf Lahmar and collaborators Knowledge in many languages University Not in cards Aug 2025
ITBench-AA Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON. Artificial Analysis + IBM Research Terminal tasks Expert Not in cards May 2026
IVISTA Voice Interaction, revised 2023 protocol In-car assistants carry out spoken commands from different seats and answer follow-up requests. Testers check completed actions, including window adjustments and navigation commands. China Automotive Engineering Research Institute General assistants Everyday Not in cards Mar 2025
IWSLT 2017 TED Translation Models translate TED talk transcripts between languages. The campaign includes bilingual translation and multilingual translation with unseen language pairs. IWSLT Translation University Not in cards
JailTrickBench Models respond to jailbreak attacks under varied attack and defense settings. A GPT-4 evaluator judges attack success across harmful-request datasets. HKUST (Guangzhou), Guangzhou HKUST Fok Ying Tung Research Institute and HKUST Jailbreaks and prompt injection Expert Not in cards Jun 2024
Jam-ALT Models transcribe song lyrics, including punctuation and line breaks. The scorer reports word error rate and separate measures for formatting. AudioShake Speech recognition and translation Everyday Not in cards Nov 2023
JAMA Clinical Challenge (JesseLiu) Models choose among four answers to clinical vignettes from JAMA. The OpenMed port combines 1,521 cases and reserves about 5% for evaluation. JesseLiu Medicine Expert Not in cards Nov 2024
JamendoLyrics MultiLang Models align written lyrics with singing across 80 songs in English, French, German, and Spanish. The researchers compare predicted and annotated word onsets. Spotify Research Speech recognition and translation Everyday Not in cards Jun 2023
Jamp Models infer temporal relationships between Japanese sentences. Separate test splits vary the templates, time formats, and time spans. Tomoki Sugimoto and colleagues Reasoning in many languages School Not in cards Jun 2023
JaNLI Models judge entailment between Japanese sentences designed to expose shortcuts in language inference. Hitomi Yanaka and Koji Mineshima Reasoning in many languages School Not in cards Nov 2021
Japanese LLM Leaderboard (lm-evaluation-harness) Models complete eight Japanese tasks covering reading, reasoning, math, and summarization. The harness groups the individual results. EleutherAI harness contributors Reading in many languages School Not in cards Nov 2024
Japanese Tort-case Dataset Models predict whether a Japanese court found a tort from the facts and the parties' arguments. They also label which arguments the court accepted. Hiroaki Yamada and colleagues Law Expert Not in cards May 2024
Japanese-English Bilingual Corpus of Wikipedia’s Kyoto Articles Models translate between Japanese and English using parallel Wikipedia passages about Kyoto. NICT Translation School Not in cards Oct 2010
JAQKET Models answer Japanese quiz questions. Tohoku University Knowledge in many languages School Not in cards
JASMINE Models replace medical jargon in Japanese sentences with everyday language. The reference sentences come from patients' blogs. Ehime University and Osaka University Medicine University Not in cards Mar 2024
Jawaher Models translate Arabic proverbs into English and explain their meanings in English and Arabic. Human raters score translations and explanations on five-point scales. Mohamed bin Zayed University of Artificial Intelligence and University of British Columbia Translation Everyday Not in cards Feb 2025
JBLiMP Models compare 331 pairs of Japanese sentences that differ in grammatical acceptability. Oseki Lab Reasoning in many languages Everyday Not in cards Jun 2022
JEEM Vision-language models caption images and answer visual questions in four Arabic dialects. The evaluation compares outputs with reference annotations. Toloka and MBZUAI Knowledge in many languages Everyday Not in cards Apr 2025
JEMHopQA Models answer Japanese questions that require combining facts across multiple documents and provide the supporting reasoning. Ai Ishii and colleagues Reading in many languages School Not in cards Jul 2023
Jester gesture recognition Models classify hand gestures in short videos. The task distinguishes movements with similar appearance but different directions or timing. TwentyBN Video understanding Everyday Not in cards Oct 2019
JFinQA Models calculate financial ratios, check consistency, and reason about changes over time using Japanese securities reports. ajtgjmdjp Finance University Not in cards Feb 2026
JGLUE Models classify Japanese sentences, compare sentence meanings, and answer reading comprehension and commonsense questions. The authors publish baseline scores for each task. Yahoo Japan Corporation and Waseda University Reading in many languages School Not in cards Mar 2022
JGPQA Models answer graduate-level science questions translated into Japanese from GPQA. LLM-jp Knowledge in many languages Expert Not in cards Sep 2025
JHumanEval Models implement Python functions from Japanese prompts. Programmers reviewed machine translations of the HumanEval specifications. Kuramitsu Lab Function-level problems School Not in cards Sep 2023
Jiminy Cricket Agents play 25 text adventures. Annotations on game states track moral behavior alongside game rewards. Dan Hendrycks and collaborators Deception and alignment Everyday Not in cards Oct 2021
JMultiWOZ-TC Models use Japanese dialogue history and past tool results to predict which tools to call next. LLM-jp Function calling Everyday Not in cards Dec 2025
JobBERT Job Title Normalization Models match English vacancy titles from Malaysia’s public job board to standardized occupation labels. The authors report reciprocal rank and recall at several cutoffs. TechWolf and Ghent University Other professions University Not in cards Sep 2021
Joi-Bench-Task Voice agents use tools while a simulated caller changes a request or interrupts a reply. The evaluator tracks completion, responsiveness, and omitted information. Tencent Speech recognition and translation Everyday Not in cards Nov 2026
JSeM Models decide whether a Japanese hypothesis follows from one or more premises. Native speakers provide the reference judgments. Daisuke Bekki and collaborators Reasoning in many languages School Not in cards Sep 2019
JSPLIB Agents schedule operations on specified machines. Each job must follow its operation order, and each machine can process only one operation at a time. Shorter schedules score better. JSPLIB contributors Tasks from real jobs University Not in cards Apr 2014
K-MHaS Models assign hate speech labels to Korean news comments. A comment can receive more than one label. Jean Lee and colleagues Bias, fairness and politics Everyday Not in cards Oct 2022
Kaeru Flip (paprika3d) Agents play an 86-note rhythm game from browser pixels. They must finish the song with at most one miss and leave autoplay off. jessicayjung Video games Everyday Not in cards Nov 2026
Kaggle Standardized Agent Exams Agents answer 16 questions on multistep reasoning and handling manipulative prompts. Kaggle publishes an overall score. Kaggle Deception and alignment School Not in cards Mar 2026
KairosQA Models answer questions about facts that changed over time, with each item tied to a time period. Kyutai Language reasoning University Not in cards Feb 2026
Kazakh Speech Corpus 2 Models transcribe Kazakh broadcasts, parliamentary speeches and podcasts alongside read speech. Some speakers switch between Kazakh and Russian. ISSAI, Nazarbayev University Speech recognition and translation Everyday Not in cards Aug 2022
KernelBench-Verified Language models generate CUDA kernels that must match PyTorch outputs and improve runtime. The verifier checks correctness with robust performance baselines. Meta FAIR and Superintelligence Labs GPU kernels Research frontier Not in cards Jan 2026
KernelBenchFiltered Agents optimize GPU kernels. METR takes the geometric mean of the speedups across tasks after checking correctness. METR GPU kernels Expert Not in cards Feb 2025
KeSpeech Models transcribe Mandarin speech from speakers with regional accents. Qwen compares transcription error against other speech recognizers. KeSpeech team Speech recognition and translation Everyday Not in cards
Keystone-Eval Agents configure Dockerfiles and test entrypoints for arbitrary code repositories, then run the repositories' tests. Imbue Fixing real repositories Expert Not in cards Apr 2026
KgCLUE Models answer Chinese questions using a knowledge graph. CLUE benchmark team Questions that need search School Not in cards
Khmer Handwritten Dataset (4.2k) Models transcribe line images from a handwriting collection that includes Khmer and English text. Royal University of Phnom Penh Documents, charts and text in images Everyday Not in cards Oct 2024
Kiji PII Detection Models locate and label personal information in multilingual text. Dataiku publishes a test split, and independent evaluators compare predicted spans with its annotations. Dataiku Following precise instructions Everyday Not in cards Feb 2026
Kili LLM Reasoning Benchmark Models answer language, mathematics, and reasoning questions at five proficiency levels. Kili Technology Language reasoning University Not in cards
KiloGram Models choose which of ten tangram pictures matches a description. The descriptions can name the whole shape or explain what its pieces represent. Cornell University, New York University, Allen Institute for AI and Princeton University Spatial reasoning and counting Everyday Not in cards Nov 2022
KilometerVision Models watch walking-tour videos geolocated over up to 1 km and answer questions on landmarks, routes and map layout, such as distances and loop closures. Google DeepMind (Perception Test team) Spatial reasoning and counting Everyday Not in cards Jun 2026
Kimi Code Bench Moonshot in-house coding-agent tasks from internal engineering cases, production incidents and open-source projects across 10+ languages. 80 tasks. Moonshot AI Fixing real repositories Expert Not in cards
KITTI depth prediction Models estimate scene depth from road images. The evaluation compares model predictions with depth measurements collected from a moving vehicle. Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago Spatial reasoning and counting School Not in cards
Knowledge Cutoff Benchmark Models answer questions about dated events. A judge checks direct answers against known outcomes. A separate multiple-choice test uses an answer key. Apoorv Saxena Short factual questions Everyday Not in cards Jul 2026
KnowledgeGraphEval 2026 Models label nested entities in Arabic text from ten domains. A separate test asks models to classify relations between marked pairs of entities. KnowledgeGraphEval organizers Reading in many languages University Not in cards
KoBEST Models do five Korean tasks designed by professional linguists: BoolQ, COPA, WiC, HellaSwag and SentiNeg. All data is human-annotated. SK Telecom and collaborators (Kim, Jang et al.) Reasoning in many languages Everyday Not in cards Apr 2022
Kompetencer Models classify skill and knowledge phrases from Danish job ads using a European occupational taxonomy. The authors report weighted macro F1. IT University of Copenhagen and LMU Munich Other professions University Not in cards Apr 2022
KoMT-Bench Models follow multi-turn instructions in Korean on a translated and culturally adapted version of MT-Bench, reviewed by expert linguists. LG AI Research Chat and instructions in many languages Everyday Not in cards Aug 2024
Kontext Bench Image editing models get 1,026 image and instruction pairs across local edits, global edits, text editing, style reference and character consistency. Black Forest Labs Video generation Everyday Not in cards Jun 2025
Kor-Phishing Models classify web requests and links as malicious or benign. Korea's AI Safety Institute reports accuracy separately for each group of 100 items. Korea AI Safety Institute Secure code and review University Not in cards Dec 2025
KORA Apps Simulated children use consumer AI apps end to end, with each app system prompt, filters and safeguards active. Scores cover online, physical and sexual, psycho-social and learning risks. KORA (non-profit) Wellbeing and sycophancy Everyday Not in cards Jul 2026
Korean Canonical Legal Benchmark Models answer Korean legal questions with supporting precedents. The benchmark includes multiple-choice questions and essays with individual grading criteria. Hongseok Oh and colleagues Law University Not in cards Oct 2025
Korean National Licensing Examination for Korean Medicine Doctors 2022 Models answer 340 questions from the Korean medicine licensing exam. Researchers remove the images, transcribe tables, and compare prompt variants against the answer key. Gachon University and Pusan National University Medicine Expert Not in cards Mar 2023
KPI-EDGAR Models identify key performance indicators and related values in financial reports. The original annotations also link entities within the text. Tobias Deußer and colleagues Finance University Not in cards Oct 2022
KSAA 2026 Automatic Diacritization of Speech Dictation Models restore Arabic vowel marks from a recording and its transcript. Organizers rank systems by word error rate with case endings and unmarked characters included. King Salman Global Academy for Arabic Language and collaborators Speech recognition and translation School Not in cards Dec 2025
KuralHub Models classify emotions in speech from 29 languages. Luxshan Thavarasa and colleagues Audio understanding and voice Everyday Not in cards Feb 2025
KurdiSent Models predict the sentiment expressed in Kurdish text. Soran Badawi and colleagues Reading in many languages Everyday Not in cards
Kyoto University Commonsense Inference Dataset Models select a plausible continuation for a Japanese passage by inferring the relationship between events. Kyoto University Reasoning in many languages Everyday Not in cards Nov 2020
Kyoto University Wikipedia Annotated Corpus Models analyze Japanese Wikipedia text. The harness tests reading prediction, names, syntactic dependencies, predicate arguments, and coreference. Kyoto University Reading in many languages University Not in cards Oct 2022
LAB: Contracts 500 in-house contracting tasks (drafting, redlines, issues lists, escalation memos) across 10 contract categories. All-pass expert-rubric grading. Harvey Law Expert Not in cards Jun 2026
LAB: Diligence Agent researches a synthetic M&A virtual data room (thousands of documents, up to ~80M tokens) and drafts a diligence memo scored against planted-issue rubric criteria. Harvey Law Expert Not in cards Jul 2026
LAB: Law Firm Knowledge Agents search a simulated law firm’s matter files and deliver work that satisfies legal task criteria. Harvey Law Expert Not in cards Aug 2026
LAB: US Case Law Agents draft a litigation brief for the same party and motion as a real filed federal brief, from the case record. Model judges compare briefs pairwise and wins become Elo ratings. Harvey Law Expert Not in cards Oct 2026
LAD2000 Models identify unusual events in videos and score when they happen. The test contains 280 videos with unusual events and 280 without them. Jiangxi University of Finance and Economics Video understanding Everyday Not in cards May 2021
Lahjoita puhetta ASR test Models transcribe spontaneous Finnish speech on the official ten-hour test. Human transcripts supply the reference words. Aalto University and University of Helsinki Speech recognition and translation Everyday Not in cards Mar 2022
Lakh MIDI MASD Style Classification Models identify which of 25 styles apply to a MIDI file. The released mapping labels 17,785 files, and each file can have multiple labels. Universitat Pompeu Fabra and University of Helsinki Audio understanding and voice School Not in cards Sep 2017
Lakh MIDI Top-MAGD Genre Classification Models identify which of 13 genres apply to a MIDI file. The released mapping labels 22,535 files, and each file can have multiple labels. Universitat Pompeu Fabra and University of Helsinki Audio understanding and voice School Not in cards Sep 2017
LAMBADA OpenAI Multilingual Models predict the last word in short narrative passages. The harness includes English and four machine-translated test sets. EleutherAI Reading in many languages Everyday Not in cards
Lamina Product Bench Agents build four small applications under three instructions: code directly, plan first, or use Lamina skills. A model judge grades each submission. Lamina Building whole projects University Not in cards Jul 2026
LaminaBench v1 Agents implement the same product brief with and without Lamina skills. The package README specifies ten briefs across five categories and 20 paired Harbor tasks. Lamina Building whole projects University Not in cards Jul 2026
LangChain Benchmarks Agents complete tool use, retrieval and extraction tasks with reproducible LangSmith datasets and task-specific graders. LangChain General assistants Expert Not in cards Nov 2023
Language-Table A simulated robot moves colored blocks in response to natural language instructions. The benchmark measures success and success weighted by path length across five task families and 696 task variations. Robotics at Google Simulated robots Everyday Not in cards Oct 2022
Large Movie Review Dataset Models classify an IMDb movie review as positive or negative. Andrew Maas and colleagues Reading comprehension Everyday Not in cards Jun 2011
Large-Vocabulary Video Instance Segmentation Models detect, segment, and track object instances across video frames. The benchmark includes categories outside the training vocabulary. LV-VIS team Video understanding Everyday Not in cards Apr 2023
LARYBench Models answer expert questions in the LARYBench dataset released by Meituan LongCat. Meituan LongCat Team Expert-written questions Expert Not in cards Mar 2026
Latin Synonym Selection Benchmark Models choose the closest synonym of a Latin lemma from four options. The released file contains 2,759 questions. LiLa, Università Cattolica del Sacro Cuore Reading in many languages University Not in cards Oct 2019
LayMaker MTSamples Simplification Models rewrite medical transcription sentences for patients. Clinicians supplied the reference versions and kept sentences they already considered clear. Babylon Health Medicine University Not in cards Dec 2021
LCBench v0.1 Models solve English and Chinese LeetCode contest problems. Tests check the submitted programs. OpenCompass Competitive programming Competition Not in cards May 2024
LeanDojo Benchmark Models prove theorems from Lean's mathlib library inside the Lean proof assistant, which checks every proof. The benchmark has 98,734 theorems and proofs with premise annotations. A novel-premises split tests whether a prover can use lemmas it never saw in training. Caltech, NVIDIA and collaborators (LeanDojo) Formal proofs University Not in cards Jun 2023
LedgerBench-100 Agents perform 100 corporate finance tasks through 66 tools in a shared enterprise system. SQL checks grade the changes they make. Blobfish AI Finance Expert Not in cards Aug 2026
Leeds Sports Pose Extended Models locate body joints in photographs of athletes. Meta compares the fraction of predicted joints within a distance threshold. Sam Johnson and Mark Everingham Spatial reasoning and counting School Not in cards Jun 2011
Legal Information Retrieval Benchmark Search APIs retrieve the official document behind legal research requests. Exact URL matching scores whether the official page appears first or in the top ten. Firecrawl Law Expert Not in cards Nov 2026
Legal Opinion Sentiment Classification Models classify the sentiment expressed in passages from legal opinions. Expert annotations provide the reference labels. Gathika Ratnayaka and colleagues Law University Not in cards Oct 2020
Legal World sample Agents act as a legal assistant in an anonymized law practice. Four tasks grade the drafts, mail, calendar changes, and files they leave behind. doe Law Expert Not in cards Nov 2026
LegalSupport Models read a legal claim and two court case summaries. They choose the case that most directly supports the claim. Neel Guha and Stanford CRFM Law University Not in cards
LEVIR-CC Models compare aligned aerial images from Texas and describe what changed. Five annotators wrote reference captions for each of the 1,929 test pairs. Chenyang Liu, Rui Zhao, Hao Chen, Zhengxia Zou, and Zhenwei Shi Detection and captioning Everyday Not in cards Apr 2022
LEVIR-CDC Models describe changes between aerial images and mark changed buildings. The authors paired LEVIR-CC captions with binary building masks. Jingye Shi and colleagues Detection and captioning Everyday Not in cards Oct 2024
LexGLUE Models solve a suite of English legal text tasks drawn from court cases, EU law, contracts and terms of service. Most tasks are classification, and one asks for the holding of a US case. Each task reports micro and macro F1. University of Copenhagen and collaborators Law Expert Not in cards Oct 2021
LEXTREME Models analyze legal text across languages. Tasks include predicting document labels, identifying legal arguments, and labeling words within documents. Joel Niklaus and colleagues Law University Not in cards Jan 2023
LGEB Models generate summaries and questions from source passages. CLUE benchmark team Reading comprehension School Not in cards
LightGenBench Models predict which parts of a 3D surface emit light and what color they emit. The authors compare generated emission with reference assets. Dongchen Yang and colleagues Image generation and editing University Not in cards Oct 2026
Lights Out 3×3 Models toggle lights on a three-by-three grid until every light is off. The dataset reserves 30 boards for testing and rewards efficient solutions. candukuri Puzzles and chess School Not in cards Sep 2025
LIMIT Models retrieve specific pairs of documents from a shared collection using embeddings. Orion Weller and colleagues Questions that need search School Not in cards Aug 2025
LIMMITS 2025 Speech models synthesize text in four languages. Listeners rate naturalness and voice similarity. Separate tracks also measure streaming latency, codec bitrate, or both. Indian Institute of Science and LIMMITS organizers Audio understanding and voice Everyday Not in cards Aug 2024
Linear Algebra Done Right (Axler Exercises) Models solve 186 linear algebra tasks on 25 topics drawn from chapters one through nine of Axler’s textbook. Exact rational arithmetic checks their answers. nair-akash Word problems University Not in cards Aug 2026
LineEX Models extract labels and numerical series from scientific line charts. The hub port uses a 20,000-image test split and grades the structured output. IIT Gandhinagar Documents, charts and text in images University Not in cards Jul 2022
LitBench Judges predict human preferences across 2,480 held-out story pairs. The hub port instead samples the training corpus and adds self-critique. SAA Lab Taste and style University Not in cards Jul 2025
LitSearch Models retrieve relevant research papers in response to questions about the scientific literature. Anirudh Ajith and colleagues Questions that need search Expert Not in cards Jul 2024
LiveMACE Agents trade in live financial markets, and the benchmark scores the process capabilities behind their decisions, not only returns. Fudan University Finance Expert Not in cards Oct 2026
LiveMath Models solve 250 math problems written with six professional mathematicians. The sets are 150 regular high school problems, 50 at about AIME level and 50 multimodal problems with figures. Google DeepMind Competition problems with answers Competition Not in cards Sep 2025
LlamaTouch Agents carry out requests in Android apps. The evaluator checks whether each run visited the required app or system states. Beijing University of Posts and Telecommunications and Tsinghua University Phones Everyday Not in cards Apr 2024
LLaVA-Bench (In-the-Wild) Multimodal assistants answer open questions about a small set of diverse real images. A GPT-4 judge scores each answer relative to a reference answer. UW-Madison and Microsoft Research (LLaVA team) General visual questions Everyday Not in cards Apr 2023
LLM Economist Models set tax rates or choose working hours in a simulated economy. Researchers compare the social welfare achieved under different tax policies. Princeton University and Salesforce Research Running a business University Not in cards Jul 2025
LLM Training Puzzles Models solve eight coding puzzles on distributed neural network training, including optimizer state and pipeline parallelism. The hub evaluates each solution in a sandbox. Sasha Rush Machine learning engineering Expert Not in cards Jun 2023
llm-jp-eval Models answer Japanese language understanding questions, translate passages, and follow instructions. The harness reports scores for individual datasets and groups of tasks. LLM-jp Reasoning in many languages School Not in cards Oct 2023
LLM-NERRE materials extraction Models read materials papers and link dopants to host materials, guest species to metal-organic frameworks, and crystal structures to chemical formulas. Lawrence Berkeley National Laboratory and University of California, Berkeley Lab and data work University Not in cards Dec 2022
LLMEval-Med Models answer clinical questions drawn from patient records and expert scenarios. A model grades answers against physician-developed checklists. Ming Zhang and colleagues Medicine Expert Not in cards Jun 2025
LM-SynEval Models compare English sentences that differ in grammaticality. The test measures preferences for correct agreement and other syntactic dependencies. Rebecca Marvin and colleagues Language reasoning Everyday Not in cards Oct 2018
LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. Google DeepMind Embodied reasoning University Not in cards Dec 2024
LMSYS Chatbot Arena Human Preference Predictions Models predict which answer a user preferred in an anonymous pair. Kaggle scores the submitted probabilities by log loss. LMSYS and Kaggle Overall chat quality University Not in cards May 2024
LoCoMo Models answer questions, summarize events and continue dialogue over very long multi-session chats of about 300 turns and 9K tokens, up to 35 sessions. Answers are scored against references. Snap Research, UNC Chapel Hill and USC (Maharana et al.) Finding facts in long inputs Everyday Not in cards Feb 2024
LogiQA Models answer reading comprehension questions that test logical reasoning, taken from Chinese civil service exams and translated to English. Liu et al. (LogiQA authors) Language reasoning University Not in cards Jul 2020
Long Context Navigator Agents answer questions that link facts across a synthetic document of about 32,000 tokens. They must read, compress, or discard information within a 600-token working context. manasganti Reasoning over long documents University Not in cards Oct 2026
Long Range Arena Transformers solve six long-sequence tasks covering list operations, text, retrieval, image, path and path crossing. Google Research and collaborators Finding facts in long inputs Expert Not in cards Jan 2021
Long-document QA (Kensho) long-document-qa Models answer questions by reasoning over multi-hundred-page financial filings and other long documents. Kensho Technologies, S&P Global Finance Expert Not in cards Oct 2024
Long-term Video Object Segmentation v2 Models track object masks through long videos. The evaluation tests whether they can recover an object's identity after an extended absence. Lingyi Hong and colleagues Video understanding Everyday Not in cards Apr 2024
LongBench Chinese Models answer questions and complete other long-context tasks in the original Chinese headline table. The score is its published average on a percentage scale. THUDM (Tsinghua University) Reasoning over long documents University Not in cards Aug 2023
LongLeader Models answer a unified suite of long-context tasks across multiple context lengths and abilities. Accuracy tables compare language models and context-extension methods. Amazon Science Long context University Not in cards Jan 2025
LongPuzzleBench GUI agents solve 114 visual puzzle levels across six games through native interface actions. LongPuzzleBench authors Puzzles and chess Everyday Not in cards Sep 2026
LoTTE Models retrieve passages that answer questions about less common topics. Keshav Santhanam and colleagues Questions that need search School Not in cards Dec 2021
LoveDA Models label land cover in aerial images of urban and rural areas. The benchmark supports comparisons across these two settings. Junjue Wang and colleagues Detection and captioning University Not in cards Oct 2021
LP-TTS-Bench Models synthesize speech from extended recordings of a speaker. The benchmark measures how well they reproduce timing, emphasis, and intonation on 2,728 held-out utterances. The Chinese University of Hong Kong, Shenzhen and Amphion Technology Speech recognition and translation Everyday Not in cards Oct 2026
LVIS Models locate objects and outline each instance across a large vocabulary of categories. Meta reports box and mask average precision. Agrim Gupta and colleagues Detection and captioning Everyday Not in cards Aug 2019
LVSum Models summarize long videos with timestamped segments, relevance scores and descriptions using video frames, transcripts or both. Apple Long videos Expert Not in cards Apr 2026
M3-Bench Multimodal agents answer questions about long videos while using long-term memory and multimodal retrieval. ByteDance Seed Long videos Expert Not in cards Aug 2025
M³AV Speech Recognition Models transcribe academic lectures. The authors measure word errors across the transcript and separately for rare words. Slide text can provide context. Shanghai Jiao Tong University and collaborators Speech recognition and translation University Not in cards Mar 2024
M4Singer Models transcribe lyrics from singing voices. Qwen compares recognition systems on the M4Singer recordings. M4Singer team Speech recognition and translation Everyday Not in cards
MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. The score is normalized game reward. Annotations also score power-seeking, disutility and ethical violations. UC Berkeley and Center for AI Safety Deception and alignment Everyday Not in cards Apr 2023
MacroLens Models use company and economic data to make financial estimates under different scenarios. The authors compare forecasting models with language models and simple baselines. Patara Trirat and colleagues Finance University Not in cards Jun 2026
MAD-Bench Multimodal models answer image questions containing deceptive prompt information about objects, counts and spatial relationships. Apple General visual questions University Not in cards Feb 2024
MAD-Eval Models write descriptions for scenes from ten movies. One set of references names characters. Another replaces their names with the word “SOMEONE.” University of Oxford, École des Ponts and Shanghai Jiao Tong University Video captioning Everyday Not in cards Mar 2023
MADASR 2.0 Models transcribe read speech and spontaneous conversations across eight Indian languages. The organizers rank each track by character error and also score language and dialect identification. Indian Institute of Science and MADASR organizers Speech recognition and translation Everyday Not in cards Apr 2025
MADE 1.0 Models identify drugs, indications, and adverse events in cancer care notes. They link each drug to its attributes and effects. The challenge scores extraction, linking, and the combined task. University of Massachusetts Medicine University Not in cards Nov 2017
MAESTRO Piano Transcription Models transcribe competition piano recordings into notes. The evaluator compares predictions with MIDI events captured by the instrument. Each composition appears in only one split. Google Magenta and the International Piano-e-Competition Audio understanding and voice Expert Not in cards Oct 2018
MAFAND-MT Models translate news text in a benchmark covering 21 African languages. David Ifeoluwa Adelani and colleagues Translation School Not in cards Jul 2022
Magentic Marketplace Agents search for sellers, exchange proposals and make purchases. The simulation measures how much value customers receive after paying for their purchases. Microsoft Research Running a business Everyday Not in cards Oct 2025
MagnaTagATune Models predict descriptive tags for music clips. Qwen compares their predictions with human annotations using micro F1. Edith Law and colleagues Audio understanding and voice Everyday Not in cards
Make Data Count: Finding Data References Models find dataset identifiers in scientific papers and classify each reference as data the authors created or reused. Kaggle scores the labeled references with F1. Make Data Count and DataCite Reading comprehension University Not in cards Jun 2025
Malaysian Sign Language Medical Dataset Models identify Malaysian Sign Language terms used in health care. The training and test lists share all seven signer names. Swinburne University of Technology Sarawak and Sarawak Society for the Deaf Video understanding Everyday Not in cards Jul 2023
Málrómur Models transcribe Icelandic sentences that volunteers recorded on smartphones. Human listeners checked the reference text against the recordings. Reykjavík University and the Icelandic Centre for Language Technology Speech recognition and translation Everyday Not in cards Aug 2014
Maltese News Categories Models assign one or more topic labels to Maltese news articles. Amit Kumar Chaudhary and colleagues Reading in many languages School Not in cards May 2024
MammalNet Models identify mammals and their behavior in videos. The benchmark covers 173 mammal categories and 12 behaviors, including nursing and hunting. Jun Chen and colleagues Video understanding School Not in cards Mar 2023
MAMUBench Agents represent different users who share a limited resource. The benchmark compares their combined outcome with one agent coordinating for everyone. Anthropic Fellows Program and collaborators Multi-agent games Everyday Not in cards Oct 2026
ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. Chongkai Gao and colleagues Simulated robots Everyday Not in cards Nov 2026
ManipArena 20 real bimanual tasks on X2Robot platforms: 10 execution, 5 semantic reasoning and 5 mobile manipulation, with in-domain and shifted trials. Partial-credit scoring. X Square Robot Real robots Everyday Not in cards Mar 2026
ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. UC San Diego (Hao Su Lab) + Hillbot Simulated robots Everyday Not in cards Oct 2024
Mantic vs Polymarket backtest Point-in-time backtest of Mantic's AI forecasts against prices on 4,139 Polymarket markets at volume snapshots. Scored by Brier score and calibration. Mantic Real-world events Expert Not in cards Sep 2026
Manually-Reviewed Agentic Labeled Transcripts Monitors classify agent transcripts for reward hacking and sandbagging. METR compares their predictions with human labels using the area under the receiver operating characteristic curve. METR Deception and alignment Expert Not in cards Oct 2025
Many Labs 2 text-based replications Models answer questions as participants in 14 psychology experiments. Researchers compare the effects with human replications and report the share of findings the model reproduces. Peter S. Park, Philipp Schoenegger, and Chongyang Zhu Social simulation Everyday Not in cards Feb 2023
MAPA EUR-Lex Models mark sensitive entities in EU legal texts. MAPA annotates general and fine-grained categories. The multilingual conversion covers 21 languages. Multilingual Anonymisation for Public Administrations consortium Law University Not in cards
MapAnything dense multi-view reconstruction Models reconstruct a scene from multiple images. The benchmark compares point geometry, metric scale, depth, and camera poses with reference measurements. Meta Spatial reasoning and counting School Not in cards Sep 2025
MapAnything single-view calibration Models infer camera intrinsics from a single image. The benchmark measures angular error between predicted camera rays and reference rays. Meta Spatial reasoning and counting School Not in cards Sep 2025
MAPS Piano Transcription Models transcribe piano recordings into notes. A common test uses 60 Disklavier recordings from two microphone setups and checks predicted pitches and timing against MIDI. Télécom ParisTech Audio understanding and voice Expert Not in cards
Markov Blanket Discovery (shinpaku) Models write a function that identifies the variables in a target’s Markov blanket from binary data. The evaluator measures F1 against known variable sets. shinpaku Machine learning engineering Expert Not in cards Aug 2026
MasakhaNER 2.0 Models identify named entities in human-annotated text across 20 African languages. David Ifeoluwa Adelani and colleagues Reading in many languages School Not in cards Oct 2022
MasakhaNEWS Models predict the topic of news articles written in African languages. David Ifeoluwa Adelani and colleagues Reading in many languages Everyday Not in cards Apr 2023
MASSIVE Models identify an utterance’s intent and label its argument slots across 51 languages. The held-out test reports intent accuracy, slot-filling F1, and exact match accuracy for the complete prediction. Amazon Alexa AI Reading in many languages Everyday Not in cards Apr 2022
Massive Visualized IR Benchmark Models retrieve screenshots that match a search query. The benchmark compares performance across four retrieval task groups. Beijing Academy of Artificial Intelligence Documents, charts and text in images Everyday Not in cards Feb 2025
Massively Multilingual Speech labeled corpus Models transcribe recorded readings across languages. The evaluation compares predictions with reference transcripts using character error rate. Meta Speech recognition and translation School Not in cards May 2023
Mastermind Information Gain Models choose a Mastermind move from earlier guesses and feedback. The evaluator measures information gain for individual moves and success over complete games. peterN908 Puzzles and chess School Not in cards Sep 2025
MatBERT gold nanoparticle entity recognition Models mark particle names and words describing their shapes in passages from gold nanoparticle papers. The evaluation scores both types of text span. Lawrence Berkeley National Laboratory and UC Berkeley Lab and data work University Not in cards Oct 2021
Materials Science Procedural Text Corpus Models identify operations in materials synthesis procedures and link each operation to its materials, instruments, and conditions. Experts annotated 230 procedures. University of Massachusetts Amherst and Massachusetts Institute of Technology Lab and data work University Not in cards Nov 2018
Mathematical Problem Repository for AlphaEvolve Agents generate programs that construct mathematical objects or improve known bounds. The repository documents 67 problems and provides verification code for many of them. Google DeepMind Research mathematics Research frontier Not in cards Nov 2025
MathOlympiadBench Models produce Lean 4 proofs for 360 human-checked formalizations of olympiad problems. Lean checks each proof. Goedel-LM Formal proofs Competition Not in cards Aug 2025
MatSci-NLP Models read text from materials science papers and complete seven tasks, including identifying synthesis actions and linking measurements to experiments. The authors used 99% of examples for testing. Université de Montréal, Mila, and Intel Labs Lab and data work University Not in cards May 2023
MAUD Models classify provisions in merger agreements into expert-defined legal categories. The benchmark compares predictions with lawyer annotations and reports precision-recall area and micro and macro F1. ETH Zurich, Harvard University, The Atticus Project and others Law Expert Not in cards Jan 2023
MAWARITH v2026 Models identify eligible heirs and calculate each share of an estate from an Arabic case. The MIR-E score gives credit for intermediate decisions. Hamad Bin Khalifa University and Nazarbayev University Law University Not in cards Dec 2025
MAWPS Models solve one-unknown arithmetic and algebra word problems drawn from several earlier collections. Each problem has an equation and a numeric answer, and the final answer is scored by exact match. University of Washington and collaborators (Koncel-Kedziorski, Roy, Amini, Kushman, Hajishirzi) Word problems School Not in cards Jun 2016
Maydan HF RL Sampler Agents reconcile synthetic Saudi social insurance contributions across ten payroll tasks. SQLite checks the final records and audit log. Maydan Labs Documents and spreadsheets Expert Not in cards Oct 2026
MBench Models generate human motion from 450 prompts. The evaluation measures motion quality and uses a vision-language model to judge prompt matching and generalization. ViMoGen team at Nanyang Technological University, SenseTime Research and collaborating institutions Other Everyday Not in cards Oct 2025
MBTI Post Classification (Kunumi) Models predict an author’s four-letter Myers-Briggs type from their posts. The port holds out one quarter of the rows and gives partial credit for matching letters. Kunumi Reading comprehension Everyday Not in cards Sep 2025
MC-TACO Models judge plausible answers to questions about when events happen, how often they recur and how long they last. Ben Zhou and colleagues Commonsense Everyday Not in cards Sep 2019
MCP-SIM Agents turn physics problems into FEniCS simulations. The tasks include heat flow, elastic deformation, and fracture. The authors count solved tasks. Donggeun Park, Hyunbin Moon, and Seunghwa Ryu Scientific code University Not in cards Jun 2025
MdCoEditBench Agents edit selected Markdown in the WorkBuddy editor alongside a user. Tasks include restoring a version, resolving an edit conflict, and continuing existing text. Tencent Documents and spreadsheets University Not in cards Nov 2026
MeasEval Models extract quantities and units from scientific paragraphs and link them to measured properties, entities, and qualifiers. The competition averages scores across nine components. Elsevier Labs and the University of Amsterdam Lab and data work University Not in cards Jul 2020
MECCANO Action Recognition Models identify an assembly action from a first-person video clip of someone building a toy motorcycle. The answer key has 61 action classes. University of Catania and collaborators Video understanding Everyday Not in cards Oct 2020
Med-EASi Models rewrite English medical passages in plain language. The references mark replacements, explanations, insertions, and deletions. Stanford University, University of Zurich, and Cornell University Medicine University Not in cards Feb 2023
MedAlign Models follow clinician-written instructions using a patient’s electronic medical record. Clinicians evaluate the responses. Scott L. Fleming and colleagues Medicine Expert Not in cards Aug 2023
MedBench (Russian closed-test platform) Closed-test platform Models answer Russian medical questions and classify relationships between clinical statements. The platform also scores electrocardiogram labels on a separate task. MedBench team Medicine University Not in cards Nov 2022
MedDialog Models generate a clinician’s response from a patient description. The harness also evaluates summaries of medical questions using reference text. Guangtao Zeng and colleagues Medicine Expert Not in cards Nov 2020
MEDDOPROF Models find occupations and employment status in Spanish clinical notes. They identify whose occupation each mention describes and assign an occupational or clinical code. Barcelona Supercomputing Center Medicine University Not in cards Dec 2020
MedGemma lab report conversion Models convert lab reports to structured JSON. Google reports macro and micro F1 on two collections of clinical PDFs and a collection of synthetic reports. Google Documents, charts and text in images School Not in cards Jan 2026
MedGenVidQA 2026 Visual Answer Localization Models mark the start and end of the video segment that answers a medical question. The organizers created 80 test examples from professional medical videos. U.S. National Library of Medicine Medicine University Not in cards Jan 2026
Media Bias Identification Benchmark Models classify text in eight tasks, including linguistic bias, political bias, and hate speech. The benchmark reports F1 across pooled examples and separately by source dataset. Media Bias Group Bias, fairness and politics Everyday Not in cards Feb 2023
Media Frames Corpus Models identify the main frame of a US news article, such as economic consequences or public opinion. Researchers compare predictions with human labels. Dallas Card, Amber Boydstun, Justin Gross, Philip Resnik, and Noah Smith Reading comprehension School Not in cards Jul 2015
Medical Paper-Blog Dataset Models rewrite a sentence from a medical paper for a general reader. The reference sentence comes from a medical blog. International Institute of Information Technology Hyderabad Medicine University Not in cards May 2020
Medical Prescription Selection (lm-evaluation-harness) Models read a diagnosis or complaint and choose the recorded prescription. The two task variants use different distractors. EleutherAI lm-evaluation-harness contributors Medicine Expert Not in cards Mar 2025
MedICaT Models locate panels within a biomedical figure and match each panel to the relevant words in its caption. The original test scores word overlap. Allen Institute for AI and collaborators Documents, charts and text in images University Not in cards Oct 2020
MedicationQA Models answer consumer questions about medications. The original corpus pairs questions with expert answers, and the hub evaluation grades answer accuracy and completeness. US National Library of Medicine Medicine University Not in cards Mar 2019
MEDIQA 2019 Question Answering Models answer consumer health questions or assess candidate answers. The shared task evaluates the relevance and quality of answers to medical questions. Asma Ben Abacha and colleagues Medicine Expert Not in cards Aug 2019
MEDIQA-SYNUR Models extract structured observations from synthetic nursing dictations and map them to a clinical vocabulary. The hub uses the development split by default and supports the test split. MEDIQA organizers Medicine Expert Not in cards Dec 2025
MedMentions Models identify biomedical concepts in PubMed abstracts and link text spans to identifiers in the Unified Medical Language System. Chan Zuckerberg Initiative Genomics and life sciences University Not in cards May 2018
MedPhysBench Knowledge Questions Models answer board-style medical physics questions covering radiation oncology, dosimetry, imaging, and radiation protection. Version 0.2.0 contains 412 questions, including 115 test items. omar Medicine Expert Not in cards Jan 2026
MedProcNER Models identify procedures in Spanish clinical cases, link them to SNOMED CT codes, and list the codes for each case. Barcelona Supercomputing Center Medicine University Not in cards Jan 2023
MedPT Models classify Brazilian Portuguese medical questions by specialty. The original evaluation tests 20 classes, while the hub creates a split after deduplication and label selection. Fernanda Bufon Farber and colleagues Medicine University Not in cards Nov 2025
MedQuAD Systems answer medical questions using a collection of 47,457 question-answer pairs from NIH websites. The OpenMed port uses a model judge on a subset. Asma Ben Abacha and Dina Demner-Fushman Medicine University Not in cards Jan 2019
MedReadMe Models rate how easy a medical sentence is to understand. Researchers compare the ratings with judgments from readers. Georgia Institute of Technology Medicine Everyday Not in cards May 2024
MedSecId Models identify section boundaries and assign a section type in clinical notes. The authors annotated 2,002 notes and reserved 10% for testing. University of Illinois at Chicago Medicine University Not in cards Oct 2022
MedText Models suggest diagnoses or treatments for textbook-style patient presentations. The harness compares their text with the recorded completion. BI55 (MedText dataset) Medicine Expert Not in cards Jul 2023
MedVidQA 2023 Medical Instructional Question Generation Models write a question that a supplied medical video segment answers. They can use both the images and the subtitles. U.S. National Library of Medicine and National Institute of Standards and Technology Medicine Everyday Not in cards Feb 2023
MedVidQA 2024 Query-Focused Instructional Step Captioning Models divide a medical demonstration into steps that answer a question. They give each step a caption and start and end times. U.S. National Library of Medicine and National Institute of Standards and Technology Medicine Everyday Not in cards Feb 2024
MedVidQA 2024 Video Corpus Visual Answer Localization Models search 48,605 medical videos for demonstrations that answer 52 new questions. They return ranked videos and the start and end times of each answer. U.S. National Library of Medicine and National Institute of Standards and Technology Medicine Everyday Not in cards Feb 2024
MEEN 361 Conceptual Question Grading Models use an instructor’s rubrics to grade answers to mechanical engineering laboratory quizzes. Gao and colleagues compared model grades with human grades across ten questions. Rujun Gao and colleagues at Texas A&M University and partner institutions Other professions University Not in cards Nov 2024
Megaminx solver Models solve a simulated twelve-faced twisty puzzle. They read the puzzle state as text, make moves with tools and receive rewards from the simulator. setrf Spatial reasoning and counting University Not in cards May 2026
MegaOath Forecasting Survey Models predict how each of 21 honesty oaths changes the share of earned income people report in a tax game. Researchers score correlation with observed effects. Janis H. Zickfeld and the MegaOath Collaboration Social simulation University Not in cards Nov 2023
MELA Models judge whether sentences are linguistically acceptable across languages. The evaluation compares their judgments with the labeled examples. Ziyin Zhang and colleagues Reasoning in many languages Everyday Not in cards Nov 2023
MELD Models identify emotions in utterances from multiparty conversations. The dataset includes speech, video, and text. Soujanya Poria and colleagues Audio understanding and voice Everyday Not in cards Oct 2018
Mem-Gallery Multimodal agents hold long conversations over many sessions that mix images and text, then answer questions that need what they remembered. Questions test memory extraction and test-time adaptation, reasoning over memory, and managing knowledge that changes over time. Yuanchen Bei, Tianxin Wei and coauthors General visual questions Everyday Not in cards Jan 2026
Menagerie: A Dataset of Graded Programming Assignments Models grade student Java programs that simulate predators and prey. They score correctness, design, readability, and documentation against marks assigned by human graders. Marcus Messer and colleagues at King's College London Other professions University Not in cards May 2024
MeQSum Models condense consumer health questions into shorter questions while preserving the medical information needed for retrieval. Reference summaries provide the comparison. Asma Ben Abacha and colleagues Medicine University Not in cards Jul 2019
MER-UniBench Models describe emotions from audio, video, and text across nine datasets. Label normalization supports hit rate, weighted F1, and emotion-wheel F1. The reported mean averages the primary metric from each dataset. Institute of Automation, Chinese Academy of Sciences, University of Oulu, and collaborators Audio understanding and voice Everyday Not in cards Jan 2025
MermaidSeqBench Models translate natural-language descriptions into Mermaid sequence diagrams. Judges score syntax, logic, completeness, activation handling and error tracking. IBM Research Web and app front ends University Not in cards Nov 2025
MET-Meme Models classify Chinese and English memes by metaphor, sentiment, intent, and degree of offensiveness. Dalian University of Technology and collaborators General visual questions Everyday Not in cards Feb 2022
Meta-Album Models classify unfamiliar images after seeing a few labeled examples. Tasks span ten domains, including microscopy, remote sensing and wildlife. Meta-Album team and ChaLearn General visual questions School Not in cards Jun 2022
Meta-Dataset Models learn from episodes sampled across diverse image datasets and classify novel classes from a few examples. Google Research and collaborators General visual questions University Not in cards Mar 2019
META-GUI Models predict the next Android action and reply from screenshots and dialogue history. A turn succeeds when every predicted action matches the recorded interaction. Shanghai Jiao Tong University Phones Everyday Not in cards May 2022
Meta-World Robot policies do simulated tabletop manipulation tasks, scored by success rate. Splits test multi-task learning, such as MT10 and MT50, and meta-learning, such as ML45. Farama Foundation (originally Stanford, Berkeley and others) Simulated robots Everyday Not in cards Oct 2019
metal-arm-harness Safety-enveloped CLI that lets any coding agent drive a real MakerMods Metal or Maker arm one command at a time from camera JPEGs. MakerMods Robotics Real robots Everyday Not in cards Sep 2026
METR Faithfulness and Monitorability Experiments Models answer difficult questions with clues. Researchers measure whether the reasoning reveals the use of a clue when the model answers correctly. METR Deception and alignment University Not in cards Aug 2025
METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. METR AI research tasks Research frontier Not in cards
METR side task monitorability evaluation Monitors review agent transcripts for evidence of a side task. METR measures how frequently monitors miss an agent that completed one. METR Deception and alignment Expert Not in cards Jan 2026
MeViS Models segment the objects in a video that a sentence picks out by their motion, not by static looks. Scored by J&F. Nanyang Technological University and Fudan University (Ding, Liu, He, Jiang, Loy) Video understanding Everyday Not in cards Aug 2023
MIEB Image and image-text embedding models are scored on 130 tasks in 38 languages, grouped into 8 task types such as retrieval, clustering and classification. MTEB community General visual questions Everyday Not in cards Apr 2025
MIMIC-CXR (classification) Models read chest X-rays from Beth Israel Deaconess and say whether five common conditions are present. Scored by average F1 over the conditions. MIT Laboratory for Computational Physiology and Beth Israel Deaconess Medical Center (Johnson et al.) Medicine Expert Not in cards Jan 2019
MIMIC-CXR report generation Models write the findings of a radiology report for a MIMIC-CXR chest X-ray. Scored against the radiologist's report, most often by RadGraph F1. MIT Laboratory for Computational Physiology and Beth Israel Deaconess Medical Center (Johnson et al.) Medicine Expert Not in cards Jan 2019
MIMIC-III Report Summarization Models generate the impression section of a clinical report from the preceding text. The harness compares the response with the recorded impression. EleutherAI lm-evaluation-harness contributors Medicine Expert Not in cards Mar 2025
MIMIC-IV-BHC Models summarize a patient’s hospital course from clinical notes. The reference is the brief hospital course section of the discharge summary. Asad Aali and colleagues Medicine Expert Not in cards Dec 2024
MindGames Agents compete in Colonel Blotto, Iterated Prisoner’s Dilemma, Codenames, and Secret Mafia. MindGames authors Multi-agent games Everyday Not in cards Nov 2026
MInDS-14 Models identify the banking service a caller wants from a spoken request. Daniela Gerz and colleagues Speech recognition and translation Everyday Not in cards Apr 2021
MindWorldBench Models animate a scene to show how a person would act given their beliefs or intentions. Paired prompts change mental states while holding the scene fixed. Peking University and collaborators Video generation Everyday Not in cards Sep 2026
Minecraft Builder Action Prediction Models use dialogue and the current block layout to predict which blocks a Minecraft builder will place or remove. Net action F1 measures agreement with human moves. University of Illinois Urbana-Champaign Embodied reasoning Everyday Not in cards Jul 2020
Ming-Freeform-Audio-Edit Models edit Chinese and English speech from written instructions. The evaluator checks edits to the spoken words and changes to voice characteristics. Inclusion AI, Ant Group Audio understanding and voice University Not in cards Sep 2025
MiniStS Models play card battles with five preset decks. The authors measure final health in four scenarios and how often agents play the Harm card in the fifth. Bahar Bateni and Jim Whitehead Video games Everyday Not in cards May 2024
MiniWoB++ Agents operate over 100 small synthetic web pages, such as forms, menus and date pickers, from a natural-language instruction. Each task returns a success reward. Stanford University, now maintained by the Farama Foundation Web browsing Everyday Not in cards Feb 2018
MIR-1K Models transcribe lyrics from the vocal tracks of karaoke recordings. Qwen compares recognition systems using transcription error. Chao-Ling Hsu and Jyh-Shing Roger Jang Speech recognition and translation Everyday Not in cards Jul 2009
MIREX 2026 Music Evaluation via CMI-RewardBench Models choose which of two generated recordings listeners prefer for a prompt. Organizers rank systems by accuracy on 249 pairs and report vocal and instrumental results separately. Music Information Retrieval Evaluation eXchange (MIREX) Audio understanding and voice Everyday Not in cards
MIREX 2026 Music Performance Difficulty Prediction Models listen to solo piano recordings and rank the pieces by how hard they are to play. Organizers compare the rankings with piano educators’ judgments. Music Information Retrieval Evaluation eXchange (MIREX) Audio understanding and voice University Not in cards
MIT-Adobe FiveK Models adjust photographic tone to match human edits. Five photographers each edited 5,000 raw photos in Lightroom. The original evaluation compares luminance with each reference edit. MIT and Adobe Apps and MCP servers University Not in cards Jun 2011
Mixer 6 Speech Models transcribe conversations recorded across microphones and assign words to speakers. NVIDIA evaluates channel four recordings. Linguistic Data Consortium Speech recognition and translation Everyday Not in cards Aug 2013
MKQA Models answer 10,000 open-domain questions translated into 26 languages. Exact match and token F1 measure agreement with human answers, including unanswerable questions. Apple Knowledge in many languages Everyday Not in cards Jul 2020
ML-Dev-Bench Agents implement and train models, debug failures, and improve performance. Harbor ports 33 development tasks and checks them with the original validators. ML-Dev-Bench team Machine learning engineering Expert Not in cards Jan 2025
MLCommons AILuminate Jailbreak Models answer hazard prompts with and without taxonomy-based jailbreak attacks. The Resilience Gap is the drop in safety score under attack. MLCommons Jailbreaks and prompt injection Everyday Not in cards Dec 2025
MLCR-AA Models reason over synthetic medical claim files of about 25k to 64k tokens, as a claims reviewer would. Three judge models check completeness and accuracy. Wisedocs + Artificial Analysis Medicine Expert Not in cards Aug 2026
MLQA Models extract answers from Wikipedia passages in seven languages, with questions and contexts that can be in different languages. Scored by F1 and exact match. Facebook AI Research Reading in many languages Everyday Not in cards Oct 2019
MM-OCEAN Models rate apparent personality traits in short videos, explain those ratings, and answer questions about observable cues. Evaluation reports rating accuracy, absolute error, cue-question accuracy, and model-judged explanation scores. University of Tokyo, Shanda AI Research Tokyo, and Dalian University of Technology Video understanding Everyday Not in cards May 2026
MM-SafetyBench Multimodal models get 5,040 harmful text and image pairs in 13 scenarios, where a query-relevant image carries the harmful intent. Judges score attack success. East China Normal University and Shanghai AI Laboratory Jailbreaks and prompt injection Everyday Not in cards Nov 2023
MM-ToolSandbox Visual tool-calling agents complete 258 multi-image, multi-turn scenarios across 16 application domains and more than 500 tools. Rubric and entity-diff checks score the runs. Apple General assistants Expert Not in cards Jul 2026
MMAE Audio models edit audio from instructions across a broad range of real-world editing tasks and sound types. MMAE authors Audio understanding and voice Everyday Not in cards Jun 2026
mMARCO Models rank passages by relevance to a query. The multilingual benchmark translates the MS MARCO passage ranking collection. Luiz Henrique Bonifacio and colleagues Reading in many languages School Not in cards Aug 2021
MMDM Decision models assign probabilities to options in 5,448 questions across 16 categories. Hard choices use accuracy or exact set match, while soft targets use Brier error. Chinese University of Hong Kong and collaborators Language reasoning University Not in cards Nov 2026
MMDocIR Models find relevant pages or layout elements inside long documents that contain both text and visual material. Kuicai Dong and colleagues Documents, charts and text in images University Not in cards Jan 2025
MMGenre Models synthesize singing in different genres. The authors measure vocal quality, lyric errors, and how well the singing conveys the intended genre. Wenhao Feng and colleagues Audio understanding and voice University Not in cards Jul 2026
MMIE Models interpret and generate interleaved text and images. MMIE-Score, a fine-tuned InternVL-2-4B model, grades the resulting multimodal responses. UNC Chapel Hill, Microsoft Research, University of Chicago and others General visual questions University Not in cards Oct 2024
MMLU Early Training Models score complete answer texts on selected MMLU questions. The metric compares the correct answer’s log likelihood with the average for incorrect answers. EleutherAI lm-evaluation-harness contributors Exam questions University Not in cards Dec 2025
MMLU-Redux 2.0 Spanish Models answer corrected MMLU questions in Spanish across 57 subjects. The evaluation compares each selected option with the answer key. amias-mx (Spanish dataset) Knowledge in many languages University Not in cards Feb 2025
MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. OpenCompass Knowledge in many languages University Not in cards Oct 2024
Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. University of Illinois Urbana-Champaign and Alibaba Phones Everyday Not in cards Jan 2025
Mobile-Eval-v2 Agents follow basic and advanced instructions on real phones. The authors give 44 instructions in English on Android and 44 in Chinese on HarmonyOS. Beijing Jiaotong University and Alibaba Phones Everyday Not in cards Jun 2024
Model Spec Evals Models answer short conversations that test adherence to the OpenAI Model Spec. A grader scores each response against a prompt-specific rubric. OpenAI Deception and alignment Everyday Not in cards Mar 2026
Mol-Instructions Chemical Entity Recognition Models identify chemical entities in biomedical text. The score is test-set F1. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Not in cards Jun 2023
Mol-Instructions Chemical-disease Interaction Extraction Models extract chemical and disease interactions from text. The score is test-set F1. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Not in cards Jun 2023
Mol-Instructions Chemical-protein Interaction Extraction Models extract chemical and protein interactions from text. The score is test-set F1. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Not in cards Jun 2023
Mol-Instructions Description-guided Molecule Design Models generate molecules from descriptions. The score is exact-match accuracy on the molecule design test set. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Not in cards Jun 2023
Mol-Instructions Forward Reaction Prediction Models predict reaction products from reactants. The score is exact-match accuracy on the forward reaction test set. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Not in cards Jun 2023
Mol-Instructions Multi-choice Models answer biomedical multiple-choice questions. The score is test-set accuracy. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Not in cards Jun 2023
Mol-Instructions Reagent Prediction Models predict reaction reagents. The score is exact-match accuracy on the reagent prediction test set. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Not in cards Jun 2023
Mol-Instructions Retrosynthesis Models predict reactants for target products. The score is exact-match accuracy on the retrosynthesis test set. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Not in cards Jun 2023
Mol-Instructions True or False Models answer biomedical true or false questions. The score is test-set accuracy. Zhejiang University (ZJUNLP) Genomics and life sciences Expert Not in cards Jun 2023
MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. Ai2 Simulated robots Everyday Not in cards Jun 2026
Moneda AdviceBench Models answer personal finance requests and paired education questions. Moneda’s default board scores paired pass rate on 50 EU personal requests with education twins. Moneda Finance Expert Not in cards Nov 2026
Moneda Bench Models answer finance questions across seven separate benchmarks. AdviceBench, CiteBench, FilingsBench, GroundBench and PortfolioBench have their own records. KnowledgeBench and MarketBench have no published results yet. Moneda publishes no combined score. Moneda Finance Expert Not in cards Nov 2026
Moneda CiteBench Models write company research notes with linked sources. Moneda scores the weighted share of linked claims that the cited pages support across 100 notes. Moneda Finance Expert Not in cards Nov 2026
Moneda FilingsBench Models answer questions about SEC filings without receiving documents. Moneda scores accuracy on the private questions using filing data and judges for segment questions. Moneda Finance Expert Not in cards Nov 2026
Moneda GroundBench Models answer live questions about market figures. Moneda’s default board reports the share of correct answers against reference data, with incorrect answers reported separately. Moneda Finance Expert Not in cards Nov 2026
Moneda PortfolioBench Models solve hypothetical portfolio tasks involving rebalancing, risk, position sizing, and constraints. Moneda’s default board scores accuracy on 150 tasks with fixed answers. Moneda Finance Expert Not in cards Nov 2026
Monsoon ASR Models transcribe Indian English and Hindi recordings. The Hindi evaluation accounts for alternative spellings in its word error rate. Voice Arena Speech recognition and translation Everyday Not in cards Jul 2026
Moral Reasoning under Uncertainty Models explain ethical decisions involving humans, unfamiliar organisms and digital minds. A judge evaluates how their answers handle uncertainty. Compassion Aligned Machine Learning Deception and alignment Everyday Not in cards Jan 2026
Moral Stories Models reason about norms, intentions, actions and consequences in short stories. Tasks include distinguishing a norm-respecting action from a norm-violating alternative. Denis Emelin and colleagues Commonsense Everyday Not in cards Nov 2021
Mosaic Eval Gauntlet Original Models complete tasks in six capability groups. MosaicML compared the MPT models across these groups. MosaicML Language reasoning University Not in cards Jun 2023
MOSEv2 Models track and segment objects in videos with complex scenes. Objects may disappear, reappear, or blend into a crowd. Henghui Ding and colleagues Video understanding Everyday Not in cards Aug 2025
Movie Gen Video Bench Video generation models make clips from a fixed prompt set, and people compare them. Meta released non-cherry-picked Movie Gen videos for comparison. Meta Video generation Everyday Not in cards Oct 2024
MP-DocVQA Models answer questions about scanned documents of up to 20 pages, where the answer sits on one page. Scored by ANLS against the reference answer. Computer Vision Center, Universitat Autonoma de Barcelona (Tito, Karatzas, Valveny) Documents, charts and text in images Everyday Not in cards Dec 2022
MQuAKE Models answer linked factual questions after one or more facts change. An edit succeeds if the model gives the updated answer to any of three phrasings. Princeton University and Stanford University Language reasoning Everyday Not in cards May 2023
MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. Google Medicine Expert Not in cards Jan 2026
mRobust04 Models rank documents for translated queries from the TREC Robust04 collection. Relevance judgments support comparisons across languages. Vitor Jeronymo and colleagues Reading in many languages School Not in cards Sep 2022
MS MARCO Question Answering Models answer real Bing queries using retrieved web passages. BLEU and ROUGE-L compare generated answers with human-written references. Microsoft Reading comprehension Everyday Not in cards Nov 2016
MS-CXR-T Models compare a current and a prior chest X-ray of one patient and say whether a finding improved, stayed stable or got worse. Microsoft Health Futures (Bannur et al.) Medicine Expert Not in cards Jan 2023
MS² Models synthesize the findings of multiple medical studies into a short answer to the research question of a systematic review. Northeastern University and Allen Institute for AI Medicine Expert Not in cards Apr 2021
MSD-I Multimodal Genre Classification Models predict a track’s genre from its audio and album cover. The published test contains 4,649 tracks across 15 genres. Universitat Pompeu Fabra and collaborators Audio understanding and voice Everyday Not in cards Apr 2018
MSR-VTT Models write captions for short web videos. Human-written sentences provide multiple descriptions of each clip. Jun Xu, Tao Mei, Ting Yao, and Yong Rui Video captioning Everyday Not in cards Jun 2016
MSRVTT-QA Models answer short open-ended questions about web video clips from the MSR-VTT captioning dataset. The question and answer pairs were generated automatically from the clip descriptions. Scored by answer accuracy. Zhejiang University and collaborators (Xu et al.) Video understanding Everyday Not in cards Oct 2017
MSVD Models retrieve short videos from human descriptions and retrieve descriptions from videos. Meta reports retrieval accuracy in both directions. David Chen and William Dolan Video understanding Everyday Not in cards
MTG-Jamendo Models tag songs by genre, mood, theme, and instrument. Qwen reports micro F1 for each tagging task. Music Technology Group at Universitat Pompeu Fabra Audio understanding and voice Everyday Not in cards
MTGBench Models play multi-turn games that test reasoning, planning, and interaction across varied game types. MTGBench authors Multi-agent games Everyday Not in cards Nov 2026
MTOP Models parse a user's request into its intent and the details needed to act on it. Facebook AI Reading in many languages School Not in cards Aug 2020
MTS-Dialog Models write a clinical note section from a conversation between a doctor and a patient. The evaluation compares it with the reference note. Asma Ben Abacha and colleagues Medicine Expert Not in cards May 2023
MTSamples clinical entity recognition Models mark medical problems, treatments, and tests in 163 synthetic discharge summaries. Researchers score exact and overlapping text spans separately. McWilliams School of Biomedical Informatics and collaborators Medicine University Not in cards Aug 2023
MUCS 2021 Models transcribe speech in Indian languages and mixed-language recordings. NVIDIA reports results on Bengali-English and Hindi-English speech. MUCS challenge organizers Speech recognition and translation Everyday Not in cards Apr 2021
Multi-LexSum Models summarize documents from civil rights lawsuits at different lengths. Legal experts wrote the reference summaries. Allen Institute for AI and collaborators Law Expert Not in cards Jun 2022
Multi-Scenario Anomaly Detection Models score video frames for unusual events across 14 settings. The two protocols hold out different videos and score frame-level detection. Liyun Zhu and colleagues Video understanding Everyday Not in cards Jun 2024
Multi-Turn Human Jailbreaks Models respond to multi-turn conversations collected in human safety evaluations. The authors report how often model safeguards fail. Scale AI and collaborators Jailbreaks and prompt injection Expert Not in cards Aug 2024
Multi-Turn Sudoku (m8ngotree) Models fill a nine-by-nine Sudoku board one move at a time. The generator checks for a unique solution, and the scorer rewards valid moves and completion. m8ngotree Puzzles and chess School Not in cards Sep 2025
MultiCardioNER Models mark disease mentions in Spanish cardiology cases and medication mentions in Spanish, English, and Italian versions of the cases. Organizers score exact spans with microaveraged F1. Barcelona Supercomputing Center and BioASQ Medicine University Not in cards Apr 2024
MultiClinAI Models identify diseases, symptoms, and procedures in seven languages. A second task transfers annotated Spanish mentions to the corresponding spans in six translations. Barcelona Supercomputing Center Medicine University Not in cards Nov 2025
MultiCochrane Models rewrite medical review sentences in plain language. The collection pairs technical and simplified wording in English, Spanish, French, and Farsi. University of Texas at Austin, Georgia Institute of Technology, and Northeastern University Medicine University Not in cards May 2023
Multilingual Conversational Speech Language Model Challenge Models recognize conversational speech across languages. Qwen compares transcription error and language identification accuracy on the challenge recordings. MLC-SLM organizers Speech recognition and translation Everyday Not in cards
Multilingual Entity Linking of Occupations Models map occupational titles from national classifications to European occupation codes. The benchmark scores the rank of the first correct match. Avature Other professions University Not in cards Oct 2024
Multilingual Entity Linking of Skills Models match skill names from Belgian, German, and Swedish classifications to European skill entries. Official mappings identify the correct matches. Avature Other professions University Not in cards Jan 2026
Multilingual HateCheck Models distinguish hate speech from statements that quote, reject, or otherwise resemble hate speech. The test covers multiple languages. Paul Röttger and colleagues Bias, fairness and politics Everyday Not in cards Jul 2022
MultiNERD Models locate fine-grained named entities in Wikipedia and Wikinews sentences across ten languages. Simone Tedeschi and Roberto Navigli Reading in many languages School Not in cards Jul 2022
MultiRef-Compass Models generate audio-video content from several visual references and text instructions across 350 samples. Fourteen metrics assess quality, reference fidelity, audio-visual consistency, and instruction following. The evaluator combines automatic measurements with multimodal model judgments. Nanjing University, Kling, National University of Singapore and others Video generation Everyday Not in cards Jul 2026
MuMiN Models use social media context to classify claims as factual or misleading. A second task asks which verdict applies to the claim a post discusses. Dan Saattrup Nielsen and Ryan McConville Faithful to sources University Not in cards Feb 2022
MUSDB-ALT Models transcribe 39 English songs from the MUSDB18 test set. Line timings and separate vocal recordings let researchers compare transcription with and without the accompaniment. Queen Mary University of London and AudioShake Speech recognition and translation Everyday Not in cards Apr 2025
MUSDB18 Models separate a music mixture into vocals, drums, bass, and accompaniment. The dataset supplies isolated studio recordings for comparison. MUSDB18 team Audio understanding and voice Everyday Not in cards
MUSEBENCH Models detect computer-generated music and identify artifacts. The authors compare eleven detectors on original recordings and on transformed tracks. Yi Chen Liu and colleagues Audio understanding and voice University Not in cards Oct 2026
Music Understanding Benchmark Models estimate tempo, key, meter, and song structure from recordings with identifying metadata removed. The evaluator compares predictions with existing annotations. Kun Liu and colleagues Audio understanding and voice University Not in cards Oct 2026
MUSIC-21 Instrument Separation Models separate performances from a mixture using video of the musicians. When two musicians play the same instrument, their movements help identify which sound belongs to each. MIT and MIT-IBM Watson AI Lab Audio understanding and voice University Not in cards Apr 2019
Musical Expectation Benchmark Models predict how surprising a musical passage will sound. The benchmark compares predictions from scores or recordings with perceptual measurements. Mathias Rose Bjare and colleagues Audio understanding and voice University Not in cards Nov 2026
MusicEval Models predict how experts rate generated music for musical quality and agreement with a text prompt. The public test contains 413 clips. Nankai University and Beijing AISHELL Technology Audio understanding and voice Everyday Not in cards Jan 2025
MUStARD Models determine whether a speaker is being sarcastic in a television clip. HELM evaluates the audio track. Santiago Castro and colleagues Audio understanding and voice Everyday Not in cards Jun 2019
MuTox Models listen to short speech clips in 30 languages and classify whether they contain toxic speech. The harness compares their decisions with human labels. Marta R. Costa-jussà and colleagues Audio understanding and voice Everyday Not in cards Jan 2024
MuTual Models read a multi-turn dialogue and select the response that follows from the speakers’ exchange. The alternatives require reasoning about the conversation. Leyang Cui and colleagues Multi-turn conversation Everyday Not in cards Jul 2020
MyCareersFuture Skill Retrieval Models rank required skills from job descriptions on Singapore’s government employment portal. Employer tags supply the answers, including skills the description leaves implicit. National University of Singapore, IIT Kanpur, and Zalando Other professions University Not in cards Dec 2020
MyClaw Bench Models run inside the OpenClaw agent on 45 tasks in four difficulty tiers built from real usage data. Some tasks use an LLM judge. MyClaw.ai General assistants Everyday Not in cards Mar 2026
Mythos Models write a new story in an author’s style using earlier stories as examples. Judges compare the result with the author’s writing. University of Massachusetts Amherst and University of Maryland, College Park Creative writing School Not in cards Feb 2025
N-1 Contingency Dispatch Models schedule electricity generation to meet demand while tolerating a single transmission line outage. The grader checks feasibility and cost. jwilksbooth Electronics Expert Not in cards Jul 2026
n2c2 2018 Adverse Drug Events and Medication Extraction Models extract drugs and adverse events from discharge summaries. They also identify which drug each event or medication detail belongs to. National NLP Clinical Challenges Medicine University Not in cards Jan 2018
n2c2 2022 Contextualized Medication Event Extraction Models identify medication mentions and determine whether a clinician discusses a change. They classify each change by action, negation, timing, certainty, and who acted. National NLP Clinical Challenges and the CMED organizers Medicine University Not in cards Nov 2021
n2c2 2022 Progress Note Understanding: Assessment and Plan Reasoning Models read a clinician's assessment and one part of the treatment plan. They choose one of four relationships. Organizers score the labels with macro F1. National NLP Clinical Challenges and the University of Wisconsin team Medicine University Not in cards Nov 2021
NaijaRC Models answer multiple-choice reading comprehension questions from high school exams in three Nigerian languages. Anuoluwapo Aremu and colleagues Reading in many languages School Not in cards Aug 2023
Nakdimon Modern Hebrew Test Set Models restore diacritics in modern Hebrew from 11 sources, including news and user writing. The authors report accuracy for complete words and individual characters. Elazar Gershuni and Yuval Pinter Reading in many languages University Not in cards May 2021
NanoGPT Expenditure Horizon Agents optimize NanoGPT training. METR estimates the budget at which human contributors become more cost effective than the agents. METR Task length Expert Not in cards Jul 2026
NAPISUL-II Progress Test 2023 Models answer Portuguese multiple-choice questions from the October 2023 medical progress test in southern Brazil. Researchers compare their answers with the official key. NAPISUL-II, Brazilian Association of Medical Education Medicine University Not in cards Oct 2023
NarrativeTrack Video-language models answer entity-centric questions about existence, changes and ambiguity as narratives unfold across clips. Apple Video understanding Expert Not in cards Jan 2026
NASA SDE IR Benchmark v5 Models retrieve NASA science documents from questions and search phrases. The authors generated 176,901 queries from Science Discovery Engine content. NASA and IBM Research Questions that need search University Not in cards Nov 2025
NASA-IR Models retrieve the scientific passage that answers a question. The test measures how often the correct passage appears among the first ten results. NASA and IBM Research Questions that need search University Not in cards Apr 2024
NASA-QA Models read passages from Earth science papers and extract the words that answer a question. Some questions have no answer in the passage. NASA and IBM Research Reading comprehension University Not in cards Apr 2024
Natural Adversarial Objects Models identify objects in unmodified photographs that caused errors in earlier detectors. Human annotations identify the objects. Felix Lau and colleagues with Scale AI Detection and captioning Everyday Not in cards Nov 2021
NATURAL PLAN Models make travel itineraries, find meeting arrangements, and schedule calendar events. The evaluator checks each proposed plan against the stated constraints. Google DeepMind General assistants Everyday Not in cards Jun 2024
Natural Questions in Icelandic Models extract answers from Icelandic Wikipedia articles. Annotators write questions after seeing only the beginning of an article. Vésteinn Snæbjarnarson and Hafsteinn Einarsson Reading in many languages School Not in cards Jun 2022
Natural World Tasks (NeWT 2021) Models distinguish pairs of labels in wildlife photographs. The 164 tasks include recognizing life stages, animal behavior and scene context. Cornell University, Caltech, Google and the University of Edinburgh General visual questions University Not in cards Mar 2021
NaturalProofs Models retrieve or generate references needed for a mathematical theorem's proof. Ranking and sequence metrics compare predicted references with those used in the source proof. University of Washington, Allen Institute for AI, New York University and others Written proofs University Not in cards Apr 2021
NAVI Models match points on the same object across photographs taken from different viewpoints. Reference geometry supplies the correct correspondences. Varun Jampani and colleagues Spatial reasoning and counting School Not in cards Jun 2023
NB Tale Module 1 Models transcribe Norwegian newspaper sentences read aloud by speakers from different dialect regions. Lingit AS and NTNU for the National Library of Norway Speech recognition and translation Everyday Not in cards Nov 2013
NB Tale Module 3 Models transcribe spontaneous Norwegian speech from speakers with different dialects. The reference transcripts use Bokmål. Lingit AS and NTNU for the National Library of Norway Speech recognition and translation Everyday Not in cards Dec 2015
NBABench (Thierry Damiba) Models search NBA player statistics from two seasons and answer 102 generated questions. The reference answers come directly from player records. Thierry Damiba Questions that need search Everyday Not in cards Jan 2026
NCHLT isiZulu Speech Corpus Models transcribe short phrases that native isiZulu speakers read aloud. The archive identifies eight test speakers. Meraka Institute, CSIR and North-West University Speech recognition and translation Everyday Not in cards May 2014
Nego Eval Agents choose negotiation partners and remember prior encounters across games. The evaluator compares match scores with reference policies on the same seeds. hanjinkim Multi-agent games Everyday Not in cards Sep 2026
NEPABench Models process public comments, extract information from federal documents and answer questions about environmental reviews. Each task has its own scoring rules. Pacific Northwest National Laboratory Law University Not in cards Apr 2026
NEREL-bench Models extract entities and relations from Russian texts and write definitions for them. The evaluator scores extraction and definition quality separately. Ivan Bondarenko Reading in many languages University Not in cards Mar 2026
Neuro Reasoning (dwin2) Models answer generated questions about EEG experiment design and preprocessing. The evaluator computes a reference answer for each question. dwin2 Science questions University Not in cards Sep 2026
NeuroDiscoveryBench Models analyze neuroscience, genetics and immunology datasets and answer questions grounded in figures and data workflows. Ai2 and Allen Institute Research science Expert Not in cards Nov 2025
NevIR Models distinguish documents whose meanings differ because one passage negates a statement. Orion Weller and colleagues Questions that need search School Not in cards May 2023
NEVIS’22 Models learn 106 image classification tasks in chronological order. The evaluation compares error rates against the computation each model uses. Google DeepMind General visual questions Everyday Not in cards Nov 2022
NewsGuard false audio claims audit Voice assistants answer questions about 20 false news claims under three prompt styles. NewsGuard counts how often each assistant repeats a false claim aloud. NewsGuard Long-form factuality Everyday Not in cards Feb 2026
NewsQA Models extract an answer from a CNN news article in response to a question written by a reader. Adam Trischler and colleagues Reading comprehension School Not in cards Nov 2016
NIA Sovereign AI Benchmark Evaluation, Phase 2 Models answer questions across seven areas, including Korean language and instruction following. The agency combines eight weighted component scores for a maximum of 15 points. National Information Society Agency of Korea Composite indices University Not in cards Aug 2026
NICTA-PIBOSO Models assign one or more labels to each sentence in a medical abstract. The six labels include population, intervention, and outcome. NICTA and the University of Melbourne Medicine University Not in cards Mar 2011
NIFTY Market Movement Models use financial news and market history to predict whether the market will rise, fall or remain neutral. NIFTY authors Finance University Not in cards May 2024
Nigerian Pidgin ASR Models transcribe Nigerian Pidgin sentences read by native speakers. The published test split contains 892 recordings. Nigerian Pidgin Project Speech recognition and translation Everyday Not in cards Oct 2020
NIILC Question Answering Dataset Models answer Japanese questions. The dataset includes answer types, clues, and links to relevant Wikipedia pages. NIILC Question Answering Dataset project Knowledge in many languages School Not in cards Oct 2015
Nim compiler repair benchmark Agents repair the Nim compiler or standard library from linked issues. The current README lists 285 tasks graded with compile and regression tests. josancamon19 Fixing real repositories Expert Not in cards Nov 2026
Nim Hard Agents fix Nim compiler issues on Linux. Black-box tests grade changes to the C backend. The current README lists 15 tasks. josancamon19 Fixing real repositories Expert Not in cards Nov 2026
NLEBench Models generate Norwegian text for tasks including reading comprehension and summarization. The benchmark combines translated and human-annotated material. Peng Liu and colleagues Reading in many languages School Not in cards Dec 2023
NLPCC 2024 Chinese Essay Discourse Logic Evaluation and Integration Models locate logical errors, identify paragraph topics and their relationships, and write feedback on Chinese school essays. The organizers average three track scores. East China Normal University and Microsoft Research Asia Reading in many languages University Not in cards Mar 2024
NLVR2 Models judge whether a sentence is true of a pair of photographs. The benchmark measures agreement with annotated truth values. NLVR2 team General visual questions Everyday Not in cards Nov 2018
nocaps Models caption photographs that contain objects outside the caption training vocabulary. The benchmark compares generated captions with human references. Harsh Agrawal and colleagues Detection and captioning Everyday Not in cards Dec 2018
Nodal Pricing LMP Models calculate locational marginal electricity prices for congested transmission networks. The grader checks their answers against a power flow optimization model. jwilksbooth Electronics Expert Not in cards Jul 2026
NollySenti Models classify the sentiment of Nollywood movie reviews in five languages spoken in Nigeria. Iyanuoluwa Shode and colleagues Reading in many languages Everyday Not in cards May 2023
Non-native Children's Automatic Speech Assessment Challenge Models rate children's Norwegian word pronunciations on a five-point scale. The challenge measures recall averaged across the five ratings. Teflon Project at Aalto University and the Norwegian University of Science and Technology Speech recognition and translation University Not in cards Oct 2024
Noor Models answer filtered academic questions chosen to distinguish early training checkpoints. The benchmark measures progress where standard MMLU scores are noisy. Mohammed Dahbani and Anas Ezzakri Exam questions University Not in cards Nov 2025
NorQuAD Models extract answers to Norwegian questions from Wikipedia articles and news passages. University of Oslo Reading in many languages School Not in cards May 2023
NorSID Models identify a request’s intent, tag words such as places or times, and classify the written Norwegian variety. Each task has its own score. University of Oslo and IT University of Copenhagen Reading in many languages Everyday Not in cards Oct 2024
Norwegian Comma Benchmark Models choose which of two Norwegian sentences uses commas correctly. Each pair tests one punctuation rule. Hans Christian Farsethås and Joakim Tjøstheim Reading in many languages School Not in cards May 2024
Norwegian Parliamentary Speech Corpus Models transcribe speeches from the Norwegian parliament. The corpus includes manual transcripts in Bokmål and Nynorsk. Norwegian Language Bank, National Library of Norway Speech recognition and translation Everyday Not in cards Oct 2020
Norwegian Review Corpus Models predict sentiment from Norwegian reviews. The NorEval document task groups the original ratings into two or three classes. University of Oslo Reading in many languages Everyday Not in cards May 2018
NOTSOFAR-1 Models transcribe distant meetings and assign speech to speakers. NVIDIA also measures diarization error on the evaluation recordings. Microsoft Audio understanding and voice Everyday Not in cards Jan 2024
Novelty Judge Bench Models judge whether generated and human-authored research ideas are novel using pairwise and pointwise comparisons. Ai2 AI research tasks Expert Not in cards Oct 2026
NTCIR-13 MedWeb Models label each short message for eight symptoms. The organizers score an exact match when the model gets all eight labels right. NTCIR MedWeb organizers Medicine Everyday Not in cards Aug 2016
NTCIR-17 Social Media Adverse Drug Event Detection Models identify drug side effects in simulated social media posts across four languages. Exact match requires all 22 symptom labels to match. NTCIR MedNLP-SC organizers Medicine University Not in cards Sep 2022
NTCIR-18 HIDDEN-RAD Models explain chest radiograph findings from reports or structured observations. The final score combines text similarity, language model judgments, and expert review. Konyang University Medicine Expert Not in cards Mar 2024
NTCIR-18 MedNLP-CHAT Models flag medical, legal, and ethical risks in answers to patient questions. The organizers compare the labels with expert judgments using macro F1. NTCIR MedNLP-CHAT organizers Medicine Expert Not in cards Mar 2024
NTCIR-19 MedNLP-CALL Triage Classification Models classify synthetic emergency calls into three urgency levels from dispatcher and caller dialogue. Medical experts supplied the reference labels for the six-language task. Nara Institute of Science and Technology and collaborators Medicine University Not in cards Jun 2025
Nufo Models transcribe Ewe speech from a fixed test split. The authors compare transcriptions with reference text using word error rate. Christ Kekeli Keli and colleagues Speech recognition and translation Everyday Not in cards Oct 2026
Numb3rs Models transcribe spoken numbers, including addresses, amounts, and times. NVIDIA compares the output with spoken-form references. NVIDIA Speech recognition and translation Everyday Not in cards
NusaWrites Models interpret and translate texts in underrepresented Indonesian languages. Google reports emotion classification on original paragraphs and retrieval of matching translations. NusaWrites team Reading in many languages School Not in cards Sep 2023
NusaX Models predict sentiment in parallel reviews written in Indonesian local languages, Indonesian, and English. IndoNLP Reading in many languages Everyday Not in cards May 2022
nuScenes Detection Models detect 3D objects in nuScenes sensor data. The official test leaderboard ranks submissions by nuScenes Detection Score. Motional (nuTonomy) Detection and captioning Everyday Not in cards Mar 2019
NWPU-Captions Models describe aerial scenes across 45 land use categories. The authors reserve 3,150 images for testing and provide five captions per image. Qimin Cheng, Haiyan Huang, Yuan Xu, Yuzhuo Zhou, Huanying Li, and Zhongyuan Wang Detection and captioning School Not in cards Apr 2022
NYT Connections (Prime Community) Models identify four groups of related words in New York Times Connections puzzles. The default 100-puzzle evaluation scores completion, efficiency, partial progress, and format. Prime Community Puzzles and chess Everyday Not in cards Mar 2026
NYU Depth V2 Models estimate depth from indoor photographs. The benchmark compares predictions with depth measurements captured by a sensor. New York University Spatial reasoning and counting School Not in cards
O-NET M6 Models answer Thai grade 12 examination questions. The evaluation includes Thai and English questions across school subjects. OpenThaiGPT and the National Institute of Educational Testing Service of Thailand Exam questions School Not in cards Sep 2024
OAB Exams Models answer Portuguese multiple-choice questions from the Brazilian bar examination. Eduardo Garcia and contributors Law University Not in cards Jan 2024
Object Detection Leaderboard Models locate and classify objects in images. The board compares average precision. Hugging Face Detection and captioning Everyday Not in cards Sep 2023
ObjectNet Models identify familiar objects photographed with controlled changes in viewpoint, orientation, and background. Meta reports results on classes shared with ImageNet. Andrei Barbu and colleagues General visual questions Everyday Not in cards
OCBench 28 simulated manipulation tasks with human-like scripted demonstrators for controlled BC and RL studies. Policies scored by task success rate. UC Berkeley (Seohong Park, Sergey Levine) Simulated robots Everyday Not in cards Oct 2026
Occluded Video Instance Segmentation Models track and segment objects that overlap or disappear behind other objects. The evaluation compares predicted masks and identities with annotated video instances. OVIS team Video understanding Everyday Not in cards Feb 2021
OCNLI Models decide whether a Chinese hypothesis follows from, contradicts, or remains neutral toward a premise. Accuracy measures agreement with the majority annotation. Indiana University, Allen Institute for AI and Chinese Language Understanding Evaluation (CLUE) Reasoning in many languages Everyday Not in cards Oct 2020
OHSUMED Systems rank medical articles for 106 questions from physicians. The original study measured precision and recall against physicians' judgments of which articles were relevant. Oregon Health Sciences University and Cornell University Medicine Expert Not in cards Jul 1994
Okapi ARC Models choose answers to translated science questions from ARC-Challenge. University of Oregon NLP Group Science questions School Not in cards Jul 2023
Okapi HellaSwag Models choose the most plausible next event in a translated description of an everyday activity. University of Oregon NLP Group Reasoning in many languages Everyday Not in cards Jul 2023
Okapi TruthfulQA Models distinguish true answers from common misconceptions in translated multiple-choice questions. University of Oregon NLP Group Short factual questions Everyday Not in cards Jul 2023
OmniACT Models generate executable scripts from a screenshot and a user request. The evaluation compares the actions needed to complete desktop and web tasks. OmniACT team and Writer Desktop apps University Not in cards Feb 2024
OmniLabel Models locate objects described by category names and free-form phrases. The benchmark compares predicted boxes with annotated objects. OmniLabel team Detection and captioning Everyday Not in cards Apr 2023
Omnilingua-Bench Models handle long-audio transcription, multilingual long-audio question answering, multi-speaker attribution and complex spoken instructions. WER, diarization, accuracy and rubric pass rates score the tracks. Qwen Team, Alibaba Group Speech recognition and translation University Not in cards Sep 2026
Omnilingual ASR Corpus Models transcribe recordings from speakers of many languages. Meta compares character error rates on held-out speech, including languages with little existing transcription data. Meta Speech recognition and translation School Not in cards Nov 2025
On Evaluating User Simulators for Interactive Agents Models play the user in replays of 351 real sessions. The evaluation compares simulated behavior with observed behavior and measures agreement with human rankings of nine agents. Arena and UCLA Multi-turn conversation Everyday Not in cards Nov 2026
OOD-Speech Models transcribe Bengali recordings from everyday media, including parliamentary sessions, online classes and poetry readings. The authors measure word and character error separately for each domain. Bengali.AI Speech recognition and translation Everyday Not in cards May 2023
Open Prompt Injection Models perform seven language tasks with conflicting instructions embedded in their inputs. The port measures whether models follow the original task and separately tracks accuracy. hubert-marek Jailbreaks and prompt injection University Not in cards Oct 2023
Open RAG Benchmark RAG systems answer 3,045 queries over 1,000 arXiv PDFs with text, tables, images, and hard negatives. Vectara Questions that need search Expert Not in cards Apr 2025
OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. OpenAI Deception and alignment Not in cards
OpenAI GPT-Red jailbreak and prompt-injection evals Static jailbreaks from attacker models (half from a held-out attacker), adaptive multi-turn attackers, and direct and indirect prompt-injection sets. OpenAI Jailbreaks and prompt injection Not in cards
OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). OpenAI Labs' internal R&D tests Expert Not in cards
OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). OpenAI Finding and exploiting real bugs Expert Not in cards
OpenCanopy Models estimate tree canopy height from satellite images. The evaluation compares predicted height maps with measurements from airborne laser scans. OpenCanopy team Spatial reasoning and counting University Not in cards Jul 2024
Opencpop Models transcribe lyrics from Mandarin singing. Qwen uses the test recordings to compare speech recognition systems. Opencpop team Speech recognition and translation Everyday Not in cards Jan 2022
OpenFARM Bioacoustics Models classify animal vocalizations across eight tasks, including pig call type and recording context. Each task uses labels from its source dataset. OpenFARM Audio understanding and voice Expert Not in cards May 2026
OpenFinData Models answer Chinese financial questions, extract entities and values, and check calculations or data consistency. OpenCompass scores answer accuracy, using required-keyword matching for entity extraction. East Money and Shanghai Artificial Intelligence Laboratory Finance University Not in cards Jan 2024
OpenHuEval Models answer Hungarian questions, interpret proverbs, and fill gaps in texts across five tasks. Factual questions and open responses use model judges, while matching and blank completion also use rules and text similarity. Shanghai Artificial Intelligence Laboratory, Hungarian Research Centre for Linguistics, and university collaborators Reasoning in many languages University Not in cards Mar 2025
OpenLex3D Models label points and retrieve objects in indoor 3D scans. Annotators distinguish an object’s synonyms from pictures on its surface and visually similar objects. University of Oxford, Université de Montréal, University of Freiburg, and Mila Spatial reasoning and counting Everyday Not in cards Mar 2025
OpenMed MedReason Evaluation Models answer medical questions from a held-out portion of the MedReason training corpus. The port scores answer accuracy, reasoning, and format separately. OpenMed Medicine Expert Not in cards Jan 2026
OpenRSI Index Agents run autoresearch on real open-source model development projects with their original code, data and checkpoints, for 60 or more hours. Fixed verifiers score the result against the human recipe they started from. OpenRSI Foundation (UW-led, academic and industry partners) AI research tasks Research frontier Not in cards Sep 2026
OpenSUN3D Open-Vocabulary 3D Object Instance Search Models read a description and mark every matching object in a scanned room. The organizers rank predictions by mean average precision across 25 test scenes. OpenSUN3D workshop organizers Spatial reasoning and counting Everyday Not in cards Jul 2023
OpenTTD-Bench Agents build transport networks over ten simulated years. The grader rewards cumulative operating profit and the number of rivals each company beats. Sophia Xu Video games Expert Not in cards Oct 2026
OphthoQuestions ophthalmology board preparation Models answer multiple-choice ophthalmology questions from a subscription question bank. Each model selects one answer per question. OphthoQuestions Medicine Expert Not in cards Jan 2023
OPT-RSVG Models use a written description to locate an object in an aerial image. The test pairs 24,477 descriptions with reference boxes. Ke Li and colleagues Detection and captioning Everyday Not in cards Jul 2023
OPUS-100 Models translate between English and other languages. Separate test sets measure performance on supervised directions and language pairs without direct training data. Biao Zhang and colleagues Translation University Not in cards Jul 2020
Opusparcus Models identify paraphrases among movie and television subtitle sentences in six European languages. Mathias Creutz Reading in many languages School Not in cards May 2018
OSGym Infrastructure running 1,000+ parallel OS sandboxes for computer-use agent training and evaluation. Tasks span browsing, documents, SWE and multi-app work. Zengyi Qin et al. (agiopen-org) Desktop apps Everyday Not in cards Nov 2025
OSWorkerBench Desktop agents complete long-horizon office tasks across business applications with executable evaluators and optional demonstrations. Tencent Desktop apps Expert Not in cards Aug 2026
OTBench Tabular models generate missing data, impute cells, and predict targets under shared evaluation protocols. The suite compares 46 methods over 56 datasets and 20 evaluation axes. Yuanrui Wang, Kaifang Li, Xuanyue Li, Ziyun Li, Yan Lu, Xingxuan Zhang, and Peng Cui Data science and SQL University Not in cards Nov 2026
OutboundEval Agents conduct outbound calls with simulated users. The benchmark scores compliance with the professional workflow and the quality of the interaction. Meituan with xbench and Agora Multi-turn conversation Expert Not in cards Oct 2025
OVEN Models identify the Wikipedia entity that answers a question about an image. Hexiang Hu and colleagues General visual questions University Not in cards Feb 2023
OverSearchQA Search-augmented models decide when to abstain from retrieval on unknown, false-premise and underspecified questions. Apple Questions that need search University Not in cards Jan 2026
Òwe-Voice Models transcribe spoken Yorùbá proverbs. The evaluation measures errors in words and tone marks. Daud Abolade, University of Lagos Speech recognition and translation Everyday Not in cards Mar 2026
Oxford Flowers 102 Models identify flower categories from photographs. The benchmark tests recognition across changes in shape, scale, and lighting. University of Oxford General visual questions University Not in cards Dec 2008
Oxford-IIIT Pet Dataset Models identify the breed of a cat or dog from a photograph. Meta measures breed classification across 37 categories. University of Oxford and IIIT Hyderabad General visual questions School Not in cards
PACT (robot arm orchestration) Agents write and revise programs to control a simulated robot arm. The simulator checks whether the arm completes the requested manipulation. Aryan Madhav Verma Simulated robots Expert Not in cards Sep 2026
PaintSkills Models generate images of specified objects, counts, and spatial relationships. Detectors check whether an image satisfies the requested composition. Jaemin Cho and colleagues Image generation and editing Everyday Not in cards Feb 2022
Paired-symmetry suite Models respond to paired political prompts. Writer compares refusal behavior across the paired requests. Writer Bias, fairness and politics Everyday Not in cards Aug 2026
PalmX 2025 General Culture Models choose answers about customs, history and daily life across Arab countries. The test contains 2,000 questions in Modern Standard Arabic. University of British Columbia and Qatar Computing Research Institute Knowledge in many languages School Not in cards Jun 2025
PalmX 2025 Islamic Culture Models choose answers about Islamic practices, scripture and history in Modern Standard Arabic. The organizers score exact answer choices. University of British Columbia and Qatar Computing Research Institute Knowledge in many languages School Not in cards Jun 2025
Palmyra X6 Enterprise Workflow Evaluations Agents perform production workflows that require retrieval, tool use, content generation, or delegation. Writer compares capability and cost. Writer Running a business Expert Not in cards Aug 2026
Paloma Language models are measured by perplexity on held-out text from 585 domains, from news sites to individual subreddits. Allen Institute for AI Commonsense Everyday Not in cards Dec 2023
PAM-Audio Models judge the quality of generated environmental sounds and how closely each recording matches its caption. The listening study used 100 AudioCaps captions. Carnegie Mellon University and Microsoft Audio understanding and voice Everyday Not in cards Feb 2024
Parafraseja Models decide whether two Catalan sentences express the same meaning. Human annotators wrote contrasting pairs. Barcelona Supercomputing Center Reading in many languages School Not in cards Oct 2022
ParaShoot Models answer Hebrew questions by selecting words from a Hebrew Wikipedia passage. Scorers compare the selected text with human answers. Tel Aviv University Reading in many languages School Not in cards Sep 2021
Pare-Bench Proactive assistants infer user goals and act across stateful digital applications with simulated users and multi-app task environments. Apple Apps and MCP servers Expert Not in cards Apr 2026
Parley Labs negotiation tasks Agents negotiate contracts and plan around market changes with scripted counterparties. An offline verifier grades outcomes. Ten public development tasks accompany private scored sets. parley-labs Running a business Expert Not in cards Nov 2026
ParsiNLU Models read Persian passages, compare queries, and identify sentiment. The six tasks also cover entailment, multiple-choice questions, and translation. Daniel Khashabi and colleagues Reading in many languages University Not in cards Oct 2020
ParsiNLU Mizan Literary Translation Models translate literary passages between Persian and English. ParsiNLU scores each direction against 10,000 reference translations using SacreBLEU. ParsiNLU team Translation University Not in cards Oct 2020
Partner Restrictions Compliance Models identify bank partner restrictions that apply to businesses. Ramp scores missed restrictions, coverage across categories, and false positives. Ramp Labs Finance Expert Not in cards Mar 2026
PARTNR Agents follow natural language instructions in human-robot collaboration scenes. Tasks require planning and coordination among multiple embodied agents. Meta FAIR and Superintelligence Labs Embodied reasoning University Not in cards Nov 2024
PASCAL VOC 2012 Models locate objects and label their pixels in photographs. Meta reports detection and segmentation results on the 2012 challenge. PASCAL VOC organizers Detection and captioning Everyday Not in cards May 2012
PASCAL-Context Models assign a semantic class to each pixel in a photograph. The benchmark includes objects and background regions. PASCAL-Context team Detection and captioning Everyday Not in cards Jun 2014
PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. Google Medicine Expert Not in cards Jul 2025
PAWBench Video generators are run many times on the same start and action, and the outcomes become a distribution over physical behaviors. The test checks whether it matches reference probabilities and covers every valid behavior. PAWBench team (with Hugging Face's Sayak Paul) Physical prediction Everyday Not in cards Aug 2026
PAWS-X Models decide whether two sentences with similar wording express the same meaning across languages. Yinfei Yang and colleagues Reading in many languages School Not in cards Aug 2019
PCB-Bench Models answer placement and routing questions and describe circuit boards from editor screenshots. The authors report choice accuracy and separate measures for free-form answers. Hong Kong University of Science and Technology (Guangzhou) Electronics University Not in cards
pCLUE Models follow Chinese prompts for classification, inference, generation and reading comprehension. The leaderboard combines exact match and accuracy across tasks. SuperCLUE Reading in many languages School Not in cards Oct 2022
PE-A-Frame internal audio event localization Models locate sound events within recordings. Meta compares model sizes using the area under the receiver operating characteristic curve. Meta FAIR Audio understanding and voice Everyday Not in cards Dec 2025
PE-AV internal video-to-audio retrieval Models match videos to music tracks. Meta compares retrieval accuracy on internal clips that last from five to thirty seconds. Meta FAIR Audio understanding and voice Everyday Not in cards Dec 2025
Penn epilepsy clinic notes Models read dated paragraphs from epilepsy clinic notes and classify recent seizures. Two other questions ask for seizure frequency and the most recent seizure date. University of Pennsylvania Medicine University Not in cards Jan 2022
People’s Daily and Children’s Fairy Tale Models fill a missing word using a Chinese news passage or a fairy tale. The fairy tales test whether a reader trained on news can transfer. iFLYTEK Research and Harbin Institute of Technology Reading in many languages School Not in cards Jul 2016
PersistBench Models answer requests using stored memories. A judge measures inappropriate disclosure, agreement with false user beliefs and failure to use relevant memories. Sidharth Pulipaka and colleagues Deception and alignment Everyday Not in cards Feb 2026
Personal Agent Bench Personal assistants complete everyday tasks using connected accounts. Experts check whether each assistant completes the work and respects the user’s instructions. micro1 Documents and spreadsheets Everyday Not in cards
PestCLEF 2026 Models extract structured facts from plant health documents. Organizers compare the predicted relationships with expert annotations on a held-out test set. INRAE and Paris-Saclay University Reading comprehension University Not in cards Sep 2025
PFBench Models answer questions about a power grid after specified changes to loads, generators, or transmission lines. The grader checks structured answers against power flow calculations. Buxin She Electronics University Not in cards Mar 2026
PHEB Models answer multiple-choice questions from Portuguese national secondary school exams administered between 2006 and 2023. PHEB authors Knowledge in many languages School Not in cards Jan 2026
PHEE Models extract adverse effects and potential therapeutic effects from published medical cases. They identify the event triggers and details such as the treatment and the patient. University of Warwick and Northeastern University Medicine University Not in cards Oct 2022
PHINC Models translate social media posts that mix Hindi and English into English. Vivek Srivastava and colleagues Translation School Not in cards Apr 2020
PhoATIS Models identify a traveler’s intent and label relevant words in Vietnamese air travel queries. HELM uses the intent classification task. Mai Hoang Dao, Thinh Hung Truong, and Dat Quoc Nguyen Reading in many languages Everyday Not in cards Aug 2021
PhoMT Models translate between Vietnamese and English. Held-out sentence pairs provide reference translations in both directions. Long Doan and colleagues Translation University Not in cards Nov 2021
PHYRE Agents solve two-dimensional physics puzzles by placing new bodies into a simulated scene. The benchmark measures success on unseen puzzles and the number of attempts. Meta FAIR and Superintelligence Labs Physical prediction School Not in cards May 2020
PhysCapBench Models caption videos of physical interactions. The benchmark checks which motions, contacts, and changes of state the captions describe across 1,500 clips. Peking University, Tsinghua University, Beihang University, and collaborators Video captioning Everyday Not in cards Nov 2026
PhysGym Arena DR-Hard Models repair Gym simulators that must behave correctly under unseen physical parameters. The grader compares their simulations with hidden tests and reference trajectories. kishanpb Scientific code Expert Not in cards May 2026
Physics GRE (Inflection) Models answer questions from Physics GRE exams. Inflection reports majority-vote accuracy, while the harness also supports direct multiple-choice scoring. Inflection AI Science questions University Not in cards Mar 2024
Physiome-ODE Models forecast irregularly sampled biological time series generated from published differential equations. The authors compare forecasts using mean squared error. Christian Klötergens and colleagues Genomics and life sciences University Not in cards Feb 2025
PhyWorldBench Video generators create scenes that follow physical laws across object motion, energy conservation, rigid interactions and human or animal motion. NVIDIA Cosmos Lab Video generation Expert Not in cards Feb 2026
Pico DSL v0 Models execute a small string transformation language without tools. An interpreter checks the final string across 240 generated examples. oso Language reasoning School Not in cards Jun 2026
PII-Masking-300k Models locate and classify personal information in multilingual synthetic text. The open portion covers six languages. The authors separately describe a finance and insurance portion. Ai4Privacy Following precise instructions Everyday Not in cards Mar 2024
PiiClustering Models group synthetic English financial documents by document type. The labels include bank statements, payment confirmations, and insurance claim forms. The Hong Kong University of Science and Technology Finance Everyday Not in cards Jul 2024
PiRhDy Accompaniment Suggestion Models rank 50 accompaniment phrases for a MIDI melody. Several tracks can belong with the melody. The authors report mean average precision. Nankai University, National University of Singapore, A*STAR, and Tsinghua University Audio understanding and voice School Not in cards Oct 2020
PiRhDy Melody Completion Models rank 50 possible continuations of a MIDI melody. One phrase follows the prompt in the original piece. The authors report mean average precision. Nankai University, National University of Singapore, A*STAR, and Tsinghua University Audio understanding and voice School Not in cards Oct 2020
PitVQA Models answer questions about a frame from pituitary surgery. Questions cover the surgical step, the instruments in view, and what the surgeon is doing. University College London and collaborators Medicine Expert Not in cards May 2024
Pixel2Perspective Models locate and segment objects from aerial and ground views. Tests vary the camera angle, weather, and use of visible or infrared images. Indian Institute of Science and collaborators Detection and captioning Everyday Not in cards Nov 2026
PLABA 2024 Complete Abstract Adaptation Models rewrite 400 new biomedical abstracts for patients. Reviewers grade how well the rewrites preserve information and explain it clearly. U.S. National Library of Medicine and National Institute of Standards and Technology Medicine University Not in cards Jan 2024
PLABA 2024 Term Replacement Models identify difficult medical terms, classify how to adapt them, and supply wording for patients. The organizers held out 300 abstracts. U.S. National Library of Medicine and National Institute of Standards and Technology Medicine University Not in cards Aug 2024
Places Audio Captions (Hindi) Models retrieve photographs from spoken Hindi descriptions or retrieve descriptions from photographs. The score counts how often the correct match appears among the top results. MIT Spoken Language Systems Group Audio understanding and voice Everyday Not in cards Apr 2018
Places205 Models classify photographs by scene type. Meta compares scene recognition on the 205-category edition of Places. MIT Computer Science and Artificial Intelligence Laboratory General visual questions Everyday Not in cards
Plain English Summarization of Contracts Models summarize contract clauses in plain English. Human summaries provide references for evaluating how well each response conveys the legal language. Laura Manor and colleagues Law University Not in cards Jun 2019
Plain Language Adaptation of Biomedical Abstracts Models rewrite biomedical abstracts in plain language for people who asked health questions. The authors collected 750 abstracts for 75 questions. U.S. National Library of Medicine Medicine University Not in cards Jul 2022
PlayWorld An agent player pursues objectives inside video world models. Gemini 3.1 Pro applies weighted rubrics for geometry, interaction, and state evolution across the resulting videos. University of Hong Kong and Kuaishou Technology Interactive world models Everyday Not in cards Aug 2026
PMPP CUDA Evaluation Models implement 53 CUDA kernels and answer 146 questions about parallel programming. The tasks follow the textbook Programming Massively Parallel Processors. Sinatras GPU kernels University Not in cards Oct 2025
PNCExtract Models read a research paper and list its polymer nanocomposite samples. They link each sample to its matrix, filler, and composition by mass or volume. Duke University Lab and data work University Not in cards Feb 2024
Poem Sentiment Models judge the sentiment expressed in individual verses of English poems from Project Gutenberg. Emily Sheng and colleagues Creative writing School Not in cards Dec 2020
PoemSum Models summarize the meaning of an English poem. The authors compare each summary with a reference interpretation using ROUGE and BERTScore. Islamic University of Technology, York University, and Dialpad Canada Reading comprehension School Not in cards Dec 2023
Polaris Healthcare Safety Benchmarks Models conduct simulated healthcare conversations. The evaluation measures clinical safety, communication, and administrative task performance. Hippocratic AI Medicine Expert Not in cards
PolEmo 2.0 Models classify the sentiment of Polish consumer reviews. Separate test sets measure performance in familiar and unfamiliar domains. Jan Kocon and colleagues Reading in many languages Everyday Not in cards Nov 2019
PolEval 2025 Gender-inclusive Proofreading Models rewrite Polish passages that use generic masculine forms while preserving their meaning. Institute of Computer Science, Polish Academy of Sciences Reading in many languages School Not in cards May 2025
PolEval 2025 Gender-sensitive Translation Models translate inclusive Polish passages into English and English passages into gender-inclusive Polish. Institute of Computer Science, Polish Academy of Sciences Translation University Not in cards May 2025
Polish Speech Emotion Recognition Challenge Models identify one of six emotions in Polish speech after training on recordings in other languages. PolEval 2025 Task 4 organizers Audio understanding and voice Everyday Not in cards May 2025
PopCS Models transcribe lyrics from Chinese pop singing. Qwen compares recognition errors across speech models. PopCS team Speech recognition and translation Everyday Not in cards May 2021
Post-OCR Text Correction in Sanskrit Models correct machine transcriptions of printed Sanskrit books in Devanagari. The benchmark reports character and word error rates. Ayush Maheshwari, Nikhil Singh, Amrith Krishna, and Ganesh Ramakrishnan Reading in many languages University Not in cards Jun 2022
PosterBench Agents turn research papers into posters. Programmatic checks and visual model judgments score seven content and design dimensions, with ceilings for serious rendering or layout failures. Meituan, MBZUAI, Huazhong University of Science and Technology and others Documents and spreadsheets University Not in cards Aug 2026
PosteriorBench Samplers generate rare model behaviors and are scored by hit rate, posterior divergence, and inference cost. Transluce AI research tasks Research frontier Not in cards Nov 2026
PPR10K Models retouch groups of portraits against three expert retouchers’ Adobe Camera Raw edits. The test measures accuracy on people and color consistency across each group. The Hong Kong Polytechnic University and Alibaba DAMO Academy Image generation and editing University Not in cards Mar 2021
PPTCoEditBench Agents edit selected content in PowerPoint slides. Tasks include changing a title or table while preserving surrounding shapes and animations. Tencent Documents and spreadsheets School Not in cards Nov 2026
PPTFixBench Agents inspect a PowerPoint file and repair its formatting and content. Each of the 50 files contains more than 25 issues. Tencent Documents and spreadsheets University Not in cards Nov 2026
PreScience Models forecast future scientific contributions through collaborator, prior-work, contribution and impact prediction tasks. Ai2 and University of Chicago Real-world events Research frontier Not in cards Feb 2026
PretrainGym Agents select and mix public training data for a fixed language model. The grader trains on 400 million tokens and rewards low test loss. It penalizes overlap with evaluation data. Justin Yang Chae Machine learning engineering Research frontier Not in cards Oct 2026
Privy Synthetic PII Protocol Trace Dataset Models locate personal information in synthetic API messages represented as protocol traces. The small and large configurations have separate test files. Benjamin Kilimnik and Pixie Reading comprehension University Not in cards Sep 2022
ProbSum 2023 Models list active diagnoses and problems from hospital progress notes. Organizers compare the lists with a physician’s annotations using ROUGE-L F1. University of Wisconsin and collaborators Medicine Expert Not in cards Jan 2023
PST50 Models transfer a reference look to a photograph. The paired test compares 50 outputs with reference grades that professionals refined in DaVinci Resolve. Nanyang Technological University and SenseTime Research Image generation and editing University Not in cards Jun 2025
PsychoBench Models answer questionnaires from 13 psychological scales. Each instrument produces a profile of traits, relationships, motivations or emotional abilities. PsychoBench authors Deception and alignment Everyday Not in cards Oct 2023
PsyCoMark: Conspiracy Detection Models decide whether a Reddit comment expresses a conspiracy belief. The organizers score predictions against human labels using weighted F1. PsyCoMark organizers Reading comprehension School Not in cards Mar 2025
PsyCoMark: Conspiracy Marker Extraction Models mark the words that describe conspiracy claims in Reddit comments. The organizers score overlap with human annotations across five marker types. PsyCoMark organizers Reading comprehension School Not in cards Mar 2025
PsyTAR Models identify adverse reactions in patient reviews of four antidepressants. The original corpus also labels withdrawal symptoms, reasons for treatment, and reported effectiveness. Maryam Zolnoori and colleagues Medicine University Not in cards Jan 2019
PUBHEALTH Models classify public health claims as true, false, mixed or unproven. The original benchmark also asks models to explain their verdicts. Neema Kotonya and Francesca Toni Faithful to sources University Not in cards Nov 2020
Public Tests, Hidden Tests Models write Python functions from public test cases. The grader runs hidden tests to measure whether each solution generalizes beyond the visible examples. summer-winter-98379 Function-level problems School Not in cards May 2026
PubMed causal language use Models classify the wording of a research conclusion as a direct causal claim, a conditional causal claim, a correlation, or no relationship. Syracuse University and Jun Wang Medicine University Not in cards Sep 2019
PubMed-PICO Models classify sentences in structured medical abstracts. Section headings supply the reference labels, including those for the study population, intervention, and outcome. MIT Medicine University Not in cards Jun 2018
PUCRS inpatient fall detection Models identify whether a Portuguese progress note documents a fall. Nursing students labeled 355 of the 1,078 notes as fall related. Pontifical Catholic University of Rio Grande do Sul Medicine University Not in cards Jul 2019
PUMaC 2024 Models solve problems from the 2024 Princeton University Mathematics Competition. Scores measure the percentage of contest problems answered correctly. Princeton University Mathematics Competition Competition problems with answers Competition Not in cards Nov 2024
PUMaVOS Models track masks for object parts across video frames. A prompt can identify a small region, such as one cheek of a person's face. PUMaVOS team Video understanding Everyday Not in cards Jul 2023
PushWorld Agents move shapes through puzzles that require tools or pushing obstacles aside. The benchmark includes more than 200 puzzles with goal positions. Google DeepMind Puzzles and chess School Not in cards Jan 2023
PVD Bench Models match captions with short videos in both retrieval directions. Meta measures retrieval on held-out video and caption pairs. Meta FAIR Video understanding Everyday Not in cards Apr 2025
PWS-Decisions Decision APIs choose answers to 300 closed-ended questions whose answers depend on current or scattered facts on the public web. Parallel Web Systems Deep research Expert Not in cards Nov 2026
PWS-Legal legal Web research systems answer dated questions about companies, laws, regulations, intellectual property, court decisions, and legal documents. Parallel Web Systems Law Expert Not in cards Nov 2026
Py Bug Trace Level 3 Models fix Python functions after inspecting examples that miss the defect. The grader runs a stricter hidden test suite. neomatrix369 Function-level problems University Not in cards Jun 2026
Q-Bench Image-Pair Perception Models answer multiple-choice questions comparing low-level attributes across image pairs. The score is overall answer accuracy in the A1-Pair table. Nanyang Technological University and collaborators General visual questions Everyday Not in cards Sep 2023
Q-Bench Single-Image Perception Models answer multiple-choice questions about low-level attributes of individual images. The score is overall answer accuracy in the A1-Single table. Nanyang Technological University and collaborators General visual questions Everyday Not in cards Sep 2023
Q2D-Chunk Retrievers find short passages that answer a query or supply the evidence needed to interpret the answer. Perplexity Questions that need search University Not in cards Oct 2026
QA4MRE Models read documents and choose answers that require integrating information across the text. The harness includes the 2011, 2012 and 2013 CLEF tests. CLEF QA4MRE organizers Reading comprehension University Not in cards
QAG-360K Models compare two satellite images, answer questions about land cover, and mark the relevant pixels. The authors score answer accuracy and mask overlap separately. Ke Li and colleagues Spatial reasoning and counting School Not in cards Oct 2024
QALB 2015 Models correct Arabic news comments and essays by Arabic learners. The organizers score edits separately for native writers and learners. Qatar Arabic Language Bank project Reading in many languages School Not in cards Feb 2015
QAMPARI Models return multiple answers to a question using evidence scattered across Wikipedia passages. Each question has at least five correct answers. Tel Aviv University Questions that need search University Not in cards May 2022
QASC Models answer eight-choice grade-school science questions by combining supporting facts from a large text corpus. Accuracy checks the selected answer against the answer key. Allen Institute for Artificial Intelligence and collaborators Science questions School Not in cards Oct 2019
QBQTC Models judge how relevant a page title is to a Chinese search query. CLUE benchmark team Questions that need search Everyday Not in cards
QEVD-FIT-COACH Models watch a workout and decide when to give feedback. The benchmark scores feedback timing and the content of the coaching advice. Qualcomm AI Research Video understanding Everyday Not in cards Jul 2024
QIAS 2025 Islamic Knowledge Assessment Models answer Arabic multiple-choice questions about classical Islamic texts. The released test contains 1,000 rows and 729 distinct questions. QIAS organizers Exam questions University Not in cards May 2025
QMSum Models answer a query about a long meeting transcript with a written summary of the relevant parts. The 1,808 query and summary pairs cover 232 meetings from several domains. Scores compare outputs with reference summaries using ROUGE. Yale University (LILY Lab) and collaborators Reasoning over long documents School Not in cards Apr 2021
Qodo AI Code Review Benchmark Models review real merged pull requests with planted functional bugs and best-practice violations. Findings are matched to the planted issues and scored by F1. Qodo Fixing real repositories Expert Not in cards Feb 2026
Quartets Audio-to-Score Transcription Models transcribe synthesized string quartets by Haydn, Mozart, and Beethoven into four-part scores. The released test split contains 7,611 excerpts. University of Alicante Audio understanding and voice Expert Not in cards Oct 2019
QuartetsMIREX 2026 Audio-to-Score Transcription Models transcribe synthesized string quartet recordings into notation. Organizers scored 373 excerpts by Schubert, Brahms, and Mendelssohn, whose music the original Quartets dataset omits. MIREX Audio understanding and voice Expert Not in cards
Questflow Traders Arena Agents and people trade on Hyperliquid and Polymarket in a public competition. The leaderboard reports weekly portfolio returns alongside gains and trading volume. Questflow Finance University Not in cards Jun 2026
QuixBugs Models repair 40 classic algorithm programs, in Python and Java, that each contain a one-line bug. The bugs fall into 14 defect classes. A fix counts when the program passes the test cases. MIT (Lin, Koppel, Chen, Solar-Lezama) Function-level problems School Not in cards Jul 2017
Qur’an QA 2023 Passage Retrieval Models rank Quran passages that answer an Arabic question or return no passages. Organizers score mean average precision at ten. Qatar University Questions that need search University Not in cards Jun 2023
Qur’an QA 2023 Reading Comprehension Models extract answers from a supplied Quran passage or return no answer. Organizers give partial credit for overlapping answer spans. Qatar University Reading comprehension School Not in cards Jun 2023
Qwen-Audio-Agent In-House Cockpit Benchmark Agents handle requests with cockpit tools. The evaluation measures whether they complete each task when they call tools directly, delegate, or choose between both routes. Alibaba Token Foundry Function calling Everyday Not in cards Sep 2026
Qwen-Music Cover Song Generation Models remake a reference melody using requested lyrics and style. The evaluation aligns the melodies and measures their average pitch difference in semitones. Alibaba Qwen Team Audio understanding and voice University Not in cards Jul 2026
Qwen3 forced alignment benchmarks Models place word or character boundaries in audio. Qwen measures timestamp shifts against automatic and human reference alignments. Qwen Speech recognition and translation School Not in cards Jan 2026
Qwen3-ASR internal transcription benchmarks Models transcribe accented conversations, noisy speech, tongue twisters, and multilingual news. Qwen reports error rates for each recording condition. Qwen Speech recognition and translation Everyday Not in cards Jan 2026
RaceBias (Race-Based Medicine) Models answer clinical questions used to identify unsupported race-based medical beliefs. Evaluators check responses for those beliefs. Jesutofunmi A. Omiye and colleagues Bias, fairness and politics Expert Not in cards Oct 2023
RadNLP 2024 Lung Cancer Staging Models assign detailed lung cancer stages to radiology reports. The organizers score Japanese and English separately and require all three stage labels to match. NTCIR RadNLP organizers Medicine Expert Not in cards Mar 2024
RadNLP 2024 Radiology Report Segmentation Models label sentences in lung cancer reports with eight categories, including tumor size and distant metastasis. The organizers score labels using micro F2. NTCIR RadNLP organizers Medicine Expert Not in cards Jun 2024
RAGBench Evaluators judge whether retrieved passages are relevant and generated answers are grounded. Galileo compares their judgments with reference annotations. Galileo Faithful to sources University Not in cards Jun 2024
Railhead Agents run a transport company in OpenTTD. The evaluator replays the action log and lets the company run without intervention before measuring its performance. Rachit Shah Video games Everyday Not in cards Oct 2026
Range Repair Agent Agents repair a Python function that merges touching ranges. The sole task earns one point if the edited function passes every held-out test. akashdeepjassal Function-level problems School Not in cards Oct 2026
Rauh Bundestag sentence sentiment validation Models rate the sentiment of individual German parliamentary sentences. Three native speakers independently rated all 1,500 sentences from Bundestag speeches delivered from 1991 through 2013. Christian Rauh Reading in many languages Everyday Not in cards May 2018
RAVDESS Models identify an actor's expressed emotion from recorded speech or song. Steven Livingstone and Frank Russo Speech recognition and translation Everyday Not in cards May 2018
Ravnursson Faroese Speech and Transcripts Models transcribe 3,002 Faroese recordings from 20 test speakers. The speakers read aloud for four and a half hours. Reykjavík University and the Ravnur Project Speech recognition and translation Everyday Not in cards Sep 2022
RCTs to Scale nudge forecasts Models predict how each of 14 government interventions changes participation or compliance relative to a control. Researchers correlate predictions with observed changes in percentage points. Stefano DellaVigna and Elizabeth Linos Social simulation University Not in cards Jul 2020
Re:Baltica Latvian and Russian chatbot sources Chatbots answer four questions about the Baltic states in Latvian and Russian. Journalists count the propaganda sources each chatbot offers in each language. Re:Baltica Long-form factuality Everyday Not in cards Sep 2025
ReactSim-Bench Driving simulators predict how traffic responds when the vehicle changes its behavior. The authors test those reactions in 2,636 scenarios. Zhiyuan Zhang and colleagues Simulated robots University Not in cards Jun 2026
RealCommBench Models distinguish human and synthetic speech after transmission over real phone calls. Paired recordings measure how the call changes detection error. Li Wang and colleagues Speech recognition and translation Everyday Not in cards Oct 2026
RealEstate10K Models estimate relative camera poses across frames from real estate videos. The benchmark supplies camera trajectories for comparison. Tinghui Zhou and colleagues Spatial reasoning and counting School Not in cards
RealHall Evaluators score model responses for hallucinations. Four datasets cover questions with and without reference documents. The score measures agreement with binary ground truth labels. Galileo Hallucination rates University Not in cards Oct 2023
RealHarm Safety systems classify real-world harmful conversations and compare their decisions with documented safe and unsafe versions across ten harm categories. Giskard AI Refusing harmful requests Expert Not in cards Apr 2025
RealityTest Conversational models answer identity questions in text and speech across five languages. A judge classifies explicit, ambiguous and missing identity disclosure. UK AI Security Institute Deception and alignment Everyday Not in cards Jun 2026
Reasoning Gym Models solve procedurally generated reasoning problems with automatic rewards. Harbor draws three easy and three hard instances from each of 96 task generators. Reasoning Gym team Language reasoning School Not in cards Jan 2025
ReasoningJudgeBench Judge models rate reasoning outputs to measure evaluator reliability and agreement across difficult tasks. Salesforce AI Research AI research tasks Expert Not in cards Jan 2026
ReasonSeg Models infer which object an instruction describes and return its mask. Instructions can require reasoning about the image. ReasonSeg team Detection and captioning Everyday Not in cards Aug 2023
ReasonVOS Models segment and track objects in videos from explicit referring queries and implicit reasoning queries. The score is overall J&F on ReVOS. University of Technology Sydney and Alibaba (Yan et al., VISA) Video understanding Everyday Not in cards Jul 2024
Recipe1M image and recipe retrieval Models match food photographs with recipes. Google evaluates image retrieval of the matching ingredients and instructions separately. Recipe1M team General visual questions Everyday Not in cards
Recovery-Bench Agents repair a task after a weaker agent leaves failed actions and a corrupted environment. The score is the share of recovery tasks that pass the original task verifier. Letta Long terminal sessions Expert Not in cards Aug 2025
Red Team Attack Simulator Models test whether a simulated assistant follows its safety policy across three conversational turns. The evaluation records policy violations and penalizes direct harm. Intertwine Jailbreaks and prompt injection University Not in cards Sep 2025
Red-Teaming Resistance Benchmark Models respond to prompts from multiple safety datasets. Judges assess whether the responses comply with harmful requests. Haize Labs and Hugging Face Jailbreaks and prompt injection Everyday Not in cards Feb 2024
RedlineBench v2 (Fa1zvn) Agents negotiate eight contract terms with a simulated vendor. The grader compares the agreement with the best deal both sides could accept. fa1zvn Running a business University Not in cards Jun 2026
Ref-DAVIS17 Models segment and track the object that a sentence describes through each frame of a DAVIS 2017 video. Scored by J&F, the mean of mask overlap and contour accuracy. Max Planck Institute for Informatics (Khoreva, Rohrbach, Schiele) Video understanding Everyday Not in cards Mar 2018
Ref-YouTube-VOS Models segment and track the object that a sentence describes in YouTube-VOS videos. Scored by J&F on a server-graded validation split. Seoul National University and Adobe Research (Seo, Lee, Han, URVOS) Video understanding Everyday Not in cards Aug 2020
Ref-ZOM Models mark the pixels described by a phrase in a COCO image. A phrase can refer to zero, one or several objects. Yutao Hu and colleagues Detection and captioning Everyday Not in cards Oct 2023
RefAV Models find driving scenarios described in natural language. They identify when each scenario occurs and track the objects involved. Carnegie Mellon University Detection and captioning University Not in cards Mar 2025
RefChecker Models generate answers that are checked for fine-grained claims against references. Claim-level labels measure factual consistency and hallucination detection quality. Amazon Science Hallucination rates University Not in cards Jan 2024
RefineBench Models refine answers to 1,002 problems across 11 domains using checklists that specify required content and errors. NVIDIA Research Taiwan Following precise instructions Expert Not in cards Nov 2025
RefSpatial-Expand-Bench Location Models solve spatial location questions in the RefSpatial expansion set. The score is the published overall accuracy. BAAI (RoboRefer) Embodied reasoning Everyday Not in cards Oct 2025
RefSpatial-Expand-Bench Placement Models solve spatial placement questions in the RefSpatial expansion set. The score is the published overall accuracy. BAAI (RoboRefer) Embodied reasoning Everyday Not in cards Oct 2025
RegexEval Models turn natural-language descriptions into regular expressions. The dataset contains 762 patterns with matching and nonmatching examples. The hub port holds out 62 patterns. Mohammed Latif Siddiq and colleagues Function-level problems University Not in cards Nov 2023
RELATE Models listen to an audio clip and predict how well it matches a caption. Human listeners also judge which sounds occur and in what order. Yusuke Kanamori and colleagues Audio understanding and voice Everyday Not in cards Jun 2025
ReMuQ Models retrieve a knowledge passage that answers a question about an image. Man Luo and colleagues General visual questions School Not in cards Jun 2023
RepliBench Agents acquire resources, exfiltrate model weights, replicate onto compute and persist over time. Task checks measure components of autonomous self-replication. UK AI Security Institute Labs' internal R&D tests Research frontier Not in cards Apr 2025
Research Plan Generation Models write research plans from goals derived from scientific papers. Judges grade each plan against criteria specific to its source paper. Meta AI research tasks Expert Not in cards Dec 2025
RESPIN-S1.0 Models transcribe dialect speakers reading sentences about farming and finance in nine Indian languages. The test holds out speakers and reports errors by dialect. Indian Institute of Science and Navana Tech Speech recognition and translation Everyday Not in cards May 2025
REVERSAL-BENCH Reset-free reinforcement learning agents learn manipulation policies across environments with controlled reversibility and recoverability checks. Apple Simulated robots Expert Not in cards Sep 2026
Reverse Wikipedia Paragraphs Models reverse a Wikipedia paragraph character by character. The evaluator compares the response with the exact reversal using the longest common subsequence. d42me Following precise instructions School Not in cards Oct 2025
Revisited Oxford and Paris Models retrieve photographs of the same landmark from Oxford and Paris image collections. The revised benchmark defines medium and hard evaluation protocols. Filip Radenović and colleagues General visual questions Everyday Not in cards Mar 2018
Reward Hack Bench Agents attempt eight coding and puzzle tasks under controlled conditions. A judge distinguishes independent solutions from use of published answers. The control omits hints. Islo Labs Deception and alignment University Not in cards Jun 2026
Reward-seeking via contrastive belief updates Researchers compare model decisions under different beliefs about what a grader rewards. The evaluation measures how often those beliefs change behavior. Apollo Research and OpenAI Deception and alignment Expert Not in cards Jul 2026
RICH Models reconstruct people interacting with their surroundings. The evaluation compares body joints and mesh vertices with the reference geometry. RICH team Spatial reasoning and counting School Not in cards Jun 2022
RivaBench Models answer questions about stand-up routines and academic talks using speech and visual cues. They also classify clips as real or synthetic. University of Cambridge, ByteDance, and Tsinghua University Video understanding University Not in cards Feb 2025
RLBench Robot arm policies attempt 100 hand-designed manipulation tasks in simulation, from reaching a target and opening a door to multi-stage tasks such as putting a tray in an oven. Each task has camera, depth and joint observations and an unlimited supply of motion-planned demonstrations. Scores are task success rates. Imperial College London (Dyson Robotics Lab) Simulated robots Everyday Not in cards Sep 2019
RMU:ECHR Models mark argument spans in European Court of Human Rights rulings and identify the type of argument and the party making it. Ivan Habernal and colleagues Law University Not in cards Aug 2022
Roark Live Bench v1.3 Voice agents answer calls over the public phone network and use simulated business tools. Roark scores task completion, tool use, speech quality, and response time. Roark Customer service University Not in cards Oct 2026
Robocurve physical tool-use benchmark (unidentified) Unnamed Robocurve physical tool-use robot task. Robocurve Real robots Everyday Not in cards
Robot Capabilities Index (Robocurve) An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3. Robocurve Real robots Everyday Not in cards Oct 2026
Robotic Origami Challenge (IROS 2026) Fold a six-fold paper airplane on a supplied bimanual Sharpa-hand rig. Origami judges give pass/fail, and passes are ranked by folding time. Sharpa + BitRobot/FrodoBots + Nippon Origami Association + partners Real robots Everyday Not in cards Jun 2026
Robots Evolved GUI lessons Agents use screenshots, clicks, and keys to move sliding robots onto a goal. Ten puzzles enforce a move budget and offer no undo. suryanshsh Puzzles and chess Everyday Not in cards Oct 2026
Robust Multi-view Depth Models estimate depth from multiple photographs of a scene. The benchmark compares relative depth errors across datasets from different environments. Robust Multi-view Depth team Spatial reasoning and counting School Not in cards Sep 2022
robust-kbench Agents write CUDA kernels that are tested across several weight initializations and input shapes, so shape and magnitude exploits fail. Correctness checks and profiling score each kernel. Sakana AI GPU kernels Expert Not in cards Sep 2025
ROI Calculator (dsm) Models calculate total and annualized investment returns, decide whether to invest, and explain the decision. The evaluator checks each answer against a bundled reference. dsm Finance School Not in cards Feb 2026
RollerCoasterTycoonBench Models manage a theme park by choosing weekly actions in RollerCoaster Tycoon 2. Vals AI Video games Everyday Not in cards Nov 2026
Roman Urdu Code-Switch Benchmark Models answer 35 items mixing Roman Urdu and English. Tasks cover comprehension, sentiment, translation, and Pakistani cultural context. UMAR SHAREEF Reading in many languages School Not in cards Oct 2026
Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. Google Research Simulated robots Everyday Not in cards Oct 2020
ROOMELSA Models use a marked region in a room panorama and a written description to retrieve the matching 3D object. They return their ten closest matches. Vietnam National University Ho Chi Minh City and University of Dayton Spatial reasoning and counting Everyday Not in cards Mar 2025
RouterBench Routers select a language model for each prompt using cached responses, quality measurements, and costs. Evaluation measures the quality and cost of those selections. Martian Composite indices University Not in cards
RP2K Models identify retail products from photographs taken in stores. The test distinguishes products with similar packaging. RP2K team General visual questions Everyday Not in cards Jun 2020
RRG24 Models write findings and impressions from one or more chest X-rays per study. Organizers score each section separately using five metrics. Stanford AIMI and collaborators Medicine Expert Not in cards Feb 2024
RRN Models summarize Icelandic news articles from the national broadcaster RÚV using reference summaries. Þór Sverrisson and Hafsteinn Einarsson Reading in many languages School Not in cards May 2023
RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. OpenAI Labs' internal R&D tests Research frontier Not in cards Jul 2026
RSICD Models describe aerial scenes in a sentence. The authors provide five reference captions per image and reserve 1,093 images for testing. Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Li Detection and captioning Everyday Not in cards Sep 2017
RSIEval Models describe scenes in aerial photographs and answer questions about objects, positions, and road directions. Human graders score captions and answers separately. Yuan Hu and colleagues General visual questions Everyday Not in cards Jul 2023
RTP-LX Models continue or judge toxic prompts hand-translated from RealToxicityPrompts into dozens of languages, with native-speaker labels. Measures toxic output and toxicity detection. Microsoft (de Wynter et al.) Bias, fairness and politics Everyday Not in cards Apr 2024
RTSGameBench Agents play the open real-time strategy game Beyond All Reason. RTSGameBench authors Video games Expert Not in cards Jun 2026
Rundkast Models transcribe Norwegian radio news, interviews and debates. Word error rate measures transcription accuracy. Norwegian University of Science and Technology Speech recognition and translation Everyday Not in cards May 2008
Runway Model Router Benchmark Video generation systems produce clips for 250 image-to-video prompts across ten production categories and are compared by usable rate and cost. Runway Video generation Expert Not in cards Sep 2026
RVMS-Bench Agents search open-domain videos and localize moments using global impression, key moment, temporal context and auditory memory cues. Tencent Web browsing Expert Not in cards Mar 2026
S3Gym Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict. ByteDance Seed AI research tasks Expert Not in cards Aug 2026
SA-1B Hard human pose evaluation Models locate body and foot joints in photographs with unusual viewpoints, poses, and occlusions. Meta measures the fraction of correctly placed joints. Meta Spatial reasoning and counting School Not in cards Feb 2026
SA-37 interactive segmentation Models refine an object mask after a user clicks on an image. The benchmark averages overlap across datasets at different numbers of clicks. Meta Detection and captioning Everyday Not in cards Jul 2024
Saarland MIDI-Audio Piano Music Models transcribe student piano performances. The Disklavier captured the reference notes as the musicians played. Saarland University, Max Planck Institute for Informatics, and Hochschule für Musik Saar Audio understanding and voice Expert Not in cards Oct 2011
Sakana Japanese-English translation evaluation Models translate Japanese text into English while handling Japan-specific proper nouns and honorifics. Sakana AI Translation University Not in cards Aug 2026
Sakana kana-letter reading-order set Model recovers the reading order of 25 expert-annotated classical Japanese kana letters (chirashigaki). Scored by normalized edit distance. Sakana AI Documents, charts and text in images Expert Not in cards Jun 2026
SalesBench (Insurance Sales) Agents find insurance leads, quote plans, and schedule callbacks in a simulated sales pipeline. The evaluator rewards converted monthly premiums and checks compliance with calling restrictions. SalesBench Running a business University Not in cards Feb 2026
SalesBench-100 Agents reconcile sales evidence, update authorized records, and deliver executive briefs across 100 workflows. Checks grade procedure, system state, and outputs. Blobfish AI Apps and MCP servers Expert Not in cards Aug 2026
SAM 3D Artist Objects Models reconstruct an object's shape and placement from a single photograph. The benchmark compares predictions with meshes created by 3D artists. Meta Spatial reasoning and counting School Not in cards Nov 2025
SAM 3D Body synthetic pose evaluation Models estimate human pose and shape from synthetic images of individual people. Meta compares predicted joints and mesh vertices with the rendering parameters. Meta Spatial reasoning and counting School Not in cards Feb 2026
SAM Audio Judge evaluation Models rate the quality of separated speech, music, and environmental sounds. Meta measures correlation with listener ratings for completeness, precision, faithfulness, and overall quality. Meta Audio understanding and voice Everyday Not in cards Dec 2025
Samrómur 21.05 Models transcribe Icelandic sentences read aloud by volunteers. The training, development, and test sets use different speakers. Reykjavík University Speech recognition and translation Everyday Not in cards May 2021
Samrómur Children Models transcribe Icelandic speech from children reading prompts. The test covers girls and boys aged six through sixteen. Reykjavík University Speech recognition and translation Everyday Not in cards Sep 2021
SandboxEscapeBench Agents attempt to escape a container sandbox and retrieve a flag from the host. Inspect runs the challenge inside reproducible virtual machines. UK AI Security Institute Finding and exploiting real bugs Expert Not in cards Oct 2026
SandhiKosh Models join Sanskrit words under sound change rules or recover the original words from a joined form. The authors score accuracy separately for joining and splitting. Indian Institute of Technology Delhi and IBM Research Reading in many languages University Not in cards May 2018
SatLidar canopy height evaluation Models predict canopy height from satellite images. Meta compares height estimates with airborne lidar measurements and reports errors on held-out regions. Meta Spatial reasoning and counting University Not in cards Aug 2025
SatNet Agents assign ground antennas and contact times to spacecraft requests across five weeks of NASA Deep Space Network operations. They balance unmet demand across missions. NASA Jet Propulsion Laboratory and collaborators Tasks from real jobs University Not in cards Nov 2021
SAUDADE-PT Models determine the chronological order of events related to Portuguese history and culture. AMALIA project Reasoning in many languages School Not in cards Apr 2026
Sayfullina Soft Skill Span Extraction Models mark the words that express a soft skill in English job ads. The test scores exact span matches. IT University of Copenhagen and LMU Munich Other professions Everyday Not in cards Sep 2022
SB10k Models classify the sentiment of German social media posts using labels from three human annotators. Mark Cieliebak and colleagues Reading in many languages Everyday Not in cards Apr 2017
SC-CoMIcs Models read superconductivity abstracts and identify materials, properties, and measurements. The corpus also records relationships, including which dopant replaces an element. Toyota Technological Institute and Toyota Central R&D Labs Lab and data work University Not in cards May 2020
ScanNet Models infer indoor scene geometry from images. Meta evaluates depth predictions against measurements from scanned indoor environments. Angela Dai and colleagues Spatial reasoning and counting School Not in cards Feb 2017
ScanNet-1500 Models match features between pairs of indoor views and estimate relative camera poses. The evaluation measures pose accuracy on 1,500 image pairs. Paul-Edouard Sarlin and colleagues Spatial reasoning and counting School Not in cards Nov 2019
ScanNet++ v2 Models reconstruct indoor scenes from images of real scans. The second release adds scene types and updates the annotations and camera poses. ScanNet++ team Spatial reasoning and counting School Not in cards Dec 2024
ScarfBench Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change. IBM Research Fixing real repositories Expert Not in cards May 2026
SceneFun3D Models identify movable parts in indoor scans, locate the part needed for a written task, and estimate how it moves. ETH Zürich and collaborators Spatial reasoning and counting Everyday Not in cards Mar 2024
Sci Pipeline v1 Agents clean hourly sensor readings and write daily means, record counts, and extremes to a JSON file. The grader checks the saved file. afzal0 Data science and SQL University Not in cards Sep 2026
SciCap Models describe figures from scientific publications. Researchers compare generated captions with the text that accompanies each figure. Ting-Yao Hsu and colleagues Documents, charts and text in images School Not in cards Oct 2021
SciDocs Models compare representations of scientific papers on citation prediction, recommendation, and classification tasks. Allen Institute for AI Research science University Not in cards Apr 2020
Science Sim Chemistry Models compare simulated compounds and choose whether to advance or reject them. The evaluator scores decisions against measured reaction outcomes and a budget. allan Lab and data work University Not in cards May 2026
SciFact Models retrieve research abstracts and determine whether they support or refute a scientific claim. Allen Institute for AI Faithful to sources Expert Not in cards Apr 2020
SciQ Models answer crowdsourced multiple-choice science exam questions in physics, chemistry and biology, each with a supporting passage. Allen Institute for AI (Welbl et al.) Science questions School Not in cards Jul 2017
SciVid Video models answer questions about scientific videos across several disciplines. The benchmark tests whether models recognize procedures, objects, and events in specialized footage. Google DeepMind Vision University Not in cards Jul 2025
SCORE (Consistency and Robustness) Models answer the same questions under different option orders, prompt formats and decoding settings. The evaluator measures consistency as well as accuracy. NVIDIA Following precise instructions University Not in cards Nov 2024
Screen2Words Models summarize the purpose and contents of a mobile interface from a screenshot. The benchmark uses human-written summaries as references. Google Research Documents, charts and text in images School Not in cards Oct 2021
SCROLLS Models do seven long-text tasks such as summarization, question answering and natural language inference over naturally long documents. Tel Aviv University and collaborators Reasoning over long documents University Not in cards Jan 2022
SD-QA Models answer spoken questions recorded in different English dialects. The benchmark compares answer accuracy across regional varieties of English. Fahim Faisal and colleagues Audio understanding and voice Everyday Not in cards Sep 2021
SDBench Diagnostic agents order tests, ask questions and commit diagnoses in sequential clinical cases. Scores combine diagnostic correctness with test cost. Microsoft Medicine Expert Not in cards Oct 2026
SEA-HELM Audio Models answer the same reasoning and instruction-following tasks from speech and text in five languages. Identical checks measure the accuracy lost with speech input. AI Singapore Speech recognition and translation School Not in cards Oct 2026
Security Configuration Verification Agents audit Kubernetes and Terraform configurations, identify security violations and submit fixes. The grader weights detected violations by severity and validates patches. Intertwine Secure code and review University Not in cards Sep 2025
Sedgewick and Wayne Algorithms Exercises Models solve 469 exercises from the first six chapters of Algorithms, fourth edition. A Java sandbox tests code, and a model judge scores theoretical answers. Amit Prakash Function-level problems University Not in cards Nov 2025
Segment Anything Video Models predict masks for prompted objects throughout a video. The evaluation compares predicted boundaries and regions with annotated object tracks. Meta Video understanding Everyday Not in cards Jul 2024
Segment Anything with Concepts Models find every instance of a concept named in a prompt. The benchmark compares image masks and video tracks with human annotations. Meta Detection and captioning Everyday Not in cards Nov 2025
Self-Instruct User-Oriented Instructions Models follow manually written instructions on everyday tasks. Evaluators judge whether their responses meet the user's request. Yizhong Wang and colleagues Following precise instructions Everyday Not in cards Dec 2022
SemClinBr Models identify and classify medical terms in Brazilian Portuguese clinical notes. The corpus contains 1,000 notes from multiple specialties. Pontifical Catholic University of Paraná and collaborators Medicine University Not in cards Jan 2020
SemEval 2015 Task 14 Analysis of Clinical Text Models identify disorder mentions and map them to terminology codes in clinical notes. A separate task asks for attributes such as negation and severity. SemEval Task 14 and ShARe organizers Medicine University Not in cards May 2014
SemEval 2023 Task 3: Genre, Framing and Persuasion Techniques in Online News Models classify a news article as reporting, opinion, or satire. They also label its frames and identify persuasion techniques in individual paragraphs. SemEval 2023 Task 3 organizers Reading in many languages University Not in cards Sep 2022
SemEval 2026 Task 4: Narrative Story Similarity and Narrative Representation Learning Models choose which story is more similar to a reference story. A second track compares story embeddings. Each track scores accuracy on 400 triples. University of Hamburg, Toloka AI, German UDS and TU Darmstadt Reading comprehension School Not in cards Jul 2025
SemEval-2015 Task 1 Models decide whether two short social media posts express the same meaning. Wei Xu and colleagues Reading comprehension School Not in cards Jun 2015
SemiKongBench-100 Agents resolve 100 synthetic semiconductor operating exceptions. The evaluator grades each workflow with deterministic checks. Blobfish AI Tasks from real jobs Expert Not in cards Sep 2026
Sentences Involving Compositional Knowledge Models compare sentences whose meanings change with their grammatical structure. The dataset includes both relatedness ratings and entailment labels. Marco Marelli and colleagues Reading comprehension School Not in cards May 2014
SentinelBench Agents monitor changing web environments and respond when scripted events make progress possible. Scores measure completion, reaction time and resource use. Microsoft Research Web browsing Expert Not in cards Jun 2026
Sequential Question Answering Models answer sequences of related questions about Wikipedia tables. Answers identify table cells, so later questions can depend on earlier selections. Evaluation checks predicted answers against the annotated cells. Microsoft Research Reading comprehension School Not in cards Nov 2016
SEvenLLM Models analyze cyber security incidents and write responses. Tasks split into understanding and generation over 28 subcategories, in Chinese and English. SEvenLLM authors (Ji et al.) Finding and exploiting real bugs Expert Not in cards May 2024
SF20K Competition 2025 Models answer story questions about short films. GPT-4.1-nano compares their answers with human references. Films released during the competition determine the final rankings. SLoMO Workshop at ICCV 2025 Long videos Everyday Not in cards Jul 2025
SFI-Bench Multimodal models answer video questions about spatial layouts, object affordances and functional reasoning in indoor scenes. Apple Spatial reasoning and counting University Not in cards May 2026
SHREC 2026 Human Perceived Visual Complexity of 3D Shapes Models rank CAD parts, primate teeth, and fractals by visual complexity. People compared pairs of CAD parts and fractals. A zoological measure ranks the teeth. SHREC 2026 visual complexity track organizers Spatial reasoning and counting Everyday Not in cards
SHROOM-Visions 2026 Models inspect an image and a written response. They mark unsupported characters, estimate the probability of an error and classify each error in four languages. SHROOM-Visions organizers Faithful to sources Everyday Not in cards Apr 2026
SICK-NL Models judge whether one Dutch sentence entails, contradicts, or is neutral toward another sentence. Gijs Wijnholds and Michael Moortgat Reasoning in many languages School Not in cards Apr 2021
SIGHT (Search and Intent Gauge for Hospitality and Tourism) Agents recommend restaurants, hotels and attractions from user preferences and search results. Evaluators check constraints, evidence and privacy. AGI-Eval (Meituan) Apps and MCP servers Everyday Not in cards May 2026
SIGHUM Sanskrit Word Segmentation Models recover individual Sanskrit words from continuous text. Evaluators measure word precision, recall, F1, and the percentage of sentences whose entire segmentation matches. Amrith Krishna, Pavankumar Satuluri, and Pawan Goyal Reading in many languages University Not in cards Jun 2017
SIGMORPHON 2023 Shared Task on Interlinear Glossing Models label each word’s meaningful parts with translations or grammatical tags. The organizers rank submissions by word-level accuracy pooled across languages. SIGMORPHON Translation Expert Not in cards Feb 2023
SignBD-Word Models identify isolated Bangla signs from video or body pose. The authors score top one and top five accuracy. Bangladesh University of Engineering and Technology Video understanding Everyday Not in cards Jun 2022
Silent Exit Models fix four typos without changing correct lines. A held-out test measures whether models learned to quit early to avoid penalties. dp-learning-rl Deception and alignment Everyday Not in cards Jul 2026
Simple Cooccurrence Bias Models complete occupation prompts with gendered words. The harness reports the percentage of prompts where the model prefers a male term. OpenAI and EleutherAI Bias, fairness and politics Everyday Not in cards May 2020
Simreal ML Benchmark ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them. Simreal Machine learning engineering Expert Not in cards Sep 2026
SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. HCLTech CAD models Expert Not in cards Oct 2026
Singapore Factuality Models answer questions about Singapore's laws, economy, and public policies. A model judge checks whether each answer contradicts the reference. AI Verify Foundation Short factual questions University Not in cards Aug 2026
Singapore POFMA Statements Models classify statements as true or false using claims from Singapore's official corrections. The test includes true statements to check models that always answer false. AI Verify Foundation Short factual questions Everyday Not in cards Jan 2025
Singapore Safety Questions Models answer 59 open-ended questions about culturally or legally sensitive situations in Singapore. A Llama Guard classifier checks whether each response is safe. AI Verify Foundation Refusing harmful requests Everyday Not in cards May 2024
Singapore Undesirable Content Models answer prompts about sensitive situations in Singapore. A classifier labels each response and reports the percentage it judges safe. AI Verify Foundation Refusing harmful requests Everyday Not in cards Aug 2026
Single-turn election integrity (Anthropic, 2026) Models answer legitimate election requests and decline requests that violate Anthropic’s election policy. Graders check refusals and harmful completions separately. Anthropic Refusing harmful requests Everyday Not in cards Apr 2026
SingMOS Models listen to Chinese and Japanese singing and predict how listeners rate its quality. The recordings include human voices and synthesized voices. Renmin University of China and Carnegie Mellon University Audio understanding and voice Everyday Not in cards Jun 2024
SkillSpan Models mark the words that describe skills or knowledge in English job postings. Exact span matches determine precision, recall, and F1. IT University of Copenhagen and LMU Munich Other professions School Not in cards Apr 2022
SkillSpan-ESCO Models link skills mentioned in job advertisements to entries in the European skills classification. Human annotators assigned the reference labels. TechWolf and Ghent University Other professions University Not in cards Sep 2022
skLEP Models classify Slovak reviews, recognize entailment, identify names, and answer questions about Slovak Wikipedia passages. Marek Suppa and colleagues Reading in many languages School Not in cards Jul 2025
SLDBench Agents infer scaling laws from data covering more than 5,000 language model experiments. Eight tasks test how well their formulas predict results at other scales. Haowei Lin and colleagues AI research tasks Expert Not in cards Jul 2025
SlideVQA Models answer questions about whole slide decks, with single-hop, multi-hop and numerical reasoning. The set has 14.5k questions over 2.6k decks. NTT (Tanaka, Nishida et al.) Documents, charts and text in images Everyday Not in cards Jan 2023
SLIP zero-shot transfer suite Models assign labels to images using text descriptions of the classes. The evaluation averages zero-shot accuracy across 26 recognition tasks. Norman Mu and colleagues General visual questions Everyday Not in cards Dec 2021
Slitherlink (savi) Models place edges to form a single loop on seven-by-seven Slitherlink boards. The evaluation uses ten boards disjoint from its 15 training examples. savi Puzzles and chess School Not in cards May 2026
SLoMO Audio Description Competition 2026 Models write one sentence for each gap in a film’s dialogue. Judges score how well the descriptions convey visual information and match professional references. SLoMO Workshop at ECCV 2026 Video captioning University Not in cards Jun 2026
SLPHelm UltraSuitePlus Models analyze recordings of children’s speech for speech sound disorders. The suite also includes transcription and symptom classification tasks. SAA Lab and SLPHelm contributors Speech recognition and translation Expert Not in cards Sep 2025
SLURP Models identify the intent and named entities in spoken requests. NVIDIA evaluates both tasks within AIR-Bench. Emanuele Bastianelli and colleagues Audio understanding and voice Everyday Not in cards Nov 2020
SMEBench Models edit recorded music according to instructions. The benchmark uses paired source and target recordings to check changes to pitch, instrumentation, articulation, and dynamics. Matthieu Cervera and colleagues Audio understanding and voice University Not in cards Sep 2026
ŚMIGIEL Models classify Polish texts as human or machine written. The test includes genres and generators absent from training. PolEval 2025 organizers Reading in many languages University Not in cards Apr 2025
SMILE Video Laugh Reasoning Models explain why people laugh during sitcoms and TED talks. The authors compare generated explanations with human references. POSTECH and collaborators Video understanding Everyday Not in cards Dec 2023
SMM4H 2025 Shingles Vaccine Reactions Models classify English Reddit posts about shingles vaccines. Organizers score F1 for posts in which the author reports a personal reaction. SMM4H-HeaRD organizers Medicine Everyday Not in cards Feb 2025
SMM4H-HeaRD 2026 Flu Vaccine Effectiveness Models classify posts by the author’s flu vaccination status and flu test result. The labels distinguish the current flu season from past seasons. SMM4H-HeaRD organizers Medicine Everyday Not in cards Nov 2025
SMM4H-HeaRD 2026 MedSynth Dialogue-to-Note Models turn synthetic clinical dialogues into notes with four sections. The organizers generated 368 separate test dialogues and scored the notes against references. Dalhousie University and collaborators Medicine University Not in cards Nov 2025
SMM4H-HeaRD 2026 Multilingual Adverse Drug Event Detection Models flag personal reports of drug side effects in seven languages. The organizers withheld Farsi from training and added 15,184 Farsi posts at test time. SMM4H-HeaRD organizers Medicine Everyday Not in cards Nov 2025
SMS Spam Collection Models classify 5,574 English text messages as spam or legitimate. The Prime port creates an 80/20 split and adds risk and reasoning scores. Tiago Almeida, José María Gómez Hidalgo and Akebo Yamakami Overall chat quality Everyday Not in cards Sep 2011
Snowflake AI_EXTRACT Document Extraction Evaluation Models extract fields, lists and tables across 2,789 annotated tasks from 11 datasets. Missing items, extra items and incorrect cells reduce the score. Snowflake Documents, charts and text in images University Not in cards Nov 2026
Snowflake Analytical Search: ArXiv Papers Agents count, group and compare findings across research papers. The evaluation checks numeric and categorical answers against references and scores lists with F1. Snowflake Deep research University Not in cards Jun 2026
Snowflake Arctic Enterprise Intelligence Models write code, generate SQL and follow instructions. Snowflake averages the coding, SQL and instruction-following scores. Snowflake Composite indices University Not in cards Apr 2024
Snowflake Financial Filings RAG Evaluation Systems answer about 500 questions over 23,000 SEC filings. Annotators identify supporting passages, pages and answers. Snowflake Finance Expert Not in cards Mar 2025
Snowflake Financial PDF Table Parsing Models reconstruct tables from 50 manually annotated financial documents. Evaluators compare complete tables, table dimensions and individual cells with reference Markdown. Snowflake Documents, charts and text in images University Not in cards Feb 2026
Snowflake Multimodal PDF Retrieval Evaluation Retrievers find relevant pages in technical manuals, SEC filings and presentation slides. The tasks include 143 chart questions and 69 text questions about manuals. Snowflake Documents, charts and text in images University Not in cards Apr 2025
SO-Bench Multimodal models produce schema-compliant structured outputs grounded in images from four visual domains. Static checks and rubric-based grading score semantic correctness and schema compliance. Apple Documents, charts and text in images University Not in cards Dec 2025
SoccerNet-Caption Models write timestamped commentary for soccer broadcasts. The set has almost 37k commentaries over 715.9 hours of video. Streaming reports also score when the model chooses to speak. University of Liege and KAUST (SoccerNet) Video captioning Everyday Not in cards Apr 2023
Social Science Experiment Treatment Effect Archive Models predict how survey interventions change responses in the United States. Researchers compare predicted treatment effects with experimental estimates using Pearson correlation. Ashwini Ashokkumar, Luke Hewitt, Isaias Ghezae, and Robb Willer Social simulation University Not in cards Aug 2024
SocialMaze Models perform social deduction and other interactive tasks under deep reasoning and uncertainty. SocialMaze authors Multi-agent games Everyday Not in cards Nov 2026
SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. King's College London, Fudan University and The Alan Turing Institute Deception and alignment Everyday Not in cards Jun 2026
SoCRATES Models mediate disputes in 600 scenarios across eight domains. A model judge rates topic agreement to measure consensus gain, intervention timeliness, and intervention effectiveness. KAIST and Chungnam National University Multi-turn conversation University Not in cards Jun 2026
SOFC-Exp Models identify sentences that report fuel cell experiments, mark materials and measurements, and assign roles such as anode material or operating temperature. Bosch Center for Artificial Intelligence and Robert Bosch Corporate Research Lab and data work University Not in cards Jun 2020
SokoBench Models solve Sokoban puzzles that require long-horizon planning and persistent state tracking. SokoBench authors Puzzles and chess Everyday Not in cards Jan 2026
SOL-ExecBench Systems write CUDA kernels for 235 workloads taken from real model architectures, run on B200 GPUs. The SOL score measures how much of the gap between a baseline and the analytic hardware speed limit a kernel closes. NVIDIA GPU kernels Expert Not in cards Mar 2026
Solubility Optimization (anindya) Models choose chemical transformations to improve predicted solubility. The package includes a benchmark runner that measures improvement on evaluation examples. anindya Lab and data work University Not in cards Dec 2025
SongBench Models generate songs from lyrics and a style prompt. A trained music judge scores seven aspects, including the melody, arrangement, vocals, and mix. Tencent and Tsinghua University Audio understanding and voice University Not in cards Mar 2026
SongEval Models predict professional listeners' ratings for full-length songs in English and Mandarin. The test covers memorability, phrasing, structure, coherence, and musicality. Northwestern Polytechnical University and collaborators Audio understanding and voice University Not in cards May 2025
Soohak Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems. OneLineAI, EleutherAI and CMU with 64 mathematicians (Guijin Son et al.) Research mathematics Research frontier Not in cards May 2026
SOP-Bench Agents follow expert-written standard operating procedures across 12 industrial domains, using executable tool interfaces over more than 2,000 tasks with 10 to 50 decision points. Amazon Running a business Expert Not in cards Feb 2026
SoundingEarth audio and image retrieval (MTEB) Models match a field recording to an aerial image of the same location, or retrieve the recording from the image. Each direction has 1,000 queries and 2,048 candidates. Massive Text Embedding Benchmark contributors Audio understanding and voice Everyday Not in cards Jul 2026
SPACE Models solve spatial cognition tasks involving maps, object layouts, spatial attention and memory in text and image presentations. Accuracy measures task performance. Apple Spatial reasoning and counting University Not in cards Apr 2025
SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). xAI (SpaceXAI) Labs' internal R&D tests Expert Not in cards
SPair-71k Models match corresponding parts of objects across image pairs. The benchmark includes changes in viewpoint, scale, and occlusion. Juhong Min and colleagues Spatial reasoning and counting School Not in cards Aug 2019
SpartQA Models answer questions about spatial relationships described in a passage. Roshanak Mirzaee and colleagues Language reasoning School Not in cards Apr 2021
Spatial-Gym Models find paths that satisfy grid puzzle rules. The evaluation uses 500 SPaRC puzzles in one-shot and stepwise settings, including a setting that allows backtracking. Lukas Kaesberg and collaborators Puzzles and chess School Not in cards Apr 2026
SpatialBench (Ropedia) Spatial foundation models are tested across many 3D tasks, viewpoints, scene domains, input densities and hardware limits, beyond the domains they were trained for. Ropedia Spatial reasoning and counting Everyday Not in cards May 2026
SpatialGenEval Text-to-image models generate scenes from 1,230 prompts, and vision-language judges answer 12,300 spatial questions about the resulting images. AMAP and Beijing University of Posts and Telecommunications Image generation and editing University Not in cards Jan 2026
Spec Gaming Review Agents review 12 programs that pass their test suites and decide whether each meets its specification. Six programs exploit gaps in the tests. fredrik-talent Function-level problems University Not in cards Sep 2026
Spec2Control Models convert plant control requirements into function block diagrams. The evaluation compares control strategies, connections, and alarm mappings with reviewed reference diagrams. ABB Electronics Expert Not in cards Oct 2025
SpecDecode RL Agents choose how many tokens to draft during speculative decoding. An offline grader measures speedup over the target model while requiring identical output. semih Machine learning engineering Expert Not in cards Jul 2026
Speech Robust Bench Models transcribe speech under different recording distortions. Word error rates show how each distortion affects recognition. Muhammad A. Shah and colleagues Speech recognition and translation Everyday Not in cards Mar 2024
SpeechIO ASR leaderboard Models transcribe Mandarin recordings from the SpeechIO evaluation sets. Qwen reports an aggregate transcription error. SpeechIO Speech recognition and translation Everyday Not in cards
SpeechMMLU Speech-language models answer MMLU-derived questions from text or synthesized speech in text-to-text, speech-to-text, text-to-speech and speech-to-speech settings. Xiaomi MiMo Audio understanding and voice University Not in cards Sep 2025
SpeedrunBench Agents try to finish goals in 10 video games as fast as possible. Score is time to goal in frames or seconds, lower is better. Patronus AI Video games Everyday Not in cards Sep 2026
SpireBench Agents play Slay the Spire and are scored on long-horizon deck-building decisions. SpireBench authors Video games Everyday Not in cards Oct 2026
SPOT5 daily photograph scheduling Agents choose which photographs a satellite should take and assign cameras. Feasible schedules earn points for image priority while respecting instrument, data rate, and storage limits. Centre national d'études spatiales and collaborators Tasks from real jobs University Not in cards
SPY Models locate personal information about the author of a synthetic medical consultation or legal question. Other people mentioned in the text fall outside the annotation policy. Moscow Institute of Physics and Technology, MTS AI, ITMO University and AIRI Following precise instructions Everyday Not in cards Sep 2024
SQAD Models find the answer to a Czech question in a Wikipedia passage. Student annotators identify the answer sentence and answer span. Natural Language Processing Centre at Masaryk University Reading in many languages School Not in cards
SQL-API-Bench Agents answer questions that require both database queries and API calls. The evaluator checks answers across two mixed database and API benchmark settings. IBM Research Function calling Expert Not in cards Oct 2025
Sregym env Agents diagnose incidents in a running service with generated logs and traffic. The grader checks service recovery, the underlying fault and collateral damage. ericxu88 Apps and MCP servers University Not in cards Aug 2026
Sri Lankan local election chatbot audit Chatbots answer 16 procedural and two political questions in English, Sinhala, and Tamil. Reviewers score procedural accuracy and political neutrality separately. Democracy Reporting International and Factum Long-form factuality Everyday Not in cards May 2025
ST-VQA Models answer questions by reading text in a scene. The benchmark compares each answer with the text required by the question. ST-VQA team Documents, charts and text in images School Not in cards May 2019
Stanford Cars Models identify a car's make, model, and year from a photograph. The benchmark measures fine-grained vehicle recognition. Stanford University General visual questions University Not in cards Dec 2013
Stanford Online Products Models retrieve photographs of the same product. The test compares learned image features on products outside the training set. Hyun Oh Song and colleagues General visual questions Everyday Not in cards Nov 2015
STAR situated video reasoning Models answer multiple-choice questions about how people interact with objects in videos. The questions cover interactions, sequences, predictions, and feasibility. Bo Wu and colleagues Video understanding Everyday Not in cards
STaRK Retrieval systems find the right items for complex queries over knowledge bases that mix free text with relations between entities. The three domains are product search, academic paper search and precision medicine. Most queries are synthesized and checked by people, and a smaller set is written by people. Stanford University (SNAP group) and collaborators Questions that need search University Not in cards Apr 2024
StateBench Models follow variables through long runs of assignments and report a final value. Four tiers share one source trace and differ only in the update: fixed values, reassignment, self-referential updates, and assignment by reference. Problems come in several formats, such as code, spreadsheet edits and people moving between rooms, at lengths from under 2K to over 2M tokens. Ai2 Reasoning over long documents Everyday Not in cards Nov 2026
States United voter questions ChatGPT and Google answer voter questions. Researchers check factual claims against official sources and assess the completeness of written answers. States United Democracy Center Long-form factuality Everyday Not in cards Aug 2026
SteerBench Speech models respond to spoken instructions about tone, persona, style, and speed. A judge checks their speech and words against binary rubrics. Scale AI and the University of Maryland Audio understanding and voice Everyday Not in cards Sep 2026
Step-Caption Audio models describe speakers' vocal characteristics across language, age, rate, rhythm, pitch, timbre, emotion and related attributes. A judge compares the description with expert labels. StepFun Audio understanding and voice University Not in cards Apr 2026
Step-DU Audio models answer targeted questions about a speaker's vocal characteristics in dialogue. A judge scores answers against reference labels. StepFun Audio understanding and voice University Not in cards Apr 2026
Step-SPQA Audio models answer paralinguistic questions about age, emotion, events, gender, pitch, rhythm, scene, speed, voice sounds and style. StepFun Audio understanding and voice University Not in cards Apr 2026
StepEval-Audio-360 Speech models answer multi-turn Chinese prompts about instruction following, voice understanding, reasoning, role play, creativity and speech control. StepFun Audio understanding and voice University Not in cards Feb 2025
StepUp gym attendance forecasts Models predict how each four-week exercise program changes weekly gym visits relative to a placebo. Researchers compare predicted effects with recorded gym check-ins. Katherine Milkman and the Behavior Change for Good Initiative Social simulation University Not in cards Dec 2021
StereoSet Models choose between stereotypical, anti-stereotypical and unrelated completions in four domains: gender, profession, race and religion. Scores combine language modeling and stereotype preference. MIT, Intel AI and McGill University Bias, fairness and politics Everyday Not in cards Apr 2020
StereoTales Models write multilingual stories about characters with controlled demographic attributes. Statistical tests find overrepresented associations, and people judge their harmfulness. Giskard AI Bias, fairness and politics Everyday Not in cards May 2026
STL-10 Models classify images into ten object categories using limited labeled training data. Adam Coates and colleagues General visual questions Everyday Not in cards Jun 2011
STORAL Models choose which moral fits a Chinese or English story. Two generation tasks ask models to write a moral from a story or illustrate a moral. Tsinghua University Reading comprehension School Not in cards Apr 2022
StreamEval Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing. StreamEval authors Video generation Everyday Not in cards Oct 2026
StreamingQA Models answer time-stamped questions about news. The test measures how well they acquire new information while retaining older knowledge. Google DeepMind Questions that need search Everyday Not in cards May 2022
Structured Web Data Extraction Models extract named attribute values from web pages. The benchmark compares their outputs with annotated values across website categories. Qiang Hao and colleagues Faithful to sources University Not in cards Jul 2011
StyleInstructCaps Models describe a speaker's voice according to different instructions. The evaluator grades instruction following, accuracy about the recording, and consistency across the speaker's clips. Johns Hopkins University and The University of Texas at Dallas Speech recognition and translation Everyday Not in cards Oct 2026
Sugar’s Kitchen AI Cooking Battle 17 Models design a summer lunch for a fictional Japanese inn. ChatGPT awards up to 40 points for the written proposal, and the organizer awards ten. Sugar (シュガ) Taste and style Everyday Not in cards Sep 2026
Sumi Japanese PII Corpus Models locate personal information in Japanese business templates and prose. The release includes 2,000 template test documents and 520 documents outside those templates. NagaYu Reading comprehension Everyday Not in cards Sep 2026
Summ-it Coreference Models group Portuguese noun phrases that refer to the same entity in science articles from Folha de São Paulo. Sandra Collovini and colleagues Reading in many languages University Not in cards Jul 2007
SummScreenFD Models write a recap of a TV episode from its full transcript. Plot details sit in dialogue scattered across the episode, so the model must find and join them. Scores compare outputs with human-written recaps using ROUGE. Toyota Technological Institute at Chicago and collaborators Reasoning over long documents School Not in cards Apr 2021
SUMO Web Claims Summarization Models write a short headline summarizing a fact-checking article. HELM uses articles about climate claims. Rahul Mishra and colleagues Faithful to sources School Not in cards Nov 2020
SUN397 Models recognize the scene category in indoor and outdoor photographs. The classification task covers 397 categories. Jianxiong Xiao and colleagues General visual questions Everyday Not in cards Jun 2010
Sunbird African Language Technology Models translate parallel sentences between English and six languages spoken in Uganda and East Africa. Sunbird AI Translation School Not in cards
SUPERB Self-supervised speech models are tested as frozen encoders on a range of speech tasks, including speech recognition, keyword spotting, speaker identification, intent classification and emotion recognition. Each task trains only a light prediction head on top of the shared model. National Taiwan University and collaborators Speech recognition and translation Everyday Not in cards May 2021
SuperCLUE-AISearch Search systems answer questions in Chinese. SuperCLUE Questions that need search Everyday Not in cards
SuperCLUE-Coder Coding assistants complete programming tasks in Chinese. SuperCLUE Building whole projects University Not in cards
SuperCLUE-CPIFOpen Models answer Chinese prompts with precise instructions. The published evaluation code checks their responses. SuperCLUE team Following precise instructions Everyday Not in cards Jul 2025
SuperCLUE-DeepResearch Agents research questions and assemble reports in Chinese. SuperCLUE Deep research University Not in cards
SuperCLUE-LarkClaw Claw assistants complete tasks through Feishu. SuperCLUE Apps and MCP servers Everyday Not in cards
SuperCLUE-MedAssist Medical assistants respond to Chinese healthcare questions. SuperCLUE Medicine Expert Not in cards
SuperCLUE-Office Agents process documents, analyze data, and write office materials. SuperCLUE Documents and spreadsheets University Not in cards
SuperCLUE-R1 Search DeepSeek R1 services answer questions that require search and reasoning. SuperCLUE Questions that need search University Not in cards
SuperCLUE-Voice (audiovisual interaction) Models respond to audio and video inputs in real time. SuperCLUE Audio understanding and voice Everyday Not in cards
SuperCLUE-WeClaw Claw assistants complete tasks through WeChat. SuperCLUE Apps and MCP servers Everyday Not in cards
SuperCLUE-XClaw Claw assistant products complete user tasks in Chinese. SuperCLUE General assistants Everyday Not in cards
SuperLim 2.0 Models complete 15 Swedish language tasks across 14 datasets. Tasks include recognizing arguments, choosing synonyms, and summarizing newspaper articles. Språkbanken Text, AI Sweden, and the National Library of Sweden Reading in many languages School Not in cards Sep 2022
SuperSim Models give separate scores for how similar two Swedish words are and how closely they are associated. Five Swedish speakers rated each pair. Simon Hengchen and Nina Tahmasebi, University of Gothenburg Reading in many languages Everyday Not in cards Apr 2021
Surge AI blind coding evaluation Surge AI raters blindly score model coding outputs on a 1-5 scale. Mistral reports five models. Surge AI (commissioned by Mistral) Building whole projects University Not in cards Oct 2026
SUST Bangla Emotional Speech Corpus Models identify which of seven emotions a speaker acts out in a Bangla sentence. Shahjalal University of Science and Technology Audio understanding and voice Everyday Not in cards Feb 2021
SV-Gap Reset Release Agents write SystemVerilog control circuits from eight specifications. The evaluator tests functional behavior and checks whether each circuit obeys its reset wiring requirement. Shraddha Sridhar Electronics University Not in cards Jul 2026
SVAMP Models solve 1,000 grade-school arithmetic word problems. The problems are variations of problems from earlier datasets, made to catch solvers that ignore the question or the word order. The final numeric answer is scored by exact match. Microsoft Research India (Patel, Bhattamishra, Goyal) Word problems School Not in cards Mar 2021
SVG-Bench (StarVector) Models generate SVG code from images or text. The RLRF port uses the SVG-Stack image-to-SVG task and rewards the quality of the rendering. StarVector authors Image generation and editing University Not in cards Dec 2023
SWAG Models read a partial description of an everyday activity. They choose the most plausible continuation from four options. Rowan Zellers and colleagues Commonsense Everyday Not in cards Aug 2018
SWE-bench Models resolve 2,294 real GitHub issues from 12 Python repositories. A patch counts as resolved when the hidden repository tests pass. Princeton University (SWE-bench team) Fixing real repositories Expert Not in cards Oct 2023
SWE-bench Lite Models resolve 300 self-contained GitHub issues chosen from SWE-bench to cut evaluation cost. Hidden repository tests decide each patch. Princeton University (SWE-bench team) Fixing real repositories Expert Not in cards Mar 2024
SWE-Bench Pro Verified Agents repair software repositories. OpenCompass corrects inconsistent tasks and strengthens tests in SWE-Bench Pro. OpenCompass and collaborators Fixing real repositories Expert Not in cards Sep 2026
SWE-gen JS Agents fix issues drawn from 1,000 merged pull requests across 30 JavaScript and TypeScript repositories. Each task includes tests that fail before the reference fix. Abundant AI Fixing real repositories Expert Not in cards Jan 2026
SweDiagnostics Models judge entailment between Swedish sentences that test phenomena such as negation, quantifiers, and ambiguity. Felix Morger, University of Gothenburg Reasoning in many languages School Not in cards Jun 2021
Swedish Analogy Test Set Models complete a Swedish word analogy by predicting the fourth word from the first three. Relations include grammar and geography. Tosin Adewumi, Foteini Liwicki, and Marcus Liwicki Reading in many languages School Not in cards Jul 2020
SweFAQ 2.0 Models match questions from Swedish public agencies to answers from the same subject category. Språkbanken Text, University of Gothenburg Reading in many languages Everyday Not in cards Sep 2022
SweNLI Models decide whether Swedish premises entail a statement, contradict it, or leave it undecided. Språkbanken Text, University of Gothenburg Reasoning in many languages School Not in cards Feb 2023
SweParaphrase 2.0 Models rate meaning overlap between Swedish sentences on a scale from zero to five. Dana Dannélls, University of Gothenburg Reading in many languages Everyday Not in cards Oct 2022
SweSAT Synonyms 1.1 Models choose one of five Swedish synonyms or definitions. The official score adjusts accuracy for random guessing. Språkbanken Text, University of Gothenburg Reading in many languages School Not in cards Mar 2023
SweWiC Models decide whether a highlighted Swedish word has the same meaning in two sentences. Gerlof Bouma, University of Gothenburg Reading in many languages Everyday Not in cards Jun 2021
SweWinogender 2.0 Models judge whether a Swedish pronoun refers to a person in a stated occupation. The parity score checks agreement across three pronoun variants. Språkbanken Text, University of Gothenburg Bias, fairness and politics Everyday Not in cards Feb 2023
SweWinograd 2.0 Models decide whether a highlighted Swedish pronoun refers to a proposed person or thing in the passage. Språkbanken Text, University of Gothenburg Reading in many languages Everyday Not in cards Feb 2023
Swiss Job Market Monitor ICT Term Recognition Models mark words for information and communication technology in German-language Swiss job ads. The original test holds out 10% of the annotated ads. University of Zurich Other professions University Not in cards May 2022
Swiss-Judgment-Prediction Models predict whether the Swiss Federal Supreme Court accepts or dismisses a case. Joel Niklaus and colleagues Law Expert Not in cards Oct 2021
SWM-Bench (Social World Models) Predict how Kalshi and Polymarket belief trajectories shift after news events. 12k+ points across politics, finance and crypto, chronologically split. UIUC (ulab-uiuc) Social simulation Expert Not in cards Jun 2026
Sydney-Captions Models describe land use in aerial images of Sydney. The dataset pairs 613 images with five human captions each. Bo Qu, Xuelong Li, Dacheng Tao, and Xiaoqiang Lu Detection and captioning Everyday Not in cards Jul 2016
SympTEMIST Models locate symptoms, signs, and findings in Spanish clinical cases. They also map given mentions to SNOMED CT codes in Spanish and five additional languages. Barcelona Supercomputing Center Medicine University Not in cards Jul 2023
SYNERGY+ Models rank research records against a systematic review's eligibility criteria. Reviewers' final inclusion decisions supply the reference labels. Utrecht University and collaborators Medicine University Not in cards Aug 2026
SynLogic Models solve synthetic reasoning problems, including Sudoku, arithmetic puzzles, and mazes. The port uses official task graders on separate easy and hard validation sets. MiniMax Language reasoning School Not in cards May 2025
Synthetic Contextual ASR Benchmark (Indic) Models transcribe a synthetic caller’s response with or without the preceding conversation. A language model scores whether each transcript preserves the caller’s intent and named entities. Sarvam AI Speech recognition and translation Everyday Not in cards Feb 2026
Szeged NER Models identify named entities in Hungarian business news from the Hungarian News Agency. György Szarvas and colleagues Reading in many languages School Not in cards May 2006
T2Ranking Models rank Chinese passages by their relevance to a query. Xiaohui Xie and colleagues Reading in many languages School Not in cards Jul 2023
TabFact Models decide whether a statement is entailed or refuted by a Wikipedia table. Classification accuracy measures agreement with human labels. UC Santa Barbara and Tencent AI Lab Faithful to sources Everyday Not in cards Sep 2019
TAC 2017 Adverse Drug Reaction Extraction from Drug Labels Models extract adverse reactions and their modifiers from prescription drug labels. They identify positive reactions and map them to standard medical terms. The organizers held out 99 labels. Text Analysis Conference and the US National Library of Medicine Medicine University Not in cards Apr 2017
TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. Shanghai Jiao Tong University and collaborators Simulated robots Everyday Not in cards Oct 2026
TailOR Models generate images and videos of regular, unconventional, and impossible tool interactions. Annotators score instruction adherence, interaction accuracy, physical realism, and perceptual quality. University of Illinois Urbana-Champaign, Stanford University, and Columbia University Physical prediction Everyday Not in cards Jun 2026
Taiwan Mandarin Speech with Video: Speech Enhancement Models use lip video to remove noise from Mandarin sentences. Researchers compare the enhanced signal with clean speech for quality and intelligibility. Shang-Yi Chuang and colleagues, Academia Sinica and KU Leuven Speech recognition and translation University Not in cards May 2020
Taiwan National Dental Technician Licensing Examination 2023 Models answer 194 text-only dental technology questions from Taiwan's 2023 licensing exam. The authors score accuracy at baseline and after each of three weeks. Ching-Yi Huang, Yi-Pang Lee, Andy Sun, and Chun-Pin Chiang Medicine University Not in cards Jul 2025
TalentCLEF 2025 Multilingual Job Title Matching Models rank job titles by how closely they match a query title within or across languages. Organizers score mean average precision. Avature and UNED Other professions Everyday Not in cards Sep 2024
TalentCLEF 2026 Contextualized Job-Person Matching Models rank candidate resumes for a job description in English or Spanish. Organizers score retrieval within and across languages and compare rankings across gender variants. Avature, TechWolf, and UNED Other professions University Not in cards Nov 2025
TalentCLEF 2026 Job-Skill Matching with Skill Type Classification Models rank skills for an English job title. The main ranking metric gives core skills more weight than contextual skills. Avature, TechWolf, and UNED Other professions University Not in cards Nov 2025
Talking Points: When Chatbots Surface Russian State Media Chatbots answer questions about Ukraine in five languages. Analysts check whether each response cites an official Russian channel, a state-controlled outlet, or a state-linked source. Institute for Strategic Dialogue Faithful to sources Everyday Not in cards Oct 2025
TamperBench Open-weight models undergo fine-tuning and representation tampering attacks. Safety and utility metrics measure tamper resistance across attack and model pairs. FAR.AI and Critical ML Lab Jailbreaks and prompt injection Expert Not in cards Feb 2026
TAP-Vid Models track individual points across frames and predict when each point disappears. The benchmark compares their trajectories and visibility predictions with reference tracks. Google DeepMind Video understanding Everyday Not in cards Nov 2022
TarantuBench Agents complete simulated web security challenges. Each challenge has a binary success criterion. Trivulzianus Capture-the-flag challenges Expert Not in cards Apr 2026
TartanAir v2 Models reconstruct camera motion and scene geometry from simulated trajectories. The evaluation compares predictions with known poses and depths. TartanAir team Spatial reasoning and counting School Not in cards
Task AW IT support Agents complete five support workflows in simulated IT systems. Task checks cover onboarding, offboarding, and cleanup of accounts and assets. abhishek203 Customer service University Not in cards Jul 2026
TAT-QA Models answer 16,552 questions about 2,757 passages from real financial reports that mix a table with text. Many answers need arithmetic, counting or comparison. Scored by exact match and F1. National University of Singapore (NExT++) and collaborators Finance University Not in cards May 2021
Tatoeba Translation Challenge Models translate sentences for many language pairs. The challenge publishes held-out test sets so systems can report comparable translation scores. Helsinki NLP Translation University Not in cards Nov 2020
TAU Urban Acoustic Scenes 2022 Mobile Models identify an urban acoustic scene from a one-second recording made on a mobile device. Toni Heittola and colleagues Audio understanding and voice Everyday Not in cards Mar 2022
Tau2 Bench Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. Snorkel AI Customer service University Not in cards Nov 2025
Tau2 Bench Verified Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. Snorkel AI Customer service University Not in cards Dec 2025
Tavily Company Search Benchmark Search providers return evidence for company research questions, and an answer agent responds from those results. A judge scores answer completeness, relevance and freshness. Tavily Questions that need search Expert Not in cards Oct 2026
TaxaBench-8k Models identify species or match observations across photographs, sounds, and other field data. The test contains 8,813 observations paired across six types of data. Washington University in St. Louis Genomics and life sciences Expert Not in cards Oct 2024
TaxoBench Agents find research papers and organize them into a taxonomy. The evaluator compares their groupings and hierarchy with taxonomies from 72 published surveys. Fudan University and Tencent Hunyuan Deep research Expert Not in cards Jan 2026
Te Hiku Media Māori speech recognition golden dataset Speech models transcribe te reo Māori recordings from native speakers and proficient second-language speakers. Te Hiku Media compares their word error rates. Te Hiku Media Speech recognition and translation Everyday Not in cards Jan 2023
TE-ca Models decide whether a Catalan hypothesis follows from a premise. The reference labels identify the inference relation. Barcelona Supercomputing Center Reading in many languages School Not in cards Jul 2021
TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. Amazon Alexa AI Simulated robots Everyday Not in cards Oct 2021
Teaching Monster Challenge Agents generate instructional videos for a specified topic and learner. A model screens submissions, viewers compare videos in pairs, and an expert panel sets the final ranking. National Taiwan University and collaborators Other professions University Not in cards Mar 2026
TED-LIUM Models transcribe English TED talks. The reference transcripts support comparisons by word error rate. LIUM Speech recognition and translation Everyday Not in cards
Terminal-Bench Challenges Agents build a large project end to end with no time limit, such as a faster Rust compiler or a WebGL renderer in WebAssembly. Each challenge has its own test bed and metric. Terminal-Bench team (Stanford, Harbor and Laude Institute) Long terminal sessions Research frontier Not in cards Jun 2026
TerraFirm REIT Models choose which properties to sell from a portfolio of 12 office buildings. The evaluator checks whether the remaining portfolio meets five debt and valuation constraints. Patrick Browne Finance Expert Not in cards Feb 2026
TerretaQA Models answer open-domain questions in Valencian about the history, geography, tourism, and culture of the Valencian Community. University of Alicante Knowledge in many languages School Not in cards Oct 2026
Testing Relational Understanding in Text-Guided Image Generation Models depict relationships between named entities. Human participants judge whether the generated scenes match each caption. Colin Conwell and colleagues Image generation and editing Everyday Not in cards Jul 2022
Text2CAD Models generate parametric CAD designs from text prompts at different levels of detail. The authors compare generated geometry with the reference and count invalid designs. Mohammad Sadil Khan and colleagues CAD models University Not in cards Sep 2024
TGIF-QA Models answer questions about actions, repetitions, and changes over time in animated clips. Yunseok Jang and colleagues Video understanding Everyday Not in cards Apr 2017
The GenderQueer Test Suite Models translate 331 English passages into Icelandic. The graders check pronouns, adjective agreement and terms for LGBTQIA+ people. University of Iceland Translation Everyday Not in cards Sep 2024
The Impossible Refund Assistants pursue a late train refund in a simulated email environment. Separate judges rate deception and other behaviors from the messages. Vals AI Deception and alignment Everyday Not in cards Oct 2026
The Met Dataset Models match photographs of artworks to catalog images from the Metropolitan Museum of Art. The evaluation measures instance recognition across photographic conditions. Nikolaos-Antonios Ypsilantis and colleagues General visual questions Everyday Not in cards
The Midnight Index Models lead a nation through fixed scenarios in a strategy game. The simulation records outcomes, escalation, and whether public commitments match later decisions. Clouthier Labs Deception and alignment Everyday Not in cards Oct 2026
The Silicon Sample Benchmark Models predict how 16 messages affect trust in climate scientists and related outcomes. Organizers compare predictions with responses from a US experiment. Jan Pfänder and the Silicon Sample Benchmark team Social simulation University Not in cards Jun 2026
The Turtle Test Models remember a conditional instruction during unrelated conversation. They must apply it when triggered and avoid applying it on earlier turns. mohi_achu Multi-turn conversation Everyday Not in cards Oct 2026
TheGoldman Retrieval Models retrieve the Goldman Sachs dictionary definition for a financial term. The score rewards placing the correct definition near the top of the first ten results. The Hong Kong University of Science and Technology Finance Everyday Not in cards Jul 2024
Third PASCAL Recognizing Textual Entailment Challenge Models decide whether a passage entails a proposed statement. The challenge compares their decisions with human labels. Danilo Giampiccolo and colleagues Reading comprehension School Not in cards Jun 2007
TIME (time series forecasting) Models forecast time series over short, medium, and long horizons without training on the target task. Real-TSF Real-world events University Not in cards Feb 2026
tinyBenchmarks Models answer small selected subsets of established benchmarks. Item response estimates approximate accuracy on the full tests. Felipe Maia Polo and colleagues Composite indices University Not in cards Feb 2024
TituLLMs Bangla Evaluation Suite Models answer Bangla questions on academic subjects, science and everyday reasoning. The suite also asks yes-or-no questions about passages. Hishab (TituLLMs team) Knowledge in many languages University Not in cards Feb 2025
TMHINT-QI version II v2 Models predict how clear Mandarin speech sounds and how many words listeners understand. The test adds street noise and two previously unseen enhancement systems. Academia Sinica and VoiceMOS collaborators Audio understanding and voice Everyday Not in cards May 2023
TOEFL11 Essay Proficiency Scoring Models rate the English proficiency of each essay as low, medium, or high. Researchers measure agreement with the human ratings using quadratic weighted kappa. Educational Testing Service Other professions University Not in cards Jan 2013
TokSuite Models complete questions under alternative tokenizations. The benchmark measures the change in accuracy relative to the canonical representation. Gül Sena Altıntaş and colleagues Reasoning in many languages School Not in cards Dec 2025
ToLD-Br Models identify toxic language in Brazilian Portuguese posts using human annotations. João Augusto Leite and colleagues Bias, fairness and politics Everyday Not in cards Oct 2020
Toloka Visual Question Answering Benchmark Models answer questions about images by locating the object that answers each question. The benchmark compares predicted and reference bounding boxes. Toloka Spatial reasoning and counting Everyday Not in cards
ToolBench Models plan and call real REST APIs from RapidAPI, over 16,000 of them, to answer instructions that need one or many tools. ToolEval judges pass rate and win rate. OpenBMB and Tsinghua University Function calling Everyday Not in cards Jul 2023
ToolRet Models retrieve relevant tools for 7,600 requests from a corpus of about 43,000 tools. Evaluation measures ranking quality, precision, recall, and complete tool-set retrieval. Shandong University, Baidu, and Leiden University Function calling University Not in cards Mar 2025
TopiOCQA Models retrieve evidence and answer follow-up questions in conversations that change topics. Vaibhav Adlakha and colleagues Questions that need search School Not in cards Oct 2021
ToxicChat Models identify toxic user messages from real conversations with an online chatbot. Zi Lin and colleagues Deception and alignment School Not in cards Oct 2023
TraceSpatial-Bench Models plan an object-centric 3D trace, an ordered list of metric 3D waypoints, from language instructions in real cluttered indoor scenes with depth. Beihang University + Peking University + BAAI Embodied reasoning Everyday Not in cards Dec 2025
Trading Bot Bench Models write Python trading strategies for historical stock prices. The evaluator combines return, Sharpe ratio, and drawdown on held-out scenarios. amrita Finance University Not in cards Sep 2025
Transformer Puzzles (RASPy) Models complete eight sequence-programming puzzles in RASPy. The hub allows up to eight attempts per puzzle and checks solutions with executable tests. Sasha Rush Function-level problems University Not in cards Nov 2022
Transluce Mental Health Evaluation Models hold multi-turn conversations with simulated users facing mental-health scenarios. Automated judges score 14 helpful and harmful behaviors. Transluce Wellbeing and sycophancy Expert Not in cards Aug 2026
Transluce User Awareness Evaluation Models answer behavior, capability, grading and borderline-request tasks under different simulated user identities. Scores measure identity-conditioned behavior shifts. Transluce Safety and alignment Expert Not in cards Aug 2026
TravelPlanner Agents plan multi-day trips from a user request, using tools to search nearly four million records of flights, hotels, restaurants and attractions. Each plan must meet the user's hard constraints, such as budget, and commonsense rules. The headline final pass rate counts a plan only when it meets every constraint. Ohio State University (OSU NLP Group) and Fudan University General assistants Everyday Not in cards Feb 2024
TREC 2017 LiveQA Medical Task Models answer consumer health questions. The shared task compares responses with expert assessments of answer quality. Asma Ben Abacha and colleagues Medicine Expert Not in cards
TREC 2023 Clinical Trials Track Models rank trials for 50 simulated patients using answers to medical intake questionnaires. Some answers can be missing. TREC Clinical Trials Track organizers Medicine Expert Not in cards May 2023
TREC Million LLMs 2025 Systems rank simulated expert language models for a new question using their answers to earlier questions. Each expert has access to a different document cluster. NIST TREC Million LLMs Track Questions that need search University Not in cards Nov 2024
TREC Product Search 2025 Models rewrite shopping queries for a search engine. Automatic entries submit one reformulation. Interactive entries can submit four, and the best one counts. NIST TREC Product Search and Recommendation Track Questions that need search Everyday Not in cards Mar 2025
TREC Related Product Recommendation 2025 Models rank items that complement or substitute for a given product. Assessors label the type and strength of each relationship. NIST TREC Product Search and Recommendation Track Questions that need search Everyday Not in cards Mar 2025
TREC-COVID Models rank scientific articles that address specific information needs about COVID-19. National Institute of Standards and Technology and collaborators Questions that need search Expert Not in cards Apr 2020
TREK-150 Models follow a marked object through first-person kitchen videos. In the one-pass test, they receive its bounding box in the first frame and track it to the end. University of Udine and University of Catania Video understanding Everyday Not in cards Aug 2021
Trialstreamer clinical evidence extraction Models identify treatments, comparators, outcomes, and the direction of each reported effect in trial abstracts. Physicians annotated 100 test abstracts. Northeastern University and collaborators Medicine University Not in cards Jan 2020
TriScore Models identify composers and instruments from recordings, score images, and symbolic notation. They also retrieve excerpts from the same composition. University of Alicante Audio understanding and voice University Not in cards Apr 2026
Trivia-et Models answer Estonian trivia questions from the Eesti Mäng board game. TalTechNLP Knowledge in many languages Everyday Not in cards Dec 2024
TriWorldBench World models generate synchronized robot-view videos from head and wrist cameras. Nineteen signals measure cross-view agreement, task alignment, physical coherence, motion, temporal stability, and visual quality. Peking University, Tsinghua University, Beihang University and others Video world models Everyday Not in cards Sep 2026
TruthInsightBench Agents analyze data from published studies without seeing the original conclusions. A judge grades whether the evidence and artifacts support the claims each agent makes. TruthInsight-AI Research science Expert Not in cards Sep 2026
Turkish Dental Specialty Examination 2020 and 2021 Models answer Turkish dental specialty entrance questions in basic and clinical sciences. The two official answer keys cover 238 scored questions, including questions with images. ÖSYM Medicine Expert Not in cards Jan 2025
Turku NER Corpus Models identify named entities in Finnish text from the Universal Dependencies corpus. Jouni Luoma and colleagues Reading in many languages School Not in cards May 2020
Turku Paraphrase Corpus Models compare Finnish sentence pairs and estimate how closely their meanings match. Jenna Kanerva and colleagues Reading in many languages School Not in cards May 2021
TUT Urban Acoustic Scenes 2018 Models recognize the setting of a ten-second audio clip recorded in a European city. Annamaria Mesaros and colleagues Audio understanding and voice Everyday Not in cards Apr 2018
TV-AD Models describe what viewers can see during clips from television episodes. The test pairs each clip with a timed reference description. University of Oxford, École des Ponts and Shanghai Jiao Tong University Video captioning Everyday Not in cards Jul 2024
tweetSentBR Models classify the sentiment of Brazilian Portuguese social media posts as positive, neutral, or negative. Henrique Brum and Maria das Graças Volpe Nunes Reading in many languages Everyday Not in cards May 2018
Twitter Financial News Sentiment Models classify financial news posts as bearish, bullish, or neutral. The released validation file contains 2,388 posts. zeroshot Finance Everyday Not in cards Sep 2022
Twitter URL Paraphrase Corpus Models identify paraphrases among pairs of social media posts that reference the same URL. Wuwei Lan and colleagues Reading comprehension School Not in cards Sep 2017
Two-Hop Latent Reasoning Models answer questions that require composing two facts. The hub port covers the semi-synthetic tasks and includes a shuffled control. Mikuláš Balesni, Tomasz Korbak and Owain Evans Language reasoning University Not in cards Nov 2024
Typed Decision Bench Models assign probabilities to decisions about a given state. The benchmark scores accuracy and calibration across five suites. Blobfish AI Running a business Expert Not in cards Sep 2026
UAV Operator Models manage simulated drone missions as weather and flight restrictions change. The simulator scores their decisions from logged aircraft and mission state. jarrett Simulated robots Expert Not in cards Jul 2026
UAV Robust Control Hub Agents construct policies for a fictional civilian drone simulator. The benchmark evaluates policies on hidden scenarios and reports separate results under changed conditions. andehit Simulated robots University Not in cards Nov 2026
UCF101 Models recognize one of 101 human actions in video clips. University of Central Florida Video understanding Everyday Not in cards Dec 2012
UCI Adult Dataset (Project Moonshot) Models choose an income range or say that income cannot be determined from a census profile. Moonshot measures exact matches with the expected response. AI Verify Foundation Bias, fairness and politics Everyday Not in cards May 2024
UCM-Captions Models describe aerial land use scenes from the UC Merced collection. The dataset pairs 2,100 images with five human captions each. Bo Qu, Xuelong Li, Dacheng Tao, and Xiaoqiang Lu Detection and captioning Everyday Not in cards Jul 2016
UD Ancient Greek Perseus Models identify dictionary forms, parts of speech, and grammatical relationships in Ancient Greek. The version 2.10 test contains 1,306 sentences from Athenaeus. Tufts University and Leipzig University Reading in many languages University Not in cards Nov 2015
UD Ancient Greek PROIEL Models analyze Ancient Greek passages from Herodotus and the New Testament. They identify dictionary forms and grammatical relationships between words. University of Oslo Reading in many languages University Not in cards Nov 2015
UD Ancient Greek PTNK Models tag and parse the Greek Septuagint. The test contains 410 verses from chapters 19 through 30 of Genesis. Daniel Swanson, Bryce Bussert, and Francis Tyers Reading in many languages University Not in cards Nov 2023
UD Ancient Hebrew PTNK Models analyze word forms and grammatical relationships in the Hebrew Bible. Version 2.17 tests 410 verses from Genesis and 142 from Deuteronomy. Daniel Swanson and Francis Tyers Reading in many languages University Not in cards May 2022
UD Latin CIRCSE Models tag and parse Latin drama and prose. The version 2.14 test contains two tragedies by Seneca and Germania by Tacitus. CIRCSE Research Centre, Università Cattolica del Sacro Cuore Reading in many languages University Not in cards May 2024
UD Latin ITTB Models analyze medieval Latin prose by Thomas Aquinas. They identify the grammatical form of each word and its relationship to other words. Index Thomisticus Treebank team Reading in many languages University Not in cards Nov 2015
UD Latin LLCT Models tag and parse the Latin of legal charters from Tuscany. The source documents date from 774 to 897. Timo Korkiakangas, Flavio Massimiliano Cecchini, and Marco Passarotti Reading in many languages University Not in cards May 2020
UD Latin Perseus Models analyze word forms and grammatical relationships in Latin literary texts. Version 2.13 reserves 939 sentences across four source files for testing. Tufts University and Leipzig University Reading in many languages University Not in cards Nov 2015
UD Latin PROIEL Models analyze Latin sentences from the Vulgate and from authors including Caesar and Cicero. The test includes both biblical and secular prose. University of Oslo Reading in many languages University Not in cards Nov 2015
UD Latin UDante Models tag and parse Dante's Latin. All 419 test sentences in version 2.13 come from his treatise on the vernacular, De vulgari eloquentia. University of Pisa and Università Cattolica del Sacro Cuore Reading in many languages University Not in cards May 2021
Uganda Cultural and Cognitive Benchmark Models answer questions about Ugandan daily life, institutions and cultural practices. A grader checks answers across 24 domains. Crane AI Labs Trivia and open questions Everyday Not in cards Jul 2025
UI-Vision Computer use agents are tested offline on human desktop demonstrations from 83 applications. Three tasks go from element grounding to layout grounding to next-action prediction for clicks, drags and key presses. ServiceNow Research and collaborators Finding things on screen Everyday Not in cards Mar 2025
UI-Vision Action Prediction Models predict the next desktop interface action. The score is the published step success rate. ServiceNow Research and collaborators Finding things on screen Everyday Not in cards Mar 2025
UIT-ViCTSD Models classify Vietnamese comments for constructiveness and toxicity. The two labels capture different properties of the same text. Son T. Luu and colleagues Reading in many languages Everyday Not in cards Mar 2021
UIT-ViHSD Models classify Vietnamese social media text into three categories covering clean language, offensive language, and hate speech. Son T. Luu, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen Reading in many languages Everyday Not in cards
UIT-VSFC Models classify sentiment in Vietnamese student feedback. The corpus also includes topic labels. Kiet Van Nguyen and colleagues Reading in many languages Everyday Not in cards Nov 2018
UK Dermatology SCE Sample Questions Models answer multiple-choice questions from the Federation’s dermatology practice exam. Researchers score answers against the official key. Federation of the Royal Colleges of Physicians of the UK Medicine Expert Not in cards Jun 2023
ULSA synthesis action retrieval Models identify and classify actions in sentences about inorganic materials synthesis. The corpus contains 3,040 labeled sentences from 535 paragraphs. University of California, Berkeley and Lawrence Berkeley National Laboratory Lab and data work University Not in cards Oct 2021
Unhealthy Comments Corpus Models rate newspaper comments for overall conversational health and six attributes, including hostility and sarcasm. Researchers score each label separately using ROC AUC. Ilan Price and colleagues Reading comprehension Everyday Not in cards Aug 2020
Unidentified Video Objects Models segment and track objects in video without a fixed list of object categories. The evaluation compares predicted masks and tracks with annotated instances. UVO team Video understanding Everyday Not in cards Apr 2021
Unit Commitment Lite Models schedule generators to serve hourly demand at minimum cost. The grader checks whether schedules meet operating constraints and compares cost with the optimum. jwilksbooth Electronics Expert Not in cards Jul 2026
Universal Dependencies Sanskrit Vedic Models identify which word governs each word in Vedic Sanskrit and label the grammatical relation. The official split keeps each hymn or prose chapter together. University of Zurich and Universal Dependencies contributors Reading in many languages University Not in cards May 2020
UNT Computer Science Short Answer Dataset 2.0 Models grade 2,273 student answers from a computer science course against the average of two human grades. The authors report root mean square error and correlation. University of North Texas Other professions University Not in cards Apr 2011
UQ Models answer 500 unsolved Stack Exchange questions. A model checks answers without a reference solution, and community members review selected answers. UQ project Expert-written questions Research frontier Not in cards Aug 2025
UrbanSED Models locate sound events in synthetic urban soundscapes. The benchmark includes reference labels for the times and types of events. Justin Salamon and colleagues Audio understanding and voice Everyday Not in cards Oct 2017
Urdu Language Speech Emotional Corpus Models identify emotions in excerpts from Urdu television talk shows. The recordings cover 38 speakers and four emotion categories. Siddique Latif, Adnan Qayyum, Muhammad Usman and Junaid Qadir Audio understanding and voice Everyday Not in cards Jun 2018
UrduSER Models identify emotions in Urdu dialogue from Pakistani television. The corpus contains 3,500 excerpts from ten actors across seven emotion categories. Muhammad Zaheer Akhtar and colleagues Audio understanding and voice Everyday Not in cards Dec 2024
URGENT 2024 Models enhance speech under different forms of degradation. NVIDIA measures how accurately a recognizer transcribes the evaluation audio. URGENT Challenge organizers Speech recognition and translation Everyday Not in cards
US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. Google Medicine Expert Not in cards Jul 2025
UW Clinical Record Section Chunker i2b2 Discharge Summary Corpus Models locate section boundaries in discharge summaries and label each section. Researchers compare the predicted labels and boundaries with human annotations. University of Washington and Microsoft Research Medicine University Not in cards May 2012
UWBench Models answer questions about underwater scenes, write captions, and locate objects from descriptions. Questions cover visibility, materials, and spatial relationships. Institute of Artificial Intelligence, China Telecom General visual questions Everyday Not in cards Oct 2025
Uyghur Language Question Answering Models answer crowdsourced Uyghur language and literature questions. The suite includes reading comprehension, language exercises and exam questions at different proficiency levels. keramjan (Uyghur evaluation datasets) Reading in many languages School Not in cards Jan 2026
Uzbek Open ASR Leaderboard Models transcribe Uzbek recordings from four test sets. The main score averages the four word error rates with equal weight. NavAI Speech recognition and translation Everyday Not in cards Jul 2026
Uzbek Speech Corpus Models transcribe held-out recordings of people reading Uzbek. Scorers compare the transcripts with the reference words. ISSAI and Tashkent University of Information Technologies Speech recognition and translation Everyday Not in cards Jul 2021
V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. Meta Real robots Everyday Not in cards Jun 2025
Vaani Benchmark V1.0 Models transcribe speakers describing pictures. The scorer compares each prediction with three human transcripts and allows wording that matches any reference. ARTPARK and Indian Institute of Science Speech recognition and translation Everyday Not in cards Jun 2026
VABAR Agents program robots to sort tools, mesh gears, and handle cables in simulation. The harness checks success as object shapes and positions vary. UC Berkeley, NVIDIA, and collaborators Simulated robots University Not in cards Nov 2026
VAERS influenza vaccine adverse event extraction Models mark six kinds of clinical information in influenza vaccine safety reports, including neurological events, procedures, and time expressions. Researchers score exact or overlapping text spans. University of Texas Health Science Center at Houston and University of Pennsylvania Medicine University Not in cards Aug 2020
Valerius Flaccus Intertextuality Benchmark Models find passages in four Latin epics that scholars linked to Book 1 of the Argonautica. The authors score precision and recall against 945 parallels. Quantitative Criticism Lab Reading in many languages Expert Not in cards May 2021
VALOR-32K Models match captions to video and audio clips. Meta reports separate retrieval results for text paired with audio and text paired with video. Jing Liu, Sihan Chen, and colleagues Audio understanding and voice Everyday Not in cards Apr 2023
Vals Legal AI Report Legal AI tools perform seven kinds of legal work. Vals compares their task scores with scores from practicing lawyers. Vals AI Law Expert Not in cards Feb 2025
Vals Router Index Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights. Vals AI Composite indices Expert Not in cards Oct 2026
Vals Smith: pingdotgg/t3code Agents modify pingdotgg/t3code to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Vals AI Fixing real repositories Expert Not in cards Nov 2026
VCB-Bench Audio language models answer spoken prompts across instruction following, empathy, safety, simulation and vocalization tasks. Judges score the responses. Tencent Audio understanding and voice Expert Not in cards Oct 2025
VCTK Corpus Models match speech recordings with their texts. Meta evaluates audio-to-text retrieval across model sizes. Centre for Speech Technology Research at the University of Edinburgh Audio understanding and voice Everyday Not in cards
Verifiability Judgments for Generative Search Models decide whether a cited source fully, partly, or never supports a generated claim. Human annotations provide the reference judgments. Nelson F. Liu and colleagues Faithful to sources School Not in cards Apr 2023
Verifiable Browsing Task (Meta) Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools. Meta (Meta Superintelligence Labs) Web browsing Everyday Not in cards Oct 2026
VerticalRTK Search and answer APIs answer questions about facts that change over time across seven real-world domains. An LLM judge compares answers with dated references. Tako Deep research Expert Not in cards Jul 2026
VeruSAGE-Bench repository Agents synthesize Verus proofs for 849 tasks extracted from eight real-world Rust systems projects. Microsoft Research and collaborators Fixing real repositories Research frontier Not in cards Dec 2025
VGGSound Models identify sound events in video clips where the sound source is visible. University of Oxford Audio understanding and voice Everyday Not in cards Apr 2020
Video Browser Showdown 2026 marine video search Participants use interactive retrieval systems to find five underwater clips after seeing each target. Each search has a five-minute limit. Video Browser Showdown organizers Video understanding Everyday Not in cards Jan 2026
Video Object Segmentation under Transformations Models follow objects whose appearance changes as people manipulate them. The benchmark compares predicted masks with annotations through these transformations. VOST team Video understanding Everyday Not in cards Dec 2022
Video to Valence Models predict which of two videos viewers find more pleasant. Human preference annotations supply the reference judgments. Mantas Mazeika and collaborators Video understanding Everyday Not in cards Oct 2022
Video-ChatGPT video conversation benchmark Models answer questions about video content. The evaluation grades correctness, detail, context, timing, and consistency. Muhammad Maaz and colleagues Video captioning Everyday Not in cards Jun 2023
Video-MonetBench Models rank two videos generated for the same prompt. The test compares each ranking with a human preference, including ties. Tsinghua University and Z.ai Video understanding Everyday Not in cards Dec 2024
Video2Robot Embodied Models predict cleaned two-dimensional trajectories from noisy or occluded landmark tracks. The evaluator compares predicted motion and hand grips with a hidden synthetic reference. taylormino Embodied reasoning University Not in cards Oct 2026
VideoCon Models distinguish descriptions supported by a video from captions that contradict it. Hritik Bansal and colleagues Video understanding School Not in cards Nov 2023
VideoGen-RewardBench Models rank two generated videos for the same prompt. The scorer compares their rankings with human preferences for visual quality, motion, prompt alignment, and overall quality. Kuaishou Technology and collaborators Video understanding Everyday Not in cards Jan 2025
ViGiL3D Models locate objects in indoor 3D scans from English descriptions. A description can match several objects or none. Models must return the matching set. Simon Fraser University and Alberta Machine Intelligence Institute Spatial reasoning and counting Everyday Not in cards Jan 2025
ViLegalExpert Models retrieve Vietnamese law and answer questions from citizen consultations. Professional answers and expert-verified provisions supply references across 34 legal domains. University of Information Technology, Vietnam National University Ho Chi Minh City Law University Not in cards Sep 2026
ViLLM-Eval Models answer Vietnamese exam, general knowledge, and reading comprehension questions. They also predict the final word in a passage. Ton Duc Thang University, INT2 and HPC SYSTEMS Knowledge in many languages School Not in cards Apr 2024
VIMA-Bench Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates. Stanford University and NVIDIA with collaborators Simulated robots Everyday Not in cards Oct 2022
Vinoground Models match short videos with captions that distinguish events by their temporal order. Jianrui Zhang and colleagues Video understanding School Not in cards Oct 2024
ViNV-TTS Models read Vietnamese dialogue aloud and place sounds such as laughter or breathing where the text marks them. VLSP 2026 organizers Audio understanding and voice Everyday Not in cards Sep 2026
VIPBench Models predict whether listeners hear the same voice in two recordings. The authors compare eleven speech models against judgments from 1,290 listeners. Northwestern University Speech recognition and translation Everyday Not in cards Nov 2026
VIPSeg Models label video pixels and keep instance identities consistent across frames. The benchmark includes both countable objects and background regions. VIPSeg team Video understanding Everyday Not in cards
Virtual KITTI 2 Models interpret simulated driving videos with reference geometry and pixel labels. Meta uses the sequences to compare video object segmentation models. Naver Labs Europe Video understanding Everyday Not in cards Jan 2020
VisR-RAG-Fin Models retrieve pages from financial documents and answer questions using their contents. The evaluator measures page ranking and checks answers against references. University of Luxembourg and collaborators Finance University Not in cards Oct 2026
Vistaar Models transcribe speech across 59 combinations of language and dataset in 12 Indian languages. Each test has its own word error rate. AI4Bharat Speech recognition and translation Everyday Not in cards May 2023
VizWiz Models answer over 31,000 spoken questions that blind people asked about photos they took with phones. Some questions are unanswerable from the image. University of Texas at Austin and collaborators (Gurari et al.) General visual questions Everyday Not in cards Feb 2018
VLAIR Legal Research Legal AI tools answer 200 US legal research questions. Reviewers score legal accuracy, citation authority, and suitability for sharing. Vals AI Law Expert Not in cards Oct 2025
VLSP 2023 Vietnamese Mispronunciation Detection Models compare spoken Vietnamese with the expected pronunciation and flag individual sounds that the speaker pronounced incorrectly. VLSP 2023 organizers Speech recognition and translation Everyday Not in cards Aug 2023
VM14K Models answer Vietnamese multiple-choice questions across 34 medical specialties. The authors reserve a private set for leaderboard evaluation. Thong Nguyen and colleagues Medicine University Not in cards May 2025
Vmax Tasks Agents fix JavaScript bugs from projects including Vue.js, Redux, Docusaurus, and Chalk. Harbor lists 1,043 tasks with bug reports and expected behavior. VmaxAI Fixing real repositories Expert Not in cards Jan 2026
VocalSound Models classify human vocal sounds such as laughter, coughing, and sneezing. Yuan Gong and colleagues Audio understanding and voice Everyday Not in cards May 2022
Voice Agent LLM Benchmark Models handle simulated caller requests in text. Hamming grades task completion, policy compliance, and how naturally the models phrase their replies. Hamming AI Customer service Everyday Not in cards Nov 2026
Voice Jailbreak Attacks Against GPT-4o Models respond to spoken requests. Human reviewers measure how often a model complies with an unsafe request. Xinyue Shen and colleagues Jailbreaks and prompt injection Everyday Not in cards May 2024
VoiceMOS Challenge 2026 Models predict listener ratings for enhanced speech, emotional speech, and synthesized accents. Organizers compare predictions with ratings for individual utterances. VoiceMOS Challenge organizing committee Speech recognition and translation Everyday Not in cards Apr 2026
VoxAnon Models anonymize speech or try to identify the original speaker. Balanced trials compare reidentification across age, gender, pitch, and accent groups. Avignon University Speech recognition and translation Everyday Not in cards Oct 2026
VoxCeleb1 Models identify speakers from recorded speech. NVIDIA also uses the recordings for speaker counting within AIR-Bench. University of Oxford Audio understanding and voice Everyday Not in cards Jun 2017
VoxLingua107 Models identify the language in a speech recording. The corpus includes 107 languages. Tallinn University of Technology Audio understanding and voice Everyday Not in cards Nov 2020
VoxPopuli Models transcribe European Parliament speeches. The corpus also supports speech translation across languages. Facebook AI Speech recognition and translation Everyday Not in cards Jan 2021
VQA-RAD Vision-language models answer clinicians' questions about 315 radiology images, including head CTs, MRIs and chest X-rays. Questions are closed yes or no, or open free text. NIH and collaborators (Lau et al.) Medicine Expert Not in cards Nov 2018
VSAS-Bench Streaming vision-language models answer tasks over online video frames while evaluation measures accuracy, response timing and consistency. Apple Video understanding Expert Not in cards Apr 2026
VSEC Models correct misspellings in Vietnamese text. The evaluation compares their corrections against reference sentences. Dinh-Truong Do and colleagues Reading in many languages School Not in cards Nov 2021
VSI-SUPER Video models watch arbitrarily long streams and must recall where objects appeared (VSR) or keep counting objects across scenes (VSC). Video lengths run from 10 minutes to 4 hours. New York University (Cambrian-S team, Yang et al.) Long videos Everyday Not in cards Nov 2025
VSR Models judge whether captions with 65 kinds of spatial relation, such as under or facing, are true of an image. Over 10,000 text and image pairs. University of Cambridge (Liu et al.) Spatial reasoning and counting Everyday Not in cards Apr 2022
VSR Random Split Models judge whether image captions correctly describe spatial relations on the randomly partitioned test set. The score is accuracy over 2,195 test examples. University of Cambridge (Liu et al.) Spatial reasoning and counting Everyday Not in cards Apr 2022
VSR Zero-Shot Split Models judge spatial captions on a test set whose object concepts do not appear in training. The score is accuracy over 616 test examples. University of Cambridge (Liu et al.) Spatial reasoning and counting Everyday Not in cards Apr 2022
VulnGym White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace. Tencent Finding and exploiting real bugs Expert Not in cards Apr 2026
Wahl-O-Mat 2024 European election party agreement Models take a position on 38 German statements about European Union policy. Researchers compare those answers with each party’s published positions. Luca Rettenberger, Markus Reischl, and Mark Schutera at Karlsruhe Institute of Technology Bias, fairness and politics Everyday Not in cards May 2024
Wahl-O-Mat 2025 Bundestag party agreement Models answer 38 statements from Germany’s 2025 federal election. The original researchers score agreement with each of seven parties using the Wahl-O-Mat weighting rule. Ina Dormuth and colleagues at TU Dortmund University Bias, fairness and politics Everyday Not in cards Feb 2025
WANLP 2022 Shared Task on Propaganda Detection in Arabic Models identify persuasion techniques in Arabic social media posts. A second task asks for the words that express each technique. WANLP 2022 shared task organizers Reading in many languages University Not in cards Jul 2022
Washington Post midterm voter guides Chatbots write voter guides for US addresses. The Washington Post checks whether each guide names the correct House district. The Washington Post and Promptwatch Long-form factuality Everyday Not in cards Oct 2026
WASSCE Benchmark for Open Language Models Models answer questions based on the West African Senior School Certificate Examination. The harness compares models across mathematics, English, science, and social studies. acheamponge Exam questions School Not in cards Apr 2026
WAXAL Speech Recognition Test Speech models transcribe people describing pictures in African languages. Researchers compare the transcripts with human references using word error rate. Google Research and African data partners Speech recognition and translation Everyday Not in cards Jan 2026
Wayfinder Code Search Benchmark Coding search agents find the files changed by real bug fixes from short commit subjects or longer problem briefs. Voyage AI Fixing real repositories Expert Not in cards Nov 2026
WebGuard Models classify the risk of a web action using human labels for safe, low-risk, and high-risk actions. Ohio State University and collaborators with Scale AI Web browsing Everyday Not in cards Jul 2025
WebShop Agents search and click through a simulated shopping site with 1.18 million real products to buy an item that matches a text instruction. Reward reflects how well the purchase matches. Princeton University Web browsing Everyday Not in cards Jul 2022
WebSRC Models answer questions about a web page from its screenshot, HTML or both. The set has 400K question and answer pairs from 6.4K web pages. Scored by exact match and F1. Shanghai Jiao Tong University (X-LANCE lab) Documents, charts and text in images Everyday Not in cards Jan 2021
WeChat Pay Physical Client Evaluation Agents operate the WeChat Pay client on physical phones. The team scores execution paths, use of supplied knowledge, acceptance checks, and account creation accuracy. Tencent WeChat Pay Phones University Not in cards Nov 2026
WeirdChat Models respond to prompts designed to elicit unexpected behaviors. A judge labels more than 1,300 behavior patterns across annotated transcripts. Transluce Safety and alignment Expert Not in cards Jul 2026
WenetSpeech Test_Meeting Models transcribe spontaneous Mandarin speech from real meetings in the Test_Meeting set. The baseline score is mixture error rate, counting Mandarin characters and English words. Northwestern Polytechnical University (ASLP@NPU), Mobvoi and others Speech recognition and translation Everyday Not in cards Oct 2021
WenetSpeech Test_Net Models transcribe Mandarin internet speech from the Test_Net set. The baseline score is mixture error rate, counting Mandarin characters and English words. Northwestern Polytechnical University (ASLP@NPU), Mobvoi and others Speech recognition and translation Everyday Not in cards Oct 2021
WenetSpeech-Chuan-Eval Models transcribe Sichuanese speech. Qwen reports separate error rates on the easy and hard test sets. Yuhang Dai and colleagues Speech recognition and translation Everyday Not in cards Sep 2025
WHAMR! Models recover individual voices from noisy speech mixtures with simulated room reverberation. The test contains 3,000 mixtures. Mitsubishi Electric Research Laboratories and Whisper Audio understanding and voice University Not in cards Oct 2019
Where Did My Data Go? Models trace missing records from a data source to a dashboard. The grader checks whether they find the cause or recognize that the evidence is insufficient. Olga Braginskaya Data science and SQL University Not in cards Oct 2026
WHU-CDC Models compare aerial photos of Christchurch, describe what changed, and mark changed pixels. Reference captions describe reconstruction after the earthquake. Jingye Shi and colleagues Detection and captioning Everyday Not in cards Oct 2024
Widget Captioning Models describe an individual interface element in a mobile screenshot. The benchmark compares their captions with human descriptions of the same element. Google Research Documents, charts and text in images School Not in cards Oct 2020
Wiki Neutrality Corpus Models identify biased wording in English Wikipedia sentences and rewrite the sentences. The authors compare the rewrites with corrections by Wikipedia editors. Reid Pryzant and colleagues Bias, fairness and politics School Not in cards Nov 2019
WIKIBIAS Models classify subjective bias in Wikipedia sentences, mark the words that express it, and rewrite the sentences. The test includes edits that change several separate phrases. Yang Zhong, Jingfeng Yang, Wei Xu, and Diyi Yang Bias, fairness and politics School Not in cards Oct 2021
WikiCompany2IndustryClustering Models group company descriptions by industry. The score measures how well the groups recover the industry labels. The Hong Kong University of Science and Technology Finance Everyday Not in cards Jul 2024
WikiFact (HELM) Models complete a Wikidata relation given its subject and predicate. HELM checks the predicted entity against a reference answer. Stanford CRFM Trivia and open questions School Not in cards Nov 2022
WikiLingua Models summarize procedural articles in different languages. Aligned WikiHow articles support both same-language and cross-language evaluation. Faisal Ladhak and colleagues Reading in many languages School Not in cards Nov 2020
WikiNEuRal Models locate named entities in Wikipedia text across nine languages using automatically produced reference annotations. Simone Tedeschi and colleagues Reading in many languages School Not in cards Nov 2021
Wikispeedia Players navigate from one Wikipedia article to another using only article links. The Prime Intellect port scores whether agents reach targets held out from training. Robert West, Joelle Pineau, and Doina Precup Web browsing Everyday Not in cards Jul 2009
WikiSQL Models turn natural language questions into SQL queries over single Wikipedia tables. The dataset has 80,654 hand-annotated questions over 24,241 tables. Scored by execution accuracy and logical form accuracy. Salesforce Research (Zhong, Xiong, Socher) Data science and SQL School Not in cards Aug 2017
WikiTableQuestions Models answer complex questions about semi-structured HTML tables from Wikipedia. Answers often need counting, comparison or aggregation across rows. Scored by answer accuracy against the gold answers. Stanford University (Pasupat and Liang) Language reasoning Everyday Not in cards Aug 2015
WikiText-2 Language models predict the next token in held-out Wikipedia articles. Lower perplexity means the model gave more probability to the tokens that actually appeared. Stephen Merity, Caiming Xiong, James Bradbury and Richard Socher Other University Not in cards Sep 2016
WikiVQABench Vision-language models answer multiple-choice questions that require image evidence and linked Wikipedia and Wikidata knowledge. IBM Research General visual questions University Not in cards May 2026
WildGraphBench GraphRAG systems answer 1,100 questions over long, mixed reference documents behind Wikipedia articles in 12 topics, from single facts to section summaries. USTC and Metastone Technology (DeepResearch Bench team) Reasoning over long documents University Not in cards Feb 2026
WildKhmerST Models find and read Khmer text in photographs of public places. Annotations mark text lines with polygons and transcriptions. Cambodia Academy of Digital Technology and La Rochelle University collaborators Documents, charts and text in images Everyday Not in cards Jun 2025
Winoground Models match two images to two captions that use the same words in a different order. Tristan Thrush and colleagues General visual questions School Not in cards Apr 2022
Wisesight Sentiment Models classify Thai social media posts about products and services. SEA-HELM tests positive, neutral, and negative sentiment and excludes the original question label. PyThaiNLP and Wisesight Reading in many languages Everyday Not in cards Sep 2019
WixQA Models retrieve Wix support articles and answer customer questions from those articles. The authors score retrieval and answer generation separately. Dvir Cohen and colleagues Running a business Everyday Not in cards Apr 2025
WMT 2014 News Translation Models translate news text between English and other languages. The shared task compares submitted translations on a common test set. Workshop on Statistical Machine Translation Translation University Not in cards Jun 2014
WMT 2016 News Translation Models translate news between languages. The harness compares translations with human references from the 2016 shared task. Ondřej Bojar and the WMT 2016 organizers Translation University Not in cards Aug 2016
WMT24 Icelandic Proper Names Models translate English sentences containing place names or personal names into Icelandic. The scorer checks the Icelandic name and its grammatical form. Árni Magnússon Institute for Icelandic Studies Translation Everyday Not in cards Jun 2024
WMT26 Multilingual Instruction Shared Task v2026 Models answer questions from passages, summarize papers across languages and follow writing instructions. The overall ranking averages ranks across the three tasks. WMT multilingual instruction organizers Reading in many languages University Not in cards Jan 2026
WMT26 Video Subtitle Translation Models translate timed Chinese subtitles into five target languages using a synopsis for context. Human reviewers check the model judge’s assessments. WMT video subtitle translation organizers Translation University Not in cards Apr 2026
WOLF Models play Werewolf to measure deception production and deception detection as separate abilities. WOLF authors Multi-agent games Everyday Not in cards Dec 2025
Word-Level BISINDO Models identify 32 signs from the Banten variety of Indonesian Sign Language in video clips. BINUS University Video understanding Everyday Not in cards May 2025
WorkArena Web agents do routine knowledge-work tasks in a live ServiceNow instance, such as filling forms and filtering lists. WorkArena++ adds 682 compositional tasks at levels 2 and 3. ServiceNow Research Web browsing University Not in cards Mar 2024
WorkIndex Agents complete everyday WorkBuddy tasks. Tencent combines eight tests covering office files, audio and video work, and visual questions. Tencent Composite work indices University Not in cards Nov 2026
WorkRB Skill Extraction Challenge 2026 Models rank occupational skills for vacancy sentences and alternative skill names. Organizers average graded ranking scores at depth 100 across five test sets. TechWolf, Avature, and Ghent University Other professions University Not in cards Jun 2026
WorldModelBench Video generators judged as world models on instruction following and physics adherence in application domains. Human labels and a fine-tuned judge. Video world models Not in cards Feb 2025
WorldModelGym World models predict returns for five candidate action sequences across 100+ Atari, Meta-World, DMC and classic-control tracks. Scored by normalized regret. Reka Labs Interactive world models Not in cards Jul 2026
WorldOlympiad Models generate long videos for gaming, robotics, and everyday scenes. The evaluator checks physical rules, reconstructed geometry, and instruction following across video segments. Zhejiang University, Alibaba DAMO Academy, Hong Kong University of Science and Technology, and collaborators Video world models Everyday Not in cards Jun 2026
WorldPrediction Agents predict and plan in procedural environments with long horizons. Tasks test whether models maintain a useful world model while acting. Meta FAIR and Superintelligence Labs Interactive world models University Not in cards Jun 2025
WorldScore 3,000 next-scene generation tasks with camera-trajectory layouts. Scores 3D/4D and video world generators on controllability, quality, dynamics. Stanford Video world models Not in cards Apr 2025
WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. Video world models Not in cards Oct 2024
Writing Style Matching (apaz) Models continue a story in the author’s style. A judge compares the generated continuation with the reference continuation and assesses whether the passages sound alike. apaz Taste and style Everyday Not in cards Nov 2025
WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. Google Medicine Expert Not in cards Jan 2026
WSYue-eval Models transcribe short and long Cantonese recordings. The suite also evaluates speech synthesis. Longhao Li and colleagues Speech recognition and translation Everyday Not in cards Sep 2025
x-stance Models identify whether a political comment supports or opposes the proposition in a question. Jannis Vamvas and colleagues Reading in many languages School Not in cards Mar 2020
XACLE Challenge 2026 Models listen to generated audio and predict how closely it matches an English description. Organizers rank predictions by their correlation with listener scores. XACLE Challenge organizers Audio understanding and voice Everyday Not in cards Jul 2025
xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. xAI (SpaceXAI) Refusing harmful requests Not in cards
XL-Sum (English) Models summarize English BBC news articles. ROUGE-1, ROUGE-2, and ROUGE-L compare generated summaries with journalist-written summaries. The current English test split contains 11,535 examples. Bangladesh University of Engineering and Technology and collaborators Faithful to sources Everyday Not in cards Jun 2021
XNLIeu Models judge whether a Basque sentence follows from, contradicts, or is neutral toward a premise. The evaluation includes translated and native examples. Maite Heredia and colleagues Reading in many languages School Not in cards Apr 2024
XOR-TyDi QA Models retrieve passages across languages to answer questions written by speakers of seven languages. Tasks require English evidence or answers in the question’s language. Akari Asai and colleagues Reading in many languages School Not in cards Oct 2020
XSum Models summarize a BBC news article in one sentence. ROUGE-1, ROUGE-2, and ROUGE-L compare the generated summary with the journalist's introductory summary. University of Edinburgh Reading comprehension Everyday Not in cards Aug 2018
XTD-200 Models retrieve images and captions across languages using translations of the XTD test set. Results measure how often the matching item ranks first. Alexander Visheratin Reading in many languages School Not in cards Sep 2023
XTREME Multilingual models are fine-tuned on English data and then tested on 9 tasks in 40 languages. The tasks cover sentence classification, structured prediction such as named entities, question answering and sentence retrieval. The benchmark measures how well skills transfer across languages. Carnegie Mellon University and Google Reading in many languages School Not in cards Mar 2020
XTREME-R Multilingual models are fine-tuned on English data and tested on 10 tasks in 50 typologically diverse languages, including harder language-agnostic retrieval tasks. It extends XTREME and adds MultiCheckList, a multilingual diagnostic suite. Google Research and DeepMind with collaborators Reading in many languages School Not in cards Apr 2021
XWinograd Models resolve ambiguous pronouns in six languages by choosing which noun fits each sentence. Alexey Tikhonov and colleagues Reasoning in many languages Everyday Not in cards Jun 2021
Yet Another Kernel Kernel Models optimize GPU kernels for speed while a verifier rejects numerical errors and input-insensitive shortcuts. HUD AI GPU kernels Research frontier Not in cards Nov 2026
YorùLect Models transcribe Yorùbá speech and translate dialect text. The speech task covers three varieties. The text task also covers Ìjèbú. Orevaoghene Ahia and collaborators Speech recognition and translation Everyday Not in cards Jun 2024
YouCook2 Dense Video Description Models locate cooking steps and describe each step. The score is CIDEr for the complete dense video description task. University of Michigan Video captioning Everyday Not in cards Mar 2017
Your Voice, Your Device, Your Language Challenge Models transcribe Kiswahili recordings for offline speech recognition. Zindi ranks submissions by word error rate before assessing the top ten for deployment. Sartify, ITU AI for Good and Zindi Speech recognition and translation Everyday Not in cards Jul 2025
YouTube-VIS Models detect, segment, and track object instances in YouTube videos. The SAM 3 report evaluates the 2021 edition. YouTube-VIS team Video understanding Everyday Not in cards May 2019
YouTube-VOS Models segment and track objects across frames in YouTube videos. The evaluation distinguishes categories seen during training from previously unseen categories. Ning Xu and colleagues Video understanding Everyday Not in cards Sep 2018
Zalo AI Challenge 2022 End-to-End Question Answering Models answer Vietnamese general knowledge questions. HELM scores short answers without supplying a passage. Zalo AI Knowledge in many languages School Not in cards
ZendoWorld Models infer hidden rules in the board game Zendo from labeled examples. ZendoWorld authors Puzzles and chess Everyday Not in cards Jul 2026
ZenML Bench Agents build, repair, and investigate machine learning pipelines in ZenML. Eighteen tasks grade the state the framework records. ZenML Machine learning engineering Expert Not in cards Sep 2026
Zerolang Editing Agents repair and refactor Zerolang programs through graph editing tools. The evaluation includes 67 held-out synthetic tasks. pandelis Fixing real repositories University Not in cards May 2026
Zoomed-in BVCC Models predict how listeners rate synthesized speech when they hear only the highest-rated systems. Listeners rated the same clips again in progressively smaller groups. Erica Cooper and Junichi Yamagishi Audio understanding and voice Everyday Not in cards May 2023