Knowledge
We track 127 benchmarks in knowledge. They run from everyday level to research frontier level.
Qwen held the top knowledge score 31% of the contested time
That's about 14 of the 46 months since January 2023, averaged over these benchmarks. We only count time while other labs are still testing, and stop once a model passes 85%. Faded stretches no longer count.
OpenAI Anthropic Shanghai AI Lab Other labs Microsoft Meta NVIDIA Google DeepMind Baidu Meituan ByteDance Seed Qwen DeepSeek xAI Tencent MiniMax
Everyday Most adults, without training
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Expert Specialists with years of training, often over hours of work
Research frontier Researchers, on open problems or tasks that take days
Sources: Official leaderboards and labs’ model cards
Share of contested time at number one
Qwen 31% 
Google DeepMind 18% OpenAI 16% Anthropic 8% Other labs 8% 
Meta 5% 
ByteDance Seed 4% 
Baidu 3% xAI 3% 
DeepSeek 3% Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
Closed out: 12 benchmarks passed 85%
OpenAI 4 Other labs 3 Google DeepMind 1 Baidu 1 
Tencent 1 DeepSeek 1 Anthropic 1
Google DeepMind 1 Baidu 1 Tencent 1 DeepSeek 1 Anthropic 1 Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
- Nov 2025 Extended NYT Connections Gemini 3 Pro
- Sep 2025 MuSR ERNIE-4.5-21B-A3B-Thinking
- Mar 2025 MMLU-Pro Hunyuan-T1
- Dec 2024 C-Eval DeepSeek-V3
- Mar 2024 BIG-Bench Hard Claude 3 Opus
- May 2023 BoolQ PaLM 2-L
- May 2023 PIQA PaLM 2-L
- May 2023 TriviaQA PaLM 2-L
- Mar 2023 HellaSwag GPT-4 Mar 2023
- Mar 2023 WinoGrande GPT-4 Mar 2023
- Mar 2023 ARC-Challenge GPT-4 Mar 2023
- Mar 2023 MMLU GPT-4 Mar 2023
Each benchmark counts from the one source its own page leads with: its official leaderboard when that is current, else Epoch AI’s own runs, else labs’ own numbers. We date independent runs by the model’s release and lab reports by when they came out.
Google DeepMind OpenAI Meta Meituan Anthropic Technology Innovation Institute Other labs Qwen Mistral AI DeepSeek
Everyday Most adults, without training
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Expert Specialists with years of training, often over hours of work
Research frontier Researchers, on open problems or tasks that take days
Sources: Official leaderboards and boards copied by Epoch AI (CC BY 4.0)
Share of contested time at number one, independent runs only
OpenAI 31% DeepSeek 27% Google DeepMind 21% Anthropic 8% Meta 4% 
Mistral AI 3% Other labs 2% Qwen 2% 
Technology Innovation Institute 1%
DeepSeek 27% Google DeepMind 21% Anthropic 8% Meta 4% Mistral AI 3% Other labs 2% Qwen 2% Technology Innovation Institute 1% Sources: Official leaderboards and Epoch AI (CC BY 4.0)
Closed out: 3 benchmarks passed 85%
Google DeepMind 1 OpenAI 1 Qwen 1 Sources: Official leaderboards and Epoch AI (CC BY 4.0)
- Nov 2025 Extended NYT Connections Gemini 3 Pro
- Dec 2024 MMLU-Pro o1
- Apr 2024 CMMLU Qwen1.5-110B
Every independent run: each benchmark’s own leaderboard, Epoch AI’s runs and boards Epoch AI copied, dated by the model’s release. We leave out benchmarks that only labs report.
Google DeepMind OpenAI Anthropic Shanghai AI Lab Microsoft Other labs Meta NVIDIA Baidu Meituan ByteDance Seed Qwen DeepSeek Mistral AI xAI Moonshot AI
Everyday Most adults, without training
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Expert Specialists with years of training, often over hours of work
Research frontier Researchers, on open problems or tasks that take days
Sources: Official leaderboards, boards copied by Epoch AI (CC BY 4.0), labs’ model cards and claims collected by Epoch AI (CC BY 4.0)
Share of contested time at number one, claimed scores included
DeepSeek 18% Google DeepMind 16% OpenAI 16% Qwen 13% Anthropic 13% Other labs 7% xAI 4% Meta 3% 
Microsoft 2% ByteDance Seed 2% Baidu 2% Mistral AI 2% Moonshot AI 1% Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
Closed out: 8 benchmarks passed 85%
OpenAI 2 Google DeepMind 1 Baidu 1 DeepSeek 1 Anthropic 1 Other labs 1 Meta 1
Google DeepMind 1 Baidu 1 DeepSeek 1 Anthropic 1 Other labs 1 Meta 1 Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
- Nov 2025 Extended NYT Connections Gemini 3 Pro
- Sep 2025 MuSR ERNIE-4.5-21B-A3B-Thinking
- Dec 2024 C-Eval DeepSeek-V3
- Dec 2024 MMLU-Pro o1
- Mar 2024 BIG-Bench Hard Claude 3 Opus
- May 2023 PIQA PaLM 2-L
- Mar 2023 MMLU GPT-4 Mar 2023
- Feb 2023 TriviaQA LLaMA-65B
Every score, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.
From everyday to research frontier level
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
Everyday
Most adults, without training
- HellaSwag
- WinoGrande
- PIQA
- BoolQ
- TriviaQA
- WideSearch
- SIQA
- LAMBADA
- Natural Questions
- CommonSenseQA
- SkateBench v2
- FRAMES
- ANLI
- PopQA
- SQuAD v2
- SuperGLUE
- CommonsenseQA 2.0
- SQuAD
- CB
- ComplexNLU (Tencent)
- WorldSense
- WSC
- RTE
- StrategyQA
- ChineseQA
- COPA
- QuAC
- ReCoRD
- StoryCloze
- WebQuestions
- WiC
- Winograd
- Word-Task (Tencent)
- Bamboogle
- DeepSeek search Q&A evaluation
- GLM-4 All Tools information seeking (Z.ai)
- HotpotQA
- Knowledge Q&A (MiniMax in-house)
- Temporal reasoning with search
- Adversarial SQuAD
- ANLI R1
- ANLI R2
- ANLI R3
- BLiMP
- CoQA
- Dynabench SQuAD
- FreshQA
- HANS
- Hindsight Neglect
- MultiHop-RAG
- NarrativeQA
- Paloma
- PAWS
- Story Cloze Winter 2018
- WikiTableQuestions
School
Students, from grade school to high school
- BIG-Bench Hard
- ARC-Challenge
- Extended NYT Connections
- DROP
- OpenBookQA
- AGIEval
- ARC-Easy
- SnorkelSpatial
- RACE-H
- GAOKAO-Bench
- RACE
- MultiRC
- RACE-M
- AP Calculus
- AP Physics
- SAT Reading
- AP Art History
- AP Biology
- AP Calculus BC
- AP Chemistry
- AP English Language and Composition
- AP English Literature and Composition
- AP Environmental Science
- AP Macroeconomics
- AP Microeconomics
- AP Physics 2
- AP Psychology
- AP Statistics
- AP US Government
- AP US History
- AP World History
- SAT Evidence-Based Reading & Writing
University
Undergraduates and entry-level professionals
Expert
Specialists with years of training, often over hours of work
By kind
Exam questions
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| MMLU-Pro Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. | University | Checked automatically |  89.8% | In use |
| MMLU 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. | University | Checked automatically | 92.3% | Fading |
| ARC-Challenge Models answer grade-school science multiple-choice questions that retrieval and word co-occurrence baselines got wrong. Scored by accuracy. | School | Checked automatically |  97.3% | Dropped |
| C-Eval 13,948 Chinese multiple-choice exam questions across 52 disciplines and four difficulty levels. Accuracy. | University | Checked automatically | 93.7% | Fading |
| MMLU-Redux Manually re-annotated subset of MMLU that fixes wrong labels. Multiple-choice accuracy. | University | Checked automatically | 95% | Fading |
| OpenBookQA Models answer elementary science multiple-choice questions that need a given core science fact plus common knowledge. Scored by accuracy. | School | Checked automatically | 96.6% | Fading |
| AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. | School | Checked automatically |  89.2% | Fading |
| CMMLU Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy. | University | Checked automatically | 90.8% | Fading |
| MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. | University | Checked automatically | 73.4% | Not in cards |
| ARC-Easy Models answer grade-school science multiple-choice questions from the ARC Easy Set, which has 5,197 questions. Scoring is accuracy. | School | Checked automatically | 98.9% | Dropped |
| GAOKAO-Bench Models answer Chinese national college entrance exam questions from 2010 to 2022 across nine subjects, with objective and subjective questions. | School | Checked automatically | 71.6% | Dropped |
| AP Calculus Models answer an official AP Calculus practice exam without image questions. Scored as accuracy. | School | Checked automatically | 88.6% | System cards only |
| AP Physics Models answer an official AP Physics practice exam without image questions. Scored as accuracy. | School | Checked automatically | 92.9% | System cards only |
| GRE Analytical Writing Models write GRE issue and argument essays graded by trained contractors. Scored on the 0 to 6 scale. | University | Graded by people | 5.00 | System cards only |
| MMLU-stem Models answer the STEM subjects of MMLU, such as physics, chemistry, mathematics, and computer science, as multiple choice. Scoring is accuracy. | University | Checked automatically | 83.6% | Dropped |
| KMMLU-Redux Models answer multiple-choice questions from Korean National Technical Qualification exams. The set is rebuilt from KMMLU with erroneous questions removed. | University | Checked automatically |  72.7% | Dropped |
| AP Art History Models take a simulated AP Art History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Biology Models take a simulated AP Biology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Calculus BC Models take a simulated AP Calculus BC exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 4.00 | System cards only |
| AP Chemistry Models take a simulated AP Chemistry exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 4.00 | System cards only |
| AP English Language and Composition Models take a simulated AP English Language and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 2.00 | System cards only |
| AP English Literature and Composition Models take a simulated AP English Literature and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 2.00 | System cards only |
| AP Environmental Science Models take a simulated AP Environmental Science exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Macroeconomics Models take a simulated AP Macroeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Microeconomics Models take a simulated AP Microeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Physics 2 Models take a simulated AP Physics 2 exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 4.00 | System cards only |
| AP Psychology Models take a simulated AP Psychology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP Statistics Models take a simulated AP Statistics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP US Government Models take a simulated AP US Government exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP US History Models take a simulated AP US History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 5.00 | System cards only |
| AP World History Models take a simulated AP World History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | Graded by people | 4.00 | System cards only |
| MMLU College Biology Models answer MMLU multiple-choice questions from the College Biology subject. Scoring is accuracy. | University | Checked automatically | 95% | System cards only |
| MMLU-Redux 2.0 Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them. | University | Checked automatically | Not in cards |
Expert-written questions
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| Humanity's Last Exam 2,500 expert-written closed-ended questions across 100+ subjects. Exact-match or multiple-choice answers graded by an LLM judge. | Expert | Checked automatically | 54.8% | In use |
| SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. | Expert | Checked automatically | 73.9% | In use |
| KINA Models answer 899 knowledge questions across 261 fine-grained disciplines, each with ten lettered options. Scored by accuracy against the key. | Expert | Checked automatically | 53.2% | In use |
| HLE Extended Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores. | Expert | Checked automatically | 77.3% | Not in cards |
| Encyclo-K Models judge questions composed at test time from knowledge statements extracted from authoritative textbooks. Random composition is meant to resist contamination and test several facts at once. | University | Checked automatically | 51.8% | Fading |
| HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. | Research frontier | Checked automatically | 77.9% | In use |
| KMMLU-Pro Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. | Expert | Checked automatically |  79.2% | In use |
| HLE-Verified Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. | Expert | Checked automatically | 73.6% | In use |
| HLE-Verified (text-only Full) Models answer the text-only questions of Humanity's Last Exam after expert verification and revision. The Full set keeps the questions experts marked uncertain, and the score is pass@1 accuracy. | Expert | Checked automatically | 46.8% | Not in cards |
| LPFQA Models answer 430 long-tail professional questions drawn from real forum discussions in 7 academic and industrial domains. | Expert | Checked automatically | 52.6% | Fading |
| HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. | Research frontier | Checked automatically | Not in cards |
Commonsense
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| HellaSwag Commonsense sentence completion with adversarially filtered wrong endings. Multiple-choice accuracy. | Everyday | Checked automatically | 97% | Fading |
| WinoGrande 44K Winograd-style pronoun-resolution problems. Binary-choice accuracy. | Everyday | Checked automatically | 91.7% | Fading |
| PIQA Models pick which of two solutions better achieves an everyday physical goal. Scoring is accuracy. | Everyday | Checked automatically | 88.6% | Dropped |
| SIQA Models answer multiple-choice questions about people's motives, reactions, and emotions in everyday social situations. Scored by accuracy. | Everyday | Checked automatically | 80.2% | Dropped |
| CommonSenseQA Models answer five-way multiple-choice questions built from ConceptNet relations that need everyday background knowledge. Scored by accuracy. | Everyday | Checked automatically | 90.4% | Dropped |
| CommonsenseQA 2.0 Models answer 14,343 yes or no commonsense questions that players wrote to fool an AI in a game. Scored by accuracy. | Everyday | Checked automatically | 67.8% | Not in cards |
| WSC Models decide whether a pronoun refers to a given noun phrase in sentences that need commonsense to resolve. Scored by accuracy. | Everyday | Checked automatically | 86.9% | Dropped |
| COPA Models choose which of two alternatives is the more plausible cause or effect of a premise, across 1,000 questions. Scored by accuracy. | Everyday | Checked automatically | 96% | Dropped |
| StoryCloze Models pick the correct ending for a four-sentence everyday story from two candidate endings. Scored by accuracy. | Everyday | Checked automatically | 87.4% | Dropped |
| Winograd Models pick which noun an ambiguous pronoun refers to in Winograd schema sentence pairs that need world knowledge. Scored by accuracy. | Everyday | Checked automatically | 89.5% | Dropped |
| BLiMP Language models must assign higher probability to the grammatical sentence in each of 67,000 minimal pairs covering English syntax, morphology and semantics. | Everyday | Checked automatically | Not in cards | |
| Paloma Language models are measured by perplexity on held-out text from 585 domains, from news sites to individual subreddits. | Everyday | Checked automatically | Not in cards | |
| Story Cloze Winter 2018 Systems select the correct story ending on the Winter 2018 test set. CodaLab reports percentage accuracy under submission identities. | Everyday | Checked automatically | Not in cards |
Reading comprehension
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| BoolQ Models read a Wikipedia passage and answer a naturally occurring yes or no question about it. Scored by accuracy. | Everyday | Checked automatically | 90.9% | Dropped |
| DROP Reading comprehension requiring discrete operations (counting, sorting, arithmetic) over paragraphs. Scored by F1. | School | Checked automatically | 93.5% | Fading |
| LAMBADA Models predict the last word of narrative passages where humans can guess it only from the whole passage. Scored by accuracy. | Everyday | Checked automatically | 86.9% | Dropped |
| RACE-H Models answer multiple-choice reading questions from English exams for Chinese high school students. The test split has 3,498 questions, scored by accuracy. | School | Checked automatically | 92.9% | Dropped |
| SQuAD v2 Models extract answers from Wikipedia paragraphs and must abstain on over 50,000 adversarial unanswerable questions. Scored by exact match and F1. | Everyday | Checked automatically | 81.8% | Dropped |
| RACE Models answer reading comprehension questions from Chinese middle and high school English exams. Reported here without a subset split. | School | Checked automatically | 94.2% | System cards only |
| GRE Verbal Models take the GRE verbal reasoning section. Scored on the 130 to 170 scale. | University | Checked automatically | 169.0 | System cards only |
| SQuAD Models extract answer spans from Wikipedia paragraphs for over 100,000 crowdsourced questions. Scored by exact match and F1. | Everyday | Checked automatically | 89.3% | Dropped |
| ComplexNLU (Tencent) Models answer Tencent in-house natural language understanding tasks with complex inputs. Task and scoring details are not published. | Everyday | Checked automatically | 61.2% | Dropped |
| MultiRC Models pick all correct answers to questions that need reasoning over several sentences of a paragraph. Scored by F1 over answer options. | School | Checked automatically | 88.2% | Dropped |
| RACE-M Models read passages from English exams for Chinese middle school students and answer multiple-choice questions. Scored by accuracy. | School | Checked automatically | 77% | Dropped |
| GMAT Verbal Models answer the verbal section of the official GMAT online exam. Scored as accuracy. | University | Checked automatically | 86.6% | System cards only |
| SAT Reading Models answer the reading section of official SAT practice tests. Scored as accuracy. | School | Checked automatically | 74.8% | System cards only |
| QuAC Models answer free-form questions in information-seeking dialogs about a hidden Wikipedia section, across 100K questions. Scored by word-level F1. | Everyday | Checked automatically | 51.1% | Dropped |
| ReCoRD Models fill in a masked entity in a query about a news article, using the article as context. Scoring is exact match and token F1. | Everyday | Checked automatically | 93.8% | Dropped |
| Word-Task (Tencent) Models solve Tencent in-house word level language tasks. Task and scoring details are not published. | Everyday | Checked automatically | 62.9% | Dropped |
| HotpotQA Models answer 113k Wikipedia questions that need facts from several documents. The distractor setting supplies gold and distractor paragraphs. Scored by exact match or F1. | Everyday | Checked automatically |  92.1% | System cards only |
| SAT Evidence-Based Reading & Writing Models take the SAT reading and writing section. Scored on the 200 to 800 scale. | School | Checked automatically | 710.0 | System cards only |
| Adversarial SQuAD Models answer SQuAD questions whose passages contain an added distracting sentence. | Everyday | Checked automatically | System cards only | |
| CoQA Models answer a series of linked conversational questions about a passage, where later questions depend on earlier turns. Scored by word-level F1. | Everyday | Checked automatically | Not in cards | |
| Dynabench SQuAD Models answer reading comprehension questions written by people to fool earlier models in the loop. | Everyday | Checked automatically | System cards only | |
| NarrativeQA Models answer questions about books and movie scripts, either from a Wikipedia summary or from the full story. The questions need an understanding of the whole narrative rather than one matching passage. Answers are free text, scored against two reference answers with F1, ROUGE-L or BLEU. | Everyday | Checked automatically | Not in cards |
Trivia and open questions
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| TriviaQA Trivia questions with evidence documents. Scored by exact match. | Everyday | Checked automatically | 91.8% | Fading |
| Natural Questions Models answer real Google search questions. Model cards usually use the open-domain version, scored by exact match against short answers from Wikipedia. | Everyday | Checked automatically | 53.4% | Dropped |
| SkateBench v2 Models name skateboard tricks from technical trick definitions. The score is the share named correctly. | Everyday | Checked automatically | 96.9% | Not in cards |
| PopQA Models answer about 14k entity questions built from Wikidata, spread from popular to rare entities. Scored by whether the answer matches a gold alias. | Everyday | Checked automatically | 55.5% | Fading |
| ChineseQA Models answer Chinese trivia questions in a closed-book format modeled on TriviaQA. | Everyday | Checked automatically | 85.1% | Dropped |
| WebQuestions Models answer factual questions mined from Google Suggest, with answers taken from the Freebase knowledge base. Scored by exact match. | Everyday | Checked automatically | 28.2% | Dropped |
| Knowledge Q&A (MiniMax in-house) Models answer knowledge questions drawn from real Hailuo AI user queries. | Everyday | Graded by a model |  78.6% | System cards only |
Questions that need search
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| WideSearch 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. | Everyday | Checked automatically | 83.9% | In use |
| Seal-0 Models answer 111 fact-seeking questions where web search returns conflicting or unhelpful results. Answers are graded for accuracy, usually with search enabled. | University | Checked automatically | 57.4% | Fading |
| FRAMES Models answer 824 multi-hop questions that need facts from 2 to 15 Wikipedia articles. Answers are graded against gold answers for accuracy. | Everyday | Checked automatically | 87.6% | Fading |
| Bamboogle Models answer 125 two-hop questions written so that a search engine cannot answer them directly. Models usually search and compose the facts. | Everyday | Checked automatically | 76% | System cards only |
| DeepSeek search Q&A evaluation Models answer objective and subjective Chinese search questions with retrieval or agentic search. Pairwise judging gives win, loss and tie rates. | Everyday | Ranked by matches | 28.1% | System cards only |
| GLM-4 All Tools information seeking (Z.ai) Models use a web browser to find information for user questions. The ChatGLM report gives accuracy but does not describe the test set. | Everyday | Graded by a model | 78.1% | System cards only |
| Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. | Expert | Graded by a model | 94.3% | System cards only |
| Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. | Everyday | Graded by a model | 11.8% | System cards only |
| BRIGHT Retrieval systems find documents for queries from StackExchange, LeetCode and math competitions where relevance needs reasoning. Scored by nDCG@10. | Expert | Checked automatically | Not in cards | |
| FreshQA Models answer questions whose answers change over time, never change, or rest on a false premise that they should reject. FreshEval, an LLM judge, grades answers in a Relaxed mode or a Strict mode that also penalizes any hallucinated detail. | Everyday | Graded by a model | Not in cards | |
| MultiHop-RAG Retrieval-augmented systems answer 2,556 queries whose evidence is spread across two to four news articles. The queries also use document metadata, such as source and date. The benchmark scores both retrieval quality and answer accuracy. | Everyday | Checked automatically | Not in cards | |
| OBLIQ-Bench Retrievers search large real corpora for documents that match a latent pattern, such as an implicit stance. The suite has five such oblique search problems. | Expert | Checked automatically | Not in cards | |
| STaRK Retrieval systems find the right items for complex queries over knowledge bases that mix free text with relations between entities. The three domains are product search, academic paper search and precision medicine. Most queries are synthesized and checked by people, and a smaller set is written by people. | University | Checked automatically | Not in cards |
Language reasoning
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| BIG-Bench Hard Models solve 23 BIG-Bench tasks where earlier models trailed average human raters, such as logic puzzles and date math. Scoring is exact match. | School | Checked automatically | 94.3% | Fading |
| MuSR Models answer questions about long generated narratives such as murder mysteries, object placement and team allocation. Each item needs several soft reasoning steps over the story. | University | Checked automatically | 86.7% | Fading |
| Extended NYT Connections Models solve New York Times Connections puzzles with up to four extra trick words added. Each puzzle scores by the square of the share of groups found. | School | Checked automatically | 99.3% | Not in cards |
| BBEH micro average Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. | University | Checked automatically | 74.4% | In use |
| SnorkelSpatial Models track particles on a 2D grid through 10 to 200 text-described moves, then report final positions and orientations. The 330 samples test allocentric and egocentric spatial reasoning. | School | Checked automatically | 99% | Not in cards |
| ANLI Models label premise and hypothesis pairs as entailment, neutral, or contradiction. People wrote the pairs over three rounds to fool strong models. | Everyday | Checked automatically | 73.1% | Dropped |
| ToRR Models answer questions over tables given in several formats, and the board checks how much accuracy changes when the table format or prompt changes. | University | Checked automatically | 49.9 | Not in cards |
| BIG-Bench Extra Hard Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. | University | Checked automatically | 44.8% | Not in cards |
| SuperGLUE Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score. | Everyday | Checked automatically | 90.4% | Not in cards |
| TableBench Models answer 886 questions about real-world tables across 18 fields in four groups: fact checking, numerical reasoning, data analysis and visualization. Fact and numeric answers are scored by exact match, analysis answers by ROUGE-L, and chart code by pass@1. | University | Checked automatically | 61.7% | Not in cards |
| CB Models label whether a text commits its speaker to an embedded clause as entailment, contradiction, or neutral. Scored by accuracy and F1. | Everyday | Checked automatically | 87.5% | Dropped |
| WorldSense Models answer synthetic questions that test consistent reasoning about simple arrangements of entities described in text. | Everyday | Checked automatically | 67.3% | Fading |
| LSAT Models take a simulated Law School Admission Test. Scored on the 120 to 180 scale. | University | Checked automatically | 163.0 | System cards only |
| RTE Models decide whether a premise sentence entails a hypothesis, using data from the PASCAL RTE challenges. Scored by accuracy. | Everyday | Checked automatically | 79.3% | Dropped |
| StrategyQA Models answer 2,780 yes or no questions that need implicit multi-step reasoning, such as whether Aristotle used a laptop. Scoring is accuracy. | Everyday | Checked automatically | 90.4% | Dropped |
| WiC Models decide whether a target word has the same meaning in two sentence contexts. Scoring is binary accuracy. | Everyday | Checked automatically | 66.8% | Dropped |
| ACPBench Models answer boolean and multiple-choice questions on 7 planning reasoning tasks over 13 formally described planning domains. Answers are provably correct. | University | Checked automatically |  62.9% | System cards only |
| ANLI R1 Models label premise and hypothesis pairs from the first adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. | Everyday | Checked automatically | Not in cards | |
| ANLI R2 Models label premise and hypothesis pairs from the second adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. | Everyday | Checked automatically | Not in cards | |
| ANLI R3 Models label premise and hypothesis pairs from the third adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. | Everyday | Checked automatically | Not in cards | |
| BIG-bench Language models do over 200 tasks contributed by hundreds of authors, from linguistics and math to social bias. Most tasks are scored by exact match or multiple choice. | University | Checked automatically | Not in cards | |
| HANS An entailment test built to catch three shallow shortcuts: lexical overlap, subsequence and constituent. Every premise and hypothesis pair shares words, but only some pairs are entailments. A model that relies on word overlap fails the non-entailment cases. | Everyday | Checked automatically | Not in cards | |
| Hindsight Neglect Models judge whether a bet was worth taking based on expected value rather than its outcome. Scoring is accuracy. | Everyday | Checked automatically | System cards only | |
| LogiQA Models answer reading comprehension questions that test logical reasoning, taken from Chinese civil service exams and translated to English. | University | Checked automatically | Not in cards | |
| PAWS Models decide whether two sentences with heavy word overlap are paraphrases. | Everyday | Checked automatically | System cards only | |
| PlanningBench Models solve generated planning problems that coordinate goals, constraints and resources. Instances are scalable and come with verifiers. | Expert | Checked automatically | Not in cards | |
| RuleTrek Agents must find and correctly apply domain-specific rules when the scenario's facts are spread across many files, unlike rule benchmarks built from a few statements. | Expert | Checked automatically | Not in cards | |
| WikiTableQuestions Models answer complex questions about semi-structured HTML tables from Wikipedia. Answers often need counting, comparison or aggregation across rows. Scored by answer accuracy against the gold answers. | Everyday | Checked automatically | Not in cards |