QuALITY
Models answer multiple-choice questions about English passages that average about 5,000 tokens. Scoring is accuracy.
About half the questions were too hard for annotators to answer under tight time limits.
- Made by
- New York University
- Released
- December 2021
- Kind
- Reasoning over long documents
- Level
- School
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Strongest model tested
-
Llama 3.1-405B #162 on the capability index - Last in a launch post
- July 2023, dropped before it saturated
- Last model added
- November 2024
- Scores refreshed
- November 12th, 2026
In the authors’ words
To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).
Claude 3 Opus leads with 90.5% among models released by July 2024
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
GPT-4o leads with 83.1% across all independent runs
Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.
Claude 3 Opus leads with 90.5%, claims included
Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.
The newest model with a score here came out in July 2024, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- Claude Fable 5.2
-
Gemini 4 Carbon - GPT-6 Astra
- Claude Opus 5.5
- and 16 more
- 1 Claude 3 Opus 90.5%
- 2 Llama 3.1-405B 87.6%
- 3 Claude 3 Sonnet 85.9%
- 4 Claude 2 83.2%
- 5 Llama 3.1-70B 82.8%
- 6 Claude 3 Haiku 80.2%
- 7 Llama 3.1-8B 56%
Source: Labs’ model cards
- 1 GPT-4o Aug 2024 83.1%
- 2 GPT-4 Mar 2023 82.6%
- 3 GPT-3.5 Turbo Jun 2023 74.7%
- 4 GPT-4o mini 71.7%
Source: Official QuALITY leaderboard
- 1 Claude 3 Opus 90.5%
- 2 Llama 3.1-405B 87.6%
- 3 Claude 3 Sonnet 85.9%
- 4 Claude 2 83.2%
- 5 GPT-4o Aug 2024 83.1%
- 6 Llama 3.1-70B 82.8%
- 7 GPT-4 Mar 2023 82.6%
- 8 Claude 3 Haiku 80.2%
- 9 GPT-3.5 Turbo Jun 2023 74.7%
- 10 GPT-4o mini 71.7%
- 11 Llama 3.1-8B 56%
Sources: Labs’ model cards and official QuALITY leaderboard
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude 3 Opus | 90.5% | Mar 2024 | Lab model card | |
| 2 | Llama 3.1-405B | 87.6% | Nov 2024 | Lab model card | |
| 3 | Claude 3 Sonnet | 85.9% | Mar 2024 | Lab model card | |
| 4 | Claude 2 | 83.2% | Jul 2023 | Lab model card | |
| 5 | Llama 3.1-70B | 82.8% | Nov 2024 | Lab model card | |
| 6 | Claude 3 Haiku | 80.2% | Mar 2024 | Lab model card | |
| 7 | Llama 3.1-8B | 56% | Nov 2024 | Lab model card |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-4o Aug 2024 | 83.1% | RAPTOR | Aug 2024 | Official leaderboard |
| 2 | GPT-4 Mar 2023 | 82.6% | RAPTOR (collapsed tree) | Mar 2023 | Official leaderboard |
| 3 | GPT-3.5 Turbo Jun 2023 | 74.7% | Jun 2023 | Official leaderboard | |
| 4 | GPT-4o mini | 71.7% | RAPTOR | Jul 2024 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude 3 Opus | 90.5% | Mar 2024 | Lab model card | |
| 2 | Llama 3.1-405B | 87.6% | Nov 2024 | Lab model card | |
| 3 | Claude 3 Sonnet | 85.9% | Mar 2024 | Lab model card | |
| 4 | Claude 2 | 83.2% | Jul 2023 | Lab model card | |
| 5 | GPT-4o Aug 2024 | 83.1% | RAPTOR | Aug 2024 | Official leaderboard |
| 6 | Llama 3.1-70B | 82.8% | Nov 2024 | Lab model card | |
| 7 | GPT-4 Mar 2023 | 82.6% | RAPTOR (collapsed tree) | Mar 2023 | Official leaderboard |
| 8 | Claude 3 Haiku | 80.2% | Mar 2024 | Lab model card | |
| 9 | GPT-3.5 Turbo Jun 2023 | 74.7% | Jun 2023 | Official leaderboard | |
| 10 | GPT-4o mini | 71.7% | RAPTOR | Jul 2024 | Official leaderboard |
| 11 | Llama 3.1-8B | 56% | Nov 2024 | Lab model card |
The benchmark's own leaderboard has 17 entries, 4 of them models we track. Switch the scores above to rank them together.
2 models have held number one in labs’ own numbers since 2023 2 models have held number one across all independent runs since 2023 3 models have held number one with claims included since 2023
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
Sources: Official QuALITY leaderboard and labs’ model cards
Anthropic’s own number for Claude 3 Opus passed 85% in March 2024
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Sources: Official QuALITY leaderboard and labs’ model cards
No launch post has listed it since July 2023
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Jul 11, 2023 | Claude 2 | 83.2 | QuALITY (5-shot) |
7 system cards report it, from 2 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Nov 23, 2024 | Llama 3.1-8B technical report | 56 | QuALITY (5-shot), pretrained long-context (Table 14) | Capabilities |
| Nov 23, 2024 | Llama 3.1-70B technical report | 82.8 | QuALITY (5-shot), pretrained long-context (Table 14) | Capabilities |
| Nov 23, 2024 | Llama 3.1-405B technical report | 87.6 | QuALITY (5-shot), pretrained long-context (Table 14) | Capabilities |
| Mar 4, 2024 | Claude 3 Sonnet model card | 85.9 | QuALITY, 1-shot. 0-shot 84.9 (Table 6) | Capabilities |
| Mar 4, 2024 | Claude 3 Opus model card | 90.5 | QuALITY, 1-shot. 0-shot 89.2 (Table 6) | Capabilities |
| Mar 4, 2024 | Claude 3 Haiku model card | 80.2 | QuALITY, 1-shot. 0-shot 79.4 (Table 6) | Capabilities |
| Jul 11, 2023 | Claude 2 model card | 83.2 | QuALITY (5-shot) (4.3) | Capabilities |
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
- LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy.
Inkling 2 Small 70.6% - GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. Gemini 3.7 Flash 89%
- CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.
Step 5 Preview 88.3%