BoolQ
The official leaderboard hasn’t scored a model released after PaLM (540B) (April 2022). Labs report their own numbers for 64 newer models, such as PaLM 2-L at 90.9%.
Models read a Wikipedia passage and answer a naturally occurring yes or no question about it. Scored by accuracy.
Part of SuperGLUE. The paper reports 90 percent human accuracy, which large models now approach.
- Made by
- Released
- May 2019
- Kind
- Reading comprehension
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Real work and use
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
-
Llama 3.1-405B #163 on the capability index
- Last in a launch post
- June 2025, dropped after it saturated
- Last model added
- June 2025
- Scores refreshed
- November 19th, 2026
In the authors’ words
In this paper we study yes/no questions that are naturally occurring — meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.
PaLM 2-L leads with 90.9% among models released by June 2025
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
PaLM (540B) leads with 91.9% across all independent runs
Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.
PaLM (540B) leads with 91.9%, claims included
Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.
The newest model with a score here came out in June 2025, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 90.9 PaLM 2-L
- 87.9 ChatGLM3-6B
- 86.6
phi-3-medium 14B
- 85.0
Llama 2-70B
- 84.8
Gemma 2 27B
- 84.8
phi-3-small 7.4B
- 84.6
Phi-3.5-MoE
- 84.2
Gemma 2 9B
- 83.7
Llama 2-34B
- 83.3
Phi-2
- 83.2
Gemma 7B
- 82.4
Gemma 3 27B
- 81.7
Llama 2-13B
- 81.6
Gemma 3n E4B
- 81.2
Phi-4 Mini
- 80.2
BitNet b1.58 2B4T
- 80.0
Llama 3.1-405B
- 79.4
Llama 3.1-70B
- 79.0
Llama 3-70B
- 78.8
Gemma 3 12B
- 78.0
Phi-3.5-mini
- 77.4
Llama 2-7B
- 77.2
phi-3-mini 3.8B
- 75.7
Llama 3-8B
- 75.0
Llama 3.1-8B
- 72.3
Gemma 3 4B
- 69.4
Gemma 2B
- 63.2
Gemma 3 1B
- 1 PaLM 2-L 90.9%
- 2 ChatGLM3-6B 87.9%
- 3
phi-3-medium 14B 86.6%
- 4
Llama 2-70B 85%
- 5
Gemma 2 27B 84.8%
- 6
phi-3-small 7.4B 84.8%
- 7
Phi-3.5-MoE 84.6%
- 8
Gemma 2 9B 84.2%
- 9
Llama 2-34B 83.7%
- 10
Phi-2 83.3%
- 11
Gemma 7B 83.2%
- 12
Gemma 3 27B 82.4%
- 13
Llama 2-13B 81.7%
- 14
Gemma 3n E4B 81.6%
- 15
Phi-4 Mini 81.2%
- 16
BitNet b1.58 2B4T 80.2%
- 17
Llama 3.1-405B 80%
- 18
Llama 3.1-70B 79.4%
- 19
Llama 3-70B 79%
- 20
Gemma 3 12B 78.8%
- 21
Phi-3.5-mini 78%
- 22
Llama 2-7B 77.4%
- 23
phi-3-mini 3.8B 77.2%
- 24
Llama 3-8B 75.7%
- 25
Llama 3.1-8B 75%
- 26
Gemma 3 4B 72.3%
- 27
Gemma 2B 69.4%
- 28
Gemma 3 1B 63.2%
Source: Labs’ model cards
- 1
PaLM (540B) ✓ 91.9%
- 2
T5-11B ✓ 91.2%
- 3 GPT-3 175B (davinci) ✓ 76.4%
Source: Official BoolQ leaderboard
- 91.9
PaLM (540B)
- 91.2
T5-11B
- 90.9 PaLM 2-L
- 89.9
T5-3B
- 89.7 Inflection-1
- 89.4
Stable Beluga 2
- 89.0
Falcon-180B
- 88.7 GPT-4o mini
- 88.6 PaLM 2-M
- 88.1 PaLM 2-S
- 87.9 ChatGLM3-6B
- 87.7
Llama 2-70B
- 87.5 internlm-20b
- 86.6
LLaMA-65B
- 86.6
phi-3-medium 14B
- 86.2
Qwen-14B
- 85.8
Gemini 1.5 Flash
- 85.7
Gemma 2 9B
- 85.4 T5-Large
- 84.8
Megatron-Turing NLG 530B
- 84.8 PaLM 62B
- 84.8
Gemma 2 27B
- 84.8
phi-3-small 7.4B
- 84.6
Phi-3.5-MoE
- 84.4
LLaMA-33B
- 83.7
Chinchilla
- 83.7
Llama 2-34B
- 83.5 vicuna-13b-v1.1
- 83.3
Phi-2
- 83.2
Mistral 7B v0.2
- 83.2
Gemma 7B
- 83.1
Falcon-40B
- 82.8
Llama 3.1-8B
- 82.5
Mistral NeMo
- 82.4
Llama 2-13B
- 82.4
Gemma 3 27B
- 81.6
Gemma 3n E4B
- 81.4 T5-Base
- 81.2
Phi-4 Mini
- 80.5
Mistral 7B v0.1
- 80.2
BitNet b1.58 2B4T
- 80.0
Llama 3.1-405B
- 79.4
Gopher (280B)
- 79.4
Llama 3.1-70B
- 79.0
MPT-30B
- 79.0 chatglm2-6b
- 79.0
Llama 3-70B
- 78.8
Gemma 3 12B
- 78.7
LLaMA-13B
- 78.0
Phi-3.5-mini
- 77.9
Llama 2-7B
- 77.5 InstructGPT 175B
- 77.2
phi-3-mini 3.8B
- 76.5
LLaMA-7B
- 76.4 T5-Small
- 76.4
Qwen-7B
- 76.4 GPT-3 175B (davinci)
- 75.8
Phi-1.5
- 75.7
Llama 3-8B
- 75.0
MPT-7B
- 74.3
XGen-7B
- 73.8
Falcon-7B
- 72.3
Gemma 3 4B
- 70.6 open_llama_7b
- 69.4
Gemma 2B
- 69.3 RedPajama-INCITE-7B-Base
- 68.0 Qwen-1_8B
- 67.0
Baichuan2-13B
- 65.4 GPT-J-6B
- 65.0
OPT-1.3B
- 64.9 GPT-NeoX-20B
- 64.1 internlm-7b
- 63.2
Baichuan 2-7B
- 63.2
Gemma 3 1B
- 61.8 GPT-2 (1.5B)
- 61.8 GPT-Neo-2.7B
- 61.1
Cerebras-GPT-13B
- 59.0 stablelm-tuned-alpha-7b
- 56.3
Dolly 2.0-12b
- 1
PaLM (540B) ✓ 91.9%
- 2
T5-11B 91.2%
- 3 PaLM 2-L 90.9%
- 4
T5-3B 89.9%
- 5 Inflection-1 89.7%
- 6
Stable Beluga 2 89.4%
- 7
Falcon-180B 89%
- 8 GPT-4o mini 88.7%
- 9 PaLM 2-M 88.6%
- 10 PaLM 2-S 88.1%
- 11 ChatGLM3-6B 87.9%
- 12
Llama 2-70B 87.7%
- 13 internlm-20b 87.5%
- 14
LLaMA-65B 86.6%
- 15
phi-3-medium 14B 86.6%
- 16
Qwen-14B 86.2%
- 17
Gemini 1.5 Flash May 2024 85.8%
- 18
Gemma 2 9B 85.7%
- 19 T5-Large 85.4%
- 20
Megatron-Turing NLG 530B 84.8%
- 21 PaLM 62B 84.8%
- 22
Gemma 2 27B 84.8%
- 23
phi-3-small 7.4B 84.8%
- 24
Phi-3.5-MoE 84.6%
- 25
LLaMA-33B 84.4%
- 26
Chinchilla 83.7%
- 27
Llama 2-34B 83.7%
- 28 vicuna-13b-v1.1 83.5%
- 29
Phi-2 83.3%
- 30
Mistral 7B v0.2 83.2%
- 31
Gemma 7B 83.2%
- 32
Falcon-40B 83.1%
- 33
Llama 3.1-8B 82.8%
- 34
Mistral NeMo 82.5%
- 35
Llama 2-13B 82.4%
- 36
Gemma 3 27B 82.4%
- 37
Gemma 3n E4B 81.6%
- 38 T5-Base 81.4%
- 39
Phi-4 Mini 81.2%
- 40
Mistral 7B v0.1 80.5%
- 41
BitNet b1.58 2B4T 80.2%
- 42
Llama 3.1-405B 80%
- 43
Gopher (280B) 79.4%
- 44
Llama 3.1-70B 79.4%
- 45
MPT-30B 79%
- 46 chatglm2-6b 79%
- 47
Llama 3-70B 79%
- 48
Gemma 3 12B 78.8%
- 49
LLaMA-13B 78.7%
- 50
Phi-3.5-mini 78%
- 51
Llama 2-7B 77.9%
- 52 InstructGPT 175B 77.5%
- 53
phi-3-mini 3.8B 77.2%
- 54
LLaMA-7B 76.5%
- 55 T5-Small 76.4%
- 56
Qwen-7B 76.4%
- 57 GPT-3 175B (davinci) ✓ 76.4%
- 58
Phi-1.5 75.8%
- 59
Llama 3-8B 75.7%
- 60
MPT-7B 75%
- 61
XGen-7B 74.3%
- 62
Falcon-7B 73.8%
- 63
Gemma 3 4B 72.3%
- 64 open_llama_7b 70.6%
- 65
Gemma 2B 69.4%
- 66 RedPajama-INCITE-7B-Base 69.3%
- 67 Qwen-1_8B 68%
- 68
Baichuan2-13B 67%
- 69 GPT-J-6B 65.4%
- 70
OPT-1.3B 65%
- 71 GPT-NeoX-20B 64.9%
- 72 internlm-7b 64.1%
- 73
Baichuan 2-7B 63.2%
- 74
Gemma 3 1B 63.2%
- 75 GPT-2 (1.5B) 61.8%
- 76 GPT-Neo-2.7B 61.8%
- 77
Cerebras-GPT-13B 61.1%
- 78 stablelm-tuned-alpha-7b 59%
- 79
Dolly 2.0-12b 56.3%
Sources: Official BoolQ leaderboard, another board, via Epoch AI (CC BY 4.0) and labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | PaLM 2-L | 90.9% | May 2023 | Lab model card | |
| 2 | ChatGLM3-6B | 87.9% | Jun 2024 | Lab model card | |
| 3 | | 86.6% | Apr 2024 | Lab model card | |
| 4 | | 85% | Jul 2023 | Lab model card | |
| 5 | | 84.8% | Jun 2024 | Lab model card | |
| 6 | | 84.8% | Aug 2024 | Lab model card | |
| 7 | | 84.6% | Aug 2024 | Lab model card | |
| 8 | | 84.2% | Jul 2024 | Lab model card | |
| 9 | | 83.7% | Jul 2023 | Lab model card | |
| 10 | | 83.3% | Dec 2023 | Lab model card | |
| 11 | | 83.2% | Feb 2024 | Lab model card | |
| 12 | | 82.4% | Mar 2025 | Lab model card | |
| 13 | | 81.7% | Jul 2023 | Lab model card | |
| 14 | | 81.6% | Jun 2025 | Lab model card | |
| 15 | | 81.2% | Feb 2025 | Lab model card | |
| 16 | | 80.2% | Apr 2025 | Lab model card | |
| 17 | | 80% | Jul 2024 | Lab model card | |
| 18 | | 79.4% | Jul 2024 | Lab model card | |
| 19 | | 79% | Apr 2024 | Lab model card | |
| 20 | | 78.8% | Mar 2025 | Lab model card | |
| 21 | | 78% | Aug 2024 | Lab model card | |
| 22 | | 77.4% | Jul 2023 | Lab model card | |
| 23 | | 77.2% | Aug 2024 | Lab model card | |
| 24 | | 75.7% | Apr 2024 | Lab model card | |
| 25 | | 75% | Jul 2024 | Lab model card | |
| 26 | | 72.3% | Mar 2025 | Lab model card | |
| 27 | | 69.4% | Mar 2024 | Lab model card | |
| 28 | | 63.2% | Mar 2025 | Lab model card |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 91.9% | Apr 2022 | Official leaderboard | ||
| 2 | 91.2% | Oct 2019 | Official leaderboard | ||
| 3 | GPT-3 175B (davinci) | 76.4% | May 2020 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 91.9% | Apr 2022 | Official leaderboard | ||
| 2 | 91.2% | Oct 2019 | Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI | ||
| 3 | PaLM 2-L | 90.9% | May 2023 | Lab model card | |
| 4 | 89.9% | Oct 2019 | Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI | ||
| 5 | Inflection-1 | 89.7% | Jun 2023 | Claim in The Falcon Series of Open Language Models, via Epoch AI | |
| 6 | 89.4% | Jul 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 7 | 89% | Sep 2023 | Claim in The Falcon Series of Open Language Models, via Epoch AI | ||
| 8 | GPT-4o mini | 88.7% | Jul 2024 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| 9 | PaLM 2-M | 88.6% | May 2023 | Claim in The Falcon Series of Open Language Models, via Epoch AI | |
| 10 | PaLM 2-S | 88.1% | May 2023 | Claim in The Falcon Series of Open Language Models, via Epoch AI | |
| 11 | ChatGLM3-6B | 87.9% | Jun 2024 | Lab model card | |
| 12 | 87.7% | Jul 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 13 | internlm-20b | 87.5% | Sep 2023 | Claim in Qwen Technical Report, via Epoch AI | |
| 14 | 86.6% | Feb 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 15 | 86.6% | Apr 2024 | Lab model card | ||
| 16 | 86.2% | Sep 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 17 | 85.8% | May 2024 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | ||
| 18 | 85.7% | Jun 2024 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | ||
| 19 | T5-Large | 85.4% | Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI | ||
| 20 | 84.8% | Oct 2021 | Claim in Using DeepSpeed and Megatron to Train Megatron-Turing NLG…, via Epoch AI | ||
| 21 | PaLM 62B | 84.8% | Apr 2022 | Claim in LLaMA: Open and Efficient Foundation Language Models, via Epoch AI | |
| 22 | 84.8% | Jun 2024 | Lab model card | ||
| 23 | 84.8% | Aug 2024 | Lab model card | ||
| 24 | 84.6% | Aug 2024 | Lab model card | ||
| 25 | 84.4% | Feb 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 26 | 83.7% | Mar 2022 | Claim in LLaMA: Open and Efficient Foundation Language Models, via Epoch AI | ||
| 27 | 83.7% | Jul 2023 | Lab model card | ||
| 28 | vicuna-13b-v1.1 | 83.5% | Apr 2023 | Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI | |
| 29 | 83.3% | Dec 2023 | Lab model card | ||
| 30 | 83.2% | Dec 2023 | Claim in Gemma: Open Models Based on Gemini Research and Technology, via Epoch AI | ||
| 31 | 83.2% | Feb 2024 | Lab model card | ||
| 32 | 83.1% | Mar 2023 | Claim in Llama 2: Open Foundation and Fine-Tuned Chat Models, via Epoch AI | ||
| 33 | 82.8% | Jul 2024 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | ||
| 34 | 82.5% | Jul 2024 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | ||
| 35 | 82.4% | Jul 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 36 | 82.4% | Mar 2025 | Lab model card | ||
| 37 | 81.6% | Jun 2025 | Lab model card | ||
| 38 | T5-Base | 81.4% | Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI | ||
| 39 | 81.2% | Feb 2025 | Lab model card | ||
| 40 | 80.5% | Sep 2023 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | ||
| 41 | 80.2% | Apr 2025 | Lab model card | ||
| 42 | 80% | Jul 2024 | Lab model card | ||
| 43 | 79.4% | Dec 2021 | Claim in The Falcon Series of Open Language Models, via Epoch AI | ||
| 44 | 79.4% | Jul 2024 | Lab model card | ||
| 45 | 79% | Jun 2023 | Claim in Llama 2: Open Foundation and Fine-Tuned Chat Models, via Epoch AI | ||
| 46 | chatglm2-6b | 79% | Jun 2023 | Claim in Qwen Technical Report, via Epoch AI | |
| 47 | 79% | Apr 2024 | Lab model card | ||
| 48 | 78.8% | Mar 2025 | Lab model card | ||
| 49 | 78.7% | Feb 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 50 | 78% | Aug 2024 | Lab model card | ||
| 51 | 77.9% | Jul 2023 | Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI | ||
| 52 | InstructGPT 175B | 77.5% | Jan 2022 | Claim in GLaM: Efficient Scaling of Language Models with Mixture-o…, via Epoch AI | |
| 53 | 77.2% | Aug 2024 | Lab model card | ||
| 54 | 76.5% | Feb 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 55 | T5-Small | 76.4% | Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI | ||
| 56 | 76.4% | Sep 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 57 | GPT-3 175B (davinci) | 76.4% | May 2020 | Official leaderboard | |
| 58 | 75.8% | Sep 2023 | Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI | ||
| 59 | 75.7% | Apr 2024 | Lab model card | ||
| 60 | 75% | May 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 61 | 74.3% | Jun 2023 | Claim in XGen-7B Technical Report, via Epoch AI | ||
| 62 | 73.8% | Apr 2023 | Claim in The Falcon Series of Open Language Models, via Epoch AI | ||
| 63 | 72.3% | Mar 2025 | Lab model card | ||
| 64 | open_llama_7b | 70.6% | Jun 2023 | Claim in XGen-7B Technical Report, via Epoch AI | |
| 65 | 69.4% | Mar 2024 | Lab model card | ||
| 66 | RedPajama-INCITE-7B-Base | 69.3% | May 2023 | Claim in XGen-7B Technical Report, via Epoch AI | |
| 67 | Qwen-1_8B | 68% | Nov 2023 | Claim in Qwen Technical Report, via Epoch AI | |
| 68 | 67% | Sep 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 69 | GPT-J-6B | 65.4% | Aug 2021 | Claim in XGen-7B Technical Report, via Epoch AI | |
| 70 | 65% | May 2022 | Claim in XGen-7B Technical Report, via Epoch AI | ||
| 71 | GPT-NeoX-20B | 64.9% | Apr 2022 | Claim in XGen-7B Technical Report, via Epoch AI | |
| 72 | internlm-7b | 64.1% | Jul 2023 | Claim in Qwen Technical Report, via Epoch AI | |
| 73 | 63.2% | Sep 2023 | Claim in Qwen Technical Report, via Epoch AI | ||
| 74 | 63.2% | Mar 2025 | Lab model card | ||
| 75 | GPT-2 (1.5B) | 61.8% | Nov 2019 | Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI | |
| 76 | GPT-Neo-2.7B | 61.8% | Mar 2021 | Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI | |
| 77 | 61.1% | Mar 2023 | Claim in XGen-7B Technical Report, via Epoch AI | ||
| 78 | stablelm-tuned-alpha-7b | 59% | Apr 2023 | Claim in XGen-7B Technical Report, via Epoch AI | |
| 79 | 56.3% | Apr 2023 | Claim in XGen-7B Technical Report, via Epoch AI |
The benchmark's own leaderboard has 26 entries, 3 of them models we track. Epoch AI collected claims for 61 models from labs’ reports and papers. Switch the scores above to rank them together.
Number one over time Number one over time Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
Sources: Official BoolQ leaderboard and labs’ model cards
A claim for T5-11B that Epoch AI collected passed 85% in October 2019
That was 5 months after its release. All the scores we have, one dot each, so a model tested at several settings appears more than once.
Sources: Official BoolQ leaderboard, labs’ model cards and claims collected by Epoch AI (CC BY 4.0)
Labs dropped it from launch posts once the best score passed 85%
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out. Google DeepMind and Microsoft account for 8 of the 10 releases that named it since 2024.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Jun 26, 2025 | 81.6 | BoolQ 0-shot (PT) | |
| Apr 15, 2025 | 80.2 | BoolQ | |
| Mar 12, 2025 | 82.4 | BoolQ 0-shot (PT) | |
| Feb 26, 2025 | 81.2 | BoolQ (2-shot) | |
| Aug 20, 2024 | 84.6 | BoolQ (2-shot) | |
| Jun 27, 2024 | 84.8 | BoolQ (0-shot) | |
| Apr 22, 2024 | 86.6 | BoolQ (0-Shot) | |
| Apr 18, 2024 | 79 | BoolQ (0-shot) [pretrained] | |
| Feb 21, 2024 | 83.2 | BoolQ (0-shot) | |
| Dec 12, 2023 | 83.3 | BoolQ, Table 2 | |
| May 10, 2023 | 90.9 | BoolQ, 1-shot |
25 system cards report it, from 4 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Apr 16, 2025 | | 80.2 | BoolQ (0-shot; Acc) (Table 1, Table 3) | Capabilities |
| Mar 25, 2025 | | 72.3 | BoolQ, PT, 0-shot (Table 9) | Capabilities |
| Mar 25, 2025 | | 82.4 | BoolQ, PT, 0-shot (Table 9) | Capabilities |
| Mar 25, 2025 | | 63.2 | BoolQ, PT, 0-shot (Table 9) | Capabilities |
| Mar 25, 2025 | | 78.8 | BoolQ, PT, 0-shot (Table 9) | Capabilities |
| Mar 3, 2025 | | 81.2 | BoolQ (2-Shot) (Table 7) | Capabilities |
| Aug 30, 2024 | | 84.8 | BoolQ (2-Shot) (Section 3 table) | Capabilities |
| Aug 30, 2024 | | 77.2 | BoolQ (2-Shot) (Section 3 table) | Capabilities |
| Aug 30, 2024 | | 86.5 | BoolQ (2-Shot) (Section 3 table) | Capabilities |
| Aug 30, 2024 | | 84.6 | BoolQ (2-shot) (Table 3) | Capabilities |
| Aug 30, 2024 | | 78 | BoolQ (2-shot) (Table 3) | Capabilities |
| Jul 31, 2024 | | 84.2 | Boolq, PT, 0-shot (Table 13) | Capabilities |
| Jul 31, 2024 | | 84.8 | Boolq, PT, 0-shot (Table 13) | Capabilities |
| Jul 23, 2024 | | 75 | BoolQ, 0-shot, acc_char, pretrained (Base pretrained models table) | Capabilities |
| Jul 23, 2024 | | 79.4 | BoolQ, 0-shot, acc_char, pretrained (Base pretrained models table) | Capabilities |
| Jul 23, 2024 | | 80 | BoolQ, 0-shot, acc_char, pretrained (Base pretrained models table) | Capabilities |
| Jun 18, 2024 | ChatGLM3-6B technical report | 87.9 | ChatGLM3-6B-Base, BoolQ (Table 1) | Capabilities |
| Apr 18, 2024 | | 75.7 | BoolQ (0-shot), pretrained (Base pretrained models table) | Capabilities |
| Apr 18, 2024 | | 79 | BoolQ (0-shot), pretrained (Base pretrained models table) | Capabilities |
| Mar 13, 2024 | | 83.2 | Boolq, PT, 0-shot (Table 6) | Capabilities |
| Mar 13, 2024 | | 69.4 | Boolq, PT, 0-shot (Table 6) | Capabilities |
| Jul 19, 2023 | | 77.4 | BoolQ, 0-shot, pretrained (Table 20) | Capabilities |
| Jul 19, 2023 | | 85 | BoolQ, 0-shot, pretrained (Table 20) | Capabilities |
| Jul 19, 2023 | | 83.7 | BoolQ, 0-shot, pretrained (Table 20) | Capabilities |
| Jul 19, 2023 | | 81.7 | BoolQ, 0-shot, pretrained (Table 20) | Capabilities |
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the official leaderboard (26 entries), launch posts (11 scores), system cards (25 scores) and claims Epoch AI collected (61 models).
- Pass Models scored 79 models we track, plus 23 other entries on its leaderboard.
- Note Newest score The newest came 17 months ago, in June 2025, for Gemma 3n E4B. That’s expected for a saturated test.
- Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is Llama 3.1-405B (#163).
- Note Headroom Saturated. The best is 91.9%, past 85% since October 2019.
Leaderboard import 3 pass
- Pass Importer We read the board on super.gluebenchmark.com: 3 rows for models we track and 23 other entries at the last read.
- Pass Last read November 19th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.