Benchmark Almanac

BoolQ

PaLM 2-L 90.9% leads since May 2023
PaLM (540B) 91.9% leads since April 2022
PaLM (540B) 91.9% leads since April 2022
Dropped

Models read a Wikipedia passage and answer a naturally occurring yes or no question about it. Scored by accuracy.

Part of SuperGLUE. The paper reports 90 percent human accuracy, which large models now approach.

Made by
Google
Released
May 2019
Kind
Reading comprehension
Level
Everyday
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Strongest model tested
Llama 3.1-405B #162 on the capability index
Last in a launch post
June 2025, dropped after it saturated
Last model added
June 2025
Scores refreshed
November 12th, 2026

In the authors’ words

In this paper we study yes/no questions that are naturally occurring — meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.

Abstract of BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (arXiv:1905.10044, CC0)
28
Models scored in lab cards
11
Launch posts list it, from 3 labs
25
System cards report it, from 4 labs

PaLM 2-L leads with 90.9% among models released by June 2025

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

PaLM (540B) leads with 91.9% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.

PaLM (540B) leads with 91.9%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in June 2025, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

28 models. Scroll sideways for the rest.

  1. 90.9 PaLM 2-L
  2. 87.9 ChatGLM3-6B
  3. 86.6 phi-3-medium 14B
  4. 85.0 Llama 2-70B
  5. 84.8 Gemma 2 27B
  6. 84.8 phi-3-small 7.4B
  7. 84.6 Phi-3.5-MoE
  8. 84.2 Gemma 2 9B
  9. 83.7 Llama 2-34B
  10. 83.3 Phi-2
  11. 83.2 Gemma 7B
  12. 82.4 Gemma 3 27B
  13. 81.7 Llama 2-13B
  14. 81.6 Gemma 3n E4B
  15. 81.2 Phi-4 Mini
  16. 80.2 BitNet b1.58 2B4T
  17. 80.0 Llama 3.1-405B
  18. 79.4 Llama 3.1-70B
  19. 79.0 Llama 3-70B
  20. 78.8 Gemma 3 12B
  21. 78.0 Phi-3.5-mini
  22. 77.4 Llama 2-7B
  23. 77.2 phi-3-mini 3.8B
  24. 75.7 Llama 3-8B
  25. 75.0 Llama 3.1-8B
  26. 72.3 Gemma 3 4B
  27. 69.4 Gemma 2B
  28. 63.2 Gemma 3 1B
  1. 1 PaLM 2-L 90.9%
  2. 2 ChatGLM3-6B 87.9%
  3. 3 phi-3-medium 14B 86.6%
  4. 4 Llama 2-70B 85%
  5. 5 Gemma 2 27B 84.8%
  6. 6 phi-3-small 7.4B 84.8%
  7. 7 Phi-3.5-MoE 84.6%
  8. 8 Gemma 2 9B 84.2%
  9. 9 Llama 2-34B 83.7%
  10. 10 Phi-2 83.3%
  11. 11 Gemma 7B 83.2%
  12. 12 Gemma 3 27B 82.4%
  13. 13 Llama 2-13B 81.7%
  14. 14 Gemma 3n E4B 81.6%
  15. 15 Phi-4 Mini 81.2%
  16. 16 BitNet b1.58 2B4T 80.2%
  17. 17 Llama 3.1-405B 80%
  18. 18 Llama 3.1-70B 79.4%
  19. 19 Llama 3-70B 79%
  20. 20 Gemma 3 12B 78.8%
  21. 21 Phi-3.5-mini 78%
  22. 22 Llama 2-7B 77.4%
  23. 23 phi-3-mini 3.8B 77.2%
  24. 24 Llama 3-8B 75.7%
  25. 25 Llama 3.1-8B 75%
  26. 26 Gemma 3 4B 72.3%
  27. 27 Gemma 2B 69.4%
  28. 28 Gemma 3 1B 63.2%

Source: Labs’ model cards

Source: Official BoolQ leaderboard

79 models. Scroll sideways for the rest.

  1. 91.9 PaLM (540B)
  2. 91.2 T5-11B
  3. 90.9 PaLM 2-L
  4. 89.9 T5-3B
  5. 89.7 Inflection-1
  6. 89.4 Stable Beluga 2
  7. 89.0 Falcon-180B
  8. 88.7 GPT-4o mini
  9. 88.6 PaLM 2-M
  10. 88.1 PaLM 2-S
  11. 87.9 ChatGLM3-6B
  12. 87.7 Llama 2-70B
  13. 87.5 internlm-20b
  14. 86.6 LLaMA-65B
  15. 86.6 phi-3-medium 14B
  16. 86.2 Qwen-14B
  17. 85.8 Gemini 1.5 Flash
  18. 85.7 Gemma 2 9B
  19. 85.4 T5-Large
  20. 84.8 Megatron-Turing NLG 530B
  21. 84.8 PaLM 62B
  22. 84.8 Gemma 2 27B
  23. 84.8 phi-3-small 7.4B
  24. 84.6 Phi-3.5-MoE
  25. 84.4 LLaMA-33B
  26. 83.7 Chinchilla
  27. 83.7 Llama 2-34B
  28. 83.5 vicuna-13b-v1.1
  29. 83.3 Phi-2
  30. 83.2 Mistral 7B v0.2
  31. 83.2 Gemma 7B
  32. 83.1 Falcon-40B
  33. 82.8 Llama 3.1-8B
  34. 82.5 Mistral NeMo
  35. 82.4 Llama 2-13B
  36. 82.4 Gemma 3 27B
  37. 81.6 Gemma 3n E4B
  38. 81.4 T5-Base
  39. 81.2 Phi-4 Mini
  40. 80.5 Mistral 7B v0.1
  41. 80.2 BitNet b1.58 2B4T
  42. 80.0 Llama 3.1-405B
  43. 79.4 Gopher (280B)
  44. 79.4 Llama 3.1-70B
  45. 79.0 MPT-30B
  46. 79.0 chatglm2-6b
  47. 79.0 Llama 3-70B
  48. 78.8 Gemma 3 12B
  49. 78.7 LLaMA-13B
  50. 78.0 Phi-3.5-mini
  51. 77.9 Llama 2-7B
  52. 77.5 InstructGPT 175B
  53. 77.2 phi-3-mini 3.8B
  54. 76.5 LLaMA-7B
  55. 76.4 T5-Small
  56. 76.4 Qwen-7B
  57. 76.4 GPT-3 175B (davinci)
  58. 75.8 Phi-1.5
  59. 75.7 Llama 3-8B
  60. 75.0 MPT-7B
  61. 74.3 XGen-7B
  62. 73.8 Falcon-7B
  63. 72.3 Gemma 3 4B
  64. 70.6 open_llama_7b
  65. 69.4 Gemma 2B
  66. 69.3 RedPajama-INCITE-7B-Base
  67. 68.0 Qwen-1_8B
  68. 67.0 Baichuan2-13B
  69. 65.4 GPT-J-6B
  70. 65.0 OPT-1.3B
  71. 64.9 GPT-NeoX-20B
  72. 64.1 internlm-7b
  73. 63.2 Baichuan 2-7B
  74. 63.2 Gemma 3 1B
  75. 61.8 GPT-2 (1.5B)
  76. 61.8 GPT-Neo-2.7B
  77. 61.1 Cerebras-GPT-13B
  78. 59.0 stablelm-tuned-alpha-7b
  79. 56.3 Dolly 2.0-12b
  1. 1 PaLM (540B) 91.9%
  2. 2 T5-11B 91.2%
  3. 3 PaLM 2-L 90.9%
  4. 4 T5-3B 89.9%
  5. 5 Inflection-1 89.7%
  6. 6 Stable Beluga 2 89.4%
  7. 7 Falcon-180B 89%
  8. 8 GPT-4o mini 88.7%
  9. 9 PaLM 2-M 88.6%
  10. 10 PaLM 2-S 88.1%
  11. 11 ChatGLM3-6B 87.9%
  12. 12 Llama 2-70B 87.7%
  13. 13 internlm-20b 87.5%
  14. 14 LLaMA-65B 86.6%
  15. 15 phi-3-medium 14B 86.6%
  16. 16 Qwen-14B 86.2%
  17. 17 Gemini 1.5 Flash May 2024 85.8%
  18. 18 Gemma 2 9B 85.7%
  19. 19 T5-Large 85.4%
  20. 20 Megatron-Turing NLG 530B 84.8%
  21. 21 PaLM 62B 84.8%
  22. 22 Gemma 2 27B 84.8%
  23. 23 phi-3-small 7.4B 84.8%
  24. 24 Phi-3.5-MoE 84.6%
  25. 25 LLaMA-33B 84.4%
  26. 26 Chinchilla 83.7%
  27. 27 Llama 2-34B 83.7%
  28. 28 vicuna-13b-v1.1 83.5%
  29. 29 Phi-2 83.3%
  30. 30 Mistral 7B v0.2 83.2%
  31. 31 Gemma 7B 83.2%
  32. 32 Falcon-40B 83.1%
  33. 33 Llama 3.1-8B 82.8%
  34. 34 Mistral NeMo 82.5%
  35. 35 Llama 2-13B 82.4%
  36. 36 Gemma 3 27B 82.4%
  37. 37 Gemma 3n E4B 81.6%
  38. 38 T5-Base 81.4%
  39. 39 Phi-4 Mini 81.2%
  40. 40 Mistral 7B v0.1 80.5%
  41. 41 BitNet b1.58 2B4T 80.2%
  42. 42 Llama 3.1-405B 80%
  43. 43 Gopher (280B) 79.4%
  44. 44 Llama 3.1-70B 79.4%
  45. 45 MPT-30B 79%
  46. 46 chatglm2-6b 79%
  47. 47 Llama 3-70B 79%
  48. 48 Gemma 3 12B 78.8%
  49. 49 LLaMA-13B 78.7%
  50. 50 Phi-3.5-mini 78%
  51. 51 Llama 2-7B 77.9%
  52. 52 InstructGPT 175B 77.5%
  53. 53 phi-3-mini 3.8B 77.2%
  54. 54 LLaMA-7B 76.5%
  55. 55 T5-Small 76.4%
  56. 56 Qwen-7B 76.4%
  57. 57 GPT-3 175B (davinci) 76.4%
  58. 58 Phi-1.5 75.8%
  59. 59 Llama 3-8B 75.7%
  60. 60 MPT-7B 75%
  61. 61 XGen-7B 74.3%
  62. 62 Falcon-7B 73.8%
  63. 63 Gemma 3 4B 72.3%
  64. 64 open_llama_7b 70.6%
  65. 65 Gemma 2B 69.4%
  66. 66 RedPajama-INCITE-7B-Base 69.3%
  67. 67 Qwen-1_8B 68%
  68. 68 Baichuan2-13B 67%
  69. 69 GPT-J-6B 65.4%
  70. 70 OPT-1.3B 65%
  71. 71 GPT-NeoX-20B 64.9%
  72. 72 internlm-7b 64.1%
  73. 73 Baichuan 2-7B 63.2%
  74. 74 Gemma 3 1B 63.2%
  75. 75 GPT-2 (1.5B) 61.8%
  76. 76 GPT-Neo-2.7B 61.8%
  77. 77 Cerebras-GPT-13B 61.1%
  78. 78 stablelm-tuned-alpha-7b 59%
  79. 79 Dolly 2.0-12b 56.3%

Sources: Official BoolQ leaderboard, Another board, via Epoch AI (CC BY 4.0) and labs’ model cards

#ModelScoreSettingDateSource
1 PaLM 2-L 90.9% May 2023 Lab model card
2 ChatGLM3-6B 87.9% Jun 2024 Lab model card
3 phi-3-medium 14B 86.6% Apr 2024 Lab model card
4 Llama 2-70B 85% Jul 2023 Lab model card
5 Gemma 2 27B 84.8% Jun 2024 Lab model card
6 phi-3-small 7.4B 84.8% Aug 2024 Lab model card
7 Phi-3.5-MoE 84.6% Aug 2024 Lab model card
8 Gemma 2 9B 84.2% Jul 2024 Lab model card
9 Llama 2-34B 83.7% Jul 2023 Lab model card
10 Phi-2 83.3% Dec 2023 Lab model card
11 Gemma 7B 83.2% Feb 2024 Lab model card
12 Gemma 3 27B 82.4% Mar 2025 Lab model card
13 Llama 2-13B 81.7% Jul 2023 Lab model card
14 Gemma 3n E4B 81.6% Jun 2025 Lab model card
15 Phi-4 Mini 81.2% Feb 2025 Lab model card
16 BitNet b1.58 2B4T 80.2% Apr 2025 Lab model card
17 Llama 3.1-405B 80% Jul 2024 Lab model card
18 Llama 3.1-70B 79.4% Jul 2024 Lab model card
19 Llama 3-70B 79% Apr 2024 Lab model card
20 Gemma 3 12B 78.8% Mar 2025 Lab model card
21 Phi-3.5-mini 78% Aug 2024 Lab model card
22 Llama 2-7B 77.4% Jul 2023 Lab model card
23 phi-3-mini 3.8B 77.2% Aug 2024 Lab model card
24 Llama 3-8B 75.7% Apr 2024 Lab model card
25 Llama 3.1-8B 75% Jul 2024 Lab model card
26 Gemma 3 4B 72.3% Mar 2025 Lab model card
27 Gemma 2B 69.4% Mar 2024 Lab model card
28 Gemma 3 1B 63.2% Mar 2025 Lab model card
#ModelScoreSettingDateSource
1 PaLM (540B) 91.9% Apr 2022 Official leaderboard
2 T5-11B 91.2% Oct 2019 Official leaderboard
3 GPT-3 175B (davinci) 76.4% May 2020 Official leaderboard
#ModelScoreSettingDateSource
1 PaLM (540B) 91.9% Apr 2022 Official leaderboard
2 T5-11B 91.2% Oct 2019 Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI
3 PaLM 2-L 90.9% May 2023 Lab model card
4 T5-3B 89.9% Oct 2019 Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI
5 Inflection-1 89.7% Jun 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
6 Stable Beluga 2 89.4% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
7 Falcon-180B 89% Sep 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
8 GPT-4o mini 88.7% Jul 2024 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
9 PaLM 2-M 88.6% May 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
10 PaLM 2-S 88.1% May 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
11 ChatGLM3-6B 87.9% Jun 2024 Lab model card
12 Llama 2-70B 87.7% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
13 internlm-20b 87.5% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
14 LLaMA-65B 86.6% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
15 phi-3-medium 14B 86.6% Apr 2024 Lab model card
16 Qwen-14B 86.2% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
17 Gemini 1.5 Flash May 2024 85.8% May 2024 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
18 Gemma 2 9B 85.7% Jun 2024 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
19 T5-Large 85.4% Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI
20 Megatron-Turing NLG 530B 84.8% Oct 2021 Claim in Using DeepSpeed and Megatron to Train Megatron-Turing NLG…, via Epoch AI
21 PaLM 62B 84.8% Apr 2022 Claim in LLaMA: Open and Efficient Foundation Language Models, via Epoch AI
22 Gemma 2 27B 84.8% Jun 2024 Lab model card
23 phi-3-small 7.4B 84.8% Aug 2024 Lab model card
24 Phi-3.5-MoE 84.6% Aug 2024 Lab model card
25 LLaMA-33B 84.4% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
26 Chinchilla 83.7% Mar 2022 Claim in LLaMA: Open and Efficient Foundation Language Models, via Epoch AI
27 Llama 2-34B 83.7% Jul 2023 Lab model card
28 vicuna-13b-v1.1 83.5% Apr 2023 Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI
29 Phi-2 83.3% Dec 2023 Lab model card
30 Mistral 7B v0.2 83.2% Dec 2023 Claim in Gemma: Open Models Based on Gemini Research and Technology, via Epoch AI
31 Gemma 7B 83.2% Feb 2024 Lab model card
32 Falcon-40B 83.1% Mar 2023 Claim in Llama 2: Open Foundation and Fine-Tuned Chat Models, via Epoch AI
33 Llama 3.1-8B 82.8% Jul 2024 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
34 Mistral NeMo 82.5% Jul 2024 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
35 Llama 2-13B 82.4% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
36 Gemma 3 27B 82.4% Mar 2025 Lab model card
37 Gemma 3n E4B 81.6% Jun 2025 Lab model card
38 T5-Base 81.4% Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI
39 Phi-4 Mini 81.2% Feb 2025 Lab model card
40 Mistral 7B v0.1 80.5% Sep 2023 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
41 BitNet b1.58 2B4T 80.2% Apr 2025 Lab model card
42 Llama 3.1-405B 80% Jul 2024 Lab model card
43 Gopher (280B) 79.4% Dec 2021 Claim in The Falcon Series of Open Language Models, via Epoch AI
44 Llama 3.1-70B 79.4% Jul 2024 Lab model card
45 MPT-30B 79% Jun 2023 Claim in Llama 2: Open Foundation and Fine-Tuned Chat Models, via Epoch AI
46 chatglm2-6b 79% Jun 2023 Claim in Qwen Technical Report, via Epoch AI
47 Llama 3-70B 79% Apr 2024 Lab model card
48 Gemma 3 12B 78.8% Mar 2025 Lab model card
49 LLaMA-13B 78.7% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
50 Phi-3.5-mini 78% Aug 2024 Lab model card
51 Llama 2-7B 77.9% Jul 2023 Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI
52 InstructGPT 175B 77.5% Jan 2022 Claim in GLaM: Efficient Scaling of Language Models with Mixture-o…, via Epoch AI
53 phi-3-mini 3.8B 77.2% Aug 2024 Lab model card
54 LLaMA-7B 76.5% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
55 T5-Small 76.4% Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI
56 Qwen-7B 76.4% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
57 GPT-3 175B (davinci) 76.4% May 2020 Official leaderboard
58 Phi-1.5 75.8% Sep 2023 Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI
59 Llama 3-8B 75.7% Apr 2024 Lab model card
60 MPT-7B 75% May 2023 Claim in Qwen Technical Report, via Epoch AI
61 XGen-7B 74.3% Jun 2023 Claim in XGen-7B Technical Report, via Epoch AI
62 Falcon-7B 73.8% Apr 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
63 Gemma 3 4B 72.3% Mar 2025 Lab model card
64 open_llama_7b 70.6% Jun 2023 Claim in XGen-7B Technical Report, via Epoch AI
65 Gemma 2B 69.4% Mar 2024 Lab model card
66 RedPajama-INCITE-7B-Base 69.3% May 2023 Claim in XGen-7B Technical Report, via Epoch AI
67 Qwen-1_8B 68% Nov 2023 Claim in Qwen Technical Report, via Epoch AI
68 Baichuan2-13B 67% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
69 GPT-J-6B 65.4% Aug 2021 Claim in XGen-7B Technical Report, via Epoch AI
70 OPT-1.3B 65% May 2022 Claim in XGen-7B Technical Report, via Epoch AI
71 GPT-NeoX-20B 64.9% Apr 2022 Claim in XGen-7B Technical Report, via Epoch AI
72 internlm-7b 64.1% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
73 Baichuan 2-7B 63.2% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
74 Gemma 3 1B 63.2% Mar 2025 Lab model card
75 GPT-2 (1.5B) 61.8% Nov 2019 Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI
76 GPT-Neo-2.7B 61.8% Mar 2021 Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI
77 Cerebras-GPT-13B 61.1% Mar 2023 Claim in XGen-7B Technical Report, via Epoch AI
78 stablelm-tuned-alpha-7b 59% Apr 2023 Claim in XGen-7B Technical Report, via Epoch AI
79 Dolly 2.0-12b 56.3% Apr 2023 Claim in XGen-7B Technical Report, via Epoch AI

The benchmark's own leaderboard has 26 entries, 3 of them models we track. Epoch AI collected claims for 61 models from labs’ reports and papers. Switch the scores above to rank them together.

Number one over time Number one over time Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Google DeepMind Other labs
Own leaderboard Passed 85% in Oct 2019, by T5-11B
PaLM (540B) 91.9%
Lab claims Passed 85% in May 2023, by PaLM 2-L
PaLM 2-L 90.9%

Sources: Official BoolQ leaderboard and labs’ model cards

A claim for T5-11B that Epoch AI collected passed 85% in October 2019

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card A claim Epoch AI collected from a report or paper Best so far
0% 20% 40% 60% 80% 100% 2020 2021 2022 2023 2024 2025 Near the ceiling (85%) Random guessing PaLM (540B) T5-11B 0% 20% 40% 60% 80% 100% 2020 2022 2024 Near the ceiling (85%) Random guessing PaLM (540B)

Sources: Official BoolQ leaderboard, labs’ model cards and claims collected by Epoch AI (CC BY 4.0)

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Google DeepMind
Microsoft
Meta
DateLaunch postScoreAs the post writes it
Jun 26, 2025 Gemma 3n E4B 81.6 BoolQ 0-shot (PT)
Apr 15, 2025 BitNet b1.58 2B4T 80.2 BoolQ
Mar 12, 2025 Gemma 3 27B 82.4 BoolQ 0-shot (PT)
Feb 26, 2025 Phi-4 Mini 81.2 BoolQ (2-shot)
Aug 20, 2024 Phi-3.5-MoE 84.6 BoolQ (2-shot)
Jun 27, 2024 Gemma 2 27B 84.8 BoolQ (0-shot)
Apr 22, 2024 phi-3-medium 14B 86.6 BoolQ (0-Shot)
Apr 18, 2024 Llama 3-70B 79 BoolQ (0-shot) [pretrained]
Feb 21, 2024 Gemma 7B 83.2 BoolQ (0-shot)
Dec 12, 2023 Phi-2 83.3 BoolQ, Table 2
May 10, 2023 PaLM 2-L 90.9 BoolQ, 1-shot

25 system cards report it, from 4 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Apr 16, 2025 BitNet b1.58 2B4T technical report 80.2 BoolQ (0-shot; Acc) (Table 1, Table 3) Capabilities
Mar 25, 2025 Gemma 3 4B technical report 72.3 BoolQ, PT, 0-shot (Table 9) Capabilities
Mar 25, 2025 Gemma 3 27B technical report 82.4 BoolQ, PT, 0-shot (Table 9) Capabilities
Mar 25, 2025 Gemma 3 1B technical report 63.2 BoolQ, PT, 0-shot (Table 9) Capabilities
Mar 25, 2025 Gemma 3 12B technical report 78.8 BoolQ, PT, 0-shot (Table 9) Capabilities
Mar 3, 2025 Phi-4 Mini technical report 81.2 BoolQ (2-Shot) (Table 7) Capabilities
Aug 30, 2024 phi-3-small 7.4B technical report 84.8 BoolQ (2-Shot) (Section 3 table) Capabilities
Aug 30, 2024 phi-3-mini 3.8B technical report 77.2 BoolQ (2-Shot) (Section 3 table) Capabilities
Aug 30, 2024 phi-3-medium 14B technical report 86.5 BoolQ (2-Shot) (Section 3 table) Capabilities
Aug 30, 2024 Phi-3.5-MoE technical report 84.6 BoolQ (2-shot) (Table 3) Capabilities
Aug 30, 2024 Phi-3.5-mini technical report 78 BoolQ (2-shot) (Table 3) Capabilities
Jul 31, 2024 Gemma 2 9B technical report 84.2 Boolq, PT, 0-shot (Table 13) Capabilities
Jul 31, 2024 Gemma 2 27B technical report 84.8 Boolq, PT, 0-shot (Table 13) Capabilities
Jul 23, 2024 Llama 3.1-8B model card 75 BoolQ, 0-shot, acc_char, pretrained (Base pretrained models table) Capabilities
Jul 23, 2024 Llama 3.1-70B model card 79.4 BoolQ, 0-shot, acc_char, pretrained (Base pretrained models table) Capabilities
Jul 23, 2024 Llama 3.1-405B model card 80 BoolQ, 0-shot, acc_char, pretrained (Base pretrained models table) Capabilities
Jun 18, 2024 ChatGLM3-6B technical report 87.9 ChatGLM3-6B-Base, BoolQ (Table 1) Capabilities
Apr 18, 2024 Llama 3-8B model card 75.7 BoolQ (0-shot), pretrained (Base pretrained models table) Capabilities
Apr 18, 2024 Llama 3-70B model card 79 BoolQ (0-shot), pretrained (Base pretrained models table) Capabilities
Mar 13, 2024 Gemma 7B technical report 83.2 Boolq, PT, 0-shot (Table 6) Capabilities
Mar 13, 2024 Gemma 2B technical report 69.4 Boolq, PT, 0-shot (Table 6) Capabilities
Jul 19, 2023 Llama 2-7B technical report 77.4 BoolQ, 0-shot, pretrained (Table 20) Capabilities
Jul 19, 2023 Llama 2-70B technical report 85 BoolQ, 0-shot, pretrained (Table 20) Capabilities
Jul 19, 2023 Llama 2-34B technical report 83.7 BoolQ, 0-shot, pretrained (Table 20) Capabilities
Jul 19, 2023 Llama 2-13B technical report 81.7 BoolQ, 0-shot, pretrained (Table 20) Capabilities

More in Knowledge

All 121 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • KMMLU-Pro In use Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. Solar Pro 4 79.2%
  • HLE-Verified In use Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Seed2.0 Pro 73.6%