Benchmark Almanac

ARC-Challenge

Nemotron 3 Ultra 550B-A55B Base 97.3% leads since June 2026
Nemotron 3 Ultra 550B-A55B Base 97.3% leads since June 2026
Dropped

Models answer grade-school science multiple-choice questions that retrieval and word co-occurrence baselines got wrong. Scored by accuracy.

Saturated by frontier models, which score above 95%.

Released
March 2018
Kind
Exam questions
Level
School
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Strongest model tested
DeepSeek-V3 #154 on the capability index
Last in a launch post
August 2025, dropped after it saturated
Last model added
October 2026
Scores refreshed
November 12th, 2026

In the authors’ words

We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.

Abstract of Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (arXiv:1803.05457, CC0)
49
Models scored in lab cards
25
Launch posts list it, from 10 labs
43
System cards report it, from 10 labs

Nemotron 3 Ultra 550B-A55B Base leads with 97.3%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Nemotron 3 Ultra 550B-A55B Base leads with 97.3%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

49 models. Scroll sideways for the rest.

  1. 97.3 Nemotron 3 Ultra 550B-A55B Base
  2. 96.9 Llama 3.1-405B
  3. 96.4 Claude 3 Opus
  4. 96.3 GPT-4
  5. 96.0 Nemotron 3.5 Super 120B-A12B Base
  6. 95.3 DeepSeek-V3
  7. 95.1 PaLM 2-L
  8. 95.0 Sarvam-M
  9. 94.8 Llama 3.1-70B
  10. 94.0 Mistral Large
  11. 93.2 Claude 3 Sonnet
  12. 93.2 Nemotron 3.5 Lightning 30B-A3B Base
  13. 93.0 Llama 3-70B
  14. 92.3 DeepSeek-V2 (MoE-236B, May 2024)
  15. 91.9 Nemotron 3 Nano 30B-A3B Base
  16. 91.6 phi-3-medium 14B
  17. 91.3 Mistral Small 3
  18. 91.0 Claude 2
  19. 91.0 Phi-3.5-MoE
  20. 90.7 phi-3-small 7.4B
  21. 89.2 Claude 3 Haiku
  22. 85.8 Mixtral 8x7B
  23. 85.0 Apertus 70B
  24. 84.9 phi-3-mini 3.8B
  25. 84.6 Phi-3.5-mini
  26. 83.7 Phi-4 Mini
  27. 83.4 Llama 3.1-8B
  28. 78.6 Llama 3-8B
  29. 72.4 Qwen2.5-72B
  30. 71.4 Gemma 2 27B
  31. 70.6 Gemma 3 27B
  32. 68.9 Qwen2-72B
  33. 68.9 Gemma 3 12B
  34. 68.4 Gemma 2 9B
  35. 64.1 DeepSeek LLM 67B
  36. 61.6 Gemma 3n E4B
  37. 61.0 Falcon-H1-34B
  38. 57.4 Llama 2-70B
  39. 56.2 Gemma 3 4B
  40. 55.5 Mistral 7B v0.1
  41. 54.5 Llama 2-34B
  42. 53.5 Falcon-Arabic-7B
  43. 53.2 Gemma 7B
  44. 49.9 BitNet b1.58 2B4T
  45. 49.4 Llama 2-13B
  46. 45.9 Llama 2-7B
  47. 42.1 Gemma 2B
  48. 38.4 Gemma 3 1B
  49. 28.2 Gemma 3 270M
  1. 1 Nemotron 3 Ultra 550B-A55B Base 97.3%
  2. 2 Llama 3.1-405B 96.9%
  3. 3 Claude 3 Opus 96.4%
  4. 4 GPT-4 Mar 2023 96.3%
  5. 5 Nemotron 3.5 Super 120B-A12B Base 96%
  6. 6 DeepSeek-V3 95.3%
  7. 7 PaLM 2-L 95.1%
  8. 8 Sarvam-M 95%
  9. 9 Llama 3.1-70B 94.8%
  10. 10 Mistral Large 94%
  11. 11 Claude 3 Sonnet 93.2%
  12. 12 Nemotron 3.5 Lightning 30B-A3B Base 93.2%
  13. 13 Llama 3-70B 93%
  14. 14 DeepSeek-V2 (MoE-236B, May 2024) 92.3%
  15. 15 Nemotron 3 Nano 30B-A3B Base 91.9%
  16. 16 phi-3-medium 14B 91.6%
  17. 17 Mistral Small 3 91.3%
  18. 18 Claude 2 91%
  19. 19 Phi-3.5-MoE 91%
  20. 20 phi-3-small 7.4B 90.7%
  21. 21 Claude 3 Haiku 89.2%
  22. 22 Mixtral 8x7B 85.8%
  23. 23 Apertus 70B 85%
  24. 24 phi-3-mini 3.8B 84.9%
  25. 25 Phi-3.5-mini 84.6%
  26. 26 Phi-4 Mini 83.7%
  27. 27 Llama 3.1-8B 83.4%
  28. 28 Llama 3-8B 78.6%
  29. 29 Qwen2.5-72B 72.4%
  30. 30 Gemma 2 27B 71.4%
  31. 31 Gemma 3 27B 70.6%
  32. 32 Qwen2-72B 68.9%
  33. 33 Gemma 3 12B 68.9%
  34. 34 Gemma 2 9B 68.4%
  35. 35 DeepSeek LLM 67B 64.1%
  36. 36 Gemma 3n E4B 61.6%
  37. 37 Falcon-H1-34B 61%
  38. 38 Llama 2-70B 57.4%
  39. 39 Gemma 3 4B 56.2%
  40. 40 Mistral 7B v0.1 55.5%
  41. 41 Llama 2-34B 54.5%
  42. 42 Falcon-Arabic-7B 53.5%
  43. 43 Gemma 7B 53.2%
  44. 44 BitNet b1.58 2B4T 49.9%
  45. 45 Llama 2-13B 49.4%
  46. 46 Llama 2-7B 45.9%
  47. 47 Gemma 2B 42.1%
  48. 48 Gemma 3 1B 38.4%
  49. 49 Gemma 3 270M 28.2%

Source: Labs’ model cards

107 models. Scroll sideways for the rest.

  1. 97.3 Nemotron 3 Ultra 550B-A55B Base
  2. 96.9 Llama 3.1-405B
  3. 96.4 Claude 3 Opus
  4. 96.3 GPT-4
  5. 96.0 Nemotron 3.5 Super 120B-A12B Base
  6. 95.3 DeepSeek-V3
  7. 95.1 PaLM 2-L
  8. 95.0 Sarvam-M
  9. 94.8 Llama 3.1-70B
  10. 94.5 Qwen2.5-72B
  11. 94.0 Mistral Large
  12. 93.2 Claude 3 Sonnet
  13. 93.2 Nemotron 3.5 Lightning 30B-A3B Base
  14. 93.0 Llama 3-70B
  15. 92.3 DeepSeek-V2 (MoE-236B, May 2024)
  16. 91.9 Nemotron 3 Nano 30B-A3B Base
  17. 91.6 phi-3-medium 14B
  18. 91.3 Mistral Small 3
  19. 91.0 Claude 2
  20. 91.0 Phi-3.5-MoE
  21. 90.7 phi-3-small 7.4B
  22. 89.2 Claude 3 Haiku
  23. 87.4 GPT-3.5 Turbo
  24. 87.3 Mixtral 8x7B
  25. 86.3 Claude Instant
  26. 86.1 Stable Beluga 2
  27. 85.2 davinci-002
  28. 85.2 PaLM (540B)
  29. 85.0 Apertus 70B
  30. 84.9 phi-3-mini 3.8B
  31. 84.6 Phi-3.5-mini
  32. 84.4 Qwen-14B
  33. 83.7 Phi-4 Mini
  34. 83.4 Llama 3.1-8B
  35. 82.8 Llama 3-8B
  36. 81.7 internlm-20b
  37. 78.6 Mistral 7B v0.1
  38. 78.3 Llama 2-70B
  39. 78.3 Gemma 7B
  40. 75.9 Phi-2
  41. 75.3 Qwen-7B
  42. 71.4 Gemma 2 27B
  43. 70.6 Gemma 3 27B
  44. 70.5 Qwen2.5-Coder-32B
  45. 69.5 LLaMA-65B
  46. 69.5 internlm-7b
  47. 68.9 Qwen2-72B
  48. 68.9 Gemma 3 12B
  49. 68.4 Gemma 2 9B
  50. 67.8 Falcon-180B
  51. 67.5 LLaMA-33B
  52. 66.0 Qwen2.5-Coder-14B
  53. 64.9 PaLM 2-M
  54. 64.3 DeepSeek-Coder-V2 236B
  55. 64.1 DeepSeek LLM 67B
  56. 61.9 Falcon-40B
  57. 61.6 Gemma 3n E4B
  58. 61.0 Falcon-H1-34B
  59. 61.0 chatglm2-6b
  60. 60.9 Qwen2.5-Coder (7B)
  61. 60.3 Llama 2-13B
  62. 59.6 PaLM 2-S
  63. 57.3 DeepSeek-Coder-V2-Lite-Base
  64. 56.2 Gemma 3 4B
  65. 55.6 Yi-9B
  66. 55.5 Nemotron-4 15B
  67. 54.5 INTELLECT-1
  68. 54.5 Llama 2-34B
  69. 53.5 Falcon-Arabic-7B
  70. 53.2 InstructGPT 175B
  71. 53.2 Qwen-1_8B
  72. 52.9 Qwen2.5-Coder-3B
  73. 52.7 LLaMA-13B
  74. 52.5 PaLM 62B
  75. 50.6 MPT-30B
  76. 50.3 Yi 6B
  77. 49.9 BitNet b1.58 2B4T
  78. 47.9 Falcon-7B
  79. 47.6 LLaMA-7B
  80. 47.2 StarCoder 2 15B
  81. 45.9 Llama 2-7B
  82. 45.2 Qwen2.5-Coder (1.5B)
  83. 44.4 Phi-1.5
  84. 43.2 vicuna-13b-v1.1
  85. 42.6 MPT-7B
  86. 42.2 DeepSeek Coder 33B
  87. 42.1 Gemma 2B
  88. 41.2 XGen-7B
  89. 41.1 GPT-NeoX-20B
  90. 39.6 Dolly 2.0-12b
  91. 39.1 RedPajama-INCITE-7B-Base
  92. 38.7 open_llama_7b
  93. 38.7 StarCoder 2 7B
  94. 38.4 Gemma 3 1B
  95. 38.0 Baichuan2-13B
  96. 36.4 DeepSeek Coder 6.7B
  97. 36.3 GPT-J-6B
  98. 35.8 OPT-1.3B
  99. 35.7 CodeQwen1.5-7B
  100. 34.4 Qwen2.5-Coder-0.5B
  101. 34.2 StarCoder 2 3B
  102. 32.5 Baichuan 2-7B
  103. 32.4 Cerebras-GPT-13B
  104. 28.2 Gemma 3 270M
  105. 27.0 stablelm-tuned-alpha-7b
  106. 25.4 DeepSeek Coder 1.3B
  107. 25.0 GPT-2 (1.5B)
  1. 1 Nemotron 3 Ultra 550B-A55B Base 97.3%
  2. 2 Llama 3.1-405B 96.9%
  3. 3 Claude 3 Opus 96.4%
  4. 4 GPT-4 Mar 2023 96.3%
  5. 5 Nemotron 3.5 Super 120B-A12B Base 96%
  6. 6 DeepSeek-V3 95.3%
  7. 7 PaLM 2-L 95.1%
  8. 8 Sarvam-M 95%
  9. 9 Llama 3.1-70B 94.8%
  10. 10 Qwen2.5-72B 94.5%
  11. 11 Mistral Large 94%
  12. 12 Claude 3 Sonnet 93.2%
  13. 13 Nemotron 3.5 Lightning 30B-A3B Base 93.2%
  14. 14 Llama 3-70B 93%
  15. 15 DeepSeek-V2 (MoE-236B, May 2024) 92.3%
  16. 16 Nemotron 3 Nano 30B-A3B Base 91.9%
  17. 17 phi-3-medium 14B 91.6%
  18. 18 Mistral Small 3 91.3%
  19. 19 Claude 2 91%
  20. 20 Phi-3.5-MoE 91%
  21. 21 phi-3-small 7.4B 90.7%
  22. 22 Claude 3 Haiku 89.2%
  23. 23 GPT-3.5 Turbo Nov 2023 87.4%
  24. 24 Mixtral 8x7B 87.3%
  25. 25 Claude Instant 86.3%
  26. 26 Stable Beluga 2 86.1%
  27. 27 davinci-002 85.2%
  28. 28 PaLM (540B) 85.2%
  29. 29 Apertus 70B 85%
  30. 30 phi-3-mini 3.8B 84.9%
  31. 31 Phi-3.5-mini 84.6%
  32. 32 Qwen-14B 84.4%
  33. 33 Phi-4 Mini 83.7%
  34. 34 Llama 3.1-8B 83.4%
  35. 35 Llama 3-8B 82.8%
  36. 36 internlm-20b 81.7%
  37. 37 Mistral 7B v0.1 78.6%
  38. 38 Llama 2-70B 78.3%
  39. 39 Gemma 7B 78.3%
  40. 40 Phi-2 75.9%
  41. 41 Qwen-7B 75.3%
  42. 42 Gemma 2 27B 71.4%
  43. 43 Gemma 3 27B 70.6%
  44. 44 Qwen2.5-Coder-32B 70.5%
  45. 45 LLaMA-65B 69.5%
  46. 46 internlm-7b 69.5%
  47. 47 Qwen2-72B 68.9%
  48. 48 Gemma 3 12B 68.9%
  49. 49 Gemma 2 9B 68.4%
  50. 50 Falcon-180B 67.8%
  51. 51 LLaMA-33B 67.5%
  52. 52 Qwen2.5-Coder-14B 66%
  53. 53 PaLM 2-M 64.9%
  54. 54 DeepSeek-Coder-V2 236B 64.3%
  55. 55 DeepSeek LLM 67B 64.1%
  56. 56 Falcon-40B 61.9%
  57. 57 Gemma 3n E4B 61.6%
  58. 58 Falcon-H1-34B 61%
  59. 59 chatglm2-6b 61%
  60. 60 Qwen2.5-Coder (7B) 60.9%
  61. 61 Llama 2-13B 60.3%
  62. 62 PaLM 2-S 59.6%
  63. 63 DeepSeek-Coder-V2-Lite-Base 57.3%
  64. 64 Gemma 3 4B 56.2%
  65. 65 Yi-9B 55.6%
  66. 66 Nemotron-4 15B 55.5%
  67. 67 INTELLECT-1 54.5%
  68. 68 Llama 2-34B 54.5%
  69. 69 Falcon-Arabic-7B 53.5%
  70. 70 InstructGPT 175B 53.2%
  71. 71 Qwen-1_8B 53.2%
  72. 72 Qwen2.5-Coder-3B 52.9%
  73. 73 LLaMA-13B 52.7%
  74. 74 PaLM 62B 52.5%
  75. 75 MPT-30B 50.6%
  76. 76 Yi 6B 50.3%
  77. 77 BitNet b1.58 2B4T 49.9%
  78. 78 Falcon-7B 47.9%
  79. 79 LLaMA-7B 47.6%
  80. 80 StarCoder 2 15B 47.2%
  81. 81 Llama 2-7B 45.9%
  82. 82 Qwen2.5-Coder (1.5B) 45.2%
  83. 83 Phi-1.5 44.4%
  84. 84 vicuna-13b-v1.1 43.2%
  85. 85 MPT-7B 42.6%
  86. 86 DeepSeek Coder 33B 42.2%
  87. 87 Gemma 2B 42.1%
  88. 88 XGen-7B 41.2%
  89. 89 GPT-NeoX-20B 41.1%
  90. 90 Dolly 2.0-12b 39.6%
  91. 91 RedPajama-INCITE-7B-Base 39.1%
  92. 92 open_llama_7b 38.7%
  93. 93 StarCoder 2 7B 38.7%
  94. 94 Gemma 3 1B 38.4%
  95. 95 Baichuan2-13B 38%
  96. 96 DeepSeek Coder 6.7B 36.4%
  97. 97 GPT-J-6B 36.3%
  98. 98 OPT-1.3B 35.8%
  99. 99 CodeQwen1.5-7B 35.7%
  100. 100 Qwen2.5-Coder-0.5B 34.4%
  101. 101 StarCoder 2 3B 34.2%
  102. 102 Baichuan 2-7B 32.5%
  103. 103 Cerebras-GPT-13B 32.4%
  104. 104 Gemma 3 270M 28.2%
  105. 105 stablelm-tuned-alpha-7b 27%
  106. 106 DeepSeek Coder 1.3B 25.4%
  107. 107 GPT-2 (1.5B) 25%

Sources: Labs’ model cards and Another board, via Epoch AI (CC BY 4.0)

#ModelScoreSettingDateSource
1 Nemotron 3 Ultra 550B-A55B Base 97.3% Jun 2026 Lab model card
2 Llama 3.1-405B 96.9% Jul 2024 Lab model card
3 Claude 3 Opus 96.4% Mar 2024 Lab model card
4 GPT-4 Mar 2023 96.3% Mar 2023 Lab model card
5 Nemotron 3.5 Super 120B-A12B Base 96% Oct 2026 Lab model card
6 DeepSeek-V3 95.3% Dec 2024 Lab model card
7 PaLM 2-L 95.1% May 2023 Lab model card
8 Sarvam-M 95% May 2025 Lab model card
9 Llama 3.1-70B 94.8% Jul 2024 Lab model card
10 Mistral Large 94% Feb 2024 Lab model card
11 Claude 3 Sonnet 93.2% Mar 2024 Lab model card
12 Nemotron 3.5 Lightning 30B-A3B Base 93.2% Oct 2026 Lab model card
13 Llama 3-70B 93% Apr 2024 Lab model card
14 DeepSeek-V2 (MoE-236B, May 2024) 92.3% May 2024 Lab model card
15 Nemotron 3 Nano 30B-A3B Base 91.9% Dec 2025 Lab model card
16 phi-3-medium 14B 91.6% Aug 2024 Lab model card
17 Mistral Small 3 91.3% Jan 2025 Lab model card
18 Claude 2 91% Jul 2023 Lab model card
19 Phi-3.5-MoE 91% Aug 2024 Lab model card
20 phi-3-small 7.4B 90.7% Aug 2024 Lab model card
21 Claude 3 Haiku 89.2% Mar 2024 Lab model card
22 Mixtral 8x7B 85.8% Dec 2023 Lab model card
23 Apertus 70B 85% Sep 2025 Lab model card
24 phi-3-mini 3.8B 84.9% Aug 2024 Lab model card
25 Phi-3.5-mini 84.6% Aug 2024 Lab model card
26 Phi-4 Mini 83.7% Feb 2025 Lab model card
27 Llama 3.1-8B 83.4% Jul 2024 Lab model card
28 Llama 3-8B 78.6% Apr 2024 Lab model card
29 Qwen2.5-72B 72.4% Sep 2024 Lab model card
30 Gemma 2 27B 71.4% Jun 2024 Lab model card
31 Gemma 3 27B 70.6% Mar 2025 Lab model card
32 Qwen2-72B 68.9% Jun 2024 Lab model card
33 Gemma 3 12B 68.9% Mar 2025 Lab model card
34 Gemma 2 9B 68.4% Jul 2024 Lab model card
35 DeepSeek LLM 67B 64.1% Jan 2024 Lab model card
36 Gemma 3n E4B 61.6% Jun 2025 Lab model card
37 Falcon-H1-34B 61% May 2025 Lab model card
38 Llama 2-70B 57.4% Jul 2023 Lab model card
39 Gemma 3 4B 56.2% Mar 2025 Lab model card
40 Mistral 7B v0.1 55.5% Sep 2023 Lab model card
41 Llama 2-34B 54.5% Jul 2023 Lab model card
42 Falcon-Arabic-7B 53.5% Nov 2025 Lab model card
43 Gemma 7B 53.2% Feb 2024 Lab model card
44 BitNet b1.58 2B4T 49.9% Apr 2025 Lab model card
45 Llama 2-13B 49.4% Jul 2023 Lab model card
46 Llama 2-7B 45.9% Jul 2023 Lab model card
47 Gemma 2B 42.1% Mar 2024 Lab model card
48 Gemma 3 1B 38.4% Mar 2025 Lab model card
49 Gemma 3 270M 28.2% Aug 2025 Lab model card
#ModelScoreSettingDateSource
1 Nemotron 3 Ultra 550B-A55B Base 97.3% Jun 2026 Lab model card
2 Llama 3.1-405B 96.9% Jul 2024 Lab model card
3 Claude 3 Opus 96.4% Mar 2024 Lab model card
4 GPT-4 Mar 2023 96.3% Mar 2023 Lab model card
5 Nemotron 3.5 Super 120B-A12B Base 96% Oct 2026 Lab model card
6 DeepSeek-V3 95.3% Dec 2024 Lab model card
7 PaLM 2-L 95.1% May 2023 Lab model card
8 Sarvam-M 95% May 2025 Lab model card
9 Llama 3.1-70B 94.8% Jul 2024 Lab model card
10 Qwen2.5-72B 94.5% Sep 2024 Claim in DeepSeek-V3 Technical Report, via Epoch AI
11 Mistral Large 94% Feb 2024 Lab model card
12 Claude 3 Sonnet 93.2% Mar 2024 Lab model card
13 Nemotron 3.5 Lightning 30B-A3B Base 93.2% Oct 2026 Lab model card
14 Llama 3-70B 93% Apr 2024 Lab model card
15 DeepSeek-V2 (MoE-236B, May 2024) 92.3% May 2024 Lab model card
16 Nemotron 3 Nano 30B-A3B Base 91.9% Dec 2025 Lab model card
17 phi-3-medium 14B 91.6% Aug 2024 Lab model card
18 Mistral Small 3 91.3% Jan 2025 Lab model card
19 Claude 2 91% Jul 2023 Lab model card
20 Phi-3.5-MoE 91% Aug 2024 Lab model card
21 phi-3-small 7.4B 90.7% Aug 2024 Lab model card
22 Claude 3 Haiku 89.2% Mar 2024 Lab model card
23 GPT-3.5 Turbo Nov 2023 87.4% Nov 2023 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
24 Mixtral 8x7B 87.3% Dec 2023 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
25 Claude Instant 86.3% Aug 2023 Claim in Releasing Claude Instant 1.2, via Epoch AI
26 Stable Beluga 2 86.1% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
27 davinci-002 85.2% Mar 2022 Claim in GPT-4 Technical Report, via Epoch AI
28 PaLM (540B) 85.2% Apr 2022 Claim in GPT-4 Technical Report, via Epoch AI
29 Apertus 70B 85% Sep 2025 Lab model card
30 phi-3-mini 3.8B 84.9% Aug 2024 Lab model card
31 Phi-3.5-mini 84.6% Aug 2024 Lab model card
32 Qwen-14B 84.4% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
33 Phi-4 Mini 83.7% Feb 2025 Lab model card
34 Llama 3.1-8B 83.4% Jul 2024 Lab model card
35 Llama 3-8B 82.8% Apr 2024 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
36 internlm-20b 81.7% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
37 Mistral 7B v0.1 78.6% Sep 2023 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
38 Llama 2-70B 78.3% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
39 Gemma 7B 78.3% Feb 2024 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
40 Phi-2 75.9% Dec 2023 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
41 Qwen-7B 75.3% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
42 Gemma 2 27B 71.4% Jun 2024 Lab model card
43 Gemma 3 27B 70.6% Mar 2025 Lab model card
44 Qwen2.5-Coder-32B 70.5% Sep 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
45 LLaMA-65B 69.5% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
46 internlm-7b 69.5% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
47 Qwen2-72B 68.9% Jun 2024 Lab model card
48 Gemma 3 12B 68.9% Mar 2025 Lab model card
49 Gemma 2 9B 68.4% Jul 2024 Lab model card
50 Falcon-180B 67.8% Sep 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
51 LLaMA-33B 67.5% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
52 Qwen2.5-Coder-14B 66% Sep 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
53 PaLM 2-M 64.9% May 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
54 DeepSeek-Coder-V2 236B 64.3% Jun 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
55 DeepSeek LLM 67B 64.1% Jan 2024 Lab model card
56 Falcon-40B 61.9% Mar 2023 Claim in Falcon2-11B Technical Report, via Epoch AI
57 Gemma 3n E4B 61.6% Jun 2025 Lab model card
58 Falcon-H1-34B 61% May 2025 Lab model card
59 chatglm2-6b 61% Jun 2023 Claim in Qwen Technical Report, via Epoch AI
60 Qwen2.5-Coder (7B) 60.9% Sep 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
61 Llama 2-13B 60.3% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
62 PaLM 2-S 59.6% May 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
63 DeepSeek-Coder-V2-Lite-Base 57.3% Jun 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
64 Gemma 3 4B 56.2% Mar 2025 Lab model card
65 Yi-9B 55.6% Mar 2024 Claim in Yi: Open Foundation Models by 01.AI, via Epoch AI
66 Nemotron-4 15B 55.5% Feb 2024 Claim in Nemotron-4 15B Technical Report, via Epoch AI
67 INTELLECT-1 54.5% Nov 2024 Claim in INTELLECT-1 Technical Report, via Epoch AI
68 Llama 2-34B 54.5% Jul 2023 Lab model card
69 Falcon-Arabic-7B 53.5% Nov 2025 Lab model card
70 InstructGPT 175B 53.2% Jan 2022 Claim in GLaM: Efficient Scaling of Language Models with Mixture-o…, via Epoch AI
71 Qwen-1_8B 53.2% Nov 2023 Claim in Qwen Technical Report, via Epoch AI
72 Qwen2.5-Coder-3B 52.9% Sep 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
73 LLaMA-13B 52.7% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
74 PaLM 62B 52.5% Apr 2022 Claim in LLaMA: Open and Efficient Foundation Language Models, via Epoch AI
75 MPT-30B 50.6% Jun 2023 Claim in Llama 2: Open Foundation and Fine-Tuned Chat Models, via Epoch AI
76 Yi 6B 50.3% Nov 2023 Claim in Yi: Open Foundation Models by 01.AI, via Epoch AI
77 BitNet b1.58 2B4T 49.9% Apr 2025 Lab model card
78 Falcon-7B 47.9% Apr 2023 Claim in Falcon2-11B Technical Report, via Epoch AI
79 LLaMA-7B 47.6% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
80 StarCoder 2 15B 47.2% Feb 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
81 Llama 2-7B 45.9% Jul 2023 Lab model card
82 Qwen2.5-Coder (1.5B) 45.2% Sep 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
83 Phi-1.5 44.4% Sep 2023 Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI
84 vicuna-13b-v1.1 43.2% Apr 2023 Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI
85 MPT-7B 42.6% May 2023 Claim in Qwen Technical Report, via Epoch AI
86 DeepSeek Coder 33B 42.2% Nov 2023 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
87 Gemma 2B 42.1% Mar 2024 Lab model card
88 XGen-7B 41.2% Jun 2023 Claim in XGen-7B Technical Report, via Epoch AI
89 GPT-NeoX-20B 41.1% Apr 2022 Claim in XGen-7B Technical Report, via Epoch AI
90 Dolly 2.0-12b 39.6% Apr 2023 Claim in XGen-7B Technical Report, via Epoch AI
91 RedPajama-INCITE-7B-Base 39.1% May 2023 Claim in XGen-7B Technical Report, via Epoch AI
92 open_llama_7b 38.7% Jun 2023 Claim in XGen-7B Technical Report, via Epoch AI
93 StarCoder 2 7B 38.7% Feb 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
94 Gemma 3 1B 38.4% Mar 2025 Lab model card
95 Baichuan2-13B 38% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
96 DeepSeek Coder 6.7B 36.4% Nov 2023 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
97 GPT-J-6B 36.3% Aug 2021 Claim in XGen-7B Technical Report, via Epoch AI
98 OPT-1.3B 35.8% May 2022 Claim in XGen-7B Technical Report, via Epoch AI
99 CodeQwen1.5-7B 35.7% Apr 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
100 Qwen2.5-Coder-0.5B 34.4% Sep 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
101 StarCoder 2 3B 34.2% Feb 2024 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
102 Baichuan 2-7B 32.5% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
103 Cerebras-GPT-13B 32.4% Mar 2023 Claim in XGen-7B Technical Report, via Epoch AI
104 Gemma 3 270M 28.2% Aug 2025 Lab model card
105 stablelm-tuned-alpha-7b 27% Apr 2023 Claim in XGen-7B Technical Report, via Epoch AI
106 DeepSeek Coder 1.3B 25.4% Nov 2023 Claim in Qwen2.5-Coder Technical Report, via Epoch AI
107 GPT-2 (1.5B) 25% Nov 2019 Claim in Textbooks Are All You Need II: phi-1.5 technical report, via Epoch AI

The benchmark's own leaderboard has 7 entries, 0 of them models we track. Epoch AI collected claims for 75 models from labs’ reports and papers. Switch the scores above to rank them together.

4 models have held number one in labs’ own numbers since 2023 5 models have held number one with claims included since 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Anthropic Meta NVIDIA
Lab claims Passed 85% in Mar 2023, by GPT-4
Nemotron 3 Ultra 550B-A55B Base 97.3%

Source: Labs’ model cards

A claim for davinci-002 that Epoch AI collected passed 85% in March 2022

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card A claim Epoch AI collected from a report or paper Best so far
0% 20% 40% 60% 80% 100% 2020 2021 2022 2023 2024 2025 2026 Near the ceiling (85%) Random guessing Nemotron 3 Ultra 550B-A55B Base Llama 3.1-405B GPT-4 davinci-002 InstructGPT 175B GPT-J-6B GPT-2 (1.5B) 0% 20% 40% 60% 80% 100% 2020 2022 2024 2026 Near the ceiling (85%) Random guessing Nemotron 3 Ultra 550B-A55B Base davinci-002

Sources: Labs’ model cards and claims collected by Epoch AI (CC BY 4.0)

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

OpenAI
Google DeepMind
Anthropic
Mistral AI
Microsoft
Meta
Qwen
DeepSeek
Technology Innovation Institute
Sarvam AI
DateLaunch postScoreAs the post writes it
Aug 14, 2025 Gemma 3 270M 28.2 ARC-c 0-shot (IT)
Jun 26, 2025 Gemma 3n E4B 61.6 ARC-c 25-shot (PT)
May 23, 2025 Sarvam-M 95 ARC-C (0.95)
May 20, 2025 Falcon-H1-34B 61 ARC-C
Apr 15, 2025 BitNet b1.58 2B4T 49.9 ARC-Challenge
Mar 12, 2025 Gemma 3 27B 70.6 ARC-c 25-shot (PT)
Feb 26, 2025 Phi-4 Mini 83.7 ARC Challenge (10-shot)
Jan 30, 2025 Mistral Small 3 91.3 ARC-c (0-shot) [base]
Dec 26, 2024 DeepSeek-V3 95.3 ARC-Challenge (Acc.)
Sep 19, 2024 Qwen2.5-72B 72.4 ARC-C
Aug 20, 2024 Phi-3.5-MoE 91 ARC Challenge (10-shot)
Jul 23, 2024 Llama 3.1-405B 96.9 ARC Challenge (0-shot)
Jun 27, 2024 Gemma 2 27B 71.4 ARC-c (25-shot)
Jun 7, 2024 Qwen2-72B 68.9 ARC-C
Apr 22, 2024 phi-3-medium 14B 91 Arc-C (10-Shot)
Apr 18, 2024 Llama 3-70B 93 ARC-Challenge (25-shot) [pretrained]
Mar 4, 2024 Claude 3 Opus 96.4 ARC-Challenge (25-shot)
Feb 26, 2024 Mistral Large 94 Best of Arc C (25) [Arc C (5) = 94.2] / French Arc-C / German Arc-C / Spanish Arc-C / Italian Arc-C
Feb 21, 2024 Gemma 7B 53.2 ARC-c
Dec 12, 2023 Phi-2 ARC challenge (in Commonsense Reasoning average only)
Dec 11, 2023 Mixtral 8x7B 85.8 Best of ARC Challenge (25-shot) [detailed table Arc-c: 59.7] / French Arc-c / German Arc-c / Spanish Arc-c / Italian Arc-c
Sep 27, 2023 Mistral 7B v0.1 55.5 Arc-c
Jul 11, 2023 Claude 2 91 ARC-Challenge (5-shot)
May 10, 2023 PaLM 2-L 95.1 ARC-C (4), CoT + SC, Table 5 (Table 2 1-shot = 69.2)
Mar 14, 2023 GPT-4 96.3 AI2 Reasoning Challenge (ARC) (25-shot)

43 system cards report it, from 10 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 15, 2026 Nemotron 3.5 Super 120B-A12B Base technical report 96 ARC-Challenge, 25-shot acc_norm, Table 6 Capabilities
Oct 15, 2026 Nemotron 3.5 Lightning 30B-A3B Base technical report 93.2 ARC-Challenge, 25-shot acc_norm, Table 6 Capabilities
Jun 4, 2026 Nemotron 3 Ultra 550B-A55B Base model card 97.3 ARC-Challenge, 25-shot acc_norm Capabilities
Dec 23, 2025 Nemotron 3 Nano 30B-A3B Base technical report 91.9 ARC-Challenge, 25-shot acc_norm, base model Table 2 Capabilities
Nov 8, 2025 Falcon-Arabic-7B technical report 53.5 ARC Challenge 25-shot (Table 8) Capabilities
Sep 17, 2025 Apertus 70B technical report 85 ARC Challenge Chat, 0-shot CoT, held-out test suite (Table 21). ARC Challenge Multilingual 37.3 Capabilities
Jul 30, 2025 Falcon-H1-34B technical report 61 ARC-C, logprobs 0-shot (Table 24) Capabilities
Apr 16, 2025 BitNet b1.58 2B4T technical report 49.9 ARC-Challange (0-shot; Acc,norm) (Table 1, Table 3) Capabilities
Mar 25, 2025 Gemma 3 4B technical report 56.2 ARC-C, PT, 25-shot (Table 9) Capabilities
Mar 25, 2025 Gemma 3 27B technical report 70.6 ARC-C, PT, 25-shot (Table 9) Capabilities
Mar 25, 2025 Gemma 3 1B technical report 38.4 ARC-C, PT, 25-shot (Table 9) Capabilities
Mar 25, 2025 Gemma 3 12B technical report 68.9 ARC-C, PT, 25-shot (Table 9) Capabilities
Mar 3, 2025 Phi-4 Mini technical report 83.7 Arc-C (10-Shot) (Table 7) Capabilities
Nov 23, 2024 Llama 3.1-8B technical report 83.4 ARC Challenge (0-shot), post-trained (Table 2) Capabilities
Nov 23, 2024 Llama 3.1-70B technical report 94.8 ARC Challenge (0-shot), post-trained (Table 2) Capabilities
Nov 23, 2024 Llama 3.1-405B technical report 96.9 ARC Challenge (0-shot), post-trained (Table 2) Capabilities
Aug 30, 2024 phi-3-small 7.4B technical report 90.7 Arc-C (10-Shot) (Section 3 table) Capabilities
Aug 30, 2024 phi-3-mini 3.8B technical report 84.9 Arc-C (10-Shot) (Section 3 table) Capabilities
Aug 30, 2024 phi-3-medium 14B technical report 91.6 Arc-C (10-Shot) (Section 3 table) Capabilities
Aug 30, 2024 Phi-3.5-MoE technical report 91 ARC Challenge (10-shot) (Table 3) Capabilities
Aug 30, 2024 Phi-3.5-mini technical report 84.6 ARC Challenge (10-shot) (Table 3) Capabilities
Jul 31, 2024 Gemma 2 9B technical report 68.4 ARC-C, PT, 25-shot (Table 13) Capabilities
Jul 31, 2024 Gemma 2 27B technical report 71.4 ARC-C, PT, 25-shot (Table 13) Capabilities
Jul 23, 2024 Llama 3.1-8B model card 83.4 ARC-C, 0-shot, acc, instruction tuned (Instruction tuned models table) Capabilities
Jul 23, 2024 Llama 3.1-70B model card 94.8 ARC-C, 0-shot, acc, instruction tuned (Instruction tuned models table) Capabilities
Jul 23, 2024 Llama 3.1-405B model card 96.9 ARC-C, 0-shot, acc, instruction tuned (Instruction tuned models table) Capabilities
May 7, 2024 DeepSeek-V2 (MoE-236B, May 2024) technical report 92.3 ARC-Challenge 25-shot, DeepSeek-V2 Chat (RL) (Table 3) Capabilities
Apr 18, 2024 Llama 3-8B model card 78.6 ARC-Challenge (25-shot), pretrained (Base pretrained models table) Capabilities
Apr 18, 2024 Llama 3-70B model card 93 ARC-Challenge (25-shot), pretrained (Base pretrained models table) Capabilities
Mar 13, 2024 Gemma 7B technical report 53.2 ARC-c, PT (Table 6). HuggingFace H6 run (Table 7): 61.9 Capabilities
Mar 13, 2024 Gemma 2B technical report 42.1 ARC-c, PT (Table 6) Capabilities
Mar 4, 2024 Claude 3 Sonnet model card 93.2 ARC-Challenge, 25-shot (Table 1) Capabilities
Mar 4, 2024 Claude 3 Opus model card 96.4 ARC-Challenge, 25-shot (Table 1) Capabilities
Mar 4, 2024 Claude 3 Haiku model card 89.2 ARC-Challenge, 25-shot (Table 1) Capabilities
Jan 8, 2024 Mixtral 8x7B technical report 85.8 ARC Challenge, 25-shot (Table 3). 0-shot 59.7 (Table 2) Capabilities
Jan 5, 2024 DeepSeek LLM 67B technical report 64.1 ARC-Challenge, DeepSeek 67B Chat (Table 6) Capabilities
Oct 10, 2023 Mistral 7B v0.1 technical report 55.5 Arc-c, 0-shot (Table 2) Capabilities
Jul 19, 2023 Llama 2-7B technical report 45.9 ARC-c, 0-shot, pretrained (Table 20) Capabilities
Jul 19, 2023 Llama 2-70B technical report 57.4 ARC-c, 0-shot, pretrained (Table 20) Capabilities
Jul 19, 2023 Llama 2-34B technical report 54.5 ARC-c, 0-shot, pretrained (Table 20) Capabilities
Jul 19, 2023 Llama 2-13B technical report 49.4 ARC-c, 0-shot, pretrained (Table 20) Capabilities
Jul 11, 2023 Claude 2 model card 91 ARC-Challenge (5-shot) (4.3) Capabilities
Mar 14, 2023 GPT-4 technical report 96.3 AI2 Reasoning Challenge (ARC), Challenge-set, 25-shot (Table 2, base model) Capabilities

More in Knowledge

All 121 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • KMMLU-Pro In use Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. Solar Pro 4 79.2%
  • HLE-Verified In use Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Seed2.0 Pro 73.6%