Benchmark Almanac

MATH-500

LongCat-Flash-Thinking 99.2% leads since September 2025
Gemini 3 Pro 96.4% leads since November 2025
LongCat-Flash-Thinking 99.2% leads since September 2025
Fading

Models solve 500 competition math problems drawn from the MATH test set. Final answers are checked against the reference answers.

Near saturation, since DeepSeek-R1 already scored 97.3 percent in January 2025.

Made by
OpenAI
Released
May 2023
Kind
Competition problems with answers
Level
Competition
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Strongest model tested
Gemini 3 Pro #55 on the capability index
Last in a launch post
October 2026, dropped after it saturated
Last model added
October 2026
Scores refreshed
November 12th, 2026
70
Models scored in lab cards
50
Launch posts list it, from 29 labs
46
System cards report it, from 18 labs

LongCat-Flash-Thinking leads with 99.2%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Gemini 3 Pro leads with 96.4% across all independent runs

Best score per model across Vals AI’s leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.

LongCat-Flash-Thinking leads with 99.2%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

70 models. Scroll sideways for the rest.

  1. 99.2 LongCat-Flash-Thinking
  2. 98.6 Sarvam 105B
  3. 98.6 Cogito v2.1
  4. 98.2 GLM-4.5
  5. 98.1 GLM-4.5-Air
  6. 98.1 INTELLECT-3
  7. 98.0 Qwen3-235B-A22B
  8. 98.0 Qwen3-30B-A3B
  9. 98.0 InternVL3.5-241B-A28B
  10. 97.9 o3-mini
  11. 97.8 nvidia-nemotron-nano-9b-v2
  12. 97.6 LongCat-Flash-Omni
  13. 97.4 Qwen3-8B
  14. 97.4 Kimi K2
  15. 97.4 Llama-3.3-Nemotron-Super-49B-v1.5
  16. 97.4 Falcon H1R 7B
  17. 97.3 DeepSeek-R1
  18. 97.2 Qwen3-32B
  19. 97.0 Llama-3.1-Nemotron-Ultra-253B-v1
  20. 97.0 Qwen3-4B
  21. 97.0 Sarvam 30B
  22. 96.9 Nemotron Nano 12B v2 VL
  23. 96.8 Qwen3-14B
  24. 96.8 MiniMax-M1-80k
  25. 96.6 Llama-3.3-Nemotron-Super-49B-v1
  26. 96.6 Code World Model (CWM)
  27. 96.4 o1
  28. 96.4 ERNIE-4.5-300B-A47B
  29. 96.4 LongCat-Flash-Chat
  30. 96.3 Hermes 4 405B
  31. 96.2 Kimi k1.5
  32. 96.2 Claude 3.7 Sonnet
  33. 96.2 Hunyuan-T1
  34. 96.0 LFM2.5-24B-A2B
  35. 95.9 Magistral Small 1.0
  36. 95.8 MiMo-7B-RL
  37. 95.7 EXAONE Deep 32B
  38. 95.4 Solar Pro 2
  39. 95.4 Nemotron 3 Nano 4B
  40. 95.0 Llama 4 Behemoth
  41. 94.6 Phi-4-mini-reasoning
  42. 94.5 DeepSeek-R1-Distill-Llama-70B
  43. 94.3 DeepSeek-R1-Distill-Qwen-32B
  44. 94.3 Magistral Medium 1.0
  45. 93.9 DeepSeek-R1-Distill-Qwen-14B
  46. 93.8 Hermes 4.3 36B
  47. 93.4 Qwen3-1.7B
  48. 92.5 Phi-4-mini-flash-reasoning
  49. 92.1 Trinity Mini
  50. 91.0 Mistral Medium 3
  51. 90.6 Reka Flash 3
  52. 90.2 DeepSeek-V3
  53. 88.8 LFM2.5-8B-A1B
  54. 88.0 LFM2.5-1.2B-Thinking
  55. 85.5 o1-preview
  56. 83.9 DeepSeek-R1-Distill-Qwen-1.5B
  57. 83.8 Falcon-H1-34B
  58. 83.0 InternLM3-8B-Instruct
  59. 83.0 Mercury
  60. 82.0 Amazon Nova Premier
  61. 81.6 Nemotron 3.5 Super 120B-A12B Base
  62. 79.5 Nemotron 3.5 Lightning 30B-A3B Base
  63. 78.6 Nemotron 3 Nano 30B-A3B Base
  64. 74.2 LFM2-8B-A1B
  65. 71.8 Phi-4 Mini
  66. 69.0 Granite 3.3 8B Instruct
  67. 43.4 BitNet b1.58 2B4T
  68. 42.4 LFM2-1.2B
  69. 18.4 Qwen-14B
  70. 6.80 Qwen-7B
  1. 1 LongCat-Flash-Thinking 99.2%
  2. 2 Sarvam 105B 98.6%
  3. 3 Cogito v2.1 98.6%
  4. 4 GLM-4.5 98.2%
  5. 5 GLM-4.5-Air 98.1%
  6. 6 INTELLECT-3 98.1%
  7. 7 Qwen3-235B-A22B 98%
  8. 8 Qwen3-30B-A3B 98%
  9. 9 InternVL3.5-241B-A28B 98%
  10. 10 o3-mini 97.9%
  11. 11 nvidia-nemotron-nano-9b-v2 97.8%
  12. 12 LongCat-Flash-Omni 97.6%
  13. 13 Qwen3-8B 97.4%
  14. 14 Kimi K2 Jul 2025 97.4%
  15. 15 Llama-3.3-Nemotron-Super-49B-v1.5 97.4%
  16. 16 Falcon H1R 7B 97.4%
  17. 17 DeepSeek-R1 97.3%
  18. 18 Qwen3-32B 97.2%
  19. 19 Llama-3.1-Nemotron-Ultra-253B-v1 97%
  20. 20 Qwen3-4B 97%
  21. 21 Sarvam 30B 97%
  22. 22 Nemotron Nano 12B v2 VL 96.9%
  23. 23 Qwen3-14B 96.8%
  24. 24 MiniMax-M1-80k 96.8%
  25. 25 Llama-3.3-Nemotron-Super-49B-v1 96.6%
  26. 26 Code World Model (CWM) 96.6%
  27. 27 o1 96.4%
  28. 28 ERNIE-4.5-300B-A47B 96.4%
  29. 29 LongCat-Flash-Chat 96.4%
  30. 30 Hermes 4 405B 96.3%
  31. 31 Kimi k1.5 96.2%
  32. 32 Claude 3.7 Sonnet 96.2%
  33. 33 Hunyuan-T1 96.2%
  34. 34 LFM2.5-24B-A2B 96%
  35. 35 Magistral Small 1.0 95.9%
  36. 36 MiMo-7B-RL 95.8%
  37. 37 EXAONE Deep 32B 95.7%
  38. 38 Solar Pro 2 95.4%
  39. 39 Nemotron 3 Nano 4B 95.4%
  40. 40 Llama 4 Behemoth 95%
  41. 41 Phi-4-mini-reasoning 94.6%
  42. 42 DeepSeek-R1-Distill-Llama-70B 94.5%
  43. 43 DeepSeek-R1-Distill-Qwen-32B 94.3%
  44. 44 Magistral Medium 1.0 94.3%
  45. 45 DeepSeek-R1-Distill-Qwen-14B 93.9%
  46. 46 Hermes 4.3 36B 93.8%
  47. 47 Qwen3-1.7B 93.4%
  48. 48 Phi-4-mini-flash-reasoning 92.5%
  49. 49 Trinity Mini 92.1%
  50. 50 Mistral Medium 3 91%
  51. 51 Reka Flash 3 90.6%
  52. 52 DeepSeek-V3 90.2%
  53. 53 LFM2.5-8B-A1B 88.8%
  54. 54 LFM2.5-1.2B-Thinking 88%
  55. 55 o1-preview 85.5%
  56. 56 DeepSeek-R1-Distill-Qwen-1.5B 83.9%
  57. 57 Falcon-H1-34B 83.8%
  58. 58 InternLM3-8B-Instruct 83%
  59. 59 Mercury 83%
  60. 60 Amazon Nova Premier 82%
  61. 61 Nemotron 3.5 Super 120B-A12B Base 81.6%
  62. 62 Nemotron 3.5 Lightning 30B-A3B Base 79.5%
  63. 63 Nemotron 3 Nano 30B-A3B Base 78.6%
  64. 64 LFM2-8B-A1B 74.2%
  65. 65 Phi-4 Mini 71.8%
  66. 66 Granite 3.3 8B Instruct 69%
  67. 67 BitNet b1.58 2B4T 43.4%
  68. 68 LFM2-1.2B 42.4%
  69. 69 Qwen-14B 18.4%
  70. 70 Qwen-7B 6.80%

Source: Labs’ model cards

38 models. Scroll sideways for the rest.

  1. 96.4 Gemini 3 Pro
  2. 96.2 Grok 4
  3. 96.0 GPT-5
  4. 95.4 Claude Opus 4.1
  5. 95.2 Gemini 2.5 Pro
  6. 94.8 gpt-oss-120b
  7. 94.8 GPT-5 mini
  8. 94.6 o3
  9. 94.6 Qwen3-235B-A22B
  10. 94.2 Grok-3 mini
  11. 94.2 o4-mini
  12. 94.2 Kimi K2
  13. 94.2 gpt-oss-20b
  14. 94.0 GLM-4.5
  15. 93.8 Claude Sonnet 4
  16. 93.8 GPT-5 nano
  17. 92.2 DeepSeek-R1
  18. 91.8 o3-mini
  19. 91.8 Gemini 2.5 Flash
  20. 91.6 Claude 3.7 Sonnet
  21. 90.4 o1
  22. 90.4 Claude Opus 4
  23. 89.8 Grok 3
  24. 89.0 Gemini 2.0 Flash
  25. 89.0 MiniMax-M2.1
  26. 88.6 DeepSeek-V3
  27. 88.0 GPT-4.1 mini
  28. 87.2 GPT-4.1
  29. 85.2 Llama 4 Maverick
  30. 84.6 Gemini 2.0 Flash Thinking
  31. 80.2 GPT-4.1 nano
  32. 79.2 Llama 4 Scout
  33. 78.4 Grok-2
  34. 72.6 GPT-4o mini
  35. 72.4 Claude 3.5 Sonnet
  36. 64.2 Claude 3.5 Haiku
  37. 54.8 Jamba 1.6 Large
  38. 25.4 Jamba 1.6 Mini
  1. 1 Gemini 3 Pro Vals AI harness 96.4%
  2. 2 Grok 4 Vals AI harness 96.2%
  3. 3 GPT-5 Vals AI harness 96%
  4. 4 Claude Opus 4.1 Vals AI harness 95.4%
  5. 5 Gemini 2.5 Pro Jun 2025, Vals AI harness 95.2%
  6. 6 gpt-oss-120b Vals AI harness 94.8%
  7. 7 GPT-5 mini Vals AI harness 94.8%
  8. 8 o3 Vals AI harness 94.6%
  9. 9 Qwen3-235B-A22B Vals AI harness 94.6%
  10. 10 Grok-3 mini Vals AI harness 94.2%
  11. 11 o4-mini Vals AI harness 94.2%
  12. 12 Kimi K2 Jul 2025, Vals AI harness 94.2%
  13. 13 gpt-oss-20b Vals AI harness 94.2%
  14. 14 GLM-4.5 Vals AI harness 94%
  15. 15 Claude Sonnet 4 Vals AI harness 93.8%
  16. 16 GPT-5 nano Vals AI harness 93.8%
  17. 17 DeepSeek-R1 May 2025, Vals AI harness 92.2%
  18. 18 o3-mini Vals AI harness 91.8%
  19. 19 Gemini 2.5 Flash Apr 2025, Vals AI harness 91.8%
  20. 20 Claude 3.7 Sonnet Vals AI harness 91.6%
  21. 21 o1 Vals AI harness 90.4%
  22. 22 Claude Opus 4 Vals AI harness 90.4%
  23. 23 Grok 3 Vals AI harness 89.8%
  24. 24 Gemini 2.0 Flash Feb 2025, Vals AI harness 89%
  25. 25 MiniMax-M2.1 Vals AI harness 89%
  26. 26 DeepSeek-V3 Mar 2025, Vals AI harness 88.6%
  27. 27 GPT-4.1 mini Vals AI harness 88%
  28. 28 GPT-4.1 Vals AI harness 87.2%
  29. 29 Llama 4 Maverick Vals AI harness 85.2%
  30. 30 Gemini 2.0 Flash Thinking Jan 2025, Vals AI harness 84.6%
  31. 31 GPT-4.1 nano Vals AI harness 80.2%
  32. 32 Llama 4 Scout Vals AI harness 79.2%
  33. 33 Grok-2 Dec 2024, Vals AI harness 78.4%
  34. 34 GPT-4o mini Vals AI harness 72.6%
  35. 35 Claude 3.5 Sonnet Vals AI harness 72.4%
  36. 36 Claude 3.5 Haiku Vals AI harness 64.2%
  37. 37 Jamba 1.6 Large Vals AI harness 54.8%
  38. 38 Jamba 1.6 Mini Vals AI harness 25.4%

Source: Vals AI’s leaderboard

102 models. Scroll sideways for the rest.

  1. 99.2 LongCat-Flash-Thinking
  2. 98.6 Sarvam 105B
  3. 98.6 Cogito v2.1
  4. 98.2 GLM-4.5
  5. 98.1 GLM-4.5-Air
  6. 98.1 INTELLECT-3
  7. 98.0 Qwen3-235B-A22B
  8. 98.0 Qwen3-30B-A3B
  9. 98.0 InternVL3.5-241B-A28B
  10. 97.9 o3-mini
  11. 97.8 nvidia-nemotron-nano-9b-v2
  12. 97.6 LongCat-Flash-Omni
  13. 97.4 Qwen3-8B
  14. 97.4 Kimi K2
  15. 97.4 Llama-3.3-Nemotron-Super-49B-v1.5
  16. 97.4 Falcon H1R 7B
  17. 97.3 DeepSeek-R1
  18. 97.2 Qwen3-32B
  19. 97.0 Llama-3.1-Nemotron-Ultra-253B-v1
  20. 97.0 Qwen3-4B
  21. 97.0 Sarvam 30B
  22. 96.9 Nemotron Nano 12B v2 VL
  23. 96.8 Qwen3-14B
  24. 96.8 MiniMax-M1-80k
  25. 96.6 Llama-3.3-Nemotron-Super-49B-v1
  26. 96.6 Code World Model (CWM)
  27. 96.4 o1
  28. 96.4 ERNIE-4.5-300B-A47B
  29. 96.4 LongCat-Flash-Chat
  30. 96.4 Gemini 3 Pro
  31. 96.3 Hermes 4 405B
  32. 96.2 Kimi k1.5
  33. 96.2 Claude 3.7 Sonnet
  34. 96.2 Hunyuan-T1
  35. 96.2 Grok 4
  36. 96.0 LFM2.5-24B-A2B
  37. 96.0 GPT-5
  38. 95.9 Magistral Small 1.0
  39. 95.8 MiMo-7B-RL
  40. 95.7 EXAONE Deep 32B
  41. 95.4 Solar Pro 2
  42. 95.4 Claude Opus 4.1
  43. 95.4 Nemotron 3 Nano 4B
  44. 95.2 Gemini 2.5 Pro
  45. 95.0 Llama 4 Behemoth
  46. 94.8 gpt-oss-120b
  47. 94.8 GPT-5 mini
  48. 94.6 o3
  49. 94.6 Phi-4-mini-reasoning
  50. 94.5 DeepSeek-R1-Distill-Llama-70B
  51. 94.3 DeepSeek-R1-Distill-Qwen-32B
  52. 94.3 Magistral Medium 1.0
  53. 94.2 Grok-3 mini
  54. 94.2 o4-mini
  55. 94.2 gpt-oss-20b
  56. 93.9 DeepSeek-R1-Distill-Qwen-14B
  57. 93.8 Claude Sonnet 4
  58. 93.8 GPT-5 nano
  59. 93.8 Hermes 4.3 36B
  60. 93.4 Qwen3-1.7B
  61. 92.5 Phi-4-mini-flash-reasoning
  62. 92.2 DeepSeek-R1
  63. 92.1 Trinity Mini
  64. 91.8 Gemini 2.5 Flash
  65. 91.0 Mistral Medium 3
  66. 90.6 Reka Flash 3
  67. 90.4 Claude Opus 4
  68. 90.2 DeepSeek-V3
  69. 89.8 Grok 3
  70. 89.0 Gemini 2.0 Flash
  71. 89.0 MiniMax-M2.1
  72. 88.8 LFM2.5-8B-A1B
  73. 88.6 DeepSeek-V3
  74. 88.0 GPT-4.1 mini
  75. 88.0 LFM2.5-1.2B-Thinking
  76. 87.2 GPT-4.1
  77. 85.5 o1-preview
  78. 85.2 Llama 4 Maverick
  79. 84.6 Gemini 2.0 Flash Thinking
  80. 83.9 DeepSeek-R1-Distill-Qwen-1.5B
  81. 83.8 Falcon-H1-34B
  82. 83.0 InternLM3-8B-Instruct
  83. 83.0 Mercury
  84. 82.0 Amazon Nova Premier
  85. 81.6 Nemotron 3.5 Super 120B-A12B Base
  86. 80.2 GPT-4.1 nano
  87. 79.5 Nemotron 3.5 Lightning 30B-A3B Base
  88. 79.2 Llama 4 Scout
  89. 78.6 Nemotron 3 Nano 30B-A3B Base
  90. 78.4 Grok-2
  91. 74.2 LFM2-8B-A1B
  92. 72.6 GPT-4o mini
  93. 72.4 Claude 3.5 Sonnet
  94. 71.8 Phi-4 Mini
  95. 69.0 Granite 3.3 8B Instruct
  96. 64.2 Claude 3.5 Haiku
  97. 54.8 Jamba 1.6 Large
  98. 43.4 BitNet b1.58 2B4T
  99. 42.4 LFM2-1.2B
  100. 25.4 Jamba 1.6 Mini
  101. 18.4 Qwen-14B
  102. 6.80 Qwen-7B
  1. 1 LongCat-Flash-Thinking 99.2%
  2. 2 Sarvam 105B 98.6%
  3. 3 Cogito v2.1 98.6%
  4. 4 GLM-4.5 98.2%
  5. 5 GLM-4.5-Air 98.1%
  6. 6 INTELLECT-3 98.1%
  7. 7 Qwen3-235B-A22B 98%
  8. 8 Qwen3-30B-A3B 98%
  9. 9 InternVL3.5-241B-A28B 98%
  10. 10 o3-mini 97.9%
  11. 11 nvidia-nemotron-nano-9b-v2 97.8%
  12. 12 LongCat-Flash-Omni 97.6%
  13. 13 Qwen3-8B 97.4%
  14. 14 Kimi K2 Jul 2025 97.4%
  15. 15 Llama-3.3-Nemotron-Super-49B-v1.5 97.4%
  16. 16 Falcon H1R 7B 97.4%
  17. 17 DeepSeek-R1 97.3%
  18. 18 Qwen3-32B 97.2%
  19. 19 Llama-3.1-Nemotron-Ultra-253B-v1 97%
  20. 20 Qwen3-4B 97%
  21. 21 Sarvam 30B 97%
  22. 22 Nemotron Nano 12B v2 VL 96.9%
  23. 23 Qwen3-14B 96.8%
  24. 24 MiniMax-M1-80k 96.8%
  25. 25 Llama-3.3-Nemotron-Super-49B-v1 96.6%
  26. 26 Code World Model (CWM) 96.6%
  27. 27 o1 96.4%
  28. 28 ERNIE-4.5-300B-A47B 96.4%
  29. 29 LongCat-Flash-Chat 96.4%
  30. 30 Gemini 3 Pro Vals AI harness 96.4%
  31. 31 Hermes 4 405B 96.3%
  32. 32 Kimi k1.5 96.2%
  33. 33 Claude 3.7 Sonnet 96.2%
  34. 34 Hunyuan-T1 96.2%
  35. 35 Grok 4 Vals AI harness 96.2%
  36. 36 LFM2.5-24B-A2B 96%
  37. 37 GPT-5 Vals AI harness 96%
  38. 38 Magistral Small 1.0 95.9%
  39. 39 MiMo-7B-RL 95.8%
  40. 40 EXAONE Deep 32B 95.7%
  41. 41 Solar Pro 2 95.4%
  42. 42 Claude Opus 4.1 Vals AI harness 95.4%
  43. 43 Nemotron 3 Nano 4B 95.4%
  44. 44 Gemini 2.5 Pro Jun 2025, Vals AI harness 95.2%
  45. 45 Llama 4 Behemoth 95%
  46. 46 gpt-oss-120b Vals AI harness 94.8%
  47. 47 GPT-5 mini Vals AI harness 94.8%
  48. 48 o3 Vals AI harness 94.6%
  49. 49 Phi-4-mini-reasoning 94.6%
  50. 50 DeepSeek-R1-Distill-Llama-70B 94.5%
  51. 51 DeepSeek-R1-Distill-Qwen-32B 94.3%
  52. 52 Magistral Medium 1.0 94.3%
  53. 53 Grok-3 mini Vals AI harness 94.2%
  54. 54 o4-mini Vals AI harness 94.2%
  55. 55 gpt-oss-20b Vals AI harness 94.2%
  56. 56 DeepSeek-R1-Distill-Qwen-14B 93.9%
  57. 57 Claude Sonnet 4 Vals AI harness 93.8%
  58. 58 GPT-5 nano Vals AI harness 93.8%
  59. 59 Hermes 4.3 36B 93.8%
  60. 60 Qwen3-1.7B 93.4%
  61. 61 Phi-4-mini-flash-reasoning 92.5%
  62. 62 DeepSeek-R1 May 2025, Vals AI harness 92.2%
  63. 63 Trinity Mini 92.1%
  64. 64 Gemini 2.5 Flash Apr 2025, Vals AI harness 91.8%
  65. 65 Mistral Medium 3 91%
  66. 66 Reka Flash 3 90.6%
  67. 67 Claude Opus 4 Vals AI harness 90.4%
  68. 68 DeepSeek-V3 90.2%
  69. 69 Grok 3 Vals AI harness 89.8%
  70. 70 Gemini 2.0 Flash Feb 2025, Vals AI harness 89%
  71. 71 MiniMax-M2.1 Vals AI harness 89%
  72. 72 LFM2.5-8B-A1B 88.8%
  73. 73 DeepSeek-V3 Mar 2025, Vals AI harness 88.6%
  74. 74 GPT-4.1 mini Vals AI harness 88%
  75. 75 LFM2.5-1.2B-Thinking 88%
  76. 76 GPT-4.1 Vals AI harness 87.2%
  77. 77 o1-preview 85.5%
  78. 78 Llama 4 Maverick Vals AI harness 85.2%
  79. 79 Gemini 2.0 Flash Thinking Jan 2025, Vals AI harness 84.6%
  80. 80 DeepSeek-R1-Distill-Qwen-1.5B 83.9%
  81. 81 Falcon-H1-34B 83.8%
  82. 82 InternLM3-8B-Instruct 83%
  83. 83 Mercury 83%
  84. 84 Amazon Nova Premier 82%
  85. 85 Nemotron 3.5 Super 120B-A12B Base 81.6%
  86. 86 GPT-4.1 nano Vals AI harness 80.2%
  87. 87 Nemotron 3.5 Lightning 30B-A3B Base 79.5%
  88. 88 Llama 4 Scout Vals AI harness 79.2%
  89. 89 Nemotron 3 Nano 30B-A3B Base 78.6%
  90. 90 Grok-2 Dec 2024, Vals AI harness 78.4%
  91. 91 LFM2-8B-A1B 74.2%
  92. 92 GPT-4o mini Vals AI harness 72.6%
  93. 93 Claude 3.5 Sonnet Vals AI harness 72.4%
  94. 94 Phi-4 Mini 71.8%
  95. 95 Granite 3.3 8B Instruct 69%
  96. 96 Claude 3.5 Haiku Vals AI harness 64.2%
  97. 97 Jamba 1.6 Large Vals AI harness 54.8%
  98. 98 BitNet b1.58 2B4T 43.4%
  99. 99 LFM2-1.2B 42.4%
  100. 100 Jamba 1.6 Mini Vals AI harness 25.4%
  101. 101 Qwen-14B 18.4%
  102. 102 Qwen-7B 6.80%

Sources: Labs’ model cards and Vals AI’s leaderboard

#ModelScoreSettingDateSource
1 LongCat-Flash-Thinking 99.2% Sep 2025 Lab model card
2 Sarvam 105B 98.6% Mar 2026 Lab model card
3 Cogito v2.1 98.6% Nov 2025 Lab model card
4 GLM-4.5 98.2% Jul 2025 Lab model card
5 GLM-4.5-Air 98.1% Aug 2025 Lab model card
6 INTELLECT-3 98.1% Nov 2025 Lab model card
7 Qwen3-235B-A22B 98% Apr 2025 Lab model card
8 Qwen3-30B-A3B 98% May 2025 Lab model card
9 InternVL3.5-241B-A28B 98% Aug 2025 Lab model card
10 o3-mini 97.9% Jan 2025 Lab model card
11 nvidia-nemotron-nano-9b-v2 97.8% Aug 2025 Lab model card
12 LongCat-Flash-Omni 97.6% Oct 2025 Lab model card
13 Qwen3-8B 97.4% May 2025 Lab model card
14 Kimi K2 Jul 2025 97.4% Jul 2025 Lab model card
15 Llama-3.3-Nemotron-Super-49B-v1.5 97.4% Jul 2025 Lab model card
16 Falcon H1R 7B 97.4% Jan 2026 Lab model card
17 DeepSeek-R1 97.3% Jan 2025 Lab model card
18 Qwen3-32B 97.2% May 2025 Lab model card
19 Llama-3.1-Nemotron-Ultra-253B-v1 97% Apr 2025 Lab model card
20 Qwen3-4B 97% May 2025 Lab model card
21 Sarvam 30B 97% Mar 2026 Lab model card
22 Nemotron Nano 12B v2 VL 96.9% Nov 2025 Lab model card
23 Qwen3-14B 96.8% May 2025 Lab model card
24 MiniMax-M1-80k 96.8% Jun 2025 Lab model card
25 Llama-3.3-Nemotron-Super-49B-v1 96.6% Mar 2025 Lab model card
26 Code World Model (CWM) 96.6% Sep 2025 Lab model card
27 o1 96.4% Dec 2024 Lab model card
28 ERNIE-4.5-300B-A47B 96.4% Jun 2025 Lab model card
29 LongCat-Flash-Chat 96.4% Aug 2025 Lab model card
30 Hermes 4 405B 96.3% Aug 2025 Lab model card
31 Kimi k1.5 96.2% Jan 2025 Lab model card
32 Claude 3.7 Sonnet 96.2% Feb 2025 Lab model card
33 Hunyuan-T1 96.2% Mar 2025 Lab model card
34 LFM2.5-24B-A2B 96% Oct 2026 Lab model card
35 Magistral Small 1.0 95.9% Jun 2025 Lab model card
36 MiMo-7B-RL 95.8% Apr 2025 Lab model card
37 EXAONE Deep 32B 95.7% Mar 2025 Lab model card
38 Solar Pro 2 95.4% Jul 2025 Lab model card
39 Nemotron 3 Nano 4B 95.4% Mar 2026 Lab model card
40 Llama 4 Behemoth 95% Apr 2025 Lab model card
41 Phi-4-mini-reasoning 94.6% Apr 2025 Lab model card
42 DeepSeek-R1-Distill-Llama-70B 94.5% Jan 2025 Lab model card
43 DeepSeek-R1-Distill-Qwen-32B 94.3% Jan 2025 Lab model card
44 Magistral Medium 1.0 94.3% Jun 2025 Lab model card
45 DeepSeek-R1-Distill-Qwen-14B 93.9% Jan 2025 Lab model card
46 Hermes 4.3 36B 93.8% Dec 2025 Lab model card
47 Qwen3-1.7B 93.4% May 2025 Lab model card
48 Phi-4-mini-flash-reasoning 92.5% Jul 2025 Lab model card
49 Trinity Mini 92.1% Dec 2025 Lab model card
50 Mistral Medium 3 91% May 2025 Lab model card
51 Reka Flash 3 90.6% Mar 2025 Lab model card
52 DeepSeek-V3 90.2% Dec 2024 Lab model card
53 LFM2.5-8B-A1B 88.8% May 2026 Lab model card
54 LFM2.5-1.2B-Thinking 88% Jan 2026 Lab model card
55 o1-preview 85.5% Sep 2024 Lab model card
56 DeepSeek-R1-Distill-Qwen-1.5B 83.9% Jan 2025 Lab model card
57 Falcon-H1-34B 83.8% May 2025 Lab model card
58 InternLM3-8B-Instruct 83% Jan 2025 Lab model card
59 Mercury 83% Jun 2025 Lab model card
60 Amazon Nova Premier 82% Apr 2025 Lab model card
61 Nemotron 3.5 Super 120B-A12B Base 81.6% Oct 2026 Lab model card
62 Nemotron 3.5 Lightning 30B-A3B Base 79.5% Oct 2026 Lab model card
63 Nemotron 3 Nano 30B-A3B Base 78.6% Dec 2025 Lab model card
64 LFM2-8B-A1B 74.2% Oct 2025 Lab model card
65 Phi-4 Mini 71.8% Mar 2025 Lab model card
66 Granite 3.3 8B Instruct 69% Apr 2025 Lab model card
67 BitNet b1.58 2B4T 43.4% Apr 2025 Lab model card
68 LFM2-1.2B 42.4% Nov 2025 Lab model card
69 Qwen-14B 18.4% Sep 2023 Lab model card
70 Qwen-7B 6.80% Sep 2023 Lab model card
#ModelScoreSettingDateSource
1 Gemini 3 Pro 96.4% Vals AI harness Nov 2025 Official leaderboard
2 Grok 4 96.2% Vals AI harness Jul 2025 Official leaderboard
3 GPT-5 96% Vals AI harness Aug 2025 Official leaderboard
4 Claude Opus 4.1 95.4% Vals AI harness Aug 2025 Official leaderboard
5 Gemini 2.5 Pro Jun 2025 95.2% Vals AI harness Jun 2025 Official leaderboard
6 gpt-oss-120b 94.8% Vals AI harness Aug 2025 Official leaderboard
7 GPT-5 mini 94.8% Vals AI harness Aug 2025 Official leaderboard
8 o3 94.6% Vals AI harness Apr 2025 Official leaderboard
9 Qwen3-235B-A22B 94.6% Vals AI harness Apr 2025 Official leaderboard
10 Grok-3 mini 94.2% Vals AI harness Apr 2025 Official leaderboard
11 o4-mini 94.2% Vals AI harness Apr 2025 Official leaderboard
12 Kimi K2 Jul 2025 94.2% Vals AI harness Jul 2025 Official leaderboard
13 gpt-oss-20b 94.2% Vals AI harness Aug 2025 Official leaderboard
14 GLM-4.5 94% Vals AI harness Aug 2025 Official leaderboard
15 Claude Sonnet 4 93.8% Vals AI harness May 2025 Official leaderboard
16 GPT-5 nano 93.8% Vals AI harness Aug 2025 Official leaderboard
17 DeepSeek-R1 May 2025 92.2% Vals AI harness May 2025 Official leaderboard
18 o3-mini 91.8% Vals AI harness Jan 2025 Official leaderboard
19 Gemini 2.5 Flash Apr 2025 91.8% Vals AI harness Apr 2025 Official leaderboard
20 Claude 3.7 Sonnet 91.6% Vals AI harness Feb 2025 Official leaderboard
21 o1 90.4% Vals AI harness Dec 2024 Official leaderboard
22 Claude Opus 4 90.4% Vals AI harness May 2025 Official leaderboard
23 Grok 3 89.8% Vals AI harness Apr 2025 Official leaderboard
24 Gemini 2.0 Flash Feb 2025 89% Vals AI harness Feb 2025 Official leaderboard
25 MiniMax-M2.1 89% Vals AI harness Dec 2025 Official leaderboard
26 DeepSeek-V3 Mar 2025 88.6% Vals AI harness Mar 2025 Official leaderboard
27 GPT-4.1 mini 88% Vals AI harness Apr 2025 Official leaderboard
28 GPT-4.1 87.2% Vals AI harness Apr 2025 Official leaderboard
29 Llama 4 Maverick 85.2% Vals AI harness Apr 2025 Official leaderboard
30 Gemini 2.0 Flash Thinking Jan 2025 84.6% Vals AI harness Jan 2025 Official leaderboard
31 GPT-4.1 nano 80.2% Vals AI harness Apr 2025 Official leaderboard
32 Llama 4 Scout 79.2% Vals AI harness Apr 2025 Official leaderboard
33 Grok-2 Dec 2024 78.4% Vals AI harness Dec 2024 Official leaderboard
34 GPT-4o mini 72.6% Vals AI harness Jul 2024 Official leaderboard
35 Claude 3.5 Sonnet 72.4% Vals AI harness Jun 2024 Official leaderboard
36 Claude 3.5 Haiku 64.2% Vals AI harness Oct 2024 Official leaderboard
37 Jamba 1.6 Large 54.8% Vals AI harness Mar 2025 Official leaderboard
38 Jamba 1.6 Mini 25.4% Vals AI harness Mar 2025 Official leaderboard
#ModelScoreSettingDateSource
1 LongCat-Flash-Thinking 99.2% Sep 2025 Lab model card
2 Sarvam 105B 98.6% Mar 2026 Lab model card
3 Cogito v2.1 98.6% Nov 2025 Lab model card
4 GLM-4.5 98.2% Jul 2025 Lab model card
5 GLM-4.5-Air 98.1% Aug 2025 Lab model card
6 INTELLECT-3 98.1% Nov 2025 Lab model card
7 Qwen3-235B-A22B 98% Apr 2025 Lab model card
8 Qwen3-30B-A3B 98% May 2025 Lab model card
9 InternVL3.5-241B-A28B 98% Aug 2025 Lab model card
10 o3-mini 97.9% Jan 2025 Lab model card
11 nvidia-nemotron-nano-9b-v2 97.8% Aug 2025 Lab model card
12 LongCat-Flash-Omni 97.6% Oct 2025 Lab model card
13 Qwen3-8B 97.4% May 2025 Lab model card
14 Kimi K2 Jul 2025 97.4% Jul 2025 Lab model card
15 Llama-3.3-Nemotron-Super-49B-v1.5 97.4% Jul 2025 Lab model card
16 Falcon H1R 7B 97.4% Jan 2026 Lab model card
17 DeepSeek-R1 97.3% Jan 2025 Lab model card
18 Qwen3-32B 97.2% May 2025 Lab model card
19 Llama-3.1-Nemotron-Ultra-253B-v1 97% Apr 2025 Lab model card
20 Qwen3-4B 97% May 2025 Lab model card
21 Sarvam 30B 97% Mar 2026 Lab model card
22 Nemotron Nano 12B v2 VL 96.9% Nov 2025 Lab model card
23 Qwen3-14B 96.8% May 2025 Lab model card
24 MiniMax-M1-80k 96.8% Jun 2025 Lab model card
25 Llama-3.3-Nemotron-Super-49B-v1 96.6% Mar 2025 Lab model card
26 Code World Model (CWM) 96.6% Sep 2025 Lab model card
27 o1 96.4% Dec 2024 Lab model card
28 ERNIE-4.5-300B-A47B 96.4% Jun 2025 Lab model card
29 LongCat-Flash-Chat 96.4% Aug 2025 Lab model card
30 Gemini 3 Pro 96.4% Vals AI harness Nov 2025 Official leaderboard
31 Hermes 4 405B 96.3% Aug 2025 Lab model card
32 Kimi k1.5 96.2% Jan 2025 Lab model card
33 Claude 3.7 Sonnet 96.2% Feb 2025 Lab model card
34 Hunyuan-T1 96.2% Mar 2025 Lab model card
35 Grok 4 96.2% Vals AI harness Jul 2025 Official leaderboard
36 LFM2.5-24B-A2B 96% Oct 2026 Lab model card
37 GPT-5 96% Vals AI harness Aug 2025 Official leaderboard
38 Magistral Small 1.0 95.9% Jun 2025 Lab model card
39 MiMo-7B-RL 95.8% Apr 2025 Lab model card
40 EXAONE Deep 32B 95.7% Mar 2025 Lab model card
41 Solar Pro 2 95.4% Jul 2025 Lab model card
42 Claude Opus 4.1 95.4% Vals AI harness Aug 2025 Official leaderboard
43 Nemotron 3 Nano 4B 95.4% Mar 2026 Lab model card
44 Gemini 2.5 Pro Jun 2025 95.2% Vals AI harness Jun 2025 Official leaderboard
45 Llama 4 Behemoth 95% Apr 2025 Lab model card
46 gpt-oss-120b 94.8% Vals AI harness Aug 2025 Official leaderboard
47 GPT-5 mini 94.8% Vals AI harness Aug 2025 Official leaderboard
48 o3 94.6% Vals AI harness Apr 2025 Official leaderboard
49 Phi-4-mini-reasoning 94.6% Apr 2025 Lab model card
50 DeepSeek-R1-Distill-Llama-70B 94.5% Jan 2025 Lab model card
51 DeepSeek-R1-Distill-Qwen-32B 94.3% Jan 2025 Lab model card
52 Magistral Medium 1.0 94.3% Jun 2025 Lab model card
53 Grok-3 mini 94.2% Vals AI harness Apr 2025 Official leaderboard
54 o4-mini 94.2% Vals AI harness Apr 2025 Official leaderboard
55 gpt-oss-20b 94.2% Vals AI harness Aug 2025 Official leaderboard
56 DeepSeek-R1-Distill-Qwen-14B 93.9% Jan 2025 Lab model card
57 Claude Sonnet 4 93.8% Vals AI harness May 2025 Official leaderboard
58 GPT-5 nano 93.8% Vals AI harness Aug 2025 Official leaderboard
59 Hermes 4.3 36B 93.8% Dec 2025 Lab model card
60 Qwen3-1.7B 93.4% May 2025 Lab model card
61 Phi-4-mini-flash-reasoning 92.5% Jul 2025 Lab model card
62 DeepSeek-R1 May 2025 92.2% Vals AI harness May 2025 Official leaderboard
63 Trinity Mini 92.1% Dec 2025 Lab model card
64 Gemini 2.5 Flash Apr 2025 91.8% Vals AI harness Apr 2025 Official leaderboard
65 Mistral Medium 3 91% May 2025 Lab model card
66 Reka Flash 3 90.6% Mar 2025 Lab model card
67 Claude Opus 4 90.4% Vals AI harness May 2025 Official leaderboard
68 DeepSeek-V3 90.2% Dec 2024 Lab model card
69 Grok 3 89.8% Vals AI harness Apr 2025 Official leaderboard
70 Gemini 2.0 Flash Feb 2025 89% Vals AI harness Feb 2025 Official leaderboard
71 MiniMax-M2.1 89% Vals AI harness Dec 2025 Official leaderboard
72 LFM2.5-8B-A1B 88.8% May 2026 Lab model card
73 DeepSeek-V3 Mar 2025 88.6% Vals AI harness Mar 2025 Official leaderboard
74 GPT-4.1 mini 88% Vals AI harness Apr 2025 Official leaderboard
75 LFM2.5-1.2B-Thinking 88% Jan 2026 Lab model card
76 GPT-4.1 87.2% Vals AI harness Apr 2025 Official leaderboard
77 o1-preview 85.5% Sep 2024 Lab model card
78 Llama 4 Maverick 85.2% Vals AI harness Apr 2025 Official leaderboard
79 Gemini 2.0 Flash Thinking Jan 2025 84.6% Vals AI harness Jan 2025 Official leaderboard
80 DeepSeek-R1-Distill-Qwen-1.5B 83.9% Jan 2025 Lab model card
81 Falcon-H1-34B 83.8% May 2025 Lab model card
82 InternLM3-8B-Instruct 83% Jan 2025 Lab model card
83 Mercury 83% Jun 2025 Lab model card
84 Amazon Nova Premier 82% Apr 2025 Lab model card
85 Nemotron 3.5 Super 120B-A12B Base 81.6% Oct 2026 Lab model card
86 GPT-4.1 nano 80.2% Vals AI harness Apr 2025 Official leaderboard
87 Nemotron 3.5 Lightning 30B-A3B Base 79.5% Oct 2026 Lab model card
88 Llama 4 Scout 79.2% Vals AI harness Apr 2025 Official leaderboard
89 Nemotron 3 Nano 30B-A3B Base 78.6% Dec 2025 Lab model card
90 Grok-2 Dec 2024 78.4% Vals AI harness Dec 2024 Official leaderboard
91 LFM2-8B-A1B 74.2% Oct 2025 Lab model card
92 GPT-4o mini 72.6% Vals AI harness Jul 2024 Official leaderboard
93 Claude 3.5 Sonnet 72.4% Vals AI harness Jun 2024 Official leaderboard
94 Phi-4 Mini 71.8% Mar 2025 Lab model card
95 Granite 3.3 8B Instruct 69% Apr 2025 Lab model card
96 Claude 3.5 Haiku 64.2% Vals AI harness Oct 2024 Official leaderboard
97 Jamba 1.6 Large 54.8% Vals AI harness Mar 2025 Official leaderboard
98 BitNet b1.58 2B4T 43.4% Apr 2025 Lab model card
99 LFM2-1.2B 42.4% Nov 2025 Lab model card
100 Jamba 1.6 Mini 25.4% Vals AI harness Mar 2025 Official leaderboard
101 Qwen-14B 18.4% Sep 2023 Lab model card
102 Qwen-7B 6.80% Sep 2023 Lab model card

Vals AI’s leaderboard has 55 entries, 38 of them models we track. Switch the scores above to rank them together.

8 models have held number one in labs’ own numbers since its release in May 2023 10 models have held number one across all independent runs since its release in May 2023 10 models have held number one with claims included since its release in May 2023

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for Vals AI’s leaderboard and what labs claim in their cards.

Anthropic OpenAI xAI Google DeepMind Qwen DeepSeek Z.ai Meituan
Vals AI leaderboard Passed 85% in Dec 2024, by o1
Gemini 3 Pro 96.4%
Lab claims Passed 85% in Sep 2024, by o1-preview
LongCat-Flash-Thinking 99.2%

Sources: Vals AI’s leaderboard and labs’ model cards

OpenAI’s own number for o1-preview passed 85% in September 2024

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On Vals AI’s leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) LongCat-Flash-Thinking o3-mini o1-preview GPT-4o mini Qwen-14B 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) LongCat-Flash-Thinking o1-preview

Sources: Vals AI’s leaderboard and labs’ model cards

Labs headline it less often now, last in October 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

OpenAI
DeepSeek
Shanghai AI Lab
Moonshot AI
Anthropic
Reka
Baidu
NVIDIA
LG AI Research
Tencent
Meta
Microsoft
IBM
Qwen
Xiaomi Corp
Amazon
Mistral AI
Technology Innovation Institute
MiniMax
Inception Labs
Upstage
Z.ai
Nous Research
Meituan
Liquid AI
Deep Cogito
Prime Intellect
Arcee AI
Sarvam AI
DateLaunch postScoreAs the post writes it
Oct 13, 2026 LFM2.5-24B-A2B 96 MATH-500
May 28, 2026 LFM2.5-8B-A1B 88.8 MATH500
Mar 16, 2026 Nemotron 3 Nano 4B 95.4 MATH500 (reasoning on)
Mar 6, 2026 Sarvam 30B 97 Math500
Mar 6, 2026 Sarvam 105B 98.6 Math500
Jan 20, 2026 LFM2.5-1.2B-Thinking 88 MATH-500 (± 0.72)
Dec 3, 2025 Hermes 4.3 36B 93.8 MATH-500
Dec 1, 2025 Trinity Mini 92.1 Math-500
Nov 26, 2025 INTELLECT-3 98.1 MATH-500
Nov 19, 2025 Cogito v2.1 98.6 MATH-500
Oct 31, 2025 LongCat-Flash-Omni 97.6 MATH500(acc)
Oct 7, 2025 LFM2-8B-A1B 74.2 MATH 500
Sep 24, 2025 Code World Model (CWM) 96.6 Math-500
Sep 21, 2025 LongCat-Flash-Thinking 99.2 MATH500(Mean@1)
Aug 29, 2025 LongCat-Flash-Chat 96.4 MATH500(acc)
Aug 26, 2025 Hermes 4 405B 96.3 MATH-500 (reasoning)
Aug 25, 2025 InternVL3.5-241B-A28B 98 MATH500
Aug 18, 2025 nvidia-nemotron-nano-9b-v2 97.8 MATH500
Jul 28, 2025 GLM-4.5 98.2 MATH 500
Jul 25, 2025 Llama-3.3-Nemotron-Super-49B-v1.5 97.4 MATH500 (pass@1, avg of 4)
Jul 11, 2025 Kimi K2 97.4 MATH-500, Acc
Jul 10, 2025 Solar Pro 2 95.4 Math500
Jul 9, 2025 Phi-4-mini-flash-reasoning 92.5 Math500
Jun 30, 2025 ERNIE-4.5-300B-A47B 96.4 MATH-500
Jun 26, 2025 Mercury 83 MATH-500 (% accuracy)
Jun 16, 2025 MiniMax-M1-80k 96.8 MATH-500
May 20, 2025 Falcon-H1-34B 83.8 MATH-500
May 7, 2025 Mistral Medium 3 91 Math500 Instruct 0-shot
Apr 30, 2025 Phi-4-mini-reasoning 94.6 MATH-500
Apr 30, 2025 Amazon Nova Premier 82 MATH-500
Apr 29, 2025 MiMo-7B-RL 95.8 MATH-500 (Pass@1)
Apr 29, 2025 Qwen3-235B-A22B 98 MATH-500
Apr 16, 2025 Granite 3.3 8B Instruct 69 MATH-500
Apr 15, 2025 BitNet b1.58 2B4T 43.4 MATH-500
Apr 7, 2025 Llama-3.1-Nemotron-Ultra-253B-v1 97 MATH500 (reasoning on)
Apr 5, 2025 Llama 4 Behemoth 95 MATH-500
Mar 24, 2025 DeepSeek-V3 MATH-500 (Pass@1)
Mar 21, 2025 Hunyuan-T1 96.2 MATH-500
Mar 18, 2025 EXAONE Deep 32B 95.7 MATH-500 (pass@1)
Mar 18, 2025 Llama-3.3-Nemotron-Super-49B-v1 96.6 MATH500 (reasoning on)
Mar 16, 2025 ERNIE 4.5 Math-500
Mar 10, 2025 Reka Flash 3 90.6 MATH-500
Feb 24, 2025 Claude 3.7 Sonnet 96.2 Math problem-solving MATH 500 (64K extended thinking)
Jan 31, 2025 o3-mini 97.9 Math (pass@1)
Jan 20, 2025 Kimi k1.5 96.2 MATH-500 (EM), long-CoT
Jan 20, 2025 DeepSeek-R1 97.3 MATH-500 (Pass@1)
Jan 13, 2025 InternLM3-8B-Instruct 83 MATH-500(0-shot), Thinking Mode
Dec 26, 2024 DeepSeek-V3 90.2 MATH-500 (EM)
Dec 17, 2024 o1 96.4 MATH (pass @1)
Sep 12, 2024 o1-preview 85.5 MATH pass@1 (500-problem split)

46 system cards report it, from 18 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 15, 2026 Nemotron 3.5 Super 120B-A12B Base technical report 81.6 MATH-500, 4-shot sampled pass@1, n=32, Table 6 Capabilities
Oct 15, 2026 Nemotron 3.5 Lightning 30B-A3B Base technical report 79.5 MATH-500, 4-shot sampled pass@1, n=32, Table 6 Capabilities
Jan 5, 2026 Falcon H1R 7B technical report 97.4 MATH500 pass@1 (Table 4) Capabilities
Dec 23, 2025 Nemotron 3 Nano 30B-A3B Base technical report 78.6 MATH-500, 4-shot avg@32, base model Table 2 Capabilities
Nov 28, 2025 LFM2-8B-A1B technical report 74.2 MATH 500, LFM2-8B-A1B (Table 7) Capabilities
Nov 28, 2025 LFM2-1.2B technical report 42.4 MATH 500, LFM2-1.2B (Table 6) Capabilities
Nov 6, 2025 Nemotron Nano 12B v2 VL technical report 96.9 MATH-500, reasoning on, SFT stage 4 final model, mean of 4 runs (Table 6) Capabilities
Oct 31, 2025 LongCat-Flash-Omni technical report 97.6 MATH500 (acc) (Table 13) Capabilities
Sep 30, 2025 Code World Model (CWM) technical report 96.6 Math-500, pass@1 averaged over 20 samples (Table 12) Capabilities
Sep 23, 2025 LongCat-Flash-Thinking technical report 99.2 MATH-500(Mean@1) (Table 2) Capabilities
Sep 1, 2025 LongCat-Flash-Chat technical report 96.4 MATH500 (acc) (Table 3) Capabilities
Aug 25, 2025 InternVL3.5-241B-A28B technical report 98 MATH500 (Table 2) Capabilities
Aug 25, 2025 Hermes 4 405B technical report 96.2 MATH-500, reasoning mode (Table 3). Non-reasoning 73.8. v1 printed 96.3 Capabilities
Aug 8, 2025 GLM-4.5 technical report 98.2 MATH 500 (Table 4) Capabilities
Aug 8, 2025 GLM-4.5-Air technical report 98.1 MATH 500 (Table 4) Capabilities
Jul 30, 2025 Falcon-H1-34B technical report 83.8 MATH-500 (Table 24) Capabilities
Jul 21, 2025 Kimi K2 technical report 97.4 MATH-500 (Acc.) (Table 3) Capabilities
Jul 9, 2025 Phi-4-mini-flash-reasoning technical report 92.5 Math500, pass@1 averaged over 8 samples (Table 4) Capabilities
Jun 29, 2025 ERNIE-4.5-300B-A47B technical report 96.4 MATH-500 (Table 5) Capabilities
Jun 16, 2025 MiniMax-M1-80k technical report 96.8 MATH-500 (Table 2) Capabilities
Jun 12, 2025 Magistral Small 1.0 technical report 95.9 MATH-500, SFT + RL (Magistral Small) column (Table 3) Capabilities
Jun 12, 2025 Magistral Medium 1.0 technical report 94.3 MATH-500 (Table 2) Capabilities
May 14, 2025 Qwen3-8B technical report 97.4 MATH-500, thinking mode (Table 17). Non-thinking 87.4 (Table 18) Capabilities
May 14, 2025 Qwen3-4B technical report 97 MATH-500, thinking mode (Table 17). Non-thinking 84.8 (Table 18) Capabilities
May 14, 2025 Qwen3-32B technical report 97.2 MATH-500, thinking mode (Table 13). Non-thinking 88.6 (Table 14) Capabilities
May 14, 2025 Qwen3-30B-A3B technical report 98 MATH-500, thinking mode (Table 15). Non-thinking 89.8 (Table 16) Capabilities
May 14, 2025 Qwen3-235B-A22B technical report 98 MATH-500, thinking mode (Table 11). Non-thinking 91.2 (Table 12) Capabilities
May 14, 2025 Qwen3-14B technical report 96.8 MATH-500, thinking mode (Table 15). Non-thinking 90.0 (Table 16) Capabilities
May 14, 2025 Qwen3-1.7B technical report 93.4 MATH-500, thinking mode (Table 19). Non-thinking 73.0 (Table 20) Capabilities
May 12, 2025 MiMo-7B-RL technical report 95.8 MATH500 (Pass@1) (Table 4, Table 5, Table 6) Capabilities
May 2, 2025 Llama-3.3-Nemotron-Super-49B-v1 technical report 96.6 MATH500, reasoning on, reasoning off 74.0 (Table 4) Capabilities
May 2, 2025 Llama-3.1-Nemotron-Ultra-253B-v1 technical report 97 MATH500, reasoning on, reasoning off 80.4 (Table 5) Capabilities
Apr 30, 2025 Phi-4-mini-reasoning technical report 94.6 MATH-500, pass@1 (Table 3) Capabilities
Apr 30, 2025 Amazon Nova Premier technical report 82 MATH-500, 0-shot CoT, +/-3.4 (Table 1) Capabilities
Apr 16, 2025 BitNet b1.58 2B4T technical report 43.4 MATH-500 (0-shot; EM) (Table 1) Capabilities
Mar 16, 2025 EXAONE Deep 32B technical report 95.7 MATH-500, pass@1 with k=8 (Table 2) Capabilities
Mar 3, 2025 Phi-4 Mini technical report 71.8 MATH-500, Phi-4-Mini row of the CoT reasoning comparison (Table 9) Capabilities
Jan 20, 2025 Kimi k1.5 technical report 96.2 MATH-500 (EM), long-CoT (Table 2). Short-CoT 94.6 (Table 3) Capabilities
Jan 20, 2025 DeepSeek-R1 technical report 97.3 MATH-500 (Pass@1), DeepSeek-R1 (Table 8, same as final column of Table 3) Capabilities
Jan 20, 2025 DeepSeek-R1-Distill-Qwen-32B technical report 94.3 MATH-500 pass@1, DeepSeek-R1-Distill-Qwen-32B (Table 15) Capabilities
Jan 20, 2025 DeepSeek-R1-Distill-Qwen-14B technical report 93.9 MATH-500 pass@1, DeepSeek-R1-Distill-Qwen-14B (Table 15) Capabilities
Jan 20, 2025 DeepSeek-R1-Distill-Qwen-1.5B technical report 83.9 MATH-500 pass@1, DeepSeek-R1-Distill-Qwen-1.5B (Table 15) Capabilities
Jan 20, 2025 DeepSeek-R1-Distill-Llama-70B technical report 94.5 MATH-500 pass@1, DeepSeek-R1-Distill-Llama-70B (Table 15) Capabilities
Dec 26, 2024 DeepSeek-V3 technical report 90.2 MATH-500 (EM), DeepSeek-V3 (Table 6) Capabilities
Sep 28, 2023 Qwen-7B technical report 6.8 MATH on the Lightman et al. (2023) test set, greedy decoding (Table 12) Capabilities
Sep 28, 2023 Qwen-14B technical report 18.4 MATH on the Lightman et al. (2023) test set, greedy decoding (Table 12) Capabilities

The lab's number comes out ahead in 6 of 6 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. Every benchmark.

Model Lab says Board says Gap
o3-mini 97.9% Math (pass@1) 91.8% Leaderboard, Vals AI harness +6.1 pts
o1 96.4% MATH (pass @1) 90.4% Leaderboard, Vals AI harness +6.0 pts
Claude 3.7 Sonnet 96.2% Math problem-solving MATH 500 (64K extended thinking) 91.6% Leaderboard, Vals AI harness +4.6 pts
GLM-4.5 98.2% MATH 500 94% Leaderboard, Vals AI harness +4.2 pts
Qwen3-235B-A22B 98% MATH-500 94.6% Leaderboard, Vals AI harness +3.4 pts
Kimi K2 Jul 2025 97.4% MATH-500, Acc 94.2% Leaderboard, Vals AI harness +3.2 pts

Versions

Replaces MATH Replaced by AIME 2025

More in Math

All 113 in Math
  • AIME 2026 In use Models solve the 30 integer-answer problems from the 2026 AIME I and II math competitions. Scored by exact answer match. GPT-5.5 100%
  • IMO-AnswerBench In use 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. GLM-5.2 91%
  • HMMT 2026 In use Models solve the 33 final-answer problems from the February 2026 Harvard-MIT Mathematics Tournament. Scored by exact answer match. GPT-5.5 98.5%
  • PolyMath In use Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. Qwen3.7-Max 86.5%
  • MathArena (Apex, IMO, USAMO) In use Evaluates models on newly released math competitions after release. Final-answer or human/LLM-judged proof scores. Claude Opus 4.8 81.3%
  • MathArena Apex Shortlist In use Models solve 47 hard final-answer problems that MathArena selected from 2025 competitions, where the best models scored about 50% at selection. Scored by accuracy over repeated attempts. GPT-5.5 98.4%