Benchmark Almanac

WeirdML

GPT-6 Astra Pro 93.6% leads since September 2026
Not in model cards

Models write PyTorch code to solve 17 unusual machine learning tasks, with 5 attempts and feedback each. Scoring is mean best test accuracy.

This record is WeirdML v2 (17 tasks), launched in June 2025 and run on Epoch AI Benchmarking Hub. v1 (6 tasks, December 2024) is archived. WeirdML v3 replaced v2 in September 2026.

Released
June 2025
Kind
Machine learning engineering
Level
Expert
Scored
Checked automatically
Answers
Code
Strongest model tested
GPT-6 Astra #3 on the capability index
Last model added
September 2026
Scores refreshed
November 12th, 2026
131
Entries on its own leaderboard, 112 models we track

GPT-6 Astra Pro leads with 93.6% among models released by September 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in September 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

131 entries. Scroll sideways for the rest.

  1. 93.6 GPT-6 Astra Pro
  2. 93.3 GPT-6 Astra
  3. 92.9 Claude Fable 5.1
  4. 92.0 Claude Fable 5
  5. 91.8 Claude Opus 5
  6. 89.4 GPT-5.6 Sol
  7. 84.9 GPT-5.5
  8. 84.8 Gemini 3.8 Flash
  9. 82.9 Claude Opus 4.8
  10. 82.6 Kimi K3
  11. 78.3 GPT-5.6 Terra
  12. 78.0 Claude Opus 4.6
  13. 77.9 GPT-5.3 Codex
  14. 77.7 GPT-5.4
  15. 76.5 Qwen3.8-2.4T-A95B
  16. 76.4 Claude Opus 4.7
  17. 75.4 GLM-5.3
  18. 72.2 GPT-5.2
  19. 72.1 Gemini 3.1 Pro
  20. 70.1 GLM-5.2
  21. 69.9 Gemini 3 Pro
  22. 69.0 Claude Sonnet 5
  23. 67.3 Grok 4.6
  24. 66.2 DeepSeek V4 Pro 0813
  25. 66.1 Claude Sonnet 4.6
  26. 63.7 Claude Opus 4.5
  27. 63.0 DeepSeek V4 Flash 0731
  28. 62.6 Gemini 3.5 Flash
  29. 61.6 Gemini 3 Flash
  30. 60.9 GPT-5.6 Luna
  31. 60.8 GPT-5.1
  32. 60.7 GPT-5
  33. 60.4 GPT-5 Pro
  34. 60.4 Muse Spark 1.2
  35. 60.3 GPT-5.4 Mini
  36. 58.2 o3-pro
  37. 57.1 GLM-5.1
  38. 56.1 Gemini 3.6 Flash
  39. 55.9 Kimi K2.6
  40. 54.5 GPT-5-Codex
  41. 54.1 Kimi K2.7 Code
  42. 54.0 Gemini 2.5 Pro
  43. 52.7 GPT-5 mini
  44. 52.6 o4-mini
  45. 52.4 o3
  46. 52.3 x-ai/grok-4.20-beta
  47. 52.3 gemma4:31b-it-q4_K_M
  48. 52.2 Gemini 3.1 Flash-Lite
  49. 49.9 Grok 4.3
  50. 49.2 GPT-5.4 Nano
  51. 48.6 DeepSeek-V4-Pro
  52. 48.2 gpt-oss-120b
  53. 48.2 GLM-5
  54. 47.7 Claude Sonnet 4.5
  55. 47.6 o1-preview
  56. 46.7 DeepSeek-V3.2-Speciale
  57. 46.4 Grok 4.5
  58. 46.1 Claude Sonnet 4
  59. 46.1 o1
  60. 45.9 Claude Opus 4.1
  61. 45.7 Grok 4
  62. 45.6 DeepSeek-V4-Flash
  63. 45.6 Kimi K2.5
  64. 45.4 Claude Haiku 4.5
  65. 43.8 o3-mini
  66. 43.7 Claude Opus 4
  67. 43.7 mistral-medium-3.5:128b-q4_K_M
  68. 43.5 Nemotron 3 Ultra
  69. 43.2 Mercury 2
  70. 42.9 Grok 4 Fast
  71. 42.8 Kimi K2 Thinking
  72. 42.6 Grok-3 mini
  73. 41.9 Gemini 2.5 Flash
  74. 41.6 DeepSeek-R1
  75. 41.2 Qwen3-Coder-480B-A35B
  76. 41.0 Qwen3-235B-A22B-Thinking
  77. 41.0 Gemini 2.5 Flash
  78. 40.9 gpt-oss-20b
  79. 40.6 GLM-4.5
  80. 40.0 Claude 3.5 Sonnet
  81. 39.8 GPT-5 Chat
  82. 39.5 qwen3.5:27b-q4_K_M
  83. 39.5 DeepSeek-V3.2-Exp
  84. 39.4 GPT-4.5
  85. 39.4 Kimi K2
  86. 39.0 GPT-4.1
  87. 39.0 Gemini 3.5 Flash-Lite
  88. 38.7 Qwen3-235B-A22B-Instruct
  89. 38.4 DeepSeek-V3.1
  90. 38.1 GPT-5 nano
  91. 38.0 nemotron-3-super:120b-a12b-q4_K_M
  92. 37.6 GPT-4.1 mini
  93. 37.3 Qwen3-235B-A22B
  94. 37.2 Grok 3
  95. 37.0 MiniMax-M2.7
  96. 36.7 moonshotai/kimi-k2-0905
  97. 36.5 DeepSeek-R1
  98. 36.3 o1-mini
  99. 36.1 deepseek/deepseek-chat-v3-0324
  100. 35.2 Gemini 2.5 Flash-Lite
  101. 35.2 gemma4:26b-a4b-it-q4_K_M
  102. 35.1 grok-code-fast-1
  103. 34.5 qwen3.6:35b-a3b-q4_K_M
  104. 34.4 Qwen3-Coder-Next
  105. 33.1 Mistral Medium 3.1
  106. 32.3 Inkling
  107. 30.7 Claude 3.5 Haiku
  108. 29.8 Qwen3-30B-A3B
  109. 25.8 google/gemini-2.0-flash-001
  110. 25.1 GPT-4o
  111. 24.9 google/gemini-flash-1.5
  112. 24.5 Llama 4 Maverick
  113. 22.2 x-ai/grok-2-1212
  114. 22.2 google/gemini-pro-1.5
  115. 21.4 meta-llama/llama-3.1-405b-instruct
  116. 19.2 Claude 3 Opus
  117. 19.0 GPT-4.1 nano
  118. 18.0 GPT-4 Turbo
  119. 16.0 Qwen2.5-72B
  120. 14.4 Llama 3.3 70B Instruct
  121. 13.0 deepseek-r1-0528-qwen3-8b
  122. 12.4 GPT-4
  123. 11.8 GPT-4o mini
  124. 11.3 qwen2:72b-instruct-q8_0
  125. 10.2 Claude 3 Sonnet
  126. 9.84 Claude 3 Haiku
  127. 8.97 meta-llama/llama-3.1-70b-instruct
  128. 7.06 Claude 2.1
  129. 3.48 openai/gpt-3.5-turbo-16k
  130. 3.17 mistralai/mixtral-8x22b-instruct
  131. 1.73 meta-llama/llama-3.1-8b-instruct
  1. 1 GPT-6 Astra Pro max 93.6%
  2. 2 GPT-6 Astra max 93.3%
  3. 3 Claude Fable 5.1 max 92.9%
  4. 4 Claude Fable 5 max 92%
  5. 5 Claude Opus 5 max 91.8%
  6. 6 GPT-5.6 Sol max 89.4%
  7. 7 GPT-5.5 xhigh 84.9%
  8. 8 Gemini 3.8 Flash high 84.8%
  9. 9 Claude Opus 4.8 xhigh 82.9%
  10. 10 Kimi K3 max 82.6%
  11. 11 GPT-5.6 Terra high 78.3%
  12. 12 Claude Opus 4.6 high 78%
  13. 13 GPT-5.3 Codex xhigh 77.9%
  14. 14 GPT-5.4 xhigh 77.7%
  15. 15 Qwen3.8-2.4T-A95B xhigh 76.5%
  16. 16 Claude Opus 4.7 high 76.4%
  17. 17 GLM-5.3 max 75.4%
  18. 18 GPT-5.2 xhigh 72.2%
  19. 19 Gemini 3.1 Pro high 72.1%
  20. 20 GLM-5.2 max 70.1%
  21. 21 Gemini 3 Pro high 69.9%
  22. 22 Claude Sonnet 5 high 69%
  23. 23 Grok 4.6 high 67.3%
  24. 24 DeepSeek V4 Pro 0813 max 66.2%
  25. 25 Claude Sonnet 4.6 medium 66.1%
  26. 26 Claude Opus 4.5 high, 16k 63.7%
  27. 27 DeepSeek V4 Flash 0731 max 63%
  28. 28 Gemini 3.5 Flash high 62.6%
  29. 29 Gemini 3 Flash high 61.6%
  30. 30 GPT-5.6 Luna high 60.9%
  31. 31 GPT-5.1 high 60.8%
  32. 32 GPT-5 high 60.7%
  33. 33 GPT-5 Pro high 60.4%
  34. 34 Muse Spark 1.2 xhigh 60.4%
  35. 35 GPT-5.4 Mini high 60.3%
  36. 36 o3-pro high 58.2%
  37. 37 GLM-5.1 57.1%
  38. 38 Gemini 3.6 Flash high 56.1%
  39. 39 Kimi K2.6 55.9%
  40. 40 GPT-5-Codex high 54.5%
  41. 41 Kimi K2.7 Code 54.1%
  42. 42 Gemini 2.5 Pro Jun 2025, thinking 16k 54%
  43. 43 GPT-5 mini high 52.7%
  44. 44 o4-mini high 52.6%
  45. 45 o3 high 52.4%
  46. 46 x-ai/grok-4.20-beta 52.3%
  47. 47 gemma4:31b-it-q4_K_M 52.3%
  48. 48 Gemini 3.1 Flash-Lite high 52.2%
  49. 49 Grok 4.3 49.9%
  50. 50 GPT-5.4 Nano high 49.2%
  51. 51 DeepSeek-V4-Pro max 48.6%
  52. 52 gpt-oss-120b high 48.2%
  53. 53 GLM-5 thinking 48.2%
  54. 54 Claude Sonnet 4.5 thinking 16k 47.7%
  55. 55 o1-preview 47.6%
  56. 56 DeepSeek-V3.2-Speciale 46.7%
  57. 57 Grok 4.5 46.4%
  58. 58 Claude Sonnet 4 thinking 16k 46.1%
  59. 59 o1 high 46.1%
  60. 60 Claude Opus 4.1 thinking 16k 45.9%
  61. 61 Grok 4 45.7%
  62. 62 DeepSeek-V4-Flash max 45.6%
  63. 63 Kimi K2.5 45.6%
  64. 64 Claude Haiku 4.5 no thinking 45.4%
  65. 65 o3-mini high 43.8%
  66. 66 Claude Opus 4 thinking 16k 43.7%
  67. 67 mistral-medium-3.5:128b-q4_K_M 43.7%
  68. 68 Nemotron 3 Ultra 43.5%
  69. 69 Mercury 2 high 43.2%
  70. 70 Grok 4 Fast 42.9%
  71. 71 Kimi K2 Thinking 42.8%
  72. 72 Grok-3 mini high 42.6%
  73. 73 Gemini 2.5 Flash Sep 2025, thinking 16k 41.9%
  74. 74 DeepSeek-R1 May 2025 41.6%
  75. 75 Qwen3-Coder-480B-A35B 41.2%
  76. 76 Qwen3-235B-A22B-Thinking Jul 2025 41%
  77. 77 Gemini 2.5 Flash Jun 2025, thinking 16k 41%
  78. 78 gpt-oss-20b high 40.9%
  79. 79 GLM-4.5 thinking 40.6%
  80. 80 Claude 3.5 Sonnet 40%
  81. 81 GPT-5 Chat 39.8%
  82. 82 qwen3.5:27b-q4_K_M 39.5%
  83. 83 DeepSeek-V3.2-Exp thinking 39.5%
  84. 84 GPT-4.5 39.4%
  85. 85 Kimi K2 Jul 2025 39.4%
  86. 86 GPT-4.1 39%
  87. 87 Gemini 3.5 Flash-Lite high 39%
  88. 88 Qwen3-235B-A22B-Instruct Jul 2025 38.7%
  89. 89 DeepSeek-V3.1 thinking 38.4%
  90. 90 GPT-5 nano high 38.1%
  91. 91 nemotron-3-super:120b-a12b-q4_K_M 38%
  92. 92 GPT-4.1 mini 37.6%
  93. 93 Qwen3-235B-A22B thinking 37.3%
  94. 94 Grok 3 37.2%
  95. 95 MiniMax-M2.7 37%
  96. 96 moonshotai/kimi-k2-0905 36.7%
  97. 97 DeepSeek-R1 36.5%
  98. 98 o1-mini 36.3%
  99. 99 deepseek/deepseek-chat-v3-0324 36.1%
  100. 100 Gemini 2.5 Flash-Lite Jun 2025, thinking 16k 35.2%
  101. 101 gemma4:26b-a4b-it-q4_K_M 35.2%
  102. 102 grok-code-fast-1 35.1%
  103. 103 qwen3.6:35b-a3b-q4_K_M 34.5%
  104. 104 Qwen3-Coder-Next 34.4%
  105. 105 Mistral Medium 3.1 33.1%
  106. 106 Inkling high 32.3%
  107. 107 Claude 3.5 Haiku 30.7%
  108. 108 Qwen3-30B-A3B thinking 29.8%
  109. 109 google/gemini-2.0-flash-001 25.8%
  110. 110 GPT-4o Nov 2024 25.1%
  111. 111 google/gemini-flash-1.5 24.9%
  112. 112 Llama 4 Maverick 24.5%
  113. 113 x-ai/grok-2-1212 22.2%
  114. 114 google/gemini-pro-1.5 22.2%
  115. 115 meta-llama/llama-3.1-405b-instruct 21.4%
  116. 116 Claude 3 Opus 19.2%
  117. 117 GPT-4.1 nano 19%
  118. 118 GPT-4 Turbo Apr 2024 18%
  119. 119 Qwen2.5-72B 16%
  120. 120 Llama 3.3 70B Instruct 14.4%
  121. 121 deepseek-r1-0528-qwen3-8b 13%
  122. 122 GPT-4 Jun 2023 12.4%
  123. 123 GPT-4o mini 11.8%
  124. 124 qwen2:72b-instruct-q8_0 11.3%
  125. 125 Claude 3 Sonnet 10.2%
  126. 126 Claude 3 Haiku 9.84%
  127. 127 meta-llama/llama-3.1-70b-instruct 8.97%
  128. 128 Claude 2.1 7.06%
  129. 129 openai/gpt-3.5-turbo-16k 3.48%
  130. 130 mistralai/mixtral-8x22b-instruct 3.17%
  131. 131 meta-llama/llama-3.1-8b-instruct 1.73%

Source: Official WeirdML leaderboard

#EntryScoreSettingDateSource
1 GPT-6 Astra Pro 93.6% max Sep 2026 Official leaderboard
2 GPT-6 Astra 93.3% max Sep 2026 Official leaderboard
3 Claude Fable 5.1 92.9% max Sep 2026 Official leaderboard
4 Claude Fable 5 92% max Jun 2026 Official leaderboard
5 Claude Opus 5 91.8% max Jul 2026 Official leaderboard
6 GPT-5.6 Sol 89.4% max Jul 2026 Official leaderboard
7 GPT-5.5 84.9% xhigh Apr 2026 Official leaderboard
8 Gemini 3.8 Flash 84.8% high Sep 2026 Official leaderboard
9 Claude Opus 4.8 82.9% xhigh May 2026 Official leaderboard
10 Kimi K3 82.6% max Jul 2026 Official leaderboard
11 GPT-5.6 Terra 78.3% high Jul 2026 Official leaderboard
12 Claude Opus 4.6 78% high Feb 2026 Official leaderboard
13 GPT-5.3 Codex 77.9% xhigh Feb 2026 Official leaderboard
14 GPT-5.4 77.7% xhigh Mar 2026 Official leaderboard
15 Qwen3.8-2.4T-A95B 76.5% xhigh Aug 2026 Official leaderboard
16 Claude Opus 4.7 76.4% high Apr 2026 Official leaderboard
17 GLM-5.3 75.4% max Aug 2026 Official leaderboard
18 GPT-5.2 72.2% xhigh Dec 2025 Official leaderboard
19 Gemini 3.1 Pro 72.1% high Feb 2026 Official leaderboard
20 GLM-5.2 70.1% max Jun 2026 Official leaderboard
21 Gemini 3 Pro 69.9% high Nov 2025 Official leaderboard
22 Claude Sonnet 5 69% high Jun 2026 Official leaderboard
23 Grok 4.6 67.3% high Aug 2026 Official leaderboard
24 DeepSeek V4 Pro 0813 66.2% max Aug 2026 Official leaderboard
25 Claude Sonnet 4.6 66.1% medium Feb 2026 Official leaderboard
26 Claude Opus 4.5 63.7% high, 16k Nov 2025 Official leaderboard
27 DeepSeek V4 Flash 0731 63% max Jul 2026 Official leaderboard
28 Gemini 3.5 Flash 62.6% high May 2026 Official leaderboard
29 Gemini 3 Flash 61.6% high Dec 2025 Official leaderboard
30 GPT-5.6 Luna 60.9% high Jul 2026 Official leaderboard
31 GPT-5.1 60.8% high Nov 2025 Official leaderboard
32 GPT-5 60.7% high Aug 2025 Official leaderboard
33 GPT-5 Pro 60.4% high Oct 2025 Official leaderboard
34 Muse Spark 1.2 60.4% xhigh Aug 2026 Official leaderboard
35 GPT-5.4 Mini 60.3% high Mar 2026 Official leaderboard
36 o3-pro 58.2% high Jun 2025 Official leaderboard
37 GLM-5.1 57.1% Apr 2026 Official leaderboard
38 Gemini 3.6 Flash 56.1% high Jul 2026 Official leaderboard
39 Kimi K2.6 55.9% Apr 2026 Official leaderboard
40 GPT-5-Codex 54.5% high Sep 2025 Official leaderboard
41 Kimi K2.7 Code 54.1% Jun 2026 Official leaderboard
42 Gemini 2.5 Pro Jun 2025 54% thinking 16k Jun 2025 Official leaderboard
43 GPT-5 mini 52.7% high Aug 2025 Official leaderboard
44 o4-mini 52.6% high Apr 2025 Official leaderboard
45 o3 52.4% high Apr 2025 Official leaderboard
46 x-ai/grok-4.20-beta 52.3% Mar 2026 Official leaderboard
47 gemma4:31b-it-q4_K_M 52.3% Apr 2026 Official leaderboard
48 Gemini 3.1 Flash-Lite 52.2% high Mar 2026 Official leaderboard
49 Grok 4.3 49.9% Apr 2026 Official leaderboard
50 GPT-5.4 Nano 49.2% high Mar 2026 Official leaderboard
51 DeepSeek-V4-Pro 48.6% max Apr 2026 Official leaderboard
52 gpt-oss-120b 48.2% high Aug 2025 Official leaderboard
53 GLM-5 48.2% thinking Feb 2026 Official leaderboard
54 Claude Sonnet 4.5 47.7% thinking 16k Sep 2025 Official leaderboard
55 o1-preview 47.6% Sep 2024 Official leaderboard
56 DeepSeek-V3.2-Speciale 46.7% Dec 2025 Official leaderboard
57 Grok 4.5 46.4% Jul 2026 Official leaderboard
58 Claude Sonnet 4 46.1% thinking 16k May 2025 Official leaderboard
59 o1 46.1% high Dec 2024 Official leaderboard
60 Claude Opus 4.1 45.9% thinking 16k Aug 2025 Official leaderboard
61 Grok 4 45.7% Jul 2025 Official leaderboard
62 DeepSeek-V4-Flash 45.6% max Apr 2026 Official leaderboard
63 Kimi K2.5 45.6% Jan 2026 Official leaderboard
64 Claude Haiku 4.5 45.4% no thinking Oct 2025 Official leaderboard
65 o3-mini 43.8% high Jan 2025 Official leaderboard
66 Claude Opus 4 43.7% thinking 16k May 2025 Official leaderboard
67 mistral-medium-3.5:128b-q4_K_M 43.7% Apr 2026 Official leaderboard
68 Nemotron 3 Ultra 43.5% Jun 2026 Official leaderboard
69 Mercury 2 43.2% high Feb 2026 Official leaderboard
70 Grok 4 Fast 42.9% Sep 2025 Official leaderboard
71 Kimi K2 Thinking 42.8% Nov 2025 Official leaderboard
72 Grok-3 mini 42.6% high Apr 2025 Official leaderboard
73 Gemini 2.5 Flash Sep 2025 41.9% thinking 16k Sep 2025 Official leaderboard
74 DeepSeek-R1 May 2025 41.6% May 2025 Official leaderboard
75 Qwen3-Coder-480B-A35B 41.2% Jul 2025 Official leaderboard
76 Qwen3-235B-A22B-Thinking Jul 2025 41% Jul 2025 Official leaderboard
77 Gemini 2.5 Flash Jun 2025 41% thinking 16k Jun 2025 Official leaderboard
78 gpt-oss-20b 40.9% high Aug 2025 Official leaderboard
79 GLM-4.5 40.6% thinking Aug 2025 Official leaderboard
80 Claude 3.5 Sonnet 40% Jun 2024 Official leaderboard
81 GPT-5 Chat 39.8% Aug 2025 Official leaderboard
82 qwen3.5:27b-q4_K_M 39.5% Feb 2026 Official leaderboard
83 DeepSeek-V3.2-Exp 39.5% thinking Sep 2025 Official leaderboard
84 GPT-4.5 39.4% Feb 2025 Official leaderboard
85 Kimi K2 Jul 2025 39.4% Jul 2025 Official leaderboard
86 GPT-4.1 39% Apr 2025 Official leaderboard
87 Gemini 3.5 Flash-Lite 39% high Jul 2026 Official leaderboard
88 Qwen3-235B-A22B-Instruct Jul 2025 38.7% Jul 2025 Official leaderboard
89 DeepSeek-V3.1 38.4% thinking Aug 2025 Official leaderboard
90 GPT-5 nano 38.1% high Aug 2025 Official leaderboard
91 nemotron-3-super:120b-a12b-q4_K_M 38% Mar 2026 Official leaderboard
92 GPT-4.1 mini 37.6% Apr 2025 Official leaderboard
93 Qwen3-235B-A22B 37.3% thinking Apr 2025 Official leaderboard
94 Grok 3 37.2% Apr 2025 Official leaderboard
95 MiniMax-M2.7 37% Mar 2026 Official leaderboard
96 moonshotai/kimi-k2-0905 36.7% Sep 2025 Official leaderboard
97 DeepSeek-R1 36.5% Jan 2025 Official leaderboard
98 o1-mini 36.3% Sep 2024 Official leaderboard
99 deepseek/deepseek-chat-v3-0324 36.1% Mar 2025 Official leaderboard
100 Gemini 2.5 Flash-Lite Jun 2025 35.2% thinking 16k Jun 2025 Official leaderboard
101 gemma4:26b-a4b-it-q4_K_M 35.2% Apr 2026 Official leaderboard
102 grok-code-fast-1 35.1% Aug 2025 Official leaderboard
103 qwen3.6:35b-a3b-q4_K_M 34.5% Apr 2026 Official leaderboard
104 Qwen3-Coder-Next 34.4% Feb 2026 Official leaderboard
105 Mistral Medium 3.1 33.1% Aug 2025 Official leaderboard
106 Inkling 32.3% high Jul 2026 Official leaderboard
107 Claude 3.5 Haiku 30.7% Oct 2024 Official leaderboard
108 Qwen3-30B-A3B 29.8% thinking Apr 2025 Official leaderboard
109 google/gemini-2.0-flash-001 25.8% Feb 2025 Official leaderboard
110 GPT-4o Nov 2024 25.1% Nov 2024 Official leaderboard
111 google/gemini-flash-1.5 24.9% Sep 2024 Official leaderboard
112 Llama 4 Maverick 24.5% Apr 2025 Official leaderboard
113 x-ai/grok-2-1212 22.2% Dec 2024 Official leaderboard
114 google/gemini-pro-1.5 22.2% Sep 2024 Official leaderboard
115 meta-llama/llama-3.1-405b-instruct 21.4% Jul 2024 Official leaderboard
116 Claude 3 Opus 19.2% Feb 2024 Official leaderboard
117 GPT-4.1 nano 19% Apr 2025 Official leaderboard
118 GPT-4 Turbo Apr 2024 18% Apr 2024 Official leaderboard
119 Qwen2.5-72B 16% Sep 2024 Official leaderboard
120 Llama 3.3 70B Instruct 14.4% Dec 2024 Official leaderboard
121 deepseek-r1-0528-qwen3-8b 13% May 2025 Official leaderboard
122 GPT-4 Jun 2023 12.4% Jun 2023 Official leaderboard
123 GPT-4o mini 11.8% Jul 2024 Official leaderboard
124 qwen2:72b-instruct-q8_0 11.3% Jun 2024 Official leaderboard
125 Claude 3 Sonnet 10.2% Feb 2024 Official leaderboard
126 Claude 3 Haiku 9.84% Mar 2024 Official leaderboard
127 meta-llama/llama-3.1-70b-instruct 8.97% Jul 2024 Official leaderboard
128 Claude 2.1 7.06% Nov 2023 Official leaderboard
129 openai/gpt-3.5-turbo-16k 3.48% Jan 2024 Official leaderboard
130 mistralai/mixtral-8x22b-instruct 3.17% Apr 2024 Official leaderboard
131 meta-llama/llama-3.1-8b-instruct 1.73% Jul 2024 Official leaderboard

Average accuracy against output tokens

Average total output tokens per response (reasoning plus completion) against average accuracy over all tasks. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 80% 100% 1001,00010K100K Output tokens per response (tokens), log scale, lower is better Average accuracy Claude 3 Opus: Average accuracy 19.2%, output tokens per response 1,326 tokens Claude 3 Sonnet: Average accuracy 10.2%, output tokens per response 1,389 tokens Claude 3 Haiku: Average accuracy 9.84%, output tokens per response 1,275 tokens GPT-4 Turbo: Average accuracy 18%, output tokens per response 915 tokens Claude 3.5 Sonnet: Average accuracy 40%, output tokens per response 2,121 tokens Claude 3.5 Sonnet: Average accuracy 31%, output tokens per response 1,966 tokens GPT-4o mini: Average accuracy 11.8%, output tokens per response 1,112 tokens o1-mini: Average accuracy 36.3%, output tokens per response 3,877 tokens o1-preview: Average accuracy 47.6%, output tokens per response 5,129 tokens Qwen2.5-72B: Average accuracy 16%, output tokens per response 1,380 tokens Claude 3.5 Haiku: Average accuracy 30.7%, output tokens per response 1,764 tokens GPT-4o: Average accuracy 25.1%, output tokens per response 1,135 tokens Llama 3.3 70B Instruct: Average accuracy 14.4%, output tokens per response 1,109 tokens o1 (high): Average accuracy 46.1%, output tokens per response 5,767 tokens DeepSeek-R1: Average accuracy 36.5%, output tokens per response 5,170 tokens o3-mini (high): Average accuracy 43.8%, output tokens per response 6,930 tokens Llama 4 Maverick: Average accuracy 24.5%, output tokens per response 992 tokens Grok 3: Average accuracy 37.2%, output tokens per response 2,122 tokens Grok-3 mini (high): Average accuracy 42.6%, output tokens per response 4,919 tokens GPT-4.1: Average accuracy 39%, output tokens per response 1,723 tokens GPT-4.1 mini: Average accuracy 37.6%, output tokens per response 1,532 tokens GPT-4.1 nano: Average accuracy 19%, output tokens per response 945 tokens o3 (high): Average accuracy 52.4%, output tokens per response 4,622 tokens o4-mini (high): Average accuracy 52.6%, output tokens per response 7,922 tokens Qwen3-235B-A22B (thinking): Average accuracy 37.3%, output tokens per response 5,575 tokens Qwen3-30B-A3B (thinking): Average accuracy 29.8%, output tokens per response 4,614 tokens Claude Opus 4 (thinking 16k): Average accuracy 43.7%, output tokens per response 3,136 tokens Claude Sonnet 4 (thinking 16k): Average accuracy 46.1%, output tokens per response 3,147 tokens Claude Sonnet 4 (no thinking): Average accuracy 43.9%, output tokens per response 2,693 tokens DeepSeek-R1: Average accuracy 41.6%, output tokens per response 5,243 tokens deepseek-r1-0528-qwen3-8b: Average accuracy 13%, output tokens per response 5,756 tokens Gemini 2.5 Pro (thinking 16k): Average accuracy 54%, output tokens per response 7,312 tokens o3-pro (high): Average accuracy 58.2%, output tokens per response 5,243 tokens Gemini 2.5 Flash (thinking 16k): Average accuracy 41%, output tokens per response 7,457 tokens Gemini 2.5 Flash-Lite (thinking 16k): Average accuracy 35.2%, output tokens per response 12K tokens Grok 4: Average accuracy 45.7%, output tokens per response 5,154 tokens Kimi K2: Average accuracy 39.4%, output tokens per response 1,710 tokens Qwen3-235B-A22B-Instruct: Average accuracy 38.7%, output tokens per response 1,961 tokens Qwen3-235B-A22B-Thinking: Average accuracy 41%, output tokens per response 5,771 tokens Qwen3-Coder-480B-A35B: Average accuracy 41.2%, output tokens per response 1,730 tokens GLM-4.5 (thinking): Average accuracy 40.6%, output tokens per response 4,988 tokens Claude Opus 4.1 (thinking 16k): Average accuracy 45.9%, output tokens per response 3,180 tokens gpt-oss-120b (high): Average accuracy 48.2%, output tokens per response 9,314 tokens gpt-oss-120b (medium): Average accuracy 41.9%, output tokens per response 2,346 tokens gpt-oss-20b (high): Average accuracy 40.9%, output tokens per response 9,131 tokens gpt-oss-20b (medium): Average accuracy 36.8%, output tokens per response 2,593 tokens GPT-5 (high): Average accuracy 60.7%, output tokens per response 11K tokens GPT-5 mini (high): Average accuracy 52.7%, output tokens per response 10K tokens GPT-5 nano (high): Average accuracy 38.1%, output tokens per response 12K tokens GPT-5 nano (low): Average accuracy 25.9%, output tokens per response 1,673 tokens Mistral Medium 3.1: Average accuracy 33.1%, output tokens per response 2,428 tokens DeepSeek-V3.1 (thinking): Average accuracy 38.4%, output tokens per response 4,545 tokens DeepSeek-V3.1: Average accuracy 37.5%, output tokens per response 2,058 tokens grok-code-fast-1: Average accuracy 35.1%, output tokens per response 2,850 tokens GPT-5-Codex (high): Average accuracy 54.5%, output tokens per response 21K tokens Grok 4 Fast: Average accuracy 42.9%, output tokens per response 4,009 tokens Gemini 2.5 Flash (thinking 16k): Average accuracy 41.9%, output tokens per response 5,190 tokens Claude Sonnet 4.5 (thinking 16k): Average accuracy 47.7%, output tokens per response 3,260 tokens Claude Sonnet 4.5 (no thinking): Average accuracy 46.7%, output tokens per response 2,406 tokens DeepSeek-V3.2-Exp (thinking): Average accuracy 39.5%, output tokens per response 4,236 tokens GPT-5 Pro (high): Average accuracy 60.4%, output tokens per response 11K tokens Claude Haiku 4.5 (no thinking): Average accuracy 45.4%, output tokens per response 2,286 tokens Claude Haiku 4.5 (thinking 16k): Average accuracy 44.1%, output tokens per response 3,705 tokens Kimi K2 Thinking: Average accuracy 42.8%, output tokens per response 5,090 tokens GPT-5.1 (high): Average accuracy 60.8%, output tokens per response 13K tokens Gemini 3 Pro (high): Average accuracy 69.9%, output tokens per response 7,795 tokens Claude Opus 4.5 (high, 16k): Average accuracy 63.7%, output tokens per response 3,684 tokens DeepSeek-V3.2-Speciale: Average accuracy 46.7%, output tokens per response 12K tokens GPT-5.2 (xhigh): Average accuracy 72.2%, output tokens per response 28K tokens GPT-5.2 (medium): Average accuracy 63.4%, output tokens per response 4,928 tokens Gemini 3 Flash (high): Average accuracy 61.6%, output tokens per response 14K tokens Kimi K2.5: Average accuracy 45.6%, output tokens per response 2,892 tokens Qwen3-Coder-Next: Average accuracy 34.4%, output tokens per response 2,279 tokens Claude Opus 4.6 (high): Average accuracy 78%, output tokens per response 32K tokens Claude Opus 4.6 (no thinking): Average accuracy 65.9%, output tokens per response 3,877 tokens GPT-5.3 Codex (xhigh): Average accuracy 77.9%, output tokens per response 34K tokens GLM-5 (thinking): Average accuracy 48.2%, output tokens per response 3,326 tokens Claude Sonnet 4.6 (medium): Average accuracy 66.1%, output tokens per response 28K tokens Gemini 3.1 Pro (high): Average accuracy 72.1%, output tokens per response 21K tokens Mercury 2 (high): Average accuracy 43.2%, output tokens per response 6,449 tokens Gemini 3.1 Flash-Lite (high): Average accuracy 52.2%, output tokens per response 8,128 tokens GPT-5.4 (xhigh): Average accuracy 77.7%, output tokens per response 72K tokens GPT-5.4 (no thinking): Average accuracy 57.4%, output tokens per response 2,780 tokens GPT-5.4 Mini (high): Average accuracy 60.3%, output tokens per response 55K tokens GPT-5.4 Mini (no thinking): Average accuracy 37.9%, output tokens per response 2,133 tokens GPT-5.4 Nano (high): Average accuracy 49.2%, output tokens per response 6,445 tokens GPT-5.4 Nano (no thinking): Average accuracy 38%, output tokens per response 2,155 tokens MiniMax-M2.7: Average accuracy 37%, output tokens per response 4,305 tokens GLM-5.1: Average accuracy 57.1%, output tokens per response 12K tokens Claude Opus 4.7 (high): Average accuracy 76.4%, output tokens per response 9,965 tokens Claude Opus 4.7 (no thinking): Average accuracy 76.4%, output tokens per response 3,594 tokens Claude Opus 4.7 (max): Average accuracy 75.5%, output tokens per response 44K tokens Kimi K2.6: Average accuracy 55.9%, output tokens per response 17K tokens GPT-5.5 (xhigh): Average accuracy 84.9%, output tokens per response 15K tokens GPT-5.5 (high): Average accuracy 83.9%, output tokens per response 13K tokens GPT-5.5 (no thinking): Average accuracy 67.2%, output tokens per response 2,493 tokens DeepSeek-V4-Flash (max): Average accuracy 45.6%, output tokens per response 7,978 tokens DeepSeek-V4-Flash (high): Average accuracy 43.8%, output tokens per response 3,570 tokens DeepSeek-V4-Pro (max): Average accuracy 48.6%, output tokens per response 17K tokens DeepSeek-V4-Pro (high): Average accuracy 46.5%, output tokens per response 5,183 tokens Grok 4.3: Average accuracy 49.9%, output tokens per response 3,525 tokens Gemini 3.5 Flash (high): Average accuracy 62.6%, output tokens per response 13K tokens Claude Opus 4.8 (xhigh): Average accuracy 82.9%, output tokens per response 13K tokens Claude Opus 4.8 (medium): Average accuracy 76%, output tokens per response 4,332 tokens Nemotron 3 Ultra: Average accuracy 43.5%, output tokens per response 4,368 tokens Claude Fable 5 (max): Average accuracy 92%, output tokens per response 22K tokens Claude Fable 5 (high): Average accuracy 87.8%, output tokens per response 8,393 tokens Kimi K2.7 Code: Average accuracy 54.1%, output tokens per response 9,342 tokens GLM-5.2 (max): Average accuracy 70.1%, output tokens per response 22K tokens GLM-5.2 (high): Average accuracy 67.3%, output tokens per response 12K tokens Claude Sonnet 5 (high): Average accuracy 69%, output tokens per response 12K tokens GPT-5.6 Luna (high): Average accuracy 60.9%, output tokens per response 5,867 tokens GPT-5.6 Sol (max): Average accuracy 89.4%, output tokens per response 20K tokens GPT-5.6 Sol (max): Average accuracy 87%, output tokens per response 11K tokens Inkling (high): Average accuracy 32.3%, output tokens per response 4,300 tokens Kimi K3 (max): Average accuracy 82.6%, output tokens per response 16K tokens Gemini 3.5 Flash-Lite (high): Average accuracy 39%, output tokens per response 3,366 tokens Gemini 3.6 Flash (high): Average accuracy 56.1%, output tokens per response 15K tokens Claude Opus 5 (max): Average accuracy 91.8%, output tokens per response 25K tokens Claude Opus 5 (high): Average accuracy 91.6%, output tokens per response 12K tokens Claude Opus 5 (no thinking): Average accuracy 86.3%, output tokens per response 7,394 tokens DeepSeek V4 Flash 0731 (max): Average accuracy 63%, output tokens per response 18K tokens DeepSeek V4 Flash 0731 (high): Average accuracy 57%, output tokens per response 16K tokens Muse Spark 1.2 (xhigh): Average accuracy 60.4%, output tokens per response 9,948 tokens Grok 4.6 (high): Average accuracy 67.3%, output tokens per response 7,558 tokens Qwen3.8-2.4T-A95B (xhigh): Average accuracy 76.5%, output tokens per response 32K tokens DeepSeek V4 Pro 0813 (max): Average accuracy 66.2%, output tokens per response 18K tokens GLM-5.3 (max): Average accuracy 75.4%, output tokens per response 17K tokens Claude Fable 5.1 (max): Average accuracy 92.9%, output tokens per response 32K tokens Claude Fable 5.1 (high): Average accuracy 92.3%, output tokens per response 8,369 tokens Gemini 3.8 Flash (high): Average accuracy 84.8%, output tokens per response 25K tokens gemma4:31b-it-q4_K_M: Average accuracy 52.3%, output tokens per response 4,921 tokens. Not in our model list x-ai/grok-4.20-beta: Average accuracy 52.3%, output tokens per response 2,945 tokens. Not in our model list mistral-medium-3.5:128b-q4_K_M: Average accuracy 43.7%, output tokens per response 6,593 tokens. Not in our model list qwen3.5:27b-q4_K_M: Average accuracy 39.5%, output tokens per response 2,664 tokens. Not in our model list nemotron-3-super:120b-a12b-q4_K_M: Average accuracy 38%, output tokens per response 3,543 tokens. Not in our model list moonshotai/kimi-k2-0905: Average accuracy 36.7%, output tokens per response 1,953 tokens. Not in our model list deepseek/deepseek-chat-v3-0324: Average accuracy 36.1%, output tokens per response 1,548 tokens. Not in our model list gemma4:26b-a4b-it-q4_K_M: Average accuracy 35.2%, output tokens per response 8,239 tokens. Not in our model list qwen3.6:35b-a3b-q4_K_M: Average accuracy 34.5%, output tokens per response 11K tokens. Not in our model list google/gemini-2.0-flash-001: Average accuracy 25.8%, output tokens per response 1,524 tokens. Not in our model list x-ai/grok-2-1212: Average accuracy 22.2%, output tokens per response 1,337 tokens. Not in our model list google/gemini-pro-1.5: Average accuracy 22.2%, output tokens per response 1,010 tokens. Not in our model list meta-llama/llama-3.1-405b-instruct: Average accuracy 21.4%, output tokens per response 1,008 tokens. Not in our model list qwen2:72b-instruct-q8_0: Average accuracy 11.3%, output tokens per response 1,257 tokens. Not in our model list meta-llama/llama-3.1-70b-instruct: Average accuracy 8.97%, output tokens per response 911 tokens. Not in our model list mistralai/mixtral-8x22b-instruct: Average accuracy 3.17%, output tokens per response 2,523 tokens. Not in our model list meta-llama/llama-3.1-8b-instruct: Average accuracy 1.73%, output tokens per response 1,052 tokens. Not in our model list GPT-4: Average accuracy 12.4%, output tokens per response 728 tokens Claude 2.1: Average accuracy 7.06%, output tokens per response 480 tokens GPT-4.5: Average accuracy 39.4%, output tokens per response 979 tokens GPT-5 Chat: Average accuracy 39.8%, output tokens per response 1,437 tokens GPT-5.2 (no thinking): Average accuracy 49.6%, output tokens per response 2,243 tokens Claude Opus 4.8 (no thinking): Average accuracy 70.5%, output tokens per response 2,414 tokens Grok 4.5: Average accuracy 46.4%, output tokens per response 1,723 tokens GPT-5.6 Sol (high): Average accuracy 88.8%, output tokens per response 7,245 tokens GPT-5.6 Terra (high): Average accuracy 78.3%, output tokens per response 3,479 tokens GPT-6 Astra (max): Average accuracy 93.3%, output tokens per response 22K tokens GPT-6 Astra (high): Average accuracy 92.9%, output tokens per response 8,056 tokens GPT-6 Astra Pro (max): Average accuracy 93.6%, output tokens per response 64K tokens google/gemini-flash-1.5: Average accuracy 24.9%, output tokens per response 833 tokens. Not in our model list openai/gpt-3.5-turbo-16k: Average accuracy 3.48%, output tokens per response 393 tokens. Not in our model list GPT-6 Astra Pro (max)GPT-6 Astra (max)GPT-6 Astra (high)GPT-5.6 Sol (high)GPT-5.6 Terra (high)Claude Opus 4.8 (no thinking)GPT-5.2 (no thinking)Grok 4.5GPT-5 ChatGPT-4.5google/gemini-flash-1.5GPT-4Claude 2.1openai/gpt-3.5-turbo-16k 0% 20% 40% 60% 80% 100% 1001,00010K100K Output tokens per response (tokens), log scale, lower is better Average accuracy Claude 3 Opus: Average accuracy 19.2%, output tokens per response 1,326 tokens Claude 3 Sonnet: Average accuracy 10.2%, output tokens per response 1,389 tokens Claude 3 Haiku: Average accuracy 9.84%, output tokens per response 1,275 tokens GPT-4 Turbo: Average accuracy 18%, output tokens per response 915 tokens Claude 3.5 Sonnet: Average accuracy 40%, output tokens per response 2,121 tokens Claude 3.5 Sonnet: Average accuracy 31%, output tokens per response 1,966 tokens GPT-4o mini: Average accuracy 11.8%, output tokens per response 1,112 tokens o1-mini: Average accuracy 36.3%, output tokens per response 3,877 tokens o1-preview: Average accuracy 47.6%, output tokens per response 5,129 tokens Qwen2.5-72B: Average accuracy 16%, output tokens per response 1,380 tokens Claude 3.5 Haiku: Average accuracy 30.7%, output tokens per response 1,764 tokens GPT-4o: Average accuracy 25.1%, output tokens per response 1,135 tokens Llama 3.3 70B Instruct: Average accuracy 14.4%, output tokens per response 1,109 tokens o1 (high): Average accuracy 46.1%, output tokens per response 5,767 tokens DeepSeek-R1: Average accuracy 36.5%, output tokens per response 5,170 tokens o3-mini (high): Average accuracy 43.8%, output tokens per response 6,930 tokens Llama 4 Maverick: Average accuracy 24.5%, output tokens per response 992 tokens Grok 3: Average accuracy 37.2%, output tokens per response 2,122 tokens Grok-3 mini (high): Average accuracy 42.6%, output tokens per response 4,919 tokens GPT-4.1: Average accuracy 39%, output tokens per response 1,723 tokens GPT-4.1 mini: Average accuracy 37.6%, output tokens per response 1,532 tokens GPT-4.1 nano: Average accuracy 19%, output tokens per response 945 tokens o3 (high): Average accuracy 52.4%, output tokens per response 4,622 tokens o4-mini (high): Average accuracy 52.6%, output tokens per response 7,922 tokens Qwen3-235B-A22B (thinking): Average accuracy 37.3%, output tokens per response 5,575 tokens Qwen3-30B-A3B (thinking): Average accuracy 29.8%, output tokens per response 4,614 tokens Claude Opus 4 (thinking 16k): Average accuracy 43.7%, output tokens per response 3,136 tokens Claude Sonnet 4 (thinking 16k): Average accuracy 46.1%, output tokens per response 3,147 tokens Claude Sonnet 4 (no thinking): Average accuracy 43.9%, output tokens per response 2,693 tokens DeepSeek-R1: Average accuracy 41.6%, output tokens per response 5,243 tokens deepseek-r1-0528-qwen3-8b: Average accuracy 13%, output tokens per response 5,756 tokens Gemini 2.5 Pro (thinking 16k): Average accuracy 54%, output tokens per response 7,312 tokens o3-pro (high): Average accuracy 58.2%, output tokens per response 5,243 tokens Gemini 2.5 Flash (thinking 16k): Average accuracy 41%, output tokens per response 7,457 tokens Gemini 2.5 Flash-Lite (thinking 16k): Average accuracy 35.2%, output tokens per response 12K tokens Grok 4: Average accuracy 45.7%, output tokens per response 5,154 tokens Kimi K2: Average accuracy 39.4%, output tokens per response 1,710 tokens Qwen3-235B-A22B-Instruct: Average accuracy 38.7%, output tokens per response 1,961 tokens Qwen3-235B-A22B-Thinking: Average accuracy 41%, output tokens per response 5,771 tokens Qwen3-Coder-480B-A35B: Average accuracy 41.2%, output tokens per response 1,730 tokens GLM-4.5 (thinking): Average accuracy 40.6%, output tokens per response 4,988 tokens Claude Opus 4.1 (thinking 16k): Average accuracy 45.9%, output tokens per response 3,180 tokens gpt-oss-120b (high): Average accuracy 48.2%, output tokens per response 9,314 tokens gpt-oss-120b (medium): Average accuracy 41.9%, output tokens per response 2,346 tokens gpt-oss-20b (high): Average accuracy 40.9%, output tokens per response 9,131 tokens gpt-oss-20b (medium): Average accuracy 36.8%, output tokens per response 2,593 tokens GPT-5 (high): Average accuracy 60.7%, output tokens per response 11K tokens GPT-5 mini (high): Average accuracy 52.7%, output tokens per response 10K tokens GPT-5 nano (high): Average accuracy 38.1%, output tokens per response 12K tokens GPT-5 nano (low): Average accuracy 25.9%, output tokens per response 1,673 tokens Mistral Medium 3.1: Average accuracy 33.1%, output tokens per response 2,428 tokens DeepSeek-V3.1 (thinking): Average accuracy 38.4%, output tokens per response 4,545 tokens DeepSeek-V3.1: Average accuracy 37.5%, output tokens per response 2,058 tokens grok-code-fast-1: Average accuracy 35.1%, output tokens per response 2,850 tokens GPT-5-Codex (high): Average accuracy 54.5%, output tokens per response 21K tokens Grok 4 Fast: Average accuracy 42.9%, output tokens per response 4,009 tokens Gemini 2.5 Flash (thinking 16k): Average accuracy 41.9%, output tokens per response 5,190 tokens Claude Sonnet 4.5 (thinking 16k): Average accuracy 47.7%, output tokens per response 3,260 tokens Claude Sonnet 4.5 (no thinking): Average accuracy 46.7%, output tokens per response 2,406 tokens DeepSeek-V3.2-Exp (thinking): Average accuracy 39.5%, output tokens per response 4,236 tokens GPT-5 Pro (high): Average accuracy 60.4%, output tokens per response 11K tokens Claude Haiku 4.5 (no thinking): Average accuracy 45.4%, output tokens per response 2,286 tokens Claude Haiku 4.5 (thinking 16k): Average accuracy 44.1%, output tokens per response 3,705 tokens Kimi K2 Thinking: Average accuracy 42.8%, output tokens per response 5,090 tokens GPT-5.1 (high): Average accuracy 60.8%, output tokens per response 13K tokens Gemini 3 Pro (high): Average accuracy 69.9%, output tokens per response 7,795 tokens Claude Opus 4.5 (high, 16k): Average accuracy 63.7%, output tokens per response 3,684 tokens DeepSeek-V3.2-Speciale: Average accuracy 46.7%, output tokens per response 12K tokens GPT-5.2 (xhigh): Average accuracy 72.2%, output tokens per response 28K tokens GPT-5.2 (medium): Average accuracy 63.4%, output tokens per response 4,928 tokens Gemini 3 Flash (high): Average accuracy 61.6%, output tokens per response 14K tokens Kimi K2.5: Average accuracy 45.6%, output tokens per response 2,892 tokens Qwen3-Coder-Next: Average accuracy 34.4%, output tokens per response 2,279 tokens Claude Opus 4.6 (high): Average accuracy 78%, output tokens per response 32K tokens Claude Opus 4.6 (no thinking): Average accuracy 65.9%, output tokens per response 3,877 tokens GPT-5.3 Codex (xhigh): Average accuracy 77.9%, output tokens per response 34K tokens GLM-5 (thinking): Average accuracy 48.2%, output tokens per response 3,326 tokens Claude Sonnet 4.6 (medium): Average accuracy 66.1%, output tokens per response 28K tokens Gemini 3.1 Pro (high): Average accuracy 72.1%, output tokens per response 21K tokens Mercury 2 (high): Average accuracy 43.2%, output tokens per response 6,449 tokens Gemini 3.1 Flash-Lite (high): Average accuracy 52.2%, output tokens per response 8,128 tokens GPT-5.4 (xhigh): Average accuracy 77.7%, output tokens per response 72K tokens GPT-5.4 (no thinking): Average accuracy 57.4%, output tokens per response 2,780 tokens GPT-5.4 Mini (high): Average accuracy 60.3%, output tokens per response 55K tokens GPT-5.4 Mini (no thinking): Average accuracy 37.9%, output tokens per response 2,133 tokens GPT-5.4 Nano (high): Average accuracy 49.2%, output tokens per response 6,445 tokens GPT-5.4 Nano (no thinking): Average accuracy 38%, output tokens per response 2,155 tokens MiniMax-M2.7: Average accuracy 37%, output tokens per response 4,305 tokens GLM-5.1: Average accuracy 57.1%, output tokens per response 12K tokens Claude Opus 4.7 (high): Average accuracy 76.4%, output tokens per response 9,965 tokens Claude Opus 4.7 (no thinking): Average accuracy 76.4%, output tokens per response 3,594 tokens Claude Opus 4.7 (max): Average accuracy 75.5%, output tokens per response 44K tokens Kimi K2.6: Average accuracy 55.9%, output tokens per response 17K tokens GPT-5.5 (xhigh): Average accuracy 84.9%, output tokens per response 15K tokens GPT-5.5 (high): Average accuracy 83.9%, output tokens per response 13K tokens GPT-5.5 (no thinking): Average accuracy 67.2%, output tokens per response 2,493 tokens DeepSeek-V4-Flash (max): Average accuracy 45.6%, output tokens per response 7,978 tokens DeepSeek-V4-Flash (high): Average accuracy 43.8%, output tokens per response 3,570 tokens DeepSeek-V4-Pro (max): Average accuracy 48.6%, output tokens per response 17K tokens DeepSeek-V4-Pro (high): Average accuracy 46.5%, output tokens per response 5,183 tokens Grok 4.3: Average accuracy 49.9%, output tokens per response 3,525 tokens Gemini 3.5 Flash (high): Average accuracy 62.6%, output tokens per response 13K tokens Claude Opus 4.8 (xhigh): Average accuracy 82.9%, output tokens per response 13K tokens Claude Opus 4.8 (medium): Average accuracy 76%, output tokens per response 4,332 tokens Nemotron 3 Ultra: Average accuracy 43.5%, output tokens per response 4,368 tokens Claude Fable 5 (max): Average accuracy 92%, output tokens per response 22K tokens Claude Fable 5 (high): Average accuracy 87.8%, output tokens per response 8,393 tokens Kimi K2.7 Code: Average accuracy 54.1%, output tokens per response 9,342 tokens GLM-5.2 (max): Average accuracy 70.1%, output tokens per response 22K tokens GLM-5.2 (high): Average accuracy 67.3%, output tokens per response 12K tokens Claude Sonnet 5 (high): Average accuracy 69%, output tokens per response 12K tokens GPT-5.6 Luna (high): Average accuracy 60.9%, output tokens per response 5,867 tokens GPT-5.6 Sol (max): Average accuracy 89.4%, output tokens per response 20K tokens GPT-5.6 Sol (max): Average accuracy 87%, output tokens per response 11K tokens Inkling (high): Average accuracy 32.3%, output tokens per response 4,300 tokens Kimi K3 (max): Average accuracy 82.6%, output tokens per response 16K tokens Gemini 3.5 Flash-Lite (high): Average accuracy 39%, output tokens per response 3,366 tokens Gemini 3.6 Flash (high): Average accuracy 56.1%, output tokens per response 15K tokens Claude Opus 5 (max): Average accuracy 91.8%, output tokens per response 25K tokens Claude Opus 5 (high): Average accuracy 91.6%, output tokens per response 12K tokens Claude Opus 5 (no thinking): Average accuracy 86.3%, output tokens per response 7,394 tokens DeepSeek V4 Flash 0731 (max): Average accuracy 63%, output tokens per response 18K tokens DeepSeek V4 Flash 0731 (high): Average accuracy 57%, output tokens per response 16K tokens Muse Spark 1.2 (xhigh): Average accuracy 60.4%, output tokens per response 9,948 tokens Grok 4.6 (high): Average accuracy 67.3%, output tokens per response 7,558 tokens Qwen3.8-2.4T-A95B (xhigh): Average accuracy 76.5%, output tokens per response 32K tokens DeepSeek V4 Pro 0813 (max): Average accuracy 66.2%, output tokens per response 18K tokens GLM-5.3 (max): Average accuracy 75.4%, output tokens per response 17K tokens Claude Fable 5.1 (max): Average accuracy 92.9%, output tokens per response 32K tokens Claude Fable 5.1 (high): Average accuracy 92.3%, output tokens per response 8,369 tokens Gemini 3.8 Flash (high): Average accuracy 84.8%, output tokens per response 25K tokens gemma4:31b-it-q4_K_M: Average accuracy 52.3%, output tokens per response 4,921 tokens. Not in our model list x-ai/grok-4.20-beta: Average accuracy 52.3%, output tokens per response 2,945 tokens. Not in our model list mistral-medium-3.5:128b-q4_K_M: Average accuracy 43.7%, output tokens per response 6,593 tokens. Not in our model list qwen3.5:27b-q4_K_M: Average accuracy 39.5%, output tokens per response 2,664 tokens. Not in our model list nemotron-3-super:120b-a12b-q4_K_M: Average accuracy 38%, output tokens per response 3,543 tokens. Not in our model list moonshotai/kimi-k2-0905: Average accuracy 36.7%, output tokens per response 1,953 tokens. Not in our model list deepseek/deepseek-chat-v3-0324: Average accuracy 36.1%, output tokens per response 1,548 tokens. Not in our model list gemma4:26b-a4b-it-q4_K_M: Average accuracy 35.2%, output tokens per response 8,239 tokens. Not in our model list qwen3.6:35b-a3b-q4_K_M: Average accuracy 34.5%, output tokens per response 11K tokens. Not in our model list google/gemini-2.0-flash-001: Average accuracy 25.8%, output tokens per response 1,524 tokens. Not in our model list x-ai/grok-2-1212: Average accuracy 22.2%, output tokens per response 1,337 tokens. Not in our model list google/gemini-pro-1.5: Average accuracy 22.2%, output tokens per response 1,010 tokens. Not in our model list meta-llama/llama-3.1-405b-instruct: Average accuracy 21.4%, output tokens per response 1,008 tokens. Not in our model list qwen2:72b-instruct-q8_0: Average accuracy 11.3%, output tokens per response 1,257 tokens. Not in our model list meta-llama/llama-3.1-70b-instruct: Average accuracy 8.97%, output tokens per response 911 tokens. Not in our model list mistralai/mixtral-8x22b-instruct: Average accuracy 3.17%, output tokens per response 2,523 tokens. Not in our model list meta-llama/llama-3.1-8b-instruct: Average accuracy 1.73%, output tokens per response 1,052 tokens. Not in our model list GPT-4: Average accuracy 12.4%, output tokens per response 728 tokens Claude 2.1: Average accuracy 7.06%, output tokens per response 480 tokens GPT-4.5: Average accuracy 39.4%, output tokens per response 979 tokens GPT-5 Chat: Average accuracy 39.8%, output tokens per response 1,437 tokens GPT-5.2 (no thinking): Average accuracy 49.6%, output tokens per response 2,243 tokens Claude Opus 4.8 (no thinking): Average accuracy 70.5%, output tokens per response 2,414 tokens Grok 4.5: Average accuracy 46.4%, output tokens per response 1,723 tokens GPT-5.6 Sol (high): Average accuracy 88.8%, output tokens per response 7,245 tokens GPT-5.6 Terra (high): Average accuracy 78.3%, output tokens per response 3,479 tokens GPT-6 Astra (max): Average accuracy 93.3%, output tokens per response 22K tokens GPT-6 Astra (high): Average accuracy 92.9%, output tokens per response 8,056 tokens GPT-6 Astra Pro (max): Average accuracy 93.6%, output tokens per response 64K tokens google/gemini-flash-1.5: Average accuracy 24.9%, output tokens per response 833 tokens. Not in our model list openai/gpt-3.5-turbo-16k: Average accuracy 3.48%, output tokens per response 393 tokens. Not in our model list GPT-6 Astra Pro (max)GPT-6 Astra (max)GPT-6 Astra (high)GPT-5.6 Sol (high)GPT-5.6 Terra (high)Claude Opus 4.8 (no thinking)GPT-5.2 (no thinking)Grok 4.5GPT-5 ChatGPT-4.5google/gemini-flash-1.5GPT-4Claude 2.1openai/gpt-3.5-turbo-16k
ModelSettingAverage accuracyOutput tokens per response
GPT-6 Astra Pro max 93.6% 64,161 tokens
GPT-6 Astra max 93.3% 21,941 tokens
Claude Fable 5.1 max 92.9% 31,644 tokens
GPT-6 Astra high 92.9% 8,056 tokens
Claude Fable 5.1 high 92.3% 8,369 tokens
Claude Fable 5 max 92% 22,153 tokens
Claude Opus 5 max 91.8% 25,400 tokens
Claude Opus 5 high 91.6% 11,841 tokens
GPT-5.6 Sol max 89.4% 20,241 tokens
GPT-5.6 Sol high 88.8% 7,245 tokens
Claude Fable 5 high 87.9% 8,393 tokens
GPT-5.6 Sol max 87% 10,828 tokens
Claude Opus 5 no thinking 86.3% 7,394 tokens
GPT-5.5 xhigh 84.9% 15,144 tokens
Gemini 3.8 Flash high 84.8% 24,698 tokens
GPT-5.5 high 83.9% 12,659 tokens
Claude Opus 4.8 xhigh 82.9% 12,515 tokens
Kimi K3 max 82.6% 16,306 tokens
GPT-5.6 Terra high 78.3% 3,479 tokens
Claude Opus 4.6 high 78% 31,733 tokens
GPT-5.3 Codex xhigh 77.9% 33,792 tokens
GPT-5.4 xhigh 77.7% 71,878 tokens
Qwen3.8-2.4T-A95B xhigh 76.5% 31,909 tokens
Claude Opus 4.7 high 76.4% 9,965 tokens
Claude Opus 4.7 no thinking 76.4% 3,594 tokens
Claude Opus 4.8 medium 76% 4,332 tokens
Claude Opus 4.7 max 75.5% 43,546 tokens
GLM-5.3 max 75.4% 16,943 tokens
GPT-5.2 xhigh 72.2% 27,867 tokens
Gemini 3.1 Pro high 72.1% 21,437 tokens
Claude Opus 4.8 no thinking 70.5% 2,414 tokens
GLM-5.2 max 70.1% 21,907 tokens
Gemini 3 Pro high 69.9% 7,795 tokens
Claude Sonnet 5 high 69% 11,634 tokens
GLM-5.2 high 67.3% 11,795 tokens
Grok 4.6 high 67.3% 7,558 tokens
GPT-5.5 no thinking 67.2% 2,493 tokens
DeepSeek V4 Pro 0813 max 66.2% 18,019 tokens
Claude Sonnet 4.6 medium 66.1% 27,707 tokens
Claude Opus 4.6 no thinking 65.9% 3,877 tokens
Claude Opus 4.5 high, 16k 63.7% 3,684 tokens
GPT-5.2 medium 63.4% 4,928 tokens
DeepSeek V4 Flash 0731 max 63% 18,083 tokens
Gemini 3.5 Flash high 62.6% 13,380 tokens
Gemini 3 Flash high 61.6% 13,959 tokens
GPT-5.6 Luna high 60.9% 5,867 tokens
GPT-5.1 high 60.8% 13,207 tokens
GPT-5 high 60.7% 10,675 tokens
GPT-5 Pro high 60.4% 11,491 tokens
Muse Spark 1.2 xhigh 60.4% 9,948 tokens
GPT-5.4 Mini high 60.3% 54,770 tokens
o3-pro high 58.2% 5,243 tokens
GPT-5.4 no thinking 57.4% 2,780 tokens
GLM-5.1 57.1% 12,380 tokens
DeepSeek V4 Flash 0731 high 57.1% 15,646 tokens
Gemini 3.6 Flash high 56.1% 14,696 tokens
Kimi K2.6 55.9% 16,808 tokens
GPT-5-Codex high 54.5% 21,302 tokens
Kimi K2.7 Code 54.1% 9,342 tokens
Gemini 2.5 Pro Jun 2025 thinking 16k 54% 7,312 tokens
GPT-5 mini high 52.7% 10,290 tokens
o4-mini high 52.6% 7,922 tokens
o3 high 52.4% 4,622 tokens
x-ai/grok-4.20-beta 52.3% 2,945 tokens
gemma4:31b-it-q4_K_M 52.3% 4,921 tokens
Gemini 3.1 Flash-Lite high 52.2% 8,128 tokens
Grok 4.3 49.9% 3,525 tokens
GPT-5.2 no thinking 49.6% 2,243 tokens
GPT-5.4 Nano high 49.2% 6,445 tokens
DeepSeek-V4-Pro max 48.6% 16,877 tokens
GLM-5 thinking 48.2% 3,326 tokens
gpt-oss-120b high 48.2% 9,314 tokens
Claude Sonnet 4.5 thinking 16k 47.7% 3,260 tokens
o1-preview 47.6% 5,129 tokens
DeepSeek-V3.2-Speciale 46.7% 11,930 tokens
Claude Sonnet 4.5 no thinking 46.7% 2,406 tokens
DeepSeek-V4-Pro high 46.5% 5,183 tokens
Grok 4.5 46.4% 1,723 tokens
Claude Sonnet 4 thinking 16k 46.1% 3,147 tokens
o1 high 46.1% 5,767 tokens
Claude Opus 4.1 thinking 16k 45.9% 3,180 tokens
Grok 4 45.7% 5,154 tokens
DeepSeek-V4-Flash max 45.6% 7,978 tokens
Kimi K2.5 45.6% 2,892 tokens
Claude Haiku 4.5 no thinking 45.4% 2,286 tokens
Claude Haiku 4.5 thinking 16k 44.2% 3,705 tokens
Claude Sonnet 4 no thinking 43.9% 2,693 tokens
DeepSeek-V4-Flash high 43.8% 3,570 tokens
o3-mini high 43.8% 6,930 tokens
Claude Opus 4 thinking 16k 43.7% 3,136 tokens
mistral-medium-3.5:128b-q4_K_M 43.7% 6,593 tokens
Nemotron 3 Ultra 43.5% 4,368 tokens
Mercury 2 high 43.2% 6,449 tokens
Grok 4 Fast 42.9% 4,009 tokens
Kimi K2 Thinking 42.8% 5,090 tokens
Grok-3 mini high 42.6% 4,919 tokens
Gemini 2.5 Flash Sep 2025 thinking 16k 41.9% 5,190 tokens
gpt-oss-120b medium 41.9% 2,346 tokens
DeepSeek-R1 May 2025 41.6% 5,243 tokens
Qwen3-Coder-480B-A35B 41.2% 1,730 tokens
Qwen3-235B-A22B-Thinking Jul 2025 41% 5,771 tokens
Gemini 2.5 Flash Jun 2025 thinking 16k 41% 7,457 tokens
gpt-oss-20b high 40.9% 9,131 tokens
GLM-4.5 thinking 40.6% 4,988 tokens
Claude 3.5 Sonnet 40% 2,121 tokens
GPT-5 Chat 39.8% 1,437 tokens
qwen3.5:27b-q4_K_M 39.5% 2,664 tokens
DeepSeek-V3.2-Exp thinking 39.5% 4,236 tokens
GPT-4.5 39.4% 979.08 tokens
Kimi K2 Jul 2025 39.4% 1,710 tokens
GPT-4.1 39% 1,723 tokens
Gemini 3.5 Flash-Lite high 39% 3,366 tokens
Qwen3-235B-A22B-Instruct Jul 2025 38.7% 1,961 tokens
DeepSeek-V3.1 thinking 38.4% 4,545 tokens
GPT-5 nano high 38.1% 12,377 tokens
nemotron-3-super:120b-a12b-q4_K_M 38% 3,543 tokens
GPT-5.4 Nano no thinking 38% 2,155 tokens
GPT-5.4 Mini no thinking 37.9% 2,133 tokens
GPT-4.1 mini 37.6% 1,532 tokens
DeepSeek-V3.1 37.5% 2,058 tokens
Qwen3-235B-A22B thinking 37.3% 5,575 tokens
Grok 3 37.2% 2,122 tokens
MiniMax-M2.7 37% 4,305 tokens
gpt-oss-20b medium 36.8% 2,593 tokens
moonshotai/kimi-k2-0905 36.7% 1,953 tokens
DeepSeek-R1 36.5% 5,170 tokens
o1-mini 36.3% 3,877 tokens
deepseek/deepseek-chat-v3-0324 36.1% 1,548 tokens
Gemini 2.5 Flash-Lite Jun 2025 thinking 16k 35.2% 12,204 tokens
gemma4:26b-a4b-it-q4_K_M 35.2% 8,239 tokens
grok-code-fast-1 35.1% 2,850 tokens
qwen3.6:35b-a3b-q4_K_M 34.5% 10,729 tokens
Qwen3-Coder-Next 34.4% 2,279 tokens
Mistral Medium 3.1 33.1% 2,428 tokens
Inkling high 32.3% 4,300 tokens
Claude 3.5 Sonnet 31% 1,966 tokens
Claude 3.5 Haiku 30.7% 1,764 tokens
Qwen3-30B-A3B thinking 29.8% 4,614 tokens
GPT-5 nano low 25.9% 1,673 tokens
google/gemini-2.0-flash-001 25.8% 1,524 tokens
GPT-4o Nov 2024 25.1% 1,135 tokens
google/gemini-flash-1.5 24.9% 832.55 tokens
Llama 4 Maverick 24.5% 991.82 tokens
x-ai/grok-2-1212 22.2% 1,337 tokens
google/gemini-pro-1.5 22.2% 1,010 tokens
meta-llama/llama-3.1-405b-instruct 21.4% 1,008 tokens
Claude 3 Opus 19.2% 1,326 tokens
GPT-4.1 nano 19% 944.76 tokens
GPT-4 Turbo Apr 2024 18% 914.73 tokens
Qwen2.5-72B 16% 1,380 tokens
Llama 3.3 70B Instruct 14.4% 1,109 tokens
deepseek-r1-0528-qwen3-8b 13% 5,756 tokens
GPT-4 Jun 2023 12.4% 728.3 tokens
GPT-4o mini 11.8% 1,112 tokens
qwen2:72b-instruct-q8_0 11.3% 1,257 tokens
Claude 3 Sonnet 10.2% 1,389 tokens
Claude 3 Haiku 9.8% 1,275 tokens
meta-llama/llama-3.1-70b-instruct 9% 911.09 tokens
Claude 2.1 7.1% 479.89 tokens
openai/gpt-3.5-turbo-16k 3.5% 392.55 tokens
mistralai/mixtral-8x22b-instruct 3.2% 2,523 tokens
meta-llama/llama-3.1-8b-instruct 1.7% 1,052 tokens

The best score, 93.6%, costs $29 a task

Runs on the cost frontier score higher than any run that's cheaper. GPT-5.6 Sol at high comes within 4.8 points of the best for $1.25 a task, 23 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Meta OpenAI Other labs DeepSeek Anthropic Google DeepMind Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra Pro, max GPT-6 Astra, high GPT-5.6 Sol, high GPT-5.6 Terra, high DeepSeek V4 Pro 0813, max DeepSeek V4 Flash 0731, max DeepSeek V4 Flash 0731, high 0% 20% 40% 60% 80% 100% $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra Pro, max Gemini 3.8 Flash, high

Source: Official WeirdML leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-6 Astra Pro max 93.6% $29
GPT-6 Astra max 93.3% $7.75
Claude Fable 5.1 max 92.9% $12
GPT-6 Astra high 92.9% $2.98
Claude Fable 5.1 high 92.3% $3.20
Claude Fable 5 max 92% $8.42
Claude Opus 5 max 91.8% $4.74
Claude Opus 5 high 91.6% $2.19
GPT-5.6 Sol max 89.4% $3.87
GPT-5.6 Sol high 88.8% $1.25
Claude Fable 5 high 87.8% $3.11
GPT-5.6 Sol max 87% $2.31
Claude Opus 5 no thinking 86.3% $1.42
GPT-5.5 xhigh 84.9% $2.57
Gemini 3.8 Flash high 84.8% $0.51
GPT-5.5 high 83.9% $2.22
Claude Opus 4.8 xhigh 82.9% $2.35
Kimi K3 max 82.6% $2.05
GPT-5.6 Terra high 78.3% $0.25
Claude Opus 4.6 high 78% $5.78
GPT-5.3 Codex xhigh 77.9% $2.45
GPT-5.4 xhigh 77.7% $5.72
Qwen3.8-2.4T-A95B xhigh 76.5% $1.42
Claude Opus 4.7 high 76.4% $1.82
Claude Opus 4.7 no thinking 76.4% $0.69
Claude Opus 4.8 medium 76% $0.85
Claude Opus 4.7 max 75.5% $7.99
GLM-5.3 max 75.4% $0.54
GPT-5.2 xhigh 72.2% $2.05
Gemini 3.1 Pro high 72.1% $1.36
Claude Opus 4.8 no thinking 70.5% $0.48
GLM-5.2 max 70.1% $0.51
Gemini 3 Pro high 69.9% $0.53
Claude Sonnet 5 high 69% $0.90
GLM-5.2 high 67.3% $0.28
Grok 4.6 high 67.3% $0.33
GPT-5.5 no thinking 67.2% $0.54
DeepSeek V4 Pro 0813 max 66.2% $0.19
Claude Sonnet 4.6 medium 66.1% $2.86
Claude Opus 4.6 no thinking 65.9% $0.74
Claude Opus 4.5 high, 16k 63.7% $0.74
GPT-5.2 medium 63.4% $0.42
DeepSeek V4 Flash 0731 max 63% $0.03
Gemini 3.5 Flash high 62.6% $0.76
Gemini 3 Flash high 61.6% $0.22
GPT-5.6 Luna high 60.9% $0.04
GPT-5.1 high 60.8% $0.69
GPT-5 high 60.7% $0.59
GPT-5 Pro high 60.4% $7.65
Muse Spark 1.2 xhigh 60.4% $0.38
GPT-5.4 Mini high 60.3% $1.29
o3-pro high 58.2% $2.54
GPT-5.4 no thinking 57.4% $0.27
GLM-5.1 57.1% $0.32
DeepSeek V4 Flash 0731 high 57% $0.03
Gemini 3.6 Flash high 56.1% $0.62
Kimi K2.6 55.9% $0.35
GPT-5-Codex high 54.5% $1.10
Kimi K2.7 Code 54.1% $0.23
Gemini 2.5 Pro Jun 2025 thinking 16k 54% $0.41
GPT-5 mini high 52.7% $0.11
o4-mini high 52.6% $0.20
o3 high 52.4% $0.23
Gemini 3.1 Flash-Lite high 52.2% $0.07
Grok 4.3 49.9% $0.09
GPT-5.2 no thinking 49.6% $0.21
GPT-5.4 Nano high 49.2% $0.05
DeepSeek-V4-Pro max 48.6% $0.09
gpt-oss-120b high 48.2% $0.03
GLM-5 thinking 48.2% $0.08
Claude Sonnet 4.5 thinking 16k 47.7% $0.33
o1-preview 47.6% $1.85
DeepSeek-V3.2-Speciale 46.7% $0.04
Claude Sonnet 4.5 no thinking 46.7% $0.28
DeepSeek-V4-Pro high 46.5% $0.03
Grok 4.5 46.4% $0.08
Claude Sonnet 4 thinking 16k 46.1% $0.34
o1 high 46.1% $2.03
Claude Opus 4.1 thinking 16k 45.9% $1.76
Grok 4 45.7% $0.45
DeepSeek-V4-Flash max 45.6% $0.01
Kimi K2.5 45.6% $0.05
Claude Haiku 4.5 no thinking 45.4% $0.09
Claude Haiku 4.5 thinking 16k 44.1% $0.12
Claude Sonnet 4 no thinking 43.9% $0.31
DeepSeek-V4-Flash high 43.8% $0.007
o3-mini high 43.8% $0.17
Claude Opus 4 thinking 16k 43.7% $1.74
Nemotron 3 Ultra 43.5% $0.08
Mercury 2 high 43.2% $0.03
Grok 4 Fast 42.9% $0.01
Kimi K2 Thinking 42.8% $0.08
Grok-3 mini high 42.6% $0.02
Gemini 2.5 Flash Sep 2025 thinking 16k 41.9% $0.07
gpt-oss-120b medium 41.9% $0.01
DeepSeek-R1 May 2025 41.6% $0.07
Qwen3-Coder-480B-A35B 41.2% $0.07
Qwen3-235B-A22B-Thinking Jul 2025 41% $0.02
Gemini 2.5 Flash Jun 2025 thinking 16k 41% $0.11
gpt-oss-20b high 40.9% $0.01
GLM-4.5 thinking 40.6% $0.07
Claude 3.5 Sonnet 40% $0.25
GPT-5 Chat 39.8% $0.08
DeepSeek-V3.2-Exp thinking 39.5% $0.02
GPT-4.5 39.4% $1.57
Kimi K2 Jul 2025 39.4% $0.03
GPT-4.1 39% $0.10
Gemini 3.5 Flash-Lite high 39% $0.05
Qwen3-235B-A22B-Instruct Jul 2025 38.7% $0.01
DeepSeek-V3.1 thinking 38.4% $0.03
GPT-5 nano high 38.1% $0.03
GPT-5.4 Nano no thinking 38% $0.02
GPT-5.4 Mini no thinking 37.9% $0.07
GPT-4.1 mini 37.6% $0.02
DeepSeek-V3.1 37.5% $0.02
Qwen3-235B-A22B thinking 37.3% $0.02
Grok 3 37.2% $0.25
MiniMax-M2.7 37% $0.03
gpt-oss-20b medium 36.8% $0.004
DeepSeek-R1 36.5% $0.08
o1-mini 36.3% $0.11
Gemini 2.5 Flash-Lite Jun 2025 thinking 16k 35.2% $0.03
grok-code-fast-1 35.1% $0.02
Qwen3-Coder-Next 34.4% $0.03
Mistral Medium 3.1 33.1% $0.04
Inkling high 32.3% $0.13
Claude 3.5 Sonnet 31% $0.25
Claude 3.5 Haiku 30.7% $0.06
Qwen3-30B-A3B thinking 29.8% $0.008
GPT-5 nano low 25.9% $0.004
GPT-4o Nov 2024 25.1% $0.08
Llama 4 Maverick 24.5% $0.006
Claude 3 Opus 19.2% $0.86
GPT-4.1 nano 19% $0.003
GPT-4 Turbo Apr 2024 18% $0.32
Qwen2.5-72B 16% $0.01
Llama 3.3 70B Instruct 14.4% $0.001
deepseek-r1-0528-qwen3-8b 13% $0.001
GPT-4 Jun 2023 12.4% $0.64
GPT-4o mini 11.8% $0.005
Claude 3 Sonnet 10.2% $0.18
Claude 3 Haiku 9.84% $0.01
Claude 2.1 7.06% $0.17

12 models have held number one on the official leaderboard since its release in June 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Google DeepMind Anthropic
Own leaderboard Passed 85% in Jun 2026, by Claude Fable 5
GPT-6 Astra Pro 93.6%

Source: Official WeirdML leaderboard

Claude Fable 5 on the official leaderboard passed 85% in June 2026

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) GPT-6 Astra Pro GPT-5.5 Claude Opus 4.6 GPT-5.2 GPT-5.1 Gemini 2.5 Pro o1-preview 0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) GPT-6 Astra Pro GPT-5.5

Source: Official WeirdML leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

OpenAI Google DeepMind Anthropic
WeirdML v3 15 leaders
Claude Fable 5.2 43%
WeirdML Passed 85% in Jun 2026, by Claude Fable 5
GPT-6 Astra Pro 93.6%
WeirdML v3 3 leaders
Claude Fable 5.2 43%

Source: Official leaderboards

Replaced by WeirdML v3

More in AI research

All 104 in AI research
  • PaperBench In use Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. Qwen 3.8 Max 93%
  • MLS-Bench In use Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Claude Fable 5.1 50.3%
  • PostTrainBench 1.2 In use CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Claude Fable 5.1 44.6%
  • MLE-bench In use Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. GPT-6 Astra 93.8%
  • KernelBench In use Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. MiniMax-M3 28.8%
  • PMPP-Hard In use Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours. Inkling 2 Small 494.3