Benchmark Almanac

ALE-Bench

GPT-6 Astra 3,164 leads since September 2026
Not in model cards

Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants.

Made by
Sakana AI and AtCoder
Released
June 2025
Kind
Competitive programming
Level
Competition
Scored
Checked automatically
Answers
Code
Tasks
All public
Strongest model tested
Claude Fable 5.2 #1 on the capability index
Last model added
November 2026
Scores refreshed
November 12th, 2026

In the authors’ words

How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons. Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations. Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements.

Abstract of ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering (arXiv:2506.09050, CC0)
139
Entries on its own leaderboard, 126 models we track

GPT-6 Astra leads with 3,164

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

139 entries. Scroll sideways for the rest.

  1. 3,164 GPT-6 Astra
  2. 2,990 GPT-6.1 Sol
  3. 2,986 Claude Fable 5.2
  4. 2,929 Gemini 4 Carbon
  5. 2,875 GPT-6.1 Terra
  6. 2,849 Claude Fable 5.1
  7. 2,805 Gemini 4 Argon
  8. 2,748 Claude Opus 5.5
  9. 2,637 GPT-6 Sol
  10. 2,622 Claude Opus 5
  11. 2,463 Claude Fable 5
  12. 2,448 Claude Sonnet 5.5
  13. 2,410 GPT-5.6 Sol
  14. 2,182 Gemini 3.8 Flash
  15. 2,154 GPT-5.6 Terra
  16. 2,132 Kimi K3.1 Pro
  17. 2,108 GPT-5.5
  18. 2,102 Claude Opus 4.8
  19. 2,041 Gemini 3.1 Pro
  20. 2,012 DeepSeek V4.1 Flash 1017
  21. 1,996 Claude Haiku 5.5
  22. 1,991 Kimi K3
  23. 1,981 Muse Spark 1.3
  24. 1,964 Qwen3.8 Max (0902)
  25. 1,937 GLM-5.3
  26. 1,934 GPT-5.3 Codex
  27. 1,897 Claude Opus 4.7
  28. 1,876 Gemini 3.7 Flash
  29. 1,830 Muse Spark Open
  30. 1,829 DeepSeek V4.1 Flash
  31. 1,829 GPT-5.6 Luna
  32. 1,825 Gemini 3.6 Flash
  33. 1,823 Grok 4.7
  34. 1,822 Grok 4.6
  35. 1,811 GPT-5.4
  36. 1,791 GPT-6 Luna
  37. 1,783 Claude Sonnet 5
  38. 1,768 Claude Sonnet 4.6
  39. 1,730 DeepSeek V4 Pro 0813
  40. 1,729 Gemini 3 Pro
  41. 1,722 Gemini 3 Flash
  42. 1,715 GPT-5.2
  43. 1,712 Gemini 3.5 Flash
  44. 1,695 Kimi K2.6
  45. 1,692 Mercury 3
  46. 1,686 DeepSeek V4 Flash 0731
  47. 1,686 GPT-5.2 Codex
  48. 1,685 GLM-5.2
  49. 1,657 Mistral Large 4
  50. 1,646 GLM-5.1
  51. 1,607 Claude Opus 4.6
  52. 1,548 qwen3.7
  53. 1,545 GPT-5
  54. 1,538 GPT-5.1-Codex-Max
  55. 1,528 GPT-5.1-Codex
  56. 1,516 Tencent Hy3
  57. 1,495 MiniMax-M3.1-Flash
  58. 1,493 Beam 1.0
  59. 1,489 DeepSeek-V4-Pro
  60. 1,487 GPT-5.4 Mini
  61. 1,476 MiMo-V2.6 Pro
  62. 1,467 Grok 4.5
  63. 1,455 MiMo-V2.6 Flash
  64. 1,429 Claude Opus 4.5
  65. 1,425 GPT-5.1
  66. 1,422 Kimi K2.7 Code
  67. 1,421 GPT-5 mini
  68. 1,394 Gemma 4 31B IT
  69. 1,388 grok-4.20-beta
  70. 1,380 DeepSeek-V4-Flash
  71. 1,352 Nemotron 3.5 Super VL
  72. 1,347 MiMo-V2-Pro
  73. 1,313 GLM-5-Turbo
  74. 1,312 GPT-5.4 Nano
  75. 1,304 o3
  76. 1,283 GLM-5.3-Flash
  77. 1,279 Kimi K2.5
  78. 1,277 Step 3.7 Flash
  79. 1,273 Qwen 3.6 Plus
  80. 1,271 o4-mini
  81. 1,256 Solar Pro 4
  82. 1,254 MiMo-V2.5-Pro
  83. 1,226 muse-glimmer-30b
  84. 1,212 LongCat-2.0
  85. 1,211 Grok 4.3
  86. 1,195 Inkling
  87. 1,194 GLM-5
  88. 1,192 Claude Sonnet 4.5
  89. 1,190 Mercury 2
  90. 1,183 Claude Opus 4.1
  91. 1,175 gemini-2.5-pro-thinking (snapshot unspecified)
  92. 1,165 Gemini 3.1 Flash-Lite
  93. 1,165 gemma-4-26b-a4b-it
  94. 1,137 Gemini 3.5 Flash-Lite
  95. 1,131 Mistral Medium 3.5
  96. 1,119 gpt-oss-120b
  97. 1,111 Grok 4.1 Fast
  98. 1,108 Ling-2.6-1T
  99. 1,106 MiMo-V2.5
  100. 1,103 Kimi K2 Thinking
  101. 1,088 Claude Sonnet 4
  102. 1,078 DeepSeek-V3.1-Terminus
  103. 1,071 Qwen3.5-Plus
  104. 1,059 DeepSeek-R1
  105. 1,053 MiniMax-M3
  106. 1,033 MiniMax-M2.5
  107. 1,022 gemini-2.5-flash-thinking (snapshot unspecified)
  108. 1,021 MiniMax-M2.7
  109. 1,017 Nemotron 3 Super
  110. 1,017 Nemotron 3.5 Lightning 30B-A3B
  111. 1,016 GPT-5 nano
  112. 1,015 Qwen 3.6 Flash
  113. 1,011 GLM-4.7
  114. 1,007 Qwen3.5 397B-A17B
  115. 1,006 Claude Haiku 4.5
  116. 1,000 MiMo-V2-Flash
  117. 998 MiniMax-M2.1
  118. 985 GPT-4.1
  119. 977 Mercury 2.5
  120. 946 Qwen3.7 Flash
  121. 942 gpt-oss-20b
  122. 894 qwen3
  123. 882 Qwen3.5-27B
  124. 862 Mistral Small 4
  125. 853 grok-code-fast-1
  126. 838 GLM-4.6
  127. 827 Qwen3-Coder-Plus
  128. 826 Qwen3.5-Flash
  129. 822 Qwen3-Coder-480B-A35B
  130. 776 kimi-k2-0905
  131. 721 mistral-large-3-2512
  132. 701 Mistral Medium 3.1
  133. 648 Qwen3.5-35B-A3B
  134. 552 GLM-4.5
  135. 552 Llama 4 Maverick
  136. 542 gemini-2.5-flash-lite-thinking (snapshot unspecified)
  137. 529 nova-premier-v1
  138. 507 nova-2-lite-v1
  139. 341 codestral-2508
  1. 1 GPT-6 Astra self-refine 8; max 3,164
  2. 2 GPT-6.1 Sol self-refine 16; max 2,990
  3. 3 Claude Fable 5.2 self-refine 16; high 2,986
  4. 4 Gemini 4 Carbon self-refine 8; high 2,929
  5. 5 GPT-6.1 Terra self-refine 16; max 2,875
  6. 6 Claude Fable 5.1 self-refine 16; high 2,849
  7. 7 Gemini 4 Argon self-refine 8; high 2,805
  8. 8 Claude Opus 5.5 self-refine 16; high 2,748
  9. 9 GPT-6 Sol self-refine 16; max 2,637
  10. 10 Claude Opus 5 self-refine 16; high 2,622
  11. 11 Claude Fable 5 self-refine 16; high 2,463
  12. 12 Claude Sonnet 5.5 self-refine 16; high 2,448
  13. 13 GPT-5.6 Sol self-refine 16; max 2,410
  14. 14 Gemini 3.8 Flash self-refine 16; high 2,182
  15. 15 GPT-5.6 Terra self-refine 16; max 2,154
  16. 16 Kimi K3.1 Pro self-refine 16; max 2,132
  17. 17 GPT-5.5 self-refine 16; xhigh 2,108
  18. 18 Claude Opus 4.8 self-refine 16; high 2,102
  19. 19 Gemini 3.1 Pro self-refine 16; high 2,041
  20. 20 DeepSeek V4.1 Flash 1017 self-refine 16; max 2,012
  21. 21 Claude Haiku 5.5 self-refine 16; high 1,996
  22. 22 Kimi K3 self-refine 16; max 1,991
  23. 23 Muse Spark 1.3 self-refine 16; xhigh 1,981
  24. 24 Qwen3.8 Max (0902) self-refine 16; xhigh 1,964
  25. 25 GLM-5.3 self-refine 16; high 1,937
  26. 26 GPT-5.3 Codex self-refine 16; xhigh 1,934
  27. 27 Claude Opus 4.7 self-refine 16; no-thinking 1,897
  28. 28 Gemini 3.7 Flash self-refine 16; high 1,876
  29. 29 Muse Spark Open self-refine 16; xhigh 1,830
  30. 30 DeepSeek V4.1 Flash self-refine 16; max 1,829
  31. 31 GPT-5.6 Luna self-refine 16; max 1,829
  32. 32 Gemini 3.6 Flash self-refine 16; high 1,825
  33. 33 Grok 4.7 self-refine 16; xhigh 1,823
  34. 34 Grok 4.6 self-refine 16; xhigh 1,822
  35. 35 GPT-5.4 self-refine 16; high 1,811
  36. 36 GPT-6 Luna self-refine 16; xhigh 1,791
  37. 37 Claude Sonnet 5 self-refine 16; high 1,783
  38. 38 Claude Sonnet 4.6 self-refine 16; medium 1,768
  39. 39 DeepSeek V4 Pro 0813 self-refine 16; max 1,730
  40. 40 Gemini 3 Pro self-refine 16; high 1,729
  41. 41 Gemini 3 Flash self-refine 16; high 1,722
  42. 42 GPT-5.2 self-refine 16; high 1,715
  43. 43 Gemini 3.5 Flash self-refine 16; high 1,712
  44. 44 Kimi K2.6 self-refine 16 1,695
  45. 45 Mercury 3 self-refine 16; high 1,692
  46. 46 DeepSeek V4 Flash 0731 self-refine 8; max 1,686
  47. 47 GPT-5.2 Codex self-refine 8; xhigh 1,686
  48. 48 GLM-5.2 self-refine 16; max 1,685
  49. 49 Mistral Large 4 self-refine 16 1,657
  50. 50 GLM-5.1 self-refine 16 1,646
  51. 51 Claude Opus 4.6 self-refine 16; no-thinking 1,607
  52. 52 qwen3.7 self-refine 8; max 1,548
  53. 53 GPT-5 self-refine 16 1,545
  54. 54 GPT-5.1-Codex-Max self-refine 16; xhigh 1,538
  55. 55 GPT-5.1-Codex self-refine 16; high 1,528
  56. 56 Tencent Hy3 self-refine 16; high 1,516
  57. 57 MiniMax-M3.1-Flash self-refine 16; max 1,495
  58. 58 Beam 1.0 self-refine 16; max 1,493
  59. 59 DeepSeek-V4-Pro self-refine 16; high 1,489
  60. 60 GPT-5.4 Mini self-refine 16; high 1,487
  61. 61 MiMo-V2.6 Pro self-refine 8 1,476
  62. 62 Grok 4.5 self-refine 8; high 1,467
  63. 63 MiMo-V2.6 Flash self-refine 16 1,455
  64. 64 Claude Opus 4.5 self-refine 16 1,429
  65. 65 GPT-5.1 self-refine 16 1,425
  66. 66 Kimi K2.7 Code self-refine 16 1,422
  67. 67 GPT-5 mini self-refine 16 1,421
  68. 68 Gemma 4 31B IT self-refine 16 1,394
  69. 69 grok-4.20-beta self-refine 16 1,388
  70. 70 DeepSeek-V4-Flash self-refine 16; high 1,380
  71. 71 Nemotron 3.5 Super VL self-refine 16 1,352
  72. 72 MiMo-V2-Pro self-refine 16 1,347
  73. 73 GLM-5-Turbo self-refine 16 1,313
  74. 74 GPT-5.4 Nano self-refine 16; high 1,312
  75. 75 o3 self-refine 16; high 1,304
  76. 76 GLM-5.3-Flash self-refine 16; high 1,283
  77. 77 Kimi K2.5 self-refine 16 1,279
  78. 78 Step 3.7 Flash self-refine 16; medium 1,277
  79. 79 Qwen 3.6 Plus self-refine 16 1,273
  80. 80 o4-mini self-refine 16; high 1,271
  81. 81 Solar Pro 4 self-refine 16; max 1,256
  82. 82 MiMo-V2.5-Pro self-refine 16 1,254
  83. 83 muse-glimmer-30b self-refine 16; high 1,226
  84. 84 LongCat-2.0 self-refine 8 1,212
  85. 85 Grok 4.3 self-refine 16 1,211
  86. 86 Inkling self-refine 16; max 1,195
  87. 87 GLM-5 self-refine 16 1,194
  88. 88 Claude Sonnet 4.5 self-refine 16 1,192
  89. 89 Mercury 2 self-refine 16 1,190
  90. 90 Claude Opus 4.1 self-refine 16 1,183
  91. 91 gemini-2.5-pro-thinking (snapshot unspecified) self-refine 16 1,175
  92. 92 Gemini 3.1 Flash-Lite self-refine 8; high 1,165
  93. 93 gemma-4-26b-a4b-it self-refine 16 1,165
  94. 94 Gemini 3.5 Flash-Lite self-refine 16; high 1,137
  95. 95 Mistral Medium 3.5 self-refine 16 1,131
  96. 96 gpt-oss-120b self-refine 16 1,119
  97. 97 Grok 4.1 Fast self-refine 16 1,111
  98. 98 Ling-2.6-1T self-refine 16 1,108
  99. 99 MiMo-V2.5 self-refine 16 1,106
  100. 100 Kimi K2 Thinking self-refine 16 1,103
  101. 101 Claude Sonnet 4 self-refine 16 1,088
  102. 102 DeepSeek-V3.1-Terminus self-refine 16 1,078
  103. 103 Qwen3.5-Plus self-refine 16 1,071
  104. 104 DeepSeek-R1 May 2025, self-refine 16 1,059
  105. 105 MiniMax-M3 self-refine 16 1,053
  106. 106 MiniMax-M2.5 self-refine 16 1,033
  107. 107 gemini-2.5-flash-thinking (snapshot unspecified) self-refine 16 1,022
  108. 108 MiniMax-M2.7 self-refine 16 1,021
  109. 109 Nemotron 3 Super self-refine 16 1,017
  110. 110 Nemotron 3.5 Lightning 30B-A3B self-refine 16 1,017
  111. 111 GPT-5 nano self-refine 16 1,016
  112. 112 Qwen 3.6 Flash self-refine 16 1,015
  113. 113 GLM-4.7 self-refine 16 1,011
  114. 114 Qwen3.5 397B-A17B self-refine 16 1,007
  115. 115 Claude Haiku 4.5 self-refine 16 1,006
  116. 116 MiMo-V2-Flash self-refine 16 1,000
  117. 117 MiniMax-M2.1 self-refine 16 998
  118. 118 GPT-4.1 self-refine 16 985
  119. 119 Mercury 2.5 self-refine 16; high 977
  120. 120 Qwen3.7 Flash self-refine 16 946
  121. 121 gpt-oss-20b self-refine 16 942
  122. 122 qwen3 self-refine 16; max 894
  123. 123 Qwen3.5-27B self-refine 16 882
  124. 124 Mistral Small 4 self-refine 16 862
  125. 125 grok-code-fast-1 self-refine 16 853
  126. 126 GLM-4.6 self-refine 16 838
  127. 127 Qwen3-Coder-Plus self-refine 16 827
  128. 128 Qwen3.5-Flash self-refine 16 826
  129. 129 Qwen3-Coder-480B-A35B self-refine 16 822
  130. 130 kimi-k2-0905 self-refine 16 776
  131. 131 mistral-large-3-2512 self-refine 16 721
  132. 132 Mistral Medium 3.1 self-refine 16 701
  133. 133 Qwen3.5-35B-A3B self-refine 16 648
  134. 134 GLM-4.5 self-refine 16 552
  135. 135 Llama 4 Maverick self-refine 16 552
  136. 136 gemini-2.5-flash-lite-thinking (snapshot unspecified) self-refine 16 542
  137. 137 nova-premier-v1 self-refine 16 529
  138. 138 nova-2-lite-v1 self-refine 16 507
  139. 139 codestral-2508 self-refine 16 341

Source: Official ALE-Bench leaderboard

#EntryScoreSettingDateSource
1 GPT-6 Astra 3,164 self-refine 8; max Sep 2026 Official leaderboard
2 GPT-6.1 Sol 2,990 self-refine 16; max Sep 2026 Official leaderboard
3 Claude Fable 5.2 2,986 self-refine 16; high Nov 2026 Official leaderboard
4 Gemini 4 Carbon 2,929 self-refine 8; high Oct 2026 Official leaderboard
5 GPT-6.1 Terra 2,875 self-refine 16; max Oct 2026 Official leaderboard
6 Claude Fable 5.1 2,849 self-refine 16; high Sep 2026 Official leaderboard
7 Gemini 4 Argon 2,805 self-refine 8; high Sep 2026 Official leaderboard
8 Claude Opus 5.5 2,748 self-refine 16; high Sep 2026 Official leaderboard
9 GPT-6 Sol 2,637 self-refine 16; max Sep 2026 Official leaderboard
10 Claude Opus 5 2,622 self-refine 16; high Jul 2026 Official leaderboard
11 Claude Fable 5 2,463 self-refine 16; high Jun 2026 Official leaderboard
12 Claude Sonnet 5.5 2,448 self-refine 16; high Sep 2026 Official leaderboard
13 GPT-5.6 Sol 2,410 self-refine 16; max Jul 2026 Official leaderboard
14 Gemini 3.8 Flash 2,182 self-refine 16; high Sep 2026 Official leaderboard
15 GPT-5.6 Terra 2,154 self-refine 16; max Jul 2026 Official leaderboard
16 Kimi K3.1 Pro 2,132 self-refine 16; max Oct 2026 Official leaderboard
17 GPT-5.5 2,108 self-refine 16; xhigh Apr 2026 Official leaderboard
18 Claude Opus 4.8 2,102 self-refine 16; high May 2026 Official leaderboard
19 Gemini 3.1 Pro 2,041 self-refine 16; high Feb 2026 Official leaderboard
20 DeepSeek V4.1 Flash 1017 2,012 self-refine 16; max Oct 2026 Official leaderboard
21 Claude Haiku 5.5 1,996 self-refine 16; high Oct 2026 Official leaderboard
22 Kimi K3 1,991 self-refine 16; max Jul 2026 Official leaderboard
23 Muse Spark 1.3 1,981 self-refine 16; xhigh Sep 2026 Official leaderboard
24 Qwen3.8 Max (0902) 1,964 self-refine 16; xhigh Sep 2026 Official leaderboard
25 GLM-5.3 1,937 self-refine 16; high Aug 2026 Official leaderboard
26 GPT-5.3 Codex 1,934 self-refine 16; xhigh Feb 2026 Official leaderboard
27 Claude Opus 4.7 1,897 self-refine 16; no-thinking Apr 2026 Official leaderboard
28 Gemini 3.7 Flash 1,876 self-refine 16; high Aug 2026 Official leaderboard
29 Muse Spark Open 1,830 self-refine 16; xhigh Oct 2026 Official leaderboard
30 DeepSeek V4.1 Flash 1,829 self-refine 16; max Sep 2026 Official leaderboard
31 GPT-5.6 Luna 1,829 self-refine 16; max Jul 2026 Official leaderboard
32 Gemini 3.6 Flash 1,825 self-refine 16; high Jul 2026 Official leaderboard
33 Grok 4.7 1,823 self-refine 16; xhigh Sep 2026 Official leaderboard
34 Grok 4.6 1,822 self-refine 16; xhigh Aug 2026 Official leaderboard
35 GPT-5.4 1,811 self-refine 16; high Mar 2026 Official leaderboard
36 GPT-6 Luna 1,791 self-refine 16; xhigh Sep 2026 Official leaderboard
37 Claude Sonnet 5 1,783 self-refine 16; high Jun 2026 Official leaderboard
38 Claude Sonnet 4.6 1,768 self-refine 16; medium Feb 2026 Official leaderboard
39 DeepSeek V4 Pro 0813 1,730 self-refine 16; max Aug 2026 Official leaderboard
40 Gemini 3 Pro 1,729 self-refine 16; high Nov 2025 Official leaderboard
41 Gemini 3 Flash 1,722 self-refine 16; high Dec 2025 Official leaderboard
42 GPT-5.2 1,715 self-refine 16; high Dec 2025 Official leaderboard
43 Gemini 3.5 Flash 1,712 self-refine 16; high May 2026 Official leaderboard
44 Kimi K2.6 1,695 self-refine 16 Apr 2026 Official leaderboard
45 Mercury 3 1,692 self-refine 16; high Oct 2026 Official leaderboard
46 DeepSeek V4 Flash 0731 1,686 self-refine 8; max Jul 2026 Official leaderboard
47 GPT-5.2 Codex 1,686 self-refine 8; xhigh Dec 2025 Official leaderboard
48 GLM-5.2 1,685 self-refine 16; max Jun 2026 Official leaderboard
49 Mistral Large 4 1,657 self-refine 16 Oct 2026 Official leaderboard
50 GLM-5.1 1,646 self-refine 16 Apr 2026 Official leaderboard
51 Claude Opus 4.6 1,607 self-refine 16; no-thinking Feb 2026 Official leaderboard
52 qwen3.7 1,548 self-refine 8; max Official leaderboard
53 GPT-5 1,545 self-refine 16 Aug 2025 Official leaderboard
54 GPT-5.1-Codex-Max 1,538 self-refine 16; xhigh Nov 2025 Official leaderboard
55 GPT-5.1-Codex 1,528 self-refine 16; high Nov 2025 Official leaderboard
56 Tencent Hy3 1,516 self-refine 16; high Jul 2026 Official leaderboard
57 MiniMax-M3.1-Flash 1,495 self-refine 16; max Oct 2026 Official leaderboard
58 Beam 1.0 1,493 self-refine 16; max Oct 2026 Official leaderboard
59 DeepSeek-V4-Pro 1,489 self-refine 16; high Apr 2026 Official leaderboard
60 GPT-5.4 Mini 1,487 self-refine 16; high Mar 2026 Official leaderboard
61 MiMo-V2.6 Pro 1,476 self-refine 8 Sep 2026 Official leaderboard
62 Grok 4.5 1,467 self-refine 8; high Jul 2026 Official leaderboard
63 MiMo-V2.6 Flash 1,455 self-refine 16 Sep 2026 Official leaderboard
64 Claude Opus 4.5 1,429 self-refine 16 Nov 2025 Official leaderboard
65 GPT-5.1 1,425 self-refine 16 Nov 2025 Official leaderboard
66 Kimi K2.7 Code 1,422 self-refine 16 Jun 2026 Official leaderboard
67 GPT-5 mini 1,421 self-refine 16 Aug 2025 Official leaderboard
68 Gemma 4 31B IT 1,394 self-refine 16 Apr 2026 Official leaderboard
69 grok-4.20-beta 1,388 self-refine 16 Official leaderboard
70 DeepSeek-V4-Flash 1,380 self-refine 16; high Apr 2026 Official leaderboard
71 Nemotron 3.5 Super VL 1,352 self-refine 16 Oct 2026 Official leaderboard
72 MiMo-V2-Pro 1,347 self-refine 16 Mar 2026 Official leaderboard
73 GLM-5-Turbo 1,313 self-refine 16 Mar 2026 Official leaderboard
74 GPT-5.4 Nano 1,312 self-refine 16; high Mar 2026 Official leaderboard
75 o3 1,304 self-refine 16; high Apr 2025 Official leaderboard
76 GLM-5.3-Flash 1,283 self-refine 16; high Aug 2026 Official leaderboard
77 Kimi K2.5 1,279 self-refine 16 Jan 2026 Official leaderboard
78 Step 3.7 Flash 1,277 self-refine 16; medium May 2026 Official leaderboard
79 Qwen 3.6 Plus 1,273 self-refine 16 Mar 2026 Official leaderboard
80 o4-mini 1,271 self-refine 16; high Apr 2025 Official leaderboard
81 Solar Pro 4 1,256 self-refine 16; max Aug 2026 Official leaderboard
82 MiMo-V2.5-Pro 1,254 self-refine 16 Apr 2026 Official leaderboard
83 muse-glimmer-30b 1,226 self-refine 16; high Official leaderboard
84 LongCat-2.0 1,212 self-refine 8 Jul 2026 Official leaderboard
85 Grok 4.3 1,211 self-refine 16 Apr 2026 Official leaderboard
86 Inkling 1,195 self-refine 16; max Jul 2026 Official leaderboard
87 GLM-5 1,194 self-refine 16 Feb 2026 Official leaderboard
88 Claude Sonnet 4.5 1,192 self-refine 16 Sep 2025 Official leaderboard
89 Mercury 2 1,190 self-refine 16 Feb 2026 Official leaderboard
90 Claude Opus 4.1 1,183 self-refine 16 Aug 2025 Official leaderboard
91 gemini-2.5-pro-thinking (snapshot unspecified) 1,175 self-refine 16 Official leaderboard
92 Gemini 3.1 Flash-Lite 1,165 self-refine 8; high Mar 2026 Official leaderboard
93 gemma-4-26b-a4b-it 1,165 self-refine 16 Official leaderboard
94 Gemini 3.5 Flash-Lite 1,137 self-refine 16; high Jul 2026 Official leaderboard
95 Mistral Medium 3.5 1,131 self-refine 16 Apr 2026 Official leaderboard
96 gpt-oss-120b 1,119 self-refine 16 Aug 2025 Official leaderboard
97 Grok 4.1 Fast 1,111 self-refine 16 Nov 2025 Official leaderboard
98 Ling-2.6-1T 1,108 self-refine 16 Apr 2026 Official leaderboard
99 MiMo-V2.5 1,106 self-refine 16 Apr 2026 Official leaderboard
100 Kimi K2 Thinking 1,103 self-refine 16 Nov 2025 Official leaderboard
101 Claude Sonnet 4 1,088 self-refine 16 May 2025 Official leaderboard
102 DeepSeek-V3.1-Terminus 1,078 self-refine 16 Sep 2025 Official leaderboard
103 Qwen3.5-Plus 1,071 self-refine 16 Feb 2026 Official leaderboard
104 DeepSeek-R1 May 2025 1,059 self-refine 16 May 2025 Official leaderboard
105 MiniMax-M3 1,053 self-refine 16 Jun 2026 Official leaderboard
106 MiniMax-M2.5 1,033 self-refine 16 Feb 2026 Official leaderboard
107 gemini-2.5-flash-thinking (snapshot unspecified) 1,022 self-refine 16 Official leaderboard
108 MiniMax-M2.7 1,021 self-refine 16 Mar 2026 Official leaderboard
109 Nemotron 3 Super 1,017 self-refine 16 Mar 2026 Official leaderboard
110 Nemotron 3.5 Lightning 30B-A3B 1,017 self-refine 16 Aug 2026 Official leaderboard
111 GPT-5 nano 1,016 self-refine 16 Aug 2025 Official leaderboard
112 Qwen 3.6 Flash 1,015 self-refine 16 Apr 2026 Official leaderboard
113 GLM-4.7 1,011 self-refine 16 Dec 2025 Official leaderboard
114 Qwen3.5 397B-A17B 1,007 self-refine 16 Feb 2026 Official leaderboard
115 Claude Haiku 4.5 1,006 self-refine 16 Oct 2025 Official leaderboard
116 MiMo-V2-Flash 1,000 self-refine 16 Dec 2025 Official leaderboard
117 MiniMax-M2.1 998 self-refine 16 Dec 2025 Official leaderboard
118 GPT-4.1 985 self-refine 16 Apr 2025 Official leaderboard
119 Mercury 2.5 977 self-refine 16; high Sep 2026 Official leaderboard
120 Qwen3.7 Flash 946 self-refine 16 Jul 2026 Official leaderboard
121 gpt-oss-20b 942 self-refine 16 Aug 2025 Official leaderboard
122 qwen3 894 self-refine 16; max Official leaderboard
123 Qwen3.5-27B 882 self-refine 16 Feb 2026 Official leaderboard
124 Mistral Small 4 862 self-refine 16 Mar 2026 Official leaderboard
125 grok-code-fast-1 853 self-refine 16 Aug 2025 Official leaderboard
126 GLM-4.6 838 self-refine 16 Sep 2025 Official leaderboard
127 Qwen3-Coder-Plus 827 self-refine 16 Sep 2025 Official leaderboard
128 Qwen3.5-Flash 826 self-refine 16 Feb 2026 Official leaderboard
129 Qwen3-Coder-480B-A35B 822 self-refine 16 Jul 2025 Official leaderboard
130 kimi-k2-0905 776 self-refine 16 Official leaderboard
131 mistral-large-3-2512 721 self-refine 16 Official leaderboard
132 Mistral Medium 3.1 701 self-refine 16 Aug 2025 Official leaderboard
133 Qwen3.5-35B-A3B 648 self-refine 16 Feb 2026 Official leaderboard
134 GLM-4.5 552 self-refine 16 Aug 2025 Official leaderboard
135 Llama 4 Maverick 552 self-refine 16 Apr 2025 Official leaderboard
136 gemini-2.5-flash-lite-thinking (snapshot unspecified) 542 self-refine 16 Official leaderboard
137 nova-premier-v1 529 self-refine 16 Official leaderboard
138 nova-2-lite-v1 507 self-refine 16 Official leaderboard
139 codestral-2508 341 self-refine 16 Official leaderboard

Performance versus cost

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0 1,000 2,000 3,000 4,000 $0.0001$0.001$0.01$0.1$1$10$100 Mean task cost ($), log scale, lower is better Mean performance (rating) Llama 4 Maverick (self-refine 1): Mean performance 173 rating, mean task cost $0.001 Llama 4 Maverick (self-refine 2): Mean performance 349 rating, mean task cost $0.004 Llama 4 Maverick (self-refine 4): Mean performance 499 rating, mean task cost $0.01 Llama 4 Maverick (self-refine 8): Mean performance 530 rating, mean task cost $0.03 Llama 4 Maverick (self-refine 16): Mean performance 552 rating, mean task cost $0.08 GPT-4.1 (self-refine 1): Mean performance 558 rating, mean task cost $0.03 GPT-4.1 (self-refine 2): Mean performance 679 rating, mean task cost $0.06 GPT-4.1 (self-refine 4): Mean performance 797 rating, mean task cost $0.11 GPT-4.1 (self-refine 8): Mean performance 915 rating, mean task cost $0.27 GPT-4.1 (self-refine 16): Mean performance 985 rating, mean task cost $0.76 o3 (self-refine 1; high): Mean performance 934 rating, mean task cost $0.1 o3 (self-refine 2; high): Mean performance 1,067 rating, mean task cost $0.2 o3 (self-refine 4; high): Mean performance 1,186 rating, mean task cost $0.38 o3 (self-refine 8; high): Mean performance 1,223 rating, mean task cost $0.68 o3 (self-refine 16; high): Mean performance 1,304 rating, mean task cost $1.28 o4-mini (self-refine 1; high): Mean performance 826 rating, mean task cost $0.04 o4-mini (self-refine 2; high): Mean performance 1,014 rating, mean task cost $0.1 o4-mini (self-refine 4; high): Mean performance 1,110 rating, mean task cost $0.21 o4-mini (self-refine 8; high): Mean performance 1,192 rating, mean task cost $0.42 o4-mini (self-refine 16; high): Mean performance 1,271 rating, mean task cost $0.82 Claude Sonnet 4 (self-refine 1): Mean performance 655 rating, mean task cost $0.19 Claude Sonnet 4 (self-refine 2): Mean performance 834 rating, mean task cost $0.32 Claude Sonnet 4 (self-refine 4): Mean performance 990 rating, mean task cost $0.56 Claude Sonnet 4 (self-refine 8): Mean performance 1,042 rating, mean task cost $1.08 Claude Sonnet 4 (self-refine 16): Mean performance 1,088 rating, mean task cost $2.53 DeepSeek-R1 (self-refine 1): Mean performance 804 rating, mean task cost $0.04 DeepSeek-R1 (self-refine 2): Mean performance 909 rating, mean task cost $0.07 DeepSeek-R1 (self-refine 4): Mean performance 967 rating, mean task cost $0.11 DeepSeek-R1 (self-refine 8): Mean performance 1,009 rating, mean task cost $0.19 DeepSeek-R1 (self-refine 16): Mean performance 1,059 rating, mean task cost $0.4 Qwen3-Coder-480B-A35B (self-refine 1): Mean performance 461 rating, mean task cost $0.004 Qwen3-Coder-480B-A35B (self-refine 2): Mean performance 551 rating, mean task cost $0.009 Qwen3-Coder-480B-A35B (self-refine 4): Mean performance 681 rating, mean task cost $0.02 Qwen3-Coder-480B-A35B (self-refine 8): Mean performance 784 rating, mean task cost $0.06 Qwen3-Coder-480B-A35B (self-refine 16): Mean performance 822 rating, mean task cost $0.17 GLM-4.5 (self-refine 1): Mean performance 345 rating, mean task cost $0.03 GLM-4.5 (self-refine 2): Mean performance 431 rating, mean task cost $0.05 GLM-4.5 (self-refine 4): Mean performance 509 rating, mean task cost $0.08 GLM-4.5 (self-refine 8): Mean performance 540 rating, mean task cost $0.15 GLM-4.5 (self-refine 16): Mean performance 552 rating, mean task cost $0.28 Claude Opus 4.1 (self-refine 1): Mean performance 675 rating, mean task cost $0.43 Claude Opus 4.1 (self-refine 2): Mean performance 894 rating, mean task cost $0.78 Claude Opus 4.1 (self-refine 4): Mean performance 1,014 rating, mean task cost $1.62 Claude Opus 4.1 (self-refine 8): Mean performance 1,099 rating, mean task cost $4.08 Claude Opus 4.1 (self-refine 16): Mean performance 1,183 rating, mean task cost $12.1 gpt-oss-120b (self-refine 4): Mean performance 957 rating, mean task cost $0.009 gpt-oss-120b (self-refine 8): Mean performance 1,066 rating, mean task cost $0.02 gpt-oss-120b (self-refine 16): Mean performance 1,119 rating, mean task cost $0.07 gpt-oss-20b (self-refine 4): Mean performance 775 rating, mean task cost $0.005 gpt-oss-20b (self-refine 8): Mean performance 867 rating, mean task cost $0.01 gpt-oss-20b (self-refine 16): Mean performance 942 rating, mean task cost $0.03 GPT-5 (self-refine 1): Mean performance 1,162 rating, mean task cost $0.24 GPT-5 (self-refine 2): Mean performance 1,262 rating, mean task cost $0.44 GPT-5 (self-refine 4): Mean performance 1,431 rating, mean task cost $0.75 GPT-5 (self-refine 8): Mean performance 1,491 rating, mean task cost $1.3 GPT-5 (self-refine 16): Mean performance 1,545 rating, mean task cost $2.41 GPT-5 (self-refine 1): Mean performance 808 rating, mean task cost $0.04 GPT-5 (self-refine 2): Mean performance 956 rating, mean task cost $0.09 GPT-5 (self-refine 4): Mean performance 1,029 rating, mean task cost $0.18 GPT-5 (self-refine 8): Mean performance 1,066 rating, mean task cost $0.4 GPT-5 (self-refine 16): Mean performance 1,105 rating, mean task cost $0.96 GPT-5 mini (self-refine 1): Mean performance 800 rating, mean task cost $0.02 GPT-5 mini (self-refine 2): Mean performance 994 rating, mean task cost $0.05 GPT-5 mini (self-refine 4): Mean performance 1,172 rating, mean task cost $0.11 GPT-5 mini (self-refine 8): Mean performance 1,311 rating, mean task cost $0.21 GPT-5 mini (self-refine 16): Mean performance 1,421 rating, mean task cost $0.44 GPT-5 nano (self-refine 1): Mean performance 719 rating, mean task cost $0.007 GPT-5 nano (self-refine 2): Mean performance 839 rating, mean task cost $0.02 GPT-5 nano (self-refine 4): Mean performance 916 rating, mean task cost $0.03 GPT-5 nano (self-refine 8): Mean performance 992 rating, mean task cost $0.06 GPT-5 nano (self-refine 16): Mean performance 1,016 rating, mean task cost $0.1 Mistral Medium 3.1 (self-refine 1): Mean performance 210 rating, mean task cost $0.007 Mistral Medium 3.1 (self-refine 2): Mean performance 379 rating, mean task cost $0.02 Mistral Medium 3.1 (self-refine 4): Mean performance 487 rating, mean task cost $0.04 Mistral Medium 3.1 (self-refine 8): Mean performance 611 rating, mean task cost $0.11 Mistral Medium 3.1 (self-refine 16): Mean performance 701 rating, mean task cost $0.32 grok-code-fast-1 (self-refine 1): Mean performance 588 rating, mean task cost $0.009 grok-code-fast-1 (self-refine 2): Mean performance 694 rating, mean task cost $0.02 grok-code-fast-1 (self-refine 4): Mean performance 782 rating, mean task cost $0.02 grok-code-fast-1 (self-refine 8): Mean performance 839 rating, mean task cost $0.04 grok-code-fast-1 (self-refine 16): Mean performance 853 rating, mean task cost $0.07 DeepSeek-V3.1-Terminus (self-refine 1): Mean performance 745 rating, mean task cost $0.02 DeepSeek-V3.1-Terminus (self-refine 2): Mean performance 865 rating, mean task cost $0.03 DeepSeek-V3.1-Terminus (self-refine 4): Mean performance 993 rating, mean task cost $0.05 DeepSeek-V3.1-Terminus (self-refine 8): Mean performance 1,052 rating, mean task cost $0.1 DeepSeek-V3.1-Terminus (self-refine 16): Mean performance 1,078 rating, mean task cost $0.21 Qwen3-Coder-Plus (self-refine 1): Mean performance 457 rating, mean task cost $0.01 Qwen3-Coder-Plus (self-refine 2): Mean performance 570 rating, mean task cost $0.03 Qwen3-Coder-Plus (self-refine 4): Mean performance 747 rating, mean task cost $0.06 Qwen3-Coder-Plus (self-refine 8): Mean performance 796 rating, mean task cost $0.13 Qwen3-Coder-Plus (self-refine 16): Mean performance 827 rating, mean task cost $0.29 Claude Sonnet 4.5 (self-refine 1): Mean performance 796 rating, mean task cost $0.15 Claude Sonnet 4.5 (self-refine 2): Mean performance 919 rating, mean task cost $0.29 Claude Sonnet 4.5 (self-refine 4): Mean performance 1,011 rating, mean task cost $0.53 Claude Sonnet 4.5 (self-refine 8): Mean performance 1,072 rating, mean task cost $1 Claude Sonnet 4.5 (self-refine 16): Mean performance 1,192 rating, mean task cost $2.3 GLM-4.6 (self-refine 1): Mean performance 341 rating, mean task cost $0.02 GLM-4.6 (self-refine 2): Mean performance 461 rating, mean task cost $0.03 GLM-4.6 (self-refine 4): Mean performance 724 rating, mean task cost $0.05 GLM-4.6 (self-refine 8): Mean performance 789 rating, mean task cost $0.12 GLM-4.6 (self-refine 16): Mean performance 838 rating, mean task cost $0.31 Claude Haiku 4.5 (self-refine 1): Mean performance 653 rating, mean task cost $0.07 Claude Haiku 4.5 (self-refine 2): Mean performance 823 rating, mean task cost $0.13 Claude Haiku 4.5 (self-refine 4): Mean performance 936 rating, mean task cost $0.24 Claude Haiku 4.5 (self-refine 8): Mean performance 991 rating, mean task cost $0.45 Claude Haiku 4.5 (self-refine 16): Mean performance 1,006 rating, mean task cost $0.94 Kimi K2 Thinking (self-refine 1): Mean performance 598 rating, mean task cost $0.09 Kimi K2 Thinking (self-refine 2): Mean performance 848 rating, mean task cost $0.13 Kimi K2 Thinking (self-refine 4): Mean performance 991 rating, mean task cost $0.18 Kimi K2 Thinking (self-refine 8): Mean performance 1,087 rating, mean task cost $0.3 Kimi K2 Thinking (self-refine 16): Mean performance 1,103 rating, mean task cost $0.53 GPT-5.1-Codex (self-refine 1; high): Mean performance 1,245 rating, mean task cost $0.51 GPT-5.1-Codex (self-refine 2; high): Mean performance 1,373 rating, mean task cost $0.94 GPT-5.1-Codex (self-refine 4; high): Mean performance 1,497 rating, mean task cost $1.41 GPT-5.1-Codex (self-refine 8; high): Mean performance 1,522 rating, mean task cost $1.98 GPT-5.1-Codex (self-refine 16; high): Mean performance 1,528 rating, mean task cost $2.75 GPT-5.1 (self-refine 1): Mean performance 1,192 rating, mean task cost $0.32 GPT-5.1 (self-refine 2): Mean performance 1,281 rating, mean task cost $0.55 GPT-5.1 (self-refine 4): Mean performance 1,336 rating, mean task cost $0.86 GPT-5.1 (self-refine 8): Mean performance 1,401 rating, mean task cost $1.34 GPT-5.1 (self-refine 16): Mean performance 1,425 rating, mean task cost $2.24 Gemini 3 Pro (self-refine 1; high): Mean performance 1,177 rating, mean task cost $0.26 Gemini 3 Pro (self-refine 2; high): Mean performance 1,348 rating, mean task cost $0.48 Gemini 3 Pro (self-refine 4; high): Mean performance 1,571 rating, mean task cost $0.85 Gemini 3 Pro (self-refine 8; high): Mean performance 1,675 rating, mean task cost $1.56 Gemini 3 Pro (self-refine 16; high): Mean performance 1,729 rating, mean task cost $3.03 Gemini 3 Pro (self-refine 1; low): Mean performance 988 rating, mean task cost $0.1 Gemini 3 Pro (self-refine 2; low): Mean performance 1,241 rating, mean task cost $0.24 Gemini 3 Pro (self-refine 4; low): Mean performance 1,463 rating, mean task cost $0.46 Gemini 3 Pro (self-refine 8; low): Mean performance 1,628 rating, mean task cost $0.9 Gemini 3 Pro (self-refine 16; low): Mean performance 1,715 rating, mean task cost $1.88 GPT-5.1-Codex-Max (self-refine 1; high): Mean performance 1,209 rating, mean task cost $0.25 GPT-5.1-Codex-Max (self-refine 2; high): Mean performance 1,316 rating, mean task cost $0.4 GPT-5.1-Codex-Max (self-refine 4; high): Mean performance 1,364 rating, mean task cost $0.6 GPT-5.1-Codex-Max (self-refine 8; high): Mean performance 1,436 rating, mean task cost $0.92 GPT-5.1-Codex-Max (self-refine 16; high): Mean performance 1,458 rating, mean task cost $1.52 GPT-5.1-Codex-Max (self-refine 1; xhigh): Mean performance 1,228 rating, mean task cost $0.32 GPT-5.1-Codex-Max (self-refine 2; xhigh): Mean performance 1,348 rating, mean task cost $0.52 GPT-5.1-Codex-Max (self-refine 4; xhigh): Mean performance 1,457 rating, mean task cost $0.75 GPT-5.1-Codex-Max (self-refine 8; xhigh): Mean performance 1,519 rating, mean task cost $1.13 GPT-5.1-Codex-Max (self-refine 16; xhigh): Mean performance 1,538 rating, mean task cost $1.76 Grok 4.1 Fast (self-refine 1): Mean performance 395 rating, mean task cost $0.009 Grok 4.1 Fast (self-refine 2): Mean performance 701 rating, mean task cost $0.02 Grok 4.1 Fast (self-refine 4): Mean performance 887 rating, mean task cost $0.03 Grok 4.1 Fast (self-refine 8): Mean performance 1,023 rating, mean task cost $0.05 Grok 4.1 Fast (self-refine 16): Mean performance 1,111 rating, mean task cost $0.08 Claude Opus 4.5 (self-refine 1): Mean performance 1,025 rating, mean task cost $0.29 Claude Opus 4.5 (self-refine 2): Mean performance 1,149 rating, mean task cost $0.58 Claude Opus 4.5 (self-refine 4): Mean performance 1,297 rating, mean task cost $1.28 Claude Opus 4.5 (self-refine 8): Mean performance 1,377 rating, mean task cost $2.44 Claude Opus 4.5 (self-refine 16): Mean performance 1,429 rating, mean task cost $5.38 GPT-5.2 (self-refine 1; high): Mean performance 1,294 rating, mean task cost $0.3 GPT-5.2 (self-refine 2; high): Mean performance 1,453 rating, mean task cost $0.54 GPT-5.2 (self-refine 4; high): Mean performance 1,586 rating, mean task cost $0.93 GPT-5.2 (self-refine 8; high): Mean performance 1,660 rating, mean task cost $1.66 GPT-5.2 (self-refine 16; high): Mean performance 1,715 rating, mean task cost $3.19 GPT-5.2 (self-refine 1; medium): Mean performance 1,250 rating, mean task cost $0.14 GPT-5.2 (self-refine 2; medium): Mean performance 1,372 rating, mean task cost $0.26 GPT-5.2 (self-refine 4; medium): Mean performance 1,502 rating, mean task cost $0.48 GPT-5.2 (self-refine 8; medium): Mean performance 1,616 rating, mean task cost $0.95 GPT-5.2 (self-refine 16; medium): Mean performance 1,672 rating, mean task cost $1.93 MiMo-V2-Flash (self-refine 1): Mean performance 738 rating, mean task cost $0.007 MiMo-V2-Flash (self-refine 2): Mean performance 782 rating, mean task cost $0.01 MiMo-V2-Flash (self-refine 4): Mean performance 873 rating, mean task cost $0.02 MiMo-V2-Flash (self-refine 8): Mean performance 974 rating, mean task cost $0.03 MiMo-V2-Flash (self-refine 16): Mean performance 1,000 rating, mean task cost $0.07 Gemini 3 Flash (self-refine 1; high): Mean performance 1,367 rating, mean task cost $0.09 Gemini 3 Flash (self-refine 2; high): Mean performance 1,472 rating, mean task cost $0.16 Gemini 3 Flash (self-refine 4; high): Mean performance 1,598 rating, mean task cost $0.28 Gemini 3 Flash (self-refine 8; high): Mean performance 1,632 rating, mean task cost $0.48 Gemini 3 Flash (self-refine 16; high): Mean performance 1,722 rating, mean task cost $0.85 GPT-5.2 Codex (self-refine 1; xhigh): Mean performance 1,300 rating, mean task cost $0.59 GPT-5.2 Codex (self-refine 2; xhigh): Mean performance 1,512 rating, mean task cost $1.11 GPT-5.2 Codex (self-refine 4; xhigh): Mean performance 1,599 rating, mean task cost $2.1 GPT-5.2 Codex (self-refine 8; xhigh): Mean performance 1,686 rating, mean task cost $3.85 GPT-5.2 Codex (self-refine 16; xhigh): Mean performance 1,613 rating, mean task cost $6.22 GLM-4.7 (self-refine 1): Mean performance 399 rating, mean task cost $0.04 GLM-4.7 (self-refine 2): Mean performance 560 rating, mean task cost $0.07 GLM-4.7 (self-refine 4): Mean performance 805 rating, mean task cost $0.13 GLM-4.7 (self-refine 8): Mean performance 923 rating, mean task cost $0.23 GLM-4.7 (self-refine 16): Mean performance 1,011 rating, mean task cost $0.47 MiniMax-M2.1 (self-refine 1): Mean performance 624 rating, mean task cost $0.05 MiniMax-M2.1 (self-refine 2): Mean performance 858 rating, mean task cost $0.11 MiniMax-M2.1 (self-refine 4): Mean performance 915 rating, mean task cost $0.19 MiniMax-M2.1 (self-refine 8): Mean performance 931 rating, mean task cost $0.3 MiniMax-M2.1 (self-refine 16): Mean performance 998 rating, mean task cost $0.43 Kimi K2.5 (self-refine 1): Mean performance 822 rating, mean task cost $0.05 Kimi K2.5 (self-refine 2): Mean performance 1,077 rating, mean task cost $0.08 Kimi K2.5 (self-refine 4): Mean performance 1,127 rating, mean task cost $0.13 Kimi K2.5 (self-refine 8): Mean performance 1,240 rating, mean task cost $0.2 Kimi K2.5 (self-refine 16): Mean performance 1,279 rating, mean task cost $0.33 Claude Opus 4.6 (self-refine 1; no-thinking): Mean performance 997 rating, mean task cost $0.06 Claude Opus 4.6 (self-refine 2; no-thinking): Mean performance 1,186 rating, mean task cost $0.15 Claude Opus 4.6 (self-refine 4; no-thinking): Mean performance 1,320 rating, mean task cost $0.39 Claude Opus 4.6 (self-refine 8; no-thinking): Mean performance 1,457 rating, mean task cost $1.15 Claude Opus 4.6 (self-refine 16; no-thinking): Mean performance 1,607 rating, mean task cost $3.64 GPT-5.3 Codex (self-refine 1; xhigh): Mean performance 1,655 rating, mean task cost $0.56 GPT-5.3 Codex (self-refine 2; xhigh): Mean performance 1,803 rating, mean task cost $1.06 GPT-5.3 Codex (self-refine 4; xhigh): Mean performance 1,891 rating, mean task cost $1.93 GPT-5.3 Codex (self-refine 8; xhigh): Mean performance 1,919 rating, mean task cost $3.37 GPT-5.3 Codex (self-refine 16; xhigh): Mean performance 1,934 rating, mean task cost $5.91 GLM-5 (self-refine 1): Mean performance 766 rating, mean task cost $0.07 GLM-5 (self-refine 2): Mean performance 828 rating, mean task cost $0.12 GLM-5 (self-refine 4): Mean performance 1,009 rating, mean task cost $0.19 GLM-5 (self-refine 8): Mean performance 1,090 rating, mean task cost $0.29 GLM-5 (self-refine 16): Mean performance 1,194 rating, mean task cost $0.5 MiniMax-M2.5 (self-refine 1): Mean performance 618 rating, mean task cost $0.04 MiniMax-M2.5 (self-refine 2): Mean performance 750 rating, mean task cost $0.06 MiniMax-M2.5 (self-refine 4): Mean performance 1,007 rating, mean task cost $0.1 MiniMax-M2.5 (self-refine 8): Mean performance 988 rating, mean task cost $0.14 MiniMax-M2.5 (self-refine 16): Mean performance 1,033 rating, mean task cost $0.22 Qwen3.5 397B-A17B (self-refine 1): Mean performance 622 rating, mean task cost $0.04 Qwen3.5 397B-A17B (self-refine 2): Mean performance 725 rating, mean task cost $0.06 Qwen3.5 397B-A17B (self-refine 4): Mean performance 898 rating, mean task cost $0.11 Qwen3.5 397B-A17B (self-refine 8): Mean performance 978 rating, mean task cost $0.23 Qwen3.5 397B-A17B (self-refine 16): Mean performance 1,007 rating, mean task cost $0.62 Qwen3.5-Plus (self-refine 1): Mean performance 609 rating, mean task cost $0.03 Qwen3.5-Plus (self-refine 2): Mean performance 813 rating, mean task cost $0.04 Qwen3.5-Plus (self-refine 4): Mean performance 936 rating, mean task cost $0.07 Qwen3.5-Plus (self-refine 8): Mean performance 1,039 rating, mean task cost $0.14 Qwen3.5-Plus (self-refine 16): Mean performance 1,071 rating, mean task cost $0.39 Claude Sonnet 4.6 (self-refine 1; medium): Mean performance 1,327 rating, mean task cost $1.77 Claude Sonnet 4.6 (self-refine 2; medium): Mean performance 1,487 rating, mean task cost $3.5 Claude Sonnet 4.6 (self-refine 4; medium): Mean performance 1,560 rating, mean task cost $6.6 Claude Sonnet 4.6 (self-refine 8; medium): Mean performance 1,688 rating, mean task cost $12.8 Claude Sonnet 4.6 (self-refine 16; medium): Mean performance 1,768 rating, mean task cost $27.9 Gemini 3.1 Pro (self-refine 1; high): Mean performance 1,161 rating, mean task cost $0.39 Gemini 3.1 Pro (self-refine 2; high): Mean performance 1,270 rating, mean task cost $0.79 Gemini 3.1 Pro (self-refine 4; high): Mean performance 1,677 rating, mean task cost $1.59 Gemini 3.1 Pro (self-refine 8; high): Mean performance 1,952 rating, mean task cost $3.14 Gemini 3.1 Pro (self-refine 16; high): Mean performance 2,041 rating, mean task cost $6.68 Gemini 3.1 Pro (self-refine 1; low): Mean performance 1,055 rating, mean task cost $0.04 Gemini 3.1 Pro (self-refine 2; low): Mean performance 1,223 rating, mean task cost $0.09 Gemini 3.1 Pro (self-refine 4; low): Mean performance 1,373 rating, mean task cost $0.2 Gemini 3.1 Pro (self-refine 8; low): Mean performance 1,522 rating, mean task cost $0.43 Gemini 3.1 Pro (self-refine 16; low): Mean performance 1,607 rating, mean task cost $0.93 Mercury 2 (self-refine 1): Mean performance 786 rating, mean task cost $0.007 Mercury 2 (self-refine 2): Mean performance 982 rating, mean task cost $0.02 Mercury 2 (self-refine 4): Mean performance 1,162 rating, mean task cost $0.04 Mercury 2 (self-refine 8): Mean performance 1,183 rating, mean task cost $0.08 Mercury 2 (self-refine 16): Mean performance 1,190 rating, mean task cost $0.17 Qwen3.5-Flash (self-refine 1): Mean performance 266 rating, mean task cost $0.01 Qwen3.5-Flash (self-refine 2): Mean performance 369 rating, mean task cost $0.02 Qwen3.5-Flash (self-refine 4): Mean performance 526 rating, mean task cost $0.02 Qwen3.5-Flash (self-refine 8): Mean performance 758 rating, mean task cost $0.04 Qwen3.5-Flash (self-refine 16): Mean performance 826 rating, mean task cost $0.1 Qwen3.5-27B (self-refine 1): Mean performance 349 rating, mean task cost $0.04 Qwen3.5-27B (self-refine 2): Mean performance 506 rating, mean task cost $0.06 Qwen3.5-27B (self-refine 4): Mean performance 715 rating, mean task cost $0.09 Qwen3.5-27B (self-refine 8): Mean performance 784 rating, mean task cost $0.16 Qwen3.5-27B (self-refine 16): Mean performance 882 rating, mean task cost $0.36 Qwen3.5-35B-A3B (self-refine 1): Mean performance 222 rating, mean task cost $0.04 Qwen3.5-35B-A3B (self-refine 2): Mean performance 333 rating, mean task cost $0.06 Qwen3.5-35B-A3B (self-refine 4): Mean performance 399 rating, mean task cost $0.09 Qwen3.5-35B-A3B (self-refine 8): Mean performance 582 rating, mean task cost $0.16 Qwen3.5-35B-A3B (self-refine 16): Mean performance 648 rating, mean task cost $0.36 Gemini 3.1 Flash-Lite (self-refine 1; high): Mean performance 798 rating, mean task cost $0.03 Gemini 3.1 Flash-Lite (self-refine 2; high): Mean performance 925 rating, mean task cost $0.04 Gemini 3.1 Flash-Lite (self-refine 4; high): Mean performance 1,083 rating, mean task cost $0.08 Gemini 3.1 Flash-Lite (self-refine 8; high): Mean performance 1,165 rating, mean task cost $0.14 Gemini 3.1 Flash-Lite (self-refine 16; high): Mean performance 1,110 rating, mean task cost $0.26 GPT-5.4 (self-refine 1; high): Mean performance 1,607 rating, mean task cost $0.61 GPT-5.4 (self-refine 2; high): Mean performance 1,710 rating, mean task cost $1.07 GPT-5.4 (self-refine 4; high): Mean performance 1,807 rating, mean task cost $1.69 GPT-5.4 (self-refine 8; high): Mean performance 1,752 rating, mean task cost $2.76 GPT-5.4 (self-refine 16; high): Mean performance 1,811 rating, mean task cost $5.02 GPT-5.4 (self-refine 1; medium): Mean performance 1,521 rating, mean task cost $0.26 GPT-5.4 (self-refine 2; medium): Mean performance 1,538 rating, mean task cost $0.53 GPT-5.4 (self-refine 4; medium): Mean performance 1,686 rating, mean task cost $0.94 GPT-5.4 (self-refine 8; medium): Mean performance 1,592 rating, mean task cost $1.69 GPT-5.4 (self-refine 16; medium): Mean performance 1,717 rating, mean task cost $3.32 GPT-5.4 (self-refine 1): Mean performance 1,086 rating, mean task cost $0.05 GPT-5.4 (self-refine 2): Mean performance 1,200 rating, mean task cost $0.12 GPT-5.4 (self-refine 4): Mean performance 1,304 rating, mean task cost $0.27 GPT-5.4 (self-refine 8): Mean performance 1,397 rating, mean task cost $0.65 GPT-5.4 (self-refine 16): Mean performance 1,445 rating, mean task cost $1.58 Nemotron 3 Super (self-refine 1): Mean performance 214 rating, mean task cost $0.008 Nemotron 3 Super (self-refine 2): Mean performance 494 rating, mean task cost $0.01 Nemotron 3 Super (self-refine 4): Mean performance 707 rating, mean task cost $0.02 Nemotron 3 Super (self-refine 8): Mean performance 880 rating, mean task cost $0.03 Nemotron 3 Super (self-refine 16): Mean performance 1,017 rating, mean task cost $0.07 GLM-5-Turbo (self-refine 1): Mean performance 634 rating, mean task cost $0.11 GLM-5-Turbo (self-refine 2): Mean performance 900 rating, mean task cost $0.2 GLM-5-Turbo (self-refine 4): Mean performance 1,053 rating, mean task cost $0.35 GLM-5-Turbo (self-refine 8): Mean performance 1,208 rating, mean task cost $0.62 GLM-5-Turbo (self-refine 16): Mean performance 1,313 rating, mean task cost $1.06 Mistral Small 4 (self-refine 1): Mean performance 498 rating, mean task cost $0.02 Mistral Small 4 (self-refine 2): Mean performance 588 rating, mean task cost $0.02 Mistral Small 4 (self-refine 4): Mean performance 729 rating, mean task cost $0.04 Mistral Small 4 (self-refine 8): Mean performance 787 rating, mean task cost $0.06 Mistral Small 4 (self-refine 16): Mean performance 862 rating, mean task cost $0.12 GPT-5.4 Mini (self-refine 1; high): Mean performance 1,189 rating, mean task cost $0.27 GPT-5.4 Mini (self-refine 2; high): Mean performance 1,277 rating, mean task cost $0.55 GPT-5.4 Mini (self-refine 4; high): Mean performance 1,415 rating, mean task cost $1.04 GPT-5.4 Mini (self-refine 8; high): Mean performance 1,452 rating, mean task cost $1.84 GPT-5.4 Mini (self-refine 16; high): Mean performance 1,487 rating, mean task cost $3.01 GPT-5.4 Nano (self-refine 1; high): Mean performance 1,005 rating, mean task cost $0.03 GPT-5.4 Nano (self-refine 2; high): Mean performance 1,096 rating, mean task cost $0.05 GPT-5.4 Nano (self-refine 4; high): Mean performance 1,214 rating, mean task cost $0.08 GPT-5.4 Nano (self-refine 8; high): Mean performance 1,292 rating, mean task cost $0.14 GPT-5.4 Nano (self-refine 16; high): Mean performance 1,312 rating, mean task cost $0.25 MiMo-V2-Pro (self-refine 1): Mean performance 785 rating, mean task cost $0.08 MiMo-V2-Pro (self-refine 2): Mean performance 954 rating, mean task cost $0.12 MiMo-V2-Pro (self-refine 4): Mean performance 1,122 rating, mean task cost $0.2 MiMo-V2-Pro (self-refine 8): Mean performance 1,262 rating, mean task cost $0.34 MiMo-V2-Pro (self-refine 16): Mean performance 1,347 rating, mean task cost $0.66 MiniMax-M2.7 (self-refine 1): Mean performance 599 rating, mean task cost $0.05 MiniMax-M2.7 (self-refine 2): Mean performance 711 rating, mean task cost $0.09 MiniMax-M2.7 (self-refine 4): Mean performance 821 rating, mean task cost $0.12 MiniMax-M2.7 (self-refine 8): Mean performance 991 rating, mean task cost $0.16 MiniMax-M2.7 (self-refine 16): Mean performance 1,021 rating, mean task cost $0.22 Qwen 3.6 Plus (self-refine 1): Mean performance 670 rating, mean task cost $0.06 Qwen 3.6 Plus (self-refine 2): Mean performance 840 rating, mean task cost $0.1 Qwen 3.6 Plus (self-refine 4): Mean performance 1,045 rating, mean task cost $0.21 Qwen 3.6 Plus (self-refine 8): Mean performance 1,158 rating, mean task cost $0.46 Qwen 3.6 Plus (self-refine 16): Mean performance 1,273 rating, mean task cost $1.07 Gemma 4 31B IT (self-refine 4): Mean performance 1,226 rating, mean task cost $0.02 Gemma 4 31B IT (self-refine 8): Mean performance 1,386 rating, mean task cost $0.04 Gemma 4 31B IT (self-refine 16): Mean performance 1,394 rating, mean task cost $0.11 GLM-5.1 (self-refine 1): Mean performance 887 rating, mean task cost $0.13 GLM-5.1 (self-refine 2): Mean performance 1,170 rating, mean task cost $0.29 GLM-5.1 (self-refine 4): Mean performance 1,353 rating, mean task cost $0.57 GLM-5.1 (self-refine 8): Mean performance 1,489 rating, mean task cost $1.05 GLM-5.1 (self-refine 16): Mean performance 1,646 rating, mean task cost $1.91 Claude Opus 4.7 (self-refine 1; no-thinking): Mean performance 1,323 rating, mean task cost $0.22 Claude Opus 4.7 (self-refine 2; no-thinking): Mean performance 1,521 rating, mean task cost $0.53 Claude Opus 4.7 (self-refine 4; no-thinking): Mean performance 1,695 rating, mean task cost $1.44 Claude Opus 4.7 (self-refine 8; no-thinking): Mean performance 1,836 rating, mean task cost $4.15 Claude Opus 4.7 (self-refine 16; no-thinking): Mean performance 1,897 rating, mean task cost $12 Kimi K2.6 (self-refine 1): Mean performance 1,093 rating, mean task cost $0.2 Kimi K2.6 (self-refine 2): Mean performance 1,297 rating, mean task cost $0.36 Kimi K2.6 (self-refine 4): Mean performance 1,520 rating, mean task cost $0.61 Kimi K2.6 (self-refine 8): Mean performance 1,593 rating, mean task cost $1 Kimi K2.6 (self-refine 16): Mean performance 1,695 rating, mean task cost $1.65 GPT-5.5 (self-refine 1; medium): Mean performance 1,589 rating, mean task cost $0.33 GPT-5.5 (self-refine 2; medium): Mean performance 1,735 rating, mean task cost $0.63 GPT-5.5 (self-refine 4; medium): Mean performance 1,791 rating, mean task cost $1.26 GPT-5.5 (self-refine 8; medium): Mean performance 1,892 rating, mean task cost $2.58 GPT-5.5 (self-refine 16; medium): Mean performance 1,938 rating, mean task cost $5.49 GPT-5.5 (self-refine 1): Mean performance 1,128 rating, mean task cost $0.07 GPT-5.5 (self-refine 2): Mean performance 1,394 rating, mean task cost $0.19 GPT-5.5 (self-refine 4): Mean performance 1,545 rating, mean task cost $0.47 GPT-5.5 (self-refine 8): Mean performance 1,558 rating, mean task cost $1.18 GPT-5.5 (self-refine 16): Mean performance 1,633 rating, mean task cost $2.86 GPT-5.5 (self-refine 1; xhigh): Mean performance 1,943 rating, mean task cost $1.52 GPT-5.5 (self-refine 2; xhigh): Mean performance 2,006 rating, mean task cost $2.66 GPT-5.5 (self-refine 4; xhigh): Mean performance 2,061 rating, mean task cost $4.56 GPT-5.5 (self-refine 8; xhigh): Mean performance 2,095 rating, mean task cost $8.03 GPT-5.5 (self-refine 16; xhigh): Mean performance 2,108 rating, mean task cost $14.5 MiMo-V2.5 (self-refine 1): Mean performance 514 rating, mean task cost $0.04 MiMo-V2.5 (self-refine 2): Mean performance 735 rating, mean task cost $0.06 MiMo-V2.5 (self-refine 4): Mean performance 942 rating, mean task cost $0.09 MiMo-V2.5 (self-refine 8): Mean performance 1,052 rating, mean task cost $0.15 MiMo-V2.5 (self-refine 16): Mean performance 1,106 rating, mean task cost $0.29 MiMo-V2.5-Pro (self-refine 1): Mean performance 900 rating, mean task cost $0.1 MiMo-V2.5-Pro (self-refine 2): Mean performance 1,080 rating, mean task cost $0.16 MiMo-V2.5-Pro (self-refine 4): Mean performance 1,138 rating, mean task cost $0.22 MiMo-V2.5-Pro (self-refine 8): Mean performance 1,204 rating, mean task cost $0.33 MiMo-V2.5-Pro (self-refine 16): Mean performance 1,254 rating, mean task cost $0.61 DeepSeek-V4-Flash (self-refine 1; high): Mean performance 678 rating, mean task cost $0.005 DeepSeek-V4-Flash (self-refine 8; high): Mean performance 1,296 rating, mean task cost $0.02 DeepSeek-V4-Flash (self-refine 16; high): Mean performance 1,380 rating, mean task cost $0.05 DeepSeek-V4-Flash (self-refine 1; no-thinking): Mean performance 325 rating, mean task cost $0.0009 DeepSeek-V4-Flash (self-refine 2; no-thinking): Mean performance 533 rating, mean task cost $0.002 DeepSeek-V4-Flash (self-refine 4; no-thinking): Mean performance 744 rating, mean task cost $0.006 DeepSeek-V4-Flash (self-refine 8; no-thinking): Mean performance 929 rating, mean task cost $0.01 DeepSeek-V4-Flash (self-refine 16; no-thinking): Mean performance 1,051 rating, mean task cost $0.03 DeepSeek-V4-Pro (self-refine 1; high): Mean performance 1,006 rating, mean task cost $0.15 DeepSeek-V4-Pro (self-refine 2; high): Mean performance 1,182 rating, mean task cost $0.23 DeepSeek-V4-Pro (self-refine 4; high): Mean performance 1,308 rating, mean task cost $0.35 DeepSeek-V4-Pro (self-refine 8; high): Mean performance 1,445 rating, mean task cost $0.56 DeepSeek-V4-Pro (self-refine 16; high): Mean performance 1,489 rating, mean task cost $0.96 DeepSeek-V4-Pro (self-refine 1; no-thinking): Mean performance 522 rating, mean task cost $0.02 DeepSeek-V4-Pro (self-refine 2; no-thinking): Mean performance 725 rating, mean task cost $0.05 DeepSeek-V4-Pro (self-refine 4; no-thinking): Mean performance 971 rating, mean task cost $0.1 DeepSeek-V4-Pro (self-refine 8; no-thinking): Mean performance 1,188 rating, mean task cost $0.21 DeepSeek-V4-Pro (self-refine 16; no-thinking): Mean performance 1,272 rating, mean task cost $0.45 Qwen 3.6 Flash (self-refine 1): Mean performance 326 rating, mean task cost $0.04 Qwen 3.6 Flash (self-refine 2): Mean performance 562 rating, mean task cost $0.06 Qwen 3.6 Flash (self-refine 4): Mean performance 668 rating, mean task cost $0.11 Qwen 3.6 Flash (self-refine 8): Mean performance 874 rating, mean task cost $0.19 Qwen 3.6 Flash (self-refine 16): Mean performance 1,015 rating, mean task cost $0.41 Mistral Medium 3.5 (self-refine 1): Mean performance 764 rating, mean task cost $0.27 Mistral Medium 3.5 (self-refine 2): Mean performance 896 rating, mean task cost $0.44 Mistral Medium 3.5 (self-refine 4): Mean performance 1,010 rating, mean task cost $0.69 Mistral Medium 3.5 (self-refine 8): Mean performance 1,064 rating, mean task cost $1.14 Mistral Medium 3.5 (self-refine 16): Mean performance 1,131 rating, mean task cost $1.96 Ling-2.6-1T (self-refine 1): Mean performance 433 rating, mean task cost $0.008 Ling-2.6-1T (self-refine 2): Mean performance 733 rating, mean task cost $0.02 Ling-2.6-1T (self-refine 4): Mean performance 846 rating, mean task cost $0.04 Ling-2.6-1T (self-refine 8): Mean performance 1,017 rating, mean task cost $0.08 Ling-2.6-1T (self-refine 16): Mean performance 1,108 rating, mean task cost $0.17 Grok 4.3 (self-refine 1): Mean performance 944 rating, mean task cost $0.06 Grok 4.3 (self-refine 2): Mean performance 1,088 rating, mean task cost $0.1 Grok 4.3 (self-refine 4): Mean performance 1,131 rating, mean task cost $0.18 Grok 4.3 (self-refine 8): Mean performance 1,155 rating, mean task cost $0.33 Grok 4.3 (self-refine 16): Mean performance 1,211 rating, mean task cost $0.71 Gemini 3.5 Flash (self-refine 1; high): Mean performance 911 rating, mean task cost $0.44 Gemini 3.5 Flash (self-refine 2; high): Mean performance 1,178 rating, mean task cost $0.68 Gemini 3.5 Flash (self-refine 4; high): Mean performance 1,552 rating, mean task cost $1.04 Gemini 3.5 Flash (self-refine 8; high): Mean performance 1,657 rating, mean task cost $1.73 Gemini 3.5 Flash (self-refine 16; high): Mean performance 1,712 rating, mean task cost $2.94 Claude Opus 4.8 (self-refine 1; high): Mean performance 1,564 rating, mean task cost $0.96 Claude Opus 4.8 (self-refine 2; high): Mean performance 1,690 rating, mean task cost $1.8 Claude Opus 4.8 (self-refine 4; high): Mean performance 1,802 rating, mean task cost $3.48 Claude Opus 4.8 (self-refine 8; high): Mean performance 1,970 rating, mean task cost $7.56 Claude Opus 4.8 (self-refine 16; high): Mean performance 2,102 rating, mean task cost $18.9 Claude Opus 4.8 (self-refine 1; no-thinking): Mean performance 1,412 rating, mean task cost $0.42 Claude Opus 4.8 (self-refine 2; no-thinking): Mean performance 1,613 rating, mean task cost $0.84 Claude Opus 4.8 (self-refine 4; no-thinking): Mean performance 1,799 rating, mean task cost $1.8 Claude Opus 4.8 (self-refine 8; no-thinking): Mean performance 1,820 rating, mean task cost $4.28 Claude Opus 4.8 (self-refine 16; no-thinking): Mean performance 1,913 rating, mean task cost $11.5 Step 3.7 Flash (self-refine 1; medium): Mean performance 694 rating, mean task cost $0.03 Step 3.7 Flash (self-refine 2; medium): Mean performance 855 rating, mean task cost $0.06 Step 3.7 Flash (self-refine 4; medium): Mean performance 1,059 rating, mean task cost $0.13 Step 3.7 Flash (self-refine 8; medium): Mean performance 1,236 rating, mean task cost $0.22 Step 3.7 Flash (self-refine 16; medium): Mean performance 1,277 rating, mean task cost $0.38 MiniMax-M3 (self-refine 1): Mean performance 640 rating, mean task cost $0.11 MiniMax-M3 (self-refine 2): Mean performance 773 rating, mean task cost $0.17 MiniMax-M3 (self-refine 4): Mean performance 903 rating, mean task cost $0.26 MiniMax-M3 (self-refine 8): Mean performance 1,025 rating, mean task cost $0.4 MiniMax-M3 (self-refine 16): Mean performance 1,053 rating, mean task cost $0.68 Claude Fable 5 (self-refine 1; high): Mean performance 2,041 rating, mean task cost $2.11 Claude Fable 5 (self-refine 2; high): Mean performance 2,194 rating, mean task cost $3.96 Claude Fable 5 (self-refine 4; high): Mean performance 2,326 rating, mean task cost $7.78 Claude Fable 5 (self-refine 8; high): Mean performance 2,378 rating, mean task cost $16.6 Claude Fable 5 (self-refine 16; high): Mean performance 2,463 rating, mean task cost $41 Kimi K2.7 Code (self-refine 1): Mean performance 886 rating, mean task cost $0.18 Kimi K2.7 Code (self-refine 2): Mean performance 1,046 rating, mean task cost $0.3 Kimi K2.7 Code (self-refine 4): Mean performance 1,243 rating, mean task cost $0.46 Kimi K2.7 Code (self-refine 8): Mean performance 1,316 rating, mean task cost $0.77 Kimi K2.7 Code (self-refine 16): Mean performance 1,422 rating, mean task cost $1.38 GLM-5.2 (self-refine 1; high): Mean performance 1,047 rating, mean task cost $0.16 GLM-5.2 (self-refine 2; high): Mean performance 1,201 rating, mean task cost $0.3 GLM-5.2 (self-refine 4; high): Mean performance 1,372 rating, mean task cost $0.48 GLM-5.2 (self-refine 8; high): Mean performance 1,510 rating, mean task cost $0.77 GLM-5.2 (self-refine 16; high): Mean performance 1,463 rating, mean task cost $1.32 GLM-5.2 (self-refine 1; max): Mean performance 1,010 rating, mean task cost $0.15 GLM-5.2 (self-refine 2; max): Mean performance 1,255 rating, mean task cost $0.33 GLM-5.2 (self-refine 4; max): Mean performance 1,500 rating, mean task cost $0.64 GLM-5.2 (self-refine 8; max): Mean performance 1,610 rating, mean task cost $1.13 GLM-5.2 (self-refine 16; max): Mean performance 1,685 rating, mean task cost $1.95 Claude Sonnet 5 (self-refine 1; high): Mean performance 1,463 rating, mean task cost $0.96 Claude Sonnet 5 (self-refine 2; high): Mean performance 1,533 rating, mean task cost $1.62 Claude Sonnet 5 (self-refine 4; high): Mean performance 1,610 rating, mean task cost $2.79 Claude Sonnet 5 (self-refine 8; high): Mean performance 1,717 rating, mean task cost $5.4 Claude Sonnet 5 (self-refine 16; high): Mean performance 1,783 rating, mean task cost $11.9 Tencent Hy3 (self-refine 1; high): Mean performance 839 rating, mean task cost $0.03 Tencent Hy3 (self-refine 2; high): Mean performance 1,159 rating, mean task cost $0.05 Tencent Hy3 (self-refine 4; high): Mean performance 1,321 rating, mean task cost $0.07 Tencent Hy3 (self-refine 8; high): Mean performance 1,343 rating, mean task cost $0.12 Tencent Hy3 (self-refine 16; high): Mean performance 1,516 rating, mean task cost $0.21 LongCat-2.0 (self-refine 1): Mean performance 793 rating, mean task cost $0.04 LongCat-2.0 (self-refine 2): Mean performance 908 rating, mean task cost $0.06 LongCat-2.0 (self-refine 4): Mean performance 1,110 rating, mean task cost $0.09 LongCat-2.0 (self-refine 8): Mean performance 1,212 rating, mean task cost $0.13 LongCat-2.0 (self-refine 16): Mean performance 1,204 rating, mean task cost $0.19 Grok 4.5 (self-refine 1; high): Mean performance 1,309 rating, mean task cost $0.18 Grok 4.5 (self-refine 2; high): Mean performance 1,348 rating, mean task cost $0.28 Grok 4.5 (self-refine 4; high): Mean performance 1,417 rating, mean task cost $0.4 Grok 4.5 (self-refine 8; high): Mean performance 1,467 rating, mean task cost $0.64 Grok 4.5 (self-refine 16; high): Mean performance 1,441 rating, mean task cost $1.23 GPT-5.6 Luna (self-refine 1; max): Mean performance 1,667 rating, mean task cost $0.36 GPT-5.6 Luna (self-refine 2; max): Mean performance 1,725 rating, mean task cost $0.64 GPT-5.6 Luna (self-refine 4; max): Mean performance 1,763 rating, mean task cost $0.93 GPT-5.6 Luna (self-refine 8; max): Mean performance 1,785 rating, mean task cost $1.38 GPT-5.6 Luna (self-refine 16; max): Mean performance 1,829 rating, mean task cost $2.29 GPT-5.6 Sol (self-refine 1; max): Mean performance 2,177 rating, mean task cost $1.53 GPT-5.6 Sol (self-refine 2; max): Mean performance 2,316 rating, mean task cost $2.67 GPT-5.6 Sol (self-refine 4; max): Mean performance 2,365 rating, mean task cost $4.33 GPT-5.6 Sol (self-refine 8; max): Mean performance 2,391 rating, mean task cost $7.36 GPT-5.6 Sol (self-refine 16; max): Mean performance 2,410 rating, mean task cost $15.1 GPT-5.6 Terra (self-refine 1; max): Mean performance 1,951 rating, mean task cost $1.25 GPT-5.6 Terra (self-refine 2; max): Mean performance 2,056 rating, mean task cost $2.14 GPT-5.6 Terra (self-refine 4; max): Mean performance 2,101 rating, mean task cost $3.1 GPT-5.6 Terra (self-refine 8; max): Mean performance 2,122 rating, mean task cost $4.56 GPT-5.6 Terra (self-refine 16; max): Mean performance 2,154 rating, mean task cost $7.74 Inkling (self-refine 1; max): Mean performance 946 rating, mean task cost $0.2 Inkling (self-refine 2; max): Mean performance 1,062 rating, mean task cost $0.32 Inkling (self-refine 4; max): Mean performance 1,150 rating, mean task cost $0.53 Inkling (self-refine 8; max): Mean performance 1,135 rating, mean task cost $0.95 Inkling (self-refine 16; max): Mean performance 1,195 rating, mean task cost $1.9 Kimi K3 (self-refine 1; max): Mean performance 1,524 rating, mean task cost $1.28 Kimi K3 (self-refine 2; max): Mean performance 1,633 rating, mean task cost $2.26 Kimi K3 (self-refine 4; max): Mean performance 1,788 rating, mean task cost $3.37 Kimi K3 (self-refine 8; max): Mean performance 1,908 rating, mean task cost $5.58 Kimi K3 (self-refine 16; max): Mean performance 1,991 rating, mean task cost $9.8 Gemini 3.5 Flash-Lite (self-refine 1; high): Mean performance 765 rating, mean task cost $0.05 Gemini 3.5 Flash-Lite (self-refine 2; high): Mean performance 929 rating, mean task cost $0.07 Gemini 3.5 Flash-Lite (self-refine 4; high): Mean performance 1,072 rating, mean task cost $0.11 Gemini 3.5 Flash-Lite (self-refine 8; high): Mean performance 1,111 rating, mean task cost $0.19 Gemini 3.5 Flash-Lite (self-refine 16; high): Mean performance 1,137 rating, mean task cost $0.34 Gemini 3.6 Flash (self-refine 1; high): Mean performance 716 rating, mean task cost $0.24 Gemini 3.6 Flash (self-refine 2; high): Mean performance 1,121 rating, mean task cost $0.4 Gemini 3.6 Flash (self-refine 4; high): Mean performance 1,409 rating, mean task cost $0.68 Gemini 3.6 Flash (self-refine 8; high): Mean performance 1,601 rating, mean task cost $1.22 Gemini 3.6 Flash (self-refine 16; high): Mean performance 1,825 rating, mean task cost $2.53 Claude Opus 5 (self-refine 1; high): Mean performance 2,165 rating, mean task cost $1.89 Claude Opus 5 (self-refine 2; high): Mean performance 2,340 rating, mean task cost $3.38 Claude Opus 5 (self-refine 4; high): Mean performance 2,436 rating, mean task cost $6.5 Claude Opus 5 (self-refine 8; high): Mean performance 2,582 rating, mean task cost $13.6 Claude Opus 5 (self-refine 16; high): Mean performance 2,622 rating, mean task cost $32 Qwen3.7 Flash (self-refine 1): Mean performance 420 rating, mean task cost $0.004 Qwen3.7 Flash (self-refine 2): Mean performance 601 rating, mean task cost $0.006 Qwen3.7 Flash (self-refine 4): Mean performance 798 rating, mean task cost $0.01 Qwen3.7 Flash (self-refine 8): Mean performance 872 rating, mean task cost $0.02 Qwen3.7 Flash (self-refine 16): Mean performance 946 rating, mean task cost $0.06 DeepSeek V4 Flash 0731 (self-refine 1; max): Mean performance 1,306 rating, mean task cost $0.02 DeepSeek V4 Flash 0731 (self-refine 2; max): Mean performance 1,501 rating, mean task cost $0.03 DeepSeek V4 Flash 0731 (self-refine 4; max): Mean performance 1,590 rating, mean task cost $0.04 DeepSeek V4 Flash 0731 (self-refine 8; max): Mean performance 1,686 rating, mean task cost $0.07 DeepSeek V4 Flash 0731 (self-refine 16; max): Mean performance 1,679 rating, mean task cost $0.1 Nemotron 3.5 Lightning 30B-A3B (self-refine 1): Mean performance 769 rating, mean task cost $0.009 Nemotron 3.5 Lightning 30B-A3B (self-refine 2): Mean performance 914 rating, mean task cost $0.02 Nemotron 3.5 Lightning 30B-A3B (self-refine 4): Mean performance 957 rating, mean task cost $0.03 Nemotron 3.5 Lightning 30B-A3B (self-refine 8): Mean performance 993 rating, mean task cost $0.04 Nemotron 3.5 Lightning 30B-A3B (self-refine 16): Mean performance 1,017 rating, mean task cost $0.07 Solar Pro 4 (self-refine 1; max): Mean performance 179 rating, mean task cost $0.04 Solar Pro 4 (self-refine 2; max): Mean performance 465 rating, mean task cost $0.06 Solar Pro 4 (self-refine 4; max): Mean performance 919 rating, mean task cost $0.11 Solar Pro 4 (self-refine 8; max): Mean performance 1,073 rating, mean task cost $0.2 Solar Pro 4 (self-refine 16; max): Mean performance 1,256 rating, mean task cost $0.42 Grok 4.6 (self-refine 1; xhigh): Mean performance 1,508 rating, mean task cost $0.24 Grok 4.6 (self-refine 2; xhigh): Mean performance 1,592 rating, mean task cost $0.44 Grok 4.6 (self-refine 4; xhigh): Mean performance 1,673 rating, mean task cost $0.71 Grok 4.6 (self-refine 8; xhigh): Mean performance 1,745 rating, mean task cost $1.16 Grok 4.6 (self-refine 16; xhigh): Mean performance 1,822 rating, mean task cost $2.32 DeepSeek V4 Pro 0813 (self-refine 1; max): Mean performance 1,403 rating, mean task cost $0.07 DeepSeek V4 Pro 0813 (self-refine 2; max): Mean performance 1,550 rating, mean task cost $0.12 DeepSeek V4 Pro 0813 (self-refine 4; max): Mean performance 1,628 rating, mean task cost $0.18 DeepSeek V4 Pro 0813 (self-refine 8; max): Mean performance 1,649 rating, mean task cost $0.27 DeepSeek V4 Pro 0813 (self-refine 16; max): Mean performance 1,730 rating, mean task cost $0.41 Gemini 3.7 Flash (self-refine 1; high): Mean performance 904 rating, mean task cost $0.24 Gemini 3.7 Flash (self-refine 2; high): Mean performance 1,210 rating, mean task cost $0.41 Gemini 3.7 Flash (self-refine 4; high): Mean performance 1,567 rating, mean task cost $0.68 Gemini 3.7 Flash (self-refine 8; high): Mean performance 1,840 rating, mean task cost $1.2 Gemini 3.7 Flash (self-refine 16; high): Mean performance 1,876 rating, mean task cost $2.34 GLM-5.3 (self-refine 1; high): Mean performance 1,317 rating, mean task cost $0.28 GLM-5.3 (self-refine 2; high): Mean performance 1,454 rating, mean task cost $0.5 GLM-5.3 (self-refine 4; high): Mean performance 1,724 rating, mean task cost $0.72 GLM-5.3 (self-refine 8; high): Mean performance 1,863 rating, mean task cost $1.1 GLM-5.3 (self-refine 16; high): Mean performance 1,937 rating, mean task cost $1.86 GLM-5.3-Flash (self-refine 1; high): Mean performance 304 rating, mean task cost $0.01 GLM-5.3-Flash (self-refine 2; high): Mean performance 668 rating, mean task cost $0.02 GLM-5.3-Flash (self-refine 4; high): Mean performance 961 rating, mean task cost $0.03 GLM-5.3-Flash (self-refine 8; high): Mean performance 1,136 rating, mean task cost $0.06 GLM-5.3-Flash (self-refine 16; high): Mean performance 1,283 rating, mean task cost $0.11 Claude Fable 5.1 (self-refine 1; high): Mean performance 2,143 rating, mean task cost $2.29 Claude Fable 5.1 (self-refine 2; high): Mean performance 2,287 rating, mean task cost $3.79 Claude Fable 5.1 (self-refine 4; high): Mean performance 2,470 rating, mean task cost $7.05 Claude Fable 5.1 (self-refine 8; high): Mean performance 2,581 rating, mean task cost $15.3 Claude Fable 5.1 (self-refine 16; high): Mean performance 2,849 rating, mean task cost $39 Qwen3.8 Max (0902) (self-refine 1; xhigh): Mean performance 1,679 rating, mean task cost $0.53 Qwen3.8 Max (0902) (self-refine 2; xhigh): Mean performance 1,744 rating, mean task cost $0.96 Qwen3.8 Max (0902) (self-refine 4; xhigh): Mean performance 1,848 rating, mean task cost $1.5 Qwen3.8 Max (0902) (self-refine 8; xhigh): Mean performance 1,908 rating, mean task cost $2.43 Qwen3.8 Max (0902) (self-refine 16; xhigh): Mean performance 1,964 rating, mean task cost $4.36 Gemini 3.8 Flash (self-refine 1; high): Mean performance 1,270 rating, mean task cost $0.34 Gemini 3.8 Flash (self-refine 2; high): Mean performance 1,637 rating, mean task cost $0.58 Gemini 3.8 Flash (self-refine 4; high): Mean performance 2,007 rating, mean task cost $0.96 Gemini 3.8 Flash (self-refine 8; high): Mean performance 2,086 rating, mean task cost $1.67 Gemini 3.8 Flash (self-refine 16; high): Mean performance 2,182 rating, mean task cost $3.18 Muse Spark 1.3 (self-refine 1; xhigh): Mean performance 1,688 rating, mean task cost $0.22 Muse Spark 1.3 (self-refine 2; xhigh): Mean performance 1,765 rating, mean task cost $0.4 Muse Spark 1.3 (self-refine 4; xhigh): Mean performance 1,837 rating, mean task cost $0.68 Muse Spark 1.3 (self-refine 8; xhigh): Mean performance 1,924 rating, mean task cost $1.37 Muse Spark 1.3 (self-refine 16; xhigh): Mean performance 1,981 rating, mean task cost $2.95 GPT-6 Astra (self-refine 16; max): Mean performance 3,128 rating, mean task cost $46.7 Mercury 2.5 (self-refine 1; high): Mean performance 302 rating, mean task cost $0.006 Mercury 2.5 (self-refine 2; high): Mean performance 464 rating, mean task cost $0.01 Mercury 2.5 (self-refine 4; high): Mean performance 665 rating, mean task cost $0.02 Mercury 2.5 (self-refine 8; high): Mean performance 897 rating, mean task cost $0.04 Mercury 2.5 (self-refine 16; high): Mean performance 977 rating, mean task cost $0.1 DeepSeek V4.1 Flash (self-refine 1; max): Mean performance 1,092 rating, mean task cost $0.08 DeepSeek V4.1 Flash (self-refine 2; max): Mean performance 1,287 rating, mean task cost $0.14 DeepSeek V4.1 Flash (self-refine 4; max): Mean performance 1,476 rating, mean task cost $0.24 DeepSeek V4.1 Flash (self-refine 8; max): Mean performance 1,727 rating, mean task cost $0.39 DeepSeek V4.1 Flash (self-refine 16; max): Mean performance 1,829 rating, mean task cost $0.65 Grok 4.7 (self-refine 1; xhigh): Mean performance 1,546 rating, mean task cost $0.67 Grok 4.7 (self-refine 2; xhigh): Mean performance 1,601 rating, mean task cost $1.12 Grok 4.7 (self-refine 4; xhigh): Mean performance 1,650 rating, mean task cost $1.73 Grok 4.7 (self-refine 8; xhigh): Mean performance 1,732 rating, mean task cost $3.24 Grok 4.7 (self-refine 16; xhigh): Mean performance 1,823 rating, mean task cost $7.12 MiMo-V2.6 Flash (self-refine 1): Mean performance 1,244 rating, mean task cost $0.04 MiMo-V2.6 Flash (self-refine 2): Mean performance 1,321 rating, mean task cost $0.06 MiMo-V2.6 Flash (self-refine 4): Mean performance 1,349 rating, mean task cost $0.08 MiMo-V2.6 Flash (self-refine 8): Mean performance 1,406 rating, mean task cost $0.1 MiMo-V2.6 Flash (self-refine 16): Mean performance 1,455 rating, mean task cost $0.15 MiMo-V2.6 Pro (self-refine 1): Mean performance 1,158 rating, mean task cost $0.11 MiMo-V2.6 Pro (self-refine 2): Mean performance 1,270 rating, mean task cost $0.19 MiMo-V2.6 Pro (self-refine 4): Mean performance 1,350 rating, mean task cost $0.26 MiMo-V2.6 Pro (self-refine 8): Mean performance 1,476 rating, mean task cost $0.36 MiMo-V2.6 Pro (self-refine 16): Mean performance 1,441 rating, mean task cost $0.48 Claude Opus 5.5 (self-refine 2; high): Mean performance 2,335 rating, mean task cost $0.52 Claude Opus 5.5 (self-refine 4; high): Mean performance 2,504 rating, mean task cost $1.16 Claude Opus 5.5 (self-refine 8; high): Mean performance 2,670 rating, mean task cost $2.93 Claude Opus 5.5 (self-refine 16; high): Mean performance 2,748 rating, mean task cost $8.59 GPT-6 Luna (self-refine 4; xhigh): Mean performance 1,714 rating, mean task cost $0.04 GPT-6 Luna (self-refine 16; xhigh): Mean performance 1,791 rating, mean task cost $0.12 GPT-6 Sol (self-refine 1; max): Mean performance 2,462 rating, mean task cost $0.84 GPT-6 Sol (self-refine 2; max): Mean performance 2,573 rating, mean task cost $1.42 GPT-6 Sol (self-refine 4; max): Mean performance 2,629 rating, mean task cost $2.39 GPT-6 Sol (self-refine 8; max): Mean performance 2,633 rating, mean task cost $4.11 GPT-6 Sol (self-refine 16; max): Mean performance 2,637 rating, mean task cost $8.42 Claude Sonnet 5.5 (self-refine 1; high): Mean performance 1,819 rating, mean task cost $0.11 Claude Sonnet 5.5 (self-refine 4; high): Mean performance 2,151 rating, mean task cost $0.57 Claude Sonnet 5.5 (self-refine 8; high): Mean performance 2,289 rating, mean task cost $1.49 Claude Sonnet 5.5 (self-refine 16; high): Mean performance 2,448 rating, mean task cost $4.05 GPT-6.1 Sol (self-refine 8; max): Mean performance 2,942 rating, mean task cost $4.29 GPT-6.1 Sol (self-refine 16; max): Mean performance 2,990 rating, mean task cost $10.3 Gemini 4 Argon (self-refine 1; high): Mean performance 2,100 rating, mean task cost $1.95 Gemini 4 Argon (self-refine 2; high): Mean performance 2,359 rating, mean task cost $3.17 Gemini 4 Argon (self-refine 4; high): Mean performance 2,675 rating, mean task cost $4.92 Gemini 4 Argon (self-refine 8; high): Mean performance 2,805 rating, mean task cost $8.1 Gemini 4 Argon (self-refine 16; high): Mean performance 2,747 rating, mean task cost $15.4 Gemini 4 Argon (self-refine 1; low): Mean performance 1,705 rating, mean task cost $0.17 Gemini 4 Argon (self-refine 2; low): Mean performance 1,990 rating, mean task cost $0.41 Gemini 4 Argon (self-refine 4; low): Mean performance 2,298 rating, mean task cost $1.06 Gemini 4 Argon (self-refine 8; low): Mean performance 2,481 rating, mean task cost $3.02 Gemini 4 Argon (self-refine 16; low): Mean performance 2,599 rating, mean task cost $6.85 Beam 1.0 (self-refine 1; max): Mean performance 1,098 rating, mean task cost $0.03 Beam 1.0 (self-refine 2; max): Mean performance 1,139 rating, mean task cost $0.05 Beam 1.0 (self-refine 4; max): Mean performance 1,304 rating, mean task cost $0.08 Beam 1.0 (self-refine 8; max): Mean performance 1,430 rating, mean task cost $0.14 Beam 1.0 (self-refine 16; max): Mean performance 1,493 rating, mean task cost $0.26 Mistral Large 4 (self-refine 1): Mean performance 1,213 rating, mean task cost $0.46 Mistral Large 4 (self-refine 2): Mean performance 1,332 rating, mean task cost $0.73 Mistral Large 4 (self-refine 4): Mean performance 1,492 rating, mean task cost $1.45 Mistral Large 4 (self-refine 8): Mean performance 1,589 rating, mean task cost $3.08 Mistral Large 4 (self-refine 16): Mean performance 1,657 rating, mean task cost $7.37 Claude Haiku 5.5 (self-refine 16; high): Mean performance 1,996 rating, mean task cost $0.87 Muse Spark Open (self-refine 1; xhigh): Mean performance 1,590 rating, mean task cost $0.33 Muse Spark Open (self-refine 2; xhigh): Mean performance 1,640 rating, mean task cost $0.52 Muse Spark Open (self-refine 4; xhigh): Mean performance 1,704 rating, mean task cost $0.75 Muse Spark Open (self-refine 8; xhigh): Mean performance 1,783 rating, mean task cost $1.1 Muse Spark Open (self-refine 16; xhigh): Mean performance 1,830 rating, mean task cost $1.72 Nemotron 3.5 Super VL (self-refine 1): Mean performance 957 rating, mean task cost $0.02 Nemotron 3.5 Super VL (self-refine 2): Mean performance 1,142 rating, mean task cost $0.03 Nemotron 3.5 Super VL (self-refine 4): Mean performance 1,152 rating, mean task cost $0.04 Nemotron 3.5 Super VL (self-refine 8): Mean performance 1,276 rating, mean task cost $0.05 Nemotron 3.5 Super VL (self-refine 16): Mean performance 1,352 rating, mean task cost $0.08 DeepSeek V4.1 Flash 1017 (self-refine 1; max): Mean performance 1,622 rating, mean task cost $0.1 DeepSeek V4.1 Flash 1017 (self-refine 2; max): Mean performance 1,677 rating, mean task cost $0.17 DeepSeek V4.1 Flash 1017 (self-refine 4; max): Mean performance 1,769 rating, mean task cost $0.29 DeepSeek V4.1 Flash 1017 (self-refine 8; max): Mean performance 1,872 rating, mean task cost $0.47 DeepSeek V4.1 Flash 1017 (self-refine 16; max): Mean performance 2,012 rating, mean task cost $0.76 Mercury 3 (self-refine 1; high): Mean performance 1,209 rating, mean task cost $0.05 Mercury 3 (self-refine 2; high): Mean performance 1,360 rating, mean task cost $0.08 Mercury 3 (self-refine 4; high): Mean performance 1,483 rating, mean task cost $0.12 Mercury 3 (self-refine 8; high): Mean performance 1,561 rating, mean task cost $0.17 Mercury 3 (self-refine 16; high): Mean performance 1,692 rating, mean task cost $0.28 MiniMax-M3.1-Flash (self-refine 1; max): Mean performance 218 rating, mean task cost $0.05 MiniMax-M3.1-Flash (self-refine 2; max): Mean performance 671 rating, mean task cost $0.07 MiniMax-M3.1-Flash (self-refine 4; max): Mean performance 1,041 rating, mean task cost $0.09 MiniMax-M3.1-Flash (self-refine 8; max): Mean performance 1,314 rating, mean task cost $0.13 MiniMax-M3.1-Flash (self-refine 16; max): Mean performance 1,495 rating, mean task cost $0.22 Kimi K3.1 Pro (self-refine 1; max): Mean performance 1,933 rating, mean task cost $0.66 Kimi K3.1 Pro (self-refine 2; max): Mean performance 1,996 rating, mean task cost $1.11 Kimi K3.1 Pro (self-refine 4; max): Mean performance 2,064 rating, mean task cost $1.7 Kimi K3.1 Pro (self-refine 8; max): Mean performance 2,098 rating, mean task cost $2.72 Kimi K3.1 Pro (self-refine 16; max): Mean performance 2,132 rating, mean task cost $5.06 GPT-6.1 Terra (self-refine 2; max): Mean performance 2,661 rating, mean task cost $0.75 GPT-6.1 Terra (self-refine 4; max): Mean performance 2,826 rating, mean task cost $1.33 GPT-6.1 Terra (self-refine 8; max): Mean performance 2,843 rating, mean task cost $2.42 GPT-6.1 Terra (self-refine 16; max): Mean performance 2,875 rating, mean task cost $5.77 Gemini 4 Carbon (self-refine 1; high): Mean performance 2,375 rating, mean task cost $2.07 Gemini 4 Carbon (self-refine 2; high): Mean performance 2,631 rating, mean task cost $3.43 Gemini 4 Carbon (self-refine 4; high): Mean performance 2,834 rating, mean task cost $6.18 Gemini 4 Carbon (self-refine 8; high): Mean performance 2,929 rating, mean task cost $13.1 Gemini 4 Carbon (self-refine 16; high): Mean performance 2,897 rating, mean task cost $34.5 Claude Fable 5.2 (self-refine 1; high): Mean performance 2,510 rating, mean task cost $1.5 Claude Fable 5.2 (self-refine 2; high): Mean performance 2,615 rating, mean task cost $2.7 Claude Fable 5.2 (self-refine 4; high): Mean performance 2,787 rating, mean task cost $5.53 Claude Fable 5.2 (self-refine 8; high): Mean performance 2,889 rating, mean task cost $12.5 Claude Fable 5.2 (self-refine 16; high): Mean performance 2,986 rating, mean task cost $31.8 qwen3.7 (self-refine 8; max): Mean performance 1,548 rating, mean task cost $2.42. Not in our model list qwen3.7 (self-refine 16; max): Mean performance 1,547 rating, mean task cost $4.68. Not in our model list qwen3.7 (self-refine 4; max): Mean performance 1,462 rating, mean task cost $1.32. Not in our model list grok-4.20-beta (self-refine 16): Mean performance 1,388 rating, mean task cost $0.94. Not in our model list grok-4.20-beta (self-refine 8): Mean performance 1,348 rating, mean task cost $0.54. Not in our model list qwen3.7 (self-refine 2; max): Mean performance 1,327 rating, mean task cost $0.73. Not in our model list grok-4.20-beta (self-refine 4): Mean performance 1,260 rating, mean task cost $0.34. Not in our model list muse-glimmer-30b (self-refine 16; high): Mean performance 1,226 rating, mean task cost $0.21. Not in our model list muse-glimmer-30b (self-refine 8; high): Mean performance 1,212 rating, mean task cost $0.1. Not in our model list qwen3.7 (self-refine 1; max): Mean performance 1,189 rating, mean task cost $0.38. Not in our model list muse-glimmer-30b (self-refine 4; high): Mean performance 1,183 rating, mean task cost $0.07. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 16): Mean performance 1,175 rating, mean task cost $2.22. Not in our model list gemma-4-26b-a4b-it (self-refine 16): Mean performance 1,165 rating, mean task cost $0.12. Not in our model list grok-4.20-beta (self-refine 2): Mean performance 1,164 rating, mean task cost $0.24. Not in our model list grok-4.20-beta (self-refine 1): Mean performance 1,150 rating, mean task cost $0.14. Not in our model list muse-glimmer-30b (self-refine 2; high): Mean performance 1,117 rating, mean task cost $0.04. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 8): Mean performance 1,117 rating, mean task cost $1.3. Not in our model list gemma-4-26b-a4b-it (self-refine 8): Mean performance 1,105 rating, mean task cost $0.06. Not in our model list gemma-4-26b-a4b-it (self-refine 4): Mean performance 1,072 rating, mean task cost $0.03. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 4): Mean performance 1,071 rating, mean task cost $0.78. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 16): Mean performance 1,022 rating, mean task cost $0.58. Not in our model list gemma-4-26b-a4b-it (self-refine 2): Mean performance 998 rating, mean task cost $0.02. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 8): Mean performance 986 rating, mean task cost $0.31. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 4): Mean performance 974 rating, mean task cost $0.19. Not in our model list muse-glimmer-30b (self-refine 1; high): Mean performance 962 rating, mean task cost $0.03. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 2): Mean performance 952 rating, mean task cost $0.47. Not in our model list gemma-4-26b-a4b-it (self-refine 1): Mean performance 927 rating, mean task cost $0.01. Not in our model list qwen3 (self-refine 16; max): Mean performance 894 rating, mean task cost $1.14. Not in our model list qwen3 (self-refine 8; max): Mean performance 863 rating, mean task cost $0.36. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 2): Mean performance 842 rating, mean task cost $0.11. Not in our model list qwen3 (self-refine 4; max): Mean performance 792 rating, mean task cost $0.13. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 1): Mean performance 786 rating, mean task cost $0.27. Not in our model list kimi-k2-0905 (self-refine 16): Mean performance 776 rating, mean task cost $0.33. Not in our model list kimi-k2-0905 (self-refine 8): Mean performance 730 rating, mean task cost $0.11. Not in our model list mistral-large-3-2512 (self-refine 16): Mean performance 721 rating, mean task cost $0.26. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 1): Mean performance 662 rating, mean task cost $0.07. Not in our model list kimi-k2-0905 (self-refine 4): Mean performance 661 rating, mean task cost $0.04. Not in our model list mistral-large-3-2512 (self-refine 8): Mean performance 659 rating, mean task cost $0.08. Not in our model list qwen3 (self-refine 2; max): Mean performance 627 rating, mean task cost $0.05. Not in our model list mistral-large-3-2512 (self-refine 4): Mean performance 606 rating, mean task cost $0.03. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 16): Mean performance 542 rating, mean task cost $0.21. Not in our model list nova-premier-v1 (self-refine 16): Mean performance 529 rating, mean task cost $1.48. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 8): Mean performance 519 rating, mean task cost $0.07. Not in our model list nova-2-lite-v1 (self-refine 16): Mean performance 507 rating, mean task cost $0.18. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 4): Mean performance 500 rating, mean task cost $0.04. Not in our model list kimi-k2-0905 (self-refine 2): Mean performance 476 rating, mean task cost $0.02. Not in our model list mistral-large-3-2512 (self-refine 2): Mean performance 444 rating, mean task cost $0.01. Not in our model list nova-2-lite-v1 (self-refine 8): Mean performance 440 rating, mean task cost $0.06. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 2): Mean performance 435 rating, mean task cost $0.02. Not in our model list nova-premier-v1 (self-refine 8): Mean performance 414 rating, mean task cost $0.47. Not in our model list nova-2-lite-v1 (self-refine 4): Mean performance 372 rating, mean task cost $0.02. Not in our model list qwen3 (self-refine 1; max): Mean performance 370 rating, mean task cost $0.02. Not in our model list codestral-2508 (self-refine 16): Mean performance 341 rating, mean task cost $0.16. Not in our model list codestral-2508 (self-refine 8): Mean performance 340 rating, mean task cost $0.05. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 1): Mean performance 326 rating, mean task cost $0.01. Not in our model list codestral-2508 (self-refine 4): Mean performance 294 rating, mean task cost $0.02. Not in our model list nova-premier-v1 (self-refine 4): Mean performance 289 rating, mean task cost $0.18. Not in our model list nova-2-lite-v1 (self-refine 2): Mean performance 289 rating, mean task cost $0.01. Not in our model list kimi-k2-0905 (self-refine 1): Mean performance 267 rating, mean task cost $0.008. Not in our model list mistral-large-3-2512 (self-refine 1): Mean performance 265 rating, mean task cost $0.005. Not in our model list nova-2-lite-v1 (self-refine 1): Mean performance 236 rating, mean task cost $0.004. Not in our model list nova-premier-v1 (self-refine 2): Mean performance 204 rating, mean task cost $0.07. Not in our model list codestral-2508 (self-refine 2): Mean performance 196 rating, mean task cost $0.006. Not in our model list nova-premier-v1 (self-refine 1): Mean performance 147 rating, mean task cost $0.03. Not in our model list codestral-2508 (self-refine 1): Mean performance 138 rating, mean task cost $0.003. Not in our model list gpt-oss-120b (self-refine 1): Mean performance 576 rating, mean task cost $0.001 gpt-oss-120b (self-refine 2): Mean performance 807 rating, mean task cost $0.003 gpt-oss-20b (self-refine 1): Mean performance 566 rating, mean task cost $0.0007 gpt-oss-20b (self-refine 2): Mean performance 657 rating, mean task cost $0.002 Gemma 4 31B IT (self-refine 1): Mean performance 926 rating, mean task cost $0.005 Gemma 4 31B IT (self-refine 2): Mean performance 1,091 rating, mean task cost $0.009 DeepSeek-V4-Flash (self-refine 2; high): Mean performance 967 rating, mean task cost $0.008 DeepSeek-V4-Flash (self-refine 4; high): Mean performance 1,158 rating, mean task cost $0.01 GPT-6 Astra (self-refine 1; max): Mean performance 2,951 rating, mean task cost $2.86 GPT-6 Astra (self-refine 2; max): Mean performance 3,093 rating, mean task cost $5.38 GPT-6 Astra (self-refine 4; max): Mean performance 3,132 rating, mean task cost $10.1 GPT-6 Astra (self-refine 8; max): Mean performance 3,164 rating, mean task cost $19.5 Claude Opus 5.5 (self-refine 1; high): Mean performance 2,147 rating, mean task cost $0.27 GPT-6 Luna (self-refine 1; xhigh): Mean performance 1,577 rating, mean task cost $0.02 GPT-6 Luna (self-refine 2; xhigh): Mean performance 1,642 rating, mean task cost $0.03 GPT-6 Luna (self-refine 8; xhigh): Mean performance 1,749 rating, mean task cost $0.07 Claude Sonnet 5.5 (self-refine 2; high): Mean performance 1,953 rating, mean task cost $0.24 GPT-6.1 Sol (self-refine 1; max): Mean performance 2,744 rating, mean task cost $0.6 GPT-6.1 Sol (self-refine 2; max): Mean performance 2,878 rating, mean task cost $1.15 GPT-6.1 Sol (self-refine 4; max): Mean performance 2,921 rating, mean task cost $2.21 Claude Haiku 5.5 (self-refine 1; high): Mean performance 1,587 rating, mean task cost $0.02 Claude Haiku 5.5 (self-refine 2; high): Mean performance 1,738 rating, mean task cost $0.03 Claude Haiku 5.5 (self-refine 4; high): Mean performance 1,852 rating, mean task cost $0.07 Claude Haiku 5.5 (self-refine 8; high): Mean performance 1,931 rating, mean task cost $0.24 GPT-6.1 Terra (self-refine 1; max): Mean performance 2,609 rating, mean task cost $0.41 GPT-6 Astra (self-refine 8; max)GPT-6 Astra (self-refine 4; max)GPT-6 Astra (self-refine 2; max)GPT-6 Astra (self-refine 1; max)GPT-6.1 Sol (self-refine 4; max)GPT-6.1 Sol (self-refine 2; max)GPT-6.1 Sol (self-refine 1; max)GPT-6.1 Terra (self-refine 1; max)Claude Opus 5.5 (self-refine 1; high)Claude Sonnet 5.5 (self-refine 2; high)Claude Haiku 5.5 (self-refine 8; high)Claude Haiku 5.5 (self-refine 4; high)GPT-6 Luna (self-refine 8; xhigh)Claude Haiku 5.5 (self-refine 2; high)GPT-6 Luna (self-refine 2; xhigh)Claude Haiku 5.5 (self-refine 1; high)GPT-6 Luna (self-refine 1; xhigh)DeepSeek-V4-Flash (self-refine 4; high)Gemma 4 31B IT (self-refine 2)DeepSeek-V4-Flash (self-refine 2; high)Gemma 4 31B IT (self-refine 1)gpt-oss-120b (self-refine 2)gpt-oss-20b (self-refine 2)gpt-oss-120b (self-refine 1)gpt-oss-20b (self-refine 1) 0 1,000 2,000 3,000 4,000 $0.0001$0.001$0.01$0.1$1$10$100 Mean task cost ($), log scale, lower is better Mean performance (rating) Llama 4 Maverick (self-refine 1): Mean performance 173 rating, mean task cost $0.001 Llama 4 Maverick (self-refine 2): Mean performance 349 rating, mean task cost $0.004 Llama 4 Maverick (self-refine 4): Mean performance 499 rating, mean task cost $0.01 Llama 4 Maverick (self-refine 8): Mean performance 530 rating, mean task cost $0.03 Llama 4 Maverick (self-refine 16): Mean performance 552 rating, mean task cost $0.08 GPT-4.1 (self-refine 1): Mean performance 558 rating, mean task cost $0.03 GPT-4.1 (self-refine 2): Mean performance 679 rating, mean task cost $0.06 GPT-4.1 (self-refine 4): Mean performance 797 rating, mean task cost $0.11 GPT-4.1 (self-refine 8): Mean performance 915 rating, mean task cost $0.27 GPT-4.1 (self-refine 16): Mean performance 985 rating, mean task cost $0.76 o3 (self-refine 1; high): Mean performance 934 rating, mean task cost $0.1 o3 (self-refine 2; high): Mean performance 1,067 rating, mean task cost $0.2 o3 (self-refine 4; high): Mean performance 1,186 rating, mean task cost $0.38 o3 (self-refine 8; high): Mean performance 1,223 rating, mean task cost $0.68 o3 (self-refine 16; high): Mean performance 1,304 rating, mean task cost $1.28 o4-mini (self-refine 1; high): Mean performance 826 rating, mean task cost $0.04 o4-mini (self-refine 2; high): Mean performance 1,014 rating, mean task cost $0.1 o4-mini (self-refine 4; high): Mean performance 1,110 rating, mean task cost $0.21 o4-mini (self-refine 8; high): Mean performance 1,192 rating, mean task cost $0.42 o4-mini (self-refine 16; high): Mean performance 1,271 rating, mean task cost $0.82 Claude Sonnet 4 (self-refine 1): Mean performance 655 rating, mean task cost $0.19 Claude Sonnet 4 (self-refine 2): Mean performance 834 rating, mean task cost $0.32 Claude Sonnet 4 (self-refine 4): Mean performance 990 rating, mean task cost $0.56 Claude Sonnet 4 (self-refine 8): Mean performance 1,042 rating, mean task cost $1.08 Claude Sonnet 4 (self-refine 16): Mean performance 1,088 rating, mean task cost $2.53 DeepSeek-R1 (self-refine 1): Mean performance 804 rating, mean task cost $0.04 DeepSeek-R1 (self-refine 2): Mean performance 909 rating, mean task cost $0.07 DeepSeek-R1 (self-refine 4): Mean performance 967 rating, mean task cost $0.11 DeepSeek-R1 (self-refine 8): Mean performance 1,009 rating, mean task cost $0.19 DeepSeek-R1 (self-refine 16): Mean performance 1,059 rating, mean task cost $0.4 Qwen3-Coder-480B-A35B (self-refine 1): Mean performance 461 rating, mean task cost $0.004 Qwen3-Coder-480B-A35B (self-refine 2): Mean performance 551 rating, mean task cost $0.009 Qwen3-Coder-480B-A35B (self-refine 4): Mean performance 681 rating, mean task cost $0.02 Qwen3-Coder-480B-A35B (self-refine 8): Mean performance 784 rating, mean task cost $0.06 Qwen3-Coder-480B-A35B (self-refine 16): Mean performance 822 rating, mean task cost $0.17 GLM-4.5 (self-refine 1): Mean performance 345 rating, mean task cost $0.03 GLM-4.5 (self-refine 2): Mean performance 431 rating, mean task cost $0.05 GLM-4.5 (self-refine 4): Mean performance 509 rating, mean task cost $0.08 GLM-4.5 (self-refine 8): Mean performance 540 rating, mean task cost $0.15 GLM-4.5 (self-refine 16): Mean performance 552 rating, mean task cost $0.28 Claude Opus 4.1 (self-refine 1): Mean performance 675 rating, mean task cost $0.43 Claude Opus 4.1 (self-refine 2): Mean performance 894 rating, mean task cost $0.78 Claude Opus 4.1 (self-refine 4): Mean performance 1,014 rating, mean task cost $1.62 Claude Opus 4.1 (self-refine 8): Mean performance 1,099 rating, mean task cost $4.08 Claude Opus 4.1 (self-refine 16): Mean performance 1,183 rating, mean task cost $12.1 gpt-oss-120b (self-refine 4): Mean performance 957 rating, mean task cost $0.009 gpt-oss-120b (self-refine 8): Mean performance 1,066 rating, mean task cost $0.02 gpt-oss-120b (self-refine 16): Mean performance 1,119 rating, mean task cost $0.07 gpt-oss-20b (self-refine 4): Mean performance 775 rating, mean task cost $0.005 gpt-oss-20b (self-refine 8): Mean performance 867 rating, mean task cost $0.01 gpt-oss-20b (self-refine 16): Mean performance 942 rating, mean task cost $0.03 GPT-5 (self-refine 1): Mean performance 1,162 rating, mean task cost $0.24 GPT-5 (self-refine 2): Mean performance 1,262 rating, mean task cost $0.44 GPT-5 (self-refine 4): Mean performance 1,431 rating, mean task cost $0.75 GPT-5 (self-refine 8): Mean performance 1,491 rating, mean task cost $1.3 GPT-5 (self-refine 16): Mean performance 1,545 rating, mean task cost $2.41 GPT-5 (self-refine 1): Mean performance 808 rating, mean task cost $0.04 GPT-5 (self-refine 2): Mean performance 956 rating, mean task cost $0.09 GPT-5 (self-refine 4): Mean performance 1,029 rating, mean task cost $0.18 GPT-5 (self-refine 8): Mean performance 1,066 rating, mean task cost $0.4 GPT-5 (self-refine 16): Mean performance 1,105 rating, mean task cost $0.96 GPT-5 mini (self-refine 1): Mean performance 800 rating, mean task cost $0.02 GPT-5 mini (self-refine 2): Mean performance 994 rating, mean task cost $0.05 GPT-5 mini (self-refine 4): Mean performance 1,172 rating, mean task cost $0.11 GPT-5 mini (self-refine 8): Mean performance 1,311 rating, mean task cost $0.21 GPT-5 mini (self-refine 16): Mean performance 1,421 rating, mean task cost $0.44 GPT-5 nano (self-refine 1): Mean performance 719 rating, mean task cost $0.007 GPT-5 nano (self-refine 2): Mean performance 839 rating, mean task cost $0.02 GPT-5 nano (self-refine 4): Mean performance 916 rating, mean task cost $0.03 GPT-5 nano (self-refine 8): Mean performance 992 rating, mean task cost $0.06 GPT-5 nano (self-refine 16): Mean performance 1,016 rating, mean task cost $0.1 Mistral Medium 3.1 (self-refine 1): Mean performance 210 rating, mean task cost $0.007 Mistral Medium 3.1 (self-refine 2): Mean performance 379 rating, mean task cost $0.02 Mistral Medium 3.1 (self-refine 4): Mean performance 487 rating, mean task cost $0.04 Mistral Medium 3.1 (self-refine 8): Mean performance 611 rating, mean task cost $0.11 Mistral Medium 3.1 (self-refine 16): Mean performance 701 rating, mean task cost $0.32 grok-code-fast-1 (self-refine 1): Mean performance 588 rating, mean task cost $0.009 grok-code-fast-1 (self-refine 2): Mean performance 694 rating, mean task cost $0.02 grok-code-fast-1 (self-refine 4): Mean performance 782 rating, mean task cost $0.02 grok-code-fast-1 (self-refine 8): Mean performance 839 rating, mean task cost $0.04 grok-code-fast-1 (self-refine 16): Mean performance 853 rating, mean task cost $0.07 DeepSeek-V3.1-Terminus (self-refine 1): Mean performance 745 rating, mean task cost $0.02 DeepSeek-V3.1-Terminus (self-refine 2): Mean performance 865 rating, mean task cost $0.03 DeepSeek-V3.1-Terminus (self-refine 4): Mean performance 993 rating, mean task cost $0.05 DeepSeek-V3.1-Terminus (self-refine 8): Mean performance 1,052 rating, mean task cost $0.1 DeepSeek-V3.1-Terminus (self-refine 16): Mean performance 1,078 rating, mean task cost $0.21 Qwen3-Coder-Plus (self-refine 1): Mean performance 457 rating, mean task cost $0.01 Qwen3-Coder-Plus (self-refine 2): Mean performance 570 rating, mean task cost $0.03 Qwen3-Coder-Plus (self-refine 4): Mean performance 747 rating, mean task cost $0.06 Qwen3-Coder-Plus (self-refine 8): Mean performance 796 rating, mean task cost $0.13 Qwen3-Coder-Plus (self-refine 16): Mean performance 827 rating, mean task cost $0.29 Claude Sonnet 4.5 (self-refine 1): Mean performance 796 rating, mean task cost $0.15 Claude Sonnet 4.5 (self-refine 2): Mean performance 919 rating, mean task cost $0.29 Claude Sonnet 4.5 (self-refine 4): Mean performance 1,011 rating, mean task cost $0.53 Claude Sonnet 4.5 (self-refine 8): Mean performance 1,072 rating, mean task cost $1 Claude Sonnet 4.5 (self-refine 16): Mean performance 1,192 rating, mean task cost $2.3 GLM-4.6 (self-refine 1): Mean performance 341 rating, mean task cost $0.02 GLM-4.6 (self-refine 2): Mean performance 461 rating, mean task cost $0.03 GLM-4.6 (self-refine 4): Mean performance 724 rating, mean task cost $0.05 GLM-4.6 (self-refine 8): Mean performance 789 rating, mean task cost $0.12 GLM-4.6 (self-refine 16): Mean performance 838 rating, mean task cost $0.31 Claude Haiku 4.5 (self-refine 1): Mean performance 653 rating, mean task cost $0.07 Claude Haiku 4.5 (self-refine 2): Mean performance 823 rating, mean task cost $0.13 Claude Haiku 4.5 (self-refine 4): Mean performance 936 rating, mean task cost $0.24 Claude Haiku 4.5 (self-refine 8): Mean performance 991 rating, mean task cost $0.45 Claude Haiku 4.5 (self-refine 16): Mean performance 1,006 rating, mean task cost $0.94 Kimi K2 Thinking (self-refine 1): Mean performance 598 rating, mean task cost $0.09 Kimi K2 Thinking (self-refine 2): Mean performance 848 rating, mean task cost $0.13 Kimi K2 Thinking (self-refine 4): Mean performance 991 rating, mean task cost $0.18 Kimi K2 Thinking (self-refine 8): Mean performance 1,087 rating, mean task cost $0.3 Kimi K2 Thinking (self-refine 16): Mean performance 1,103 rating, mean task cost $0.53 GPT-5.1-Codex (self-refine 1; high): Mean performance 1,245 rating, mean task cost $0.51 GPT-5.1-Codex (self-refine 2; high): Mean performance 1,373 rating, mean task cost $0.94 GPT-5.1-Codex (self-refine 4; high): Mean performance 1,497 rating, mean task cost $1.41 GPT-5.1-Codex (self-refine 8; high): Mean performance 1,522 rating, mean task cost $1.98 GPT-5.1-Codex (self-refine 16; high): Mean performance 1,528 rating, mean task cost $2.75 GPT-5.1 (self-refine 1): Mean performance 1,192 rating, mean task cost $0.32 GPT-5.1 (self-refine 2): Mean performance 1,281 rating, mean task cost $0.55 GPT-5.1 (self-refine 4): Mean performance 1,336 rating, mean task cost $0.86 GPT-5.1 (self-refine 8): Mean performance 1,401 rating, mean task cost $1.34 GPT-5.1 (self-refine 16): Mean performance 1,425 rating, mean task cost $2.24 Gemini 3 Pro (self-refine 1; high): Mean performance 1,177 rating, mean task cost $0.26 Gemini 3 Pro (self-refine 2; high): Mean performance 1,348 rating, mean task cost $0.48 Gemini 3 Pro (self-refine 4; high): Mean performance 1,571 rating, mean task cost $0.85 Gemini 3 Pro (self-refine 8; high): Mean performance 1,675 rating, mean task cost $1.56 Gemini 3 Pro (self-refine 16; high): Mean performance 1,729 rating, mean task cost $3.03 Gemini 3 Pro (self-refine 1; low): Mean performance 988 rating, mean task cost $0.1 Gemini 3 Pro (self-refine 2; low): Mean performance 1,241 rating, mean task cost $0.24 Gemini 3 Pro (self-refine 4; low): Mean performance 1,463 rating, mean task cost $0.46 Gemini 3 Pro (self-refine 8; low): Mean performance 1,628 rating, mean task cost $0.9 Gemini 3 Pro (self-refine 16; low): Mean performance 1,715 rating, mean task cost $1.88 GPT-5.1-Codex-Max (self-refine 1; high): Mean performance 1,209 rating, mean task cost $0.25 GPT-5.1-Codex-Max (self-refine 2; high): Mean performance 1,316 rating, mean task cost $0.4 GPT-5.1-Codex-Max (self-refine 4; high): Mean performance 1,364 rating, mean task cost $0.6 GPT-5.1-Codex-Max (self-refine 8; high): Mean performance 1,436 rating, mean task cost $0.92 GPT-5.1-Codex-Max (self-refine 16; high): Mean performance 1,458 rating, mean task cost $1.52 GPT-5.1-Codex-Max (self-refine 1; xhigh): Mean performance 1,228 rating, mean task cost $0.32 GPT-5.1-Codex-Max (self-refine 2; xhigh): Mean performance 1,348 rating, mean task cost $0.52 GPT-5.1-Codex-Max (self-refine 4; xhigh): Mean performance 1,457 rating, mean task cost $0.75 GPT-5.1-Codex-Max (self-refine 8; xhigh): Mean performance 1,519 rating, mean task cost $1.13 GPT-5.1-Codex-Max (self-refine 16; xhigh): Mean performance 1,538 rating, mean task cost $1.76 Grok 4.1 Fast (self-refine 1): Mean performance 395 rating, mean task cost $0.009 Grok 4.1 Fast (self-refine 2): Mean performance 701 rating, mean task cost $0.02 Grok 4.1 Fast (self-refine 4): Mean performance 887 rating, mean task cost $0.03 Grok 4.1 Fast (self-refine 8): Mean performance 1,023 rating, mean task cost $0.05 Grok 4.1 Fast (self-refine 16): Mean performance 1,111 rating, mean task cost $0.08 Claude Opus 4.5 (self-refine 1): Mean performance 1,025 rating, mean task cost $0.29 Claude Opus 4.5 (self-refine 2): Mean performance 1,149 rating, mean task cost $0.58 Claude Opus 4.5 (self-refine 4): Mean performance 1,297 rating, mean task cost $1.28 Claude Opus 4.5 (self-refine 8): Mean performance 1,377 rating, mean task cost $2.44 Claude Opus 4.5 (self-refine 16): Mean performance 1,429 rating, mean task cost $5.38 GPT-5.2 (self-refine 1; high): Mean performance 1,294 rating, mean task cost $0.3 GPT-5.2 (self-refine 2; high): Mean performance 1,453 rating, mean task cost $0.54 GPT-5.2 (self-refine 4; high): Mean performance 1,586 rating, mean task cost $0.93 GPT-5.2 (self-refine 8; high): Mean performance 1,660 rating, mean task cost $1.66 GPT-5.2 (self-refine 16; high): Mean performance 1,715 rating, mean task cost $3.19 GPT-5.2 (self-refine 1; medium): Mean performance 1,250 rating, mean task cost $0.14 GPT-5.2 (self-refine 2; medium): Mean performance 1,372 rating, mean task cost $0.26 GPT-5.2 (self-refine 4; medium): Mean performance 1,502 rating, mean task cost $0.48 GPT-5.2 (self-refine 8; medium): Mean performance 1,616 rating, mean task cost $0.95 GPT-5.2 (self-refine 16; medium): Mean performance 1,672 rating, mean task cost $1.93 MiMo-V2-Flash (self-refine 1): Mean performance 738 rating, mean task cost $0.007 MiMo-V2-Flash (self-refine 2): Mean performance 782 rating, mean task cost $0.01 MiMo-V2-Flash (self-refine 4): Mean performance 873 rating, mean task cost $0.02 MiMo-V2-Flash (self-refine 8): Mean performance 974 rating, mean task cost $0.03 MiMo-V2-Flash (self-refine 16): Mean performance 1,000 rating, mean task cost $0.07 Gemini 3 Flash (self-refine 1; high): Mean performance 1,367 rating, mean task cost $0.09 Gemini 3 Flash (self-refine 2; high): Mean performance 1,472 rating, mean task cost $0.16 Gemini 3 Flash (self-refine 4; high): Mean performance 1,598 rating, mean task cost $0.28 Gemini 3 Flash (self-refine 8; high): Mean performance 1,632 rating, mean task cost $0.48 Gemini 3 Flash (self-refine 16; high): Mean performance 1,722 rating, mean task cost $0.85 GPT-5.2 Codex (self-refine 1; xhigh): Mean performance 1,300 rating, mean task cost $0.59 GPT-5.2 Codex (self-refine 2; xhigh): Mean performance 1,512 rating, mean task cost $1.11 GPT-5.2 Codex (self-refine 4; xhigh): Mean performance 1,599 rating, mean task cost $2.1 GPT-5.2 Codex (self-refine 8; xhigh): Mean performance 1,686 rating, mean task cost $3.85 GPT-5.2 Codex (self-refine 16; xhigh): Mean performance 1,613 rating, mean task cost $6.22 GLM-4.7 (self-refine 1): Mean performance 399 rating, mean task cost $0.04 GLM-4.7 (self-refine 2): Mean performance 560 rating, mean task cost $0.07 GLM-4.7 (self-refine 4): Mean performance 805 rating, mean task cost $0.13 GLM-4.7 (self-refine 8): Mean performance 923 rating, mean task cost $0.23 GLM-4.7 (self-refine 16): Mean performance 1,011 rating, mean task cost $0.47 MiniMax-M2.1 (self-refine 1): Mean performance 624 rating, mean task cost $0.05 MiniMax-M2.1 (self-refine 2): Mean performance 858 rating, mean task cost $0.11 MiniMax-M2.1 (self-refine 4): Mean performance 915 rating, mean task cost $0.19 MiniMax-M2.1 (self-refine 8): Mean performance 931 rating, mean task cost $0.3 MiniMax-M2.1 (self-refine 16): Mean performance 998 rating, mean task cost $0.43 Kimi K2.5 (self-refine 1): Mean performance 822 rating, mean task cost $0.05 Kimi K2.5 (self-refine 2): Mean performance 1,077 rating, mean task cost $0.08 Kimi K2.5 (self-refine 4): Mean performance 1,127 rating, mean task cost $0.13 Kimi K2.5 (self-refine 8): Mean performance 1,240 rating, mean task cost $0.2 Kimi K2.5 (self-refine 16): Mean performance 1,279 rating, mean task cost $0.33 Claude Opus 4.6 (self-refine 1; no-thinking): Mean performance 997 rating, mean task cost $0.06 Claude Opus 4.6 (self-refine 2; no-thinking): Mean performance 1,186 rating, mean task cost $0.15 Claude Opus 4.6 (self-refine 4; no-thinking): Mean performance 1,320 rating, mean task cost $0.39 Claude Opus 4.6 (self-refine 8; no-thinking): Mean performance 1,457 rating, mean task cost $1.15 Claude Opus 4.6 (self-refine 16; no-thinking): Mean performance 1,607 rating, mean task cost $3.64 GPT-5.3 Codex (self-refine 1; xhigh): Mean performance 1,655 rating, mean task cost $0.56 GPT-5.3 Codex (self-refine 2; xhigh): Mean performance 1,803 rating, mean task cost $1.06 GPT-5.3 Codex (self-refine 4; xhigh): Mean performance 1,891 rating, mean task cost $1.93 GPT-5.3 Codex (self-refine 8; xhigh): Mean performance 1,919 rating, mean task cost $3.37 GPT-5.3 Codex (self-refine 16; xhigh): Mean performance 1,934 rating, mean task cost $5.91 GLM-5 (self-refine 1): Mean performance 766 rating, mean task cost $0.07 GLM-5 (self-refine 2): Mean performance 828 rating, mean task cost $0.12 GLM-5 (self-refine 4): Mean performance 1,009 rating, mean task cost $0.19 GLM-5 (self-refine 8): Mean performance 1,090 rating, mean task cost $0.29 GLM-5 (self-refine 16): Mean performance 1,194 rating, mean task cost $0.5 MiniMax-M2.5 (self-refine 1): Mean performance 618 rating, mean task cost $0.04 MiniMax-M2.5 (self-refine 2): Mean performance 750 rating, mean task cost $0.06 MiniMax-M2.5 (self-refine 4): Mean performance 1,007 rating, mean task cost $0.1 MiniMax-M2.5 (self-refine 8): Mean performance 988 rating, mean task cost $0.14 MiniMax-M2.5 (self-refine 16): Mean performance 1,033 rating, mean task cost $0.22 Qwen3.5 397B-A17B (self-refine 1): Mean performance 622 rating, mean task cost $0.04 Qwen3.5 397B-A17B (self-refine 2): Mean performance 725 rating, mean task cost $0.06 Qwen3.5 397B-A17B (self-refine 4): Mean performance 898 rating, mean task cost $0.11 Qwen3.5 397B-A17B (self-refine 8): Mean performance 978 rating, mean task cost $0.23 Qwen3.5 397B-A17B (self-refine 16): Mean performance 1,007 rating, mean task cost $0.62 Qwen3.5-Plus (self-refine 1): Mean performance 609 rating, mean task cost $0.03 Qwen3.5-Plus (self-refine 2): Mean performance 813 rating, mean task cost $0.04 Qwen3.5-Plus (self-refine 4): Mean performance 936 rating, mean task cost $0.07 Qwen3.5-Plus (self-refine 8): Mean performance 1,039 rating, mean task cost $0.14 Qwen3.5-Plus (self-refine 16): Mean performance 1,071 rating, mean task cost $0.39 Claude Sonnet 4.6 (self-refine 1; medium): Mean performance 1,327 rating, mean task cost $1.77 Claude Sonnet 4.6 (self-refine 2; medium): Mean performance 1,487 rating, mean task cost $3.5 Claude Sonnet 4.6 (self-refine 4; medium): Mean performance 1,560 rating, mean task cost $6.6 Claude Sonnet 4.6 (self-refine 8; medium): Mean performance 1,688 rating, mean task cost $12.8 Claude Sonnet 4.6 (self-refine 16; medium): Mean performance 1,768 rating, mean task cost $27.9 Gemini 3.1 Pro (self-refine 1; high): Mean performance 1,161 rating, mean task cost $0.39 Gemini 3.1 Pro (self-refine 2; high): Mean performance 1,270 rating, mean task cost $0.79 Gemini 3.1 Pro (self-refine 4; high): Mean performance 1,677 rating, mean task cost $1.59 Gemini 3.1 Pro (self-refine 8; high): Mean performance 1,952 rating, mean task cost $3.14 Gemini 3.1 Pro (self-refine 16; high): Mean performance 2,041 rating, mean task cost $6.68 Gemini 3.1 Pro (self-refine 1; low): Mean performance 1,055 rating, mean task cost $0.04 Gemini 3.1 Pro (self-refine 2; low): Mean performance 1,223 rating, mean task cost $0.09 Gemini 3.1 Pro (self-refine 4; low): Mean performance 1,373 rating, mean task cost $0.2 Gemini 3.1 Pro (self-refine 8; low): Mean performance 1,522 rating, mean task cost $0.43 Gemini 3.1 Pro (self-refine 16; low): Mean performance 1,607 rating, mean task cost $0.93 Mercury 2 (self-refine 1): Mean performance 786 rating, mean task cost $0.007 Mercury 2 (self-refine 2): Mean performance 982 rating, mean task cost $0.02 Mercury 2 (self-refine 4): Mean performance 1,162 rating, mean task cost $0.04 Mercury 2 (self-refine 8): Mean performance 1,183 rating, mean task cost $0.08 Mercury 2 (self-refine 16): Mean performance 1,190 rating, mean task cost $0.17 Qwen3.5-Flash (self-refine 1): Mean performance 266 rating, mean task cost $0.01 Qwen3.5-Flash (self-refine 2): Mean performance 369 rating, mean task cost $0.02 Qwen3.5-Flash (self-refine 4): Mean performance 526 rating, mean task cost $0.02 Qwen3.5-Flash (self-refine 8): Mean performance 758 rating, mean task cost $0.04 Qwen3.5-Flash (self-refine 16): Mean performance 826 rating, mean task cost $0.1 Qwen3.5-27B (self-refine 1): Mean performance 349 rating, mean task cost $0.04 Qwen3.5-27B (self-refine 2): Mean performance 506 rating, mean task cost $0.06 Qwen3.5-27B (self-refine 4): Mean performance 715 rating, mean task cost $0.09 Qwen3.5-27B (self-refine 8): Mean performance 784 rating, mean task cost $0.16 Qwen3.5-27B (self-refine 16): Mean performance 882 rating, mean task cost $0.36 Qwen3.5-35B-A3B (self-refine 1): Mean performance 222 rating, mean task cost $0.04 Qwen3.5-35B-A3B (self-refine 2): Mean performance 333 rating, mean task cost $0.06 Qwen3.5-35B-A3B (self-refine 4): Mean performance 399 rating, mean task cost $0.09 Qwen3.5-35B-A3B (self-refine 8): Mean performance 582 rating, mean task cost $0.16 Qwen3.5-35B-A3B (self-refine 16): Mean performance 648 rating, mean task cost $0.36 Gemini 3.1 Flash-Lite (self-refine 1; high): Mean performance 798 rating, mean task cost $0.03 Gemini 3.1 Flash-Lite (self-refine 2; high): Mean performance 925 rating, mean task cost $0.04 Gemini 3.1 Flash-Lite (self-refine 4; high): Mean performance 1,083 rating, mean task cost $0.08 Gemini 3.1 Flash-Lite (self-refine 8; high): Mean performance 1,165 rating, mean task cost $0.14 Gemini 3.1 Flash-Lite (self-refine 16; high): Mean performance 1,110 rating, mean task cost $0.26 GPT-5.4 (self-refine 1; high): Mean performance 1,607 rating, mean task cost $0.61 GPT-5.4 (self-refine 2; high): Mean performance 1,710 rating, mean task cost $1.07 GPT-5.4 (self-refine 4; high): Mean performance 1,807 rating, mean task cost $1.69 GPT-5.4 (self-refine 8; high): Mean performance 1,752 rating, mean task cost $2.76 GPT-5.4 (self-refine 16; high): Mean performance 1,811 rating, mean task cost $5.02 GPT-5.4 (self-refine 1; medium): Mean performance 1,521 rating, mean task cost $0.26 GPT-5.4 (self-refine 2; medium): Mean performance 1,538 rating, mean task cost $0.53 GPT-5.4 (self-refine 4; medium): Mean performance 1,686 rating, mean task cost $0.94 GPT-5.4 (self-refine 8; medium): Mean performance 1,592 rating, mean task cost $1.69 GPT-5.4 (self-refine 16; medium): Mean performance 1,717 rating, mean task cost $3.32 GPT-5.4 (self-refine 1): Mean performance 1,086 rating, mean task cost $0.05 GPT-5.4 (self-refine 2): Mean performance 1,200 rating, mean task cost $0.12 GPT-5.4 (self-refine 4): Mean performance 1,304 rating, mean task cost $0.27 GPT-5.4 (self-refine 8): Mean performance 1,397 rating, mean task cost $0.65 GPT-5.4 (self-refine 16): Mean performance 1,445 rating, mean task cost $1.58 Nemotron 3 Super (self-refine 1): Mean performance 214 rating, mean task cost $0.008 Nemotron 3 Super (self-refine 2): Mean performance 494 rating, mean task cost $0.01 Nemotron 3 Super (self-refine 4): Mean performance 707 rating, mean task cost $0.02 Nemotron 3 Super (self-refine 8): Mean performance 880 rating, mean task cost $0.03 Nemotron 3 Super (self-refine 16): Mean performance 1,017 rating, mean task cost $0.07 GLM-5-Turbo (self-refine 1): Mean performance 634 rating, mean task cost $0.11 GLM-5-Turbo (self-refine 2): Mean performance 900 rating, mean task cost $0.2 GLM-5-Turbo (self-refine 4): Mean performance 1,053 rating, mean task cost $0.35 GLM-5-Turbo (self-refine 8): Mean performance 1,208 rating, mean task cost $0.62 GLM-5-Turbo (self-refine 16): Mean performance 1,313 rating, mean task cost $1.06 Mistral Small 4 (self-refine 1): Mean performance 498 rating, mean task cost $0.02 Mistral Small 4 (self-refine 2): Mean performance 588 rating, mean task cost $0.02 Mistral Small 4 (self-refine 4): Mean performance 729 rating, mean task cost $0.04 Mistral Small 4 (self-refine 8): Mean performance 787 rating, mean task cost $0.06 Mistral Small 4 (self-refine 16): Mean performance 862 rating, mean task cost $0.12 GPT-5.4 Mini (self-refine 1; high): Mean performance 1,189 rating, mean task cost $0.27 GPT-5.4 Mini (self-refine 2; high): Mean performance 1,277 rating, mean task cost $0.55 GPT-5.4 Mini (self-refine 4; high): Mean performance 1,415 rating, mean task cost $1.04 GPT-5.4 Mini (self-refine 8; high): Mean performance 1,452 rating, mean task cost $1.84 GPT-5.4 Mini (self-refine 16; high): Mean performance 1,487 rating, mean task cost $3.01 GPT-5.4 Nano (self-refine 1; high): Mean performance 1,005 rating, mean task cost $0.03 GPT-5.4 Nano (self-refine 2; high): Mean performance 1,096 rating, mean task cost $0.05 GPT-5.4 Nano (self-refine 4; high): Mean performance 1,214 rating, mean task cost $0.08 GPT-5.4 Nano (self-refine 8; high): Mean performance 1,292 rating, mean task cost $0.14 GPT-5.4 Nano (self-refine 16; high): Mean performance 1,312 rating, mean task cost $0.25 MiMo-V2-Pro (self-refine 1): Mean performance 785 rating, mean task cost $0.08 MiMo-V2-Pro (self-refine 2): Mean performance 954 rating, mean task cost $0.12 MiMo-V2-Pro (self-refine 4): Mean performance 1,122 rating, mean task cost $0.2 MiMo-V2-Pro (self-refine 8): Mean performance 1,262 rating, mean task cost $0.34 MiMo-V2-Pro (self-refine 16): Mean performance 1,347 rating, mean task cost $0.66 MiniMax-M2.7 (self-refine 1): Mean performance 599 rating, mean task cost $0.05 MiniMax-M2.7 (self-refine 2): Mean performance 711 rating, mean task cost $0.09 MiniMax-M2.7 (self-refine 4): Mean performance 821 rating, mean task cost $0.12 MiniMax-M2.7 (self-refine 8): Mean performance 991 rating, mean task cost $0.16 MiniMax-M2.7 (self-refine 16): Mean performance 1,021 rating, mean task cost $0.22 Qwen 3.6 Plus (self-refine 1): Mean performance 670 rating, mean task cost $0.06 Qwen 3.6 Plus (self-refine 2): Mean performance 840 rating, mean task cost $0.1 Qwen 3.6 Plus (self-refine 4): Mean performance 1,045 rating, mean task cost $0.21 Qwen 3.6 Plus (self-refine 8): Mean performance 1,158 rating, mean task cost $0.46 Qwen 3.6 Plus (self-refine 16): Mean performance 1,273 rating, mean task cost $1.07 Gemma 4 31B IT (self-refine 4): Mean performance 1,226 rating, mean task cost $0.02 Gemma 4 31B IT (self-refine 8): Mean performance 1,386 rating, mean task cost $0.04 Gemma 4 31B IT (self-refine 16): Mean performance 1,394 rating, mean task cost $0.11 GLM-5.1 (self-refine 1): Mean performance 887 rating, mean task cost $0.13 GLM-5.1 (self-refine 2): Mean performance 1,170 rating, mean task cost $0.29 GLM-5.1 (self-refine 4): Mean performance 1,353 rating, mean task cost $0.57 GLM-5.1 (self-refine 8): Mean performance 1,489 rating, mean task cost $1.05 GLM-5.1 (self-refine 16): Mean performance 1,646 rating, mean task cost $1.91 Claude Opus 4.7 (self-refine 1; no-thinking): Mean performance 1,323 rating, mean task cost $0.22 Claude Opus 4.7 (self-refine 2; no-thinking): Mean performance 1,521 rating, mean task cost $0.53 Claude Opus 4.7 (self-refine 4; no-thinking): Mean performance 1,695 rating, mean task cost $1.44 Claude Opus 4.7 (self-refine 8; no-thinking): Mean performance 1,836 rating, mean task cost $4.15 Claude Opus 4.7 (self-refine 16; no-thinking): Mean performance 1,897 rating, mean task cost $12 Kimi K2.6 (self-refine 1): Mean performance 1,093 rating, mean task cost $0.2 Kimi K2.6 (self-refine 2): Mean performance 1,297 rating, mean task cost $0.36 Kimi K2.6 (self-refine 4): Mean performance 1,520 rating, mean task cost $0.61 Kimi K2.6 (self-refine 8): Mean performance 1,593 rating, mean task cost $1 Kimi K2.6 (self-refine 16): Mean performance 1,695 rating, mean task cost $1.65 GPT-5.5 (self-refine 1; medium): Mean performance 1,589 rating, mean task cost $0.33 GPT-5.5 (self-refine 2; medium): Mean performance 1,735 rating, mean task cost $0.63 GPT-5.5 (self-refine 4; medium): Mean performance 1,791 rating, mean task cost $1.26 GPT-5.5 (self-refine 8; medium): Mean performance 1,892 rating, mean task cost $2.58 GPT-5.5 (self-refine 16; medium): Mean performance 1,938 rating, mean task cost $5.49 GPT-5.5 (self-refine 1): Mean performance 1,128 rating, mean task cost $0.07 GPT-5.5 (self-refine 2): Mean performance 1,394 rating, mean task cost $0.19 GPT-5.5 (self-refine 4): Mean performance 1,545 rating, mean task cost $0.47 GPT-5.5 (self-refine 8): Mean performance 1,558 rating, mean task cost $1.18 GPT-5.5 (self-refine 16): Mean performance 1,633 rating, mean task cost $2.86 GPT-5.5 (self-refine 1; xhigh): Mean performance 1,943 rating, mean task cost $1.52 GPT-5.5 (self-refine 2; xhigh): Mean performance 2,006 rating, mean task cost $2.66 GPT-5.5 (self-refine 4; xhigh): Mean performance 2,061 rating, mean task cost $4.56 GPT-5.5 (self-refine 8; xhigh): Mean performance 2,095 rating, mean task cost $8.03 GPT-5.5 (self-refine 16; xhigh): Mean performance 2,108 rating, mean task cost $14.5 MiMo-V2.5 (self-refine 1): Mean performance 514 rating, mean task cost $0.04 MiMo-V2.5 (self-refine 2): Mean performance 735 rating, mean task cost $0.06 MiMo-V2.5 (self-refine 4): Mean performance 942 rating, mean task cost $0.09 MiMo-V2.5 (self-refine 8): Mean performance 1,052 rating, mean task cost $0.15 MiMo-V2.5 (self-refine 16): Mean performance 1,106 rating, mean task cost $0.29 MiMo-V2.5-Pro (self-refine 1): Mean performance 900 rating, mean task cost $0.1 MiMo-V2.5-Pro (self-refine 2): Mean performance 1,080 rating, mean task cost $0.16 MiMo-V2.5-Pro (self-refine 4): Mean performance 1,138 rating, mean task cost $0.22 MiMo-V2.5-Pro (self-refine 8): Mean performance 1,204 rating, mean task cost $0.33 MiMo-V2.5-Pro (self-refine 16): Mean performance 1,254 rating, mean task cost $0.61 DeepSeek-V4-Flash (self-refine 1; high): Mean performance 678 rating, mean task cost $0.005 DeepSeek-V4-Flash (self-refine 8; high): Mean performance 1,296 rating, mean task cost $0.02 DeepSeek-V4-Flash (self-refine 16; high): Mean performance 1,380 rating, mean task cost $0.05 DeepSeek-V4-Flash (self-refine 1; no-thinking): Mean performance 325 rating, mean task cost $0.0009 DeepSeek-V4-Flash (self-refine 2; no-thinking): Mean performance 533 rating, mean task cost $0.002 DeepSeek-V4-Flash (self-refine 4; no-thinking): Mean performance 744 rating, mean task cost $0.006 DeepSeek-V4-Flash (self-refine 8; no-thinking): Mean performance 929 rating, mean task cost $0.01 DeepSeek-V4-Flash (self-refine 16; no-thinking): Mean performance 1,051 rating, mean task cost $0.03 DeepSeek-V4-Pro (self-refine 1; high): Mean performance 1,006 rating, mean task cost $0.15 DeepSeek-V4-Pro (self-refine 2; high): Mean performance 1,182 rating, mean task cost $0.23 DeepSeek-V4-Pro (self-refine 4; high): Mean performance 1,308 rating, mean task cost $0.35 DeepSeek-V4-Pro (self-refine 8; high): Mean performance 1,445 rating, mean task cost $0.56 DeepSeek-V4-Pro (self-refine 16; high): Mean performance 1,489 rating, mean task cost $0.96 DeepSeek-V4-Pro (self-refine 1; no-thinking): Mean performance 522 rating, mean task cost $0.02 DeepSeek-V4-Pro (self-refine 2; no-thinking): Mean performance 725 rating, mean task cost $0.05 DeepSeek-V4-Pro (self-refine 4; no-thinking): Mean performance 971 rating, mean task cost $0.1 DeepSeek-V4-Pro (self-refine 8; no-thinking): Mean performance 1,188 rating, mean task cost $0.21 DeepSeek-V4-Pro (self-refine 16; no-thinking): Mean performance 1,272 rating, mean task cost $0.45 Qwen 3.6 Flash (self-refine 1): Mean performance 326 rating, mean task cost $0.04 Qwen 3.6 Flash (self-refine 2): Mean performance 562 rating, mean task cost $0.06 Qwen 3.6 Flash (self-refine 4): Mean performance 668 rating, mean task cost $0.11 Qwen 3.6 Flash (self-refine 8): Mean performance 874 rating, mean task cost $0.19 Qwen 3.6 Flash (self-refine 16): Mean performance 1,015 rating, mean task cost $0.41 Mistral Medium 3.5 (self-refine 1): Mean performance 764 rating, mean task cost $0.27 Mistral Medium 3.5 (self-refine 2): Mean performance 896 rating, mean task cost $0.44 Mistral Medium 3.5 (self-refine 4): Mean performance 1,010 rating, mean task cost $0.69 Mistral Medium 3.5 (self-refine 8): Mean performance 1,064 rating, mean task cost $1.14 Mistral Medium 3.5 (self-refine 16): Mean performance 1,131 rating, mean task cost $1.96 Ling-2.6-1T (self-refine 1): Mean performance 433 rating, mean task cost $0.008 Ling-2.6-1T (self-refine 2): Mean performance 733 rating, mean task cost $0.02 Ling-2.6-1T (self-refine 4): Mean performance 846 rating, mean task cost $0.04 Ling-2.6-1T (self-refine 8): Mean performance 1,017 rating, mean task cost $0.08 Ling-2.6-1T (self-refine 16): Mean performance 1,108 rating, mean task cost $0.17 Grok 4.3 (self-refine 1): Mean performance 944 rating, mean task cost $0.06 Grok 4.3 (self-refine 2): Mean performance 1,088 rating, mean task cost $0.1 Grok 4.3 (self-refine 4): Mean performance 1,131 rating, mean task cost $0.18 Grok 4.3 (self-refine 8): Mean performance 1,155 rating, mean task cost $0.33 Grok 4.3 (self-refine 16): Mean performance 1,211 rating, mean task cost $0.71 Gemini 3.5 Flash (self-refine 1; high): Mean performance 911 rating, mean task cost $0.44 Gemini 3.5 Flash (self-refine 2; high): Mean performance 1,178 rating, mean task cost $0.68 Gemini 3.5 Flash (self-refine 4; high): Mean performance 1,552 rating, mean task cost $1.04 Gemini 3.5 Flash (self-refine 8; high): Mean performance 1,657 rating, mean task cost $1.73 Gemini 3.5 Flash (self-refine 16; high): Mean performance 1,712 rating, mean task cost $2.94 Claude Opus 4.8 (self-refine 1; high): Mean performance 1,564 rating, mean task cost $0.96 Claude Opus 4.8 (self-refine 2; high): Mean performance 1,690 rating, mean task cost $1.8 Claude Opus 4.8 (self-refine 4; high): Mean performance 1,802 rating, mean task cost $3.48 Claude Opus 4.8 (self-refine 8; high): Mean performance 1,970 rating, mean task cost $7.56 Claude Opus 4.8 (self-refine 16; high): Mean performance 2,102 rating, mean task cost $18.9 Claude Opus 4.8 (self-refine 1; no-thinking): Mean performance 1,412 rating, mean task cost $0.42 Claude Opus 4.8 (self-refine 2; no-thinking): Mean performance 1,613 rating, mean task cost $0.84 Claude Opus 4.8 (self-refine 4; no-thinking): Mean performance 1,799 rating, mean task cost $1.8 Claude Opus 4.8 (self-refine 8; no-thinking): Mean performance 1,820 rating, mean task cost $4.28 Claude Opus 4.8 (self-refine 16; no-thinking): Mean performance 1,913 rating, mean task cost $11.5 Step 3.7 Flash (self-refine 1; medium): Mean performance 694 rating, mean task cost $0.03 Step 3.7 Flash (self-refine 2; medium): Mean performance 855 rating, mean task cost $0.06 Step 3.7 Flash (self-refine 4; medium): Mean performance 1,059 rating, mean task cost $0.13 Step 3.7 Flash (self-refine 8; medium): Mean performance 1,236 rating, mean task cost $0.22 Step 3.7 Flash (self-refine 16; medium): Mean performance 1,277 rating, mean task cost $0.38 MiniMax-M3 (self-refine 1): Mean performance 640 rating, mean task cost $0.11 MiniMax-M3 (self-refine 2): Mean performance 773 rating, mean task cost $0.17 MiniMax-M3 (self-refine 4): Mean performance 903 rating, mean task cost $0.26 MiniMax-M3 (self-refine 8): Mean performance 1,025 rating, mean task cost $0.4 MiniMax-M3 (self-refine 16): Mean performance 1,053 rating, mean task cost $0.68 Claude Fable 5 (self-refine 1; high): Mean performance 2,041 rating, mean task cost $2.11 Claude Fable 5 (self-refine 2; high): Mean performance 2,194 rating, mean task cost $3.96 Claude Fable 5 (self-refine 4; high): Mean performance 2,326 rating, mean task cost $7.78 Claude Fable 5 (self-refine 8; high): Mean performance 2,378 rating, mean task cost $16.6 Claude Fable 5 (self-refine 16; high): Mean performance 2,463 rating, mean task cost $41 Kimi K2.7 Code (self-refine 1): Mean performance 886 rating, mean task cost $0.18 Kimi K2.7 Code (self-refine 2): Mean performance 1,046 rating, mean task cost $0.3 Kimi K2.7 Code (self-refine 4): Mean performance 1,243 rating, mean task cost $0.46 Kimi K2.7 Code (self-refine 8): Mean performance 1,316 rating, mean task cost $0.77 Kimi K2.7 Code (self-refine 16): Mean performance 1,422 rating, mean task cost $1.38 GLM-5.2 (self-refine 1; high): Mean performance 1,047 rating, mean task cost $0.16 GLM-5.2 (self-refine 2; high): Mean performance 1,201 rating, mean task cost $0.3 GLM-5.2 (self-refine 4; high): Mean performance 1,372 rating, mean task cost $0.48 GLM-5.2 (self-refine 8; high): Mean performance 1,510 rating, mean task cost $0.77 GLM-5.2 (self-refine 16; high): Mean performance 1,463 rating, mean task cost $1.32 GLM-5.2 (self-refine 1; max): Mean performance 1,010 rating, mean task cost $0.15 GLM-5.2 (self-refine 2; max): Mean performance 1,255 rating, mean task cost $0.33 GLM-5.2 (self-refine 4; max): Mean performance 1,500 rating, mean task cost $0.64 GLM-5.2 (self-refine 8; max): Mean performance 1,610 rating, mean task cost $1.13 GLM-5.2 (self-refine 16; max): Mean performance 1,685 rating, mean task cost $1.95 Claude Sonnet 5 (self-refine 1; high): Mean performance 1,463 rating, mean task cost $0.96 Claude Sonnet 5 (self-refine 2; high): Mean performance 1,533 rating, mean task cost $1.62 Claude Sonnet 5 (self-refine 4; high): Mean performance 1,610 rating, mean task cost $2.79 Claude Sonnet 5 (self-refine 8; high): Mean performance 1,717 rating, mean task cost $5.4 Claude Sonnet 5 (self-refine 16; high): Mean performance 1,783 rating, mean task cost $11.9 Tencent Hy3 (self-refine 1; high): Mean performance 839 rating, mean task cost $0.03 Tencent Hy3 (self-refine 2; high): Mean performance 1,159 rating, mean task cost $0.05 Tencent Hy3 (self-refine 4; high): Mean performance 1,321 rating, mean task cost $0.07 Tencent Hy3 (self-refine 8; high): Mean performance 1,343 rating, mean task cost $0.12 Tencent Hy3 (self-refine 16; high): Mean performance 1,516 rating, mean task cost $0.21 LongCat-2.0 (self-refine 1): Mean performance 793 rating, mean task cost $0.04 LongCat-2.0 (self-refine 2): Mean performance 908 rating, mean task cost $0.06 LongCat-2.0 (self-refine 4): Mean performance 1,110 rating, mean task cost $0.09 LongCat-2.0 (self-refine 8): Mean performance 1,212 rating, mean task cost $0.13 LongCat-2.0 (self-refine 16): Mean performance 1,204 rating, mean task cost $0.19 Grok 4.5 (self-refine 1; high): Mean performance 1,309 rating, mean task cost $0.18 Grok 4.5 (self-refine 2; high): Mean performance 1,348 rating, mean task cost $0.28 Grok 4.5 (self-refine 4; high): Mean performance 1,417 rating, mean task cost $0.4 Grok 4.5 (self-refine 8; high): Mean performance 1,467 rating, mean task cost $0.64 Grok 4.5 (self-refine 16; high): Mean performance 1,441 rating, mean task cost $1.23 GPT-5.6 Luna (self-refine 1; max): Mean performance 1,667 rating, mean task cost $0.36 GPT-5.6 Luna (self-refine 2; max): Mean performance 1,725 rating, mean task cost $0.64 GPT-5.6 Luna (self-refine 4; max): Mean performance 1,763 rating, mean task cost $0.93 GPT-5.6 Luna (self-refine 8; max): Mean performance 1,785 rating, mean task cost $1.38 GPT-5.6 Luna (self-refine 16; max): Mean performance 1,829 rating, mean task cost $2.29 GPT-5.6 Sol (self-refine 1; max): Mean performance 2,177 rating, mean task cost $1.53 GPT-5.6 Sol (self-refine 2; max): Mean performance 2,316 rating, mean task cost $2.67 GPT-5.6 Sol (self-refine 4; max): Mean performance 2,365 rating, mean task cost $4.33 GPT-5.6 Sol (self-refine 8; max): Mean performance 2,391 rating, mean task cost $7.36 GPT-5.6 Sol (self-refine 16; max): Mean performance 2,410 rating, mean task cost $15.1 GPT-5.6 Terra (self-refine 1; max): Mean performance 1,951 rating, mean task cost $1.25 GPT-5.6 Terra (self-refine 2; max): Mean performance 2,056 rating, mean task cost $2.14 GPT-5.6 Terra (self-refine 4; max): Mean performance 2,101 rating, mean task cost $3.1 GPT-5.6 Terra (self-refine 8; max): Mean performance 2,122 rating, mean task cost $4.56 GPT-5.6 Terra (self-refine 16; max): Mean performance 2,154 rating, mean task cost $7.74 Inkling (self-refine 1; max): Mean performance 946 rating, mean task cost $0.2 Inkling (self-refine 2; max): Mean performance 1,062 rating, mean task cost $0.32 Inkling (self-refine 4; max): Mean performance 1,150 rating, mean task cost $0.53 Inkling (self-refine 8; max): Mean performance 1,135 rating, mean task cost $0.95 Inkling (self-refine 16; max): Mean performance 1,195 rating, mean task cost $1.9 Kimi K3 (self-refine 1; max): Mean performance 1,524 rating, mean task cost $1.28 Kimi K3 (self-refine 2; max): Mean performance 1,633 rating, mean task cost $2.26 Kimi K3 (self-refine 4; max): Mean performance 1,788 rating, mean task cost $3.37 Kimi K3 (self-refine 8; max): Mean performance 1,908 rating, mean task cost $5.58 Kimi K3 (self-refine 16; max): Mean performance 1,991 rating, mean task cost $9.8 Gemini 3.5 Flash-Lite (self-refine 1; high): Mean performance 765 rating, mean task cost $0.05 Gemini 3.5 Flash-Lite (self-refine 2; high): Mean performance 929 rating, mean task cost $0.07 Gemini 3.5 Flash-Lite (self-refine 4; high): Mean performance 1,072 rating, mean task cost $0.11 Gemini 3.5 Flash-Lite (self-refine 8; high): Mean performance 1,111 rating, mean task cost $0.19 Gemini 3.5 Flash-Lite (self-refine 16; high): Mean performance 1,137 rating, mean task cost $0.34 Gemini 3.6 Flash (self-refine 1; high): Mean performance 716 rating, mean task cost $0.24 Gemini 3.6 Flash (self-refine 2; high): Mean performance 1,121 rating, mean task cost $0.4 Gemini 3.6 Flash (self-refine 4; high): Mean performance 1,409 rating, mean task cost $0.68 Gemini 3.6 Flash (self-refine 8; high): Mean performance 1,601 rating, mean task cost $1.22 Gemini 3.6 Flash (self-refine 16; high): Mean performance 1,825 rating, mean task cost $2.53 Claude Opus 5 (self-refine 1; high): Mean performance 2,165 rating, mean task cost $1.89 Claude Opus 5 (self-refine 2; high): Mean performance 2,340 rating, mean task cost $3.38 Claude Opus 5 (self-refine 4; high): Mean performance 2,436 rating, mean task cost $6.5 Claude Opus 5 (self-refine 8; high): Mean performance 2,582 rating, mean task cost $13.6 Claude Opus 5 (self-refine 16; high): Mean performance 2,622 rating, mean task cost $32 Qwen3.7 Flash (self-refine 1): Mean performance 420 rating, mean task cost $0.004 Qwen3.7 Flash (self-refine 2): Mean performance 601 rating, mean task cost $0.006 Qwen3.7 Flash (self-refine 4): Mean performance 798 rating, mean task cost $0.01 Qwen3.7 Flash (self-refine 8): Mean performance 872 rating, mean task cost $0.02 Qwen3.7 Flash (self-refine 16): Mean performance 946 rating, mean task cost $0.06 DeepSeek V4 Flash 0731 (self-refine 1; max): Mean performance 1,306 rating, mean task cost $0.02 DeepSeek V4 Flash 0731 (self-refine 2; max): Mean performance 1,501 rating, mean task cost $0.03 DeepSeek V4 Flash 0731 (self-refine 4; max): Mean performance 1,590 rating, mean task cost $0.04 DeepSeek V4 Flash 0731 (self-refine 8; max): Mean performance 1,686 rating, mean task cost $0.07 DeepSeek V4 Flash 0731 (self-refine 16; max): Mean performance 1,679 rating, mean task cost $0.1 Nemotron 3.5 Lightning 30B-A3B (self-refine 1): Mean performance 769 rating, mean task cost $0.009 Nemotron 3.5 Lightning 30B-A3B (self-refine 2): Mean performance 914 rating, mean task cost $0.02 Nemotron 3.5 Lightning 30B-A3B (self-refine 4): Mean performance 957 rating, mean task cost $0.03 Nemotron 3.5 Lightning 30B-A3B (self-refine 8): Mean performance 993 rating, mean task cost $0.04 Nemotron 3.5 Lightning 30B-A3B (self-refine 16): Mean performance 1,017 rating, mean task cost $0.07 Solar Pro 4 (self-refine 1; max): Mean performance 179 rating, mean task cost $0.04 Solar Pro 4 (self-refine 2; max): Mean performance 465 rating, mean task cost $0.06 Solar Pro 4 (self-refine 4; max): Mean performance 919 rating, mean task cost $0.11 Solar Pro 4 (self-refine 8; max): Mean performance 1,073 rating, mean task cost $0.2 Solar Pro 4 (self-refine 16; max): Mean performance 1,256 rating, mean task cost $0.42 Grok 4.6 (self-refine 1; xhigh): Mean performance 1,508 rating, mean task cost $0.24 Grok 4.6 (self-refine 2; xhigh): Mean performance 1,592 rating, mean task cost $0.44 Grok 4.6 (self-refine 4; xhigh): Mean performance 1,673 rating, mean task cost $0.71 Grok 4.6 (self-refine 8; xhigh): Mean performance 1,745 rating, mean task cost $1.16 Grok 4.6 (self-refine 16; xhigh): Mean performance 1,822 rating, mean task cost $2.32 DeepSeek V4 Pro 0813 (self-refine 1; max): Mean performance 1,403 rating, mean task cost $0.07 DeepSeek V4 Pro 0813 (self-refine 2; max): Mean performance 1,550 rating, mean task cost $0.12 DeepSeek V4 Pro 0813 (self-refine 4; max): Mean performance 1,628 rating, mean task cost $0.18 DeepSeek V4 Pro 0813 (self-refine 8; max): Mean performance 1,649 rating, mean task cost $0.27 DeepSeek V4 Pro 0813 (self-refine 16; max): Mean performance 1,730 rating, mean task cost $0.41 Gemini 3.7 Flash (self-refine 1; high): Mean performance 904 rating, mean task cost $0.24 Gemini 3.7 Flash (self-refine 2; high): Mean performance 1,210 rating, mean task cost $0.41 Gemini 3.7 Flash (self-refine 4; high): Mean performance 1,567 rating, mean task cost $0.68 Gemini 3.7 Flash (self-refine 8; high): Mean performance 1,840 rating, mean task cost $1.2 Gemini 3.7 Flash (self-refine 16; high): Mean performance 1,876 rating, mean task cost $2.34 GLM-5.3 (self-refine 1; high): Mean performance 1,317 rating, mean task cost $0.28 GLM-5.3 (self-refine 2; high): Mean performance 1,454 rating, mean task cost $0.5 GLM-5.3 (self-refine 4; high): Mean performance 1,724 rating, mean task cost $0.72 GLM-5.3 (self-refine 8; high): Mean performance 1,863 rating, mean task cost $1.1 GLM-5.3 (self-refine 16; high): Mean performance 1,937 rating, mean task cost $1.86 GLM-5.3-Flash (self-refine 1; high): Mean performance 304 rating, mean task cost $0.01 GLM-5.3-Flash (self-refine 2; high): Mean performance 668 rating, mean task cost $0.02 GLM-5.3-Flash (self-refine 4; high): Mean performance 961 rating, mean task cost $0.03 GLM-5.3-Flash (self-refine 8; high): Mean performance 1,136 rating, mean task cost $0.06 GLM-5.3-Flash (self-refine 16; high): Mean performance 1,283 rating, mean task cost $0.11 Claude Fable 5.1 (self-refine 1; high): Mean performance 2,143 rating, mean task cost $2.29 Claude Fable 5.1 (self-refine 2; high): Mean performance 2,287 rating, mean task cost $3.79 Claude Fable 5.1 (self-refine 4; high): Mean performance 2,470 rating, mean task cost $7.05 Claude Fable 5.1 (self-refine 8; high): Mean performance 2,581 rating, mean task cost $15.3 Claude Fable 5.1 (self-refine 16; high): Mean performance 2,849 rating, mean task cost $39 Qwen3.8 Max (0902) (self-refine 1; xhigh): Mean performance 1,679 rating, mean task cost $0.53 Qwen3.8 Max (0902) (self-refine 2; xhigh): Mean performance 1,744 rating, mean task cost $0.96 Qwen3.8 Max (0902) (self-refine 4; xhigh): Mean performance 1,848 rating, mean task cost $1.5 Qwen3.8 Max (0902) (self-refine 8; xhigh): Mean performance 1,908 rating, mean task cost $2.43 Qwen3.8 Max (0902) (self-refine 16; xhigh): Mean performance 1,964 rating, mean task cost $4.36 Gemini 3.8 Flash (self-refine 1; high): Mean performance 1,270 rating, mean task cost $0.34 Gemini 3.8 Flash (self-refine 2; high): Mean performance 1,637 rating, mean task cost $0.58 Gemini 3.8 Flash (self-refine 4; high): Mean performance 2,007 rating, mean task cost $0.96 Gemini 3.8 Flash (self-refine 8; high): Mean performance 2,086 rating, mean task cost $1.67 Gemini 3.8 Flash (self-refine 16; high): Mean performance 2,182 rating, mean task cost $3.18 Muse Spark 1.3 (self-refine 1; xhigh): Mean performance 1,688 rating, mean task cost $0.22 Muse Spark 1.3 (self-refine 2; xhigh): Mean performance 1,765 rating, mean task cost $0.4 Muse Spark 1.3 (self-refine 4; xhigh): Mean performance 1,837 rating, mean task cost $0.68 Muse Spark 1.3 (self-refine 8; xhigh): Mean performance 1,924 rating, mean task cost $1.37 Muse Spark 1.3 (self-refine 16; xhigh): Mean performance 1,981 rating, mean task cost $2.95 GPT-6 Astra (self-refine 16; max): Mean performance 3,128 rating, mean task cost $46.7 Mercury 2.5 (self-refine 1; high): Mean performance 302 rating, mean task cost $0.006 Mercury 2.5 (self-refine 2; high): Mean performance 464 rating, mean task cost $0.01 Mercury 2.5 (self-refine 4; high): Mean performance 665 rating, mean task cost $0.02 Mercury 2.5 (self-refine 8; high): Mean performance 897 rating, mean task cost $0.04 Mercury 2.5 (self-refine 16; high): Mean performance 977 rating, mean task cost $0.1 DeepSeek V4.1 Flash (self-refine 1; max): Mean performance 1,092 rating, mean task cost $0.08 DeepSeek V4.1 Flash (self-refine 2; max): Mean performance 1,287 rating, mean task cost $0.14 DeepSeek V4.1 Flash (self-refine 4; max): Mean performance 1,476 rating, mean task cost $0.24 DeepSeek V4.1 Flash (self-refine 8; max): Mean performance 1,727 rating, mean task cost $0.39 DeepSeek V4.1 Flash (self-refine 16; max): Mean performance 1,829 rating, mean task cost $0.65 Grok 4.7 (self-refine 1; xhigh): Mean performance 1,546 rating, mean task cost $0.67 Grok 4.7 (self-refine 2; xhigh): Mean performance 1,601 rating, mean task cost $1.12 Grok 4.7 (self-refine 4; xhigh): Mean performance 1,650 rating, mean task cost $1.73 Grok 4.7 (self-refine 8; xhigh): Mean performance 1,732 rating, mean task cost $3.24 Grok 4.7 (self-refine 16; xhigh): Mean performance 1,823 rating, mean task cost $7.12 MiMo-V2.6 Flash (self-refine 1): Mean performance 1,244 rating, mean task cost $0.04 MiMo-V2.6 Flash (self-refine 2): Mean performance 1,321 rating, mean task cost $0.06 MiMo-V2.6 Flash (self-refine 4): Mean performance 1,349 rating, mean task cost $0.08 MiMo-V2.6 Flash (self-refine 8): Mean performance 1,406 rating, mean task cost $0.1 MiMo-V2.6 Flash (self-refine 16): Mean performance 1,455 rating, mean task cost $0.15 MiMo-V2.6 Pro (self-refine 1): Mean performance 1,158 rating, mean task cost $0.11 MiMo-V2.6 Pro (self-refine 2): Mean performance 1,270 rating, mean task cost $0.19 MiMo-V2.6 Pro (self-refine 4): Mean performance 1,350 rating, mean task cost $0.26 MiMo-V2.6 Pro (self-refine 8): Mean performance 1,476 rating, mean task cost $0.36 MiMo-V2.6 Pro (self-refine 16): Mean performance 1,441 rating, mean task cost $0.48 Claude Opus 5.5 (self-refine 2; high): Mean performance 2,335 rating, mean task cost $0.52 Claude Opus 5.5 (self-refine 4; high): Mean performance 2,504 rating, mean task cost $1.16 Claude Opus 5.5 (self-refine 8; high): Mean performance 2,670 rating, mean task cost $2.93 Claude Opus 5.5 (self-refine 16; high): Mean performance 2,748 rating, mean task cost $8.59 GPT-6 Luna (self-refine 4; xhigh): Mean performance 1,714 rating, mean task cost $0.04 GPT-6 Luna (self-refine 16; xhigh): Mean performance 1,791 rating, mean task cost $0.12 GPT-6 Sol (self-refine 1; max): Mean performance 2,462 rating, mean task cost $0.84 GPT-6 Sol (self-refine 2; max): Mean performance 2,573 rating, mean task cost $1.42 GPT-6 Sol (self-refine 4; max): Mean performance 2,629 rating, mean task cost $2.39 GPT-6 Sol (self-refine 8; max): Mean performance 2,633 rating, mean task cost $4.11 GPT-6 Sol (self-refine 16; max): Mean performance 2,637 rating, mean task cost $8.42 Claude Sonnet 5.5 (self-refine 1; high): Mean performance 1,819 rating, mean task cost $0.11 Claude Sonnet 5.5 (self-refine 4; high): Mean performance 2,151 rating, mean task cost $0.57 Claude Sonnet 5.5 (self-refine 8; high): Mean performance 2,289 rating, mean task cost $1.49 Claude Sonnet 5.5 (self-refine 16; high): Mean performance 2,448 rating, mean task cost $4.05 GPT-6.1 Sol (self-refine 8; max): Mean performance 2,942 rating, mean task cost $4.29 GPT-6.1 Sol (self-refine 16; max): Mean performance 2,990 rating, mean task cost $10.3 Gemini 4 Argon (self-refine 1; high): Mean performance 2,100 rating, mean task cost $1.95 Gemini 4 Argon (self-refine 2; high): Mean performance 2,359 rating, mean task cost $3.17 Gemini 4 Argon (self-refine 4; high): Mean performance 2,675 rating, mean task cost $4.92 Gemini 4 Argon (self-refine 8; high): Mean performance 2,805 rating, mean task cost $8.1 Gemini 4 Argon (self-refine 16; high): Mean performance 2,747 rating, mean task cost $15.4 Gemini 4 Argon (self-refine 1; low): Mean performance 1,705 rating, mean task cost $0.17 Gemini 4 Argon (self-refine 2; low): Mean performance 1,990 rating, mean task cost $0.41 Gemini 4 Argon (self-refine 4; low): Mean performance 2,298 rating, mean task cost $1.06 Gemini 4 Argon (self-refine 8; low): Mean performance 2,481 rating, mean task cost $3.02 Gemini 4 Argon (self-refine 16; low): Mean performance 2,599 rating, mean task cost $6.85 Beam 1.0 (self-refine 1; max): Mean performance 1,098 rating, mean task cost $0.03 Beam 1.0 (self-refine 2; max): Mean performance 1,139 rating, mean task cost $0.05 Beam 1.0 (self-refine 4; max): Mean performance 1,304 rating, mean task cost $0.08 Beam 1.0 (self-refine 8; max): Mean performance 1,430 rating, mean task cost $0.14 Beam 1.0 (self-refine 16; max): Mean performance 1,493 rating, mean task cost $0.26 Mistral Large 4 (self-refine 1): Mean performance 1,213 rating, mean task cost $0.46 Mistral Large 4 (self-refine 2): Mean performance 1,332 rating, mean task cost $0.73 Mistral Large 4 (self-refine 4): Mean performance 1,492 rating, mean task cost $1.45 Mistral Large 4 (self-refine 8): Mean performance 1,589 rating, mean task cost $3.08 Mistral Large 4 (self-refine 16): Mean performance 1,657 rating, mean task cost $7.37 Claude Haiku 5.5 (self-refine 16; high): Mean performance 1,996 rating, mean task cost $0.87 Muse Spark Open (self-refine 1; xhigh): Mean performance 1,590 rating, mean task cost $0.33 Muse Spark Open (self-refine 2; xhigh): Mean performance 1,640 rating, mean task cost $0.52 Muse Spark Open (self-refine 4; xhigh): Mean performance 1,704 rating, mean task cost $0.75 Muse Spark Open (self-refine 8; xhigh): Mean performance 1,783 rating, mean task cost $1.1 Muse Spark Open (self-refine 16; xhigh): Mean performance 1,830 rating, mean task cost $1.72 Nemotron 3.5 Super VL (self-refine 1): Mean performance 957 rating, mean task cost $0.02 Nemotron 3.5 Super VL (self-refine 2): Mean performance 1,142 rating, mean task cost $0.03 Nemotron 3.5 Super VL (self-refine 4): Mean performance 1,152 rating, mean task cost $0.04 Nemotron 3.5 Super VL (self-refine 8): Mean performance 1,276 rating, mean task cost $0.05 Nemotron 3.5 Super VL (self-refine 16): Mean performance 1,352 rating, mean task cost $0.08 DeepSeek V4.1 Flash 1017 (self-refine 1; max): Mean performance 1,622 rating, mean task cost $0.1 DeepSeek V4.1 Flash 1017 (self-refine 2; max): Mean performance 1,677 rating, mean task cost $0.17 DeepSeek V4.1 Flash 1017 (self-refine 4; max): Mean performance 1,769 rating, mean task cost $0.29 DeepSeek V4.1 Flash 1017 (self-refine 8; max): Mean performance 1,872 rating, mean task cost $0.47 DeepSeek V4.1 Flash 1017 (self-refine 16; max): Mean performance 2,012 rating, mean task cost $0.76 Mercury 3 (self-refine 1; high): Mean performance 1,209 rating, mean task cost $0.05 Mercury 3 (self-refine 2; high): Mean performance 1,360 rating, mean task cost $0.08 Mercury 3 (self-refine 4; high): Mean performance 1,483 rating, mean task cost $0.12 Mercury 3 (self-refine 8; high): Mean performance 1,561 rating, mean task cost $0.17 Mercury 3 (self-refine 16; high): Mean performance 1,692 rating, mean task cost $0.28 MiniMax-M3.1-Flash (self-refine 1; max): Mean performance 218 rating, mean task cost $0.05 MiniMax-M3.1-Flash (self-refine 2; max): Mean performance 671 rating, mean task cost $0.07 MiniMax-M3.1-Flash (self-refine 4; max): Mean performance 1,041 rating, mean task cost $0.09 MiniMax-M3.1-Flash (self-refine 8; max): Mean performance 1,314 rating, mean task cost $0.13 MiniMax-M3.1-Flash (self-refine 16; max): Mean performance 1,495 rating, mean task cost $0.22 Kimi K3.1 Pro (self-refine 1; max): Mean performance 1,933 rating, mean task cost $0.66 Kimi K3.1 Pro (self-refine 2; max): Mean performance 1,996 rating, mean task cost $1.11 Kimi K3.1 Pro (self-refine 4; max): Mean performance 2,064 rating, mean task cost $1.7 Kimi K3.1 Pro (self-refine 8; max): Mean performance 2,098 rating, mean task cost $2.72 Kimi K3.1 Pro (self-refine 16; max): Mean performance 2,132 rating, mean task cost $5.06 GPT-6.1 Terra (self-refine 2; max): Mean performance 2,661 rating, mean task cost $0.75 GPT-6.1 Terra (self-refine 4; max): Mean performance 2,826 rating, mean task cost $1.33 GPT-6.1 Terra (self-refine 8; max): Mean performance 2,843 rating, mean task cost $2.42 GPT-6.1 Terra (self-refine 16; max): Mean performance 2,875 rating, mean task cost $5.77 Gemini 4 Carbon (self-refine 1; high): Mean performance 2,375 rating, mean task cost $2.07 Gemini 4 Carbon (self-refine 2; high): Mean performance 2,631 rating, mean task cost $3.43 Gemini 4 Carbon (self-refine 4; high): Mean performance 2,834 rating, mean task cost $6.18 Gemini 4 Carbon (self-refine 8; high): Mean performance 2,929 rating, mean task cost $13.1 Gemini 4 Carbon (self-refine 16; high): Mean performance 2,897 rating, mean task cost $34.5 Claude Fable 5.2 (self-refine 1; high): Mean performance 2,510 rating, mean task cost $1.5 Claude Fable 5.2 (self-refine 2; high): Mean performance 2,615 rating, mean task cost $2.7 Claude Fable 5.2 (self-refine 4; high): Mean performance 2,787 rating, mean task cost $5.53 Claude Fable 5.2 (self-refine 8; high): Mean performance 2,889 rating, mean task cost $12.5 Claude Fable 5.2 (self-refine 16; high): Mean performance 2,986 rating, mean task cost $31.8 qwen3.7 (self-refine 8; max): Mean performance 1,548 rating, mean task cost $2.42. Not in our model list qwen3.7 (self-refine 16; max): Mean performance 1,547 rating, mean task cost $4.68. Not in our model list qwen3.7 (self-refine 4; max): Mean performance 1,462 rating, mean task cost $1.32. Not in our model list grok-4.20-beta (self-refine 16): Mean performance 1,388 rating, mean task cost $0.94. Not in our model list grok-4.20-beta (self-refine 8): Mean performance 1,348 rating, mean task cost $0.54. Not in our model list qwen3.7 (self-refine 2; max): Mean performance 1,327 rating, mean task cost $0.73. Not in our model list grok-4.20-beta (self-refine 4): Mean performance 1,260 rating, mean task cost $0.34. Not in our model list muse-glimmer-30b (self-refine 16; high): Mean performance 1,226 rating, mean task cost $0.21. Not in our model list muse-glimmer-30b (self-refine 8; high): Mean performance 1,212 rating, mean task cost $0.1. Not in our model list qwen3.7 (self-refine 1; max): Mean performance 1,189 rating, mean task cost $0.38. Not in our model list muse-glimmer-30b (self-refine 4; high): Mean performance 1,183 rating, mean task cost $0.07. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 16): Mean performance 1,175 rating, mean task cost $2.22. Not in our model list gemma-4-26b-a4b-it (self-refine 16): Mean performance 1,165 rating, mean task cost $0.12. Not in our model list grok-4.20-beta (self-refine 2): Mean performance 1,164 rating, mean task cost $0.24. Not in our model list grok-4.20-beta (self-refine 1): Mean performance 1,150 rating, mean task cost $0.14. Not in our model list muse-glimmer-30b (self-refine 2; high): Mean performance 1,117 rating, mean task cost $0.04. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 8): Mean performance 1,117 rating, mean task cost $1.3. Not in our model list gemma-4-26b-a4b-it (self-refine 8): Mean performance 1,105 rating, mean task cost $0.06. Not in our model list gemma-4-26b-a4b-it (self-refine 4): Mean performance 1,072 rating, mean task cost $0.03. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 4): Mean performance 1,071 rating, mean task cost $0.78. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 16): Mean performance 1,022 rating, mean task cost $0.58. Not in our model list gemma-4-26b-a4b-it (self-refine 2): Mean performance 998 rating, mean task cost $0.02. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 8): Mean performance 986 rating, mean task cost $0.31. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 4): Mean performance 974 rating, mean task cost $0.19. Not in our model list muse-glimmer-30b (self-refine 1; high): Mean performance 962 rating, mean task cost $0.03. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 2): Mean performance 952 rating, mean task cost $0.47. Not in our model list gemma-4-26b-a4b-it (self-refine 1): Mean performance 927 rating, mean task cost $0.01. Not in our model list qwen3 (self-refine 16; max): Mean performance 894 rating, mean task cost $1.14. Not in our model list qwen3 (self-refine 8; max): Mean performance 863 rating, mean task cost $0.36. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 2): Mean performance 842 rating, mean task cost $0.11. Not in our model list qwen3 (self-refine 4; max): Mean performance 792 rating, mean task cost $0.13. Not in our model list gemini-2.5-pro-thinking (snapshot unspecified) (self-refine 1): Mean performance 786 rating, mean task cost $0.27. Not in our model list kimi-k2-0905 (self-refine 16): Mean performance 776 rating, mean task cost $0.33. Not in our model list kimi-k2-0905 (self-refine 8): Mean performance 730 rating, mean task cost $0.11. Not in our model list mistral-large-3-2512 (self-refine 16): Mean performance 721 rating, mean task cost $0.26. Not in our model list gemini-2.5-flash-thinking (snapshot unspecified) (self-refine 1): Mean performance 662 rating, mean task cost $0.07. Not in our model list kimi-k2-0905 (self-refine 4): Mean performance 661 rating, mean task cost $0.04. Not in our model list mistral-large-3-2512 (self-refine 8): Mean performance 659 rating, mean task cost $0.08. Not in our model list qwen3 (self-refine 2; max): Mean performance 627 rating, mean task cost $0.05. Not in our model list mistral-large-3-2512 (self-refine 4): Mean performance 606 rating, mean task cost $0.03. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 16): Mean performance 542 rating, mean task cost $0.21. Not in our model list nova-premier-v1 (self-refine 16): Mean performance 529 rating, mean task cost $1.48. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 8): Mean performance 519 rating, mean task cost $0.07. Not in our model list nova-2-lite-v1 (self-refine 16): Mean performance 507 rating, mean task cost $0.18. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 4): Mean performance 500 rating, mean task cost $0.04. Not in our model list kimi-k2-0905 (self-refine 2): Mean performance 476 rating, mean task cost $0.02. Not in our model list mistral-large-3-2512 (self-refine 2): Mean performance 444 rating, mean task cost $0.01. Not in our model list nova-2-lite-v1 (self-refine 8): Mean performance 440 rating, mean task cost $0.06. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 2): Mean performance 435 rating, mean task cost $0.02. Not in our model list nova-premier-v1 (self-refine 8): Mean performance 414 rating, mean task cost $0.47. Not in our model list nova-2-lite-v1 (self-refine 4): Mean performance 372 rating, mean task cost $0.02. Not in our model list qwen3 (self-refine 1; max): Mean performance 370 rating, mean task cost $0.02. Not in our model list codestral-2508 (self-refine 16): Mean performance 341 rating, mean task cost $0.16. Not in our model list codestral-2508 (self-refine 8): Mean performance 340 rating, mean task cost $0.05. Not in our model list gemini-2.5-flash-lite-thinking (snapshot unspecified) (self-refine 1): Mean performance 326 rating, mean task cost $0.01. Not in our model list codestral-2508 (self-refine 4): Mean performance 294 rating, mean task cost $0.02. Not in our model list nova-premier-v1 (self-refine 4): Mean performance 289 rating, mean task cost $0.18. Not in our model list nova-2-lite-v1 (self-refine 2): Mean performance 289 rating, mean task cost $0.01. Not in our model list kimi-k2-0905 (self-refine 1): Mean performance 267 rating, mean task cost $0.008. Not in our model list mistral-large-3-2512 (self-refine 1): Mean performance 265 rating, mean task cost $0.005. Not in our model list nova-2-lite-v1 (self-refine 1): Mean performance 236 rating, mean task cost $0.004. Not in our model list nova-premier-v1 (self-refine 2): Mean performance 204 rating, mean task cost $0.07. Not in our model list codestral-2508 (self-refine 2): Mean performance 196 rating, mean task cost $0.006. Not in our model list nova-premier-v1 (self-refine 1): Mean performance 147 rating, mean task cost $0.03. Not in our model list codestral-2508 (self-refine 1): Mean performance 138 rating, mean task cost $0.003. Not in our model list gpt-oss-120b (self-refine 1): Mean performance 576 rating, mean task cost $0.001 gpt-oss-120b (self-refine 2): Mean performance 807 rating, mean task cost $0.003 gpt-oss-20b (self-refine 1): Mean performance 566 rating, mean task cost $0.0007 gpt-oss-20b (self-refine 2): Mean performance 657 rating, mean task cost $0.002 Gemma 4 31B IT (self-refine 1): Mean performance 926 rating, mean task cost $0.005 Gemma 4 31B IT (self-refine 2): Mean performance 1,091 rating, mean task cost $0.009 DeepSeek-V4-Flash (self-refine 2; high): Mean performance 967 rating, mean task cost $0.008 DeepSeek-V4-Flash (self-refine 4; high): Mean performance 1,158 rating, mean task cost $0.01 GPT-6 Astra (self-refine 1; max): Mean performance 2,951 rating, mean task cost $2.86 GPT-6 Astra (self-refine 2; max): Mean performance 3,093 rating, mean task cost $5.38 GPT-6 Astra (self-refine 4; max): Mean performance 3,132 rating, mean task cost $10.1 GPT-6 Astra (self-refine 8; max): Mean performance 3,164 rating, mean task cost $19.5 Claude Opus 5.5 (self-refine 1; high): Mean performance 2,147 rating, mean task cost $0.27 GPT-6 Luna (self-refine 1; xhigh): Mean performance 1,577 rating, mean task cost $0.02 GPT-6 Luna (self-refine 2; xhigh): Mean performance 1,642 rating, mean task cost $0.03 GPT-6 Luna (self-refine 8; xhigh): Mean performance 1,749 rating, mean task cost $0.07 Claude Sonnet 5.5 (self-refine 2; high): Mean performance 1,953 rating, mean task cost $0.24 GPT-6.1 Sol (self-refine 1; max): Mean performance 2,744 rating, mean task cost $0.6 GPT-6.1 Sol (self-refine 2; max): Mean performance 2,878 rating, mean task cost $1.15 GPT-6.1 Sol (self-refine 4; max): Mean performance 2,921 rating, mean task cost $2.21 Claude Haiku 5.5 (self-refine 1; high): Mean performance 1,587 rating, mean task cost $0.02 Claude Haiku 5.5 (self-refine 2; high): Mean performance 1,738 rating, mean task cost $0.03 Claude Haiku 5.5 (self-refine 4; high): Mean performance 1,852 rating, mean task cost $0.07 Claude Haiku 5.5 (self-refine 8; high): Mean performance 1,931 rating, mean task cost $0.24 GPT-6.1 Terra (self-refine 1; max): Mean performance 2,609 rating, mean task cost $0.41 GPT-6 Astra (self-refine 8; max)GPT-6 Astra (self-refine 4; max)GPT-6 Astra (self-refine 2; max)GPT-6 Astra (self-refine 1; max)GPT-6.1 Sol (self-refine 4; max)GPT-6.1 Sol (self-refine 2; max)GPT-6.1 Sol (self-refine 1; max)GPT-6.1 Terra (self-refine 1; max)Claude Opus 5.5 (self-refine 1; high)Claude Sonnet 5.5 (self-refine 2; high)Claude Haiku 5.5 (self-refine 8; high)Claude Haiku 5.5 (self-refine 4; high)GPT-6 Luna (self-refine 8; xhigh)Claude Haiku 5.5 (self-refine 2; high)GPT-6 Luna (self-refine 2; xhigh)Claude Haiku 5.5 (self-refine 1; high)GPT-6 Luna (self-refine 1; xhigh)DeepSeek-V4-Flash (self-refine 4; high)Gemma 4 31B IT (self-refine 2)DeepSeek-V4-Flash (self-refine 2; high)Gemma 4 31B IT (self-refine 1)gpt-oss-120b (self-refine 2)gpt-oss-20b (self-refine 2)gpt-oss-120b (self-refine 1)gpt-oss-20b (self-refine 1)
ModelSettingMean performanceMean task cost
GPT-6 Astra self-refine 8; max 3,164 rating $19.51
GPT-6 Astra self-refine 4; max 3,132 rating $10.08
GPT-6 Astra self-refine 16; max 3,128 rating $46.69
GPT-6 Astra self-refine 2; max 3,093 rating $5.38
GPT-6.1 Sol self-refine 16; max 2,990 rating $10.35
Claude Fable 5.2 self-refine 16; high 2,986 rating $31.81
GPT-6 Astra self-refine 1; max 2,951 rating $2.86
GPT-6.1 Sol self-refine 8; max 2,942 rating $4.29
Gemini 4 Carbon self-refine 8; high 2,929 rating $13.11
GPT-6.1 Sol self-refine 4; max 2,921 rating $2.21
Gemini 4 Carbon self-refine 16; high 2,897 rating $34.55
Claude Fable 5.2 self-refine 8; high 2,889 rating $12.5
GPT-6.1 Sol self-refine 2; max 2,878 rating $1.15
GPT-6.1 Terra self-refine 16; max 2,875 rating $5.77
Claude Fable 5.1 self-refine 16; high 2,849 rating $38.97
GPT-6.1 Terra self-refine 8; max 2,843 rating $2.42
Gemini 4 Carbon self-refine 4; high 2,834 rating $6.19
GPT-6.1 Terra self-refine 4; max 2,826 rating $1.33
Gemini 4 Argon self-refine 8; high 2,805 rating $8.1
Claude Fable 5.2 self-refine 4; high 2,787 rating $5.53
Claude Opus 5.5 self-refine 16; high 2,748 rating $8.59
Gemini 4 Argon self-refine 16; high 2,747 rating $15.44
GPT-6.1 Sol self-refine 1; max 2,744 rating $0.6
Gemini 4 Argon self-refine 4; high 2,675 rating $4.92
Claude Opus 5.5 self-refine 8; high 2,670 rating $2.93
GPT-6.1 Terra self-refine 2; max 2,661 rating $0.75
GPT-6 Sol self-refine 16; max 2,637 rating $8.42
GPT-6 Sol self-refine 8; max 2,633 rating $4.11
Gemini 4 Carbon self-refine 2; high 2,631 rating $3.43
GPT-6 Sol self-refine 4; max 2,629 rating $2.39
Claude Opus 5 self-refine 16; high 2,622 rating $32.04
Claude Fable 5.2 self-refine 2; high 2,615 rating $2.7
GPT-6.1 Terra self-refine 1; max 2,609 rating $0.41
Gemini 4 Argon self-refine 16; low 2,599 rating $6.85
Claude Opus 5 self-refine 8; high 2,582 rating $13.6
Claude Fable 5.1 self-refine 8; high 2,581 rating $15.35
GPT-6 Sol self-refine 2; max 2,573 rating $1.42
Claude Fable 5.2 self-refine 1; high 2,510 rating $1.5
Claude Opus 5.5 self-refine 4; high 2,504 rating $1.16
Gemini 4 Argon self-refine 8; low 2,481 rating $3.02
Claude Fable 5.1 self-refine 4; high 2,470 rating $7.05
Claude Fable 5 self-refine 16; high 2,463 rating $40.97
GPT-6 Sol self-refine 1; max 2,462 rating $0.84
Claude Sonnet 5.5 self-refine 16; high 2,448 rating $4.05
Claude Opus 5 self-refine 4; high 2,436 rating $6.5
GPT-5.6 Sol self-refine 16; max 2,410 rating $15.14
GPT-5.6 Sol self-refine 8; max 2,391 rating $7.36
Claude Fable 5 self-refine 8; high 2,378 rating $16.65
Gemini 4 Carbon self-refine 1; high 2,375 rating $2.07
GPT-5.6 Sol self-refine 4; max 2,365 rating $4.33
Gemini 4 Argon self-refine 2; high 2,359 rating $3.17
Claude Opus 5 self-refine 2; high 2,340 rating $3.39
Claude Opus 5.5 self-refine 2; high 2,335 rating $0.52
Claude Fable 5 self-refine 4; high 2,326 rating $7.78
GPT-5.6 Sol self-refine 2; max 2,316 rating $2.67
Gemini 4 Argon self-refine 4; low 2,298 rating $1.06
Claude Sonnet 5.5 self-refine 8; high 2,289 rating $1.49
Claude Fable 5.1 self-refine 2; high 2,287 rating $3.79
Claude Fable 5 self-refine 2; high 2,194 rating $3.96
Gemini 3.8 Flash self-refine 16; high 2,182 rating $3.18
GPT-5.6 Sol self-refine 1; max 2,177 rating $1.53
Claude Opus 5 self-refine 1; high 2,165 rating $1.89
GPT-5.6 Terra self-refine 16; max 2,154 rating $7.74
Claude Sonnet 5.5 self-refine 4; high 2,151 rating $0.57
Claude Opus 5.5 self-refine 1; high 2,147 rating $0.27
Claude Fable 5.1 self-refine 1; high 2,143 rating $2.3
Kimi K3.1 Pro self-refine 16; max 2,132 rating $5.06
GPT-5.6 Terra self-refine 8; max 2,122 rating $4.56
GPT-5.5 self-refine 16; xhigh 2,108 rating $14.53
Claude Opus 4.8 self-refine 16; high 2,102 rating $18.93
GPT-5.6 Terra self-refine 4; max 2,101 rating $3.11
Gemini 4 Argon self-refine 1; high 2,100 rating $1.95
Kimi K3.1 Pro self-refine 8; max 2,098 rating $2.72
GPT-5.5 self-refine 8; xhigh 2,095 rating $8.03
Gemini 3.8 Flash self-refine 8; high 2,086 rating $1.67
Kimi K3.1 Pro self-refine 4; max 2,064 rating $1.7
GPT-5.5 self-refine 4; xhigh 2,061 rating $4.56
GPT-5.6 Terra self-refine 2; max 2,056 rating $2.14
Claude Fable 5 self-refine 1; high 2,041 rating $2.11
Gemini 3.1 Pro self-refine 16; high 2,041 rating $6.68
DeepSeek V4.1 Flash 1017 self-refine 16; max 2,012 rating $0.76
Gemini 3.8 Flash self-refine 4; high 2,007 rating $0.96
GPT-5.5 self-refine 2; xhigh 2,006 rating $2.66
Kimi K3.1 Pro self-refine 2; max 1,996 rating $1.11
Claude Haiku 5.5 self-refine 16; high 1,996 rating $0.87
Kimi K3 self-refine 16; max 1,991 rating $9.8
Gemini 4 Argon self-refine 2; low 1,990 rating $0.41
Muse Spark 1.3 self-refine 16; xhigh 1,981 rating $2.95
Claude Opus 4.8 self-refine 8; high 1,970 rating $7.56
Qwen3.8 Max (0902) self-refine 16; xhigh 1,964 rating $4.36
Claude Sonnet 5.5 self-refine 2; high 1,953 rating $0.24
Gemini 3.1 Pro self-refine 8; high 1,952 rating $3.14
GPT-5.6 Terra self-refine 1; max 1,951 rating $1.25
GPT-5.5 self-refine 1; xhigh 1,943 rating $1.52
GPT-5.5 self-refine 16; medium 1,938 rating $5.49
GLM-5.3 self-refine 16; high 1,937 rating $1.86
GPT-5.3 Codex self-refine 16; xhigh 1,934 rating $5.91
Kimi K3.1 Pro self-refine 1; max 1,933 rating $0.66
Claude Haiku 5.5 self-refine 8; high 1,931 rating $0.24
Muse Spark 1.3 self-refine 8; xhigh 1,924 rating $1.37
GPT-5.3 Codex self-refine 8; xhigh 1,919 rating $3.37
Claude Opus 4.8 self-refine 16; no-thinking 1,913 rating $11.5
Kimi K3 self-refine 8; max 1,908 rating $5.58
Qwen3.8 Max (0902) self-refine 8; xhigh 1,908 rating $2.43
Claude Opus 4.7 self-refine 16; no-thinking 1,897 rating $12.03
GPT-5.5 self-refine 8; medium 1,892 rating $2.58
GPT-5.3 Codex self-refine 4; xhigh 1,891 rating $1.93
Gemini 3.7 Flash self-refine 16; high 1,876 rating $2.34
DeepSeek V4.1 Flash 1017 self-refine 8; max 1,872 rating $0.47
GLM-5.3 self-refine 8; high 1,863 rating $1.1
Claude Haiku 5.5 self-refine 4; high 1,852 rating $0.07
Qwen3.8 Max (0902) self-refine 4; xhigh 1,848 rating $1.51
Gemini 3.7 Flash self-refine 8; high 1,840 rating $1.2
Muse Spark 1.3 self-refine 4; xhigh 1,837 rating $0.68
Claude Opus 4.7 self-refine 8; no-thinking 1,836 rating $4.15
Muse Spark Open self-refine 16; xhigh 1,830 rating $1.72
DeepSeek V4.1 Flash self-refine 16; max 1,829 rating $0.65
GPT-5.6 Luna self-refine 16; max 1,829 rating $2.29
Gemini 3.6 Flash self-refine 16; high 1,825 rating $2.53
Grok 4.7 self-refine 16; xhigh 1,823 rating $7.12
Grok 4.6 self-refine 16; xhigh 1,822 rating $2.32
Claude Opus 4.8 self-refine 8; no-thinking 1,820 rating $4.28
Claude Sonnet 5.5 self-refine 1; high 1,819 rating $0.11
GPT-5.4 self-refine 16; high 1,811 rating $5.02
GPT-5.4 self-refine 4; high 1,807 rating $1.69
GPT-5.3 Codex self-refine 2; xhigh 1,803 rating $1.06
Claude Opus 4.8 self-refine 4; high 1,802 rating $3.48
Claude Opus 4.8 self-refine 4; no-thinking 1,799 rating $1.8
GPT-6 Luna self-refine 16; xhigh 1,791 rating $0.12
GPT-5.5 self-refine 4; medium 1,791 rating $1.26
Kimi K3 self-refine 4; max 1,788 rating $3.37
GPT-5.6 Luna self-refine 8; max 1,785 rating $1.38
Muse Spark Open self-refine 8; xhigh 1,783 rating $1.1
Claude Sonnet 5 self-refine 16; high 1,783 rating $11.89
DeepSeek V4.1 Flash 1017 self-refine 4; max 1,769 rating $0.29
Claude Sonnet 4.6 self-refine 16; medium 1,768 rating $27.94
Muse Spark 1.3 self-refine 2; xhigh 1,765 rating $0.4
GPT-5.6 Luna self-refine 4; max 1,763 rating $0.93
GPT-5.4 self-refine 8; high 1,752 rating $2.76
GPT-6 Luna self-refine 8; xhigh 1,749 rating $0.07
Grok 4.6 self-refine 8; xhigh 1,745 rating $1.16
Qwen3.8 Max (0902) self-refine 2; xhigh 1,744 rating $0.96
Claude Haiku 5.5 self-refine 2; high 1,738 rating $0.03
GPT-5.5 self-refine 2; medium 1,735 rating $0.63
Grok 4.7 self-refine 8; xhigh 1,732 rating $3.24
DeepSeek V4 Pro 0813 self-refine 16; max 1,730 rating $0.41
Gemini 3 Pro self-refine 16; high 1,729 rating $3.03
DeepSeek V4.1 Flash self-refine 8; max 1,727 rating $0.39
GPT-5.6 Luna self-refine 2; max 1,725 rating $0.64
GLM-5.3 self-refine 4; high 1,724 rating $0.72
Gemini 3 Flash self-refine 16; high 1,722 rating $0.85
GPT-5.4 self-refine 16; medium 1,717 rating $3.32
Claude Sonnet 5 self-refine 8; high 1,717 rating $5.4
Gemini 3 Pro self-refine 16; low 1,715 rating $1.88
GPT-5.2 self-refine 16; high 1,715 rating $3.19
GPT-6 Luna self-refine 4; xhigh 1,714 rating $0.04
Gemini 3.5 Flash self-refine 16; high 1,712 rating $2.94
GPT-5.4 self-refine 2; high 1,710 rating $1.07
Gemini 4 Argon self-refine 1; low 1,705 rating $0.17
Muse Spark Open self-refine 4; xhigh 1,704 rating $0.75
Claude Opus 4.7 self-refine 4; no-thinking 1,695 rating $1.44
Kimi K2.6 self-refine 16 1,695 rating $1.65
Mercury 3 self-refine 16; high 1,692 rating $0.28
Claude Opus 4.8 self-refine 2; high 1,690 rating $1.8
Muse Spark 1.3 self-refine 1; xhigh 1,688 rating $0.22
Claude Sonnet 4.6 self-refine 8; medium 1,688 rating $12.79
DeepSeek V4 Flash 0731 self-refine 8; max 1,686 rating $0.07
GPT-5.4 self-refine 4; medium 1,686 rating $0.94
GPT-5.2 Codex self-refine 8; xhigh 1,686 rating $3.85
GLM-5.2 self-refine 16; max 1,685 rating $1.95
Qwen3.8 Max (0902) self-refine 1; xhigh 1,679 rating $0.53
DeepSeek V4 Flash 0731 self-refine 16; max 1,679 rating $0.1
DeepSeek V4.1 Flash 1017 self-refine 2; max 1,677 rating $0.17
Gemini 3.1 Pro self-refine 4; high 1,677 rating $1.59
Gemini 3 Pro self-refine 8; high 1,675 rating $1.56
Grok 4.6 self-refine 4; xhigh 1,673 rating $0.71
GPT-5.2 self-refine 16; medium 1,672 rating $1.93
GPT-5.6 Luna self-refine 1; max 1,667 rating $0.36
GPT-5.2 self-refine 8; high 1,660 rating $1.66
Gemini 3.5 Flash self-refine 8; high 1,657 rating $1.73
Mistral Large 4 self-refine 16 1,657 rating $7.37
GPT-5.3 Codex self-refine 1; xhigh 1,655 rating $0.56
Grok 4.7 self-refine 4; xhigh 1,650 rating $1.73
DeepSeek V4 Pro 0813 self-refine 8; max 1,649 rating $0.27
GLM-5.1 self-refine 16 1,646 rating $1.91
GPT-6 Luna self-refine 2; xhigh 1,642 rating $0.03
Muse Spark Open self-refine 2; xhigh 1,640 rating $0.52
Gemini 3.8 Flash self-refine 2; high 1,637 rating $0.58
GPT-5.5 self-refine 16 1,633 rating $2.86
Kimi K3 self-refine 2; max 1,633 rating $2.26
Gemini 3 Flash self-refine 8; high 1,632 rating $0.48
DeepSeek V4 Pro 0813 self-refine 4; max 1,628 rating $0.18
Gemini 3 Pro self-refine 8; low 1,628 rating $0.9
DeepSeek V4.1 Flash 1017 self-refine 1; max 1,622 rating $0.1
GPT-5.2 self-refine 8; medium 1,616 rating $0.95
Claude Opus 4.8 self-refine 2; no-thinking 1,613 rating $0.84
GPT-5.2 Codex self-refine 16; xhigh 1,613 rating $6.22
GLM-5.2 self-refine 8; max 1,610 rating $1.13
Claude Sonnet 5 self-refine 4; high 1,610 rating $2.8
GPT-5.4 self-refine 1; high 1,607 rating $0.61
Gemini 3.1 Pro self-refine 16; low 1,607 rating $0.93
Claude Opus 4.6 self-refine 16; no-thinking 1,607 rating $3.64
Gemini 3.6 Flash self-refine 8; high 1,601 rating $1.22
Grok 4.7 self-refine 2; xhigh 1,601 rating $1.12
GPT-5.2 Codex self-refine 4; xhigh 1,599 rating $2.1
Gemini 3 Flash self-refine 4; high 1,598 rating $0.28
Kimi K2.6 self-refine 8 1,593 rating $1
Grok 4.6 self-refine 2; xhigh 1,592 rating $0.44
GPT-5.4 self-refine 8; medium 1,592 rating $1.69
Muse Spark Open self-refine 1; xhigh 1,590 rating $0.33
DeepSeek V4 Flash 0731 self-refine 4; max 1,590 rating $0.04
GPT-5.5 self-refine 1; medium 1,589 rating $0.33
Mistral Large 4 self-refine 8 1,589 rating $3.08
Claude Haiku 5.5 self-refine 1; high 1,587 rating $0.02
GPT-5.2 self-refine 4; high 1,586 rating $0.93
GPT-6 Luna self-refine 1; xhigh 1,577 rating $0.02
Gemini 3 Pro self-refine 4; high 1,571 rating $0.85
Gemini 3.7 Flash self-refine 4; high 1,567 rating $0.68
Claude Opus 4.8 self-refine 1; high 1,564 rating $0.96
Mercury 3 self-refine 8; high 1,561 rating $0.17
Claude Sonnet 4.6 self-refine 4; medium 1,560 rating $6.6
GPT-5.5 self-refine 8 1,558 rating $1.18
Gemini 3.5 Flash self-refine 4; high 1,552 rating $1.04
DeepSeek V4 Pro 0813 self-refine 2; max 1,550 rating $0.12
qwen3.7 self-refine 8; max 1,548 rating $2.42
qwen3.7 self-refine 16; max 1,547 rating $4.68
Grok 4.7 self-refine 1; xhigh 1,546 rating $0.67
GPT-5.5 self-refine 4 1,545 rating $0.47
GPT-5 self-refine 16 1,545 rating $2.41
GPT-5.4 self-refine 2; medium 1,538 rating $0.53
GPT-5.1-Codex-Max self-refine 16; xhigh 1,538 rating $1.76
Claude Sonnet 5 self-refine 2; high 1,533 rating $1.62
GPT-5.1-Codex self-refine 16; high 1,528 rating $2.75
Kimi K3 self-refine 1; max 1,524 rating $1.28
GPT-5.1-Codex self-refine 8; high 1,522 rating $1.98
Gemini 3.1 Pro self-refine 8; low 1,522 rating $0.43
GPT-5.4 self-refine 1; medium 1,521 rating $0.26
Claude Opus 4.7 self-refine 2; no-thinking 1,521 rating $0.53
Kimi K2.6 self-refine 4 1,520 rating $0.61
GPT-5.1-Codex-Max self-refine 8; xhigh 1,519 rating $1.13
Tencent Hy3 self-refine 16; high 1,516 rating $0.21
GPT-5.2 Codex self-refine 2; xhigh 1,512 rating $1.11
GLM-5.2 self-refine 8; high 1,510 rating $0.77
Grok 4.6 self-refine 1; xhigh 1,508 rating $0.24
GPT-5.2 self-refine 4; medium 1,502 rating $0.48
DeepSeek V4 Flash 0731 self-refine 2; max 1,501 rating $0.03
GLM-5.2 self-refine 4; max 1,500 rating $0.64
GPT-5.1-Codex self-refine 4; high 1,497 rating $1.41
MiniMax-M3.1-Flash self-refine 16; max 1,495 rating $0.22
Beam 1.0 self-refine 16; max 1,493 rating $0.26
Mistral Large 4 self-refine 4 1,492 rating $1.45
GPT-5 self-refine 8 1,491 rating $1.3
DeepSeek-V4-Pro self-refine 16; high 1,489 rating $0.96
GLM-5.1 self-refine 8 1,489 rating $1.05
Claude Sonnet 4.6 self-refine 2; medium 1,487 rating $3.5
GPT-5.4 Mini self-refine 16; high 1,487 rating $3.01
Mercury 3 self-refine 4; high 1,483 rating $0.12
MiMo-V2.6 Pro self-refine 8 1,476 rating $0.36
DeepSeek V4.1 Flash self-refine 4; max 1,476 rating $0.24
Gemini 3 Flash self-refine 2; high 1,472 rating $0.16
Grok 4.5 self-refine 8; high 1,467 rating $0.64
GLM-5.2 self-refine 16; high 1,463 rating $1.32
Claude Sonnet 5 self-refine 1; high 1,463 rating $0.96
Gemini 3 Pro self-refine 4; low 1,463 rating $0.46
qwen3.7 self-refine 4; max 1,462 rating $1.32
GPT-5.1-Codex-Max self-refine 16; high 1,458 rating $1.52
Claude Opus 4.6 self-refine 8; no-thinking 1,457 rating $1.15
GPT-5.1-Codex-Max self-refine 4; xhigh 1,457 rating $0.75
MiMo-V2.6 Flash self-refine 16 1,455 rating $0.15
GLM-5.3 self-refine 2; high 1,454 rating $0.5
GPT-5.2 self-refine 2; high 1,453 rating $0.54
GPT-5.4 Mini self-refine 8; high 1,452 rating $1.84
DeepSeek-V4-Pro self-refine 8; high 1,445 rating $0.56
GPT-5.4 self-refine 16 1,445 rating $1.58
Grok 4.5 self-refine 16; high 1,441 rating $1.23
MiMo-V2.6 Pro self-refine 16 1,441 rating $0.48
GPT-5.1-Codex-Max self-refine 8; high 1,436 rating $0.92
GPT-5 self-refine 4 1,431 rating $0.75
Beam 1.0 self-refine 8; max 1,430 rating $0.14
Claude Opus 4.5 self-refine 16 1,429 rating $5.38
GPT-5.1 self-refine 16 1,425 rating $2.24
Kimi K2.7 Code self-refine 16 1,422 rating $1.38
GPT-5 mini self-refine 16 1,421 rating $0.44
Grok 4.5 self-refine 4; high 1,417 rating $0.4
GPT-5.4 Mini self-refine 4; high 1,415 rating $1.04
Claude Opus 4.8 self-refine 1; no-thinking 1,412 rating $0.42
Gemini 3.6 Flash self-refine 4; high 1,409 rating $0.68
MiMo-V2.6 Flash self-refine 8 1,406 rating $0.1
DeepSeek V4 Pro 0813 self-refine 1; max 1,403 rating $0.07
GPT-5.1 self-refine 8 1,401 rating $1.34
GPT-5.4 self-refine 8 1,397 rating $0.65
GPT-5.5 self-refine 2 1,394 rating $0.19
Gemma 4 31B IT self-refine 16 1,394 rating $0.11
grok-4.20-beta self-refine 16 1,388 rating $0.94
Gemma 4 31B IT self-refine 8 1,386 rating $0.04
DeepSeek-V4-Flash self-refine 16; high 1,380 rating $0.05
Claude Opus 4.5 self-refine 8 1,377 rating $2.44
GPT-5.1-Codex self-refine 2; high 1,373 rating $0.94
Gemini 3.1 Pro self-refine 4; low 1,373 rating $0.2
GPT-5.2 self-refine 2; medium 1,372 rating $0.26
GLM-5.2 self-refine 4; high 1,372 rating $0.48
Gemini 3 Flash self-refine 1; high 1,367 rating $0.09
GPT-5.1-Codex-Max self-refine 4; high 1,364 rating $0.6
Mercury 3 self-refine 2; high 1,360 rating $0.08
GLM-5.1 self-refine 4 1,353 rating $0.57
Nemotron 3.5 Super VL self-refine 16 1,352 rating $0.08
MiMo-V2.6 Pro self-refine 4 1,350 rating $0.26
MiMo-V2.6 Flash self-refine 4 1,349 rating $0.08
grok-4.20-beta self-refine 8 1,348 rating $0.54
Gemini 3 Pro self-refine 2; high 1,348 rating $0.48
GPT-5.1-Codex-Max self-refine 2; xhigh 1,348 rating $0.52
Grok 4.5 self-refine 2; high 1,348 rating $0.28
MiMo-V2-Pro self-refine 16 1,347 rating $0.66
Tencent Hy3 self-refine 8; high 1,343 rating $0.12
GPT-5.1 self-refine 4 1,336 rating $0.86
Mistral Large 4 self-refine 2 1,332 rating $0.73
Claude Sonnet 4.6 self-refine 1; medium 1,327 rating $1.77
qwen3.7 self-refine 2; max 1,327 rating $0.73
Claude Opus 4.7 self-refine 1; no-thinking 1,323 rating $0.22
MiMo-V2.6 Flash self-refine 2 1,321 rating $0.06
Tencent Hy3 self-refine 4; high 1,321 rating $0.07
Claude Opus 4.6 self-refine 4; no-thinking 1,320 rating $0.39
GLM-5.3 self-refine 1; high 1,317 rating $0.28
Kimi K2.7 Code self-refine 8 1,316 rating $0.77
GPT-5.1-Codex-Max self-refine 2; high 1,316 rating $0.4
MiniMax-M3.1-Flash self-refine 8; max 1,314 rating $0.13
GLM-5-Turbo self-refine 16 1,313 rating $1.06
GPT-5.4 Nano self-refine 16; high 1,312 rating $0.25
GPT-5 mini self-refine 8 1,311 rating $0.21
Grok 4.5 self-refine 1; high 1,309 rating $0.18
DeepSeek-V4-Pro self-refine 4; high 1,308 rating $0.35
DeepSeek V4 Flash 0731 self-refine 1; max 1,306 rating $0.02
GPT-5.4 self-refine 4 1,304 rating $0.27
Beam 1.0 self-refine 4; max 1,304 rating $0.08
o3 self-refine 16; high 1,304 rating $1.28
GPT-5.2 Codex self-refine 1; xhigh 1,300 rating $0.59
Claude Opus 4.5 self-refine 4 1,297 rating $1.28
Kimi K2.6 self-refine 2 1,297 rating $0.36
DeepSeek-V4-Flash self-refine 8; high 1,296 rating $0.02
GPT-5.2 self-refine 1; high 1,294 rating $0.3
GPT-5.4 Nano self-refine 8; high 1,292 rating $0.14
DeepSeek V4.1 Flash self-refine 2; max 1,287 rating $0.14
GLM-5.3-Flash self-refine 16; high 1,283 rating $0.11
GPT-5.1 self-refine 2 1,281 rating $0.55
Kimi K2.5 self-refine 16 1,279 rating $0.33
GPT-5.4 Mini self-refine 2; high 1,277 rating $0.55
Step 3.7 Flash self-refine 16; medium 1,277 rating $0.38
Nemotron 3.5 Super VL self-refine 8 1,276 rating $0.05
Qwen 3.6 Plus self-refine 16 1,273 rating $1.07
DeepSeek-V4-Pro self-refine 16; no-thinking 1,272 rating $0.45
o4-mini self-refine 16; high 1,271 rating $0.82
MiMo-V2.6 Pro self-refine 2 1,270 rating $0.19
Gemini 3.8 Flash self-refine 1; high 1,270 rating $0.34
Gemini 3.1 Pro self-refine 2; high 1,270 rating $0.79
MiMo-V2-Pro self-refine 8 1,262 rating $0.34
GPT-5 self-refine 2 1,262 rating $0.44
grok-4.20-beta self-refine 4 1,260 rating $0.34
Solar Pro 4 self-refine 16; max 1,256 rating $0.42
GLM-5.2 self-refine 2; max 1,255 rating $0.33
MiMo-V2.5-Pro self-refine 16 1,254 rating $0.61
GPT-5.2 self-refine 1; medium 1,250 rating $0.14
GPT-5.1-Codex self-refine 1; high 1,245 rating $0.51
MiMo-V2.6 Flash self-refine 1 1,244 rating $0.04
Kimi K2.7 Code self-refine 4 1,243 rating $0.46
Gemini 3 Pro self-refine 2; low 1,241 rating $0.24
Kimi K2.5 self-refine 8 1,240 rating $0.2
Step 3.7 Flash self-refine 8; medium 1,236 rating $0.22
GPT-5.1-Codex-Max self-refine 1; xhigh 1,228 rating $0.32
Gemma 4 31B IT self-refine 4 1,226 rating $0.02
muse-glimmer-30b self-refine 16; high 1,226 rating $0.21
o3 self-refine 8; high 1,223 rating $0.68
Gemini 3.1 Pro self-refine 2; low 1,223 rating $0.09
GPT-5.4 Nano self-refine 4; high 1,214 rating $0.08
Mistral Large 4 self-refine 1 1,213 rating $0.46
muse-glimmer-30b self-refine 8; high 1,212 rating $0.1
LongCat-2.0 self-refine 8 1,212 rating $0.13
Grok 4.3 self-refine 16 1,211 rating $0.71
Gemini 3.7 Flash self-refine 2; high 1,210 rating $0.41
GPT-5.1-Codex-Max self-refine 1; high 1,209 rating $0.25
Mercury 3 self-refine 1; high 1,209 rating $0.05
GLM-5-Turbo self-refine 8 1,208 rating $0.62
MiMo-V2.5-Pro self-refine 8 1,204 rating $0.33
LongCat-2.0 self-refine 16 1,204 rating $0.19
GLM-5.2 self-refine 2; high 1,201 rating $0.3
GPT-5.4 self-refine 2 1,200 rating $0.12
Inkling self-refine 16; max 1,195 rating $1.9
GLM-5 self-refine 16 1,194 rating $0.5
GPT-5.1 self-refine 1 1,192 rating $0.32
Claude Sonnet 4.5 self-refine 16 1,192 rating $2.3
o4-mini self-refine 8; high 1,192 rating $0.42
Mercury 2 self-refine 16 1,190 rating $0.17
qwen3.7 self-refine 1; max 1,189 rating $0.38
GPT-5.4 Mini self-refine 1; high 1,189 rating $0.27
DeepSeek-V4-Pro self-refine 8; no-thinking 1,188 rating $0.21
o3 self-refine 4; high 1,186 rating $0.38
Claude Opus 4.6 self-refine 2; no-thinking 1,186 rating $0.15
muse-glimmer-30b self-refine 4; high 1,183 rating $0.07
Claude Opus 4.1 self-refine 16 1,183 rating $12.09
Mercury 2 self-refine 8 1,183 rating $0.08
DeepSeek-V4-Pro self-refine 2; high 1,182 rating $0.23
Gemini 3.5 Flash self-refine 2; high 1,178 rating $0.68
Gemini 3 Pro self-refine 1; high 1,177 rating $0.26
gemini-2.5-pro-thinking (snapshot unspecified) self-refine 16 1,175 rating $2.22
GPT-5 mini self-refine 4 1,172 rating $0.11
GLM-5.1 self-refine 2 1,170 rating $0.29
Gemini 3.1 Flash-Lite self-refine 8; high 1,165 rating $0.14
gemma-4-26b-a4b-it self-refine 16 1,165 rating $0.12
grok-4.20-beta self-refine 2 1,164 rating $0.24
GPT-5 self-refine 1 1,162 rating $0.24
Mercury 2 self-refine 4 1,162 rating $0.04
Gemini 3.1 Pro self-refine 1; high 1,161 rating $0.39
Tencent Hy3 self-refine 2; high 1,159 rating $0.05
MiMo-V2.6 Pro self-refine 1 1,158 rating $0.11
DeepSeek-V4-Flash self-refine 4; high 1,158 rating $0.01
Qwen 3.6 Plus self-refine 8 1,158 rating $0.46
Grok 4.3 self-refine 8 1,155 rating $0.33
Nemotron 3.5 Super VL self-refine 4 1,152 rating $0.04
grok-4.20-beta self-refine 1 1,150 rating $0.14
Inkling self-refine 4; max 1,150 rating $0.53
Claude Opus 4.5 self-refine 2 1,149 rating $0.58
Nemotron 3.5 Super VL self-refine 2 1,142 rating $0.03
Beam 1.0 self-refine 2; max 1,139 rating $0.05
MiMo-V2.5-Pro self-refine 4 1,138 rating $0.22
Gemini 3.5 Flash-Lite self-refine 16; high 1,137 rating $0.34
GLM-5.3-Flash self-refine 8; high 1,136 rating $0.06
Inkling self-refine 8; max 1,135 rating $0.95
Mistral Medium 3.5 self-refine 16 1,131 rating $1.96
Grok 4.3 self-refine 4 1,131 rating $0.18
GPT-5.5 self-refine 1 1,128 rating $0.07
Kimi K2.5 self-refine 4 1,127 rating $0.13
MiMo-V2-Pro self-refine 4 1,122 rating $0.2
Gemini 3.6 Flash self-refine 2; high 1,121 rating $0.4
gpt-oss-120b self-refine 16 1,119 rating $0.07
muse-glimmer-30b self-refine 2; high 1,117 rating $0.04
gemini-2.5-pro-thinking (snapshot unspecified) self-refine 8 1,117 rating $1.3
Gemini 3.5 Flash-Lite self-refine 8; high 1,111 rating $0.19
Grok 4.1 Fast self-refine 16 1,111 rating $0.08
o4-mini self-refine 4; high 1,110 rating $0.21
LongCat-2.0 self-refine 4 1,110 rating $0.09
Gemini 3.1 Flash-Lite self-refine 16; high 1,110 rating $0.26
Ling-2.6-1T self-refine 16 1,108 rating $0.17
MiMo-V2.5 self-refine 16 1,106 rating $0.29
GPT-5 self-refine 16 1,105 rating $0.96
gemma-4-26b-a4b-it self-refine 8 1,105 rating $0.06
Kimi K2 Thinking self-refine 16 1,103 rating $0.53
Claude Opus 4.1 self-refine 8 1,099 rating $4.08
Beam 1.0 self-refine 1; max 1,098 rating $0.03
GPT-5.4 Nano self-refine 2; high 1,096 rating $0.05
Kimi K2.6 self-refine 1 1,093 rating $0.2
DeepSeek V4.1 Flash self-refine 1; max 1,092 rating $0.08
Gemma 4 31B IT self-refine 2 1,091 rating $0.01
GLM-5 self-refine 8 1,090 rating $0.29
Grok 4.3 self-refine 2 1,088 rating $0.1
Claude Sonnet 4 self-refine 16 1,088 rating $2.53
Kimi K2 Thinking self-refine 8 1,087 rating $0.3
GPT-5.4 self-refine 1 1,086 rating $0.05
Gemini 3.1 Flash-Lite self-refine 4; high 1,083 rating $0.08
MiMo-V2.5-Pro self-refine 2 1,080 rating $0.16
DeepSeek-V3.1-Terminus self-refine 16 1,078 rating $0.21
Kimi K2.5 self-refine 2 1,077 rating $0.08
Solar Pro 4 self-refine 8; max 1,073 rating $0.2
Claude Sonnet 4.5 self-refine 8 1,072 rating $1
Gemini 3.5 Flash-Lite self-refine 4; high 1,072 rating $0.11
gemma-4-26b-a4b-it self-refine 4 1,072 rating $0.03
gemini-2.5-pro-thinking (snapshot unspecified) self-refine 4 1,071 rating $0.78
Qwen3.5-Plus self-refine 16 1,071 rating $0.39
o3 self-refine 2; high 1,067 rating $0.2
GPT-5 self-refine 8 1,066 rating $0.4
gpt-oss-120b self-refine 8 1,066 rating $0.02
Mistral Medium 3.5 self-refine 8 1,064 rating $1.14
Inkling self-refine 2; max 1,062 rating $0.32
DeepSeek-R1 May 2025 self-refine 16 1,059 rating $0.4
Step 3.7 Flash self-refine 4; medium 1,059 rating $0.13
Gemini 3.1 Pro self-refine 1; low 1,055 rating $0.04
MiniMax-M3 self-refine 16 1,053 rating $0.68
GLM-5-Turbo self-refine 4 1,053 rating $0.35
DeepSeek-V3.1-Terminus self-refine 8 1,052 rating $0.1
MiMo-V2.5 self-refine 8 1,052 rating $0.15
DeepSeek-V4-Flash self-refine 16; no-thinking 1,051 rating $0.03
GLM-5.2 self-refine 1; high 1,047 rating $0.16
Kimi K2.7 Code self-refine 2 1,046 rating $0.3
Qwen 3.6 Plus self-refine 4 1,045 rating $0.21
Claude Sonnet 4 self-refine 8 1,042 rating $1.08
MiniMax-M3.1-Flash self-refine 4; max 1,041 rating $0.09
Qwen3.5-Plus self-refine 8 1,039 rating $0.14
MiniMax-M2.5 self-refine 16 1,033 rating $0.22
GPT-5 self-refine 4 1,029 rating $0.18
Claude Opus 4.5 self-refine 1 1,025 rating $0.29
MiniMax-M3 self-refine 8 1,025 rating $0.4
Grok 4.1 Fast self-refine 8 1,023 rating $0.05
gemini-2.5-flash-thinking (snapshot unspecified) self-refine 16 1,022 rating $0.58
MiniMax-M2.7 self-refine 16 1,021 rating $0.22
Nemotron 3 Super self-refine 16 1,017 rating $0.07
Nemotron 3.5 Lightning 30B-A3B self-refine 16 1,017 rating $0.07
Ling-2.6-1T self-refine 8 1,017 rating $0.08
GPT-5 nano self-refine 16 1,016 rating $0.1
Qwen 3.6 Flash self-refine 16 1,015 rating $0.41
o4-mini self-refine 2; high 1,014 rating $0.1
Claude Opus 4.1 self-refine 4 1,014 rating $1.62
GLM-4.7 self-refine 16 1,011 rating $0.47
Claude Sonnet 4.5 self-refine 4 1,011 rating $0.53
Mistral Medium 3.5 self-refine 4 1,010 rating $0.69
GLM-5.2 self-refine 1; max 1,010 rating $0.15
DeepSeek-R1 May 2025 self-refine 8 1,009 rating $0.19
GLM-5 self-refine 4 1,009 rating $0.19
MiniMax-M2.5 self-refine 4 1,007 rating $0.1
Qwen3.5 397B-A17B self-refine 16 1,007 rating $0.62
Claude Haiku 4.5 self-refine 16 1,006 rating $0.94
DeepSeek-V4-Pro self-refine 1; high 1,006 rating $0.15
GPT-5.4 Nano self-refine 1; high 1,005 rating $0.03
MiMo-V2-Flash self-refine 16 999.68 rating $0.07
gemma-4-26b-a4b-it self-refine 2 998.48 rating $0.02
MiniMax-M2.1 self-refine 16 998.18 rating $0.43
Claude Opus 4.6 self-refine 1; no-thinking 996.5 rating $0.06
GPT-5 mini self-refine 2 993.98 rating $0.05
Nemotron 3.5 Lightning 30B-A3B self-refine 8 992.85 rating $0.04
DeepSeek-V3.1-Terminus self-refine 4 992.73 rating $0.05
GPT-5 nano self-refine 8 991.95 rating $0.06
MiniMax-M2.7 self-refine 8 991.48 rating $0.16
Kimi K2 Thinking self-refine 4 991.38 rating $0.18
Claude Haiku 4.5 self-refine 8 991.38 rating $0.45
Claude Sonnet 4 self-refine 4 989.95 rating $0.56
Gemini 3 Pro self-refine 1; low 988.23 rating $0.1
MiniMax-M2.5 self-refine 8 987.83 rating $0.14
gemini-2.5-flash-thinking (snapshot unspecified) self-refine 8 986.4 rating $0.31
GPT-4.1 self-refine 16 984.75 rating $0.76
Mercury 2 self-refine 2 982.48 rating $0.02
Qwen3.5 397B-A17B self-refine 8 978.08 rating $0.23
Mercury 2.5 self-refine 16; high 977.35 rating $0.1
gemini-2.5-flash-thinking (snapshot unspecified) self-refine 4 974.45 rating $0.19
MiMo-V2-Flash self-refine 8 974.35 rating $0.03
DeepSeek-V4-Pro self-refine 4; no-thinking 971.05 rating $0.1
DeepSeek-V4-Flash self-refine 2; high 967.43 rating $0.01
DeepSeek-R1 May 2025 self-refine 4 967.25 rating $0.11
muse-glimmer-30b self-refine 1; high 962.45 rating $0.03
GLM-5.3-Flash self-refine 4; high 960.58 rating $0.03
gpt-oss-120b self-refine 4 957.2 rating $0.01
Nemotron 3.5 Lightning 30B-A3B self-refine 4 956.9 rating $0.03
Nemotron 3.5 Super VL self-refine 1 956.73 rating $0.02
GPT-5 self-refine 2 956.23 rating $0.09
MiMo-V2-Pro self-refine 2 953.63 rating $0.12
gemini-2.5-pro-thinking (snapshot unspecified) self-refine 2 951.78 rating $0.47
Qwen3.7 Flash self-refine 16 946 rating $0.06
Inkling self-refine 1; max 946 rating $0.2
Grok 4.3 self-refine 1 944.18 rating $0.06
gpt-oss-20b self-refine 16 942.3 rating $0.03
MiMo-V2.5 self-refine 4 941.68 rating $0.09
Qwen3.5-Plus self-refine 4 936.23 rating $0.07
Claude Haiku 4.5 self-refine 4 935.85 rating $0.24
o3 self-refine 1; high 933.55 rating $0.1
MiniMax-M2.1 self-refine 8 931.03 rating $0.3
DeepSeek-V4-Flash self-refine 8; no-thinking 928.85 rating $0.01
Gemini 3.5 Flash-Lite self-refine 2; high 928.63 rating $0.07
gemma-4-26b-a4b-it self-refine 1 927.18 rating $0.01
Gemma 4 31B IT self-refine 1 925.5 rating $0
Gemini 3.1 Flash-Lite self-refine 2; high 925.03 rating $0.04
GLM-4.7 self-refine 8 922.68 rating $0.23
Solar Pro 4 self-refine 4; max 918.98 rating $0.11
Claude Sonnet 4.5 self-refine 2 918.63 rating $0.29
GPT-5 nano self-refine 4 916.3 rating $0.03
MiniMax-M2.1 self-refine 4 915.35 rating $0.19
GPT-4.1 self-refine 8 915.35 rating $0.27
Nemotron 3.5 Lightning 30B-A3B self-refine 2 914.33 rating $0.02
Gemini 3.5 Flash self-refine 1; high 911.03 rating $0.44
DeepSeek-R1 May 2025 self-refine 2 908.78 rating $0.07
LongCat-2.0 self-refine 2 908.48 rating $0.06
Gemini 3.7 Flash self-refine 1; high 904.3 rating $0.24
MiniMax-M3 self-refine 4 903.08 rating $0.26
MiMo-V2.5-Pro self-refine 1 899.8 rating $0.1
GLM-5-Turbo self-refine 2 899.63 rating $0.2
Qwen3.5 397B-A17B self-refine 4 897.68 rating $0.11
Mercury 2.5 self-refine 8; high 897.28 rating $0.04
Mistral Medium 3.5 self-refine 2 896.33 rating $0.44
qwen3 self-refine 16; max 894.35 rating $1.14
Claude Opus 4.1 self-refine 2 893.85 rating $0.78
Grok 4.1 Fast self-refine 4 887.1 rating $0.03
GLM-5.1 self-refine 1 887.1 rating $0.13
Kimi K2.7 Code self-refine 1 886.23 rating $0.18
Qwen3.5-27B self-refine 16 882.48 rating $0.36
Nemotron 3 Super self-refine 8 879.68 rating $0.03
Qwen 3.6 Flash self-refine 8 873.55 rating $0.19
MiMo-V2-Flash self-refine 4 872.55 rating $0.02
Qwen3.7 Flash self-refine 8 872.3 rating $0.02
gpt-oss-20b self-refine 8 867.2 rating $0.01
DeepSeek-V3.1-Terminus self-refine 2 865.4 rating $0.03
qwen3 self-refine 8; max 863.1 rating $0.36
Mistral Small 4 self-refine 16 861.9 rating $0.12
MiniMax-M2.1 self-refine 2 857.73 rating $0.11
Step 3.7 Flash self-refine 2; medium 855.18 rating $0.06
grok-code-fast-1 self-refine 16 853.48 rating $0.07
Kimi K2 Thinking self-refine 2 848.05 rating $0.13
Ling-2.6-1T self-refine 4 845.58 rating $0.04
gemini-2.5-flash-thinking (snapshot unspecified) self-refine 2 841.83 rating $0.11
Qwen 3.6 Plus self-refine 2 840.33 rating $0.1
GPT-5 nano self-refine 2 839.38 rating $0.02
grok-code-fast-1 self-refine 8 839.38 rating $0.04
Tencent Hy3 self-refine 1; high 838.65 rating $0.03
GLM-4.6 self-refine 16 838.4 rating $0.31
Claude Sonnet 4 self-refine 2 833.55 rating $0.32
GLM-5 self-refine 2 828.13 rating $0.12
Qwen3-Coder-Plus self-refine 16 827.23 rating $0.29
o4-mini self-refine 1; high 826.18 rating $0.04
Qwen3.5-Flash self-refine 16 826 rating $0.1
Claude Haiku 4.5 self-refine 2 822.55 rating $0.13
Qwen3-Coder-480B-A35B self-refine 16 822.43 rating $0.17
Kimi K2.5 self-refine 1 821.65 rating $0.05
MiniMax-M2.7 self-refine 4 820.68 rating $0.12
Qwen3.5-Plus self-refine 2 812.7 rating $0.04
GPT-5 self-refine 1 807.65 rating $0.04
gpt-oss-120b self-refine 2 806.93 rating $0
GLM-4.7 self-refine 4 805.25 rating $0.13
DeepSeek-R1 May 2025 self-refine 1 804.13 rating $0.04
GPT-5 mini self-refine 1 799.78 rating $0.02
Qwen3.7 Flash self-refine 4 798.1 rating $0.01
Gemini 3.1 Flash-Lite self-refine 1; high 797.73 rating $0.03
GPT-4.1 self-refine 4 796.6 rating $0.11
Qwen3-Coder-Plus self-refine 8 796.45 rating $0.13
Claude Sonnet 4.5 self-refine 1 796.15 rating $0.15
LongCat-2.0 self-refine 1 793.45 rating $0.04
qwen3 self-refine 4; max 791.68 rating $0.13
GLM-4.6 self-refine 8 788.9 rating $0.12
Mistral Small 4 self-refine 8 786.53 rating $0.06
Mercury 2 self-refine 1 785.58 rating $0.01
gemini-2.5-pro-thinking (snapshot unspecified) self-refine 1 785.53 rating $0.27
MiMo-V2-Pro self-refine 1 785.18 rating $0.08
Qwen3.5-27B self-refine 8 784.4 rating $0.16
Qwen3-Coder-480B-A35B self-refine 8 784.23 rating $0.06
MiMo-V2-Flash self-refine 2 782.45 rating $0.01
grok-code-fast-1 self-refine 4 782.05 rating $0.02
kimi-k2-0905 self-refine 16 776.33 rating $0.33
gpt-oss-20b self-refine 4 774.98 rating $0
MiniMax-M3 self-refine 2 772.58 rating $0.17
Nemotron 3.5 Lightning 30B-A3B self-refine 1 769.23 rating $0.01
GLM-5 self-refine 1 765.63 rating $0.07
Gemini 3.5 Flash-Lite self-refine 1; high 765.28 rating $0.05
Mistral Medium 3.5 self-refine 1 763.98 rating $0.27
Qwen3.5-Flash self-refine 8 757.93 rating $0.04
MiniMax-M2.5 self-refine 2 749.9 rating $0.06
Qwen3-Coder-Plus self-refine 4 746.53 rating $0.06
DeepSeek-V3.1-Terminus self-refine 1 745.18 rating $0.02
DeepSeek-V4-Flash self-refine 4; no-thinking 743.55 rating $0.01
MiMo-V2-Flash self-refine 1 737.95 rating $0.01
MiMo-V2.5 self-refine 2 735.03 rating $0.06
Ling-2.6-1T self-refine 2 732.7 rating $0.02
kimi-k2-0905 self-refine 8 729.68 rating $0.11
Mistral Small 4 self-refine 4 728.6 rating $0.04
Qwen3.5 397B-A17B self-refine 2 725.1 rating $0.06
DeepSeek-V4-Pro self-refine 2; no-thinking 725.05 rating $0.05
GLM-4.6 self-refine 4 723.58 rating $0.05
mistral-large-3-2512 self-refine 16 721.05 rating $0.26
GPT-5 nano self-refine 1 718.68 rating $0.01
Gemini 3.6 Flash self-refine 1; high 715.53 rating $0.24
Qwen3.5-27B self-refine 4 715.48 rating $0.09
MiniMax-M2.7 self-refine 2 711.45 rating $0.09
Nemotron 3 Super self-refine 4 706.9 rating $0.02
Grok 4.1 Fast self-refine 2 701.1 rating $0.02
Mistral Medium 3.1 self-refine 16 700.73 rating $0.32
Step 3.7 Flash self-refine 1; medium 694.13 rating $0.03
grok-code-fast-1 self-refine 2 693.65 rating $0.02
Qwen3-Coder-480B-A35B self-refine 4 680.65 rating $0.02
GPT-4.1 self-refine 2 679.05 rating $0.06
DeepSeek-V4-Flash self-refine 1; high 678.2 rating $0.01
Claude Opus 4.1 self-refine 1 674.78 rating $0.43
MiniMax-M3.1-Flash self-refine 2; max 670.8 rating $0.07
Qwen 3.6 Plus self-refine 1 670.15 rating $0.06
GLM-5.3-Flash self-refine 2; high 667.85 rating $0.02
Qwen 3.6 Flash self-refine 4 667.68 rating $0.11
Mercury 2.5 self-refine 4; high 664.6 rating $0.02
gemini-2.5-flash-thinking (snapshot unspecified) self-refine 1 661.88 rating $0.07
kimi-k2-0905 self-refine 4 661.35 rating $0.04
mistral-large-3-2512 self-refine 8 658.6 rating $0.08
gpt-oss-20b self-refine 2 657 rating $0
Claude Sonnet 4 self-refine 1 655.35 rating $0.19
Claude Haiku 4.5 self-refine 1 653.48 rating $0.07
Qwen3.5-35B-A3B self-refine 16 647.8 rating $0.36
MiniMax-M3 self-refine 1 640.03 rating $0.11
GLM-5-Turbo self-refine 1 633.98 rating $0.11
qwen3 self-refine 2; max 626.8 rating $0.05
MiniMax-M2.1 self-refine 1 623.83 rating $0.05
Qwen3.5 397B-A17B self-refine 1 621.92 rating $0.04
MiniMax-M2.5 self-refine 1 618.17 rating $0.04
Mistral Medium 3.1 self-refine 8 611.05 rating $0.11
Qwen3.5-Plus self-refine 1 608.92 rating $0.03
mistral-large-3-2512 self-refine 4 605.65 rating $0.03
Qwen3.7 Flash self-refine 2 600.98 rating $0.01
MiniMax-M2.7 self-refine 1 599.25 rating $0.05
Kimi K2 Thinking self-refine 1 597.5 rating $0.09
Mistral Small 4 self-refine 2 588.08 rating $0.02
grok-code-fast-1 self-refine 1 587.73 rating $0.01
Qwen3.5-35B-A3B self-refine 8 581.8 rating $0.16
gpt-oss-120b self-refine 1 575.63 rating $0
Qwen3-Coder-Plus self-refine 2 569.58 rating $0.03
gpt-oss-20b self-refine 1 566.05 rating $0
Qwen 3.6 Flash self-refine 2 562.08 rating $0.06
GLM-4.7 self-refine 2 559.67 rating $0.07
GPT-4.1 self-refine 1 558.1 rating $0.03
GLM-4.5 self-refine 16 552.45 rating $0.28
Llama 4 Maverick self-refine 16 552 rating $0.08
Qwen3-Coder-480B-A35B self-refine 2 551.08 rating $0.01
gemini-2.5-flash-lite-thinking (snapshot unspecified) self-refine 16 541.58 rating $0.21
GLM-4.5 self-refine 8 540.13 rating $0.15
DeepSeek-V4-Flash self-refine 2; no-thinking 532.95 rating $0
Llama 4 Maverick self-refine 8 530.45 rating $0.03
nova-premier-v1 self-refine 16 528.78 rating $1.48
Qwen3.5-Flash self-refine 4 525.7 rating $0.02
DeepSeek-V4-Pro self-refine 1; no-thinking 521.67 rating $0.02
gemini-2.5-flash-lite-thinking (snapshot unspecified) self-refine 8 519.4 rating $0.07
MiMo-V2.5 self-refine 1 513.95 rating $0.04
GLM-4.5 self-refine 4 509.48 rating $0.08
nova-2-lite-v1 self-refine 16 507.48 rating $0.18
Qwen3.5-27B self-refine 2 505.93 rating $0.06
gemini-2.5-flash-lite-thinking (snapshot unspecified) self-refine 4 499.63 rating $0.04
Llama 4 Maverick self-refine 4 498.8 rating $0.01
Mistral Small 4 self-refine 1 497.63 rating $0.02
Nemotron 3 Super self-refine 2 494.25 rating $0.01
Mistral Medium 3.1 self-refine 4 487.13 rating $0.04
kimi-k2-0905 self-refine 2 476.13 rating $0.02
Solar Pro 4 self-refine 2; max 465.1 rating $0.06
Mercury 2.5 self-refine 2; high 464.03 rating $0.01
Qwen3-Coder-480B-A35B self-refine 1 461.45 rating $0
GLM-4.6 self-refine 2 460.98 rating $0.03
Qwen3-Coder-Plus self-refine 1 456.5 rating $0.01
mistral-large-3-2512 self-refine 2 444.45 rating $0.01
nova-2-lite-v1 self-refine 8 439.85 rating $0.06
gemini-2.5-flash-lite-thinking (snapshot unspecified) self-refine 2 434.98 rating $0.02
Ling-2.6-1T self-refine 1 432.58 rating $0.01
GLM-4.5 self-refine 2 431.2 rating $0.05
Qwen3.7 Flash self-refine 1 420.1 rating $0
nova-premier-v1 self-refine 8 413.93 rating $0.47
GLM-4.7 self-refine 1 399.48 rating $0.04
Qwen3.5-35B-A3B self-refine 4 398.78 rating $0.09
Grok 4.1 Fast self-refine 1 394.93 rating $0.01
Mistral Medium 3.1 self-refine 2 378.6 rating $0.02
nova-2-lite-v1 self-refine 4 372.48 rating $0.02
qwen3 self-refine 1; max 370.45 rating $0.02
Qwen3.5-Flash self-refine 2 368.75 rating $0.02
Qwen3.5-27B self-refine 1 349.45 rating $0.04
Llama 4 Maverick self-refine 2 349.13 rating $0
GLM-4.5 self-refine 1 344.83 rating $0.03
GLM-4.6 self-refine 1 340.83 rating $0.02
codestral-2508 self-refine 16 340.65 rating $0.16
codestral-2508 self-refine 8 340.48 rating $0.05
Qwen3.5-35B-A3B self-refine 2 332.7 rating $0.06
Qwen 3.6 Flash self-refine 1 326.4 rating $0.04
gemini-2.5-flash-lite-thinking (snapshot unspecified) self-refine 1 325.9 rating $0.01
DeepSeek-V4-Flash self-refine 1; no-thinking 324.98 rating $0
GLM-5.3-Flash self-refine 1; high 303.55 rating $0.01
Mercury 2.5 self-refine 1; high 301.65 rating $0.01
codestral-2508 self-refine 4 294.13 rating $0.02
nova-premier-v1 self-refine 4 289.48 rating $0.18
nova-2-lite-v1 self-refine 2 289.43 rating $0.01
kimi-k2-0905 self-refine 1 267.13 rating $0.01
Qwen3.5-Flash self-refine 1 265.93 rating $0.01
mistral-large-3-2512 self-refine 1 264.7 rating $0.01
nova-2-lite-v1 self-refine 1 236.25 rating $0
Qwen3.5-35B-A3B self-refine 1 221.8 rating $0.04
MiniMax-M3.1-Flash self-refine 1; max 217.75 rating $0.05
Nemotron 3 Super self-refine 1 213.9 rating $0.01
Mistral Medium 3.1 self-refine 1 210.18 rating $0.01
nova-premier-v1 self-refine 2 204.25 rating $0.07
codestral-2508 self-refine 2 195.9 rating $0.01
Solar Pro 4 self-refine 1; max 179.3 rating $0.04
Llama 4 Maverick self-refine 1 172.98 rating $0
nova-premier-v1 self-refine 1 147.38 rating $0.03
codestral-2508 self-refine 1 137.78 rating $0

The best score, 3,164, costs $20 a task

Runs on the cost frontier score higher than any run that's cheaper. GPT-6 Astra at self-refine 2; max comes within 71.3 of the best for $5.38 a task, 4 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI Google DeepMind DeepSeek Anthropic Cost frontier One model at each effort level
0 1,000 2,000 3,000 4,000 $0.0001 $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra, self-refine 8; max GPT-6 Astra, self-refine 1; max GPT-6.1 Sol, self-refine 1; max Claude Opus 5.5, self-refine 1; high Claude Sonnet 5.5, self-refine 2; high GPT-6 Luna, self-refine 8; xhigh Claude Haiku 5.5, self-refine 1; high 0 1,000 2,000 3,000 4,000 $0.0001 $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra, self-refine 8; max GPT-6.1 Sol, self-refine 4; max

Source: Official ALE-Bench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-6 Astra self-refine 8; max 3,164 $20 347,606
GPT-6 Astra self-refine 4; max 3,132 $10 188,322
GPT-6 Astra self-refine 16; max 3,128 $47 670,598
GPT-6 Astra self-refine 2; max 3,093 $5.38 104,233
GPT-6.1 Sol self-refine 16; max 2,990 $10 737,678
Claude Fable 5.2 self-refine 16; high 2,986 $32 281,340
GPT-6 Astra self-refine 1; max 2,951 $2.86 57,124
GPT-6.1 Sol self-refine 8; max 2,942 $4.29 382,336
Gemini 4 Carbon self-refine 8; high 2,929 $13 391,255
GPT-6.1 Sol self-refine 4; max 2,921 $2.21 206,510
Gemini 4 Carbon self-refine 16; high 2,897 $35 732,671
Claude Fable 5.2 self-refine 8; high 2,889 $13 166,885
GPT-6.1 Sol self-refine 2; max 2,878 $1.15 110,954
GPT-6.1 Terra self-refine 16; max 2,875 $5.77 813,089
Claude Fable 5.1 self-refine 16; high 2,849 $39 274,920
GPT-6.1 Terra self-refine 8; max 2,843 $2.42 431,166
Gemini 4 Carbon self-refine 4; high 2,834 $6.18 222,756
GPT-6.1 Terra self-refine 4; max 2,826 $1.33 247,607
Gemini 4 Argon self-refine 8; high 2,805 $8.10 280,242
Claude Fable 5.2 self-refine 4; high 2,787 $5.53 99,086
Claude Opus 5.5 self-refine 16; high 2,748 $8.59 160,277
Gemini 4 Argon self-refine 16; high 2,747 $15 471,517
GPT-6.1 Sol self-refine 1; max 2,744 $0.60 59,569
Gemini 4 Argon self-refine 4; high 2,675 $4.92 177,119
Claude Opus 5.5 self-refine 8; high 2,670 $2.93 78,852
GPT-6.1 Terra self-refine 2; max 2,661 $0.75 144,716
GPT-6 Sol self-refine 16; max 2,637 $8.42 613,072
GPT-6 Sol self-refine 8; max 2,633 $4.11 364,234
Gemini 4 Carbon self-refine 2; high 2,631 $3.43 136,364
GPT-6 Sol self-refine 4; max 2,629 $2.39 222,392
Claude Opus 5 self-refine 16; high 2,622 $32 436,075
Claude Fable 5.2 self-refine 2; high 2,615 $2.70 58,266
GPT-6.1 Terra self-refine 1; max 2,609 $0.41 82,725
Gemini 4 Argon self-refine 16; low 2,599 $6.85 214,213
Claude Opus 5 self-refine 8; high 2,582 $14 284,182
Claude Fable 5.1 self-refine 8; high 2,581 $15 160,732
GPT-6 Sol self-refine 2; max 2,573 $1.42 137,070
Claude Fable 5.2 self-refine 1; high 2,510 $1.50 36,749
Claude Opus 5.5 self-refine 4; high 2,504 $1.16 40,667
Gemini 4 Argon self-refine 8; low 2,481 $3.02 97,520
Claude Fable 5.1 self-refine 4; high 2,470 $7.05 98,600
Claude Fable 5 self-refine 16; high 2,463 $41 283,789
GPT-6 Sol self-refine 1; max 2,462 $0.84 84,338
Claude Sonnet 5.5 self-refine 16; high 2,448 $4.05 143,200
Claude Opus 5 self-refine 4; high 2,436 $6.50 186,068
GPT-5.6 Sol self-refine 16; max 2,410 $15 324,870
GPT-5.6 Sol self-refine 8; max 2,391 $7.36 194,033
Claude Fable 5 self-refine 8; high 2,378 $17 173,878
Gemini 4 Carbon self-refine 1; high 2,375 $2.07 88,129
GPT-5.6 Sol self-refine 4; max 2,365 $4.33 118,903
Gemini 4 Argon self-refine 2; high 2,359 $3.17 118,327
Claude Opus 5 self-refine 2; high 2,340 $3.38 118,659
Claude Opus 5.5 self-refine 2; high 2,335 $0.52 21,703
Claude Fable 5 self-refine 4; high 2,326 $7.78 111,234
GPT-5.6 Sol self-refine 2; max 2,316 $2.67 76,165
Gemini 4 Argon self-refine 4; low 2,298 $1.06 40,570
Claude Sonnet 5.5 self-refine 8; high 2,289 $1.49 80,800
Claude Fable 5.1 self-refine 2; high 2,287 $3.79 64,890
Claude Fable 5 self-refine 2; high 2,194 $3.96 69,099
Gemini 3.8 Flash self-refine 16; high 2,182 $3.18 298,127
GPT-5.6 Sol self-refine 1; max 2,177 $1.53 50,026
Claude Opus 5 self-refine 1; high 2,165 $1.89 74,788
GPT-5.6 Terra self-refine 16; max 2,154 $7.74 325,392
Claude Sonnet 5.5 self-refine 4; high 2,151 $0.57 40,834
Claude Opus 5.5 self-refine 1; high 2,147 $0.27 12,726
Claude Fable 5.1 self-refine 1; high 2,143 $2.29 45,064
Kimi K3.1 Pro self-refine 16; max 2,132 $5.06 335,122
GPT-5.6 Terra self-refine 8; max 2,122 $4.56 239,257
GPT-5.5 self-refine 16; xhigh 2,108 $15 423,109
Claude Opus 4.8 self-refine 16; high 2,102 $19 264,965
GPT-5.6 Terra self-refine 4; max 2,101 $3.10 171,863
Gemini 4 Argon self-refine 1; high 2,100 $1.95 79,328
Kimi K3.1 Pro self-refine 8; max 2,098 $2.72 208,767
GPT-5.5 self-refine 8; xhigh 2,095 $8.03 247,271
Gemini 3.8 Flash self-refine 8; high 2,086 $1.67 171,249
Kimi K3.1 Pro self-refine 4; max 2,064 $1.70 140,348
GPT-5.5 self-refine 4; xhigh 2,061 $4.56 145,446
GPT-5.6 Terra self-refine 2; max 2,056 $2.14 124,862
Claude Fable 5 self-refine 1; high 2,041 $2.11 41,344
Gemini 3.1 Pro self-refine 16; high 2,041 $6.68 435,611
DeepSeek V4.1 Flash 1017 self-refine 16; max 2,012 $0.76 584,138
Gemini 3.8 Flash self-refine 4; high 2,007 $0.96 104,011
GPT-5.5 self-refine 2; xhigh 2,006 $2.66 86,807
Kimi K3.1 Pro self-refine 2; max 1,996 $1.11 97,319
Claude Haiku 5.5 self-refine 16; high 1,996 $0.87 159,132
Kimi K3 self-refine 16; max 1,991 $9.80 389,282
Gemini 4 Argon self-refine 2; low 1,990 $0.41 17,152
Muse Spark 1.3 self-refine 16; xhigh 1,981 $2.95 270,896
Claude Opus 4.8 self-refine 8; high 1,970 $7.56 158,026
Qwen3.8 Max (0902) self-refine 16; xhigh 1,964 $4.36 424,895
Claude Sonnet 5.5 self-refine 2; high 1,953 $0.24 20,470
Gemini 3.1 Pro self-refine 8; high 1,952 $3.14 233,175
GPT-5.6 Terra self-refine 1; max 1,951 $1.25 83,497
GPT-5.5 self-refine 1; xhigh 1,943 $1.52 50,121
GPT-5.5 self-refine 16; medium 1,938 $5.49 149,622
GLM-5.3 self-refine 16; high 1,937 $1.86 318,004
GPT-5.3 Codex self-refine 16; xhigh 1,934 $5.91 365,412
Kimi K3.1 Pro self-refine 1; max 1,933 $0.66 65,883
Claude Haiku 5.5 self-refine 8; high 1,931 $0.24 105,528
Muse Spark 1.3 self-refine 8; xhigh 1,924 $1.37 166,019
GPT-5.3 Codex self-refine 8; xhigh 1,919 $3.37 220,760
Claude Opus 4.8 self-refine 16; no-thinking 1,913 $12 163,896
Kimi K3 self-refine 8; max 1,908 $5.58 256,363
Qwen3.8 Max (0902) self-refine 8; xhigh 1,908 $2.43 273,273
Claude Opus 4.7 self-refine 16; no-thinking 1,897 $12 178,376
GPT-5.5 self-refine 8; medium 1,892 $2.58 74,573
GPT-5.3 Codex self-refine 4; xhigh 1,891 $1.93 130,249
Gemini 3.7 Flash self-refine 16; high 1,876 $2.34 210,320
DeepSeek V4.1 Flash 1017 self-refine 8; max 1,872 $0.47 370,024
GLM-5.3 self-refine 8; high 1,863 $1.10 218,161
Claude Haiku 5.5 self-refine 4; high 1,852 $0.07 71,958
Qwen3.8 Max (0902) self-refine 4; xhigh 1,848 $1.50 186,845
Gemini 3.7 Flash self-refine 8; high 1,840 $1.20 120,585
Muse Spark 1.3 self-refine 4; xhigh 1,837 $0.68 103,433
Claude Opus 4.7 self-refine 8; no-thinking 1,836 $4.15 94,706
Muse Spark Open self-refine 16; xhigh 1,830 $1.72 218,114
DeepSeek V4.1 Flash self-refine 16; max 1,829 $0.65 491,134
GPT-5.6 Luna self-refine 16; max 1,829 $2.29 248,854
Gemini 3.6 Flash self-refine 16; high 1,825 $2.53 210,474
Grok 4.7 self-refine 16; xhigh 1,823 $7.12 338,309
Grok 4.6 self-refine 16; xhigh 1,822 $2.32 155,981
Claude Opus 4.8 self-refine 8; no-thinking 1,820 $4.28 90,883
Claude Sonnet 5.5 self-refine 1; high 1,819 $0.11 10,259
GPT-5.4 self-refine 16; high 1,811 $5.02 274,123
GPT-5.4 self-refine 4; high 1,807 $1.69 107,445
GPT-5.3 Codex self-refine 2; xhigh 1,803 $1.06 72,549
Claude Opus 4.8 self-refine 4; high 1,802 $3.48 98,872
Claude Opus 4.8 self-refine 4; no-thinking 1,799 $1.80 50,806
GPT-6 Luna self-refine 16; xhigh 1,791 $0.12 205,947
GPT-5.5 self-refine 4; medium 1,791 $1.26 38,074
Kimi K3 self-refine 4; max 1,788 $3.37 181,800
GPT-5.6 Luna self-refine 8; max 1,785 $1.38 171,336
Muse Spark Open self-refine 8; xhigh 1,783 $1.10 162,049
Claude Sonnet 5 self-refine 16; high 1,783 $12 252,107
DeepSeek V4.1 Flash 1017 self-refine 4; max 1,769 $0.29 232,059
Claude Sonnet 4.6 self-refine 16; medium 1,768 $28 606,412
Muse Spark 1.3 self-refine 2; xhigh 1,765 $0.40 73,770
GPT-5.6 Luna self-refine 4; max 1,763 $0.93 119,309
GPT-5.4 self-refine 8; high 1,752 $2.76 167,111
GPT-6 Luna self-refine 8; xhigh 1,749 $0.07 121,527
Grok 4.6 self-refine 8; xhigh 1,745 $1.16 109,540
Qwen3.8 Max (0902) self-refine 2; xhigh 1,744 $0.96 131,244
Claude Haiku 5.5 self-refine 2; high 1,738 $0.03 52,685
GPT-5.5 self-refine 2; medium 1,735 $0.63 20,032
Grok 4.7 self-refine 8; xhigh 1,732 $3.24 248,030
DeepSeek V4 Pro 0813 self-refine 16; max 1,730 $0.41 407,082
Gemini 3 Pro self-refine 16; high 1,729 $3.03 211,744
DeepSeek V4.1 Flash self-refine 8; max 1,727 $0.39 310,246
GPT-5.6 Luna self-refine 2; max 1,725 $0.64 85,010
GLM-5.3 self-refine 4; high 1,724 $0.72 154,248
Gemini 3 Flash self-refine 16; high 1,722 $0.85 248,192
GPT-5.4 self-refine 16; medium 1,717 $3.32 183,870
Claude Sonnet 5 self-refine 8; high 1,717 $5.40 177,970
Gemini 3 Pro self-refine 16; low 1,715 $1.88 107,911
GPT-5.2 self-refine 16; high 1,715 $3.19 198,415
GPT-6 Luna self-refine 4; xhigh 1,714 $0.04 77,788
Gemini 3.5 Flash self-refine 16; high 1,712 $2.94 206,748
GPT-5.4 self-refine 2; high 1,710 $1.07 69,846
Gemini 4 Argon self-refine 1; low 1,705 $0.17 7,696
Muse Spark Open self-refine 4; xhigh 1,704 $0.75 121,985
Claude Opus 4.7 self-refine 4; no-thinking 1,695 $1.44 42,395
Kimi K2.6 self-refine 16 1,695 $1.65 363,128
Mercury 3 self-refine 16; high 1,692 $0.28 349,305
Claude Opus 4.8 self-refine 2; high 1,690 $1.80 62,602
Muse Spark 1.3 self-refine 1; xhigh 1,688 $0.22 46,065
Claude Sonnet 4.6 self-refine 8; medium 1,688 $13 431,416
DeepSeek V4 Flash 0731 self-refine 8; max 1,686 $0.07 211,483
GPT-5.4 self-refine 4; medium 1,686 $0.94 58,237
GPT-5.2 Codex self-refine 8; xhigh 1,686 $3.85 266,602
GLM-5.2 self-refine 16; max 1,685 $1.95 379,028
Qwen3.8 Max (0902) self-refine 1; xhigh 1,679 $0.53 87,126
DeepSeek V4 Flash 0731 self-refine 16; max 1,679 $0.10 315,684
DeepSeek V4.1 Flash 1017 self-refine 2; max 1,677 $0.17 142,213
Gemini 3.1 Pro self-refine 4; high 1,677 $1.59 124,633
Gemini 3 Pro self-refine 8; high 1,675 $1.56 116,306
Grok 4.6 self-refine 4; xhigh 1,673 $0.71 80,499
GPT-5.2 self-refine 16; medium 1,672 $1.93 114,434
GPT-5.6 Luna self-refine 1; max 1,667 $0.36 55,171
GPT-5.2 self-refine 8; high 1,660 $1.66 109,311
Gemini 3.5 Flash self-refine 8; high 1,657 $1.73 137,471
Mistral Large 4 self-refine 16 1,657 $7.37 443,669
GPT-5.3 Codex self-refine 1; xhigh 1,655 $0.56 38,820
Grok 4.7 self-refine 4; xhigh 1,650 $1.73 190,639
DeepSeek V4 Pro 0813 self-refine 8; max 1,649 $0.27 280,757
GLM-5.1 self-refine 16 1,646 $1.91 367,278
GPT-6 Luna self-refine 2; xhigh 1,642 $0.03 52,298
Muse Spark Open self-refine 2; xhigh 1,640 $0.52 95,196
Gemini 3.8 Flash self-refine 2; high 1,637 $0.58 67,184
GPT-5.5 self-refine 16 1,633 $2.86 70,475
Kimi K3 self-refine 2; max 1,633 $2.26 132,952
Gemini 3 Flash self-refine 8; high 1,632 $0.48 143,540
Gemini 3 Pro self-refine 8; low 1,628 $0.90 57,653
DeepSeek V4 Pro 0813 self-refine 4; max 1,628 $0.18 196,021
DeepSeek V4.1 Flash 1017 self-refine 1; max 1,622 $0.10 82,681
GPT-5.2 self-refine 8; medium 1,616 $0.95 59,567
Claude Opus 4.8 self-refine 2; no-thinking 1,613 $0.84 28,581
GPT-5.2 Codex self-refine 16; xhigh 1,613 $6.22 419,890
GLM-5.2 self-refine 8; max 1,610 $1.13 238,364
Claude Sonnet 5 self-refine 4; high 1,610 $2.79 128,326
GPT-5.4 self-refine 1; high 1,607 $0.61 40,236
Gemini 3.1 Pro self-refine 16; low 1,607 $0.93 51,851
Claude Opus 4.6 self-refine 16; no-thinking 1,607 $3.64 49,867
Gemini 3.6 Flash self-refine 8; high 1,601 $1.22 117,533
Grok 4.7 self-refine 2; xhigh 1,601 $1.12 150,569
GPT-5.2 Codex self-refine 4; xhigh 1,599 $2.10 147,540
Gemini 3 Flash self-refine 4; high 1,598 $0.28 86,466
Kimi K2.6 self-refine 8 1,593 $1.00 234,051
Grok 4.6 self-refine 2; xhigh 1,592 $0.44 59,616
GPT-5.4 self-refine 8; medium 1,592 $1.69 100,103
Muse Spark Open self-refine 1; xhigh 1,590 $0.33 68,206
DeepSeek V4 Flash 0731 self-refine 4; max 1,590 $0.04 146,360
GPT-5.5 self-refine 1; medium 1,589 $0.33 10,726
Mistral Large 4 self-refine 8 1,589 $3.08 297,056
Claude Haiku 5.5 self-refine 1; high 1,587 $0.02 40,947
GPT-5.2 self-refine 4; high 1,586 $0.93 63,256
GPT-6 Luna self-refine 1; xhigh 1,577 $0.02 36,004
Gemini 3 Pro self-refine 4; high 1,571 $0.85 65,594
Gemini 3.7 Flash self-refine 4; high 1,567 $0.68 72,777
Claude Opus 4.8 self-refine 1; high 1,564 $0.96 37,725
Mercury 3 self-refine 8; high 1,561 $0.17 230,078
Claude Sonnet 4.6 self-refine 4; medium 1,560 $6.60 305,748
GPT-5.5 self-refine 8 1,558 $1.18 30,164
Gemini 3.5 Flash self-refine 4; high 1,552 $1.04 94,109
DeepSeek V4 Pro 0813 self-refine 2; max 1,550 $0.12 135,811
Grok 4.7 self-refine 1; xhigh 1,546 $0.67 111,375
GPT-5.5 self-refine 4 1,545 $0.47 12,780
GPT-5 self-refine 16 1,545 $2.41 214,703
GPT-5.4 self-refine 2; medium 1,538 $0.53 33,469
GPT-5.1-Codex-Max self-refine 16; xhigh 1,538 $1.76 157,492
Claude Sonnet 5 self-refine 2; high 1,533 $1.62 93,656
GPT-5.1-Codex self-refine 16; high 1,528 $2.75 248,267
Kimi K3 self-refine 1; max 1,524 $1.28 84,872
GPT-5.1-Codex self-refine 8; high 1,522 $1.98 188,560
Gemini 3.1 Pro self-refine 8; low 1,522 $0.43 25,678
GPT-5.4 self-refine 1; medium 1,521 $0.26 16,667
Claude Opus 4.7 self-refine 2; no-thinking 1,521 $0.53 17,643
Kimi K2.6 self-refine 4 1,520 $0.61 146,374
GPT-5.1-Codex-Max self-refine 8; xhigh 1,519 $1.13 106,581
Tencent Hy3 self-refine 16; high 1,516 $0.21 256,528
GPT-5.2 Codex self-refine 2; xhigh 1,512 $1.11 78,612
GLM-5.2 self-refine 8; high 1,510 $0.77 156,480
Grok 4.6 self-refine 1; xhigh 1,508 $0.24 39,584
GPT-5.2 self-refine 4; medium 1,502 $0.48 31,916
DeepSeek V4 Flash 0731 self-refine 2; max 1,501 $0.03 103,996
GLM-5.2 self-refine 4; max 1,500 $0.64 139,654
GPT-5.1-Codex self-refine 4; high 1,497 $1.41 137,597
MiniMax-M3.1-Flash self-refine 16; max 1,495 $0.22 259,858
Beam 1.0 self-refine 16; max 1,493 $0.26 129,876
Mistral Large 4 self-refine 4 1,492 $1.45 201,177
GPT-5 self-refine 8 1,491 $1.30 121,683
DeepSeek-V4-Pro self-refine 16; high 1,489 $0.96 203,670
GLM-5.1 self-refine 8 1,489 $1.05 220,208
Claude Sonnet 4.6 self-refine 2; medium 1,487 $3.50 193,607
GPT-5.4 Mini self-refine 16; high 1,487 $3.01 613,006
Mercury 3 self-refine 4; high 1,483 $0.12 157,458
MiMo-V2.6 Pro self-refine 8 1,476 $0.36 259,556
DeepSeek V4.1 Flash self-refine 4; max 1,476 $0.24 197,117
Gemini 3 Flash self-refine 2; high 1,472 $0.16 52,643
Grok 4.5 self-refine 8; high 1,467 $0.64 61,626
GLM-5.2 self-refine 16; high 1,463 $1.32 236,942
Claude Sonnet 5 self-refine 1; high 1,463 $0.96 63,198
Gemini 3 Pro self-refine 4; low 1,463 $0.46 32,543
GPT-5.1-Codex-Max self-refine 16; high 1,458 $1.52 132,446
Claude Opus 4.6 self-refine 8; no-thinking 1,457 $1.15 22,283
GPT-5.1-Codex-Max self-refine 4; xhigh 1,457 $0.75 73,338
MiMo-V2.6 Flash self-refine 16 1,455 $0.15 298,521
GLM-5.3 self-refine 2; high 1,454 $0.50 110,512
GPT-5.2 self-refine 2; high 1,453 $0.54 37,526
GPT-5.4 Mini self-refine 8; high 1,452 $1.84 385,986
DeepSeek-V4-Pro self-refine 8; high 1,445 $0.56 137,320
GPT-5.4 self-refine 16 1,445 $1.58 73,596
Grok 4.5 self-refine 16; high 1,441 $1.23 88,395
MiMo-V2.6 Pro self-refine 16 1,441 $0.48 330,356
GPT-5.1-Codex-Max self-refine 8; high 1,436 $0.92 84,979
GPT-5 self-refine 4 1,431 $0.75 72,012
Beam 1.0 self-refine 8; max 1,430 $0.14 78,936
Claude Opus 4.5 self-refine 16 1,429 $5.38 90,618
GPT-5.1 self-refine 16 1,425 $2.24 200,979
Kimi K2.7 Code self-refine 16 1,422 $1.38 201,905
GPT-5 mini self-refine 16 1,421 $0.44 185,945
Grok 4.5 self-refine 4; high 1,417 $0.40 46,003
GPT-5.4 Mini self-refine 4; high 1,415 $1.04 219,981
Claude Opus 4.8 self-refine 1; no-thinking 1,412 $0.42 16,037
Gemini 3.6 Flash self-refine 4; high 1,409 $0.68 71,443
MiMo-V2.6 Flash self-refine 8 1,406 $0.10 228,748
DeepSeek V4 Pro 0813 self-refine 1; max 1,403 $0.07 82,224
GPT-5.1 self-refine 8 1,401 $1.34 126,771
GPT-5.4 self-refine 8 1,397 $0.65 33,667
GPT-5.5 self-refine 2 1,394 $0.19 5,494
Gemma 4 31B IT self-refine 16 1,394 $0.11 113,077
Gemma 4 31B IT self-refine 8 1,386 $0.04 63,897
DeepSeek-V4-Flash self-refine 16; high 1,380 $0.05 95,549
Claude Opus 4.5 self-refine 8 1,377 $2.44 58,097
GPT-5.1-Codex self-refine 2; high 1,373 $0.94 93,318
Gemini 3.1 Pro self-refine 4; low 1,373 $0.20 12,254
GPT-5.2 self-refine 2; medium 1,372 $0.26 18,090
GLM-5.2 self-refine 4; high 1,372 $0.48 103,743
Gemini 3 Flash self-refine 1; high 1,367 $0.09 29,988
GPT-5.1-Codex-Max self-refine 4; high 1,364 $0.60 57,645
Mercury 3 self-refine 2; high 1,360 $0.08 107,498
GLM-5.1 self-refine 4 1,353 $0.57 124,395
Nemotron 3.5 Super VL self-refine 16 1,352 $0.08 129,309
MiMo-V2.6 Pro self-refine 4 1,350 $0.26 193,822
MiMo-V2.6 Flash self-refine 4 1,349 $0.08 180,307
Gemini 3 Pro self-refine 2; high 1,348 $0.48 38,451
GPT-5.1-Codex-Max self-refine 2; xhigh 1,348 $0.52 51,618
Grok 4.5 self-refine 2; high 1,348 $0.28 36,784
MiMo-V2-Pro self-refine 16 1,347 $0.66 94,621
Tencent Hy3 self-refine 8; high 1,343 $0.12 176,544
GPT-5.1 self-refine 4 1,336 $0.86 83,092
Mistral Large 4 self-refine 2 1,332 $0.73 134,854
Claude Sonnet 4.6 self-refine 1; medium 1,327 $1.77 107,010
Claude Opus 4.7 self-refine 1; no-thinking 1,323 $0.22 8,148
MiMo-V2.6 Flash self-refine 2 1,321 $0.06 153,410
Tencent Hy3 self-refine 4; high 1,321 $0.07 116,366
Claude Opus 4.6 self-refine 4; no-thinking 1,320 $0.39 9,728
GLM-5.3 self-refine 1; high 1,317 $0.28 63,718
Kimi K2.7 Code self-refine 8 1,316 $0.77 135,466
GPT-5.1-Codex-Max self-refine 2; high 1,316 $0.40 39,631
MiniMax-M3.1-Flash self-refine 8; max 1,314 $0.13 174,556
GLM-5-Turbo self-refine 16 1,313 $1.06 220,695
GPT-5.4 Nano self-refine 16; high 1,312 $0.25 180,258
GPT-5 mini self-refine 8 1,311 $0.21 98,554
Grok 4.5 self-refine 1; high 1,309 $0.18 28,869
DeepSeek-V4-Pro self-refine 4; high 1,308 $0.35 92,343
DeepSeek V4 Flash 0731 self-refine 1; max 1,306 $0.02 68,384
GPT-5.4 self-refine 4 1,304 $0.27 15,523
o3 self-refine 16; high 1,304 $1.28 118,202
Beam 1.0 self-refine 4; max 1,304 $0.08 51,482
GPT-5.2 Codex self-refine 1; xhigh 1,300 $0.59 41,699
Claude Opus 4.5 self-refine 4 1,297 $1.28 37,133
Kimi K2.6 self-refine 2 1,297 $0.36 89,462
DeepSeek-V4-Flash self-refine 8; high 1,296 $0.02 60,492
GPT-5.2 self-refine 1; high 1,294 $0.30 21,051
GPT-5.4 Nano self-refine 8; high 1,292 $0.14 102,050
DeepSeek V4.1 Flash self-refine 2; max 1,287 $0.14 115,240
GLM-5.3-Flash self-refine 16; high 1,283 $0.11 126,453
GPT-5.1 self-refine 2 1,281 $0.55 54,217
Kimi K2.5 self-refine 16 1,279 $0.33 81,961
GPT-5.4 Mini self-refine 2; high 1,277 $0.55 116,425
Step 3.7 Flash self-refine 16; medium 1,277 $0.38 302,269
Nemotron 3.5 Super VL self-refine 8 1,276 $0.05 96,372
Qwen 3.6 Plus self-refine 16 1,273 $1.07 279,130
DeepSeek-V4-Pro self-refine 16; no-thinking 1,272 $0.45 60,531
o4-mini self-refine 16; high 1,271 $0.82 146,340
MiMo-V2.6 Pro self-refine 2 1,270 $0.19 143,599
Gemini 3.8 Flash self-refine 1; high 1,270 $0.34 45,251
Gemini 3.1 Pro self-refine 2; high 1,270 $0.79 63,876
MiMo-V2-Pro self-refine 8 1,262 $0.34 61,062
GPT-5 self-refine 2 1,262 $0.44 42,754
Solar Pro 4 self-refine 16; max 1,256 $0.42 195,531
GLM-5.2 self-refine 2; max 1,255 $0.33 72,575
MiMo-V2.5-Pro self-refine 16 1,254 $0.61 97,456
GPT-5.2 self-refine 1; medium 1,250 $0.14 10,094
GPT-5.1-Codex self-refine 1; high 1,245 $0.51 51,031
MiMo-V2.6 Flash self-refine 1 1,244 $0.04 115,534
Kimi K2.7 Code self-refine 4 1,243 $0.46 92,350
Gemini 3 Pro self-refine 2; low 1,241 $0.24 17,802
Kimi K2.5 self-refine 8 1,240 $0.20 56,361
Step 3.7 Flash self-refine 8; medium 1,236 $0.22 178,875
GPT-5.1-Codex-Max self-refine 1; xhigh 1,228 $0.32 31,586
Gemma 4 31B IT self-refine 4 1,226 $0.02 36,628
o3 self-refine 8; high 1,223 $0.68 72,433
Gemini 3.1 Pro self-refine 2; low 1,223 $0.09 5,841
GPT-5.4 Nano self-refine 4; high 1,214 $0.08 60,593
Mistral Large 4 self-refine 1 1,213 $0.46 103,988
LongCat-2.0 self-refine 8 1,212 $0.13 79,843
Grok 4.3 self-refine 16 1,211 $0.71 65,362
Gemini 3.7 Flash self-refine 2; high 1,210 $0.41 46,830
GPT-5.1-Codex-Max self-refine 1; high 1,209 $0.25 25,138
Mercury 3 self-refine 1; high 1,209 $0.05 68,745
GLM-5-Turbo self-refine 8 1,208 $0.62 141,498
MiMo-V2.5-Pro self-refine 8 1,204 $0.33 66,220
LongCat-2.0 self-refine 16 1,204 $0.19 113,498
GLM-5.2 self-refine 2; high 1,201 $0.30 66,097
GPT-5.4 self-refine 2 1,200 $0.12 7,221
Inkling self-refine 16; max 1,195 $1.90 143,557
GLM-5 self-refine 16 1,194 $0.50 96,788
GPT-5.1 self-refine 1 1,192 $0.32 31,365
Claude Sonnet 4.5 self-refine 16 1,192 $2.30 76,490
o4-mini self-refine 8; high 1,192 $0.42 84,693
Mercury 2 self-refine 16 1,190 $0.17 116,467
GPT-5.4 Mini self-refine 1; high 1,189 $0.27 58,094
DeepSeek-V4-Pro self-refine 8; no-thinking 1,188 $0.21 31,831
o3 self-refine 4; high 1,186 $0.38 43,139
Claude Opus 4.6 self-refine 2; no-thinking 1,186 $0.15 4,329
Claude Opus 4.1 self-refine 16 1,183 $12 60,856
Mercury 2 self-refine 8 1,183 $0.08 72,634
DeepSeek-V4-Pro self-refine 2; high 1,182 $0.23 65,315
Gemini 3.5 Flash self-refine 2; high 1,178 $0.68 66,876
Gemini 3 Pro self-refine 1; high 1,177 $0.26 21,301
GPT-5 mini self-refine 4 1,172 $0.11 50,785
GLM-5.1 self-refine 2 1,170 $0.29 64,093
Gemini 3.1 Flash-Lite self-refine 8; high 1,165 $0.14 77,549
GPT-5 self-refine 1 1,162 $0.24 24,148
Mercury 2 self-refine 4 1,162 $0.04 41,906
Gemini 3.1 Pro self-refine 1; high 1,161 $0.39 31,824
Tencent Hy3 self-refine 2; high 1,159 $0.05 73,481
MiMo-V2.6 Pro self-refine 1 1,158 $0.11 103,073
DeepSeek-V4-Flash self-refine 4; high 1,158 $0.01 40,205
Qwen 3.6 Plus self-refine 8 1,158 $0.46 132,858
Grok 4.3 self-refine 8 1,155 $0.33 48,197
Nemotron 3.5 Super VL self-refine 4 1,152 $0.04 76,953
Inkling self-refine 4; max 1,150 $0.53 83,781
Claude Opus 4.5 self-refine 2 1,149 $0.58 19,838
Nemotron 3.5 Super VL self-refine 2 1,142 $0.03 62,916
Beam 1.0 self-refine 2; max 1,139 $0.05 35,464
MiMo-V2.5-Pro self-refine 4 1,138 $0.22 49,721
Gemini 3.5 Flash-Lite self-refine 16; high 1,137 $0.34 74,638
GLM-5.3-Flash self-refine 8; high 1,136 $0.06 72,649
Inkling self-refine 8; max 1,135 $0.95 108,952
Mistral Medium 3.5 self-refine 16 1,131 $1.96 185,928
Grok 4.3 self-refine 4 1,131 $0.18 35,998
GPT-5.5 self-refine 1 1,128 $0.07 2,208
Kimi K2.5 self-refine 4 1,127 $0.13 38,217
MiMo-V2-Pro self-refine 4 1,122 $0.20 42,482
Gemini 3.6 Flash self-refine 2; high 1,121 $0.40 45,255
gpt-oss-120b self-refine 16 1,119 $0.07 51,576
Gemini 3.5 Flash-Lite self-refine 8; high 1,111 $0.19 49,873
Grok 4.1 Fast self-refine 16 1,111 $0.08 113,265
o4-mini self-refine 4; high 1,110 $0.21 44,725
LongCat-2.0 self-refine 4 1,110 $0.09 57,101
Gemini 3.1 Flash-Lite self-refine 16; high 1,110 $0.26 130,252
Ling-2.6-1T self-refine 16 1,108 $0.17 43,480
MiMo-V2.5 self-refine 16 1,106 $0.29 85,911
GPT-5 self-refine 16 1,105 $0.96 75,657
Kimi K2 Thinking self-refine 16 1,103 $0.53 96,943
Claude Opus 4.1 self-refine 8 1,099 $4.08 29,847
Beam 1.0 self-refine 1; max 1,098 $0.03 25,595
GPT-5.4 Nano self-refine 2; high 1,096 $0.05 37,082
Kimi K2.6 self-refine 1 1,093 $0.20 48,987
DeepSeek V4.1 Flash self-refine 1; max 1,092 $0.08 67,313
Gemma 4 31B IT self-refine 2 1,091 $0.009 20,367
GLM-5 self-refine 8 1,090 $0.29 72,282
Grok 4.3 self-refine 2 1,088 $0.10 27,098
Claude Sonnet 4 self-refine 16 1,088 $2.53 79,799
Kimi K2 Thinking self-refine 8 1,087 $0.30 65,632
GPT-5.4 self-refine 1 1,086 $0.05 3,225
Gemini 3.1 Flash-Lite self-refine 4; high 1,083 $0.08 47,034
MiMo-V2.5-Pro self-refine 2 1,080 $0.16 40,403
DeepSeek-V3.1-Terminus self-refine 16 1,078 $0.21 112,586
Kimi K2.5 self-refine 2 1,077 $0.08 24,803
Solar Pro 4 self-refine 8; max 1,073 $0.20 117,742
Claude Sonnet 4.5 self-refine 8 1,072 $1.00 47,587
Gemini 3.5 Flash-Lite self-refine 4; high 1,072 $0.11 34,398
Qwen3.5-Plus self-refine 16 1,071 $0.39 70,465
o3 self-refine 2; high 1,067 $0.20 23,659
GPT-5 self-refine 8 1,066 $0.40 33,295
gpt-oss-120b self-refine 8 1,066 $0.02 24,520
Mistral Medium 3.5 self-refine 8 1,064 $1.14 132,331
Inkling self-refine 2; max 1,062 $0.32 64,945
DeepSeek-R1 May 2025 self-refine 16 1,059 $0.40 131,485
Step 3.7 Flash self-refine 4; medium 1,059 $0.13 106,930
Gemini 3.1 Pro self-refine 1; low 1,055 $0.04 2,848
MiniMax-M3 self-refine 16 1,053 $0.68 150,432
GLM-5-Turbo self-refine 4 1,053 $0.35 83,121
DeepSeek-V3.1-Terminus self-refine 8 1,052 $0.10 70,875
MiMo-V2.5 self-refine 8 1,052 $0.15 52,764
DeepSeek-V4-Flash self-refine 16; no-thinking 1,051 $0.03 42,018
GLM-5.2 self-refine 1; high 1,047 $0.16 35,036
Kimi K2.7 Code self-refine 2 1,046 $0.30 64,991
Qwen 3.6 Plus self-refine 4 1,045 $0.21 64,359
Claude Sonnet 4 self-refine 8 1,042 $1.08 48,749
MiniMax-M3.1-Flash self-refine 4; max 1,041 $0.09 124,023
Qwen3.5-Plus self-refine 8 1,039 $0.14 37,637
MiniMax-M2.5 self-refine 16 1,033 $0.22 124,985
GPT-5 self-refine 4 1,029 $0.18 15,443
Claude Opus 4.5 self-refine 1 1,025 $0.29 11,025
MiniMax-M3 self-refine 8 1,025 $0.40 108,643
Grok 4.1 Fast self-refine 8 1,023 $0.05 88,858
MiniMax-M2.7 self-refine 16 1,021 $0.22 136,430
Nemotron 3 Super self-refine 16 1,017 $0.07 80,540
Nemotron 3.5 Lightning 30B-A3B self-refine 16 1,017 $0.07 183,145
Ling-2.6-1T self-refine 8 1,017 $0.08 22,189
GPT-5 nano self-refine 16 1,016 $0.10 226,017
Qwen 3.6 Flash self-refine 16 1,015 $0.41 197,982
o4-mini self-refine 2; high 1,014 $0.10 22,131
Claude Opus 4.1 self-refine 4 1,014 $1.62 15,417
GLM-4.7 self-refine 16 1,011 $0.47 143,093
Claude Sonnet 4.5 self-refine 4 1,011 $0.53 29,817
Mistral Medium 3.5 self-refine 4 1,010 $0.69 86,752
GLM-5.2 self-refine 1; max 1,010 $0.15 33,098
DeepSeek-R1 May 2025 self-refine 8 1,009 $0.19 82,140
GLM-5 self-refine 4 1,009 $0.19 52,253
MiniMax-M2.5 self-refine 4 1,007 $0.10 73,089
Qwen3.5 397B-A17B self-refine 16 1,007 $0.62 76,390
Claude Haiku 4.5 self-refine 16 1,006 $0.94 113,295
DeepSeek-V4-Pro self-refine 1; high 1,006 $0.15 42,362
GPT-5.4 Nano self-refine 1; high 1,005 $0.03 21,398
MiMo-V2-Flash self-refine 16 1,000 $0.07 109,164
MiniMax-M2.1 self-refine 16 998 $0.43 291,132
Claude Opus 4.6 self-refine 1; no-thinking 997 $0.06 1,796
GPT-5 mini self-refine 2 994 $0.05 24,564
Nemotron 3.5 Lightning 30B-A3B self-refine 8 993 $0.04 138,605
DeepSeek-V3.1-Terminus self-refine 4 993 $0.05 46,556
GPT-5 nano self-refine 8 992 $0.06 132,399
MiniMax-M2.7 self-refine 8 991 $0.16 110,122
Claude Haiku 4.5 self-refine 8 991 $0.45 71,208
Kimi K2 Thinking self-refine 4 991 $0.18 48,813
Claude Sonnet 4 self-refine 4 990 $0.56 30,740
Gemini 3 Pro self-refine 1; low 988 $0.10 7,996
MiniMax-M2.5 self-refine 8 988 $0.14 96,159
GPT-4.1 self-refine 16 985 $0.76 46,311
Mercury 2 self-refine 2 982 $0.02 21,148
Qwen3.5 397B-A17B self-refine 8 978 $0.23 40,843
Mercury 2.5 self-refine 16; high 977 $0.10 66,320
MiMo-V2-Flash self-refine 8 974 $0.03 70,313
DeepSeek-V4-Pro self-refine 4; no-thinking 971 $0.10 16,510
DeepSeek-V4-Flash self-refine 2; high 967 $0.008 27,768
DeepSeek-R1 May 2025 self-refine 4 967 $0.11 50,983
GLM-5.3-Flash self-refine 4; high 961 $0.03 43,023
gpt-oss-120b self-refine 4 957 $0.009 11,509
Nemotron 3.5 Lightning 30B-A3B self-refine 4 957 $0.03 98,720
Nemotron 3.5 Super VL self-refine 1 957 $0.02 45,954
GPT-5 self-refine 2 956 $0.09 7,697
MiMo-V2-Pro self-refine 2 954 $0.12 31,227
Inkling self-refine 1; max 946 $0.20 48,499
Qwen3.7 Flash self-refine 16 946 $0.06 188,386
Grok 4.3 self-refine 1 944 $0.06 21,053
gpt-oss-20b self-refine 16 942 $0.03 66,352
MiMo-V2.5 self-refine 4 942 $0.09 35,111
Qwen3.5-Plus self-refine 4 936 $0.07 22,255
Claude Haiku 4.5 self-refine 4 936 $0.24 43,084
o3 self-refine 1; high 934 $0.10 12,304
MiniMax-M2.1 self-refine 8 931 $0.30 224,375
DeepSeek-V4-Flash self-refine 8; no-thinking 929 $0.01 20,340
Gemini 3.5 Flash-Lite self-refine 2; high 929 $0.07 23,837
Gemma 4 31B IT self-refine 1 926 $0.005 10,709
Gemini 3.1 Flash-Lite self-refine 2; high 925 $0.04 27,763
GLM-4.7 self-refine 8 923 $0.23 83,101
Solar Pro 4 self-refine 4; max 919 $0.11 74,316
Claude Sonnet 4.5 self-refine 2 919 $0.29 17,703
GPT-5 nano self-refine 4 916 $0.03 75,549
GPT-4.1 self-refine 8 915 $0.27 21,343
MiniMax-M2.1 self-refine 4 915 $0.19 152,999
Nemotron 3.5 Lightning 30B-A3B self-refine 2 914 $0.02 65,147
Gemini 3.5 Flash self-refine 1; high 911 $0.44 48,620
DeepSeek-R1 May 2025 self-refine 2 909 $0.07 32,910
LongCat-2.0 self-refine 2 908 $0.06 39,363
Gemini 3.7 Flash self-refine 1; high 904 $0.24 31,850
MiniMax-M3 self-refine 4 903 $0.26 80,144
MiMo-V2.5-Pro self-refine 1 900 $0.10 34,021
GLM-5-Turbo self-refine 2 900 $0.20 47,953
Qwen3.5 397B-A17B self-refine 4 898 $0.11 23,441
Mercury 2.5 self-refine 8; high 897 $0.04 38,416
Mistral Medium 3.5 self-refine 2 896 $0.44 57,158
Claude Opus 4.1 self-refine 2 894 $0.78 8,604
Grok 4.1 Fast self-refine 4 887 $0.03 57,802
GLM-5.1 self-refine 1 887 $0.13 28,926
Kimi K2.7 Code self-refine 1 886 $0.18 45,506
Qwen3.5-27B self-refine 16 882 $0.36 86,078
Nemotron 3 Super self-refine 8 880 $0.03 43,887
Qwen 3.6 Flash self-refine 8 874 $0.19 107,997
MiMo-V2-Flash self-refine 4 873 $0.02 47,729
Qwen3.7 Flash self-refine 8 872 $0.02 108,872
gpt-oss-20b self-refine 8 867 $0.01 34,231
DeepSeek-V3.1-Terminus self-refine 2 865 $0.03 29,792
Mistral Small 4 self-refine 16 862 $0.12 111,446
MiniMax-M2.1 self-refine 2 858 $0.11 90,905
Step 3.7 Flash self-refine 2; medium 855 $0.06 54,004
grok-code-fast-1 self-refine 16 853 $0.07 37,512
Kimi K2 Thinking self-refine 2 848 $0.13 38,234
Ling-2.6-1T self-refine 4 846 $0.04 10,928
Qwen 3.6 Plus self-refine 2 840 $0.10 33,118
GPT-5 nano self-refine 2 839 $0.02 38,144
grok-code-fast-1 self-refine 8 839 $0.04 23,931
Tencent Hy3 self-refine 1; high 839 $0.03 43,035
GLM-4.6 self-refine 16 838 $0.31 48,580
Claude Sonnet 4 self-refine 2 834 $0.32 19,574
GLM-5 self-refine 2 828 $0.12 36,199
Qwen3-Coder-Plus self-refine 16 827 $0.29 29,220
o4-mini self-refine 1; high 826 $0.04 9,564
Qwen3.5-Flash self-refine 16 826 $0.10 89,514
Claude Haiku 4.5 self-refine 2 823 $0.13 25,214
Qwen3-Coder-480B-A35B self-refine 16 822 $0.17 32,827
Kimi K2.5 self-refine 1 822 $0.05 15,508
MiniMax-M2.7 self-refine 4 821 $0.12 92,119
Qwen3.5-Plus self-refine 2 813 $0.04 14,824
GPT-5 self-refine 1 808 $0.04 4,027
gpt-oss-120b self-refine 2 807 $0.003 5,138
GLM-4.7 self-refine 4 805 $0.13 50,239
DeepSeek-R1 May 2025 self-refine 1 804 $0.04 20,931
GPT-5 mini self-refine 1 800 $0.02 9,960
Qwen3.7 Flash self-refine 4 798 $0.01 66,946
Gemini 3.1 Flash-Lite self-refine 1; high 798 $0.03 16,242
GPT-4.1 self-refine 4 797 $0.11 10,177
Qwen3-Coder-Plus self-refine 8 796 $0.13 16,107
Claude Sonnet 4.5 self-refine 1 796 $0.15 9,427
LongCat-2.0 self-refine 1 793 $0.04 29,709
GLM-4.6 self-refine 8 789 $0.12 28,449
Mistral Small 4 self-refine 8 787 $0.06 75,471
Mercury 2 self-refine 1 786 $0.007 9,229
MiMo-V2-Pro self-refine 1 785 $0.08 24,936
Qwen3.5-27B self-refine 8 784 $0.16 47,334
Qwen3-Coder-480B-A35B self-refine 8 784 $0.06 16,788
MiMo-V2-Flash self-refine 2 782 $0.01 32,971
grok-code-fast-1 self-refine 4 782 $0.02 15,172
gpt-oss-20b self-refine 4 775 $0.005 18,249
MiniMax-M3 self-refine 2 773 $0.17 58,406
Nemotron 3.5 Lightning 30B-A3B self-refine 1 769 $0.009 35,923
GLM-5 self-refine 1 766 $0.07 22,201
Gemini 3.5 Flash-Lite self-refine 1; high 765 $0.05 17,899
Mistral Medium 3.5 self-refine 1 764 $0.27 35,775
Qwen3.5-Flash self-refine 8 758 $0.04 61,645
MiniMax-M2.5 self-refine 2 750 $0.06 47,991
Qwen3-Coder-Plus self-refine 4 747 $0.06 8,780
DeepSeek-V3.1-Terminus self-refine 1 745 $0.02 18,032
DeepSeek-V4-Flash self-refine 4; no-thinking 744 $0.006 9,735
MiMo-V2-Flash self-refine 1 738 $0.007 21,750
MiMo-V2.5 self-refine 2 735 $0.06 24,672
Ling-2.6-1T self-refine 2 733 $0.02 5,115
Mistral Small 4 self-refine 4 729 $0.04 54,737
Qwen3.5 397B-A17B self-refine 2 725 $0.06 15,432
DeepSeek-V4-Pro self-refine 2; no-thinking 725 $0.05 8,370
GLM-4.6 self-refine 4 724 $0.05 17,705
GPT-5 nano self-refine 1 719 $0.007 17,654
Gemini 3.6 Flash self-refine 1; high 716 $0.24 31,944
Qwen3.5-27B self-refine 4 715 $0.09 32,082
MiniMax-M2.7 self-refine 2 711 $0.09 70,837
Nemotron 3 Super self-refine 4 707 $0.02 28,634
Grok 4.1 Fast self-refine 2 701 $0.02 32,632
Mistral Medium 3.1 self-refine 16 701 $0.32 50,974
Step 3.7 Flash self-refine 1; medium 694 $0.03 28,040
grok-code-fast-1 self-refine 2 694 $0.02 9,363
Qwen3-Coder-480B-A35B self-refine 4 681 $0.02 8,748
GPT-4.1 self-refine 2 679 $0.06 5,240
DeepSeek-V4-Flash self-refine 1; high 678 $0.005 18,163
Claude Opus 4.1 self-refine 1 675 $0.43 5,114
MiniMax-M3.1-Flash self-refine 2; max 671 $0.07 90,525
Qwen 3.6 Plus self-refine 1 670 $0.06 18,481
GLM-5.3-Flash self-refine 2; high 668 $0.02 28,833
Qwen 3.6 Flash self-refine 4 668 $0.11 65,102
Mercury 2.5 self-refine 4; high 665 $0.02 22,234
gpt-oss-20b self-refine 2 657 $0.002 8,428
Claude Sonnet 4 self-refine 1 655 $0.19 11,714
Claude Haiku 4.5 self-refine 1 653 $0.07 13,689
Qwen3.5-35B-A3B self-refine 16 648 $0.36 119,116
MiniMax-M3 self-refine 1 640 $0.11 43,775
GLM-5-Turbo self-refine 1 634 $0.11 26,853
MiniMax-M2.1 self-refine 1 624 $0.05 44,297
Qwen3.5 397B-A17B self-refine 1 622 $0.04 10,516
MiniMax-M2.5 self-refine 1 618 $0.04 30,193
Mistral Medium 3.1 self-refine 8 611 $0.11 24,405
Qwen3.5-Plus self-refine 1 609 $0.03 10,182
Qwen3.7 Flash self-refine 2 601 $0.006 42,408
MiniMax-M2.7 self-refine 1 599 $0.05 38,884
Kimi K2 Thinking self-refine 1 598 $0.09 30,387
Mistral Small 4 self-refine 2 588 $0.02 38,384
grok-code-fast-1 self-refine 1 588 $0.009 5,820
Qwen3.5-35B-A3B self-refine 8 582 $0.16 62,842
gpt-oss-120b self-refine 1 576 $0.001 1,997
Qwen3-Coder-Plus self-refine 2 570 $0.03 4,664
gpt-oss-20b self-refine 1 566 $0.0007 2,906
Qwen 3.6 Flash self-refine 2 562 $0.06 39,204
GLM-4.7 self-refine 2 560 $0.07 30,585
GPT-4.1 self-refine 1 558 $0.03 2,670
GLM-4.5 self-refine 16 552 $0.28 68,693
Llama 4 Maverick self-refine 16 552 $0.08 14,719
Qwen3-Coder-480B-A35B self-refine 2 551 $0.009 4,474
GLM-4.5 self-refine 8 540 $0.15 44,131
DeepSeek-V4-Flash self-refine 2; no-thinking 533 $0.002 4,633
Llama 4 Maverick self-refine 8 530 $0.03 8,272
Qwen3.5-Flash self-refine 4 526 $0.02 46,028
DeepSeek-V4-Pro self-refine 1; no-thinking 522 $0.02 4,198
MiMo-V2.5 self-refine 1 514 $0.04 18,431
GLM-4.5 self-refine 4 509 $0.08 29,551
Qwen3.5-27B self-refine 2 506 $0.06 21,815
Llama 4 Maverick self-refine 4 499 $0.01 4,635
Mistral Small 4 self-refine 1 498 $0.02 25,336
Nemotron 3 Super self-refine 2 494 $0.01 19,935
Mistral Medium 3.1 self-refine 4 487 $0.04 11,762
Solar Pro 4 self-refine 2; max 465 $0.06 46,847
Mercury 2.5 self-refine 2; high 464 $0.01 13,101
Qwen3-Coder-480B-A35B self-refine 1 461 $0.004 2,304
GLM-4.6 self-refine 2 461 $0.03 11,593
Qwen3-Coder-Plus self-refine 1 457 $0.01 2,389
Ling-2.6-1T self-refine 1 433 $0.008 2,649
GLM-4.5 self-refine 2 431 $0.05 20,482
Qwen3.7 Flash self-refine 1 420 $0.004 26,631
GLM-4.7 self-refine 1 399 $0.04 18,953
Qwen3.5-35B-A3B self-refine 4 399 $0.09 39,961
Grok 4.1 Fast self-refine 1 395 $0.009 17,413
Mistral Medium 3.1 self-refine 2 379 $0.02 5,813
Qwen3.5-Flash self-refine 2 369 $0.02 37,573
Qwen3.5-27B self-refine 1 349 $0.04 17,070
Llama 4 Maverick self-refine 2 349 $0.004 2,493
GLM-4.5 self-refine 1 345 $0.03 13,479
GLM-4.6 self-refine 1 341 $0.02 7,877
Qwen3.5-35B-A3B self-refine 2 333 $0.06 28,043
Qwen 3.6 Flash self-refine 1 326 $0.04 23,209
DeepSeek-V4-Flash self-refine 1; no-thinking 325 $0.0009 2,257
GLM-5.3-Flash self-refine 1; high 304 $0.01 20,965
Mercury 2.5 self-refine 1; high 302 $0.006 7,539
Qwen3.5-Flash self-refine 1 266 $0.01 29,723
Qwen3.5-35B-A3B self-refine 1 222 $0.04 18,528
MiniMax-M3.1-Flash self-refine 1; max 218 $0.05 76,343
Nemotron 3 Super self-refine 1 214 $0.008 15,788
Mistral Medium 3.1 self-refine 1 210 $0.007 3,062
Solar Pro 4 self-refine 1; max 179 $0.04 31,664
Llama 4 Maverick self-refine 1 173 $0.001 1,406

10 models have held number one on the official leaderboard since its release in June 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Google DeepMind Anthropic
Own leaderboard 10 leaders
GPT-6 Astra 3,164

Source: Official ALE-Bench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0 1,000 2,000 3,000 4,000 2025 2026 2027 0 1,000 2,000 3,000 4,000 2025 2026 2027

Source: Official ALE-Bench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471