Benchmark Almanac

MacroscopeBench

Claude Opus 5.5 83.9% leads since September 2026
Not in model cards

Models review commits from public repositories and must flag the known bugs a commit introduced without false reports. LLM judges match findings to the bugs.

195 tasks include control commits with no known bugs. Scored by combined precision and recall. 41 models in the Fireworks index.

Made by
Macroscope
Kind
Fixing real repositories
Level
Expert
Scored
Graded by a model
Answers
Open-ended
Tasks
Kept private
Strongest model tested
Claude Fable 5.2 #1 on the capability index
Last model added
November 2026
Scores refreshed
November 12th, 2026
16
Models scored on its own leaderboard

Claude Opus 5.5 leads with 83.9%

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

16 models. Scroll sideways for the rest.

  1. 83.9 Claude Opus 5.5
  2. 82.3 GPT-6.1 Terra
  3. 80.3 GPT-6 Sol
  4. 78.5 GPT-6.1 Sol
  5. 78.5 Claude Fable 5.2
  6. 78.0 GPT-6 Astra
  7. 77.8 GPT-5.6 Sol
  8. 77.4 GLM-5.3
  9. 75.9 Grok 4.6
  10. 72.9 Kimi K3
  11. 72.8 Gemini 4 Argon
  12. 72.5 GPT-6 Luna
  13. 72.0 DeepSeek V4.1 Flash
  14. 70.7 Gemini 3.8 Flash
  15. 69.9 Claude Opus 5
  16. 64.2 Muse Wonder
  1. 1 Claude Opus 5.5 max effort 83.9%
  2. 2 GPT-6.1 Terra high effort 82.3%
  3. 3 GPT-6 Sol max effort 80.3%
  4. 4 GPT-6.1 Sol max effort 78.5%
  5. 5 Claude Fable 5.2 xhigh effort 78.5%
  6. 6 GPT-6 Astra max effort 78%
  7. 7 GPT-5.6 Sol high effort 77.8%
  8. 8 GLM-5.3 max effort 77.4%
  9. 9 Grok 4.6 xhigh effort 75.9%
  10. 10 Kimi K3 max effort 72.9%
  11. 11 Gemini 4 Argon high effort 72.8%
  12. 12 GPT-6 Luna max effort 72.5%
  13. 13 DeepSeek V4.1 Flash max effort 72%
  14. 14 Gemini 3.8 Flash medium effort 70.7%
  15. 15 Claude Opus 5 high effort 69.9%
  16. 16 Muse Wonder low effort 64.2%

Source: Official MacroscopeBench leaderboard

#ModelScoreSettingDateSource
1 Claude Opus 5.5 83.9% max effort Sep 2026 Official leaderboard
2 GPT-6.1 Terra 82.3% high effort Oct 2026 Official leaderboard
3 GPT-6 Sol 80.3% max effort Sep 2026 Official leaderboard
4 GPT-6.1 Sol 78.5% max effort Sep 2026 Official leaderboard
5 Claude Fable 5.2 78.5% xhigh effort Nov 2026 Official leaderboard
6 GPT-6 Astra 78% max effort Sep 2026 Official leaderboard
7 GPT-5.6 Sol 77.8% high effort Jul 2026 Official leaderboard
8 GLM-5.3 77.4% max effort Aug 2026 Official leaderboard
9 Grok 4.6 75.9% xhigh effort Aug 2026 Official leaderboard
10 Kimi K3 72.9% max effort Jul 2026 Official leaderboard
11 Gemini 4 Argon 72.8% high effort Sep 2026 Official leaderboard
12 GPT-6 Luna 72.5% max effort Sep 2026 Official leaderboard
13 DeepSeek V4.1 Flash 72% max effort Sep 2026 Official leaderboard
14 Gemini 3.8 Flash 70.7% medium effort Sep 2026 Official leaderboard
15 Claude Opus 5 69.9% high effort Jul 2026 Official leaderboard
16 Muse Wonder 64.2% low effort Oct 2026 Official leaderboard

Score vs. inference cost

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

40 60 80 100 $0$2.5$5$7.5$10$12.5 Cost per code review ($), lower is better Score GPT-5.6 Sol (high effort): Score 77.8, cost per code review $3.9 GPT-5.6 Sol (low effort): Score 66.1, cost per code review $0.81 Kimi K3 (max effort): Score 72.9, cost per code review $2.82 Kimi K3 (high effort): Score 68.6, cost per code review $1.9 Kimi K3 (low effort): Score 53.5, cost per code review $0.48 Claude Opus 5 (high effort): Score 69.9, cost per code review $6.5 Claude Opus 5 (medium effort): Score 65.5, cost per code review $3.5 Claude Opus 5 (low effort): Score 59.4, cost per code review $1.4 Grok 4.6 (xhigh effort): Score 75.9, cost per code review $4.68 Grok 4.6 (high effort): Score 75.1, cost per code review $3.51 Grok 4.6 (medium effort): Score 72.8, cost per code review $2.56 Grok 4.6 (low effort): Score 57.1, cost per code review $0.3 GLM-5.3 (max effort): Score 77.4, cost per code review $3.86 GLM-5.3 (high effort): Score 69.4, cost per code review $2.02 GLM-5.3 (low effort): Score 54.6, cost per code review $0.41 Gemini 3.8 Flash (medium effort): Score 70.7, cost per code review $3.38 Gemini 3.8 Flash (low effort): Score 51.3, cost per code review $0.9 GPT-6 Astra (max effort): Score 78, cost per code review $7.87 DeepSeek V4.1 Flash (max effort): Score 72, cost per code review $0.75 DeepSeek V4.1 Flash (low effort): Score 70.7, cost per code review $0.38 DeepSeek V4.1 Flash (high effort): Score 70.5, cost per code review $0.38 Claude Opus 5.5 (xhigh effort): Score 78.9, cost per code review $3.51 Claude Opus 5.5 (high effort): Score 73.1, cost per code review $1.3 Claude Opus 5.5 (medium effort): Score 70.1, cost per code review $0.97 Claude Opus 5.5 (low effort): Score 61.4, cost per code review $0.68 GPT-6 Sol (max effort): Score 80.3, cost per code review $3.68 GPT-6 Sol (xhigh effort): Score 77.2, cost per code review $2 GPT-6 Sol (high effort): Score 74.3, cost per code review $1.23 GPT-6 Sol (medium effort): Score 69.2, cost per code review $0.69 GPT-6 Sol (low effort): Score 61.8, cost per code review $0.36 GPT-6.1 Sol (max effort): Score 78.5, cost per code review $1.05 GPT-6.1 Sol (xhigh effort): Score 77.2, cost per code review $0.79 GPT-6.1 Sol (high effort): Score 76.7, cost per code review $0.56 GPT-6.1 Sol (medium effort): Score 73.5, cost per code review $0.28 GPT-6.1 Sol (low effort): Score 70.6, cost per code review $0.16 Gemini 4 Argon (high effort): Score 72.8, cost per code review $1.71 Gemini 4 Argon (medium effort): Score 70.3, cost per code review $0.51 Gemini 4 Argon (low effort): Score 61.3, cost per code review $0.43 Muse Wonder (low effort): Score 64.2, cost per code review $3.3 Claude Fable 5.2 (xhigh effort): Score 78.5, cost per code review $7.41 Claude Fable 5.2 (high effort): Score 76, cost per code review $3.01 Claude Fable 5.2 (medium effort): Score 72.7, cost per code review $1.22 Claude Fable 5.2 (low effort): Score 71.1, cost per code review $1 Claude Opus 5.5 (max effort): Score 83.9, cost per code review $11.6 GPT-6 Luna (max effort): Score 72.5, cost per code review $0.13 GPT-6 Luna (xhigh effort): Score 68.2, cost per code review $0.06 GPT-6 Luna (high effort): Score 65.6, cost per code review $0.05 GPT-6 Luna (medium effort): Score 62.4, cost per code review $0.03 GPT-6 Luna (low effort): Score 44.7, cost per code review $0.02 GPT-6.1 Terra (high effort): Score 82.3, cost per code review $0.57 GPT-6.1 Terra (medium effort): Score 81.2, cost per code review $0.35 GPT-6.1 Terra (low effort): Score 76.1, cost per code review $0.18 Claude Opus 5.5 (max effort)GPT-6.1 Terra (high effort)GPT-6.1 Terra (medium effort)GPT-6.1 Terra (low effort)GPT-6 Luna (max effort)GPT-6 Luna (xhigh effort)GPT-6 Luna (high effort)GPT-6 Luna (medium effort)GPT-6 Luna (low effort) 40 60 80 100 $0$2.5$5$7.5$10$12.5 Cost per code review ($), lower is better Score GPT-5.6 Sol (high effort): Score 77.8, cost per code review $3.9 GPT-5.6 Sol (low effort): Score 66.1, cost per code review $0.81 Kimi K3 (max effort): Score 72.9, cost per code review $2.82 Kimi K3 (high effort): Score 68.6, cost per code review $1.9 Kimi K3 (low effort): Score 53.5, cost per code review $0.48 Claude Opus 5 (high effort): Score 69.9, cost per code review $6.5 Claude Opus 5 (medium effort): Score 65.5, cost per code review $3.5 Claude Opus 5 (low effort): Score 59.4, cost per code review $1.4 Grok 4.6 (xhigh effort): Score 75.9, cost per code review $4.68 Grok 4.6 (high effort): Score 75.1, cost per code review $3.51 Grok 4.6 (medium effort): Score 72.8, cost per code review $2.56 Grok 4.6 (low effort): Score 57.1, cost per code review $0.3 GLM-5.3 (max effort): Score 77.4, cost per code review $3.86 GLM-5.3 (high effort): Score 69.4, cost per code review $2.02 GLM-5.3 (low effort): Score 54.6, cost per code review $0.41 Gemini 3.8 Flash (medium effort): Score 70.7, cost per code review $3.38 Gemini 3.8 Flash (low effort): Score 51.3, cost per code review $0.9 GPT-6 Astra (max effort): Score 78, cost per code review $7.87 DeepSeek V4.1 Flash (max effort): Score 72, cost per code review $0.75 DeepSeek V4.1 Flash (low effort): Score 70.7, cost per code review $0.38 DeepSeek V4.1 Flash (high effort): Score 70.5, cost per code review $0.38 Claude Opus 5.5 (xhigh effort): Score 78.9, cost per code review $3.51 Claude Opus 5.5 (high effort): Score 73.1, cost per code review $1.3 Claude Opus 5.5 (medium effort): Score 70.1, cost per code review $0.97 Claude Opus 5.5 (low effort): Score 61.4, cost per code review $0.68 GPT-6 Sol (max effort): Score 80.3, cost per code review $3.68 GPT-6 Sol (xhigh effort): Score 77.2, cost per code review $2 GPT-6 Sol (high effort): Score 74.3, cost per code review $1.23 GPT-6 Sol (medium effort): Score 69.2, cost per code review $0.69 GPT-6 Sol (low effort): Score 61.8, cost per code review $0.36 GPT-6.1 Sol (max effort): Score 78.5, cost per code review $1.05 GPT-6.1 Sol (xhigh effort): Score 77.2, cost per code review $0.79 GPT-6.1 Sol (high effort): Score 76.7, cost per code review $0.56 GPT-6.1 Sol (medium effort): Score 73.5, cost per code review $0.28 GPT-6.1 Sol (low effort): Score 70.6, cost per code review $0.16 Gemini 4 Argon (high effort): Score 72.8, cost per code review $1.71 Gemini 4 Argon (medium effort): Score 70.3, cost per code review $0.51 Gemini 4 Argon (low effort): Score 61.3, cost per code review $0.43 Muse Wonder (low effort): Score 64.2, cost per code review $3.3 Claude Fable 5.2 (xhigh effort): Score 78.5, cost per code review $7.41 Claude Fable 5.2 (high effort): Score 76, cost per code review $3.01 Claude Fable 5.2 (medium effort): Score 72.7, cost per code review $1.22 Claude Fable 5.2 (low effort): Score 71.1, cost per code review $1 Claude Opus 5.5 (max effort): Score 83.9, cost per code review $11.6 GPT-6 Luna (max effort): Score 72.5, cost per code review $0.13 GPT-6 Luna (xhigh effort): Score 68.2, cost per code review $0.06 GPT-6 Luna (high effort): Score 65.6, cost per code review $0.05 GPT-6 Luna (medium effort): Score 62.4, cost per code review $0.03 GPT-6 Luna (low effort): Score 44.7, cost per code review $0.02 GPT-6.1 Terra (high effort): Score 82.3, cost per code review $0.57 GPT-6.1 Terra (medium effort): Score 81.2, cost per code review $0.35 GPT-6.1 Terra (low effort): Score 76.1, cost per code review $0.18 Claude Opus 5.5 (max effort)GPT-6.1 Terra (high effort)GPT-6.1 Terra (medium effort)GPT-6.1 Terra (low effort)GPT-6 Luna (max effort)GPT-6 Luna (xhigh effort)GPT-6 Luna (high effort)GPT-6 Luna (medium effort)GPT-6 Luna (low effort)
ModelSettingScoreCost per code review
Claude Opus 5.5 max effort 83.9 $11.58
GPT-6.1 Terra high effort 82.3 $0.57
GPT-6.1 Terra medium effort 81.2 $0.35
GPT-6 Sol max effort 80.3 $3.68
Claude Opus 5.5 xhigh effort 78.9 $3.51
GPT-6.1 Sol max effort 78.5 $1.05
Claude Fable 5.2 xhigh effort 78.5 $7.41
GPT-6 Astra max effort 78 $7.87
GPT-5.6 Sol high effort 77.8 $3.9
GLM-5.3 max effort 77.4 $3.86
GPT-6.1 Sol xhigh effort 77.2 $0.79
GPT-6 Sol xhigh effort 77.2 $2
GPT-6.1 Sol high effort 76.7 $0.56
GPT-6.1 Terra low effort 76.1 $0.18
Claude Fable 5.2 high effort 76 $3.01
Grok 4.6 xhigh effort 75.9 $4.68
Grok 4.6 high effort 75.1 $3.51
GPT-6 Sol high effort 74.3 $1.23
GPT-6.1 Sol medium effort 73.5 $0.28
Claude Opus 5.5 high effort 73.1 $1.3
Kimi K3 max effort 72.9 $2.82
Gemini 4 Argon high effort 72.8 $1.71
Grok 4.6 medium effort 72.8 $2.56
Claude Fable 5.2 medium effort 72.7 $1.22
GPT-6 Luna max effort 72.5 $0.13
DeepSeek V4.1 Flash max effort 72 $0.75
Claude Fable 5.2 low effort 71.1 $1
DeepSeek V4.1 Flash low effort 70.7 $0.38
Gemini 3.8 Flash medium effort 70.7 $3.38
GPT-6.1 Sol low effort 70.6 $0.16
DeepSeek V4.1 Flash high effort 70.5 $0.38
Gemini 4 Argon medium effort 70.3 $0.51
Claude Opus 5.5 medium effort 70.1 $0.97
Claude Opus 5 high effort 69.9 $6.5
GLM-5.3 high effort 69.4 $2.02
GPT-6 Sol medium effort 69.2 $0.69
Kimi K3 high effort 68.6 $1.9
GPT-6 Luna xhigh effort 68.2 $0.06
GPT-5.6 Sol low effort 66.1 $0.81
GPT-6 Luna high effort 65.6 $0.05
Claude Opus 5 medium effort 65.5 $3.5
Muse Wonder low effort 64.2 $3.3
GPT-6 Luna medium effort 62.4 $0.03
GPT-6 Sol low effort 61.8 $0.36
Claude Opus 5.5 low effort 61.4 $0.68
Gemini 4 Argon low effort 61.3 $0.43
Claude Opus 5 low effort 59.4 $1.4
Grok 4.6 low effort 57.1 $0.3
GLM-5.3 low effort 54.6 $0.41
Kimi K3 low effort 53.5 $0.48
Gemini 3.8 Flash low effort 51.3 $0.9
GPT-6 Luna low effort 44.7 $0.02

The best score, 83.9%, costs $12 a task

Runs on the cost frontier score higher than any run that's cheaper. GPT-6.1 Terra at medium effort comes within 2.7 points of the best for $0.35 a task, 33 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Anthropic OpenAI Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5, max effort GPT-6.1 Terra, high effort GPT-6.1 Terra, low effort GPT-6 Luna, xhigh effort GPT-6 Luna, medium effort GPT-6 Luna, low effort 0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5, max effort GPT-6.1 Terra, low effort

Source: Official MacroscopeBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 5.5 max effort 83.9% $12
GPT-6.1 Terra high effort 82.3% $0.57
GPT-6.1 Terra medium effort 81.2% $0.35
GPT-6 Sol max effort 80.3% $3.68
Claude Opus 5.5 xhigh effort 78.9% $3.51
GPT-6.1 Sol max effort 78.5% $1.05
Claude Fable 5.2 xhigh effort 78.5% $7.41
GPT-6 Astra max effort 78% $7.87
GPT-5.6 Sol high effort 77.8% $3.90
GLM-5.3 max effort 77.4% $3.86
GPT-6 Sol xhigh effort 77.2% $2.00
GPT-6.1 Sol xhigh effort 77.2% $0.79
GPT-6.1 Sol high effort 76.7% $0.56
GPT-6.1 Terra low effort 76.1% $0.18
Claude Fable 5.2 high effort 76% $3.01
Grok 4.6 xhigh effort 75.9% $4.68
Grok 4.6 high effort 75.1% $3.51
GPT-6 Sol high effort 74.3% $1.23
GPT-6.1 Sol medium effort 73.5% $0.28
Claude Opus 5.5 high effort 73.1% $1.30
Kimi K3 max effort 72.9% $2.82
Grok 4.6 medium effort 72.8% $2.56
Gemini 4 Argon high effort 72.8% $1.71
Claude Fable 5.2 medium effort 72.7% $1.22
GPT-6 Luna max effort 72.5% $0.13
DeepSeek V4.1 Flash max effort 72% $0.75
Claude Fable 5.2 low effort 71.1% $1.00
Gemini 3.8 Flash medium effort 70.7% $3.38
DeepSeek V4.1 Flash low effort 70.7% $0.38
GPT-6.1 Sol low effort 70.6% $0.16
DeepSeek V4.1 Flash high effort 70.5% $0.38
Gemini 4 Argon medium effort 70.3% $0.51
Claude Opus 5.5 medium effort 70.1% $0.97
Claude Opus 5 high effort 69.9% $6.50
GLM-5.3 high effort 69.4% $2.02
GPT-6 Sol medium effort 69.2% $0.69
Kimi K3 high effort 68.6% $1.90
GPT-6 Luna xhigh effort 68.2% $0.06
GPT-5.6 Sol low effort 66.1% $0.81
GPT-6 Luna high effort 65.6% $0.05
Claude Opus 5 medium effort 65.5% $3.50
Muse Wonder low effort 64.2% $3.30
GPT-6 Luna medium effort 62.4% $0.03
GPT-6 Sol low effort 61.8% $0.36
Claude Opus 5.5 low effort 61.4% $0.68
Gemini 4 Argon low effort 61.3% $0.43
Claude Opus 5 low effort 59.4% $1.40
Grok 4.6 low effort 57.1% $0.30
GLM-5.3 low effort 54.6% $0.41
Kimi K3 low effort 53.5% $0.48
Gemini 3.8 Flash low effort 51.3% $0.90
GPT-6 Luna low effort 44.7% $0.02

3 models have held number one on the official leaderboard since 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Anthropic
Own leaderboard 3 leaders
Claude Opus 5.5 83.9%

Source: Official MacroscopeBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2027 Near the ceiling (85%) Claude Opus 5.5 GPT-6 Astra GPT-5.6 Sol 0% 20% 40% 60% 80% 100% 2027 Near the ceiling (85%) Claude Opus 5.5

Source: Official MacroscopeBench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471