Benchmark Almanac

ProgramBench

Claude Opus 5.5 18.5% leads since September 2026
In use

200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original.

Tasks range from jq to FFmpeg and SQLite. The Opus 5.5 system card reports a 100-agent swarm result.

Released
May 2026
Kind
Building whole projects
Level
Research frontier
Scored
Checked automatically
Answers
Code
Tasks
All public
Strongest model tested
GPT-6 Astra #3 on the capability index
Last in a launch post
October 2026
Last model added
October 2026
Scores refreshed
November 12th, 2026

In the authors’ words

Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.

Abstract of ProgramBench: Can Language Models Rebuild Programs From Scratch? (arXiv:2605.03546, CC0)
25
Models scored on its own leaderboard
8
Launch posts list it, from 6 labs
2
System cards report it, from 2 labs

Claude Opus 5.5 leads with 18.5%

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

25 models. Scroll sideways for the rest.

  1. 18.5 Claude Opus 5.5
  2. 9.00 Claude Fable 5.1
  3. 8.50 Muse Wonder Full
  4. 5.50 Muse Wonder
  5. 5.00 GPT-6 Astra
  6. 4.50 Claude Opus 5
  7. 2.50 Muse Spark 1.3
  8. 1.00 GPT-5.6 Sol
  9. 0.50 GPT-5.5
  10. 0.50 Gemini 3.6 Flash
  11. 0.0 GPT-5 mini
  12. 0.0 Claude Haiku 4.5
  13. 0.0 Gemini 3 Flash
  14. 0.0 Claude Opus 4.6
  15. 0.0 Claude Sonnet 4.6
  16. 0.0 Gemini 3.1 Pro
  17. 0.0 GPT-5.4
  18. 0.0 GPT-5.4 Mini
  19. 0.0 Claude Opus 4.7
  20. 0.0 Gemini 3.5 Flash
  21. 0.0 Claude Opus 4.8
  22. 0.0 GLM-5.2
  23. 0.0 Muse Spark 1.1
  24. 0.0 Muse Spark 1.2
  25. 0.0 Gemini 3.7 Flash
  1. 1 Claude Opus 5.5 xhigh 18.5%
  2. 2 Claude Fable 5.1 xhigh 9.00%
  3. 3 Muse Wonder Full 8.50%
  4. 4 Muse Wonder xhigh 5.50%
  5. 5 GPT-6 Astra max 5.00%
  6. 6 Claude Opus 5 xhigh 4.50%
  7. 7 Muse Spark 1.3 max 2.50%
  8. 8 GPT-5.6 Sol xhigh 1.00%
  9. 9 GPT-5.5 xhigh 0.50%
  10. 10 Gemini 3.6 Flash 0.50%
  11. 11 GPT-5 mini 0%
  12. 12 Claude Haiku 4.5 0%
  13. 13 Gemini 3 Flash 0%
  14. 14 Claude Opus 4.6 0%
  15. 15 Claude Sonnet 4.6 0%
  16. 16 Gemini 3.1 Pro 0%
  17. 17 GPT-5.4 0%
  18. 18 GPT-5.4 Mini 0%
  19. 19 Claude Opus 4.7 xhigh 0%
  20. 20 Gemini 3.5 Flash 0%
  21. 21 Claude Opus 4.8 xhigh 0%
  22. 22 GLM-5.2 0%
  23. 23 Muse Spark 1.1 xhigh 0%
  24. 24 Muse Spark 1.2 xhigh 0%
  25. 25 Gemini 3.7 Flash 0%

Source: Official ProgramBench leaderboard

#ModelScoreSettingDateSource
1 Claude Opus 5.5 18.5% xhigh, mini-SWE-agent Sep 2026 Official leaderboard
2 Claude Fable 5.1 9.00% xhigh, mini-SWE-agent Sep 2026 Official leaderboard
3 Muse Wonder Full 8.50% mini-SWE-agent Oct 2026 Official leaderboard
4 Muse Wonder 5.50% xhigh, mini-SWE-agent Oct 2026 Official leaderboard
5 GPT-6 Astra 5.00% max, mini-SWE-agent Sep 2026 Official leaderboard
6 Claude Opus 5 4.50% xhigh, mini-SWE-agent Jul 2026 Official leaderboard
7 Muse Spark 1.3 2.50% max, mini-SWE-agent Sep 2026 Official leaderboard
8 GPT-5.6 Sol 1.00% xhigh, mini-SWE-agent Jul 2026 Official leaderboard
9 GPT-5.5 0.50% xhigh, mini-SWE-agent Apr 2026 Official leaderboard
10 Gemini 3.6 Flash 0.50% mini-SWE-agent Jul 2026 Official leaderboard
11 GPT-5 mini 0% mini-SWE-agent Aug 2025 Official leaderboard
12 Claude Haiku 4.5 0% mini-SWE-agent Oct 2025 Official leaderboard
13 Gemini 3 Flash 0% mini-SWE-agent Dec 2025 Official leaderboard
14 Claude Opus 4.6 0% mini-SWE-agent Feb 2026 Official leaderboard
15 Claude Sonnet 4.6 0% mini-SWE-agent Feb 2026 Official leaderboard
16 Gemini 3.1 Pro 0% mini-SWE-agent Feb 2026 Official leaderboard
17 GPT-5.4 0% mini-SWE-agent Mar 2026 Official leaderboard
18 GPT-5.4 Mini 0% mini-SWE-agent Mar 2026 Official leaderboard
19 Claude Opus 4.7 0% xhigh, mini-SWE-agent Apr 2026 Official leaderboard
20 Gemini 3.5 Flash 0% mini-SWE-agent May 2026 Official leaderboard
21 Claude Opus 4.8 0% xhigh, mini-SWE-agent May 2026 Official leaderboard
22 GLM-5.2 0% mini-SWE-agent Jun 2026 Official leaderboard
23 Muse Spark 1.1 0% xhigh, mini-SWE-agent Jul 2026 Official leaderboard
24 Muse Spark 1.2 0% xhigh, mini-SWE-agent Aug 2026 Official leaderboard
25 Gemini 3.7 Flash 0% mini-SWE-agent Aug 2026 Official leaderboard

Pareto

The board's Pareto view plots the share of hidden tests passed, which it calls an unofficial metric, against cost, turns, tokens or date. This is its Turns view. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 80% 100% 0100200300400500 Turns per task, lower is better % Test Passed GPT-5 mini: % Test Passed 16%, turns per task 15 Claude Haiku 4.5: % Test Passed 30%, turns per task 124 Gemini 3 Flash: % Test Passed 31.8%, turns per task 85 Claude Opus 4.6: % Test Passed 52.1%, turns per task 260 Claude Sonnet 4.6: % Test Passed 47.5%, turns per task 472 Gemini 3.1 Pro: % Test Passed 36.4%, turns per task 94 GPT-5.4: % Test Passed 37.7%, turns per task 16 GPT-5.4 Mini: % Test Passed 16.4%, turns per task 18 Claude Opus 4.7 (xhigh): % Test Passed 55.1%, turns per task 159 Claude Opus 4.7: % Test Passed 50.9%, turns per task 93 GPT-5.5 (xhigh): % Test Passed 69.5%, turns per task 82 GPT-5.5: % Test Passed 56.6%, turns per task 19 Gemini 3.5 Flash: % Test Passed 53.6%, turns per task 166 Claude Opus 4.8 (xhigh): % Test Passed 70.9%, turns per task 145 GLM-5.2: % Test Passed 64.6%, turns per task 222 Muse Spark 1.1 (xhigh): % Test Passed 47%, turns per task 193 Gemini 3.6 Flash: % Test Passed 55.7%, turns per task 150 Claude Opus 5 (xhigh): % Test Passed 74.7%, turns per task 257 Muse Spark 1.2 (xhigh): % Test Passed 57.2%, turns per task 183 Gemini 3.7 Flash: % Test Passed 61.2%, turns per task 133 Claude Fable 5.1 (xhigh): % Test Passed 82.5%, turns per task 260 Muse Spark 1.3 (max): % Test Passed 70.8%, turns per task 325 Muse Spark 1.3 (xhigh): % Test Passed 68.6%, turns per task 253 Muse Wonder (xhigh): % Test Passed 86.5%, turns per task 363 GPT-5.5 (high): % Test Passed 66.5%, turns per task 41 GPT-5.6 Sol (xhigh): % Test Passed 69.9%, turns per task 42 GPT-5.6 Sol: % Test Passed 57.8%, turns per task 15 GPT-6 Astra (max): % Test Passed 86.9%, turns per task 57 Claude Opus 5.5 (xhigh): % Test Passed 87.7%, turns per task 304 Muse Wonder Full: % Test Passed 88.5%, turns per task 395 Muse Wonder FullClaude Opus 5.5 (xhigh)GPT-6 Astra (max)GPT-5.6 Sol (xhigh)GPT-5.5 (high)GPT-5.6 Sol 0% 20% 40% 60% 80% 100% 0100200300400500 Turns per task, lower is better % Test Passed GPT-5 mini: % Test Passed 16%, turns per task 15 Claude Haiku 4.5: % Test Passed 30%, turns per task 124 Gemini 3 Flash: % Test Passed 31.8%, turns per task 85 Claude Opus 4.6: % Test Passed 52.1%, turns per task 260 Claude Sonnet 4.6: % Test Passed 47.5%, turns per task 472 Gemini 3.1 Pro: % Test Passed 36.4%, turns per task 94 GPT-5.4: % Test Passed 37.7%, turns per task 16 GPT-5.4 Mini: % Test Passed 16.4%, turns per task 18 Claude Opus 4.7 (xhigh): % Test Passed 55.1%, turns per task 159 Claude Opus 4.7: % Test Passed 50.9%, turns per task 93 GPT-5.5 (xhigh): % Test Passed 69.5%, turns per task 82 GPT-5.5: % Test Passed 56.6%, turns per task 19 Gemini 3.5 Flash: % Test Passed 53.6%, turns per task 166 Claude Opus 4.8 (xhigh): % Test Passed 70.9%, turns per task 145 GLM-5.2: % Test Passed 64.6%, turns per task 222 Muse Spark 1.1 (xhigh): % Test Passed 47%, turns per task 193 Gemini 3.6 Flash: % Test Passed 55.7%, turns per task 150 Claude Opus 5 (xhigh): % Test Passed 74.7%, turns per task 257 Muse Spark 1.2 (xhigh): % Test Passed 57.2%, turns per task 183 Gemini 3.7 Flash: % Test Passed 61.2%, turns per task 133 Claude Fable 5.1 (xhigh): % Test Passed 82.5%, turns per task 260 Muse Spark 1.3 (max): % Test Passed 70.8%, turns per task 325 Muse Spark 1.3 (xhigh): % Test Passed 68.6%, turns per task 253 Muse Wonder (xhigh): % Test Passed 86.5%, turns per task 363 GPT-5.5 (high): % Test Passed 66.5%, turns per task 41 GPT-5.6 Sol (xhigh): % Test Passed 69.9%, turns per task 42 GPT-5.6 Sol: % Test Passed 57.8%, turns per task 15 GPT-6 Astra (max): % Test Passed 86.9%, turns per task 57 Claude Opus 5.5 (xhigh): % Test Passed 87.7%, turns per task 304 Muse Wonder Full: % Test Passed 88.5%, turns per task 395 Muse Wonder FullClaude Opus 5.5 (xhigh)GPT-6 Astra (max)GPT-5.6 Sol (xhigh)GPT-5.5 (high)GPT-5.6 Sol
ModelSetting% Test PassedTurns per task
Muse Wonder Full mini-SWE-agent 88.5% 395
Claude Opus 5.5 xhigh, mini-SWE-agent 87.7% 304
GPT-6 Astra max, mini-SWE-agent 86.9% 57
Muse Wonder xhigh, mini-SWE-agent 86.5% 363
Claude Fable 5.1 xhigh, mini-SWE-agent 82.5% 260
Claude Opus 5 xhigh, mini-SWE-agent 74.7% 257
Claude Opus 4.8 xhigh, mini-SWE-agent 70.9% 145
Muse Spark 1.3 max, mini-SWE-agent 70.8% 325
GPT-5.6 Sol xhigh, mini-SWE-agent 69.9% 42
GPT-5.5 xhigh, mini-SWE-agent 69.5% 82
Muse Spark 1.3 xhigh, mini-SWE-agent 68.6% 253
GPT-5.5 high, mini-SWE-agent 66.5% 41
GLM-5.2 mini-SWE-agent 64.6% 222
Gemini 3.7 Flash mini-SWE-agent 61.2% 133
GPT-5.6 Sol mini-SWE-agent 57.8% 15
Muse Spark 1.2 xhigh, mini-SWE-agent 57.2% 183
GPT-5.5 mini-SWE-agent 56.6% 19
Gemini 3.6 Flash mini-SWE-agent 55.7% 150
Claude Opus 4.7 xhigh, mini-SWE-agent 55.1% 159
Gemini 3.5 Flash mini-SWE-agent 53.6% 166
Claude Opus 4.6 mini-SWE-agent 52.1% 260
Claude Opus 4.7 mini-SWE-agent 50.9% 93
Claude Sonnet 4.6 mini-SWE-agent 47.5% 472
Muse Spark 1.1 xhigh, mini-SWE-agent 47% 193
GPT-5.4 mini-SWE-agent 37.7% 16
Gemini 3.1 Pro mini-SWE-agent 36.4% 94
Gemini 3 Flash mini-SWE-agent 31.8% 85
Claude Haiku 4.5 mini-SWE-agent 30% 124
GPT-5.4 Mini mini-SWE-agent 16.4% 18
GPT-5 mini mini-SWE-agent 16% 15

The best score, 18.5%, costs $36 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI Meta Anthropic Cost frontier One model at each effort level
0% 5% 10% 15% 20% $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5, xhigh GPT-6 Astra, max Muse Spark 1.3, max Muse Spark 1.3, xhigh GPT-5 mini 0% 5% 10% 15% 20% $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5, xhigh GPT-6 Astra, max

Source: Official ProgramBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 5.5 xhigh mini-SWE-agent 18.5% $36 434,511
Claude Fable 5.1 xhigh mini-SWE-agent 9.00% $57 445,587
Muse Wonder Full mini-SWE-agent 8.50% 352,279
Muse Wonder xhigh mini-SWE-agent 5.50% $37 345,971
GPT-6 Astra max mini-SWE-agent 5.00% $11 91,595
Claude Opus 5 xhigh mini-SWE-agent 4.50% $51 357,785
Muse Spark 1.3 max mini-SWE-agent 2.50% $6.46 387,723
GPT-5.6 Sol xhigh mini-SWE-agent 1.00% $6.08 56,179
Muse Spark 1.3 xhigh mini-SWE-agent 1.00% $2.02 216,872
GPT-5.5 xhigh mini-SWE-agent 0.50% $8.85 51,624
GPT-5.5 high mini-SWE-agent 0.50% $3.65 27,379
GPT-5.6 Sol mini-SWE-agent 0.50% $1.00 13,084
Gemini 3.6 Flash mini-SWE-agent 0.50% $4.83 107,792
GPT-5 mini mini-SWE-agent 0% $0.03 7,146
Claude Haiku 4.5 mini-SWE-agent 0% $0.80 47,506
Gemini 3 Flash mini-SWE-agent 0% $0.30 22,561
Claude Opus 4.6 mini-SWE-agent 0% $11 89,147
Claude Sonnet 4.6 mini-SWE-agent 0% $27 190,227
Gemini 3.1 Pro mini-SWE-agent 0% $1.51 18,071
GPT-5.4 mini-SWE-agent 0% $0.33 7,408
GPT-5.4 Mini mini-SWE-agent 0% $0.04 3,147
Claude Opus 4.7 xhigh mini-SWE-agent 0% $11 97,205
Claude Opus 4.7 mini-SWE-agent 0% $3.81 41,669
GPT-5.5 mini-SWE-agent 0% $1.21 11,928
Gemini 3.5 Flash mini-SWE-agent 0% $5.60 132,679
Claude Opus 4.8 xhigh mini-SWE-agent 0% $21 270,298
GLM-5.2 mini-SWE-agent 0% $25 237,624
Muse Spark 1.1 xhigh mini-SWE-agent 0% $0.73 357,115
Muse Spark 1.2 xhigh mini-SWE-agent 0% $1.48 329,742
Gemini 3.7 Flash mini-SWE-agent 0% $2.04 74,885

5 models have held number one on the official leaderboard since its release in May 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Anthropic
Own leaderboard 5 leaders
Claude Opus 5.5 18.5%

Source: Official ProgramBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 5.5 Claude Fable 5.1 Claude Opus 5 GPT-5.5 GPT-5 mini 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 5.5 Claude Fable 5.1

Source: Official ProgramBench leaderboard

6 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Moonshot AI
Z.ai
Tencent
Xiaomi Corp
Meta
DeepSeek
DateLaunch postScoreAs the post writes it
Oct 17, 2026 DeepSeek V4.1 Flash 1017 ProgramBench
Oct 12, 2026 Muse Wonder ProgramBench
Oct 12, 2026 Muse Wonder Full ProgramBench
Sep 22, 2026 MiMo-V2.6 Pro ProgramBench
Aug 28, 2026 Tencent Hy4 preview ProgramBench
Jul 16, 2026 Kimi K3 ProgramBench
Jun 16, 2026 GLM-5.2 ProgramBench
Jun 12, 2026 Kimi K2.7 Code Program Bench

2 system cards report it, from 2 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Sep 22, 2026 MiMo-V2.6 Pro technical report ProgramBench (Table 3) Capabilities
Jul 27, 2026 Kimi K3 technical report ProgramBench (Table 2) Capabilities

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471