Benchmark Almanac

ProgramBench

Claude Opus 5.5 18.5% leads since September 2026
MiMo-V2.6 Flash 26% leads since September 2026
In use

200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original.

Tasks range from jq to FFmpeg and SQLite. The Opus 5.5 system card reports a 100-agent swarm result.

Released
May 2026
Announced on X
4.5K views, 45 likes
Kind
Building whole projects
Level
Research frontier
Scored
Checked automatically
Answers
Code
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Code → Code, Actions
Interaction
Multi-agent
Environment
Terminal
Items from
Real work and use
Refresh
One fixed set
Run by
Anyone
In views
Agents
Strongest model tested
GPT-6 Astra #4 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 20th, 2026

In the authors’ words

Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.

Abstract of ProgramBench: Can Language Models Rebuild Programs From Scratch? (arXiv:2605.03546, CC0)
25
Models scored on its own leaderboard
18
Launch posts list it, from 10 labs
15
System cards report it, from 7 labs

Claude Opus 5.5 leads with 18.5%

Best score per model from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

MiMo-V2.6 Flash leads with 26%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

  1. 1 Claude Opus 5.5 xhigh 18.5%
  2. 2 Claude Fable 5.1 xhigh 9.00%
  3. 3 Muse Wonder Full 8.50%
  4. 4 Muse Wonder xhigh 5.50%
  5. 5 GPT-6 Astra max 5.00%
  6. 6 Claude Opus 5 xhigh 4.50%
  7. 7 Muse Spark 1.3 max 2.50%
  8. 8 GPT-5.6 Sol xhigh 1.00%
  9. 9 GPT-5.5 xhigh 0.50%
  10. 10 Gemini 3.6 Flash 0.50%
  11. 11 GPT-5 mini 0%
  12. 12 Claude Haiku 4.5 0%
  13. 13 Gemini 3 Flash 0%
  14. 14 Claude Opus 4.6 0%
  15. 15 Claude Sonnet 4.6 0%
  16. 16 Gemini 3.1 Pro 0%
  17. 17 GPT-5.4 0%
  18. 18 GPT-5.4 Mini 0%
  19. 19 Claude Opus 4.7 xhigh 0%
  20. 20 Gemini 3.5 Flash 0%
  21. 21 Claude Opus 4.8 xhigh 0%
  22. 22 GLM-5.2 0%
  23. 23 Muse Spark 1.1 xhigh 0%
  24. 24 Muse Spark 1.2 xhigh 0%
  25. 25 Gemini 3.7 Flash 0%

Source: Official ProgramBench leaderboard

#ModelScoreSettingReleasedRun onSource
1 Claude Opus 5.5 18.5% xhigh, mini-SWE-agent Sep 2026 Sep 2026 Official leaderboard
2 Claude Fable 5.1 9.00% xhigh, mini-SWE-agent Sep 2026 Sep 2026 Official leaderboard
3 Muse Wonder Full 8.50% mini-SWE-agent Oct 2026 Oct 2026 Official leaderboard
4 Muse Wonder 5.50% xhigh, mini-SWE-agent Oct 2026 Sep 2026 Official leaderboard
5 GPT-6 Astra 5.00% max, mini-SWE-agent Sep 2026 Sep 2026 Official leaderboard
6 Claude Opus 5 4.50% xhigh, mini-SWE-agent Jul 2026 Jul 2026 Official leaderboard
7 Muse Spark 1.3 2.50% max, mini-SWE-agent Sep 2026 Sep 2026 Official leaderboard
8 GPT-5.6 Sol 1.00% xhigh, mini-SWE-agent Jul 2026 Jul 2026 Official leaderboard
9 GPT-5.5 0.50% xhigh, mini-SWE-agent Apr 2026 Apr 2026 Official leaderboard
10 Gemini 3.6 Flash 0.50% mini-SWE-agent Jul 2026 Jul 2026 Official leaderboard
11 GPT-5 mini 0% mini-SWE-agent Aug 2025 Aug 2025 Official leaderboard
12 Claude Haiku 4.5 0% mini-SWE-agent Oct 2025 Oct 2025 Official leaderboard
13 Gemini 3 Flash 0% mini-SWE-agent Dec 2025 Dec 2025 Official leaderboard
14 Claude Opus 4.6 0% mini-SWE-agent Feb 2026 Feb 2026 Official leaderboard
15 Claude Sonnet 4.6 0% mini-SWE-agent Feb 2026 Feb 2026 Official leaderboard
16 Gemini 3.1 Pro 0% mini-SWE-agent Feb 2026 Feb 2026 Official leaderboard
17 GPT-5.4 0% mini-SWE-agent Mar 2026 Mar 2026 Official leaderboard
18 GPT-5.4 Mini 0% mini-SWE-agent Mar 2026 Mar 2026 Official leaderboard
19 Claude Opus 4.7 0% xhigh, mini-SWE-agent Apr 2026 Apr 2026 Official leaderboard
20 Gemini 3.5 Flash 0% mini-SWE-agent May 2026 May 2026 Official leaderboard
21 Claude Opus 4.8 0% xhigh, mini-SWE-agent May 2026 May 2026 Official leaderboard
22 GLM-5.2 0% mini-SWE-agent Jun 2026 Jun 2026 Official leaderboard
23 Muse Spark 1.1 0% xhigh, mini-SWE-agent Jul 2026 Jul 2026 Official leaderboard
24 Muse Spark 1.2 0% xhigh, mini-SWE-agent Aug 2026 Aug 2026 Official leaderboard
25 Gemini 3.7 Flash 0% mini-SWE-agent Aug 2026 Aug 2026 Official leaderboard
#ModelScoreSettingDateSource
1 MiMo-V2.6 Flash 26% Sep 2026 Lab model card
2 Claude Opus 5.5 18.5% xhigh, mini-SWE-agent Sep 2026 Official leaderboard
3 Claude Fable 5.1 9.00% xhigh, mini-SWE-agent Sep 2026 Official leaderboard
4 Muse Wonder Full 8.50% mini-SWE-agent Oct 2026 Official leaderboard
5 Muse Wonder 5.50% xhigh, mini-SWE-agent Oct 2026 Official leaderboard
6 GPT-6 Astra 5.00% max, mini-SWE-agent Sep 2026 Official leaderboard
7 Claude Opus 5 4.50% xhigh, mini-SWE-agent Jul 2026 Official leaderboard
8 Muse Spark 1.3 2.50% max, mini-SWE-agent Sep 2026 Official leaderboard
9 GPT-5.6 Sol 1.00% xhigh, mini-SWE-agent Jul 2026 Official leaderboard
10 GPT-5.5 0.50% xhigh, mini-SWE-agent Apr 2026 Official leaderboard
11 Gemini 3.6 Flash 0.50% mini-SWE-agent Jul 2026 Official leaderboard
12 GPT-5 mini 0% mini-SWE-agent Aug 2025 Official leaderboard
13 Claude Haiku 4.5 0% mini-SWE-agent Oct 2025 Official leaderboard
14 Gemini 3 Flash 0% mini-SWE-agent Dec 2025 Official leaderboard
15 Claude Opus 4.6 0% mini-SWE-agent Feb 2026 Official leaderboard
16 Claude Sonnet 4.6 0% mini-SWE-agent Feb 2026 Official leaderboard
17 Gemini 3.1 Pro 0% mini-SWE-agent Feb 2026 Official leaderboard
18 GPT-5.4 0% mini-SWE-agent Mar 2026 Official leaderboard
19 GPT-5.4 Mini 0% mini-SWE-agent Mar 2026 Official leaderboard
20 Claude Opus 4.7 0% xhigh, mini-SWE-agent Apr 2026 Official leaderboard
21 Gemini 3.5 Flash 0% mini-SWE-agent May 2026 Official leaderboard
22 Claude Opus 4.8 0% xhigh, mini-SWE-agent May 2026 Official leaderboard
23 GLM-5.2 0% mini-SWE-agent Jun 2026 Official leaderboard
24 Muse Spark 1.1 0% xhigh, mini-SWE-agent Jul 2026 Official leaderboard
25 Muse Spark 1.2 0% xhigh, mini-SWE-agent Aug 2026 Official leaderboard
26 Gemini 3.7 Flash 0% mini-SWE-agent Aug 2026 Official leaderboard

Labs report scores for 1 model in their cards. Switch the scores above to rank them together.

Other ways to score it

The benchmark’s own leaderboard also reports Almost solved and Tests passed, other ways to score the whole test. Labs sometimes quote these instead of the headline. Claude Opus 5.5 leads Almost solved and Muse Wonder Full leads Tests passed. Arrows show how far a model moved from its overall rank among the models we track.

Almost solved

The share of tasks whose rebuilt program passes at least 95% of the hidden tests, the board's Almost column. Most labs quote it. 25 models we track have a score here.

  1. 1 Claude Opus 5.5 70.5%
  2. 2 Muse Wonder Full ↑1 60%
  3. 3 Claude Fable 5.1 ↓1 55.5%
  4. 4 GPT-6 Astra ↑1 52.5%
  5. 5 Muse Wonder ↓1 50.5%
  6. 6 Claude Opus 5 37%
  7. 7 Muse Spark 1.3 25%
  8. 8 Claude Opus 4.8 ↑13 16.5%
  9. 9 GPT-5.6 Sol ↓1 15.5%
  10. 10 GPT-5.5 ↓1 13.5%

Pareto

The board's Pareto view plots the share of hidden tests passed, which it calls an unofficial metric, against cost, turns, tokens or date. This is its Turns view. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 80% 100% 0100200300400500 Turns per task, lower is better % Test Passed Muse Wonder FullClaude Opus 5.5 (xhigh)GPT-6 Astra (max)GPT-5.6 Sol (xhigh)GPT-5.5 (high)GPT-5.6 Sol 0% 20% 40% 60% 80% 100% 0100200300400500 Turns per task, lower is better % Test Passed Muse Wonder FullClaude Opus 5.5 (xhigh)GPT-6 Astra (max)GPT-5.6 Sol (xhigh)GPT-5.5 (high)GPT-5.6 Sol
ModelSetting% Test PassedTurns per task
Muse Wonder Full mini-SWE-agent 88.5% 395
Claude Opus 5.5 xhigh, mini-SWE-agent 87.7% 304
GPT-6 Astra max, mini-SWE-agent 86.9% 57
Muse Wonder xhigh, mini-SWE-agent 86.5% 363
Claude Fable 5.1 xhigh, mini-SWE-agent 82.5% 260
Claude Opus 5 xhigh, mini-SWE-agent 74.7% 257
Claude Opus 4.8 xhigh, mini-SWE-agent 70.9% 145
Muse Spark 1.3 max, mini-SWE-agent 70.8% 325
GPT-5.6 Sol xhigh, mini-SWE-agent 69.9% 42
GPT-5.5 xhigh, mini-SWE-agent 69.5% 82
Muse Spark 1.3 xhigh, mini-SWE-agent 68.6% 253
GPT-5.5 high, mini-SWE-agent 66.5% 41
GLM-5.2 mini-SWE-agent 64.6% 222
Gemini 3.7 Flash mini-SWE-agent 61.2% 133
GPT-5.6 Sol mini-SWE-agent 57.8% 15
Muse Spark 1.2 xhigh, mini-SWE-agent 57.2% 183
GPT-5.5 mini-SWE-agent 56.6% 19
Gemini 3.6 Flash mini-SWE-agent 55.7% 150
Claude Opus 4.7 xhigh, mini-SWE-agent 55.1% 159
Gemini 3.5 Flash mini-SWE-agent 53.6% 166
Claude Opus 4.6 mini-SWE-agent 52.1% 260
Claude Opus 4.7 mini-SWE-agent 50.9% 93
Claude Sonnet 4.6 mini-SWE-agent 47.5% 472
Muse Spark 1.1 xhigh, mini-SWE-agent 47% 193
GPT-5.4 mini-SWE-agent 37.7% 16
Gemini 3.1 Pro mini-SWE-agent 36.4% 94
Gemini 3 Flash mini-SWE-agent 31.8% 85
Claude Haiku 4.5 mini-SWE-agent 30% 124
GPT-5.4 Mini mini-SWE-agent 16.4% 18
GPT-5 mini mini-SWE-agent 16% 15

The best score, 18.5%, costs $36 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI Meta Anthropic Cost frontier One model at each effort level
0% 5% 10% 15% 20% $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5, xhigh GPT-6 Astra, max Muse Spark 1.3, max Muse Spark 1.3, xhigh GPT-5 mini 0% 5% 10% 15% 20% $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5, xhigh GPT-6 Astra, max

Source: Official ProgramBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 5.5 xhigh mini-SWE-agent 18.5% $36 434,511
Claude Fable 5.1 xhigh mini-SWE-agent 9.00% $57 445,587
Muse Wonder Full mini-SWE-agent 8.50% 352,279
Muse Wonder xhigh mini-SWE-agent 5.50% $37 345,971
GPT-6 Astra max mini-SWE-agent 5.00% $11 91,595
Claude Opus 5 xhigh mini-SWE-agent 4.50% $51 357,785
Muse Spark 1.3 max mini-SWE-agent 2.50% $6.46 387,723
GPT-5.6 Sol xhigh mini-SWE-agent 1.00% $6.08 56,179
Muse Spark 1.3 xhigh mini-SWE-agent 1.00% $2.02 216,872
GPT-5.5 xhigh mini-SWE-agent 0.50% $8.85 51,624
GPT-5.5 high mini-SWE-agent 0.50% $3.65 27,379
GPT-5.6 Sol mini-SWE-agent 0.50% $1.00 13,084
Gemini 3.6 Flash mini-SWE-agent 0.50% $4.83 107,792
GPT-5 mini mini-SWE-agent 0% $0.03 7,146
Claude Haiku 4.5 mini-SWE-agent 0% $0.80 47,506
Gemini 3 Flash mini-SWE-agent 0% $0.30 22,561
Claude Opus 4.6 mini-SWE-agent 0% $11 89,147
Claude Sonnet 4.6 mini-SWE-agent 0% $27 190,227
Gemini 3.1 Pro mini-SWE-agent 0% $1.51 18,071
GPT-5.4 mini-SWE-agent 0% $0.33 7,408
GPT-5.4 Mini mini-SWE-agent 0% $0.04 3,147
Claude Opus 4.7 xhigh mini-SWE-agent 0% $11 97,205
Claude Opus 4.7 mini-SWE-agent 0% $3.81 41,669
GPT-5.5 mini-SWE-agent 0% $1.21 11,928
Gemini 3.5 Flash mini-SWE-agent 0% $5.60 132,679
Claude Opus 4.8 xhigh mini-SWE-agent 0% $21 270,298
GLM-5.2 mini-SWE-agent 0% $25 237,624
Muse Spark 1.1 xhigh mini-SWE-agent 0% $0.73 357,115
Muse Spark 1.2 xhigh mini-SWE-agent 0% $1.48 329,742
Gemini 3.7 Flash mini-SWE-agent 0% $2.04 74,885

5 models have held number one on the official leaderboard since its release in May 2026 5 models have held number one with claims included since its release in May 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Anthropic
Own leaderboard 5 leaders

Source: Official ProgramBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) MiMo-V2.6 Flash Claude Fable 5.1 Claude Opus 5 GPT-5.5 GPT-5 mini 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) MiMo-V2.6 Flash Claude Fable 5.1

Sources: Official ProgramBench leaderboard and labs’ model cards

10 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Moonshot AI
Z.ai
ByteDance Seed
Tencent
Qwen
DeepSeek
StepFun
Xiaomi Corp
Meta
Meituan
DateLaunch postScoreAs the post writes it
Nov 15, 2026 Qwen4-Max 31.5 Black-box software replication, ProgramBench
Nov 4, 2026 Tencent Hy4 30 ProgramBench (almost solved) *
Nov 2, 2026 LongCat-2.5 25.5 Program Bench (Almost Resolved)
Oct 24, 2026 Kimi K3.1 Pro 31.5 Program Bench (binary score, >=95% counts as solved)
Oct 17, 2026 DeepSeek V4.1 Flash 1017 ProgramBench
Oct 12, 2026 Muse Wonder ProgramBench
Oct 12, 2026 Muse Wonder Full ProgramBench
Sep 22, 2026 MiMo-V2.6 Pro ProgramBench
Sep 21, 2026 MiMo-V2.6 Flash 26 ProgramBench
Sep 20, 2026 Step 5 Preview 80.5 ProgramBench (Pass Rate)
Sep 10, 2026 DeepSeek V4.1 Flash 20.3 ProgramBench (Almost@1)
Sep 2, 2026 Qwen3.8 Max (0902) 28 Black-box software replication, ProgramBench (Almost Solved)
Aug 28, 2026 Tencent Hy4 preview ProgramBench
Aug 14, 2026 GLM-5.3 19 ProgramBench (Almost Solved)
Jul 16, 2026 Kimi K3 ProgramBench
Jun 23, 2026 Seed2.1 Pro 50.3 ProgramBench (resolved / almost resolved / average pass rate) 0/1/50.3
Jun 16, 2026 GLM-5.2 ProgramBench
Jun 12, 2026 Kimi K2.7 Code Program Bench

15 system cards report it, from 7 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 9, 2026 Claude Fable 5.2 system card 85.6 ProgramBench hidden-test pass rate on 166 golden tasks, single agent, mini-swe-agent harness, no time limit, 1M-token window. Showing remaining context raises the score to 91.6% (Section 8.10.1) Capabilities
Oct 12, 2026 Muse Wonder model card 50 ProgramBench, rollouts passing at least 95% of tests, mini-swe-agent, Muse Wonder (max) (capabilities table image, page 3) Capabilities
Oct 12, 2026 Muse Wonder Full model card 59 ProgramBench, rollouts passing at least 95% of tests, mini-swe-agent, Muse Wonder Full (max) (capabilities table image, page 3) Capabilities
Oct 7, 2026 Claude Haiku 5.5 system card 82 ProgramBench, hidden test pass rate on 166 of 200 tasks, mini-swe-agent without time limit (8.7.1) Capabilities
Sep 28, 2026 Claude Sonnet 5.5 system card 79.7 ProgramBench hidden test pass rate on 166 tasks, single agent, mini-swe-agent harness (8.10.1) Capabilities
Sep 22, 2026 MiMo-V2.6 Pro technical report ProgramBench (Table 3) Capabilities
Sep 22, 2026 Claude Opus 5.5 system card 91.2 ProgramBench hidden test pass rate on 166 of 200 tasks, single agent, mini-swe-agent harness, no time limit (8.10.1) Capabilities
Sep 10, 2026 DeepSeek V4.1 Flash technical report 20.3 ProgramBench (Almost@1) (Table 3). Figure 10 text: 30.04 multi-agent and 20.39 single-agent at 8h on a 172-task high-confidence subset Capabilities
Sep 1, 2026 Claude Fable 5.1 system card 87.6 ProgramBench hidden test pass rate on 166 tasks, single agent, mini-swe-agent harness, no time limit (8.11.1) Capabilities
Jul 27, 2026 Kimi K3 technical report ProgramBench (Table 2) Capabilities
Jul 24, 2026 Claude Opus 5 system card 93 ProgramBench hidden test pass rate on 166 of 200 tasks after five 1M-token episodes, single agent (83% after the first episode) (8.9.1) Capabilities
Jul 14, 2026 Grok 4.5 model card 57.2 ProgramBench, tests-passed rate, high (2.8) Capabilities
Jun 30, 2026 Claude Sonnet 5 system card ProgramBench, hidden test pass rate on 166 of 200 tasks across 5 episodes, given only as a range 76 to 86%, no single value (8.8) Capabilities
Jun 23, 2026 Seed2.1 Pro model card 50.3 ProgramBench, resolved 0 / almost resolved 1 / average pass rate 50.3 (Table 5) Capabilities
May 28, 2026 Claude Opus 4.8 system card 79 ProgramBench hidden test pass rate on 166 of 200 tasks, one episode of up to 1M tokens, near-final snapshot. Text gives 79 to 88% across 1 to 5 episodes (8.5, Figure 8.5.A) Capabilities

The lab's number comes out ahead in 4 of 6 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. Every benchmark.

Model Lab says Board says Gap
Claude Opus 5 Tests passed 93% ProgramBench hidden test pass rate on 166 of 200 tasks after five 1M-token episodes, single agent (83% after the first episode) (8.9.1) 74.7% Leaderboard, xhigh, mini-SWE-agent +18 pts
Claude Opus 4.8 Tests passed 79% ProgramBench hidden test pass rate on 166 of 200 tasks, one episode of up to 1M tokens, near-final snapshot. Text gives 79 to 88% across 1 to 5 episodes (8.5, Figure 8.5.A) 70.9% Leaderboard, xhigh, mini-SWE-agent +8.1 pts
Claude Fable 5.1 Tests passed 87.6% ProgramBench hidden test pass rate on 166 tasks, single agent, mini-swe-agent harness, no time limit (8.11.1) 82.5% Leaderboard, xhigh, mini-SWE-agent +5.1 pts
Claude Opus 5.5 Tests passed 91.2% ProgramBench hidden test pass rate on 166 of 200 tasks, single agent, mini-swe-agent harness, no time limit (8.10.1) 87.7% Leaderboard, xhigh, mini-SWE-agent +3.5 pts
Muse Wonder Full Almost solved 59% ProgramBench, rollouts passing at least 95% of tests, mini-swe-agent, Muse Wonder Full (max) (capabilities table image, page 3) 60% Leaderboard, mini-SWE-agent −1.0 pts
Muse Wonder Almost solved 50% ProgramBench, rollouts passing at least 95% of tests, mini-swe-agent, Muse Wonder (max) (capabilities table image, page 3) 50.5% Leaderboard, xhigh, mini-SWE-agent −0.5 pts

More in Coding

All 523 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471

Data audit

11 of 12 checks pass, 1 warning. Look first at the Lab claims check. 1 of 6 pairs differs by more than 10 points, the largest for Claude Opus 5 (+18 pts). Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 5 pass
  • Pass Independent runs Scores from the official leaderboard (25 entries), launch posts (10 scores) and system cards (12 scores).
  • Pass Models scored 26 models we track.
  • Pass Newest score The newest is from November 2026, for Qwen4-Max.
  • Pass Frontier models 3 of the top 10 have a score, led by GPT-6 Astra (#4).
  • Pass Headroom The best is 26%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on programbench.com: 30 rows for models we track at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 warning 1 pass
  • Warning Lab claims 1 of 6 pairs differs by more than 10 points, the largest for Claude Opus 5 (+18 pts).
  • Pass Known issues No published audit found problems.

How we check