Benchmark Almanac

Real-SWE 1.1

Claude Opus 5.5 56.8% leads since October 2026
Not in model cards

Resolution rate on private enterprise coding tasks, averaged across five runs per task.

Retains highest effort, harness, estimated rollout cost, and standard errors. Separate from the original Real-SWE.

Made by
Specific Labs
Released
October 2026
Kind
Fixing real repositories
Level
Expert
Scored
Checked automatically
Answers
Code
Tasks
Kept private
Strongest model tested
GPT-6 Astra #3 on the capability index
Last model added
October 2026
Scores refreshed
November 12th, 2026
18
Entries on its own leaderboard, 15 models we track

Claude Opus 5.5 leads with 56.8%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

18 entries. Scroll sideways for the rest.

  1. 56.8 Claude Opus 5.5
  2. 56.0 GPT-6 Astra
  3. 51.2 Claude Sonnet 5.5
  4. 47.2 GPT-6.1 Sol
  5. 45.6 Claude Fable 5.1
  6. 42.4 Muse Wonder 1.0
  7. 39.2 Gemini 4 Argon
  8. 39.2 Kimi K3.1 Pro
  9. 38.4 Particle-1
  10. 38.4 DeepSeek V4.1 Flash
  11. 33.6 Tencent Hy4 preview
  12. 28.0 Grok 4.7
  13. 26.4 GLM-5.3
  14. 24.0 Qwen3.8-Max prime
  15. 10.4 GPT-6 Luna
  16. 8.00 Inkling
  17. 5.60 Doubao Seed 2.1 Turbo
  18. 4.80 Nemotron 3 Ultra
  1. 1 Claude Opus 5.5 highest available effort; five runs per task 56.8%
  2. 2 GPT-6 Astra highest available effort; five runs per task 56%
  3. 3 Claude Sonnet 5.5 highest available effort; five runs per task 51.2%
  4. 4 GPT-6.1 Sol highest available effort; five runs per task 47.2%
  5. 5 Claude Fable 5.1 highest available effort; five runs per task 45.6%
  6. 6 Muse Wonder 1.0 highest available effort; five runs per task 42.4%
  7. 7 Gemini 4 Argon highest available effort; five runs per task 39.2%
  8. 8 Kimi K3.1 Pro highest available effort; five runs per task 39.2%
  9. 9 Particle-1 highest available effort; five runs per task 38.4%
  10. 10 DeepSeek V4.1 Flash highest available effort; five runs per task 38.4%
  11. 11 Tencent Hy4 preview highest available effort; five runs per task 33.6%
  12. 12 Grok 4.7 highest available effort; five runs per task 28%
  13. 13 GLM-5.3 highest available effort; five runs per task 26.4%
  14. 14 Qwen3.8-Max prime highest available effort; five runs per task 24%
  15. 15 GPT-6 Luna highest available effort; five runs per task 10.4%
  16. 16 Inkling highest available effort; five runs per task 8.00%
  17. 17 Doubao Seed 2.1 Turbo highest available effort; five runs per task 5.60%
  18. 18 Nemotron 3 Ultra highest available effort; five runs per task 4.80%

Source: Official Real-SWE 1.1 leaderboard

#EntryScoreSettingDateSource
1 Claude Opus 5.5 56.8% highest available effort; five runs per task, Claude Code Sep 2026 Official leaderboard
2 GPT-6 Astra 56% highest available effort; five runs per task, Codex CLI Sep 2026 Official leaderboard
3 Claude Sonnet 5.5 51.2% highest available effort; five runs per task, Claude Code Sep 2026 Official leaderboard
4 GPT-6.1 Sol 47.2% highest available effort; five runs per task, Codex CLI Sep 2026 Official leaderboard
5 Claude Fable 5.1 45.6% highest available effort; five runs per task, Claude Code Sep 2026 Official leaderboard
6 Muse Wonder 1.0 42.4% highest available effort; five runs per task, Muse Code Official leaderboard
7 Gemini 4 Argon 39.2% highest available effort; five runs per task, Antigravity CLI Sep 2026 Official leaderboard
8 Kimi K3.1 Pro 39.2% highest available effort; five runs per task, Kimi Code Oct 2026 Official leaderboard
9 Particle-1 38.4% highest available effort; five runs per task, mini-swe-agent Official leaderboard
10 DeepSeek V4.1 Flash 38.4% highest available effort; five runs per task, mini-swe-agent Sep 2026 Official leaderboard
11 Tencent Hy4 preview 33.6% highest available effort; five runs per task, mini-swe-agent Aug 2026 Official leaderboard
12 Grok 4.7 28% highest available effort; five runs per task, Grok Build Sep 2026 Official leaderboard
13 GLM-5.3 26.4% highest available effort; five runs per task, Claude Code Aug 2026 Official leaderboard
14 Qwen3.8-Max prime 24% highest available effort; five runs per task, Qwen Code Official leaderboard
15 GPT-6 Luna 10.4% highest available effort; five runs per task, Codex CLI Sep 2026 Official leaderboard
16 Inkling 8.00% highest available effort; five runs per task, mini-swe-agent Jul 2026 Official leaderboard
17 Doubao Seed 2.1 Turbo 5.60% highest available effort; five runs per task, mini-swe-agent Jun 2026 Official leaderboard
18 Nemotron 3 Ultra 4.80% highest available effort; five runs per task, mini-swe-agent Jun 2026 Official leaderboard

Resolution rate vs. estimated cost per rollout

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% $0$5$10$15 Estimated cost per rollout ($), lower is better Resolution rate Nemotron 3 Ultra (highest available effort; five runs per task): Resolution rate 4.8%, estimated cost per rollout $1.39 Doubao Seed 2.1 Turbo (highest available effort; five runs per task): Resolution rate 5.6%, estimated cost per rollout $3.03 Inkling (highest available effort; five runs per task): Resolution rate 8%, estimated cost per rollout $5.13 GLM-5.3 (highest available effort; five runs per task): Resolution rate 26.4%, estimated cost per rollout $3.35 Tencent Hy4 preview (highest available effort; five runs per task): Resolution rate 33.6%, estimated cost per rollout $1.13 Claude Fable 5.1 (highest available effort; five runs per task): Resolution rate 45.6%, estimated cost per rollout $14.2 Grok 4.7 (highest available effort; five runs per task): Resolution rate 28%, estimated cost per rollout $2.65 Claude Sonnet 5.5 (highest available effort; five runs per task): Resolution rate 51.2%, estimated cost per rollout $10.1 Gemini 4 Argon (highest available effort; five runs per task): Resolution rate 39.2%, estimated cost per rollout $4.66 Kimi K3.1 Pro (highest available effort; five runs per task): Resolution rate 39.2%, estimated cost per rollout $2.03 Muse Wonder 1.0 (highest available effort; five runs per task): Resolution rate 42.4%, estimated cost per rollout $3.76. Not in our model list Particle-1 (highest available effort; five runs per task): Resolution rate 38.4%, estimated cost per rollout $0.56. Not in our model list Qwen3.8-Max prime (highest available effort; five runs per task): Resolution rate 24%, estimated cost per rollout $7.06. Not in our model list GPT-6 Astra (highest available effort; five runs per task): Resolution rate 56%, estimated cost per rollout $3.78 DeepSeek V4.1 Flash (highest available effort; five runs per task): Resolution rate 38.4%, estimated cost per rollout $0.19 Claude Opus 5.5 (highest available effort; five runs per task): Resolution rate 56.8%, estimated cost per rollout $13.9 GPT-6 Luna (highest available effort; five runs per task): Resolution rate 10.4%, estimated cost per rollout $0.08 GPT-6.1 Sol (highest available effort; five runs per task): Resolution rate 47.2%, estimated cost per rollout $0.79 Claude Opus 5.5 (highest available effort; five runs per task)GPT-6 Astra (highest available effort; five runs per task)GPT-6.1 Sol (highest available effort; five runs per task)DeepSeek V4.1 Flash (highest available effort; five runs per task)GPT-6 Luna (highest available effort; five runs per task) 0% 20% 40% 60% $0$5$10$15 Estimated cost per rollout ($), lower is better Resolution rate Nemotron 3 Ultra (highest available effort; five runs per task): Resolution rate 4.8%, estimated cost per rollout $1.39 Doubao Seed 2.1 Turbo (highest available effort; five runs per task): Resolution rate 5.6%, estimated cost per rollout $3.03 Inkling (highest available effort; five runs per task): Resolution rate 8%, estimated cost per rollout $5.13 GLM-5.3 (highest available effort; five runs per task): Resolution rate 26.4%, estimated cost per rollout $3.35 Tencent Hy4 preview (highest available effort; five runs per task): Resolution rate 33.6%, estimated cost per rollout $1.13 Claude Fable 5.1 (highest available effort; five runs per task): Resolution rate 45.6%, estimated cost per rollout $14.2 Grok 4.7 (highest available effort; five runs per task): Resolution rate 28%, estimated cost per rollout $2.65 Claude Sonnet 5.5 (highest available effort; five runs per task): Resolution rate 51.2%, estimated cost per rollout $10.1 Gemini 4 Argon (highest available effort; five runs per task): Resolution rate 39.2%, estimated cost per rollout $4.66 Kimi K3.1 Pro (highest available effort; five runs per task): Resolution rate 39.2%, estimated cost per rollout $2.03 Muse Wonder 1.0 (highest available effort; five runs per task): Resolution rate 42.4%, estimated cost per rollout $3.76. Not in our model list Particle-1 (highest available effort; five runs per task): Resolution rate 38.4%, estimated cost per rollout $0.56. Not in our model list Qwen3.8-Max prime (highest available effort; five runs per task): Resolution rate 24%, estimated cost per rollout $7.06. Not in our model list GPT-6 Astra (highest available effort; five runs per task): Resolution rate 56%, estimated cost per rollout $3.78 DeepSeek V4.1 Flash (highest available effort; five runs per task): Resolution rate 38.4%, estimated cost per rollout $0.19 Claude Opus 5.5 (highest available effort; five runs per task): Resolution rate 56.8%, estimated cost per rollout $13.9 GPT-6 Luna (highest available effort; five runs per task): Resolution rate 10.4%, estimated cost per rollout $0.08 GPT-6.1 Sol (highest available effort; five runs per task): Resolution rate 47.2%, estimated cost per rollout $0.79 Claude Opus 5.5 (highest available effort; five runs per task)GPT-6 Astra (highest available effort; five runs per task)GPT-6.1 Sol (highest available effort; five runs per task)DeepSeek V4.1 Flash (highest available effort; five runs per task)GPT-6 Luna (highest available effort; five runs per task)
ModelSettingResolution rateEstimated cost per rollout
Claude Opus 5.5 highest available effort; five runs per task, Claude Code 56.8% $13.95
GPT-6 Astra highest available effort; five runs per task, Codex CLI 56% $3.78
Claude Sonnet 5.5 highest available effort; five runs per task, Claude Code 51.2% $10.09
GPT-6.1 Sol highest available effort; five runs per task, Codex CLI 47.2% $0.79
Claude Fable 5.1 highest available effort; five runs per task, Claude Code 45.6% $14.24
Muse Wonder 1.0 highest available effort; five runs per task, Muse Code 42.4% $3.76
Kimi K3.1 Pro highest available effort; five runs per task, Kimi Code 39.2% $2.03
Gemini 4 Argon highest available effort; five runs per task, Antigravity CLI 39.2% $4.66
DeepSeek V4.1 Flash highest available effort; five runs per task, mini-swe-agent 38.4% $0.19
Particle-1 highest available effort; five runs per task, mini-swe-agent 38.4% $0.56
Tencent Hy4 preview highest available effort; five runs per task, mini-swe-agent 33.6% $1.13
Grok 4.7 highest available effort; five runs per task, Grok Build 28% $2.66
GLM-5.3 highest available effort; five runs per task, Claude Code 26.4% $3.35
Qwen3.8-Max prime highest available effort; five runs per task, Qwen Code 24% $7.06
GPT-6 Luna highest available effort; five runs per task, Codex CLI 10.4% $0.08
Inkling highest available effort; five runs per task, mini-swe-agent 8% $5.13
Doubao Seed 2.1 Turbo highest available effort; five runs per task, mini-swe-agent 5.6% $3.03
Nemotron 3 Ultra highest available effort; five runs per task, mini-swe-agent 4.8% $1.39

The best score, 56.8%, costs $14 a task

Runs on the cost frontier score higher than any run that's cheaper. GPT-6 Astra at highest available effort; five runs per task comes within 0.8 points of the best for $3.78 a task, 4 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI DeepSeek Anthropic Cost frontier One model at each effort level
0% 20% 40% 60% $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5, highest available effort; five runs per task GPT-6.1 Sol, highest available effort; five runs per task DeepSeek V4.1 Flash, highest available effort; five runs per task GPT-6 Luna, highest available effort; five runs per task 0% 20% 40% 60% $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale Claude Opus 5.5, highest available effort; five runs per task GPT-6.1 Sol, highest available effort; five runs per task

Source: Official Real-SWE 1.1 leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 5.5 highest available effort; five runs per task Claude Code 56.8% $14 291,909
GPT-6 Astra highest available effort; five runs per task Codex CLI 56% $3.78 23,729
Claude Sonnet 5.5 highest available effort; five runs per task Claude Code 51.2% $10 388,359
GPT-6.1 Sol highest available effort; five runs per task Codex CLI 47.2% $0.79 31,913
Claude Fable 5.1 highest available effort; five runs per task Claude Code 45.6% $14 105,194
Gemini 4 Argon highest available effort; five runs per task Antigravity CLI 39.2% $4.66 135,946
Kimi K3.1 Pro highest available effort; five runs per task Kimi Code 39.2% $2.03 42,625
DeepSeek V4.1 Flash highest available effort; five runs per task mini-swe-agent 38.4% $0.19 104,862
Tencent Hy4 preview highest available effort; five runs per task mini-swe-agent 33.6% $1.13 91,750
Grok 4.7 highest available effort; five runs per task Grok Build 28% $2.65 39,715
GLM-5.3 highest available effort; five runs per task Claude Code 26.4% $3.35 414,581
GPT-6 Luna highest available effort; five runs per task Codex CLI 10.4% $0.08 45,551
Inkling highest available effort; five runs per task mini-swe-agent 8.00% $5.13 24,896
Doubao Seed 2.1 Turbo highest available effort; five runs per task mini-swe-agent 5.60% $3.03 29,987
Nemotron 3 Ultra highest available effort; five runs per task mini-swe-agent 4.80% $1.39 36,853

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard 1 leader
Claude Opus 5.5 56.8%

Source: Official Real-SWE 1.1 leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Claude Opus 5.5 Claude Fable 5.1 Tencent Hy4 preview GLM-5.3 Inkling 0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Claude Opus 5.5 Claude Fable 5.1

Source: Official Real-SWE 1.1 leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

Versions

Replaces Real-SWE

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471