Benchmark Almanac

SWE-Bench Pro Private

Muse Spark 1.1 51.5% leads since July 2026
Not in model cards

Agents resolve held-out software issues from the private SWE-Bench Pro split. The leaderboard reports the percentage of resolved tasks.

Made by
Scale AI
Released
September 2025
Kind
Fixing real repositories
Level
Expert
Scored
Checked automatically
Answers
Code
Tasks
Kept private
Strongest model tested
GPT-5.4 #27 on the capability index
Last model added
July 2026
Scores refreshed
November 12th, 2026
14
Entries on its own leaderboard, 12 models we track

Muse Spark 1.1 leads with 51.5% among models released by July 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in July 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

14 entries. Scroll sideways for the rest.

  1. 51.5 Muse Spark 1.1
  2. 47.1 Claude Opus 4.6
  3. 44.7 Muse Spark
  4. 43.4 GPT-5.4
  5. 32.2 Gemini 3.1 Pro
  6. 27.7 GPT-5.2 Codex
  7. 23.8 GPT-5.2
  8. 23.4 Claude Opus 4.5
  9. 17.9 Gemini 3 Pro
  10. 17.8 Claude Opus 4.1
  11. 14.9 OpenAI GPT-5
  12. 10.1 Gemini 2.5 Pro
  13. 9.06 Claude Sonnet 4
  14. 3.62 OpenAI GPT-4o
  1. 1 Muse Spark 1.1 51.5%
  2. 2 Claude Opus 4.6 thinking 47.1%
  3. 3 Muse Spark 44.7%
  4. 4 GPT-5.4 xhigh 43.4%
  5. 5 Gemini 3.1 Pro thinking 32.2%
  6. 6 GPT-5.2 Codex 27.7%
  7. 7 GPT-5.2 23.8%
  8. 8 Claude Opus 4.5 23.4%
  9. 9 Gemini 3 Pro 17.9%
  10. 10 Claude Opus 4.1 17.8%
  11. 11 OpenAI GPT-5 14.9%
  12. 12 Gemini 2.5 Pro Jun 2025 10.1%
  13. 13 Claude Sonnet 4 9.06%
  14. 14 OpenAI GPT-4o 3.62%

Source: Official SWE-Bench Pro Private leaderboard

#EntryScoreSettingDateSource
1 Muse Spark 1.1 51.5% mini-SWE-agent Jul 2026 Official leaderboard
2 Claude Opus 4.6 47.1% thinking, mini-SWE-agent Feb 2026 Official leaderboard
3 Muse Spark 44.7% mini-SWE-agent Apr 2026 Official leaderboard
4 GPT-5.4 43.4% xhigh, mini-SWE-agent Mar 2026 Official leaderboard
5 Gemini 3.1 Pro 32.2% thinking, mini-SWE-agent Feb 2026 Official leaderboard
6 GPT-5.2 Codex 27.7% SWE-agent Dec 2025 Official leaderboard
7 GPT-5.2 23.8% SWE-agent Dec 2025 Official leaderboard
8 Claude Opus 4.5 23.4% SWE-agent Nov 2025 Official leaderboard
9 Gemini 3 Pro 17.9% SWE-agent Nov 2025 Official leaderboard
10 Claude Opus 4.1 17.8% SWE-agent Aug 2025 Official leaderboard
11 OpenAI GPT-5 14.9% SWE-agent Official leaderboard
12 Gemini 2.5 Pro Jun 2025 10.1% SWE-agent Jun 2025 Official leaderboard
13 Claude Sonnet 4 9.06% SWE-agent May 2025 Official leaderboard
14 OpenAI GPT-4o 3.62% SWE-agent Official leaderboard

7 models have held number one on the official leaderboard since its release in September 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic Google DeepMind OpenAI Meta
Own leaderboard Not tested since Jul 2026
Muse Spark 1.1 51.5%

Source: Official SWE-Bench Pro Private leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Muse Spark 1.1 Claude Opus 4.6 GPT-5.2 Codex Claude Opus 4.5 Gemini 3 Pro Claude Opus 4.1 Gemini 2.5 Pro 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Muse Spark 1.1 GPT-5.2 Codex

Source: Official SWE-Bench Pro Private leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471