Benchmark Almanac

SWE Atlas Codebase QnA Extended

Claude Sonnet 5.5 44.7% leads since September 2026
Not in model cards

Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores.

Held-out extension of swe-atlas. Mercor shows the original and extended score for each model. The month is approximate. The Wayback Machine first saved the earliest extended pages on August 15th, 2026.

Made by
Mercor
Released
August 2026
Kind
Fixing real repositories
Level
Expert
Scored
Graded by a model
Answers
Open-ended
Tasks
Public, plus a private test set
Strongest model tested
Claude Opus 5.5 #4 on the capability index
Last model added
October 2026
Scores refreshed
November 12th, 2026
38
Models scored on its own leaderboard

Claude Sonnet 5.5 leads with 44.7%

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best.

38 models. Scroll sideways for the rest.

  1. 44.7 Claude Sonnet 5.5
  2. 43.3 Claude Opus 5.5
  3. 40.0 Claude Opus 5
  4. 38.0 Claude Fable 5.1
  5. 36.7 DeepSeek V4.1 Flash
  6. 33.3 Qwen 3.8 Max
  7. 32.7 Claude Sonnet 5
  8. 31.3 GPT-6 Sol
  9. 31.3 GPT-6.1 Sol
  10. 24.7 GPT-5.6 Sol
  11. 24.0 Claude Opus 4.8
  12. 24.0 Kimi K3
  13. 19.3 GPT-5.6 Luna
  14. 19.3 Grok 4.6
  15. 18.0 DeepSeek-V4-Flash
  16. 17.3 GPT-5.5
  17. 16.0 GLM-5.2
  18. 16.0 GPT-5.6 Terra
  19. 15.3 Grok 4.7
  20. 15.3 GPT-6 Luna
  21. 12.0 Grok 4.5
  22. 11.3 Claude Opus 4.7
  23. 10.7 GPT-5.4
  24. 10.0 MiniMax-M3
  25. 6.70 Claude Sonnet 4.6
  26. 5.30 DeepSeek-V4-Pro
  27. 5.00 Muse Spark Open
  28. 4.70 Kimi K2.7 Code
  29. 4.70 Gemini 3.6 Flash
  30. 2.70 Gemini 3.1 Pro
  31. 2.70 Gemini 3.5 Flash
  32. 1.30 Inkling
  33. 0.70 MiniMax-M2.7
  34. 0.0 gpt-oss-120b
  35. 0.0 Kimi K2
  36. 0.0 DeepSeek-V3.2
  37. 0.0 Gemma 4 31B IT
  38. 0.0 Claude Fable 5
  1. 1 Claude Sonnet 5.5 max 44.7%
  2. 2 Claude Opus 5.5 max 43.3%
  3. 3 Claude Opus 5 max 40%
  4. 4 Claude Fable 5.1 max 38%
  5. 5 DeepSeek V4.1 Flash max 36.7%
  6. 6 Qwen 3.8 Max xhigh 33.3%
  7. 7 Claude Sonnet 5 max 32.7%
  8. 8 GPT-6 Sol max 31.3%
  9. 9 GPT-6.1 Sol max 31.3%
  10. 10 GPT-5.6 Sol max 24.7%
  11. 11 Claude Opus 4.8 max 24%
  12. 12 Kimi K3 max 24%
  13. 13 GPT-5.6 Luna max 19.3%
  14. 14 Grok 4.6 xhigh 19.3%
  15. 15 DeepSeek-V4-Flash max 18%
  16. 16 GPT-5.5 xhigh 17.3%
  17. 17 GLM-5.2 max 16%
  18. 18 GPT-5.6 Terra max 16%
  19. 19 Grok 4.7 xhigh 15.3%
  20. 20 GPT-6 Luna max 15.3%
  21. 21 Grok 4.5 high 12%
  22. 22 Claude Opus 4.7 max 11.3%
  23. 23 GPT-5.4 xhigh 10.7%
  24. 24 MiniMax-M3 high 10%
  25. 25 Claude Sonnet 4.6 high 6.70%
  26. 26 DeepSeek-V4-Pro max 5.30%
  27. 27 Muse Spark Open xhigh 5.00%
  28. 28 Kimi K2.7 Code high 4.70%
  29. 29 Gemini 3.6 Flash high 4.70%
  30. 30 Gemini 3.1 Pro high 2.70%
  31. 31 Gemini 3.5 Flash high 2.70%
  32. 32 Inkling high 1.30%
  33. 33 MiniMax-M2.7 high 0.70%
  34. 34 gpt-oss-120b high 0%
  35. 35 Kimi K2 Sep 2025, high 0%
  36. 36 DeepSeek-V3.2 0%
  37. 37 Gemma 4 31B IT 0%
  38. 38 Claude Fable 5 max 0%

Source: Official SWE Atlas Codebase QnA Extended leaderboard

#ModelScoreSettingDateSource
1 Claude Sonnet 5.5 44.7% max Sep 2026 Official leaderboard
2 Claude Opus 5.5 43.3% max Sep 2026 Official leaderboard
3 Claude Opus 5 40% max Jul 2026 Official leaderboard
4 Claude Fable 5.1 38% max Sep 2026 Official leaderboard
5 DeepSeek V4.1 Flash 36.7% max Sep 2026 Official leaderboard
6 Qwen 3.8 Max 33.3% xhigh Aug 2026 Official leaderboard
7 Claude Sonnet 5 32.7% max Jun 2026 Official leaderboard
8 GPT-6 Sol 31.3% max Sep 2026 Official leaderboard
9 GPT-6.1 Sol 31.3% max Sep 2026 Official leaderboard
10 GPT-5.6 Sol 24.7% max Jul 2026 Official leaderboard
11 Claude Opus 4.8 24% max May 2026 Official leaderboard
12 Kimi K3 24% max Jul 2026 Official leaderboard
13 GPT-5.6 Luna 19.3% max Jul 2026 Official leaderboard
14 Grok 4.6 19.3% xhigh Aug 2026 Official leaderboard
15 DeepSeek-V4-Flash 18% max Apr 2026 Official leaderboard
16 GPT-5.5 17.3% xhigh Apr 2026 Official leaderboard
17 GLM-5.2 16% max Jun 2026 Official leaderboard
18 GPT-5.6 Terra 16% max Jul 2026 Official leaderboard
19 Grok 4.7 15.3% xhigh Sep 2026 Official leaderboard
20 GPT-6 Luna 15.3% max Sep 2026 Official leaderboard
21 Grok 4.5 12% high Jul 2026 Official leaderboard
22 Claude Opus 4.7 11.3% max Apr 2026 Official leaderboard
23 GPT-5.4 10.7% xhigh Mar 2026 Official leaderboard
24 MiniMax-M3 10% high Jun 2026 Official leaderboard
25 Claude Sonnet 4.6 6.70% high Feb 2026 Official leaderboard
26 DeepSeek-V4-Pro 5.30% max Apr 2026 Official leaderboard
27 Muse Spark Open 5.00% xhigh Oct 2026 Official leaderboard
28 Kimi K2.7 Code 4.70% high Jun 2026 Official leaderboard
29 Gemini 3.6 Flash 4.70% high Jul 2026 Official leaderboard
30 Gemini 3.1 Pro 2.70% high Feb 2026 Official leaderboard
31 Gemini 3.5 Flash 2.70% high May 2026 Official leaderboard
32 Inkling 1.30% high Jul 2026 Official leaderboard
33 MiniMax-M2.7 0.70% high Mar 2026 Official leaderboard
34 gpt-oss-120b 0% high Aug 2025 Official leaderboard
35 Kimi K2 Sep 2025 0% high Sep 2025 Official leaderboard
36 DeepSeek-V3.2 0% Dec 2025 Official leaderboard
37 Gemma 4 31B IT 0% Apr 2026 Official leaderboard
38 Claude Fable 5 0% max Jun 2026 Official leaderboard

3 models have held number one on the official leaderboard since its release in August 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard 3 leaders
Claude Sonnet 5.5 44.7%

Source: Official SWE Atlas Codebase QnA Extended leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Sonnet 5.5 Claude Opus 5 Claude Sonnet 5 Claude Opus 4.8 DeepSeek-V4-Flash Claude Opus 4.7 Claude Sonnet 4.6 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Sonnet 5.5 Claude Sonnet 5

Source: Official SWE Atlas Codebase QnA Extended leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471