Benchmark Almanac

Aider Refactoring Benchmark

Claude 3.5 Sonnet (October 2024) 92.1% leads since October 2024
Not in model cards

Models refactor large Python methods while preserving their implementation. The benchmark checks whether edits complete correctly.

Aider tests 89 methods from Python classes.

Made by
Aider
Released
December 2023
Kind
Fixing real repositories
Level
University
Scored
Checked automatically
Answers
Code
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Code → Code
Interaction
Single turn
Items from
Real work and use
Refresh
One fixed set
Run by
Anyone
Strongest model tested
o1-mini #142 on the capability index
Last model added
October 2024
Scores refreshed
November 20th, 2026
13
Entries on its own leaderboard, 12 models we track

Claude 3.5 Sonnet (October 2024) leads with 92.1% among models released by October 2024 Claude 3.5 Sonnet (October 2024) leads with 92.1% among models released by October 2024

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in October 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Claude 3.5 Sonnet (October 2024) diff 92.1%
  2. 2 o1-preview diff 75.3%
  3. 3 Claude 3 Opus diff 72.3%
  4. 4 Claude 3.5 Sonnet diff 64%
  5. 5 GPT-4o May 2024, diff 62.9%
  6. 6 GPT-4 Turbo Nov 2023, udiff 50.6%
  7. 7 Gemini 1.5 Pro Feb 2024, diff-fenced 49.4%
  8. 8 GPT-4o Aug 2024, diff 49.4%
  9. 9 o1-mini diff 44.9%
  10. 10 GPT-4 Turbo Apr 2024, udiff 34.1%
  11. 11 GPT-4 Turbo (Jan 2024) udiff 33.7%
  12. 12 DeepSeek Coder V2 0724 (deprecated) diff 32.6%
  13. 13 DeepSeek-V2.5 Sep 2024, diff 31.5%

Source: Official Aider Refactoring Benchmark leaderboard

#EntryScoreSettingReleasedRun onSource
1 Claude 3.5 Sonnet (October 2024) 92.1% diff, aider Oct 2024 Oct 2024 Official leaderboard
2 o1-preview 75.3% diff, aider Sep 2024 Oct 2024 Official leaderboard
3 Claude 3 Opus 72.3% diff, aider Feb 2024 May 2024 Official leaderboard
4 Claude 3.5 Sonnet 64% diff, aider Jun 2024 Jul 2024 Official leaderboard
5 GPT-4o May 2024 62.9% diff, aider May 2024 May 2024 Official leaderboard
6 GPT-4 Turbo Nov 2023 50.6% udiff, aider Nov 2023 May 2024 Official leaderboard
7 Gemini 1.5 Pro Feb 2024 49.4% diff-fenced, aider Feb 2024 May 2024 Official leaderboard
8 GPT-4o Aug 2024 49.4% diff, aider Aug 2024 Aug 2024 Official leaderboard
9 o1-mini 44.9% diff, aider Sep 2024 Oct 2024 Official leaderboard
10 GPT-4 Turbo Apr 2024 34.1% udiff, aider Apr 2024 Apr 2024 Official leaderboard
11 GPT-4 Turbo (Jan 2024) 33.7% udiff, aider Jan 2024 May 2024 Official leaderboard
12 DeepSeek Coder V2 0724 (deprecated) 32.6% diff, aider Jul 2024 Official leaderboard
13 DeepSeek-V2.5 Sep 2024 31.5% diff, aider Sep 2024 Sep 2024 Official leaderboard
#ModelScoreSettingReleasedRun onSource
1 Claude 3.5 Sonnet (October 2024) 92.1% diff, aider Oct 2024 Oct 2024 Official leaderboard
2 o1-preview 75.3% diff, aider Sep 2024 Oct 2024 Official leaderboard
3 Claude 3 Opus 72.3% diff, aider Feb 2024 May 2024 Official leaderboard
4 Claude 3.5 Sonnet 64% diff, aider Jun 2024 Jul 2024 Official leaderboard
5 GPT-4o May 2024 62.9% diff, aider May 2024 May 2024 Official leaderboard
6 GPT-4 Turbo Nov 2023 50.6% udiff, aider Nov 2023 May 2024 Official leaderboard
7 Gemini 1.5 Pro Feb 2024 49.4% diff-fenced, aider Feb 2024 May 2024 Official leaderboard
8 GPT-4o Aug 2024 49.4% diff, aider Aug 2024 Aug 2024 Official leaderboard
9 o1-mini 44.9% diff, aider Sep 2024 Oct 2024 Official leaderboard
10 GPT-4 Turbo Apr 2024 34.1% udiff, aider Apr 2024 Apr 2024 Official leaderboard
11 GPT-4 Turbo (Jan 2024) 33.7% udiff, aider Jan 2024 May 2024 Official leaderboard
12 DeepSeek-V2.5 Sep 2024 31.5% diff, aider Sep 2024 Sep 2024 Official leaderboard

The best score, 92.1%, costs $0.10 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI DeepSeek Anthropic Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 Cost per task, log scale Claude 3.5 Sonnet (October 2024), diff GPT-4o, diff DeepSeek-V2.5, diff 0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 Cost per task, log scale Claude 3.5 Sonnet (October 2024), diff GPT-4o, diff

Source: Official Aider Refactoring Benchmark leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude 3.5 Sonnet (October 2024) diff aider 92.1% $0.10
o1-preview diff aider 75.3% $1.36
Claude 3 Opus diff aider 72.3% $0.34
Claude 3.5 Sonnet diff aider 64% $0.13
GPT-4o May 2024 diff aider 62.9%
GPT-4 Turbo Nov 2023 udiff aider 50.6% $0.21
Gemini 1.5 Pro Feb 2024 diff-fenced aider 49.4%
GPT-4o Aug 2024 diff aider 49.4% $0.05
o1-mini diff aider 44.9% $0.33
GPT-4 Turbo Apr 2024 udiff aider 34.1% $0.22
GPT-4 Turbo (Jan 2024) udiff aider 33.7% $0.23
DeepSeek-V2.5 Sep 2024 diff aider 31.5% $0.01
GPT-4 Turbo Apr 2024 diff aider 21.4% $0.23

The best score, 92.1%, costs $0.10 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI DeepSeek Anthropic Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 Cost per task, log scale Claude 3.5 Sonnet (October 2024), diff GPT-4o, diff DeepSeek-V2.5, diff 0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 Cost per task, log scale Claude 3.5 Sonnet (October 2024), diff GPT-4o, diff

Source: Official Aider Refactoring Benchmark leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude 3.5 Sonnet (October 2024) diff aider 92.1% $0.10
o1-preview diff aider 75.3% $1.36
Claude 3 Opus diff aider 72.3% $0.34
Claude 3.5 Sonnet diff aider 64% $0.13
GPT-4o May 2024 diff aider 62.9%
GPT-4 Turbo Nov 2023 udiff aider 50.6% $0.21
Gemini 1.5 Pro Feb 2024 diff-fenced aider 49.4%
GPT-4o Aug 2024 diff aider 49.4% $0.05
o1-mini diff aider 44.9% $0.33
GPT-4 Turbo Apr 2024 udiff aider 34.1% $0.22
GPT-4 Turbo (Jan 2024) udiff aider 33.7% $0.23
DeepSeek-V2.5 Sep 2024 diff aider 31.5% $0.01
GPT-4 Turbo Apr 2024 diff aider 21.4% $0.23

4 models have held number one on the official leaderboard since its release in December 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Anthropic
Own leaderboard Passed 85% in Oct 2024, by Claude 3.5 Sonnet (October 2024)

Source: Official Aider Refactoring Benchmark leaderboard

Claude 3.5 Sonnet (October 2024) on the official leaderboard passed 85% in October 2024

That was 10 months after its release. By the same age, 13% of the benchmarks from 2023 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 2025 Near the ceiling (85%) Claude 3.5 Sonnet (October 2024) o1-preview Claude 3 Opus GPT-4 Turbo 0% 20% 40% 60% 80% 100% 2024 2025 Near the ceiling (85%) Claude 3.5 Sonnet (October 2024) o1-preview

Source: Official Aider Refactoring Benchmark leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Coding

All 523 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471

Data audit

All 8 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the official leaderboard (13 entries).
  • Pass Models scored 12 models we track, plus 1 other entry on its leaderboard.
  • Note Newest score The newest came 2 years ago, in October 2024, for Claude 3.5 Sonnet (October 2024). That’s expected for a saturated test.
  • Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is o1-mini (#142).
  • Note Headroom Saturated. The best is 92.1%, past 85% since October 2024.
Leaderboard import 3 pass
  • Pass Importer We read the board on aider.chat: 13 rows for models we track and 1 other entry at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check