Aider Refactoring Benchmark
Models refactor large Python methods while preserving their implementation. The benchmark checks whether edits complete correctly.
Aider tests 89 methods from Python classes.
- Made by
- Aider
- Released
- December 2023
- Kind
- Fixing real repositories
- Level
- University
- Scored
- Checked automatically
- Answers
- Code
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Real work and use
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
- o1-mini #142 on the capability index
- Last model added
- October 2024
- Scores refreshed
- November 20th, 2026
Claude 3.5 Sonnet (October 2024) leads with 92.1% among models released by October 2024 Claude 3.5 Sonnet (October 2024) leads with 92.1% among models released by October 2024
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
The newest model with a score here came out in October 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 1 Claude 3.5 Sonnet (October 2024) diff 92.1%
- 2 o1-preview diff 75.3%
- 3 Claude 3 Opus diff 72.3%
- 4 Claude 3.5 Sonnet diff 64%
- 5 GPT-4o May 2024, diff 62.9%
- 6 GPT-4 Turbo Nov 2023, udiff 50.6%
- 7
Gemini 1.5 Pro Feb 2024, diff-fenced 49.4%
- 8 GPT-4o Aug 2024, diff 49.4%
- 9 o1-mini diff 44.9%
- 10 GPT-4 Turbo Apr 2024, udiff 34.1%
- 11 GPT-4 Turbo (Jan 2024) udiff 33.7%
- 12 DeepSeek Coder V2 0724 (deprecated) diff 32.6%
- 13
DeepSeek-V2.5 Sep 2024, diff 31.5%
Source: Official Aider Refactoring Benchmark leaderboard
- 1 Claude 3.5 Sonnet (October 2024) diff 92.1%
- 2 o1-preview diff 75.3%
- 3 Claude 3 Opus diff 72.3%
- 4 Claude 3.5 Sonnet diff 64%
- 5 GPT-4o May 2024, diff 62.9%
- 6 GPT-4 Turbo Nov 2023, udiff 50.6%
- 7
Gemini 1.5 Pro Feb 2024, diff-fenced 49.4%
- 8 GPT-4o Aug 2024, diff 49.4%
- 9 o1-mini diff 44.9%
- 10 GPT-4 Turbo Apr 2024, udiff 34.1%
- 11 GPT-4 Turbo (Jan 2024) udiff 33.7%
- 12
DeepSeek-V2.5 Sep 2024, diff 31.5%
Source: Official Aider Refactoring Benchmark leaderboard
| # | Entry | Score | Setting | Released | Run on | Source |
|---|---|---|---|---|---|---|
| 1 | Claude 3.5 Sonnet (October 2024) | 92.1% | diff, aider | Oct 2024 | Oct 2024 | Official leaderboard |
| 2 | o1-preview | 75.3% | diff, aider | Sep 2024 | Oct 2024 | Official leaderboard |
| 3 | Claude 3 Opus | 72.3% | diff, aider | Feb 2024 | May 2024 | Official leaderboard |
| 4 | Claude 3.5 Sonnet | 64% | diff, aider | Jun 2024 | Jul 2024 | Official leaderboard |
| 5 | GPT-4o May 2024 | 62.9% | diff, aider | May 2024 | May 2024 | Official leaderboard |
| 6 | GPT-4 Turbo Nov 2023 | 50.6% | udiff, aider | Nov 2023 | May 2024 | Official leaderboard |
| 7 | | 49.4% | diff-fenced, aider | Feb 2024 | May 2024 | Official leaderboard |
| 8 | GPT-4o Aug 2024 | 49.4% | diff, aider | Aug 2024 | Aug 2024 | Official leaderboard |
| 9 | o1-mini | 44.9% | diff, aider | Sep 2024 | Oct 2024 | Official leaderboard |
| 10 | GPT-4 Turbo Apr 2024 | 34.1% | udiff, aider | Apr 2024 | Apr 2024 | Official leaderboard |
| 11 | GPT-4 Turbo (Jan 2024) | 33.7% | udiff, aider | Jan 2024 | May 2024 | Official leaderboard |
| 12 | DeepSeek Coder V2 0724 (deprecated) | 32.6% | diff, aider | Jul 2024 | Official leaderboard | |
| 13 | | 31.5% | diff, aider | Sep 2024 | Sep 2024 | Official leaderboard |
| # | Model | Score | Setting | Released | Run on | Source |
|---|---|---|---|---|---|---|
| 1 | Claude 3.5 Sonnet (October 2024) | 92.1% | diff, aider | Oct 2024 | Oct 2024 | Official leaderboard |
| 2 | o1-preview | 75.3% | diff, aider | Sep 2024 | Oct 2024 | Official leaderboard |
| 3 | Claude 3 Opus | 72.3% | diff, aider | Feb 2024 | May 2024 | Official leaderboard |
| 4 | Claude 3.5 Sonnet | 64% | diff, aider | Jun 2024 | Jul 2024 | Official leaderboard |
| 5 | GPT-4o May 2024 | 62.9% | diff, aider | May 2024 | May 2024 | Official leaderboard |
| 6 | GPT-4 Turbo Nov 2023 | 50.6% | udiff, aider | Nov 2023 | May 2024 | Official leaderboard |
| 7 | | 49.4% | diff-fenced, aider | Feb 2024 | May 2024 | Official leaderboard |
| 8 | GPT-4o Aug 2024 | 49.4% | diff, aider | Aug 2024 | Aug 2024 | Official leaderboard |
| 9 | o1-mini | 44.9% | diff, aider | Sep 2024 | Oct 2024 | Official leaderboard |
| 10 | GPT-4 Turbo Apr 2024 | 34.1% | udiff, aider | Apr 2024 | Apr 2024 | Official leaderboard |
| 11 | GPT-4 Turbo (Jan 2024) | 33.7% | udiff, aider | Jan 2024 | May 2024 | Official leaderboard |
| 12 | | 31.5% | diff, aider | Sep 2024 | Sep 2024 | Official leaderboard |
The best score, 92.1%, costs $0.10 a task
Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Source: Official Aider Refactoring Benchmark leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Claude 3.5 Sonnet (October 2024) | diff | aider | 92.1% | $0.10 | |
| o1-preview | diff | aider | 75.3% | $1.36 | |
| Claude 3 Opus | diff | aider | 72.3% | $0.34 | |
| Claude 3.5 Sonnet | diff | aider | 64% | $0.13 | |
| GPT-4o May 2024 | diff | aider | 62.9% | ||
| GPT-4 Turbo Nov 2023 | udiff | aider | 50.6% | $0.21 | |
| diff-fenced | aider | 49.4% | |||
| GPT-4o Aug 2024 | diff | aider | 49.4% | $0.05 | |
| o1-mini | diff | aider | 44.9% | $0.33 | |
| GPT-4 Turbo Apr 2024 | udiff | aider | 34.1% | $0.22 | |
| GPT-4 Turbo (Jan 2024) | udiff | aider | 33.7% | $0.23 | |
| diff | aider | 31.5% | $0.01 | ||
| GPT-4 Turbo Apr 2024 | diff | aider | 21.4% | $0.23 |
The best score, 92.1%, costs $0.10 a task
Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Source: Official Aider Refactoring Benchmark leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Claude 3.5 Sonnet (October 2024) | diff | aider | 92.1% | $0.10 | |
| o1-preview | diff | aider | 75.3% | $1.36 | |
| Claude 3 Opus | diff | aider | 72.3% | $0.34 | |
| Claude 3.5 Sonnet | diff | aider | 64% | $0.13 | |
| GPT-4o May 2024 | diff | aider | 62.9% | ||
| GPT-4 Turbo Nov 2023 | udiff | aider | 50.6% | $0.21 | |
| diff-fenced | aider | 49.4% | |||
| GPT-4o Aug 2024 | diff | aider | 49.4% | $0.05 | |
| o1-mini | diff | aider | 44.9% | $0.33 | |
| GPT-4 Turbo Apr 2024 | udiff | aider | 34.1% | $0.22 | |
| GPT-4 Turbo (Jan 2024) | udiff | aider | 33.7% | $0.23 | |
| diff | aider | 31.5% | $0.01 | ||
| GPT-4 Turbo Apr 2024 | diff | aider | 21.4% | $0.23 |
4 models have held number one on the official leaderboard since its release in December 2023
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official Aider Refactoring Benchmark leaderboard
Claude 3.5 Sonnet (October 2024) on the official leaderboard passed 85% in October 2024
That was 10 months after its release. By the same age, 13% of the benchmarks from 2023 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official Aider Refactoring Benchmark leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Coding
- SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
- LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests.
DeepSeek-V4-Pro 93.5%
- SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
- DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
- SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases.
Seed2.1 Pro 59.8%
- Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance.
DeepSeek V4.1 Flash 3,471
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the official leaderboard (13 entries).
- Pass Models scored 12 models we track, plus 1 other entry on its leaderboard.
- Note Newest score The newest came 2 years ago, in October 2024, for Claude 3.5 Sonnet (October 2024). That’s expected for a saturated test.
- Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is o1-mini (#142).
- Note Headroom Saturated. The best is 92.1%, past 85% since October 2024.
Leaderboard import 3 pass
- Pass Importer We read the board on aider.chat: 13 rows for models we track and 1 other entry at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.