Benchmark Almanac

Aider-Edit

DeepSeek-V3 79.7% leads since December 2024
Claude 3.5 Sonnet (October 2024) 84.2% leads since October 2024
Claude 3.5 Sonnet (October 2024) 84.2% leads since October 2024
Dropped

Models edit Python source files to complete 133 Exercism exercises through Aider. Scored by the share of exercises whose unit tests all pass.

Aider replaced it with the polyglot benchmark in December 2024 because top scores passed 80% on its 133 Python exercises.

Made by
Aider
Released
July 2023
Kind
Function-level problems
Level
School
Scored
Checked automatically
Answers
Code
Tasks
All public
Strongest model tested
o1 #117 on the capability index
Last in a launch post
July 2025, dropped for a newer version
Last model added
October 2025
Scores refreshed
November 12th, 2026
5
Models scored in lab cards
3
Launch posts list it, from 2 labs
4
System cards report it, from 3 labs

DeepSeek-V3 leads with 79.7% among models released by October 2025

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Claude 3.5 Sonnet (October 2024) leads with 84.2% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.

Claude 3.5 Sonnet (October 2024) leads with 84.2%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in October 2025, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 79.7 DeepSeek-V3
  2. 78.6 Ring-1T
  3. 72.2 DeepSeek-V2.5
  4. 57.1 Seed-Coder-8B-Instruct
  5. 44.4 Seed Diffusion Preview
  1. 1 DeepSeek-V3 79.7%
  2. 2 Ring-1T 78.6%
  3. 3 DeepSeek-V2.5 Sep 2024 72.2%
  4. 4 Seed-Coder-8B-Instruct 57.1%
  5. 5 Seed Diffusion Preview 44.4%

Source: Labs’ model cards

34 models. Scroll sideways for the rest.

  1. 84.2 Claude 3.5 Sonnet (October 2024)
  2. 84.2 o1
  3. 79.7 o1-preview
  4. 77.4 Claude 3.5 Sonnet
  5. 75.2 Claude 3.5 Haiku
  6. 72.9 GPT-4o
  7. 72.9 Qwen2.5-Coder-32B
  8. 72.2 DeepSeek-V2.5
  9. 72.2 DeepSeek-V2.5-1210
  10. 71.4 GPT-4o
  11. 71.4 GPT-4o
  12. 71.4 Qwen2.5-Coder-32B-Instruct
  13. 69.9 Gemini 2.0 Flash
  14. 69.2 Qwen2.5-Coder-14B-Instruct
  15. 68.4 Claude 3 Opus
  16. 67.7 GPT-4
  17. 66.9 Dracarys2-72B-Instruct
  18. 66.2 GPT-4 Turbo (Jan 2024)
  19. 65.4 Yi-Lightning
  20. 61.7 Qwen2.5-Coder-14B
  21. 61.1 o1-mini
  22. 60.2 Mistral Large 2
  23. 59.4 Llama 3.3 70B Instruct
  24. 58.6 Grok-2
  25. 57.9 GPT-3.5 Turbo (Mar 2023)
  26. 55.6 Qwen2-72B
  27. 55.6 GPT-4o mini
  28. 54.9 Claude 3 Sonnet
  29. 54.9 Grok-2 mini
  30. 54.9 Llama-3.1-Nemotron-70B-Instruct
  31. 54.1 Qwen2.5-32B
  32. 47.4 Claude 3 Haiku
  33. 44.4 WizardLM-2 8x22B
  34. 39.1 Qwen2.5-Coder-3B-Instruct
  1. 1 Claude 3.5 Sonnet (October 2024) 84.2%
  2. 2 o1 84.2%
  3. 3 o1-preview 79.7%
  4. 4 Claude 3.5 Sonnet 77.4%
  5. 5 Claude 3.5 Haiku 75.2%
  6. 6 GPT-4o May 2024 72.9%
  7. 7 Qwen2.5-Coder-32B 72.9%
  8. 8 DeepSeek-V2.5 Sep 2024 72.2%
  9. 9 DeepSeek-V2.5-1210 72.2%
  10. 10 GPT-4o Aug 2024 71.4%
  11. 11 GPT-4o Nov 2024 71.4%
  12. 12 Qwen2.5-Coder-32B-Instruct 71.4%
  13. 13 Gemini 2.0 Flash Dec 2024 69.9%
  14. 14 Qwen2.5-Coder-14B-Instruct 69.2%
  15. 15 Claude 3 Opus 68.4%
  16. 16 GPT-4 Jun 2023 67.7%
  17. 17 Dracarys2-72B-Instruct 66.9%
  18. 18 GPT-4 Turbo (Jan 2024) 66.2%
  19. 19 Yi-Lightning 65.4%
  20. 20 Qwen2.5-Coder-14B 61.7%
  21. 21 o1-mini 61.1%
  22. 22 Mistral Large 2 Jul 2024 60.2%
  23. 23 Llama 3.3 70B Instruct 59.4%
  24. 24 Grok-2 Aug 2024 58.6%
  25. 25 GPT-3.5 Turbo (Mar 2023) 57.9%
  26. 26 Qwen2-72B 55.6%
  27. 27 GPT-4o mini 55.6%
  28. 28 Claude 3 Sonnet 54.9%
  29. 29 Grok-2 mini 54.9%
  30. 30 Llama-3.1-Nemotron-70B-Instruct 54.9%
  31. 31 Qwen2.5-32B 54.1%
  32. 32 Claude 3 Haiku 47.4%
  33. 33 WizardLM-2 8x22B 44.4%
  34. 34 Qwen2.5-Coder-3B-Instruct 39.1%

Source: Official Aider-Edit leaderboard

38 models. Scroll sideways for the rest.

  1. 84.2 Claude 3.5 Sonnet (October 2024)
  2. 84.2 o1
  3. 79.7 o1-preview
  4. 79.7 DeepSeek-V3
  5. 78.6 Ring-1T
  6. 77.4 Claude 3.5 Sonnet
  7. 75.2 Claude 3.5 Haiku
  8. 72.9 GPT-4o
  9. 72.9 Qwen2.5-Coder-32B
  10. 72.2 DeepSeek-V2.5
  11. 72.2 DeepSeek-V2.5-1210
  12. 71.4 GPT-4o
  13. 71.4 GPT-4o
  14. 71.4 Qwen2.5-Coder-32B-Instruct
  15. 69.9 Gemini 2.0 Flash
  16. 69.2 Qwen2.5-Coder-14B-Instruct
  17. 68.4 Claude 3 Opus
  18. 67.7 GPT-4
  19. 66.9 Dracarys2-72B-Instruct
  20. 66.2 GPT-4 Turbo (Jan 2024)
  21. 65.4 Yi-Lightning
  22. 61.7 Qwen2.5-Coder-14B
  23. 61.1 o1-mini
  24. 60.2 Mistral Large 2
  25. 59.4 Llama 3.3 70B Instruct
  26. 58.6 Grok-2
  27. 57.9 GPT-3.5 Turbo (Mar 2023)
  28. 57.1 Seed-Coder-8B-Instruct
  29. 55.6 Qwen2-72B
  30. 55.6 GPT-4o mini
  31. 54.9 Claude 3 Sonnet
  32. 54.9 Grok-2 mini
  33. 54.9 Llama-3.1-Nemotron-70B-Instruct
  34. 54.1 Qwen2.5-32B
  35. 47.4 Claude 3 Haiku
  36. 44.4 WizardLM-2 8x22B
  37. 44.4 Seed Diffusion Preview
  38. 39.1 Qwen2.5-Coder-3B-Instruct
  1. 1 Claude 3.5 Sonnet (October 2024) 84.2%
  2. 2 o1 84.2%
  3. 3 o1-preview 79.7%
  4. 4 DeepSeek-V3 79.7%
  5. 5 Ring-1T 78.6%
  6. 6 Claude 3.5 Sonnet 77.4%
  7. 7 Claude 3.5 Haiku 75.2%
  8. 8 GPT-4o May 2024 72.9%
  9. 9 Qwen2.5-Coder-32B 72.9%
  10. 10 DeepSeek-V2.5 Sep 2024 72.2%
  11. 11 DeepSeek-V2.5-1210 72.2%
  12. 12 GPT-4o Aug 2024 71.4%
  13. 13 GPT-4o Nov 2024 71.4%
  14. 14 Qwen2.5-Coder-32B-Instruct 71.4%
  15. 15 Gemini 2.0 Flash Dec 2024 69.9%
  16. 16 Qwen2.5-Coder-14B-Instruct 69.2%
  17. 17 Claude 3 Opus 68.4%
  18. 18 GPT-4 Jun 2023 67.7%
  19. 19 Dracarys2-72B-Instruct 66.9%
  20. 20 GPT-4 Turbo (Jan 2024) 66.2%
  21. 21 Yi-Lightning 65.4%
  22. 22 Qwen2.5-Coder-14B 61.7%
  23. 23 o1-mini 61.1%
  24. 24 Mistral Large 2 Jul 2024 60.2%
  25. 25 Llama 3.3 70B Instruct 59.4%
  26. 26 Grok-2 Aug 2024 58.6%
  27. 27 GPT-3.5 Turbo (Mar 2023) 57.9%
  28. 28 Seed-Coder-8B-Instruct 57.1%
  29. 29 Qwen2-72B 55.6%
  30. 30 GPT-4o mini 55.6%
  31. 31 Claude 3 Sonnet 54.9%
  32. 32 Grok-2 mini 54.9%
  33. 33 Llama-3.1-Nemotron-70B-Instruct 54.9%
  34. 34 Qwen2.5-32B 54.1%
  35. 35 Claude 3 Haiku 47.4%
  36. 36 WizardLM-2 8x22B 44.4%
  37. 37 Seed Diffusion Preview 44.4%
  38. 38 Qwen2.5-Coder-3B-Instruct 39.1%

Sources: Official Aider-Edit leaderboard and labs’ model cards

#ModelScoreSettingDateSource
1 DeepSeek-V3 79.7% Dec 2024 Lab model card
2 Ring-1T 78.6% Oct 2025 Lab model card
3 DeepSeek-V2.5 Sep 2024 72.2% Sep 2024 Lab model card
4 Seed-Coder-8B-Instruct 57.1% Jun 2025 Lab model card
5 Seed Diffusion Preview 44.4% Jul 2025 Lab model card
#ModelScoreSettingDateSource
1 Claude 3.5 Sonnet (October 2024) 84.2% aider, diff edits Oct 2024 Official leaderboard
2 o1 84.2% aider, diff edits Dec 2024 Official leaderboard
3 o1-preview 79.7% aider, diff edits Sep 2024 Official leaderboard
4 Claude 3.5 Sonnet 77.4% aider, diff edits Jun 2024 Official leaderboard
5 Claude 3.5 Haiku 75.2% aider, diff edits Oct 2024 Official leaderboard
6 GPT-4o May 2024 72.9% aider, diff edits May 2024 Official leaderboard
7 Qwen2.5-Coder-32B 72.9% aider, whole edits Sep 2024 Official leaderboard
8 DeepSeek-V2.5 Sep 2024 72.2% aider, diff edits Sep 2024 Official leaderboard
9 DeepSeek-V2.5-1210 72.2% aider, diff edits Dec 2024 Official leaderboard
10 GPT-4o Aug 2024 71.4% aider, diff edits Aug 2024 Official leaderboard
11 GPT-4o Nov 2024 71.4% aider, diff edits Nov 2024 Official leaderboard
12 Qwen2.5-Coder-32B-Instruct 71.4% aider, diff edits Nov 2024 Official leaderboard
13 Gemini 2.0 Flash Dec 2024 69.9% aider, diff edits Dec 2024 Official leaderboard
14 Qwen2.5-Coder-14B-Instruct 69.2% aider, whole edits Nov 2024 Official leaderboard
15 Claude 3 Opus 68.4% aider, diff edits Feb 2024 Official leaderboard
16 GPT-4 Jun 2023 67.7% aider, diff edits Jun 2023 Official leaderboard
17 Dracarys2-72B-Instruct 66.9% aider, whole edits Sep 2024 Official leaderboard
18 GPT-4 Turbo (Jan 2024) 66.2% aider, udiff edits Jan 2024 Official leaderboard
19 Yi-Lightning 65.4% aider, whole edits Dec 2024 Official leaderboard
20 Qwen2.5-Coder-14B 61.7% aider, whole edits Sep 2024 Official leaderboard
21 o1-mini 61.1% aider, diff edits Sep 2024 Official leaderboard
22 Mistral Large 2 Jul 2024 60.2% aider, whole edits Jul 2024 Official leaderboard
23 Llama 3.3 70B Instruct 59.4% aider, diff edits Dec 2024 Official leaderboard
24 Grok-2 Aug 2024 58.6% aider, whole edits Aug 2024 Official leaderboard
25 GPT-3.5 Turbo (Mar 2023) 57.9% aider, whole edits Mar 2023 Official leaderboard
26 Qwen2-72B 55.6% aider, whole edits Jun 2024 Official leaderboard
27 GPT-4o mini 55.6% aider, whole edits Jul 2024 Official leaderboard
28 Claude 3 Sonnet 54.9% aider, whole edits Feb 2024 Official leaderboard
29 Grok-2 mini 54.9% aider, whole edits Aug 2024 Official leaderboard
30 Llama-3.1-Nemotron-70B-Instruct 54.9% aider, whole edits Oct 2024 Official leaderboard
31 Qwen2.5-32B 54.1% aider, whole edits Sep 2024 Official leaderboard
32 Claude 3 Haiku 47.4% aider, whole edits Mar 2024 Official leaderboard
33 WizardLM-2 8x22B 44.4% aider, whole edits Apr 2024 Official leaderboard
34 Qwen2.5-Coder-3B-Instruct 39.1% aider, whole edits Nov 2024 Official leaderboard
#ModelScoreSettingDateSource
1 Claude 3.5 Sonnet (October 2024) 84.2% aider, diff edits Oct 2024 Official leaderboard
2 o1 84.2% aider, diff edits Dec 2024 Official leaderboard
3 o1-preview 79.7% aider, diff edits Sep 2024 Official leaderboard
4 DeepSeek-V3 79.7% Dec 2024 Lab model card
5 Ring-1T 78.6% Oct 2025 Lab model card
6 Claude 3.5 Sonnet 77.4% aider, diff edits Jun 2024 Official leaderboard
7 Claude 3.5 Haiku 75.2% aider, diff edits Oct 2024 Official leaderboard
8 GPT-4o May 2024 72.9% aider, diff edits May 2024 Official leaderboard
9 Qwen2.5-Coder-32B 72.9% aider, whole edits Sep 2024 Official leaderboard
10 DeepSeek-V2.5 Sep 2024 72.2% aider, diff edits Sep 2024 Official leaderboard
11 DeepSeek-V2.5-1210 72.2% aider, diff edits Dec 2024 Official leaderboard
12 GPT-4o Aug 2024 71.4% aider, diff edits Aug 2024 Official leaderboard
13 GPT-4o Nov 2024 71.4% aider, diff edits Nov 2024 Official leaderboard
14 Qwen2.5-Coder-32B-Instruct 71.4% aider, diff edits Nov 2024 Official leaderboard
15 Gemini 2.0 Flash Dec 2024 69.9% aider, diff edits Dec 2024 Official leaderboard
16 Qwen2.5-Coder-14B-Instruct 69.2% aider, whole edits Nov 2024 Official leaderboard
17 Claude 3 Opus 68.4% aider, diff edits Feb 2024 Official leaderboard
18 GPT-4 Jun 2023 67.7% aider, diff edits Jun 2023 Official leaderboard
19 Dracarys2-72B-Instruct 66.9% aider, whole edits Sep 2024 Official leaderboard
20 GPT-4 Turbo (Jan 2024) 66.2% aider, udiff edits Jan 2024 Official leaderboard
21 Yi-Lightning 65.4% aider, whole edits Dec 2024 Official leaderboard
22 Qwen2.5-Coder-14B 61.7% aider, whole edits Sep 2024 Official leaderboard
23 o1-mini 61.1% aider, diff edits Sep 2024 Official leaderboard
24 Mistral Large 2 Jul 2024 60.2% aider, whole edits Jul 2024 Official leaderboard
25 Llama 3.3 70B Instruct 59.4% aider, diff edits Dec 2024 Official leaderboard
26 Grok-2 Aug 2024 58.6% aider, whole edits Aug 2024 Official leaderboard
27 GPT-3.5 Turbo (Mar 2023) 57.9% aider, whole edits Mar 2023 Official leaderboard
28 Seed-Coder-8B-Instruct 57.1% Jun 2025 Lab model card
29 Qwen2-72B 55.6% aider, whole edits Jun 2024 Official leaderboard
30 GPT-4o mini 55.6% aider, whole edits Jul 2024 Official leaderboard
31 Claude 3 Sonnet 54.9% aider, whole edits Feb 2024 Official leaderboard
32 Grok-2 mini 54.9% aider, whole edits Aug 2024 Official leaderboard
33 Llama-3.1-Nemotron-70B-Instruct 54.9% aider, whole edits Oct 2024 Official leaderboard
34 Qwen2.5-32B 54.1% aider, whole edits Sep 2024 Official leaderboard
35 Claude 3 Haiku 47.4% aider, whole edits Mar 2024 Official leaderboard
36 WizardLM-2 8x22B 44.4% aider, whole edits Apr 2024 Official leaderboard
37 Seed Diffusion Preview 44.4% Jul 2025 Lab model card
38 Qwen2.5-Coder-3B-Instruct 39.1% aider, whole edits Nov 2024 Official leaderboard

The benchmark's own leaderboard has 96 entries, 34 of them models we track. Switch the scores above to rank them together.

Of runs with a cost, the best scores 79.7% for $0.48 a task

Runs on the cost frontier score higher than any run that's cheaper. Claude 3.5 Sonnet comes within 2.3 points of the best for $0.03 a task, 18 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Anthropic OpenAI DeepSeek Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale o1-preview Claude 3.5 Sonnet DeepSeek-V2.5 GPT-4o mini 0% 20% 40% 60% 80% 100% $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale o1-preview Claude 3.5 Sonnet

Source: Official Aider-Edit leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude 3.5 Sonnet (October 2024) aider, diff edits 84.2%
o1 aider, diff edits 84.2%
o1-preview aider, diff edits 79.7% $0.48
Claude 3.5 Sonnet aider, diff edits 77.4% $0.03
Claude 3.5 Haiku aider, diff edits 75.2%
GPT-4o May 2024 aider, diff edits 72.9%
Qwen2.5-Coder-32B aider, whole edits 72.9%
DeepSeek-V2.5 Sep 2024 aider, diff edits 72.2% $0.0008
DeepSeek-V2.5-1210 aider, diff edits 72.2% $0.0008
GPT-4o Aug 2024 aider, diff edits 71.4%
GPT-4o Nov 2024 aider, diff edits 71.4%
Qwen2.5-Coder-32B-Instruct aider, diff edits 71.4%
Gemini 2.0 Flash Dec 2024 aider, diff edits 69.9%
Qwen2.5-Coder-14B-Instruct aider, whole edits 69.2%
Claude 3 Opus aider, diff edits 68.4% $0.10
GPT-4 Jun 2023 aider, diff edits 67.7% $0.13
Dracarys2-72B-Instruct aider, whole edits 66.9%
GPT-4 Turbo (Jan 2024) aider, udiff edits 66.2% $0.11
Yi-Lightning aider, whole edits 65.4%
Qwen2.5-Coder-14B aider, whole edits 61.7%
o1-mini aider, diff edits 61.1% $0.07
Mistral Large 2 Jul 2024 aider, whole edits 60.2%
Llama 3.3 70B Instruct aider, diff edits 59.4%
Grok-2 Aug 2024 aider, whole edits 58.6%
GPT-3.5 Turbo (Mar 2023) aider, whole edits 57.9% $0.004
Qwen2-72B aider, whole edits 55.6%
GPT-4o mini aider, whole edits 55.6% $0.0007
Claude 3 Sonnet aider, whole edits 54.9%
Grok-2 mini aider, whole edits 54.9%
Llama-3.1-Nemotron-70B-Instruct aider, whole edits 54.9%
Qwen2.5-32B aider, whole edits 54.1%
Claude 3 Haiku aider, whole edits 47.4% $0.001
WizardLM-2 8x22B aider, whole edits 44.4%
Qwen2.5-Coder-3B-Instruct aider, whole edits 39.1%
o1-mini aider, whole edits 27.1% $0.08

2 models have held number one in labs’ own numbers since its release in July 2023 6 models have held number one across all independent runs since its release in July 2023 6 models have held number one with claims included since its release in July 2023

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Anthropic DeepSeek
Own leaderboard Replaced by Aider Polyglot, Dec 2024
Claude 3.5 Sonnet (October 2024) 84.2%
Lab claims Not tested since Oct 2025
DeepSeek-V3 79.7%

Sources: Official Aider-Edit leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) Claude 3.5 Sonnet (October 2024) o1-preview GPT-4o Claude 3 Opus GPT-4 GPT-3.5 Turbo (Mar 2023) 0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) Claude 3.5 Sonnet (October 2024) Claude 3.5 Sonnet

Sources: Official Aider-Edit leaderboard and labs’ model cards

Labs moved on to Aider Polyglot

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

DeepSeek
ByteDance Seed
DateLaunch postScoreAs the post writes it
Jul 31, 2025 Seed Diffusion Preview 44.4 Aider (whole format, tries=2)
Dec 26, 2024 DeepSeek-V3 79.7 Aider-Edit (Acc.)
Sep 5, 2024 DeepSeek-V2.5 72.2 Aider

4 system cards report it, from 3 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 21, 2025 Ring-1T technical report 78.6 Aider, cited as aider.chat/docs/benchmarks.html#the-benchmark (Table 1) Capabilities
Aug 4, 2025 Seed Diffusion Preview technical report 44.4 Aider, whole format, tries=2 (Table 1) Capabilities
Jun 4, 2025 Seed-Coder-8B-Instruct technical report 57.1 Aider ("whole" format) tries=2 (Table 13) Capabilities
Dec 26, 2024 DeepSeek-V3 technical report 79.7 Aider-Edit (Acc.), DeepSeek-V3 (Table 6) Capabilities

One lab claim next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 1 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
DeepSeek-V2.5 Sep 2024 72.2% Aider 72.2% Leaderboard, aider, diff edits Same

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

DeepSeek Google DeepMind OpenAI
Aider Polyglot Passed 85% in Aug 2025, by GPT-5
GPT-5 88%
Aider-Edit Not tested since Oct 2025
DeepSeek-V3 79.7%
Aider Polyglot Passed 85% in Aug 2025, by GPT-5
GPT-5 88%

Source: Labs’ model cards

Replaced by Aider Polyglot

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471