Benchmark Almanac

SWE-PolyBench

Amazon Q Developer Agent (v20250402) 22.6% tops its leaderboard Claude 3.5 Sonnet (October 2024) 14.1% leads the models we track
Not in model cards

Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution.

A stratified 500-task subset (SWE-PolyBench500) is common.

Made by
Amazon
Released
April 2025
Kind
Fixing real repositories
Level
Expert
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Strongest model tested
DeepSeek-R1 #129 on the capability index
Last model added
January 2025
Scores refreshed
November 12th, 2026

In the authors’ words

Coding agents powered by large language models have shown impressive capabilities in software engineering tasks, but evaluating their performance across diverse programming languages and real-world scenarios remains challenging. We introduce SWE-PolyBench, a new multi-language benchmark for repository-level, execution-based evaluation of coding agents. SWE-PolyBench contains 2110 instances from 21 repositories and includes tasks in Java (165), JavaScript (1017), TypeScript (729) and Python (199), covering bug fixes, feature additions, and code refactoring. We provide a task and repository-stratified subsample (SWE-PolyBench500) and release an evaluation harness allowing for fully automated evaluation. To enable a more comprehensive comparison of coding agents, this work also presents a novel set of metrics rooted in syntax tree analysis. We evaluate leading open source coding agents on SWE-PolyBench, revealing their strengths and limitations across languages, task types, and complexity classes. Our experiments show that current agents exhibit uneven performances across languages and struggle with complex problems while showing higher performance on simpler tasks. SWE-PolyBench aims to drive progress in developing more versatile and robust AI coding assistants for real-world software engineering. Our datasets and code are available at: https://github.com/amazon-science/SWE-PolyBench

Abstract of SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents (arXiv:2504.08703, CC0)
7
Entries on its own leaderboard, 6 models we track

Amazon Q Developer Agent (v20250402) tops the leaderboard with 22.6%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Claude 3.5 Sonnet (October 2024), at 14.1%.

The newest model with a score here came out in January 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 22.6 Amazon Q Developer Agent (v20250402)
  2. 14.1 Claude 3.5 Sonnet (October 2024)
  3. 11.5 DeepSeek-R1
  4. 9.86 Claude 3.5 Haiku
  5. 6.02 Llama 3.3 70B
  6. 5.88 Mistral Large 2
  7. 5.31 DeepSeek-R1-Distill-Llama-70B
  1. 1 Amazon Q Developer Agent (v20250402) Full 22.6%
  2. 2 Claude 3.5 Sonnet (October 2024) Full 14.1%
  3. 3 DeepSeek-R1 Full 11.5%
  4. 4 Claude 3.5 Haiku Full 9.86%
  5. 5 Llama 3.3 70B Full 6.02%
  6. 6 Mistral Large 2 Nov 2024, Full 5.88%
  7. 7 DeepSeek-R1-Distill-Llama-70B Full 5.31%

Source: Official SWE-PolyBench leaderboard

#EntryScoreSettingDateSource
1 Amazon Q Developer Agent (v20250402) 22.6% Full Apr 2025 Official leaderboard
2 Claude 3.5 Sonnet (October 2024) 14.1% Full, Aider-PB Oct 2024 Official leaderboard
3 DeepSeek-R1 11.5% Full, Aider-PB Jan 2025 Official leaderboard
4 Claude 3.5 Haiku 9.86% Full, Aider-PB Oct 2024 Official leaderboard
5 Llama 3.3 70B 6.02% Full, Aider-PB Dec 2024 Official leaderboard
6 Mistral Large 2 Nov 2024 5.88% Full, Aider-PB Nov 2024 Official leaderboard
7 DeepSeek-R1-Distill-Llama-70B 5.31% Full, Aider-PB Jan 2025 Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard Not tested since Apr 2025
Claude 3.5 Sonnet (October 2024) 14.1%

Source: Official SWE-PolyBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Claude 3.5 Sonnet (October 2024) 0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Claude 3.5 Sonnet (October 2024)

Source: Official SWE-PolyBench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471