We track 2 benchmarks and 1 arena that SWE-bench team made or helped make.
Top five in each of its 1 arena with three or more rated models, the most rated first.
Models build and improve their own codebases over many rounds, then compete in arenas such as Halite, poker and BattleSnake. Results give Elo ratings.
Official leaderboard
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| SWE-bench Multilingual 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. | Fixing real repositories | Apr 2025 | 96.6% | In use |
| SWE-bench Multimodal SWE-bench tasks whose issues include screenshots or design mockups from visual JavaScript repos. Patch graded by hidden tests. | Fixing real repositories | Oct 2024 | 74.1% | In use |