Ramp SWE-Bench v2
Version 2
Agents solve 500 engineering tasks from Ramp's production backend. A panel of model judges compares each patch with the code engineers merged.
Ramp runs every model in a shared Bash harness (mini-SWE-agent, v2.0) and the labs' own coding agents in their native harnesses (v2.1) on the same tasks.
- Made by
- Ramp Labs
- Released
- November 2026
- Kind
- Fixing real repositories
- Level
- Expert
- Scored
- Graded by a model
- Answers
- Code
- Tasks
- Kept private
- Measures
- Capability
- Language
- English
- Interaction
- Agentic
- Environment
- Terminal
- Items from
- Real work and use
- Refresh
- New versions replace it
- Run by
- A host
- Based on
- Ramp SWE-Bench v1
- In views
- Agents
- Strongest model tested
- GPT-6.2 Astra #1 on the capability index
- Last model added
- November 2026
- Scores refreshed
- November 20th, 2026
GPT-6.2 Astra leads with 76.4% GPT-6.2 Astra leads with 76.4% GPT-6.2 Astra leads with 75% in mini-SWE-agent
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran on its own with several settings, we keep its best. Every run is in the next chart. Agents other companies built around a model get a pale shade of that model's lab color. Gray marks blends and entries not in our model list, such as fine-tunes and older models. None of these count toward leaders. Agents built around a model, such as OpenCode + Gemini 4 Carbon, are their own entries. We don't count them as the model's score. This view shows only models we track, each at its best. This view keeps only runs in mini-SWE-agent, a harness no model’s maker built. A maker’s own agent, such as Claude Code with a Claude model, is left out.
- 76.4 GPT-6.2 Astra
- 75.4 OpenCode + Gemini 4 Carbon
- 74.0
Gemini 4 Carbon
- 73.4 Claude Fable 5.2
- 73.4 GPT-6.2 Sol
- 71.6 OpenCode + Gemini 4 Hydrogen
- 71.2 Claude Opus 5.5
- 70.8
Gemini 4 Hydrogen
- 70.4 GPT-6.1 Terra
- 64.4 Grok 4.7
- 63.8 Claude Fable 5.1
- 63.4 GPT-6 Astra
- 63.2 Claude Sonnet 5.5
- 60.8 DeepSeek Harness + DeepSeek V4.1 Flash 1017
- 59.6 GPT-6.1 Sol
- 58.6
Muse Wonder
- 57.4 Claude Haiku 5.5
- 54.8
Gemini 4 Argon
- 54.4
Muse Ember
- 53.4
DeepSeek V4.1 Flash 1017
- 52.4 GPT-6 Luna
- 47.0
Gemini 3.8 Flash
- 45.4 Kimi K3.1 Pro
- 34.0 Beam 1.0
- 12.0
Nemotron 3.5 Super VL
- 1 GPT-6.2 Astra high 76.4%
- 2 OpenCode + Gemini 4 Carbon high 75.4%
- 3
Gemini 4 Carbon high 74%
- 4 Claude Fable 5.2 high 73.4%
- 5 GPT-6.2 Sol high 73.4%
- 6 OpenCode + Gemini 4 Hydrogen high 71.6%
- 7 Claude Opus 5.5 high 71.2%
- 8
Gemini 4 Hydrogen high 70.8%
- 9 GPT-6.1 Terra high 70.4%
- 10 Grok 4.7 high 64.4%
- 11 Claude Fable 5.1 high 63.8%
- 12 GPT-6 Astra high 63.4%
- 13 Claude Sonnet 5.5 high 63.2%
- 14 DeepSeek Harness + DeepSeek V4.1 Flash 1017 high 60.8%
- 15 GPT-6.1 Sol high 59.6%
- 16
Muse Wonder high 58.6%
- 17 Claude Haiku 5.5 high 57.4%
- 18
Gemini 4 Argon high 54.8%
- 19
Muse Ember high 54.4%
- 20
DeepSeek V4.1 Flash 1017 high 53.4%
- 21 GPT-6 Luna high 52.4%
- 22
Gemini 3.8 Flash high 47%
- 23 Kimi K3.1 Pro high 45.4%
- 24 Beam 1.0 high 34%
- 25
Nemotron 3.5 Super VL high 12%
Source: Official Ramp SWE-Bench v2 leaderboard
- 76.4 GPT-6.2 Astra
- 74.0
Gemini 4 Carbon
- 73.4 Claude Fable 5.2
- 73.4 GPT-6.2 Sol
- 71.2 Claude Opus 5.5
- 70.8
Gemini 4 Hydrogen
- 70.4 GPT-6.1 Terra
- 64.4 Grok 4.7
- 63.8 Claude Fable 5.1
- 63.4 GPT-6 Astra
- 63.2 Claude Sonnet 5.5
- 59.6 GPT-6.1 Sol
- 58.6
Muse Wonder
- 57.4 Claude Haiku 5.5
- 54.8
Gemini 4 Argon
- 54.4
Muse Ember
- 53.4
DeepSeek V4.1 Flash 1017
- 52.4 GPT-6 Luna
- 47.0
Gemini 3.8 Flash
- 45.4 Kimi K3.1 Pro
- 34.0 Beam 1.0
- 12.0
Nemotron 3.5 Super VL
- 1 GPT-6.2 Astra high 76.4%
- 2
Gemini 4 Carbon high 74%
- 3 Claude Fable 5.2 high 73.4%
- 4 GPT-6.2 Sol high 73.4%
- 5 Claude Opus 5.5 high 71.2%
- 6
Gemini 4 Hydrogen high 70.8%
- 7 GPT-6.1 Terra high 70.4%
- 8 Grok 4.7 high 64.4%
- 9 Claude Fable 5.1 high 63.8%
- 10 GPT-6 Astra high 63.4%
- 11 Claude Sonnet 5.5 high 63.2%
- 12 GPT-6.1 Sol high 59.6%
- 13
Muse Wonder high 58.6%
- 14 Claude Haiku 5.5 high 57.4%
- 15
Gemini 4 Argon high 54.8%
- 16
Muse Ember high 54.4%
- 17
DeepSeek V4.1 Flash 1017 high 53.4%
- 18 GPT-6 Luna high 52.4%
- 19
Gemini 3.8 Flash high 47%
- 20 Kimi K3.1 Pro high 45.4%
- 21 Beam 1.0 high 34%
- 22
Nemotron 3.5 Super VL high 12%
Source: Official Ramp SWE-Bench v2 leaderboard
- 75.0 GPT-6.2 Astra
- 74.0
Gemini 4 Carbon
- 73.0 Claude Fable 5.2
- 72.4 GPT-6.2 Sol
- 71.2 Claude Opus 5.5
- 70.8
Gemini 4 Hydrogen
- 63.8 Claude Fable 5.1
- 60.6 GPT-6.1 Terra
- 57.8
Muse Wonder
- 57.6 Claude Sonnet 5.5
- 55.4 Grok 4.7
- 54.8
Gemini 4 Argon
- 54.6 GPT-6 Astra
- 54.4
Muse Ember
- 53.4
DeepSeek V4.1 Flash 1017
- 52.8 Claude Haiku 5.5
- 52.0 GPT-6.1 Sol
- 47.0
Gemini 3.8 Flash
- 45.4 Kimi K3.1 Pro
- 42.6 GPT-6 Luna
- 34.0 Beam 1.0
- 12.0
Nemotron 3.5 Super VL
- 1 GPT-6.2 Astra high 75%
- 2
Gemini 4 Carbon high 74%
- 3 Claude Fable 5.2 high 73%
- 4 GPT-6.2 Sol high 72.4%
- 5 Claude Opus 5.5 high 71.2%
- 6
Gemini 4 Hydrogen high 70.8%
- 7 Claude Fable 5.1 high 63.8%
- 8 GPT-6.1 Terra high 60.6%
- 9
Muse Wonder high 57.8%
- 10 Claude Sonnet 5.5 high 57.6%
- 11 Grok 4.7 high 55.4%
- 12
Gemini 4 Argon high 54.8%
- 13 GPT-6 Astra high 54.6%
- 14
Muse Ember high 54.4%
- 15
DeepSeek V4.1 Flash 1017 high 53.4%
- 16 Claude Haiku 5.5 high 52.8%
- 17 GPT-6.1 Sol high 52%
- 18
Gemini 3.8 Flash high 47%
- 19 Kimi K3.1 Pro high 45.4%
- 20 GPT-6 Luna high 42.6%
- 21 Beam 1.0 high 34%
- 22
Nemotron 3.5 Super VL high 12%
Source: Official Ramp SWE-Bench v2 leaderboard
| # | Entry | Score ± | Setting | Released | Run on | Source |
|---|---|---|---|---|---|---|
| 1 | GPT-6.2 Astra | 76.4% 72.5 to 79.9 | high, Codex | Nov 2026 | Nov 2026 | Official leaderboard |
| 2 | OpenCode + Gemini 4 Carbon | 75.4% 71.4 to 79 | high | Oct 2026 | Nov 2026 | Official leaderboard |
| 3 | | 74% 70 to 77.7 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 4 | Claude Fable 5.2 | 73.4% 69.4 to 77.1 | high, Claude Code | Nov 2026 | Nov 2026 | Official leaderboard |
| 5 | GPT-6.2 Sol | 73.4% 69.4 to 77.1 | high, Codex | Nov 2026 | Nov 2026 | Official leaderboard |
| 6 | OpenCode + Gemini 4 Hydrogen | 71.6% 67.5 to 75.4 | high | Nov 2026 | Nov 2026 | Official leaderboard |
| 7 | Claude Opus 5.5 | 71.2% 67.1 to 75 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 8 | | 70.8% 66.7 to 74.6 | high, mini-SWE-agent | Nov 2026 | Nov 2026 | Official leaderboard |
| 9 | GPT-6.1 Terra | 70.4% 66.3 to 74.2 | high, Codex | Oct 2026 | Nov 2026 | Official leaderboard |
| 10 | Grok 4.7 | 64.4% 60.1 to 68.5 | high, Grok Build | Sep 2026 | Nov 2026 | Official leaderboard |
| 11 | Claude Fable 5.1 | 63.8% 59.5 to 67.9 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 12 | GPT-6 Astra | 63.4% 59.1 to 67.5 | high, Codex | Sep 2026 | Nov 2026 | Official leaderboard |
| 13 | Claude Sonnet 5.5 | 63.2% 58.9 to 67.3 | high, Claude Code | Sep 2026 | Nov 2026 | Official leaderboard |
| 14 | DeepSeek Harness + DeepSeek V4.1 Flash 1017 | 60.8% ±4.3 | high | Oct 2026 | Nov 2026 | Official leaderboard |
| 15 | GPT-6.1 Sol | 59.6% 55.2 to 63.8 | high, Codex | Sep 2026 | Nov 2026 | Official leaderboard |
| 16 | | 58.6% 54.2 to 62.8 | high, Muse Code | Oct 2026 | Nov 2026 | Official leaderboard |
| 17 | Claude Haiku 5.5 | 57.4% ±4.4 | high, Claude Code | Oct 2026 | Nov 2026 | Official leaderboard |
| 18 | | 54.8% ±4.4 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 19 | | 54.4% ±4.4 | high, mini-SWE-agent | Nov 2026 | Nov 2026 | Official leaderboard |
| 20 | | 53.4% ±4.4 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 21 | GPT-6 Luna | 52.4% ±4.4 | high, Codex | Sep 2026 | Nov 2026 | Official leaderboard |
| 22 | | 47% ±4.3 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 23 | Kimi K3.1 Pro | 45.4% ±4.3 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 24 | Beam 1.0 | 34% 30 to 38.3 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 25 | | 12% 9.4 to 15.1 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| # | Model | Score ± | Setting | Released | Run on | Source |
|---|---|---|---|---|---|---|
| 1 | GPT-6.2 Astra | 76.4% 72.5 to 79.9 | high, Codex | Nov 2026 | Nov 2026 | Official leaderboard |
| 2 | | 74% 70 to 77.7 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 3 | Claude Fable 5.2 | 73.4% 69.4 to 77.1 | high, Claude Code | Nov 2026 | Nov 2026 | Official leaderboard |
| 4 | GPT-6.2 Sol | 73.4% 69.4 to 77.1 | high, Codex | Nov 2026 | Nov 2026 | Official leaderboard |
| 5 | Claude Opus 5.5 | 71.2% 67.1 to 75 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 6 | | 70.8% 66.7 to 74.6 | high, mini-SWE-agent | Nov 2026 | Nov 2026 | Official leaderboard |
| 7 | GPT-6.1 Terra | 70.4% 66.3 to 74.2 | high, Codex | Oct 2026 | Nov 2026 | Official leaderboard |
| 8 | Grok 4.7 | 64.4% 60.1 to 68.5 | high, Grok Build | Sep 2026 | Nov 2026 | Official leaderboard |
| 9 | Claude Fable 5.1 | 63.8% 59.5 to 67.9 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 10 | GPT-6 Astra | 63.4% 59.1 to 67.5 | high, Codex | Sep 2026 | Nov 2026 | Official leaderboard |
| 11 | Claude Sonnet 5.5 | 63.2% 58.9 to 67.3 | high, Claude Code | Sep 2026 | Nov 2026 | Official leaderboard |
| 12 | GPT-6.1 Sol | 59.6% 55.2 to 63.8 | high, Codex | Sep 2026 | Nov 2026 | Official leaderboard |
| 13 | | 58.6% 54.2 to 62.8 | high, Muse Code | Oct 2026 | Nov 2026 | Official leaderboard |
| 14 | Claude Haiku 5.5 | 57.4% ±4.4 | high, Claude Code | Oct 2026 | Nov 2026 | Official leaderboard |
| 15 | | 54.8% ±4.4 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 16 | | 54.4% ±4.4 | high, mini-SWE-agent | Nov 2026 | Nov 2026 | Official leaderboard |
| 17 | | 53.4% ±4.4 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 18 | GPT-6 Luna | 52.4% ±4.4 | high, Codex | Sep 2026 | Nov 2026 | Official leaderboard |
| 19 | | 47% ±4.3 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 20 | Kimi K3.1 Pro | 45.4% ±4.3 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 21 | Beam 1.0 | 34% 30 to 38.3 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 22 | | 12% 9.4 to 15.1 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| # | Model | Score ± | Setting | Released | Run on | Source |
|---|---|---|---|---|---|---|
| 1 | GPT-6.2 Astra | 75% 71 to 78.6 | high, mini-SWE-agent | Nov 2026 | Nov 2026 | Official leaderboard |
| 2 | | 74% 70 to 77.7 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 3 | Claude Fable 5.2 | 73% 68.9 to 76.7 | high, mini-SWE-agent | Nov 2026 | Nov 2026 | Official leaderboard |
| 4 | GPT-6.2 Sol | 72.4% 68.3 to 76.1 | high, mini-SWE-agent | Nov 2026 | Nov 2026 | Official leaderboard |
| 5 | Claude Opus 5.5 | 71.2% 67.1 to 75 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 6 | | 70.8% 66.7 to 74.6 | high, mini-SWE-agent | Nov 2026 | Nov 2026 | Official leaderboard |
| 7 | Claude Fable 5.1 | 63.8% 59.5 to 67.9 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 8 | GPT-6.1 Terra | 60.6% ±4.3 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 9 | | 57.8% ±4.4 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 10 | Claude Sonnet 5.5 | 57.6% ±4.3 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 11 | Grok 4.7 | 55.4% ±4.4 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 12 | | 54.8% ±4.4 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 13 | GPT-6 Astra | 54.6% ±4.3 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 14 | | 54.4% ±4.4 | high, mini-SWE-agent | Nov 2026 | Nov 2026 | Official leaderboard |
| 15 | | 53.4% ±4.4 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 16 | Claude Haiku 5.5 | 52.8% ±4.4 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 17 | GPT-6.1 Sol | 52% ±4.3 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 18 | | 47% ±4.3 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 19 | Kimi K3.1 Pro | 45.4% ±4.3 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 20 | GPT-6 Luna | 42.6% ±4.4 | high, mini-SWE-agent | Sep 2026 | Nov 2026 | Official leaderboard |
| 21 | Beam 1.0 | 34% 30 to 38.3 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
| 22 | | 12% 9.4 to 15.1 | high, mini-SWE-agent | Oct 2026 | Nov 2026 | Official leaderboard |
Score versus spend
The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.
| Model | Setting | Resolved | Mean cost per task |
|---|---|---|---|
| GPT-6.2 Astra | high, Codex | 76.4% | $3.31 |
| OpenCode + Gemini 4 Carbon | high, OpenCode | 75.4% | $4.25 |
| GPT-6.2 Astra | high, mini-SWE-agent | 75% | $1.03 |
| high, mini-SWE-agent | 74% | $5.33 | |
| GPT-6.2 Sol | high, Codex | 73.4% | $1.05 |
| Claude Fable 5.2 | high, Claude Code | 73.4% | $3.79 |
| Claude Fable 5.2 | high, mini-SWE-agent | 73% | $6.45 |
| GPT-6.2 Sol | high, mini-SWE-agent | 72.4% | $0.28 |
| OpenCode + Gemini 4 Hydrogen | high, OpenCode | 71.6% | $2.11 |
| Claude Opus 5.5 | high, mini-SWE-agent | 71.2% | $0.99 |
| high, mini-SWE-agent | 70.8% | $1.75 | |
| GPT-6.1 Terra | high, Codex | 70.4% | $0.46 |
| Claude Opus 5.5 | high, Claude Code | 67.8% | $1.53 |
| Grok 4.7 | high, Grok Build | 64.4% | $4.32 |
| Claude Fable 5.1 | high, mini-SWE-agent | 63.8% | $2.43 |
| GPT-6 Astra | high, Codex | 63.4% | $5.22 |
| Claude Sonnet 5.5 | high, Claude Code | 63.2% | $0.47 |
| DeepSeek Harness + DeepSeek V4.1 Flash 1017 | high, DeepSeek Harness | 60.8% | $0.22 |
| GPT-6.1 Terra | high, mini-SWE-agent | 60.6% | $0.13 |
| GPT-6.1 Sol | high, Codex | 59.6% | $0.74 |
| high, Muse Code | 58.6% | $4.59 | |
| high, mini-SWE-agent | 57.8% | $2.65 | |
| Claude Sonnet 5.5 | high, mini-SWE-agent | 57.6% | $0.24 |
| Claude Haiku 5.5 | high, Claude Code | 57.4% | $0.28 |
| Grok 4.7 | high, mini-SWE-agent | 55.4% | $3.98 |
| high, mini-SWE-agent | 54.8% | $6.51 | |
| GPT-6 Astra | high, mini-SWE-agent | 54.6% | $1.69 |
| high, mini-SWE-agent | 54.4% | $0.06 | |
| high, mini-SWE-agent | 53.4% | $0.16 | |
| Claude Haiku 5.5 | high, mini-SWE-agent | 52.8% | $0.12 |
| GPT-6 Luna | high, Codex | 52.4% | $0.07 |
| GPT-6.1 Sol | high, mini-SWE-agent | 52% | $0.27 |
| high, Muse Code | 51.8% | $0.11 | |
| high, mini-SWE-agent | 47% | $2.62 | |
| Kimi K3.1 Pro | high, mini-SWE-agent | 45.4% | $0.7 |
| GPT-6 Luna | high, mini-SWE-agent | 42.6% | $0.02 |
| Beam 1.0 | high, mini-SWE-agent | 34% | $0.08 |
| high, mini-SWE-agent | 12% | $0.11 |
Score versus spend
The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.
| Model | Setting | Resolved | Mean cost per task |
|---|---|---|---|
| GPT-6.2 Astra | high, Codex | 76.4% | $3.31 |
| GPT-6.2 Astra | high, mini-SWE-agent | 75% | $1.03 |
| high, mini-SWE-agent | 74% | $5.33 | |
| GPT-6.2 Sol | high, Codex | 73.4% | $1.05 |
| Claude Fable 5.2 | high, Claude Code | 73.4% | $3.79 |
| Claude Fable 5.2 | high, mini-SWE-agent | 73% | $6.45 |
| GPT-6.2 Sol | high, mini-SWE-agent | 72.4% | $0.28 |
| Claude Opus 5.5 | high, mini-SWE-agent | 71.2% | $0.99 |
| high, mini-SWE-agent | 70.8% | $1.75 | |
| GPT-6.1 Terra | high, Codex | 70.4% | $0.46 |
| Claude Opus 5.5 | high, Claude Code | 67.8% | $1.53 |
| Grok 4.7 | high, Grok Build | 64.4% | $4.32 |
| Claude Fable 5.1 | high, mini-SWE-agent | 63.8% | $2.43 |
| GPT-6 Astra | high, Codex | 63.4% | $5.22 |
| Claude Sonnet 5.5 | high, Claude Code | 63.2% | $0.47 |
| GPT-6.1 Terra | high, mini-SWE-agent | 60.6% | $0.13 |
| GPT-6.1 Sol | high, Codex | 59.6% | $0.74 |
| high, Muse Code | 58.6% | $4.59 | |
| high, mini-SWE-agent | 57.8% | $2.65 | |
| Claude Sonnet 5.5 | high, mini-SWE-agent | 57.6% | $0.24 |
| Claude Haiku 5.5 | high, Claude Code | 57.4% | $0.28 |
| Grok 4.7 | high, mini-SWE-agent | 55.4% | $3.98 |
| high, mini-SWE-agent | 54.8% | $6.51 | |
| GPT-6 Astra | high, mini-SWE-agent | 54.6% | $1.69 |
| high, mini-SWE-agent | 54.4% | $0.06 | |
| high, mini-SWE-agent | 53.4% | $0.16 | |
| Claude Haiku 5.5 | high, mini-SWE-agent | 52.8% | $0.12 |
| GPT-6 Luna | high, Codex | 52.4% | $0.07 |
| GPT-6.1 Sol | high, mini-SWE-agent | 52% | $0.27 |
| high, Muse Code | 51.8% | $0.11 | |
| high, mini-SWE-agent | 47% | $2.62 | |
| Kimi K3.1 Pro | high, mini-SWE-agent | 45.4% | $0.7 |
| GPT-6 Luna | high, mini-SWE-agent | 42.6% | $0.02 |
| Beam 1.0 | high, mini-SWE-agent | 34% | $0.08 |
| high, mini-SWE-agent | 12% | $0.11 |
Score versus spend
The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.
| Model | Setting | Resolved | Mean cost per task |
|---|---|---|---|
| GPT-6.2 Astra | high, mini-SWE-agent | 75% | $1.03 |
| high, mini-SWE-agent | 74% | $5.33 | |
| Claude Fable 5.2 | high, mini-SWE-agent | 73% | $6.45 |
| GPT-6.2 Sol | high, mini-SWE-agent | 72.4% | $0.28 |
| Claude Opus 5.5 | high, mini-SWE-agent | 71.2% | $0.99 |
| high, mini-SWE-agent | 70.8% | $1.75 | |
| Claude Fable 5.1 | high, mini-SWE-agent | 63.8% | $2.43 |
| GPT-6.1 Terra | high, mini-SWE-agent | 60.6% | $0.13 |
| high, mini-SWE-agent | 57.8% | $2.65 | |
| Claude Sonnet 5.5 | high, mini-SWE-agent | 57.6% | $0.24 |
| Grok 4.7 | high, mini-SWE-agent | 55.4% | $3.98 |
| high, mini-SWE-agent | 54.8% | $6.51 | |
| GPT-6 Astra | high, mini-SWE-agent | 54.6% | $1.69 |
| high, mini-SWE-agent | 54.4% | $0.06 | |
| high, mini-SWE-agent | 53.4% | $0.16 | |
| Claude Haiku 5.5 | high, mini-SWE-agent | 52.8% | $0.12 |
| GPT-6.1 Sol | high, mini-SWE-agent | 52% | $0.27 |
| high, mini-SWE-agent | 47% | $2.62 | |
| Kimi K3.1 Pro | high, mini-SWE-agent | 45.4% | $0.7 |
| GPT-6 Luna | high, mini-SWE-agent | 42.6% | $0.02 |
| Beam 1.0 | high, mini-SWE-agent | 34% | $0.08 |
| high, mini-SWE-agent | 12% | $0.11 |
The best score, 76.4%, costs $3.31 a task
Runs on the cost frontier score higher than any run that's cheaper. GPT-6.2 Sol at high comes within 4 points of the best for $0.28 a task, 12 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Source: Official Ramp SWE-Bench v2 leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| GPT-6.2 Astra | high | Codex | 76.4% | $3.31 | 9,593 |
| OpenCode + Gemini 4 Carbon | high | OpenCode | 75.4% | $4.25 | 72,348 |
| GPT-6.2 Astra | high | mini-SWE-agent | 75% | $1.03 | 7,097 |
| high | mini-SWE-agent | 74% | $5.33 | 73,704 | |
| Claude Fable 5.2 | high | Claude Code | 73.4% | $3.79 | 30,618 |
| GPT-6.2 Sol | high | Codex | 73.4% | $1.05 | 10,919 |
| Claude Fable 5.2 | high | mini-SWE-agent | 73% | $6.45 | 63,016 |
| GPT-6.2 Sol | high | mini-SWE-agent | 72.4% | $0.28 | 8,066 |
| OpenCode + Gemini 4 Hydrogen | high | OpenCode | 71.6% | $2.11 | 48,264 |
| Claude Opus 5.5 | high | mini-SWE-agent | 71.2% | $0.99 | 13,707 |
| high | mini-SWE-agent | 70.8% | $1.75 | 47,277 | |
| GPT-6.1 Terra | high | Codex | 70.4% | $0.46 | 16,043 |
| Claude Opus 5.5 | high | Claude Code | 67.8% | $1.53 | 16,447 |
| Grok 4.7 | high | Grok Build | 64.4% | $4.32 | 50,767 |
| Claude Fable 5.1 | high | mini-SWE-agent | 63.8% | $2.43 | 18,100 |
| GPT-6 Astra | high | Codex | 63.4% | $5.22 | 14,662 |
| Claude Sonnet 5.5 | high | Claude Code | 63.2% | $0.47 | 10,602 |
| DeepSeek Harness + DeepSeek V4.1 Flash 1017 | high | DeepSeek Harness | 60.8% | $0.22 | 40,170 |
| GPT-6.1 Terra | high | mini-SWE-agent | 60.6% | $0.13 | 7,661 |
| GPT-6.1 Sol | high | Codex | 59.6% | $0.74 | 16,113 |
| high | Muse Code | 58.6% | $4.59 | 38,797 | |
| high | mini-SWE-agent | 57.8% | $2.65 | 26,331 | |
| Claude Sonnet 5.5 | high | mini-SWE-agent | 57.6% | $0.24 | 6,798 |
| Claude Haiku 5.5 | high | Claude Code | 57.4% | $0.28 | 35,854 |
| Grok 4.7 | high | mini-SWE-agent | 55.4% | $3.98 | 17,674 |
| high | mini-SWE-agent | 54.8% | $6.51 | 92,379 | |
| GPT-6 Astra | high | mini-SWE-agent | 54.6% | $1.69 | 9,030 |
| high | mini-SWE-agent | 54.4% | $0.06 | 28,963 | |
| high | mini-SWE-agent | 53.4% | $0.16 | 22,857 | |
| Claude Haiku 5.5 | high | mini-SWE-agent | 52.8% | $0.12 | 26,909 |
| GPT-6 Luna | high | Codex | 52.4% | $0.07 | 28,442 |
| GPT-6.1 Sol | high | mini-SWE-agent | 52% | $0.27 | 9,111 |
| high | Muse Code | 51.8% | $0.11 | 41,751 | |
| high | mini-SWE-agent | 47% | $2.62 | 80,934 | |
| Kimi K3.1 Pro | high | mini-SWE-agent | 45.4% | $0.70 | 11,519 |
| GPT-6 Luna | high | mini-SWE-agent | 42.6% | $0.02 | 11,048 |
| Beam 1.0 | high | mini-SWE-agent | 34% | $0.08 | 8,639 |
| high | mini-SWE-agent | 12% | $0.11 | 11,766 |
The best score, 76.4%, costs $3.31 a task
Runs on the cost frontier score higher than any run that's cheaper. GPT-6.2 Sol at high comes within 4 points of the best for $0.28 a task, 12 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Source: Official Ramp SWE-Bench v2 leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| GPT-6.2 Astra | high | Codex | 76.4% | $3.31 | 9,593 |
| GPT-6.2 Astra | high | mini-SWE-agent | 75% | $1.03 | 7,097 |
| high | mini-SWE-agent | 74% | $5.33 | 73,704 | |
| Claude Fable 5.2 | high | Claude Code | 73.4% | $3.79 | 30,618 |
| GPT-6.2 Sol | high | Codex | 73.4% | $1.05 | 10,919 |
| Claude Fable 5.2 | high | mini-SWE-agent | 73% | $6.45 | 63,016 |
| GPT-6.2 Sol | high | mini-SWE-agent | 72.4% | $0.28 | 8,066 |
| Claude Opus 5.5 | high | mini-SWE-agent | 71.2% | $0.99 | 13,707 |
| high | mini-SWE-agent | 70.8% | $1.75 | 47,277 | |
| GPT-6.1 Terra | high | Codex | 70.4% | $0.46 | 16,043 |
| Claude Opus 5.5 | high | Claude Code | 67.8% | $1.53 | 16,447 |
| Grok 4.7 | high | Grok Build | 64.4% | $4.32 | 50,767 |
| Claude Fable 5.1 | high | mini-SWE-agent | 63.8% | $2.43 | 18,100 |
| GPT-6 Astra | high | Codex | 63.4% | $5.22 | 14,662 |
| Claude Sonnet 5.5 | high | Claude Code | 63.2% | $0.47 | 10,602 |
| GPT-6.1 Terra | high | mini-SWE-agent | 60.6% | $0.13 | 7,661 |
| GPT-6.1 Sol | high | Codex | 59.6% | $0.74 | 16,113 |
| high | Muse Code | 58.6% | $4.59 | 38,797 | |
| high | mini-SWE-agent | 57.8% | $2.65 | 26,331 | |
| Claude Sonnet 5.5 | high | mini-SWE-agent | 57.6% | $0.24 | 6,798 |
| Claude Haiku 5.5 | high | Claude Code | 57.4% | $0.28 | 35,854 |
| Grok 4.7 | high | mini-SWE-agent | 55.4% | $3.98 | 17,674 |
| high | mini-SWE-agent | 54.8% | $6.51 | 92,379 | |
| GPT-6 Astra | high | mini-SWE-agent | 54.6% | $1.69 | 9,030 |
| high | mini-SWE-agent | 54.4% | $0.06 | 28,963 | |
| high | mini-SWE-agent | 53.4% | $0.16 | 22,857 | |
| Claude Haiku 5.5 | high | mini-SWE-agent | 52.8% | $0.12 | 26,909 |
| GPT-6 Luna | high | Codex | 52.4% | $0.07 | 28,442 |
| GPT-6.1 Sol | high | mini-SWE-agent | 52% | $0.27 | 9,111 |
| high | Muse Code | 51.8% | $0.11 | 41,751 | |
| high | mini-SWE-agent | 47% | $2.62 | 80,934 | |
| Kimi K3.1 Pro | high | mini-SWE-agent | 45.4% | $0.70 | 11,519 |
| GPT-6 Luna | high | mini-SWE-agent | 42.6% | $0.02 | 11,048 |
| Beam 1.0 | high | mini-SWE-agent | 34% | $0.08 | 8,639 |
| high | mini-SWE-agent | 12% | $0.11 | 11,766 |
The best score, 75%, costs $1.03 a task
Runs on the cost frontier score higher than any run that's cheaper. GPT-6.2 Sol at high comes within 2.6 points of the best for $0.28 a task, 4 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Source: Official Ramp SWE-Bench v2 leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| GPT-6.2 Astra | high | mini-SWE-agent | 75% | $1.03 | 7,097 |
| high | mini-SWE-agent | 74% | $5.33 | 73,704 | |
| Claude Fable 5.2 | high | mini-SWE-agent | 73% | $6.45 | 63,016 |
| GPT-6.2 Sol | high | mini-SWE-agent | 72.4% | $0.28 | 8,066 |
| Claude Opus 5.5 | high | mini-SWE-agent | 71.2% | $0.99 | 13,707 |
| high | mini-SWE-agent | 70.8% | $1.75 | 47,277 | |
| Claude Fable 5.1 | high | mini-SWE-agent | 63.8% | $2.43 | 18,100 |
| GPT-6.1 Terra | high | mini-SWE-agent | 60.6% | $0.13 | 7,661 |
| high | mini-SWE-agent | 57.8% | $2.65 | 26,331 | |
| Claude Sonnet 5.5 | high | mini-SWE-agent | 57.6% | $0.24 | 6,798 |
| Grok 4.7 | high | mini-SWE-agent | 55.4% | $3.98 | 17,674 |
| high | mini-SWE-agent | 54.8% | $6.51 | 92,379 | |
| GPT-6 Astra | high | mini-SWE-agent | 54.6% | $1.69 | 9,030 |
| high | mini-SWE-agent | 54.4% | $0.06 | 28,963 | |
| high | mini-SWE-agent | 53.4% | $0.16 | 22,857 | |
| Claude Haiku 5.5 | high | mini-SWE-agent | 52.8% | $0.12 | 26,909 |
| GPT-6.1 Sol | high | mini-SWE-agent | 52% | $0.27 | 9,111 |
| high | mini-SWE-agent | 47% | $2.62 | 80,934 | |
| Kimi K3.1 Pro | high | mini-SWE-agent | 45.4% | $0.70 | 11,519 |
| GPT-6 Luna | high | mini-SWE-agent | 42.6% | $0.02 | 11,048 |
| Beam 1.0 | high | mini-SWE-agent | 34% | $0.08 | 8,639 |
| high | mini-SWE-agent | 12% | $0.11 | 11,766 |
2 models have held number one on the official leaderboard since its release in November 2026
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official Ramp SWE-Bench v2 leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official Ramp SWE-Bench v2 leaderboard
Source: Official Ramp SWE-Bench v2 leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
One logged change that moves scores
Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.
- Harness changed November 2026
Ramp added runs in each lab's own coding agent (v2.1) beside the shared Bash harness (v2.0), on the same 500 tasks. Source
Versions
We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.
Source: Official leaderboards
More in Coding
- SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
- LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests.
DeepSeek-V4-Pro 93.5%
- SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
- DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
- SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases.
Seed2.1 Pro 59.8%
- Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance.
DeepSeek V4.1 Flash 3,471
Data audit
All 11 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access The tasks are private.
- Pass Confidence Checked against a primary source.
Scores 5 pass
- Pass Independent runs Scores from the official leaderboard (25 entries).
- Pass Models scored 22 models we track, plus 3 other entries on its leaderboard.
- Pass Newest score The newest is from November 2026, for GPT-6.2 Astra.
- Pass Frontier models 10 of the top 10 have a score, led by GPT-6.2 Astra (#1).
- Pass Headroom The best is 76.4%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on labs.ramp.com: 38 rows for models we track at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.