RoadmapBench

Claude Opus 4.7 39.1% leads since May 2026

Step 5 Preview 54.3% leads since September 2026

In use

Official leaderboard Include claimed scores

Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests.

The median task changes about 3,700 lines across 51 files. The paper reports a Completion Score for partial progress next to the resolve rate.

Made by: UniPat AI
Released: May 2026
Kind: Building whole projects
Level: Expert
Scored: Checked automatically
Answers: Agent task
Tasks: All public

Strongest model tested: GPT-5.4 #27 on the capability index
Last in a launch post: September 2026
Last model added: September 2026
Scores refreshed: November 12th, 2026

In the authors’ words

Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem.

Abstract of RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (arXiv:2605.15846, CC0)

Entries on its own leaderboard, 11 models we track

Launch post lists it, from 1 lab

Claude Opus 4.7 leads with 39.1% among models released by April 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

Step 5 Preview leads with 54.3%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

Bars Table

The newest model with a score here came out in April 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

1 Claude Opus 4.7 single trial per model 39.1%
2 Claude Opus 4.6 single trial per model 32.2%
3 GPT-5.4 single trial per model 29.6%
4 Gemini 3.1 Pro single trial per model 20.9%
5 GLM-5.1 single trial per model 18.3%
6 DeepSeek-V4-Pro single trial per model 18.3%
7 Kimi K2.6 single trial per model 14.8%
8 MiMo-V2.5-Pro single trial per model 13.9%
9 Qwen 3.6 Plus single trial per model 12.2%
10 Kimi K2.5 single trial per model 11.3%
11 MiniMax-M2.7 single trial per model 10.4%
12 Qwen3.5-397B single trial per model 9.60%

Source: Official RoadmapBench leaderboard

1 Step 5 Preview 54.3%
2 Claude Opus 4.7 single trial per model 39.1%
3 Claude Opus 4.6 single trial per model 32.2%
4 GPT-5.4 single trial per model 29.6%
5 Gemini 3.1 Pro single trial per model 20.9%
6 GLM-5.1 single trial per model 18.3%
7 DeepSeek-V4-Pro single trial per model 18.3%
8 Kimi K2.6 single trial per model 14.8%
9 MiMo-V2.5-Pro single trial per model 13.9%
10 Qwen 3.6 Plus single trial per model 12.2%
11 Kimi K2.5 single trial per model 11.3%
12 MiniMax-M2.7 single trial per model 10.4%

Sources: Labs’ model cards and official RoadmapBench leaderboard

#	Entry	Score	Setting	Date	Source
1	Claude Opus 4.7	39.1%	single trial per model, OpenHands	Apr 2026	Official leaderboard
2	Claude Opus 4.6	32.2%	single trial per model, OpenHands	Feb 2026	Official leaderboard
3	GPT-5.4	29.6%	single trial per model, OpenHands	Mar 2026	Official leaderboard
4	Gemini 3.1 Pro	20.9%	single trial per model, OpenHands	Feb 2026	Official leaderboard
5	GLM-5.1	18.3%	single trial per model, OpenHands	Apr 2026	Official leaderboard
6	DeepSeek-V4-Pro	18.3%	single trial per model, OpenHands	Apr 2026	Official leaderboard
7	Kimi K2.6	14.8%	single trial per model, OpenHands	Apr 2026	Official leaderboard
8	MiMo-V2.5-Pro	13.9%	single trial per model, OpenHands	Apr 2026	Official leaderboard
9	Qwen 3.6 Plus	12.2%	single trial per model, OpenHands	Mar 2026	Official leaderboard
10	Kimi K2.5	11.3%	single trial per model, OpenHands	Jan 2026	Official leaderboard
11	MiniMax-M2.7	10.4%	single trial per model, OpenHands	Mar 2026	Official leaderboard
12	Qwen3.5-397B	9.60%	single trial per model, OpenHands		Official leaderboard

#	Model	Score	Setting	Date	Source
1	Step 5 Preview	54.3%		Sep 2026	Lab model card
2	Claude Opus 4.7	39.1%	single trial per model, OpenHands	Apr 2026	Official leaderboard
3	Claude Opus 4.6	32.2%	single trial per model, OpenHands	Feb 2026	Official leaderboard
4	GPT-5.4	29.6%	single trial per model, OpenHands	Mar 2026	Official leaderboard
5	Gemini 3.1 Pro	20.9%	single trial per model, OpenHands	Feb 2026	Official leaderboard
6	GLM-5.1	18.3%	single trial per model, OpenHands	Apr 2026	Official leaderboard
7	DeepSeek-V4-Pro	18.3%	single trial per model, OpenHands	Apr 2026	Official leaderboard
8	Kimi K2.6	14.8%	single trial per model, OpenHands	Apr 2026	Official leaderboard
9	MiMo-V2.5-Pro	13.9%	single trial per model, OpenHands	Apr 2026	Official leaderboard
10	Qwen 3.6 Plus	12.2%	single trial per model, OpenHands	Mar 2026	Official leaderboard
11	Kimi K2.5	11.3%	single trial per model, OpenHands	Jan 2026	Official leaderboard
12	MiniMax-M2.7	10.4%	single trial per model, OpenHands	Mar 2026	Official leaderboard

Labs report scores for 1 model in their cards. Switch the scores above to rank them together.

Efficiency Landscape

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

Chart Table

Model	Setting	Resolved rate	Average agent steps
Claude Opus 4.7	single trial per model, OpenHands	39.1%	140 steps
Claude Opus 4.6	single trial per model, OpenHands	32.2%	141 steps
GPT-5.4	single trial per model, OpenHands	29.6%	171 steps
Gemini 3.1 Pro	single trial per model, OpenHands	20.9%	133 steps
DeepSeek-V4-Pro	single trial per model, OpenHands	18.3%	140 steps
GLM-5.1	single trial per model, OpenHands	18.3%	163 steps
Kimi K2.6	single trial per model, OpenHands	14.8%	159 steps
MiMo-V2.5-Pro	single trial per model, OpenHands	13.9%	156 steps
Qwen 3.6 Plus	single trial per model, OpenHands	12.2%	150 steps
Kimi K2.5	single trial per model, OpenHands	11.3%	110 steps
MiniMax-M2.7	single trial per model, OpenHands	10.4%	124 steps
Qwen3.5-397B	single trial per model, OpenHands	9.6%	111 steps

Number one over time 2 models have held number one with claims included since its release in May 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic

Own leaderboard Not tested since May 2026

Claude Opus 4.7 39.1%

Source: Official RoadmapBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far

Sources: Official RoadmapBench leaderboard and labs’ model cards

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

StepFun

Date	Launch post	Score	As the post writes it
Sep 20, 2026	Step 5 Preview	54.3	RoadmapBench