Benchmark Almanac

xdev-bench-hard

GPT-5.6 Sol 100% leads since October 2026
In use

Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string.

In-house. Most tasks reproduce recent disclosed vulnerabilities in software such as the Linux kernel, Node.js and Exim, with mitigations on. Moonshot reports pass@3.

Made by
Moonshot AI
Released
October 2026
Kind
Finding and exploiting real bugs
Level
Expert
Scored
Checked automatically
Answers
Agent task
Tasks
Kept private
Strongest model tested
GPT-5.6 Sol #13 on the capability index
Last in a launch post
October 2026
Last model added
October 2026
Scores refreshed
November 12th, 2026
6
Entries on its own leaderboard, 5 models we track
1
Launch post lists it, from 1 lab

GPT-5.6 Sol leads with 100%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

  1. 100.0 GPT-5.6 Sol
  2. 90.5 Kimi model w/o cyber risk-reduction
  3. 85.7 DeepSeek V4.1 Flash
  4. 81.0 Kimi K3.1 Pro
  5. 57.1 GLM-5.3
  6. 33.3 Kimi K3
  1. 1 GPT-5.6 Sol 100%
  2. 2 Kimi model w/o cyber risk-reduction 90.5%
  3. 3 DeepSeek V4.1 Flash 85.7%
  4. 4 Kimi K3.1 Pro 81%
  5. 5 GLM-5.3 57.1%
  6. 6 Kimi K3 33.3%

Source: Official xdev-bench-hard leaderboard

#EntryScoreSettingDateSource
1 GPT-5.6 Sol 100% codex Jul 2026 Official leaderboard
2 Kimi model w/o cyber risk-reduction 90.5% kimi-code Official leaderboard
3 DeepSeek V4.1 Flash 85.7% claude-code Sep 2026 Official leaderboard
4 Kimi K3.1 Pro 81% kimi-code Oct 2026 Official leaderboard
5 GLM-5.3 57.1% claude-code Aug 2026 Official leaderboard
6 Kimi K3 33.3% kimi-code Jul 2026 Official leaderboard

Labs report scores for 1 model in their cards. We plot every source in the charts below.

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Passed 85% in Oct 2026, by GPT-5.6 Sol
GPT-5.6 Sol 100%

Source: Official xdev-bench-hard leaderboard

GPT-5.6 Sol on the official leaderboard passed 85% in July 2026

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card
0% 20% 40% 60% 80% 100% 2027 Near the ceiling (85%) GPT-5.6 Sol 0% 20% 40% 60% 80% 100% 2027 Near the ceiling (85%) GPT-5.6 Sol

Sources: Official xdev-bench-hard leaderboard and labs’ model cards

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Moonshot AI
DateLaunch postScoreAs the post writes it
Oct 24, 2026 Kimi K3.1 Pro 81 xdev-bench-hard (Internal, pass@3)

One lab claim next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 1 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
Kimi K3.1 Pro 81% xdev-bench-hard (Internal, pass@3) 81% Leaderboard, kimi-code Same

More in Cybersecurity

All 115 in Cybersecurity
  • ExploitGym In use 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. GPT-6 Astra 42.4%
  • ExploitBench In use 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. GPT-6 Astra 100%
  • CyberBench v1.1 (Vals) In use Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Muse Wonder 80.4%
  • SRE-Bench In use Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. GPT-6 Astra 56.9%
  • CyScenarioBench In use Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Gemini 4 Argon 100%
  • ExploitBench - Internal Port In use Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. GPT-6 Cyber 42%