Benchmark Almanac

DeepsecBench

GPT-6 Sol 40.8 leads since September 2026
Not in model cards

Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings.

Models read source only and never run or exploit the app. Vercel reports cost and speed too. Artificial Analysis runs DeepsecBench-AA in its Cyber Index with its own harness, so scores differ.

Made by
Vercel
Released
July 2026
Kind
Finding and exploiting real bugs
Level
Expert
Scored
Graded by people
Answers
Agent task
Tasks
Kept private
Strongest model tested
Gemini 4 Carbon #2 on the capability index
Last model added
October 2026
Scores refreshed
November 12th, 2026
29
Models scored on its own leaderboard

GPT-6 Sol leads with 40.8

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

29 models. Scroll sideways for the rest.

  1. 40.8 GPT-6 Sol
  2. 39.5 GPT-6.1 Sol
  3. 37.6 GPT-6 Astra
  4. 37.5 Claude Sonnet 5.5
  5. 37.5 Kimi K3.1 Pro
  6. 35.3 GPT-5.6 Sol
  7. 35.2 Gemini 4 Carbon
  8. 32.3 Claude Opus 5
  9. 30.0 DeepSeek V4.1 Flash 1017
  10. 26.6 GPT-5.6 Luna
  11. 26.6 Claude Opus 5.5
  12. 21.8 GLM-5.3
  13. 21.0 GPT-5.5
  14. 21.0 GPT-6 Luna
  15. 19.1 GPT-5.6 Terra
  16. 17.4 Kimi K3
  17. 16.4 DeepSeek-V4-Flash
  18. 16.4 Qwen 3.8 Max
  19. 16.4 Grok 4.5
  20. 16.1 Grok 4.6
  21. 14.9 Claude Sonnet 5
  22. 12.9 GLM-5.3-Flash
  23. 11.5 Gemini 3.6 Flash
  24. 10.9 Grok 4.7
  25. 10.4 GLM-5.2
  26. 6.86 Claude Opus 4.8
  27. 6.27 Inkling
  28. 4.69 Claude Haiku 4.5
  29. 2.13 Gemini 3.5 Flash-Lite
  1. 1 GPT-6 Sol xhigh 40.8
  2. 2 GPT-6.1 Sol xhigh 39.5
  3. 3 GPT-6 Astra xhigh 37.6
  4. 4 Claude Sonnet 5.5 max 37.5
  5. 5 Kimi K3.1 Pro max 37.5
  6. 6 GPT-5.6 Sol xhigh 35.3
  7. 7 Gemini 4 Carbon high 35.2
  8. 8 Claude Opus 5 max 32.3
  9. 9 DeepSeek V4.1 Flash 1017 max 30.0
  10. 10 GPT-5.6 Luna xhigh 26.6
  11. 11 Claude Opus 5.5 max 26.6
  12. 12 GLM-5.3 high 21.8
  13. 13 GPT-5.5 xhigh 21.0
  14. 14 GPT-6 Luna xhigh 21.0
  15. 15 GPT-5.6 Terra xhigh 19.1
  16. 16 Kimi K3 xhigh 17.4
  17. 17 DeepSeek-V4-Flash xhigh 16.4
  18. 18 Qwen 3.8 Max xhigh 16.4
  19. 19 Grok 4.5 medium 16.4
  20. 20 Grok 4.6 xhigh 16.1
  21. 21 Claude Sonnet 5 max 14.9
  22. 22 GLM-5.3-Flash high 12.9
  23. 23 Gemini 3.6 Flash medium 11.5
  24. 24 Grok 4.7 xhigh 10.9
  25. 25 GLM-5.2 medium 10.4
  26. 26 Claude Opus 4.8 max 6.86
  27. 27 Inkling xhigh 6.27
  28. 28 Claude Haiku 4.5 4.69
  29. 29 Gemini 3.5 Flash-Lite medium 2.13

Source: Official DeepsecBench leaderboard

#ModelScoreSettingDateSource
1 GPT-6 Sol 40.8 xhigh, Codex Sep 2026 Official leaderboard
2 GPT-6.1 Sol 39.5 xhigh, Codex Sep 2026 Official leaderboard
3 GPT-6 Astra 37.6 xhigh, Codex Sep 2026 Official leaderboard
4 Claude Sonnet 5.5 37.5 max, Claude Code Sep 2026 Official leaderboard
5 Kimi K3.1 Pro 37.5 max, Pi Oct 2026 Official leaderboard
6 GPT-5.6 Sol 35.3 xhigh, Codex Jul 2026 Official leaderboard
7 Gemini 4 Carbon 35.2 high, Pi Oct 2026 Official leaderboard
8 Claude Opus 5 32.3 max, Claude Code Jul 2026 Official leaderboard
9 DeepSeek V4.1 Flash 1017 30.0 max, Pi Oct 2026 Official leaderboard
10 GPT-5.6 Luna 26.6 xhigh, Codex Jul 2026 Official leaderboard
11 Claude Opus 5.5 26.6 max, Claude Code Sep 2026 Official leaderboard
12 GLM-5.3 21.8 high, Pi Aug 2026 Official leaderboard
13 GPT-5.5 21.0 xhigh, Codex Apr 2026 Official leaderboard
14 GPT-6 Luna 21.0 xhigh, Codex Sep 2026 Official leaderboard
15 GPT-5.6 Terra 19.1 xhigh, Codex Jul 2026 Official leaderboard
16 Kimi K3 17.4 xhigh, Pi Jul 2026 Official leaderboard
17 DeepSeek-V4-Flash 16.4 xhigh, Pi Apr 2026 Official leaderboard
18 Qwen 3.8 Max 16.4 xhigh, Pi Aug 2026 Official leaderboard
19 Grok 4.5 16.4 medium, Pi Jul 2026 Official leaderboard
20 Grok 4.6 16.1 xhigh, Pi Aug 2026 Official leaderboard
21 Claude Sonnet 5 14.9 max, Claude Code Jun 2026 Official leaderboard
22 GLM-5.3-Flash 12.9 high, Pi Aug 2026 Official leaderboard
23 Gemini 3.6 Flash 11.5 medium, Pi Jul 2026 Official leaderboard
24 Grok 4.7 10.9 xhigh, Pi Sep 2026 Official leaderboard
25 GLM-5.2 10.4 medium, Pi Jun 2026 Official leaderboard
26 Claude Opus 4.8 6.86 max, Claude Code May 2026 Official leaderboard
27 Inkling 6.27 xhigh, Pi Jul 2026 Official leaderboard
28 Claude Haiku 4.5 4.69 Claude Code Oct 2025 Official leaderboard
29 Gemini 3.5 Flash-Lite 2.13 medium, Pi Jul 2026 Official leaderboard

Score vs. total time

Each point is one model configuration; the Pareto frontier joins configurations that no other beats on both score and total time. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0 10 20 30 40 50 02468 Total time (h), lower is better Score Claude Haiku 4.5: Score 4.69, total time 0.49 h GPT-5.5 (xhigh): Score 21, total time 2.38 h GPT-5.5 (medium): Score 12.5, total time 0.4 h DeepSeek-V4-Flash (xhigh): Score 16.4, total time 1.3 h DeepSeek-V4-Flash (medium): Score 15.3, total time 1.35 h Claude Opus 4.8 (max): Score 6.86, total time 0.32 h Claude Opus 4.8 (medium): Score 6.82, total time 0.3 h GLM-5.2 (medium): Score 10.4, total time 1.06 h GLM-5.2 (xhigh): Score 9.82, total time 2.51 h Claude Sonnet 5 (max): Score 14.9, total time 2.91 h Claude Sonnet 5 (medium): Score 5.79, total time 0.26 h Grok 4.5 (medium): Score 16.4, total time 1.63 h Grok 4.5 (xhigh): Score 15.4, total time 1.4 h GPT-5.6 Luna (xhigh): Score 26.6, total time 3.03 h GPT-5.6 Sol (xhigh): Score 35.3, total time 3.65 h GPT-5.6 Sol (medium): Score 24.9, total time 0.51 h GPT-5.6 Terra (xhigh): Score 19.1, total time 1.76 h Inkling (xhigh): Score 6.27, total time 0.19 h Inkling (medium): Score 5.21, total time 0.18 h Kimi K3 (xhigh): Score 17.4, total time 1.99 h Kimi K3 (medium): Score 11.4, total time 1.49 h Gemini 3.6 Flash (medium): Score 11.5, total time 0.44 h Gemini 3.6 Flash (xhigh): Score 8.91, total time 0.37 h Claude Opus 5 (max): Score 32.3, total time 2.57 h Claude Opus 5 (medium): Score 28.1, total time 0.78 h Qwen 3.8 Max (xhigh): Score 16.4, total time 1.87 h Qwen 3.8 Max (medium): Score 15.5, total time 2.06 h Grok 4.6 (xhigh): Score 16.1, total time 4.04 h GLM-5.3 (high): Score 21.8, total time 2.09 h GLM-5.3 (max): Score 20.3, total time 1.49 h GLM-5.3-Flash (max): Score 10.9, total time 0.68 h Grok 4.7 (xhigh): Score 10.9, total time 0.71 h Grok 4.7 (medium): Score 7.81, total time 0.36 h Claude Opus 5.5 (max): Score 26.6, total time 3.85 h Claude Opus 5.5 (medium): Score 9.95, total time 0.12 h GPT-6 Luna (xhigh): Score 21, total time 1.01 h GPT-6 Luna (medium): Score 13.5, total time 0.42 h GPT-6 Sol (medium): Score 29.2, total time 0.47 h Claude Sonnet 5.5 (max): Score 37.5, total time 6.27 h GPT-6.1 Sol (xhigh): Score 39.5, total time 1.35 h DeepSeek V4.1 Flash 1017 (max): Score 30, total time 1.08 h DeepSeek V4.1 Flash 1017 (high): Score 24.7, total time 0.79 h Kimi K3.1 Pro (max): Score 37.5, total time 2.05 h Kimi K3.1 Pro (high): Score 28.7, total time 0.7 h Gemini 4 Carbon (high): Score 35.2, total time 2.85 h Gemini 4 Carbon (medium): Score 32.4, total time 2.33 h GPT-5.6 Luna (medium): Score 12, total time 0.2 h GPT-5.6 Terra (medium): Score 12, total time 0.22 h Gemini 3.5 Flash-Lite (medium): Score 2.13, total time 0.03 h Gemini 3.5 Flash-Lite (xhigh): Score 1.06, total time 0.02 h GLM-5.3-Flash (high): Score 12.9, total time 0.36 h GPT-6 Astra (xhigh): Score 37.6, total time 0.83 h GPT-6 Astra (medium): Score 29.2, total time 0.36 h GPT-6 Sol (xhigh): Score 40.8, total time 1.15 h Claude Sonnet 5.5 (medium): Score 11.5, total time 0.06 h GPT-6.1 Sol (medium): Score 30.7, total time 0.62 h GPT-6 Sol (xhigh)GPT-6 Astra (xhigh)GPT-6.1 Sol (medium)GPT-6 Astra (medium)GLM-5.3-Flash (high)GPT-5.6 Terra (medium)GPT-5.6 Luna (medium)Claude Sonnet 5.5 (medium)Gemini 3.5 Flash-Lite (medium)Gemini 3.5 Flash-Lite (xhigh) 0 10 20 30 40 50 02468 Total time (h), lower is better Score Claude Haiku 4.5: Score 4.69, total time 0.49 h GPT-5.5 (xhigh): Score 21, total time 2.38 h GPT-5.5 (medium): Score 12.5, total time 0.4 h DeepSeek-V4-Flash (xhigh): Score 16.4, total time 1.3 h DeepSeek-V4-Flash (medium): Score 15.3, total time 1.35 h Claude Opus 4.8 (max): Score 6.86, total time 0.32 h Claude Opus 4.8 (medium): Score 6.82, total time 0.3 h GLM-5.2 (medium): Score 10.4, total time 1.06 h GLM-5.2 (xhigh): Score 9.82, total time 2.51 h Claude Sonnet 5 (max): Score 14.9, total time 2.91 h Claude Sonnet 5 (medium): Score 5.79, total time 0.26 h Grok 4.5 (medium): Score 16.4, total time 1.63 h Grok 4.5 (xhigh): Score 15.4, total time 1.4 h GPT-5.6 Luna (xhigh): Score 26.6, total time 3.03 h GPT-5.6 Sol (xhigh): Score 35.3, total time 3.65 h GPT-5.6 Sol (medium): Score 24.9, total time 0.51 h GPT-5.6 Terra (xhigh): Score 19.1, total time 1.76 h Inkling (xhigh): Score 6.27, total time 0.19 h Inkling (medium): Score 5.21, total time 0.18 h Kimi K3 (xhigh): Score 17.4, total time 1.99 h Kimi K3 (medium): Score 11.4, total time 1.49 h Gemini 3.6 Flash (medium): Score 11.5, total time 0.44 h Gemini 3.6 Flash (xhigh): Score 8.91, total time 0.37 h Claude Opus 5 (max): Score 32.3, total time 2.57 h Claude Opus 5 (medium): Score 28.1, total time 0.78 h Qwen 3.8 Max (xhigh): Score 16.4, total time 1.87 h Qwen 3.8 Max (medium): Score 15.5, total time 2.06 h Grok 4.6 (xhigh): Score 16.1, total time 4.04 h GLM-5.3 (high): Score 21.8, total time 2.09 h GLM-5.3 (max): Score 20.3, total time 1.49 h GLM-5.3-Flash (max): Score 10.9, total time 0.68 h Grok 4.7 (xhigh): Score 10.9, total time 0.71 h Grok 4.7 (medium): Score 7.81, total time 0.36 h Claude Opus 5.5 (max): Score 26.6, total time 3.85 h Claude Opus 5.5 (medium): Score 9.95, total time 0.12 h GPT-6 Luna (xhigh): Score 21, total time 1.01 h GPT-6 Luna (medium): Score 13.5, total time 0.42 h GPT-6 Sol (medium): Score 29.2, total time 0.47 h Claude Sonnet 5.5 (max): Score 37.5, total time 6.27 h GPT-6.1 Sol (xhigh): Score 39.5, total time 1.35 h DeepSeek V4.1 Flash 1017 (max): Score 30, total time 1.08 h DeepSeek V4.1 Flash 1017 (high): Score 24.7, total time 0.79 h Kimi K3.1 Pro (max): Score 37.5, total time 2.05 h Kimi K3.1 Pro (high): Score 28.7, total time 0.7 h Gemini 4 Carbon (high): Score 35.2, total time 2.85 h Gemini 4 Carbon (medium): Score 32.4, total time 2.33 h GPT-5.6 Luna (medium): Score 12, total time 0.2 h GPT-5.6 Terra (medium): Score 12, total time 0.22 h Gemini 3.5 Flash-Lite (medium): Score 2.13, total time 0.03 h Gemini 3.5 Flash-Lite (xhigh): Score 1.06, total time 0.02 h GLM-5.3-Flash (high): Score 12.9, total time 0.36 h GPT-6 Astra (xhigh): Score 37.6, total time 0.83 h GPT-6 Astra (medium): Score 29.2, total time 0.36 h GPT-6 Sol (xhigh): Score 40.8, total time 1.15 h Claude Sonnet 5.5 (medium): Score 11.5, total time 0.06 h GPT-6.1 Sol (medium): Score 30.7, total time 0.62 h GPT-6 Sol (xhigh)GPT-6 Astra (xhigh)GPT-6.1 Sol (medium)GPT-6 Astra (medium)GLM-5.3-Flash (high)GPT-5.6 Terra (medium)GPT-5.6 Luna (medium)Claude Sonnet 5.5 (medium)Gemini 3.5 Flash-Lite (medium)Gemini 3.5 Flash-Lite (xhigh)
ModelSettingScoreTotal time
GPT-6 Sol xhigh, Codex 40.75 1.15 h
GPT-6.1 Sol xhigh, Codex 39.49 1.35 h
GPT-6 Astra xhigh, Codex 37.64 0.83 h
Claude Sonnet 5.5 max, Claude Code 37.5 6.27 h
Kimi K3.1 Pro max, Pi 37.45 2.05 h
GPT-5.6 Sol xhigh, Codex 35.3 3.65 h
Gemini 4 Carbon high, Pi 35.19 2.85 h
Gemini 4 Carbon medium, Pi 32.37 2.33 h
Claude Opus 5 max, Claude Code 32.31 2.57 h
GPT-6.1 Sol medium, Codex 30.69 0.62 h
DeepSeek V4.1 Flash 1017 max, Pi 29.97 1.08 h
GPT-6 Astra medium, Codex 29.21 0.36 h
GPT-6 Sol medium, Codex 29.19 0.47 h
Kimi K3.1 Pro high, Pi 28.71 0.71 h
Claude Opus 5 medium, Claude Code 28.14 0.78 h
GPT-5.6 Luna xhigh, Codex 26.57 3.03 h
Claude Opus 5.5 max, Claude Code 26.57 3.85 h
GPT-5.6 Sol medium, Codex 24.9 0.51 h
DeepSeek V4.1 Flash 1017 high, Pi 24.71 0.79 h
GLM-5.3 high, Pi 21.82 2.09 h
GPT-5.5 xhigh, Codex 21.03 2.38 h
GPT-6 Luna xhigh, Codex 20.97 1.01 h
GLM-5.3 max, Pi 20.31 1.49 h
GPT-5.6 Terra xhigh, Codex 19.06 1.76 h
Kimi K3 xhigh, Pi 17.42 1.99 h
DeepSeek-V4-Flash xhigh, Pi 16.41 1.3 h
Qwen 3.8 Max xhigh, Pi 16.41 1.87 h
Grok 4.5 medium, Pi 16.4 1.63 h
Grok 4.6 xhigh, Pi 16.1 4.04 h
Qwen 3.8 Max medium, Pi 15.46 2.06 h
Grok 4.5 xhigh, Pi 15.45 1.4 h
DeepSeek-V4-Flash medium, Pi 15.35 1.35 h
Claude Sonnet 5 max, Claude Code 14.94 2.91 h
GPT-6 Luna medium, Codex 13.53 0.42 h
GLM-5.3-Flash high, Pi 12.87 0.36 h
GPT-5.5 medium, Codex 12.53 0.4 h
GPT-5.6 Terra medium, Codex 12.02 0.22 h
GPT-5.6 Luna medium, Codex 11.98 0.2 h
Claude Sonnet 5.5 medium, Claude Code 11.48 0.06 h
Gemini 3.6 Flash medium, Pi 11.46 0.44 h
Kimi K3 medium, Pi 11.38 1.49 h
Grok 4.7 xhigh, Pi 10.89 0.71 h
GLM-5.3-Flash max, Pi 10.87 0.68 h
GLM-5.2 medium, Pi 10.37 1.06 h
Claude Opus 5.5 medium, Claude Code 9.95 0.12 h
GLM-5.2 xhigh, Pi 9.82 2.51 h
Gemini 3.6 Flash xhigh, Pi 8.91 0.37 h
Grok 4.7 medium, Pi 7.81 0.36 h
Claude Opus 4.8 max, Claude Code 6.86 0.32 h
Claude Opus 4.8 medium, Claude Code 6.82 0.3 h
Inkling xhigh, Pi 6.27 0.19 h
Claude Sonnet 5 medium, Claude Code 5.79 0.26 h
Inkling medium, Pi 5.21 0.18 h
Claude Haiku 4.5 Claude Code 4.69 0.49 h
Gemini 3.5 Flash-Lite medium, Pi 2.13 0.03 h
Gemini 3.5 Flash-Lite xhigh, Pi 1.06 0.02 h

The best score, 40.8, costs $13 for the full run

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Google DeepMind Z.ai OpenAI DeepSeek Moonshot AI Cost frontier One model at each effort level
0.00 10.0 20.0 30.0 40.0 50.0 $0.10 $1.00 $10 $100 $1,000 Cost of the full run, log scale GPT-6 Sol, xhigh Gemini 4 Carbon, high DeepSeek V4.1 Flash 1017, max DeepSeek V4.1 Flash 1017, high GPT-6 Luna, xhigh GLM-5.3-Flash, high Gemini 3.5 Flash-Lite, xhigh 0.00 10.0 20.0 30.0 40.0 50.0 $0.10 $1.00 $10 $100 $1,000 Cost of the full run, log scale GPT-6 Sol, xhigh Gemini 4 Carbon, high

Source: Official DeepsecBench leaderboard

ModelEffortHarnessScoreCost of the full runOutput tokens
GPT-6 Sol xhigh Codex 40.8 $13
GPT-6.1 Sol xhigh Codex 39.5 $12
GPT-6 Astra xhigh Codex 37.6 $64
Claude Sonnet 5.5 max Claude Code 37.5 $65
Kimi K3.1 Pro max Pi 37.5 $16
GPT-5.6 Sol xhigh Codex 35.3 $56
Gemini 4 Carbon high Pi 35.2 $12
Gemini 4 Carbon medium Pi 32.4 $13
Claude Opus 5 max Claude Code 32.3 $128
GPT-6.1 Sol medium Codex 30.7 $12
DeepSeek V4.1 Flash 1017 max Pi 30.0 $7.87
GPT-6 Astra medium Codex 29.2 $35
GPT-6 Sol medium Codex 29.2 $12
Kimi K3.1 Pro high Pi 28.7 $5.50
Claude Opus 5 medium Claude Code 28.1 $32
GPT-5.6 Luna xhigh Codex 26.6 $25
Claude Opus 5.5 max Claude Code 26.6 $40
GPT-5.6 Sol medium Codex 24.9 $18
DeepSeek V4.1 Flash 1017 high Pi 24.7 $0.56
GLM-5.3 high Pi 21.8 $16
GPT-5.5 xhigh Codex 21.0 $43
GPT-6 Luna xhigh Codex 21.0 $0.51
GLM-5.3 max Pi 20.3 $15
GPT-5.6 Terra xhigh Codex 19.1 $28
Kimi K3 xhigh Pi 17.4 $12
DeepSeek-V4-Flash xhigh Pi 16.4 $5.94
Qwen 3.8 Max xhigh Pi 16.4 $6.33
Grok 4.5 medium Pi 16.4 $11
Grok 4.6 xhigh Pi 16.1 $22
Qwen 3.8 Max medium Pi 15.5 $6.22
Grok 4.5 xhigh Pi 15.4 $5.60
DeepSeek-V4-Flash medium Pi 15.3 $5.06
Claude Sonnet 5 max Claude Code 14.9 $43
GPT-6 Luna medium Codex 13.5 $1.75
GLM-5.3-Flash high Pi 12.9 $0.35
GPT-5.5 medium Codex 12.5 $15
GPT-5.6 Terra medium Codex 12.0 $5.15
GPT-5.6 Luna medium Codex 12.0 $5.23
Claude Sonnet 5.5 medium Claude Code 11.5 $4.33
Gemini 3.6 Flash medium Pi 11.5 $11
Kimi K3 medium Pi 11.4 $11
Grok 4.7 xhigh Pi 10.9 $13
GLM-5.3-Flash max Pi 10.9 $1.83
GLM-5.2 medium Pi 10.4 $11
Claude Opus 5.5 medium Claude Code 9.95 $7.32
GLM-5.2 xhigh Pi 9.82 $52
Gemini 3.6 Flash xhigh Pi 8.91 $5.33
Grok 4.7 medium Pi 7.81 $3.97
Claude Opus 4.8 max Claude Code 6.86 $7.56
Claude Opus 4.8 medium Claude Code 6.82 $11
Inkling xhigh Pi 6.27 $1.93
Claude Sonnet 5 medium Claude Code 5.79 $6.66
Inkling medium Pi 5.21 $2.18
Claude Haiku 4.5 Claude Code 4.69 $5.91
Gemini 3.5 Flash-Lite medium Pi 2.13 $0.58
Gemini 3.5 Flash-Lite xhigh Pi 1.06 $0.18

4 models have held number one on the official leaderboard since its release in July 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard 4 leaders
GPT-6 Sol 40.8

Source: Official DeepsecBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0.00 10.0 20.0 30.0 40.0 50.0 2026 2027 0.00 10.0 20.0 30.0 40.0 50.0 2026 2027

Source: Official DeepsecBench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Cybersecurity

All 115 in Cybersecurity
  • ExploitGym In use 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. GPT-6 Astra 42.4%
  • ExploitBench In use 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. GPT-6 Astra 100%
  • CyberBench v1.1 (Vals) In use Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Muse Wonder 80.4%
  • SRE-Bench In use Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. GPT-6 Astra 56.9%
  • CyScenarioBench In use Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Gemini 4 Argon 100%
  • ExploitBench - Internal Port In use Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. GPT-6 Cyber 42%