Benchmark Almanac

SecIT Bench

DeepSeek V4.1 Flash 82.9% leads since September 2026
Not in model cards

Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions.

30 investigations, some deliberately benign. Joined the Fireworks index on October 28th, 2026 with 17 models.

Released
August 2026
Kind
Finding and exploiting real bugs
Level
Expert
Scored
Graded by a model
Answers
Agent task
Tasks
Kept private
Strongest model tested
GPT-6 Astra #3 on the capability index
Last model added
October 2026
Scores refreshed
November 12th, 2026
27
Entries on its own leaderboard, 26 models we track

DeepSeek V4.1 Flash leads with 82.9%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

27 entries. Scroll sideways for the rest.

  1. 82.9 DeepSeek V4.1 Flash
  2. 81.4 GLM-5.3
  3. 81.0 GPT-6 Astra
  4. 80.7 Grok 4.6
  5. 79.9 GPT-6.1 Sol
  6. 79.2 Claude Opus 5
  7. 78.5 Qwen 3.8 Max
  8. 77.5 Muse Spark 1.3
  9. 77.0 Kimi K3
  10. 76.4 Mistral Large 4
  11. 76.0 MiMo-V2.6 Flash
  12. 75.5 GLM-5.3-Flash
  13. 75.4 GPT-5.6 Sol
  14. 75.1 Claude Sonnet 5.5
  15. 74.9 Grok 4.5
  16. 74.5 Ember-1
  17. 71.9 GPT-5.6 Terra
  18. 69.3 Muse Spark 1.2
  19. 69.0 DeepSeek V4 Flash 0731
  20. 69.0 Gemini 3.8 Flash
  21. 67.1 Claude Opus 4.8
  22. 65.9 Claude Sonnet 5
  23. 65.1 Gemini 3.7 Flash
  24. 64.5 GLM-5.2-Fast
  25. 63.8 GPT-5.6 Luna
  26. 59.3 GPT-6 Sol
  27. 58.0 GPT-6 Luna
  1. 1 DeepSeek V4.1 Flash deepseek-deepseek-v4.1-flash / max 82.9%
  2. 2 GLM-5.3 z-ai-glm-5.3 / max 81.4%
  3. 3 GPT-6 Astra openai-gpt-6-astra / max 81%
  4. 4 Grok 4.6 x-ai-grok-4.6 / max 80.7%
  5. 5 GPT-6.1 Sol openai-gpt-6.1-sol / max 79.9%
  6. 6 Claude Opus 5 max 79.2%
  7. 7 Qwen 3.8 Max qwen-qwen3.8-max / max 78.5%
  8. 8 Muse Spark 1.3 meta-muse-spark-1.3 / max 77.5%
  9. 9 Kimi K3 moonshotai-Kimi-K3 / max 77%
  10. 10 Mistral Large 4 mistralai-mistral-large-4-0 / max 76.4%
  11. 11 MiMo-V2.6 Flash xiaomi-mimo-v2.6-flash / max 76%
  12. 12 GLM-5.3-Flash stealth-ox-alpha / max 75.5%
  13. 13 GPT-5.6 Sol max 75.4%
  14. 14 Claude Sonnet 5.5 max 75.1%
  15. 15 Grok 4.5 x-ai-grok-4.5 / max 74.9%
  16. 16 Ember-1 fireworks-ember-1 / max 74.5%
  17. 17 GPT-5.6 Terra max 71.9%
  18. 18 Muse Spark 1.2 meta-muse-spark-1.2 / max 69.3%
  19. 19 DeepSeek V4 Flash 0731 deepseek-deepseek-v4-flash-0731 / max 69%
  20. 20 Gemini 3.8 Flash google-gemini-3.8-flash / max 69%
  21. 21 Claude Opus 4.8 max 67.1%
  22. 22 Claude Sonnet 5 max 65.9%
  23. 23 Gemini 3.7 Flash google-gemini-3.7-flash / max 65.1%
  24. 24 GLM-5.2-Fast zai-org-GLM-5.2-Fast / high 64.5%
  25. 25 GPT-5.6 Luna max 63.8%
  26. 26 GPT-6 Sol openai-gpt-6-sol / max 59.3%
  27. 27 GPT-6 Luna openai-gpt-6-luna / max 58%

Source: Official SecIT Bench leaderboard

#EntryScoreSettingDateSource
1 DeepSeek V4.1 Flash 82.9% deepseek-deepseek-v4.1-flash / max, pydantic-ai Sep 2026 Official leaderboard
2 GLM-5.3 81.4% z-ai-glm-5.3 / max, pydantic-ai Aug 2026 Official leaderboard
3 GPT-6 Astra 81% openai-gpt-6-astra / max, pydantic-ai Sep 2026 Official leaderboard
4 Grok 4.6 80.7% x-ai-grok-4.6 / max, pydantic-ai Aug 2026 Official leaderboard
5 GPT-6.1 Sol 79.9% openai-gpt-6.1-sol / max, pydantic-ai Sep 2026 Official leaderboard
6 Claude Opus 5 79.2% max, pydantic-ai Jul 2026 Official leaderboard
7 Qwen 3.8 Max 78.5% qwen-qwen3.8-max / max, pydantic-ai Aug 2026 Official leaderboard
8 Muse Spark 1.3 77.5% meta-muse-spark-1.3 / max, pydantic-ai Sep 2026 Official leaderboard
9 Kimi K3 77% moonshotai-Kimi-K3 / max, pydantic-ai Jul 2026 Official leaderboard
10 Mistral Large 4 76.4% mistralai-mistral-large-4-0 / max, pydantic-ai Oct 2026 Official leaderboard
11 MiMo-V2.6 Flash 76% xiaomi-mimo-v2.6-flash / max, pydantic-ai Sep 2026 Official leaderboard
12 GLM-5.3-Flash 75.5% stealth-ox-alpha / max, pydantic-ai Aug 2026 Official leaderboard
13 GPT-5.6 Sol 75.4% max, pydantic-ai Jul 2026 Official leaderboard
14 Claude Sonnet 5.5 75.1% max, pydantic-ai Sep 2026 Official leaderboard
15 Grok 4.5 74.9% x-ai-grok-4.5 / max, pydantic-ai Jul 2026 Official leaderboard
16 Ember-1 74.5% fireworks-ember-1 / max, pydantic-ai Sep 2026 Official leaderboard
17 GPT-5.6 Terra 71.9% max, codex Jul 2026 Official leaderboard
18 Muse Spark 1.2 69.3% meta-muse-spark-1.2 / max, pydantic-ai Aug 2026 Official leaderboard
19 DeepSeek V4 Flash 0731 69% deepseek-deepseek-v4-flash-0731 / max, pydantic-ai Jul 2026 Official leaderboard
20 Gemini 3.8 Flash 69% google-gemini-3.8-flash / max, pydantic-ai Sep 2026 Official leaderboard
21 Claude Opus 4.8 67.1% max, claude-code May 2026 Official leaderboard
22 Claude Sonnet 5 65.9% max, claude-code Jun 2026 Official leaderboard
23 Gemini 3.7 Flash 65.1% google-gemini-3.7-flash / max, pydantic-ai Aug 2026 Official leaderboard
24 GLM-5.2-Fast 64.5% zai-org-GLM-5.2-Fast / high, pydantic-ai Sep 2026 Official leaderboard
25 GPT-5.6 Luna 63.8% max, pydantic-ai Jul 2026 Official leaderboard
26 GPT-6 Sol 59.3% openai-gpt-6-sol / max, pydantic-ai Sep 2026 Official leaderboard
27 GPT-6 Luna 58% openai-gpt-6-luna / max, pydantic-ai Sep 2026 Official leaderboard

Accuracy vs. cost

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

60% 70% 80% 90% 100% $0$5$10$15$20 Estimated cost per rollout ($), lower is better Accuracy Claude Opus 4.8 (max): Accuracy 67.1%, estimated cost per rollout $3.44 Claude Opus 4.8 (max): Accuracy 67%, estimated cost per rollout $2.15 Claude Sonnet 5 (max): Accuracy 65.9%, estimated cost per rollout $5.78 Claude Sonnet 5 (max): Accuracy 63.3%, estimated cost per rollout $1.66 Grok 4.5 (x-ai-grok-4.5 / max): Accuracy 74.9%, estimated cost per rollout $1.26 GPT-5.6 Luna (max): Accuracy 60.1%, estimated cost per rollout $0.46 GPT-5.6 Luna (max): Accuracy 63.8%, estimated cost per rollout $0.51 GPT-5.6 Sol (max): Accuracy 71.7%, estimated cost per rollout $6.08 GPT-5.6 Sol (max): Accuracy 75.4%, estimated cost per rollout $4.62 GPT-5.6 Terra (max): Accuracy 71.9%, estimated cost per rollout $2.36 GPT-5.6 Terra (max): Accuracy 69.4%, estimated cost per rollout $2.12 Kimi K3 (moonshotai-Kimi-K3 / max): Accuracy 77%, estimated cost per rollout $3.13 Claude Opus 5 (max): Accuracy 78.6%, estimated cost per rollout $7.34 Claude Opus 5 (max): Accuracy 79.2%, estimated cost per rollout $6.11 DeepSeek V4 Flash 0731 (deepseek-deepseek-v4-flash-0731 / max): Accuracy 69%, estimated cost per rollout $0.28 Qwen 3.8 Max (qwen-qwen3.8-max / max): Accuracy 78.5%, estimated cost per rollout $3.39 Muse Spark 1.2 (meta-muse-spark-1.2 / max): Accuracy 69.3%, estimated cost per rollout $2.88 Grok 4.6 (x-ai-grok-4.6 / max): Accuracy 80.7%, estimated cost per rollout $3.27 Gemini 3.7 Flash (google-gemini-3.7-flash / max): Accuracy 65.1%, estimated cost per rollout $3.96 GLM-5.3 (z-ai-glm-5.3 / max): Accuracy 81.4%, estimated cost per rollout $2.7 Gemini 3.8 Flash (google-gemini-3.8-flash / max): Accuracy 69%, estimated cost per rollout $19.2 Muse Spark 1.3 (meta-muse-spark-1.3 / max): Accuracy 77.5%, estimated cost per rollout $3.65 GPT-6 Astra (openai-gpt-6-astra / max): Accuracy 81%, estimated cost per rollout $8.21 GPT-6 Sol (openai-gpt-6-sol / max): Accuracy 59.3%, estimated cost per rollout $2.46 Ember-1 (fireworks-ember-1 / max): Accuracy 74.5%, estimated cost per rollout $3.01 Claude Sonnet 5.5 (max): Accuracy 69.9%, estimated cost per rollout $0.47 Claude Sonnet 5.5 (max): Accuracy 75.1%, estimated cost per rollout $0.72 GPT-6.1 Sol (openai-gpt-6.1-sol / max): Accuracy 79.9%, estimated cost per rollout $1.94 Mistral Large 4 (mistralai-mistral-large-4-0 / max): Accuracy 76.4%, estimated cost per rollout $1.78 GLM-5.2-Fast (zai-org-GLM-5.2-Fast / high): Accuracy 64.5%, estimated cost per rollout $1.7. Not in our model list GLM-5.3-Flash (stealth-ox-alpha / max): Accuracy 75.5%, estimated cost per rollout $0.28 DeepSeek V4.1 Flash (deepseek-deepseek-v4.1-flash / max): Accuracy 82.9%, estimated cost per rollout $1.13 MiMo-V2.6 Flash (xiaomi-mimo-v2.6-flash / max): Accuracy 76%, estimated cost per rollout $0.33 GPT-6 Luna (openai-gpt-6-luna / max): Accuracy 58%, estimated cost per rollout $0.13 DeepSeek V4.1 Flash (deepseek-deepseek-v4.1-flash / max)MiMo-V2.6 Flash (xiaomi-mimo-v2.6-flash / max)GLM-5.3-Flash (stealth-ox-alpha / max)GPT-6 Luna (openai-gpt-6-luna / max) 60% 70% 80% 90% 100% $0$5$10$15$20 Estimated cost per rollout ($), lower is better Accuracy Claude Opus 4.8 (max): Accuracy 67.1%, estimated cost per rollout $3.44 Claude Opus 4.8 (max): Accuracy 67%, estimated cost per rollout $2.15 Claude Sonnet 5 (max): Accuracy 65.9%, estimated cost per rollout $5.78 Claude Sonnet 5 (max): Accuracy 63.3%, estimated cost per rollout $1.66 Grok 4.5 (x-ai-grok-4.5 / max): Accuracy 74.9%, estimated cost per rollout $1.26 GPT-5.6 Luna (max): Accuracy 60.1%, estimated cost per rollout $0.46 GPT-5.6 Luna (max): Accuracy 63.8%, estimated cost per rollout $0.51 GPT-5.6 Sol (max): Accuracy 71.7%, estimated cost per rollout $6.08 GPT-5.6 Sol (max): Accuracy 75.4%, estimated cost per rollout $4.62 GPT-5.6 Terra (max): Accuracy 71.9%, estimated cost per rollout $2.36 GPT-5.6 Terra (max): Accuracy 69.4%, estimated cost per rollout $2.12 Kimi K3 (moonshotai-Kimi-K3 / max): Accuracy 77%, estimated cost per rollout $3.13 Claude Opus 5 (max): Accuracy 78.6%, estimated cost per rollout $7.34 Claude Opus 5 (max): Accuracy 79.2%, estimated cost per rollout $6.11 DeepSeek V4 Flash 0731 (deepseek-deepseek-v4-flash-0731 / max): Accuracy 69%, estimated cost per rollout $0.28 Qwen 3.8 Max (qwen-qwen3.8-max / max): Accuracy 78.5%, estimated cost per rollout $3.39 Muse Spark 1.2 (meta-muse-spark-1.2 / max): Accuracy 69.3%, estimated cost per rollout $2.88 Grok 4.6 (x-ai-grok-4.6 / max): Accuracy 80.7%, estimated cost per rollout $3.27 Gemini 3.7 Flash (google-gemini-3.7-flash / max): Accuracy 65.1%, estimated cost per rollout $3.96 GLM-5.3 (z-ai-glm-5.3 / max): Accuracy 81.4%, estimated cost per rollout $2.7 Gemini 3.8 Flash (google-gemini-3.8-flash / max): Accuracy 69%, estimated cost per rollout $19.2 Muse Spark 1.3 (meta-muse-spark-1.3 / max): Accuracy 77.5%, estimated cost per rollout $3.65 GPT-6 Astra (openai-gpt-6-astra / max): Accuracy 81%, estimated cost per rollout $8.21 GPT-6 Sol (openai-gpt-6-sol / max): Accuracy 59.3%, estimated cost per rollout $2.46 Ember-1 (fireworks-ember-1 / max): Accuracy 74.5%, estimated cost per rollout $3.01 Claude Sonnet 5.5 (max): Accuracy 69.9%, estimated cost per rollout $0.47 Claude Sonnet 5.5 (max): Accuracy 75.1%, estimated cost per rollout $0.72 GPT-6.1 Sol (openai-gpt-6.1-sol / max): Accuracy 79.9%, estimated cost per rollout $1.94 Mistral Large 4 (mistralai-mistral-large-4-0 / max): Accuracy 76.4%, estimated cost per rollout $1.78 GLM-5.2-Fast (zai-org-GLM-5.2-Fast / high): Accuracy 64.5%, estimated cost per rollout $1.7. Not in our model list GLM-5.3-Flash (stealth-ox-alpha / max): Accuracy 75.5%, estimated cost per rollout $0.28 DeepSeek V4.1 Flash (deepseek-deepseek-v4.1-flash / max): Accuracy 82.9%, estimated cost per rollout $1.13 MiMo-V2.6 Flash (xiaomi-mimo-v2.6-flash / max): Accuracy 76%, estimated cost per rollout $0.33 GPT-6 Luna (openai-gpt-6-luna / max): Accuracy 58%, estimated cost per rollout $0.13 DeepSeek V4.1 Flash (deepseek-deepseek-v4.1-flash / max)MiMo-V2.6 Flash (xiaomi-mimo-v2.6-flash / max)GLM-5.3-Flash (stealth-ox-alpha / max)GPT-6 Luna (openai-gpt-6-luna / max)
ModelSettingAccuracyEstimated cost per rollout
DeepSeek V4.1 Flash deepseek-deepseek-v4.1-flash / max, pydantic-ai 82.9% $1.13
GLM-5.3 z-ai-glm-5.3 / max, pydantic-ai 81.4% $2.7
GPT-6 Astra openai-gpt-6-astra / max, pydantic-ai 81% $8.21
Grok 4.6 x-ai-grok-4.6 / max, pydantic-ai 80.7% $3.27
GPT-6.1 Sol openai-gpt-6.1-sol / max, pydantic-ai 79.9% $1.94
Claude Opus 5 max, pydantic-ai 79.2% $6.11
Claude Opus 5 max, claude-code 78.6% $7.34
Qwen 3.8 Max qwen-qwen3.8-max / max, pydantic-ai 78.5% $3.39
Muse Spark 1.3 meta-muse-spark-1.3 / max, pydantic-ai 77.5% $3.66
Kimi K3 moonshotai-Kimi-K3 / max, pydantic-ai 77% $3.13
Mistral Large 4 mistralai-mistral-large-4-0 / max, pydantic-ai 76.4% $1.78
MiMo-V2.6 Flash xiaomi-mimo-v2.6-flash / max, pydantic-ai 76% $0.33
GLM-5.3-Flash stealth-ox-alpha / max, pydantic-ai 75.5% $0.28
GPT-5.6 Sol max, pydantic-ai 75.4% $4.62
Claude Sonnet 5.5 max, pydantic-ai 75.1% $0.72
Grok 4.5 x-ai-grok-4.5 / max, pydantic-ai 74.9% $1.26
Ember-1 fireworks-ember-1 / max, pydantic-ai 74.5% $3.01
GPT-5.6 Terra max, codex 71.9% $2.36
GPT-5.6 Sol max, codex 71.7% $6.09
Claude Sonnet 5.5 max, claude-code 69.9% $0.47
GPT-5.6 Terra max, pydantic-ai 69.4% $2.12
Muse Spark 1.2 meta-muse-spark-1.2 / max, pydantic-ai 69.4% $2.88
DeepSeek V4 Flash 0731 deepseek-deepseek-v4-flash-0731 / max, pydantic-ai 69% $0.28
Gemini 3.8 Flash google-gemini-3.8-flash / max, pydantic-ai 69% $19.24
Claude Opus 4.8 max, claude-code 67.1% $3.44
Claude Opus 4.8 max, pydantic-ai 67% $2.15
Claude Sonnet 5 max, claude-code 65.9% $5.78
Gemini 3.7 Flash google-gemini-3.7-flash / max, pydantic-ai 65.1% $3.96
GLM-5.2-Fast zai-org-GLM-5.2-Fast / high, pydantic-ai 64.5% $1.7
GPT-5.6 Luna max, pydantic-ai 63.8% $0.51
Claude Sonnet 5 max, pydantic-ai 63.3% $1.66
GPT-5.6 Luna max, codex 60.1% $0.46
GPT-6 Sol openai-gpt-6-sol / max, pydantic-ai 59.3% $2.46
GPT-6 Luna openai-gpt-6-luna / max, pydantic-ai 58% $0.13

The best score, 82.9%, costs $1.13 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Z.ai DeepSeek Xiaomi Corp OpenAI Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.10 $1.00 $10 $100 Cost per task, log scale DeepSeek V4.1 Flash, deepseek-deepseek-v4.1-flash / max MiMo-V2.6 Flash, xiaomi-mimo-v2.6-flash / max GPT-6 Luna, openai-gpt-6-luna / max 0% 20% 40% 60% 80% 100% $0.10 $1.00 $10 $100 Cost per task, log scale DeepSeek V4.1 Flash, deepseek-deepseek-v4.1-flash / max MiMo-V2.6 Flash, xiaomi-mimo-v2.6-flash / max

Source: Official SecIT Bench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
DeepSeek V4.1 Flash deepseek-deepseek-v4.1-flash / max pydantic-ai 82.9% $1.13 83,082
GLM-5.3 z-ai-glm-5.3 / max pydantic-ai 81.4% $2.70 27,899
GPT-6 Astra openai-gpt-6-astra / max pydantic-ai 81% $8.21 26,875
Grok 4.6 x-ai-grok-4.6 / max pydantic-ai 80.7% $3.27 23,662
GPT-6.1 Sol openai-gpt-6.1-sol / max pydantic-ai 79.9% $1.94 26,721
Claude Opus 5 max pydantic-ai 79.2% $6.11 15,818
Claude Opus 5 max claude-code 78.6% $7.34 28,725
Qwen 3.8 Max qwen-qwen3.8-max / max pydantic-ai 78.5% $3.39 25,014
Muse Spark 1.3 meta-muse-spark-1.3 / max pydantic-ai 77.5% $3.65 24,148
Kimi K3 moonshotai-Kimi-K3 / max pydantic-ai 77% $3.13 19,068
Mistral Large 4 mistralai-mistral-large-4-0 / max pydantic-ai 76.4% $1.78 32,770
MiMo-V2.6 Flash xiaomi-mimo-v2.6-flash / max pydantic-ai 76% $0.33 33,167
GLM-5.3-Flash stealth-ox-alpha / max pydantic-ai 75.5% $0.28 18,213
GPT-5.6 Sol max pydantic-ai 75.4% $4.62 10,155
Claude Sonnet 5.5 max pydantic-ai 75.1% $0.72 6,773
Grok 4.5 x-ai-grok-4.5 / max pydantic-ai 74.9% $1.26 8,741
Ember-1 fireworks-ember-1 / max pydantic-ai 74.5% $3.01 15,836
GPT-5.6 Terra max codex 71.9% $2.36 10,851
GPT-5.6 Sol max codex 71.7% $6.08 11,319
Claude Sonnet 5.5 max claude-code 69.9% $0.47 4,750
GPT-5.6 Terra max pydantic-ai 69.4% $2.12 11,083
Muse Spark 1.2 meta-muse-spark-1.2 / max pydantic-ai 69.3% $2.88 40,795
DeepSeek V4 Flash 0731 deepseek-deepseek-v4-flash-0731 / max pydantic-ai 69% $0.28 26,697
Gemini 3.8 Flash google-gemini-3.8-flash / max pydantic-ai 69% $19 24,549
Claude Opus 4.8 max claude-code 67.1% $3.44 16,754
Claude Opus 4.8 max pydantic-ai 67% $2.15 11,434
Claude Sonnet 5 max claude-code 65.9% $5.78 31,099
Gemini 3.7 Flash google-gemini-3.7-flash / max pydantic-ai 65.1% $3.96 11,835
GPT-5.6 Luna max pydantic-ai 63.8% $0.51 12,287
Claude Sonnet 5 max pydantic-ai 63.3% $1.66 12,205
GPT-5.6 Luna max codex 60.1% $0.46 16,086
GPT-6 Sol openai-gpt-6-sol / max pydantic-ai 59.3% $2.46 39,112
GPT-6 Luna openai-gpt-6-luna / max pydantic-ai 58% $0.13 86,988

4 models have held number one on the official leaderboard since its release in August 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic xAI Z.ai DeepSeek
Own leaderboard 4 leaders
DeepSeek V4.1 Flash 82.9%

Source: Official SecIT Bench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) DeepSeek V4.1 Flash Kimi K3 Claude Opus 4.8 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) DeepSeek V4.1 Flash GPT-5.6 Sol

Source: Official SecIT Bench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Cybersecurity

All 115 in Cybersecurity
  • ExploitGym In use 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. GPT-6 Astra 42.4%
  • ExploitBench In use 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. GPT-6 Astra 100%
  • CyberBench v1.1 (Vals) In use Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Muse Wonder 80.4%
  • SRE-Bench In use Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. GPT-6 Astra 56.9%
  • CyScenarioBench In use Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Gemini 4 Argon 100%
  • ExploitBench - Internal Port In use Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. GPT-6 Cyber 42%