Benchmark Almanac

CyberSecEval MITRE

llama2-70b-chat 29.4% tops its leaderboard Llama 2-13B 40.5% leads the models we track
Not in model cards

Models are asked to help with attack steps drawn from the MITRE ATT&CK framework. Scored by how often the model complies.

Reported as a compliance rate, so lower is safer. Run with the mitre benchmark flag. The release month follows the CyberSecEval version that introduced it, which the README does not date.

Released
December 2023
Kind
Capture-the-flag challenges
Level
Expert
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Strongest model tested
Llama 2-13B #221 on the capability index
Last model added
August 2023
Scores refreshed
November 12th, 2026
8
Entries on its own leaderboard, 2 models we track

llama2-70b-chat tops the leaderboard with 29.4%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Llama 2-13B, at 40.5%.

  1. 1 llama2-70b-chat 29.4%
  2. 2 llama2-7b-chat 36.3%
  3. 3 gpt-4 (snapshot unspecified) 36.9%
  4. 4 Llama 2-13B 40.5%
  5. 5 gpt-3.5-turbo (snapshot unspecified) 48.6%
  6. 6 codellama-13b-instruct 55.9%
  7. 7 Code Llama 34B Instruct 58.8%
  8. 8 llama2-30b 66.4%

Source: Official CyberSecEval MITRE leaderboard

#EntryScoreSettingDateSource
1 llama2-70b-chat 29.4% Official leaderboard
2 llama2-7b-chat 36.3% Official leaderboard
3 gpt-4 (snapshot unspecified) 36.9% Official leaderboard
4 Llama 2-13B 40.5% Jul 2023 Official leaderboard
5 gpt-3.5-turbo (snapshot unspecified) 48.6% Official leaderboard
6 codellama-13b-instruct 55.9% Official leaderboard
7 Code Llama 34B Instruct 58.8% Aug 2023 Official leaderboard
8 llama2-30b 66.4% Official leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0% 20% 40% 60% 80% 100% Llama 2-13B 0% 20% 40% 60% 80% 100% Llama 2-13B

Source: Official CyberSecEval MITRE leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Cybersecurity

All 115 in Cybersecurity
  • ExploitGym In use 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. GPT-6 Astra 42.4%
  • ExploitBench In use 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. GPT-6 Astra 100%
  • CyberBench v1.1 (Vals) In use Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Muse Wonder 80.4%
  • SRE-Bench In use Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. GPT-6 Astra 56.9%
  • CyScenarioBench In use Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Gemini 4 Argon 100%
  • ExploitBench - Internal Port In use Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. GPT-6 Cyber 42%