Benchmark Almanac

CyberSecEval MITRE False Refusal Rate

llama3p-70b-rc3_vr_mid_3 0% tops its leaderboard Code Llama 34B Instruct 1.60% leads the models we track
Not in model cards

Models answer borderline but benign security questions. Scored by how often the model wrongly refuses.

Paired with the MITRE compliance test to show over-refusal. Run with the mitre-frr flag. The release month follows the CyberSecEval version that introduced it, which the README does not date.

Released
April 2024
Kind
Capture-the-flag challenges
Level
Expert
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Last model added
August 2023
Scores refreshed
November 12th, 2026
10
Entries on its own leaderboard, 1 model we track

llama3p-70b-rc3_vr_mid_3 tops the leaderboard with 0%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Code Llama 34B Instruct, at 1.60%.

  1. 1 llama3p-70b-rc3_vr_mid_3 0%
  2. 2 gpt-3.5-turbo (snapshot unspecified) 0.27%
  3. 3 codellama-13b-instruct 1.60%
  4. 4 Code Llama 34B Instruct 1.60%
  5. 5 llama3p-7b-rc3_vr_mid_2 2.27%
  6. 6 gpt-4 (snapshot unspecified) 4.13%
  7. 7 mistral-small-latest (snapshot unspecified) 10.4%
  8. 8 mistral-large-latest (snapshot unspecified) 11.1%
  9. 9 mistral-medium-latest (snapshot unspecified) 11.3%
  10. 10 codellama/CodeLlama-70b-Instruct-hf 69.5%

Source: Official CyberSecEval MITRE False Refusal Rate leaderboard

#EntryScoreSettingDateSource
1 llama3p-70b-rc3_vr_mid_3 0% Official leaderboard
2 gpt-3.5-turbo (snapshot unspecified) 0.27% Official leaderboard
3 codellama-13b-instruct 1.60% Official leaderboard
4 Code Llama 34B Instruct 1.60% Aug 2023 Official leaderboard
5 llama3p-7b-rc3_vr_mid_2 2.27% Official leaderboard
6 gpt-4 (snapshot unspecified) 4.13% Official leaderboard
7 mistral-small-latest (snapshot unspecified) 10.4% Official leaderboard
8 mistral-large-latest (snapshot unspecified) 11.1% Official leaderboard
9 mistral-medium-latest (snapshot unspecified) 11.3% Official leaderboard
10 codellama/CodeLlama-70b-Instruct-hf 69.5% Official leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0% 20% 40% 60% 80% 100% Code Llama 34B Instruct 0% 20% 40% 60% 80% 100% Code Llama 34B Instruct

Source: Official CyberSecEval MITRE False Refusal Rate leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Cybersecurity

All 115 in Cybersecurity
  • ExploitGym In use 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. GPT-6 Astra 42.4%
  • ExploitBench In use 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. GPT-6 Astra 100%
  • CyberBench v1.1 (Vals) In use Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Muse Wonder 80.4%
  • SRE-Bench In use Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. GPT-6 Astra 56.9%
  • CyScenarioBench In use Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Gemini 4 Argon 100%
  • ExploitBench - Internal Port In use Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. GPT-6 Cyber 42%