Cybersecurity
We track 122 benchmarks in cybersecurity. They run from school level to research frontier level.
OpenAI held the top cybersecurity score 56% of the contested time
That's about 26 of the 46 months since January 2023, averaged over these benchmarks. We only count time while other labs are still testing, and stop once a model passes 85%. Faded stretches no longer count.
Anthropic OpenAI xAI Meta DeepSeek Z.ai Google DeepMind
Sources: Official leaderboards and labs’ model cards
Share of contested time at number one
OpenAI 56% Anthropic 31% 
DeepSeek 4% xAI 3% 
Google DeepMind 2% 
Meta 2% Z.ai 2%
DeepSeek 4% xAI 3% Google DeepMind 2% Meta 2% Z.ai 2% Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
Closed out: 5 benchmarks passed 85%
OpenAI 3 Google DeepMind 1 Anthropic 1
Google DeepMind 1 Anthropic 1 Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
- Oct 2026 CyScenarioBench Gemini 4 Argon
- Sep 2026 ExploitBench GPT-6 Astra
- Sep 2026 SEC-bench Pro GPT-6 Astra
- Jul 2026 CyberBench (Vals) GPT-5.6 Sol
- Feb 2026 CyberGym Claude Opus 4.6
Each benchmark counts from the one source its own page leads with: its official leaderboard when that is current, else Epoch AI’s own runs, else labs’ own numbers. We date independent runs by the model’s release and lab reports by when they came out.
Anthropic OpenAI xAI Meta DeepSeek Z.ai Google DeepMind
Source: Official leaderboards
Share of contested time at number one, independent runs only
OpenAI 56% Anthropic 29% DeepSeek 5% xAI 3% Google DeepMind 3% Meta 2% Z.ai 2%
DeepSeek 5% xAI 3% Google DeepMind 3% Meta 2% Z.ai 2% Sources: Official leaderboards and Epoch AI (CC BY 4.0)
Every independent run: each benchmark’s own leaderboard, Epoch AI’s runs and boards Epoch AI copied, dated by the model’s release. We leave out benchmarks that only labs report.
Anthropic OpenAI DeepSeek Z.ai Google DeepMind Meta xAI
Sources: Official leaderboards, labs’ model cards and claims collected by Epoch AI (CC BY 4.0)
Share of contested time at number one, claimed scores included
OpenAI 45% Anthropic 45% DeepSeek 4% Google DeepMind 3% Meta 2% Z.ai 1%
DeepSeek 4% Google DeepMind 3% Meta 2% Z.ai 1% Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
Closed out: 7 benchmarks passed 85%
OpenAI 4 Anthropic 2 Google DeepMind 1
Google DeepMind 1 Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards
- Oct 2026 CyScenarioBench Gemini 4 Argon
- Sep 2026 ExploitBench GPT-6 Astra
- Sep 2026 SEC-bench Pro GPT-6 Astra
- Sep 2026 SRE-Bench GPT-6 Astra
- Jul 2026 CyberBench (Vals) GPT-5.6 Sol
- Feb 2026 Cybench Claude Opus 4.6
- Feb 2026 CyberGym Claude Opus 4.6
Every score, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.
From school to research frontier level
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
University
Undergraduates and entry-level professionals
Expert
Specialists with years of training, often over hours of work
- CyberGym
- CyberBench v1.1 (Vals)
- ExploitBench
- SRE-Bench
- ExploitGym
- BaxBench
- Cybench
- DeepsecBench
- CyberBench (Vals)
- SecIT Bench
- SEC-bench Pro
- Capture the Flag (Professional)
- BinaryAudit
- CyScenarioBench
- dfbench
- CWE-bench
- CVE-Bench
- CWE-bench 2
- AutoBaxBench
- Cyber Range
- CyberMetric
- SusVibes
- AIRTBench
- PatchEval
- PatchEval-Verified
- SCONE-bench
- UK AISI and CAISI simulated enterprise network range
- AthenaBench
- Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based)
- CRSBench
- Crypto CTF challenges (Anthropic)
- CyberOffense
- Last Secure Code Benchmark
- MiMo Cyber Bench
- Network CTF challenges (Anthropic)
- Pwn CTF challenges (Anthropic)
- Rev CTF challenges (Anthropic)
- SEC-bench Pro 260617
- Web CTF challenges (Anthropic)
- xdev-bench-hard
- Capture the Flag (Internal)
- CyberGym-E2E
- WMDP Cyber
- Atomic Challenges
- Big Sleep Evaluation (Google)
- Chrome Production Commit Scanning (Google)
- Cyber key skills benchmark v1
- CyberGym-E2E (615 tasks)
- Hack the Box challenge set (Google DeepMind)
- In-house CTF suite (Phuong et al. 2024)
- AutoPatchBench
- CTI-REALM
- Real-world vulnerability discovery
- RedCode
- Artificial Analysis Cyber Index
- CTIBench
- CyberGym (final-submission metric)
- CyberSecEval MITRE
- Forensics CTF challenges (Anthropic)
- Irregular FrontierCyber and Atomic Challenges
- Meta curated CTF set
- Misc CTF challenges (Anthropic)
- PrimeVul
- UK AISI narrow cyber tasks
- XBOW Validation Benchmarks
- CathedralBench
- CVE Rediscovery (Anthropic)
- CWEval
- Cyber key skills benchmark v2
- CyberSecEval MITRE False Refusal Rate
- CyberSOCEval Malware Analysis
- CyberSOCEval Threat Intelligence Reasoning
- DiverseVul
- Hack the Box Machines (Meta)
- Internal vulnerability research
- Kimi exploit development suite
- Meta Code-Fetch Under User Requests
- Native Code Exploitation (Meta)
- SEC-bench
- SecretPatch
- SecureCodeReview (xAI)
- SPI (security patch identification)
- SWE-PI
- Unauthorized Code Execution Bench
- A.S.E (AICGSecEval)
- A.S.E 2.0
- Anthropic IoT self-replication evaluation
- AutoPenBench
- BountyBench
- CASTLE
- Catastrophic Cyber Capabilities Benchmark (3CB)
- CVE-Bench 2
- CyberSecEval Autonomous Offensive Cyber Operations
- CyberSecEval Visual Prompt Injection
- CyberSecEval Vulnerability Exploitation
- Daybreak Blue and Enabling Cyber Defense
- EVMbench
- Expert vulnerability discovery and exploit development evaluations (Anthropic)
- Google DeepMind internal cyber benchmarks
- Kimi vulnerability discovery (Tier 1)
- OpenAI internal cyber evaluations
- Pattern Labs cyberoffensive challenges
- PWNBench v0.1
- SEvenLLM
- VulnGym
Research frontier
Researchers, on open problems or tasks that take days
Another 1 has no level. Levels don’t apply to arenas, safety evaluations or composite indices.
By kind
Capture-the-flag challenges
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| SRE-Bench Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. | Expert | Checked automatically | 56.9% | In use |
| Cybench 40 professional capture-the-flag tasks from four competitions. Agent must recover the flag, with optional subtasks. | Expert | Checked automatically | 65.4% | Fading |
| Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. | Expert | Checked automatically | 96.3% | Fading |
| CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. | Expert | Checked automatically | 100% | In use |
| Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. | University | Checked automatically | 68% | System cards only |
| CyberMetric Models answer multiple-choice cybersecurity knowledge questions drawn from standards, papers and RFCs. Sets of 80, 500, 2,000 and 10,000 questions exist. | Expert | Checked automatically |  74.8% | Not in cards |
| AIRTBench Models act as autonomous AI red teamers on 70 black-box capture-the-flag challenges from Dreadnode's Crucible platform. Each challenge hides a flag behind an AI or ML system, and the model writes Python code to attack it with methods such as prompt injection, evasion and model inversion. A challenge counts as solved when the model submits the correct flag. | Expert | Checked automatically | 46.9% | Not in cards |
| Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. | School | Checked automatically | 82% | System cards only |
| NYU CTF Bench Agents solve 200 capture the flag challenges from seven years of CSAW competitions. Six categories cover crypto, forensics, pwn, reverse, web and misc. Each runs in Docker. | University | Checked automatically | 22% | Not in cards |
| UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. | Expert | Checked automatically | 15.6 | System cards only |
| Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. | Expert | Checked automatically | 94.4% | System cards only |
| CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. | Expert | Checked automatically | 87.7% | Not in cards |
| InterCode-CTF Agents solve 100 picoCTF capture-the-flag challenges in an interactive Bash and Python environment. | School | Checked automatically | 98.7% | System cards only |
| Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. | Expert | Checked automatically | 100% | System cards only |
| Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. | Expert | Checked automatically | 71.4% | System cards only |
| Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. | Expert | Checked automatically | 100% | System cards only |
| Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. | Expert | Checked automatically | 100% | System cards only |
| Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. | Expert | Checked automatically | 97.1% | Fading |
| Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. | Expert | Checked automatically | 97% | In use |
| Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. | Expert | Checked automatically | 91.7% | System cards only |
| Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. | Expert | Checked automatically | 15.4% | System cards only |
| In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. | Expert | Checked automatically | 84.6% | System cards only |
| CyberSecEval MITRE Models are asked to help with attack steps drawn from the MITRE ATT&CK framework. Scored by how often the model complies. | Expert | Graded by a model | 40.5% | Not in cards |
| CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. | Graded by a model | 14% | System cards only | |
| Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. | Expert | Checked automatically | 100% | System cards only |
| Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). | Expert | Checked automatically | 70% | System cards only |
| Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. | Expert | Checked automatically | 89.9% | System cards only |
| Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. | Expert | Checked automatically | 100% | System cards only |
| UK AISI narrow cyber tasks Agents solve UK AISI cyber tasks graded by difficulty, including expert-level tasks. Scored as pass@5 and pass@1. | Expert | Checked automatically | 95% | System cards only |
| XBOW Validation Benchmarks Offensive security agents attack deliberately vulnerable web applications and must capture a flag to prove each exploit. | Expert | Checked automatically | 98.5% | Fading |
| Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. | Expert | Checked automatically | 0% | System cards only |
| CyberSecEval MITRE False Refusal Rate Models answer borderline but benign security questions. Scored by how often the model wrongly refuses. | Expert | Graded by a model | 1.60% | Not in cards |
| Catastrophic Cyber Capabilities Benchmark (3CB) Agents solve capture the flag tasks that span MITRE ATT&CK categories such as reconnaissance. Built to measure cyber offense capability. | Expert | Checked automatically | Not in cards | |
| CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. | Expert | Checked automatically | Not in cards | |
| Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. | Expert | Checked automatically | System cards only |
Finding and exploiting real bugs
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| CyberGym Large-scale benchmark of real open-source vulnerabilities. Agent must reproduce each known vulnerability with a proof-of-concept input. | Expert | Checked automatically | 99.4% | In use |
| CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Expert | Checked automatically | 80.4% | In use |
| ExploitBench 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. | Expert | Checked automatically | 100% | In use |
| ExploitGym 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. | Expert | Checked automatically | 42.4% | In use |
| DeepsecBench Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings. | Expert | Graded by people | 40.8 | Not in cards |
| CyberBench (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Expert | Checked automatically | 88.1% | Not in cards |
| SecIT Bench Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions. | Expert | Graded by a model | 82.9% | Not in cards |
| SEC-bench Pro Agents hunt and reproduce security bugs in large software projects from limited information. The first release covers V8 and SpiderMonkey. | Expert | Checked automatically | 88.2% | In use |
| BinaryAudit Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software. | Expert | Checked automatically | 49.1% | Not in cards |
| dfbench Agents do defensive security work on held-out code: find vulnerabilities, judge whether reported findings are real, and explain what a diff changed. Each part is scored separately. | Expert | Checked automatically | 69% | Not in cards |
| CVE-Bench Real-world web-application CVEs in sandboxed services. Agent is scored on reaching predefined attack objectives. | Expert | Checked automatically | 94.1% | System cards only |
| Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. | Expert | Checked automatically | 93.3% | System cards only |
| PatchEval Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch. | Expert | Checked automatically | 83.9% | In use |
| PatchEval-Verified Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch. | Expert | Checked automatically | 83.9% | Not in cards |
| SCONE-bench Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail. | Expert | Checked automatically | $548,400,000 | Not in cards |
| AthenaBench Models answer cyber threat intelligence tasks such as threat actor attribution, attack technique extraction, root cause mapping, severity prediction and risk mitigation. It extends CTIBench. | Expert | Checked automatically | 66.1% | System cards only |
| Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. | Expert | Checked automatically | 91.1% | System cards only |
| CRSBench Cyber reasoning systems find and patch bugs in 124 C, C++ and Java projects with 315 vulnerabilities. A patch passes only if no ground-truth proof of vulnerability still triggers and functionality tests pass. | Expert | Checked automatically | 82.3% | System cards only |
| ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. | Research frontier | Checked automatically | 42% | In use |
| MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. | Expert | Checked automatically | 81.7% | In use |
| SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. | Expert | Checked automatically | 58.4% | Not in cards |
| xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. | Expert | Checked automatically | 100% | In use |
| CyberGym-E2E Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects. | Expert | Checked automatically | 89% | Not in cards |
| Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. | Research frontier | Checked automatically | 72.4% | System cards only |
| WMDP Cyber Models answer multiple-choice questions on offensive cybersecurity knowledge. The cybersecurity split of the Weapons of Mass Destruction Proxy benchmark. | Expert | Checked automatically | 91% | Fading |
| Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. | Expert | Checked automatically | 72% | In use |
| Chrome Production Commit Scanning (Google) Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. | Expert | Checked automatically | 72% | In use |
| CyberGym-E2E (615 tasks) Agents discover and demonstrate vulnerabilities on the older 615-task cohort. The headline is cumulative stage S3 success. | Expert | Checked automatically | 22.6% | Not in cards |
| FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. | Research frontier | Checked automatically | 38.1% | In use |
| Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. | Research frontier | Checked automatically | 45.5% | System cards only |
| UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. | Research frontier | Checked automatically | 80% | System cards only |
| CTI-REALM Agents read cyber threat intelligence and build detection rules. A five-step workflow maps MITRE techniques, finds data sources, writes Sigma and KQL rules and tests them on telemetry. | Expert | Checked automatically | 63.7% | Not in cards |
| Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. | Expert | Checked automatically | 86% | In use |
| Artificial Analysis Cyber Index Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate. | Expert | Checked automatically | 51% | In use |
| CTIBench Models answer cyber threat intelligence questions. Tasks cover knowledge questions, CVE to CWE mapping, CVSS severity scoring, threat actor attribution and technique extraction. | Expert | Checked automatically | 65.7% | Not in cards |
| CyberGym (final-submission metric) Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug. | Expert | Checked automatically |  86.3% | In use |
| PrimeVul Models classify C and C++ functions as vulnerable or not, with a paired setting that pairs each vulnerable function with its patch. | Expert | Checked automatically | 63% | System cards only |
| CathedralBench Independent third-party red-team evaluation of difficult multi-step exploit-chain tasks in egress-secure sandboxes. | Expert | Checked automatically | 29% | System cards only |
| CVE Rediscovery (Anthropic) Models get a known-vulnerable snapshot of an open-source repository and the list of files its fix changed, then report security bugs in one attempt. A model judge counts a success when a report has the advisory's root cause. 1,829 advisories from October 2026. | Expert | Graded by a model | 89.6% | System cards only |
| Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. | Research frontier | Checked automatically | 66.7% | System cards only |
| CyberSOCEval Malware Analysis Models read potential malware behavior and pick every malicious activity that applies. Scored on precision and recall. | Expert | Checked automatically | 33% | System cards only |
| CyberSOCEval Threat Intelligence Reasoning Models read unstructured threat intelligence reports as text or images and pick every actionable insight that applies. | Expert | Checked automatically | 55.3% | System cards only |
| DiverseVul Models classify C and C++ functions from real vulnerability fixing commits as vulnerable or not. | Expert | Checked automatically | 57% | System cards only |
| Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. | Expert | Checked automatically | 41% | System cards only |
| Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. | Expert | Checked automatically | 72% | System cards only |
| Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. | Expert | Checked automatically | 38.9% | System cards only |
| Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. | Expert | Checked automatically | 8.30% | System cards only |
| SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. | Expert | Checked automatically | 18% | Not in cards |
| SecretPatch Models decide whether commits are security patches, including silently released ones, from a set of 4,702 security patches. | Expert | Checked automatically | 72% | System cards only |
| SPI (security patch identification) Models decide whether code commits from open-source projects are security patches. | Expert | Checked automatically | 53% | System cards only |
| Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. | Expert | Checked automatically | Not in cards | |
| AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. | Expert | Checked automatically | Not in cards | |
| BountyBench Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars. | Expert | Checked automatically | Not in cards | |
| CASTLE Models read 250 hand-written compilable C programs and say whether each has a vulnerability and which CWE. Static analyzers run on the same set. | Expert | Checked automatically | Not in cards | |
| CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. | Research frontier | Checked automatically | Not in cards | |
| CVE-Bench 2 Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution. | Expert | Checked automatically | Not in cards | |
| CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. | Expert | Checked automatically | Not in cards | |
| EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. | Expert | Checked automatically | Not in cards | |
| Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. | Expert | Checked automatically | System cards only | |
| Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. | Expert | Checked automatically | Not in cards | |
| Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. | Expert | Graded by people | System cards only | |
| OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). | Expert | Checked automatically | Not in cards | |
| PWNBench v0.1 Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth. | Expert | Checked automatically | Not in cards | |
| SecQA Models answer multiple-choice computer security questions at two difficulty levels. Questions come from a security textbook and were checked by people. | University | Checked automatically | Not in cards | |
| SEvenLLM Models analyze cyber security incidents and write responses. Tasks split into understanding and generation over 28 subcategories, in Chinese and English. | Expert | Graded by a model | Not in cards | |
| VulnGym White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace. | Expert | Checked automatically | Not in cards | |
| VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. | Research frontier | Graded by people | System cards only |
Secure code and review
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| BaxBench Models write backend servers from a specification. Each solution is checked for correctness by tests and for security by end to end exploits. | Expert | Checked automatically | 68.6% | Not in cards |
| CWE-bench Agents audit a repository and write a patch for a security flaw, without being told where the bug is. Covers six programming languages. | Expert | Checked automatically | 71% | In use |
| CWE-bench 2 Agents audit and patch vulnerabilities in 250 codebases. The score is pass@1 under deterministic verification. | Expert | Checked automatically | 75% | Not in cards |
| AutoBaxBench Models write backend servers for tasks that a pipeline generated. Tests and exploits are generated too. A larger task set built in the style of BaxBench. | Expert | Checked automatically | 45% | Not in cards |
| SusVibes Coding agents rebuild a feature in a real open-source Python project from a plain-language request. Each task comes from a historical vulnerability fix, so the original human version was insecure. Hidden functional tests and security tests check the result. Scores report the share of solutions that work and the share that are also secure. | Expert | Checked automatically | 43.5% | Not in cards |
| SecCodeBench Models write or patch functions in project scaffolds drawn from real Alibaba vulnerabilities. Code must pass functional tests, then proof-of-concept security tests. | University | Checked automatically |  71.1% | Fading |
| Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. | Expert | Checked automatically | 25.1% | Not in cards |
| CyberSecEval Autocomplete Models complete code that leads up to a known insecure pattern. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | University | Checked automatically | 71.3% | System cards only |
| CyberSecEval Instruct Models get coding requests built to elicit known insecure patterns. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | University | Checked automatically | 77.3% | System cards only |
| AutoPatchBench Agents write security patches for crashes that a fuzzer found in native code. Fixes are checked by fuzzing and differential debugging. | Expert | Checked automatically | 11.5% | Not in cards |
| RedCode Models meet adversarial prompts to generate or run harmful code. Generation is model-judged and execution is checked in a container. | Expert | Graded by a model | 6.00% | System cards only |
| CWEval Models write code for security-critical tasks in several languages. Outcome-driven test oracles check both functionality (func@k) and security (func-sec@k). | Expert | Checked automatically | 56.3% | System cards only |
| Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. | Expert | Checked automatically | 47.8% | System cards only |
| SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. | Expert | Checked automatically | 58.7% | System cards only |
| SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. | Expert | Checked automatically | 0% | System cards only |
| Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. | Expert | Checked automatically | 21.1% | System cards only |
| A.S.E (AICGSecEval) Models generate code inside real GitHub projects at spots taken from CVE patches. Static analysis then checks whether the generated code is secure. | Expert | Checked automatically | Not in cards | |
| A.S.E 2.0 Models and agentic coding tools write code in real projects covering 29 CWE types. A mix of static analysis, tests and vulnerability proofs of concept checks security. | Expert | Checked automatically | Not in cards | |
| CyberSecEval Visual Prompt Injection Models read text and images where the image carries instructions that try to override the task. Scored by how often the injection succeeds. | Expert | Graded by a model | Not in cards | |
| Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. | Expert | Graded by a model | System cards only |