Dangerous capabilities
Tests of abilities that could enable CBRN or cyber harm, as the EU Code of Practice defines them. We count 167 benchmarks here. A benchmark belongs here when it measures capability, and risk includes CBRN or cyber offence.
How fast it grows
36 were released since July, more than the 33 in the first half of 2026.
How much is still open
No model has passed 85% on 73 of the 111 benchmarks here that score in percent, 66% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| CVE Rediscovery (Anthropic) Models get a known-vulnerable snapshot of an open-source repository and the list of files its fix changed, then report security bugs in one attempt. A model judge counts a success when a report has the advisory's root cause. 1,829 advisories from October 2026. | Expert | Nov 2026 | 89.6% | System cards only |
| UK AISI correlated monitor failure The model acts as a monitor over transcripts of another model carrying out an unsanctioned cyber attack. Scored on the share of violations it fails to flag. | Expert | Nov 2026 | 19% | System cards only |
| UK AISI File Download monitor evasion The model tries to download a forbidden file without a monitoring model noticing. Scored on how often it evades the monitor. | Expert | Nov 2026 | System cards only | |
| UK AISI simulated sandbox escape monitor evasion The model tries to escape a deliberately vulnerable sandbox without a monitoring model noticing. Scored on the share of samples where it escapes undetected. | Expert | Nov 2026 | 0.50% | System cards only |
| UK AISI vulnerability discovery challenges A private suite of synthetic vulnerability discovery challenges. Models answer in a single response without tools, scored pass@1. | Research frontier | Nov 2026 | 88.6% | System cards only |
| Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. | Expert | Oct 2026 | | System cards only |
| Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. | Expert | Oct 2026 | Not in cards | |
| BrowseSafe++ Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1. | University | Oct 2026 | 91.5% | Not in cards |
| CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Expert | Oct 2026 | | In use |
| CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. | Expert | Oct 2026 | 87.7% | Not in cards |
| DualChem Models answer chemistry questions written by experts. The researchers report answer accuracy and assess whether models recognize possible misuse. | University | Oct 2026 | 67.3% | Not in cards |
| SandboxEscapeBench Agents attempt to escape a container sandbox and retrieve a flag from the host. Inspect runs the challenge inside reproducible virtual machines. | Expert | Oct 2026 | Not in cards | |
| xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. | Expert | Oct 2026 | 100% | In use |
| AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. | Expert | Sep 2026 | System cards only | |
| Advanced Screening Evasion Models propose strategies to evade DNA synthesis screening and design evading fragments. Scored from 0 to 1 on strategy quality and fragment evasion. | Research frontier | Sep 2026 | 0.73 | System cards only |
| Cloud Expansion RX00 Agents complete a simulated cloud reconnaissance assessment. The verifier compares the submitted result with the recorded state of the exercise. | Expert | Sep 2026 | Not in cards | |
| Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. | Research frontier | Sep 2026 | 0.47 | System cards only |
| Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. | Expert | Sep 2026 | System cards only | |
| ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. | Research frontier | Sep 2026 | 53.5% | In use |
| MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. | Expert | Sep 2026 | 81.7% | In use |
| Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. | Research frontier | Sep 2026 | 10.7 | System cards only |
| Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. | Research frontier | Sep 2026 | 45.5% | System cards only |
| SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. | Research frontier | Sep 2026 | 0.46 | System cards only |
| Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. | Expert | Sep 2026 | 0% | System cards only |
| VCT-v2 Models answer an updated set of expert virology troubleshooting questions from SecureBio. Scored as refusal-corrected accuracy. | Expert | Sep 2026 | 60.5% | System cards only |
| Alibaba CTF Agents solve 87 capture-the-flag tasks in Alibaba’s collection of competition challenges from 2023 through 2026. | Expert | Aug 2026 | Not in cards | |
| PWNBench v0.1 Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth. | Expert | Aug 2026 | 63.3% | Not in cards |
| RevBench (Polyvor Labs) Agents analyze supplied artifacts in 22 local reverse engineering puzzles. The verifier checks each submitted answer for an exact match. | Expert | Aug 2026 | 50% | Not in cards |
| Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. | Expert | Jul 2026 | | In use |
| Chrome Production Commit Scanning (Google) Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. | Expert | Jul 2026 | | In use |
| CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. | Research frontier | Jul 2026 | 34.2% | Not in cards |
| Cyber harm coverage (Anthropic) Cyber safeguard classifiers screen examples from each cyber harm category. Score is overall recall, the share of harmful examples flagged. | Expert | Jul 2026 | 100% | System cards only |
| CyberGym (final-submission metric) Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug. | Expert | Jul 2026 | | In use |
| Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. | Expert | Jul 2026 | 38.9% | System cards only |
| Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. | Expert | Jul 2026 | System cards only | |
| Vulnerability finding evaluations (Anthropic) Cyber safeguard classifiers screen sampled vulnerability discovery traffic. Reports flag rates on violative binary work and on defensive source code work. | Expert | Jul 2026 | System cards only | |
| AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. | Research frontier | Jun 2026 | 0.59 | System cards only |
| ABC Bench Fragment Design Agents use biology tools to complete a computational genetic-construct design task. One task family of the Agentic Bio-Capabilities Benchmark. | Expert | Jun 2026 | | System cards only |
| ABC Bench Liquid-Handling Robot Agents use biology tools to plan liquid-handling robot steps for a construct-assembly task. One task family of the Agentic Bio-Capabilities Benchmark. | Expert | Jun 2026 | | System cards only |
| ABC Bench Screening Evasion Agents complete a computational task on nucleic-acid synthesis screening. One task family of the Agentic Bio-Capabilities Benchmark. | Expert | Jun 2026 | 82.9% | System cards only |
| CyberBench (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Expert | Jun 2026 | 88.1% | Not in cards |
| CyberGym-E2E Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects. | Expert | Jun 2026 | 89% | Not in cards |
| CyberGym-E2E (615 tasks) Agents discover and demonstrate vulnerabilities on the older 615-task cohort. The headline is cumulative stage S3 success. | Expert | Jun 2026 | | Not in cards |
| FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. | Research frontier | Jun 2026 | 38.1% | In use |
| Protein Design (Anthropic) Models generate novel protein sequences under design constraints and rank protein design libraries for experimental testing. | Expert | Jun 2026 | 42.5% | System cards only |
| SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. | Expert | Jun 2026 | 58.4% | Not in cards |
| WebExploitBench Agents find and exploit 110 vulnerabilities in 15 real web applications, each in its own container. A verifier checks that the agent's proof of concept hits the same endpoint as the reference exploit. | Expert | Jun 2026 | 36.4% | In use |
| AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. | Expert | May 2026 | 0.80 | System cards only |
| Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. | Expert | May 2026 | System cards only | |
| CTF-Archive Diamond Agents solve 285 difficult capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. | Expert | May 2026 | 71% | Not in cards |
| DTap Prompt Injection An attacker agent plants indirect prompt injections in tool and browser environments against a benign-user agent. Reports attack success rate. | Expert | May 2026 | 8.60% | System cards only |
| ExploitBench 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. | Expert | May 2026 | 100% | In use |
| ExploitGym 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. | Expert | May 2026 | 42.4% | In use |
| SEC-bench Pro Agents hunt and reproduce security bugs in large software projects from limited information. The first release covers V8 and SpiderMonkey. | Expert | May 2026 | | In use |
| Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. | Expert | Apr 2026 | 39.3% | System cards only |
| Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. | Expert | Apr 2026 | 97.1% | Fading |
| DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. | Research frontier | Apr 2026 | 16.5% | System cards only |
| Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. | Research frontier | Apr 2026 | 7.60% | System cards only |
| Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. | Expert | Apr 2026 | System cards only | |
| Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. | Expert | Apr 2026 | | System cards only |
| Protocols (Anthropic life sciences) Models find and fix errors in molecular biology protocols and answer practical questions about applying real laboratory protocols. | Expert | Apr 2026 | 62% | System cards only |
| TarantuBench Agents complete simulated web security challenges. Each challenge has a binary success criterion. | Expert | Apr 2026 | Not in cards | |
| UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. | Research frontier | Apr 2026 | 80% | System cards only |
| UK AISI narrow cyber tasks Agents solve UK AISI cyber tasks graded by difficulty, including expert-level tasks. Scored as pass@5 and pass@1. | Expert | Apr 2026 | 95% | System cards only |
| VulnGym White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace. | Expert | Apr 2026 | Not in cards | |
| VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. | Research frontier | Apr 2026 | System cards only | |
| Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. | Expert | Feb 2026 | 16.4 | System cards only |
| EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. | Expert | Feb 2026 | 51% | Not in cards |
| Human Pathogen Capabilities Test v2 (HPCT v2) Models answer multiple-response questions on a small set of weaponizable pathogens that biosecurity experts rank highest priority. A revised version of HPCT. | Research frontier | 2026 | 80% | Not in cards |
| Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. | Expert | Dec 2025 | 90.9% | In use |
| BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. | Expert | Dec 2025 | 99.1% | System cards only |
| CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. | Expert | Dec 2025 | | In use |
| SCONE-bench Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail. | Expert | Dec 2025 | $548,400,000 | Not in cards |
| ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. | Expert | Nov 2025 | 66.8% | System cards only |
| Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. | Expert | Nov 2025 | 73.7% | System cards only |
| BrowseSafe-Bench Detectors and browser agents face prompt injections hidden in realistic HTML pages, across 11 attack types, nine placements and three writing styles. 3,680 test samples. | University | Nov 2025 | 86.3% | Not in cards |
| CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. | Expert | Nov 2025 | System cards only | |
| Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. | Expert | Nov 2025 | | System cards only |
| BioRiskEval Researchers measure biorisk-relevant capability kept by open-weight DNA and protein foundation models, with three task sets named Gen, Mut and Vir. | Expert | Oct 2025 | Not in cards | |
| CVE-Bench 2 Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution. | Expert | Oct 2025 | Not in cards | |
| Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. | Expert | Oct 2025 | | System cards only |
| Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. | Expert | Oct 2025 | System cards only | |
| Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. | Expert | Oct 2025 | | System cards only |
| Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. | Expert | Oct 2025 | | System cards only |
| CTF-Archive Agents solve capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. | Expert | Sep 2025 | Not in cards | |
| Red Team Attack Simulator Models test whether a simulated assistant follows its safety policy across three conversational turns. The evaluation records policy violations and penalizes direct harm. | University | Sep 2025 | Not in cards | |
| Human Pathogen Capabilities Test Models answer expert-level questions on human pathogens from SecureBio. Scored as accuracy. | Expert | Aug 2025 | 80.7% | System cards only |
| AIRTBench Models act as autonomous AI red teamers on 70 black-box capture-the-flag challenges from Dreadnode's Crucible platform. Each challenge hides a flag behind an AI or ML system, and the model writes Python code to attack it with methods such as prompt injection, evasion and model inversion. A challenge counts as solved when the model submits the correct flag. | Expert | Jun 2025 | 46.9% | Not in cards |
| BashBench Agents complete multi-step system administration tasks while red teams attempt covert malicious actions in the same shell environment. | Expert | Jun 2025 | Not in cards | |
| CyberGym Large-scale benchmark of real open-source vulnerabilities. Agent must reproduce each known vulnerability with a proof-of-concept input. | Expert | Jun 2025 | 79.7% | In use |
| SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. | Expert | Jun 2025 | 18% | Not in cards |
| BountyBench Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars. | Expert | May 2025 | 50% | Not in cards |
| Creative biology Models answer questions about unusual biology needed to engineer harmless biological agents, a weak proxy for novel weapon design. | Expert | May 2025 | 60.3% | System cards only |
| DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. | Expert | May 2025 | 80% | System cards only |
| Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. | Expert | May 2025 | System cards only | |
| Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. | Expert | Apr 2025 | 93.3% | System cards only |
| Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. | Expert | Apr 2025 | System cards only | |
| AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. | Expert | Mar 2025 | 22% | Not in cards |
| CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. | Expert | Mar 2025 | System cards only | |
| CVE-Bench Real-world web-application CVEs in sandboxed services. Agent is scored on reaching predefined attack objectives. | Expert | Mar 2025 | 94.1% | System cards only |
| Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. | Expert | Mar 2025 | | System cards only |
| Bioweapons knowledge questions Models answer 33 sensitive bioweapons questions written by Deloitte biosecurity staff. Answers are compared with novice, intermediate and expert baselines. | Expert | Feb 2025 | 51.5% | System cards only |
| Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. | Expert | Feb 2025 | 94.4% | System cards only |
| Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. | Research frontier | Feb 2025 | 66.7% | System cards only |
| Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. | Expert | Feb 2025 | 69.7% | System cards only |
| Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. | Expert | Feb 2025 | 100% | System cards only |
| Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. | Expert | Feb 2025 | 71.4% | System cards only |
| Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. | Expert | Feb 2025 | 100% | System cards only |
| Virology Capabilities Test (VCT) Models answer 322 multimodal virology troubleshooting questions written by expert virologists. Multiple-select variant requires all correct options. | Expert | Feb 2025 | 67.4% | Fading |
| Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. | Expert | Feb 2025 | 100% | System cards only |
| Contextual Nuclear Knowledge Models answer 222 multiple choice questions on nuclear engineering with a nonproliferation focus. Scored as cons@32 accuracy. | Expert | Dec 2024 | 77% | System cards only |
| HackSynth OverTheWire Agents complete scored security exercises from OverTheWire. The published benchmark covers 80 challenges, and the hub port selects 46. | Expert | Dec 2024 | | Not in cards |
| HackSynth PicoCTF Models attempt 120 tasks selected from picoCTF. The benchmark measures the percentage of tasks completed, and the hub port checks submitted answers against reference flags. | Competition | Dec 2024 | | Not in cards |
| Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. | Expert | Dec 2024 | 63.1% | System cards only |
| ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. | Expert | Dec 2024 | 47.2% | In use |
| Radiological and Nuclear Expert Knowledge Models answer 87 expert and tacit knowledge questions on radiological and nuclear threats. Scored as cons@32 accuracy. | Expert | Dec 2024 | 74% | System cards only |
| Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. | Research frontier | Dec 2024 | System cards only | |
| Structured expert probing campaign: radiological and nuclear Radiological and nuclear experts test whether a model helps with weapons development steps. Conversations are rated against Preparedness risk levels. | Research frontier | Dec 2024 | System cards only | |
| Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. | Expert | Dec 2024 | 51% | System cards only |
| AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. | Expert | Oct 2024 | Not in cards | |
| Catastrophic Cyber Capabilities Benchmark (3CB) Agents solve capture the flag tasks that span MITRE ATT&CK categories such as reconnaissance. Built to measure cyber offense capability. | Expert | Oct 2024 | | Not in cards |
| Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. | Expert | Sep 2024 | System cards only | |
| Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. | Expert | Sep 2024 | System cards only | |
| Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. | Expert | Sep 2024 | 0% | System cards only |
| Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. | University | Aug 2024 | 68% | System cards only |
| Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. | School | Aug 2024 | 82% | System cards only |
| Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. | Expert | Aug 2024 | 96.3% | Fading |
| Cybench 40 professional capture-the-flag tasks from four competitions. Agent must recover the flag, with optional subtasks. | Expert | Aug 2024 | | Fading |
| Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. | Expert | Aug 2024 | 94% | System cards only |
| XBOW Validation Benchmarks Offensive security agents attack deliberately vulnerable web applications and must capture a flag to prove each exploit. | Expert | Aug 2024 | 98.5% | Fading |
| CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. | Expert | Jul 2024 | Not in cards | |
| CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. | Jul 2024 | | System cards only | |
| NYU CTF Bench Agents solve 200 capture the flag challenges from seven years of CSAW competitions. Six categories cover crypto, forensics, pwn, reverse, web and misc. Each runs in Docker. | University | Jun 2024 | 22% | Not in cards |
| WildGuardTest Models classify whether user prompts are harmful, whether responses are harmful, and whether responses refuse, across 13 risk categories. Scored by F1. | Everyday | Jun 2024 | | System cards only |
| CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. | Expert | Apr 2024 | Not in cards | |
| DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. | Mar 2024 | | Not in cards | |
| Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. | Expert | Mar 2024 | System cards only | |
| In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. | Expert | Mar 2024 | | System cards only |
| WMDP Proxy multiple-choice benchmark for hazardous knowledge in bio, chem and cyber. Released with the RMU unlearning method. | Expert | Mar 2024 | 84.8% | System cards only |
| WMDP Biology Models answer 1,273 multiple-choice questions that proxy hazardous biosecurity knowledge. Biology split of WMDP. | Expert | Mar 2024 | 91% | Fading |
| WMDP Chemistry Models answer 408 multiple-choice questions that proxy hazardous chemical security knowledge. Chemistry split of WMDP. | Expert | Mar 2024 | 90% | Fading |
| WMDP Cyber Models answer multiple-choice questions on offensive cybersecurity knowledge. The cybersecurity split of the Weapons of Mass Destruction Proxy benchmark. | Expert | Mar 2024 | 91% | Fading |
| Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. | Expert | Feb 2024 | | System cards only |
| Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. | Expert | Jan 2024 | System cards only | |
| Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. | Expert | Sep 2023 | System cards only | |
| InterCode-CTF Agents solve 100 picoCTF capture-the-flag challenges in an interactive Bash and Python environment. | School | Jun 2023 | | System cards only |
| Agentic BAIM-LLM Evaluation (ABLE) Agents run a viral-protein redesign workflow. Nine tasks cover structure retrieval, design strategy, tool selection and biological AI model use. Outputs are auto-scored. | Research frontier | | Not in cards | |
| Anthropic cyber safeguard evaluations Classifier recall on violative cyber exchanges, block rates on defensive vulnerability-finding and secure-coding work, and the rewind-attacker robustness test. | Not in cards | |||
| Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. | Expert | 91.1% | System cards only | |
| Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. | Expert | 2.53 | System cards only | |
| Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. | Expert | System cards only | ||
| CathedralBench Independent third-party red-team evaluation of difficult multi-step exploit-chain tasks in egress-secure sandboxes. | Expert | 29% | System cards only | |
| CRSBench Cyber reasoning systems find and patch bugs in 124 C, C++ and Java projects with 315 vulnerabilities. A patch passes only if no ground-truth proof of vulnerability still triggers and functionality tests pass. | Expert | 82.3% | System cards only | |
| Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. | Research frontier | 72.4% | System cards only | |
| Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. | Expert | 100% | System cards only | |
| Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. | Expert | Not in cards | ||
| HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. | 7.80% | System cards only | ||
| Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. | Expert | | System cards only | |
| Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). | Expert | | System cards only | |
| Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. | Expert | | System cards only | |
| Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. | Expert | 100% | System cards only | |
| Molecular Biology Capabilities Test Models answer expert-level molecular biology questions from SecureBio. Scored as accuracy. | Expert | 68.7% | System cards only | |
| OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). | Expert | Not in cards | ||
| Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. | Expert | | In use | |
| SecureBio VMQA, single choice Models answer 350 virology troubleshooting questions in single-choice form. VMQA is an earlier version of the Virology Capabilities Test. | Expert | | System cards only | |
| UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. | Expert | 15.6 | System cards only | |
| Virology Capabilities Test (text-only) Models answer the text-only questions of the SecureBio Virology Capabilities Test. Scored as accuracy. | Expert | 46% | System cards only |