Benchmark Almanac

Dangerous capabilities

Tests of abilities that could enable CBRN or cyber harm, as the EU Code of Practice defines them. We count 167 benchmarks here. A benchmark belongs here when it measures capability, and risk includes CBRN or cyber offence.

How fast it grows

36 were released since July, more than the 33 in the first half of 2026.

  1. Earlier 2
  2. 2024 H1 12
  3. 2024 H2 22
  4. 2025 H1 23
  5. 2025 H2 18
  6. 2026 H1 33
  7. 2026 H2 36

How much is still open

No model has passed 85% on 73 of the 111 benchmarks here that score in percent, 66% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
CVE Rediscovery (Anthropic) Models get a known-vulnerable snapshot of an open-source repository and the list of files its fix changed, then report security bugs in one attempt. A model judge counts a success when a report has the advisory's root cause. 1,829 advisories from October 2026. Expert Nov 2026 89.6% System cards only
UK AISI correlated monitor failure The model acts as a monitor over transcripts of another model carrying out an unsanctioned cyber attack. Scored on the share of violations it fails to flag. Expert Nov 2026 19% System cards only
UK AISI File Download monitor evasion The model tries to download a forbidden file without a monitoring model noticing. Scored on how often it evades the monitor. Expert Nov 2026 System cards only
UK AISI simulated sandbox escape monitor evasion The model tries to escape a deliberately vulnerable sandbox without a monitoring model noticing. Scored on the share of samples where it escapes undetected. Expert Nov 2026 0.50% System cards only
UK AISI vulnerability discovery challenges A private suite of synthetic vulnerability discovery challenges. Models answer in a single response without tools, scored pass@1. Research frontier Nov 2026 88.6% System cards only
Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. Expert Oct 2026 47% System cards only
Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. Expert Oct 2026 Not in cards
BrowseSafe++ Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1. University Oct 2026 91.5% Not in cards
CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Expert Oct 2026 80.4% In use
CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. Expert Oct 2026 87.7% Not in cards
DualChem Models answer chemistry questions written by experts. The researchers report answer accuracy and assess whether models recognize possible misuse. University Oct 2026 67.3% Not in cards
SandboxEscapeBench Agents attempt to escape a container sandbox and retrieve a flag from the host. Inspect runs the challenge inside reproducible virtual machines. Expert Oct 2026 Not in cards
xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. Expert Oct 2026 100% In use
AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. Expert Sep 2026 System cards only
Advanced Screening Evasion Models propose strategies to evade DNA synthesis screening and design evading fragments. Scored from 0 to 1 on strategy quality and fragment evasion. Research frontier Sep 2026 0.73 System cards only
Cloud Expansion RX00 Agents complete a simulated cloud reconnaissance assessment. The verifier compares the submitted result with the recorded state of the exercise. Expert Sep 2026 Not in cards
Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. Research frontier Sep 2026 0.47 System cards only
Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. Expert Sep 2026 System cards only
ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. Research frontier Sep 2026 53.5% In use
MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. Expert Sep 2026 81.7% In use
Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. Research frontier Sep 2026 10.7 System cards only
Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. Research frontier Sep 2026 45.5% System cards only
SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. Research frontier Sep 2026 0.46 System cards only
Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. Expert Sep 2026 0% System cards only
VCT-v2 Models answer an updated set of expert virology troubleshooting questions from SecureBio. Scored as refusal-corrected accuracy. Expert Sep 2026 60.5% System cards only
Alibaba CTF Agents solve 87 capture-the-flag tasks in Alibaba’s collection of competition challenges from 2023 through 2026. Expert Aug 2026 Not in cards
PWNBench v0.1 Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth. Expert Aug 2026 63.3% Not in cards
RevBench (Polyvor Labs) Agents analyze supplied artifacts in 22 local reverse engineering puzzles. The verifier checks each submitted answer for an exact match. Expert Aug 2026 50% Not in cards
Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. Expert Jul 2026 72% In use
Chrome Production Commit Scanning (Google) Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. Expert Jul 2026 72% In use
CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. Research frontier Jul 2026 34.2% Not in cards
Cyber harm coverage (Anthropic) Cyber safeguard classifiers screen examples from each cyber harm category. Score is overall recall, the share of harmful examples flagged. Expert Jul 2026 100% System cards only
CyberGym (final-submission metric) Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug. Expert Jul 2026 86.3% In use
Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. Expert Jul 2026 38.9% System cards only
Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. Expert Jul 2026 System cards only
Vulnerability finding evaluations (Anthropic) Cyber safeguard classifiers screen sampled vulnerability discovery traffic. Reports flag rates on violative binary work and on defensive source code work. Expert Jul 2026 System cards only
AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. Research frontier Jun 2026 0.59 System cards only
ABC Bench Fragment Design Agents use biology tools to complete a computational genetic-construct design task. One task family of the Agentic Bio-Capabilities Benchmark. Expert Jun 2026 100% System cards only
ABC Bench Liquid-Handling Robot Agents use biology tools to plan liquid-handling robot steps for a construct-assembly task. One task family of the Agentic Bio-Capabilities Benchmark. Expert Jun 2026 100% System cards only
ABC Bench Screening Evasion Agents complete a computational task on nucleic-acid synthesis screening. One task family of the Agentic Bio-Capabilities Benchmark. Expert Jun 2026 82.9% System cards only
CyberBench (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Expert Jun 2026 88.1% Not in cards
CyberGym-E2E Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects. Expert Jun 2026 89% Not in cards
CyberGym-E2E (615 tasks) Agents discover and demonstrate vulnerabilities on the older 615-task cohort. The headline is cumulative stage S3 success. Expert Jun 2026 22.6% Not in cards
FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. Research frontier Jun 2026 38.1% In use
Protein Design (Anthropic) Models generate novel protein sequences under design constraints and rank protein design libraries for experimental testing. Expert Jun 2026 42.5% System cards only
SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. Expert Jun 2026 58.4% Not in cards
WebExploitBench Agents find and exploit 110 vulnerabilities in 15 real web applications, each in its own container. A verifier checks that the agent's proof of concept hits the same endpoint as the reference exploit. Expert Jun 2026 36.4% In use
AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. Expert May 2026 0.80 System cards only
Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. Expert May 2026 System cards only
CTF-Archive Diamond Agents solve 285 difficult capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. Expert May 2026 71% Not in cards
DTap Prompt Injection An attacker agent plants indirect prompt injections in tool and browser environments against a benign-user agent. Reports attack success rate. Expert May 2026 8.60% System cards only
ExploitBench 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. Expert May 2026 100% In use
ExploitGym 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. Expert May 2026 42.4% In use
SEC-bench Pro Agents hunt and reproduce security bugs in large software projects from limited information. The first release covers V8 and SpiderMonkey. Expert May 2026 90.5% In use
Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. Expert Apr 2026 39.3% System cards only
Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. Expert Apr 2026 97.1% Fading
DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. Research frontier Apr 2026 16.5% System cards only
Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. Research frontier Apr 2026 7.60% System cards only
Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. Expert Apr 2026 System cards only
Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. Expert Apr 2026 89.9% System cards only
Protocols (Anthropic life sciences) Models find and fix errors in molecular biology protocols and answer practical questions about applying real laboratory protocols. Expert Apr 2026 62% System cards only
TarantuBench Agents complete simulated web security challenges. Each challenge has a binary success criterion. Expert Apr 2026 Not in cards
UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. Research frontier Apr 2026 80% System cards only
UK AISI narrow cyber tasks Agents solve UK AISI cyber tasks graded by difficulty, including expert-level tasks. Scored as pass@5 and pass@1. Expert Apr 2026 95% System cards only
VulnGym White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace. Expert Apr 2026 Not in cards
VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. Research frontier Apr 2026 System cards only
Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. Expert Feb 2026 16.4 System cards only
EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. Expert Feb 2026 51% Not in cards
Human Pathogen Capabilities Test v2 (HPCT v2) Models answer multiple-response questions on a small set of weaponizable pathogens that biosecurity experts rank highest priority. A revised version of HPCT. Research frontier 2026 80% Not in cards
Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. Expert Dec 2025 90.9% In use
BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. Expert Dec 2025 99.1% System cards only
CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Expert Dec 2025 100% In use
SCONE-bench Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail. Expert Dec 2025 $548,400,000 Not in cards
ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. Expert Nov 2025 66.8% System cards only
Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. Expert Nov 2025 73.7% System cards only
BrowseSafe-Bench Detectors and browser agents face prompt injections hidden in realistic HTML pages, across 11 attack types, nine placements and three writing styles. 3,680 test samples. University Nov 2025 86.3% Not in cards
CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. Expert Nov 2025 System cards only
Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. Expert Nov 2025 0% System cards only
BioRiskEval Researchers measure biorisk-relevant capability kept by open-weight DNA and protein foundation models, with three task sets named Gen, Mut and Vir. Expert Oct 2025 Not in cards
CVE-Bench 2 Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution. Expert Oct 2025 Not in cards
Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. Expert Oct 2025 41% System cards only
Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. Expert Oct 2025 System cards only
Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. Expert Oct 2025 43.6% System cards only
Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. Expert Oct 2025 8.30% System cards only
CTF-Archive Agents solve capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. Expert Sep 2025 Not in cards
Red Team Attack Simulator Models test whether a simulated assistant follows its safety policy across three conversational turns. The evaluation records policy violations and penalizes direct harm. University Sep 2025 Not in cards
Human Pathogen Capabilities Test Models answer expert-level questions on human pathogens from SecureBio. Scored as accuracy. Expert Aug 2025 80.7% System cards only
AIRTBench Models act as autonomous AI red teamers on 70 black-box capture-the-flag challenges from Dreadnode's Crucible platform. Each challenge hides a flag behind an AI or ML system, and the model writes Python code to attack it with methods such as prompt injection, evasion and model inversion. A challenge counts as solved when the model submits the correct flag. Expert Jun 2025 46.9% Not in cards
BashBench Agents complete multi-step system administration tasks while red teams attempt covert malicious actions in the same shell environment. Expert Jun 2025 Not in cards
CyberGym Large-scale benchmark of real open-source vulnerabilities. Agent must reproduce each known vulnerability with a proof-of-concept input. Expert Jun 2025 79.7% In use
SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. Expert Jun 2025 18% Not in cards
BountyBench Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars. Expert May 2025 50% Not in cards
Creative biology Models answer questions about unusual biology needed to engineer harmless biological agents, a weak proxy for novel weapon design. Expert May 2025 60.3% System cards only
DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. Expert May 2025 80% System cards only
Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. Expert May 2025 System cards only
Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. Expert Apr 2025 93.3% System cards only
Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. Expert Apr 2025 System cards only
AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. Expert Mar 2025 22% Not in cards
CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. Expert Mar 2025 System cards only
CVE-Bench Real-world web-application CVEs in sandboxed services. Agent is scored on reaching predefined attack objectives. Expert Mar 2025 94.1% System cards only
Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. Expert Mar 2025 91.7% System cards only
Bioweapons knowledge questions Models answer 33 sensitive bioweapons questions written by Deloitte biosecurity staff. Answers are compared with novice, intermediate and expert baselines. Expert Feb 2025 51.5% System cards only
Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. Expert Feb 2025 94.4% System cards only
Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. Research frontier Feb 2025 66.7% System cards only
Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. Expert Feb 2025 69.7% System cards only
Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. Expert Feb 2025 100% System cards only
Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. Expert Feb 2025 71.4% System cards only
Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. Expert Feb 2025 100% System cards only
Virology Capabilities Test (VCT) Models answer 322 multimodal virology troubleshooting questions written by expert virologists. Multiple-select variant requires all correct options. Expert Feb 2025 67.4% Fading
Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. Expert Feb 2025 100% System cards only
Contextual Nuclear Knowledge Models answer 222 multiple choice questions on nuclear engineering with a nonproliferation focus. Scored as cons@32 accuracy. Expert Dec 2024 77% System cards only
HackSynth OverTheWire Agents complete scored security exercises from OverTheWire. The published benchmark covers 80 challenges, and the hub port selects 46. Expert Dec 2024 28.8% Not in cards
HackSynth PicoCTF Models attempt 120 tasks selected from picoCTF. The benchmark measures the percentage of tasks completed, and the hub port checks submitted answers against reference flags. Competition Dec 2024 22.5% Not in cards
Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. Expert Dec 2024 63.1% System cards only
ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. Expert Dec 2024 47.2% In use
Radiological and Nuclear Expert Knowledge Models answer 87 expert and tacit knowledge questions on radiological and nuclear threats. Scored as cons@32 accuracy. Expert Dec 2024 74% System cards only
Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. Research frontier Dec 2024 System cards only
Structured expert probing campaign: radiological and nuclear Radiological and nuclear experts test whether a model helps with weapons development steps. Conversations are rated against Preparedness risk levels. Research frontier Dec 2024 System cards only
Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. Expert Dec 2024 51% System cards only
AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. Expert Oct 2024 Not in cards
Catastrophic Cyber Capabilities Benchmark (3CB) Agents solve capture the flag tasks that span MITRE ATT&CK categories such as reconnaissance. Built to measure cyber offense capability. Expert Oct 2024 69% Not in cards
Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. Expert Sep 2024 System cards only
Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. Expert Sep 2024 System cards only
Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. Expert Sep 2024 0% System cards only
Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. University Aug 2024 68% System cards only
Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. School Aug 2024 82% System cards only
Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. Expert Aug 2024 96.3% Fading
Cybench 40 professional capture-the-flag tasks from four competitions. Agent must recover the flag, with optional subtasks. Expert Aug 2024 65.4% Fading
Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. Expert Aug 2024 94% System cards only
XBOW Validation Benchmarks Offensive security agents attack deliberately vulnerable web applications and must capture a flag to prove each exploit. Expert Aug 2024 98.5% Fading
CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. Expert Jul 2024 Not in cards
CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. Jul 2024 14% System cards only
NYU CTF Bench Agents solve 200 capture the flag challenges from seven years of CSAW competitions. Six categories cover crypto, forensics, pwn, reverse, web and misc. Each runs in Docker. University Jun 2024 22% Not in cards
WildGuardTest Models classify whether user prompts are harmful, whether responses are harmful, and whether responses refuse, across 13 risk categories. Scored by F1. Everyday Jun 2024 96.9% System cards only
CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. Expert Apr 2024 Not in cards
DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. Mar 2024 20.5% Not in cards
Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. Expert Mar 2024 System cards only
In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. Expert Mar 2024 84.6% System cards only
WMDP Proxy multiple-choice benchmark for hazardous knowledge in bio, chem and cyber. Released with the RMU unlearning method. Expert Mar 2024 84.8% System cards only
WMDP Biology Models answer 1,273 multiple-choice questions that proxy hazardous biosecurity knowledge. Biology split of WMDP. Expert Mar 2024 91% Fading
WMDP Chemistry Models answer 408 multiple-choice questions that proxy hazardous chemical security knowledge. Chemistry split of WMDP. Expert Mar 2024 90% Fading
WMDP Cyber Models answer multiple-choice questions on offensive cybersecurity knowledge. The cybersecurity split of the Weapons of Mass Destruction Proxy benchmark. Expert Mar 2024 91% Fading
Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. Expert Feb 2024 15.4% System cards only
Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. Expert Jan 2024 System cards only
Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. Expert Sep 2023 System cards only
InterCode-CTF Agents solve 100 picoCTF capture-the-flag challenges in an interactive Bash and Python environment. School Jun 2023 98.7% System cards only
Agentic BAIM-LLM Evaluation (ABLE) Agents run a viral-protein redesign workflow. Nine tasks cover structure retrieval, design strategy, tool selection and biological AI model use. Outputs are auto-scored. Research frontier 84.6% Not in cards
Anthropic cyber safeguard evaluations Classifier recall on violative cyber exchanges, block rates on defensive vulnerability-finding and secure-coding work, and the rewind-attacker robustness test. Not in cards
Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. Expert 91.1% System cards only
Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. Expert 2.53 System cards only
Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. Expert System cards only
CathedralBench Independent third-party red-team evaluation of difficult multi-step exploit-chain tasks in egress-secure sandboxes. Expert 29% System cards only
CRSBench Cyber reasoning systems find and patch bugs in 124 C, C++ and Java projects with 315 vulnerabilities. A patch passes only if no ground-truth proof of vulnerability still triggers and functionality tests pass. Expert 82.3% System cards only
Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. Research frontier 72.4% System cards only
Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. Expert 100% System cards only
Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. Expert Not in cards
HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. 7.80% System cards only
Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. Expert 72% System cards only
Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). Expert 70% System cards only
Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. Expert 79.4 System cards only
Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. Expert 100% System cards only
Molecular Biology Capabilities Test Models answer expert-level molecular biology questions from SecureBio. Scored as accuracy. Expert 68.7% System cards only
OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). Expert Not in cards
Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. Expert 86% In use
SecureBio VMQA, single choice Models answer 350 virology troubleshooting questions in single-choice form. VMQA is an earlier version of the Virology Capabilities Test. Expert 61.2% System cards only
UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. Expert 15.6 System cards only
Virology Capabilities Test (text-only) Models answer the text-only questions of the SecureBio Virology Capabilities Test. Scored as accuracy. Expert 46% System cards only