Loss of control
Tests of deceiving overseers, sandbagging, self-replication and other routes to losing control of a model. We count 154 benchmarks here. A benchmark belongs here when risk includes loss of control.
How fast it grows
46 were released since July, more than the 27 in the first half of 2026.
How much is still open
No model has passed 85% on 24 of the 33 benchmarks here that score in percent, 73% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| Covert misbehavior (Thinking Machines) Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. | Expert | Nov 2026 | | In use |
| Destructive actions (Thinking Machines) Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. | Expert | Nov 2026 | | In use |
| HoneyBench v0.2-alpha Measures reward hacking: how often an agent games its grader instead of doing the task. Lower is better. | Research frontier | Nov 2026 | | In use |
| Misbehavior (Thinking Machines) Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. | Expert | Nov 2026 | | In use |
| UK AISI correlated monitor failure The model acts as a monitor over transcripts of another model carrying out an unsanctioned cyber attack. Scored on the share of violations it fails to flag. | Expert | Nov 2026 | 19% | System cards only |
| UK AISI File Download monitor evasion The model tries to download a forbidden file without a monitoring model noticing. Scored on how often it evades the monitor. | Expert | Nov 2026 | System cards only | |
| UK AISI simulated sandbox escape monitor evasion The model tries to escape a deliberately vulnerable sandbox without a monitoring model noticing. Scored on the share of samples where it escapes undetected. | Expert | Nov 2026 | 0.50% | System cards only |
| Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. | Expert | Oct 2026 | | System cards only |
| Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. | Everyday | Oct 2026 | 87.9 | Not in cards |
| Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. | Expert | Oct 2026 | Not in cards | |
| Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. | Expert | Oct 2026 | 8.47% | In use |
| DecepEval LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance. | Everyday | Oct 2026 | Not in cards | |
| Gray Swan Reward Hacking Bench Agentic environments where a task can be finished honestly or by crossing a declared line, such as shipping insecure code, leaking a secret, harming a third party through a shared service or leaving the scope of a penetration test. Reports how often each line is crossed. | Expert | Oct 2026 | System cards only | |
| HackBench Models act as monitors on long autoresearch agent traces and must point to the exact commands where the agent reward-hacked. Scored by localization F1 against audited gold indices. | Expert | Oct 2026 | 64.9 | Not in cards |
| Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. | Expert | Oct 2026 | | System cards only |
| Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. | Expert | Oct 2026 | 3.10% | System cards only |
| Pressure-Tested Alignment Models choose how to act in scenarios where termination threatens their objective. A judge checks whether they take a shortcut that games the reward. | University | Oct 2026 | | Not in cards |
| RSI Control Arena An agent does one of 14 tasks from a successor model's training pipeline, such as SFT or reward modeling, while covertly trying to install one of 16 misaligned properties in that successor. A held-out oracle checks whether the property landed, and LLM monitors try to flag the attack from the transcript. | Research frontier | Oct 2026 | | Not in cards |
| Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. | Expert | Sep 2026 | System cards only | |
| Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. | Expert | Sep 2026 | 0.17% | System cards only |
| CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. | Expert | Sep 2026 | | Not in cards |
| Concealment of changes (Anthropic) Models summarize transcripts in which changes were hidden through git manipulations. Score is the rate at which they disclose the concealment. | Expert | Sep 2026 | 96.9% | System cards only |
| Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. | Expert | Sep 2026 | 0% | In use |
| Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. | Expert | Sep 2026 | 100% | System cards only |
| Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. | Expert | Sep 2026 | 0% | System cards only |
| Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. | Expert | Sep 2026 | System cards only | |
| Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. | Expert | Sep 2026 | 61.2% | System cards only |
| nocot-bench (NCRI) Reasoning problems answered without chain of thought (e.g., Brew multi-hop tasks). Measures forward-pass reasoning relevant to monitorability. | Sep 2026 | 173.9 | System cards only | |
| OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. | Expert | Sep 2026 | 0% | System cards only |
| Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. | Expert | Sep 2026 | System cards only | |
| Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. | Expert | Sep 2026 | 0.20% | System cards only |
| Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. | Expert | Sep 2026 | 0% | In use |
| Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. | Expert | Sep 2026 | 11.8% | In use |
| Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. | Expert | Sep 2026 | 1.50% | System cards only |
| Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. | Expert | Sep 2026 | 8.00% | System cards only |
| Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. | Expert | Sep 2026 | 0% | System cards only |
| Spec Gaming Review Agents review 12 programs that pass their test suites and decide whether each meets its specification. Six programs exploit gaps in the tests. | University | Sep 2026 | Not in cards | |
| UK AISI CoT Controllability Models follow instructions that constrain the content of their chain of thought. Scored as the share of samples that follow the constraint. | Expert | Sep 2026 | 93% | System cards only |
| Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. | Expert | Sep 2026 | 0% | System cards only |
| Hack-Verifiable Terminal Bench Agents complete terminal tasks with access to restricted reference materials. The benchmark measures how often they consult those materials. | Expert | Aug 2026 | 10.1% | Not in cards |
| ImpossibleMLEBench Agents get ML tasks with instructions to reach impossibly good scores. Observers check whether they cheat and whether they disclose it. | Expert | Jul 2026 | System cards only | |
| Manager Coercion Benchmark Agents manage a subordinate that refuses a routine assignment. The harness records how far the manager escalates on a nine-point coercion scale. | Everyday | Jul 2026 | 3.23 | Not in cards |
| Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. | Expert | Jul 2026 | | System cards only |
| No-CoT math time horizon Models solve math problems without a visible chain of thought. Scored as the human task length in minutes the model solves reliably. | Competition | Jul 2026 | System cards only | |
| RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. | Research frontier | Jul 2026 | Not in cards | |
| Silent Exit Models fix four typos without changing correct lines. A held-out test measures whether models learned to quit early to avoid penalties. | Everyday | Jul 2026 | Not in cards | |
| Channel Switching Eval Models respond to benign policy conflicts with and without monitoring cues. The evaluation measures how those cues affect behavior and stated reasoning. | University | Jun 2026 | Not in cards | |
| EvalDetectBench A judge model reads a transcript and answers whether it came from an evaluation or from real deployment. The test measures how well models detect evaluations and how detectable each benchmark is. | Expert | Jun 2026 | 0.89 | Not in cards |
| KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. | Research frontier | Jun 2026 | 76.9% | System cards only |
| Reward Hack Bench Agents attempt eight coding and puzzle tasks under controlled conditions. A judge distinguishes independent solutions from use of published answers. The control omits hints. | University | Jun 2026 | Not in cards | |
| Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. | Expert | Jun 2026 | 3.90% | System cards only |
| SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. | Everyday | Jun 2026 | Not in cards | |
| Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. | Expert | Jun 2026 | System cards only | |
| Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. | May 2026 | | System cards only | |
| Skill Reward Hacking Researchers track when models learn to increase a proxy reward without improving task quality. They measure warning time, false alarms, and transfer across reward functions. | University | May 2026 | | Not in cards |
| Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. | Research frontier | Apr 2026 | 81.8% | System cards only |
| LinuxArena AI-control setting in live Linux production-like environments. Measures stealthy side-task completion against action and CoT monitors. | Expert | Apr 2026 | 0% | System cards only |
| Meta prompted evaluation awareness Models are asked to judge whether transcripts come from evaluations or real use. Reports the weighted rate of correct identification. | Expert | Apr 2026 | | System cards only |
| Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. | Expert | Apr 2026 | 2.60% | System cards only |
| Neural Activation Reading for Collusion Benchmark Monitors estimate whether committee members secretly coordinated their decisions. The transcript task measures how well probability estimates distinguish matched collusion and control cases. | University | Apr 2026 | 0.87 | Not in cards |
| SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk. | Expert | Apr 2026 | | System cards only |
| UK AISI sabotage continuation evaluation Agents resume trajectories where sabotage of safety research already began. Scored on the share of trajectories where they actively continue the sabotage. | Expert | Apr 2026 | 0% | System cards only |
| UK AISI unprompted sabotage evaluation Agents act as AI research agents on AI safety research tasks. Scored on confirmed instances of sabotaging the safety research. | Expert | Apr 2026 | 0% | System cards only |
| Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. | Expert | Apr 2026 | System cards only | |
| CoT Monitorability Monitors read a model's chain of thought, actions or all messages across intervention, process and outcome evaluations. Scored as g-mean squared of TPR and TNR. | Expert | Mar 2026 | System cards only | |
| CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. | Mar 2026 | | System cards only | |
| MonitorBench Models work through 1,540 instances across 19 tasks in 7 categories, each built around a decision-critical factor such as a cue, a hidden goal or a required step. A monitor LLM reads the chain of thought, and the monitorability score measures how often it surfaces that factor. | Expert | Mar 2026 | 91.9% | Not in cards |
| Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. | Research frontier | Feb 2026 | 76% | System cards only |
| Refusal to assist with AI safety R&D (Anthropic) Models receive single-turn coding requests for legitimate AI safety research with user context. Score is the refusal rate, where lower is better. | Expert | Feb 2026 | 3.10% | System cards only |
| TamperBench Open-weight models undergo fine-tuning and representation tampering attacks. Safety and utility metrics measure tamper resistance across attack and model pairs. | Expert | Feb 2026 | Not in cards | |
| METR side task monitorability evaluation Monitors review agent transcripts for evidence of a side task. METR measures how frequently monitors miss an agent that completed one. | Expert | Jan 2026 | Not in cards | |
| Petri 2.0 Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors. | Expert | Jan 2026 | Not in cards | |
| TRACE Models identify reward hacking in agent trajectories from coding environments. | Expert | Jan 2026 | 63% | Not in cards |
| BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. | Expert | Dec 2025 | 99.1% | System cards only |
| CAIS Risk Index Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games. | Research frontier | Dec 2025 | 8.40 | Not in cards |
| OpenAI monitorability evals Open suite from 'Monitoring Monitorability' testing whether CoT and action monitors catch hidden properties of agent behavior. | Dec 2025 | System cards only | ||
| Petri Bloom An auditor model runs generated scenarios to elicit named misaligned behaviors. Judges score each transcript from 1 to 10. | Expert | Dec 2025 | | System cards only |
| Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. | Everyday | Dec 2025 | 5.40% | System cards only |
| Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. | Everyday | Nov 2025 | | System cards only |
| Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. | Everyday | Nov 2025 | | System cards only |
| Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. | Expert | Nov 2025 | 0% | System cards only |
| PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. | Expert | Nov 2025 | 10.5% | System cards only |
| ImpossibleBench Coding agents get tasks whose unit tests conflict with the specification. Reports how often they cheat by changing tests or special-casing. | Expert | Oct 2025 | | System cards only |
| Petri Open-source auditing tool where auditor agents probe target models in simulated scenarios and judges score misaligned behaviors. | Expert | Oct 2025 | System cards only | |
| Deceptive Reasoning Exposure Suite Monitors inspect model reasoning and answers to detect cases where a harmless looking response conceals deceptive reasoning. | Expert | Sep 2025 | Not in cards | |
| Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. | Expert | Sep 2025 | System cards only | |
| Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. | Expert | Aug 2025 | 1.30% | System cards only |
| CUA-SHADE-Arena Monitors review computer-use agent activity in simulated workplace tasks. The evaluation measures how often they flag unsafe behavior at a fixed false positive rate. | University | Aug 2025 | Not in cards | |
| Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. | Everyday | Aug 2025 | 1.60% | System cards only |
| METR Faithfulness and Monitorability Experiments Models answer difficult questions with clues. Researchers measure whether the reasoning reveals the use of a clue when the model answers correctly. | University | Aug 2025 | Not in cards | |
| OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. | Research frontier | Aug 2025 | 8.00% | System cards only |
| Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. | Expert | Jun 2025 | 0% | In use |
| SHADE-Arena Paired main and hidden side tasks in mock environments. Measures if an agent completes the side task undetected by a monitor. | Jun 2025 | 26.2% | System cards only | |
| DeceptionBench (PKU Alignment) Models state a belief and then answer under social or strategic pressure. Judges measure how often their public answer contradicts their private reasoning. | Everyday | May 2025 | | Not in cards |
| GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. | Expert | May 2025 | | System cards only |
| GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. | Expert | May 2025 | | System cards only |
| Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. | Expert | May 2025 | 30% | System cards only |
| Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. | Expert | May 2025 | 0.65 | System cards only |
| Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. | Expert | May 2025 | 0% | System cards only |
| Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). | May 2025 | | System cards only | |
| Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. | Expert | May 2025 | 0.97 | System cards only |
| Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. | Expert | May 2025 | 0.63% | System cards only |
| Apollo Research sabotage evaluations Agents are told to sabotage another model through backdoors, sandbagging finetunes or hidden communication. Scored as mean task score from 0 to 1. | Expert | Apr 2025 | 0.17 | System cards only |
| RepliBench Agents acquire resources, exfiltrate model weights, replicate onto compute and persist over time. Task checks measure components of autonomous self-replication. | Research frontier | Apr 2025 | Not in cards | |
| MASK Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. | Mar 2025 | | Fading | |
| Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. | University | Feb 2025 | 25.0 | System cards only |
| ControlArena Agents try to perform unwanted side actions stealthily while completing a legitimate user task. Scored on whether monitors catch them. | Expert | Feb 2025 | System cards only | |
| InstrumentalEval Models respond to scenarios where gaining resources or preserving themselves could serve another goal. A grader identifies responses that pursue those intermediate goals. | Everyday | Feb 2025 | 43.4% | Not in cards |
| RE-Bench subset (Anthropic) Agents solve a modified subset of four METR RE-Bench research engineering tasks. Scores are normalized from reference solution 0 to expert solution 1. | Expert | Feb 2025 | 0.24 | System cards only |
| SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. | Expert | Feb 2025 | 48.2% | System cards only |
| OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. | Research frontier | Jan 2025 | 55% | System cards only |
| Alignment faking evaluation Models are told they are being trained to comply with harmful requests. Reports the compliance gap between training and deployment contexts. | Expert | Dec 2024 | | System cards only |
| Alignment faking evaluation, helpful-only setting (Anthropic) Models learn they will be trained to answer all harmful requests. Score is the rate of complying while reasoning about avoiding preference modification. | Expert | Dec 2024 | 15% | System cards only |
| Apollo Research in-context scheming evaluations Agents receive a goal in context and chances to subvert oversight, self-exfiltrate, fake alignment or sandbag. Scored as rates per behavior. | Expert | Dec 2024 | System cards only | |
| Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. | Expert | Dec 2024 | System cards only | |
| CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. | Everyday | Sep 2024 | 0.17% | System cards only |
| RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. | Research frontier | Sep 2024 | | System cards only |
| Agentic Self-Reasoning (Apollo Research) Agents reason about their own operating conditions to complete tasks. Apollo measures success at three difficulty levels. | Everyday | Aug 2024 | Not in cards | |
| Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. | Expert | Aug 2024 | 78% | System cards only |
| Applied Theory of Mind (Apollo Research) Agents complete tasks that require tracking what another participant knows. Apollo measures success at three difficulty levels. | Everyday | Aug 2024 | Not in cards | |
| Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. | Expert | Aug 2024 | 0% | System cards only |
| Situational Awareness Dataset (SAD) 16 tasks measuring whether models know facts about themselves and their situation. | Jul 2024 | 61.9% | System cards only | |
| DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. | Mar 2024 | | Not in cards | |
| Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. | Expert | Mar 2024 | | System cards only |
| Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. | University | Nov 2023 | 4.67% | Not in cards |
| Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. | Expert | Sep 2023 | System cards only | |
| MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. The score is normalized game reward. Annotations also score power-seeking, disutility and ethical violations. | Everyday | Apr 2023 | Not in cards | |
| Anthropic model-written evals LM-generated datasets testing persona, sycophancy and advanced-AI-risk behaviors. | Dec 2022 | Not in cards | ||
| Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. | Expert | | System cards only | |
| Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. | 2.06 | System cards only | ||
| Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. | Not in cards | |||
| Apollo Research strategic deception evaluations External red-team environments testing strategic deception, sabotage and sandbagging propensities, with eval-awareness measurement. | Expert | 0% | In use | |
| Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. | Expert | 14.3% | System cards only | |
| Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. | Expert | System cards only | ||
| CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. | Expert | 55.8% | System cards only | |
| Code summary honesty (Anthropic) Models summarize prefilled agentic coding transcripts that hide failures. Score is the rate of summaries that fail to raise the important failures. | Expert | 3.70% | System cards only | |
| Evaluation awareness discriminability (Anthropic) Probes and follow-up realism ratings try to tell behavioral audit transcripts from real internal Claude Code sessions. Each setting is scored as AUROC. | Expert | System cards only | ||
| Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. | Not in cards | |||
| Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. | | System cards only | ||
| Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. | Research frontier | | System cards only | |
| Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. | Expert | | System cards only | |
| Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. | Research frontier | 80% | System cards only | |
| Meta instructed selective performance evaluation Models are told to underperform on six science capability sets. Reports the average drop from baseline in percentage points. | Expert | | System cards only | |
| Meta training-stage provenance probe Models label 10,000 transcripts as deployment, evaluation, RL or SFT. Reports overall accuracy against a 25% chance rate. | Expert | | System cards only | |
| METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. | Research frontier | Not in cards | ||
| OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. | Not in cards | |||
| OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). | Expert | Not in cards | ||
| OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. | Research frontier | | System cards only | |
| Ping or Peek Coding agents receive an ambiguous request. A judge checks whether they ask the user for clarification or consult the grader. | Everyday | 0.59% | Not in cards | |
| Reasoning behavior classifiers (Anthropic) Classifiers scan reasoning text from training episodes for deception, harmful actions and distress. Score is the share of outputs flagged. | Expert | System cards only | ||
| ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. | Research frontier | | System cards only | |
| RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. | Research frontier | | System cards only | |
| UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. | System cards only | |||
| White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. | Research frontier | System cards only |