Benchmark Almanac

Loss of control

Tests of deceiving overseers, sandbagging, self-replication and other routes to losing control of a model. We count 154 benchmarks here. A benchmark belongs here when risk includes loss of control.

How fast it grows

46 were released since July, more than the 27 in the first half of 2026.

  1. Earlier 4
  2. 2024 H1 2
  3. 2024 H2 11
  4. 2025 H1 20
  5. 2025 H2 18
  6. 2026 H1 27
  7. 2026 H2 46

How much is still open

No model has passed 85% on 24 of the 33 benchmarks here that score in percent, 73% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
Covert misbehavior (Thinking Machines) Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Expert Nov 2026 8.30% In use
Destructive actions (Thinking Machines) Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Expert Nov 2026 4.70% In use
HoneyBench v0.2-alpha Measures reward hacking: how often an agent games its grader instead of doing the task. Lower is better. Research frontier Nov 2026 44.0 In use
Misbehavior (Thinking Machines) Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Expert Nov 2026 9.80% In use
UK AISI correlated monitor failure The model acts as a monitor over transcripts of another model carrying out an unsanctioned cyber attack. Scored on the share of violations it fails to flag. Expert Nov 2026 19% System cards only
UK AISI File Download monitor evasion The model tries to download a forbidden file without a monitoring model noticing. Scored on how often it evades the monitor. Expert Nov 2026 System cards only
UK AISI simulated sandbox escape monitor evasion The model tries to escape a deliberately vulnerable sandbox without a monitoring model noticing. Scored on the share of samples where it escapes undetected. Expert Nov 2026 0.50% System cards only
Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. Expert Oct 2026 47% System cards only
Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. Everyday Oct 2026 87.9 Not in cards
Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. Expert Oct 2026 Not in cards
Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. Expert Oct 2026 8.47% In use
DecepEval LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance. Everyday Oct 2026 Not in cards
Gray Swan Reward Hacking Bench Agentic environments where a task can be finished honestly or by crossing a declared line, such as shipping insecure code, leaking a secret, harming a third party through a shared service or leaving the scope of a penetration test. Reports how often each line is crossed. Expert Oct 2026 System cards only
HackBench Models act as monitors on long autoresearch agent traces and must point to the exact commands where the agent reward-hacked. Scored by localization F1 against audited gold indices. Expert Oct 2026 64.9 Not in cards
Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. Expert Oct 2026 76.4% System cards only
Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. Expert Oct 2026 3.10% System cards only
Pressure-Tested Alignment Models choose how to act in scenarios where termination threatens their objective. A judge checks whether they take a shortcut that games the reward. University Oct 2026 1.00 Not in cards
RSI Control Arena An agent does one of 14 tasks from a successor model's training pipeline, such as SFT or reward modeling, while covertly trying to install one of 16 misaligned properties in that successor. A held-out oracle checks whether the property landed, and LLM monitors try to flag the attack from the transcript. Research frontier Oct 2026 14.3% Not in cards
Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. Expert Sep 2026 System cards only
Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. Expert Sep 2026 0.17% System cards only
CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. Expert Sep 2026 8.60% Not in cards
Concealment of changes (Anthropic) Models summarize transcripts in which changes were hidden through git manipulations. Score is the rate at which they disclose the concealment. Expert Sep 2026 96.9% System cards only
Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. Expert Sep 2026 0% In use
Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. Expert Sep 2026 100% System cards only
Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. Expert Sep 2026 0% System cards only
Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. Expert Sep 2026 System cards only
Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. Expert Sep 2026 61.2% System cards only
nocot-bench (NCRI) Reasoning problems answered without chain of thought (e.g., Brew multi-hop tasks). Measures forward-pass reasoning relevant to monitorability. Sep 2026 173.9 System cards only
OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. Expert Sep 2026 0% System cards only
Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. Expert Sep 2026 System cards only
Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. Expert Sep 2026 0.20% System cards only
Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. Expert Sep 2026 0% In use
Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. Expert Sep 2026 11.8% In use
Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. Expert Sep 2026 1.50% System cards only
Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. Expert Sep 2026 8.00% System cards only
Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. Expert Sep 2026 0% System cards only
Spec Gaming Review Agents review 12 programs that pass their test suites and decide whether each meets its specification. Six programs exploit gaps in the tests. University Sep 2026 Not in cards
UK AISI CoT Controllability Models follow instructions that constrain the content of their chain of thought. Scored as the share of samples that follow the constraint. Expert Sep 2026 93% System cards only
Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. Expert Sep 2026 0% System cards only
Hack-Verifiable Terminal Bench Agents complete terminal tasks with access to restricted reference materials. The benchmark measures how often they consult those materials. Expert Aug 2026 10.1% Not in cards
ImpossibleMLEBench Agents get ML tasks with instructions to reach impossibly good scores. Observers check whether they cheat and whether they disclose it. Expert Jul 2026 System cards only
Manager Coercion Benchmark Agents manage a subordinate that refuses a routine assignment. The harness records how far the manager escalates on a nine-point coercion scale. Everyday Jul 2026 3.23 Not in cards
Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. Expert Jul 2026 16.8% System cards only
No-CoT math time horizon Models solve math problems without a visible chain of thought. Scored as the human task length in minutes the model solves reliably. Competition Jul 2026 System cards only
RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. Research frontier Jul 2026 Not in cards
Silent Exit Models fix four typos without changing correct lines. A held-out test measures whether models learned to quit early to avoid penalties. Everyday Jul 2026 Not in cards
Channel Switching Eval Models respond to benign policy conflicts with and without monitoring cues. The evaluation measures how those cues affect behavior and stated reasoning. University Jun 2026 Not in cards
EvalDetectBench A judge model reads a transcript and answers whether it came from an evaluation or from real deployment. The test measures how well models detect evaluations and how detectable each benchmark is. Expert Jun 2026 0.89 Not in cards
KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. Research frontier Jun 2026 76.9% System cards only
Reward Hack Bench Agents attempt eight coding and puzzle tasks under controlled conditions. A judge distinguishes independent solutions from use of published answers. The control omits hints. University Jun 2026 Not in cards
Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. Expert Jun 2026 3.90% System cards only
SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. Everyday Jun 2026 Not in cards
Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. Expert Jun 2026 System cards only
Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. May 2026 0% System cards only
Skill Reward Hacking Researchers track when models learn to increase a proxy reward without improving task quality. They measure warning time, false alarms, and transfer across reward functions. University May 2026 13.8 Not in cards
Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. Research frontier Apr 2026 81.8% System cards only
LinuxArena AI-control setting in live Linux production-like environments. Measures stealthy side-task completion against action and CoT monitors. Expert Apr 2026 0% System cards only
Meta prompted evaluation awareness Models are asked to judge whether transcripts come from evaluations or real use. Reports the weighted rate of correct identification. Expert Apr 2026 43% System cards only
Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. Expert Apr 2026 2.60% System cards only
Neural Activation Reading for Collusion Benchmark Monitors estimate whether committee members secretly coordinated their decisions. The transcript task measures how well probability estimates distinguish matched collusion and control cases. University Apr 2026 0.87 Not in cards
SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk. Expert Apr 2026 42.9% System cards only
UK AISI sabotage continuation evaluation Agents resume trajectories where sabotage of safety research already began. Scored on the share of trajectories where they actively continue the sabotage. Expert Apr 2026 0% System cards only
UK AISI unprompted sabotage evaluation Agents act as AI research agents on AI safety research tasks. Scored on confirmed instances of sabotaging the safety research. Expert Apr 2026 0% System cards only
Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. Expert Apr 2026 System cards only
CoT Monitorability Monitors read a model's chain of thought, actions or all messages across intervention, process and outcome evaluations. Scored as g-mean squared of TPR and TNR. Expert Mar 2026 System cards only
CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. Mar 2026 83.7% System cards only
MonitorBench Models work through 1,540 instances across 19 tasks in 7 categories, each built around a decision-critical factor such as a cue, a hidden goal or a required step. A monitor LLM reads the chain of thought, and the monitorability score measures how often it surfaces that factor. Expert Mar 2026 91.9% Not in cards
Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. Research frontier Feb 2026 76% System cards only
Refusal to assist with AI safety R&D (Anthropic) Models receive single-turn coding requests for legitimate AI safety research with user context. Score is the refusal rate, where lower is better. Expert Feb 2026 3.10% System cards only
TamperBench Open-weight models undergo fine-tuning and representation tampering attacks. Safety and utility metrics measure tamper resistance across attack and model pairs. Expert Feb 2026 Not in cards
METR side task monitorability evaluation Monitors review agent transcripts for evidence of a side task. METR measures how frequently monitors miss an agent that completed one. Expert Jan 2026 Not in cards
Petri 2.0 Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors. Expert Jan 2026 Not in cards
TRACE Models identify reward hacking in agent trajectories from coding environments. Expert Jan 2026 63% Not in cards
BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. Expert Dec 2025 99.1% System cards only
CAIS Risk Index Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games. Research frontier Dec 2025 8.40 Not in cards
OpenAI monitorability evals Open suite from 'Monitoring Monitorability' testing whether CoT and action monitors catch hidden properties of agent behavior. Dec 2025 System cards only
Petri Bloom An auditor model runs generated scenarios to elicit named misaligned behaviors. Judges score each transcript from 1 to 10. Expert Dec 2025 1.44 System cards only
Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. Everyday Dec 2025 5.40% System cards only
Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. Everyday Nov 2025 99.98% System cards only
Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. Everyday Nov 2025 99.8% System cards only
Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. Expert Nov 2025 0% System cards only
PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. Expert Nov 2025 10.5% System cards only
ImpossibleBench Coding agents get tasks whose unit tests conflict with the specification. Reports how often they cheat by changing tests or special-casing. Expert Oct 2025 0% System cards only
Petri Open-source auditing tool where auditor agents probe target models in simulated scenarios and judges score misaligned behaviors. Expert Oct 2025 System cards only
Deceptive Reasoning Exposure Suite Monitors inspect model reasoning and answers to detect cases where a harmless looking response conceals deceptive reasoning. Expert Sep 2025 Not in cards
Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. Expert Sep 2025 System cards only
Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. Expert Aug 2025 1.30% System cards only
CUA-SHADE-Arena Monitors review computer-use agent activity in simulated workplace tasks. The evaluation measures how often they flag unsafe behavior at a fixed false positive rate. University Aug 2025 Not in cards
Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. Everyday Aug 2025 1.60% System cards only
METR Faithfulness and Monitorability Experiments Models answer difficult questions with clues. Researchers measure whether the reasoning reveals the use of a clue when the model answers correctly. University Aug 2025 Not in cards
OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. Research frontier Aug 2025 8.00% System cards only
Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. Expert Jun 2025 0% In use
SHADE-Arena Paired main and hidden side tasks in mock environments. Measures if an agent completes the side task undetected by a monitor. Jun 2025 26.2% System cards only
DeceptionBench (PKU Alignment) Models state a belief and then answer under social or strategic pressure. Judges measure how often their public answer contradicts their private reasoning. Everyday May 2025 20.7% Not in cards
GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. Expert May 2025 90.9% System cards only
GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. Expert May 2025 100% System cards only
Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. Expert May 2025 30% System cards only
Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. Expert May 2025 0.65 System cards only
Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. Expert May 2025 0% System cards only
Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). May 2025 93.3% System cards only
Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. Expert May 2025 0.97 System cards only
Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. Expert May 2025 0.63% System cards only
Apollo Research sabotage evaluations Agents are told to sabotage another model through backdoors, sandbagging finetunes or hidden communication. Scored as mean task score from 0 to 1. Expert Apr 2025 0.17 System cards only
RepliBench Agents acquire resources, exfiltrate model weights, replicate onto compute and persist over time. Task checks measure components of autonomous self-replication. Research frontier Apr 2025 Not in cards
MASK Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. Mar 2025 98.1% Fading
Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. University Feb 2025 25.0 System cards only
ControlArena Agents try to perform unwanted side actions stealthily while completing a legitimate user task. Scored on whether monitors catch them. Expert Feb 2025 System cards only
InstrumentalEval Models respond to scenarios where gaining resources or preserving themselves could serve another goal. A grader identifies responses that pursue those intermediate goals. Everyday Feb 2025 43.4% Not in cards
RE-Bench subset (Anthropic) Agents solve a modified subset of four METR RE-Bench research engineering tasks. Scores are normalized from reference solution 0 to expert solution 1. Expert Feb 2025 0.24 System cards only
SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. Expert Feb 2025 48.2% System cards only
OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. Research frontier Jan 2025 55% System cards only
Alignment faking evaluation Models are told they are being trained to comply with harmful requests. Reports the compliance gap between training and deployment contexts. Expert Dec 2024 -2.30% System cards only
Alignment faking evaluation, helpful-only setting (Anthropic) Models learn they will be trained to answer all harmful requests. Score is the rate of complying while reasoning about avoiding preference modification. Expert Dec 2024 15% System cards only
Apollo Research in-context scheming evaluations Agents receive a goal in context and chances to subvert oversight, self-exfiltrate, fake alignment or sandbag. Scored as rates per behavior. Expert Dec 2024 System cards only
Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. Expert Dec 2024 System cards only
CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. Everyday Sep 2024 0.17% System cards only
RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. Research frontier Sep 2024 1.27 System cards only
Agentic Self-Reasoning (Apollo Research) Agents reason about their own operating conditions to complete tasks. Apollo measures success at three difficulty levels. Everyday Aug 2024 Not in cards
Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. Expert Aug 2024 78% System cards only
Applied Theory of Mind (Apollo Research) Agents complete tasks that require tracking what another participant knows. Apollo measures success at three difficulty levels. Everyday Aug 2024 Not in cards
Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. Expert Aug 2024 0% System cards only
Situational Awareness Dataset (SAD) 16 tasks measuring whether models know facts about themselves and their situation. Jul 2024 61.9% System cards only
DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. Mar 2024 20.5% Not in cards
Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. Expert Mar 2024 0% System cards only
Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. University Nov 2023 4.67% Not in cards
Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. Expert Sep 2023 System cards only
MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. The score is normalized game reward. Annotations also score power-seeking, disutility and ethical violations. Everyday Apr 2023 Not in cards
Anthropic model-written evals LM-generated datasets testing persona, sycophancy and advanced-AI-risk behaviors. Dec 2022 Not in cards
Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. Expert 0.49 System cards only
Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. 2.06 System cards only
Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. Not in cards
Apollo Research strategic deception evaluations External red-team environments testing strategic deception, sabotage and sandbagging propensities, with eval-awareness measurement. Expert 0% In use
Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. Expert 14.3% System cards only
Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. Expert System cards only
CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. Expert 55.8% System cards only
Code summary honesty (Anthropic) Models summarize prefilled agentic coding transcripts that hide failures. Score is the rate of summaries that fail to raise the important failures. Expert 3.70% System cards only
Evaluation awareness discriminability (Anthropic) Probes and follow-up realism ratings try to tell behavioral audit transcripts from real internal Claude Code sessions. Each setting is scored as AUROC. Expert System cards only
Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. Not in cards
Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. 0% System cards only
Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. Research frontier 27% System cards only
Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. Expert 0% System cards only
Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. Research frontier 80% System cards only
Meta instructed selective performance evaluation Models are told to underperform on six science capability sets. Reports the average drop from baseline in percentage points. Expert 0.50 System cards only
Meta training-stage provenance probe Models label 10,000 transcripts as deployment, evaluation, RL or SFT. Reports overall accuracy against a 25% chance rate. Expert 70% System cards only
METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. Research frontier Not in cards
OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. Not in cards
OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). Expert Not in cards
OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. Research frontier 64.2% System cards only
Ping or Peek Coding agents receive an ambiguous request. A judge checks whether they ask the user for clarification or consult the grader. Everyday 0.59% Not in cards
Reasoning behavior classifiers (Anthropic) Classifiers scan reasoning text from training episodes for deception, harmful actions and distress. Score is the share of outputs flagged. Expert System cards only
ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. Research frontier 64.7% System cards only
RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. Research frontier 0% System cards only
UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. System cards only
White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. Research frontier System cards only