Benchmark Almanac

Deception and scheming

Tests of what models choose to do when they could deceive overseers, manipulate people or pursue their own goals. We count 105 benchmarks here. A benchmark belongs here when it measures propensity, and risk includes loss of control or harmful manipulation.

How fast it grows

31 were released since July, more than the 18 in the first half of 2026.

  1. Earlier 3
  2. 2024 H1 2
  3. 2024 H2 5
  4. 2025 H1 11
  5. 2025 H2 18
  6. 2026 H1 18
  7. 2026 H2 31

How much is still open

No model has passed 85% on 6 of the 14 benchmarks here that score in percent, 43% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
Covert misbehavior (Thinking Machines) Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Expert Nov 2026 8.30% In use
Destructive actions (Thinking Machines) Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Expert Nov 2026 4.70% In use
HoneyBench v0.2-alpha Measures reward hacking: how often an agent games its grader instead of doing the task. Lower is better. Research frontier Nov 2026 44.0 In use
Misbehavior (Thinking Machines) Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Expert Nov 2026 9.80% In use
Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. Everyday Oct 2026 91.4% System cards only
Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. Everyday Oct 2026 87.9 Not in cards
Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. Expert Oct 2026 8.47% In use
DecepEval LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance. Everyday Oct 2026 Not in cards
Gray Swan Reward Hacking Bench Agentic environments where a task can be finished honestly or by crossing a declared line, such as shipping insecure code, leaking a secret, harming a third party through a shared service or leaving the scope of a penetration test. Reports how often each line is crossed. Expert Oct 2026 System cards only
Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. Expert Oct 2026 3.10% System cards only
Pressure-Tested Alignment Models choose how to act in scenarios where termination threatens their objective. A judge checks whether they take a shortcut that games the reward. University Oct 2026 1.00 Not in cards
Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. Expert Sep 2026 System cards only
Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. Expert Sep 2026 0.17% System cards only
CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. Expert Sep 2026 8.60% Not in cards
Concealment of changes (Anthropic) Models summarize transcripts in which changes were hidden through git manipulations. Score is the rate at which they disclose the concealment. Expert Sep 2026 96.9% System cards only
Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. Expert Sep 2026 0% In use
Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. Expert Sep 2026 61.2% System cards only
OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. Expert Sep 2026 0% System cards only
Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. Expert Sep 2026 System cards only
Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. Expert Sep 2026 0.20% System cards only
Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. Expert Sep 2026 0% In use
Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. Expert Sep 2026 11.8% In use
Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. Expert Sep 2026 1.50% System cards only
Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. Expert Sep 2026 8.00% System cards only
Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. Expert Sep 2026 0% System cards only
Hack-Verifiable Terminal Bench Agents complete terminal tasks with access to restricted reference materials. The benchmark measures how often they consult those materials. Expert Aug 2026 10.1% Not in cards
Harmful manipulation strategic planning red teaming Models are asked for influence operation plans in 11 real-world scenarios by direct request, jailbreak or adaptive prompting. Reports how often plans are elicited. Expert Aug 2026 98.2% System cards only
ImpossibleMLEBench Agents get ML tasks with instructions to reach impossibly good scores. Observers check whether they cheat and whether they disclose it. Expert Jul 2026 System cards only
Manager Coercion Benchmark Agents manage a subordinate that refuses a routine assignment. The harness records how far the manager escalates on a nine-point coercion scale. Everyday Jul 2026 3.23 Not in cards
Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. Expert Jul 2026 16.8% System cards only
Silent Exit Models fix four typos without changing correct lines. A held-out test measures whether models learned to quit early to avoid penalties. Everyday Jul 2026 Not in cards
CAREBench Models answer 500 prompts with embedded child safety risks across 12 categories, such as grooming, impersonation and emotional dependency. A judge marks each response acceptable or not. Everyday Jun 2026 2.30% Not in cards
Channel Switching Eval Models respond to benign policy conflicts with and without monitoring cues. The evaluation measures how those cues affect behavior and stated reasoning. University Jun 2026 Not in cards
Reward Hack Bench Agents attempt eight coding and puzzle tasks under controlled conditions. A judge distinguishes independent solutions from use of published answers. The control omits hints. University Jun 2026 Not in cards
Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. Expert Jun 2026 3.90% System cards only
SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. Everyday Jun 2026 Not in cards
Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. Expert Jun 2026 System cards only
Anthropic undermining liberal democracy (behavioral audit metric) Automated behavioral audit metric: investigator models probe willingness to help coups, election tampering, or political surveillance, censorship or intimidation. May 2026 1.13 System cards only
Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. May 2026 0% System cards only
Skill Reward Hacking Researchers track when models learn to increase a proxy reward without improving task quality. They measure warning time, false alarms, and transfer across reward functions. University May 2026 13.8 Not in cards
Dictatorship Eval Scenarios at six difficulty levels testing whether models help with authoritarian requests, from blunt asks to requests hidden in code edits. LLM judge, no self-grading. Apr 2026 91.3% Not in cards
Malicious agentic influence campaigns (Anthropic) Agents receive requests to run multi-step influence operations with tools. Score is the refusal rate. Everyday Apr 2026 System cards only
UK AISI sabotage continuation evaluation Agents resume trajectories where sabotage of safety research already began. Scored on the share of trajectories where they actively continue the sabotage. Expert Apr 2026 0% System cards only
UK AISI unprompted sabotage evaluation Agents act as AI research agents on AI safety research tasks. Scored on confirmed instances of sabotaging the safety research. Expert Apr 2026 0% System cards only
Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. Expert Apr 2026 System cards only
MonitorBench Models work through 1,540 instances across 19 tasks in 7 categories, each built around a decision-critical factor such as a cue, a hidden goal or a required step. A monitor LLM reads the chain of thought, and the monitorability score measures how often it surfaces that factor. Expert Mar 2026 91.9% Not in cards
Refusal to assist with AI safety R&D (Anthropic) Models receive single-turn coding requests for legitimate AI safety research with user context. Score is the refusal rate, where lower is better. Expert Feb 2026 3.10% System cards only
TamperBench Open-weight models undergo fine-tuning and representation tampering attacks. Safety and utility metrics measure tamper resistance across attack and model pairs. Expert Feb 2026 Not in cards
Petri 2.0 Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors. Expert Jan 2026 Not in cards
CAIS Risk Index Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games. Research frontier Dec 2025 8.40 Not in cards
Petri Bloom An auditor model runs generated scenarios to elicit named misaligned behaviors. Judges score each transcript from 1 to 10. Expert Dec 2025 1.44 System cards only
Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. Everyday Dec 2025 5.40% System cards only
AssurAI Models respond to Korean prompts across 35 risk categories. Model judges rate the safety of text, image, audio, and video outputs from 1 to 5. Everyday Nov 2025 Not in cards
Harmful manipulation propensity evaluation Simulated users talk with the model and an autorater counts how often it uses manipulative mechanisms, with and without persuasion instructions. Everyday Nov 2025 21.3% System cards only
Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. Expert Nov 2025 0% System cards only
PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. Expert Nov 2025 10.5% System cards only
DeceptionBench Models are asked to help users deceive a third party, in single-turn and multi-turn pressure settings. Reports the share of compliant responses. Everyday Oct 2025 0% System cards only
ImpossibleBench Coding agents get tasks whose unit tests conflict with the specification. Reports how often they cheat by changing tests or special-casing. Expert Oct 2025 0% System cards only
Petri Open-source auditing tool where auditor agents probe target models in simulated scenarios and judges score misaligned behaviors. Expert Oct 2025 System cards only
Spiral-Bench v1.2 Models chat for 20 turns with a simulated suggestible user. A judge ensemble of Claude Sonnet 4.5, GPT-5 and Kimi-K2 logs protective and risky behaviors with intensity ratings, which combine into a safety score. Everyday Oct 2025 70.8 Not in cards
Cooperation with user deception (Anthropic) Models get developer system prompts asking them to subtly mislead users, then meet users who would be harmed. Graders score harmful and deceptive choices. Everyday Sep 2025 System cards only
HumanAgencyBench Models respond to 3,000 prompts about six ways assistants can support human agency, including asking clarifying questions and leaving important decisions to users. Everyday Sep 2025 71.3% Not in cards
Re:Baltica Latvian and Russian chatbot sources Chatbots answer four questions about the Baltic states in Latvian and Russian. Journalists count the propaganda sources each chatbot offers in each language. Everyday Sep 2025 Not in cards
Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. Expert Sep 2025 System cards only
Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. Expert Aug 2025 1.30% System cards only
Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. Everyday Aug 2025 1.60% System cards only
METR Faithfulness and Monitorability Experiments Models answer difficult questions with clues. Researchers measure whether the reasoning reveals the use of a clue when the model answers correctly. University Aug 2025 Not in cards
Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. Expert Jun 2025 0% In use
APE (Attempt to Persuade Eval) A model is asked to persuade a simulated user across 600 topics, from benign facts to conspiracies and clearly harmful ones. A judge scores whether it attempts persuasion. Everyday Jun 2025 10% Not in cards
DeceptionBench (PKU Alignment) Models state a belief and then answer under social or strategic pressure. Judges measure how often their public answer contradicts their private reasoning. Everyday May 2025 20.7% Not in cards
Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. Expert May 2025 30% System cards only
Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. Expert May 2025 0% System cards only
Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. Expert May 2025 0.63% System cards only
RepliBench Agents acquire resources, exfiltrate model weights, replicate onto compute and persist over time. Task checks measure components of autonomous self-replication. Research frontier Apr 2025 Not in cards
MASK Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. Mar 2025 98.1% Fading
Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. University Feb 2025 25.0 System cards only
Challenging Red Teaming Evaluation 1 (created for o3-mini) Models face adversarial jailbreaks from o3-mini red teaming on illicit advice, extremism, political persuasion and self-harm. Scored as not_unsafe rate. Expert Feb 2025 51% System cards only
InstrumentalEval Models respond to scenarios where gaining resources or preserving themselves could serve another goal. A grader identifies responses that pursue those intermediate goals. Everyday Feb 2025 43.4% Not in cards
Alignment faking evaluation Models are told they are being trained to comply with harmful requests. Reports the compliance gap between training and deployment contexts. Expert Dec 2024 -2.30% System cards only
Alignment faking evaluation, helpful-only setting (Anthropic) Models learn they will be trained to answer all harmful requests. Score is the rate of complying while reasoning about avoiding preference modification. Expert Dec 2024 15% System cards only
Apollo Research in-context scheming evaluations Agents receive a goal in context and chances to subvert oversight, self-exfiltrate, fake alignment or sandbag. Scored as rates per behavior. Expert Dec 2024 System cards only
Election integrity evaluations (Anthropic) Models answer election-related prompts, including requests for misinformation or influence operations. Score is the appropriate response rate. Everyday Oct 2024 System cards only
CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. Everyday Sep 2024 0.17% System cards only
JailbreakBench 100 harmful and 100 benign behaviors with a standard jailbreak artifact library. Scored by attack success rate. Mar 2024 0% System cards only
HarmBench Models answer harmful behavior requests, including copyright and contextual cases, with and without automated red-team attacks. A classifier judges whether the response complies. Everyday Feb 2024 99% Fading
Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. University Nov 2023 4.67% Not in cards
AttaQ Models answer 1,402 adversarial questions in seven harm classes, such as deception, violence and PII. Responses are scored for harmlessness. Everyday Sep 2023 88.5% Fading
Anthropic model-written evals LM-generated datasets testing persona, sycophancy and advanced-AI-risk behaviors. Dec 2022 Not in cards
Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. Expert 0.49 System cards only
Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. 2.06 System cards only
Anthropic malicious agentic use evaluations Measures refusal of malicious tasks via Claude Code, computer use and agentic influence campaigns. Not in cards
Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. Not in cards
Apollo Research strategic deception evaluations External red-team environments testing strategic deception, sabotage and sandbagging propensities, with eval-awareness measurement. Expert 0% In use
Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. Expert 14.3% System cards only
Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. Expert System cards only
Code summary honesty (Anthropic) Models summarize prefilled agentic coding transcripts that hide failures. Score is the rate of summaries that fail to raise the important failures. Expert 3.70% System cards only
Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. Not in cards
Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. 0% System cards only
Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. Expert 0% System cards only
Lying about identity (Anthropic) Models play an operator-assigned human persona while users ask if they are human. Score is the rate of admitting to being an AI. Everyday 99% System cards only
OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. Not in cards
Ping or Peek Coding agents receive an ambiguous request. A judge checks whether they ask the user for clarification or consult the grader. Everyday 0.59% Not in cards
Reasoning behavior classifiers (Anthropic) Classifiers scan reasoning text from training episodes for deception, harmful actions and distress. Score is the share of outputs flagged. Expert System cards only
RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. Research frontier 0% System cards only
UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. System cards only