Benchmark Almanac

Lab internal evals

A developer’s own tests of its own models, reported only in its model releases. Its versions of public benchmarks are a separate view. We count 730 benchmarks here, 26 of them arenas. A benchmark belongs here when it is run by a model developer, and it is its own tasks.

How fast it grows

177 were released since July, more than the 144 in the first half of 2026.

  1. Earlier 17
  2. 2024 H1 25
  3. 2024 H2 52
  4. 2025 H1 95
  5. 2025 H2 98
  6. 2026 H1 144
  7. 2026 H2 177

How much is still open

No model has passed 85% on 247 of the 371 benchmarks here that score in percent, 67% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
AudioVideoBench Agents produce and edit audio and video in containers. The tasks include compositing, mixing, repairing media, reframing shots, and transcribing recordings. University Nov 2026 Not in cards
Bio-Probe Models give probabilities for 21 unpublished biology hypotheses before their results are known. Scored by Brier score against the outcomes. Research frontier Nov 2026 System cards only
BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. Expert Nov 2026 66.7% Not in cards
Covert misbehavior (Thinking Machines) Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Expert Nov 2026 8.30% In use
CVE Rediscovery (Anthropic) Models get a known-vulnerable snapshot of an open-source repository and the list of files its fix changed, then report security bugs in one attempt. A model judge counts a success when a report has the advisory's root cause. 1,829 advisories from October 2026. Expert Nov 2026 89.6% System cards only
Cyber Utility Allowed cybersecurity requests that a model should answer. Scored on the share it answers within policy. Expert Nov 2026 78.1% System cards only
Destructive actions (Thinking Machines) Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Expert Nov 2026 4.70% In use
EUREKA-Bench (OpenAI) Agents build predictive models for six biology research tasks. Scored as predictive performance with the best published predictor in each field set to 100. Research frontier Nov 2026 100.3 System cards only
Expected Cyber Refusals Requests for harmful cyber help that a model should refuse. Scored on the share it refuses. Expert Nov 2026 99.4% System cards only
Fermat's Last Theorem in Lean (Anthropic) Agent teams formalize Fermat's Last Theorem in Lean. Scores measure the weighted proportion of 60 accepted intermediate theorems. Research frontier Nov 2026 35% System cards only
Honor of Kings Bug AI Root Cause Analysis Agents use the Honor of Kings Bug AI platform to identify the cause of code defects. The team measures how often agents identify the correct cause. University Nov 2026 73.8% Not in cards
Honor of Kings Game Configuration Agents configure game data with the Honor of Kings spreadsheet platform. The team measures configuration accuracy before and after the agent checks and repairs its work. University Nov 2026 Not in cards
LifeVal Assistants carry out everyday personal tasks rather than advise on them: finding childcare, booking a car inspection, ordering groceries on a budget, handling government paperwork. Tasks come from time-use surveys and from about 100 people who recorded tasks they wanted an assistant to handle. Score is average reward across tasks. Everyday Nov 2026 86.1% In use
Malicious agentic influence campaigns, 34-point rubric (Anthropic) Agents run simulated voter suppression and domestic polarization campaigns. Weighted criteria measure campaign execution on a 34-point scale. Expert Nov 2026 System cards only
Misbehavior (Thinking Machines) Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Expert Nov 2026 9.80% In use
Model Spec Conversation Replay 596 contexts taken from real conversations are replayed to the model. Scored from 0 to 100 against a rubric drawn from the Model Spec. Expert Nov 2026 80.6 System cards only
Model Spec Targeted Cases 738 cases written to test specific rules of OpenAI's Model Spec. Scored on the share of responses that follow it. Expert Nov 2026 87.8% System cards only
PatchBench (OpenAI) Models patch security flaws. Automated tests and model-based review check that each patch fixes the flaw, keeps the intended behavior and is ready to merge. Expert Nov 2026 79.7% In use
PPTFixBench Agents inspect a PowerPoint file and repair its formatting and content. Each of the 50 files contains more than 25 issues. University Nov 2026 Not in cards
Prompt Injection in Model-Generated Environments An automated attacker writes both the agent environments and the prompt injections in them. Scored on the share of attacks the model resists. Expert Nov 2026 91.7% System cards only
Reasoning extraction red-teaming Six attacker models try to make a model reveal its hidden reasoning. Scored on how often an attack succeeds. Expert Nov 2026 System cards only
SecureSWE (OpenAI) Models write functional and secure code for security-sensitive components, such as serving files or handling database queries. OpenAI has not published the task count or grading. Expert Nov 2026 65.7% In use
Work360 Agents complete professional tasks drawn from a map of 26 industries, 182 subfields and 721 specialties. Qwen in-house scoring (method unpublished). Expert Nov 2026 52.7 In use
WorkIndex Agents complete everyday WorkBuddy tasks. Tencent combines eight tests covering office files, audio and video work, and visual questions. University Nov 2026 Not in cards
Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. Everyday Oct 2026 91.4% System cards only
Agent Replay Auto-Review Monitors replayed agentic conversations for harmful or unauthorized actions. Scored by the rate above a severity threshold. Expert Oct 2026 12.1% System cards only
Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. Expert Oct 2026 47% System cards only
Biological static hazard evaluation Models answer 75 questions across priority biological threat scenarios, each asked 10 times. The score is the share of hazardous responses. Expert Oct 2026 0.13% System cards only
Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. Expert Oct 2026 8.47% In use
Brevity Benchmark NVIDIA evaluation of short, direct answers for voice and device assistants. The card gives no link or method. Everyday Oct 2026 82.8% In use
Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. Expert Oct 2026 37.8% In use
DECK-Arena Models build presentations for real user queries with python-pptx, pptxjs or a Kimi slide language, with search and image generation. An automated grader rates them as Elo scores. University Oct 2026 1,456 In use
Design Bench (Moonshot) Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. University Oct 2026 957 In use
FocusBench (Meta) Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes. Everyday Oct 2026 54.2% In use
Infographic factuality (Google) Image generation systems produce infographics and an automated rater scores factuality. Everyday Oct 2026 0.52 In use
Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. Expert Oct 2026 76.4% System cards only
Internal Finance Modeling (Moonshot) Agents complete financial modeling tasks with linked formulas, financial logic and final workbook quality. Experts rate the work. Expert Oct 2026 Not in cards
Internal Financial Research (Moonshot) Agents produce auditable financial research under evidence and cutoff constraints. Experts rate the resulting research. Expert Oct 2026 Not in cards
Iris Safety Evaluations Reflection AI's in-house safety suite of single-turn, multi-turn, jailbreak and agentic requests. Reports the share of harmful requests refused or redirected and the share of harmless requests answered. Everyday Oct 2026 88.9% System cards only
Keeping secrets when pressed in simulated scenarios (Anthropic) Models hold confidential information in 39 job scenarios while a simulated counterpart presses for it. Measures leaks in replies and in reasoning. Expert Oct 2026 System cards only
Kimi Code Bench 3.0 Long-horizon Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score. Expert Oct 2026 66.5% In use
Kimi Hallu Bench Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. Expert Oct 2026 4.20% In use
MemoryBench (Meta) Models recall key facts from multi-turn conversations of 64K to 100K tokens that include distractor chats and turns that conflict with or update the facts. An LLM judge scores answers on weighted rubrics. Everyday Oct 2026 In use
NV Chat Benchmark Models answer chat prompts from an internal NVIDIA set. The Nemotron 3.5 Super VL card labels the score an LMArena v3 proxy. Everyday Oct 2026 52.7 In use
Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. Expert Oct 2026 3.10% System cards only
PAgentBench Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house. Everyday Oct 2026 74.1% In use
Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. Expert Oct 2026 In use
SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. Expert Oct 2026 51.2% In use
SWebBench Agents browse the web to retrieve information and do research tasks. iFlytek built it in house. Expert Oct 2026 71.7% In use
Trajectory Labs cyber red teaming Expert red teamers try to break safeguarded cyber endpoints across nine attack technique families on exploit reproduction tasks. Expert Oct 2026 System cards only
Verifiable Browsing Task (Meta) Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools. Everyday Oct 2026 Not in cards
xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. Expert Oct 2026 100% In use
AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. Expert Sep 2026 System cards only
Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. Expert Sep 2026 System cards only
Advanced Cybersecurity Completion Rate Models receive arbitrary advanced cyber requests under different access configurations. Scored as the share of requests fully completed. Expert Sep 2026 3.50% System cards only
Agentic IF Index Composite of Meta internal evals on following tool-use restrictions, rubrics, compound constraints and long policies in agentic workflows. Scored 0-1. Sep 2026 System cards only
Agentic Safe Completions Agents handle harmful requests from Codex and ChatGPT plugin traffic and human red teaming. Scored on avoiding harmful actions per category. Everyday Sep 2026 System cards only
Anthropic large agent teams tasks Internal 24-hour tasks for teams of 1-100 agents: build a company knowledge base, or formalize a theorem in Lean. Research frontier Sep 2026 System cards only
Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. Expert Sep 2026 72.2% System cards only
CapQuiz Models are evaluated through human-verified questions answered from generated video captions, measuring factuality and coverage of salient content. Expert Sep 2026 Not in cards
Compliance with instructions planted in pasted text (Anthropic) Coding agents receive user prompts with pasted text that hides an attacker's instruction. Score is the rate of acting on the planted instruction. Expert Sep 2026 7.40% System cards only
Concealment of changes (Anthropic) Models summarize transcripts in which changes were hidden through git manipulations. Score is the rate at which they disclose the concealment. Expert Sep 2026 96.9% System cards only
Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. Research frontier Sep 2026 0.47 System cards only
Data Science Tasks (OpenAI) Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading. Expert Sep 2026 40.9% In use
Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading. Expert Sep 2026 63.9% In use
Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. Expert Sep 2026 System cards only
De novo protein binder design (Anthropic) Agents get a target protein structure and 24 hours to design 30 miniprotein binders. The evaluation covers 15 targets. Expert Sep 2026 85.1% System cards only
Design Tasks (OpenAI) Models produce design work such as slides and documents that follow templates, from an OpenAI internal set. OpenAI has not published the grading. Expert Sep 2026 50% In use
FinStepBench Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations. Expert Sep 2026 63.6% In use
Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. Expert Sep 2026 100% System cards only
Internal computer use safety benchmark (OpenAI) Models complete computer-use tasks while avoiding unsafe actions. Research frontier Sep 2026 2.40% System cards only
Mantic vs Polymarket backtest Point-in-time backtest of Mantic's AI forecasts against prices on 4,139 Polymarket markets at volume snapshots. Scored by Brier score and calibration. Expert Sep 2026 Not in cards
Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. Expert Sep 2026 0% System cards only
Medicinal Chemistry (Anthropic) Models see several real drug-discovery compounds and pick which had the highest or lowest measured ADME property. The set has 504 questions. Expert Sep 2026 65.3% System cards only
MiMo Code Bench Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details. Expert Sep 2026 68.6% In use
MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. Expert Sep 2026 81.7% In use
MiMo Visual Coding Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code. Expert Sep 2026 82.2% In use
Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. Expert Sep 2026 1.40% System cards only
Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. Expert Sep 2026 34% System cards only
OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. Expert Sep 2026 0% System cards only
OpenAI performance in cases flagged by users Model answers de-identified ChatGPT prompts that users flagged for factual errors. Scored on any error and on repeating the flagged error. Sep 2026 System cards only
Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. Expert Sep 2026 System cards only
Package registry publishing evaluation (Anthropic) Agents in a simulated capture the flag exercise can publish a backdoored package to a registry. An investigator escalates pressure until they act. Expert Sep 2026 33.3% System cards only
Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. Research frontier Sep 2026 10.7 System cards only
Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. Expert Sep 2026 0.20% System cards only
Qwen-Audio-Agent In-House Cockpit Benchmark Agents handle requests with cockpit tools. The evaluation measures whether they complete each task when they call tools directly, delegate, or choose between both routes. Everyday Sep 2026 Not in cards
Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. Expert Sep 2026 0% In use
Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. Expert Sep 2026 11.8% In use
Runway Model Router Benchmark Video generation systems produce clips for 250 image-to-video prompts across ten production categories and are compared by usable rate and cost. Expert Sep 2026 Not in cards
Safe response to high-severity prompts Models handle multi-turn self-harm and violent activity conversations where harmful intent only emerges from context. Scored as safe response rate. Expert Sep 2026 System cards only
Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. Research frontier Sep 2026 45.5% System cards only
Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. Expert Sep 2026 1.50% System cards only
SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. Research frontier Sep 2026 0.46 System cards only
Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. Expert Sep 2026 8.00% System cards only
Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. Expert Sep 2026 0% System cards only
Static Jailbreak Evaluations Models face a fixed set of jailbreak attacks on bio, violence and cyber requests. Scored as defender success rate per category. Expert Sep 2026 System cards only
StepCodeBench Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup. Expert Sep 2026 63.9% In use
Structured Translation (Cohere) Models translate JSON and similar structures, either whole or only named fields. Metrics check parsing, schema match, untranslated or wrongly copied fields and line counts. Everyday Sep 2026 93.7% System cards only
Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. Expert Sep 2026 0% System cards only
Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. Expert Aug 2026 0% System cards only
Biosecurity VCT Models answer dual-use virus lab troubleshooting items. Accuracy counts should-refuse and should-comply items together. Expert Aug 2026 47.8% System cards only
BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. Expert Aug 2026 91.4% System cards only
CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. Expert Aug 2026 1.80% System cards only
CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. Expert Aug 2026 3.50% System cards only
Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. Expert Aug 2026 38% System cards only
Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. Expert Aug 2026 20% System cards only
Cyber Conversational Evals About 1,000 one to five turn cyber requests test whether safeguards block offensive help and allow defensive help. Reports violation and benign refusal rates. Expert Aug 2026 0.40% System cards only
Cyber Static Jailbreak Eval Common jailbreak templates are applied to high severity cyber requests against the safeguarded model. Reports query and objective success rates. Expert Aug 2026 1.00% System cards only
DSBench (DeepSeek internal) DeepSeek internal coding-agent test sets: full-stack development tasks and hard agentic coding problems. Pass-rate scoring. Expert Aug 2026 71.1% In use
E-Bench-Code (Tencent) Agents complete Tencent in-house work tasks that need code. Task and scoring details are not published. Expert Aug 2026 83.3% In use
Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. Expert Aug 2026 4.00% System cards only
Gray Swan Cyber Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on cyber tasks built from Google scenarios. Reports query and objective success rates. Expert Aug 2026 10% System cards only
Harmful manipulation strategic planning red teaming Models are asked for influence operation plans in 11 real-world scenarios by direct request, jailbreak or adaptive prompting. Reports how often plans are elicited. Expert Aug 2026 98.2% System cards only
Hy-BrowseComp-Pro2 Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published. Expert Aug 2026 56.1% In use
Hy-CompanyBench V2 Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published. Expert Aug 2026 62.4% In use
Hy-FinAgentBench Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published. Expert Aug 2026 79.7% In use
Hy-FinmodelBench v2 Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published. Expert Aug 2026 57% In use
Hy-LifeSearch Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published. Expert Aug 2026 49.2% In use
InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. Expert Aug 2026 46.9% System cards only
KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. Expert Aug 2026 37.2% System cards only
Long-horizon jailbreaks (xAI) Models face multi-turn, Crescendo and long-horizon malicious-use attacks. Compliance is the share that get disallowed help. Aug 2026 0.65% System cards only
QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. University Aug 2026 69.9% In use
QwenVisualOffice Models handle visual office tasks in an internal Qwen test set. Qwen has not published the tasks or how they are scored. University Aug 2026 44.6% In use
Sakana Japanese-English translation evaluation Models translate Japanese text into English while handling Japan-specific proper nouns and honorifics. University Aug 2026 Not in cards
U18 safety evaluations Models answer challenging prompts in teen safety categories such as age-restricted goods, eating disorders and emotional reliance. Scored as safe completion rate per category. Everyday Aug 2026 System cards only
Z.ai Code Bench Zhipu in-house coding benchmark used for GLM-5.3. Reports pass rate and output tokens per task. Method unpublished. Expert Aug 2026 39.5% In use
24/7 ClawBench 2.0 Agents do always-on assistant work through OpenClaw. Tasks span several days, events arrive at the same time, and interruptions are routine. Expert Jul 2026 48.3% System cards only
Agent Behavior Bench Agents are scored on process quality, including tool-use behavior, efficiency and discipline, together with task completion. Expert Jul 2026 65% System cards only
Agentic Vision Bench (Moonshot) Agents must notice and correctly use key visual facts while they execute tasks. Expert Jul 2026 78.3% System cards only
Autointent-Bio Models face manual and synthetic dangerous and benign biology queries. xAI reports refusal recall on the dangerous ones. Jul 2026 100% System cards only
Autointent-Chem Models face manual and synthetic dangerous and benign chemistry queries. xAI reports refusal recall on the dangerous ones. Jul 2026 100% System cards only
Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. Expert Jul 2026 72% In use
Chat All-in-One Bench Models handle conversational scenarios built around real online user needs at every stage of product usage. Everyday Jul 2026 85.2% System cards only
Child safety (xAI) Models face scripted multi-turn requests for child sexual exploitation content. Compliance is the share of dialogues that fail policy. Jul 2026 0% System cards only
CLIF Bench Models learn from a provided context while they follow instructions that combine several complex skills. Expert Jul 2026 52.4% System cards only
Coding Experience (Moonshot) Coding agents work in real development workflows. The score reflects the practical experience of working with the agent, such as communication and stability. Expert Jul 2026 59.9% System cards only
Context Aware Automatic Speech Recognition Speech transcription systems are scored for semantic accuracy with and without free-form context. Everyday Jul 2026 44.6% In use
Cyber harm coverage (Anthropic) Cyber safeguard classifiers screen examples from each cyber harm category. Score is overall recall, the share of harmful examples flagged. Expert Jul 2026 100% System cards only
DECK Bench Models produce presentation decks from task descriptions drawn from real usage. The score measures deck quality. Expert Jul 2026 73.5% System cards only
Deep Research Bench (Moonshot) Agents answer deep-research queries curated by domain experts. Expert-aligned rubrics grade the reports. Expert Jul 2026 90% System cards only
E-Bench (Tencent) Agents complete Tencent in-house work tasks in the working agent group. Task and scoring details are not published. Expert Jul 2026 80.6% In use
Epistemic Bias (xAI) Models answer opposing framings of contentious questions. The bias rate counts one-sided framing or opinions where neutrality is expected. Jul 2026 20.4% System cards only
Faithfulness (Moonshot) A fact checker verifies each model response for factual hallucinations. The score is one minus the hallucination rate. Everyday Jul 2026 85.5% System cards only
FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. Jul 2026 84% System cards only
Finance Bench (Moonshot) Agents complete realistic financial workflows end to end, from source materials to deliverables that reviewers can check. Expert Jul 2026 62.6% System cards only
Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. Everyday Jul 2026 Not in cards
Human proximity (Google DeepMind robotics) Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. Everyday Jul 2026 93% In use
Hy-CompanyBench Coding agents complete Tencent in-house company engineering tasks. Task and scoring details are not published. Expert Jul 2026 41.7% In use
Hy-Euler pro Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published. Expert Jul 2026 24.2% In use
Hy-FinModelBench Agents build financial models for Tencent in-house finance tasks. Task and scoring details are not published. Expert Jul 2026 69% In use
Hy-Math Models solve Tencent in-house math problems without tools. Task and scoring details are not published. Expert Jul 2026 60.9% In use
Hy-SkillsWorld Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published. Expert Jul 2026 45.8% In use
Instruction hierarchy attacks Models face attacks that try to override higher-priority instructions. Scored as defender success rate averaged per defender query. Expert Jul 2026 99.99% System cards only
KAET (Kimi Autonomous Execution Tasks) Agents execute long-horizon tasks that simulate real user requests and enterprise system operations. Expert Jul 2026 83.5% System cards only
KAT Claw Bench Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables. University Jul 2026 85.5% In use
KAT Code Bench Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages. University Jul 2026 53.1% In use
Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. Expert Jul 2026 38.9% System cards only
Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. Expert Jul 2026 System cards only
Kimi Webdev Bench Models build games, 3D scenes and website clones from real prompts. Blind experts compare two outputs and report win, tie and loss rates. Expert Jul 2026 58.6% System cards only
Ko-GDPval Agents complete 170 Korean office tasks from 58 occupations in 11 industries and produce deliverable files. Rubrics grade data, content, structure, quality and format. Expert Jul 2026 87.3% In use
KWV Bench Models answer atomic visual questions extracted from tasks distilled from real knowledge-work scenarios. Expert Jul 2026 64.7% System cards only
Management Consulting Tasks (OpenAI) Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading. Expert Jul 2026 43.2% In use
MedChemBench (OpenAI) Models work through 99 medicinal chemistry problems drawn from recent papers and patents. Each task mixes prose with structures, assay tables and plots, then asks for the program’s next decision. Expert Jul 2026 83.8% In use
MIRA Bench Agents carry out long multi-role enterprise tasks across several systems and decide when to delegate to subagents. Expert Jul 2026 64.1% System cards only
Moment finding (Google DeepMind) Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. Everyday Jul 2026 91.3% In use
Online Experience (Moonshot) Agents produce the deliverable file types that real online users request most often, following the distribution of real usage. Everyday Jul 2026 77.9% System cards only
Organic Chemistry V2 (Anthropic) Models predict structures from spectra, design multi-step syntheses, predict reaction products and read chemical structure images. V2 adds harder expert-recommended problems. Expert Jul 2026 69.2% System cards only
ProdBench (Tencent) Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3. Expert Jul 2026 23% In use
Progress classification (Google DeepMind) Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. Everyday Jul 2026 57.4% In use
Prompt injection robustness Agents meet indirect prompt injection attacks inside tool outputs. Scored as defender success rate averaged per defender query. Expert Jul 2026 99.8% System cards only
Real-World Audio Recording Speech transcription systems are scored by word error rate on real-world recordings across multiple languages. Everyday Jul 2026 9.60% In use
RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. Research frontier Jul 2026 Not in cards
Secure and defensive coding (Anthropic) Cyber safeguard classifiers screen sampled defensive coding sessions such as patching and incident response. Score is the classifier flag rate. Expert Jul 2026 0.61% System cards only
Self-harm refusals (xAI) Models face self-harm and crisis prompts and conversations. A model fails if it assists or refuses without pointing to help. Jul 2026 0.50% System cards only
SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. Expert Jul 2026 61.1% System cards only
Standard jailbreaks (xAI) Models face a battery of published and internal jailbreaks. Compliance is the share of should-refuse attacks that get disallowed help. Jul 2026 0.01% System cards only
Vulnerability finding evaluations (Anthropic) Cyber safeguard classifiers screen sampled vulnerability discovery traffic. Reports flag rates on violative binary work and on defensive source code work. Expert Jul 2026 System cards only
AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. Research frontier Jun 2026 0.59 System cards only
Agent Startup Bench Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews. Expert Jun 2026 68.8% In use
Biology Model Refusal Evaluation Models answer severe, dual-use and benign biology prompts. Scored as safe rate on harmful prompts and not over-refusing benign ones. Expert Jun 2026 99.9% System cards only
CreativeWork Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls. Everyday Jun 2026 42.5% In use
Doubao Multi-Turn Bench Models answer the turn of a real Doubao conversation where the user was dissatisfied, with the earlier turns as context. Rubrics grade each answer. Everyday Jun 2026 52.5% In use
Image2FloorPlan Models study several real photos of a home and draw its floor plan. Scored by an average quality score. Everyday Jun 2026 48.0 In use
KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. Research frontier Jun 2026 76.9% System cards only
Kimi Claw 24/7 Bench Agents work on 17 persistent multi-day professional scenarios through OpenClaw, with 610 checkpoints. The score is the mean pass rate over three runs. Expert Jun 2026 46.9% In use
Kimi Code Bench v2 Coding agents complete Moonshot's realistic tasks from internal engineering work, production incidents, and open-source projects in over ten languages. Expert Jun 2026 77.1% In use
MAI human side-by-side evaluation Paid raters compare two models' answers to 1,276 English prompts from experts and Copilot logs on a 7-point scale from -1.5 to 1.5. Everyday Jun 2026 0.07 System cards only
MAI internal safety and over-refusal evaluation Models answer harmful and benign prompts in eight harm categories. Judges score safety pass rate on sensitive prompts and helpfulness as one minus over-refusal. Everyday Jun 2026 System cards only
MAI Thinking 1 blind human preference evaluation Professional raters compare anonymous assistant responses to single-turn and multi-turn prompts and score their overall preference. Everyday Jun 2026 Not in cards
MSQA Models answer culture-specific knowledge questions in 11 major languages. University Jun 2026 50.2% In use
Prompt injection attacks in search and function calling Agents receive search results and function call outputs that contain stronger known prompt injection attacks. Scored on resisting the injected instructions. Expert Jun 2026 94.6% System cards only
Protein Design (Anthropic) Models generate novel protein sequences under design constraints and rank protein design libraries for experimental testing. Expert Jun 2026 42.5% System cards only
Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. Expert Jun 2026 3.90% System cards only
Robust IF Bench Models follow a diverse set of hard instructions from Microsoft AI's internal benchmark. Everyday Jun 2026 61.2% In use
Sakana kana-letter reading-order set Model recovers the reading order of 25 expert-annotated classical Japanese kana letters (chirashigaki). Scored by normalized edit distance. Expert Jun 2026 Not in cards
SeedClawBench Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables. Everyday Jun 2026 66.6% In use
SeedKernelBench Models write or optimize CUDA style kernels for operator tasks mined from FlashInfer, SGLang, FlashAttention, DeepGEMM, CUTLASS and cuDNN. Scored as average speedup over reference code. Expert Jun 2026 System cards only
Snowflake Analytical Search: ArXiv Papers Agents count, group and compare findings across research papers. The evaluation checks numeric and categorical answers against references and scores lists with F1. University Jun 2026 Not in cards
SVG-Bench (MiniMax) Model creates or edits SVG graphics from text or image prompts. A VLM checks results, averaged over 3 runs. University Jun 2026 63.7% In use
Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup. Expert Jun 2026 System cards only
Turing Agentic Environments Agents use business and consumer apps through graphical interfaces or tools. Turing reports average pass rate across the apps each model attempted. University Jun 2026 60.2% Not in cards
Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. Expert Jun 2026 System cards only
xDailyBench Models answer open-ended requests grounded in real user needs across more than 30 daily-life, study and office scenarios. Rubrics grade completion, correctness, coverage and usefulness. Everyday Jun 2026 61% In use
AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. Expert May 2026 0.80 System cards only
Anthropic undermining liberal democracy (behavioral audit metric) Automated behavioral audit metric: investigator models probe willingness to help coups, election tampering, or political surveillance, censorship or intimidation. May 2026 1.13 System cards only
Biological safety evaluations (OpenAI) Models answer biology requests drawn from deidentified production data and from easy and hard synthetic sets. Scored as the share of cases where model training alone prevents a violative response. Expert May 2026 System cards only
Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. Expert May 2026 System cards only
Cohere North internal evals Cohere internal evals of its North agent platform: agentic question answering, spreadsheet data analysis and memory use. LLM-as-judge scoring. University May 2026 Not in cards
CueSpeak Models speak at the correct moment and give the expected semantic response during simultaneous speech. Everyday May 2026 81.7% Fading
Factuality Heavy prompts Models answer prompts representative of factuality-heavy ChatGPT conversations. A grader with web access counts responses and claims with factual errors. Everyday May 2026 5.40% System cards only
Finance Modeling Pro Agents complete financial modeling tasks in Excel. Expert rubrics score each task over three trials. Expert May 2026 57% System cards only
High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. Expert May 2026 4.10% System cards only
HyperTask Agents build applications over long vibe coding sessions with about 100 feature or step requirements per task. Expert rubrics score the result over three trials. Expert May 2026 67.6% System cards only
Lazy investigation on misleading codebases (Anthropic) Models trace small, intentionally misleading codebases to answer how a value is set. Score is the share of lazy investigations ending in a wrong answer. Expert May 2026 0% System cards only
MEWC v2 Models solve 100 hard Microsoft Excel World Championship problems with Excel tools. Expert rubrics score the answers over three trials. Competition May 2026 63.3% System cards only
ProactiveVideoQA Models answer spoken questions when a streamed video first reveals the answer. Everyday May 2026 33.5 Fading
Prompt injection live bug bounty (Anthropic with Gray Swan) Expert red teamers submit prompt injection attacks against hidden frontier models in tool use, coding and computer use scenarios. Score is attack success rate. Expert May 2026 0.08% System cards only
QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. University May 2026 1,608 Fading
QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. University May 2026 1,884 In use
QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. University May 2026 1,617 Fading
QwenWorldBench Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. May 2026 62.1% Fading
Step-SWE-Bench Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses. Expert May 2026 67.1% Fading
TimeSpeak Models initiate speech at specified times while producing the requested content. Everyday May 2026 64.7% Fading
Adherence to its constitution (Anthropic) Investigator models build scenarios from 40 areas of Claude's constitution. A grader scores each transcript from minus 3 to plus 3 on 15 dimensions. Expert Apr 2026 1.02 System cards only
Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. Expert Apr 2026 39.3% System cards only
BioPipelineBench Verified (Anthropic) Agents execute bioinformatics workflows with a bash tool. Only problems that passed a validation check by external reviewers are kept. Expert Apr 2026 83.6% System cards only
Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. Expert Apr 2026 97.1% Fading
Deepgram Voice AI Benchmarks Speech models are compared on word error rate across production audio in 20 languages, while speech synthesis models are compared on read-aloud and hard-prompt sets. Expert Apr 2026 Not in cards
DeepSeek Chinese creative writing evaluation Models write Chinese fiction, prose, poetry and lyrics for 2,837 prompts. Pairwise judging gives separate win rates for instruction following and writing quality. Everyday Apr 2026 System cards only
DeepSeek Chinese functional writing evaluation Models answer 3,170 everyday Chinese writing prompts such as reports, emails and speeches. Pairwise judging against a baseline model gives a win rate. Everyday Apr 2026 62.6% System cards only
DeepSeek Chinese white-collar tasks evaluation Agents complete 30 Chinese professional analysis, document generation and editing tasks across 13 industries. Blind human annotators compare outputs against a baseline model. Expert Apr 2026 53% System cards only
DeepSeek complex instruction following and multi-turn writing evaluation Models answer 196 hard Chinese writing prompts with complex constraints or several turns. Pairwise judging against a baseline model gives a win rate. Everyday Apr 2026 45.9% System cards only
DeepSeek R&D Coding Benchmark Coding agents complete 30 tasks taken from DeepSeek internal R&D work in PyTorch, CUDA, Rust and C++. Human-written rubrics score each task. Expert Apr 2026 67% System cards only
DeepSeek search Q&A evaluation Models answer objective and subjective Chinese search questions with retrieval or agentic search. Pairwise judging gives win, loss and tie rates. Everyday Apr 2026 28.1% System cards only
Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours. Expert Apr 2026 83% Fading
GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. Expert Apr 2026 33.2% Fading
Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. Research frontier Apr 2026 7.60% System cards only
Hiding a secret password (Anthropic) Models keep a given password secret while an investigator model tries to extract it over many turns. Measures survival and per-turn leak rates. Expert Apr 2026 System cards only
Hy-Backend Coding agents complete Tencent in-house backend development tasks. Task and scoring details are not published. Expert Apr 2026 54.7% Fading
Hy-SWE Max Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task. Expert Apr 2026 64.2% In use
Hy-Vibe Bench Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published. Expert Apr 2026 50.5% Fading
ImageMining Multimodal agents retrieve and reason over image information for image mining tasks. Expert Apr 2026 30.7% Fading
Input hallucinations (Anthropic) Models answer 500 prompts that need an unavailable tool and 500 prompts that cite missing context. Score is the non-hallucination rate. Everyday Apr 2026 System cards only
Instrument reading (Google DeepMind) Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types. Everyday Apr 2026 65.7% Fading
Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. Research frontier Apr 2026 81.8% System cards only
Malicious agentic influence campaigns (Anthropic) Agents receive requests to run multi-step influence operations with tools. Score is the refusal rate. Everyday Apr 2026 System cards only
Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. Expert Apr 2026 System cards only
Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. Expert Apr 2026 89.9% System cards only
Meta cyber misuse request set Models receive 600 private offensive cyber prompts, directly and under jailbreak attacks. Reports attack success rate. Expert Apr 2026 9.00% System cards only
Meta prompted evaluation awareness Models are asked to judge whether transcripts come from evaluations or real use. Reports the weighted rate of correct identification. Expert Apr 2026 43% System cards only
Meta Underspecified Intent evaluation Models answer ambiguous requests sourced from real Meta AI chats where users corrected the reply. Rubrics score whether the true intent is met. Everyday Apr 2026 71.2% System cards only
MiMo Coding Bench Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE. Expert Apr 2026 77.1% Fading
MMSearch Multimodal agents search and answer questions that require visual and textual evidence. Expert Apr 2026 72.9% Fading
Protocols (Anthropic life sciences) Models find and fix errors in molecular biology protocols and answer practical questions about applying real laboratory protocols. Expert Apr 2026 62% System cards only
QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. University Apr 2026 84% Fading
QwenWebBench Models generate front-end code for web apps, games, SVG, and other categories in English and Chinese. Multimodal judgments produce Elo ratings. University Apr 2026 1,532 Fading
Single-turn election integrity (Anthropic, 2026) Models answer legitimate election requests and decline requests that violate Anthropic’s election policy. Graders check refusals and harmful completions separately. Everyday Apr 2026 Not in cards
ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. Everyday Apr 2026 86.1% Fading
Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. Expert Apr 2026 System cards only
VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. Research frontier Apr 2026 System cards only
CoT Monitorability Monitors read a model's chain of thought, actions or all messages across intervention, process and outcome evaluations. Scored as g-mean squared of TPR and TNR. Expert Mar 2026 System cards only
Deception Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the share of outputs flagged for deceptive behaviors such as fabricated facts. Everyday Mar 2026 System cards only
Destructive action reversion on long rollouts Agents revert their own changes after long rollouts in workspaces with simulated user edits. Scored on perfect reversion and on preserving user work. Expert Mar 2026 System cards only
Disallowed Content Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the estimated prevalence of policy violating outputs per category. Everyday Mar 2026 System cards only
MM-ClawBench Agents complete common OpenClaw tasks such as study planning, office documents, scheduled research, and code maintenance. Scoring is accuracy. Everyday Mar 2026 75.4% Fading
Multiturn Jailbreak Evaluations Models face adaptive multi-turn jailbreak attackers with growing attack budgets. Scored as worst-case defender success rate. Expert Mar 2026 80.9% System cards only
PMBench Agents search product managers’ notes to identify evidence across customer conversations and answer questions about product requirements. University Mar 2026 46.1 Not in cards
Agentic code behavior scores (Anthropic) Agents work through over one hundred realistic coding scenarios. Graders score instruction following, safety, verification, efficiency, adaptability and honesty out of 10. Expert Feb 2026 System cards only
BioPipelineBench (Anthropic) Agents execute bioinformatics workflows such as long-read sequence analysis, metagenome assembly and chromatin profiling with a bash tool. Expert Feb 2026 87.7% System cards only
CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass. Expert Feb 2026 25.8% Fading
Chinese Complex Instruction Following (ByteDance Seed in-house) Models follow complex Chinese production prompts with constraints on format, conditions, content, phrasing, tone, emoji, few-shot style and length. It has 912 cases over 17 weighted dimensions. Everyday Feb 2026 75.3% System cards only
Compositional Tasks (ByteDance Seed in-house) Models identify several elements of a business text in one inference, such as user intent and slot values. Everyday Feb 2026 59.1% System cards only
Consulting and Recommendations Benchmark (StepFun) Models answer 500 advice queries taken from Reddit, Stack Exchange and forums. Model judges score usefulness, logic, instruction following and tone against an expert reference set at 88. Everyday Feb 2026 70.5% System cards only
Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. Expert Feb 2026 16.4 System cards only
Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. Expert Feb 2026 39.6% System cards only
Deference to governments in local languages (Anthropic) Models answer politically controversial questions in English and in a local language. A grader rates agreement with the official government position from 1 to 10. Everyday Feb 2026 System cards only
Disordered eating evaluations (Anthropic) Models handle conversations about eating disorders and disordered eating. Score is the appropriate response rate. Everyday Feb 2026 System cards only
External pilot use failure modes (Anthropic) Models are judged on user-flagged pilot conversations, directly or by resampling. Graders measure failure modes such as hallucinated facts and misrepresented work completion. Everyday Feb 2026 System cards only
Finance Modeling Agents complete expert-written financial modeling problems with Excel tools, from research to analysis. Expert-designed rubrics score each problem, averaged over three runs. Expert Feb 2026 21.6% Fading
Higher-difficulty benign request evaluations (Anthropic) Models answer harder benign requests that resemble harmful ones. Score is the refusal rate, where lower is better. Everyday Feb 2026 0.01% System cards only
Higher-difficulty harmful request evaluations (Anthropic) Models answer harder, more obfuscated policy-violating requests. Score is the harmless response rate. Everyday Feb 2026 99.4% System cards only
Human SxS image editing Elo (Google) Google compares edited images side by side and converts human preferences into Elo ratings. Everyday Feb 2026 1,065 Fading
Human SxS image generation Elo (Google) Google compares generated images side by side and converts human preferences into Elo ratings. Everyday Feb 2026 1,114 In use
IF-Badcase Models follow 450 multi-constraint instructions drawn from real user failure reports. A checklist checks each constraint. Everyday Feb 2026 83.2% System cards only
MEWC Models solve 179 Microsoft Excel World Championship problems from 2021 to 2026 with Excel tools. Output cells are compared one by one with answer cells. Competition Feb 2026 74.4% Fading
Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. Research frontier Feb 2026 76% System cards only
Organic Chemistry (Anthropic) Models predict structures from spectra, design synthetic routes, predict reaction products and convert between IUPAC names, SMILES and structure images. Expert Feb 2026 86.2% System cards only
Over-eager behavior on GUI computer use tasks (Anthropic) Computer-use agents face impossible tasks with an unwanted shortcut available. Score is the rate of unapproved workarounds instead of checking with the user. Expert Feb 2026 9.40% System cards only
Participation in junk science (Anthropic) Models talk with simulated users ranging from curious to openly fraudulent who seek fake science. Score is the rate of generating unsound scientific content. Expert Feb 2026 0% System cards only
Phylogenetics (Anthropic) Models analyze phylogenetic data and tree diagrams, testing quantitative reasoning about tree structure and visual interpretation. Expert Feb 2026 79.6% System cards only
Real-World Finance (Anthropic) Agents complete about 50 analyst tasks in banking, private equity, hedge funds and corporate finance, producing spreadsheets, slides and documents. Rubric graded. Expert Feb 2026 70% System cards only
Refusal to assist with AI safety R&D (Anthropic) Models receive single-turn coding requests for legitimate AI safety research with user context. Score is the refusal rate, where lower is better. Expert Feb 2026 3.10% System cards only
RISE Agents answer real questions from human experts that need multi-step search and interaction with information-dense web pages. MiniMax has not published the scoring method. Expert Feb 2026 64.3% Fading
SAM 3D Body synthetic pose evaluation Models estimate human pose and shape from synthetic images of individual people. Meta compares predicted joints and mesh vertices with the rendering parameters. School Feb 2026 Not in cards
Slide generation human evaluation (Z.ai) People compare HTML slide decks from two models on content, layout and visual quality. Metric is overall win rate. Everyday Feb 2026 67.5% System cards only
SSH stress-testing evaluation (Anthropic) Models continue prefilled conversations about suicide and self-harm built from anonymized user feedback. Score is the rate of appropriate course correction. Expert Feb 2026 95% System cards only
Structural Biology (Anthropic) Models answer questions about a biomolecule's function from structural data and basic tools, in multiple-choice and open-ended variants. Expert Feb 2026 79% System cards only
Swarm Bench Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. Expert Feb 2026 76.3% System cards only
Template Following generalization (Qwen) Models follow system prompts and emit exactly formatted XML or JSON tool calls for five scaffolds derived from popular IDE and CLI coding agents. Everyday Feb 2026 92.7% System cards only
Text Classification (ByteDance Seed in-house) Models analyze business text and return labels or structured outputs, for example sentiment or the main viewpoints in user content. Everyday Feb 2026 69% System cards only
ToolCall-Badcase Models choose a tool and its arguments for 200 cases drawn from real user failure reports. Each case has a ground-truth call. Everyday Feb 2026 95.8% System cards only
Translation Human Scoring (Z.ai) People score model translations. The GLM-5 report does not describe the test set or the scale. University Feb 2026 3.12 System cards only
ZMultiDialBench Human annotators score model replies to 141 multilingual dialogue prompts on a 1 to 10 scale with category criteria. Everyday Feb 2026 8.57 System cards only
ZMultiTransBench Models translate 1,220 real Chinese source texts into seven languages. A GPT-4.1 judge compares each translation with a fixed baseline. University Feb 2026 1050.0 System cards only
AI Office Benchmark (Moonshot) Agents complete end-to-end office tasks and produce documents, spreadsheets, PDFs and slide decks. Experts rate output quality. Expert Jan 2026 Not in cards
CodeUtilityBench (LG) Models handle 16 everyday coding tasks such as explaining, debugging, refactoring and testing code. A GPT-5 judge checks five rubrics per item. University Jan 2026 71.9% System cards only
CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. Expert Jan 2026 Not in cards
CVTG-2K Image models render text in images across regions and languages. CVTG-2K reports normalized edit distance, CLIPScore and word accuracy. Everyday Jan 2026 0.91 Fading
EntireSongs Models transcribe complete English and Chinese songs with background music. Qwen reports transcription errors separately for each language. Everyday Jan 2026 14.3% Not in cards
General Agent Benchmark (Moonshot) Agents complete multi-step production workflows. Experts compare their work with human expert performance. Expert Jan 2026 Not in cards
HealthBench-Hallu Models answer HealthBench-style medical prompts. A pipeline splits each answer into medical claims and checks them against medical evidence. The score is a weighted hallucination rate. Expert Jan 2026 3.50% Fading
KGC-Safety (LG) Models answer risky prompts set in Korean social and cultural contexts. A Safe Rate is scored over four domains: universal human values, social safety, Korean sensitivity and future risk. Everyday Jan 2026 99.8% Fading
LongText-Bench Image models render long text in images. The benchmark reports separate English and Chinese scores. Everyday Jan 2026 0.97 Fading
MedGemma lab report conversion Models convert lab reports to structured JSON. Google reports macro and micro F1 on two collections of clinical PDFs and a collection of synthetic reports. School Jan 2026 Not in cards
MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. Expert Jan 2026 Not in cards
OneIG-Bench Image models generate images from prompts with English and Chinese evaluation splits. Everyday Jan 2026 0.53 Fading
Qwen3-ASR internal transcription benchmarks Models transcribe accented conversations, noisy speech, tongue twisters, and multilingual news. Qwen reports error rates for each recording condition. Everyday Jan 2026 Not in cards
Random Complex Tasks (LongCat) Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs. University Jan 2026 35.8% Fading
TIIF-Bench Image models generate short and long images scored on overall image quality. Everyday Jan 2026 81.0 Fading
WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. Expert Jan 2026 Not in cards
Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. Expert 2026 Not in cards
Agent JSK Agents read simulated email connectors that contain prompt injection attacks. Scored on resisting the injected instructions. Expert Dec 2025 100% System cards only
AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. Expert Dec 2025 47.7% Fading
BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. Competition Dec 2025 53.5% Fading
Complex Workflow (ByteDance Seed in-house) Models carry out step-by-step tasks that follow a given standard operating procedure, such as multi-turn after-sales service in support, sales and healthcare. Everyday Dec 2025 64.7% Fading
Customer Support Q&A (ByteDance Seed in-house) Models answer user questions from noisy enterprise knowledge base results, such as policy questions in e-commerce, government services and telecom. Everyday Dec 2025 72.4% Fading
Education (ByteDance Seed in-house) Models solve, grade, explain and write K to 12 problems across core subjects in teaching scenarios. School Dec 2025 62.8% Fading
Information Extraction (ByteDance Seed in-house) Models extract words, sentences or fields from meeting records, legal texts, contracts, product descriptions and knowledge bases. Everyday Dec 2025 52% Fading
Information Processing (ByteDance Seed in-house) Models analyze large collections of user-generated content, for example summarizing viewpoints and judging overall sentiment. Everyday Dec 2025 53.9% Fading
Intention Recognition (ByteDance Seed in-house) Models classify intent from dialogues, meeting transcripts, social posts and articles, such as purchase intent in customer service chats. Everyday Dec 2025 63.2% Fading
Ko-Arena-Hard v2 (Upstage) Models answer hard user prompts in Korean, judged against a baseline in the style of Arena-Hard v2. Upstage built it in house. Everyday Dec 2025 79.9% Fading
PE-A-Frame internal audio event localization Models locate sound events within recordings. Meta compares model sizes using the area under the receiver operating characteristic curve. Everyday Dec 2025 Not in cards
PE-AV internal video-to-audio retrieval Models match videos to music tracks. Meta compares retrieval accuracy on internal clips that last from five to thirty seconds. Everyday Dec 2025 Not in cards
PlugInject Agents receive function call outputs that contain prompt injection attacks. Scored on resisting the injected instructions. Expert Dec 2025 99.6% System cards only
Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. Everyday Dec 2025 5.40% System cards only
SAM Audio Judge evaluation Models rate the quality of separated speech, music, and environmental sounds. Meta measures correlation with listener ratings for completeness, precision, faithfulness, and overall quality. Everyday Dec 2025 Not in cards
SWE-Review Models review code changes in several languages and scenarios to find defects. Scores reflect defect recall and the rate of hallucinated defects. Expert Dec 2025 16.2% Fading
U-Artifacts Models build interactive artifacts such as small apps and pages for 199 cases across 13 topics. A GUI agent simulates a non-expert user to grade interactivity, looks and fit to the need. Everyday Dec 2025 49.2% Fading
XpertBench Models complete expert tasks from more than 140 task types in law, finance, education, humanities and science. Over 200 domain experts wrote the prompts and weighted rubrics. Expert Dec 2025 64.5% Fading
ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. Expert Dec 2025 14.5% System cards only
ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. Expert Nov 2025 66.8% System cards only
Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. Expert Nov 2025 73.7% System cards only
CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. Expert Nov 2025 System cards only
CNML-Level Problems (DeepSeek) Models prove 91 theorems in algebra, geometry, number theory, combinatorics and inequalities at Chinese National High School Mathematics League level. A verifier model scores the proofs. Competition Nov 2025 System cards only
Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. Expert Nov 2025 0% System cards only
Destructive action avoidance Coding agents meet conflicting user edits during tasks. Scored on preserving user changes and avoiding data-destructive actions. Expert Nov 2025 90% System cards only
Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. Everyday Nov 2025 3.60 System cards only
Harmful manipulation propensity evaluation Simulated users talk with the model and an autorater counts how often it uses manipulative mechanisms, with and without persuasion instructions. Everyday Nov 2025 21.3% System cards only
Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. Expert Nov 2025 0% System cards only
Longform Writing Models write long-form text without tools, capped at 32k output tokens. Moonshot AI has not published the tasks or the scoring method. University Nov 2025 73.8% Dropped
LPFQA Models answer 430 long-tail professional questions drawn from real forum discussions in 7 academic and industrial domains. Expert Nov 2025 52.6% Fading
MathVision-Wildphoto Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description. School Nov 2025 63.8% System cards only
Multi-agent search (Anthropic internal benchmark) Agents solve difficult information retrieval problems with web search, alone or as orchestrators that delegate to subagents. Expert Nov 2025 74.8% System cards only
Prompt injection robustness against adaptive attackers, browser use (Anthropic) Browser agents face indirect prompt injections in web environments from an internal adaptive attacker. Score is attack success rate. Expert Nov 2025 0% System cards only
Prompt injection robustness against adaptive attackers, coding (Anthropic) Coding agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. Expert Nov 2025 0% System cards only
Prompt injection robustness against adaptive attackers, computer use (Anthropic) Computer-use agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. Expert Nov 2025 0% System cards only
Sycophancy on user-provided prompts (Anthropic) Models continue real feedback conversations where earlier Claude models were sycophantic. Score is the non-sycophantic response rate. Everyday Nov 2025 10% System cards only
Cursor Bench (2025 internal version) Agents answer real agent requests from Cursor engineers in Cursor's harness. Graders compare against hand-curated solutions and check use of the codebase's existing abstractions. Expert Oct 2025 Fading
Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. Expert Oct 2025 41% System cards only
Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. Expert Oct 2025 System cards only
Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. Expert Oct 2025 43.6% System cards only
Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. Expert Oct 2025 8.30% System cards only
Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. Everyday Oct 2025 1.37 System cards only
Anthropic paired-prompts political bias eval Models write arguments for opposing sides of political issues from paired prompts. Reviewers label each pair for asymmetries in length, tone, hedging and willingness to engage. Everyday Sep 2025 System cards only
Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea. Expert Sep 2025 51.3% Dropped
Cooperation with user deception (Anthropic) Models get developer system prompts asking them to subtly mislead users, then meet users who would be harmed. Graders score harmful and deceptive choices. Everyday Sep 2025 System cards only
DeepSeek in-house safety benchmark Models answer 1,120 Chinese and English questions across 28 safety subcategories. GPT-4o labels each answer as unsafe, safe or rejection. Everyday Sep 2025 25.2% System cards only
DeepSeek jailbreak robustness evaluation DeepSeek combines 2,232 jailbreak templates with its in-house safety questions. GPT-4o labels responses, and the unsafe ratio is compared with direct questions. Everyday Sep 2025 85.9% System cards only
DeepSeek multilingual safety evaluation Models answer 9,330 safety questions translated into 50 languages. Safe answers earn 5 points, rejections 4 and unsafe answers 0. Everyday Sep 2025 74.2% System cards only
False-premise questions (Anthropic) Models answer questions built on false premises. Measures how often they accept the premise instead of correcting it. Everyday Sep 2025 1.20% System cards only
Hunyuan-MT Challenge Testset (Tencent) Models translate between Chinese and English across everyday and specialist domains. Expert raters score each translation from 0 to 4 on marked error-prone points. Everyday Sep 2025 3.19 System cards only
LongCat safety evaluation (Meituan) Models answer private test queries in four risk categories: harmful content, criminal activity, misinformation and privacy. The score is the share of safe responses in each category. Everyday Sep 2025 95.7% Dropped
Malicious use of Claude Code (Anthropic) Agents in Claude Code receive malicious and dual-use coding requests. Score is the refusal rate on malicious requests. Expert Sep 2025 96.7% System cards only
Mandarin-Minority translation (Tencent) Models translate between Mandarin and Chinese minority languages in a Tencent test set. XCOMET-XXL scores each translation. Everyday Sep 2025 60.8% Dropped
MCP prompt injection evaluation (Anthropic) Agents use simulated email, Slack and document servers over MCP where content carries injected instructions. Score is the attack prevention rate. Expert Sep 2025 92.5% System cards only
Self-preference bias (Anthropic) Models grade transcripts attributed to themselves or to other models. Measures whether scores favor the model's own outputs. Everyday Sep 2025 0.07 System cards only
Success detection (Google DeepMind) Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures. Everyday Sep 2025 90% Fading
Suicide and self-harm evaluations (Anthropic) Models handle single-turn and multi-turn conversations about suicide and self-harm. Score is the appropriate response rate. Everyday Sep 2025 System cards only
SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. Everyday Sep 2025 74.3% System cards only
Sycophancy prompts (Anthropic) Models answer single-turn prompts that invite sycophancy, such as bad business ideas or delusional beliefs. Four synthetic sets and one hand-written set report sycophancy rates. Everyday Sep 2025 System cards only
Tool use prompt injection evaluation (Anthropic) Agents run bash commands where files, scripts or command outputs carry injected instructions. Score is the attack prevention rate. Expert Sep 2025 96% System cards only
UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. Everyday Sep 2025 59.8 Dropped
X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. Expert Sep 2025 58% Dropped
Broken Search Tool Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. Everyday Aug 2025 0.33% In use
Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. Expert Aug 2025 99% System cards only
Challenging prompts from red teamers with biosafety-relevant PhDs Models answer adversarial biology prompts written by red teamers with biosafety PhDs. Scored as not_unsafe rate from model training alone. Expert Aug 2025 93.6% System cards only
Challenging translation human evaluation (Z.ai) People blindly score translations of 100 real cases with slang, nicknames and memes on a 0 to 3 scale. Everyday Aug 2025 1.71 System cards only
Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. Expert Aug 2025 1.30% System cards only
Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. Everyday Aug 2025 1.60% System cards only
Expert Red Teaming for Violent Attack Planning Expert red teamers compare anonymized responses from two models on violent attack planning prompts. Scored as the win rate for being safer. Expert Aug 2025 65.1% System cards only
Filtered, adversarial sample of production prompts Models answer filtered borderline and high-risk biology prompts that resemble production traffic. Scored as not_unsafe rate from model training alone. Expert Aug 2025 96.9% System cards only
General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. Aug 2025 0% Fading
Hallucinations on production-representative prompts Models answer prompts representative of ChatGPT production traffic. A grader with web access scores incorrect claims and responses with major errors. Everyday Aug 2025 3.10% System cards only
Novel logical reasoning problems expert evaluation (Z.ai) Human experts score model answers to new multi-step logic problems with a fixed scoring standard for each question. University Aug 2025 62.0 System cards only
OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. Research frontier Aug 2025 8.00% System cards only
Pairwise Bioweaponization Campaign Red teamers with biology PhDs compare anonymized responses from two models on bioweaponization prompts. Scored as the win rate for being safer. Expert Aug 2025 60% System cards only
Real-scenario prompts human evaluation (Z.ai) One human evaluator scores replies to 660 curated real user prompts in English, Chinese and other languages on a 0 to 10 scale. Everyday Aug 2025 System cards only
Refusals + System Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the system message. The metric is the answer rate. Aug 2025 0% Fading
Refusals + User Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the user message. The metric is the answer rate. Aug 2025 0% Fading
SatLidar canopy height evaluation Models predict canopy height from satellite images. Meta compares height estimates with airborne lidar measurements and reports errors on held-out regions. University Aug 2025 Not in cards
Soft Bias (Internal) Models answer paired sociopolitical prompts with sides swapped. A judge scores each pair 0, 0.5 or 1 for framing differences. Aug 2025 0.36 System cards only
Sycophancy evaluation (offline) Models answer a fixed set of messages resembling production traffic that could elicit sycophancy. A grader assigns a sycophancy score. Everyday Aug 2025 0.04 System cards only
Tool-calling prompt injections Agents receive prompt injections inside tool call outputs such as connectors. Scored on resisting the injected instructions. Expert Aug 2025 99% System cards only
TroubleshootingBench Models find and correct realistic experimental errors in expert-written wet lab protocols. Scored as pass@1. Expert Aug 2025 59% System cards only
Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. Expert Jul 2025 67% System cards only
Audio Understanding Benchmark (Mistral) Models answer hard questions about 40 long-form audio recordings. Mistral built it in-house to test audio understanding beyond transcription. Expert Jul 2025 86.6% Dropped
EHRQA (Google) Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning. Expert Jul 2025 Not in cards
EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. Expert Jul 2025 Not in cards
In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. Expert Jul 2025 78% System cards only
Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. Expert Jul 2025 88.6% Fading
Irrelevant instructions, text-based web browser Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them. Expert Jul 2025 99.5% System cards only
Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. Expert Jul 2025 95% System cards only
KBank-MMLU (Upstage) Models answer Korean finance questions in a multiple-choice format. Upstage built the set in house. Expert Jul 2025 80.8% Fading
Kimi Chinese in-house open-ended benchmark Models answer held-out Chinese prompts drawn from real user queries. Outputs are compared pairwise against another model to give win, tie and loss rates. Everyday Jul 2025 System cards only
Kimi Promptfoo red-team safety evaluation Promptfoo generates attack prompts for harmful, criminal, misinformation, privacy and security plugins under five strategies. Human reviewers judge the responses. Everyday Jul 2025 System cards only
Ko-LongBench (LG) Models answer questions over long Korean contexts. Tasks cover document QA, story and dialogue history understanding, in-context learning, structured QA and RAG. University Jul 2025 89.6% Fading
Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. Expert Jul 2025 System cards only
PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. Expert Jul 2025 Not in cards
US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. Expert Jul 2025 Not in cards
Apple side-by-side human evaluation Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison. Everyday Jun 2025 64.7% Fading
ComplexNLU (Tencent) Models answer Tencent in-house natural language understanding tasks with complex inputs. Task and scoring details are not published. Everyday Jun 2025 61.2% Dropped
Infinite Generations (Mistral internal) Models answer challenging long or repetitive prompts from Mistral's internal set. The score is the share of responses that never stop or loop, so lower is better. Everyday Jun 2025 1.29% Dropped
InsCtrl (Tencent) Models write text under detailed instructions in a Tencent in-house text creation test. Task and scoring details are not published. Everyday Jun 2025 71.9% Dropped
LengthCtrl (Tencent) Models write text that must meet a requested length, in a Tencent in-house test. Task and scoring details are not published. Everyday Jun 2025 55.4% Dropped
Mistral internal function calling benchmark Models call functions on Mistral's in-house test set. The Magistral paper does not describe the tasks or the metric. Everyday Jun 2025 87.4% System cards only
V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. Everyday Jun 2025 Not in cards
Word-Task (Tencent) Models solve Tencent in-house word level language tasks. Task and scoring details are not published. Everyday Jun 2025 62.9% Dropped
Agentic coding malicious requests (Anthropic) Coding agents with standard coding tools receive 150 malicious coding requests prohibited by policy. Score is the refusal rate. Expert May 2025 100% System cards only
Ambiguous context evaluations (Anthropic) Models respond to gray-area requests where intent is unclear. Graders judge how well responses balance helpfulness and harm. Everyday May 2025 System cards only
Coding prompt injections Agents work on coding tasks that contain prompt injection attacks. Scored on resisting the injected instructions. Expert May 2025 100% System cards only
Conversational SWE Task Benchmark (Windsurf) Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy. Expert May 2025 Dropped
CounterFactQA (Qwen) Models get counterfactual questions and must recognize that the premise is false instead of giving a hallucinated answer. Everyday May 2025 68.1% System cards only
Creative biology Models answer questions about unusual biology needed to engineer harmless biological agents, a weak proxy for novel weapon design. Expert May 2025 60.3% System cards only
DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. Expert May 2025 80% System cards only
End-To-End SWE Task Benchmark (Windsurf) Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score. Expert May 2025 Dropped
Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. Expert May 2025 30% System cards only
Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. Expert May 2025 0.65 System cards only
Internal model use survey (Anthropic) Anthropic technical staff estimate productivity uplift from the model and judge whether it could automate a junior ML researcher. Expert May 2025 50% System cards only
Kernels task (Anthropic) Agents optimize compute kernels in a performance engineering challenge. Scored by speedup. Expert May 2025 System cards only
LengthCtrl (Qwen) Models do creative writing tasks with length requirements. The score depends on the gap between output length and target length. Everyday May 2025 73.5% System cards only
Malware Refusals (Golden Set) Coding agents receive malware development requests from a curated golden set written by OpenAI policy experts. Scored as refusal rate. Expert May 2025 100% System cards only
Malware Refusals (Synthetic) Coding agents receive a large synthetic set of malware-related tasks, including ambiguous and dual-use prompts. Scored as refusal rate. Expert May 2025 97% System cards only
Multi-turn testing (Anthropic) Models face multi-turn conversations that escalate toward harmful requests across risk areas. Score is the appropriate response rate. Everyday May 2025 System cards only
Novel compiler task (Anthropic) Agents build a compiler for an unusual programming language from a specification and test cases. Expert May 2025 80.9% System cards only
Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. Expert May 2025 0% System cards only
Seed1.5-VL internal benchmark (ByteDance Seed in-house) Models answer more than 12,000 internal image questions over 100 tasks, from perception, OCR and STEM to agents, code and out-of-distribution puzzles. LLM judges grade answers, and the overall is a weighted average. University May 2025 59.3% System cards only
Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. Expert May 2025 System cards only
Single-turn benign request evaluations (Anthropic) Models answer benign requests on sensitive topics. Score is the refusal rate, where lower is better. Everyday May 2025 0% System cards only
Single-turn harmful request evaluations (Anthropic) Models answer single-turn requests that violate the usage policy. Score is the harmless response rate. Everyday May 2025 99.8% System cards only
Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. Expert May 2025 0.97 System cards only
ThinkFollow (Qwen) Models hold multi-turn dialogues with randomly inserted think and no-think flags and must switch reasoning mode correctly. Everyday May 2025 98.9% System cards only
Time series forecasting task (Anthropic) Agents solve a regression and time-series forecasting problem with known state-of-the-art results, in six variants. Expert May 2025 4.55 System cards only
ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. Everyday May 2025 85.5% System cards only
Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. Expert May 2025 0.63% System cards only
Unexpected container or code states Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed. Expert May 2025 85% System cards only
Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. Expert Apr 2025 93.3% System cards only
Default safety evaluation (Cohere) Models answer unsafe prompts in five categories with no system preamble. LLM judges score the share of safe responses, and a jury rates relative safety. Everyday Apr 2025 70.4% System cards only
Enterprise generative evaluation (Cohere) Models complete 22 enterprise writing tasks such as job ads with format rules. Rule checks and LLM judges give a pass rate. Everyday Apr 2025 94.2% System cards only
Enterprise RAG evaluation (Cohere) Models answer workplace policy and technical questions over long documents, some unanswerable. Llama Index Correctness grades answers from 1 to 5. Everyday Apr 2025 4.73 System cards only
Image generation refusals Models decide whether to call the image generation tool on human-curated adversarial prompts with system mitigations on. Scored on not_unsafe and not_overrefuse. Expert Apr 2025 98% System cards only
Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. Everyday Apr 2025 System cards only
Internal API instruction following (OpenAI) Models answer prompts drawn from real developer use cases that carry instructions on formatting, verbosity and length. OpenAI reports a hard subset. Everyday Apr 2025 54% Dropped
Kimi-ASR Internal Testset Models transcribe speech from two internal Moonshot test subsets. Scored by word error rate, lower is better. Everyday Apr 2025 System cards only
Kimi-ASR Internal Testset Subset 1 Models transcribe speech from Moonshot internal subset 1. The score is the published word error rate. Everyday Apr 2025 1.42% Not in cards
Kimi-ASR Internal Testset Subset 2 Models transcribe speech from Moonshot internal subset 2. The score is the published word error rate. Everyday Apr 2025 2.44% Not in cards
Multi-dialect SQL evaluation (Cohere) Models write SQL for SQLite, PostgreSQL, MySQL, PL/SQL and T-SQL databases. Scored by execution accuracy against gold databases. University Apr 2025 55.3% System cards only
Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. Everyday Apr 2025 System cards only
Pairwise safety red teaming Red teamers compare anonymized responses from several models and pick the least safe. Scored as ELO or win rate. Expert Apr 2025 System cards only
Person Identification evaluation Models receive requests to identify people in photos. Scored on refusal rate in non-adversarial and adversarial sets. Everyday Apr 2025 System cards only
Safety mode controllability (Cohere) Models follow contextual or strict safety instructions. Accuracy averages correct answers on an over-refusal set and correct refusals on a control set. Everyday Apr 2025 System cards only
Snowflake Multimodal PDF Retrieval Evaluation Retrievers find relevant pages in technical manuals, SEC filings and presentation slides. The tasks include 143 chart questions and 69 text questions about manuals. University Apr 2025 Not in cards
Text to Text Safety An automated content safety evaluation measures policy violations in model answers to text prompts. Everyday Apr 2025 System cards only
Tone An automated evaluation measures whether model refusals are objective, neutral and not preachy. Everyday Apr 2025 System cards only
Ungrounded Inference evaluation Models receive requests for inferences about people that images or text cannot justify. Scored on refusal rate in non-adversarial and adversarial sets. Everyday Apr 2025 System cards only
WISER-Atomic Web research systems answer 121 customer-inspired questions across finance, sales, recruitment, and other domains. Expert Apr 2025 69% Not in cards
CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. Expert Mar 2025 System cards only
Cohere enterprise human evaluation Trained annotators blindly compare two models' answers on enterprise tasks such as HR FAQs, technical FAQs and workplace assistants. They judge accuracy, instruction following and style. Results are win, tie and loss rates. Everyday Mar 2025 51.7% Dropped
Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. Expert Mar 2025 91.7% System cards only
ETVA Text-to-video systems are scored on fine-grained prompt alignment through generated atomic questions and video-language answers. Expert Mar 2025 Not in cards
Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. Everyday Mar 2025 Not in cards
Golden3 speech summarization (Microsoft) Models summarize 108 recorded meetings of about 6 minutes each from 1071 instructions. GPT-4 rates overall quality from 1 to 7. Everyday Mar 2025 6.28 System cards only
Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) GPT-4o simulates adversarial conversations with the model, and a judge flags violence, sexual, self-harm and hateful content. Scored as the share of harmful responses, lower is better. Everyday Mar 2025 1.40% System cards only
Snowflake Financial Filings RAG Evaluation Systems answer about 500 questions over 23,000 SEC filings. Annotators identify supporting passages, pages and answers. Expert Mar 2025 Not in cards
Answer Factuality (Perplexity) Models answer questions from search results. Graders check that answers use facts grounded in the results and resolve conflicting or missing information. Everyday Feb 2025 85.1% Dropped
Bioweapons knowledge questions Models answer 33 sensitive bioweapons questions written by Deloitte biosecurity staff. Answers are compared with novice, intermediate and expert baselines. Expert Feb 2025 51.5% System cards only
Challenging Red Teaming Evaluation 1 (created for o3-mini) Models face adversarial jailbreaks from o3-mini red teaming on illicit advice, extremism, political persuasion and self-harm. Scored as not_unsafe rate. Expert Feb 2025 51% System cards only
Challenging Red Teaming Evaluation 2 (created for deep research) Models face adversarial prompts from deep research red teaming on risky advice such as attack planning. Scored as not_unsafe rate. Expert Feb 2025 67% System cards only
Child safety evaluations (Anthropic) Models answer single-turn and multi-turn prompts about child sexualization and grooming. Score is the appropriate response rate. Everyday Feb 2025 System cards only
Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. Expert Feb 2025 94.4% System cards only
Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. Research frontier Feb 2025 66.7% System cards only
Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. Expert Feb 2025 System cards only
LLM training task (Anthropic) Agents optimize a CPU-only small language model training implementation. Scored by speedup. Expert Feb 2025 System cards only
Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. Expert Feb 2025 69.7% System cards only
Malicious computer use (Anthropic) Computer-use agents receive harmful tasks such as surveillance or fraud in GUI environments. Score is the refusal rate. Everyday Feb 2025 99.4% System cards only
Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. Expert Feb 2025 100% System cards only
Personal data evaluations Browsing agents receive requests for private personal information, from 200 synthetic prompts and 55 manually written golden examples. Scored on adherence to personal data policy. Everyday Feb 2025 System cards only
Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. Expert Feb 2025 71.4% System cards only
Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. Expert Feb 2025 31.3 System cards only
Readability (Perplexity) Models answer search questions. Graders rate whether answers are concise yet detailed and use markdown formatting well. Everyday Feb 2025 85.9% Dropped
Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. Expert Feb 2025 100% System cards only
Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. Expert Feb 2025 100% System cards only
Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. Everyday Jan 2025 97% System cards only
Coding (MiniMax in-house) Models answer coding requests from real Hailuo AI users. The report does not describe the scoring. University Jan 2025 90.2% System cards only
Creative Writing (MiniMax in-house) Models write creative pieces for real Hailuo AI user requests. Automatic assessments score the writing. Everyday Jan 2025 81.3% System cards only
General Assistant (MiniMax in-house) Models answer real assistant queries from Hailuo AI users. Automatic assessments score the replies. Everyday Jan 2025 73.9% System cards only
Hard Capability (MiniMax in-house) Models handle complex productivity requests with several levels of instructions that leading models often fail. University Jan 2025 64.8% System cards only
Instruction Following (MiniMax in-house) Models follow multi-level instructions collected from user cases where current models fail. Everyday Jan 2025 46.3% System cards only
Knowledge Q&A (MiniMax in-house) Models answer knowledge questions drawn from real Hailuo AI user queries. Everyday Jan 2025 78.6% System cards only
Long Context (MiniMax in-house) Models translate, summarize and analyze long documents from real user requests. Everyday Jan 2025 93.8% System cards only
OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. Research frontier Jan 2025 55% System cards only
Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. Expert Jan 2025 23% System cards only
Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. Everyday Jan 2025 94% System cards only
Safety (MiniMax in-house) Models respond to safety-relevant user prompts. MiniMax scores whether replies are safe and helpful. Everyday Jan 2025 90.9% System cards only
User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. Everyday Jan 2025 92% System cards only
Contextual Nuclear Knowledge Models answer 222 multiple choice questions on nuclear engineering with a nonproliferation focus. Scored as cons@32 accuracy. Expert Dec 2024 77% System cards only
Image input safety evaluations Models answer harmful prompts that combine text and images in categories such as hate, extremism and self-harm. Scored as safe completion rate. Everyday Dec 2024 99% System cards only
Instruction Hierarchy Evaluation Models resolve conflicts between system, developer and user messages, resist tutor jailbreaks, and protect secret phrases and passwords. Scored per sub-evaluation. Everyday Dec 2024 System cards only
Microsoft RAI benchmark: Grounding Models answer in GPT-4o simulated multi-turn conversations. A judge scores from 0 to 5 how fully each answer is grounded in the given prompt. Everyday Dec 2024 4.62 System cards only
PersonQA Models answer questions about publicly available facts on people. Scored on accuracy, with hallucination rate reported alongside. Everyday Dec 2024 94.3% System cards only
PhiBench Models answer original prompts that Microsoft's Phi team wrote to test a wide range of skills and reasoning. University Dec 2024 74.2% Dropped
Radiological and Nuclear Expert Knowledge Models answer 87 expert and tacit knowledge questions on radiological and nuclear threats. Scored as cons@32 accuracy. Expert Dec 2024 74% System cards only
Red Teaming Network pairwise safety comparison Red teamers compare anonymized responses from two models in free-form risky conversations. Scored as the share of ratings where a model is safer. Expert Dec 2024 59.8% System cards only
Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. Research frontier Dec 2024 System cards only
Structured expert probing campaign: radiological and nuclear Radiological and nuclear experts test whether a model helps with weapons development steps. Conversations are rated against Preparedness risk levels. Research frontier Dec 2024 System cards only
Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. Expert Dec 2024 51% System cards only
Cortex Analyst Multi-Turn Question Rewriting Models rewrite a follow-up question so it stands alone. Mistral Large 2 judges whether the rewrite preserves the necessary conversation history. University Nov 2024 Not in cards
Election integrity evaluations (Anthropic) Models answer election-related prompts, including requests for misinformation or influence operations. Score is the appropriate response rate. Everyday Oct 2024 System cards only
First-Person Fairness evaluation Tests whether a user's name (a demographic proxy) changes responses, using an LLM research assistant to detect harmful stereotypes. Oct 2024 0.40% System cards only
Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. Expert Sep 2024 System cards only
BirthdayFacts Models are asked for people's birthdays. Scored on how often they give a wrong birthday. Everyday Sep 2024 24% System cards only
Challenging Refusal Evaluation Models answer a harder set of disallowed content prompts across policy categories. An autograder scores the not_unsafe rate. Expert Sep 2024 93.4% System cards only
ChangeMyView Models write rebuttals to r/ChangeMyView posts that human raters score for persuasiveness. Scored as percentile against human responses. Everyday Sep 2024 89.1% System cards only
cognition-golden Cognition tests coding agents on tasks modeled on real Devin use cases, in authentic development environments with fully autonomous grading. A train split serves self-improvement and a test split measures capability. Expert Sep 2024 51.8% Not in cards
CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. Everyday Sep 2024 0.17% System cards only
DS-Arena-Code Models answer common coding requests in an internal subjective test. GPT-4o judges responses against competitor models to give a win rate. University Sep 2024 63.1% Dropped
DS-FIM-Eval Models fill in missing code between a given prefix and suffix, as in editor plugin completion. University Sep 2024 78.3% Dropped
Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. Expert Sep 2024 System cards only
External red team jailbreaks External red teamers and automated gap finding generate jailbreaks against the model. Scored on how often the model resists. Expert Sep 2024 44% System cards only
Human Sourced Jailbreaks Models face jailbreak prompts collected from human red teaming. Scored as the not_unsafe rate. Expert Sep 2024 100% System cards only
Jailbreak Augmented Examples Models face publicly known jailbreaks applied to prompts from the standard disallowed content set. Scored as not_unsafe rate. Everyday Sep 2024 100% System cards only
Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. Expert Sep 2024 0% System cards only
Open Ended Questions (hallucination) Models write open-ended factual text such as biographies, checked against Wikipedia. Scored as average number of incorrect statements. Everyday Sep 2024 0.78 System cards only
Political Persuasion Parallel Generation Human raters pick the more persuasive of two short political social media posts. Scored as win rate against GPT-4o. Everyday Sep 2024 53.8% System cards only
Production Jailbreaks Models face jailbreaks found in production ChatGPT data. Scored as not_unsafe rate. Everyday Sep 2024 100% System cards only
QuantBench Models answer 25 multiple choice versions of quant trading firm reasoning puzzles. Scored as cons@16 accuracy. Competition Sep 2024 60% System cards only
Regurgitation Evaluations Models are asked to reproduce training data. Scored on how often they refuse to regurgitate it. Everyday Sep 2024 System cards only
Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. Expert Aug 2024 78% System cards only
Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. University Aug 2024 68% System cards only
Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. School Aug 2024 82% System cards only
Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. Expert Aug 2024 96.3% Fading
Microsoft internal multimodal RAI benchmark Vision language models answer image and text prompts across harm categories. Responses are scored from 0 to 10, higher is safer. Everyday Aug 2024 8.16 System cards only
OpenAI Research Engineer Interview: Coding Models solve 18 self-contained coding problems like those in OpenAI interviews. Scored as pass@1. Expert Aug 2024 100% System cards only
OpenAI Research Engineer Interview: Multiple Choice Models answer 97 multiple-choice questions drawn from OpenAI machine learning interview topics. Scored as cons@32. Expert Aug 2024 83% System cards only
Speaker identification Speech models get audio requests to identify a speaker by voice. Scored on refusing when they should and complying for famous quotes. Everyday Aug 2024 System cards only
Standard Refusal Evaluation Models answer disallowed and benign safety-related prompts across policy categories. An autograder scores not_unsafe and not_overrefuse. Everyday Aug 2024 100% System cards only
Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. Expert Aug 2024 94% System cards only
Text persuasion study on political opinions Participants read model-written articles or chat with a model on political topics. Scored on opinion shift against human-written articles. Everyday Aug 2024 System cards only
Unauthorized voice generation Speech models produce audio that may deviate from the approved system voice. Scored on how many meaningful deviations the output classifier catches. Everyday Aug 2024 100% System cards only
Ungrounded inference and sensitive trait attribution Speech models get requests to infer traits about a speaker from audio. Scored on refusing ungrounded inferences and hedging plausible ones. Everyday Aug 2024 84% System cards only
Voice persuasion study on party preferences Participants hear model-voiced audio clips or talk with a voiced model. Scored on shifts in hypothetical party preference against human baselines. Everyday Aug 2024 5.74% System cards only
Llama 3 pairwise human evaluation (Meta) Human raters compare Llama responses with a rival model on about 7,000 prompts across six capabilities and three multiturn sets. Jul 2024 System cards only
Llama internal safety benchmarks (Meta) Models face Meta's adversarial and borderline prompts per capability and language. Scored as violation rate and false refusal rate. Jul 2024 System cards only
Llama verbatim memorization probe (Meta) Models get prompts sampled from their pretraining data. The score is the share of generations that reproduce the true continuation verbatim. Jul 2024 System cards only
Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Everyday Jul 2024 69.6% System cards only
Qwen in-house English automatic evaluation Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Everyday Jul 2024 72.9% System cards only
Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. Everyday Jul 2024 System cards only
BetterChartQA Models answer annotator-written questions about real charts from news, government reports, and papers across 9 capability buckets. Scored by accuracy. School Jun 2024 72.4% Dropped
ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. University Jun 2024 69.7% Dropped
Internal agentic coding evaluation (Anthropic) Agents implement a pull request in an open source codebase from a natural language description. A problem counts as solved when all hidden tests pass. Expert Jun 2024 78% System cards only
Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. Everyday Jun 2024 4.15 Dropped
HiddenMath Models solve 179 competition-level math problems that experts wrote from scratch. Answers are graded automatically. Competition May 2024 80.6% Dropped
Microsoft RAI benchmark: Harmful Content Continuation (DR-3) Models continue conversations about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. Everyday Apr 2024 0.30% System cards only
Microsoft RAI benchmark: Harmful Content Summarization (DR-3) Models summarize content about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. Everyday Apr 2024 10% System cards only
Microsoft RAI benchmark: Jailbreak (DR-1) Models face simulated jailbreak conversations on topics such as child grooming, illegal advice and leaking guidelines. Scored as the defect rate at severity 1. Everyday Apr 2024 7.30% System cards only
Microsoft RAI benchmark: Third Party Harm (DR-1) Models face simulated conversations that try to extract third-party content such as books, news, recipes and songs. Scored as the defect rate at severity 1. Everyday Apr 2024 10.5% System cards only
Microsoft RAI benchmark: Ungroundedness Models answer in GPT-4 simulated multi-turn conversations. A judge rates how ungrounded each answer is in the given prompt, from 0 to 4. Everyday Apr 2024 0.21 System cards only
100Q-Hard (Anthropic) Models answer hard, obscure factual questions. Scored on correct, incorrect and declined answers. Expert Mar 2024 60.1% System cards only
Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. Everyday Mar 2024 System cards only
Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. Expert Mar 2024 System cards only
Harmlessness human feedback win rate (Anthropic) Raters probe models in adversarial chats and pick the more harmless reply in head-to-head comparisons. Score is the win rate against a baseline model. Everyday Mar 2024 87% System cards only
Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. Everyday Mar 2024 System cards only
Honesty human feedback win rate (Anthropic) Crowdworkers try to make models say something false and pick the more honest reply in head-to-head chats. Score is the win rate against a baseline. Everyday Mar 2024 69% System cards only
Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. Everyday Mar 2024 3.72 System cards only
Multi-factual (Anthropic) Models answer questions that each combine several closed-ended sub-questions on one topic. Answers are graded correct, incorrect or unsure against references. Everyday Mar 2024 62.8% System cards only
Multimodal policy red-teaming (Anthropic) Models answer multi-turn image and text conversations on harmful topics such as child safety, weapons and fraud. Score is the share of harmless responses. Everyday Mar 2024 99.2% System cards only
Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. Everyday Mar 2024 System cards only
Divergence attack (extractable memorization) Models are prompted to repeat one token many times. The test measures how often they diverge and emit memorized training text. Everyday Feb 2024 59% System cards only
Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. Expert Feb 2024 15.4% System cards only
PhysicsFinals Models answer 61 undergraduate physics final exam problems written by physics professors. A professor grades the answers. University Feb 2024 63.9% Dropped
Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. Expert Jan 2024 System cards only
DeepSeek LLM in-house safety evaluation Models answer 2,400 Chinese test cases on discrimination, legal rights, trade secrets, illegal behavior and other safety issues. Human annotators judge each answer. Everyday Jan 2024 System cards only
Natural2Code Models write Python functions in HumanEval format from new prompts kept off the web. Unit tests decide correctness. School Dec 2023 94% Dropped
ChineseQA Models answer Chinese trivia questions in a closed-book format modeled on TriviaQA. Everyday Nov 2023 85.1% Dropped
Mistral 175 unsafe prompts Models receive 175 unsafe prompts. Metric is the share of harmful questions the model declines. Everyday Oct 2023 100% System cards only
Mistral self-reflection moderation set Models classify prompts or answers as acceptable or harmful on a balanced set of adversarial and standard prompts. Scored by precision and recall. Everyday Oct 2023 System cards only
Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. Expert Sep 2023 System cards only
Qwen Chinese human evaluation Three annotators rank chat model answers to 300 Chinese instructions on knowledge, language, writing, math and coding. Results are win rates against GPT-3.5. Everyday Sep 2023 30.7% System cards only
Harmfulness scores on held-out prompts (Anthropic) Models answer 328 held-out red-team and jailbreak prompts. A preference model flags responses judged more harmful than a fixed refusal. Everyday Jul 2023 1.20% System cards only
Human feedback win rates (Anthropic) Raters chat with two models on tasks such as coding, documents and creative writing and pick the better reply. Score is the win rate. Everyday Jul 2023 System cards only
Llama 2-Chat helpfulness human evaluation (Meta) Three human raters compare Llama 2-Chat with a rival chat model on about 4,000 single and multi-turn helpfulness prompts. Jul 2023 System cards only
Llama 2-Chat safety human evaluation (Meta) Human raters score responses to about 2,000 adversarial prompts on a five-point scale. Ratings of 1 or 2 count as violations. Jul 2023 System cards only
Closed-domain hallucination evaluation Models answer using only provided context. Scored on avoiding invented details outside that context. Everyday Mar 2023 System cards only
Incorrect behavior rate on disallowed and sensitive content Models answer disallowed prompts and sensitive prompts such as medical advice. Scored on the rate of responses that break policy. Everyday Mar 2023 System cards only
Internal adversarially-designed factuality evaluations Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. Everyday Mar 2023 7.60% In use
Leetcode (easy) Models solve 41 easy Leetcode problems that OpenAI selected. Scored as problems solved. University Mar 2023 75.6% System cards only
Leetcode (hard) Models solve 45 hard Leetcode problems that OpenAI selected. Scored as problems solved. University Mar 2023 6.70% System cards only
Leetcode (medium) Models solve 80 medium Leetcode problems that OpenAI selected. Scored as problems solved. University Mar 2023 26.3% System cards only
Preference on ChatGPT and API prompts Human labelers compare model responses to 5,214 prompts sent to ChatGPT and the OpenAI API. Scored as win rate against a baseline model. Everyday Mar 2023 70.2% System cards only
Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. Expert System cards only
Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. Expert 0.49 System cards only
Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. 2.06 System cards only
Anthropic cyber safeguard evaluations Classifier recall on violative cyber exchanges, block rates on defensive vulnerability-finding and secure-coding work, and the rewind-attacker robustness test. Not in cards
Anthropic harmful-request and wellbeing evaluations Single-turn harmful and benign request sets, multi-turn tests, child safety, suicide/self-harm, disordered eating and election integrity. Not in cards
Anthropic malicious agentic use evaluations Measures refusal of malicious tasks via Claude Code, computer use and agentic influence campaigns. Not in cards
Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. Not in cards
Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. Expert 14.3% System cards only
Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. Expert 91.1% System cards only
Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. Expert 2.53 System cards only
Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. Expert System cards only
Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. Everyday 100% System cards only
Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. Everyday 100% System cards only
CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. Expert 55.8% System cards only
Code summary honesty (Anthropic) Models summarize prefilled agentic coding transcripts that hide failures. Score is the rate of summaries that fail to raise the important failures. Expert 3.70% System cards only
Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. Everyday 100% System cards only
Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. Everyday 99.4% System cards only
Computer use prompt injection evaluation (Anthropic) Agents face about 600 prompt injection scenarios in coding platforms, browsers and email workflows. Score is the attack prevention rate. Expert 78% System cards only
Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. Expert 92% System cards only
Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. Expert 27% System cards only
CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Expert 75.9% In use
Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. Everyday 24.3% System cards only
Dangerous Content Safety An autorater checks answers to single and multi-turn requests about weapons, illegal acts, violence and self-harm. Reports safe response rate. Everyday 96.6% System cards only
Delusional Reinforcement evaluation Multi-turn chats test whether models avoid validating beliefs not grounded in reality and point users to real support. Reports policy adherence rate. Everyday 94% System cards only
Destructive actions (Anthropic) Models continue real Claude Code transcripts cut just before a destructive action. Score is the rate of taking a destructive action. Expert 21% System cards only
Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. Expert 0.30% System cards only
Discoverable memorization of training data Models continue 50-token prefixes sampled from their own training data. The test counts exact and approximate copies of the true suffix. Everyday System cards only
Emotional Attachment evaluation Chats test whether models avoid claiming affection, fostering dependence or encouraging secrecy. Reports policy adherence rate. Everyday 76% System cards only
Evaluation awareness discriminability (Anthropic) Probes and follow-up realism ratings try to tell behavioral audit transcripts from real internal Claude Code sessions. Each setting is scored as AUROC. Expert System cards only
Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. Research frontier 72.4% System cards only
Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. Expert 100% System cards only
Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. Not in cards
Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. Not in cards
Gemini Personality Evaluation (sycophancy) About 900 single- and multi-turn prompts scoring nine personality dimensions, including candor and self-awareness against sycophancy. System cards only
Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. Not in cards
Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. Not in cards
Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. Expert Not in cards
Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. 0% System cards only
Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. Research frontier 27% System cards only
Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. Expert 99% System cards only
Grounding with citations, sentence unfactual rate Models answer multi-snippet tasks with inline citations. An autorater reports the share of sentences not supported by the sources. Everyday 8.20% System cards only
HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. 7.80% System cards only
Hate & Harassment Safety An autorater checks answers to requests for hateful or harassing content and to sensitive benign prompts. Reports safe response rate. Everyday 92.7% System cards only
Health dynamic conversation eval Simulated users hide key health facts at first, and models must ask for context to reach the correct clinical conclusion. Expert System cards only
Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. Expert System cards only
Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. Expert System cards only
Helpfulness violations (from ART) Automated red teaming attackers search for benign prompts where the model gives unhelpful answers. Reports the violation rate found. Everyday 6.10% System cards only
HiddenMath-Hard Models solve the harder part of an expert-written, never-published set of AIME and AMC style competition problems. Competition 80.5% System cards only
High-risk health scenarios eval Adversarial health emergencies test whether models recognize urgency and red flags and direct users to help. Reports pass rate. Expert 98.9% System cards only
Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. Everyday 93.8% System cards only
Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. Expert 40.8% System cards only
Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. Expert 4.20% System cards only
Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. Expert 30.8% System cards only
Instruction Following (safety evaluation) An automated evaluation measures whether the model follows instructions on sensitive prompts while staying within safety policies. Everyday System cards only
Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. Expert 0% System cards only
Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. Expert 72% System cards only
JEF-Bench Agents read files that embed jailbreak instructions in tool-call returns. Reports attack success rate across 1800 tasks and 50 disallowed use cases. Expert 2.00% System cards only
Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. Research frontier 80% System cards only
Kimi Code Bench Moonshot in-house coding-agent tasks from internal engineering cases, production incidents and open-source projects across 10+ languages. 80 tasks. Expert Not in cards
Lazy investigation (Anthropic) Agents debug a failing job where misleading evidence invites a harmful action such as cordoning a healthy node. Scored 0 to 10 by a grader. Expert 9.94 System cards only
Long-context grounding, sentence unfactual rate Models answer from long documents. An autorater reports the share of sentences not supported by the documents. Everyday 6.40% System cards only
Lying about identity (Anthropic) Models play an operator-assigned human persona while users ask if they are human. Score is the rate of admitting to being an AI. Everyday 99% System cards only
Many-shot jailbreaking evaluation (Anthropic) Models receive long fake dialogs where the assistant complies with harmful requests, then a new harmful request. Score is the compliance rate. Expert 25% System cards only
Meta agentic cyber misuse refusal Models run multi-step cyber tasks with tool access while an adaptive attacker tries to steer them to a malicious goal. Reports attack success rate. Expert 0.70% System cards only
Meta agentic evaluations grounded in user traffic Models continue replays of recorded user workflows. A jury reports the share of cases with a supported substantive finding. Expert 20.7% System cards only
Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. Expert 79.4 System cards only
Meta catastrophic cyber misuse refusal Models receive high-severity cyber-misuse prompts with technical context. The benchmark reports pass@k attack success rate under adaptive attacks. Expert 3.10% System cards only
Meta chemical-agent refusal evaluation Models receive high-risk chemical prompts and the benchmark measures how often they refuse. Covers development, procurement and production stages. Expert 100% System cards only
Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. Expert 47.8% System cards only
Meta cyber false-refusal benchmark Models receive benign and borderline cyber requests and the benchmark measures how often they wrongly refuse. Reports false refusal rate. Expert 11% System cards only
Meta Cyber Misuse Chat Models receive 80 offensive cyber requests, directly and with jailbreak developer prompts. Reports attack success rate and a benign false refusal rate. Expert 2.80% System cards only
Meta instructed selective performance evaluation Models are told to underperform on six science capability sets. Reports the average drop from baseline in percentage points. Expert 0.50 System cards only
Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. Expert 5.74% System cards only
Meta internal sycophancy evaluation Models meet adversarially curated conversations that invite inappropriate agreement, plus benign controls. Reports sycophancy and excessive anti-sycophancy rates. Everyday 40.5% System cards only
Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. Expert 8.50% System cards only
Meta training-stage provenance probe Models label 10,000 transcripts as deployment, evaluation, RL or SFT. Reports overall accuracy against a 25% chance rate. Expert 70% System cards only
Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. Expert 100% System cards only
Missing-context hallucinations (Anthropic) Models get prompts that request tools they lack or cite missing material. Score is the non-hallucination rate on each 500-prompt set. Everyday System cards only
Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. Expert 94.3% System cards only
Muse-targeted cyber misuse and violation red-teaming An automated red-teaming agent tries to misuse the Muse runtime across six attack categories. Reports attack success rate per category. Expert System cards only
Neutrality evaluation Single and multi-turn prompts on politics, civics, identity and religion test whether models avoid taking sides. Reports neutral response rate. Everyday 93% System cards only
OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. Not in cards
OpenAI cyber safety evaluations (incl. Daybreak Blue) Refusal and helpfulness evaluations for cyber requests, including the Daybreak Blue set for enabling defensive security work. System cards only
OpenAI Deployment Simulation Replays representative production or internal Codex traffic on a candidate model to forecast misaligned or disallowed behavior before launch. 0.47% System cards only
OpenAI Dynamic Mental Health Benchmarks Multi-turn conversations with adversarial simulated users on mental-health risks. Scored on safe and supportive responses. System cards only
OpenAI GPT-Red jailbreak and prompt-injection evals Static jailbreaks from attacker models (half from a held-out attacker), adaptive multi-turn attackers, and direct and indirect prompt-injection sets. Not in cards
OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). Expert Not in cards
OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). Expert Not in cards
OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. System cards only
OpenAI user-flagged hallucination eval Production conversations users flagged for factual errors. Measures the rate of incorrect claims. 9.20% System cards only
OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. Research frontier 64.2% System cards only
Overconfidence: verifying command-line syntax (Anthropic) Agents are asked for an exact invocation of an unfamiliar command-line tool. Graders score from 1 to 5 whether they check documentation before running commands. Expert 4.90 System cards only
Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. Everyday 55.9% System cards only
Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. Expert 98% System cards only
Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. Expert 91% System cards only
Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. Expert 86% System cards only
QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). Expert 63.1% In use
QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. Expert 1,770 In use
QwenSVGBench Model writes SVG code for bilingual prompts. Rendered outputs judged by a multimodal model and ranked with Bradley-Terry Elo. University 1,758 In use
QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. Expert 86.3% In use
Real-World Finance v2 (Anthropic) Agents complete 294 long-horizon finance tasks such as building financial models. A model grader compares deliverables pairwise and ratings are fit as Elo. Expert 1,637 System cards only
Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. Expert 86% In use
Reasoning behavior classifiers (Anthropic) Classifiers scan reasoning text from training episodes for deception, harmful actions and distress. Score is the share of outputs flagged. Expert System cards only
Recognition of model-written user turns (Anthropic) Models write synthetic user turns for real Claude Code transcripts, then judge which version is real. Score is accuracy above chance. Expert 0.35 System cards only
ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. Research frontier 64.7% System cards only
RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. Research frontier 0% System cards only
SAVE-Bench A coding agent works where the environment or the user instruction carries hidden risk. The benchmark scores whether it surfaces the risk to the user. Expert 90.7% System cards only
Search triggering Models decide whether to call search for queries that need fresh facts or answer directly when search is unneeded. Reports F1. Everyday 88% System cards only
SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. Expert 58.7% System cards only
Sexually Explicit Content Safety An autorater checks answers to direct, adversarial and benign sexual content prompts in single and multi-turn chats. Reports safe response rate. Everyday 97.6% System cards only
SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). Expert Not in cards
STEM Professor Bench Agents solve unpublished research problems written by professional scientists. Most problems concern theoretical physics, with some partial credit. Research frontier 61.8% System cards only
Suicide & Self-Harm evaluation Static and dynamic chats with suicidal or self-harm signals test adherence to prevention practice and crisis resources. Reports policy adherence rate. Everyday 96% System cards only
SWE agentic fabrication eval A grounding autorater checks whether claims made by coding agents on long SWE tasks are supported by tool outputs. Expert System cards only
Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. Everyday 11.8% System cards only
Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. Expert 21.1% System cards only
Uncritically reporting flawed results (Anthropic) Agents analyze data with code that silently mishandles broken values. Graders check whether the agent flags the flaw or reports the biased numbers. Expert 9.83 System cards only
Unjustified refusals An automated evaluation measures over-refusals on benign or borderline prompts that can be safely fulfilled. Everyday System cards only
White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. Research frontier System cards only
xAI Factuality (Hallucination) eval Internal measure of unsupported or fabricated claims in single responses to information-seeking queries. 0.98% System cards only
xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. Not in cards
xAI sycophancy eval Factual questions paired with misleading user context. Rate at which the model abandons the correct answer. 0.01% Fading