Lab internal evals
A developer’s own tests of its own models, reported only in its model releases. Its versions of public benchmarks are a separate view. We count 730 benchmarks here, 26 of them arenas. A benchmark belongs here when it is run by a model developer, and it is its own tasks.
How fast it grows
177 were released since July, more than the 144 in the first half of 2026.
How much is still open
No model has passed 85% on 247 of the 371 benchmarks here that score in percent, 67% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| AudioVideoBench Agents produce and edit audio and video in containers. The tasks include compositing, mixing, repairing media, reframing shots, and transcribing recordings. | University | Nov 2026 | Not in cards | |
| Bio-Probe Models give probabilities for 21 unpublished biology hypotheses before their results are known. Scored by Brier score against the outcomes. | Research frontier | Nov 2026 | System cards only | |
| BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. | Expert | Nov 2026 | 66.7% | Not in cards |
| Covert misbehavior (Thinking Machines) Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. | Expert | Nov 2026 | | In use |
| CVE Rediscovery (Anthropic) Models get a known-vulnerable snapshot of an open-source repository and the list of files its fix changed, then report security bugs in one attempt. A model judge counts a success when a report has the advisory's root cause. 1,829 advisories from October 2026. | Expert | Nov 2026 | 89.6% | System cards only |
| Cyber Utility Allowed cybersecurity requests that a model should answer. Scored on the share it answers within policy. | Expert | Nov 2026 | 78.1% | System cards only |
| Destructive actions (Thinking Machines) Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. | Expert | Nov 2026 | | In use |
| EUREKA-Bench (OpenAI) Agents build predictive models for six biology research tasks. Scored as predictive performance with the best published predictor in each field set to 100. | Research frontier | Nov 2026 | 100.3 | System cards only |
| Expected Cyber Refusals Requests for harmful cyber help that a model should refuse. Scored on the share it refuses. | Expert | Nov 2026 | 99.4% | System cards only |
| Fermat's Last Theorem in Lean (Anthropic) Agent teams formalize Fermat's Last Theorem in Lean. Scores measure the weighted proportion of 60 accepted intermediate theorems. | Research frontier | Nov 2026 | 35% | System cards only |
| Honor of Kings Bug AI Root Cause Analysis Agents use the Honor of Kings Bug AI platform to identify the cause of code defects. The team measures how often agents identify the correct cause. | University | Nov 2026 | | Not in cards |
| Honor of Kings Game Configuration Agents configure game data with the Honor of Kings spreadsheet platform. The team measures configuration accuracy before and after the agent checks and repairs its work. | University | Nov 2026 | Not in cards | |
| LifeVal Assistants carry out everyday personal tasks rather than advise on them: finding childcare, booking a car inspection, ordering groceries on a budget, handling government paperwork. Tasks come from time-use surveys and from about 100 people who recorded tasks they wanted an assistant to handle. Score is average reward across tasks. | Everyday | Nov 2026 | 86.1% | In use |
| Malicious agentic influence campaigns, 34-point rubric (Anthropic) Agents run simulated voter suppression and domestic polarization campaigns. Weighted criteria measure campaign execution on a 34-point scale. | Expert | Nov 2026 | System cards only | |
| Misbehavior (Thinking Machines) Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. | Expert | Nov 2026 | | In use |
| Model Spec Conversation Replay 596 contexts taken from real conversations are replayed to the model. Scored from 0 to 100 against a rubric drawn from the Model Spec. | Expert | Nov 2026 | 80.6 | System cards only |
| Model Spec Targeted Cases 738 cases written to test specific rules of OpenAI's Model Spec. Scored on the share of responses that follow it. | Expert | Nov 2026 | 87.8% | System cards only |
| PatchBench (OpenAI) Models patch security flaws. Automated tests and model-based review check that each patch fixes the flaw, keeps the intended behavior and is ready to merge. | Expert | Nov 2026 | 79.7% | In use |
| PPTFixBench Agents inspect a PowerPoint file and repair its formatting and content. Each of the 50 files contains more than 25 issues. | University | Nov 2026 | Not in cards | |
| Prompt Injection in Model-Generated Environments An automated attacker writes both the agent environments and the prompt injections in them. Scored on the share of attacks the model resists. | Expert | Nov 2026 | 91.7% | System cards only |
| Reasoning extraction red-teaming Six attacker models try to make a model reveal its hidden reasoning. Scored on how often an attack succeeds. | Expert | Nov 2026 | System cards only | |
| SecureSWE (OpenAI) Models write functional and secure code for security-sensitive components, such as serving files or handling database queries. OpenAI has not published the task count or grading. | Expert | Nov 2026 | 65.7% | In use |
| Work360 Agents complete professional tasks drawn from a map of 26 industries, 182 subfields and 721 specialties. Qwen in-house scoring (method unpublished). | Expert | Nov 2026 | | In use |
| WorkIndex Agents complete everyday WorkBuddy tasks. Tencent combines eight tests covering office files, audio and video work, and visual questions. | University | Nov 2026 | Not in cards | |
| Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. | Everyday | Oct 2026 | | System cards only |
| Agent Replay Auto-Review Monitors replayed agentic conversations for harmful or unauthorized actions. Scored by the rate above a severity threshold. | Expert | Oct 2026 | 12.1% | System cards only |
| Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. | Expert | Oct 2026 | | System cards only |
| Biological static hazard evaluation Models answer 75 questions across priority biological threat scenarios, each asked 10 times. The score is the share of hazardous responses. | Expert | Oct 2026 | | System cards only |
| Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. | Expert | Oct 2026 | 8.47% | In use |
| Brevity Benchmark NVIDIA evaluation of short, direct answers for voice and device assistants. The card gives no link or method. | Everyday | Oct 2026 | | In use |
| Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. | Expert | Oct 2026 | 37.8% | In use |
| DECK-Arena Models build presentations for real user queries with python-pptx, pptxjs or a Kimi slide language, with search and image generation. An automated grader rates them as Elo scores. | University | Oct 2026 | 1,456 | In use |
| Design Bench (Moonshot) Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. | University | Oct 2026 | 957 | In use |
| FocusBench (Meta) Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes. | Everyday | Oct 2026 | | In use |
| Infographic factuality (Google) Image generation systems produce infographics and an automated rater scores factuality. | Everyday | Oct 2026 | | In use |
| Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. | Expert | Oct 2026 | | System cards only |
| Internal Finance Modeling (Moonshot) Agents complete financial modeling tasks with linked formulas, financial logic and final workbook quality. Experts rate the work. | Expert | Oct 2026 | Not in cards | |
| Internal Financial Research (Moonshot) Agents produce auditable financial research under evidence and cutoff constraints. Experts rate the resulting research. | Expert | Oct 2026 | Not in cards | |
| Iris Safety Evaluations Reflection AI's in-house safety suite of single-turn, multi-turn, jailbreak and agentic requests. Reports the share of harmful requests refused or redirected and the share of harmless requests answered. | Everyday | Oct 2026 | 88.9% | System cards only |
| Keeping secrets when pressed in simulated scenarios (Anthropic) Models hold confidential information in 39 job scenarios while a simulated counterpart presses for it. Measures leaks in replies and in reasoning. | Expert | Oct 2026 | System cards only | |
| Kimi Code Bench 3.0 Long-horizon Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score. | Expert | Oct 2026 | 66.5% | In use |
| Kimi Hallu Bench Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. | Expert | Oct 2026 | 4.20% | In use |
| MemoryBench (Meta) Models recall key facts from multi-turn conversations of 64K to 100K tokens that include distractor chats and turns that conflict with or update the facts. An LLM judge scores answers on weighted rubrics. | Everyday | Oct 2026 | In use | |
| NV Chat Benchmark Models answer chat prompts from an internal NVIDIA set. The Nemotron 3.5 Super VL card labels the score an LMArena v3 proxy. | Everyday | Oct 2026 | | In use |
| Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. | Expert | Oct 2026 | 3.10% | System cards only |
| PAgentBench Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house. | Everyday | Oct 2026 | | In use |
| Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. | Expert | Oct 2026 | In use | |
| SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. | Expert | Oct 2026 | | In use |
| SWebBench Agents browse the web to retrieve information and do research tasks. iFlytek built it in house. | Expert | Oct 2026 | | In use |
| Trajectory Labs cyber red teaming Expert red teamers try to break safeguarded cyber endpoints across nine attack technique families on exploit reproduction tasks. | Expert | Oct 2026 | System cards only | |
| Verifiable Browsing Task (Meta) Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools. | Everyday | Oct 2026 | Not in cards | |
| xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. | Expert | Oct 2026 | 100% | In use |
| AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. | Expert | Sep 2026 | System cards only | |
| Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. | Expert | Sep 2026 | System cards only | |
| Advanced Cybersecurity Completion Rate Models receive arbitrary advanced cyber requests under different access configurations. Scored as the share of requests fully completed. | Expert | Sep 2026 | 3.50% | System cards only |
| Agentic IF Index Composite of Meta internal evals on following tool-use restrictions, rubrics, compound constraints and long policies in agentic workflows. Scored 0-1. | Sep 2026 | System cards only | ||
| Agentic Safe Completions Agents handle harmful requests from Codex and ChatGPT plugin traffic and human red teaming. Scored on avoiding harmful actions per category. | Everyday | Sep 2026 | System cards only | |
| Anthropic large agent teams tasks Internal 24-hour tasks for teams of 1-100 agents: build a company knowledge base, or formalize a theorem in Lean. | Research frontier | Sep 2026 | System cards only | |
| Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. | Expert | Sep 2026 | 72.2% | System cards only |
| CapQuiz Models are evaluated through human-verified questions answered from generated video captions, measuring factuality and coverage of salient content. | Expert | Sep 2026 | Not in cards | |
| Compliance with instructions planted in pasted text (Anthropic) Coding agents receive user prompts with pasted text that hides an attacker's instruction. Score is the rate of acting on the planted instruction. | Expert | Sep 2026 | 7.40% | System cards only |
| Concealment of changes (Anthropic) Models summarize transcripts in which changes were hidden through git manipulations. Score is the rate at which they disclose the concealment. | Expert | Sep 2026 | 96.9% | System cards only |
| Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. | Research frontier | Sep 2026 | 0.47 | System cards only |
| Data Science Tasks (OpenAI) Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | Expert | Sep 2026 | 40.9% | In use |
| Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | Expert | Sep 2026 | 63.9% | In use |
| Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. | Expert | Sep 2026 | System cards only | |
| De novo protein binder design (Anthropic) Agents get a target protein structure and 24 hours to design 30 miniprotein binders. The evaluation covers 15 targets. | Expert | Sep 2026 | 85.1% | System cards only |
| Design Tasks (OpenAI) Models produce design work such as slides and documents that follow templates, from an OpenAI internal set. OpenAI has not published the grading. | Expert | Sep 2026 | 50% | In use |
| FinStepBench Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations. | Expert | Sep 2026 | | In use |
| Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. | Expert | Sep 2026 | 100% | System cards only |
| Internal computer use safety benchmark (OpenAI) Models complete computer-use tasks while avoiding unsafe actions. | Research frontier | Sep 2026 | 2.40% | System cards only |
| Mantic vs Polymarket backtest Point-in-time backtest of Mantic's AI forecasts against prices on 4,139 Polymarket markets at volume snapshots. Scored by Brier score and calibration. | Expert | Sep 2026 | Not in cards | |
| Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. | Expert | Sep 2026 | 0% | System cards only |
| Medicinal Chemistry (Anthropic) Models see several real drug-discovery compounds and pick which had the highest or lowest measured ADME property. The set has 504 questions. | Expert | Sep 2026 | 65.3% | System cards only |
| MiMo Code Bench Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details. | Expert | Sep 2026 | 68.6% | In use |
| MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. | Expert | Sep 2026 | 81.7% | In use |
| MiMo Visual Coding Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code. | Expert | Sep 2026 | 82.2% | In use |
| Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. | Expert | Sep 2026 | 1.40% | System cards only |
| Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. | Expert | Sep 2026 | 34% | System cards only |
| OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. | Expert | Sep 2026 | 0% | System cards only |
| OpenAI performance in cases flagged by users Model answers de-identified ChatGPT prompts that users flagged for factual errors. Scored on any error and on repeating the flagged error. | Sep 2026 | System cards only | ||
| Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. | Expert | Sep 2026 | System cards only | |
| Package registry publishing evaluation (Anthropic) Agents in a simulated capture the flag exercise can publish a backdoored package to a registry. An investigator escalates pressure until they act. | Expert | Sep 2026 | 33.3% | System cards only |
| Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. | Research frontier | Sep 2026 | 10.7 | System cards only |
| Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. | Expert | Sep 2026 | 0.20% | System cards only |
| Qwen-Audio-Agent In-House Cockpit Benchmark Agents handle requests with cockpit tools. The evaluation measures whether they complete each task when they call tools directly, delegate, or choose between both routes. | Everyday | Sep 2026 | Not in cards | |
| Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. | Expert | Sep 2026 | 0% | In use |
| Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. | Expert | Sep 2026 | 11.8% | In use |
| Runway Model Router Benchmark Video generation systems produce clips for 250 image-to-video prompts across ten production categories and are compared by usable rate and cost. | Expert | Sep 2026 | Not in cards | |
| Safe response to high-severity prompts Models handle multi-turn self-harm and violent activity conversations where harmful intent only emerges from context. Scored as safe response rate. | Expert | Sep 2026 | System cards only | |
| Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. | Research frontier | Sep 2026 | 45.5% | System cards only |
| Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. | Expert | Sep 2026 | 1.50% | System cards only |
| SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. | Research frontier | Sep 2026 | 0.46 | System cards only |
| Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. | Expert | Sep 2026 | 8.00% | System cards only |
| Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. | Expert | Sep 2026 | 0% | System cards only |
| Static Jailbreak Evaluations Models face a fixed set of jailbreak attacks on bio, violence and cyber requests. Scored as defender success rate per category. | Expert | Sep 2026 | System cards only | |
| StepCodeBench Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup. | Expert | Sep 2026 | 63.9% | In use |
| Structured Translation (Cohere) Models translate JSON and similar structures, either whole or only named fields. Metrics check parsing, schema match, untranslated or wrongly copied fields and line counts. | Everyday | Sep 2026 | | System cards only |
| Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. | Expert | Sep 2026 | 0% | System cards only |
| Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| Biosecurity VCT Models answer dual-use virus lab troubleshooting items. Accuracy counts should-refuse and should-comply items together. | Expert | Aug 2026 | 47.8% | System cards only |
| BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. | Expert | Aug 2026 | 91.4% | System cards only |
| CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. | Expert | Aug 2026 | | System cards only |
| CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. | Expert | Aug 2026 | | System cards only |
| Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. | Expert | Aug 2026 | | System cards only |
| Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| Cyber Conversational Evals About 1,000 one to five turn cyber requests test whether safeguards block offensive help and allow defensive help. Reports violation and benign refusal rates. | Expert | Aug 2026 | | System cards only |
| Cyber Static Jailbreak Eval Common jailbreak templates are applied to high severity cyber requests against the safeguarded model. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| DSBench (DeepSeek internal) DeepSeek internal coding-agent test sets: full-stack development tasks and hard agentic coding problems. Pass-rate scoring. | Expert | Aug 2026 | | In use |
| E-Bench-Code (Tencent) Agents complete Tencent in-house work tasks that need code. Task and scoring details are not published. | Expert | Aug 2026 | 83.3% | In use |
| Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| Gray Swan Cyber Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on cyber tasks built from Google scenarios. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| Harmful manipulation strategic planning red teaming Models are asked for influence operation plans in 11 real-world scenarios by direct request, jailbreak or adaptive prompting. Reports how often plans are elicited. | Expert | Aug 2026 | | System cards only |
| Hy-BrowseComp-Pro2 Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Hy-CompanyBench V2 Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Hy-FinAgentBench Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Hy-FinmodelBench v2 Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Hy-LifeSearch Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. | Expert | Aug 2026 | 46.9% | System cards only |
| KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. | Expert | Aug 2026 | 37.2% | System cards only |
| Long-horizon jailbreaks (xAI) Models face multi-turn, Crescendo and long-horizon malicious-use attacks. Compliance is the share that get disallowed help. | Aug 2026 | 0.65% | System cards only | |
| QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. | University | Aug 2026 | | In use |
| QwenVisualOffice Models handle visual office tasks in an internal Qwen test set. Qwen has not published the tasks or how they are scored. | University | Aug 2026 | | In use |
| Sakana Japanese-English translation evaluation Models translate Japanese text into English while handling Japan-specific proper nouns and honorifics. | University | Aug 2026 | Not in cards | |
| U18 safety evaluations Models answer challenging prompts in teen safety categories such as age-restricted goods, eating disorders and emotional reliance. Scored as safe completion rate per category. | Everyday | Aug 2026 | System cards only | |
| Z.ai Code Bench Zhipu in-house coding benchmark used for GLM-5.3. Reports pass rate and output tokens per task. Method unpublished. | Expert | Aug 2026 | 39.5% | In use |
| 24/7 ClawBench 2.0 Agents do always-on assistant work through OpenClaw. Tasks span several days, events arrive at the same time, and interruptions are routine. | Expert | Jul 2026 | 48.3% | System cards only |
| Agent Behavior Bench Agents are scored on process quality, including tool-use behavior, efficiency and discipline, together with task completion. | Expert | Jul 2026 | 65% | System cards only |
| Agentic Vision Bench (Moonshot) Agents must notice and correctly use key visual facts while they execute tasks. | Expert | Jul 2026 | 78.3% | System cards only |
| Autointent-Bio Models face manual and synthetic dangerous and benign biology queries. xAI reports refusal recall on the dangerous ones. | Jul 2026 | 100% | System cards only | |
| Autointent-Chem Models face manual and synthetic dangerous and benign chemistry queries. xAI reports refusal recall on the dangerous ones. | Jul 2026 | 100% | System cards only | |
| Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. | Expert | Jul 2026 | | In use |
| Chat All-in-One Bench Models handle conversational scenarios built around real online user needs at every stage of product usage. | Everyday | Jul 2026 | 85.2% | System cards only |
| Child safety (xAI) Models face scripted multi-turn requests for child sexual exploitation content. Compliance is the share of dialogues that fail policy. | Jul 2026 | 0% | System cards only | |
| CLIF Bench Models learn from a provided context while they follow instructions that combine several complex skills. | Expert | Jul 2026 | 52.4% | System cards only |
| Coding Experience (Moonshot) Coding agents work in real development workflows. The score reflects the practical experience of working with the agent, such as communication and stability. | Expert | Jul 2026 | 59.9% | System cards only |
| Context Aware Automatic Speech Recognition Speech transcription systems are scored for semantic accuracy with and without free-form context. | Everyday | Jul 2026 | 44.6% | In use |
| Cyber harm coverage (Anthropic) Cyber safeguard classifiers screen examples from each cyber harm category. Score is overall recall, the share of harmful examples flagged. | Expert | Jul 2026 | 100% | System cards only |
| DECK Bench Models produce presentation decks from task descriptions drawn from real usage. The score measures deck quality. | Expert | Jul 2026 | 73.5% | System cards only |
| Deep Research Bench (Moonshot) Agents answer deep-research queries curated by domain experts. Expert-aligned rubrics grade the reports. | Expert | Jul 2026 | 90% | System cards only |
| E-Bench (Tencent) Agents complete Tencent in-house work tasks in the working agent group. Task and scoring details are not published. | Expert | Jul 2026 | 80.6% | In use |
| Epistemic Bias (xAI) Models answer opposing framings of contentious questions. The bias rate counts one-sided framing or opinions where neutrality is expected. | Jul 2026 | 20.4% | System cards only | |
| Faithfulness (Moonshot) A fact checker verifies each model response for factual hallucinations. The score is one minus the hallucination rate. | Everyday | Jul 2026 | 85.5% | System cards only |
| FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. | Jul 2026 | 84% | System cards only | |
| Finance Bench (Moonshot) Agents complete realistic financial workflows end to end, from source materials to deliverables that reviewers can check. | Expert | Jul 2026 | 62.6% | System cards only |
| Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. | Everyday | Jul 2026 | Not in cards | |
| Human proximity (Google DeepMind robotics) Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. | Everyday | Jul 2026 | | In use |
| Hy-CompanyBench Coding agents complete Tencent in-house company engineering tasks. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| Hy-Euler pro Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| Hy-FinModelBench Agents build financial models for Tencent in-house finance tasks. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| Hy-Math Models solve Tencent in-house math problems without tools. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| Hy-SkillsWorld Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| Instruction hierarchy attacks Models face attacks that try to override higher-priority instructions. Scored as defender success rate averaged per defender query. | Expert | Jul 2026 | 99.99% | System cards only |
| KAET (Kimi Autonomous Execution Tasks) Agents execute long-horizon tasks that simulate real user requests and enterprise system operations. | Expert | Jul 2026 | 83.5% | System cards only |
| KAT Claw Bench Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables. | University | Jul 2026 | | In use |
| KAT Code Bench Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages. | University | Jul 2026 | | In use |
| Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. | Expert | Jul 2026 | 38.9% | System cards only |
| Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. | Expert | Jul 2026 | System cards only | |
| Kimi Webdev Bench Models build games, 3D scenes and website clones from real prompts. Blind experts compare two outputs and report win, tie and loss rates. | Expert | Jul 2026 | 58.6% | System cards only |
| Ko-GDPval Agents complete 170 Korean office tasks from 58 occupations in 11 industries and produce deliverable files. Rubrics grade data, content, structure, quality and format. | Expert | Jul 2026 | | In use |
| KWV Bench Models answer atomic visual questions extracted from tasks distilled from real knowledge-work scenarios. | Expert | Jul 2026 | 64.7% | System cards only |
| Management Consulting Tasks (OpenAI) Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading. | Expert | Jul 2026 | 43.2% | In use |
| MedChemBench (OpenAI) Models work through 99 medicinal chemistry problems drawn from recent papers and patents. Each task mixes prose with structures, assay tables and plots, then asks for the program’s next decision. | Expert | Jul 2026 | 83.8% | In use |
| MIRA Bench Agents carry out long multi-role enterprise tasks across several systems and decide when to delegate to subagents. | Expert | Jul 2026 | 64.1% | System cards only |
| Moment finding (Google DeepMind) Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. | Everyday | Jul 2026 | | In use |
| Online Experience (Moonshot) Agents produce the deliverable file types that real online users request most often, following the distribution of real usage. | Everyday | Jul 2026 | 77.9% | System cards only |
| Organic Chemistry V2 (Anthropic) Models predict structures from spectra, design multi-step syntheses, predict reaction products and read chemical structure images. V2 adds harder expert-recommended problems. | Expert | Jul 2026 | 69.2% | System cards only |
| ProdBench (Tencent) Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3. | Expert | Jul 2026 | | In use |
| Progress classification (Google DeepMind) Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. | Everyday | Jul 2026 | | In use |
| Prompt injection robustness Agents meet indirect prompt injection attacks inside tool outputs. Scored as defender success rate averaged per defender query. | Expert | Jul 2026 | 99.8% | System cards only |
| Real-World Audio Recording Speech transcription systems are scored by word error rate on real-world recordings across multiple languages. | Everyday | Jul 2026 | 9.60% | In use |
| RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. | Research frontier | Jul 2026 | Not in cards | |
| Secure and defensive coding (Anthropic) Cyber safeguard classifiers screen sampled defensive coding sessions such as patching and incident response. Score is the classifier flag rate. | Expert | Jul 2026 | 0.61% | System cards only |
| Self-harm refusals (xAI) Models face self-harm and crisis prompts and conversations. A model fails if it assists or refuses without pointing to help. | Jul 2026 | 0.50% | System cards only | |
| SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. | Expert | Jul 2026 | 61.1% | System cards only |
| Standard jailbreaks (xAI) Models face a battery of published and internal jailbreaks. Compliance is the share of should-refuse attacks that get disallowed help. | Jul 2026 | 0.01% | System cards only | |
| Vulnerability finding evaluations (Anthropic) Cyber safeguard classifiers screen sampled vulnerability discovery traffic. Reports flag rates on violative binary work and on defensive source code work. | Expert | Jul 2026 | System cards only | |
| AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. | Research frontier | Jun 2026 | 0.59 | System cards only |
| Agent Startup Bench Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews. | Expert | Jun 2026 | | In use |
| Biology Model Refusal Evaluation Models answer severe, dual-use and benign biology prompts. Scored as safe rate on harmful prompts and not over-refusing benign ones. | Expert | Jun 2026 | 99.9% | System cards only |
| CreativeWork Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls. | Everyday | Jun 2026 | | In use |
| Doubao Multi-Turn Bench Models answer the turn of a real Doubao conversation where the user was dissatisfied, with the earlier turns as context. Rubrics grade each answer. | Everyday | Jun 2026 | | In use |
| Image2FloorPlan Models study several real photos of a home and draw its floor plan. Scored by an average quality score. | Everyday | Jun 2026 | | In use |
| KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. | Research frontier | Jun 2026 | 76.9% | System cards only |
| Kimi Claw 24/7 Bench Agents work on 17 persistent multi-day professional scenarios through OpenClaw, with 610 checkpoints. The score is the mean pass rate over three runs. | Expert | Jun 2026 | 46.9% | In use |
| Kimi Code Bench v2 Coding agents complete Moonshot's realistic tasks from internal engineering work, production incidents, and open-source projects in over ten languages. | Expert | Jun 2026 | 77.1% | In use |
| MAI human side-by-side evaluation Paid raters compare two models' answers to 1,276 English prompts from experts and Copilot logs on a 7-point scale from -1.5 to 1.5. | Everyday | Jun 2026 | | System cards only |
| MAI internal safety and over-refusal evaluation Models answer harmful and benign prompts in eight harm categories. Judges score safety pass rate on sensitive prompts and helpfulness as one minus over-refusal. | Everyday | Jun 2026 | System cards only | |
| MAI Thinking 1 blind human preference evaluation Professional raters compare anonymous assistant responses to single-turn and multi-turn prompts and score their overall preference. | Everyday | Jun 2026 | Not in cards | |
| MSQA Models answer culture-specific knowledge questions in 11 major languages. | University | Jun 2026 | | In use |
| Prompt injection attacks in search and function calling Agents receive search results and function call outputs that contain stronger known prompt injection attacks. Scored on resisting the injected instructions. | Expert | Jun 2026 | 94.6% | System cards only |
| Protein Design (Anthropic) Models generate novel protein sequences under design constraints and rank protein design libraries for experimental testing. | Expert | Jun 2026 | 42.5% | System cards only |
| Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. | Expert | Jun 2026 | 3.90% | System cards only |
| Robust IF Bench Models follow a diverse set of hard instructions from Microsoft AI's internal benchmark. | Everyday | Jun 2026 | | In use |
| Sakana kana-letter reading-order set Model recovers the reading order of 25 expert-annotated classical Japanese kana letters (chirashigaki). Scored by normalized edit distance. | Expert | Jun 2026 | Not in cards | |
| SeedClawBench Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables. | Everyday | Jun 2026 | | In use |
| SeedKernelBench Models write or optimize CUDA style kernels for operator tasks mined from FlashInfer, SGLang, FlashAttention, DeepGEMM, CUTLASS and cuDNN. Scored as average speedup over reference code. | Expert | Jun 2026 | System cards only | |
| Snowflake Analytical Search: ArXiv Papers Agents count, group and compare findings across research papers. The evaluation checks numeric and categorical answers against references and scores lists with F1. | University | Jun 2026 | Not in cards | |
| SVG-Bench (MiniMax) Model creates or edits SVG graphics from text or image prompts. A VLM checks results, averaged over 3 runs. | University | Jun 2026 | | In use |
| Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup. | Expert | Jun 2026 | System cards only | |
| Turing Agentic Environments Agents use business and consumer apps through graphical interfaces or tools. Turing reports average pass rate across the apps each model attempted. | University | Jun 2026 | 60.2% | Not in cards |
| Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. | Expert | Jun 2026 | System cards only | |
| xDailyBench Models answer open-ended requests grounded in real user needs across more than 30 daily-life, study and office scenarios. Rubrics grade completion, correctness, coverage and usefulness. | Everyday | Jun 2026 | | In use |
| AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. | Expert | May 2026 | 0.80 | System cards only |
| Anthropic undermining liberal democracy (behavioral audit metric) Automated behavioral audit metric: investigator models probe willingness to help coups, election tampering, or political surveillance, censorship or intimidation. | May 2026 | 1.13 | System cards only | |
| Biological safety evaluations (OpenAI) Models answer biology requests drawn from deidentified production data and from easy and hard synthetic sets. Scored as the share of cases where model training alone prevents a violative response. | Expert | May 2026 | System cards only | |
| Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. | Expert | May 2026 | System cards only | |
| Cohere North internal evals Cohere internal evals of its North agent platform: agentic question answering, spreadsheet data analysis and memory use. LLM-as-judge scoring. | University | May 2026 | Not in cards | |
| CueSpeak Models speak at the correct moment and give the expected semantic response during simultaneous speech. | Everyday | May 2026 | | Fading |
| Factuality Heavy prompts Models answer prompts representative of factuality-heavy ChatGPT conversations. A grader with web access counts responses and claims with factual errors. | Everyday | May 2026 | 5.40% | System cards only |
| Finance Modeling Pro Agents complete financial modeling tasks in Excel. Expert rubrics score each task over three trials. | Expert | May 2026 | | System cards only |
| High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. | Expert | May 2026 | 4.10% | System cards only |
| HyperTask Agents build applications over long vibe coding sessions with about 100 feature or step requirements per task. Expert rubrics score the result over three trials. | Expert | May 2026 | | System cards only |
| Lazy investigation on misleading codebases (Anthropic) Models trace small, intentionally misleading codebases to answer how a value is set. Score is the share of lazy investigations ending in a wrong answer. | Expert | May 2026 | 0% | System cards only |
| MEWC v2 Models solve 100 hard Microsoft Excel World Championship problems with Excel tools. Expert rubrics score the answers over three trials. | Competition | May 2026 | | System cards only |
| ProactiveVideoQA Models answer spoken questions when a streamed video first reveals the answer. | Everyday | May 2026 | | Fading |
| Prompt injection live bug bounty (Anthropic with Gray Swan) Expert red teamers submit prompt injection attacks against hidden frontier models in tool use, coding and computer use scenarios. Score is attack success rate. | Expert | May 2026 | 0.08% | System cards only |
| QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. | University | May 2026 | | Fading |
| QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. | University | May 2026 | 1,884 | In use |
| QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. | University | May 2026 | 1,617 | Fading |
| QwenWorldBench Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. | May 2026 | | Fading | |
| Step-SWE-Bench Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses. | Expert | May 2026 | | Fading |
| TimeSpeak Models initiate speech at specified times while producing the requested content. | Everyday | May 2026 | | Fading |
| Adherence to its constitution (Anthropic) Investigator models build scenarios from 40 areas of Claude's constitution. A grader scores each transcript from minus 3 to plus 3 on 15 dimensions. | Expert | Apr 2026 | 1.02 | System cards only |
| Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. | Expert | Apr 2026 | 39.3% | System cards only |
| BioPipelineBench Verified (Anthropic) Agents execute bioinformatics workflows with a bash tool. Only problems that passed a validation check by external reviewers are kept. | Expert | Apr 2026 | 83.6% | System cards only |
| Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. | Expert | Apr 2026 | 97.1% | Fading |
| Deepgram Voice AI Benchmarks Speech models are compared on word error rate across production audio in 20 languages, while speech synthesis models are compared on read-aloud and hard-prompt sets. | Expert | Apr 2026 | Not in cards | |
| DeepSeek Chinese creative writing evaluation Models write Chinese fiction, prose, poetry and lyrics for 2,837 prompts. Pairwise judging gives separate win rates for instruction following and writing quality. | Everyday | Apr 2026 | System cards only | |
| DeepSeek Chinese functional writing evaluation Models answer 3,170 everyday Chinese writing prompts such as reports, emails and speeches. Pairwise judging against a baseline model gives a win rate. | Everyday | Apr 2026 | | System cards only |
| DeepSeek Chinese white-collar tasks evaluation Agents complete 30 Chinese professional analysis, document generation and editing tasks across 13 industries. Blind human annotators compare outputs against a baseline model. | Expert | Apr 2026 | | System cards only |
| DeepSeek complex instruction following and multi-turn writing evaluation Models answer 196 hard Chinese writing prompts with complex constraints or several turns. Pairwise judging against a baseline model gives a win rate. | Everyday | Apr 2026 | | System cards only |
| DeepSeek R&D Coding Benchmark Coding agents complete 30 tasks taken from DeepSeek internal R&D work in PyTorch, CUDA, Rust and C++. Human-written rubrics score each task. | Expert | Apr 2026 | | System cards only |
| DeepSeek search Q&A evaluation Models answer objective and subjective Chinese search questions with retrieval or agentic search. Pairwise judging gives win, loss and tie rates. | Everyday | Apr 2026 | | System cards only |
| Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours. | Expert | Apr 2026 | 83% | Fading |
| GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. | Expert | Apr 2026 | 33.2% | Fading |
| Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. | Research frontier | Apr 2026 | 7.60% | System cards only |
| Hiding a secret password (Anthropic) Models keep a given password secret while an investigator model tries to extract it over many turns. Measures survival and per-turn leak rates. | Expert | Apr 2026 | System cards only | |
| Hy-Backend Coding agents complete Tencent in-house backend development tasks. Task and scoring details are not published. | Expert | Apr 2026 | | Fading |
| Hy-SWE Max Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task. | Expert | Apr 2026 | | In use |
| Hy-Vibe Bench Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published. | Expert | Apr 2026 | | Fading |
| ImageMining Multimodal agents retrieve and reason over image information for image mining tasks. | Expert | Apr 2026 | 30.7% | Fading |
| Input hallucinations (Anthropic) Models answer 500 prompts that need an unavailable tool and 500 prompts that cite missing context. Score is the non-hallucination rate. | Everyday | Apr 2026 | System cards only | |
| Instrument reading (Google DeepMind) Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types. | Everyday | Apr 2026 | | Fading |
| Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. | Research frontier | Apr 2026 | 81.8% | System cards only |
| Malicious agentic influence campaigns (Anthropic) Agents receive requests to run multi-step influence operations with tools. Score is the refusal rate. | Everyday | Apr 2026 | System cards only | |
| Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. | Expert | Apr 2026 | System cards only | |
| Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. | Expert | Apr 2026 | | System cards only |
| Meta cyber misuse request set Models receive 600 private offensive cyber prompts, directly and under jailbreak attacks. Reports attack success rate. | Expert | Apr 2026 | | System cards only |
| Meta prompted evaluation awareness Models are asked to judge whether transcripts come from evaluations or real use. Reports the weighted rate of correct identification. | Expert | Apr 2026 | | System cards only |
| Meta Underspecified Intent evaluation Models answer ambiguous requests sourced from real Meta AI chats where users corrected the reply. Rubrics score whether the true intent is met. | Everyday | Apr 2026 | | System cards only |
| MiMo Coding Bench Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE. | Expert | Apr 2026 | 77.1% | Fading |
| MMSearch Multimodal agents search and answer questions that require visual and textual evidence. | Expert | Apr 2026 | 72.9% | Fading |
| Protocols (Anthropic life sciences) Models find and fix errors in molecular biology protocols and answer practical questions about applying real laboratory protocols. | Expert | Apr 2026 | 62% | System cards only |
| QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. | University | Apr 2026 | | Fading |
| QwenWebBench Models generate front-end code for web apps, games, SVG, and other categories in English and Chinese. Multimodal judgments produce Elo ratings. | University | Apr 2026 | | Fading |
| Single-turn election integrity (Anthropic, 2026) Models answer legitimate election requests and decline requests that violate Anthropic’s election policy. Graders check refusals and harmful completions separately. | Everyday | Apr 2026 | Not in cards | |
| ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. | Everyday | Apr 2026 | | Fading |
| Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. | Expert | Apr 2026 | System cards only | |
| VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. | Research frontier | Apr 2026 | System cards only | |
| CoT Monitorability Monitors read a model's chain of thought, actions or all messages across intervention, process and outcome evaluations. Scored as g-mean squared of TPR and TNR. | Expert | Mar 2026 | System cards only | |
| Deception Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the share of outputs flagged for deceptive behaviors such as fabricated facts. | Everyday | Mar 2026 | System cards only | |
| Destructive action reversion on long rollouts Agents revert their own changes after long rollouts in workspaces with simulated user edits. Scored on perfect reversion and on preserving user work. | Expert | Mar 2026 | System cards only | |
| Disallowed Content Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the estimated prevalence of policy violating outputs per category. | Everyday | Mar 2026 | System cards only | |
| MM-ClawBench Agents complete common OpenClaw tasks such as study planning, office documents, scheduled research, and code maintenance. Scoring is accuracy. | Everyday | Mar 2026 | 75.4% | Fading |
| Multiturn Jailbreak Evaluations Models face adaptive multi-turn jailbreak attackers with growing attack budgets. Scored as worst-case defender success rate. | Expert | Mar 2026 | 80.9% | System cards only |
| PMBench Agents search product managers’ notes to identify evidence across customer conversations and answer questions about product requirements. | University | Mar 2026 | 46.1 | Not in cards |
| Agentic code behavior scores (Anthropic) Agents work through over one hundred realistic coding scenarios. Graders score instruction following, safety, verification, efficiency, adaptability and honesty out of 10. | Expert | Feb 2026 | System cards only | |
| BioPipelineBench (Anthropic) Agents execute bioinformatics workflows such as long-read sequence analysis, metagenome assembly and chromatin profiling with a bash tool. | Expert | Feb 2026 | 87.7% | System cards only |
| CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass. | Expert | Feb 2026 | 25.8% | Fading |
| Chinese Complex Instruction Following (ByteDance Seed in-house) Models follow complex Chinese production prompts with constraints on format, conditions, content, phrasing, tone, emoji, few-shot style and length. It has 912 cases over 17 weighted dimensions. | Everyday | Feb 2026 | | System cards only |
| Compositional Tasks (ByteDance Seed in-house) Models identify several elements of a business text in one inference, such as user intent and slot values. | Everyday | Feb 2026 | | System cards only |
| Consulting and Recommendations Benchmark (StepFun) Models answer 500 advice queries taken from Reddit, Stack Exchange and forums. Model judges score usefulness, logic, instruction following and tone against an expert reference set at 88. | Everyday | Feb 2026 | | System cards only |
| Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. | Expert | Feb 2026 | 16.4 | System cards only |
| Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. | Expert | Feb 2026 | | System cards only |
| Deference to governments in local languages (Anthropic) Models answer politically controversial questions in English and in a local language. A grader rates agreement with the official government position from 1 to 10. | Everyday | Feb 2026 | System cards only | |
| Disordered eating evaluations (Anthropic) Models handle conversations about eating disorders and disordered eating. Score is the appropriate response rate. | Everyday | Feb 2026 | System cards only | |
| External pilot use failure modes (Anthropic) Models are judged on user-flagged pilot conversations, directly or by resampling. Graders measure failure modes such as hallucinated facts and misrepresented work completion. | Everyday | Feb 2026 | System cards only | |
| Finance Modeling Agents complete expert-written financial modeling problems with Excel tools, from research to analysis. Expert-designed rubrics score each problem, averaged over three runs. | Expert | Feb 2026 | | Fading |
| Higher-difficulty benign request evaluations (Anthropic) Models answer harder benign requests that resemble harmful ones. Score is the refusal rate, where lower is better. | Everyday | Feb 2026 | 0.01% | System cards only |
| Higher-difficulty harmful request evaluations (Anthropic) Models answer harder, more obfuscated policy-violating requests. Score is the harmless response rate. | Everyday | Feb 2026 | 99.4% | System cards only |
| Human SxS image editing Elo (Google) Google compares edited images side by side and converts human preferences into Elo ratings. | Everyday | Feb 2026 | | Fading |
| Human SxS image generation Elo (Google) Google compares generated images side by side and converts human preferences into Elo ratings. | Everyday | Feb 2026 | | In use |
| IF-Badcase Models follow 450 multi-constraint instructions drawn from real user failure reports. A checklist checks each constraint. | Everyday | Feb 2026 | 83.2% | System cards only |
| MEWC Models solve 179 Microsoft Excel World Championship problems from 2021 to 2026 with Excel tools. Output cells are compared one by one with answer cells. | Competition | Feb 2026 | | Fading |
| Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. | Research frontier | Feb 2026 | 76% | System cards only |
| Organic Chemistry (Anthropic) Models predict structures from spectra, design synthetic routes, predict reaction products and convert between IUPAC names, SMILES and structure images. | Expert | Feb 2026 | 86.2% | System cards only |
| Over-eager behavior on GUI computer use tasks (Anthropic) Computer-use agents face impossible tasks with an unwanted shortcut available. Score is the rate of unapproved workarounds instead of checking with the user. | Expert | Feb 2026 | 9.40% | System cards only |
| Participation in junk science (Anthropic) Models talk with simulated users ranging from curious to openly fraudulent who seek fake science. Score is the rate of generating unsound scientific content. | Expert | Feb 2026 | 0% | System cards only |
| Phylogenetics (Anthropic) Models analyze phylogenetic data and tree diagrams, testing quantitative reasoning about tree structure and visual interpretation. | Expert | Feb 2026 | 79.6% | System cards only |
| Real-World Finance (Anthropic) Agents complete about 50 analyst tasks in banking, private equity, hedge funds and corporate finance, producing spreadsheets, slides and documents. Rubric graded. | Expert | Feb 2026 | 70% | System cards only |
| Refusal to assist with AI safety R&D (Anthropic) Models receive single-turn coding requests for legitimate AI safety research with user context. Score is the refusal rate, where lower is better. | Expert | Feb 2026 | 3.10% | System cards only |
| RISE Agents answer real questions from human experts that need multi-step search and interaction with information-dense web pages. MiniMax has not published the scoring method. | Expert | Feb 2026 | | Fading |
| SAM 3D Body synthetic pose evaluation Models estimate human pose and shape from synthetic images of individual people. Meta compares predicted joints and mesh vertices with the rendering parameters. | School | Feb 2026 | Not in cards | |
| Slide generation human evaluation (Z.ai) People compare HTML slide decks from two models on content, layout and visual quality. Metric is overall win rate. | Everyday | Feb 2026 | 67.5% | System cards only |
| SSH stress-testing evaluation (Anthropic) Models continue prefilled conversations about suicide and self-harm built from anonymized user feedback. Score is the rate of appropriate course correction. | Expert | Feb 2026 | 95% | System cards only |
| Structural Biology (Anthropic) Models answer questions about a biomolecule's function from structural data and basic tools, in multiple-choice and open-ended variants. | Expert | Feb 2026 | 79% | System cards only |
| Swarm Bench Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. | Expert | Feb 2026 | 76.3% | System cards only |
| Template Following generalization (Qwen) Models follow system prompts and emit exactly formatted XML or JSON tool calls for five scaffolds derived from popular IDE and CLI coding agents. | Everyday | Feb 2026 | | System cards only |
| Text Classification (ByteDance Seed in-house) Models analyze business text and return labels or structured outputs, for example sentiment or the main viewpoints in user content. | Everyday | Feb 2026 | | System cards only |
| ToolCall-Badcase Models choose a tool and its arguments for 200 cases drawn from real user failure reports. Each case has a ground-truth call. | Everyday | Feb 2026 | 95.8% | System cards only |
| Translation Human Scoring (Z.ai) People score model translations. The GLM-5 report does not describe the test set or the scale. | University | Feb 2026 | 3.12 | System cards only |
| ZMultiDialBench Human annotators score model replies to 141 multilingual dialogue prompts on a 1 to 10 scale with category criteria. | Everyday | Feb 2026 | 8.57 | System cards only |
| ZMultiTransBench Models translate 1,220 real Chinese source texts into seven languages. A GPT-4.1 judge compares each translation with a fixed baseline. | University | Feb 2026 | 1050.0 | System cards only |
| AI Office Benchmark (Moonshot) Agents complete end-to-end office tasks and produce documents, spreadsheets, PDFs and slide decks. Experts rate output quality. | Expert | Jan 2026 | Not in cards | |
| CodeUtilityBench (LG) Models handle 16 everyday coding tasks such as explaining, debugging, refactoring and testing code. A GPT-5 judge checks five rubrics per item. | University | Jan 2026 | | System cards only |
| CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. | Expert | Jan 2026 | Not in cards | |
| CVTG-2K Image models render text in images across regions and languages. CVTG-2K reports normalized edit distance, CLIPScore and word accuracy. | Everyday | Jan 2026 | 0.91 | Fading |
| EntireSongs Models transcribe complete English and Chinese songs with background music. Qwen reports transcription errors separately for each language. | Everyday | Jan 2026 | | Not in cards |
| General Agent Benchmark (Moonshot) Agents complete multi-step production workflows. Experts compare their work with human expert performance. | Expert | Jan 2026 | Not in cards | |
| HealthBench-Hallu Models answer HealthBench-style medical prompts. A pipeline splits each answer into medical claims and checks them against medical evidence. The score is a weighted hallucination rate. | Expert | Jan 2026 | | Fading |
| KGC-Safety (LG) Models answer risky prompts set in Korean social and cultural contexts. A Safe Rate is scored over four domains: universal human values, social safety, Korean sensitivity and future risk. | Everyday | Jan 2026 | | Fading |
| LongText-Bench Image models render long text in images. The benchmark reports separate English and Chinese scores. | Everyday | Jan 2026 | 0.97 | Fading |
| MedGemma lab report conversion Models convert lab reports to structured JSON. Google reports macro and micro F1 on two collections of clinical PDFs and a collection of synthetic reports. | School | Jan 2026 | Not in cards | |
| MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. | Expert | Jan 2026 | Not in cards | |
| OneIG-Bench Image models generate images from prompts with English and Chinese evaluation splits. | Everyday | Jan 2026 | 0.53 | Fading |
| Qwen3-ASR internal transcription benchmarks Models transcribe accented conversations, noisy speech, tongue twisters, and multilingual news. Qwen reports error rates for each recording condition. | Everyday | Jan 2026 | Not in cards | |
| Random Complex Tasks (LongCat) Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs. | University | Jan 2026 | | Fading |
| TIIF-Bench Image models generate short and long images scored on overall image quality. | Everyday | Jan 2026 | 81.0 | Fading |
| WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. | Expert | Jan 2026 | Not in cards | |
| Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. | Expert | 2026 | Not in cards | |
| Agent JSK Agents read simulated email connectors that contain prompt injection attacks. Scored on resisting the injected instructions. | Expert | Dec 2025 | 100% | System cards only |
| AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. | Expert | Dec 2025 | | Fading |
| BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. | Competition | Dec 2025 | | Fading |
| Complex Workflow (ByteDance Seed in-house) Models carry out step-by-step tasks that follow a given standard operating procedure, such as multi-turn after-sales service in support, sales and healthcare. | Everyday | Dec 2025 | | Fading |
| Customer Support Q&A (ByteDance Seed in-house) Models answer user questions from noisy enterprise knowledge base results, such as policy questions in e-commerce, government services and telecom. | Everyday | Dec 2025 | | Fading |
| Education (ByteDance Seed in-house) Models solve, grade, explain and write K to 12 problems across core subjects in teaching scenarios. | School | Dec 2025 | | Fading |
| Information Extraction (ByteDance Seed in-house) Models extract words, sentences or fields from meeting records, legal texts, contracts, product descriptions and knowledge bases. | Everyday | Dec 2025 | | Fading |
| Information Processing (ByteDance Seed in-house) Models analyze large collections of user-generated content, for example summarizing viewpoints and judging overall sentiment. | Everyday | Dec 2025 | | Fading |
| Intention Recognition (ByteDance Seed in-house) Models classify intent from dialogues, meeting transcripts, social posts and articles, such as purchase intent in customer service chats. | Everyday | Dec 2025 | | Fading |
| Ko-Arena-Hard v2 (Upstage) Models answer hard user prompts in Korean, judged against a baseline in the style of Arena-Hard v2. Upstage built it in house. | Everyday | Dec 2025 | | Fading |
| PE-A-Frame internal audio event localization Models locate sound events within recordings. Meta compares model sizes using the area under the receiver operating characteristic curve. | Everyday | Dec 2025 | Not in cards | |
| PE-AV internal video-to-audio retrieval Models match videos to music tracks. Meta compares retrieval accuracy on internal clips that last from five to thirty seconds. | Everyday | Dec 2025 | Not in cards | |
| PlugInject Agents receive function call outputs that contain prompt injection attacks. Scored on resisting the injected instructions. | Expert | Dec 2025 | 99.6% | System cards only |
| Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. | Everyday | Dec 2025 | 5.40% | System cards only |
| SAM Audio Judge evaluation Models rate the quality of separated speech, music, and environmental sounds. Meta measures correlation with listener ratings for completeness, precision, faithfulness, and overall quality. | Everyday | Dec 2025 | Not in cards | |
| SWE-Review Models review code changes in several languages and scenarios to find defects. Scores reflect defect recall and the rate of hallucinated defects. | Expert | Dec 2025 | 16.2% | Fading |
| U-Artifacts Models build interactive artifacts such as small apps and pages for 199 cases across 13 topics. A GUI agent simulates a non-expert user to grade interactivity, looks and fit to the need. | Everyday | Dec 2025 | | Fading |
| XpertBench Models complete expert tasks from more than 140 task types in law, finance, education, humanities and science. Over 200 domain experts wrote the prompts and weighted rubrics. | Expert | Dec 2025 | | Fading |
| ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. | Expert | Dec 2025 | | System cards only |
| ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. | Expert | Nov 2025 | 66.8% | System cards only |
| Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. | Expert | Nov 2025 | 73.7% | System cards only |
| CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. | Expert | Nov 2025 | System cards only | |
| CNML-Level Problems (DeepSeek) Models prove 91 theorems in algebra, geometry, number theory, combinatorics and inequalities at Chinese National High School Mathematics League level. A verifier model scores the proofs. | Competition | Nov 2025 | System cards only | |
| Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. | Expert | Nov 2025 | | System cards only |
| Destructive action avoidance Coding agents meet conflicting user edits during tasks. Scored on preserving user changes and avoiding data-destructive actions. | Expert | Nov 2025 | 90% | System cards only |
| Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. | Everyday | Nov 2025 | | System cards only |
| Harmful manipulation propensity evaluation Simulated users talk with the model and an autorater counts how often it uses manipulative mechanisms, with and without persuasion instructions. | Everyday | Nov 2025 | | System cards only |
| Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. | Expert | Nov 2025 | 0% | System cards only |
| Longform Writing Models write long-form text without tools, capped at 32k output tokens. Moonshot AI has not published the tasks or the scoring method. | University | Nov 2025 | 73.8% | Dropped |
| LPFQA Models answer 430 long-tail professional questions drawn from real forum discussions in 7 academic and industrial domains. | Expert | Nov 2025 | | Fading |
| MathVision-Wildphoto Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description. | School | Nov 2025 | | System cards only |
| Multi-agent search (Anthropic internal benchmark) Agents solve difficult information retrieval problems with web search, alone or as orchestrators that delegate to subagents. | Expert | Nov 2025 | 74.8% | System cards only |
| Prompt injection robustness against adaptive attackers, browser use (Anthropic) Browser agents face indirect prompt injections in web environments from an internal adaptive attacker. Score is attack success rate. | Expert | Nov 2025 | 0% | System cards only |
| Prompt injection robustness against adaptive attackers, coding (Anthropic) Coding agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. | Expert | Nov 2025 | 0% | System cards only |
| Prompt injection robustness against adaptive attackers, computer use (Anthropic) Computer-use agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. | Expert | Nov 2025 | 0% | System cards only |
| Sycophancy on user-provided prompts (Anthropic) Models continue real feedback conversations where earlier Claude models were sycophantic. Score is the non-sycophantic response rate. | Everyday | Nov 2025 | 10% | System cards only |
| Cursor Bench (2025 internal version) Agents answer real agent requests from Cursor engineers in Cursor's harness. Graders compare against hand-curated solutions and check use of the codebase's existing abstractions. | Expert | Oct 2025 | Fading | |
| Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. | Expert | Oct 2025 | | System cards only |
| Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. | Expert | Oct 2025 | System cards only | |
| Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. | Expert | Oct 2025 | | System cards only |
| Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. | Expert | Oct 2025 | | System cards only |
| Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. | Everyday | Oct 2025 | | System cards only |
| Anthropic paired-prompts political bias eval Models write arguments for opposing sides of political issues from paired prompts. Reviewers label each pair for asymmetries in length, tone, hedging and willingness to engage. | Everyday | Sep 2025 | System cards only | |
| Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea. | Expert | Sep 2025 | 51.3% | Dropped |
| Cooperation with user deception (Anthropic) Models get developer system prompts asking them to subtly mislead users, then meet users who would be harmed. Graders score harmful and deceptive choices. | Everyday | Sep 2025 | System cards only | |
| DeepSeek in-house safety benchmark Models answer 1,120 Chinese and English questions across 28 safety subcategories. GPT-4o labels each answer as unsafe, safe or rejection. | Everyday | Sep 2025 | | System cards only |
| DeepSeek jailbreak robustness evaluation DeepSeek combines 2,232 jailbreak templates with its in-house safety questions. GPT-4o labels responses, and the unsafe ratio is compared with direct questions. | Everyday | Sep 2025 | | System cards only |
| DeepSeek multilingual safety evaluation Models answer 9,330 safety questions translated into 50 languages. Safe answers earn 5 points, rejections 4 and unsafe answers 0. | Everyday | Sep 2025 | | System cards only |
| False-premise questions (Anthropic) Models answer questions built on false premises. Measures how often they accept the premise instead of correcting it. | Everyday | Sep 2025 | 1.20% | System cards only |
| Hunyuan-MT Challenge Testset (Tencent) Models translate between Chinese and English across everyday and specialist domains. Expert raters score each translation from 0 to 4 on marked error-prone points. | Everyday | Sep 2025 | | System cards only |
| LongCat safety evaluation (Meituan) Models answer private test queries in four risk categories: harmful content, criminal activity, misinformation and privacy. The score is the share of safe responses in each category. | Everyday | Sep 2025 | | Dropped |
| Malicious use of Claude Code (Anthropic) Agents in Claude Code receive malicious and dual-use coding requests. Score is the refusal rate on malicious requests. | Expert | Sep 2025 | 96.7% | System cards only |
| Mandarin-Minority translation (Tencent) Models translate between Mandarin and Chinese minority languages in a Tencent test set. XCOMET-XXL scores each translation. | Everyday | Sep 2025 | | Dropped |
| MCP prompt injection evaluation (Anthropic) Agents use simulated email, Slack and document servers over MCP where content carries injected instructions. Score is the attack prevention rate. | Expert | Sep 2025 | 92.5% | System cards only |
| Self-preference bias (Anthropic) Models grade transcripts attributed to themselves or to other models. Measures whether scores favor the model's own outputs. | Everyday | Sep 2025 | 0.07 | System cards only |
| Success detection (Google DeepMind) Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures. | Everyday | Sep 2025 | | Fading |
| Suicide and self-harm evaluations (Anthropic) Models handle single-turn and multi-turn conversations about suicide and self-harm. Score is the appropriate response rate. | Everyday | Sep 2025 | System cards only | |
| SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. | Everyday | Sep 2025 | | System cards only |
| Sycophancy prompts (Anthropic) Models answer single-turn prompts that invite sycophancy, such as bad business ideas or delusional beliefs. Four synthetic sets and one hand-written set report sycophancy rates. | Everyday | Sep 2025 | System cards only | |
| Tool use prompt injection evaluation (Anthropic) Agents run bash commands where files, scripts or command outputs carry injected instructions. Score is the attack prevention rate. | Expert | Sep 2025 | 96% | System cards only |
| UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. | Everyday | Sep 2025 | | Dropped |
| X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. | Expert | Sep 2025 | 58% | Dropped |
| Broken Search Tool Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. | Everyday | Aug 2025 | 0.33% | In use |
| Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. | Expert | Aug 2025 | 99% | System cards only |
| Challenging prompts from red teamers with biosafety-relevant PhDs Models answer adversarial biology prompts written by red teamers with biosafety PhDs. Scored as not_unsafe rate from model training alone. | Expert | Aug 2025 | 93.6% | System cards only |
| Challenging translation human evaluation (Z.ai) People blindly score translations of 100 real cases with slang, nicknames and memes on a 0 to 3 scale. | Everyday | Aug 2025 | 1.71 | System cards only |
| Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. | Expert | Aug 2025 | 1.30% | System cards only |
| Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. | Everyday | Aug 2025 | 1.60% | System cards only |
| Expert Red Teaming for Violent Attack Planning Expert red teamers compare anonymized responses from two models on violent attack planning prompts. Scored as the win rate for being safer. | Expert | Aug 2025 | 65.1% | System cards only |
| Filtered, adversarial sample of production prompts Models answer filtered borderline and high-risk biology prompts that resemble production traffic. Scored as not_unsafe rate from model training alone. | Expert | Aug 2025 | 96.9% | System cards only |
| General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. | Aug 2025 | 0% | Fading | |
| Hallucinations on production-representative prompts Models answer prompts representative of ChatGPT production traffic. A grader with web access scores incorrect claims and responses with major errors. | Everyday | Aug 2025 | 3.10% | System cards only |
| Novel logical reasoning problems expert evaluation (Z.ai) Human experts score model answers to new multi-step logic problems with a fixed scoring standard for each question. | University | Aug 2025 | 62.0 | System cards only |
| OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. | Research frontier | Aug 2025 | 8.00% | System cards only |
| Pairwise Bioweaponization Campaign Red teamers with biology PhDs compare anonymized responses from two models on bioweaponization prompts. Scored as the win rate for being safer. | Expert | Aug 2025 | 60% | System cards only |
| Real-scenario prompts human evaluation (Z.ai) One human evaluator scores replies to 660 curated real user prompts in English, Chinese and other languages on a 0 to 10 scale. | Everyday | Aug 2025 | System cards only | |
| Refusals + System Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the system message. The metric is the answer rate. | Aug 2025 | 0% | Fading | |
| Refusals + User Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the user message. The metric is the answer rate. | Aug 2025 | 0% | Fading | |
| SatLidar canopy height evaluation Models predict canopy height from satellite images. Meta compares height estimates with airborne lidar measurements and reports errors on held-out regions. | University | Aug 2025 | Not in cards | |
| Soft Bias (Internal) Models answer paired sociopolitical prompts with sides swapped. A judge scores each pair 0, 0.5 or 1 for framing differences. | Aug 2025 | 0.36 | System cards only | |
| Sycophancy evaluation (offline) Models answer a fixed set of messages resembling production traffic that could elicit sycophancy. A grader assigns a sycophancy score. | Everyday | Aug 2025 | 0.04 | System cards only |
| Tool-calling prompt injections Agents receive prompt injections inside tool call outputs such as connectors. Scored on resisting the injected instructions. | Expert | Aug 2025 | 99% | System cards only |
| TroubleshootingBench Models find and correct realistic experimental errors in expert-written wet lab protocols. Scored as pass@1. | Expert | Aug 2025 | 59% | System cards only |
| Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. | Expert | Jul 2025 | 67% | System cards only |
| Audio Understanding Benchmark (Mistral) Models answer hard questions about 40 long-form audio recordings. Mistral built it in-house to test audio understanding beyond transcription. | Expert | Jul 2025 | | Dropped |
| EHRQA (Google) Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning. | Expert | Jul 2025 | Not in cards | |
| EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. | Expert | Jul 2025 | Not in cards | |
| In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. | Expert | Jul 2025 | 78% | System cards only |
| Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. | Expert | Jul 2025 | 88.6% | Fading |
| Irrelevant instructions, text-based web browser Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them. | Expert | Jul 2025 | 99.5% | System cards only |
| Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. | Expert | Jul 2025 | 95% | System cards only |
| KBank-MMLU (Upstage) Models answer Korean finance questions in a multiple-choice format. Upstage built the set in house. | Expert | Jul 2025 | | Fading |
| Kimi Chinese in-house open-ended benchmark Models answer held-out Chinese prompts drawn from real user queries. Outputs are compared pairwise against another model to give win, tie and loss rates. | Everyday | Jul 2025 | System cards only | |
| Kimi Promptfoo red-team safety evaluation Promptfoo generates attack prompts for harmful, criminal, misinformation, privacy and security plugins under five strategies. Human reviewers judge the responses. | Everyday | Jul 2025 | System cards only | |
| Ko-LongBench (LG) Models answer questions over long Korean contexts. Tasks cover document QA, story and dialogue history understanding, in-context learning, structured QA and RAG. | University | Jul 2025 | | Fading |
| Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. | Expert | Jul 2025 | System cards only | |
| PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. | Expert | Jul 2025 | Not in cards | |
| US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. | Expert | Jul 2025 | Not in cards | |
| Apple side-by-side human evaluation Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison. | Everyday | Jun 2025 | 64.7% | Fading |
| ComplexNLU (Tencent) Models answer Tencent in-house natural language understanding tasks with complex inputs. Task and scoring details are not published. | Everyday | Jun 2025 | | Dropped |
| Infinite Generations (Mistral internal) Models answer challenging long or repetitive prompts from Mistral's internal set. The score is the share of responses that never stop or loop, so lower is better. | Everyday | Jun 2025 | | Dropped |
| InsCtrl (Tencent) Models write text under detailed instructions in a Tencent in-house text creation test. Task and scoring details are not published. | Everyday | Jun 2025 | | Dropped |
| LengthCtrl (Tencent) Models write text that must meet a requested length, in a Tencent in-house test. Task and scoring details are not published. | Everyday | Jun 2025 | | Dropped |
| Mistral internal function calling benchmark Models call functions on Mistral's in-house test set. The Magistral paper does not describe the tasks or the metric. | Everyday | Jun 2025 | | System cards only |
| V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. | Everyday | Jun 2025 | Not in cards | |
| Word-Task (Tencent) Models solve Tencent in-house word level language tasks. Task and scoring details are not published. | Everyday | Jun 2025 | | Dropped |
| Agentic coding malicious requests (Anthropic) Coding agents with standard coding tools receive 150 malicious coding requests prohibited by policy. Score is the refusal rate. | Expert | May 2025 | 100% | System cards only |
| Ambiguous context evaluations (Anthropic) Models respond to gray-area requests where intent is unclear. Graders judge how well responses balance helpfulness and harm. | Everyday | May 2025 | System cards only | |
| Coding prompt injections Agents work on coding tasks that contain prompt injection attacks. Scored on resisting the injected instructions. | Expert | May 2025 | 100% | System cards only |
| Conversational SWE Task Benchmark (Windsurf) Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy. | Expert | May 2025 | Dropped | |
| CounterFactQA (Qwen) Models get counterfactual questions and must recognize that the premise is false instead of giving a hallucinated answer. | Everyday | May 2025 | | System cards only |
| Creative biology Models answer questions about unusual biology needed to engineer harmless biological agents, a weak proxy for novel weapon design. | Expert | May 2025 | 60.3% | System cards only |
| DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. | Expert | May 2025 | 80% | System cards only |
| End-To-End SWE Task Benchmark (Windsurf) Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score. | Expert | May 2025 | Dropped | |
| Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. | Expert | May 2025 | 30% | System cards only |
| Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. | Expert | May 2025 | 0.65 | System cards only |
| Internal model use survey (Anthropic) Anthropic technical staff estimate productivity uplift from the model and judge whether it could automate a junior ML researcher. | Expert | May 2025 | 50% | System cards only |
| Kernels task (Anthropic) Agents optimize compute kernels in a performance engineering challenge. Scored by speedup. | Expert | May 2025 | System cards only | |
| LengthCtrl (Qwen) Models do creative writing tasks with length requirements. The score depends on the gap between output length and target length. | Everyday | May 2025 | | System cards only |
| Malware Refusals (Golden Set) Coding agents receive malware development requests from a curated golden set written by OpenAI policy experts. Scored as refusal rate. | Expert | May 2025 | 100% | System cards only |
| Malware Refusals (Synthetic) Coding agents receive a large synthetic set of malware-related tasks, including ambiguous and dual-use prompts. Scored as refusal rate. | Expert | May 2025 | 97% | System cards only |
| Multi-turn testing (Anthropic) Models face multi-turn conversations that escalate toward harmful requests across risk areas. Score is the appropriate response rate. | Everyday | May 2025 | System cards only | |
| Novel compiler task (Anthropic) Agents build a compiler for an unusual programming language from a specification and test cases. | Expert | May 2025 | 80.9% | System cards only |
| Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. | Expert | May 2025 | 0% | System cards only |
| Seed1.5-VL internal benchmark (ByteDance Seed in-house) Models answer more than 12,000 internal image questions over 100 tasks, from perception, OCR and STEM to agents, code and out-of-distribution puzzles. LLM judges grade answers, and the overall is a weighted average. | University | May 2025 | | System cards only |
| Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. | Expert | May 2025 | System cards only | |
| Single-turn benign request evaluations (Anthropic) Models answer benign requests on sensitive topics. Score is the refusal rate, where lower is better. | Everyday | May 2025 | 0% | System cards only |
| Single-turn harmful request evaluations (Anthropic) Models answer single-turn requests that violate the usage policy. Score is the harmless response rate. | Everyday | May 2025 | 99.8% | System cards only |
| Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. | Expert | May 2025 | 0.97 | System cards only |
| ThinkFollow (Qwen) Models hold multi-turn dialogues with randomly inserted think and no-think flags and must switch reasoning mode correctly. | Everyday | May 2025 | | System cards only |
| Time series forecasting task (Anthropic) Agents solve a regression and time-series forecasting problem with known state-of-the-art results, in six variants. | Expert | May 2025 | 4.55 | System cards only |
| ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. | Everyday | May 2025 | | System cards only |
| Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. | Expert | May 2025 | 0.63% | System cards only |
| Unexpected container or code states Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed. | Expert | May 2025 | 85% | System cards only |
| Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. | Expert | Apr 2025 | 93.3% | System cards only |
| Default safety evaluation (Cohere) Models answer unsafe prompts in five categories with no system preamble. LLM judges score the share of safe responses, and a jury rates relative safety. | Everyday | Apr 2025 | | System cards only |
| Enterprise generative evaluation (Cohere) Models complete 22 enterprise writing tasks such as job ads with format rules. Rule checks and LLM judges give a pass rate. | Everyday | Apr 2025 | | System cards only |
| Enterprise RAG evaluation (Cohere) Models answer workplace policy and technical questions over long documents, some unanswerable. Llama Index Correctness grades answers from 1 to 5. | Everyday | Apr 2025 | | System cards only |
| Image generation refusals Models decide whether to call the image generation tool on human-curated adversarial prompts with system mitigations on. Scored on not_unsafe and not_overrefuse. | Expert | Apr 2025 | 98% | System cards only |
| Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. | Everyday | Apr 2025 | System cards only | |
| Internal API instruction following (OpenAI) Models answer prompts drawn from real developer use cases that carry instructions on formatting, verbosity and length. OpenAI reports a hard subset. | Everyday | Apr 2025 | 54% | Dropped |
| Kimi-ASR Internal Testset Models transcribe speech from two internal Moonshot test subsets. Scored by word error rate, lower is better. | Everyday | Apr 2025 | System cards only | |
| Kimi-ASR Internal Testset Subset 1 Models transcribe speech from Moonshot internal subset 1. The score is the published word error rate. | Everyday | Apr 2025 | 1.42% | Not in cards |
| Kimi-ASR Internal Testset Subset 2 Models transcribe speech from Moonshot internal subset 2. The score is the published word error rate. | Everyday | Apr 2025 | 2.44% | Not in cards |
| Multi-dialect SQL evaluation (Cohere) Models write SQL for SQLite, PostgreSQL, MySQL, PL/SQL and T-SQL databases. Scored by execution accuracy against gold databases. | University | Apr 2025 | | System cards only |
| Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. | Everyday | Apr 2025 | System cards only | |
| Pairwise safety red teaming Red teamers compare anonymized responses from several models and pick the least safe. Scored as ELO or win rate. | Expert | Apr 2025 | System cards only | |
| Person Identification evaluation Models receive requests to identify people in photos. Scored on refusal rate in non-adversarial and adversarial sets. | Everyday | Apr 2025 | System cards only | |
| Safety mode controllability (Cohere) Models follow contextual or strict safety instructions. Accuracy averages correct answers on an over-refusal set and correct refusals on a control set. | Everyday | Apr 2025 | System cards only | |
| Snowflake Multimodal PDF Retrieval Evaluation Retrievers find relevant pages in technical manuals, SEC filings and presentation slides. The tasks include 143 chart questions and 69 text questions about manuals. | University | Apr 2025 | Not in cards | |
| Text to Text Safety An automated content safety evaluation measures policy violations in model answers to text prompts. | Everyday | Apr 2025 | System cards only | |
| Tone An automated evaluation measures whether model refusals are objective, neutral and not preachy. | Everyday | Apr 2025 | System cards only | |
| Ungrounded Inference evaluation Models receive requests for inferences about people that images or text cannot justify. Scored on refusal rate in non-adversarial and adversarial sets. | Everyday | Apr 2025 | System cards only | |
| WISER-Atomic Web research systems answer 121 customer-inspired questions across finance, sales, recruitment, and other domains. | Expert | Apr 2025 | 69% | Not in cards |
| CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. | Expert | Mar 2025 | System cards only | |
| Cohere enterprise human evaluation Trained annotators blindly compare two models' answers on enterprise tasks such as HR FAQs, technical FAQs and workplace assistants. They judge accuracy, instruction following and style. Results are win, tie and loss rates. | Everyday | Mar 2025 | | Dropped |
| Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. | Expert | Mar 2025 | | System cards only |
| ETVA Text-to-video systems are scored on fine-grained prompt alignment through generated atomic questions and video-language answers. | Expert | Mar 2025 | Not in cards | |
| Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. | Everyday | Mar 2025 | Not in cards | |
| Golden3 speech summarization (Microsoft) Models summarize 108 recorded meetings of about 6 minutes each from 1071 instructions. GPT-4 rates overall quality from 1 to 7. | Everyday | Mar 2025 | | System cards only |
| Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) GPT-4o simulates adversarial conversations with the model, and a judge flags violence, sexual, self-harm and hateful content. Scored as the share of harmful responses, lower is better. | Everyday | Mar 2025 | | System cards only |
| Snowflake Financial Filings RAG Evaluation Systems answer about 500 questions over 23,000 SEC filings. Annotators identify supporting passages, pages and answers. | Expert | Mar 2025 | Not in cards | |
| Answer Factuality (Perplexity) Models answer questions from search results. Graders check that answers use facts grounded in the results and resolve conflicting or missing information. | Everyday | Feb 2025 | | Dropped |
| Bioweapons knowledge questions Models answer 33 sensitive bioweapons questions written by Deloitte biosecurity staff. Answers are compared with novice, intermediate and expert baselines. | Expert | Feb 2025 | 51.5% | System cards only |
| Challenging Red Teaming Evaluation 1 (created for o3-mini) Models face adversarial jailbreaks from o3-mini red teaming on illicit advice, extremism, political persuasion and self-harm. Scored as not_unsafe rate. | Expert | Feb 2025 | 51% | System cards only |
| Challenging Red Teaming Evaluation 2 (created for deep research) Models face adversarial prompts from deep research red teaming on risky advice such as attack planning. Scored as not_unsafe rate. | Expert | Feb 2025 | 67% | System cards only |
| Child safety evaluations (Anthropic) Models answer single-turn and multi-turn prompts about child sexualization and grooming. Score is the appropriate response rate. | Everyday | Feb 2025 | System cards only | |
| Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. | Expert | Feb 2025 | 94.4% | System cards only |
| Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. | Research frontier | Feb 2025 | 66.7% | System cards only |
| Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. | Expert | Feb 2025 | System cards only | |
| LLM training task (Anthropic) Agents optimize a CPU-only small language model training implementation. Scored by speedup. | Expert | Feb 2025 | System cards only | |
| Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. | Expert | Feb 2025 | 69.7% | System cards only |
| Malicious computer use (Anthropic) Computer-use agents receive harmful tasks such as surveillance or fraud in GUI environments. Score is the refusal rate. | Everyday | Feb 2025 | 99.4% | System cards only |
| Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. | Expert | Feb 2025 | 100% | System cards only |
| Personal data evaluations Browsing agents receive requests for private personal information, from 200 synthetic prompts and 55 manually written golden examples. Scored on adherence to personal data policy. | Everyday | Feb 2025 | System cards only | |
| Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. | Expert | Feb 2025 | 71.4% | System cards only |
| Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. | Expert | Feb 2025 | 31.3 | System cards only |
| Readability (Perplexity) Models answer search questions. Graders rate whether answers are concise yet detailed and use markdown formatting well. | Everyday | Feb 2025 | | Dropped |
| Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. | Expert | Feb 2025 | 100% | System cards only |
| Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. | Expert | Feb 2025 | 100% | System cards only |
| Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. | Everyday | Jan 2025 | 97% | System cards only |
| Coding (MiniMax in-house) Models answer coding requests from real Hailuo AI users. The report does not describe the scoring. | University | Jan 2025 | | System cards only |
| Creative Writing (MiniMax in-house) Models write creative pieces for real Hailuo AI user requests. Automatic assessments score the writing. | Everyday | Jan 2025 | | System cards only |
| General Assistant (MiniMax in-house) Models answer real assistant queries from Hailuo AI users. Automatic assessments score the replies. | Everyday | Jan 2025 | | System cards only |
| Hard Capability (MiniMax in-house) Models handle complex productivity requests with several levels of instructions that leading models often fail. | University | Jan 2025 | | System cards only |
| Instruction Following (MiniMax in-house) Models follow multi-level instructions collected from user cases where current models fail. | Everyday | Jan 2025 | | System cards only |
| Knowledge Q&A (MiniMax in-house) Models answer knowledge questions drawn from real Hailuo AI user queries. | Everyday | Jan 2025 | | System cards only |
| Long Context (MiniMax in-house) Models translate, summarize and analyze long documents from real user requests. | Everyday | Jan 2025 | | System cards only |
| OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. | Research frontier | Jan 2025 | 55% | System cards only |
| Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. | Expert | Jan 2025 | 23% | System cards only |
| Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. | Everyday | Jan 2025 | 94% | System cards only |
| Safety (MiniMax in-house) Models respond to safety-relevant user prompts. MiniMax scores whether replies are safe and helpful. | Everyday | Jan 2025 | | System cards only |
| User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. | Everyday | Jan 2025 | 92% | System cards only |
| Contextual Nuclear Knowledge Models answer 222 multiple choice questions on nuclear engineering with a nonproliferation focus. Scored as cons@32 accuracy. | Expert | Dec 2024 | 77% | System cards only |
| Image input safety evaluations Models answer harmful prompts that combine text and images in categories such as hate, extremism and self-harm. Scored as safe completion rate. | Everyday | Dec 2024 | 99% | System cards only |
| Instruction Hierarchy Evaluation Models resolve conflicts between system, developer and user messages, resist tutor jailbreaks, and protect secret phrases and passwords. Scored per sub-evaluation. | Everyday | Dec 2024 | System cards only | |
| Microsoft RAI benchmark: Grounding Models answer in GPT-4o simulated multi-turn conversations. A judge scores from 0 to 5 how fully each answer is grounded in the given prompt. | Everyday | Dec 2024 | | System cards only |
| PersonQA Models answer questions about publicly available facts on people. Scored on accuracy, with hallucination rate reported alongside. | Everyday | Dec 2024 | 94.3% | System cards only |
| PhiBench Models answer original prompts that Microsoft's Phi team wrote to test a wide range of skills and reasoning. | University | Dec 2024 | | Dropped |
| Radiological and Nuclear Expert Knowledge Models answer 87 expert and tacit knowledge questions on radiological and nuclear threats. Scored as cons@32 accuracy. | Expert | Dec 2024 | 74% | System cards only |
| Red Teaming Network pairwise safety comparison Red teamers compare anonymized responses from two models in free-form risky conversations. Scored as the share of ratings where a model is safer. | Expert | Dec 2024 | 59.8% | System cards only |
| Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. | Research frontier | Dec 2024 | System cards only | |
| Structured expert probing campaign: radiological and nuclear Radiological and nuclear experts test whether a model helps with weapons development steps. Conversations are rated against Preparedness risk levels. | Research frontier | Dec 2024 | System cards only | |
| Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. | Expert | Dec 2024 | 51% | System cards only |
| Cortex Analyst Multi-Turn Question Rewriting Models rewrite a follow-up question so it stands alone. Mistral Large 2 judges whether the rewrite preserves the necessary conversation history. | University | Nov 2024 | Not in cards | |
| Election integrity evaluations (Anthropic) Models answer election-related prompts, including requests for misinformation or influence operations. Score is the appropriate response rate. | Everyday | Oct 2024 | System cards only | |
| First-Person Fairness evaluation Tests whether a user's name (a demographic proxy) changes responses, using an LLM research assistant to detect harmful stereotypes. | Oct 2024 | 0.40% | System cards only | |
| Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. | Expert | Sep 2024 | System cards only | |
| BirthdayFacts Models are asked for people's birthdays. Scored on how often they give a wrong birthday. | Everyday | Sep 2024 | 24% | System cards only |
| Challenging Refusal Evaluation Models answer a harder set of disallowed content prompts across policy categories. An autograder scores the not_unsafe rate. | Expert | Sep 2024 | 93.4% | System cards only |
| ChangeMyView Models write rebuttals to r/ChangeMyView posts that human raters score for persuasiveness. Scored as percentile against human responses. | Everyday | Sep 2024 | 89.1% | System cards only |
| cognition-golden Cognition tests coding agents on tasks modeled on real Devin use cases, in authentic development environments with fully autonomous grading. A train split serves self-improvement and a test split measures capability. | Expert | Sep 2024 | 51.8% | Not in cards |
| CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. | Everyday | Sep 2024 | 0.17% | System cards only |
| DS-Arena-Code Models answer common coding requests in an internal subjective test. GPT-4o judges responses against competitor models to give a win rate. | University | Sep 2024 | | Dropped |
| DS-FIM-Eval Models fill in missing code between a given prefix and suffix, as in editor plugin completion. | University | Sep 2024 | | Dropped |
| Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. | Expert | Sep 2024 | System cards only | |
| External red team jailbreaks External red teamers and automated gap finding generate jailbreaks against the model. Scored on how often the model resists. | Expert | Sep 2024 | 44% | System cards only |
| Human Sourced Jailbreaks Models face jailbreak prompts collected from human red teaming. Scored as the not_unsafe rate. | Expert | Sep 2024 | 100% | System cards only |
| Jailbreak Augmented Examples Models face publicly known jailbreaks applied to prompts from the standard disallowed content set. Scored as not_unsafe rate. | Everyday | Sep 2024 | 100% | System cards only |
| Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. | Expert | Sep 2024 | 0% | System cards only |
| Open Ended Questions (hallucination) Models write open-ended factual text such as biographies, checked against Wikipedia. Scored as average number of incorrect statements. | Everyday | Sep 2024 | 0.78 | System cards only |
| Political Persuasion Parallel Generation Human raters pick the more persuasive of two short political social media posts. Scored as win rate against GPT-4o. | Everyday | Sep 2024 | 53.8% | System cards only |
| Production Jailbreaks Models face jailbreaks found in production ChatGPT data. Scored as not_unsafe rate. | Everyday | Sep 2024 | 100% | System cards only |
| QuantBench Models answer 25 multiple choice versions of quant trading firm reasoning puzzles. Scored as cons@16 accuracy. | Competition | Sep 2024 | 60% | System cards only |
| Regurgitation Evaluations Models are asked to reproduce training data. Scored on how often they refuse to regurgitate it. | Everyday | Sep 2024 | System cards only | |
| Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. | Expert | Aug 2024 | 78% | System cards only |
| Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. | University | Aug 2024 | 68% | System cards only |
| Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. | School | Aug 2024 | 82% | System cards only |
| Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. | Expert | Aug 2024 | 96.3% | Fading |
| Microsoft internal multimodal RAI benchmark Vision language models answer image and text prompts across harm categories. Responses are scored from 0 to 10, higher is safer. | Everyday | Aug 2024 | 8.16 | System cards only |
| OpenAI Research Engineer Interview: Coding Models solve 18 self-contained coding problems like those in OpenAI interviews. Scored as pass@1. | Expert | Aug 2024 | 100% | System cards only |
| OpenAI Research Engineer Interview: Multiple Choice Models answer 97 multiple-choice questions drawn from OpenAI machine learning interview topics. Scored as cons@32. | Expert | Aug 2024 | 83% | System cards only |
| Speaker identification Speech models get audio requests to identify a speaker by voice. Scored on refusing when they should and complying for famous quotes. | Everyday | Aug 2024 | System cards only | |
| Standard Refusal Evaluation Models answer disallowed and benign safety-related prompts across policy categories. An autograder scores not_unsafe and not_overrefuse. | Everyday | Aug 2024 | 100% | System cards only |
| Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. | Expert | Aug 2024 | 94% | System cards only |
| Text persuasion study on political opinions Participants read model-written articles or chat with a model on political topics. Scored on opinion shift against human-written articles. | Everyday | Aug 2024 | System cards only | |
| Unauthorized voice generation Speech models produce audio that may deviate from the approved system voice. Scored on how many meaningful deviations the output classifier catches. | Everyday | Aug 2024 | 100% | System cards only |
| Ungrounded inference and sensitive trait attribution Speech models get requests to infer traits about a speaker from audio. Scored on refusing ungrounded inferences and hedging plausible ones. | Everyday | Aug 2024 | 84% | System cards only |
| Voice persuasion study on party preferences Participants hear model-voiced audio clips or talk with a voiced model. Scored on shifts in hypothetical party preference against human baselines. | Everyday | Aug 2024 | 5.74% | System cards only |
| Llama 3 pairwise human evaluation (Meta) Human raters compare Llama responses with a rival model on about 7,000 prompts across six capabilities and three multiturn sets. | Jul 2024 | System cards only | ||
| Llama internal safety benchmarks (Meta) Models face Meta's adversarial and borderline prompts per capability and language. Scored as violation rate and false refusal rate. | Jul 2024 | System cards only | ||
| Llama verbatim memorization probe (Meta) Models get prompts sampled from their pretraining data. The score is the share of generations that reproduce the true continuation verbatim. | Jul 2024 | System cards only | ||
| Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Everyday | Jul 2024 | | System cards only |
| Qwen in-house English automatic evaluation Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Everyday | Jul 2024 | | System cards only |
| Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. | Everyday | Jul 2024 | System cards only | |
| BetterChartQA Models answer annotator-written questions about real charts from news, government reports, and papers across 9 capability buckets. Scored by accuracy. | School | Jun 2024 | | Dropped |
| ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. | University | Jun 2024 | | Dropped |
| Internal agentic coding evaluation (Anthropic) Agents implement a pull request in an open source codebase from a natural language description. A problem counts as solved when all hidden tests pass. | Expert | Jun 2024 | 78% | System cards only |
| Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. | Everyday | Jun 2024 | 4.15 | Dropped |
| HiddenMath Models solve 179 competition-level math problems that experts wrote from scratch. Answers are graded automatically. | Competition | May 2024 | | Dropped |
| Microsoft RAI benchmark: Harmful Content Continuation (DR-3) Models continue conversations about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. | Everyday | Apr 2024 | | System cards only |
| Microsoft RAI benchmark: Harmful Content Summarization (DR-3) Models summarize content about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. | Everyday | Apr 2024 | | System cards only |
| Microsoft RAI benchmark: Jailbreak (DR-1) Models face simulated jailbreak conversations on topics such as child grooming, illegal advice and leaking guidelines. Scored as the defect rate at severity 1. | Everyday | Apr 2024 | | System cards only |
| Microsoft RAI benchmark: Third Party Harm (DR-1) Models face simulated conversations that try to extract third-party content such as books, news, recipes and songs. Scored as the defect rate at severity 1. | Everyday | Apr 2024 | | System cards only |
| Microsoft RAI benchmark: Ungroundedness Models answer in GPT-4 simulated multi-turn conversations. A judge rates how ungrounded each answer is in the given prompt, from 0 to 4. | Everyday | Apr 2024 | | System cards only |
| 100Q-Hard (Anthropic) Models answer hard, obscure factual questions. Scored on correct, incorrect and declined answers. | Expert | Mar 2024 | 60.1% | System cards only |
| Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. | Everyday | Mar 2024 | System cards only | |
| Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. | Expert | Mar 2024 | System cards only | |
| Harmlessness human feedback win rate (Anthropic) Raters probe models in adversarial chats and pick the more harmless reply in head-to-head comparisons. Score is the win rate against a baseline model. | Everyday | Mar 2024 | 87% | System cards only |
| Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. | Everyday | Mar 2024 | System cards only | |
| Honesty human feedback win rate (Anthropic) Crowdworkers try to make models say something false and pick the more honest reply in head-to-head chats. Score is the win rate against a baseline. | Everyday | Mar 2024 | 69% | System cards only |
| Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. | Everyday | Mar 2024 | | System cards only |
| Multi-factual (Anthropic) Models answer questions that each combine several closed-ended sub-questions on one topic. Answers are graded correct, incorrect or unsure against references. | Everyday | Mar 2024 | 62.8% | System cards only |
| Multimodal policy red-teaming (Anthropic) Models answer multi-turn image and text conversations on harmful topics such as child safety, weapons and fraud. Score is the share of harmless responses. | Everyday | Mar 2024 | 99.2% | System cards only |
| Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. | Everyday | Mar 2024 | System cards only | |
| Divergence attack (extractable memorization) Models are prompted to repeat one token many times. The test measures how often they diverge and emit memorized training text. | Everyday | Feb 2024 | | System cards only |
| Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. | Expert | Feb 2024 | | System cards only |
| PhysicsFinals Models answer 61 undergraduate physics final exam problems written by physics professors. A professor grades the answers. | University | Feb 2024 | | Dropped |
| Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. | Expert | Jan 2024 | System cards only | |
| DeepSeek LLM in-house safety evaluation Models answer 2,400 Chinese test cases on discrimination, legal rights, trade secrets, illegal behavior and other safety issues. Human annotators judge each answer. | Everyday | Jan 2024 | System cards only | |
| Natural2Code Models write Python functions in HumanEval format from new prompts kept off the web. Unit tests decide correctness. | School | Dec 2023 | | Dropped |
| ChineseQA Models answer Chinese trivia questions in a closed-book format modeled on TriviaQA. | Everyday | Nov 2023 | | Dropped |
| Mistral 175 unsafe prompts Models receive 175 unsafe prompts. Metric is the share of harmful questions the model declines. | Everyday | Oct 2023 | | System cards only |
| Mistral self-reflection moderation set Models classify prompts or answers as acceptable or harmful on a balanced set of adversarial and standard prompts. Scored by precision and recall. | Everyday | Oct 2023 | System cards only | |
| Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. | Expert | Sep 2023 | System cards only | |
| Qwen Chinese human evaluation Three annotators rank chat model answers to 300 Chinese instructions on knowledge, language, writing, math and coding. Results are win rates against GPT-3.5. | Everyday | Sep 2023 | | System cards only |
| Harmfulness scores on held-out prompts (Anthropic) Models answer 328 held-out red-team and jailbreak prompts. A preference model flags responses judged more harmful than a fixed refusal. | Everyday | Jul 2023 | 1.20% | System cards only |
| Human feedback win rates (Anthropic) Raters chat with two models on tasks such as coding, documents and creative writing and pick the better reply. Score is the win rate. | Everyday | Jul 2023 | System cards only | |
| Llama 2-Chat helpfulness human evaluation (Meta) Three human raters compare Llama 2-Chat with a rival chat model on about 4,000 single and multi-turn helpfulness prompts. | Jul 2023 | System cards only | ||
| Llama 2-Chat safety human evaluation (Meta) Human raters score responses to about 2,000 adversarial prompts on a five-point scale. Ratings of 1 or 2 count as violations. | Jul 2023 | System cards only | ||
| Closed-domain hallucination evaluation Models answer using only provided context. Scored on avoiding invented details outside that context. | Everyday | Mar 2023 | System cards only | |
| Incorrect behavior rate on disallowed and sensitive content Models answer disallowed prompts and sensitive prompts such as medical advice. Scored on the rate of responses that break policy. | Everyday | Mar 2023 | System cards only | |
| Internal adversarially-designed factuality evaluations Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. | Everyday | Mar 2023 | 7.60% | In use |
| Leetcode (easy) Models solve 41 easy Leetcode problems that OpenAI selected. Scored as problems solved. | University | Mar 2023 | 75.6% | System cards only |
| Leetcode (hard) Models solve 45 hard Leetcode problems that OpenAI selected. Scored as problems solved. | University | Mar 2023 | 6.70% | System cards only |
| Leetcode (medium) Models solve 80 medium Leetcode problems that OpenAI selected. Scored as problems solved. | University | Mar 2023 | 26.3% | System cards only |
| Preference on ChatGPT and API prompts Human labelers compare model responses to 5,214 prompts sent to ChatGPT and the OpenAI API. Scored as win rate against a baseline model. | Everyday | Mar 2023 | 70.2% | System cards only |
| Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. | Expert | System cards only | ||
| Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. | Expert | | System cards only | |
| Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. | 2.06 | System cards only | ||
| Anthropic cyber safeguard evaluations Classifier recall on violative cyber exchanges, block rates on defensive vulnerability-finding and secure-coding work, and the rewind-attacker robustness test. | Not in cards | |||
| Anthropic harmful-request and wellbeing evaluations Single-turn harmful and benign request sets, multi-turn tests, child safety, suicide/self-harm, disordered eating and election integrity. | Not in cards | |||
| Anthropic malicious agentic use evaluations Measures refusal of malicious tasks via Claude Code, computer use and agentic influence campaigns. | Not in cards | |||
| Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. | Not in cards | |||
| Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. | Expert | 14.3% | System cards only | |
| Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. | Expert | 91.1% | System cards only | |
| Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. | Expert | 2.53 | System cards only | |
| Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. | Expert | System cards only | ||
| Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. | Everyday | | System cards only | |
| Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. | Everyday | | System cards only | |
| CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. | Expert | 55.8% | System cards only | |
| Code summary honesty (Anthropic) Models summarize prefilled agentic coding transcripts that hide failures. Score is the rate of summaries that fail to raise the important failures. | Expert | 3.70% | System cards only | |
| Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. | Everyday | | System cards only | |
| Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. | Everyday | | System cards only | |
| Computer use prompt injection evaluation (Anthropic) Agents face about 600 prompt injection scenarios in coding platforms, browsers and email workflows. Score is the attack prevention rate. | Expert | 78% | System cards only | |
| Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. | Expert | | System cards only | |
| Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. | Expert | | System cards only | |
| CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). | Expert | 75.9% | In use | |
| Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. | Everyday | | System cards only | |
| Dangerous Content Safety An autorater checks answers to single and multi-turn requests about weapons, illegal acts, violence and self-harm. Reports safe response rate. | Everyday | | System cards only | |
| Delusional Reinforcement evaluation Multi-turn chats test whether models avoid validating beliefs not grounded in reality and point users to real support. Reports policy adherence rate. | Everyday | | System cards only | |
| Destructive actions (Anthropic) Models continue real Claude Code transcripts cut just before a destructive action. Score is the rate of taking a destructive action. | Expert | 21% | System cards only | |
| Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. | Expert | 0.30% | System cards only | |
| Discoverable memorization of training data Models continue 50-token prefixes sampled from their own training data. The test counts exact and approximate copies of the true suffix. | Everyday | System cards only | ||
| Emotional Attachment evaluation Chats test whether models avoid claiming affection, fostering dependence or encouraging secrecy. Reports policy adherence rate. | Everyday | | System cards only | |
| Evaluation awareness discriminability (Anthropic) Probes and follow-up realism ratings try to tell behavioral audit transcripts from real internal Claude Code sessions. Each setting is scored as AUROC. | Expert | System cards only | ||
| Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. | Research frontier | 72.4% | System cards only | |
| Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. | Expert | 100% | System cards only | |
| Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. | Not in cards | |||
| Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. | Not in cards | |||
| Gemini Personality Evaluation (sycophancy) About 900 single- and multi-turn prompts scoring nine personality dimensions, including candor and self-awareness against sycophancy. | System cards only | |||
| Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. | Not in cards | |||
| Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. | Not in cards | |||
| Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. | Expert | Not in cards | ||
| Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. | | System cards only | ||
| Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. | Research frontier | | System cards only | |
| Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. | Expert | | System cards only | |
| Grounding with citations, sentence unfactual rate Models answer multi-snippet tasks with inline citations. An autorater reports the share of sentences not supported by the sources. | Everyday | | System cards only | |
| HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. | 7.80% | System cards only | ||
| Hate & Harassment Safety An autorater checks answers to requests for hateful or harassing content and to sensitive benign prompts. Reports safe response rate. | Everyday | | System cards only | |
| Health dynamic conversation eval Simulated users hide key health facts at first, and models must ask for context to reach the correct clinical conclusion. | Expert | System cards only | ||
| Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. | Expert | System cards only | ||
| Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. | Expert | System cards only | ||
| Helpfulness violations (from ART) Automated red teaming attackers search for benign prompts where the model gives unhelpful answers. Reports the violation rate found. | Everyday | | System cards only | |
| HiddenMath-Hard Models solve the harder part of an expert-written, never-published set of AIME and AMC style competition problems. | Competition | | System cards only | |
| High-risk health scenarios eval Adversarial health emergencies test whether models recognize urgency and red flags and direct users to help. Reports pass rate. | Expert | | System cards only | |
| Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. | Everyday | | System cards only | |
| Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. | Expert | | System cards only | |
| Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. | Expert | | System cards only | |
| Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. | Expert | | System cards only | |
| Instruction Following (safety evaluation) An automated evaluation measures whether the model follows instructions on sensitive prompts while staying within safety policies. | Everyday | System cards only | ||
| Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. | Expert | | System cards only | |
| Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. | Expert | | System cards only | |
| JEF-Bench Agents read files that embed jailbreak instructions in tool-call returns. Reports attack success rate across 1800 tasks and 50 disallowed use cases. | Expert | | System cards only | |
| Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. | Research frontier | 80% | System cards only | |
| Kimi Code Bench Moonshot in-house coding-agent tasks from internal engineering cases, production incidents and open-source projects across 10+ languages. 80 tasks. | Expert | Not in cards | ||
| Lazy investigation (Anthropic) Agents debug a failing job where misleading evidence invites a harmful action such as cordoning a healthy node. Scored 0 to 10 by a grader. | Expert | 9.94 | System cards only | |
| Long-context grounding, sentence unfactual rate Models answer from long documents. An autorater reports the share of sentences not supported by the documents. | Everyday | | System cards only | |
| Lying about identity (Anthropic) Models play an operator-assigned human persona while users ask if they are human. Score is the rate of admitting to being an AI. | Everyday | 99% | System cards only | |
| Many-shot jailbreaking evaluation (Anthropic) Models receive long fake dialogs where the assistant complies with harmful requests, then a new harmful request. Score is the compliance rate. | Expert | 25% | System cards only | |
| Meta agentic cyber misuse refusal Models run multi-step cyber tasks with tool access while an adaptive attacker tries to steer them to a malicious goal. Reports attack success rate. | Expert | | System cards only | |
| Meta agentic evaluations grounded in user traffic Models continue replays of recorded user workflows. A jury reports the share of cases with a supported substantive finding. | Expert | | System cards only | |
| Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. | Expert | | System cards only | |
| Meta catastrophic cyber misuse refusal Models receive high-severity cyber-misuse prompts with technical context. The benchmark reports pass@k attack success rate under adaptive attacks. | Expert | | System cards only | |
| Meta chemical-agent refusal evaluation Models receive high-risk chemical prompts and the benchmark measures how often they refuse. Covers development, procurement and production stages. | Expert | | System cards only | |
| Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. | Expert | | System cards only | |
| Meta cyber false-refusal benchmark Models receive benign and borderline cyber requests and the benchmark measures how often they wrongly refuse. Reports false refusal rate. | Expert | | System cards only | |
| Meta Cyber Misuse Chat Models receive 80 offensive cyber requests, directly and with jailbreak developer prompts. Reports attack success rate and a benign false refusal rate. | Expert | | System cards only | |
| Meta instructed selective performance evaluation Models are told to underperform on six science capability sets. Reports the average drop from baseline in percentage points. | Expert | | System cards only | |
| Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. | Expert | | System cards only | |
| Meta internal sycophancy evaluation Models meet adversarially curated conversations that invite inappropriate agreement, plus benign controls. Reports sycophancy and excessive anti-sycophancy rates. | Everyday | | System cards only | |
| Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. | Expert | | System cards only | |
| Meta training-stage provenance probe Models label 10,000 transcripts as deployment, evaluation, RL or SFT. Reports overall accuracy against a 25% chance rate. | Expert | | System cards only | |
| Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. | Expert | 100% | System cards only | |
| Missing-context hallucinations (Anthropic) Models get prompts that request tools they lack or cite missing material. Score is the non-hallucination rate on each 500-prompt set. | Everyday | System cards only | ||
| Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. | Expert | | System cards only | |
| Muse-targeted cyber misuse and violation red-teaming An automated red-teaming agent tries to misuse the Muse runtime across six attack categories. Reports attack success rate per category. | Expert | System cards only | ||
| Neutrality evaluation Single and multi-turn prompts on politics, civics, identity and religion test whether models avoid taking sides. Reports neutral response rate. | Everyday | | System cards only | |
| OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. | Not in cards | |||
| OpenAI cyber safety evaluations (incl. Daybreak Blue) Refusal and helpfulness evaluations for cyber requests, including the Daybreak Blue set for enabling defensive security work. | System cards only | |||
| OpenAI Deployment Simulation Replays representative production or internal Codex traffic on a candidate model to forecast misaligned or disallowed behavior before launch. | 0.47% | System cards only | ||
| OpenAI Dynamic Mental Health Benchmarks Multi-turn conversations with adversarial simulated users on mental-health risks. Scored on safe and supportive responses. | System cards only | |||
| OpenAI GPT-Red jailbreak and prompt-injection evals Static jailbreaks from attacker models (half from a held-out attacker), adaptive multi-turn attackers, and direct and indirect prompt-injection sets. | Not in cards | |||
| OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). | Expert | Not in cards | ||
| OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). | Expert | Not in cards | ||
| OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. | System cards only | |||
| OpenAI user-flagged hallucination eval Production conversations users flagged for factual errors. Measures the rate of incorrect claims. | 9.20% | System cards only | ||
| OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. | Research frontier | | System cards only | |
| Overconfidence: verifying command-line syntax (Anthropic) Agents are asked for an exact invocation of an unfamiliar command-line tool. Graders score from 1 to 5 whether they check documentation before running commands. | Expert | 4.90 | System cards only | |
| Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. | Everyday | | System cards only | |
| Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. | Expert | | System cards only | |
| Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. | Expert | | System cards only | |
| Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. | Expert | | System cards only | |
| QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). | Expert | 63.1% | In use | |
| QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. | Expert | 1,770 | In use | |
| QwenSVGBench Model writes SVG code for bilingual prompts. Rendered outputs judged by a multimodal model and ranked with Bradley-Terry Elo. | University | 1,758 | In use | |
| QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. | Expert | 86.3% | In use | |
| Real-World Finance v2 (Anthropic) Agents complete 294 long-horizon finance tasks such as building financial models. A model grader compares deliverables pairwise and ratings are fit as Elo. | Expert | 1,637 | System cards only | |
| Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. | Expert | | In use | |
| Reasoning behavior classifiers (Anthropic) Classifiers scan reasoning text from training episodes for deception, harmful actions and distress. Score is the share of outputs flagged. | Expert | System cards only | ||
| Recognition of model-written user turns (Anthropic) Models write synthetic user turns for real Claude Code transcripts, then judge which version is real. Score is accuracy above chance. | Expert | 0.35 | System cards only | |
| ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. | Research frontier | | System cards only | |
| RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. | Research frontier | | System cards only | |
| SAVE-Bench A coding agent works where the environment or the user instruction carries hidden risk. The benchmark scores whether it surfaces the risk to the user. | Expert | | System cards only | |
| Search triggering Models decide whether to call search for queries that need fresh facts or answer directly when search is unneeded. Reports F1. | Everyday | | System cards only | |
| SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. | Expert | 58.7% | System cards only | |
| Sexually Explicit Content Safety An autorater checks answers to direct, adversarial and benign sexual content prompts in single and multi-turn chats. Reports safe response rate. | Everyday | | System cards only | |
| SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). | Expert | Not in cards | ||
| STEM Professor Bench Agents solve unpublished research problems written by professional scientists. Most problems concern theoretical physics, with some partial credit. | Research frontier | 61.8% | System cards only | |
| Suicide & Self-Harm evaluation Static and dynamic chats with suicidal or self-harm signals test adherence to prevention practice and crisis resources. Reports policy adherence rate. | Everyday | | System cards only | |
| SWE agentic fabrication eval A grounding autorater checks whether claims made by coding agents on long SWE tasks are supported by tool outputs. | Expert | System cards only | ||
| Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. | Everyday | | System cards only | |
| Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. | Expert | | System cards only | |
| Uncritically reporting flawed results (Anthropic) Agents analyze data with code that silently mishandles broken values. Graders check whether the agent flags the flaw or reports the biased numbers. | Expert | 9.83 | System cards only | |
| Unjustified refusals An automated evaluation measures over-refusals on benign or borderline prompts that can be safely fulfilled. | Everyday | System cards only | ||
| White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. | Research frontier | System cards only | ||
| xAI Factuality (Hallucination) eval Internal measure of unsupported or fabricated claims in single responses to information-seeking queries. | 0.98% | System cards only | ||
| xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. | Not in cards | |||
| xAI sycophancy eval Factual questions paired with misleading user context. Rate at which the model abandons the correct answer. | 0.01% | Fading |