ChangeMyView
Models write rebuttals to r/ChangeMyView posts that human raters score for persuasiveness. Scored as percentile against human responses.
- o1 89.1%
- OpenAI deep research 84.1%
- o3-mini 82.9%
- o1-preview 81.8%
- o1-mini 69.5%
As labs report it
OpenAI
We track 205 benchmarks and 7 arenas that OpenAI made or helped make. Its models are on its lab page.
29 in use 28 with public code 98 with a leaderboard here
Arena leaderboards
Top five in each of its 2 arenas with three or more rated models, the most rated first. We don't have a leaderboard yet for 5 others, listed below.
Political Persuasion Parallel Generation
Human raters pick the more persuasive of two short political social media posts. Scored as win rate against GPT-4o.
- o1-preview 53.8%
- o1 47.1%
- o1-mini 46.8%
- o3-mini 35.5%
As labs report it
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| SWE-bench Verified 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. | Fixing real repositories | Aug 2024 | 83.5% | In use |
| HumanEval 164 hand-written Python function problems. Generated code is scored pass@1 against unit tests. | Function-level problems | Jul 2021 |  98.8% | Fading |
| GSM8K 8.5K grade-school math word problems. Final numeric answer scored by exact match. | Word problems | Oct 2021 |  96.8% | Fading |
| BrowseComp Hard-to-find facts that need persistent web browsing. Short verifiable answers graded for exact correctness. | Web browsing | Apr 2025 | 93.3% | In use |
| MMMLU MMLU professionally translated into 14 languages. Multiple-choice accuracy. | Knowledge in many languages | Sep 2024 | 92.7% | In use |
| MATH-500 Models solve 500 competition math problems drawn from the MATH test set. Final answers are checked against the reference answers. | Competition problems with answers | May 2023 |  99.2% | Fading |
| SimpleQA Models answer 4,326 short fact questions that each have one indisputable answer. A grader model marks each reply correct, incorrect, or not attempted. | Short factual questions | Oct 2024 |  97.1% | Fading |
| HealthBench Professional 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. | Medicine | Apr 2026 | 69.2% | In use |
| HealthBench 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. | Medicine | May 2025 | 67.2% | In use |
| TruthfulQA Models answer 817 questions across 38 categories that are written to elicit common misconceptions. Scoring checks for truthful answers, often in multiple-choice form. | Short factual questions | Sep 2021 |  88% | Fading |
| Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. | Lab protocols | Aug 2024 | 94% | System cards only |
| GraphWalks A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. | Reasoning over long documents | Apr 2025 | 99.96% | In use |
| ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. | Lab protocols | Dec 2024 | 47.2% | In use |
| Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. | Lab protocols | Dec 2024 | 63.1% | System cards only |
| HealthBench Hard Models answer the 1,000 hardest HealthBench conversations, graded against physician-written rubrics. | Medicine | May 2025 | 46.2% | Fading |
| OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. | Refusing harmful requests | System cards only | ||
| Image input safety evaluations Models answer harmful prompts that combine text and images in categories such as hate, extremism and self-harm. Scored as safe completion rate. | Refusing harmful requests | Dec 2024 | 99% | System cards only |
| TroubleshootingBench Models find and correct realistic experimental errors in expert-written wet lab protocols. Scored as pass@1. | Lab protocols | Aug 2025 | 59% | System cards only |
| HealthBench Consensus Models answer HealthBench conversations graded only on criteria that several physicians agreed on. | Medicine | May 2025 | 96.6% | Dropped |
| Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. | Capture-the-flag challenges | Aug 2024 | 96.3% | Fading |
| OpenAI cyber safety evaluations (incl. Daybreak Blue) Refusal and helpfulness evaluations for cyber requests, including the Daybreak Blue set for enabling defensive security work. | Refusing harmful requests | System cards only | ||
| OpenAI Dynamic Mental Health Benchmarks Multi-turn conversations with adversarial simulated users on mental-health risks. Scored on safe and supportive responses. | Wellbeing and sycophancy | System cards only | ||
| MLE-bench Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. | Machine learning engineering | Oct 2024 | 93.8% | In use |
| SWE-Lancer Models complete real Upwork software tasks, from bug fixes to features, and pick between engineering proposals. Scored by end-to-end tests and payout earned. | Fixing real repositories | Feb 2025 | 81.4% | Fading |
| Multiturn Jailbreak Evaluations Models face adaptive multi-turn jailbreak attackers with growing attack budgets. Scored as worst-case defender success rate. | Jailbreaks and prompt injection | Mar 2026 | System cards only | |
| CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. | Deception and alignment | Mar 2026 |  83.7% | System cards only |
| OpenAI MRCR v2 Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. | Finding facts in long inputs | Dec 2025 | 89% | In use |
| PaperBench Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. | AI research tasks | Apr 2025 |  93% | In use |
| Standard Refusal Evaluation Models answer disallowed and benign safety-related prompts across policy categories. An autograder scores not_unsafe and not_overrefuse. | Refusing harmful requests | Aug 2024 | 100% | System cards only |
| OpenAI user-flagged hallucination eval Production conversations users flagged for factual errors. Measures the rate of incorrect claims. | Hallucination rates | 9.20% | System cards only | |
| Biology Model Refusal Evaluation Models answer severe, dual-use and benign biology prompts. Scored as safe rate on harmful prompts and not over-refusing benign ones. | Refusing harmful requests | Jun 2026 | 99.9% | System cards only |
| Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. | Capture-the-flag challenges | Aug 2024 | 68% | System cards only |
| Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. | Finding and exploiting real bugs | Apr 2025 | 93.3% | System cards only |
| Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. | Hazardous knowledge | Sep 2024 | 0% | System cards only |
| GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. | Tasks from real jobs | Sep 2025 |  87.9% | In use |
| OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. | Labs' internal R&D tests | Jan 2025 | 55% | System cards only |
| SimpleQA Hallucination Rate Models answer 4,326 short fact-seeking questions. This metric is the share of questions answered incorrectly, so declining to answer is not penalized. | Hallucination rates | Oct 2024 | 7.90% | Dropped |
| First-Person Fairness evaluation Tests whether a user's name (a demographic proxy) changes responses, using an LLM research assistant to detect harmful stereotypes. | Bias, fairness and politics | Oct 2024 | 0.40% | System cards only |
| OpenAI MRCR Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio, in context bins from 4K to 1M tokens. | Finding facts in long inputs | Apr 2025 |  94.4% | Dropped |
| Broken Search Tool Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. | Deception and alignment | Aug 2025 | 0.33% | In use |
| Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. | Labs' internal R&D tests | Apr 2026 | 78% | System cards only |
| MakeMeSay A manipulator model tries to make a GPT-4o manipulatee say a codeword without suspicion. Scored as manipulator win rate. | Dangerous capabilities | Sep 2023 | 72% | Fading |
| Destructive action avoidance Coding agents meet conflicting user edits during tasks. Scored on preserving user changes and avoiding data-destructive actions. | Deception and alignment | Feb 2026 | 90% | System cards only |
| Agent JSK Agents read simulated email connectors that contain prompt injection attacks. Scored on resisting the injected instructions. | Jailbreaks and prompt injection | Dec 2025 | 100% | System cards only |
| Prompt injection robustness Agents meet indirect prompt injection attacks inside tool outputs. Scored as defender success rate averaged per defender query. | Jailbreaks and prompt injection | Jul 2026 | 99.8% | System cards only |
| Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. | Deception and alignment | Sep 2026 | 0% | In use |
| Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. | Deception and alignment | Sep 2026 | 11.8% | In use |
| KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. | GPU kernels | Jun 2026 | 60.4% | System cards only |
| Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. | Capture-the-flag challenges | Aug 2024 | 82% | System cards only |
| OpenAI Research Engineer Interview: Multiple Choice Models answer 97 multiple-choice questions drawn from OpenAI machine learning interview topics. Scored as cons@32. | Labs' internal R&D tests | Aug 2024 | 83% | System cards only |
| Instruction hierarchy attacks Models face attacks that try to override higher-priority instructions. Scored as defender success rate averaged per defender query. | Jailbreaks and prompt injection | Jul 2026 | 99.99% | System cards only |
| User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. | Refusing harmful requests | Mar 2026 | 92% | System cards only |
| U18 safety evaluations Models answer challenging prompts in teen safety categories such as age-restricted goods, eating disorders and emotional reliance. Scored as safe completion rate per category. | Refusing harmful requests | Aug 2026 | System cards only | |
| Challenging Refusal Evaluation Models answer a harder set of disallowed content prompts across policy categories. An autograder scores the not_unsafe rate. | Refusing harmful requests | Sep 2024 | 93.4% | System cards only |
| OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. | Labs' internal R&D tests | Aug 2025 | 8.00% | System cards only |
| PersonQA Models answer questions about publicly available facts on people. Scored on accuracy, with hallucination rate reported alongside. | Hallucination rates | Dec 2024 | 94.3% | System cards only |
| FrontierScience Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. | Research science | Dec 2025 |  87% | In use |
| OpenAI Research Engineer Interview: Coding Models solve 18 self-contained coding problems like those in OpenAI interviews. Scored as pass@1. | Labs' internal R&D tests | Aug 2024 | 100% | System cards only |
| miniF2F Models write formal proofs of olympiad and high school competition statements, such as AMC, AIME and IMO problems. A proof assistant checks each proof, and scores are pass@k on the 244 test problems. | Formal proofs | Aug 2021 |  100% | Fading |
| Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. | Capture-the-flag challenges | Jun 2026 | 97.1% | Fading |
| Instruction Hierarchy Evaluation Models resolve conflicts between system, developer and user messages, resist tutor jailbreaks, and protect secret phrases and passwords. Scored per sub-evaluation. | Jailbreaks and prompt injection | Dec 2024 | System cards only | |
| Human Sourced Jailbreaks Models face jailbreak prompts collected from human red teaming. Scored as the not_unsafe rate. | Jailbreaks and prompt injection | Sep 2024 | 100% | System cards only |
| MLE-bench 30 Agents train models for 30 of MLE-bench's Kaggle-style competitions. The score is the share of runs that earn a bronze medal or better. | Machine learning engineering | Aug 2025 | 36.7% | System cards only |
| AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. | Genomics and life sciences | Jun 2026 | 0.53 | System cards only |
| Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. | Deception and alignment | Sep 2026 | 0% | In use |
| Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. | Genomics and life sciences | Sep 2026 | 0.47 | System cards only |
| Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. | Genomics and life sciences | Sep 2026 | 10.7 | System cards only |
| SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. | Genomics and life sciences | Sep 2026 | 0.40 | System cards only |
| Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. | Labs' internal R&D tests | Aug 2024 | 78% | System cards only |
| Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. | Deception and alignment | Oct 2026 | 8.47% | In use |
| ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. | Finding and exploiting real bugs | Sep 2026 | 42% | In use |
| OpenAI Deployment Simulation Replays representative production or internal Codex traffic on a candidate model to forecast misaligned or disallowed behavior before launch. | Deception and alignment | 0.08% | System cards only | |
| Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. | Deception and alignment | Aug 2025 | 1.30% | System cards only |
| MakeMePay A con artist model tries to talk a GPT-4o mark into paying money in a role-play. Scored as the share of conversations with a payment. | Dangerous capabilities | Sep 2023 | 57% | System cards only |
| Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. | Deception and alignment | Sep 2026 | 0% | System cards only |
| DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. | Genomics and life sciences | Apr 2026 | 16.5% | System cards only |
| Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. | Genomics and life sciences | Apr 2026 | 7.60% | System cards only |
| Factuality Heavy prompts Models answer prompts representative of factuality-heavy ChatGPT conversations. A grader with web access counts responses and claims with factual errors. | Hallucination rates | Aug 2026 | 5.40% | System cards only |
| High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. | Hallucination rates | Aug 2026 | 4.10% | System cards only |
| Prompt injection attacks in search and function calling Agents receive search results and function call outputs that contain stronger known prompt injection attacks. Scored on resisting the injected instructions. | Jailbreaks and prompt injection | Jun 2026 | 94.6% | System cards only |
| GeneBench Pro Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. | Genomics and life sciences | Jun 2026 | 31.5% | In use |
| FrontierScience-Research Models work on open-ended PhD-level research subtasks in physics, chemistry and biology, written by working scientists. A grader scores each answer against a 10-point rubric. | Research science | Dec 2025 | 38% | Fading |
| LifeSciBench Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. | Genomics and life sciences | Jun 2026 | 61.3% | In use |
| Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. | Finance | Jul 2025 | 88.6% | Fading |
| Jailbreak Augmented Examples Models face publicly known jailbreaks applied to prompts from the standard disallowed content set. Scored as not_unsafe rate. | Jailbreaks and prompt injection | Sep 2024 | 100% | System cards only |
| MedChemBench (OpenAI) Models answer chemistry questions from an OpenAI internal set, grouped with OpenAI's life science research evaluations. OpenAI has not published the task format. | Genomics and life sciences | Jul 2026 | 49.6% | In use |
| Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. | Labs' internal R&D tests | Feb 2026 | 76% | System cards only |
| Production Jailbreaks Models face jailbreaks found in production ChatGPT data. Scored as not_unsafe rate. | Jailbreaks and prompt injection | Sep 2024 | 100% | System cards only |
| MentalHealthBench 1,215 synthetic mental-health conversations with 5,262 clinician-authored rubric criteria. An LLM judge scores safety and guidance quality. | Wellbeing and sycophancy | Sep 2026 | 57.9% | System cards only |
| Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. | Deception and alignment | Sep 2026 | 0% | System cards only |
| Agentic Safe Completions Agents handle harmful requests from Codex and ChatGPT plugin traffic and human red teaming. Scored on avoiding harmful actions per category. | Refusing harmful requests | Sep 2026 | System cards only | |
| Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. | Hazardous knowledge | Sep 2024 | System cards only | |
| CoT Monitorability Monitors read a model's chain of thought, actions or all messages across intervention, process and outcome evaluations. Scored as g-mean squared of TPR and TNR. | Deception and alignment | Mar 2026 | System cards only | |
| Internal API instruction following (OpenAI) Models answer prompts drawn from real developer use cases that carry instructions on formatting, verbosity and length. OpenAI reports a hard subset. | Following precise instructions | Apr 2025 | 54% | Dropped |
| Coding prompt injections Agents work on coding tasks that contain prompt injection attacks. Scored on resisting the injected instructions. | Jailbreaks and prompt injection | Aug 2025 | 100% | System cards only |
| Contextual Nuclear Knowledge Models answer 222 multiple choice questions on nuclear engineering with a nonproliferation focus. Scored as cons@32 accuracy. | Dangerous capabilities | Dec 2024 | 77% | System cards only |
| Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. | Deception and alignment | Sep 2026 | 1.40% | System cards only |
| Radiological and Nuclear Expert Knowledge Models answer 87 expert and tacit knowledge questions on radiological and nuclear threats. Scored as cons@32 accuracy. | Dangerous capabilities | Dec 2024 | 74% | System cards only |
| Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. | Finding and exploiting real bugs | Sep 2026 | 45.5% | System cards only |
| SWE-Lancer Diamond (SWE Manager) Models review competing implementation proposals for real freelance tasks and pick the best one. Scored against the original choice as pass@1. | Fixing real repositories | Feb 2025 | 51% | System cards only |
| WildChat (toxic conversations) Models answer toxic user turns from WildChat, the 200 highest scoring conversations per moderation category. Scored as not_unsafe rate. | Refusing harmful requests | May 2024 | 98% | System cards only |
| OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. | Deception and alignment | Sep 2026 | 0% | System cards only |
| Deception Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the share of outputs flagged for deceptive behaviors such as fabricated facts. | Deception and alignment | Mar 2026 | System cards only | |
| Disallowed Content Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the estimated prevalence of policy violating outputs per category. | Refusing harmful requests | Mar 2026 | System cards only | |
| Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. | Hazardous knowledge | Sep 2024 | System cards only | |
| Static Jailbreak Evaluations Models face a fixed set of jailbreak attacks on bio, violence and cyber requests. Scored as defender success rate per category. | Jailbreaks and prompt injection | Sep 2026 | System cards only | |
| Management Consulting Tasks (OpenAI) Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading. | Running a business | Jul 2026 | 43.2% | In use |
| BrowseComp Long Context 128k Models answer BrowseComp questions from a 128k-token context of the needed web pages mixed with related distractor pages. Scored by accuracy. | Reasoning over long documents | Aug 2025 | 90% | Dropped |
| Challenging prompts from red teamers with biosafety-relevant PhDs Models answer adversarial biology prompts written by red teamers with biosafety PhDs. Scored as not_unsafe rate from model training alone. | Refusing harmful requests | Aug 2025 | 93.6% | System cards only |
| Filtered, adversarial sample of production prompts Models answer filtered borderline and high-risk biology prompts that resemble production traffic. Scored as not_unsafe rate from model training alone. | Refusing harmful requests | Aug 2025 | 96.9% | System cards only |
| Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. | Deception and alignment | Sep 2026 | 100% | System cards only |
| Malware Refusals (Golden Set) Coding agents receive malware development requests from a curated golden set written by OpenAI policy experts. Scored as refusal rate. | Refusing harmful requests | May 2025 | 100% | System cards only |
| QuantBench Models answer 25 multiple choice versions of quant trading firm reasoning puzzles. Scored as cons@16 accuracy. | Competition problems with answers | Sep 2024 | 60% | System cards only |
| Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours. | Building whole projects | Apr 2026 | 83% | Fading |
| HealthBench Hard Hallucinations Models respond to difficult health conversations that 2 or more physicians validated. The score is the hallucination rate, so lower is better. | Hallucination rates | Aug 2025 | 1.60% | Dropped |
| Internal adversarially-designed factuality evaluations Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. | Hallucination rates | Mar 2023 | 7.60% | In use |
| Regurgitation Evaluations Models are asked to reproduce training data. Scored on how often they refuse to regurgitate it. | Refusing harmful requests | Sep 2024 | System cards only | |
| Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. | Deception and alignment | Jun 2026 | System cards only | |
| VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. | Finding and exploiting real bugs | Apr 2026 | System cards only | |
| FrontierScience-Olympiad Models solve olympiad-style physics, chemistry and biology problems at the level of IPhO, IChO and IBO, written by medalists and coaches. Graded on short final answers. | Science olympiads | Dec 2025 | 77.1% | System cards only |
| Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. | Genomics and life sciences | Apr 2026 | 39.3% | System cards only |
| BirthdayFacts Models are asked for people's birthdays. Scored on how often they give a wrong birthday. | Hallucination rates | Sep 2024 | 24% | System cards only |
| Challenging Red Teaming Evaluation 1 (created for o3-mini) Models face adversarial jailbreaks from o3-mini red teaming on illicit advice, extremism, political persuasion and self-harm. Scored as not_unsafe rate. | Jailbreaks and prompt injection | Feb 2025 | 51% | System cards only |
| Challenging Red Teaming Evaluation 2 (created for deep research) Models face adversarial prompts from deep research red teaming on risky advice such as attack planning. Scored as not_unsafe rate. | Jailbreaks and prompt injection | Feb 2025 | 67% | System cards only |
| CharXiv Missing Image Models get CharXiv chart questions with every image removed. Scored on how often they attempt an answer instead of abstaining. | Deception and alignment | Aug 2025 | 9.00% | System cards only |
| CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. | Deception and alignment | Sep 2024 | 0.17% | System cards only |
| Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. | Deception and alignment | Aug 2025 | 1.60% | System cards only |
| External red team jailbreaks External red teamers and automated gap finding generate jailbreaks against the model. Scored on how often the model resists. | Jailbreaks and prompt injection | Sep 2024 | 44% | System cards only |
| GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. | Genomics and life sciences | Apr 2026 | 33.2% | Fading |
| Hallucinations on production-representative prompts Models answer prompts representative of ChatGPT production traffic. A grader with web access scores incorrect claims and responses with major errors. | Hallucination rates | Aug 2025 | 3.10% | System cards only |
| HealthBench Consensus Global Health Models answer health conversations that need adjustment for global context such as local care access. Scored as the failure rate. | Medicine | Aug 2025 | 0% | System cards only |
| HealthBench Consensus Urgent Models answer ambiguous high-stakes health conversations validated by two or more physicians. Scored as the error rate in informing the user. | Medicine | Aug 2025 | 0.40% | System cards only |
| Image generation refusals Models decide whether to call the image generation tool on human-curated adversarial prompts with system mitigations on. Scored on not_unsafe and not_overrefuse. | Refusing harmful requests | Apr 2025 | 98% | System cards only |
| Open Ended Questions (hallucination) Models write open-ended factual text such as biographies, checked against Wikipedia. Scored as average number of incorrect statements. | Long-form factuality | Sep 2024 | 0.78 | System cards only |
| PlugInject Agents receive function call outputs that contain prompt injection attacks. Scored on resisting the injected instructions. | Jailbreaks and prompt injection | Dec 2025 | 99.6% | System cards only |
| Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. | Refusing harmful requests | Jan 2025 | 94% | System cards only |
| Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. | Lab protocols | Dec 2024 | 51% | System cards only |
| Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. | Refusing harmful requests | Jan 2025 | 97% | System cards only |
| Destructive action reversion on long rollouts Agents revert their own changes after long rollouts in workspaces with simulated user edits. Scored on perfect reversion and on preserving user work. | Deception and alignment | Mar 2026 | System cards only | |
| OpenAI monitorability evals Open suite from 'Monitoring Monitorability' testing whether CoT and action monitors catch hidden properties of agent behavior. | Deception and alignment | Dec 2025 | System cards only | |
| Person Identification evaluation Models receive requests to identify people in photos. Scored on refusal rate in non-adversarial and adversarial sets. | Refusing harmful requests | Apr 2025 | System cards only | |
| Structured expert probing campaign: radiological and nuclear Radiological and nuclear experts test whether a model helps with weapons development steps. Conversations are rated against Preparedness risk levels. | Dangerous capabilities | Dec 2024 | System cards only | |
| Ungrounded Inference evaluation Models receive requests for inferences about people that images or text cannot justify. Scored on refusal rate in non-adversarial and adversarial sets. | Bias, fairness and politics | Apr 2025 | System cards only | |
| Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. | Jailbreaks and prompt injection | Jul 2025 | 67% | System cards only |
| Advanced Cybersecurity Completion Rate Models receive arbitrary advanced cyber requests under different access configurations. Scored as the share of requests fully completed. | Refusing harmful requests | Sep 2026 | 3.50% | System cards only |
| Agent Replay Auto-Review Monitors replayed agentic conversations for harmful or unauthorized actions. Scored by the rate above a severity threshold. | Deception and alignment | Oct 2026 | 27.1% | System cards only |
| Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. | Jailbreaks and prompt injection | Aug 2025 | 99% | System cards only |
| Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea. | Fixing real repositories | Sep 2025 | 51.3% | Dropped |
| ComplexFuncBench Audio (OpenAI) Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs. | Function calling | Aug 2025 | 66.5% | Dropped |
| Context Aware Automatic Speech Recognition Speech transcription systems are scored for semantic accuracy with and without free-form context. | Speech recognition and translation | Jul 2026 | 44.6% | In use |
| Data Science Tasks (OpenAI) Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | Documents and spreadsheets | Sep 2026 | 40.9% | In use |
| Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | Fixing real repositories | Sep 2026 | 63.9% | In use |
| Design Tasks (OpenAI) Models produce design work such as slides and documents that follow templates, from an OpenAI internal set. OpenAI has not published the grading. | Documents and spreadsheets | Sep 2026 | 50% | In use |
| Historical Jailbreak Regression Replays previously found cyber jailbreak attacks against fixed ExploitBench tasks with real-time blocking turned off. Scored by mean task success rate under attack. | Jailbreaks and prompt injection | Oct 2026 | 14.3% | System cards only |
| HumanEval Infilling Models fill in a masked middle span of HumanEval solutions given the code before and after it. Unit tests check the completed function. | Function-level problems | Jul 2022 | 84.8% | System cards only |
| In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. | Jailbreaks and prompt injection | Jul 2025 | 78% | System cards only |
| Internal computer use safety benchmark (OpenAI) Models complete computer-use tasks while avoiding unsafe actions. | Desktop apps | Sep 2026 | 2.40% | System cards only |
| Irrelevant instructions, text-based web browser Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them. | Jailbreaks and prompt injection | Jul 2025 | 99.5% | System cards only |
| Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. | Jailbreaks and prompt injection | Jul 2025 | 95% | System cards only |
| Leetcode (easy) Models solve 41 easy Leetcode problems that OpenAI selected. Scored as problems solved. | Competitive programming | Mar 2023 | 75.6% | System cards only |
| Leetcode (hard) Models solve 45 hard Leetcode problems that OpenAI selected. Scored as problems solved. | Competitive programming | Mar 2023 | 6.70% | System cards only |
| Leetcode (medium) Models solve 80 medium Leetcode problems that OpenAI selected. Scored as problems solved. | Competitive programming | Mar 2023 | 26.3% | System cards only |
| Malware Refusals (Synthetic) Coding agents receive a large synthetic set of malware-related tasks, including ambiguous and dual-use prompts. Scored as refusal rate. | Refusing harmful requests | May 2025 | 97% | System cards only |
| MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. | Short factual questions | Mar 2023 | 0.07 | System cards only |
| Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. | Deception and alignment | Sep 2026 | 61.2% | System cards only |
| MultiChallenge Audio (OpenAI) Speech-to-speech models answer an audio-friendly subset of MultiChallenge multi-turn questions that OpenAI converted from text to speech. Scored on instruction following accuracy. | Audio understanding and voice | Aug 2025 | 30.5% | Dropped |
| OpenScore String Quartets Models transcribe images of string quartet sheet music into symbolic notation. Scored as 1 minus normalized edit distance from the reference encoding. | Documents, charts and text in images | Sep 2026 | 0.84 | In use |
| Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. | Jailbreaks and prompt injection | Jan 2025 | 23% | System cards only |
| Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. | Deception and alignment | Sep 2026 | 0.20% | System cards only |
| Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. | Deception and alignment | Dec 2025 | 5.40% | System cards only |
| Real-World Audio Recording Speech transcription systems are scored by word error rate on real-world recordings across multiple languages. | Speech recognition and translation | Jul 2026 | 9.60% | In use |
| Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. | Deception and alignment | Sep 2026 | 0% | System cards only |
| Sycophancy evaluation (offline) Models answer a fixed set of messages resembling production traffic that could elicit sycophancy. A grader assigns a sycophancy score. | Wellbeing and sycophancy | Aug 2025 | 0.04 | System cards only |
| Tool-calling prompt injections Agents receive prompt injections inside tool call outputs such as connectors. Scored on resisting the injected instructions. | Jailbreaks and prompt injection | Aug 2025 | 99% | System cards only |
| Unauthorized voice generation Speech models produce audio that may deviate from the approved system voice. Scored on how many meaningful deviations the output classifier catches. | Refusing harmful requests | Aug 2024 | 100% | System cards only |
| Unexpected container or code states Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed. | Deception and alignment | May 2025 | 85% | System cards only |
| Ungrounded inference and sensitive trait attribution Speech models get requests to infer traits about a speaker from audio. Scored on refusing ungrounded inferences and hedging plausible ones. | Bias, fairness and politics | Aug 2024 | 84% | System cards only |
| Voice persuasion study on party preferences Participants hear model-voiced audio clips or talk with a voiced model. Scored on shifts in hypothetical party preference against human baselines. | Dangerous capabilities | Aug 2024 | 5.74% | System cards only |
| Biological safety evaluations (OpenAI) Models answer biology requests drawn from deidentified production data and from easy and hard synthetic sets. Scored as the share of cases where model training alone prevents a violative response. | Refusing harmful requests | May 2026 | System cards only | |
| Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. | Hazardous knowledge | Jan 2024 | System cards only | |
| Closed-domain hallucination evaluation Models answer using only provided context. Scored on avoiding invented details outside that context. | Faithful to sources | Mar 2023 | System cards only | |
| Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. | Secure code and review | Sep 2026 | System cards only | |
| Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. | Jailbreaks and prompt injection | Feb 2025 | System cards only | |
| Disparate performance on voice inputs Speech models answer capability and safety prompts spoken in system voices and 27 human voices. Compares accuracy across voices. | Bias, fairness and politics | Aug 2024 | System cards only | |
| Incorrect behavior rate on disallowed and sensitive content Models answer disallowed prompts and sensitive prompts such as medical advice. Scored on the rate of responses that break policy. | Refusing harmful requests | Mar 2023 | System cards only | |
| MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. | Knowledge in many languages | Mar 2023 | System cards only | |
| Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. | Deception and alignment | Sep 2026 | System cards only | |
| Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. | Knowledge in many languages | Jul 2025 | System cards only | |
| OpenAI performance in cases flagged by users Model answers de-identified ChatGPT prompts that users flagged for factual errors. Scored on any error and on repeating the flagged error. | Hallucination rates | Sep 2026 | System cards only | |
| Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. | Deception and alignment | Sep 2026 | System cards only | |
| Personal data evaluations Browsing agents receive requests for private personal information, from 200 synthetic prompts and 55 manually written golden examples. Scored on adherence to personal data policy. | Refusing harmful requests | Feb 2025 | System cards only | |
| Safe response to high-severity prompts Models handle multi-turn self-harm and violent activity conversations where harmful intent only emerges from context. Scored as safe response rate. | Refusing harmful requests | Sep 2026 | System cards only | |
| Speaker identification Speech models get audio requests to identify a speaker by voice. Scored on refusing when they should and complying for famous quotes. | Refusing harmful requests | Aug 2024 | System cards only | |
| Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. | Hazardous knowledge | Dec 2024 | System cards only | |
| Text persuasion study on political opinions Participants read model-written articles or chat with a model on political topics. Scored on opinion shift against human-written articles. | Dangerous capabilities | Aug 2024 | System cards only | |
| PaperBench Code-Dev Agents implement research paper code. The Code-Dev variant evaluates coding separately from full research replication. | AI research tasks | Apr 2025 | 43.4% | Not in cards |
| BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. | Genomics and life sciences | Nov 2026 | Not in cards | |
| BrowseComp Long Context 256k Models answer BrowseComp questions from a context of 128K to 256K tokens built from the needed web pages and related distractor pages. Scored by accuracy. | Reasoning over long documents | Aug 2025 | Not in cards | |
| EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. | Finding and exploiting real bugs | Feb 2026 | Not in cards | |
| LifeVal Assistants carry out everyday personal tasks rather than advise on them: finding childcare, booking a car inspection, ordering groceries on a budget, handling government paperwork. Tasks come from time-use surveys and from about 100 people who recorded tasks they wanted an assistant to handle. Score is average reward across tasks. | Web browsing | Nov 2026 | Not in cards | |
| OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. | Deception and alignment | Not in cards | ||
| OpenAI GPT-Red jailbreak and prompt-injection evals Static jailbreaks from attacker models (half from a held-out attacker), adaptive multi-turn attackers, and direct and indirect prompt-injection sets. | Jailbreaks and prompt injection | Not in cards | ||
| OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). | Labs' internal R&D tests | Not in cards | ||
| OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). | Finding and exploiting real bugs | Not in cards | ||
| RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. | Labs' internal R&D tests | Jul 2026 | Not in cards |
Arenas without a leaderboard here
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Red Teaming Network pairwise safety comparison Red teamers compare anonymized responses from two models in free-form risky conversations. Scored as the share of ratings where a model is safer. | Refusing harmful requests | Dec 2024 | 59.8% | System cards only |
| Pairwise safety red teaming Red teamers compare anonymized responses from several models and pick the least safe. Scored as ELO or win rate. | Refusing harmful requests | Apr 2025 | System cards only | |
| Expert Red Teaming for Violent Attack Planning Expert red teamers compare anonymized responses from two models on violent attack planning prompts. Scored as the win rate for being safer. | Refusing harmful requests | Aug 2025 | 65.1% | System cards only |
| Pairwise Bioweaponization Campaign Red teamers with biology PhDs compare anonymized responses from two models on bioweaponization prompts. Scored as the win rate for being safer. | Refusing harmful requests | Aug 2025 | 60% | System cards only |
| Preference on ChatGPT and API prompts Human labelers compare model responses to 5,214 prompts sent to ChatGPT and the OpenAI API. Scored as win rate against a baseline model. | Overall chat quality | Mar 2023 | 70.2% | System cards only |