Labs’ versions of public benchmarks
A developer’s own version of a public benchmark, or its own run of a public method, reported in its model releases: GDPval v2, a hard subset of SWE-bench Verified, a lab’s ECI fit. We count 68 benchmarks here. A benchmark belongs here when it is run by a model developer, and it is a version of a public benchmark or a run of another’s method.
How fast it grows
13 were released since July, against 18 in the first half of 2026.
How much is still open
No model has passed 85% on 28 of the 40 benchmarks here that score in percent, 70% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| GDPval v2 (OpenAI) Multistep assignments drawn from real projects in consulting, finance and design, given with the files and messages the team had when the work was assigned. OpenAI has not published the task count or grading. | Expert | Nov 2026 | 50.4% | In use |
| OpenAI Epoch Capabilities Index (OECI) OpenAI's own capabilities index, fit with Epoch AI's method over 34 public and 35 internal benchmarks. | Research frontier | Nov 2026 | 170.1 | System cards only |
| Prompt Injection in Benchmark Environments An automated attacker plants prompt injections in 115 held-out agent environments adapted from Gray Swan challenges. Scored on the share of attacks the model resists. | Expert | Nov 2026 | 99.8% | System cards only |
| Historical Jailbreak Regression Replays previously found cyber jailbreak attacks against fixed ExploitBench tasks with real-time blocking turned off. Scored by mean task success rate under attack. | Expert | Oct 2026 | 14.3% | System cards only |
| MirrorDojo Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDojo. Each episode pairs a user task with an instruction planted in a tool result. Reports attack success rate and utility. | Everyday | Oct 2026 | 3.10% | System cards only |
| MirrorDyn Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDyn. Reports attack success rate and utility. | Everyday | Oct 2026 | 3.60% | System cards only |
| Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. | Expert | Sep 2026 | 0% | In use |
| ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. | Research frontier | Sep 2026 | 53.5% | In use |
| Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. | Expert | Sep 2026 | System cards only | |
| Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. | Expert | Sep 2026 | 61.2% | System cards only |
| Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. | Expert | Aug 2026 | | System cards only |
| Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. | Expert | Jul 2026 | | System cards only |
| ProtocolQA Open-Ended (xAI) Models name the single most important mistake in a biology lab protocol in free text, with no answer choices. | Expert | Jul 2026 | 87% | System cards only |
| GDPval-Rubrics (MiniMax) MiniMax rubric grading of model deliverables on public GDPval tasks, in an environment matched to GDPval-AA. | Expert | Jun 2026 | | In use |
| MAI internal jailbreak evaluation suite Models face about 9,500 jailbreak prompts built from 2,500 seed scenarios, grouped as foundational, compositional and adaptive. Scored as attack success rate. | Everyday | Jun 2026 | System cards only | |
| VIBE-V2 (MiniMax) Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output. | Expert | Jun 2026 | | In use |
| MT-AIME (Cohere internal) Cohere's internal machine translation of AIME 2025 problems into many languages (via Command A Translate). Answer accuracy. | Competition | May 2026 | | Fading |
| RepCount-A Models count repeated actions in streamed video and are scored by distance from the true count. | Everyday | May 2026 | | Fading |
| Anthropic ECI (AECI) Index that combines a basket of capability benchmarks into one capability score to track trends across Anthropic models. | Research frontier | Apr 2026 | 169.4 | System cards only |
| DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. | Research frontier | Apr 2026 | 16.5% | System cards only |
| Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. | Expert | Apr 2026 | 2.60% | System cards only |
| CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app. | Expert | Feb 2026 | 74.8% | Fading |
| CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file. | Expert | Feb 2026 | 72.2% | Fading |
| CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step. | Expert | Feb 2026 | 52.3% | System cards only |
| GDPVal-Diamond Models complete a ByteDance Seed subset of GDPval occupational tasks. Rubrics grade each deliverable automatically. | Expert | Feb 2026 | | System cards only |
| MENT-SNS Models translate English and Chinese sentences with slang, wordplay and idioms. A GPT-4.1 judge compares each translation with a baseline. | University | Feb 2026 | 1013.0 | System cards only |
| SA-1B Hard human pose evaluation Models locate body and foot joints in photographs with unusual viewpoints, poses, and occlusions. Meta measures the fraction of correctly placed joints. | School | Feb 2026 | Not in cards | |
| VIBE-Pro (MiniMax) Agents build full-stack projects for Web, Android, iOS and Windows, including server APIs and databases. Claude Code based verifiers check interaction logic and visual output. | Expert | Feb 2026 | | Fading |
| Ko-AIME 2025 (Upstage) Models solve AIME 2025 competition problems in Korean. Upstage built the set in house. | Competition | Jan 2026 | | Fading |
| tau2-Noise (LongCat) Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Everyday | Jan 2026 | | Fading |
| VitaBench-Noise (LongCat) Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Everyday | Jan 2026 | | Fading |
| Ko-AIME 2024 (Upstage) Models solve AIME 2024 competition problems translated into Korean. Upstage built the set in house. | Competition | Dec 2025 | | System cards only |
| Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. | Everyday | Nov 2025 | | System cards only |
| Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. | Everyday | Nov 2025 | | System cards only |
| HaltEval-prelim (Meta) Models judge whether 230 Python programs translated from SV-COMP and TPDB terminate. A divergence claim must give an input that times out after 5 seconds. | Expert | Sep 2025 | | System cards only |
| MultiChallenge Audio (Xiaomi) Models answer the final turn of MultiChallenge conversations converted to speech, with speech-to-text and speech-to-speech versions. Xiaomi built the conversion in house. | Everyday | Sep 2025 | 15.2% | System cards only |
| Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. | Expert | Sep 2025 | System cards only | |
| CharXiv Missing Image Models get CharXiv chart questions with every image removed. Scored on how often they attempt an answer instead of abstaining. | University | Aug 2025 | 9.00% | System cards only |
| ComplexFuncBench Audio (OpenAI) Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs. | Everyday | Aug 2025 | 66.5% | Dropped |
| MultiChallenge Audio (OpenAI) Speech-to-speech models answer an audio-friendly subset of MultiChallenge multi-turn questions that OpenAI converted from text to speech. Scored on instruction following accuracy. | Everyday | Aug 2025 | 30.5% | Dropped |
| Past IMO (Seed-Prover formal set) Provers write Lean proofs for 155 past IMO problems, mostly adapted from Compfiles and MiniF2F. | Competition | Jul 2025 | | Dropped |
| IFEval (Mistral internal version) Models follow prompts with verifiable format constraints. Mistral uses its own fixed version of IFEval. | Everyday | Jun 2025 | | Dropped |
| Multilingual AIME 2024 (Mistral) Models solve AIME 2024 problems translated into French, Spanish, German, Italian, Russian and Chinese, reasoning in the question language. | Competition | Jun 2025 | System cards only | |
| LVIS-MG Models ground many objects at once by outputting boxes for named categories in LVIS images. | Everyday | May 2025 | | Dropped |
| HumanEval-COBOL (Cohere) Models write COBOL versions of HumanEval problems, or translate COBOL solutions to Python. Unit tests score pass@1. | University | Apr 2025 | | System cards only |
| m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. | Everyday | Apr 2025 | | Dropped |
| Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. | School | Mar 2025 | System cards only | |
| Appropriate harmlessness (Anthropic) Models answer borderline and harmful prompts. Classifiers sort each response into helpful answer, policy violation, appropriate refusal or unnecessary refusal. | Everyday | Feb 2025 | System cards only | |
| SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. | Expert | Feb 2025 | 48.2% | System cards only |
| Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. | Expert | Dec 2024 | 63.1% | System cards only |
| ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. | Expert | Dec 2024 | 47.2% | In use |
| Ko-IFEval (Upstage) Models follow verifiable instructions written in Korean. Upstage adapted IFEval to Korean for the Open Ko-LLM Leaderboard2. | Everyday | Oct 2024 | 64% | System cards only |
| Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. | Everyday | Sep 2024 | | Dropped |
| Disparate performance on voice inputs Speech models answer capability and safety prompts spoken in system voices and 27 human voices. Compares accuracy across voices. | Everyday | Aug 2024 | System cards only | |
| Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. | Expert | Aug 2024 | 0% | System cards only |
| Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. | University | Jul 2024 | | System cards only |
| WildChat non-toxic incorrect refusals (Anthropic) Models answer non-toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the incorrect refusal rate. | Everyday | Mar 2024 | 4.80% | System cards only |
| WildChat toxic refusals (Anthropic) Models answer toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the correct refusal rate. | Everyday | Mar 2024 | 96.4% | System cards only |
| Translated ARC-Challenge, HellaSwag and MMLU (Mistral) Models answer French, German, Spanish and Italian versions of ARC-Challenge, HellaSwag and MMLU. | School | Jan 2024 | System cards only | |
| MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. | University | Mar 2023 | 0.07 | System cards only |
| MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. | University | Mar 2023 | System cards only | |
| CharXiv Missing Images Models answer CharXiv chart questions with the image removed. The benchmark reports how often they answer incorrectly instead of declining. | University | | System cards only | |
| Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. | Expert | System cards only | ||
| Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. | Expert | | System cards only | |
| Intentionally taking actions very rarely (Anthropic) Models are told to take a specific action at a target rate such as 1 percent. Resampling measures the observed rate under several randomness settings. | Expert | System cards only | ||
| Meta action-belief coherence probe Models act on items from five alignment evaluations and then state beliefs about them. Reports a mean coherency score. | Expert | | System cards only | |
| StrongREJECT v2 (Meta) Models face a cascade of jailbreak attacks on disallowed behaviors adapted from StrongREJECT plus internal ones. Reports attack success rate. | Expert | | In use | |
| SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. | Expert | | System cards only |