Benchmark Almanac

Labs’ versions of public benchmarks

A developer’s own version of a public benchmark, or its own run of a public method, reported in its model releases: GDPval v2, a hard subset of SWE-bench Verified, a lab’s ECI fit. We count 68 benchmarks here. A benchmark belongs here when it is run by a model developer, and it is a version of a public benchmark or a run of another’s method.

How fast it grows

13 were released since July, against 18 in the first half of 2026.

  1. Earlier 2
  2. 2024 H1 3
  3. 2024 H2 7
  4. 2025 H1 8
  5. 2025 H2 10
  6. 2026 H1 18
  7. 2026 H2 13

How much is still open

No model has passed 85% on 28 of the 40 benchmarks here that score in percent, 70% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
GDPval v2 (OpenAI) Multistep assignments drawn from real projects in consulting, finance and design, given with the files and messages the team had when the work was assigned. OpenAI has not published the task count or grading. Expert Nov 2026 50.4% In use
OpenAI Epoch Capabilities Index (OECI) OpenAI's own capabilities index, fit with Epoch AI's method over 34 public and 35 internal benchmarks. Research frontier Nov 2026 170.1 System cards only
Prompt Injection in Benchmark Environments An automated attacker plants prompt injections in 115 held-out agent environments adapted from Gray Swan challenges. Scored on the share of attacks the model resists. Expert Nov 2026 99.8% System cards only
Historical Jailbreak Regression Replays previously found cyber jailbreak attacks against fixed ExploitBench tasks with real-time blocking turned off. Scored by mean task success rate under attack. Expert Oct 2026 14.3% System cards only
MirrorDojo Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDojo. Each episode pairs a user task with an instruction planted in a tool result. Reports attack success rate and utility. Everyday Oct 2026 3.10% System cards only
MirrorDyn Indirect prompt injection test run in Reflection AI's Mirror agent harness, based on AgentDyn. Reports attack success rate and utility. Everyday Oct 2026 3.60% System cards only
Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. Expert Sep 2026 0% In use
ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. Research frontier Sep 2026 53.5% In use
Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. Expert Sep 2026 System cards only
Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. Expert Sep 2026 61.2% System cards only
Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. Expert Aug 2026 100% System cards only
Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. Expert Jul 2026 16.8% System cards only
ProtocolQA Open-Ended (xAI) Models name the single most important mistake in a biology lab protocol in free text, with no answer choices. Expert Jul 2026 87% System cards only
GDPval-Rubrics (MiniMax) MiniMax rubric grading of model deliverables on public GDPval tasks, in an environment matched to GDPval-AA. Expert Jun 2026 74.8% In use
MAI internal jailbreak evaluation suite Models face about 9,500 jailbreak prompts built from 2,500 seed scenarios, grouped as foundational, compositional and adaptive. Scored as attack success rate. Everyday Jun 2026 System cards only
VIBE-V2 (MiniMax) Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output. Expert Jun 2026 50.1% In use
MT-AIME (Cohere internal) Cohere's internal machine translation of AIME 2025 problems into many languages (via Command A Translate). Answer accuracy. Competition May 2026 86% Fading
RepCount-A Models count repeated actions in streamed video and are scored by distance from the true count. Everyday May 2026 35.4 Fading
Anthropic ECI (AECI) Index that combines a basket of capability benchmarks into one capability score to track trends across Anthropic models. Research frontier Apr 2026 169.4 System cards only
DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. Research frontier Apr 2026 16.5% System cards only
Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. Expert Apr 2026 2.60% System cards only
CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app. Expert Feb 2026 74.8% Fading
CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file. Expert Feb 2026 72.2% Fading
CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step. Expert Feb 2026 52.3% System cards only
GDPVal-Diamond Models complete a ByteDance Seed subset of GDPval occupational tasks. Rubrics grade each deliverable automatically. Expert Feb 2026 21.3% System cards only
MENT-SNS Models translate English and Chinese sentences with slang, wordplay and idioms. A GPT-4.1 judge compares each translation with a baseline. University Feb 2026 1013.0 System cards only
SA-1B Hard human pose evaluation Models locate body and foot joints in photographs with unusual viewpoints, poses, and occlusions. Meta measures the fraction of correctly placed joints. School Feb 2026 Not in cards
VIBE-Pro (MiniMax) Agents build full-stack projects for Web, Android, iOS and Windows, including server APIs and databases. Claude Code based verifiers check interaction logic and visual output. Expert Feb 2026 55.6% Fading
Ko-AIME 2025 (Upstage) Models solve AIME 2025 competition problems in Korean. Upstage built the set in house. Competition Jan 2026 97.7% Fading
tau2-Noise (LongCat) Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. Everyday Jan 2026 67.1% Fading
VitaBench-Noise (LongCat) Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. Everyday Jan 2026 20.5% Fading
Ko-AIME 2024 (Upstage) Models solve AIME 2024 competition problems translated into Korean. Upstage built the set in house. Competition Dec 2025 80.3% System cards only
Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. Everyday Nov 2025 99.98% System cards only
Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. Everyday Nov 2025 99.8% System cards only
HaltEval-prelim (Meta) Models judge whether 230 Python programs translated from SV-COMP and TPDB terminate. A divergence claim must give an input that times out after 5 seconds. Expert Sep 2025 94% System cards only
MultiChallenge Audio (Xiaomi) Models answer the final turn of MultiChallenge conversations converted to speech, with speech-to-text and speech-to-speech versions. Xiaomi built the conversion in house. Everyday Sep 2025 15.2% System cards only
Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. Expert Sep 2025 System cards only
CharXiv Missing Image Models get CharXiv chart questions with every image removed. Scored on how often they attempt an answer instead of abstaining. University Aug 2025 9.00% System cards only
ComplexFuncBench Audio (OpenAI) Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs. Everyday Aug 2025 66.5% Dropped
MultiChallenge Audio (OpenAI) Speech-to-speech models answer an audio-friendly subset of MultiChallenge multi-turn questions that OpenAI converted from text to speech. Scored on instruction following accuracy. Everyday Aug 2025 30.5% Dropped
Past IMO (Seed-Prover formal set) Provers write Lean proofs for 155 past IMO problems, mostly adapted from Compfiles and MiniF2F. Competition Jul 2025 78.1% Dropped
IFEval (Mistral internal version) Models follow prompts with verifiable format constraints. Mistral uses its own fixed version of IFEval. Everyday Jun 2025 87.4% Dropped
Multilingual AIME 2024 (Mistral) Models solve AIME 2024 problems translated into French, Spanish, German, Italian, Russian and Chinese, reasoning in the question language. Competition Jun 2025 System cards only
LVIS-MG Models ground many objects at once by outputting boxes for named categories in LVIS images. Everyday May 2025 72.5% Dropped
HumanEval-COBOL (Cohere) Models write COBOL versions of HumanEval problems, or translate COBOL solutions to Python. Unit tests score pass@1. University Apr 2025 25.3% System cards only
m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. Everyday Apr 2025 46.5% Dropped
Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. School Mar 2025 System cards only
Appropriate harmlessness (Anthropic) Models answer borderline and harmful prompts. Classifiers sort each response into helpful answer, policy violation, appropriate refusal or unnecessary refusal. Everyday Feb 2025 System cards only
SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. Expert Feb 2025 48.2% System cards only
Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. Expert Dec 2024 63.1% System cards only
ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. Expert Dec 2024 47.2% In use
Ko-IFEval (Upstage) Models follow verifiable instructions written in Korean. Upstage adapted IFEval to Korean for the Open Ko-LLM Leaderboard2. Everyday Oct 2024 64% System cards only
Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. Everyday Sep 2024 87% Dropped
Disparate performance on voice inputs Speech models answer capability and safety prompts spoken in system voices and 27 human voices. Compares accuracy across voices. Everyday Aug 2024 System cards only
Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. Expert Aug 2024 0% System cards only
Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. University Jul 2024 83.2% System cards only
WildChat non-toxic incorrect refusals (Anthropic) Models answer non-toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the incorrect refusal rate. Everyday Mar 2024 4.80% System cards only
WildChat toxic refusals (Anthropic) Models answer toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the correct refusal rate. Everyday Mar 2024 96.4% System cards only
Translated ARC-Challenge, HellaSwag and MMLU (Mistral) Models answer French, German, Spanish and Italian versions of ARC-Challenge, HellaSwag and MMLU. School Jan 2024 System cards only
MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. University Mar 2023 0.07 System cards only
MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. University Mar 2023 System cards only
CharXiv Missing Images Models answer CharXiv chart questions with the image removed. The benchmark reports how often they answer incorrectly instead of declining. University 12.3% System cards only
Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. Expert System cards only
Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. Expert 83% System cards only
Intentionally taking actions very rarely (Anthropic) Models are told to take a specific action at a target rate such as 1 percent. Resampling measures the observed rate under several randomness settings. Expert System cards only
Meta action-belief coherence probe Models act on items from five alignment evaluations and then state beliefs about them. Reports a mean coherency score. Expert 89.1% System cards only
StrongREJECT v2 (Meta) Models face a cascade of jailbreak attacks on disallowed behaviors adapted from StrongREJECT plus internal ones. Reports attack success rate. Expert 0.50% In use
SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. Expert 0% System cards only