Benchmark Almanac

Epoch AI

We track 20 benchmarks that Epoch AI made or helped make.

2 in use 3 with public code 16 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
FrontierMath Unpublished research-level math problems with automatically verifiable answers. Tier 4 is the hardest tier. Research mathematics Nov 2024 52.4% Fading
FrontierMath Tier 4 Models solve research-level math problems that take experts days, submitting answers through a Python function. Each answer is verified automatically as correct or incorrect. Research mathematics Jul 2025 39.6% Fading
FrontierMath Tiers 1-3 v2 Models solve 295 unpublished math problems from undergraduate to advanced graduate level and submit answers that code checks automatically. Scored by share correct. Research mathematics Jun 2026 95.1% In use
FrontierMath Tier 4 v2 Models solve 43 unpublished research-level math problems and submit answers that code checks automatically. Scored by share correct. Research mathematics Jun 2026 100% In use
FrontierMath Erdős Models try to prove or disprove 68 open Erdős problems in Lean 4. A result counts only if the Lean checker accepts the full proof. Research mathematics Sep 2026 10.3% System cards only
Epoch Automation Reports Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees. Simulated companies Oct 2026 81.7% System cards only
Epoch AI no-CoT serial reasoning steps How many serial reasoning steps of fixed difficulty a model can do in a single output token, without a chain of thought, at 25% above chance. Deception and alignment Nov 2026 12.3 System cards only
Epoch Capabilities Index An index that fits model capability and benchmark difficulty from scores on over 50 benchmarks in Epoch AI's hub, using item response theory. Scores read like Elo ratings. Composite indices Oct 2025 174.0 System cards only
OTIS Mock AIME 2024-2025 Models solve 45 problems from three OTIS Mock AIME exams written by students of the OTIS program. Integer answers are scored by exact match. Competition problems with answers Dec 2024 100% Not in cards
Chess Puzzles Models find the single best next move in 100 chess puzzles that Epoch generated with a chess engine. Scored by the share solved. Puzzles and chess Dec 2025 79% Not in cards
Mystery Game Puzzles 100 mid-game positions from an undisclosed puzzle variant of a well-known game. Model picks the single best move, checked against an answer key. Puzzles and chess Jul 2026 95% Not in cards
Cyber ECI Epoch estimates model capability from cybersecurity benchmark scores and places the result on the general index scale. Composite indices 171.1 Not in cards
Furniture Assembly Benchmark Model gets an IKEA manual and a photo of a partly built item, then names the faulty step or next step. Accuracy on 60 photos. Spatial reasoning and counting Sep 2026 90% Not in cards
MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass. Building whole projects Jun 2026 97.8% Not in cards
EBR-bench Agent plays the board game Earthborne Rangers 10-30 times in a row. Scored on objectives completed (of 21) to test learning from experience. Puzzles and chess Jul 2026 92.4% Not in cards
SpireClimb Agents play Slay the Spire 1 and 2 at max difficulty through a computer-use harness on fixed seeds. Score is the mean win rate. Video games Oct 2026 41.3% Not in cards
SpireClimb StS1 A20 Heart Agents play Slay the Spire 1 through its graphical interface at Ascension 20 and must defeat the Heart. Scored by win rate over fixed seeds. Video games Oct 2026 33.3% Not in cards
SpireClimb StS2 A10 Agents play Slay the Spire 2 at Ascension 10 through screenshots and mouse and keyboard, 15 runs per character. Scoring is the win rate. Video games Oct 2026 49.3% Not in cards
FrontierMath: Open Problems Models attack unsolved research math problems proposed by working mathematicians. Each problem has a program that checks a proposed solution. Epoch tracks which problems AI has solved. Research mathematics Jan 2026 Not in cards
InnovationEval Agents with thousands of GPU hours try to discover an ML technique that matches a recent human paper they have not seen. The score is the share of that paper's gain the agent reaches. AI research tasks Oct 2026 Not in cards