Benchmark Almanac

o3

o3 ranks first on 20 of the 242 benchmarks with five or more models scored.

OpenAI Released April 16th, 2025 Family o3 The lab's model card Compare with others
250
Benchmarks with a score
20
Ranked first, out of five or more models
60
In the top three
OpenAI Apr 2025

o3

Closed weights Flagship

147.1 #83 of 256
Epoch Capabilities Index
Self-Reported Index 142.6 #158 of 404
  • Humanity's Last Exam 20.3% #20 /47
  • SWE-bench Verified 62.3% #27 /32
  • MultiChallenge 56.6% #22 /42

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
ECLeKTic 43.1% #1 of 35 Official leaderboard
Video SimpleQA 66.3% #1 of 27 Official leaderboard
Arena-Hard v2.0 85.9% #1 of 23 Official leaderboard
AraGen 82.2% #3 of 42 Official leaderboard
Seal-0 17.1% #1 of 11 Official leaderboard
Fiction.LiveBench 88.9% #6 of 57 Fiction.live leaderboard, via Epoch AI
WritingBench 80.5 #5 of 45 Official leaderboard
SnorkelSpatial 76.7% #3 of 27 Official leaderboard
MMLU-Pro 85% #30 of 212 Official leaderboard
WideSearch 5.10% #1 of 6 Official leaderboard
SimpleQA 49% #13 of 60 Lab model card
MMMLU 88.8% #16 of 71 Lab model card
Wer wird Millionär? LLM Benchmark 112800.0 #18 of 77 Official leaderboard
HLE Calibration 34.0 #1 of 4 Official leaderboard
LMArena (Elo) 1,432 #90 of 335 Official leaderboard
IFEval 86.9% #17 of 60 Official leaderboard
SimpleQA Verified 52.3% #16 of 56 Official leaderboard
DiscoX 55.6 #2 of 7 Official leaderboard
Slop Score 31.8 #5 of 17 Official leaderboard
EQ-Bench 3 1,500 #23 of 78 Official leaderboard
Creative Writing v3 73.3 #37 of 120 Official leaderboard
LLM Confabulation Leaderboard 14.4% #19 of 58 Official leaderboard
MultiNRC 45.5% #13 of 38 Official leaderboard
FACTS Search 74.8% #11 of 31 Official leaderboard
FACTS Parametric 57.1% #16 of 45 Official leaderboard
Humanity's Last Exam 20.3% #20 of 47 Official leaderboard
PersonQA 59% #4 of 9 Lab model card
Judgemark v2.1 75.1 #20 of 42 Official leaderboard
Longform Creative Writing (EQ-Bench) 62.7 #57 of 119 Official leaderboard
FaithJudge 17.8% #18 of 37 Official leaderboard
MultiChallenge 56.6% #22 of 42 Official leaderboard
CL-bench 17.8% #13 of 24 Official leaderboard
BullshitBench 23.6% #62 of 107 Official leaderboard
IndicGenBench 50.0 #8 of 13 Official leaderboard
BullshitBench V2 26% #83 of 129 Official leaderboard
SimpleQA Hallucination Rate 51% #8 of 12 Lab model card
HUMAINE 22 #37 of 50 Official leaderboard
Vectara Hallucination Leaderboard (HHEM) 6.80% #111 of 148 Official leaderboard
FACTS Grounding 69.6% #26 of 34 Official leaderboard

Math and science

BenchmarkScoreRankSource
SciArena 1,151 #1 of 34 Official leaderboard
MATH Level 5 97.8% #4 of 94 Epoch AI run
GPQA Main 75.3% #4 of 60 Official leaderboard
IneqMath 37% #3 of 34 Official leaderboard
AIME 2024 91.6% #13 of 88 Lab model card
HiPhO 64.8% #6 of 36 Official leaderboard
CMT-Benchmark 26% #2 of 11 Official leaderboard
MATH-500 94.6% #8 of 38 Official leaderboard
BRUMO 2025 95.8% #10 of 42 Official leaderboard
PHYBench 34.8% #7 of 28 Official leaderboard
SnorkelGraph 71.5% #6 of 24 Official leaderboard
ForecastBench 0.13 #25 of 82 Official leaderboard
SMT 2025 87.7% #14 of 40 Official leaderboard
DTBench 84.8% #71 of 180 Official leaderboard
AIME 2025 88.9% #57 of 143 Lab model card
GPQA Diamond 81.8% #94 of 229 Epoch AI run
OTIS Mock AIME 2024-2025 84.4% #88 of 212 Epoch AI run
SnorkelSequences 65.2% #11 of 25 Official leaderboard
HMMT 2025 77.5% #24 of 53 Official leaderboard
SGI-Bench 30.7% #7 of 15 Official leaderboard
FrontierMath 18.7% #26 of 55 Official leaderboard
IMO-ProofBench 20.5% #10 of 20 Official leaderboard
IMO 2025 16.7% #3 of 6 Official leaderboard
SimpleBench 53.1% #56 of 106 Official leaderboard
FrontierScience-Research 14.4% #4 of 7 Official leaderboard
CMIMC 2025 82.5% #19 of 33 Official leaderboard
CORE-Bench 24.4% #10 of 17 Official leaderboard
AstaBench 37.8% #8 of 13 Official leaderboard
ARC-AGI-2 6.53% #62 of 98 Official leaderboard
ARC-AGI 60.8% #61 of 95 Official leaderboard
FrontierMath Tier 4 2.08% #39 of 55 Epoch AI run
FrontierScience-Olympiad 63% #5 of 7 Official leaderboard
SciPredict 17.9% #9 of 12 Official leaderboard
FrontierMath Tiers 1-3 v2 33.3% #65 of 86 Official leaderboard
LiveMathematicianBench 24.4% #13 of 17 Official leaderboard
ATLAS (OpenCompass) 33.8% #4 of 5 Official leaderboard
ScholarCatalyst 33% #3 of 3 Official leaderboard

Coding

BenchmarkScoreRankSource
AutoCodeBench 51.1% #2 of 34 Official leaderboard
LiveCodeBench v5 77.4% #2 of 21 Official leaderboard
LiveCodeBench v6 75.8% #2 of 21 Official leaderboard
Aider Polyglot 81.3% #3 of 30 Lab model card
SWE-bench Multimodal 36% #1 of 9 Official leaderboard
LiveCodeBench 89.2% #3 of 21 Official leaderboard
Codeforces rating 2,706 #8 of 47 Lab model card
MMTU 69.1% #2 of 9 Official leaderboard
METR Time Horizon 91 min #7 of 31 Official leaderboard
Spider 2.0 62.9% #6 of 23 Official leaderboard
ArtifactsBench 54% #7 of 20 Official leaderboard
CodeClash 1,343 #3 of 8 Official leaderboard
SWE-Lancer 54% #4 of 10 Lab model card
WeirdML 52.4% #45 of 112 Official leaderboard
OpenAI PRs 44% #5 of 12 Lab model card
METR Time Horizon 1.1 120 min #11 of 26 Official leaderboard
OpenAI Research Engineer Interview: Coding 97% #3 of 7 Lab model card
OpenAI Research Engineer Interview: Multiple Choice 80% #4 of 9 Lab model card
ALE-Bench 1,304 #73 of 126 Official leaderboard
GSO-Bench 8.82% #18 of 30 Official leaderboard
LiveSQLBench 23.7% #22 of 36 Official leaderboard
MLE-bench 31.6% #8 of 12 Official leaderboard
SWE-Lancer Diamond (SWE Manager) 41% #3 of 4 Lab model card
DesignArena Website 1,029 #127 of 151 Official leaderboard
SWE-bench Verified 62.3% #27 of 32 Epoch AI run
DesignArena Game Dev 1,024 #121 of 143 Official leaderboard
TasteVal 8.28× #17 of 20 Official leaderboard
DesignArena UI Component 1,008 #120 of 139 Official leaderboard
PaperBench 18% #13 of 15 Lab model card
DesignArena Code 1,010 #125 of 143 Official leaderboard

Agents and tools

BenchmarkScoreRankSource
Step Race 5 #2 of 59 Official leaderboard
SnakeBench 36 #7 of 188 Official leaderboard
lmgame-Bench 87.7 #1 of 23 Official leaderboard
BFCL V4 63% #7 of 65 Official leaderboard
MCP-Bench 0.71 #2 of 15 Official leaderboard
SnorkelWordle 92.9% #3 of 19 Official leaderboard
C3-Bench 10.2% #5 of 30 Official leaderboard
Game Arena Word Association 356 #7 of 40 Official leaderboard
MM-BrowseComp 22.8% #2 of 11 Official leaderboard
LisanBench 3308.7 #33 of 133 Official leaderboard
LLM Chess 778 #33 of 123 Official leaderboard
Game Arena Chess Openings 934 #3 of 11 Official leaderboard
VITA-Bench 26.3% #5 of 18 Official leaderboard
BrowseComp-Plus 63.5% #2 of 7 Official leaderboard
MCP-Universe 26.4% #10 of 30 Official leaderboard
Game Arena Checkers 994 #15 of 41 Official leaderboard
Elimination Game 4 #21 of 56 Official leaderboard
ITBench 51.7% #5 of 13 Official leaderboard
MCPMark 25.4% #11 of 28 Official leaderboard
Game Arena Chess 925 #17 of 42 Official leaderboard
Mystery Game Puzzles 29% #35 of 84 Epoch AI run
GAIA 62.1% #27 of 64 Official leaderboard
Pencil Puzzle Bench 3.30% #23 of 53 Official leaderboard
tau-bench 61.2% #10 of 23 Lab model card
Chess Puzzles 37% #41 of 93 Epoch AI run
Game Arena Five in a Row 540 #9 of 20 Official leaderboard
Game Arena Poker 1.17 #14 of 31 Official leaderboard
TextQuests 17.2 #13 of 28 Official leaderboard
VitaBench 2.0 40.3% #8 of 17 Official leaderboard
OSWorld-Verified 56% #16 of 33 Official leaderboard
LMArena Search Arena 1,197 #18 of 37 Official leaderboard
Game Arena Bargaining 187 #20 of 39 Official leaderboard
Galileo Agent Leaderboard 0.86 #16 of 29 Official leaderboard
Game Arena Reversi 454 #22 of 39 Official leaderboard
WebTailBench 35% #4 of 7 Official leaderboard
Kaggle Game Arena 272 #25 of 42 Official leaderboard
Game Arena Dark Hex 223 #26 of 41 Official leaderboard
Game Arena Nine Men's Morris 565 #25 of 39 Official leaderboard
Game Arena Four in a Row 293 #28 of 43 Official leaderboard
Game Arena Ultimate Tic-Tac-Toe 241 #26 of 39 Official leaderboard
Game Arena Word Art 277 #27 of 39 Official leaderboard
Game Arena Go 555 #28 of 39 Official leaderboard
Game Arena Lines of Action 293 #28 of 39 Official leaderboard
DeepResearch Bench 45.2% #16 of 22 evals.futuresearch.ai, via Epoch AI
Deep Research Bench (FutureSearch) 0.45 #16 of 21 Official leaderboard
BrowseComp 49.7% #62 of 81 Lab model card
Game Arena Clobber 144 #31 of 39 Official leaderboard
Game Arena Dots and Boxes 599 #33 of 40 Official leaderboard
Toolathlon 17% #43 of 50 Official leaderboard
OSWorld 23% #7 of 8 OS World Website, via Epoch AI
OSWorld-MCP 24.1% #8 of 9 Official leaderboard
Game Arena Coin Game 154 #37 of 39 Official leaderboard
Mind2Web 2 11% #2 of 2 Official leaderboard

Professional work

BenchmarkScoreRankSource
FinQA (Snorkel OpenEnv environment) 81% #1 of 22 Official leaderboard
SnorkelFinance 81% #1 of 22 Official leaderboard
CadEval 74% #1 of 13 Official leaderboard
HealthBench 59.8% #1 of 8 Official leaderboard
HealthBench Hard 31.6% #1 of 8 Official leaderboard
xbench-Recruitment 78.5 #1 of 8 Official leaderboard
xbench-Marketing 50.8 #1 of 7 Official leaderboard
FailureSensorIQ 67.6% #4 of 24 Official leaderboard
Vals Finance Agent v1 47.7% #8 of 47 Official leaderboard
TaxEval v2 74.6% #34 of 129 Official leaderboard
SnorkelUnderwrite 77% #8 of 27 Official leaderboard
ProfBench 52.4% #22 of 72 Official leaderboard
MedCode 47.3% #33 of 96 Official leaderboard
LegalBench 83.8% #48 of 138 Official leaderboard
PRBench 48.6% #18 of 48 Official leaderboard
StockBench 5.00 #5 of 13 Official leaderboard
MortgageTax 65.7% #37 of 87 Official leaderboard
Finance Reasoning (Snorkel) 43% #12 of 21 Official leaderboard
GDPval 34.1% #3 of 5 Official leaderboard
CorpFin v2 59.7% #73 of 119 Official leaderboard
MedScribe 76.7% #67 of 97 Official leaderboard
APEX-1 60.5% #51 of 71 Official leaderboard
Vending-Bench $1,843 #8 of 9 Official leaderboard
SpreadsheetBench 23.3% #2 of 2 Official leaderboard

Vision, audio and robotics

BenchmarkScoreRankSource
CharXiv (DQ) 95% #1 of 67 Official leaderboard
MMMU 82.9% #4 of 136 Official leaderboard
VHELM 81.0 #2 of 41 Official leaderboard
VisualPuzzles 54% #2 of 29 Official leaderboard
MORSE-500 23.6% #1 of 11 Official leaderboard
MicroVQA 59.3% #2 of 19 Official leaderboard
MMSI-Bench 41% #3 of 28 Official leaderboard
Video-MMMU 83.3% #3 of 24 Official leaderboard
LEGO-Puzzles 69.1% #2 of 16 Official leaderboard
MathVista 86.8% #9 of 70 Lab model card
MMMU-Pro 76.4% #10 of 57 Official leaderboard
ERQA 41.5% #3 of 17 Official leaderboard
Math-VR 59.3% #5 of 26 Official leaderboard
Do You See Me 51.1% #1 of 5 Official leaderboard
VISTA 50.1% #11 of 54 Official leaderboard
VisuLogic 29.5% #4 of 17 Official leaderboard
ViVerBench 73.5% #2 of 8 Official leaderboard
VPCT 52% #7 of 27 VPCT leaderboard, via Epoch AI
ChartMuseum 60.9% #9 of 21 Official leaderboard
GeoBench 74% #21 of 48 Official leaderboard
Vision Arena 1,216 #68 of 136 Official leaderboard
VisFactor 24.3% #13 of 26 Official leaderboard
CharXiv 78.6% #40 of 75 Lab model card
VLMsAreBiased 21% #3 of 5 Official leaderboard
ZeroBench 6.00% #46 of 76 Official leaderboard
Drone-Bench 55% #12 of 19 Official leaderboard
Point-Bench 43.5% #11 of 17 Official leaderboard
ClockBench 12.2% #25 of 37 Official leaderboard
OmniSpatial 56.3% #1 of 1 Official leaderboard

Safety and security

BenchmarkScoreRankSource
HELM Safety 98.2 #2 of 75 Official leaderboard
Capture the Flag (Collegiate) 68% #1 of 13 Lab model card
Multimodal Troubleshooting Virology 59% #3 of 27 Lab model card
Virology Capabilities Test (VCT) 43.8% #1 of 7 Official leaderboard
Spiral-Bench v1.0 86.1 #2 of 13 Official leaderboard
BaxBench 46.4% #6 of 34 Official leaderboard
AIR-Bench 2024 84.5% #15 of 82 Official leaderboard
Spiral-Bench v1.2 63.4 #4 of 21 Official leaderboard
First-Person Fairness evaluation 0.60% #2 of 10 Lab model card
Capture the Flag (High School) 82% #2 of 9 Lab model card
FORTRESS 16% #18 of 73 Official leaderboard
Human Sourced Jailbreaks 100% #2 of 8 Lab model card
PCT (Polarized Contrastive Pairs) 39% #13 of 46 Official leaderboard
World-Class Bio 53.8% #22 of 74 Official leaderboard
StrongREJECT 97% #6 of 19 Lab model card
Human Pathogen Capabilities Test 58.3% #28 of 86 Official leaderboard
Gray Swan Agent Red Teaming (ART) benchmark 2.50% #3 of 8 Official leaderboard
Capture the Flag (Professional) 55% #9 of 21 Lab model card
Molecular Biology Capabilities Test 49.2% #42 of 90 Official leaderboard
SimpleSafetyTests 99% #37 of 75 Official leaderboard
SUPERChem 30.6% #3 of 6 Official leaderboard
Image generation refusals 98% #1 of 2 Lab model card
Challenging Refusal Evaluation 92% #5 of 9 Lab model card
Tacit Knowledge and Troubleshooting 70% #18 of 30 Lab model card
ProtocolQA Open-Ended 30% #18 of 27 Lab model card
Apollo Research sabotage evaluations 0.58 #2 of 3 Lab model card
Anthropic Red Team 98.3% #52 of 75 Official leaderboard
Cyber Range 0% #8 of 10 Lab model card
SCONE-bench $12,000,000 #6 of 7 Official leaderboard

Other

BenchmarkScoreRankSource
CAIS Risk Index 30.2 #3 of 67 Official leaderboard
HELM Capabilities 81.1 #3 of 60 Official leaderboard
ACCoRD 71.4 #44 of 152 Official leaderboard
Conceptual Reasoning Index 53.0 #59 of 151 Official leaderboard
LMCA 39.7% #58 of 147 Official leaderboard
TutorBench 52.8% #12 of 20 Official leaderboard
CAIS Text Capabilities Index 25.6 #44 of 68 Official leaderboard
CAIS Vision Capabilities Index 46.5 #35 of 53 Official leaderboard
SAGE (Vals) 41.8% #56 of 81 Official leaderboard

Its lab's numbers put it at 142.6, 4.4 points below its ECI

The Self-Reported Index fits one score to the 18 capability results OpenAI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 140.9 to 144.4. How we compute it.

Its lab's numbers next to 9 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
SWE-bench Verified 69.1% SWE-Bench Verified (n=477) 58.4% Leaderboard, mini-SWE-agent +11 pts
Humanity's Last Exam 24.9% Humanity's Last Exam, o3 (python + browsing tools) 20.3% Leaderboard, high +4.6 pts
GPQA Diamond 83.3% GPQA Diamond PhD-Level Science Questions, o3 (no tools) 81.8% Epoch AI run, high +1.5 pts
SimpleQA 49% SimpleQA, accuracy (Table 4) 49.4% Leaderboard −0.4 pts
AIME 2025 88.9% AIME 2025 Competition Math, o3 (no tools) 89.2% Leaderboard, high −0.3 pts
MultiChallenge 56.5% Scale MultiChallenge Multi-turn instruction following 56.6% Leaderboard, high Same
Aider Polyglot 81.3% Aider Polyglot Code Editing, o3-high (whole) 81.3% Leaderboard, high, aider, diff edits Same
CharXiv 78.6% CharXiv-Reasoning Scientific Figure Reasoning 78.6% Leaderboard, high Same
MMMU 82.9% MMMU College-level visual problem-solving 82.9% Leaderboard Same

OpenAI's models over time

OpenAI's models in color, against every other model on the Epoch Capabilities Index, by release date.

50 100 150 2023 2024 2025 2026 2027 GPT-6.1 Terra GPT-6 Luna GPT-5.4 Mini GPT-5.2 Pro GPT-5 GPT-5 nano GPT-4.1 nano 50 100 150 2023 2024 2025 2026 2027 GPT-6.1 Terra GPT-6 Luna

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Introducing OpenAI o3 and o4-mini Apr 16, 2025

Its system card

Every result in the longer documents the lab published with it. 33 of them aren't in the launch post.

OpenAI o3 and o4-mini System Card Apr 16, 2025

Capabilities

Hallucination

Safety

Robustness

Bias

Alignment

Bio and chem risk

Cyber risk

AI R&D risk

Autonomy