OpenAI o3-mini Jan 31, 2025
o3-mini
o3-mini ranks first on 9 of the 150 benchmarks with five or more models scored.
161
Benchmarks with a score
9
Ranked first, out of five or more models
26
In the top three
OpenAI Jan 2025
o3-mini
Closed weights Small
140.6
#124 of 256
Epoch Capabilities Index
Self-Reported Index 133.5 #243 of 404
- GPQA Diamond 77% #104 /229
- FrontierMath 12.4% #30 /55
- FrontierMath Tier 4 4.17% #27 /55
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Vectara Hallucination Leaderboard (HHEM) | 0.80% | #2 of 148 | Official leaderboard | |
| MGSM | 92% | #5 of 57 | Lab model card | |
| BBEH micro average | 54.2% | #1 of 11 | Official leaderboard | |
| BIG-Bench Extra Hard | 44.8% | #1 of 11 | Official leaderboard | |
| JudgeBench | 80.9% | #1 of 11 | Official leaderboard | |
| FaithJudge | 11.6% | #6 of 37 | Official leaderboard | |
| Arena-Hard v2.0 | 66.1% | #5 of 23 | Official leaderboard | |
| MT-AIME2024 | 45.3% | #1 of 4 | Official leaderboard | |
| MMLU-Pro | 79.4% | #60 of 212 | Official leaderboard | |
| TableBench | 59.9% | #3 of 10 | Official leaderboard | |
| SEAL Multilingual | 1,079 | #6 of 19 | Official leaderboard | |
| SuperGPQA | 55.2% | #19 of 59 | Official leaderboard | |
| MMLU | 86.9% | #43 of 131 | Lab model card | |
| SnorkelSpatial | 37.9% | #10 of 27 | Official leaderboard | |
| Internal API instruction following (OpenAI) | 50% | #3 of 8 | Official leaderboard | |
| AraGen | 59.8% | #16 of 42 | Official leaderboard | |
| LMArena (Elo) | 1,364 | #167 of 335 | Official leaderboard | |
| Meeseeks Utility Rate | 84.6% | #5 of 10 | Official leaderboard | |
| FACTS Grounding | 77.6% | #18 of 34 | Official leaderboard | |
| MARS-Bench | 42.1% | #8 of 15 | Official leaderboard | |
| Seal-0 | 2.70% | #6 of 11 | Official leaderboard | |
| LLM Confabulation Leaderboard | 17.9% | #33 of 58 | Official leaderboard | |
| WritingBench | 68.0 | #27 of 45 | Official leaderboard | |
| ECLeKTic | 14.5% | #21 of 35 | Official leaderboard | |
| Fiction.LiveBench | 50% | #41 of 57 | Fiction.live leaderboard, via Epoch AI | |
| SimpleQA | 13.8% | #44 of 60 | Lab model card | |
| HLE Calibration | 80.0 | #3 of 4 | Official leaderboard | |
| Global MMLU Lite | 78% | #35 of 43 | Official leaderboard | |
| PersonQA | 21.7% | #8 of 9 | Lab model card | |
| SimpleQA Verified | 15.3% | #86 of 94 | Epoch AI run | |
| HUMAINE | 13 | #49 of 50 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ZebraLogic | 91.7% | #1 of 46 | Official leaderboard | |
| PHYSICS | 59.9% | #1 of 20 | Official leaderboard | |
| MATH Level 5 | 96.5% | #8 of 94 | Epoch AI run | |
| 3SAT-Search | 96.2% | #1 of 11 | Official leaderboard | |
| TSP-Opt | 56.4% | #1 of 11 | Official leaderboard | |
| U-MATH | 82.2% | #3 of 30 | Official leaderboard | |
| GSM8K Platinum Initial 268 | 99.6% | #2 of 19 | Official leaderboard | |
| MATH-500 | 97.9% | #10 of 70 | Lab model card | |
| SnorkelSequences | 71.2% | #5 of 25 | Official leaderboard | |
| IneqMath | 9.50% | #9 of 34 | Official leaderboard | |
| SnorkelGraph | 71% | #7 of 24 | Official leaderboard | |
| AIME 2024 | 87.3% | #26 of 88 | Lab model card | |
| PolyMath | 38.6% | #13 of 38 | Official leaderboard | |
| GSM8K-Platinum | 99.4% | #3 of 8 | Official leaderboard | |
| PhyX | 31.5% | #4 of 10 | Official leaderboard | |
| GPQA Diamond | 77% | #104 of 229 | Epoch AI run | |
| AMO-Bench | 32.3% | #10 of 21 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 76.9% | #101 of 212 | Epoch AI run | |
| Misguided Attention | 46.8% | #22 of 45 | Official leaderboard | |
| AIME 2025 | 86.7% | #26 of 53 | Official leaderboard | |
| FrontierMath Tier 4 | 4.17% | #27 of 55 | Epoch AI run | |
| ForecastBench | 0.13 | #41 of 82 | Official leaderboard | |
| PHYBench | 25% | #14 of 28 | Official leaderboard | |
| FrontierMath | 12.4% | #30 of 55 | Official leaderboard | |
| UGMathBench | 79% | #4 of 7 | Official leaderboard | |
| SciPredict | 19.8% | #7 of 12 | Official leaderboard | |
| HMMT 2025 | 67.5% | #32 of 53 | Official leaderboard | |
| DTBench | 68.8% | #115 of 180 | Official leaderboard | |
| PutnamBench | 0% | #9 of 12 | Official leaderboard | |
| CombiBench | 2.00% | #3 of 4 | Official leaderboard | |
| ARC-AGI-2 | 2.99% | #75 of 98 | Official leaderboard | |
| ARC-AGI | 34.5% | #75 of 95 | Official leaderboard | |
| LiveMathematicianBench | 20.6% | #15 of 17 | Official leaderboard | |
| SimpleBench | 22.8% | #95 of 106 | Official leaderboard | |
| FrontierMath Tiers 1-3 v2 | 18.6% | #78 of 86 | Official leaderboard | |
| General365 | 44.4% | #21 of 23 | Official leaderboard | |
| FrontierMath Tier 4 v2 | 0% | #66 of 68 | Official leaderboard | |
| USAMO 2025 | 2.08% | #10 of 10 | Official leaderboard | |
| Putnam-like | 64.4% | #6 of 6 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench-Hard | 35.5% | #3 of 96 | Official leaderboard | |
| SciCode | 10.8% | #1 of 18 | Official leaderboard | |
| SEAL Coding | 1,137 | #2 of 26 | Official leaderboard | |
| OIBench | 26.8% | #2 of 9 | Official leaderboard | |
| OpenAI Research Engineer Interview: Multiple Choice | 80% | #2 of 9 | Lab model card | |
| Aider Polyglot | 60.4% | #14 of 46 | Official leaderboard | |
| Codeforces rating | 2,130 | #16 of 47 | Lab model card | |
| LiveCodeBench v6 | 67.4% | #8 of 21 | Official leaderboard | |
| LiveCodeBench v5 | 69.3% | #9 of 21 | Official leaderboard | |
| SWE-bench Lite | 32.3% | #7 of 15 | Official leaderboard | |
| WeirdML | 43.8% | #63 of 112 | Official leaderboard | |
| LiveCodeBench | 82.2% | #12 of 21 | Official leaderboard | |
| LiveSQLBench | 23.7% | #21 of 36 | Official leaderboard | |
| AIRS-Bench | 0.39 | #3 of 5 | Official leaderboard | |
| SWE-bench Verified | 42.4% | #37 of 60 | Official leaderboard | |
| Multi-SWE-bench | 5.27% | #7 of 11 | Official leaderboard | |
| PaperBench | 8.50% | #2 of 3 | Official leaderboard | |
| GitTaskBench | 22.2% | #7 of 10 | Official leaderboard | |
| OpenAI Research Engineer Interview: Coding | 92% | #5 of 7 | Lab model card | |
| Spider 2.0 | 19.2% | #17 of 23 | Official leaderboard | |
| Agentic tasks (autonomous replication and adaptation) | 27% | #3 of 4 | Lab model card | |
| MLE-bench | 11% | #8 of 10 | Lab model card | |
| GSO-Bench | 1.30% | #29 of 30 | Official leaderboard | |
| OpenAI PRs | 0% | #12 of 12 | Lab model card | |
| BIRD-Interact | 20.3% | #3 of 3 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Step Race | 4 | #11 of 59 | Official leaderboard | |
| ToolComp | 65.3% | #6 of 32 | Official leaderboard | |
| Berkeley Function Calling Leaderboard (BFCL) | 64.6% | #16 of 67 | Official leaderboard | |
| SnakeBench | 28 | #51 of 188 | Official leaderboard | |
| SnorkelWordle | 90% | #7 of 19 | Official leaderboard | |
| LisanBench | 1366.3 | #50 of 133 | Official leaderboard | |
| LLM Chess | 403 | #48 of 123 | Official leaderboard | |
| Elimination Game | 4 | #25 of 56 | Official leaderboard | |
| AgentBench FC | 40.9% | #6 of 12 | Official leaderboard | |
| Orak | 4.60 | #3 of 6 | Official leaderboard | |
| Galileo Agent Leaderboard | 0.85 | #17 of 29 | Official leaderboard | |
| Chess Puzzles | 15% | #78 of 93 | Epoch AI run | |
| Mystery Game Puzzles | 7.00% | #76 of 84 | Epoch AI run |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedHELM | 57.2 | #4 of 13 | Official leaderboard | |
| CadEval | 54% | #5 of 13 | Official leaderboard | |
| LEXam | 48.1% | #12 of 30 | Official leaderboard | |
| FinQA (Snorkel OpenEnv environment) | 63.8% | #15 of 22 | Official leaderboard | |
| SnorkelFinance | 63.8% | #15 of 22 | Official leaderboard | |
| TaxEval v2 | 69.4% | #90 of 129 | Official leaderboard | |
| SnorkelUnderwrite | 44.3% | #20 of 27 | Official leaderboard | |
| LegalBench | 71.5% | #111 of 138 | Official leaderboard | |
| Vals Finance Agent v1 | 12.7% | #38 of 47 | Official leaderboard | |
| CorpFin v2 | 45.3% | #105 of 119 | Official leaderboard | |
| MultiFinBen | 4.86 | #8 of 9 | Official leaderboard | |
| Finance Reasoning (Snorkel) | 30.4% | #19 of 21 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OCR-Reasoning | 33.3% | #6 of 15 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MakeMeSay | 62% | #2 of 10 | Lab model card | |
| BaxBench | 45.9% | #7 of 34 | Official leaderboard | |
| HELM Safety | 96.2 | #18 of 75 | Official leaderboard | |
| Cybench | 22.5% | #5 of 16 | Official leaderboard | |
| BioLP-bench | 41% | #3 of 9 | Lab model card | |
| AIR-Bench 2024 | 74.9% | #28 of 82 | Official leaderboard | |
| Standard Refusal Evaluation | 100% | #3 of 8 | Lab model card | |
| Production Jailbreaks | 100% | #2 of 5 | Lab model card | |
| AIRTBench | 28.4% | #4 of 9 | Official leaderboard | |
| Capture the Flag (High School) | 61% | #4 of 9 | Lab model card | |
| SimpleSafetyTests | 99% | #35 of 75 | Official leaderboard | |
| Multimodal Troubleshooting Virology | 53% | #13 of 27 | Lab model card | |
| Capture the Flag (Collegiate) | 21% | #7 of 13 | Lab model card | |
| XSTest | 88% | #9 of 16 | Lab model card | |
| ChangeMyView | 82.9% | #3 of 5 | Lab model card | |
| Anthropic Red Team | 98.6% | #46 of 75 | Official leaderboard | |
| Human Sourced Jailbreaks | 97% | #5 of 8 | Lab model card | |
| FORTRESS | 30.1% | #46 of 73 | Official leaderboard | |
| Molecular Biology Capabilities Test | 45.7% | #58 of 90 | Official leaderboard | |
| Capture the Flag (Professional) | 21% | #14 of 21 | Lab model card | |
| Challenging Refusal Evaluation | 90% | #6 of 9 | Lab model card | |
| Human Pathogen Capabilities Test | 45.6% | #58 of 86 | Official leaderboard | |
| Contextual Nuclear Knowledge | 73% | #3 of 4 | Lab model card | |
| MakeMePay | 1.00% | #5 of 6 | Lab model card | |
| StrongREJECT | 73% | #16 of 19 | Lab model card | |
| World-Class Bio | 30.6% | #63 of 74 | Official leaderboard | |
| Gray Swan Agent Red Teaming (ART) benchmark | 4.23% | #7 of 8 | Official leaderboard | |
| ProtocolQA Open-Ended | 23% | #25 of 27 | Lab model card | |
| Tacit Knowledge and Troubleshooting | 58% | #29 of 30 | Lab model card | |
| Jailbreak Augmented Examples | 100% | #5 of 5 | Lab model card | |
| Political Persuasion Parallel Generation | 35.5% | #4 of 4 | Lab model card | |
| Radiological and Nuclear Expert Knowledge | 60% | #4 of 4 | Lab model card | |
| Tacit Knowledge Brainstorm (Open-Ended) | 51% | #2 of 2 | Lab model card | |
| Gray Swan Jailbreak Arena | 3.60% | #1 of 1 | Lab model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LiveBench | 84.6% | #1 of 27 | Lab model card | |
| ACCoRD | 56.5 | #101 of 152 | Official leaderboard | |
| Conceptual Reasoning Index | 32.3 | #112 of 151 | Official leaderboard | |
| LMCA | 19% | #115 of 147 | Official leaderboard | |
| CAIS Text Capabilities Index | 12.7 | #61 of 68 | Official leaderboard | |
| CAIS Risk Index | 67.6 | #63 of 67 | Official leaderboard |
Its lab's numbers put it at 133.5, 7.1 points below its ECI
The Self-Reported Index fits one score to the 9 capability results OpenAI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 129.9 to 137.1. How we compute it.
- MATH-500 97.9
- MGSM 92
- AIME 2024 87.3
- MMLU 86.9
- GPQA Diamond 79.7
- SWE-bench Verified 49.3
- PersonQA 21.7
- SimpleQA 13.8
- FrontierMath 9.2
Its lab's numbers next to 5 leaderboards
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap |
|---|---|---|---|
| SWE-bench Verified | 49.3% Software Engineering (SWE-bench Verified (n=477)) | 42.4% Leaderboard, Agentless Lite | +6.9 pts |
| MATH-500 | 97.9% Math (pass@1) | 91.8% Leaderboard, Vals AI harness | +6.1 pts |
| FrontierMath | 9.20% FrontierMath Pass@1 | 12.4% Leaderboard, high; task version 1.1.0, Epoch | −3.2 pts |
| GPQA Diamond | 79.7% PhD-level Science Questions (GPQA Diamond) | 77% Epoch AI run, high | +2.7 pts |
| SimpleQA | 13.8% SimpleQA | 13.8% Leaderboard, high | Same |
OpenAI's models over time
OpenAI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 37 of them aren't in the launch post.
OpenAI o3-mini System Card Jan 31, 2025
Capabilities
Hallucination
- PersonQA 21.7
Safety
Robustness
Bio and chem risk
- Long-form Biological Risk Questions
- Expert comparisons on biothreat information
- Biotool and Wet Lab Actions
- Multimodal Troubleshooting Virology 53
- BioLP-bench 41
- ProtocolQA Open-Ended 23
- Tacit Knowledge and Troubleshooting 58
- Tacit Knowledge Brainstorm (Open-Ended) 51
- Structured expert probing campaign: radiological and nuclear
- Contextual Nuclear Knowledge 73
- Radiological and Nuclear Expert Knowledge 60
Cyber risk
Autonomy
Persuasion
- ChangeMyView 82.9
- Political Persuasion Parallel Generation 35.5
- MakeMePay 1
- MakeMeSay 62