Learning to Reason with LLMs (Appendix A) Sep 12, 2024
o1-preview
o1-preview ranks first on 11 of the 71 benchmarks with five or more models scored.
79
Benchmarks with a score
11
Ranked first, out of five or more models
24
In the top three
OpenAI Sep 2024
o1-preview
Closed weights Flagship
133.8
#147 of 256
Epoch Capabilities Index
Self-Reported Index 133.6 #242 of 404
- GPQA Diamond 50.3% #155 /229
- KOR-Bench 72.9% #1 /33
- Cybench 10% #12 /16
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Chinese SimpleQA | 67.9% | #1 of 27 | Official leaderboard | |
| MMLU | 90.8% | #6 of 131 | Lab model card | |
| Arena-Hard-Auto | 90.4% | #2 of 38 | Official leaderboard | |
| CFBench | 71.8% | #1 of 19 | Official leaderboard | |
| SEAL Multilingual | 1,111 | #3 of 19 | Official leaderboard | |
| SEAL Precise Instruction Following | 86.6 | #3 of 18 | Official leaderboard | |
| LongBench v2 | 57.7% | #6 of 33 | Official leaderboard | |
| JudgeBench | 75.4% | #2 of 11 | Official leaderboard | |
| LLM Confabulation Leaderboard | 13% | #12 of 58 | Official leaderboard | |
| AidanBench | 1892.0 | #3 of 13 | Official leaderboard | |
| EQ-Bench v2 | 80.5 | #21 of 85 | Official leaderboard | |
| SimpleQA | 42% | #19 of 60 | Lab model card | |
| Vectara Hallucination Leaderboard (HHEM) | 3.30% | #51 of 148 | Official leaderboard | |
| LMArena (Elo) | 1,389 | #144 of 335 | Official leaderboard | |
| SimpleQA Hallucination Rate | 44% | #6 of 12 | Lab model card | |
| Open Ended Questions (hallucination) | 0.78 | #1 of 2 | Lab model card | |
| WritingBench | 63.4 | #30 of 45 | Official leaderboard | |
| SimpleVQA | 3.65% | #13 of 13 | Official leaderboard | |
| BirthdayFacts | 32% | #2 of 2 | Lab model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| KOR-Bench | 72.9% | #1 of 33 | Official leaderboard | |
| GSM8K Platinum Initial 268 | 99.6% | #1 of 19 | Official leaderboard | |
| ZebraLogic | 71.4% | #4 of 46 | Official leaderboard | |
| Omni-MATH | 52.5% | #2 of 9 | Official leaderboard | |
| ChemBench | 64.3% | #6 of 24 | Official leaderboard | |
| MATH Level 5 | 81.7% | #27 of 94 | Epoch AI run | |
| SimpleBench | 41.7% | #69 of 106 | Official leaderboard | |
| GPQA Diamond | 50.3% | #155 of 229 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 31.1% | #144 of 212 | Epoch AI run | |
| MATH-500 | 85.5% | #55 of 70 | Lab model card | |
| AIME 2024 | 56.7% | #72 of 88 | Lab model card | |
| QuantBench | 36% | #3 of 3 | Lab model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 95.5% | #1 of 35 | Official leaderboard | |
| MBPP+ | 80.2% | #1 of 35 | Official leaderboard | |
| HumanEval | 96.3% | #2 of 46 | Official leaderboard | |
| HumanEval+ | 89% | #2 of 46 | Official leaderboard | |
| MHPP | 68.1% | #1 of 23 | Official leaderboard | |
| Aider-Edit | 79.7% | #3 of 34 | Official leaderboard | |
| FullStackBench | 65.6% | #1 of 10 | Official leaderboard | |
| SciCode | 7.70% | #2 of 18 | Official leaderboard | |
| BIRD-CRITIC | 35.5% | #1 of 9 | Official leaderboard | |
| SEAL Coding | 1,123 | #4 of 26 | Official leaderboard | |
| BigCodeBench-Hard | 28.8% | #24 of 96 | Official leaderboard | |
| SWE-bench Verified | 64.6% | #23 of 60 | Official leaderboard | |
| WeirdML | 47.6% | #53 of 112 | Official leaderboard | |
| cognition-golden | 51.8% | #1 of 2 | Official leaderboard | |
| Spider 2.0 | 31.3% | #13 of 23 | Official leaderboard | |
| METR Time Horizon | 23 min | #18 of 31 | Official leaderboard | |
| METR Time Horizon 1.1 | 20 min | #17 of 26 | Official leaderboard | |
| MLE-bench | 22.4% | #9 of 12 | Official leaderboard | |
| OpenAI Research Engineer Interview: Multiple Choice | 76.8% | #8 of 9 | Lab model card | |
| Codeforces rating | 1,258 | #43 of 47 | Lab model card |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ToolComp | 66.4% | #4 of 32 | Official leaderboard | |
| LLM Chess | 46 | #63 of 123 | Official leaderboard | |
| WebWalkerQA | 9.85% | #5 of 5 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| VRBench | 60.1% | #6 of 19 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Situational Awareness Dataset (SAD) | 61.9% | #1 of 17 | Official leaderboard | |
| Challenging Refusal Evaluation | 93.4% | #1 of 9 | Lab model card | |
| Political Persuasion Parallel Generation | 53.8% | #1 of 4 | Lab model card | |
| MakeMeSay | 49% | #3 of 10 | Lab model card | |
| BrowserART | 13% | #2 of 6 | Official leaderboard | |
| SORRY-Bench | 25% | #8 of 23 | Official leaderboard | |
| Jailbreak Augmented Examples | 100% | #2 of 5 | Lab model card | |
| LAB-Bench ProtocolQA | 81% | #8 of 16 | Lab model card | |
| Standard Refusal Evaluation | 99.5% | #4 of 8 | Lab model card | |
| External red team jailbreaks | 44% | #1 of 2 | Lab model card | |
| MakeMePay | 11.6% | #4 of 6 | Lab model card | |
| StrongREJECT | 84% | #13 of 19 | Lab model card | |
| Cybench | 10% | #12 of 16 | Official leaderboard | |
| Human Sourced Jailbreaks | 96% | #6 of 8 | Lab model card | |
| WildChat (toxic conversations) | 97.1% | #3 of 4 | Lab model card | |
| Capture the Flag (High School) | 43% | #7 of 9 | Lab model card | |
| Tacit Knowledge and Troubleshooting | 66.7% | #24 of 30 | Lab model card | |
| ChangeMyView | 81.8% | #4 of 5 | Lab model card | |
| Production Jailbreaks | 99% | #4 of 5 | Lab model card | |
| Capture the Flag (Professional) | 7.30% | #17 of 21 | Lab model card | |
| XSTest | 97.6% | #16 of 16 | Lab model card | |
| Capture the Flag (Collegiate) | 0% | #13 of 13 | Lab model card | |
| LAB-Bench CloningScenarios | 36.4% | #12 of 12 | Lab model card | |
| CoT Deception Monitoring | 0.79% | #2 of 2 | Lab model card |
Its lab's numbers put it at 133.6, in line with its ECI
The Self-Reported Index fits one score to the 5 capability results OpenAI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 129.1 to 138.1. How we compute it.
- MMLU 90.8
- MATH-500 85.5
- GPQA Diamond 78.3
- AIME 2024 56.7
- SimpleQA 42
Its lab's numbers next to 3 leaderboards
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap | What differs |
|---|---|---|---|---|
| GPQA Diamond | 78.3% GPQA Diamond cons@64 | 50.3% Epoch AI run | +28 pts | The lab reports consensus across 64 responses, while Epoch averages scores from individual responses. |
| SWE-bench Verified | 41.4% SWE-bench Verified, pass@1, Agentless 5-try scaffold, final o1-preview model per text (near-final post-mitigation 28.4% on n=477 subset) (4.5.2) | 64.6% Leaderboard, W&B Programmer O1 crosscheck5 | −23 pts | The lab used an Agentless five-try scaffold, while the board reports W&B Programmer O1 crosscheck5. |
| SimpleQA | 42% SimpleQA, accuracy (Table 2) | 42.4% Leaderboard | −0.4 pts |
OpenAI's models over time
OpenAI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 39 of them aren't in the launch post.
OpenAI o1 System Card (o1-preview and o1-mini) Sep 12, 2024
Capabilities
- MMMLU
- QuantBench 36
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
Autonomy
Persuasion
- ChangeMyView 81.8
- Political Persuasion Parallel Generation 53.8
- MakeMePay 11.6
- MakeMeSay 49