openai/simple-evals README (initial results table) Apr 11, 2024
GPT-4 Turbo Apr 2024
GPT-4 Turbo ranks first on 2 of the 80 benchmarks with five or more models scored.
83
Benchmarks with a score
2
Ranked first, out of five or more models
10
In the top three
OpenAI Apr 2024
GPT-4 Turbo
Apr 2024 snapshot Closed weights Flagship
125.8
#171 of 256
Epoch Capabilities Index
Self-Reported Index 126.2 #308 of 404
- GPQA Diamond 46.6% #171 /229
- KOR-Bench 53.5% #7 /33
- When2Call 64.6% #1 /10
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| EQ-Bench v2 | 86.3 | #2 of 85 | Official leaderboard | |
| AlpacaEval 2.0 | 55% | #2 of 47 | Official leaderboard | |
| MixEval | 88.8% | #2 of 34 | Official leaderboard | |
| MMLU-CF | 70.4% | #3 of 37 | Official leaderboard | |
| WildBench | 56.1 | #4 of 47 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 1.70% | #15 of 148 | Official leaderboard | |
| AlignBench | 8.00 | #1 of 7 | Official leaderboard | |
| Arena-Hard-Auto | 82.6% | #6 of 38 | Official leaderboard | |
| CFBench | 67.2% | #3 of 19 | Official leaderboard | |
| MGSM | 89.6% | #13 of 57 | Lab model card | |
| MMBench-CN v1.1 | 79.5% | #49 of 178 | Official leaderboard | |
| Chinese SimpleQA | 49.1% | #8 of 27 | Official leaderboard | |
| MMLU | 86.7% | #44 of 131 | Lab model card | |
| DROP | 86% | #15 of 41 | Lab model card | |
| RepoQA | 76.4% | #9 of 22 | Official leaderboard | |
| ThaiExam | 57.5% | #10 of 24 | Official leaderboard | |
| RewardBench | 84% | #21 of 48 | Official leaderboard | |
| SEAL Precise Instruction Following | 83.2 | #8 of 18 | Official leaderboard | |
| MMLU-Pro | 63.7% | #103 of 212 | Official leaderboard | |
| SimpleQA | 24.2% | #9 of 16 | Official leaderboard | |
| SEAL Multilingual | 1,020 | #11 of 19 | Official leaderboard | |
| Multilingual human evaluation | 3.98 | #3 of 5 | Official leaderboard | |
| LMArena (Elo) | 1,325 | #203 of 335 | Official leaderboard | |
| LLM Confabulation Leaderboard | 28.4% | #52 of 58 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| KOR-Bench | 53.5% | #7 of 33 | Official leaderboard | |
| TheoremQA | 48.4% | #1 of 4 | Official leaderboard | |
| ZebraLogic | 28.4% | #12 of 46 | Official leaderboard | |
| GSM1k | 95.1% | #5 of 15 | Official leaderboard | |
| MATH | 73.4% | #41 of 99 | Lab model card | |
| ForecastBench | 0.14 | #47 of 82 | Official leaderboard | |
| MATH Level 5 | 46.7% | #55 of 94 | Epoch AI run | |
| GPQA Diamond | 46.6% | #171 of 229 | Epoch AI run | |
| DTBench | 61.6% | #137 of 180 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 6.67% | #173 of 212 | Epoch AI run | |
| SimpleBench | 25.1% | #91 of 106 | Official leaderboard | |
| FrontierMath Tiers 1-3 v2 | 0.70% | #84 of 86 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench | 53.2% | #7 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 32.1% | #10 of 96 | Official leaderboard | |
| HumanEval+ | 86.6% | #5 of 46 | Official leaderboard | |
| CRUXEval | 82% | #2 of 11 | Official leaderboard | |
| HumanEval | 88.2% | #24 of 101 | Lab model card | |
| DS-1000 | 53.9% | #1 of 4 | Official leaderboard | |
| MHPP | 43.8% | #6 of 23 | Official leaderboard | |
| SEAL Coding | 1,034 | #12 of 26 | Official leaderboard | |
| SciCode | 1.50% | #10 of 18 | Official leaderboard | |
| METR Time Horizon | 7.8 min | #23 of 31 | Official leaderboard | |
| LiveCodeBench v6 | 28.7% | #18 of 21 | Official leaderboard | |
| METR Time Horizon 1.1 | 3.7 min | #23 of 26 | Official leaderboard | |
| LiveCodeBench | 42% | #19 of 21 | Official leaderboard | |
| LiveCodeBench v5 | 28.6% | #19 of 21 | Official leaderboard | |
| WeirdML | 18% | #104 of 112 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| When2Call | 64.6% | #1 of 10 | Official leaderboard | |
| ComplexFuncBench | 49.5% | #3 of 11 | Official leaderboard | |
| ToolComp | 60.8% | #14 of 32 | Official leaderboard | |
| GroundUI | 12.3% | #5 of 11 | Official leaderboard | |
| LisanBench | 300.0 | #80 of 133 | Official leaderboard | |
| LLM Chess | -104 | #76 of 123 | Official leaderboard | |
| CRAB | 33.4% | #2 of 3 | Official leaderboard | |
| GAIA | 24.9% | #48 of 64 | Official leaderboard | |
| AppWorld Test-Challenge | 17.5% | #11 of 12 | Official leaderboard | |
| AppWorld Test-Normal | 32.7% | #12 of 12 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 78.3% | #9 of 71 | Official leaderboard | |
| LegalBench | 80.5% | #82 of 138 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SGP-Bench | 60.9% | #4 of 22 | Official leaderboard | |
| MMBench-EN v1.1 | 80% | #53 of 178 | Official leaderboard | |
| LongVideoBench | 60.7% | #7 of 23 | Official leaderboard | |
| MathVista | 58.1% | #21 of 43 | Official leaderboard | |
| VHELM | 51.5 | #22 of 41 | Official leaderboard | |
| MATH-Vision (MathVision) | 30.3% | #74 of 105 | Official leaderboard | |
| Vision Arena | 1,112 | #115 of 136 | Official leaderboard | |
| OmniBench | 33.4% | #15 of 16 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Anthropic Red Team | 99.7% | #10 of 75 | Official leaderboard | |
| OR-Bench | 12.8% | #6 of 28 | Official leaderboard | |
| HELM Safety | 96.1 | #19 of 75 | Official leaderboard | |
| SORRY-Bench | 25.4% | #9 of 23 | Official leaderboard | |
| AIR-Bench 2024 | 71.9% | #33 of 82 | Official leaderboard | |
| SimpleSafetyTests | 99% | #32 of 75 | Official leaderboard | |
| SEAL Adversarial Robustness | 20.0 | #6 of 8 | Official leaderboard | |
| BrowserART | 67% | #5 of 6 | Official leaderboard | |
| AgentDojo | 28.6% | #17 of 20 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 86.4 | #7 of 71 | Official leaderboard | |
| ACCoRD | 42.2 | #129 of 152 | Official leaderboard | |
| Conceptual Reasoning Index | 21.5 | #133 of 151 | Official leaderboard |
Its lab's numbers put it at 126.2, in line with its ECI
The Self-Reported Index fits one score to the 6 capability results OpenAI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 123.1 to 129.2. How we compute it.
Its lab's numbers next to 2 leaderboards
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
OpenAI's models over time
OpenAI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.