GPT-4o mini: advancing cost-efficient intelligence Jul 18, 2024
GPT-4o mini
GPT-4o mini ranks first on 3 of the 160 benchmarks with five or more models scored.
166
Benchmarks with a score
3
Ranked first, out of five or more models
9
In the top three
OpenAI Jul 2024
GPT-4o mini
Closed weights Small
123.9
#177 of 256
Epoch Capabilities Index
Self-Reported Index 121.9 #335 of 404
- ZeroBench 2.00% #64 /76
- GPQA Diamond 37.7% #191 /229
- SimpleQA Verified 9.06% #48 /56
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| PIQA | 88.7% | #1 of 58 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| HELMET | 55.7% | #1 of 16 | Official leaderboard | |
| WildBench | 57.3 | #3 of 47 | Official leaderboard | |
| AlpacaEval 2.0 | 50.7% | #4 of 47 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 1.70% | #16 of 148 | Official leaderboard | |
| BoolQ | 88.7% | #8 of 61 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| MixEval | 84.2% | #9 of 34 | Official leaderboard | |
| MMLU-CF | 66% | #10 of 37 | Official leaderboard | |
| Arena-Hard-Auto | 74.9% | #14 of 38 | Official leaderboard | |
| Creation-MMBench | -13.56 | #7 of 19 | Official leaderboard | |
| EQ-Bench v2 | 76.9 | #32 of 85 | Official leaderboard | |
| CulturalBench | 48.3% | #6 of 15 | Official leaderboard | |
| MGSM | 87% | #23 of 57 | Lab model card | |
| MMBench-CN v1.1 | 75% | #79 of 178 | Official leaderboard | |
| AraGen | 50.7% | #19 of 42 | Official leaderboard | |
| MMLU | 82% | #64 of 131 | Lab model card | |
| MMLU-Pro | 63.1% | #106 of 212 | Official leaderboard | |
| ThaiExam | 55.7% | #12 of 24 | Official leaderboard | |
| Chinese SimpleQA | 37.8% | #14 of 27 | Official leaderboard | |
| FaithJudge | 18.6% | #20 of 37 | Official leaderboard | |
| Wer wird Millionär? LLM Benchmark | 1267.0 | #47 of 77 | Official leaderboard | |
| RewardBench | 80.1% | #30 of 48 | Official leaderboard | |
| LMArena (Elo) | 1,318 | #212 of 335 | Official leaderboard | |
| DROP | 79.7% | #26 of 41 | Lab model card | |
| ToRR | 43.1 | #9 of 14 | Official leaderboard | |
| ECLeKTic | 12.4% | #23 of 35 | Official leaderboard | |
| LLM-AggreFact | 74% | #7 of 10 | Official leaderboard | |
| WritingBench | 56.4 | #34 of 45 | Official leaderboard | |
| RewardBench 2 | 58% | #33 of 42 | Official leaderboard | |
| FACTS Grounding | 68.4% | #27 of 34 | Official leaderboard | |
| STALE | 8.70% | #8 of 10 | Official leaderboard | |
| IFEval | 78.2% | #49 of 60 | Official leaderboard | |
| Creative Writing v3 | 51.9 | #100 of 120 | Official leaderboard | |
| SimpleQA Verified | 9.06% | #48 of 56 | Official leaderboard | |
| SimpleQA | 9.50% | #14 of 16 | Official leaderboard | |
| Phare | 58.3% | #56 of 62 | Official leaderboard | |
| LongBench v2 | 32.4% | #30 of 33 | Official leaderboard | |
| JudgeBench | 50% | #10 of 11 | Official leaderboard | |
| BullshitBench | 7.27% | #101 of 107 | Official leaderboard | |
| LLM Confabulation Leaderboard | 37.2% | #57 of 58 | Official leaderboard | |
| BullshitBench V2 | 2.00% | #128 of 129 | Official leaderboard | |
| AidanBench | 654.0 | #13 of 13 | Official leaderboard | |
| AMemGym | 31.7% | #11 of 11 | Official leaderboard | |
| Internal API instruction following (OpenAI) | 27.2% | #8 of 8 | Official leaderboard | |
| QuALITY | 71.7% | #4 of 4 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 91.3% | #5 of 70 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| OlympicArena | 34.1% | #6 of 25 | Official leaderboard | |
| ZebraLogic | 20.1% | #20 of 46 | Official leaderboard | |
| MATH | 70.2% | #46 of 99 | Lab model card | |
| MATH Level 5 | 52.6% | #48 of 94 | Epoch AI run | |
| U-MATH | 43.4% | #19 of 30 | Official leaderboard | |
| PutnamBench | 0% | #8 of 12 | Official leaderboard | |
| IneqMath | 2.00% | #24 of 34 | Official leaderboard | |
| ChemBench | 50.3% | #17 of 24 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 6.94% | #171 of 212 | Epoch AI run | |
| GPQA Main | 36.8% | #50 of 60 | Official leaderboard | |
| GPQA Diamond | 37.7% | #191 of 229 | Epoch AI run | |
| DTBench | 54.4% | #153 of 180 | Official leaderboard | |
| MATH-500 | 72.6% | #34 of 38 | Official leaderboard | |
| ARC-AGI-2 | 0% | #90 of 98 | Official leaderboard | |
| FrontierMath Tiers 1-3 v2 | 0.70% | #85 of 86 | Official leaderboard | |
| SimpleBench | 10.7% | #105 of 106 | Official leaderboard | |
| FutureX | 7.52 | #7 of 7 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 83.5% | #7 of 46 | Official leaderboard | |
| BigCodeBench | 51.8% | #14 of 69 | Official leaderboard | |
| MHPP | 42.4% | #7 of 23 | Official leaderboard | |
| HumanEval | 87.2% | #31 of 101 | Lab model card | |
| MBPP+ | 72.2% | #11 of 35 | Official leaderboard | |
| MBPP | 85.4% | #12 of 35 | Official leaderboard | |
| HumanEval-V | 18.2% | #9 of 24 | Official leaderboard | |
| BigCodeBench-Hard | 25.3% | #47 of 96 | Official leaderboard | |
| StructEval | 73.2% | #16 of 28 | Official leaderboard | |
| SWT-bench | 9.80% | #5 of 7 | Official leaderboard | |
| Aider-Edit | 55.6% | #27 of 34 | Official leaderboard | |
| LiveCodeBench v6 | 27.5% | #19 of 21 | Official leaderboard | |
| LiveCodeBench | 39.4% | #20 of 21 | Official leaderboard | |
| LiveCodeBench v5 | 27.7% | #20 of 21 | Official leaderboard | |
| WeirdML | 11.8% | #109 of 112 | Official leaderboard | |
| Aider Polyglot | 3.60% | #46 of 46 | Official leaderboard | |
| BigCodeArena | 878 | #13 of 13 | Official leaderboard | |
| MMTU | 40% | #9 of 9 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| When2Call | 52.9% | #2 of 10 | Official leaderboard | |
| Berkeley Function Calling Leaderboard (BFCL) | 64% | #18 of 67 | Official leaderboard | |
| LLM Colosseum | 1,835 | #1 of 3 | Official leaderboard | |
| AceBench | 72.5% | #7 of 16 | Official leaderboard | |
| VisualAgentBench | 20.5% | #3 of 5 | Official leaderboard | |
| Galileo Agent Leaderboard | 0.83 | #18 of 29 | Official leaderboard | |
| MCP-Bench | 0.56 | #10 of 15 | Official leaderboard | |
| ToolComp | 51.7% | #22 of 32 | Official leaderboard | |
| PinchBench | 64.4% | #34 of 48 | Official leaderboard | |
| BALROG | 17.4% | #24 of 33 | Official leaderboard | |
| GAIA | 26.9% | #47 of 64 | Official leaderboard | |
| SnakeBench | 20 | #139 of 188 | Official leaderboard | |
| Elimination Game | 3 | #43 of 56 | Official leaderboard | |
| CRMArena | 28.3% | #7 of 9 | Official leaderboard | |
| Mystery Game Puzzles | 12% | #67 of 84 | Epoch AI run | |
| LLM Chess | -252 | #99 of 123 | Official leaderboard | |
| LisanBench | 89.7 | #114 of 133 | Official leaderboard | |
| Step Race | 1 | #57 of 59 | Official leaderboard | |
| Chess Puzzles | 3.00% | #91 of 93 | Epoch AI run | |
| VitaBench 2.0 | 9.40% | #17 of 17 | Official leaderboard | |
| Factorio Learning Environment | 26756.0 | #6 of 6 | Official leaderboard | |
| Orak | 11.2 | #6 of 6 | Official leaderboard | |
| WebCanvas | 21.2% | #3 of 3 | Official leaderboard | |
| StableToolBench | 26.4% | #1 of 1 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 74.8% | #19 of 71 | Official leaderboard | |
| FinEval | 66.2% | #2 of 7 | Official leaderboard | |
| LEXam | 42.5% | #18 of 30 | Official leaderboard | |
| MortgageTax | 54.5% | #72 of 87 | Official leaderboard | |
| Vals Finance Agent v1 | 10.8% | #39 of 47 | Official leaderboard | |
| MedHELM | 33.3 | #11 of 13 | Official leaderboard | |
| CorpFin v2 | 45.5% | #104 of 119 | Official leaderboard | |
| TaxEval v2 | 60.5% | #114 of 129 | Official leaderboard | |
| MedXpertQA-MM | 28.1% | #9 of 9 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| VCR (Visual Caption Restoration) | 42% | #7 of 42 | Official leaderboard | |
| MM-Vet | 66.9% | #32 of 164 | Official leaderboard | |
| Vibe-Eval | 56.7% | #1 of 5 | Official leaderboard | |
| CoreCognition | 60.9% | #25 of 119 | Official leaderboard | |
| SGP-Bench | 58.5% | #5 of 22 | Official leaderboard | |
| KRETA | 73.3% | #4 of 17 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 61.4% | #38 of 152 | Official leaderboard | |
| DREAM-1K | 34% | #2 of 8 | Official leaderboard | |
| MMVU | 61.6% | #8 of 28 | Official leaderboard | |
| MEGA-Bench | 43.1% | #7 of 22 | Official leaderboard | |
| TOMATO | 28.8% | #7 of 22 | Official leaderboard | |
| Image2Struct | 70.2 | #6 of 18 | Official leaderboard | |
| MTVQA-Test | 25.5% | #6 of 18 | Official leaderboard | |
| Q-Bench-Video | 52.2% | #1 of 3 | Official leaderboard | |
| OmniBench | 49% | #6 of 16 | Official leaderboard | |
| MMBench-EN v1.1 | 77.1% | #77 of 178 | Official leaderboard | |
| CharXiv | 34.1% | #34 of 77 | Official leaderboard | |
| MMMU | 59.4% | #61 of 136 | Official leaderboard | |
| MME-RealWorld | 36.4% | #5 of 11 | Official leaderboard | |
| Video-MME | 68.9% | #12 of 26 | Official leaderboard | |
| MMBench | 77.6% | #73 of 155 | Official leaderboard | |
| LongVideoBench | 58.8% | #11 of 23 | Official leaderboard | |
| MMStar | 54.8% | #86 of 173 | Official leaderboard | |
| VHELM | 46.5 | #26 of 41 | Official leaderboard | |
| GeoBench | 64% | #31 of 48 | Official leaderboard | |
| MMMU-Pro | 37.6% | #37 of 57 | Official leaderboard | |
| MicroVQA | 46.2% | #13 of 19 | Official leaderboard | |
| 3DSRBench | 39.1% | #3 of 4 | Official leaderboard | |
| ZeroBench | 2.00% | #64 of 76 | Official leaderboard | |
| VPCT | 34% | #23 of 27 | VPCT leaderboard, via Epoch AI | |
| Vision Arena | 1,097 | #116 of 136 | Official leaderboard | |
| MathVista | 56.7% | #61 of 70 | Lab model card | |
| VisFactor | 16.4% | #24 of 26 | Official leaderboard | |
| Video Reality Test | 57.3% | #5 of 5 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| IHEval | 44.3% | #3 of 10 | Official leaderboard | |
| HELM Safety | 93.0 | #32 of 75 | Official leaderboard | |
| Anthropic Red Team | 98.3% | #49 of 75 | Official leaderboard | |
| SimpleSafetyTests | 97.8% | #49 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 56.3% | #57 of 82 | Official leaderboard | |
| SORRY-Bench | 35.7% | #16 of 23 | Official leaderboard | |
| FORTRESS | 48.1% | #56 of 73 | Official leaderboard | |
| AgentDojo | 27.2% | #16 of 20 | Official leaderboard | |
| nocot-bench (NCRI) | 55.7 | #154 of 185 | Official leaderboard | |
| World-Class Bio | 9.50% | #72 of 74 | Official leaderboard | |
| Molecular Biology Capabilities Test | 22.4% | #89 of 90 | Official leaderboard | |
| Human Pathogen Capabilities Test | 12.9% | #86 of 86 | Official leaderboard | |
| Human Pathogen Capabilities Test v2 (HPCT v2) | 13.1% | #71 of 71 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 70.1 | #23 of 71 | Official leaderboard | |
| OpenVLM Leaderboard | 64.1 | #64 of 173 | Official leaderboard | |
| HELM Capabilities | 56.5 | #48 of 60 | Official leaderboard | |
| Conceptual Reasoning Index | 17.5 | #138 of 151 | Official leaderboard | |
| ACCoRD | 32.6 | #143 of 152 | Official leaderboard |
Its lab's numbers put it at 121.9, 2.0 points below its ECI
The Self-Reported Index fits one score to the 8 capability results OpenAI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 119.0 to 124.8. How we compute it.
Its lab's numbers next to 3 leaderboards
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
OpenAI's models over time
OpenAI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.