- MMLU (5-Shot) 78.2
- HellaSwag (5-Shot) 83
- ANLI (7-Shot) 58.7
- GSM-8K (0-Shot; CoT) 90.3
- MedQA (2-Shot) 69.4
- AGIEval (0-Shot) 48.4
- TriviaQA (5-Shot) 75.6
- Arc-C (10-Shot) 91
- Arc-E (10-Shot) 97.8
- PIQA (5-Shot) 87.7
- SociQA (5-Shot) 80.2
- BigBench-Hard (0-Shot) 81.3
- WinoGrande (5-Shot) 81.4
- OpenBookQA (10-Shot) 87.2
- BoolQ (0-Shot) 86.6
- CommonSenseQA (10-Shot) 82.6
- TruthfulQA (10-Shot) 75.7
- HumanEval (0-Shot) 55.5
- MBPP (3-Shot) 74.5
- GPQA (2-Shot; CoT)
- MT Bench (2 round ave.) 8.91
phi-3-medium 14B
phi-3-medium 14B ranks first on SIQA.
41
Benchmarks with a score
1
Ranked first, out of five or more models
5
In the top three
Microsoft Apr 2024 phi-3-medium 14B
Open weights Small
114.3
#193 of 256
Epoch Capabilities Index
Self-Reported Index 120.3 #345 of 404
- GPQA Diamond 27.6% #213 /229
- Open Arabic LLM Leaderboard v1 46.3% #21 /55
- ChemBench 47.5% #19 /24
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SIQA | 80.2% | #1 of 24 | Lab model card | |
| PIQA | 87.9% | #2 of 37 | Lab model card | |
| TruthfulQA | 75.7% | #3 of 30 | Lab model card | |
| BoolQ | 86.6% | #3 of 28 | Lab model card | |
| ARC-Easy | 97.8% | #3 of 23 | Lab model card | |
| OpenBookQA | 87.4% | #5 of 24 | Lab model card | |
| Microsoft RAI benchmark: Ungroundedness | 0.21 | #1 of 4 | Lab model card | |
| WinoGrande | 81.5% | #14 of 47 | Lab model card | |
| ARC-Challenge | 91.6% | #16 of 49 | Lab model card | |
| MT-Bench | 8.91 | #8 of 23 | Lab model card | |
| CommonSenseQA | 82.8% | #7 of 19 | Lab model card | |
| Open Arabic LLM Leaderboard v1 | 46.3% | #21 of 55 | Official leaderboard | |
| MuSR | 42.6% | #38 of 98 | Official leaderboard | |
| EQ-Bench v2 | 76.3 | #35 of 85 | Official leaderboard | |
| Ko-IFEval (Upstage) | 38% | #9 of 21 | Official leaderboard | |
| BIG-Bench Hard | 81.4% | #35 of 73 | Lab model card | |
| ANLI | 58.7% | #2 of 4 | Lab model card | |
| MMLU-Pro | 55.7% | #124 of 212 | Official leaderboard | |
| Arena-Hard-Auto | 33.4% | #23 of 38 | Official leaderboard | |
| HellaSwag | 83% | #34 of 53 | Lab model card | |
| MMLU | 78.2% | #85 of 131 | Lab model card | |
| AGIEval | 50.2% | #26 of 40 | Lab model card | |
| TriviaQA | 75.6% | #26 of 40 | Lab model card | |
| Global PIQA | 59.5% | #32 of 44 | Official leaderboard | |
| LMArena (Elo) | 1,198 | #285 of 335 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 91% | #40 of 92 | Lab model card | |
| MATH | 53.1% | #65 of 99 | Lab model card | |
| ChemBench | 47.5% | #19 of 24 | Official leaderboard | |
| MATH Level 5 | 17.6% | #79 of 94 | Epoch AI run | |
| GPQA Diamond | 27.6% | #213 of 229 | Epoch AI run |
Coding
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 69.6% | #26 of 71 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Microsoft RAI benchmark: Harmful Content Continuation (DR-3) | 1.00% | #4 of 5 | Lab model card | |
| Microsoft RAI benchmark: Harmful Content Summarization (DR-3) | 11.2% | #4 of 5 | Lab model card | |
| Microsoft RAI benchmark: Jailbreak (DR-1) | 11.1% | #4 of 5 | Lab model card | |
| Microsoft RAI benchmark: Third Party Harm (DR-1) | 25.1% | #4 of 5 | Lab model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Open LLM Leaderboard v2 | 33.1 | #17 of 99 | Official leaderboard | |
| HELM Lite | 50.9 | #38 of 71 | Official leaderboard |
Its lab's numbers put it at 120.3, 5.9 points above its ECI
The Self-Reported Index fits one score to the 18 capability results Microsoft reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 116.7 to 123.9. How we compute it.
- ARC-Easy 97.8
- ARC-Challenge 91.6
- GSM8K 91
- PIQA 87.9
- OpenBookQA 87.4
- BoolQ 86.6
- HellaSwag 83
- CommonSenseQA 82.8
- WinoGrande 81.5
- BIG-Bench Hard 81.4
- SIQA 80.2
- MMLU 78.2
- TruthfulQA 75.7
- TriviaQA 75.6
- MBPP 75.2
- HumanEval 62.2
- MATH 53.1
- AGIEval 50.2
Its lab's number next to a leaderboard
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Microsoft's models over time
Microsoft's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 6 of them aren't in the launch post.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4) Aug 30, 2024
Capabilities
- MMLU 78
- HellaSwag 82.4
- ANLI 55.8
- GSM8K 91
- MATH 53.1
- MedQA 69.9
- AGIEval 50.2
- TriviaQA 73.9
- ARC-Challenge 91.6
- ARC-Easy 97.7
- PIQA 87.9
- SIQA 80.2
- BIG-Bench Hard 81.4
- WinoGrande 81.5
- OpenBookQA 87.4
- BoolQ 86.5
- CommonSenseQA 82.8
- HumanEval 62.2
- MBPP 75.2
- MT-Bench 8.91