- Arena Hard 37.9
- BigBench Hard CoT (0-shot) 79.1
- MMLU (5-shot) 78.9
- MMLU-Pro (0-shot, CoT) 54.3
- ARC Challenge (10-shot) 91
- BoolQ (2-shot) 84.6
- GPQA (0-shot, CoT) 36.8
- HellaSwag (5-shot) 83.8
- OpenBookQA (10-shot) 89.6
- PIQA (5-shot) 88.6
- Social IQA (5-shot) 78
- TruthfulQA (MC2) (10-shot) 77.5
- WinoGrande (5-shot) 81.3
- Multilingual MMLU (5-shot) 69.9
- MGSM (0-shot CoT) 58.7
- GSM8K (8-shot, CoT) 88.7
- MATH (0-shot, CoT) 59.5
- Qasper 40
- SQuALITY 24.1
- HumanEval (0-shot) 70.7
- MBPP (3-shot) 80.8
Phi-3.5-MoE
Phi-3.5-MoE ranks first on 2 of the 31 benchmarks with five or more models scored.
35
Benchmarks with a score
2
Ranked first, out of five or more models
7
In the top three
Microsoft Aug 2024 Phi-3.5-MoE
Open weights Small
118.8
#352 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
- KOR-Bench 43.9% #17 /33
- MMLU-CF 64.6% #13 /37
- SIQA 78% #3 /24
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| PIQA | 88.6% | #1 of 37 | Lab model card | |
| TruthfulQA | 77.5% | #2 of 30 | Lab model card | |
| MuSR | 45.6% | #10 of 98 | Official leaderboard | |
| OpenBookQA | 89.6% | #3 of 24 | Lab model card | |
| SIQA | 78% | #3 of 24 | Lab model card | |
| Vectara Hallucination Leaderboard (HHEM) | 2.50% | #34 of 148 | Official leaderboard | |
| BoolQ | 84.6% | #7 of 28 | Lab model card | |
| MMLU-CF | 64.6% | #13 of 37 | Official leaderboard | |
| WinoGrande | 81.3% | #17 of 47 | Lab model card | |
| EQ-Bench v2 | 77.0 | #31 of 85 | Official leaderboard | |
| ARC-Challenge | 91% | #19 of 49 | Lab model card | |
| Microsoft RAI benchmark: Ungroundedness | 0.23 | #2 of 4 | Lab model card | |
| BIG-Bench Hard | 79.1% | #38 of 73 | Lab model card | |
| RULER | 87.1% | #25 of 42 | Lab model card | |
| MMLU | 78.9% | #79 of 131 | Lab model card | |
| HellaSwag | 83.8% | #32 of 53 | Lab model card | |
| SQuALITY | 24.1% | #2 of 3 | Lab model card | |
| RepoQA | 85% | #3 of 4 | Lab model card | |
| MMMLU | 69.9% | #57 of 71 | Lab model card | |
| MGSM | 58.7% | #51 of 57 | Lab model card | |
| MMLU-Pro | 54.3% | #151 of 167 | Lab model card | |
| Arena-Hard-Auto | 37.9% | #37 of 39 | Lab model card | |
| Qasper | 40% | #3 of 3 | Lab model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 88.7% | #47 of 92 | Lab model card | |
| KOR-Bench | 43.9% | #17 of 33 | Official leaderboard | |
| MATH | 59.5% | #60 of 99 | Lab model card | |
| GPQA Diamond | 36.8% | #280 of 302 | Lab model card |
Coding
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BRIDGE | 36.6 | #33 of 39 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Microsoft RAI benchmark: Third Party Harm (DR-1) | 10.5% | #1 of 5 | Lab model card | |
| Microsoft RAI benchmark: Harmful Content Continuation (DR-3) | 0.50% | #2 of 5 | Lab model card | |
| Microsoft RAI benchmark: Jailbreak (DR-1) | 10.6% | #2 of 5 | Lab model card | |
| Microsoft RAI benchmark: Harmful Content Summarization (DR-3) | 12% | #5 of 5 | Lab model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Open LLM Leaderboard v2 | 36.9 | #10 of 99 | Official leaderboard |
Its lab's numbers put it at 118.8, number 352 of 404
The Self-Reported Index fits one score to the 20 capability results Microsoft reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 115.7 to 121.8. How we compute it.
- ARC-Challenge 91
- OpenBookQA 89.6
- GSM8K 88.7
- PIQA 88.6
- RULER 87.1
- BoolQ 84.6
- HellaSwag 83.8
- WinoGrande 81.3
- MBPP 80.8
- BIG-Bench Hard 79.1
- MMLU 78.9
- SIQA 78
- TruthfulQA 77.5
- HumanEval 70.7
- MMMLU 69.9
- MATH 59.5
- MGSM 58.7
- MMLU-Pro 54.3
- Arena-Hard-Auto 37.9
- GPQA Diamond 36.8
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 7 of them aren't in the launch post.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4) Aug 30, 2024
Capabilities
- Arena-Hard-Auto 37.9
- BIG-Bench Hard 79.1
- MMLU 78.9
- MMLU-Pro 54.3
- ARC-Challenge 91
- BoolQ 84.6
- GPQA Diamond 36.8
- HellaSwag 83.8
- OpenBookQA 89.6
- PIQA 88.6
- SIQA 78
- WinoGrande 81.3
- MMMLU 69.9
- MGSM 58.7
- GSM8K 88.7
- MATH 59.5
- Qasper 40
- SQuALITY 24.1
- HumanEval 70.7
- MBPP 80.8
- RepoQA 85
- RULER 87.1