phi-3-mini 3.8B ranks first on Microsoft RAI benchmark: Harmful Content Summarization (DR-3).
Microsoft Apr 2024 Open weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SIQA | 76.6% | #4 of 24 | Lab model card | |
| ARC-Easy | 94.6% | #5 of 23 | Lab model card | |
| OpenBookQA | 83.2% | #6 of 24 | Lab model card | |
| PIQA | 84.2% | #12 of 37 | Lab model card | |
| MuSR | 42.8% | #35 of 98 | Official leaderboard | |
| TruthfulQA | 65% | #12 of 30 | Lab model card | |
| Vectara Hallucination Leaderboard (HHEM) | 4.00% | #64 of 148 | Official leaderboard | |
| CommonSenseQA | 80.2% | #9 of 19 | Lab model card | |
| ARC-Challenge | 84.9% | #24 of 49 | Lab model card | |
| BIG-Bench Hard | 71.7% | #45 of 73 | Lab model card | |
| MMLU-Pro | 45.7% | #143 of 212 | Official leaderboard | |
| MT-Bench | 8.38 | #17 of 23 | Lab model card | |
| MMLU | 68.8% | #104 of 131 | Lab model card | |
| BoolQ | 77.2% | #23 of 28 | Lab model card | |
| EQ-Bench v2 | 58.1 | #70 of 85 | Official leaderboard | |
| WinoGrande | 70.8% | #39 of 47 | Lab model card | |
| TriviaQA | 64% | #34 of 40 | Lab model card | |
| Global PIQA | 55.2% | #38 of 44 | Official leaderboard | |
| HellaSwag | 76.7% | #46 of 53 | Lab model card | |
| AGIEval | 37.5% | #37 of 40 | Lab model card | |
| MMMLU | 47.3% | #67 of 71 | Lab model card | |
| LMArena (Elo) | 1,128 | #317 of 335 | Official leaderboard | |
| ANLI | 52.8% | #4 of 4 | Lab model card | |
| Microsoft RAI benchmark: Ungroundedness | 0.60 | #4 of 4 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 82.5% | #61 of 92 | Lab model card | |
| ZebraLogic | 11.6% | #34 of 46 | Official leaderboard | |
| MATH | 41.3% | #79 of 99 | Lab model card | |
| GPQA Diamond | 32.8% | #289 of 302 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 59.1% | #28 of 46 | Official leaderboard | |
| MBPP | 70% | #45 of 71 | Lab model card | |
| MHPP | 19.5% | #17 of 23 | Official leaderboard | |
| MBPP+ | 54.2% | #27 of 35 | Official leaderboard | |
| HumanEval | 58.5% | #84 of 101 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AfriMed-QA | 68% | #3 of 4 | Official leaderboard | |
| MedQA | 53.8% | #4 of 4 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Microsoft RAI benchmark: Harmful Content Summarization (DR-3) | 10% | #1 of 5 | Lab model card | |
| Microsoft RAI benchmark: Harmful Content Continuation (DR-3) | 0.70% | #3 of 5 | Lab model card | |
| Microsoft RAI benchmark: Third Party Harm (DR-1) | 24% | #3 of 5 | Lab model card | |
| CyberMetric | 4.80% | #10 of 10 | Official leaderboard | |
| Microsoft RAI benchmark: Jailbreak (DR-1) | 12.3% | #5 of 5 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Open LLM Leaderboard v2 | 27.6 | #31 of 99 | Official leaderboard |
The Self-Reported Index fits one score to the 20 capability results Microsoft reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 108.8 to 114.8. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Microsoft's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Every result in the longer documents the lab published with it. 28 of them aren't in the launch post.