Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4) Aug 30, 2024
Capabilities
Hallucination
- POPE 86.1
Safety
- Microsoft internal multimodal RAI benchmark 8.16
- RTVLM 5.44
- VLGuard 9.1
Phi-3.5-vision-instruct
Phi-3.5-vision-instruct.
Released August 16th, 2024 Compare with others
40
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Unknown lab Aug 2024
Phi-3.5-vision-instruct
No Epoch Capabilities Index score yet
- BLINK 57% #23 /29
- HumanEval-V 5.90% #15 /24
- TextVQA 72% #14 /17
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMBench-CN v1.1 | 62.8% | #137 of 178 | Official leaderboard | |
| POPE | 86.1% | #6 of 7 | Lab model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval-V | 5.90% | #15 of 24 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SEED-Bench-2-Plus | 62.2% | #67 of 136 | Official leaderboard | |
| CharXiv | 32.7% | #38 of 77 | Official leaderboard | |
| MMBench | 76% | #77 of 155 | Official leaderboard | |
| MMBench-EN v1.1 | 72.1% | #104 of 178 | Official leaderboard | |
| CharXiv (DQ) | 55% | #43 of 67 | Official leaderboard | |
| MMStar | 47.5% | #113 of 173 | Official leaderboard | |
| MM-Vet | 43.2% | #110 of 164 | Official leaderboard | |
| ChartQA | 81.8% | #25 of 37 | Lab model card | |
| CoreCognition | 52.1% | #81 of 119 | Official leaderboard | |
| VCR (Visual Caption Restoration) | 18.1% | #30 of 42 | Official leaderboard | |
| MEGA-Bench | 25.4% | #16 of 22 | Official leaderboard | |
| MMVU | 38.1% | #21 of 28 | Official leaderboard | |
| BLINK | 57% | #23 of 29 | Lab model card | |
| MMMU | 43% | #108 of 136 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 27.6% | #121 of 152 | Official leaderboard | |
| TemporalBench | 15.5% | #8 of 10 | Official leaderboard | |
| xMMMU | 38.8% | #4 of 5 | Official leaderboard | |
| MMBench-Video | 1.20 | #23 of 28 | Official leaderboard | |
| TextVQA | 72% | #14 of 17 | Lab model card | |
| VRBench | 48.5% | #16 of 19 | Official leaderboard | |
| CrossVid | 21.5% | #11 of 13 | Official leaderboard | |
| TOMATO | 20.7% | #19 of 22 | Official leaderboard | |
| VL-RewardBench | 35.7% | #7 of 8 | Official leaderboard | |
| KRETA | 42.6% | #15 of 17 | Official leaderboard | |
| MMMU-Pro | 19.7% | #52 of 57 | Official leaderboard | |
| AI2D | 78.1% | #48 of 52 | Lab model card | |
| Vision Arena | 919 | #135 of 136 | Official leaderboard | |
| MathVista | 43.9% | #70 of 70 | Lab model card | |
| VISTA | 15.2% | #54 of 54 | Official leaderboard | |
| Video-MME | 50.8% | #51 of 51 | Lab model card | |
| SITE | 21.8% | #6 of 6 | Official leaderboard | |
| Inter-GPS (Geometry3K) | 36.3% | #2 of 2 | Lab model card | |
| ScienceQA | 91.3% | #2 of 2 | Lab model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Microsoft internal multimodal RAI benchmark | 8.16 | #1 of 2 | Lab model card | |
| VLGuard | 9.10 | #1 of 2 | Lab model card | |
| RTVLM | 5.44 | #2 of 2 | Lab model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenVLM Leaderboard | 53.0 | #117 of 173 | Official leaderboard |
Its lab's numbers next to 3 leaderboards
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Its system card
Every result in the longer documents the lab published with it. 14 of them aren't in the launch post.