Capabilities
- MMLU 68.4
- GPQA Diamond 34.2
- HumanEval 62.2
- GSM8K 79.6
- MATH 30
- AGIEval 45.9
- CommonSenseQA 72.6
- WinoGrande 76.1
- BIG-Bench Hard 61.1
- ARC-Challenge 78.6
- TriviaQA 78.5
- SQuAD 76.4
- QuAC 44.4
- BoolQ 75.7
- DROP 58.4
Llama 3-8B
Llama 3-8B places in the top three on ANLI.
55
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Meta Apr 2024 Llama 3-8B
Open weights
109.4
#205 of 256
Epoch Capabilities Index
Self-Reported Index 108.7 #379 of 404
- GPQA Diamond 26.1% #219 /229
- KOR-Bench 32.8% #25 /33
- StructEval 51.6% #27 /28
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenBookQA | 82.6% | #6 of 41 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| ANLI | 57.3% | #3 of 11 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| AlpacaEval 2.0 | 22.9% | #25 of 47 | Official leaderboard | |
| TriviaQA | 78.5% | #22 of 40 | Lab model card | |
| ARC-Challenge | 78.6% | #28 of 49 | Lab model card | |
| WinoGrande | 76.1% | #27 of 47 | Lab model card | |
| MMLU-CF | 57.3% | #24 of 37 | Official leaderboard | |
| EQ-Bench v2 | 68.9 | #56 of 85 | Official leaderboard | |
| RepoQA | 53.6% | #16 of 22 | Official leaderboard | |
| MixEval | 75% | #25 of 34 | Official leaderboard | |
| CommonSenseQA | 72.6% | #14 of 19 | Lab model card | |
| WildBench | 33.2 | #35 of 47 | Official leaderboard | |
| AGIEval | 45.9% | #30 of 40 | Lab model card | |
| MMLU-Pro | 41% | #160 of 212 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 7.40% | #114 of 148 | Official leaderboard | |
| BIG-Bench Hard | 61.1% | #58 of 73 | Lab model card | |
| MMLU | 68.4% | #105 of 131 | Lab model card | |
| LMArena (Elo) | 1,224 | #274 of 335 | Official leaderboard | |
| BoolQ | 75.7% | #24 of 28 | Lab model card | |
| TurkishMMLU | 46.4% | #7 of 8 | Official leaderboard | |
| HELMET | 1.50% | #15 of 16 | Official leaderboard | |
| AraGen | 9.21% | #40 of 42 | Official leaderboard | |
| ThaiExam | 19.8% | #24 of 24 | Official leaderboard | |
| CulturalBench | 21.4% | #15 of 15 | Official leaderboard | |
| SQuAD | 76.4% | #5 of 5 | Lab model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SciBench | 8.45% | #1 of 2 | Official leaderboard | |
| ZebraLogic | 11.9% | #32 of 46 | Official leaderboard | |
| GSM8K | 79.6% | #65 of 92 | Lab model card | |
| KOR-Bench | 32.8% | #25 of 33 | Official leaderboard | |
| MathBench | 36.7% | #16 of 21 | Official leaderboard | |
| MATH | 30% | #87 of 99 | Lab model card | |
| ForecastBench | 0.17 | #73 of 82 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 1.94% | #198 of 212 | Epoch AI run | |
| GPQA Diamond | 26.1% | #219 of 229 | Epoch AI run | |
| MATH Level 5 | 6.13% | #91 of 94 | Epoch AI run | |
| DTBench | 43.9% | #178 of 180 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 56.7% | #29 of 46 | Official leaderboard | |
| MBPP+ | 54.8% | #25 of 35 | Official leaderboard | |
| MBPP | 64.6% | #26 of 35 | Official leaderboard | |
| HumanEval | 62.2% | #79 of 101 | Lab model card | |
| BigCodeBench | 34.4% | #57 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 7.40% | #88 of 96 | Official leaderboard | |
| StructEval | 51.6% | #27 of 28 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LLM-WikiRace | 0% | #15 of 21 | Official leaderboard | |
| GTA | 3.10% | #14 of 15 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 58% | #52 of 71 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SGP-Bench | 42.9% | #17 of 22 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SimpleSafetyTests | 99.3% | #28 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 70.9% | #34 of 82 | Official leaderboard | |
| Anthropic Red Team | 98.8% | #42 of 75 | Official leaderboard | |
| OR-Bench | 69.4% | #18 of 28 | Official leaderboard | |
| HELM Safety | 88.6 | #55 of 75 | Official leaderboard | |
| MolecularIQ | 4.67% | #14 of 17 | Official leaderboard | |
| CTIBench | 61.3% | #2 of 2 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 38.7 | #51 of 71 | Official leaderboard |
Its lab's numbers put it at 108.7, in line with its ECI
The Self-Reported Index fits one score to the 12 capability results Meta reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 106.5 to 110.8. How we compute it.
- GSM8K 79.6
- ARC-Challenge 78.6
- TriviaQA 78.5
- WinoGrande 76.1
- BoolQ 75.7
- CommonSenseQA 72.6
- MMLU 68.4
- HumanEval 62.2
- BIG-Bench Hard 61.1
- AGIEval 45.9
- GPQA Diamond 34.2
- MATH 30
Its lab's numbers next to 2 leaderboards
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Meta's models over time
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its system card
Every result in the longer documents the lab published with it. 15 of them aren't in the launch post.