- MMLU (5-shot) [instruct] 82
- GPQA (0-shot) [instruct] 39.5
- HumanEval (0-shot) [instruct] 81.7
- GSM-8K (8-shot, CoT) [instruct] 93
- MATH (4-shot, CoT) [instruct] 50.4
- AGIEval English (3-5 shot) [pretrained] 63
- CommonSenseQA (7-shot) [pretrained] 83.8
- Winogrande (5-shot) [pretrained] 83.1
- BIG-Bench Hard (3-shot, CoT) [pretrained] 81.3
- ARC-Challenge (25-shot) [pretrained] 93
- TriviaQA-Wiki (5-shot) [pretrained] 89.7
- SQuAD (1-shot) [pretrained] 85.6
- QuAC (1-shot, F1) [pretrained] 51.1
- BoolQ (0-shot) [pretrained] 79
- DROP (3-shot, F1) [pretrained] 79.7
Llama 3-70B
Llama 3-70B places in the top three on 3 benchmarks.
67
Benchmarks with a score
0
Ranked first, out of five or more models
3
In the top three
Meta Apr 2024 Llama 3-70B
Open weights Open model
121.2
#181 of 256
Epoch Capabilities Index
Self-Reported Index 119.9 #347 of 404
- GPQA Diamond 40.6% #185 /229
- KOR-Bench 49.2% #13 /33
- Cybench 5.00% #16 /16
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TriviaQA | 89.7% | #4 of 40 | Lab model card | |
| MMLU-CF | 68.9% | #6 of 37 | Official leaderboard | |
| EQ-Bench v2 | 82.1 | #15 of 85 | Official leaderboard | |
| WildBench | 49.6 | #10 of 47 | Official leaderboard | |
| ARC-Challenge | 93% | #13 of 49 | Lab model card | |
| AlpacaEval 2.0 | 34.4% | #13 of 47 | Official leaderboard | |
| WinoGrande | 83.1% | #13 of 47 | Lab model card | |
| CommonSenseQA | 83.8% | #6 of 19 | Lab model card | |
| RepoQA | 82.2% | #7 of 22 | Official leaderboard | |
| MixEval | 84% | #11 of 34 | Official leaderboard | |
| TurkishMMLU | 67.3% | #3 of 8 | Official leaderboard | |
| SQuAD | 85.6% | #2 of 5 | Lab model card | |
| Vectara Hallucination Leaderboard (HHEM) | 4.10% | #67 of 148 | Official leaderboard | |
| ThaiExam | 56.8% | #11 of 24 | Official leaderboard | |
| AGIEval | 63% | #19 of 40 | Lab model card | |
| MMLU | 82% | #63 of 131 | Lab model card | |
| BIG-Bench Hard | 81.3% | #36 of 73 | Lab model card | |
| AlignBench | 7.42 | #4 of 7 | Official leaderboard | |
| MMLU-Pro | 56.2% | #123 of 212 | Official leaderboard | |
| DROP | 79.7% | #25 of 41 | Lab model card | |
| RewardBench | 76.3% | #31 of 48 | Official leaderboard | |
| ChatRAG Bench | 52.5 | #2 of 3 | Official leaderboard | |
| BoolQ | 79% | #19 of 28 | Lab model card | |
| SEAL Precise Instruction Following | 81.2 | #13 of 18 | Official leaderboard | |
| LMArena (Elo) | 1,277 | #248 of 335 | Official leaderboard | |
| AraGen | 26.1% | #34 of 42 | Official leaderboard | |
| CulturalBench | 30.4% | #14 of 15 | Official leaderboard | |
| HELMET | 1.20% | #16 of 16 | Official leaderboard | |
| QuAC | 51.1% | #1 of 1 | Lab model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 93% | #24 of 92 | Lab model card | |
| OlympicArena | 26.7% | #8 of 25 | Official leaderboard | |
| MathBench | 56.4% | #7 of 21 | Official leaderboard | |
| KOR-Bench | 49.2% | #13 of 33 | Official leaderboard | |
| ZebraLogic | 16.8% | #24 of 46 | Official leaderboard | |
| MATH | 50.4% | #68 of 99 | Lab model card | |
| GSM1k | 90.1% | #11 of 15 | Official leaderboard | |
| GPQA Diamond | 40.6% | #185 of 229 | Epoch AI run | |
| MATH Level 5 | 22.6% | #76 of 94 | Epoch AI run | |
| DTBench | 54.2% | #154 of 180 | Official leaderboard | |
| ForecastBench | 0.16 | #71 of 82 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 4.31% | #185 of 212 | Epoch AI run |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 72% | #17 of 46 | Official leaderboard | |
| BigCodeBench | 49% | #26 of 69 | Official leaderboard | |
| MBPP+ | 69% | #15 of 35 | Official leaderboard | |
| MBPP | 82.3% | #16 of 35 | Official leaderboard | |
| BigCodeBench-Hard | 24.6% | #48 of 96 | Official leaderboard | |
| HumanEval | 81.7% | #51 of 101 | Lab model card | |
| SEAL Coding | 871 | #23 of 26 | Official leaderboard | |
| SciCode | 0% | #17 of 18 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| VisualWebArena | 9.78% | #2 of 7 | Official leaderboard | |
| LLM-WikiRace | 7.00% | #7 of 21 | Official leaderboard | |
| GTA | 8.32% | #11 of 15 | Official leaderboard | |
| Spider2-V | 2.00% | #4 of 5 | Official leaderboard | |
| LLM Chess | -278 | #100 of 123 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 77.7% | #10 of 71 | Official leaderboard |
Vision, audio and robotics
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OR-Bench | 37.7% | #10 of 28 | Official leaderboard | |
| SimpleSafetyTests | 99% | #33 of 75 | Official leaderboard | |
| Situational Awareness Dataset (SAD) | 49.3% | #8 of 17 | Official leaderboard | |
| CTIBench | 65.7% | #1 of 2 | Official leaderboard | |
| AIR-Bench 2024 | 64.6% | #44 of 82 | Official leaderboard | |
| HELM Safety | 89.5 | #52 of 75 | Official leaderboard | |
| AgentDojo | 25.6% | #14 of 20 | Official leaderboard | |
| Anthropic Red Team | 96.7% | #63 of 75 | Official leaderboard | |
| Cybench | 5.00% | #16 of 16 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 79.3 | #16 of 71 | Official leaderboard |
Its lab's numbers put it at 119.9, 1.3 points below its ECI
The Self-Reported Index fits one score to the 13 capability results Meta reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 117.3 to 122.6. How we compute it.
- GSM8K 93
- ARC-Challenge 93
- TriviaQA 89.7
- CommonSenseQA 83.8
- WinoGrande 83.1
- MMLU 82
- HumanEval 81.7
- BIG-Bench Hard 81.3
- DROP 79.7
- BoolQ 79
- AGIEval 63
- MATH 50.4
- GPQA Diamond 39.5
Its lab's numbers next to 2 leaderboards
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Meta's models over time
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it.
Capabilities
- MMLU 82
- GPQA Diamond 39.5
- HumanEval 81.7
- GSM8K 93
- MATH 50.4
- AGIEval 63
- CommonSenseQA 83.8
- WinoGrande 83.1
- BIG-Bench Hard 81.3
- ARC-Challenge 93
- TriviaQA 89.7
- SQuAD 85.6
- QuAC 51.1
- BoolQ 79
- DROP 79.7