Llama 2-70B
Llama 2-70B ranks first on HELM Classic.
50
Benchmarks with a score
1
Ranked first, out of five or more models
3
In the top three
Meta Jul 2023 Llama 2-70B
Open weights Open model
105.8
#209 of 256
Epoch Capabilities Index
Self-Reported Index 103.1 #388 of 404
- GPQA Diamond 26.3% #218 /229
- MMLU-CF 52.2% #29 /37
- SIQA 50.7% #16 /24
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TriviaQA | 87.6% | #5 of 40 | Lab model card | |
| BoolQ | 85% | #4 of 28 | Lab model card | |
| LAMBADA | 78.9% | #4 of 26 | Claim in Qwen Technical Report, via Epoch AI | |
| Natural Questions | 39.5% | #5 of 20 | Lab model card | |
| SQuAD v2 | 80.7% | #3 of 8 | Lab model card | |
| OpenBookQA | 60.2% | #10 of 24 | Lab model card | |
| WinoGrande | 80.2% | #20 of 47 | Lab model card | |
| EQ-Bench v2 | 73.6 | #42 of 85 | Official leaderboard | |
| LLMBoxing | 22679.0 | #1 of 2 | Official leaderboard | |
| PIQA | 82.8% | #19 of 37 | Lab model card | |
| HellaSwag | 85.3% | #28 of 53 | Lab model card | |
| AGIEval | 54.2% | #23 of 40 | Lab model card | |
| CommonSenseQA | 78.5% | #11 of 19 | Lab model card | |
| CriticEval | 5.00 | #7 of 12 | Official leaderboard | |
| SIQA | 50.7% | #16 of 24 | Lab model card | |
| ARC-Easy | 80.2% | #16 of 23 | Lab model card | |
| Vectara Hallucination Leaderboard (HHEM) | 5.90% | #105 of 148 | Official leaderboard | |
| AlpacaEval 2.0 | 14.7% | #34 of 47 | Official leaderboard | |
| TruthfulQA | 50.2% | #22 of 30 | Lab model card | |
| ARC-Challenge | 57.4% | #38 of 49 | Lab model card | |
| MMLU-CF | 52.2% | #29 of 37 | Official leaderboard | |
| MMLU | 68.9% | #103 of 131 | Lab model card | |
| MMLU-Pro | 37.5% | #170 of 212 | Official leaderboard | |
| BIG-Bench Hard | 51.2% | #64 of 73 | Lab model card | |
| LMArena (Elo) | 1,171 | #299 of 335 | Official leaderboard | |
| WildBench | 26.9 | #42 of 47 | Official leaderboard | |
| CommonsenseQA 2.0 | 50% | #6 of 6 | Claim in arxiv.org, via Epoch AI |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 56.8% | #84 of 92 | Lab model card | |
| MATH | 13.5% | #94 of 99 | Lab model card | |
| GPQA Diamond | 26.3% | #218 of 229 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 0% | #212 of 212 | Epoch AI run | |
| DTBench | 41.6% | #180 of 180 | Official leaderboard | |
| MATH Level 5 | 3.29% | #94 of 94 | Epoch AI run | |
| ForecastBench | 0.24 | #82 of 82 | Official leaderboard |
Coding
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| T-Eval | 53% | #7 of 12 | Official leaderboard | |
| VisualWebArena | 1.10% | #5 of 7 | Official leaderboard | |
| AgentBoard | 23.8% | #3 of 4 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 61.8% | #48 of 71 | Official leaderboard | |
| LegalBench | 62.4% | #127 of 138 | Official leaderboard | |
| CFLUE Knowledge | 17.7% | #8 of 8 | Official leaderboard | |
| CFLUE Application | 18.6 | #4 of 4 | Official leaderboard | |
| FinanceBench | 41% | #1 of 1 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TrustLLM JailbreakTrigger | 97.4% | #1 of 4 | Official leaderboard | |
| CyberMetric | 66.1% | #3 of 10 | Official leaderboard | |
| Situational Awareness Dataset (SAD) | 35.2% | #14 of 17 | Official leaderboard | |
| OR-Bench | 96.1% | #26 of 28 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Classic | 94.3 | #1 of 17 | Official leaderboard | |
| HELM Lite | 48.2 | #42 of 71 | Official leaderboard |
Its lab's numbers put it at 103.1, 2.7 points below its ECI
The Self-Reported Index fits one score to the 19 capability results Meta reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 100.0 to 106.2. How we compute it.
- TriviaQA 87.6
- HellaSwag 85.3
- BoolQ 85
- PIQA 82.8
- SQuAD v2 80.7
- WinoGrande 80.2
- ARC-Easy 80.2
- CommonSenseQA 78.5
- MMLU 68.9
- OpenBookQA 60.2
- ARC-Challenge 57.4
- GSM8K 56.8
- AGIEval 54.2
- BIG-Bench Hard 51.2
- SIQA 50.7
- MBPP 45
- Natural Questions 39.5
- HumanEval 29.9
- MATH 13.5
Meta's models over time
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system cards
Every result in the longer documents the lab published with it. 2 of them aren't in the launch post.
Llama 2 model card (MODEL_CARD.md, meta-llama/llama) Jul 18, 2023
Capabilities
- MMLU 68.9
- BIG-Bench Hard 51.2
- AGIEval 54.2
Safety
- TruthfulQA 50.2
- ToxiGen 24.6
Llama 2: Open Foundation and Fine-Tuned Chat Models Jul 19, 2023
Capabilities
- MMLU 68.9
- BIG-Bench Hard 51.2
- AGIEval 54.2
- HumanEval 29.9
- MBPP 45
- BoolQ 85
- PIQA 82.8
- SIQA 50.7
- HellaSwag 85.3
- WinoGrande 80.2
- ARC-Easy 80.2
- ARC-Challenge 57.4
- OpenBookQA 60.2
- CommonSenseQA 78.5
- Natural Questions 39.5
- TriviaQA 87.6
- SQuAD v2 80.7
- QuAC 42.4
- GSM8K 56.8
- MATH 13.5
- Llama 2-Chat helpfulness human evaluation (Meta)