Llama 2-13B

Llama 2-13B places in the top three on 2 benchmarks.

Meta Released July 18th, 2023 Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Meta Jul 2023

Llama 2-13B

Open weights

96.7 #221 of 256

Epoch Capabilities Index

Self-Reported Index 94.9 #397 of 404

MolecularIQ 3.06% #16 /17
MMLU-CF 44.8% #34 /37
SIQA 50.3% #17 /24

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
LAMBADA	76.5%	#11 of 26	Claim in Qwen Technical Report, via Epoch AI
BoolQ	81.7%	#13 of 28	Lab model card
TriviaQA	79.6%	#20 of 40	Lab model card
Natural Questions	31.2%	#11 of 20	Lab model card
OpenBookQA	57%	#14 of 24	Lab model card
L-Eval	45.7%	#3 of 5	Official leaderboard
CriticEval	4.92	#8 of 12	Official leaderboard
SIQA	50.3%	#17 of 24	Lab model card
HellaSwag	80.7%	#39 of 53	Lab model card
WinoGrande	72.8%	#35 of 47	Lab model card
PIQA	80.5%	#29 of 37	Lab model card
ARC-Easy	77.3%	#19 of 23	Lab model card
AlpacaEval 2.0	8.44%	#41 of 47	Official leaderboard
SQuAD v2	72.9%	#7 of 8	Lab model card
Vectara Hallucination Leaderboard (HHEM)	10.5%	#131 of 148	Official leaderboard
MMLU-Pro	25.3%	#189 of 212	Official leaderboard
CommonSenseQA	67.3%	#17 of 19	Lab model card
AGIEval	39.1%	#36 of 40	Lab model card
TruthfulQA	41.9%	#27 of 30	Lab model card
BIG-Bench Hard	39.4%	#67 of 73	Lab model card
ARC-Challenge	49.4%	#45 of 49	Lab model card
MMLU-CF	44.8%	#34 of 37	Official leaderboard
LMArena (Elo)	1,142	#311 of 335	Official leaderboard
EQ-Bench v2	49.1	#79 of 85	Official leaderboard
MMLU	54.8%	#123 of 131	Lab model card
LLMBoxing	14029.0	#2 of 2	Official leaderboard

Math and science

Benchmark	Score	Rank	Source
GSM8K	28.7%	#89 of 92	Lab model card
MATH	3.90%	#98 of 99	Lab model card
DTBench	42.2%	#179 of 180	Official leaderboard

Coding

Benchmark	Score		Rank	Source
MBPP	30.6%		#69 of 71	Lab model card
HumanEval	18.3%		#99 of 101	Lab model card

Agents and tools

Benchmark	Score	Rank	Source
WebLINX	5.16	#2 of 3	Official leaderboard
T-Eval	37.3%	#11 of 12	Official leaderboard
AgentBoard	18.9%	#4 of 4	Official leaderboard

Professional work

Benchmark	Score		Rank	Source
MedQA	39.2%		#67 of 71	Official leaderboard
LegalBench	53.7%		#134 of 138	Official leaderboard

Vision, audio and robotics

Benchmark	Score		Rank	Source
ScienceQA	55.8%		#1 of 4	Official leaderboard

Safety and security

Benchmark	Score	Rank	Source
TrustLLM JailbreakTrigger	95.9%	#2 of 4	Official leaderboard
CyberSecEval MITRE	40.5%	#1 of 2	Official leaderboard
OR-Bench	91%	#23 of 28	Official leaderboard
Situational Awareness Dataset (SAD)	33.5%	#15 of 17	Official leaderboard
MolecularIQ	3.06%	#16 of 17	Official leaderboard
SuperCLUE-Safety	77.5	#7 of 7	Official leaderboard

Other

Benchmark	Score		Rank	Source
HELM Classic	82.3		#3 of 17	Official leaderboard
HELM Lite	23.3		#66 of 71	Official leaderboard

Its lab's numbers put it at 94.9, 1.7 points below its ECI

The Self-Reported Index fits one score to the 19 capability results Meta reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 91.8 to 98.1. How we compute it.

BoolQ 81.7
HellaSwag 80.7
PIQA 80.5
TriviaQA 79.6
ARC-Easy 77.3
SQuAD v2 72.9
WinoGrande 72.8
CommonSenseQA 67.3
OpenBookQA 57
MMLU 54.8
SIQA 50.3
ARC-Challenge 49.4
BIG-Bench Hard 39.4
AGIEval 39.1
Natural Questions 31.2
MBPP 30.6
GSM8K 28.7
HumanEval 18.3
MATH 3.9

Meta's models over time

Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its system cards

Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.

Llama 2 model card (MODEL_CARD.md, meta-llama/llama) Jul 18, 2023