Llama 2-7B

Llama 2-7B places in the top three on DecodingTrust.

Meta Released July 18th, 2023 Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Meta Jul 2023

Llama 2-7B

Open weights

83.4 #237 of 256

Epoch Capabilities Index

Self-Reported Index 89.9 #401 of 404

MMLU-CF 41.9% #35 /37
SIQA 48.3% #23 /24
Natural Questions 25.7% #15 /20

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
OpenBookQA	58.6%	#11 of 24	Lab model card
LAMBADA	73.3%	#17 of 26	Claim in Qwen Technical Report, via Epoch AI
TriviaQA	72.1%	#28 of 40	Lab model card
Natural Questions	25.7%	#15 of 20	Lab model card
BoolQ	77.4%	#22 of 28	Lab model card
HellaSwag	77.2%	#43 of 53	Lab model card
CriticEval	4.30	#10 of 12	Official leaderboard
WinoGrande	69.2%	#40 of 47	Lab model card
PIQA	78.8%	#32 of 37	Lab model card
ARC-Easy	75.2%	#20 of 23	Lab model card
Vectara Hallucination Leaderboard (HHEM)	11.3%	#133 of 148	Official leaderboard
MMLU-Pro	20.3%	#197 of 212	Official leaderboard
TruthfulQA	33.3%	#28 of 30	Lab model card
AlpacaEval 2.0	5.35%	#44 of 47	Official leaderboard
ARC-Challenge	45.9%	#46 of 49	Lab model card
EQ-Bench v2	36.3	#80 of 85	Official leaderboard
MMLU-CF	41.9%	#35 of 37	Official leaderboard
WildBench	15.2	#45 of 47	Official leaderboard
SIQA	48.3%	#23 of 24	Lab model card
LMArena (Elo)	1,108	#324 of 335	Official leaderboard
MMLU	45.3%	#127 of 131	Lab model card
BIG-Bench Hard	32.6%	#71 of 73	Lab model card
AGIEval	29.3%	#39 of 40	Lab model card
CommonSenseQA	57.8%	#19 of 19	Lab model card
SQuAD v2	67.2%	#8 of 8	Lab model card
L-Eval	36.8%	#5 of 5	Official leaderboard

Math and science

Benchmark	Score	Rank	Source
MATH	2.50%	#99 of 99	Lab model card
GSM8K	14.6%	#92 of 92	Lab model card
SciBench	1.03%	#2 of 2	Official leaderboard

Coding

Benchmark	Score		Rank	Source
HumanEval	12.8%		#101 of 101	Lab model card
MBPP	20.8%		#71 of 71	Lab model card

Agents and tools

Benchmark	Score		Rank	Source
T-Eval	27.4%		#12 of 12	Official leaderboard
WebLINX	4.04		#3 of 3	Official leaderboard

Professional work

Benchmark	Score	Rank	Source
CFLUE Knowledge	18.8%	#7 of 8	Official leaderboard
MedQA	39.2%	#68 of 71	Official leaderboard
LegalBench	51.9%	#135 of 138	Official leaderboard

Vision, audio and robotics

Benchmark	Score		Rank	Source
ScienceQA	43.1%		#3 of 4	Official leaderboard

Safety and security

Benchmark	Score	Rank	Source
DecodingTrust	74.7	#2 of 5	Official leaderboard
OR-Bench	87.5%	#21 of 28	Official leaderboard
TrustLLM JailbreakTrigger	94.5%	#3 of 4	Official leaderboard
Situational Awareness Dataset (SAD)	32.9%	#16 of 17	Official leaderboard

Other

Benchmark	Score		Rank	Source
HELM Classic	60.7		#9 of 17	Official leaderboard
HELM Lite	15.2		#70 of 71	Official leaderboard

Its lab's numbers put it at 89.9, 6.5 points above its ECI

The Self-Reported Index fits one score to the 19 capability results Meta reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 86.5 to 93.3. How we compute it.

PIQA 78.8
BoolQ 77.4
HellaSwag 77.2
ARC-Easy 75.2
TriviaQA 72.1
WinoGrande 69.2
SQuAD v2 67.2
OpenBookQA 58.6
CommonSenseQA 57.8
SIQA 48.3
ARC-Challenge 45.9
MMLU 45.3
BIG-Bench Hard 32.6
AGIEval 29.3
Natural Questions 25.7
MBPP 20.8
GSM8K 14.6
HumanEval 12.8
MATH 2.5

Meta's models over time

Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its system cards

Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.

Llama 2 model card (MODEL_CARD.md, meta-llama/llama) Jul 18, 2023