PaLM 2-L

PaLM 2-L ranks first on 2 of the 17 benchmarks with five or more models scored.

Released May 17th, 2023 Family PaLM 2 The lab's model card Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Unknown lab May 2023

PaLM 2-L

Flagship

114.1 #195 of 256

Epoch Capabilities Index

Self-Reported Index 122.1 #334 of 404

Natural Questions 37.5% #6 /20
SQuAD v2 80.5% #4 /8
BIG-Bench Hard 78.1% #40 /73

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
BoolQ	90.9%	#1 of 28	Lab model card
WinoGrande	90.9%	#2 of 47	Lab model card
CommonSenseQA	90.4%	#1 of 19	Lab model card
ARC-Challenge	95.1%	#7 of 49	Lab model card
PIQA	85%	#8 of 37	Lab model card
TriviaQA	86.1%	#9 of 40	Lab model card
ANLI	73.1%	#1 of 4	Lab model card
ARC-Easy	89.7%	#6 of 23	Lab model card
Natural Questions	37.5%	#6 of 20	Lab model card
RACE-M	77%	#1 of 3	Lab model card
MGSM	87%	#22 of 57	Lab model card
DROP	85%	#16 of 41	Lab model card
HellaSwag	86.8%	#21 of 53	Lab model card
OpenBookQA	58.5%	#12 of 24	Lab model card
SQuAD v2	80.5%	#4 of 8	Lab model card
StrategyQA	90.4%	#1 of 2	Lab model card
XCOPA	94.4%	#1 of 2	Lab model card
BIG-Bench Hard	78.1%	#40 of 73	Lab model card
RACE-H	62.3%	#5 of 7	Lab model card
CB	87.5%	#1 of 1	Lab model card
COPA	96%	#1 of 1	Lab model card
LAMBADA	86.9%	#1 of 1	Lab model card
MultiRC	88.2%	#1 of 1	Lab model card
ReCoRD	93.8%	#1 of 1	Lab model card
RTE	79.3%	#1 of 1	Lab model card
StoryCloze	87.4%	#1 of 1	Lab model card
WebQuestions	28.2%	#1 of 1	Lab model card
WiC	66.8%	#1 of 1	Lab model card
Winograd	89.5%	#1 of 1	Lab model card
WSC	86.9%	#1 of 1	Lab model card

Math and science

Benchmark	Score		Rank	Source
GSM8K	91%		#39 of 92	Lab model card
MATH	48.8%		#71 of 99	Lab model card

Its lab's numbers put it at 122.1, 8.0 points above its ECI

The Self-Reported Index fits one score to the 16 capability results the lab reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 118.2 to 125.9. How we compute it.

The lab's models over time

's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

PaLM 2 Technical Report (arXiv 2305.10403v1) May 10, 2023

TriviaQA (EM), 1-shot 86.1
NaturalQuestions (EM), 1-shot 37.5
WebQuestions (EM), 1-shot 28.2
LAMBADA, 1-shot 86.9
HellaSwag, 1-shot 86.8
StoryCloze, 1-shot 87.4
WSC, 1-shot 86.9
WinoGrande (5), instruction-tuned, Table 5 (Table 2 1-shot = 83.0) 90.9
Winograd, 1-shot 89.5
SQuAD v2 (EM), 1-shot 80.5
RACE-H, 1-shot 62.3
RACE-M, 1-shot 77
PIQA, 1-shot 85
ARC-C (4), CoT + SC, Table 5 (Table 2 1-shot = 69.2) 95.1
ARC-E, 1-shot 89.7
OpenBookQA, 1-shot 58.5
BoolQ, 1-shot 90.9
COPA, 1-shot 96
RTE, 1-shot 79.3
WiC, 1-shot 66.8
MultiRC (F1), 1-shot 88.2
ReCoRD, 1-shot 93.8
CB, 1-shot 87.5
Best of ANLI-R1, 1-shot / ANLI-R2, 1-shot / ANLI-R3, 1-shot 73.1
DROP (3) 85
StrategyQA (6), CoT + SC 90.4
CSQA (7), CoT + SC 90.4
XCOPA (4) 94.4
BB Hard (3), CoT 78.1
MATH, CoT + self-consistency (greedy CoT = 34.3) 48.8
GSM8K, CoT + self-consistency (greedy CoT = 80.7) 91
MGSM, CoT + self-consistency (greedy CoT = 72.2) 87