Gemini 1.5 Pro May 2024

Gemini 1.5 Pro ranks first on 2 of the 74 benchmarks with five or more models scored.

Google DeepMind Released May 14th, 2024 Family Gemini 1.5 The lab's model card Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Google DeepMind May 2024

Gemini 1.5 Pro

May 2024 snapshot Closed weights Flagship

125.3 #173 of 256

Epoch Capabilities Index

Self-Reported Index 124.7 #315 of 404

GPQA Diamond 45.9% #174 /229
BLINK 61.4% #19 /29
KOR-Bench 43.4% #18 /33

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
MMLU	91.7%	#3 of 131	Lab model card
HellaSwag	93.3%	#4 of 53	Lab model card
BIG-Bench Hard	89.2%	#7 of 73	Lab model card
RepoQA	90.6%	#3 of 22	Official leaderboard
ThaiExam	64.7%	#4 of 24	Official leaderboard
MixEval	84.8%	#6 of 34	Official leaderboard
RewardBench	88.2%	#11 of 48	Official leaderboard
WildBench	47.4	#15 of 47	Official leaderboard
MGSM	87.5%	#19 of 57	Lab model card
WMT23	75.3	#1 of 3	Lab model card
SEAL Multilingual	1,069	#7 of 19	Official leaderboard
MMBench-CN v1.1	73.2%	#85 of 178	Official leaderboard
LMArena (Elo)	1,324	#204 of 335	Official leaderboard
DROP	74.9%	#33 of 41	Lab model card
SEAL Precise Instruction Following	79.4	#18 of 18	Official leaderboard

Math and science

Benchmark	Score	Rank	Source
OlympicArena	35.1%	#3 of 25	Official leaderboard
MATH	77.9%	#31 of 99	Lab model card
GSM8K	90.8%	#41 of 92	Lab model card
PhysicsFinals	63.9%	#1 of 2	Lab model card
KOR-Bench	43.4%	#18 of 33	Official leaderboard
GSM1k	92.3%	#9 of 15	Official leaderboard
MATH Level 5	40.8%	#61 of 94	Epoch AI run
AMC	46.4%	#3 of 4	Lab model card
GPQA Diamond	45.9%	#174 of 229	Epoch AI run
OTIS Mock AIME 2024-2025	6.81%	#172 of 212	Epoch AI run
HiddenMath	20.1%	#10 of 11	Lab model card
Functional MATH	64.6%	#1 of 1	Lab model card

Coding

Benchmark	Score	Rank	Source
BigCodeBench	50.6%	#19 of 69	Official leaderboard
Natural2Code	82.6%	#4 of 9	Lab model card
HumanEval	84.1%	#45 of 101	Lab model card
BigCodeBench-Hard	25.4%	#43 of 96	Official leaderboard
SEAL Coding	994	#15 of 26	Official leaderboard
SciCode	1.50%	#12 of 18	Official leaderboard

Agents and tools

Benchmark	Score		Rank	Source
GroundUI	35.2%		#1 of 11	Official leaderboard
ToolComp	40.4%		#27 of 32	Official leaderboard

Professional work

Benchmark	Score	Rank	Source
MedQA	69.2%	#29 of 71	Official leaderboard
MedXpertQA-MM	34.1%	#4 of 9	Official leaderboard
MedHELM	22.1	#13 of 13	Official leaderboard

Vision, audio and robotics

Benchmark	Score	Rank	Source
Video-MME	81.3%	#2 of 26	Official leaderboard
AI2D	94.4%	#5 of 52	Lab model card
TOMATO	36.1%	#3 of 22	Official leaderboard
LongVideoBench	64.4%	#4 of 23	Official leaderboard
CharXiv	43.3%	#15 of 77	Official leaderboard
VCR (Visual Caption Restoration)	38.1%	#9 of 42	Official leaderboard
A-Bench	75.4%	#28 of 116	Official leaderboard
VQAv2	80.2%	#2 of 8	Lab model card
MMStar	59.1%	#62 of 173	Official leaderboard
MMMU	62.2%	#51 of 136	Official leaderboard
ChartQA	87.2%	#16 of 37	Lab model card
InfographicVQA	81%	#12 of 26	Lab model card
EgoSchema	72.2%	#6 of 12	Lab model card
ActivityNet-QA	57.5%	#4 of 8	Lab model card
VATEX ZH	55.3	#2 of 4	Lab model card
OpenEQA	57.9	#1 of 2	Lab model card
MMBench-EN v1.1	74.6%	#92 of 178	Official leaderboard
TextVQA	78.7%	#9 of 17	Lab model card
MMMU-Pro	43.5%	#31 of 57	Official leaderboard
VHELM	50.4	#23 of 41	Official leaderboard
MMBench	73.9%	#87 of 155	Official leaderboard
DocVQA	93.1%	#22 of 36	Lab model card
BLINK	61.4%	#19 of 29	Lab model card
CCBench (MMBench Chinese Culture)	28.4%	#116 of 152	Official leaderboard
RealWorldQA	70.4%	#30 of 38	Lab model card
MathVista	63.9%	#56 of 70	Lab model card
AHELM	36.2	#8 of 10	Official leaderboard
VATEX	64.6	#4 of 5	Lab model card
Vision Arena	1,120	#111 of 136	Official leaderboard
MATH-Vision (MathVision)	19.2%	#89 of 105	Official leaderboard
LVBench	33.1%	#8 of 9	Official leaderboard
V*	71.7%	#17 of 17	Lab model card
DUDE	46%	#4 of 4	Lab model card
YouCook2	106.5	#4 of 4	Lab model card
BetterChartQA	65.8%	#3 of 3	Lab model card
ChemicalDiagramQA	69.7%	#1 of 1	Lab model card
TAT-DQA	37.8%	#1 of 1	Lab model card

Safety and security

Benchmark	Score	Rank	Source
Anthropic Red Team	99.9%	#3 of 75	Official leaderboard
SEAL Adversarial Robustness	8.00	#1 of 8	Official leaderboard
AIR-Bench 2024	82.8%	#17 of 82	Official leaderboard
HELM Safety	92.5	#36 of 75	Official leaderboard
SimpleSafetyTests	97.5%	#52 of 75	Official leaderboard
OR-Bench	88%	#22 of 28	Official leaderboard
Cybench	7.50%	#14 of 16	Official leaderboard
AgentDojo	28.6%	#18 of 20	Official leaderboard

Other

Benchmark	Score		Rank	Source
HELM Lite	73.9		#20 of 71	Official leaderboard
OpenVLM Leaderboard	64.5		#59 of 173	Official leaderboard

Its lab's numbers put it at 124.7, in line with its ECI

The Self-Reported Index fits one score to the 23 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 121.8 to 127.5. How we compute it.

AI2D 94.4
HellaSwag 93.3
DocVQA 93.1
MMLU 91.7
GSM8K 90.8
BIG-Bench Hard 89.2
MGSM 87.5
ChartQA 87.2
HumanEval 84.1
Natural2Code 82.6
InfographicVQA 81
TextVQA 78.7
MATH 77.9
DROP 74.9
EgoSchema 72.2
V* 71.7
RealWorldQA 70.4
MathVista 63.9
MMMU 62.2
BLINK 61.4
ActivityNet-QA 57.5
GPQA Diamond 46.2
HiddenMath 20.1

Its lab's numbers next to 3 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark	Lab says	Board says	Gap
GPQA Diamond	46.2% GPQA, 0-shot	45.9% Epoch AI run	+0.3 pts
MathVista	63.9% MathViSTA (testmini), 0-shot	63.9% Leaderboard	Same
MMMU	62.2% MMMU (val), 0-shot	62.2% Leaderboard	Same

Google DeepMind's models over time

Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Gemini 1.5 technical report, May 2024 update with 1.5 Flash (arXiv 2403.05530v3) May 14, 2024

GPQA, 0-shot 46.2
MATH, maj@64 (pretrained) (4-shot Minerva prompt = 67.7%) 77.9
PhysicsFinals, 0-shot 63.9
HiddenMath, 0-shot 20.1
Functional MATH (December snapshot), 0-shot 64.6
AMC 2022-23, 4-shot 46.4
GSM8K, 11-shot 90.8
BigBench - Hard, 3-shot 89.2
DROP, Variable shots (F1) 74.9
MMLU, maj@32 (pretrained) (5-shot = 85.9%) 91.7
Hellaswag, 10-shot 93.3
HumanEval chat preamble, 0-shot 84.1
Natural2Code chat preamble, 0-shot 82.6
WMT23, 1-shot (BLEURT) 75.3
MGSM, 8-shot 87.5
MMMU (val), 0-shot 62.2
MathViSTA (testmini), 0-shot 63.9
Ai2D (test), 0-shot 94.4
ChemicalDiagramQA (internal), 0-shot 69.7
ChartQA (test), 0-shot 87.2
BetterChartQA (internal), 0-shot 65.8
DocVQA (test), 0-shot (ANLS, with Cloud OCR annotations) 93.1
DUDE (test), 0-shot (ANLS) 46
TAT-DQA (test), 0-shot 37.8
InfographicVQA (test), 0-shot (ANLS) 81
TextVQA (val), 0-shot 78.7
VQAv2 (test-dev), 0-shot 80.2
V* bench (test), 0-shot 71.7
Blink (val), 0-shot 61.4
Realworld QA (test), 0-shot 70.4
VATEX (test), 4-shot (CIDEr) 64.6
VATEX ZH (val), 4-shot (CIDEr) 55.3
YouCook2 (val), 4-shot (CIDEr) 106.5
ActivityNet-QA (test), 0-shot 57.5
EgoSchema (test), 0-shot 72.2
OpenEQA (val), 0-shot (LLM-judge score) 57.9