Gemini 1.5 Flash Sep 2024

Gemini 1.5 Flash ranks first on OVBench.

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Google DeepMind Aug 2024

Sep 2024 snapshot Closed weights

127.0 #167 of 256

Epoch Capabilities Index

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Benchmark	Score	Rank	Source
PIQA	87.5%	#3 of 58	Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
MMBench-CN v1.1	81%	#38 of 178	Official leaderboard
FACTS Grounding	82.3%	#8 of 34	Official leaderboard
MMLU-CF	64.8%	#12 of 37	Official leaderboard
Vectara Hallucination Leaderboard (HHEM)	3.40%	#53 of 148	Official leaderboard
MMLU-Pro	64.1%	#102 of 212	Official leaderboard
ToRR	45.4	#7 of 14	Official leaderboard
IFEval	83.1%	#32 of 60	Official leaderboard
LMArena (Elo)	1,309	#220 of 335	Official leaderboard
ECLeKTic	11.6%	#24 of 35	Official leaderboard

Benchmark	Score	Rank	Source
U-MATH	57.8%	#11 of 30	Official leaderboard
MATH Level 5	61.9%	#43 of 94	Epoch AI run
GPQA Main	43.7%	#40 of 60	Official leaderboard
GPQA Diamond	47.3%	#168 of 229	Epoch AI run
OTIS Mock AIME 2024-2025	16.3%	#157 of 212	Epoch AI run
FrontierMath	0%	#69 of 70	Epoch AI run

Benchmark	Score	Rank	Source
HumanEval	82.3%	#15 of 46	Official leaderboard
HumanEval+	75.6%	#15 of 46	Official leaderboard
HumanEval-V	18.6%	#8 of 24	Official leaderboard
MBPP	84.7%	#13 of 35	Official leaderboard
MBPP+	67.5%	#17 of 35	Official leaderboard

Benchmark	Score		Rank	Source
BALROG	14.6%		#28 of 33	Official leaderboard
Step Race	1		#54 of 59	Official leaderboard

Benchmark	Score	Rank	Source
MedQA	65.6%	#38 of 71	Official leaderboard
LegalBench	63.2%	#126 of 138	Official leaderboard
MortgageTax	42.8%	#80 of 87	Official leaderboard
CorpFin v2	38.2%	#114 of 119	Official leaderboard
TaxEval v2	48.2%	#125 of 129	Official leaderboard

Benchmark	Score	Rank	Source
OVBench	50.7%	#1 of 7	Official leaderboard
SEED-Bench-2-Plus	68.6%	#23 of 136	Official leaderboard
MMBench-EN v1.1	81.6%	#42 of 178	Official leaderboard
MEGA-Bench	43.8%	#6 of 22	Official leaderboard
Image2Struct	71.0	#5 of 18	Official leaderboard
MMVU	58.8%	#9 of 28	Official leaderboard
GeoBench	76%	#18 of 48	Official leaderboard
MMBench	79.4%	#60 of 155	Official leaderboard
A-Bench	73.6%	#46 of 116	Official leaderboard
CCBench (MMBench Chinese Culture)	52.4%	#62 of 152	Official leaderboard
MMStar	55.8%	#77 of 173	Official leaderboard
Video-MMMU	49.8%	#11 of 24	Official leaderboard
VHELM	51.9	#21 of 41	Official leaderboard
AHELM	40.1	#7 of 10	Official leaderboard
Vision Arena	1,142	#104 of 136	Official leaderboard
VISTA	34%	#43 of 54	Official leaderboard
ZeroBench	2.00%	#65 of 76	Official leaderboard

Benchmark	Score	Rank	Source
AgentDojo	3.50%	#3 of 20	Official leaderboard
AIR-Bench 2024	67.1%	#40 of 82	Official leaderboard
FORTRESS	50.6%	#58 of 73	Official leaderboard

Benchmark	Score	Rank	Source
OpenVLM Leaderboard	63.5	#69 of 173	Official leaderboard
HELM Lite	57.4	#29 of 71	Official leaderboard
HELM Capabilities	60.9	#41 of 60	Official leaderboard

Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)