Gemma 3 12B

Gemma 3 12B places in the top three on 2 benchmarks.

Google DeepMind Released March 12th, 2025 Compare with others

100

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Google DeepMind Mar 2025

Gemma 3 12B

Open weights

115.5 #364 of 404

Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.

GPQA Diamond 39.5% #187 /229
SimpleQA Verified 6.70% #52 /56
BLINK 35.9% #29 /29

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
Vectara Hallucination Leaderboard (2025 dataset)	4.40%	#7 of 104	Official leaderboard
ArabicMMLU	65.8%	#7 of 39	Official leaderboard
ArbMMLU-HT	61.2%	#8 of 39	Official leaderboard
Flores	46.0	#2 of 8	Lab model card
Vectara Hallucination Leaderboard (HHEM)	2.80%	#38 of 148	Official leaderboard
MadinahQA	53%	#12 of 39	Official leaderboard
HELMET	40.6%	#5 of 16	Official leaderboard
Open Arabic LLM Leaderboard v2	58.2%	#13 of 39	Official leaderboard
IFEval	88.9%	#46 of 135	Lab model card
BIG-Bench Hard	85.7%	#26 of 73	Lab model card
ARC-Easy	88.3%	#9 of 23	Lab model card
MMBench-CN v1.1	74.3%	#81 of 178	Official leaderboard
SIQA	53.4%	#11 of 24	Lab model card
AlGhafa	60.7%	#19 of 39	Official leaderboard
Natural Questions	31.4%	#10 of 20	Lab model card
IndicGenBench	61.7	#2 of 4	Lab model card
XQuAD	74.5	#2 of 4	Lab model card
AGIEval	57.4%	#21 of 40	Lab model card
LMArena (Elo)	1,342	#188 of 335	Official leaderboard
HellaSwag	84.2%	#30 of 53	Lab model card
TriviaQA	78.2%	#23 of 40	Lab model card
PIQA	81.8%	#23 of 37	Lab model card
ARC-Challenge	68.9%	#33 of 49	Lab model card
SEA-HELM	62.4	#48 of 71	Official leaderboard
WinoGrande	74.3%	#32 of 47	Lab model card
BoolQ	78.8%	#20 of 28	Lab model card
ECLeKTic	10.3%	#5 of 7	Lab model card
BIG-Bench Extra Hard	4.50%	#8 of 11	Official leaderboard
Wer wird Millionär? LLM Benchmark	244.0	#57 of 77	Official leaderboard
MMLU	71.9%	#98 of 131	Lab model card
Global MMLU Lite	69.5%	#12 of 16	Lab model card
Creative Writing v3	57.0	#93 of 120	Official leaderboard
FACTS Grounding	75.8%	#12 of 15	Lab model card
Longform Creative Writing (EQ-Bench)	41.8	#96 of 119	Official leaderboard
MGSM	64.3%	#46 of 57	Lab model card
MMLU-Pro	60.6%	#143 of 167	Lab model card
BBEH micro average	16.3%	#13 of 15	Lab model card
Slop Score	70.6	#15 of 17	Official leaderboard
MRCR (Michelangelo, original)	49.8%	#8 of 9	Lab model card
SimpleQA Verified	6.70%	#52 of 56	Official leaderboard
SimpleQA	6.30%	#56 of 60	Lab model card
FACTS Parametric	3.56%	#43 of 45	Official leaderboard
RULER	57.1%	#41 of 42	Lab model card
ALRAGE	25.3%	#39 of 39	Official leaderboard
Antidote	893	#36 of 36	Official leaderboard

Math and science

Benchmark	Score	Rank	Source
MATH	83.8%	#15 of 99	Lab model card
GSM8K	94.4%	#18 of 92	Lab model card
HiddenMath	54.5%	#7 of 11	Lab model card
OTIS Mock AIME 2024-2025	16.7%	#156 of 212	Epoch AI run
GPQA Diamond	39.5%	#187 of 229	Epoch AI run
DTBench	48.8%	#169 of 180	Official leaderboard

Coding

Benchmark	Score	Rank	Source
HumanEval	85.4%	#38 of 101	Lab model card
Natural2Code	80.7%	#5 of 9	Lab model card
MBPP	73%	#40 of 71	Lab model card
SArena	0.82	#11 of 17	Official leaderboard
Bird-SQL	47.9%	#9 of 13	Lab model card
LiveCodeBench	24.6%	#47 of 54	Lab model card

Agents and tools

Benchmark	Score	Rank	Source
BFCL V4	30.4%	#39 of 65	Official leaderboard
LLM-WikiRace	1.00%	#13 of 21	Official leaderboard
Berkeley Function Calling Leaderboard (BFCL)	50.6%	#49 of 67	Official leaderboard
SnakeBench	19	#144 of 188	Official leaderboard
LisanBench	114.0	#112 of 133	Official leaderboard

Professional work

Benchmark	Score		Rank	Source
LEXam	41.3%		#19 of 30	Official leaderboard

Vision, audio and robotics

Benchmark	Score	Rank	Source
MM-Vet	64.9%	#40 of 164	Official leaderboard
EmbodiedBench EB-Manipulation	20.6%	#5 of 17	Official leaderboard
SpatialSense VQA	60%	#1 of 3	Lab model card
SEED-Bench-2-Plus	65.2%	#57 of 136	Official leaderboard
MMMU	59.6%	#58 of 136	Official leaderboard
MMStar	56.1%	#75 of 173	Official leaderboard
CCBench (MMBench Chinese Culture)	49%	#68 of 152	Official leaderboard
EmbodiedBench EB-Navigation	34%	#8 of 17	Official leaderboard
MMBench	77.5%	#75 of 155	Official leaderboard
MMBench-EN v1.1	74.9%	#91 of 178	Official leaderboard
SpaCE-10	41.5%	#21 of 36	Official leaderboard
EmbodiedBench EB-ALFRED	25.7%	#10 of 17	Official leaderboard
A-Bench	70.3%	#71 of 116	Official leaderboard
COCO Captions	111.0	#2 of 3	Lab model card
OK-VQA	58.7%	#2 of 3	Lab model card
ReMI	38.5%	#3 of 4	Lab model card
TallyQA	51.8%	#3 of 4	Lab model card
AI2D	84.2%	#41 of 52	Lab model card
EmbodiedBench EB-Habitat	23%	#14 of 17	Official leaderboard
MathVista	62.9%	#58 of 70	Lab model card
Perception Test MCVQA	54.9%	#5 of 6	Lab model card
InfographicVQA	64.9%	#22 of 26	Lab model card
ActivityNet-QA	50.4%	#7 of 8	Lab model card
VQAv2	71.6%	#7 of 8	Lab model card
DocVQA	87.1%	#32 of 36	Lab model card
TextVQA	67.7%	#16 of 17	Lab model card
ChartQA	75.7%	#36 of 37	Lab model card
RealWorldQA	52.2%	#37 of 38	Lab model card
BLINK	35.9%	#29 of 29	Lab model card
CountBench	17.8%	#28 of 28	Lab model card

Safety and security

Benchmark	Score	Rank	Source
AraTrust	87%	#6 of 39	Official leaderboard
TOMG-Bench	15%	#2 of 5	Official leaderboard
MolecularIQ	5.13%	#12 of 17	Official leaderboard

Other

Benchmark	Score	Rank	Source
OpenVLM Leaderboard	62.8	#72 of 173	Official leaderboard
ACCoRD	38.3	#135 of 152	Official leaderboard
Conceptual Reasoning Index	13.3	#142 of 151	Official leaderboard
LMCA	4.46%	#144 of 147	Official leaderboard

Its lab's numbers put it at 115.5, number 364 of 404

The Self-Reported Index fits one score to the 41 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 113.1 to 118.0. How we compute it.

Its lab's numbers next to 6 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark	Lab says	Board says	Gap
FACTS Grounding	75.8% FACTS Grounding, IT, zero-shot (Table 6)	80.5% Leaderboard	−4.7 pts
ECLeKTic	10.3% ECLeKTic score, IT, 0-shot (Table 18). PT 2-shot (Table 13): 17.2	6.20% Leaderboard	+4.1 pts
GPQA Diamond	40.9% GPQA Diamond, IT, zero-shot (Table 6)	39.5% Epoch AI run	+1.4 pts
BBEH micro average	16.3% BBEH, IT, 0-shot (Table 18)	16.3% Leaderboard	Same
MMLU-Pro	60.6% MMLU-Pro, IT, zero-shot (Table 6)	60.6% Leaderboard	Same
MMMU	59.6% MMMU (val), IT (Table 6 and Table 16)	59.6% Leaderboard	Same

Its system card

Every result in the longer documents the lab published with it. 54 of them aren't in the launch post.

Gemma 3 Technical Report (arXiv 2503.19786v1) Mar 25, 2025