Benchmark Almanac

Gemma 3 4B

Gemma 3 4B places in the top three on Flores.

Google DeepMind Released March 12th, 2025 Compare with others
92
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Google DeepMind Mar 2025

Gemma 3 4B

Open weights

106.9 #385 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • MultiChallenge 22.8% #42 /42
  • GPQA Diamond 23.2% #224 /229
  • SimpleQA Verified 4.60% #56 /56

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
IFEval 90.2% #29 of 135 Lab model card
Vectara Hallucination Leaderboard (2025 dataset) 6.40% #28 of 104 Official leaderboard
ArabicMMLU 55.3% #14 of 39 Official leaderboard
Flores 39.2 #3 of 8 Lab model card
Vectara Hallucination Leaderboard (HHEM) 3.70% #56 of 148 Official leaderboard
ArbMMLU-HT 45.2% #18 of 39 Official leaderboard
MadinahQA 44.1% #20 of 39 Official leaderboard
ARC-Easy 82.4% #12 of 23 Lab model card
SIQA 51.9% #13 of 24 Lab model card
AlGhafa 54.2% #22 of 39 Official leaderboard
Open Arabic LLM Leaderboard v2 50.5% #22 of 39 Official leaderboard
BIG-Bench Hard 72.2% #43 of 73 Lab model card
MMBench-CN v1.1 66.8% #114 of 178 Official leaderboard
LMArena (Elo) 1,303 #229 of 335 Official leaderboard
IndicGenBench 57.2 #3 of 4 Lab model card
XQuAD 68.0 #3 of 4 Lab model card
EQ-Bench 3 1,001 #62 of 78 Official leaderboard
ARC-Challenge 56.2% #39 of 49 Lab model card
Creative Writing v3 54.1 #98 of 120 Official leaderboard
AGIEval 42.1% #33 of 40 Lab model card
TriviaQA 65.8% #33 of 40 Lab model card
HellaSwag 77.2% #44 of 53 Lab model card
PIQA 79.6% #31 of 37 Lab model card
ECLeKTic 4.60% #6 of 7 Lab model card
SEA-HELM 47.8 #62 of 71 Official leaderboard
Natural Questions 20% #18 of 20 Lab model card
Longform Creative Writing (EQ-Bench) 34.4 #108 of 119 Official leaderboard
MMLU 58.1% #119 of 131 Lab model card
BoolQ 72.3% #26 of 28 Lab model card
BBEH micro average 11% #14 of 15 Lab model card
FACTS Grounding 70.1% #14 of 15 Lab model card
Global MMLU Lite 65.1% #41 of 43 Official leaderboard
WinoGrande 64.7% #45 of 47 Lab model card
MMLU-Pro 43.6% #160 of 167 Lab model card
Wer wird Millionär? LLM Benchmark 64.0 #74 of 77 Official leaderboard
SimpleQA 4.00% #58 of 60 Lab model card
ALRAGE 33.4% #38 of 39 Official leaderboard
MGSM 34.7% #56 of 57 Lab model card
SimpleQA Verified 4.60% #56 of 56 Official leaderboard
FACTS Parametric 1.57% #45 of 45 Official leaderboard
MultiChallenge 22.8% #42 of 42 Official leaderboard
RULER 46.8% #42 of 42 Lab model card
Slop Score 85.2 #17 of 17 Official leaderboard
BIG-Bench Extra Hard 3.40% #11 of 11 Official leaderboard
MRCR (Michelangelo, original) 44.6% #9 of 9 Lab model card

Math and science

BenchmarkScoreRankSource
MATH 75.6% #36 of 99 Lab model card
GSM8K 89.2% #45 of 92 Lab model card
HiddenMath 43% #8 of 11 Lab model card
OTIS Mock AIME 2024-2025 7.50% #169 of 212 Epoch AI run
DTBench 50.9% #163 of 180 Official leaderboard
GPQA Diamond 23.2% #224 of 229 Epoch AI run

Coding

BenchmarkScoreRankSource
HumanEval 71.3% #73 of 101 Lab model card
MBPP 63.2% #54 of 71 Lab model card
Natural2Code 70.3% #8 of 9 Lab model card
Bird-SQL 36.3% #12 of 13 Lab model card
LiveCodeBench 12.6% #53 of 54 Lab model card

Agents and tools

BenchmarkScoreRankSource
BFCL V4 19.6% #58 of 65 Official leaderboard
Berkeley Function Calling Leaderboard (BFCL) 39.6% #60 of 67 Official leaderboard
LLM-WikiRace 0% #19 of 21 Official leaderboard
LisanBench 50.0 #124 of 133 Official leaderboard
SnakeBench 8 #186 of 188 Official leaderboard

Professional work

BenchmarkScoreRankSource
MultiFinBen 12.7 #6 of 9 Official leaderboard

Vision, audio and robotics

BenchmarkScoreRankSource
MM-Vet 57.8% #65 of 164 Official leaderboard
SEED-Bench-2-Plus 60.7% #75 of 136 Official leaderboard
MMStar 47.9% #111 of 173 Official leaderboard
CCBench (MMBench Chinese Culture) 36.9% #99 of 152 Official leaderboard
MMMU 48.8% #90 of 136 Official leaderboard
MMBench 69.6% #115 of 155 Official leaderboard
A-Bench 67% #90 of 116 Official leaderboard
MMBench-EN v1.1 66.1% #141 of 178 Official leaderboard
MathVista 50% #67 of 70 Lab model card
CountBench 26.1% #27 of 28 Lab model card
BLINK 38% #28 of 29 Lab model card
AI2D 74.8% #51 of 52 Lab model card
RealWorldQA 45.5% #38 of 38 Lab model card
ChartQA 68.8% #37 of 37 Lab model card
DocVQA 75.8% #36 of 36 Lab model card
InfographicVQA 50% #26 of 26 Lab model card
TextVQA 57.8% #17 of 17 Lab model card
ActivityNet-QA 46.3% #8 of 8 Lab model card
VQAv2 62.4% #8 of 8 Lab model card
Perception Test MCVQA 50.6% #6 of 6 Lab model card
ReMI 27.3% #4 of 4 Lab model card
TallyQA 42.5% #4 of 4 Lab model card
COCO Captions 102.0 #3 of 3 Lab model card
OK-VQA 51% #3 of 3 Lab model card
SpatialSense VQA 50.9% #3 of 3 Lab model card

Safety and security

BenchmarkScoreRankSource
AraTrust 78.3% #15 of 39 Official leaderboard

Other

BenchmarkScoreRankSource
OpenVLM Leaderboard 55.4 #106 of 173 Official leaderboard
ACCoRD 23.8 #146 of 152 Official leaderboard
Conceptual Reasoning Index 10.1 #150 of 151 Official leaderboard
LMCA 2.78% #147 of 147 Official leaderboard

Its lab's numbers put it at 106.9, number 385 of 404

The Self-Reported Index fits one score to the 41 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 104.5 to 109.2. How we compute it.

Its lab's numbers next to 7 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
Global MMLU Lite 54.5% Global MMLU-Lite, IT, zero-shot (Table 6). PT 5-shot GMMLU (Table 13): 57.0 65.1% Leaderboard −11 pts
GPQA Diamond 30.8% GPQA Diamond, IT, zero-shot (Table 6) 23.2% Epoch AI run +7.6 pts
FACTS Grounding 70.1% FACTS Grounding, IT, zero-shot (Table 6) 64.8% Leaderboard +5.3 pts
ECLeKTic 4.60% ECLeKTic score, IT, 0-shot (Table 18). PT 2-shot (Table 13): 11.0 2.25% Leaderboard +2.4 pts
BBEH micro average 11% BBEH, IT, 0-shot (Table 18) 11% Leaderboard Same
MMLU-Pro 43.6% MMLU-Pro, IT, zero-shot (Table 6) 43.6% Leaderboard Same
MMMU 48.8% MMMU (val), IT (Table 6 and Table 16) 48.8% Leaderboard Same

Its system card

Every result in the longer documents the lab published with it. 54 of them aren't in the launch post.

Gemma 3 Technical Report (arXiv 2503.19786v1) Mar 25, 2025

Capabilities