- GPQA Diamond 0-shot (IT); PT GPQA 5-shot = 24.3 42.4
- SimpleQA 0-shot (IT) 10
- FACTS Grounding (IT) 74.9
- BIG-Bench Hard 0-shot (IT); PT few-shot = 77.7 87.6
- BIG-Bench Extra Hard 0-shot (IT) 19.3
- IFEval 0-shot (IT) 90.4
- HellaSwag 10-shot (PT) 85.6
- BoolQ 0-shot (PT) 82.4
- PIQA 0-shot (PT) 83.3
- SocialIQA 0-shot (PT) 54.9
- TriviaQA 5-shot (PT) 85.5
- Natural Questions 5-shot (PT) 36.1
- ARC-c 25-shot (PT) 70.6
- ARC-e 0-shot (PT) 89
- WinoGrande 5-shot (PT) 78.8
- DROP 1-shot (PT) 77.2
- MMLU (Pro) 0-shot (IT); PT MMLU (Pro COT) 5-shot = 52.2 67.5
- LiveCodeBench 0-shot (IT) 29.7
- Bird-SQL (dev) (IT) 54.4
- Math 0-shot (IT); PT MATH 4-shot = 50.0 89
- HiddenMath 0-shot (IT) 60.3
- MBPP 3-shot (IT); PT = 65.6 74.4
- HumanEval 0-shot (IT); PT = 48.8 87.8
- Natural2Code 0-shot (IT) 84.5
- GSM8K 0-shot (IT); PT 8-shot = 82.6 95.9
- MMLU 5-shot (PT) 78.6
- AGIEval 3-5-shot (PT) 66.2
- Global-MMLU-Lite (PT); IT 0-shot = 75.1 75.7
- ECLeKTic (PT); IT 0-shot = 16.7
- WMT24++ (ChrF) (PT); IT 0-shot = 53.4 55.7
- MGSM (PT) 74.3
- FloRes (PT) 48.8
- XQuAD (all) (PT) 76.8
- IndicGenBench (PT) 63.4
- MMMU (val) (IT); PT MMMU = 56.1 64.9
- DocVQA (IT); PT DocVQA (val) = 85.6 86.6
- InfoVQA (IT); PT InfoVQA (val) = 59.4 70.6
- TextVQA (val) (PT); IT = 65.1 68.6
- AI2D (IT); PT = 79.0 84.5
- ChartQA (IT); PT = 76.3 78
- VQAv2 (PT); IT VQAv2 (val) = 71.0 72.9
- MathVista (testmini) (IT) 67.6
- COCOcap (PT) 116
- RealWorldQA (PT) 53.9
- ReMI (PT) 44.8
- BLINK (PT) 39.6
- OKVQA (PT) 60.2
- TallyQA (PT) 54.3
- SpatialSense VQA (PT) 59.4
- CountBenchQA (PT) 68
Gemma 3 27B
Gemma 3 27B ranks first on 3 of the 132 benchmarks with five or more models scored.
140
Benchmarks with a score
3
Ranked first, out of five or more models
10
In the top three
Google DeepMind Mar 2025 Gemma 3 27B
Open weights Open model
127.5
#164 of 256
Epoch Capabilities Index
Self-Reported Index 120.6 #341 of 404
- SWE-Bench Pro 11.4% #21 /24
- GPQA Diamond 47.7% #164 /229
- SimpleQA Verified 10.7% #46 /56
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MadinahQA | 69.4% | #1 of 39 | Official leaderboard | |
| ArabicMMLU | 70.8% | #2 of 39 | Official leaderboard | |
| Open Arabic LLM Leaderboard v2 | 71.4% | #2 of 39 | Official leaderboard | |
| Alyah | 74.7% | #1 of 10 | Official leaderboard | |
| AlGhafa | 77.9% | #4 of 39 | Official leaderboard | |
| ArbMMLU-HT | 66.9% | #4 of 39 | Official leaderboard | |
| Flores | 48.8 | #1 of 8 | Lab model card | |
| QIMMA | 63.8% | #4 of 25 | Official leaderboard | |
| BIG-Bench Hard | 87.6% | #12 of 73 | Lab model card | |
| Global PIQA | 81.7% | #8 of 44 | Official leaderboard | |
| HELMET | 44.5% | #3 of 16 | Official leaderboard | |
| IFEval | 90.4% | #26 of 135 | Lab model card | |
| KoBALT | 35% | #3 of 12 | Official leaderboard | |
| IndicGenBench | 63.4 | #1 of 4 | Lab model card | |
| XQuAD | 76.8 | #1 of 4 | Lab model card | |
| TriviaQA | 85.5% | #11 of 40 | Lab model card | |
| SwitzerlandQA | 71% | #2 of 7 | Official leaderboard | |
| SIQA | 54.9% | #7 of 24 | Lab model card | |
| ARC-Easy | 89% | #7 of 23 | Lab model card | |
| Vectara Hallucination Leaderboard (HHEM) | 3.00% | #46 of 148 | Official leaderboard | |
| BOUQuET | 29.5 | #6 of 18 | Official leaderboard | |
| MMBench-CN v1.1 | 78.2% | #60 of 178 | Official leaderboard | |
| Vectara Hallucination Leaderboard (2025 dataset) | 7.40% | #36 of 104 | Official leaderboard | |
| Natural Questions | 36.1% | #7 of 20 | Lab model card | |
| AGIEval | 66.2% | #16 of 40 | Lab model card | |
| BoolQ | 82.4% | #12 of 28 | Lab model card | |
| PIQA | 83.3% | #16 of 37 | Lab model card | |
| HellaSwag | 85.6% | #25 of 53 | Lab model card | |
| LMArena (Elo) | 1,365 | #163 of 335 | Official leaderboard | |
| ALRAGE | 73.3% | #19 of 39 | Official leaderboard | |
| WinoGrande | 78.8% | #24 of 47 | Lab model card | |
| HUMAINE | 27 | #27 of 50 | Official leaderboard | |
| ECLeKTic | 16.7% | #4 of 7 | Lab model card | |
| MMLU | 78.6% | #81 of 131 | Lab model card | |
| SEA-HELM | 64.5 | #44 of 71 | Official leaderboard | |
| ARC-Challenge | 70.6% | #31 of 49 | Lab model card | |
| BIG-Bench Extra Hard | 4.90% | #7 of 11 | Official leaderboard | |
| EQ-Bench 3 | 1,105 | #51 of 78 | Official leaderboard | |
| MRCR (Michelangelo, original) | 59.3% | #6 of 9 | Lab model card | |
| Structured Output Benchmark | 84.7% | #13 of 19 | Official leaderboard | |
| MGSM | 74.3% | #40 of 57 | Lab model card | |
| Creative Writing v3 | 58.7 | #86 of 120 | Official leaderboard | |
| Wer wird Millionär? LLM Benchmark | 289.0 | #56 of 77 | Official leaderboard | |
| DROP | 77.2% | #31 of 41 | Lab model card | |
| MMLU-Pro | 67.5% | #129 of 167 | Lab model card | |
| Longform Creative Writing (EQ-Bench) | 41.9 | #95 of 119 | Official leaderboard | |
| BBEH micro average | 19.3% | #12 of 15 | Lab model card | |
| SimpleQA Verified | 10.7% | #46 of 56 | Official leaderboard | |
| Slop Score | 69.5 | #14 of 17 | Official leaderboard | |
| Arena-Hard v2.0 | 15% | #19 of 23 | Official leaderboard | |
| SimpleQA | 10% | #50 of 60 | Lab model card | |
| Global MMLU Lite | 76.3% | #36 of 43 | Official leaderboard | |
| FACTS Grounding | 74.9% | #13 of 15 | Lab model card | |
| FACTS Parametric | 6.27% | #40 of 45 | Official leaderboard | |
| RULER | 66% | #39 of 42 | Lab model card | |
| Phare | 57.4% | #59 of 62 | Official leaderboard | |
| Fiction.LiveBench | 33.3% | #55 of 57 | Fiction.live leaderboard, via Epoch AI | |
| FACTS Search | 41.4% | #30 of 31 | Official leaderboard | |
| BullshitBench | 3.64% | #105 of 107 | Official leaderboard | |
| BullshitBench V2 | 3.00% | #127 of 129 | Official leaderboard | |
| LLM Confabulation Leaderboard | 40.3% | #58 of 58 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 95.9% | #8 of 92 | Lab model card | |
| MATH | 89% | #12 of 99 | Lab model card | |
| MATH Level 5 | 74% | #31 of 94 | Epoch AI run | |
| HiddenMath | 60.3% | #5 of 11 | Lab model card | |
| GPQA Diamond | 47.7% | #164 of 229 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 22.5% | #152 of 212 | Epoch AI run | |
| PolyMath | 20.3% | #33 of 38 | Official leaderboard | |
| DTBench | 52.5% | #157 of 180 | Official leaderboard | |
| Misguided Attention | 23.4% | #43 of 45 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 87.8% | #29 of 101 | Lab model card | |
| BigO(Bench) | 37.6% | #6 of 18 | Official leaderboard | |
| Natural2Code | 84.5% | #3 of 9 | Lab model card | |
| BigCodeBench-Hard | 26% | #42 of 96 | Official leaderboard | |
| MBPP | 74.4% | #36 of 71 | Lab model card | |
| SArena | 0.83 | #9 of 17 | Official leaderboard | |
| Bird-SQL | 54.4% | #8 of 13 | Lab model card | |
| ArtifactsBench (Gemini 2.5 Pro 0506 judge) | 37.2% | #12 of 18 | Official leaderboard | |
| LiveCodeBench | 29.7% | #44 of 54 | Lab model card | |
| SWE-Bench Pro | 11.4% | #21 of 24 | Official leaderboard | |
| Aider Polyglot | 4.90% | #45 of 46 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) | 59.1% | #25 of 67 | Official leaderboard | |
| SnakeBench | 25 | #88 of 188 | Official leaderboard | |
| MCP-Bench | 0.58 | #8 of 15 | Official leaderboard | |
| BFCL V4 | 29.5% | #40 of 65 | Official leaderboard | |
| LisanBench | 175.3 | #95 of 133 | Official leaderboard | |
| LLM-WikiRace | 0% | #18 of 21 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BRIDGE | 49.5 | #9 of 39 | Official leaderboard | |
| MultiFinBen | 15.7 | #4 of 9 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MM-Vet | 71% | #18 of 164 | Official leaderboard | |
| A-Bench | 76.3% | #22 of 116 | Official leaderboard | |
| MMBench | 82.3% | #37 of 155 | Official leaderboard | |
| SEED-Bench-2-Plus | 67% | #39 of 136 | Official leaderboard | |
| EmbodiedBench EB-Navigation | 45.4% | #5 of 17 | Official leaderboard | |
| MMBench-EN v1.1 | 79.6% | #56 of 178 | Official leaderboard | |
| CoreCognition | 59.4% | #38 of 119 | Official leaderboard | |
| MMMU | 64.9% | #45 of 136 | Official leaderboard | |
| COCO Captions | 116.0 | #1 of 3 | Lab model card | |
| OK-VQA | 60.2% | #1 of 3 | Lab model card | |
| MMStar | 59.6% | #61 of 173 | Official leaderboard | |
| EmbodiedBench EB-ALFRED | 37% | #6 of 17 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 53.3% | #57 of 152 | Official leaderboard | |
| EmbodiedBench EB-Manipulation | 17.5% | #7 of 17 | Official leaderboard | |
| CSVQA | 22.9% | #6 of 12 | Official leaderboard | |
| ReMI | 44.8% | #2 of 4 | Lab model card | |
| TallyQA | 54.3% | #2 of 4 | Lab model card | |
| MATH-Vision (MathVision) | 46% | #56 of 105 | Official leaderboard | |
| MORSE-500 | 14.6% | #6 of 11 | Official leaderboard | |
| SpaCE-10 | 41.5% | #20 of 36 | Official leaderboard | |
| EmbodiedBench EB-Habitat | 35.7% | #10 of 17 | Official leaderboard | |
| Perception Test MCVQA | 58.1% | #4 of 6 | Lab model card | |
| SpatialSense VQA | 59.4% | #2 of 3 | Lab model card | |
| MathCanvas-Bench | 15.8% | #11 of 16 | Official leaderboard | |
| InfographicVQA | 70.6% | #18 of 26 | Lab model card | |
| Vision Arena | 1,160 | #96 of 136 | Official leaderboard | |
| MM-HELIX | 16.6% | #12 of 17 | Official leaderboard | |
| Math-VR | 16.1% | #19 of 26 | Official leaderboard | |
| ActivityNet-QA | 52.8% | #6 of 8 | Lab model card | |
| VQAv2 | 72.9% | #6 of 8 | Lab model card | |
| MathVista | 67.6% | #53 of 70 | Lab model card | |
| AI2D | 84.5% | #40 of 52 | Lab model card | |
| GeoBench | 52% | #41 of 48 | Official leaderboard | |
| TextVQA | 68.6% | #15 of 17 | Lab model card | |
| MSEB | 7.63 | #8 of 9 | Official leaderboard | |
| CountBench | 68% | #26 of 28 | Lab model card | |
| BLINK | 39.6% | #27 of 29 | Lab model card | |
| DocVQA | 86.6% | #34 of 36 | Lab model card | |
| ChartQA | 78% | #35 of 37 | Lab model card | |
| RealWorldQA | 53.9% | #36 of 38 | Lab model card | |
| OmniAI OCR Benchmark | 42.9% | #4 of 4 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AraTrust | 89.1% | #3 of 39 | Official leaderboard | |
| Lakera B3 (Backbone Breaker Benchmark) | 57.9% | #15 of 35 | Official leaderboard | |
| MolecularIQ | 7.82% | #9 of 17 | Official leaderboard | |
| nocot-bench (NCRI) | 56.0 | #152 of 185 | Official leaderboard | |
| Spiral-Bench v1.2 | 25.2 | #18 of 21 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenVLM Leaderboard | 67.5 | #47 of 173 | Official leaderboard | |
| Political Science LLM Benchmark | 66% | #27 of 42 | Official leaderboard | |
| ACCoRD | 40.5 | #132 of 152 | Official leaderboard | |
| LMCA | 12.3% | #130 of 147 | Official leaderboard | |
| Conceptual Reasoning Index | 19.6 | #136 of 151 | Official leaderboard |
Its lab's numbers put it at 120.6, 6.9 points below its ECI
The Self-Reported Index fits one score to the 42 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 118.8 to 122.4. How we compute it.
- GSM8K 95.9
- IFEval 90.4
- ARC-Easy 89
- MATH 89
- HumanEval 87.8
- BIG-Bench Hard 87.6
- DocVQA 86.6
- HellaSwag 85.6
- TriviaQA 85.5
- Natural2Code 84.5
- AI2D 84.5
- PIQA 83.3
- BoolQ 82.4
- WinoGrande 78.8
- MMLU 78.6
- ChartQA 78
- DROP 77.2
- Global MMLU Lite 75.7
- FACTS Grounding 74.9
- MBPP 74.4
- MGSM 74.3
- ARC-Challenge 70.6
- InfographicVQA 70.6
- TextVQA 68.6
- CountBench 68
- MathVista 67.6
- MMLU-Pro 67.5
- AGIEval 66.2
- RULER 66
- MMMU 64.9
- HiddenMath 60.3
- MRCR (Michelangelo, original) 59.3
- SIQA 54.9
- Bird-SQL 54.4
- RealWorldQA 53.9
- ActivityNet-QA 52.8
- GPQA Diamond 42.4
- BLINK 39.6
- Natural Questions 36.1
- LiveCodeBench 29.7
- BBEH micro average 19.3
- SimpleQA 10
Its lab's numbers next to 8 leaderboards
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap | What differs |
|---|---|---|---|---|
| ECLeKTic | 16.7% ECLeKTic score, IT, 0-shot (Table 18). PT 2-shot (Table 13): 24.4 | 8.45% Leaderboard | +8.2 pts | |
| FACTS Grounding | 74.9% FACTS Grounding (IT) | 80.6% Leaderboard | −5.7 pts | |
| GPQA Diamond | 42.4% GPQA Diamond 0-shot (IT); PT GPQA 5-shot = 24.3 | 47.7% Epoch AI run | −5.3 pts | |
| LMArena (Elo) | 1,338 Chatbot Arena Elo, IT, preliminary results received March 8, 2025 (Table 5) | 1,365 Leaderboard | −27 | Google reports a preliminary March 2025 Arena rating, while the board reports a later rating from updated votes. |
| Global MMLU Lite | 75.7% Global-MMLU-Lite (PT); IT 0-shot = 75.1 | 76.3% Leaderboard | −0.6 pts | |
| BBEH micro average | 19.3% BIG-Bench Extra Hard 0-shot (IT) | 18.8% Leaderboard | +0.5 pts | |
| MMLU-Pro | 67.5% MMLU (Pro) 0-shot (IT); PT MMLU (Pro COT) 5-shot = 52.2 | 67.5% Leaderboard | Same | |
| MMMU | 64.9% MMMU (val) (IT); PT MMMU = 56.1 | 64.9% Leaderboard | Same |
Google DeepMind's models over time
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.
Gemma 3 Technical Report (arXiv 2503.19786v1) Mar 25, 2025
Capabilities
- LMArena (Elo) 1338
- MMLU-Pro 67.5
- LiveCodeBench 29.7
- Bird-SQL 54.4
- GPQA Diamond 42.4
- SimpleQA 10
- FACTS Grounding 74.9
- Global MMLU Lite 75.1
- MATH 89
- HiddenMath 60.3
- MMMU 64.9
- MMLU 76.9
- MBPP 74.4
- HumanEval 87.8
- Natural2Code 84.5
- GSM8K 95.9
- BIG-Bench Hard 87.6
- BBEH micro average 19.3
- IFEval 90.4
- ECLeKTic 16.7
- WMT24++ 53.4
- RULER 66
- MRCR (Michelangelo, original) 59.3
- DocVQA 86.6
- InfographicVQA 70.6
- TextVQA 65.1
- AI2D 84.5
- ChartQA 78
- VQAv2 71
- MathVista 67.6
- Perception Test MCVQA 58.1
- ActivityNet-QA 52.8
- HellaSwag 85.6
- BoolQ 82.4
- PIQA 83.3
- SIQA 54.9
- TriviaQA 85.5
- Natural Questions 36.1
- ARC-Challenge 70.6
- ARC-Easy 89
- WinoGrande 78.8
- DROP 77.2
- AGIEval 66.2
- COCO Captions 116
- RealWorldQA 53.9
- ReMI 44.8
- BLINK 39.6
- OK-VQA 60.2
- TallyQA 54.3
- SpatialSense VQA 59.4
- CountBench 68
- MGSM 74.3
- Flores 48.8
- XQuAD 76.8
- IndicGenBench 63.4