Benchmark Almanac

Gemma 3 1B

Gemma 3 1B.

Google DeepMind Released March 12th, 2025 Compare with others
35
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Google DeepMind Mar 2025

Gemma 3 1B

Open weights

90.4 #400 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • GPQA Diamond 19.9% #225 /229
  • BBEH micro average 7.20% #15 /15
  • Bird-SQL 6.40% #13 /13

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
Vectara Hallucination Leaderboard (HHEM) 5.30% #98 of 148 Official leaderboard
IFEval 80.2% #105 of 135 Lab model card
SIQA 48.9% #22 of 24 Lab model card
BIG-Bench Hard 39.1% #68 of 73 Lab model card
PIQA 73.8% #36 of 37 Lab model card
TriviaQA 39.8% #39 of 40 Lab model card
WinoGrande 58.2% #46 of 47 Lab model card
ARC-Challenge 38.4% #48 of 49 Lab model card
HellaSwag 62.3% #52 of 53 Lab model card
MMLU-Pro 14.7% #167 of 167 Lab model card
MMLU 38.8% #131 of 131 Lab model card
SimpleQA 2.20% #60 of 60 Lab model card
MGSM 2.04% #57 of 57 Lab model card
BoolQ 63.2% #28 of 28 Lab model card
ARC-Easy 73% #23 of 23 Lab model card
Natural Questions 9.48% #20 of 20 Lab model card
Global MMLU Lite 34.2% #16 of 16 Lab model card
BBEH micro average 7.20% #15 of 15 Lab model card
FACTS Grounding 36.4% #15 of 15 Lab model card
Flores 29.5 #8 of 8 Lab model card
ECLeKTic 1.40% #7 of 7 Lab model card
IndicGenBench 41.4 #4 of 4 Lab model card
XQuAD 43.9 #4 of 4 Lab model card

Math and science

BenchmarkScoreRankSource
MATH 48% #73 of 99 Lab model card
GSM8K 62.8% #78 of 92 Lab model card
OTIS Mock AIME 2024-2025 1.11% #204 of 212 Epoch AI run
GPQA Diamond 19.9% #225 of 229 Epoch AI run
HiddenMath 15.8% #11 of 11 Lab model card

Coding

BenchmarkScoreRankSource
HumanEval 41.5% #90 of 101 Lab model card
MBPP 35.2% #67 of 71 Lab model card
LiveCodeBench 1.90% #54 of 54 Lab model card
Bird-SQL 6.40% #13 of 13 Lab model card
Natural2Code 56% #9 of 9 Lab model card

Agents and tools

BenchmarkScoreRankSource
Berkeley Function Calling Leaderboard (BFCL) 16.3% #67 of 67 Official leaderboard
BFCL V4 7.17% #65 of 65 Official leaderboard

Its lab's numbers put it at 90.4, number 400 of 404

The Self-Reported Index fits one score to the 27 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 86.7 to 94.1. How we compute it.

Its lab's numbers next to 2 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
GPQA Diamond 19.2% GPQA Diamond, IT, zero-shot (Table 6) 19.9% Epoch AI run −0.8 pts
MMLU-Pro 14.7% MMLU-Pro, IT, zero-shot (Table 6) 14.7% Leaderboard Same

Its system card

Every result in the longer documents the lab published with it. 33 of them aren't in the launch post.

Gemma 3 Technical Report (arXiv 2503.19786v1) Mar 25, 2025

Capabilities