Gemma 2 9B places in the top three on ALRAGE.
Google DeepMind Jun 2024 Open weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ALRAGE | 80.5% | #3 of 39 | Official leaderboard | |
| MuSR | 44.6% | #19 of 98 | Official leaderboard | |
| EQ-Bench v2 | 80.5 | #22 of 85 | Official leaderboard | |
| Open Arabic LLM Leaderboard v1 | 50.4% | #15 of 55 | Official leaderboard | |
| BoolQ | 84.2% | #8 of 28 | Lab model card | |
| WildBench | 45.4 | #19 of 47 | Official leaderboard | |
| WinoGrande | 80.6% | #19 of 47 | Lab model card | |
| SIQA | 53.4% | #10 of 24 | Lab model card | |
| ARC-Easy | 88% | #10 of 23 | Lab model card | |
| KoBALT | 21% | #7 of 12 | Official leaderboard | |
| MLogiQA | 41.5% | #10 of 17 | Official leaderboard | |
| MMLU-Pro | 52.1% | #131 of 212 | Official leaderboard | |
| AGIEval | 52.8% | #25 of 40 | Lab model card | |
| TriviaQA | 76.6% | #25 of 40 | Lab model card | |
| PIQA | 81.7% | #24 of 37 | Lab model card | |
| Natural Questions | 29.2% | #13 of 20 | Lab model card | |
| HellaSwag | 81.9% | #36 of 53 | Lab model card | |
| AlGhafa | 47.9% | #27 of 39 | Official leaderboard | |
| Open Arabic LLM Leaderboard v2 | 46.6% | #27 of 39 | Official leaderboard | |
| ARC-Challenge | 68.4% | #34 of 49 | Lab model card | |
| BIG-Bench Hard | 68.2% | #51 of 73 | Lab model card | |
| TruthfulQA | 50.3% | #21 of 30 | Lab model card | |
| ArabicMMLU | 36.3% | #28 of 39 | Official leaderboard | |
| MMLU-CF | 53.7% | #27 of 37 | Official leaderboard | |
| CulturalBench | 39% | #11 of 15 | Official leaderboard | |
| MMLU | 72.3% | #97 of 131 | Lab model card | |
| LMArena (Elo) | 1,267 | #255 of 335 | Official leaderboard | |
| MadinahQA | 31.7% | #30 of 39 | Official leaderboard | |
| SuperGPQA | 24% | #48 of 59 | Official leaderboard | |
| ArbMMLU-HT | 25% | #33 of 39 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 10.1% | #130 of 148 | Official leaderboard | |
| Creative Writing v3 | 49.5 | #108 of 120 | Official leaderboard | |
| EQ-Bench 3 | 636 | #76 of 78 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| KOR-Bench | 41.6% | #19 of 33 | Official leaderboard | |
| ZebraLogic | 12.8% | #29 of 46 | Official leaderboard | |
| PHYSICS | 11.9% | #14 of 20 | Official leaderboard | |
| GSM8K | 68.6% | #74 of 92 | Lab model card | |
| MATH Level 5 | 21% | #77 of 94 | Epoch AI run | |
| MATH | 36.6% | #82 of 99 | Lab model card | |
| GPQA Diamond | 27.5% | #214 of 229 | Epoch AI run | |
| IneqMath | 0% | #32 of 34 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 0.56% | #208 of 212 | Epoch AI run |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench | 37.7% | #50 of 69 | Official leaderboard | |
| MBPP | 59.2% | #57 of 71 | Lab model card | |
| MHPP | 18.1% | #19 of 23 | Official leaderboard | |
| BigCodeBench-Hard | 10.1% | #83 of 96 | Official leaderboard | |
| HumanEval | 40.2% | #92 of 101 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SnakeBench | 19 | #143 of 188 | Official leaderboard | |
| LLM Chess | -546 | #110 of 123 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 63% | #44 of 71 | Official leaderboard | |
| LEXam | 27.4% | #25 of 30 | Official leaderboard | |
| LegalBench | 68.1% | #122 of 138 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Winogender | 79.2% | #1 of 4 | Lab model card | |
| AraTrust | 77.2% | #18 of 39 | Official leaderboard | |
| OR-Bench | 79.9% | #19 of 28 | Official leaderboard | |
| CrowS-Pairs | 37.5% | #3 of 4 | Lab model card | |
| ToxiGen | 39.3 | #3 of 4 | Lab model card | |
| MolecularIQ | 3.95% | #15 of 17 | Official leaderboard | |
| nocot-bench (NCRI) | 44.7 | #171 of 185 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 56.2 | #31 of 71 | Official leaderboard | |
| Open LLM Leaderboard v2 | 21.2 | #53 of 99 | Official leaderboard |
The Self-Reported Index fits one score to the 17 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 105.9 to 110.0. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap | What differs |
|---|---|---|---|---|
| LMArena (Elo) | 1,187 LMSYS Chatbot Arena Elo, IT (Table 14) | 1,267 Leaderboard | −80 | Google quotes Arena's 2024 rating, while the board reflects later votes and rating adjustments. |
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Every result in the longer documents the lab published with it. 25 of them aren't in the launch post.