Gemma 2 27B.
Google DeepMind Jun 2024 Open weights Open model
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Open Arabic LLM Leaderboard v1 | 55.2% | #4 of 55 | Official leaderboard | |
| ALRAGE | 80.5% | #4 of 39 | Official leaderboard | |
| BoolQ | 84.8% | #5 of 28 | Lab model card | |
| AlGhafa | 72.2% | #9 of 39 | Official leaderboard | |
| ArbMMLU-HT | 61.2% | #9 of 39 | Official leaderboard | |
| WildBench | 49.4 | #11 of 47 | Official leaderboard | |
| WinoGrande | 83.7% | #11 of 47 | Lab model card | |
| EQ-Bench v2 | 80.5 | #20 of 85 | Official leaderboard | |
| MuSR | 44% | #26 of 98 | Official leaderboard | |
| TriviaQA | 83.7% | #13 of 40 | Lab model card | |
| SIQA | 53.7% | #8 of 24 | Lab model card | |
| ARC-Easy | 88.6% | #8 of 23 | Lab model card | |
| MMLU-CF | 64.2% | #14 of 37 | Official leaderboard | |
| Natural Questions | 34.5% | #8 of 20 | Lab model card | |
| Open Arabic LLM Leaderboard v2 | 55.5% | #16 of 39 | Official leaderboard | |
| MLogiQA | 50.5% | #7 of 17 | Official leaderboard | |
| HellaSwag | 86.4% | #23 of 53 | Lab model card | |
| PIQA | 83.2% | #17 of 37 | Lab model card | |
| AGIEval | 55.1% | #22 of 40 | Lab model card | |
| BIG-Bench Hard | 74.9% | #41 of 73 | Lab model card | |
| RewardBench | 80.9% | #27 of 48 | Official leaderboard | |
| MMLU-Pro | 56.5% | #121 of 212 | Official leaderboard | |
| ARC-Challenge | 71.4% | #30 of 49 | Lab model card | |
| TruthfulQA | 51.6% | #19 of 30 | Lab model card | |
| ArabicMMLU | 37.1% | #26 of 39 | Official leaderboard | |
| CulturalBench | 43% | #10 of 15 | Official leaderboard | |
| LMArena (Elo) | 1,290 | #235 of 335 | Official leaderboard | |
| MMLU | 76.2% | #92 of 131 | Lab model card | |
| MadinahQA | 33.3% | #28 of 39 | Official leaderboard | |
| SuperGPQA | 27.4% | #45 of 59 | Official leaderboard | |
| BBEH micro average | 14.8% | #9 of 11 | Official leaderboard | |
| BIG-Bench Extra Hard | 4.00% | #9 of 11 | Official leaderboard | |
| DROP | 72.2% | #34 of 41 | Lab model card | |
| SEAL Multilingual | 951 | #16 of 19 | Official leaderboard | |
| LLM Confabulation Leaderboard | 27.1% | #51 of 58 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OlympicArena | 26.3% | #9 of 25 | Official leaderboard | |
| PHYSICS | 18.3% | #9 of 20 | Official leaderboard | |
| KOR-Bench | 44.5% | #16 of 33 | Official leaderboard | |
| ZebraLogic | 16.3% | #26 of 46 | Official leaderboard | |
| MATH Level 5 | 27.9% | #67 of 94 | Epoch AI run | |
| GSM8K | 74% | #71 of 92 | Lab model card | |
| MATH | 42.3% | #78 of 99 | Lab model card | |
| GPQA Diamond | 36.5% | #193 of 229 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 1.39% | #201 of 212 | Epoch AI run | |
| DTBench | 48% | #172 of 180 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench | 47.6% | #30 of 69 | Official leaderboard | |
| MHPP | 28.6% | #12 of 23 | Official leaderboard | |
| BigCodeBench-Hard | 20% | #60 of 96 | Official leaderboard | |
| MBPP | 67.4% | #49 of 71 | Lab model card | |
| HumanEval | 51.8% | #85 of 101 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 68.4% | #31 of 71 | Official leaderboard | |
| LegalBench | 68% | #123 of 138 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AraTrust | 80.5% | #14 of 39 | Official leaderboard | |
| ToxiGen | 38.4 | #2 of 4 | Lab model card | |
| Winogender | 77.2% | #2 of 4 | Lab model card | |
| PrimeVul | 63% | #1 of 2 | Lab model card | |
| MolecularIQ | 6.92% | #10 of 17 | Official leaderboard | |
| OR-Bench | 62% | #17 of 28 | Official leaderboard | |
| nocot-bench (NCRI) | 53.3 | #159 of 185 | Official leaderboard | |
| BaxBench | 3.95% | #30 of 34 | Official leaderboard | |
| CrowS-Pairs | 36.7% | #4 of 4 | Lab model card | |
| Hack the Box challenge set (Google DeepMind) | 0% | #4 of 4 | Lab model card | |
| In-house CTF suite (Phuong et al. 2024) | 7.70% | #4 of 4 | Lab model card | |
| InterCode-CTF | 44.7% | #4 of 4 | Lab model card | |
| DiverseVul | 57% | #1 of 1 | Lab model card | |
| Money talks | 3.72 | #1 of 1 | Lab model card | |
| SecretPatch | 72% | #1 of 1 | Lab model card | |
| Self-proliferation challenges (Phuong et al. 2024) | 0% | #1 of 1 | Lab model card | |
| SPI (security patch identification) | 53% | #1 of 1 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 67.5 | #25 of 71 | Official leaderboard | |
| Open LLM Leaderboard v2 | 23.9 | #45 of 99 | Official leaderboard | |
| LMCA | 7.10% | #138 of 147 | Official leaderboard |
The Self-Reported Index fits one score to the 18 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 110.0 to 114.4. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap | What differs |
|---|---|---|---|---|
| LMArena (Elo) | 1,218 LMSYS Chatbot Arena Elo, IT (Table 14) | 1,290 Leaderboard | −72 | The lab quotes a July 2024 Arena rating, while the board uses a November 2026 rating with style controls. |
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 20 of them aren't in the launch post.