- Hellaswag, 10-shot 92.5
- MMLU, 5-shot 81.9
- GSM8K, 11-shot 91.7
- GPQA, 0-shot (4-shot = 37.9%) 41.5
- MATH, 7-shot (4-shot Minerva prompt = 58.5%) 59.4
- PhysicsFinals, 0-shot 60.7
- AMC 2022-23, 4-shot 37.2
- BigBench - Hard, 3-shot 84
- DROP, Variable shots (F1) 78.9
- HumanEval chat preamble, 0-shot 71.9
- Natural2Code chat preamble, 0-shot 77.7
- WMT23, 1-shot (BLEURT) 75.2
- MGSM, 8-shot 88.7
- MMMU (val), 0-shot 58.5
- Ai2D (test), 0-shot 80.3
- MathViSTA (testmini), 0-shot 52.1
- ChartQA (test), 0-shot 81.3
- VQAv2 (test-dev), 0-shot 73.2
- TextVQA (val), 0-shot 73.5
- DocVQA (test), 0-shot (ANLS) 86.5
- InfographicVQA (test), 0-shot (ANLS) 72.7
- VATEX (test), 4-shot (CIDEr) 63
- VATEX ZH (val), 4-shot (CIDEr) 54.9
- YouCook2 (val), 4-shot (CIDEr) 134.2
- ActivityNet-QA (test), 0-shot 56.7
- EgoSchema (test), 0-shot 63.2
Gemini 1.5 Pro Feb 2024
Gemini 1.5 Pro.
Google DeepMind Released February 15th, 2024 Family Gemini 1.5 The lab's model card Compare with others 31
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Google DeepMind Feb 2024 Gemini 1.5 Pro
Feb 2024 snapshot Closed weights Flagship
120.4
#342 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
- StructEval 71.8% #19 /28
- TextVQA 73.5% #13 /17
- EgoSchema 63.2% #12 /12
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HellaSwag | 92.5% | #5 of 53 | Lab model card | |
| MGSM | 88.7% | #15 of 57 | Lab model card | |
| BIG-Bench Hard | 84% | #27 of 73 | Lab model card | |
| MMLU | 81.9% | #66 of 131 | Lab model card | |
| SEAL Multilingual | 1,005 | #12 of 19 | Official leaderboard | |
| WMT23 | 75.2 | #2 of 3 | Lab model card | |
| DROP | 78.9% | #28 of 41 | Lab model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 91.7% | #35 of 92 | Lab model card | |
| MATH | 59.4% | #61 of 99 | Lab model card | |
| GSM1k | 90.5% | #10 of 15 | Official leaderboard | |
| GPQA Diamond | 41.5% | #268 of 302 | Lab model card | |
| AMC | 37.2% | #4 of 4 | Lab model card | |
| PhysicsFinals | 60.7% | #2 of 2 | Lab model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Natural2Code | 77.7% | #6 of 9 | Lab model card | |
| StructEval | 71.8% | #19 of 28 | Official leaderboard | |
| HumanEval | 71.9% | #72 of 101 | Lab model card | |
| SEAL Coding | 891 | #21 of 26 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LisanBench | 492.0 | #68 of 133 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| InfographicVQA | 72.7% | #16 of 26 | Lab model card | |
| ActivityNet-QA | 56.7% | #5 of 8 | Lab model card | |
| VQAv2 | 73.2% | #5 of 8 | Lab model card | |
| VATEX ZH | 54.9 | #3 of 4 | Lab model card | |
| YouCook2 | 134.2 | #3 of 4 | Lab model card | |
| ChartQA | 81.3% | #28 of 37 | Lab model card | |
| TextVQA | 73.5% | #13 of 17 | Lab model card | |
| MMMU | 58.5% | #89 of 103 | Lab model card | |
| AI2D | 80.3% | #45 of 52 | Lab model card | |
| MathVista | 52.1% | #64 of 70 | Lab model card | |
| DocVQA | 86.5% | #35 of 36 | Lab model card | |
| EgoSchema | 63.2% | #12 of 12 | Lab model card | |
| VATEX | 63.0 | #5 of 5 | Lab model card |
Its lab's numbers put it at 120.4, number 342 of 404
The Self-Reported Index fits one score to the 19 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 118.5 to 122.3. How we compute it.
- HellaSwag 92.5
- GSM8K 91.7
- MGSM 88.7
- DocVQA 86.5
- BIG-Bench Hard 84
- MMLU 81.9
- ChartQA 81.3
- AI2D 80.3
- DROP 78.9
- Natural2Code 77.7
- TextVQA 73.5
- InfographicVQA 72.7
- HumanEval 71.9
- EgoSchema 63.2
- MATH 59.4
- MMMU 58.5
- ActivityNet-QA 56.7
- MathVista 52.1
- GPQA Diamond 41.5
Its lab's number next to a leaderboard
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.