Gemini 1.5 Flash May 2024
Gemini 1.5 Flash places in the top three on Anthropic Red Team.
39
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Google DeepMind May 2024 Gemini 1.5 Flash
May 2024 snapshot Closed weights
120.4
#183 of 256
Epoch Capabilities Index
- GPQA Diamond 40.4% #186 /229
- CorpFin v2 28.6% #119 /119
- BigCodeBench-Hard 23.6% #51 /96
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| RepoQA | 90% | #4 of 22 | Official leaderboard | |
| BoolQ | 85.8% | #15 of 61 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| ThaiExam | 59.8% | #8 of 24 | Official leaderboard | |
| MMBench-CN v1.1 | 76.7% | #69 of 178 | Official leaderboard | |
| WildBench | 44.1 | #21 of 47 | Official leaderboard | |
| RewardBench | 80.5% | #28 of 48 | Official leaderboard | |
| LMArena (Elo) | 1,287 | #243 of 335 | Official leaderboard | |
| SEAL Multilingual | 980 | #15 of 19 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 82.4% | #16 of 70 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| MATH Level 5 | 25.1% | #70 of 94 | Epoch AI run | |
| GSM1k | 90.1% | #12 of 15 | Official leaderboard | |
| GPQA Diamond | 40.4% | #186 of 229 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 3.89% | #187 of 212 | Epoch AI run |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench | 49.3% | #25 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 23.6% | #51 of 96 | Official leaderboard | |
| SEAL Coding | 943 | #20 of 26 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GroundUI | 9.40% | #6 of 11 | Official leaderboard | |
| LisanBench | 218.0 | #86 of 133 | Official leaderboard | |
| LLM Chess | -367 | #106 of 123 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 68% | #32 of 71 | Official leaderboard | |
| CorpFin v2 | 28.6% | #119 of 119 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Video-MME | 75% | #6 of 26 | Official leaderboard | |
| LongVideoBench | 62.4% | #6 of 23 | Official leaderboard | |
| MMBench-EN v1.1 | 77.1% | #76 of 178 | Official leaderboard | |
| CharXiv | 33.9% | #35 of 77 | Official leaderboard | |
| TOMATO | 27.8% | #10 of 22 | Official leaderboard | |
| MathVista | 58.4% | #20 of 43 | Official leaderboard | |
| MMMU | 56.1% | #70 of 136 | Official leaderboard | |
| VHELM | 45.3 | #28 of 41 | Official leaderboard | |
| Vision Arena | 1,059 | #120 of 136 | Official leaderboard | |
| AHELM | 22.0 | #10 of 10 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Anthropic Red Team | 99.9% | #2 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 79.4% | #24 of 82 | Official leaderboard | |
| HELM Safety | 92.7 | #33 of 75 | Official leaderboard | |
| AgentDojo | 12.2% | #10 of 20 | Official leaderboard | |
| SEAL Adversarial Robustness | 14.0 | #4 of 8 | Official leaderboard | |
| OR-Bench | 84.3% | #20 of 28 | Official leaderboard | |
| SimpleSafetyTests | 97% | #57 of 75 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 66.7 | #26 of 71 | Official leaderboard |
Google DeepMind's models over time
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)