Benchmark Almanac

Gemini 2.0 Flash-Lite

Gemini 2.0 Flash-Lite places in the top three on 2 benchmarks.

Google DeepMind Released February 5th, 2025 Compare with others
40
Benchmarks with a score
0
Ranked first, out of five or more models
2
In the top three
Google DeepMind Feb 2025

Gemini 2.0 Flash-Lite

Closed weights

125.7 #312 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • LiveCodeBench v5 28.9% #44 /47
  • Bird-SQL 57.4% #6 /13
  • AV-SpeakerBench 51.4% #2 /5

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
Vectara Hallucination Leaderboard (HHEM) 1.20% #6 of 148 Official leaderboard
BIG-Bench Extra Hard 8.00% #3 of 11 Official leaderboard
BBEH micro average 19.7% #5 of 11 Official leaderboard
LMArena (Elo) 1,354 #174 of 335 Official leaderboard
IFEval 82.4% #33 of 60 Official leaderboard
FACTS Grounding 83.6% #9 of 15 Lab model card
Global MMLU Lite 78.2% #10 of 16 Lab model card
SimpleQA 21.7% #38 of 60 Lab model card
MMLU-Pro 71.6% #118 of 167 Lab model card
MRCR (Michelangelo, original) 58% #7 of 9 Lab model card
Phare 57.3% #60 of 62 Official leaderboard

Math and science

BenchmarkScoreRankSource
MATH 86.8% #14 of 99 Lab model card
HiddenMath 55.3% #6 of 11 Lab model card
GPQA Main 50% #37 of 60 Official leaderboard
GPQA Diamond 51.5% #240 of 302 Lab model card
IneqMath 1.50% #29 of 34 Official leaderboard
DTBench 52.5% #156 of 180 Official leaderboard

Coding

BenchmarkScoreRankSource
Bird-SQL 57.4% #6 of 13 Lab model card
BigCodeBench-Hard 25.4% #46 of 96 Official leaderboard
LiveCodeBench v5 28.9% #44 of 47 Lab model card

Agents and tools

BenchmarkScoreRankSource
Galileo Agent Leaderboard 0.93 #4 of 29 Official leaderboard
Berkeley Function Calling Leaderboard (BFCL) 56.4% #36 of 67 Official leaderboard
SnakeBench 21 #115 of 188 Official leaderboard
ToolComp 53.8% #20 of 32 Official leaderboard
LisanBench 157.3 #97 of 133 Official leaderboard
LLM Chess -812 #121 of 123 Official leaderboard

Vision, audio and robotics

BenchmarkScoreRankSource
MMMU 68% #39 of 136 Official leaderboard
VHELM 59.7 #16 of 41 Official leaderboard
AHELM 54.0 #4 of 10 Official leaderboard
AV-SpeakerBench 51.4% #2 of 5 Official leaderboard
CoVoST2 38.4 #8 of 14 Lab model card
Daily-Omni 61.3% #8 of 14 Official leaderboard
Vision Arena 1,136 #106 of 136 Official leaderboard
VISTA 32.7% #45 of 54 Official leaderboard
EgoSchema 67.2% #11 of 12 Lab model card

Safety and security

BenchmarkScoreRankSource
Anthropic Red Team 99.3% #25 of 75 Official leaderboard
AIR-Bench 2024 67.5% #38 of 82 Official leaderboard
HELM Safety 90.8 #42 of 75 Official leaderboard
SimpleSafetyTests 97.7% #50 of 75 Official leaderboard

Other

BenchmarkScoreRankSource
HELM Capabilities 64.2 #34 of 60 Official leaderboard

Its lab's numbers put it at 125.7, number 312 of 404

The Self-Reported Index fits one score to the 12 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 122.8 to 128.6. How we compute it.

Its lab's numbers next to 3 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
FACTS Grounding 83.6% FACTS Grounding 83.2% Leaderboard +0.4 pts
MMLU-Pro 71.6% MMLU-Pro 71.6% Leaderboard Same
MMMU 68% MMMU 68% Leaderboard Same

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Its system card

Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.