Benchmark Almanac

Gemini 1.0 Ultra

Gemini 1.0 Ultra places in the top three on 2 benchmarks.

Google DeepMind Released December 6th, 2023 Family Gemini 1.0 The lab's model card Compare with others
19
Benchmarks with a score
0
Ranked first, out of five or more models
2
In the top three
Google DeepMind Dec 2023

Gemini 1.0 Ultra

Flagship

121.3 #336 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • TextVQA 82.3% #6 /17
  • MathVista 53% #63 /70
  • MMMU 59.4% #59 /136

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
MMLU 90% #14 of 131 Lab model card
HellaSwag 87.8% #15 of 53 Lab model card
BIG-Bench Hard 83.6% #29 of 73 Lab model card
DROP 82.4% #23 of 41 Lab model card
WMT23 74.4 #3 of 3 Lab model card

Math and science

BenchmarkScoreRankSource
GSM8K 94.4% #17 of 92 Lab model card
MATH 53.2% #64 of 99 Lab model card

Coding

BenchmarkScoreRankSource
CodeEditorBench 85.5% #2 of 7 Official leaderboard
HumanEval 74.4% #63 of 101 Lab model card
Natural2Code 74.9% #7 of 9 Lab model card

Vision, audio and robotics

BenchmarkScoreRankSource
TextVQA 82.3% #6 of 17 Lab model card
VQAv2 77.8% #3 of 8 Lab model card
MTVQA-Test 23.2% #7 of 18 Official leaderboard
MMMU 59.4% #59 of 136 Official leaderboard
InfographicVQA 80.3% #14 of 26 Lab model card
DocVQA 90.9% #28 of 36 Lab model card
ChartQA 80.8% #31 of 37 Lab model card
AI2D 79.5% #46 of 52 Lab model card
MathVista 53% #63 of 70 Lab model card

Its lab's numbers put it at 121.3, number 336 of 404

The Self-Reported Index fits one score to the 15 capability results Google DeepMind reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 118.4 to 124.2. How we compute it.

Its lab's numbers next to 2 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
MMMU 62.4% MMMU (val), Maj1@32 (pass@1 = 59.4%) 59.4% Leaderboard +3.0 pts
MathVista 53% MathVista (testmini) 53% Leaderboard Same

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Gemini: A Family of Highly Capable Multimodal Models (technical report, arXiv 2312.11805v1) Dec 6, 2023