Benchmark Almanac

VisionArena-Bench

Gemini 1.5 Pro Sep 2024 88 leads since December 2024
Not in model cards

Models answer visual questions from Arena users. A vision language model judges their answers to approximate human preference rankings.

The offline benchmark contains 500 prompts.

Made by
Arena
Released
December 2024
Kind
General visual questions
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Measures
Preference
Language
English
In and out
Text, Images → Text
Interaction
Single turn
Items from
Real work and use
Refresh
One fixed set
Run by
Anyone
Based on
Vision Arena
In views
Vision
Strongest model tested
Gemini 1.5 Pro Sep 2024 #157 on the capability index
Last model added
October 2024
Scores refreshed
November 20th, 2026
16
Models scored in its paper

Gemini 1.5 Pro leads with 88 among models released by October 2024

Best score per model from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

The newest model with a score here came out in October 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

Source: The VisionArena-Bench paper

#ModelScore ±SettingDateSource
1 Gemini 1.5 Pro Sep 2024 88 ±1.1 Baseline: claude-3-sonnet-20240229 Sep 2024 Official leaderboard
2 GPT-4o May 2024 87 85.7 to 88.1 Baseline: claude-3-sonnet-20240229 May 2024 Official leaderboard
3 Claude 3.5 Sonnet 86 85 to 87.5 Baseline: claude-3-sonnet-20240229 Jun 2024 Official leaderboard
4 Gemini 1.5 Flash Sep 2024 84 81.8 to 84.6 Baseline: claude-3-sonnet-20240229 Aug 2024 Official leaderboard
5 GPT-4 Turbo Apr 2024 81 79.7 to 82.3 Baseline: claude-3-sonnet-20240229 Apr 2024 Official leaderboard
6 Gemini 1.5 Pro May 2024 75 73.6 to 76.7 Baseline: claude-3-sonnet-20240229 May 2024 Official leaderboard
7 GPT-4o mini 73 71.7 to 74.4 Baseline: claude-3-sonnet-20240229 Jul 2024 Official leaderboard
8 Gemini 1.5 Flash 8B 65 63.2 to 67.1 Baseline: claude-3-sonnet-20240229 Oct 2024 Official leaderboard
9 Gemini 1.5 Flash May 2024 58 56.5 to 59.9 Baseline: claude-3-sonnet-20240229 May 2024 Official leaderboard
10 InternVL2-26B 54 ±1.7 Baseline: claude-3-sonnet-20240229 Jul 2024 Official leaderboard
11 Claude 3 Opus 52 50 to 53.7 Baseline: claude-3-sonnet-20240229 Feb 2024 Official leaderboard
12 Claude 3 Sonnet 50 Baseline: claude-3-sonnet-20240229 Feb 2024 Official leaderboard
13 Reka Core 38 36 to 39.6 Baseline: claude-3-sonnet-20240229 Apr 2024 Official leaderboard
14 Llama 3.2 11B 33 30.9 to 34.4 Baseline: claude-3-sonnet-20240229 Sep 2024 Official leaderboard
15 Claude 3 Haiku 31 28.4 to 32.3 Baseline: claude-3-sonnet-20240229 Mar 2024 Official leaderboard
16 InternVL2-4B 20 17.7 to 20.9 Baseline: claude-3-sonnet-20240229 Jul 2024 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Gemini 1.5 Pro
Baseline: G… Baseline: c…
88
GPT-4o
Baseline: G… Baseline: c…
87
Claude 3.5 Sonnet
Baseline: G… Baseline: c…
86
Gemini 1.5 Flash
Baseline: G… Baseline: c…
84
GPT-4 Turbo
Baseline: G… Baseline: c…
81
Gemini 1.5 Pro
Baseline: G… Baseline: c…
75
GPT-4o mini
Baseline: G… Baseline: c…
73
Gemini 1.5 Flash 8B
Baseline: G… Baseline: c…
65
Gemini 1.5 Flash
Baseline: G… Baseline: c…
58
InternVL2-26B
Baseline: G… Baseline: c…
54
Claude 3 Opus
Baseline: G… Baseline: c…
52
Claude 3 Sonnet
Baseline: G… Baseline: c…
50

Source: The VisionArena-Bench paper

ModelEffortHarnessScoreCost per taskOutput tokens
Gemini 1.5 Pro Sep 2024 Baseline: claude-3-sonnet-20240229 88 329
GPT-4o May 2024 Baseline: claude-3-sonnet-20240229 87 316
Claude 3.5 Sonnet Baseline: claude-3-sonnet-20240229 86 262
Gemini 1.5 Flash Sep 2024 Baseline: claude-3-sonnet-20240229 84 367
GPT-4 Turbo Apr 2024 Baseline: claude-3-sonnet-20240229 81 258
Gemini 1.5 Pro May 2024 Baseline: claude-3-sonnet-20240229 75 261
GPT-4o mini Baseline: claude-3-sonnet-20240229 73 224
GPT-4o May 2024 Baseline: GPT-4-Turbo 68 316
Gemini 1.5 Pro Sep 2024 Baseline: GPT-4-Turbo 66 329
Gemini 1.5 Flash 8B Baseline: claude-3-sonnet-20240229 65 354
Gemini 1.5 Flash Sep 2024 Baseline: GPT-4-Turbo 60 367
Gemini 1.5 Flash May 2024 Baseline: claude-3-sonnet-20240229 58 271
Claude 3.5 Sonnet Baseline: GPT-4-Turbo 55 262
InternVL2-26B Baseline: claude-3-sonnet-20240229 54 515
Claude 3 Opus Baseline: claude-3-sonnet-20240229 52 201
Claude 3 Sonnet Baseline: claude-3-sonnet-20240229 50 205
GPT-4 Turbo Apr 2024 Baseline: GPT-4-Turbo 50 258
Gemini 1.5 Pro May 2024 Baseline: GPT-4-Turbo 46 261
GPT-4o mini Baseline: GPT-4-Turbo 40 224
Reka Core Baseline: claude-3-sonnet-20240229 38 252
Llama 3.2 11B Baseline: claude-3-sonnet-20240229 33 457
Claude 3 Haiku Baseline: claude-3-sonnet-20240229 31 155
Gemini 1.5 Flash 8B Baseline: GPT-4-Turbo 31 354
InternVL2-26B Baseline: GPT-4-Turbo 23 515
Gemini 1.5 Flash May 2024 Baseline: GPT-4-Turbo 23 271
InternVL2-4B Baseline: claude-3-sonnet-20240229 20 421
Claude 3 Opus Baseline: GPT-4-Turbo 19 201
Claude 3 Sonnet Baseline: GPT-4-Turbo 18 205
Reka Core Baseline: GPT-4-Turbo 16 252
Llama 3.2 11B Baseline: GPT-4-Turbo 11 457
Claude 3 Haiku Baseline: GPT-4-Turbo 10 155
InternVL2-4B Baseline: GPT-4-Turbo 7 421

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind
Benchmark paper Not tested since Dec 2024

Source: The VisionArena-Bench paper

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
In the benchmark’s paper
0 20 40 60 80 100 2024 0 20 40 60 80 100 2024

Source: Official VisionArena-Bench leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Vision

All 636 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. Qwen 3.6 Plus 86%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MathVista In use Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. Seed2.1 Pro 90.7%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%

Data audit

8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in October 2024, for Gemini 1.5 Flash 8B. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
  • Pass Independent runs Scores from the benchmark’s paper (16 entries).
  • Pass Models scored 16 models we track.
  • Warning Newest score The newest came 2 years ago, in October 2024, for Gemini 1.5 Flash 8B.
  • Note Frontier models None of the top 10 has a score. The strongest tested is Gemini 1.5 Pro (#157).
  • Note Headroom Scored as a rating, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on arxiv.org: 32 rows for models we track at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check