Benchmark Almanac

Open FlagEval VLM Leaderboard

Gemini 2.5 Pro Mar 2025 70.7 leads since March 2025
Not in model cards

Models answer questions about images. The leaderboard compares their scores across visual understanding tasks.

The publisher lists 52 model directories in the public leaderboard results.

Released
February 2025
Kind
General visual questions
Level
University
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Measures
Capability
In and out
Images, Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
In views
Vision
Strongest model tested
Gemini 2.5 Pro Mar 2025 #104 on the capability index
Last model added
April 2025
Scores refreshed
November 21st, 2026
51
Entries on its own leaderboard, 41 models we track

Gemini 2.5 Pro leads with 70.7 among models released by April 2025 Gemini 2.5 Pro leads with 70.7 among models released by April 2025

Best score per entry from the benchmark's own leaderboard, read on November 21st, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in April 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Gemini 2.5 Pro Mar 2025 70.7
  2. 2 Gemini 2.0 Pro 61.9
  3. 3 Gemini 2.0 Flash Feb 2025 60.5
  4. 4 Llama-4-maverick-instruct-basic 60.1
  5. 5 InternVL3-78B 57.8
  6. 6 Qwen2.5-VL-72B 57.6
  7. 7 Claude 3.7 Sonnet 57.5
  8. 8 Gemini 1.5 Pro Sep 2024 57.0
  9. 9 Claude 3.5 Sonnet 56.7
  10. 10 Qwen2.5-VL-32B 56.6
  11. 11 Doubao-Pro-Vision-32k-241028 56.3
  12. 12 Qwen-VL-Max 55.2
  13. 13 GPT-4o Nov 2024 55.1
  14. 14 Qwen2-VL-72B-Instruct 54.0
  15. 15 GPT-4o Aug 2024 52.7
  16. 16 InternVL3-8B 50.5
  17. 17 Mistral-Small-3.1-24B-Instruct-2503 49.9
  18. 18 Yi-Vision 49.8
  19. 19 Step-1V-32k 48.8
  20. 20 Pixtral-Large-Instruct-2411 48.8
  21. 21 Qwen2.5-VL-7B 48.4
  22. 22 InternVL2.5-26B 48.2
  23. 23 InternVL2-Llama3-76B 46.6
  24. 24 GLM-4V-Plus 46.6
  25. 25 LLaVA-OneVision 72B 46.3
  26. 26 Gemma 3 27B 46.1
  27. 27 InternVL2.5-8B 45.7
  28. 28 Qwen2-VL-7B-Instruct 44.9
  29. 29 Gemini 1.5 Flash Sep 2024 44.6
  30. 30 GPT-4o mini 44.5
  31. 31 NVLM-D-72B 44.0
  32. 32 Molmo-72B 43.2
  33. 33 MiniCPM-V-2_6 41.3
  34. 34 Llama 3.2 90B 41.2
  35. 35 InternVL2-8B 41.0
  36. 36 Claude 3 Opus 40.9
  37. 37 Aria 40.9
  38. 38 Phi-4-multimodal 38.8
  39. 39 Pixtral-12B-2409 37.9
  40. 40 Qwen2-VL-2B-Instruct 36.9
  41. 41 InternVL2.5-2B 35.7
  42. 42 Molmo-7B-D 35.3
  43. 43 LLaVA-OneVision 7B 34.6
  44. 44 Phi-3.5-vision-instruct 32.4
  45. 45 InternVL2-2B 31.4
  46. 46 Idefics3-8B-Llama3 30.9
  47. 47 XGen-MM-Instruct-Interleave-v1.5 29.1
  48. 48 LLaVA-OneVision 0.5B 26.2
  49. 49 Mono-InternVL-2B 24.9
  50. 50 Llama 3.2 11B 24.0
  51. 51 Janus-1.3B 21.7

Source: Official Open FlagEval VLM Leaderboard leaderboard

Source: Official Open FlagEval VLM Leaderboard leaderboard

#EntryScoreSettingDateSource
1 Gemini 2.5 Pro Mar 2025 70.7 Mar 2025 Official leaderboard
2 Gemini 2.0 Pro 61.9 Feb 2025 Official leaderboard
3 Gemini 2.0 Flash Feb 2025 60.5 Feb 2025 Official leaderboard
4 Llama-4-maverick-instruct-basic 60.1 Official leaderboard
5 InternVL3-78B 57.8 Apr 2025 Official leaderboard
6 Qwen2.5-VL-72B 57.6 Jan 2025 Official leaderboard
7 Claude 3.7 Sonnet 57.5 Feb 2025 Official leaderboard
8 Gemini 1.5 Pro Sep 2024 57.0 Sep 2024 Official leaderboard
9 Claude 3.5 Sonnet 56.7 Jun 2024 Official leaderboard
10 Qwen2.5-VL-32B 56.6 Mar 2025 Official leaderboard
11 Doubao-Pro-Vision-32k-241028 56.3 Official leaderboard
12 Qwen-VL-Max 55.2 Jan 2024 Official leaderboard
13 GPT-4o Nov 2024 55.1 Nov 2024 Official leaderboard
14 Qwen2-VL-72B-Instruct 54.0 Aug 2024 Official leaderboard
15 GPT-4o Aug 2024 52.7 Aug 2024 Official leaderboard
16 InternVL3-8B 50.5 Apr 2025 Official leaderboard
17 Mistral-Small-3.1-24B-Instruct-2503 49.9 Official leaderboard
18 Yi-Vision 49.8 Official leaderboard
19 Step-1V-32k 48.8 Official leaderboard
20 Pixtral-Large-Instruct-2411 48.8 Official leaderboard
21 Qwen2.5-VL-7B 48.4 Jan 2025 Official leaderboard
22 InternVL2.5-26B 48.2 Dec 2024 Official leaderboard
23 InternVL2-Llama3-76B 46.6 Jul 2024 Official leaderboard
24 GLM-4V-Plus 46.6 Official leaderboard
25 LLaVA-OneVision 72B 46.3 Aug 2024 Official leaderboard
26 Gemma 3 27B 46.1 Mar 2025 Official leaderboard
27 InternVL2.5-8B 45.7 Dec 2024 Official leaderboard
28 Qwen2-VL-7B-Instruct 44.9 Aug 2024 Official leaderboard
29 Gemini 1.5 Flash Sep 2024 44.6 Aug 2024 Official leaderboard
30 GPT-4o mini 44.5 Jul 2024 Official leaderboard
31 NVLM-D-72B 44.0 Sep 2024 Official leaderboard
32 Molmo-72B 43.2 Sep 2024 Official leaderboard
33 MiniCPM-V-2_6 41.3 Aug 2024 Official leaderboard
34 Llama 3.2 90B 41.2 Sep 2024 Official leaderboard
35 InternVL2-8B 41.0 Jul 2024 Official leaderboard
36 Claude 3 Opus 40.9 Feb 2024 Official leaderboard
37 Aria 40.9 Sep 2024 Official leaderboard
38 Phi-4-multimodal 38.8 Feb 2025 Official leaderboard
39 Pixtral-12B-2409 37.9 Official leaderboard
40 Qwen2-VL-2B-Instruct 36.9 Aug 2024 Official leaderboard
41 InternVL2.5-2B 35.7 Dec 2024 Official leaderboard
42 Molmo-7B-D 35.3 Sep 2024 Official leaderboard
43 LLaVA-OneVision 7B 34.6 Aug 2024 Official leaderboard
44 Phi-3.5-vision-instruct 32.4 Aug 2024 Official leaderboard
45 InternVL2-2B 31.4 Jul 2024 Official leaderboard
46 Idefics3-8B-Llama3 30.9 Aug 2024 Official leaderboard
47 XGen-MM-Instruct-Interleave-v1.5 29.1 Official leaderboard
48 LLaVA-OneVision 0.5B 26.2 Aug 2024 Official leaderboard
49 Mono-InternVL-2B 24.9 Official leaderboard
50 Llama 3.2 11B 24.0 Sep 2024 Official leaderboard
51 Janus-1.3B 21.7 Oct 2024 Official leaderboard
#ModelScoreSettingDateSource
1 Gemini 2.5 Pro Mar 2025 70.7 Mar 2025 Official leaderboard
2 Gemini 2.0 Pro 61.9 Feb 2025 Official leaderboard
3 Gemini 2.0 Flash Feb 2025 60.5 Feb 2025 Official leaderboard
4 InternVL3-78B 57.8 Apr 2025 Official leaderboard
5 Qwen2.5-VL-72B 57.6 Jan 2025 Official leaderboard
6 Claude 3.7 Sonnet 57.5 Feb 2025 Official leaderboard
7 Gemini 1.5 Pro Sep 2024 57.0 Sep 2024 Official leaderboard
8 Claude 3.5 Sonnet 56.7 Jun 2024 Official leaderboard
9 Qwen2.5-VL-32B 56.6 Mar 2025 Official leaderboard
10 Qwen-VL-Max 55.2 Jan 2024 Official leaderboard
11 GPT-4o Nov 2024 55.1 Nov 2024 Official leaderboard
12 Qwen2-VL-72B-Instruct 54.0 Aug 2024 Official leaderboard
13 GPT-4o Aug 2024 52.7 Aug 2024 Official leaderboard
14 InternVL3-8B 50.5 Apr 2025 Official leaderboard
15 Qwen2.5-VL-7B 48.4 Jan 2025 Official leaderboard
16 InternVL2.5-26B 48.2 Dec 2024 Official leaderboard
17 InternVL2-Llama3-76B 46.6 Jul 2024 Official leaderboard
18 LLaVA-OneVision 72B 46.3 Aug 2024 Official leaderboard
19 Gemma 3 27B 46.1 Mar 2025 Official leaderboard
20 InternVL2.5-8B 45.7 Dec 2024 Official leaderboard
21 Qwen2-VL-7B-Instruct 44.9 Aug 2024 Official leaderboard
22 Gemini 1.5 Flash Sep 2024 44.6 Aug 2024 Official leaderboard
23 GPT-4o mini 44.5 Jul 2024 Official leaderboard
24 NVLM-D-72B 44.0 Sep 2024 Official leaderboard
25 Molmo-72B 43.2 Sep 2024 Official leaderboard
26 MiniCPM-V-2_6 41.3 Aug 2024 Official leaderboard
27 Llama 3.2 90B 41.2 Sep 2024 Official leaderboard
28 InternVL2-8B 41.0 Jul 2024 Official leaderboard
29 Claude 3 Opus 40.9 Feb 2024 Official leaderboard
30 Aria 40.9 Sep 2024 Official leaderboard
31 Phi-4-multimodal 38.8 Feb 2025 Official leaderboard
32 Qwen2-VL-2B-Instruct 36.9 Aug 2024 Official leaderboard
33 InternVL2.5-2B 35.7 Dec 2024 Official leaderboard
34 Molmo-7B-D 35.3 Sep 2024 Official leaderboard
35 LLaVA-OneVision 7B 34.6 Aug 2024 Official leaderboard
36 Phi-3.5-vision-instruct 32.4 Aug 2024 Official leaderboard
37 InternVL2-2B 31.4 Jul 2024 Official leaderboard
38 Idefics3-8B-Llama3 30.9 Aug 2024 Official leaderboard
39 LLaVA-OneVision 0.5B 26.2 Aug 2024 Official leaderboard
40 Llama 3.2 11B 24.0 Sep 2024 Official leaderboard
41 Janus-1.3B 21.7 Oct 2024 Official leaderboard

Rankings by part

We read 12 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as CMMMU, MMMU and MMMU_Pro_standard. Gemini 2.5 Pro leads every one of them. Arrows show how far a model moved from its overall rank among the models we track.

CMMMU

41 models we track have a score here.

  1. 1 Gemini 2.5 Pro Mar 2025 71.2
  2. 2 Gemini 2.0 Pro 61.1
  3. 3 Qwen2.5-VL-72B ↑2 56.6
  4. 4 Gemini 2.0 Flash Feb 2025 ↓1 54.7
  5. 5 InternVL3-78B ↓1 52.9
  6. 6 Qwen2-VL-72B-Instruct ↑6 52.8
  7. 7 Claude 3.7 Sonnet ↓1 51.1
  8. 8 Claude 3.5 Sonnet 50.8
  9. 9 GPT-4o Aug 2024 ↑4 50.8
  10. 10 NVLM-D-72B ↑14 50.2

3 models have held number one on the official leaderboard since its release in February 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen Google DeepMind
Own leaderboard Not tested since Apr 2025

Source: Official Open FlagEval VLM Leaderboard leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard
20.0 40.0 60.0 80.0 2024 2025 20.0 40.0 60.0 80.0 2024 2025

Source: Official Open FlagEval VLM Leaderboard leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Vision

All 636 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. Qwen 3.6 Plus 86%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MathVista In use Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. Seed2.1 Pro 90.7%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%

Data audit

8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 19 months ago, in April 2025, for InternVL3-78B. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
  • Pass Independent runs Scores from the official leaderboard (51 entries).
  • Pass Models scored 41 models we track, plus 10 other entries on its leaderboard.
  • Warning Newest score The newest came 19 months ago, in April 2025, for InternVL3-78B.
  • Note Frontier models None of the top 10 has a score. The strongest tested is Gemini 2.5 Pro (#104).
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on huggingface.co: 42 rows for models we track and 10 other entries at the last read.
  • Pass Last read November 21st, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check