Benchmark Almanac

MMVU

Qwen 3.8 Max 82.4% leads since August 2026
o1 75.5% leads since January 2025
Qwen 3.8 Max 82.4% leads since August 2026
In use

3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy.

Made by
Yale NLP
Released
January 2025
Kind
Video understanding
Level
University
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Strongest model tested
Kimi K3 #24 on the capability index
Last in a launch post
August 2026
Last model added
August 2026
Scores refreshed
November 12th, 2026

In the authors’ words

We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark for evaluating foundation models in video understanding. MMVU includes 3,000 expert-annotated questions spanning 27 subjects across four core disciplines: Science, Healthcare, Humanities & Social Sciences, and Engineering. Compared to prior benchmarks, MMVU features three key advancements. First, it challenges models to apply domain-specific knowledge and perform expert-level reasoning to analyze specialized-domain videos, moving beyond the basic visual perception typically assessed in current video benchmarks. Second, each example is annotated by human experts from scratch. We implement strict data quality controls to ensure the high quality of the dataset. Finally, each example is enriched with expert-annotated reasoning rationals and relevant domain knowledge, facilitating in-depth analysis. We conduct an extensive evaluation of 32 frontier multimodal foundation models on MMVU. The latest System-2-capable models, o1 and Gemini 2.0 Flash Thinking, achieve the highest performance among the tested models. However, they still fall short of matching human expertise. Through in-depth error analyses and case studies, we offer actionable insights for future advancements in expert-level, knowledge-intensive video understanding for specialized domains.

Abstract of MMVU: Measuring Expert-Level Multi-Discipline Video Understanding (arXiv:2501.12380, CC0)
26
Models scored in lab cards
22
Launch posts list it, from 7 labs
14
System cards report it, from 6 labs

Qwen 3.8 Max leads with 82.4% among models released by August 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

o1 leads with 75.5% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.

Qwen 3.8 Max leads with 82.4%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in August 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

26 models. Scroll sideways for the rest.

  1. 82.4 Qwen 3.8 Max
  2. 82.1 Kimi K3
  3. 80.5 GLM-5.3-Flash
  4. 80.4 Kimi K2.5
  5. 78.2 Seed2.0 Pro
  6. 76.7 Seed2.0 Lite (0428)
  7. 75.4 Qwen3.5 397B-A17B
  8. 74.7 Qwen3.5-122B-A10B
  9. 73.3 Qwen3.5-27B
  10. 73.1 Seed1.8
  11. 72.3 Qwen3.5-35B-A3B
  12. 72.2 ERNIE 5.0
  13. 71.7 Reka Edge
  14. 71.1 Qwen3-VL-235B-A22B
  15. 70.1 Seed1.5-VL
  16. 68.7 GLM-4.5V
  17. 68.4 GLM-4.6V
  18. 67.8 Qwen3.5-9B
  19. 67.5 Qwen3.5-Omni-Plus
  20. 67.1 LongCat-Flash-Omni
  21. 64.9 Qwen3.5-4B
  22. 62.9 Qwen2.5-VL-72B
  23. 59.4 GLM-4.1V-9B-Thinking
  24. 57.5 Kimi-VL-A3B-Thinking-2506
  25. 48.6 Qwen3.5-2B
  26. 34.3 Qwen3.5-0.8B
  1. 1 Qwen 3.8 Max 82.4%
  2. 2 Kimi K3 82.1%
  3. 3 GLM-5.3-Flash 80.5%
  4. 4 Kimi K2.5 80.4%
  5. 5 Seed2.0 Pro 78.2%
  6. 6 Seed2.0 Lite (0428) 76.7%
  7. 7 Qwen3.5 397B-A17B 75.4%
  8. 8 Qwen3.5-122B-A10B 74.7%
  9. 9 Qwen3.5-27B 73.3%
  10. 10 Seed1.8 73.1%
  11. 11 Qwen3.5-35B-A3B 72.3%
  12. 12 ERNIE 5.0 72.2%
  13. 13 Reka Edge 71.7%
  14. 14 Qwen3-VL-235B-A22B 71.1%
  15. 15 Seed1.5-VL 70.1%
  16. 16 GLM-4.5V 68.7%
  17. 17 GLM-4.6V 68.4%
  18. 18 Qwen3.5-9B 67.8%
  19. 19 Qwen3.5-Omni-Plus 67.5%
  20. 20 LongCat-Flash-Omni 67.1%
  21. 21 Qwen3.5-4B 64.9%
  22. 22 Qwen2.5-VL-72B 62.9%
  23. 23 GLM-4.1V-9B-Thinking 59.4%
  24. 24 Kimi-VL-A3B-Thinking-2506 57.5%
  25. 25 Qwen3.5-2B 48.6%
  26. 26 Qwen3.5-0.8B 34.3%

Source: Labs’ model cards

28 models. Scroll sideways for the rest.

  1. 75.5 o1
  2. 69.1 Gemini 2.0 Flash Thinking
  3. 67.4 GPT-4o
  4. 65.9 Gemini 2.0 Flash
  5. 65.4 Gemini 1.5 Pro
  6. 65.2 Claude 3.5 Sonnet (October 2024)
  7. 62.7 Grok 2 Vision (1212)
  8. 61.6 GPT-4o mini
  9. 58.8 Gemini 1.5 Flash
  10. 53.0 Qwen2-VL-72B-Instruct
  11. 52.1 DeepSeek-VL2
  12. 50.5 InternVL2.5-38B
  13. 49.3 Aria
  14. 47.1 Llama 3.2 90B
  15. 46.9 DeepSeek-VL2-Small
  16. 45.1 VideoChat-Flash-Qwen2-7B_res448
  17. 45.0 VideoLLAMA3-7B
  18. 42.1 Qwen2-VL-7B-Instruct
  19. 41.1 InternVL2.5-8B
  20. 38.9 Llama 3.2 11B
  21. 38.1 Phi-3.5-vision-instruct
  22. 37.9 LLaVA-OneVision 7B
  23. 36.5 Qwen2-VL-2B-Instruct
  24. 36.3 InternVL2-8B
  25. 35.3 Idefics3-8B-Llama3
  26. 34.4 VideoLLAMA2-7B
  27. 33.0 DeepSeek-VL2-Tiny
  28. 32.3 Pixtral 12B
  1. 1 o1 32 frames 75.5%
  2. 2 Gemini 2.0 Flash Thinking Jan 2025, 32 frames 69.1%
  3. 3 GPT-4o Aug 2024, 32 frames 67.4%
  4. 4 Gemini 2.0 Flash Dec 2024, 32 frames 65.9%
  5. 5 Gemini 1.5 Pro Sep 2024, 32 frames 65.4%
  6. 6 Claude 3.5 Sonnet (October 2024) 32 frames 65.2%
  7. 7 Grok 2 Vision (1212) 32 frames 62.7%
  8. 8 GPT-4o mini 32 frames 61.6%
  9. 9 Gemini 1.5 Flash Sep 2024, 32 frames 58.8%
  10. 10 Qwen2-VL-72B-Instruct 1 fps 53%
  11. 11 DeepSeek-VL2 2 frames 52.1%
  12. 12 InternVL2.5-38B 4 frames 50.5%
  13. 13 Aria 8 frames 49.3%
  14. 14 Llama 3.2 90B 8 frames 47.1%
  15. 15 DeepSeek-VL2-Small 2 frames 46.9%
  16. 16 VideoChat-Flash-Qwen2-7B_res448 1 fps 45.1%
  17. 17 VideoLLAMA3-7B 1 fps 45%
  18. 18 Qwen2-VL-7B-Instruct 1 fps 42.1%
  19. 19 InternVL2.5-8B 4 frames 41.1%
  20. 20 Llama 3.2 11B 8 frames 38.9%
  21. 21 Phi-3.5-vision-instruct 16 frames 38.1%
  22. 22 LLaVA-OneVision 7B 1 fps 37.9%
  23. 23 Qwen2-VL-2B-Instruct 1 fps 36.5%
  24. 24 InternVL2-8B 4 frames 36.3%
  25. 25 Idefics3-8B-Llama3 4 frames 35.3%
  26. 26 VideoLLAMA2-7B 1 fps 34.4%
  27. 27 DeepSeek-VL2-Tiny 2 frames 33%
  28. 28 Pixtral 12B 8 frames 32.3%

Source: Official MMVU leaderboard

54 models. Scroll sideways for the rest.

  1. 82.4 Qwen 3.8 Max
  2. 82.1 Kimi K3
  3. 80.5 GLM-5.3-Flash
  4. 80.4 Kimi K2.5
  5. 78.2 Seed2.0 Pro
  6. 76.7 Seed2.0 Lite (0428)
  7. 75.5 o1
  8. 75.4 Qwen3.5 397B-A17B
  9. 74.7 Qwen3.5-122B-A10B
  10. 73.3 Qwen3.5-27B
  11. 73.1 Seed1.8
  12. 72.3 Qwen3.5-35B-A3B
  13. 72.2 ERNIE 5.0
  14. 71.7 Reka Edge
  15. 71.1 Qwen3-VL-235B-A22B
  16. 70.1 Seed1.5-VL
  17. 69.1 Gemini 2.0 Flash Thinking
  18. 68.7 GLM-4.5V
  19. 68.4 GLM-4.6V
  20. 67.8 Qwen3.5-9B
  21. 67.5 Qwen3.5-Omni-Plus
  22. 67.4 GPT-4o
  23. 67.1 LongCat-Flash-Omni
  24. 65.9 Gemini 2.0 Flash
  25. 65.4 Gemini 1.5 Pro
  26. 65.2 Claude 3.5 Sonnet (October 2024)
  27. 64.9 Qwen3.5-4B
  28. 62.9 Qwen2.5-VL-72B
  29. 62.7 Grok 2 Vision (1212)
  30. 61.6 GPT-4o mini
  31. 59.4 GLM-4.1V-9B-Thinking
  32. 58.8 Gemini 1.5 Flash
  33. 57.5 Kimi-VL-A3B-Thinking-2506
  34. 53.0 Qwen2-VL-72B-Instruct
  35. 52.1 DeepSeek-VL2
  36. 50.5 InternVL2.5-38B
  37. 49.3 Aria
  38. 48.6 Qwen3.5-2B
  39. 47.1 Llama 3.2 90B
  40. 46.9 DeepSeek-VL2-Small
  41. 45.1 VideoChat-Flash-Qwen2-7B_res448
  42. 45.0 VideoLLAMA3-7B
  43. 42.1 Qwen2-VL-7B-Instruct
  44. 41.1 InternVL2.5-8B
  45. 38.9 Llama 3.2 11B
  46. 38.1 Phi-3.5-vision-instruct
  47. 37.9 LLaVA-OneVision 7B
  48. 36.5 Qwen2-VL-2B-Instruct
  49. 36.3 InternVL2-8B
  50. 35.3 Idefics3-8B-Llama3
  51. 34.4 VideoLLAMA2-7B
  52. 34.3 Qwen3.5-0.8B
  53. 33.0 DeepSeek-VL2-Tiny
  54. 32.3 Pixtral 12B
  1. 1 Qwen 3.8 Max 82.4%
  2. 2 Kimi K3 82.1%
  3. 3 GLM-5.3-Flash 80.5%
  4. 4 Kimi K2.5 80.4%
  5. 5 Seed2.0 Pro 78.2%
  6. 6 Seed2.0 Lite (0428) 76.7%
  7. 7 o1 32 frames 75.5%
  8. 8 Qwen3.5 397B-A17B 75.4%
  9. 9 Qwen3.5-122B-A10B 74.7%
  10. 10 Qwen3.5-27B 73.3%
  11. 11 Seed1.8 73.1%
  12. 12 Qwen3.5-35B-A3B 72.3%
  13. 13 ERNIE 5.0 72.2%
  14. 14 Reka Edge 71.7%
  15. 15 Qwen3-VL-235B-A22B 71.1%
  16. 16 Seed1.5-VL 70.1%
  17. 17 Gemini 2.0 Flash Thinking Jan 2025, 32 frames 69.1%
  18. 18 GLM-4.5V 68.7%
  19. 19 GLM-4.6V 68.4%
  20. 20 Qwen3.5-9B 67.8%
  21. 21 Qwen3.5-Omni-Plus 67.5%
  22. 22 GPT-4o Aug 2024, 32 frames 67.4%
  23. 23 LongCat-Flash-Omni 67.1%
  24. 24 Gemini 2.0 Flash Dec 2024, 32 frames 65.9%
  25. 25 Gemini 1.5 Pro Sep 2024, 32 frames 65.4%
  26. 26 Claude 3.5 Sonnet (October 2024) 32 frames 65.2%
  27. 27 Qwen3.5-4B 64.9%
  28. 28 Qwen2.5-VL-72B 62.9%
  29. 29 Grok 2 Vision (1212) 32 frames 62.7%
  30. 30 GPT-4o mini 32 frames 61.6%
  31. 31 GLM-4.1V-9B-Thinking 59.4%
  32. 32 Gemini 1.5 Flash Sep 2024, 32 frames 58.8%
  33. 33 Kimi-VL-A3B-Thinking-2506 57.5%
  34. 34 Qwen2-VL-72B-Instruct 1 fps 53%
  35. 35 DeepSeek-VL2 2 frames 52.1%
  36. 36 InternVL2.5-38B 4 frames 50.5%
  37. 37 Aria 8 frames 49.3%
  38. 38 Qwen3.5-2B 48.6%
  39. 39 Llama 3.2 90B 8 frames 47.1%
  40. 40 DeepSeek-VL2-Small 2 frames 46.9%
  41. 41 VideoChat-Flash-Qwen2-7B_res448 1 fps 45.1%
  42. 42 VideoLLAMA3-7B 1 fps 45%
  43. 43 Qwen2-VL-7B-Instruct 1 fps 42.1%
  44. 44 InternVL2.5-8B 4 frames 41.1%
  45. 45 Llama 3.2 11B 8 frames 38.9%
  46. 46 Phi-3.5-vision-instruct 16 frames 38.1%
  47. 47 LLaVA-OneVision 7B 1 fps 37.9%
  48. 48 Qwen2-VL-2B-Instruct 1 fps 36.5%
  49. 49 InternVL2-8B 4 frames 36.3%
  50. 50 Idefics3-8B-Llama3 4 frames 35.3%
  51. 51 VideoLLAMA2-7B 1 fps 34.4%
  52. 52 Qwen3.5-0.8B 34.3%
  53. 53 DeepSeek-VL2-Tiny 2 frames 33%
  54. 54 Pixtral 12B 8 frames 32.3%

Sources: Labs’ model cards and official MMVU leaderboard

#ModelScoreSettingDateSource
1 Qwen 3.8 Max 82.4% Aug 2026 Lab model card
2 Kimi K3 82.1% Jul 2026 Lab model card
3 GLM-5.3-Flash 80.5% Aug 2026 Lab model card
4 Kimi K2.5 80.4% Jan 2026 Lab model card
5 Seed2.0 Pro 78.2% Feb 2026 Lab model card
6 Seed2.0 Lite (0428) 76.7% Apr 2026 Lab model card
7 Qwen3.5 397B-A17B 75.4% Feb 2026 Lab model card
8 Qwen3.5-122B-A10B 74.7% Feb 2026 Lab model card
9 Qwen3.5-27B 73.3% Feb 2026 Lab model card
10 Seed1.8 73.1% Dec 2025 Lab model card
11 Qwen3.5-35B-A3B 72.3% Feb 2026 Lab model card
12 ERNIE 5.0 72.2% Feb 2026 Lab model card
13 Reka Edge 71.7% Mar 2026 Lab model card
14 Qwen3-VL-235B-A22B 71.1% Nov 2025 Lab model card
15 Seed1.5-VL 70.1% May 2025 Lab model card
16 GLM-4.5V 68.7% Aug 2025 Lab model card
17 GLM-4.6V 68.4% Jan 2026 Lab model card
18 Qwen3.5-9B 67.8% Mar 2026 Lab model card
19 Qwen3.5-Omni-Plus 67.5% Mar 2026 Lab model card
20 LongCat-Flash-Omni 67.1% Oct 2025 Lab model card
21 Qwen3.5-4B 64.9% Mar 2026 Lab model card
22 Qwen2.5-VL-72B 62.9% Feb 2025 Lab model card
23 GLM-4.1V-9B-Thinking 59.4% Jul 2025 Lab model card
24 Kimi-VL-A3B-Thinking-2506 57.5% Jun 2025 Lab model card
25 Qwen3.5-2B 48.6% Mar 2026 Lab model card
26 Qwen3.5-0.8B 34.3% Mar 2026 Lab model card
#ModelScoreSettingDateSource
1 o1 75.5% 32 frames Dec 2024 Official leaderboard
2 Gemini 2.0 Flash Thinking Jan 2025 69.1% 32 frames Jan 2025 Official leaderboard
3 GPT-4o Aug 2024 67.4% 32 frames Aug 2024 Official leaderboard
4 Gemini 2.0 Flash Dec 2024 65.9% 32 frames Dec 2024 Official leaderboard
5 Gemini 1.5 Pro Sep 2024 65.4% 32 frames Sep 2024 Official leaderboard
6 Claude 3.5 Sonnet (October 2024) 65.2% 32 frames Oct 2024 Official leaderboard
7 Grok 2 Vision (1212) 62.7% 32 frames Dec 2024 Official leaderboard
8 GPT-4o mini 61.6% 32 frames Jul 2024 Official leaderboard
9 Gemini 1.5 Flash Sep 2024 58.8% 32 frames Aug 2024 Official leaderboard
10 Qwen2-VL-72B-Instruct 53% 1 fps Aug 2024 Official leaderboard
11 DeepSeek-VL2 52.1% 2 frames Dec 2024 Official leaderboard
12 InternVL2.5-38B 50.5% 4 frames Dec 2024 Official leaderboard
13 Aria 49.3% 8 frames Sep 2024 Official leaderboard
14 Llama 3.2 90B 47.1% 8 frames Sep 2024 Official leaderboard
15 DeepSeek-VL2-Small 46.9% 2 frames Dec 2024 Official leaderboard
16 VideoChat-Flash-Qwen2-7B_res448 45.1% 1 fps Jan 2025 Official leaderboard
17 VideoLLAMA3-7B 45% 1 fps Jan 2024 Official leaderboard
18 Qwen2-VL-7B-Instruct 42.1% 1 fps Aug 2024 Official leaderboard
19 InternVL2.5-8B 41.1% 4 frames Dec 2024 Official leaderboard
20 Llama 3.2 11B 38.9% 8 frames Sep 2024 Official leaderboard
21 Phi-3.5-vision-instruct 38.1% 16 frames Aug 2024 Official leaderboard
22 LLaVA-OneVision 7B 37.9% 1 fps Aug 2024 Official leaderboard
23 Qwen2-VL-2B-Instruct 36.5% 1 fps Aug 2024 Official leaderboard
24 InternVL2-8B 36.3% 4 frames Jul 2024 Official leaderboard
25 Idefics3-8B-Llama3 35.3% 4 frames Aug 2024 Official leaderboard
26 VideoLLAMA2-7B 34.4% 1 fps Jun 2024 Official leaderboard
27 DeepSeek-VL2-Tiny 33% 2 frames Dec 2024 Official leaderboard
28 Pixtral 12B 32.3% 8 frames Sep 2024 Official leaderboard
#ModelScoreSettingDateSource
1 Qwen 3.8 Max 82.4% Aug 2026 Lab model card
2 Kimi K3 82.1% Jul 2026 Lab model card
3 GLM-5.3-Flash 80.5% Aug 2026 Lab model card
4 Kimi K2.5 80.4% Jan 2026 Lab model card
5 Seed2.0 Pro 78.2% Feb 2026 Lab model card
6 Seed2.0 Lite (0428) 76.7% Apr 2026 Lab model card
7 o1 75.5% 32 frames Dec 2024 Official leaderboard
8 Qwen3.5 397B-A17B 75.4% Feb 2026 Lab model card
9 Qwen3.5-122B-A10B 74.7% Feb 2026 Lab model card
10 Qwen3.5-27B 73.3% Feb 2026 Lab model card
11 Seed1.8 73.1% Dec 2025 Lab model card
12 Qwen3.5-35B-A3B 72.3% Feb 2026 Lab model card
13 ERNIE 5.0 72.2% Feb 2026 Lab model card
14 Reka Edge 71.7% Mar 2026 Lab model card
15 Qwen3-VL-235B-A22B 71.1% Nov 2025 Lab model card
16 Seed1.5-VL 70.1% May 2025 Lab model card
17 Gemini 2.0 Flash Thinking Jan 2025 69.1% 32 frames Jan 2025 Official leaderboard
18 GLM-4.5V 68.7% Aug 2025 Lab model card
19 GLM-4.6V 68.4% Jan 2026 Lab model card
20 Qwen3.5-9B 67.8% Mar 2026 Lab model card
21 Qwen3.5-Omni-Plus 67.5% Mar 2026 Lab model card
22 GPT-4o Aug 2024 67.4% 32 frames Aug 2024 Official leaderboard
23 LongCat-Flash-Omni 67.1% Oct 2025 Lab model card
24 Gemini 2.0 Flash Dec 2024 65.9% 32 frames Dec 2024 Official leaderboard
25 Gemini 1.5 Pro Sep 2024 65.4% 32 frames Sep 2024 Official leaderboard
26 Claude 3.5 Sonnet (October 2024) 65.2% 32 frames Oct 2024 Official leaderboard
27 Qwen3.5-4B 64.9% Mar 2026 Lab model card
28 Qwen2.5-VL-72B 62.9% Feb 2025 Lab model card
29 Grok 2 Vision (1212) 62.7% 32 frames Dec 2024 Official leaderboard
30 GPT-4o mini 61.6% 32 frames Jul 2024 Official leaderboard
31 GLM-4.1V-9B-Thinking 59.4% Jul 2025 Lab model card
32 Gemini 1.5 Flash Sep 2024 58.8% 32 frames Aug 2024 Official leaderboard
33 Kimi-VL-A3B-Thinking-2506 57.5% Jun 2025 Lab model card
34 Qwen2-VL-72B-Instruct 53% 1 fps Aug 2024 Official leaderboard
35 DeepSeek-VL2 52.1% 2 frames Dec 2024 Official leaderboard
36 InternVL2.5-38B 50.5% 4 frames Dec 2024 Official leaderboard
37 Aria 49.3% 8 frames Sep 2024 Official leaderboard
38 Qwen3.5-2B 48.6% Mar 2026 Lab model card
39 Llama 3.2 90B 47.1% 8 frames Sep 2024 Official leaderboard
40 DeepSeek-VL2-Small 46.9% 2 frames Dec 2024 Official leaderboard
41 VideoChat-Flash-Qwen2-7B_res448 45.1% 1 fps Jan 2025 Official leaderboard
42 VideoLLAMA3-7B 45% 1 fps Jan 2024 Official leaderboard
43 Qwen2-VL-7B-Instruct 42.1% 1 fps Aug 2024 Official leaderboard
44 InternVL2.5-8B 41.1% 4 frames Dec 2024 Official leaderboard
45 Llama 3.2 11B 38.9% 8 frames Sep 2024 Official leaderboard
46 Phi-3.5-vision-instruct 38.1% 16 frames Aug 2024 Official leaderboard
47 LLaVA-OneVision 7B 37.9% 1 fps Aug 2024 Official leaderboard
48 Qwen2-VL-2B-Instruct 36.5% 1 fps Aug 2024 Official leaderboard
49 InternVL2-8B 36.3% 4 frames Jul 2024 Official leaderboard
50 Idefics3-8B-Llama3 35.3% 4 frames Aug 2024 Official leaderboard
51 VideoLLAMA2-7B 34.4% 1 fps Jun 2024 Official leaderboard
52 Qwen3.5-0.8B 34.3% Mar 2026 Lab model card
53 DeepSeek-VL2-Tiny 33% 2 frames Dec 2024 Official leaderboard
54 Pixtral 12B 32.3% 8 frames Sep 2024 Official leaderboard

The benchmark's own leaderboard has 36 entries, 28 of them models we track. Switch the scores above to rank them together.

7 models have held number one in labs’ own numbers since its release in January 2025 Number one over time 4 models have held number one with claims included since its release in January 2025

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Qwen ByteDance Seed Moonshot AI
Own leaderboard Not tested since Jan 2025
o1 75.5%
Lab claims Not tested since Aug 2026
Qwen 3.8 Max 82.4%

Sources: Official MMVU leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Qwen 3.8 Max Kimi K2.5 o1 GPT-4o GPT-4o mini VideoLLAMA3-7B 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Qwen 3.8 Max o1

Sources: Official MMVU leaderboard and labs’ model cards

3 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

ByteDance Seed
Moonshot AI
Z.ai
Meituan
Baidu
Qwen
Reka
DateLaunch postScoreAs the post writes it
Aug 26, 2026 GLM-5.3-Flash 80.5 MMVU
Aug 3, 2026 Qwen 3.8 Max 82.4 MMVU
Jul 16, 2026 Kimi K3 82.1 MMVU
Apr 28, 2026 Seed2.0 Lite (0428) 76.7 MMVU
Mar 30, 2026 Qwen3.5-Omni-Plus 67.5 MMVU
Mar 11, 2026 Reka Edge 71.7 MMVU
Mar 2, 2026 Qwen3.5-0.8B 34.3 MMVU 34.3/-- (with CI / without CI)
Mar 2, 2026 Qwen3.5-2B 48.6 MMVU 48.6/-- (with CI / without CI)
Mar 2, 2026 Qwen3.5-4B 64.9 MMVU
Mar 2, 2026 Qwen3.5-9B 67.8 MMVU
Feb 24, 2026 Qwen3.5-35B-A3B 72.3 MMVU
Feb 24, 2026 Qwen3.5-27B 73.3 MMVU
Feb 24, 2026 Qwen3.5-122B-A10B 74.7 MMVU
Feb 16, 2026 Qwen3.5 397B-A17B 75.4 MMVU
Feb 6, 2026 ERNIE 5.0 72.2 MMVU
Jan 27, 2026 Kimi K2.5 80.4 MMVU
Dec 18, 2025 Seed1.8 73.1 MMVU
Oct 31, 2025 LongCat-Flash-Omni 67.1 MMVU
Aug 11, 2025 GLM-4.5V 68.7 MMVU
Jul 1, 2025 GLM-4.1V-9B-Thinking 59.4 MMVU
Jun 21, 2025 Kimi-VL-A3B-Thinking-2506 57.5 MMVU (Pass@1)
May 12, 2025 Seed1.5-VL 70.1 MMVU

14 system cards report it, from 6 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Jul 27, 2026 Kimi K3 technical report 82.1 MMVU (Table 2) Capabilities
Apr 17, 2026 Qwen3.5-Omni-Plus technical report 67.5 MMVU (Table 6) Capabilities
Mar 21, 2026 Seed1.8 model card 73.1 MMVU (Table 3) Capabilities
Feb 14, 2026 Seed2.0 Pro model card 78.2 MMVU (Table 9) Capabilities
Feb 4, 2026 ERNIE 5.0 technical report 72.2 MMVU (Table 3) Capabilities
Jan 30, 2026 Kimi K2.5 technical report 80.4 MMVU (Table 4) Capabilities
Jan 1, 2026 GLM-4.6V technical report 68.4 MMVU, GLM-4.6V thinking (Table 2) Capabilities
Nov 26, 2025 Qwen3-VL-235B-A22B technical report 71.1 MMVU (Table 2) Capabilities
Oct 31, 2025 LongCat-Flash-Omni technical report 67.1 MMVU (Table 6) Capabilities
Aug 13, 2025 GLM-4.5V technical report 68.7 MMVU, GLM-4.5V thinking (Table 2) Capabilities
Jul 1, 2025 GLM-4.1V-9B-Thinking technical report 59.4 MMVU, GLM-4.1V-9B-Thinking thinking (Table 2) Capabilities
Jun 23, 2025 Kimi-VL-A3B-Thinking-2506 technical report 57.5 MMVU val (Pass@1) (Table 5) Capabilities
May 11, 2025 Seed1.5-VL technical report 70.1 MMVU, thinking (Table 7). Non-thinking 70.1 Capabilities
Feb 19, 2025 Qwen2.5-VL-72B technical report 62.9 MMVU val (Table 8) Capabilities

More in Audio and video

All 232 in Audio and video
  • Video-MMMU In use 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. Qwen 3.8 Max 88.7%
  • LVBench In use Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. Gemini 4 Argon 91.7%
  • MLVU In use Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. Qwen 3.8 Max 90.8%
  • FLEURS In use Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. Phi-4-multimodal 4.00%
  • VoiceBench In use Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. Qwen3.5-Omni-Plus 93.1%
  • TVBench In use Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. Qwen 3.8 Max 81.9%