Benchmark Almanac

MathVerse

Seed2.1 Pro 89.7% leads since June 2026
GPT-4V 39.4% leads since March 2024
Seed2.1 Pro 89.7% leads since June 2026
Fading

The official leaderboard hasn’t scored a model released after LLaVA-v1.6-34B (January 2024). Labs report their own numbers for 12 newer models, such as Seed2.1 Pro at 89.7%.

Models solve diagram-based math problems shown in six versions that move information between the text and the image. It tests whether models read the diagram.

Labs usually report the testmini split.

Released
March 2024
Kind
Visual math and diagrams
Level
School
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Measures
Capability
Subject
Mathematics
Language
English
In and out
Text, Images → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
In views
Vision
Last in a launch post
October 2026, dropped after it saturated
Last model added
October 2026
Scores refreshed
November 19th, 2026

In the authors’ words

The remarkable progress of Multi-modal Large Language Models (MLLMs) has garnered unparalleled attention, due to their superior performance in visual contexts. However, their capabilities in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows MathVerse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning. In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark may provide unique insights to guide the future development of MLLMs. Project page: https://mathverse-cuhk.github.io

Abstract of MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (arXiv:2403.14624, CC0)
13
Models scored in lab cards
7
Launch posts list it, from 6 labs
11
System cards report it, from 8 labs

Seed2.1 Pro leads with 89.7%

Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.

GPT-4V leads with 39.4% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.

Seed2.1 Pro leads with 89.7%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

  1. 1 Seed2.1 Pro 89.7%
  2. 2 MiniCPM-V-4.7-35B-A3B 86.5%
  3. 3 ERNIE 5.0 85.1%
  4. 4 Qwen3-VL-235B-A22B 85%
  5. 5 ERNIE-4.5-VL-28B-A3B-Thinking 81%
  6. 6 STEP3-VL-10B 75.7%
  7. 7 GLM-4.6V 75.4%
  8. 8 GLM-4.5V 72.1%
  9. 9 InternVL3.5-241B-A28B 68.5%
  10. 10 GLM-4.1V-9B-Thinking 68.4%
  11. 11 Nemotron Nano 12B v2 VL 58.2%
  12. 12 Qwen2.5-VL-72B 57.6%
  13. 13 Phi-4-Reasoning-Vision-15B 44.9%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Seed2.1 Pro 89.7% Jun 2026 Lab model card
2 MiniCPM-V-4.7-35B-A3B 86.5% Oct 2026 Lab model card
3 ERNIE 5.0 85.1% Feb 2026 Lab model card
4 Qwen3-VL-235B-A22B 85% Sep 2025 Lab model card
5 ERNIE-4.5-VL-28B-A3B-Thinking 81% Nov 2025 Lab model card
6 STEP3-VL-10B 75.7% Jan 2026 Lab model card
7 GLM-4.6V 75.4% Jan 2026 Lab model card
8 GLM-4.5V 72.1% Aug 2025 Lab model card
9 InternVL3.5-241B-A28B 68.5% Aug 2025 Lab model card
10 GLM-4.1V-9B-Thinking 68.4% Jul 2025 Lab model card
11 Nemotron Nano 12B v2 VL 58.2% Nov 2025 Lab model card
12 Qwen2.5-VL-72B 57.6% Feb 2025 Lab model card
13 Phi-4-Reasoning-Vision-15B 44.9% Mar 2026 Lab model card
#ModelScoreSettingDateSource
1 GPT-4V 39.4% Nov 2023 Official leaderboard
2 Qwen-VL-Max 25.3% Jan 2024 Official leaderboard
3 LLaVA-v1.6-34B 23.8% Jan 2024 Official leaderboard
4 Gemini 1.0 Pro 23.5% Dec 2023 Official leaderboard
5 InternLM-XComposer2-VL-7B 16.5% Jan 2024 Official leaderboard
6 llava-v1.6-vicuna-13b 15.6% Jan 2024 Official leaderboard
7 ShareGPT4V-13B 13.1% Dec 2023 Official leaderboard
8 Qwen-VL-Plus 11.8% Jan 2024 Official leaderboard
9 LLaVA-v1.5-13B 7.60% Oct 2023 Official leaderboard
#ModelScoreSettingDateSource
1 Seed2.1 Pro 89.7% Jun 2026 Lab model card
2 MiniCPM-V-4.7-35B-A3B 86.5% Oct 2026 Lab model card
3 ERNIE 5.0 85.1% Feb 2026 Lab model card
4 Qwen3-VL-235B-A22B 85% Sep 2025 Lab model card
5 ERNIE-4.5-VL-28B-A3B-Thinking 81% Nov 2025 Lab model card
6 STEP3-VL-10B 75.7% Jan 2026 Lab model card
7 GLM-4.6V 75.4% Jan 2026 Lab model card
8 GLM-4.5V 72.1% Aug 2025 Lab model card
9 InternVL3.5-241B-A28B 68.5% Aug 2025 Lab model card
10 GLM-4.1V-9B-Thinking 68.4% Jul 2025 Lab model card
11 Nemotron Nano 12B v2 VL 58.2% Nov 2025 Lab model card
12 Qwen2.5-VL-72B 57.6% Feb 2025 Lab model card
13 Phi-4-Reasoning-Vision-15B 44.9% Mar 2026 Lab model card
14 GPT-4V 39.4% Nov 2023 Official leaderboard
15 Qwen-VL-Max 25.3% Jan 2024 Official leaderboard
16 LLaVA-v1.6-34B 23.8% Jan 2024 Official leaderboard
17 Gemini 1.0 Pro 23.5% Dec 2023 Official leaderboard
18 InternLM-XComposer2-VL-7B 16.5% Jan 2024 Official leaderboard
19 llava-v1.6-vicuna-13b 15.6% Jan 2024 Official leaderboard
20 ShareGPT4V-13B 13.1% Dec 2023 Official leaderboard
21 Qwen-VL-Plus 11.8% Jan 2024 Official leaderboard
22 LLaVA-v1.5-13B 7.60% Oct 2023 Official leaderboard

The benchmark's own leaderboard has 19 entries, 9 of them models we track. Switch the scores above to rank them together.

6 models have held number one in labs’ own numbers since its release in March 2024 Number one over time 7 models have held number one with claims included since its release in March 2024

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Qwen Z.ai Baidu ByteDance Seed
Own leaderboard Not tested since Mar 2024
GPT-4V 39.4%
Lab claims Passed 85% in Sep 2025, by Qwen3-VL-235B-A22B
Seed2.1 Pro 89.7%

Sources: Official MathVerse leaderboard and labs’ model cards

Qwen’s own number for Qwen3-VL-235B-A22B passed 85% in September 2025

That was 18 months after its release. By the same age, 31% of the benchmarks from 2024 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Seed2.1 Pro ERNIE 5.0 GLM-4.5V Qwen2.5-VL-72B GPT-4V LLaVA-v1.5-13B 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Seed2.1 Pro GLM-4.5V

Sources: Official MathVerse leaderboard and labs’ model cards

Labs headline it less often now, last in October 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Z.ai
Shanghai AI Lab
Qwen
Baidu
Microsoft
OpenBMB (ModelBest)
DateLaunch postScoreAs the post writes it
Oct 7, 2026 MiniCPM-V-4.7-35B-A3B 86.5 MathVerse_MINI
Mar 4, 2026 Phi-4-Reasoning-Vision-15B 44.9 MathVerse_MINI (default mode)
Feb 6, 2026 ERNIE 5.0 85.1 MathVerse
Nov 11, 2025 ERNIE-4.5-VL-28B-A3B-Thinking 81 MathVerse
Sep 23, 2025 Qwen3-VL-235B-A22B 85 MathVerse_mini
Aug 25, 2025 InternVL3.5-241B-A28B 68.5 MathVerse (vision-only)
Aug 11, 2025 GLM-4.5V 72.1 MathVerse

11 system cards report it, from 8 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Jun 23, 2026 Seed2.1 Pro model card 89.7 MathVerse (Vision-Only) (Table 9) Capabilities
Mar 4, 2026 Phi-4-Reasoning-Vision-15B technical report 44.9 MathVerse MINI, default mode, 43.8 force nothink, 53.1 force thinking (Tables 5 and 6) Capabilities
Feb 4, 2026 ERNIE 5.0 technical report 85.1 MathVerse (Table 3) Capabilities
Jan 14, 2026 STEP3-VL-10B technical report 75.7 MathVerse (vision), SeRe (Tables 1 and 3). PaCoRe 78.30 Capabilities
Jan 1, 2026 GLM-4.6V technical report 75.4 MathVerse, GLM-4.6V thinking (Table 2) Capabilities
Nov 26, 2025 Qwen3-VL-235B-A22B technical report 85 MathVerse mini (Table 2) Capabilities
Nov 6, 2025 Nemotron Nano 12B v2 VL technical report 58.2 MathVerse-Mini (Vision-only), reasoning on (Table 3). Reasoning off 34.3 Capabilities
Aug 25, 2025 InternVL3.5-241B-A28B technical report 68.5 MathVerse (vision-only) (Table 2, Table 3). With parallel thinking 71.6 Capabilities
Aug 13, 2025 GLM-4.5V technical report 72.1 MathVerse, GLM-4.5V thinking (Table 2) Capabilities
Jul 1, 2025 GLM-4.1V-9B-Thinking technical report 68.4 MathVerse, GLM-4.1V-9B-Thinking thinking (Table 2) Capabilities
Feb 19, 2025 Qwen2.5-VL-72B technical report 57.6 MathVerse mini (Table 3) Capabilities

More in Vision

All 636 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. Qwen 3.6 Plus 86%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MathVista In use Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. Seed2.1 Pro 90.7%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%

Data audit

All 9 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 3 pass 2 notes
  • Pass Independent runs Scores from the official leaderboard (19 entries), launch posts (7 scores) and system cards (11 scores).
  • Pass Models scored 22 models we track, plus 10 other entries on its leaderboard.
  • Pass Newest score The newest is from October 2026, for MiniCPM-V-4.7-35B-A3B.
  • Note Frontier models None of its models are on the capabilities index.
  • Note Headroom Saturated. The best is 89.7%, past 85% since September 2025.
Leaderboard import 3 pass
  • Pass Importer We read the board on mathverse-cuhk.github.io: 9 rows for models we track and 10 other entries at the last read.
  • Pass Last read November 19th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check