SimpleVQA
The benchmark’s paper hasn’t scored a model released after Gemini 2.0 Flash (February 2025). Labs report their own numbers for 30 newer models, such as Qwen3.7-Plus at 81.7%.
Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references.
- Released
- February 2025
- Kind
- Short factual questions
- Level
- University
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
- In views
- Vision
- Strongest model tested
-
Qwen 3.8 Max #30 on the capability index
- Last in a launch post
- October 2026
- Last model added
- October 2026
- Scores refreshed
- November 19th, 2026
In the authors’ words
The increasing application of multi-modal large language models (MLLMs) across various sectors have spotlighted the essence of their output reliability and accuracy, particularly their ability to produce content grounded in factual information (e.g. common and domain-specific knowledge). In this work, we introduce SimpleVQA, the first comprehensive multi-modal benchmark to evaluate the factuality ability of MLLMs to answer natural language short questions. SimpleVQA is characterized by six key features: it covers multiple tasks and multiple scenarios, ensures high quality and challenging queries, maintains static and timeless reference answers, and is straightforward to evaluate. Our approach involves categorizing visual question-answering items into 9 different tasks around objective events or common knowledge and situating these within 9 topics. Rigorous quality control processes are implemented to guarantee high-quality, concise, and clear answers, facilitating evaluation with minimal variance via an LLM-as-a-judge scoring system. Using SimpleVQA, we perform a comprehensive assessment of leading 18 MLLMs and 8 text-only LLMs, delving into their image comprehension and text generation abilities by identifying and analyzing error cases.
Qwen3.7-Plus leads with 81.7%
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
Gemini 2.0 Flash leads with 52.8% across all independent runs
Best score per model across the benchmark’s paper, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.
Qwen3.7-Plus leads with 81.7%, claims included
Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.
- 81.7
Qwen3.7-Plus
- 79.2
Step 3.7 Flash
- 78.2 GLM-5V-Turbo
- 75.0
Qwen 3.8 Max
- 74.5
Seed2.1 Pro
- 73.6
Muse Spark Open
- 72.7
Seed2.0 Lite (0428)
- 71.4
Seed2.0 Pro
- 71.3
Muse Spark
- 71.2 Kimi K2.5
- 67.6
ERNIE 5.0
- 67.3
Qwen 3.6 Plus
- 67.1
Qwen3.5 397B-A17B
- 65.4
Seed1.8
- 65.3
Qwen3.5-Omni-Plus
- 63.4
Seed1.5-VL
- 62.2
Step 3
- 61.7
Qwen3.5-122B-A10B
- 61.3
Qwen3-VL-235B-A22B
- 58.9
Qwen 3.6 35B-A3B
- 58.3
Qwen3.5-35B-A3B
- 56.1
Qwen3.6 27B
- 56.0
Qwen3.5-27B
- 54.1
MiniCPM-V-4.7-35B-A3B
- 53.1
ERNIE-4.5-VL-28B-A3B-Thinking
- 53.1
STEP3-VL-10B
- 51.2
Qwen3.5-9B
- 43.4
Qwen3.5-4B
- 38.5
Qwen3.5-2B
- 35.4 LFM2.5-VL-3B
- 31.3
Qwen3.5-0.8B
- 1
Qwen3.7-Plus 81.7%
- 2
Step 3.7 Flash 79.2%
- 3 GLM-5V-Turbo 78.2%
- 4
Qwen 3.8 Max 75%
- 5
Seed2.1 Pro 74.5%
- 6
Muse Spark Open 73.6%
- 7
Seed2.0 Lite (0428) 72.7%
- 8
Seed2.0 Pro 71.4%
- 9
Muse Spark 71.3%
- 10 Kimi K2.5 71.2%
- 11
ERNIE 5.0 67.6%
- 12
Qwen 3.6 Plus 67.3%
- 13
Qwen3.5 397B-A17B 67.1%
- 14
Seed1.8 65.4%
- 15
Qwen3.5-Omni-Plus 65.3%
- 16
Seed1.5-VL 63.4%
- 17
Step 3 62.2%
- 18
Qwen3.5-122B-A10B 61.7%
- 19
Qwen3-VL-235B-A22B 61.3%
- 20
Qwen 3.6 35B-A3B 58.9%
- 21
Qwen3.5-35B-A3B 58.3%
- 22
Qwen3.6 27B 56.1%
- 23
Qwen3.5-27B 56%
- 24
MiniCPM-V-4.7-35B-A3B 54.1%
- 25
ERNIE-4.5-VL-28B-A3B-Thinking 53.1%
- 26
STEP3-VL-10B 53.1%
- 27
Qwen3.5-9B 51.2%
- 28
Qwen3.5-4B 43.4%
- 29
Qwen3.5-2B 38.5%
- 30 LFM2.5-VL-3B 35.4%
- 31
Qwen3.5-0.8B 31.3%
Source: Labs’ model cards
- 1
Gemini 2.0 Flash Feb 2025, image and text 52.8%
- 2
Qwen2.5-VL-72B image and text 49.4%
- 3
InternVL2.5-78B-MPO image and text 45.4%
- 4 Qwen2-VL-72B-Instruct image and text 44.7%
- 5
Qwen2.5-VL-7B image and text 43.2%
- 6
InternVL2.5-38B-MPO image and text 42.9%
- 7
InternVL2_5-78B image and text 41.5%
- 8
InternVL2.5-26B-MPO image and text 39.9%
- 9
InternVL2-Llama3-76B image and text 35.7%
- 10
InternVL2.5-8B-MPO image and text 33.6%
- 11
Janus-Pro-7B image and text 31.3%
- 12 o1-mini text only 4.10%
- 13 o1-preview text only 3.65%
Source: The SimpleVQA paper
- 81.7
Qwen3.7-Plus
- 79.2
Step 3.7 Flash
- 78.2 GLM-5V-Turbo
- 75.0
Qwen 3.8 Max
- 74.5
Seed2.1 Pro
- 73.6
Muse Spark Open
- 72.7
Seed2.0 Lite (0428)
- 71.4
Seed2.0 Pro
- 71.3
Muse Spark
- 71.2 Kimi K2.5
- 67.6
ERNIE 5.0
- 67.3
Qwen 3.6 Plus
- 67.1
Qwen3.5 397B-A17B
- 65.4
Seed1.8
- 65.3
Qwen3.5-Omni-Plus
- 63.4
Seed1.5-VL
- 62.2
Step 3
- 61.7
Qwen3.5-122B-A10B
- 61.3
Qwen3-VL-235B-A22B
- 58.9
Qwen 3.6 35B-A3B
- 58.3
Qwen3.5-35B-A3B
- 56.1
Qwen3.6 27B
- 56.0
Qwen3.5-27B
- 54.1
MiniCPM-V-4.7-35B-A3B
- 53.1
ERNIE-4.5-VL-28B-A3B-Thinking
- 53.1
STEP3-VL-10B
- 52.8
Gemini 2.0 Flash
- 51.2
Qwen3.5-9B
- 49.4
Qwen2.5-VL-72B
- 45.4
InternVL2.5-78B-MPO
- 44.7 Qwen2-VL-72B-Instruct
- 43.4
Qwen3.5-4B
- 43.2
Qwen2.5-VL-7B
- 42.9
InternVL2.5-38B-MPO
- 41.5
InternVL2_5-78B
- 39.9
InternVL2.5-26B-MPO
- 38.5
Qwen3.5-2B
- 35.7
InternVL2-Llama3-76B
- 35.4 LFM2.5-VL-3B
- 33.6
InternVL2.5-8B-MPO
- 31.3
Janus-Pro-7B
- 31.3
Qwen3.5-0.8B
- 4.10 o1-mini
- 3.65 o1-preview
- 1
Qwen3.7-Plus 81.7%
- 2
Step 3.7 Flash 79.2%
- 3 GLM-5V-Turbo 78.2%
- 4
Qwen 3.8 Max 75%
- 5
Seed2.1 Pro 74.5%
- 6
Muse Spark Open 73.6%
- 7
Seed2.0 Lite (0428) 72.7%
- 8
Seed2.0 Pro 71.4%
- 9
Muse Spark 71.3%
- 10 Kimi K2.5 71.2%
- 11
ERNIE 5.0 67.6%
- 12
Qwen 3.6 Plus 67.3%
- 13
Qwen3.5 397B-A17B 67.1%
- 14
Seed1.8 65.4%
- 15
Qwen3.5-Omni-Plus 65.3%
- 16
Seed1.5-VL 63.4%
- 17
Step 3 62.2%
- 18
Qwen3.5-122B-A10B 61.7%
- 19
Qwen3-VL-235B-A22B 61.3%
- 20
Qwen 3.6 35B-A3B 58.9%
- 21
Qwen3.5-35B-A3B 58.3%
- 22
Qwen3.6 27B 56.1%
- 23
Qwen3.5-27B 56%
- 24
MiniCPM-V-4.7-35B-A3B 54.1%
- 25
ERNIE-4.5-VL-28B-A3B-Thinking 53.1%
- 26
STEP3-VL-10B 53.1%
- 27
Gemini 2.0 Flash Feb 2025, image and text 52.8%
- 28
Qwen3.5-9B 51.2%
- 29
Qwen2.5-VL-72B image and text 49.4%
- 30
InternVL2.5-78B-MPO image and text 45.4%
- 31 Qwen2-VL-72B-Instruct image and text 44.7%
- 32
Qwen3.5-4B 43.4%
- 33
Qwen2.5-VL-7B image and text 43.2%
- 34
InternVL2.5-38B-MPO image and text 42.9%
- 35
InternVL2_5-78B image and text 41.5%
- 36
InternVL2.5-26B-MPO image and text 39.9%
- 37
Qwen3.5-2B 38.5%
- 38
InternVL2-Llama3-76B image and text 35.7%
- 39 LFM2.5-VL-3B 35.4%
- 40
InternVL2.5-8B-MPO image and text 33.6%
- 41
Janus-Pro-7B image and text 31.3%
- 42
Qwen3.5-0.8B 31.3%
- 43 o1-mini text only 4.10%
- 44 o1-preview text only 3.65%
Sources: Labs’ model cards and the SimpleVQA paper
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 81.7% | Jun 2026 | Lab model card | |
| 2 | | 79.2% | May 2026 | Lab model card | |
| 3 | GLM-5V-Turbo | 78.2% | Apr 2026 | Lab model card | |
| 4 | | 75% | Aug 2026 | Lab model card | |
| 5 | | 74.5% | Jun 2026 | Lab model card | |
| 6 | | 73.6% | Oct 2026 | Lab model card | |
| 7 | | 72.7% | Apr 2026 | Lab model card | |
| 8 | | 71.4% | Feb 2026 | Lab model card | |
| 9 | | 71.3% | Apr 2026 | Lab model card | |
| 10 | Kimi K2.5 | 71.2% | Jan 2026 | Lab model card | |
| 11 | | 67.6% | Feb 2026 | Lab model card | |
| 12 | | 67.3% | Apr 2026 | Lab model card | |
| 13 | | 67.1% | Feb 2026 | Lab model card | |
| 14 | | 65.4% | Dec 2025 | Lab model card | |
| 15 | | 65.3% | Mar 2026 | Lab model card | |
| 16 | | 63.4% | May 2025 | Lab model card | |
| 17 | | 62.2% | Jul 2025 | Lab model card | |
| 18 | | 61.7% | Feb 2026 | Lab model card | |
| 19 | | 61.3% | Sep 2025 | Lab model card | |
| 20 | | 58.9% | Apr 2026 | Lab model card | |
| 21 | | 58.3% | Feb 2026 | Lab model card | |
| 22 | | 56.1% | Apr 2026 | Lab model card | |
| 23 | | 56% | Feb 2026 | Lab model card | |
| 24 | | 54.1% | Oct 2026 | Lab model card | |
| 25 | | 53.1% | Nov 2025 | Lab model card | |
| 26 | | 53.1% | Jan 2026 | Lab model card | |
| 27 | | 51.2% | Mar 2026 | Lab model card | |
| 28 | | 43.4% | Mar 2026 | Lab model card | |
| 29 | | 38.5% | Mar 2026 | Lab model card | |
| 30 | LFM2.5-VL-3B | 35.4% | Aug 2026 | Lab model card | |
| 31 | | 31.3% | Mar 2026 | Lab model card |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 52.8% | image and text | Feb 2025 | Official leaderboard | |
| 2 | 49.4% | image and text | Jan 2025 | Official leaderboard | |
| 3 | 45.4% | image and text | Dec 2024 | Official leaderboard | |
| 4 | Qwen2-VL-72B-Instruct | 44.7% | image and text | Aug 2024 | Official leaderboard |
| 5 | 43.2% | image and text | Jan 2025 | Official leaderboard | |
| 6 | 42.9% | image and text | Dec 2024 | Official leaderboard | |
| 7 | 41.5% | image and text | Dec 2024 | Official leaderboard | |
| 8 | 39.9% | image and text | Dec 2024 | Official leaderboard | |
| 9 | 35.7% | image and text | Jul 2024 | Official leaderboard | |
| 10 | 33.6% | image and text | Dec 2024 | Official leaderboard | |
| 11 | 31.3% | image and text | Jan 2025 | Official leaderboard | |
| 12 | o1-mini | 4.10% | text only | Sep 2024 | Official leaderboard |
| 13 | o1-preview | 3.65% | text only | Sep 2024 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 81.7% | Jun 2026 | Lab model card | ||
| 2 | 79.2% | May 2026 | Lab model card | ||
| 3 | GLM-5V-Turbo | 78.2% | Apr 2026 | Lab model card | |
| 4 | 75% | Aug 2026 | Lab model card | ||
| 5 | 74.5% | Jun 2026 | Lab model card | ||
| 6 | 73.6% | Oct 2026 | Lab model card | ||
| 7 | 72.7% | Apr 2026 | Lab model card | ||
| 8 | 71.4% | Feb 2026 | Lab model card | ||
| 9 | 71.3% | Apr 2026 | Lab model card | ||
| 10 | Kimi K2.5 | 71.2% | Jan 2026 | Lab model card | |
| 11 | 67.6% | Feb 2026 | Lab model card | ||
| 12 | 67.3% | Apr 2026 | Lab model card | ||
| 13 | 67.1% | Feb 2026 | Lab model card | ||
| 14 | 65.4% | Dec 2025 | Lab model card | ||
| 15 | 65.3% | Mar 2026 | Lab model card | ||
| 16 | 63.4% | May 2025 | Lab model card | ||
| 17 | 62.2% | Jul 2025 | Lab model card | ||
| 18 | 61.7% | Feb 2026 | Lab model card | ||
| 19 | 61.3% | Sep 2025 | Lab model card | ||
| 20 | 58.9% | Apr 2026 | Lab model card | ||
| 21 | 58.3% | Feb 2026 | Lab model card | ||
| 22 | 56.1% | Apr 2026 | Lab model card | ||
| 23 | 56% | Feb 2026 | Lab model card | ||
| 24 | 54.1% | Oct 2026 | Lab model card | ||
| 25 | 53.1% | Nov 2025 | Lab model card | ||
| 26 | 53.1% | Jan 2026 | Lab model card | ||
| 27 | 52.8% | image and text | Feb 2025 | Official leaderboard | |
| 28 | 51.2% | Mar 2026 | Lab model card | ||
| 29 | 49.4% | image and text | Jan 2025 | Official leaderboard | |
| 30 | 45.4% | image and text | Dec 2024 | Official leaderboard | |
| 31 | Qwen2-VL-72B-Instruct | 44.7% | image and text | Aug 2024 | Official leaderboard |
| 32 | 43.4% | Mar 2026 | Lab model card | ||
| 33 | 43.2% | image and text | Jan 2025 | Official leaderboard | |
| 34 | 42.9% | image and text | Dec 2024 | Official leaderboard | |
| 35 | 41.5% | image and text | Dec 2024 | Official leaderboard | |
| 36 | 39.9% | image and text | Dec 2024 | Official leaderboard | |
| 37 | 38.5% | Mar 2026 | Lab model card | ||
| 38 | 35.7% | image and text | Jul 2024 | Official leaderboard | |
| 39 | LFM2.5-VL-3B | 35.4% | Aug 2026 | Lab model card | |
| 40 | 33.6% | image and text | Dec 2024 | Official leaderboard | |
| 41 | 31.3% | image and text | Jan 2025 | Official leaderboard | |
| 42 | 31.3% | Mar 2026 | Lab model card | ||
| 43 | o1-mini | 4.10% | text only | Sep 2024 | Official leaderboard |
| 44 | o1-preview | 3.65% | text only | Sep 2024 | Official leaderboard |
The benchmark's paper has 21 entries, 13 of them models we track. Switch the scores above to rank them together.
7 models have held number one in labs’ own numbers since its release in February 2025 2 models have held number one across all independent runs since its release in February 2025 9 models have held number one with claims included since its release in February 2025
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s paper and what labs claim in their cards.
Sources: The SimpleVQA paper and labs’ model cards
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Sources: Official SimpleVQA leaderboard and labs’ model cards
4 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Oct 15, 2026 | 73.6 | SimpleVQA (w/ tools) | |
| Oct 7, 2026 | 54.1 | SimpleVQA | |
| Aug 12, 2026 | LFM2.5-VL-3B | 35.4 | SimpleVQA |
| Aug 3, 2026 | 75 | SimpleVQA | |
| Jun 1, 2026 | 81.7 | SimpleVQA (with search) | |
| May 29, 2026 | 79.2 | SimpleVQA (with search) | |
| Apr 28, 2026 | 72.7 | SimpleVQA | |
| Apr 22, 2026 | 56.1 | SimpleVQA | |
| Apr 15, 2026 | 58.9 | SimpleVQA | |
| Apr 8, 2026 | 71.3 | SimpleVQA (Visual Factuality) | |
| Apr 2, 2026 | 67.3 | SimpleVQA | |
| Apr 1, 2026 | GLM-5V-Turbo | 78.2 | SimpleVQA 78.2 |
| Mar 30, 2026 | 65.3 | SimpleVQA | |
| Mar 2, 2026 | 31.3 | SimpleVQA 31.3/30.4 (with CI / without CI) | |
| Mar 2, 2026 | 38.5 | SimpleVQA 38.5/39.5 (with CI / without CI) | |
| Mar 2, 2026 | 43.4 | SimpleVQA | |
| Mar 2, 2026 | 51.2 | SimpleVQA | |
| Feb 24, 2026 | 58.3 | SimpleVQA | |
| Feb 24, 2026 | 56 | SimpleVQA | |
| Feb 24, 2026 | 61.7 | SimpleVQA | |
| Feb 16, 2026 | 67.1 | SimpleVQA | |
| Feb 6, 2026 | 67.6 | SimpleVQA | |
| Jan 27, 2026 | Kimi K2.5 | 71.2 | SimpleVQA |
| Dec 18, 2025 | 65.4 | SimpleVQA | |
| Nov 11, 2025 | 53.1 | SimpleVQA | |
| Sep 23, 2025 | 61.3 | SimpleVQA | |
| Jul 31, 2025 | 62.2 | SimpleVQA | |
| May 12, 2025 | 63.4 | SimpleVQA |
9 system cards report it, from 5 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Jun 23, 2026 | | 74.5 | SimpleVQA (Table 9) | Capabilities |
| Apr 17, 2026 | | 65.3 | SimpleVQA (Table 6) | Capabilities |
| Mar 21, 2026 | | 65.4 | SimpleVQA (Table 2) | Capabilities |
| Feb 14, 2026 | | 71.4 | SimpleVQA (Table 8) | Capabilities |
| Feb 4, 2026 | | 67.6 | SimpleVQA (Table 3) | Capabilities |
| Jan 30, 2026 | Kimi K2.5 technical report | 71.2 | SimpleVQA (Table 4) | Hallucination |
| Jan 14, 2026 | | 53.1 | SimpleVQA, SeRe (Tables 1 and 3). PaCoRe 54.64 | Capabilities |
| Nov 26, 2025 | | 61.3 | SimpleVQA (Table 2) | Capabilities |
| May 11, 2025 | | 63.4 | SimpleVQA, thinking (Table 6). Non-thinking 63.1 | Capabilities |
More in Factuality
- HallusionBench In use Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy.
Keye-VL-2.0-30B-A3B 73.5%
- SimpleQA Verified In use 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge.
Gemini 3.1 Pro 77.5%
- AA-Omniscience v1.1 In use Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones.
Inkling 2 Small 16.1
- GaRAGe In use Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages.
Amazon Nova Pro 60.7%
- Kimi Hallu Bench In use Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. GPT-6 Astra 4.20%
- Internal adversarially-designed factuality evaluations In use Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. GPT-6 Luna 7.60%
Data audit
All 10 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
- Pass Independent runs Scores from the benchmark’s paper (21 entries), launch posts (28 scores) and system cards (9 scores).
- Pass Models scored 44 models we track, plus 8 other entries on its leaderboard.
- Pass Newest score The newest is from October 2026, for Muse Spark Open.
- Note Frontier models None of the top 10 has a score. The strongest tested is Qwen 3.8 Max (#30).
- Pass Headroom The best is 81.7%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on arxiv.org: 14 rows for models we track and 8 other entries at the last read.
- Pass Last read November 19th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.