MMSI-Bench
The official leaderboard hasn’t scored a model released after Gemini 3 Pro (November 2025). Labs report their own numbers for 6 newer models, such as Nemotron 3.5 Super VL at 43.1%.
Models answer 1,000 human-written questions that need spatial reasoning across several images. Images come from indoor scans, driving, egocentric video and robot data.
Tasks cover camera, object and region relations, plus a multi-step reasoning category. arXiv v3 was posted on May 23rd, 2026.
- Made by
- Shanghai AI Lab (OpenRobotLab) and collaborators
- Released
- May 2025
- Kind
- Spatial reasoning and counting
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Real work and use
- Refresh
- One fixed set
- Run by
- Anyone
- In views
- Vision
- Strongest model tested
-
Gemini 3 Pro #56 on the capability index
- Last in a launch post
- October 2026, dropped before it saturated
- Last model added
- October 2026
- Scores refreshed
- November 19th, 2026
In the authors’ words
Spatial intelligence is essential for multimodal large language models (MLLMs) operating in the complex physical world. Existing benchmarks, however, probe only single-image relations and thus fail to assess the multi-image spatial reasoning that real-world deployments demand. We introduce MMSI-Bench, a VQA benchmark dedicated to multi-image spatial intelligence. Six 3D-vision researchers spent more than 300 hours meticulously crafting 1,000 challenging, unambiguous multiple-choice questions from over 120,000 images, each paired with carefully designed distractors and a stepwise reasoning process. We conduct extensive experiments and evaluate 37 open-source and proprietary MLLMs, observing a wide gap: the strongest open-source model attains roughly 30% accuracy and OpenAI's GPT-5 reasoning model reaches 40%, while humans score 97%. These results underscore the challenging nature of MMSI-Bench and the substantial headroom for future research. Leveraging the annotated reasoning processes, we also provide an automated error analysis pipeline that diagnoses four dominant failure modes, including (1) grounding errors, (2) overlap-matching and scene-reconstruction errors, (3) situation-transformation reasoning errors, and (4) spatial-logic errors, offering insights for advancing spatial intelligence. Project page: https://runsenxu.com/projects/MMSI_Bench .
Nemotron 3.5 Super VL leads with 43.1%
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
Gemini 3 Pro leads with 49.2% across all independent runs
Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.
Gemini 3 Pro leads with 49.2%, claims included
Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.
- 1
Nemotron 3.5 Super VL 43.1%
- 2
SenseNova-SI-1.3-InternVL3-8B 42.5%
- 3
Seed2.1 Pro 35.9%
- 4
Seed2.0 Pro 32.5%
- 5
STEP3-VL-10B 32.2%
- 6
Seed1.8 25.8%
- 7
MiniCPM-V-4.7-35B-A3B 16%
Source: Labs’ model cards
- 49.2
Gemini 3 Pro
- 41.9 GPT-5
- 41.0 o3
- 40.3 GPT-4.5
- 33.0
Doubao-1.5-pro
- 30.9 GPT-4.1
- 30.7
Qwen2.5-VL-72B
- 30.2 Claude 3.7 Sonnet
- 29.7
Seed1.5-VL
- 29.0
InternVL2.5-2B
- 28.7
InternVL2.5-8B
- 28.6
DeepSeek-VL2-Small
- 28.5
InternVL2_5-78B
- 28.5
InternVL3-78B
- 28.4 LLaVA-OneVision 72B
- 28.1
NVILA 8B
- 28.0
InternVL2.5-26B
- 27.1
DeepSeek-VL2
- 27.0
InternVL3-1B
- 26.7
InternVL3-9B
- 26.5
Qwen2.5-VL-3B
- 26.3
InternVL2.5-4B
- 26.1
InternVL2.5-1B
- 25.9
Qwen2.5-VL-7B
- 25.7
InternVL3-8B
- 25.3
InternVL3-2B
- 24.5
LLaVA-OneVision 7B
- 24.0
DeepSeek-VL2-Tiny
- 1
Gemini 3 Pro 49.2%
- 2 GPT-5 41.9%
- 3 o3 41%
- 4 GPT-4.5 40.3%
- 5
Doubao-1.5-pro 33%
- 6 GPT-4.1 30.9%
- 7
Qwen2.5-VL-72B 30.7%
- 8 Claude 3.7 Sonnet thinking 30.2%
- 9
Seed1.5-VL 29.7%
- 10
InternVL2.5-2B 29%
- 11
InternVL2.5-8B 28.7%
- 12
DeepSeek-VL2-Small 28.6%
- 13
InternVL2_5-78B 28.5%
- 14
InternVL3-78B 28.5%
- 15 LLaVA-OneVision 72B 28.4%
- 16
NVILA 8B 28.1%
- 17
InternVL2.5-26B 28%
- 18
DeepSeek-VL2 27.1%
- 19
InternVL3-1B 27%
- 20
InternVL3-9B 26.7%
- 21
Qwen2.5-VL-3B 26.5%
- 22
InternVL2.5-4B 26.3%
- 23
InternVL2.5-1B 26.1%
- 24
Qwen2.5-VL-7B 25.9%
- 25
InternVL3-8B 25.7%
- 26
InternVL3-2B 25.3%
- 27
LLaVA-OneVision 7B 24.5%
- 28
DeepSeek-VL2-Tiny 24%
Source: Official MMSI-Bench leaderboard
- 49.2
Gemini 3 Pro
- 43.1
Nemotron 3.5 Super VL
- 42.5
SenseNova-SI-1.3-InternVL3-8B
- 41.9 GPT-5
- 41.0 o3
- 40.3 GPT-4.5
- 35.9
Seed2.1 Pro
- 33.0
Doubao-1.5-pro
- 32.5
Seed2.0 Pro
- 32.2
STEP3-VL-10B
- 30.9 GPT-4.1
- 30.7
Qwen2.5-VL-72B
- 30.2 Claude 3.7 Sonnet
- 29.7
Seed1.5-VL
- 29.0
InternVL2.5-2B
- 28.7
InternVL2.5-8B
- 28.6
DeepSeek-VL2-Small
- 28.5
InternVL2_5-78B
- 28.5
InternVL3-78B
- 28.4 LLaVA-OneVision 72B
- 28.1
NVILA 8B
- 28.0
InternVL2.5-26B
- 27.1
DeepSeek-VL2
- 27.0
InternVL3-1B
- 26.7
InternVL3-9B
- 26.5
Qwen2.5-VL-3B
- 26.3
InternVL2.5-4B
- 26.1
InternVL2.5-1B
- 25.9
Qwen2.5-VL-7B
- 25.8
Seed1.8
- 25.7
InternVL3-8B
- 25.3
InternVL3-2B
- 24.5
LLaVA-OneVision 7B
- 24.0
DeepSeek-VL2-Tiny
- 16.0
MiniCPM-V-4.7-35B-A3B
- 1
Gemini 3 Pro 49.2%
- 2
Nemotron 3.5 Super VL 43.1%
- 3
SenseNova-SI-1.3-InternVL3-8B 42.5%
- 4 GPT-5 41.9%
- 5 o3 41%
- 6 GPT-4.5 40.3%
- 7
Seed2.1 Pro 35.9%
- 8
Doubao-1.5-pro 33%
- 9
Seed2.0 Pro 32.5%
- 10
STEP3-VL-10B 32.2%
- 11 GPT-4.1 30.9%
- 12
Qwen2.5-VL-72B 30.7%
- 13 Claude 3.7 Sonnet thinking 30.2%
- 14
Seed1.5-VL 29.7%
- 15
InternVL2.5-2B 29%
- 16
InternVL2.5-8B 28.7%
- 17
DeepSeek-VL2-Small 28.6%
- 18
InternVL2_5-78B 28.5%
- 19
InternVL3-78B 28.5%
- 20 LLaVA-OneVision 72B 28.4%
- 21
NVILA 8B 28.1%
- 22
InternVL2.5-26B 28%
- 23
DeepSeek-VL2 27.1%
- 24
InternVL3-1B 27%
- 25
InternVL3-9B 26.7%
- 26
Qwen2.5-VL-3B 26.5%
- 27
InternVL2.5-4B 26.3%
- 28
InternVL2.5-1B 26.1%
- 29
Qwen2.5-VL-7B 25.9%
- 30
Seed1.8 25.8%
- 31
InternVL3-8B 25.7%
- 32
InternVL3-2B 25.3%
- 33
LLaVA-OneVision 7B 24.5%
- 34
DeepSeek-VL2-Tiny 24%
- 35
MiniCPM-V-4.7-35B-A3B 16%
Sources: Official MMSI-Bench leaderboard and labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 43.1% | Oct 2026 | Lab model card | |
| 2 | | 42.5% | Jan 2026 | Lab model card | |
| 3 | | 35.9% | Jun 2026 | Lab model card | |
| 4 | | 32.5% | Feb 2026 | Lab model card | |
| 5 | | 32.2% | Jan 2026 | Lab model card | |
| 6 | | 25.8% | Dec 2025 | Lab model card | |
| 7 | | 16% | Oct 2026 | Lab model card |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 49.2% | Nov 2025 | Official leaderboard | ||
| 2 | GPT-5 | 41.9% | Aug 2025 | Official leaderboard | |
| 3 | o3 | 41% | Apr 2025 | Official leaderboard | |
| 4 | GPT-4.5 | 40.3% | Feb 2025 | Official leaderboard | |
| 5 | 33% | Jan 2025 | Official leaderboard | ||
| 6 | GPT-4.1 | 30.9% | Apr 2025 | Official leaderboard | |
| 7 | 30.7% | Jan 2025 | Official leaderboard | ||
| 8 | Claude 3.7 Sonnet | 30.2% | thinking | Feb 2025 | Official leaderboard |
| 9 | 29.7% | May 2025 | Official leaderboard | ||
| 10 | 29% | Dec 2024 | Official leaderboard | ||
| 11 | 28.7% | Dec 2024 | Official leaderboard | ||
| 12 | 28.6% | Dec 2024 | Official leaderboard | ||
| 13 | 28.5% | Dec 2024 | Official leaderboard | ||
| 14 | 28.5% | Apr 2025 | Official leaderboard | ||
| 15 | LLaVA-OneVision 72B | 28.4% | Aug 2024 | Official leaderboard | |
| 16 | 28.1% | Dec 2024 | Official leaderboard | ||
| 17 | 28% | Dec 2024 | Official leaderboard | ||
| 18 | 27.1% | Dec 2024 | Official leaderboard | ||
| 19 | 27% | Apr 2025 | Official leaderboard | ||
| 20 | 26.7% | Apr 2025 | Official leaderboard | ||
| 21 | 26.5% | Jan 2025 | Official leaderboard | ||
| 22 | 26.3% | Dec 2024 | Official leaderboard | ||
| 23 | 26.1% | Dec 2024 | Official leaderboard | ||
| 24 | 25.9% | Jan 2025 | Official leaderboard | ||
| 25 | 25.7% | Apr 2025 | Official leaderboard | ||
| 26 | 25.3% | Apr 2025 | Official leaderboard | ||
| 27 | 24.5% | Aug 2024 | Official leaderboard | ||
| 28 | 24% | Dec 2024 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 49.2% | Nov 2025 | Official leaderboard | ||
| 2 | 43.1% | Oct 2026 | Lab model card | ||
| 3 | 42.5% | Jan 2026 | Lab model card | ||
| 4 | GPT-5 | 41.9% | Aug 2025 | Official leaderboard | |
| 5 | o3 | 41% | Apr 2025 | Official leaderboard | |
| 6 | GPT-4.5 | 40.3% | Feb 2025 | Official leaderboard | |
| 7 | 35.9% | Jun 2026 | Lab model card | ||
| 8 | 33% | Jan 2025 | Official leaderboard | ||
| 9 | 32.5% | Feb 2026 | Lab model card | ||
| 10 | 32.2% | Jan 2026 | Lab model card | ||
| 11 | GPT-4.1 | 30.9% | Apr 2025 | Official leaderboard | |
| 12 | 30.7% | Jan 2025 | Official leaderboard | ||
| 13 | Claude 3.7 Sonnet | 30.2% | thinking | Feb 2025 | Official leaderboard |
| 14 | 29.7% | May 2025 | Official leaderboard | ||
| 15 | 29% | Dec 2024 | Official leaderboard | ||
| 16 | 28.7% | Dec 2024 | Official leaderboard | ||
| 17 | 28.6% | Dec 2024 | Official leaderboard | ||
| 18 | 28.5% | Dec 2024 | Official leaderboard | ||
| 19 | 28.5% | Apr 2025 | Official leaderboard | ||
| 20 | LLaVA-OneVision 72B | 28.4% | Aug 2024 | Official leaderboard | |
| 21 | 28.1% | Dec 2024 | Official leaderboard | ||
| 22 | 28% | Dec 2024 | Official leaderboard | ||
| 23 | 27.1% | Dec 2024 | Official leaderboard | ||
| 24 | 27% | Apr 2025 | Official leaderboard | ||
| 25 | 26.7% | Apr 2025 | Official leaderboard | ||
| 26 | 26.5% | Jan 2025 | Official leaderboard | ||
| 27 | 26.3% | Dec 2024 | Official leaderboard | ||
| 28 | 26.1% | Dec 2024 | Official leaderboard | ||
| 29 | 25.9% | Jan 2025 | Official leaderboard | ||
| 30 | 25.8% | Dec 2025 | Lab model card | ||
| 31 | 25.7% | Apr 2025 | Official leaderboard | ||
| 32 | 25.3% | Apr 2025 | Official leaderboard | ||
| 33 | 24.5% | Aug 2024 | Official leaderboard | ||
| 34 | 24% | Dec 2024 | Official leaderboard | ||
| 35 | 16% | Oct 2026 | Lab model card |
The benchmark's own leaderboard has 37 entries, 28 of them models we track. Switch the scores above to rank them together.
3 models have held number one in labs’ own numbers since its release in May 2025 3 models have held number one across all independent runs since its release in May 2025 3 models have held number one with claims included since its release in May 2025
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
Sources: Official MMSI-Bench leaderboard and labs’ model cards
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Sources: Official MMSI-Bench leaderboard and labs’ model cards
Labs headline it less often now, last in October 2026
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Oct 7, 2026 | 16 | MMSI-Bench | |
| Jan 8, 2026 | 42.5 | MMSI | |
| Dec 18, 2025 | 25.8 | MMSIBench (circular) |
5 system cards report it, from 3 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Oct 15, 2026 | | 43.1 | MMSI, SpatialClaw harness, with agentic vision training, Table 7 | Capabilities |
| Jun 23, 2026 | | 35.9 | MMSIBench (circular) (Table 9) | Capabilities |
| Mar 21, 2026 | | 25.8 | MMSIBench (circular) (Table 2) | Capabilities |
| Feb 14, 2026 | | 32.5 | MMSIBench (circular) (Table 8) | Capabilities |
| Jan 14, 2026 | | 32.2 | MMSI-Bench, SeRe (Tables 1 and 3). PaCoRe 36.40 | Capabilities |
More in Vision
- MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy.
Qwen 3.6 Plus 86%
- MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
- CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
- MathVista In use Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy.
Seed2.1 Pro 90.7%
- MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy.
Qwen 3.8 Max 97.7%
- MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy.
Qwen 3.8 Max 85.9%
Data audit
All 10 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
- Pass Independent runs Scores from the official leaderboard (37 entries), launch posts (3 scores) and system cards (5 scores).
- Pass Models scored 35 models we track, plus 9 other entries on its leaderboard.
- Pass Newest score The newest is from October 2026, for Nemotron 3.5 Super VL.
- Note Frontier models None of the top 10 has a score. The strongest tested is Gemini 3 Pro (#56).
- Pass Headroom The best is 49.2%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on github.com: 28 rows for models we track and 9 other entries at the last read.
- Pass Last read November 19th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.