MuirBench
The benchmark’s paper hasn’t scored a model released after VILA1.5-13B (April 2024). Labs report their own numbers for 13 newer models, such as Seed2.0 Pro at 81.8%.
Models answer multiple choice questions over sets of several images, including unanswerable variants. Scored by accuracy.
- Made by
- Fei Wang, Xingyu Fu et al.
- Released
- June 2024
- Kind
- General visual questions
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
- In views
- Vision
- Last in a launch post
- November 2026
- Last model added
- November 2026
- Scores refreshed
- November 18th, 2026
In the authors’ words
We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the importance of MuirBench in encouraging the community to develop multimodal LLMs that can look beyond a single image, suggesting potential pathways for future improvements.
Seed2.0 Pro leads with 81.8%
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
IDEFICS-9B-Instruct leads with 35.4% across all independent runs
Best score per model across the benchmark’s paper, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.
Seed2.0 Pro leads with 81.8%, claims included
Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.
- 1
Seed2.0 Pro 81.8%
- 2
Qwen3-VL-235B-A22B 80.1%
- 3
Seed1.8 78.7%
- 4
Inkling 2 Mini 77.5%
- 5
LongCat-Flash-Omni 77.1%
- 6 GLM-4.6V 77.1%
- 7 GLM-4.5V 75.3%
- 8 GLM-4.1V-9B-Thinking 74.7%
- 9
Qwen2.5-VL-72B 70.7%
- 10
Qwen2.5-Omni-7B 59.2%
- 11 LFM2.5-VL-3B 58.3%
- 12
InternVL3.5-241B-A28B 47.5%
- 13
Nemotron Nano 12B v2 VL 44.2%
Source: Labs’ model cards
- 1
IDEFICS-9B-Instruct 35.4%
- 2
LLaVA-v1.6-34B 33.3%
- 3
VILA1.5-13B 33.1%
- 4
Yi-VL-6B 28.7%
- 5
Idefics2-8B 26.1%
- 6
LLaVA-v1.5-13B 24.4%
- 7 llava-v1.5-7b 23.5%
- 8 CogVLM-17B-Chat 20.9%
Source: The MuirBench paper
- 81.8
Seed2.0 Pro
- 80.1
Qwen3-VL-235B-A22B
- 78.7
Seed1.8
- 77.5
Inkling 2 Mini
- 77.1
LongCat-Flash-Omni
- 77.1 GLM-4.6V
- 75.3 GLM-4.5V
- 74.7 GLM-4.1V-9B-Thinking
- 70.7
Qwen2.5-VL-72B
- 59.2
Qwen2.5-Omni-7B
- 58.3 LFM2.5-VL-3B
- 47.5
InternVL3.5-241B-A28B
- 44.2
Nemotron Nano 12B v2 VL
- 35.4
IDEFICS-9B-Instruct
- 33.3
LLaVA-v1.6-34B
- 33.1
VILA1.5-13B
- 28.7
Yi-VL-6B
- 26.1
Idefics2-8B
- 24.4
LLaVA-v1.5-13B
- 23.5 llava-v1.5-7b
- 20.9 CogVLM-17B-Chat
- 1
Seed2.0 Pro 81.8%
- 2
Qwen3-VL-235B-A22B 80.1%
- 3
Seed1.8 78.7%
- 4
Inkling 2 Mini 77.5%
- 5
LongCat-Flash-Omni 77.1%
- 6 GLM-4.6V 77.1%
- 7 GLM-4.5V 75.3%
- 8 GLM-4.1V-9B-Thinking 74.7%
- 9
Qwen2.5-VL-72B 70.7%
- 10
Qwen2.5-Omni-7B 59.2%
- 11 LFM2.5-VL-3B 58.3%
- 12
InternVL3.5-241B-A28B 47.5%
- 13
Nemotron Nano 12B v2 VL 44.2%
- 14
IDEFICS-9B-Instruct 35.4%
- 15
LLaVA-v1.6-34B 33.3%
- 16
VILA1.5-13B 33.1%
- 17
Yi-VL-6B 28.7%
- 18
Idefics2-8B 26.1%
- 19
LLaVA-v1.5-13B 24.4%
- 20 llava-v1.5-7b 23.5%
- 21 CogVLM-17B-Chat 20.9%
Sources: Labs’ model cards and the MuirBench paper
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 81.8% | Feb 2026 | Lab model card | |
| 2 | | 80.1% | Sep 2025 | Lab model card | |
| 3 | | 78.7% | Dec 2025 | Lab model card | |
| 4 | | 77.5% | Nov 2026 | Lab model card | |
| 5 | | 77.1% | Oct 2025 | Lab model card | |
| 6 | GLM-4.6V | 77.1% | Dec 2025 | Lab model card | |
| 7 | GLM-4.5V | 75.3% | Aug 2025 | Lab model card | |
| 8 | GLM-4.1V-9B-Thinking | 74.7% | Jul 2025 | Lab model card | |
| 9 | | 70.7% | Feb 2025 | Lab model card | |
| 10 | | 59.2% | Mar 2025 | Lab model card | |
| 11 | LFM2.5-VL-3B | 58.3% | Aug 2026 | Lab model card | |
| 12 | | 47.5% | Aug 2025 | Lab model card | |
| 13 | | 44.2% | Nov 2025 | Lab model card |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 35.4% | Jul 2023 | Official leaderboard | ||
| 2 | 33.3% | Jan 2024 | Official leaderboard | ||
| 3 | 33.1% | Apr 2024 | Official leaderboard | ||
| 4 | 28.7% | Dec 2023 | Official leaderboard | ||
| 5 | 26.1% | Apr 2024 | Official leaderboard | ||
| 6 | 24.4% | Oct 2023 | Official leaderboard | ||
| 7 | llava-v1.5-7b | 23.5% | Oct 2023 | Official leaderboard | |
| 8 | CogVLM-17B-Chat | 20.9% | Nov 2023 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 81.8% | Feb 2026 | Lab model card | ||
| 2 | 80.1% | Sep 2025 | Lab model card | ||
| 3 | 78.7% | Dec 2025 | Lab model card | ||
| 4 | 77.5% | Nov 2026 | Lab model card | ||
| 5 | 77.1% | Oct 2025 | Lab model card | ||
| 6 | GLM-4.6V | 77.1% | Dec 2025 | Lab model card | |
| 7 | GLM-4.5V | 75.3% | Aug 2025 | Lab model card | |
| 8 | GLM-4.1V-9B-Thinking | 74.7% | Jul 2025 | Lab model card | |
| 9 | 70.7% | Feb 2025 | Lab model card | ||
| 10 | 59.2% | Mar 2025 | Lab model card | ||
| 11 | LFM2.5-VL-3B | 58.3% | Aug 2026 | Lab model card | |
| 12 | 47.5% | Aug 2025 | Lab model card | ||
| 13 | 44.2% | Nov 2025 | Lab model card | ||
| 14 | 35.4% | Jul 2023 | Official leaderboard | ||
| 15 | 33.3% | Jan 2024 | Official leaderboard | ||
| 16 | 33.1% | Apr 2024 | Official leaderboard | ||
| 17 | 28.7% | Dec 2023 | Official leaderboard | ||
| 18 | 26.1% | Apr 2024 | Official leaderboard | ||
| 19 | 24.4% | Oct 2023 | Official leaderboard | ||
| 20 | llava-v1.5-7b | 23.5% | Oct 2023 | Official leaderboard | |
| 21 | CogVLM-17B-Chat | 20.9% | Nov 2023 | Official leaderboard |
The benchmark's paper has 20 entries, 8 of them models we track. Switch the scores above to rank them together.
Rankings by part
We read 12 parts from the benchmark’s paper, each ranking models on one part of the test, such as Counting, Action and Grounding. IDEFICS-9B-Instruct leads 4 of them, and 6 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.
Counting
8 models we track have a score here.
- 1
LLaVA-v1.6-34B ↑1 36.3%
- 2
IDEFICS-9B-Instruct ↓1 29.9%
- 3
Yi-VL-6B ↑1 28.2%
- 4
LLaVA-v1.5-13B ↑2 25.2%
- 5 llava-v1.5-7b ↑2 23.1%
- 6
Idefics2-8B ↓1 21.8%
- 7
VILA1.5-13B ↓4 19.7%
- 8 CogVLM-17B-Chat 14.1%
Action
8 models we track have a score here.
- 1
LLaVA-v1.5-13B ↑5 29.3%
- 2
VILA1.5-13B ↑1 28.7%
- 3
IDEFICS-9B-Instruct ↓2 28.1%
- 4 llava-v1.5-7b ↑3 27.4%
- 5
Yi-VL-6B ↓1 27.4%
- 6 CogVLM-17B-Chat ↑2 26.2%
- 7
LLaVA-v1.6-34B ↓5 26.2%
- 8
Idefics2-8B ↓3 26.2%
Grounding
8 models we track have a score here.
- 1
LLaVA-v1.6-34B ↑1 33.3%
- 2
Yi-VL-6B ↑2 28.6%
- 3
Idefics2-8B ↑2 26.2%
- 4
VILA1.5-13B ↓1 25%
- 5 CogVLM-17B-Chat ↑3 16.7%
- 6
LLaVA-v1.5-13B 14.3%
- 7 llava-v1.5-7b 14.3%
- 8
IDEFICS-9B-Instruct ↓7 13.1%
Matching
8 models we track have a score here.
- 1
VILA1.5-13B ↑2 41%
- 2
LLaVA-v1.6-34B 37.9%
- 3
IDEFICS-9B-Instruct ↓2 36%
- 4
Yi-VL-6B 25%
- 5
Idefics2-8B 24.8%
- 6 llava-v1.5-7b ↑1 23.5%
- 7 CogVLM-17B-Chat ↑1 21.3%
- 8
LLaVA-v1.5-13B ↓2 20.3%
Ordering
8 models we track have a score here.
- 1 llava-v1.5-7b ↑6 23.4%
- 2
LLaVA-v1.6-34B 21.9%
- 3
LLaVA-v1.5-13B ↑3 20.3%
- 4
Idefics2-8B ↑1 15.6%
- 5
IDEFICS-9B-Instruct ↓4 12.5%
- 6 CogVLM-17B-Chat ↑2 12.5%
- 7
VILA1.5-13B ↓4 10.9%
- 8
Yi-VL-6B ↓4 7.81%
Scene
8 models we track have a score here.
- 1
Idefics2-8B ↑4 56.5%
- 2
VILA1.5-13B ↑1 56.5%
- 3
LLaVA-v1.6-34B ↓1 54.3%
- 4 CogVLM-17B-Chat ↑4 41.4%
- 5
Yi-VL-6B ↓1 38.7%
- 6
LLaVA-v1.5-13B 36.6%
- 7 llava-v1.5-7b 35%
- 8
IDEFICS-9B-Instruct ↓7 27.4%
Difference
8 models we track have a score here.
- 1
IDEFICS-9B-Instruct 34.4%
- 2
Idefics2-8B ↑3 27.6%
- 3
Yi-VL-6B ↑1 25.6%
- 4
VILA1.5-13B ↓1 24.7%
- 5
LLaVA-v1.6-34B ↓3 22.1%
- 6
LLaVA-v1.5-13B 20%
- 7 llava-v1.5-7b 20%
- 8 CogVLM-17B-Chat 19.7%
Cartoon
8 models we track have a score here.
- 1
Yi-VL-6B ↑3 50%
- 2
IDEFICS-9B-Instruct ↓1 48.7%
- 3 CogVLM-17B-Chat ↑5 41%
- 4
LLaVA-v1.6-34B ↓2 41%
- 5
Idefics2-8B 39.7%
- 6
VILA1.5-13B ↓3 30.8%
- 7
LLaVA-v1.5-13B ↓1 25.6%
- 8 llava-v1.5-7b ↓1 24.4%
Diagram
8 models we track have a score here.
- 1
IDEFICS-9B-Instruct 47%
- 2
VILA1.5-13B ↑1 42.7%
- 3
LLaVA-v1.6-34B ↓1 38.2%
- 4
Yi-VL-6B 35.7%
- 5
LLaVA-v1.5-13B ↑1 31.7%
- 6
Idefics2-8B ↓1 25.4%
- 7 llava-v1.5-7b 25.1%
- 8 CogVLM-17B-Chat 19.6%
Geographic
8 models we track have a score here.
- 1
IDEFICS-9B-Instruct 35%
- 2
VILA1.5-13B ↑1 31%
- 3
Idefics2-8B ↑2 21%
- 4
LLaVA-v1.5-13B ↑2 20%
- 5 llava-v1.5-7b ↑2 20%
- 6
Yi-VL-6B ↓2 17%
- 7 CogVLM-17B-Chat ↑1 13%
- 8
LLaVA-v1.6-34B ↓6 12%
Attribute
8 models we track have a score here.
- 1
LLaVA-v1.6-34B ↑1 38.3%
- 2
Yi-VL-6B ↑2 34.2%
- 3
IDEFICS-9B-Instruct ↓2 32.6%
- 4
VILA1.5-13B ↓1 24.5%
- 5
LLaVA-v1.5-13B ↑1 23%
- 6 llava-v1.5-7b ↑1 23%
- 7
Idefics2-8B ↓2 17.9%
- 8 CogVLM-17B-Chat 16.3%
Retrieval
8 models we track have a score here.
- 1
IDEFICS-9B-Instruct 43.5%
- 2
VILA1.5-13B ↑1 30.1%
- 3
LLaVA-v1.6-34B ↓1 25%
- 4
Yi-VL-6B 22.6%
- 5
LLaVA-v1.5-13B ↑1 20.9%
- 6 llava-v1.5-7b ↑1 19.9%
- 7
Idefics2-8B ↓2 17.1%
- 8 CogVLM-17B-Chat 15.8%
5 models have held number one in labs’ own numbers since its release in June 2024 Number one over time 6 models have held number one with claims included since its release in June 2024
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s paper and what labs claim in their cards.
Sources: The MuirBench paper and labs’ model cards
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Sources: Official MuirBench leaderboard and labs’ model cards
2 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Nov 20, 2026 | 77.5 | MuirBench | |
| Aug 12, 2026 | LFM2.5-VL-3B | 58.3 | MuirBench |
| Dec 18, 2025 | 78.7 | MUIRBench | |
| Dec 8, 2025 | GLM-4.6V | 77.1 | MUIRBENCH |
| Oct 31, 2025 | 77.1 | MuirBench | |
| Sep 23, 2025 | 80.1 | MUIRBENCH | |
| Aug 11, 2025 | GLM-4.5V | 75.3 | MUIRBENCH |
| Jul 1, 2025 | GLM-4.1V-9B-Thinking | 74.7 | MUIRBENCH |
| Mar 27, 2025 | 59.2 | MuirBench |
12 system cards report it, from 7 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Nov 20, 2026 | | 77.5 | MuirBench | Capabilities |
| Mar 21, 2026 | | 78.7 | MUIRBench (Table 2) | Capabilities |
| Feb 14, 2026 | | 81.8 | MUIRBench (Table 8) | Capabilities |
| Jan 1, 2026 | GLM-4.6V technical report | 77.1 | MUIRBENCH, GLM-4.6V thinking (Table 2) | Capabilities |
| Nov 26, 2025 | | 80.1 | MUIRBENCH (Table 2) | Capabilities |
| Nov 6, 2025 | | 44.2 | MUIRBench, reasoning on (Table 3). Reasoning off 33.2 | Capabilities |
| Oct 31, 2025 | | 77.1 | MuirBench (Table 5) | Capabilities |
| Aug 25, 2025 | | 47.5 | MuirBench (Table 5), checked on the rendered page | Capabilities |
| Aug 13, 2025 | GLM-4.5V technical report | 75.3 | MUIRBENCH, GLM-4.5V thinking (Table 2) | Capabilities |
| Jul 1, 2025 | GLM-4.1V-9B-Thinking technical report | 74.7 | MUIRBENCH, GLM-4.1V-9B-Thinking thinking (Table 2) | Capabilities |
| Mar 26, 2025 | | 59.2 | MuirBench (Table 5) | Capabilities |
| Feb 19, 2025 | | 70.7 | MuirBench (Table 3) | Capabilities |
More in Vision
- MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy.
Qwen 3.6 Plus 86%
- MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
- CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
- MathVista In use Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy.
Seed2.1 Pro 90.7%
- MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy.
Qwen 3.8 Max 97.7%
- MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy.
Qwen 3.8 Max 85.9%
Data audit
All 10 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
- Pass Independent runs Scores from the benchmark’s paper (20 entries), launch posts (9 scores) and system cards (12 scores).
- Pass Models scored 21 models we track, plus 12 other entries on its leaderboard.
- Pass Newest score The newest is from November 2026, for Inkling 2 Mini.
- Note Frontier models None of its models are on the capabilities index.
- Pass Headroom The best is 81.8%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on arxiv.org: 8 rows for models we track and 12 other entries at the last read.
- Pass Last read November 18th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.