Benchmark Almanac

MuirBench

Seed2.0 Pro 81.8% leads since February 2026
IDEFICS-9B-Instruct 35.4% leads since June 2024
Seed2.0 Pro 81.8% leads since February 2026
In use

The benchmark’s paper hasn’t scored a model released after VILA1.5-13B (April 2024). Labs report their own numbers for 13 newer models, such as Seed2.0 Pro at 81.8%.

Models answer multiple choice questions over sets of several images, including unanswerable variants. Scored by accuracy.

Released
June 2024
Kind
General visual questions
Level
Everyday
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Images → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
In views
Vision
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 18th, 2026

In the authors’ words

We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the importance of MuirBench in encouraging the community to develop multimodal LLMs that can look beyond a single image, suggesting potential pathways for future improvements.

Abstract of MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding (arXiv:2406.09411, CC0)
13
Models scored in lab cards
9
Launch posts list it, from 6 labs
12
System cards report it, from 7 labs

Seed2.0 Pro leads with 81.8%

Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.

IDEFICS-9B-Instruct leads with 35.4% across all independent runs

Best score per model across the benchmark’s paper, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.

Seed2.0 Pro leads with 81.8%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

  1. 1 Seed2.0 Pro 81.8%
  2. 2 Qwen3-VL-235B-A22B 80.1%
  3. 3 Seed1.8 78.7%
  4. 4 Inkling 2 Mini 77.5%
  5. 5 LongCat-Flash-Omni 77.1%
  6. 6 GLM-4.6V 77.1%
  7. 7 GLM-4.5V 75.3%
  8. 8 GLM-4.1V-9B-Thinking 74.7%
  9. 9 Qwen2.5-VL-72B 70.7%
  10. 10 Qwen2.5-Omni-7B 59.2%
  11. 11 LFM2.5-VL-3B 58.3%
  12. 12 InternVL3.5-241B-A28B 47.5%
  13. 13 Nemotron Nano 12B v2 VL 44.2%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Seed2.0 Pro 81.8% Feb 2026 Lab model card
2 Qwen3-VL-235B-A22B 80.1% Sep 2025 Lab model card
3 Seed1.8 78.7% Dec 2025 Lab model card
4 Inkling 2 Mini 77.5% Nov 2026 Lab model card
5 LongCat-Flash-Omni 77.1% Oct 2025 Lab model card
6 GLM-4.6V 77.1% Dec 2025 Lab model card
7 GLM-4.5V 75.3% Aug 2025 Lab model card
8 GLM-4.1V-9B-Thinking 74.7% Jul 2025 Lab model card
9 Qwen2.5-VL-72B 70.7% Feb 2025 Lab model card
10 Qwen2.5-Omni-7B 59.2% Mar 2025 Lab model card
11 LFM2.5-VL-3B 58.3% Aug 2026 Lab model card
12 InternVL3.5-241B-A28B 47.5% Aug 2025 Lab model card
13 Nemotron Nano 12B v2 VL 44.2% Nov 2025 Lab model card
#ModelScoreSettingDateSource
1 IDEFICS-9B-Instruct 35.4% Jul 2023 Official leaderboard
2 LLaVA-v1.6-34B 33.3% Jan 2024 Official leaderboard
3 VILA1.5-13B 33.1% Apr 2024 Official leaderboard
4 Yi-VL-6B 28.7% Dec 2023 Official leaderboard
5 Idefics2-8B 26.1% Apr 2024 Official leaderboard
6 LLaVA-v1.5-13B 24.4% Oct 2023 Official leaderboard
7 llava-v1.5-7b 23.5% Oct 2023 Official leaderboard
8 CogVLM-17B-Chat 20.9% Nov 2023 Official leaderboard
#ModelScoreSettingDateSource
1 Seed2.0 Pro 81.8% Feb 2026 Lab model card
2 Qwen3-VL-235B-A22B 80.1% Sep 2025 Lab model card
3 Seed1.8 78.7% Dec 2025 Lab model card
4 Inkling 2 Mini 77.5% Nov 2026 Lab model card
5 LongCat-Flash-Omni 77.1% Oct 2025 Lab model card
6 GLM-4.6V 77.1% Dec 2025 Lab model card
7 GLM-4.5V 75.3% Aug 2025 Lab model card
8 GLM-4.1V-9B-Thinking 74.7% Jul 2025 Lab model card
9 Qwen2.5-VL-72B 70.7% Feb 2025 Lab model card
10 Qwen2.5-Omni-7B 59.2% Mar 2025 Lab model card
11 LFM2.5-VL-3B 58.3% Aug 2026 Lab model card
12 InternVL3.5-241B-A28B 47.5% Aug 2025 Lab model card
13 Nemotron Nano 12B v2 VL 44.2% Nov 2025 Lab model card
14 IDEFICS-9B-Instruct 35.4% Jul 2023 Official leaderboard
15 LLaVA-v1.6-34B 33.3% Jan 2024 Official leaderboard
16 VILA1.5-13B 33.1% Apr 2024 Official leaderboard
17 Yi-VL-6B 28.7% Dec 2023 Official leaderboard
18 Idefics2-8B 26.1% Apr 2024 Official leaderboard
19 LLaVA-v1.5-13B 24.4% Oct 2023 Official leaderboard
20 llava-v1.5-7b 23.5% Oct 2023 Official leaderboard
21 CogVLM-17B-Chat 20.9% Nov 2023 Official leaderboard

The benchmark's paper has 20 entries, 8 of them models we track. Switch the scores above to rank them together.

Rankings by part

We read 12 parts from the benchmark’s paper, each ranking models on one part of the test, such as Counting, Action and Grounding. IDEFICS-9B-Instruct leads 4 of them, and 6 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.

Counting

8 models we track have a score here.

  1. 1 LLaVA-v1.6-34B ↑1 36.3%
  2. 2 IDEFICS-9B-Instruct ↓1 29.9%
  3. 3 Yi-VL-6B ↑1 28.2%
  4. 4 LLaVA-v1.5-13B ↑2 25.2%
  5. 5 llava-v1.5-7b ↑2 23.1%
  6. 6 Idefics2-8B ↓1 21.8%
  7. 7 VILA1.5-13B ↓4 19.7%
  8. 8 CogVLM-17B-Chat 14.1%

5 models have held number one in labs’ own numbers since its release in June 2024 Number one over time 6 models have held number one with claims included since its release in June 2024

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s paper and what labs claim in their cards.

Hugging Face Qwen Z.ai ByteDance Seed
Benchmark paper Not tested since Jun 2024
Lab claims 5 leaders
Seed2.0 Pro 81.8%

Sources: The MuirBench paper and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
In the benchmark’s paper Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Seed2.0 Pro GLM-4.5V Qwen2.5-VL-72B IDEFICS-9B-Instruct 0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Seed2.0 Pro GLM-4.5V

Sources: Official MuirBench leaderboard and labs’ model cards

2 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
Z.ai
Meituan
ByteDance Seed
Liquid AI
Thinking Machines
DateLaunch postScoreAs the post writes it
Nov 20, 2026 Inkling 2 Mini 77.5 MuirBench
Aug 12, 2026 LFM2.5-VL-3B 58.3 MuirBench
Dec 18, 2025 Seed1.8 78.7 MUIRBench
Dec 8, 2025 GLM-4.6V 77.1 MUIRBENCH
Oct 31, 2025 LongCat-Flash-Omni 77.1 MuirBench
Sep 23, 2025 Qwen3-VL-235B-A22B 80.1 MUIRBENCH
Aug 11, 2025 GLM-4.5V 75.3 MUIRBENCH
Jul 1, 2025 GLM-4.1V-9B-Thinking 74.7 MUIRBENCH
Mar 27, 2025 Qwen2.5-Omni-7B 59.2 MuirBench

12 system cards report it, from 7 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 20, 2026 Inkling 2 Mini model card 77.5 MuirBench Capabilities
Mar 21, 2026 Seed1.8 model card 78.7 MUIRBench (Table 2) Capabilities
Feb 14, 2026 Seed2.0 Pro model card 81.8 MUIRBench (Table 8) Capabilities
Jan 1, 2026 GLM-4.6V technical report 77.1 MUIRBENCH, GLM-4.6V thinking (Table 2) Capabilities
Nov 26, 2025 Qwen3-VL-235B-A22B technical report 80.1 MUIRBENCH (Table 2) Capabilities
Nov 6, 2025 Nemotron Nano 12B v2 VL technical report 44.2 MUIRBench, reasoning on (Table 3). Reasoning off 33.2 Capabilities
Oct 31, 2025 LongCat-Flash-Omni technical report 77.1 MuirBench (Table 5) Capabilities
Aug 25, 2025 InternVL3.5-241B-A28B technical report 47.5 MuirBench (Table 5), checked on the rendered page Capabilities
Aug 13, 2025 GLM-4.5V technical report 75.3 MUIRBENCH, GLM-4.5V thinking (Table 2) Capabilities
Jul 1, 2025 GLM-4.1V-9B-Thinking technical report 74.7 MUIRBENCH, GLM-4.1V-9B-Thinking thinking (Table 2) Capabilities
Mar 26, 2025 Qwen2.5-Omni-7B technical report 59.2 MuirBench (Table 5) Capabilities
Feb 19, 2025 Qwen2.5-VL-72B technical report 70.7 MuirBench (Table 3) Capabilities

More in Vision

All 636 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. Qwen 3.6 Plus 86%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MathVista In use Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. Seed2.1 Pro 90.7%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%

Data audit

All 10 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
  • Pass Independent runs Scores from the benchmark’s paper (20 entries), launch posts (9 scores) and system cards (12 scores).
  • Pass Models scored 21 models we track, plus 12 other entries on its leaderboard.
  • Pass Newest score The newest is from November 2026, for Inkling 2 Mini.
  • Note Frontier models None of its models are on the capabilities index.
  • Pass Headroom The best is 81.8%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on arxiv.org: 8 rows for models we track and 12 other entries at the last read.
  • Pass Last read November 18th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check