HallusionBench
Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy.
- Made by
- University of Maryland
- Released
- October 2023
- Kind
- Hallucination rates
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
- In views
- Vision
- Strongest model tested
-
Qwen3.5 397B-A17B #78 on the capability index
- Last in a launch post
- October 2026
- Last model added
- October 2026
In the authors’ words
We introduce HallusionBench, a comprehensive benchmark designed for the evaluation of image-context reasoning. This benchmark presents significant challenges to advanced large visual-language models (LVLMs), such as GPT-4V(Vision), Gemini Pro Vision, Claude 3, and LLaVA-1.5, by emphasizing nuanced understanding and interpretation of visual data. The benchmark comprises 346 images paired with 1129 questions, all meticulously crafted by human experts. We introduce a novel structure for these visual questions designed to establish control groups. This structure enables us to conduct a quantitative analysis of the models' response tendencies, logical consistency, and various failure modes. In our evaluation on HallusionBench, we benchmarked 15 different models, highlighting a 31.42% question-pair accuracy achieved by the state-of-the-art GPT-4V. Notably, all other evaluated models achieve accuracy below 16%. Moreover, our analysis not only highlights the observed failure modes, including language hallucination and visual illusion, but also deepens an understanding of these pitfalls. Our comprehensive case studies within HallusionBench shed light on the challenges of hallucination and illusion in LVLMs. Based on these insights, we suggest potential pathways for their future improvement. The benchmark and codebase can be accessed at https://github.com/tianyi-lab/HallusionBench.
Keye-VL-2.0-30B-A3B leads with 73.5%
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
- 73.5
Keye-VL-2.0-30B-A3B
- 73.1
Nemotron Nano 12B v2 VL
- 71.4
Qwen3.5 397B-A17B
- 70.0
Qwen3.5-27B
- 69.8
Qwen 3.6 35B-A3B
- 69.3
Qwen3.5-9B
- 68.0
Seed2.0 Pro
- 67.9
Qwen3.5-35B-A3B
- 67.6
Qwen3.5-122B-A10B
- 66.7
Qwen3-VL-235B-A22B
- 65.4 GLM-4.5V
- 65.0
Qwen3.5-4B
- 64.9
STEP3-VL-10B
- 64.4
Phi-4-Reasoning-Vision-15B
- 64.2
Step 3
- 63.9
Seed1.8
- 63.9
ERNIE 5.0
- 63.8
MiniCPM-V-4.7-35B-A3B
- 63.7
EXAONE 4.5 33B
- 63.2 GLM-4.1V-9B-Thinking
- 62.3 GLM-4.6V
- 61.5
North Micro Vision Instruct
- 60.4
ERNIE-4.5-VL-28B-A3B-Thinking
- 60.3
Seed1.5-VL
- 59.7
Qwen3-Omni-30B-A3B
- 58.0
Qwen3.5-2B
- 57.3
InternVL3.5-241B-A28B
- 55.2
Qwen2.5-VL-72B
- 53.1
Qwen3.5-0.8B
- 50.5
Cosmos3-Nano
- 47.2 LFM2.5-VL-3B
- 40.7
Cosmos3-Edge
- 1
Keye-VL-2.0-30B-A3B 73.5%
- 2
Nemotron Nano 12B v2 VL 73.1%
- 3
Qwen3.5 397B-A17B 71.4%
- 4
Qwen3.5-27B 70%
- 5
Qwen 3.6 35B-A3B 69.8%
- 6
Qwen3.5-9B 69.3%
- 7
Seed2.0 Pro 68%
- 8
Qwen3.5-35B-A3B 67.9%
- 9
Qwen3.5-122B-A10B 67.6%
- 10
Qwen3-VL-235B-A22B 66.7%
- 11 GLM-4.5V 65.4%
- 12
Qwen3.5-4B 65%
- 13
STEP3-VL-10B 64.9%
- 14
Phi-4-Reasoning-Vision-15B 64.4%
- 15
Step 3 64.2%
- 16
Seed1.8 63.9%
- 17
ERNIE 5.0 63.9%
- 18
MiniCPM-V-4.7-35B-A3B 63.8%
- 19
EXAONE 4.5 33B 63.7%
- 20 GLM-4.1V-9B-Thinking 63.2%
- 21 GLM-4.6V 62.3%
- 22
North Micro Vision Instruct 61.5%
- 23
ERNIE-4.5-VL-28B-A3B-Thinking 60.4%
- 24
Seed1.5-VL 60.3%
- 25
Qwen3-Omni-30B-A3B 59.7%
- 26
Qwen3.5-2B 58%
- 27
InternVL3.5-241B-A28B 57.3%
- 28
Qwen2.5-VL-72B 55.2%
- 29
Qwen3.5-0.8B 53.1%
- 30
Cosmos3-Nano 50.5%
- 31 LFM2.5-VL-3B 47.2%
- 32
Cosmos3-Edge 40.7%
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 73.5% | May 2026 | Lab model card | |
| 2 | | 73.1% | Nov 2025 | Lab model card | |
| 3 | | 71.4% | Feb 2026 | Lab model card | |
| 4 | | 70% | Feb 2026 | Lab model card | |
| 5 | | 69.8% | Apr 2026 | Lab model card | |
| 6 | | 69.3% | Mar 2026 | Lab model card | |
| 7 | | 68% | Feb 2026 | Lab model card | |
| 8 | | 67.9% | Feb 2026 | Lab model card | |
| 9 | | 67.6% | Feb 2026 | Lab model card | |
| 10 | | 66.7% | Sep 2025 | Lab model card | |
| 11 | GLM-4.5V | 65.4% | Aug 2025 | Lab model card | |
| 12 | | 65% | Mar 2026 | Lab model card | |
| 13 | | 64.9% | Jan 2026 | Lab model card | |
| 14 | | 64.4% | Mar 2026 | Lab model card | |
| 15 | | 64.2% | Jul 2025 | Lab model card | |
| 16 | | 63.9% | Dec 2025 | Lab model card | |
| 17 | | 63.9% | Feb 2026 | Lab model card | |
| 18 | | 63.8% | Oct 2026 | Lab model card | |
| 19 | | 63.7% | Apr 2026 | Lab model card | |
| 20 | GLM-4.1V-9B-Thinking | 63.2% | Jul 2025 | Lab model card | |
| 21 | GLM-4.6V | 62.3% | Jan 2026 | Lab model card | |
| 22 | | 61.5% | Aug 2026 | Lab model card | |
| 23 | | 60.4% | Nov 2025 | Lab model card | |
| 24 | | 60.3% | May 2025 | Lab model card | |
| 25 | | 59.7% | Sep 2025 | Lab model card | |
| 26 | | 58% | Mar 2026 | Lab model card | |
| 27 | | 57.3% | Aug 2025 | Lab model card | |
| 28 | | 55.2% | Feb 2025 | Lab model card | |
| 29 | | 53.1% | Mar 2026 | Lab model card | |
| 30 | | 50.5% | May 2026 | Lab model card | |
| 31 | LFM2.5-VL-3B | 47.2% | Aug 2026 | Lab model card | |
| 32 | | 40.7% | May 2026 | Lab model card |
8 models have held number one in labs’ own numbers since its release in October 2023
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Labs’ model cards
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Labs’ model cards
3 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Oct 7, 2026 | 63.8 | Hallusion-Bench | |
| Aug 12, 2026 | LFM2.5-VL-3B | 47.2 | HallusionBench |
| Aug 12, 2026 | 61.5 | HallusionBench | |
| May 25, 2026 | 73.5 | HallusionBench | |
| Apr 15, 2026 | 69.8 | HallusionBench | |
| Apr 9, 2026 | 63.7 | HallusionBench | |
| Mar 4, 2026 | 64.4 | HallusionBench (default mode) | |
| Mar 2, 2026 | 53.1 | HallusionBench 53.1/46.7 (with CI / without CI) | |
| Mar 2, 2026 | 58 | HallusionBench 58.0/51.3 (with CI / without CI) | |
| Mar 2, 2026 | 65 | HallusionBench | |
| Mar 2, 2026 | 69.3 | HallusionBench | |
| Feb 24, 2026 | 67.9 | HallusionBench | |
| Feb 24, 2026 | 70 | HallusionBench | |
| Feb 24, 2026 | 67.6 | HallusionBench | |
| Feb 16, 2026 | 71.4 | HallusionBench | |
| Feb 6, 2026 | 63.9 | HallusionBench | |
| Dec 18, 2025 | 63.9 | HallusionBench | |
| Nov 11, 2025 | 60.4 | HallusionBench | |
| Sep 23, 2025 | 66.7 | HallusionBench | |
| Sep 22, 2025 | 59.7 | HallusionBench | |
| Aug 25, 2025 | 57.3 | HallusionBench | |
| Aug 11, 2025 | GLM-4.5V | 65.4 | HallusionBench |
| Jul 31, 2025 | 64.2 | Hallusion Bench | |
| May 12, 2025 | 60.3 | HallusionBench | |
| Jan 28, 2025 | 55.2 | Hallusion Bench |
18 system cards report it, from 10 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Jun 9, 2026 | | 73.5 | HallusionBench (Figure 5, printed value) | Hallucination |
| May 31, 2026 | | 50.5 | HallusionBench, Cosmos3-Nano 8B column, Table 10 | Capabilities |
| May 31, 2026 | | 40.7 | HallusionBench, Cosmos3-Edge 2B column, Table 10 | Capabilities |
| Apr 9, 2026 | | 63.7 | HallusionBench (Table 2, Reasoning mode) | Hallucination |
| Mar 21, 2026 | | 63.9 | HallusionBench (Table 2) | Capabilities |
| Mar 4, 2026 | | 64.4 | HallusionBench, default mode, 63.1 force nothink, 63.9 force thinking (Tables 5 and 6) | Hallucination |
| Feb 14, 2026 | | 68 | HallusionBench (Table 8) | Capabilities |
| Feb 4, 2026 | | 63.9 | HallusionBench (Table 3) | Capabilities |
| Jan 14, 2026 | | 64.9 | HallusionBench, SeRe (Tables 1 and 3). PaCoRe 64.54 | Hallucination |
| Jan 1, 2026 | GLM-4.6V technical report | 62.3 | HallusionBench, GLM-4.6V thinking (Table 2) | Capabilities |
| Nov 26, 2025 | | 66.7 | HallusionBench (Table 2) | Capabilities |
| Nov 6, 2025 | | 73.1 | HallusionBench, reasoning on (Table 3). Reasoning off 72.3 | Capabilities |
| Sep 22, 2025 | | 59.7 | HallusionBench, Instruct (Table 9). Thinking 62.8 (Table 10) | Capabilities |
| Aug 25, 2025 | | 57.3 | HallusionBench (avg) (Table 2, Table 6) | Hallucination |
| Aug 13, 2025 | GLM-4.5V technical report | 65.4 | HallusionBench, GLM-4.5V thinking (Table 2) | Capabilities |
| Jul 1, 2025 | GLM-4.1V-9B-Thinking technical report | 63.2 | HallusionBench, GLM-4.1V-9B-Thinking thinking (Table 2) | Capabilities |
| May 11, 2025 | | 60.3 | HallusionBench, thinking (Table 6). Non-thinking 60.0 | Capabilities |
| Feb 19, 2025 | | 55.2 | HallBench avg (Table 3) | Capabilities |
One logged change that moves scores
Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.
- Tasks changed November 28th, 2023
The maintainers expanded the dataset and updated the leaderboard. Scores on the expanded questions do not use the original evaluation set. Source
More in Factuality
- SimpleVQA In use Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references.
Qwen3.7-Plus 81.7%
- SimpleQA Verified In use 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge.
Gemini 3.1 Pro 77.5%
- AA-Omniscience v1.1 In use Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones.
Inkling 2 Small 16.1
- GaRAGe In use Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages.
Amazon Nova Pro 60.7%
- Kimi Hallu Bench In use Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. GPT-6 Astra 4.20%
- Internal adversarially-designed factuality evaluations In use Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. GPT-6 Luna 7.60%
Data audit
All 10 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
- Pass Independent runs Scores from the official leaderboard (17 entries), launch posts (25 scores) and system cards (18 scores).
- Pass Models scored 32 models we track, plus 17 other entries on its leaderboard.
- Pass Newest score The newest is from October 2026, for MiniCPM-V-4.7-35B-A3B.
- Note Frontier models None of the top 10 has a score. The strongest tested is Qwen3.5 397B-A17B (#78).
- Pass Headroom The best is 73.5%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on github.com: 17 entries at the last read, none a model we track.
- Pass Last read November 18th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.