Benchmark Almanac

Vectara Hallucination Leaderboard (2025 dataset)

antgroup/finix_s1_32b 1.80% tops its leaderboard GPT-5.4 Nano 3.10% leads the models we track
Not in model cards

Models summarize over 7,700 private articles of up to 32K tokens using only facts in the text. Vectara's commercial HHEM model flags hallucinated summaries.

Vectara calls it a new baseline. The articles grew from 1,000 to over 7,700 and the prompt changed, so rates are much higher than on the old board.

Made by
Vectara
Released
November 2025
Kind
Faithful to sources
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
Kept private
Strongest model tested
GPT-6 Astra #3 on the capability index
Last model added
September 2026
Scores refreshed
November 12th, 2026
105
Entries on its own leaderboard, 104 models we track

antgroup/finix_s1_32b tops the leaderboard with 1.80%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is GPT-5.4 Nano, at 3.10%.

  1. 1 antgroup/finix_s1_32b 1.80%
  2. 2 GPT-5.4 Nano 3.10%
  3. 3 Gemini 2.5 Flash-Lite Sep 2025 3.30%
  4. 4 Phi-4 3.70%
  5. 5 Llama 3.3 70B Instruct 4.10%
  6. 6 DeepSeek V4.1 Flash 4.20%
  7. 7 Snowflake Arctic Instruct 4.30%
  8. 8 Gemma 3 12B 4.40%
  9. 9 Mistral Large 2 Nov 2024 4.50%
  10. 10 Qwen3-8B 4.80%
  11. 11 Amazon Nova Pro 5.10%
  12. 12 Mistral Small 3 5.10%
  13. 13 Nova 2 Lite 5.10%
  14. 14 Granite 4.0 H Small 5.20%
  15. 15 Gemma 4 26B A4B 5.20%
  16. 16 DeepSeek-V3.2-Exp 5.30%
  17. 17 Jamba2 Mini 5.30%
  18. 18 Qwen3-14B 5.40%
  19. 19 Amazon Nova Micro 5.50%
  20. 20 DeepSeek-V3.1 5.50%
  21. 21 GPT-5.4 Mini 5.50%
  22. 22 GPT-4.1 5.60%
  23. 23 Qwen3-4B 5.70%
  24. 24 Grok 3 5.80%
  25. 25 Qwen3-32B 5.90%
  26. 26 Amazon Nova Lite 6.10%
  27. 27 DeepSeek-V3 Mar 2025 6.10%
  28. 28 DeepSeek-V3.2 6.30%
  29. 29 Gemma 3 4B 6.40%
  30. 30 GPT-6 Sol 6.50%
  31. 31 Command R+ 6.90%
  32. 32 Trinity Large Preview 6.90%
  33. 33 Gemini 2.5 Pro Jun 2025 7.00%
  34. 34 GPT-5.4 7.00%
  35. 35 Ministral 3B 7.30%
  36. 36 Ministral 8B 7.40%
  37. 37 Gemma 3 27B 7.40%
  38. 38 Gemma 4 31B IT 7.40%
  39. 39 Llama 4 Scout 7.70%
  40. 40 Gemini 2.5 Flash Sep 2025 7.80%
  41. 41 Llama 4 Maverick 8.20%
  42. 42 Gemini 3.1 Flash-Lite 8.20%
  43. 43 GPT-5.4 Pro 8.30%
  44. 44 GPT-5.2 low 8.40%
  45. 45 DeepSeek-V4-Pro 8.60%
  46. 46 GPT-6 Astra 8.70%
  47. 47 MiniMax-M2.5 9.10%
  48. 48 Cohere Command A 9.30%
  49. 49 Qwen3-235B-A22B 9.30%
  50. 50 GLM-4.5-Air 9.30%
  51. 51 Qwen3-Next-80B-A3B thinking 9.30%
  52. 52 glm-4.7-flash 9.30%
  53. 53 GPT-5.5 9.30%
  54. 54 Aya Expanse 8B 9.50%
  55. 55 GLM-4.6 9.50%
  56. 56 GPT-4o Aug 2024 9.60%
  57. 57 Nemotron 3 Nano 30B-A3B 9.60%
  58. 58 Jamba 1.7 Large 9.70%
  59. 59 Claude Haiku 4.5 9.80%
  60. 60 Claude Opus 5.5 9.90%
  61. 61 GLM-5 10.1%
  62. 62 Claude Sonnet 4 10.3%
  63. 63 Gemini 3.1 Pro 10.4%
  64. 64 GPT-5 nano 10.5%
  65. 65 Qwen3.5-35B-A3B 10.5%
  66. 66 Qwen 3.5 Flash (hosted 35B-A3B) 10.5%
  67. 67 Granite 3.3 8B Instruct 10.6%
  68. 68 Claude Sonnet 4.6 10.6%
  69. 69 Qwen3.5-Plus 10.7%
  70. 70 Kimi K2.6 10.8%
  71. 71 Aya Expanse 32B 10.9%
  72. 72 GPT-5.1 low 10.9%
  73. 73 Claude Opus 4.5 10.9%
  74. 74 Qwen3.5-122B-A10B 11.2%
  75. 75 DeepSeek-R1 May 2025 11.3%
  76. 76 GLM-4.7 11.7%
  77. 77 Claude Opus 4.1 11.8%
  78. 78 MiniMax-M2.1 11.8%
  79. 79 Claude Opus 4 12%
  80. 80 Claude Sonnet 4.5 12%
  81. 81 Claude Opus 4.7 12%
  82. 82 Qwen3.5-27B 12.1%
  83. 83 Gemini 4 Argon 12.1%
  84. 84 Claude Opus 4.6 12.2%
  85. 85 Mercury 2 12.3%
  86. 86 GPT-5.6 Sol 12.4%
  87. 87 GPT-5 mini 12.9%
  88. 88 MiniMax-M2.7 12.9%
  89. 89 Gemini 3 Flash 13.5%
  90. 90 Gemini 3 Pro 13.6%
  91. 91 gpt-oss-120b 14.2%
  92. 92 Kimi K2.5 14.2%
  93. 93 Mistral Large 3 14.5%
  94. 94 Jamba 1.7 Mini 14.7%
  95. 95 GPT-5 minimal 14.7%
  96. 96 Grok 4.1 Fast non-reasoning 17.8%
  97. 97 Kimi K2 Sep 2025 17.9%
  98. 98 o4-mini low 18.6%
  99. 99 Ministral 3 14B 19.4%
  100. 100 Grok 4 Fast non-reasoning 19.7%
  101. 101 Ministral 3 8B Instruct 21.7%
  102. 102 Mistral Medium 3.1 22.7%
  103. 103 o3-pro 23.3%
  104. 104 Phi-4 Mini 23.5%
  105. 105 Ministral 3 3B Instruct 24.2%

Source: Official Vectara Hallucination Leaderboard (2025 dataset) leaderboard

#EntryScoreSettingDateSource
1 antgroup/finix_s1_32b 1.80% Official leaderboard
2 GPT-5.4 Nano 3.10% Mar 2026 Official leaderboard
3 Gemini 2.5 Flash-Lite Sep 2025 3.30% Sep 2025 Official leaderboard
4 Phi-4 3.70% Dec 2024 Official leaderboard
5 Llama 3.3 70B Instruct 4.10% Dec 2024 Official leaderboard
6 DeepSeek V4.1 Flash 4.20% Sep 2026 Official leaderboard
7 Snowflake Arctic Instruct 4.30% Apr 2024 Official leaderboard
8 Gemma 3 12B 4.40% Mar 2025 Official leaderboard
9 Mistral Large 2 Nov 2024 4.50% Nov 2024 Official leaderboard
10 Qwen3-8B 4.80% Apr 2025 Official leaderboard
11 Amazon Nova Pro 5.10% Dec 2024 Official leaderboard
12 Mistral Small 3 5.10% Jan 2025 Official leaderboard
13 Nova 2 Lite 5.10% Dec 2025 Official leaderboard
14 Granite 4.0 H Small 5.20% Oct 2025 Official leaderboard
15 Gemma 4 26B A4B 5.20% Apr 2026 Official leaderboard
16 DeepSeek-V3.2-Exp 5.30% Sep 2025 Official leaderboard
17 Jamba2 Mini 5.30% Jan 2026 Official leaderboard
18 Qwen3-14B 5.40% Apr 2025 Official leaderboard
19 Amazon Nova Micro 5.50% Dec 2024 Official leaderboard
20 DeepSeek-V3.1 5.50% Aug 2025 Official leaderboard
21 GPT-5.4 Mini 5.50% Mar 2026 Official leaderboard
22 GPT-4.1 5.60% Apr 2025 Official leaderboard
23 Qwen3-4B 5.70% Apr 2025 Official leaderboard
24 Grok 3 5.80% Apr 2025 Official leaderboard
25 Qwen3-32B 5.90% Apr 2025 Official leaderboard
26 Amazon Nova Lite 6.10% Dec 2024 Official leaderboard
27 DeepSeek-V3 Mar 2025 6.10% Mar 2025 Official leaderboard
28 DeepSeek-V3.2 6.30% Dec 2025 Official leaderboard
29 Gemma 3 4B 6.40% Mar 2025 Official leaderboard
30 GPT-6 Sol 6.50% Sep 2026 Official leaderboard
31 Command R+ 6.90% Aug 2024 Official leaderboard
32 Trinity Large Preview 6.90% Jan 2026 Official leaderboard
33 Gemini 2.5 Pro Jun 2025 7.00% Jun 2025 Official leaderboard
34 GPT-5.4 7.00% Mar 2026 Official leaderboard
35 Ministral 3B 7.30% Oct 2024 Official leaderboard
36 Ministral 8B 7.40% Oct 2024 Official leaderboard
37 Gemma 3 27B 7.40% Mar 2025 Official leaderboard
38 Gemma 4 31B IT 7.40% Apr 2026 Official leaderboard
39 Llama 4 Scout 7.70% Apr 2025 Official leaderboard
40 Gemini 2.5 Flash Sep 2025 7.80% Sep 2025 Official leaderboard
41 Llama 4 Maverick 8.20% Apr 2025 Official leaderboard
42 Gemini 3.1 Flash-Lite 8.20% Mar 2026 Official leaderboard
43 GPT-5.4 Pro 8.30% Mar 2026 Official leaderboard
44 GPT-5.2 8.40% low Dec 2025 Official leaderboard
45 DeepSeek-V4-Pro 8.60% Apr 2026 Official leaderboard
46 GPT-6 Astra 8.70% Sep 2026 Official leaderboard
47 MiniMax-M2.5 9.10% Feb 2026 Official leaderboard
48 Cohere Command A 9.30% Mar 2025 Official leaderboard
49 Qwen3-235B-A22B 9.30% Apr 2025 Official leaderboard
50 GLM-4.5-Air 9.30% Jul 2025 Official leaderboard
51 Qwen3-Next-80B-A3B 9.30% thinking Sep 2025 Official leaderboard
52 glm-4.7-flash 9.30% Jan 2026 Official leaderboard
53 GPT-5.5 9.30% Apr 2026 Official leaderboard
54 Aya Expanse 8B 9.50% Oct 2024 Official leaderboard
55 GLM-4.6 9.50% Sep 2025 Official leaderboard
56 GPT-4o Aug 2024 9.60% Aug 2024 Official leaderboard
57 Nemotron 3 Nano 30B-A3B 9.60% Dec 2025 Official leaderboard
58 Jamba 1.7 Large 9.70% Jul 2025 Official leaderboard
59 Claude Haiku 4.5 9.80% Oct 2025 Official leaderboard
60 Claude Opus 5.5 9.90% Sep 2026 Official leaderboard
61 GLM-5 10.1% Feb 2026 Official leaderboard
62 Claude Sonnet 4 10.3% May 2025 Official leaderboard
63 Gemini 3.1 Pro 10.4% Feb 2026 Official leaderboard
64 GPT-5 nano 10.5% Aug 2025 Official leaderboard
65 Qwen3.5-35B-A3B 10.5% Feb 2026 Official leaderboard
66 Qwen 3.5 Flash (hosted 35B-A3B) 10.5% Feb 2026 Official leaderboard
67 Granite 3.3 8B Instruct 10.6% Apr 2025 Official leaderboard
68 Claude Sonnet 4.6 10.6% Feb 2026 Official leaderboard
69 Qwen3.5-Plus 10.7% Feb 2026 Official leaderboard
70 Kimi K2.6 10.8% Apr 2026 Official leaderboard
71 Aya Expanse 32B 10.9% Oct 2024 Official leaderboard
72 GPT-5.1 10.9% low Nov 2025 Official leaderboard
73 Claude Opus 4.5 10.9% Nov 2025 Official leaderboard
74 Qwen3.5-122B-A10B 11.2% Feb 2026 Official leaderboard
75 DeepSeek-R1 May 2025 11.3% May 2025 Official leaderboard
76 GLM-4.7 11.7% Dec 2025 Official leaderboard
77 Claude Opus 4.1 11.8% Aug 2025 Official leaderboard
78 MiniMax-M2.1 11.8% Dec 2025 Official leaderboard
79 Claude Opus 4 12% May 2025 Official leaderboard
80 Claude Sonnet 4.5 12% Sep 2025 Official leaderboard
81 Claude Opus 4.7 12% Apr 2026 Official leaderboard
82 Qwen3.5-27B 12.1% Feb 2026 Official leaderboard
83 Gemini 4 Argon 12.1% Sep 2026 Official leaderboard
84 Claude Opus 4.6 12.2% Feb 2026 Official leaderboard
85 Mercury 2 12.3% Feb 2026 Official leaderboard
86 GPT-5.6 Sol 12.4% Jul 2026 Official leaderboard
87 GPT-5 mini 12.9% Aug 2025 Official leaderboard
88 MiniMax-M2.7 12.9% Mar 2026 Official leaderboard
89 Gemini 3 Flash 13.5% Dec 2025 Official leaderboard
90 Gemini 3 Pro 13.6% Nov 2025 Official leaderboard
91 gpt-oss-120b 14.2% Aug 2025 Official leaderboard
92 Kimi K2.5 14.2% Jan 2026 Official leaderboard
93 Mistral Large 3 14.5% Dec 2025 Official leaderboard
94 Jamba 1.7 Mini 14.7% Jul 2025 Official leaderboard
95 GPT-5 14.7% minimal Aug 2025 Official leaderboard
96 Grok 4.1 Fast 17.8% non-reasoning Nov 2025 Official leaderboard
97 Kimi K2 Sep 2025 17.9% Sep 2025 Official leaderboard
98 o4-mini 18.6% low Apr 2025 Official leaderboard
99 Ministral 3 14B 19.4% Dec 2025 Official leaderboard
100 Grok 4 Fast 19.7% non-reasoning Sep 2025 Official leaderboard
101 Ministral 3 8B Instruct 21.7% Dec 2025 Official leaderboard
102 Mistral Medium 3.1 22.7% Aug 2025 Official leaderboard
103 o3-pro 23.3% Jun 2025 Official leaderboard
104 Phi-4 Mini 23.5% Feb 2025 Official leaderboard
105 Ministral 3 3B Instruct 24.2% Dec 2025 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Grok 4 Fast
non-reasoni…
20.2%
o4-mini
low
18.6%
Grok 4.1 Fast
non-reasoni… reasoning
19.2%
GPT-5
minimal
15.1%
GPT-5.1
low
12.1%
GPT-5.2
low high
10.8%

Source: Official Vectara Hallucination Leaderboard (2025 dataset) leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Ministral 3 3B Instruct 24.2%
Phi-4 Mini 23.5%
o3-pro 23.3%
Mistral Medium 3.1 22.7%
Ministral 3 8B Instruct 21.7%
Grok 4 Fast reasoning 20.2%
Grok 4 Fast non-reasoning 19.7%
Ministral 3 14B 19.4%
Grok 4.1 Fast reasoning 19.2%
o4-mini low 18.6%
o4-mini high 18.6%
Kimi K2 Sep 2025 17.9%
Grok 4.1 Fast non-reasoning 17.8%
GPT-5 high 15.1%
Jamba 1.7 Mini 14.7%
GPT-5 minimal 14.7%
Mistral Large 3 14.5%
gpt-oss-120b 14.2%
Kimi K2.5 14.2%
Gemini 3 Pro 13.6%
Gemini 3 Flash 13.5%
GPT-5 mini 12.9%
MiniMax-M2.7 12.9%
GPT-5.6 Sol 12.4%
Mercury 2 12.3%
Claude Opus 4.6 12.2%
GPT-5.1 12.1%
Qwen3.5-27B 12.1%
Gemini 4 Argon 12.1%
Claude Opus 4 12%
Claude Sonnet 4.5 12%
Claude Opus 4.7 12%
Claude Opus 4.1 11.8%
MiniMax-M2.1 11.8%
GLM-4.7 11.7%
DeepSeek-R1 May 2025 11.3%
Qwen3.5-122B-A10B 11.2%
Aya Expanse 32B 10.9%
GPT-5.1 low 10.9%
Claude Opus 4.5 10.9%
GPT-5.2 high 10.8%
Kimi K2.6 10.8%
Qwen3.5-Plus 10.7%
Granite 3.3 8B Instruct 10.6%
Claude Sonnet 4.6 10.6%
GPT-5 nano 10.5%
Qwen3.5-35B-A3B 10.5%
Qwen 3.5 Flash (hosted 35B-A3B) 10.5%
Gemini 3.1 Pro 10.4%
Claude Sonnet 4 10.3%
GLM-5 10.1%
Claude Opus 5.5 9.90%
Claude Haiku 4.5 9.80%
Jamba 1.7 Large 9.70%
GPT-4o Aug 2024 9.60%
Nemotron 3 Nano 30B-A3B 9.60%
Aya Expanse 8B 9.50%
GLM-4.6 9.50%
Cohere Command A 9.30%
Qwen3-235B-A22B 9.30%
GLM-4.5-Air 9.30%
Qwen3-Next-80B-A3B thinking 9.30%
glm-4.7-flash 9.30%
GPT-5.5 9.30%
MiniMax-M2.5 9.10%
GPT-6 Astra 8.70%
DeepSeek-V4-Pro 8.60%
GPT-5.2 low 8.40%
GPT-5.4 Pro 8.30%
Llama 4 Maverick 8.20%
Gemini 3.1 Flash-Lite 8.20%
Gemini 2.5 Flash Sep 2025 7.80%
Llama 4 Scout 7.70%
Ministral 8B 7.40%
Gemma 3 27B 7.40%
Gemma 4 31B IT 7.40%
Ministral 3B 7.30%
Gemini 2.5 Pro Jun 2025 7.00%
GPT-5.4 7.00%
Command R+ 6.90%
Trinity Large Preview 6.90%
GPT-6 Sol 6.50%
Gemma 3 4B 6.40%
DeepSeek-V3.2 6.30%
Amazon Nova Lite 6.10%
DeepSeek-V3 Mar 2025 6.10%
Qwen3-32B 5.90%
Grok 3 5.80%
Qwen3-4B 5.70%
GPT-4.1 5.60%
Amazon Nova Micro 5.50%
DeepSeek-V3.1 5.50%
GPT-5.4 Mini 5.50%
Qwen3-14B 5.40%
DeepSeek-V3.2-Exp 5.30%
Jamba2 Mini 5.30%
Granite 4.0 H Small 5.20%
Gemma 4 26B A4B 5.20%
Amazon Nova Pro 5.10%
Mistral Small 3 5.10%
Nova 2 Lite 5.10%
Qwen3-8B 4.80%
Mistral Large 2 Nov 2024 4.50%
Gemma 3 12B 4.40%
Snowflake Arctic Instruct 4.30%
DeepSeek V4.1 Flash 4.20%
Llama 3.3 70B Instruct 4.10%
Phi-4 3.70%
Gemini 2.5 Flash-Lite Sep 2025 3.30%
GPT-5.4 Nano 3.10%

2 models have held number one on the official leaderboard since its release in November 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind OpenAI
Own leaderboard 2 leaders
GPT-5.4 Nano 3.10%

Source: Official Vectara Hallucination Leaderboard (2025 dataset) leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 GPT-5.4 Nano Gemini 2.5 Flash-Lite Phi-4 Snowflake Arctic Instruct 0% 20% 40% 60% 80% 100% 2024 2025 2026 GPT-5.4 Nano Snowflake Arctic Instruct

Source: Official Vectara Hallucination Leaderboard (2025 dataset) leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

OpenAI Z.ai Google DeepMind
Vectara Hallucination Leaderboard (2025 dataset) 7 leaders
GPT-5.4 Nano 3.10%
Vectara Hallucination Leaderboard (HHEM) Replaced by 2, Nov 2025
Gemini 2.0 Flash 0.70%
Vectara Hallucination Leaderboard (2025 dataset) 2 leaders
GPT-5.4 Nano 3.10%

Source: Official leaderboards

Replaces Vectara Hallucination Leaderboard (HHEM)

More in Factuality

All 74 in Factuality
  • SimpleVQA In use Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. Qwen3.7-Plus 81.7%
  • HallusionBench In use Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. Keye-VL-2.0-30B-A3B 73.5%
  • SimpleQA Verified In use 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. Gemini 3.1 Pro 77.5%
  • Kimi Hallu Bench In use Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. GPT-6 Astra 4.20%
  • Internal adversarially-designed factuality evaluations In use Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. GPT-6 Luna 7.60%
  • AA-Omniscience v1.1 In use Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. Inkling 2 Small 16.1