Benchmark Almanac

Vectara Hallucination Leaderboard (HHEM)

Finix-S1-32B 0.60% tops its leaderboard Gemini 2.0 Flash Feb 2025 0.70% leads the models we track
Only in system cards

Models summarize short documents. The HHEM-2.1-Open classifier flags summaries that add unsupported facts, giving a hallucination rate.

Labs often report one minus the hallucination rate so that higher is better. The judge moved from HHEM-1.0 to HHEM-2.x in 2024-08. Vectara replaced the 1,000 article set in 2025-11.

Made by
Vectara
Released
November 2023
Kind
Faithful to sources
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Strongest model tested
GPT-5 #70 on the capability index
Last in a system card
July 2025
Last model added
October 2025
Scores refreshed
November 12th, 2026
166
Entries on its own leaderboard, 148 models we track
1
System card reports it, from 1 lab

Finix-S1-32B tops the leaderboard with 0.60%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Gemini 2.0 Flash, at 0.70%.

The newest model with a score here came out in October 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Finix-S1-32B 0.60%
  2. 2 Gemini 2.0 Flash Feb 2025 0.70%
  3. 3 o3-mini 0.80%
  4. 4 Gemini 2.0 Pro 0.80%
  5. 5 Mockingbird-2-Echo 0.90%
  6. 6 Gemini 2.5 Pro Mar 2025 1.10%
  7. 7 Kimi K2 Jul 2025 1.10%
  8. 8 Gemini 2.0 Flash-Lite 1.20%
  9. 9 GPT-4.5 1.20%
  10. 10 GLM-4-9B-Chat 1.30%
  11. 11 Gemini 2.0 Flash Dec 2024 1.30%
  12. 12 Gemini 2.5 Flash Apr 2025 1.30%
  13. 13 o1-mini 1.40%
  14. 14 GPT-5 high 1.40%
  15. 15 GPT-4o Mar 2025 1.50%
  16. 16 Amazon Nova Micro 1.60%
  17. 17 GPT-4 Turbo Apr 2024 1.70%
  18. 18 GPT-4o mini 1.70%
  19. 19 GPT-4 Jun 2023 1.80%
  20. 20 Amazon Nova Lite 1.80%
  21. 21 Amazon Nova Pro 1.80%
  22. 22 Gemini 2.0 Flash Thinking Jan 2025 1.80%
  23. 23 GPT-3.5 Turbo Jan 2024 1.90%
  24. 24 Grok-2 Dec 2024 1.90%
  25. 25 GLM-4.5-Air 1.90%
  26. 26 GPT-4.1 2.00%
  27. 27 GPT-4.1 nano 2.00%
  28. 28 Grok 3 2.10%
  29. 29 GPT-4.1 mini 2.20%
  30. 30 Qwen3-14B 2.20%
  31. 31 Jamba 1.6 Large 2.30%
  32. 32 DeepSeek-V2.5 Sep 2024 2.40%
  33. 33 o1 2.40%
  34. 34 o1-pro 2.40%
  35. 35 gpt-oss-120b 2.40%
  36. 36 Orca-2-13b 2.50%
  37. 37 Phi-3.5-MoE 2.50%
  38. 38 Neural-Chat-7B-v3-3 2.60%
  39. 39 Gemini 2.5 Pro Jun 2025 2.60%
  40. 40 Qwen3-4B 2.70%
  41. 41 Qwen2.5-7B 2.80%
  42. 42 Gemma 3 12B 2.80%
  43. 43 Qwen3-32B 2.80%
  44. 44 Jamba 1.5 Mini 2.90%
  45. 45 Grok 2 Vision (1212) 2.90%
  46. 46 Qwen2.5-Max 2.90%
  47. 47 Gemini 2.5 Flash-Lite Jun 2025 2.90%
  48. 48 Snowflake Arctic Instruct 3.00%
  49. 49 Qwen2.5-32B 3.00%
  50. 50 Gemma 3 27B 3.00%
  51. 51 Qwen3-8B 3.00%
  52. 52 Phi-3-mini-128k-instruct 3.10%
  53. 53 Mistral Small 3 3.10%
  54. 54 GPT-5 mini 3.20%
  55. 55 o1-preview 3.30%
  56. 56 Grok-3 mini 3.30%
  57. 57 Gemini 1.5 Flash Sep 2024 3.40%
  58. 58 Phi-4 Mini 3.40%
  59. 59 Yi-1.5-34B 3.70%
  60. 60 Gemma 3 4B 3.70%
  61. 61 Qwen3-0.6B 3.70%
  62. 62 GLM-4.5 3.70%
  63. 63 gpt-oss-20b 3.70%
  64. 64 Qwen3-Next-80B-A3B thinking 3.70%
  65. 65 Qwen3-Max 3.80%
  66. 66 Llama 3.1-405B 3.90%
  67. 67 DeepSeek-V3 3.90%
  68. 68 phi-3-mini 3.8B 4.00%
  69. 69 Llama 3.3 70B Instruct 4.00%
  70. 70 InternLM3-8B-Instruct 4.00%
  71. 71 Llama 3-70B 4.10%
  72. 72 Phi-3.5-mini 4.10%
  73. 73 Mistral Large 2 Nov 2024 4.10%
  74. 74 Qwen2-VL-7B-Instruct 4.20%
  75. 75 Qwen2.5-14B 4.20%
  76. 76 Claude Opus 4.1 4.20%
  77. 77 Qwen2.5-72B 4.30%
  78. 78 Llama 3.2 90B 4.30%
  79. 79 GLM-4.6 4.30%
  80. 80 Claude 3.7 Sonnet 4.40%
  81. 81 Qwen3-1.7B 4.40%
  82. 82 Cohere Command A 4.50%
  83. 83 Claude Sonnet 4 4.50%
  84. 84 Claude Haiku 4.5 4.50%
  85. 85 Grok 4.60%
  86. 86 Claude 3.5 Sonnet 4.60%
  87. 87 Jamba 1.6 Mini 4.60%
  88. 88 Llama 4 Maverick 4.60%
  89. 89 o4-mini 4.60%
  90. 90 DeepSeek-V3.2-Exp 4.60%
  91. 91 Mixtral 8x22B 4.70%
  92. 92 Qwen2-72B 4.70%
  93. 93 Phi-4 4.70%
  94. 94 Llama 4 Scout 4.70%
  95. 95 GPT-5 nano 4.70%
  96. 96 Claude Opus 4 4.80%
  97. 97 Grok 4 4.80%
  98. 98 Qwen3-30B-A3B-Thinking Jul 2025 4.80%
  99. 99 Command R (03-2024) 4.90%
  100. 100 Yi-1.5-9B-Chat 4.90%
  101. 101 Claude 3.5 Haiku 4.90%
  102. 102 Llama 3.1-70B 5.00%
  103. 103 Gemma 3 1B 5.30%
  104. 104 Llama 3.1-8B 5.40%
  105. 105 Command R+ 5.40%
  106. 106 DeepSeek-TNG-R1T2-Chimera 5.50%
  107. 107 Llama 3.2 11B 5.50%
  108. 108 DeepSeek-V3.1 5.50%
  109. 109 Claude Sonnet 4.5 5.50%
  110. 110 Mistral Small 3.1 5.60%
  111. 111 Llama 2-70B 5.90%
  112. 112 Kimi K2 Sep 2025 6.20%
  113. 113 Granite-3.0-8B-Instruct 6.50%
  114. 114 granite-4.0-micro 6.50%
  115. 115 Mistral-Pixtral 6.60%
  116. 116 Gemini 1.5 Pro Sep 2024 6.60%
  117. 117 Phi-2 6.70%
  118. 118 o3 6.80%
  119. 119 Gemma 2 2B IT 7.00%
  120. 120 Qwen2.5-3B 7.00%
  121. 121 Granite-4.0-h-tiny 7.30%
  122. 122 Llama 3-8B 7.40%
  123. 123 Ministral 8B 7.50%
  124. 124 Gemini 1.0 Pro 7.70%
  125. 125 DeepSeek-R1 May 2025 7.70%
  126. 126 Yi-1.5-6B-Chat 7.90%
  127. 127 Llama 3.2 3B 7.90%
  128. 128 DeepSeek-V3 Mar 2025 8.00%
  129. 129 DBRX 8.30%
  130. 130 Qwen2-VL-2B-Instruct 8.30%
  131. 131 Ministral 3B 8.30%
  132. 132 Granite-4.0-h-micro 8.40%
  133. 133 Aya Expanse 32B 8.50%
  134. 134 Mistral-Small2 8.60%
  135. 135 Granite-3.1-8B-Instruct 8.60%
  136. 136 Granite 3.2 8B Instruct 8.70%
  137. 137 Granite-3.0-2B-Instruct 8.80%
  138. 138 Mistral 7B v0.3 9.50%
  139. 139 Claude 3 Opus 10.1%
  140. 140 Gemma 2 9B 10.1%
  141. 141 Llama 2-13B 10.5%
  142. 142 OLMo-2-13B-Instruct 10.8%
  143. 143 OLMo-2-7B-Instruct 11.1%
  144. 144 Mistral NeMo 11.2%
  145. 145 Llama 2-7B 11.3%
  146. 146 Mistral Medium 3.1 11.4%
  147. 147 WizardLM-2 8x22B 11.7%
  148. 148 Aya Expanse 8B 12.2%
  149. 149 Titan-Express 13.5%
  150. 150 PaLM-2 14.1%
  151. 151 DeepSeek-R1 14.3%
  152. 152 Gemma 7B 14.8%
  153. 153 Granite-3.1-2B-Instruct 15.7%
  154. 154 Qwen2.5-1.5B 15.8%
  155. 155 QwQ-32B-Preview 16.1%
  156. 156 Claude 3 Sonnet 16.3%
  157. 157 Granite-3.2-2B-Instruct 16.5%
  158. 158 Gemma 1.1 7B IT 17%
  159. 159 Claude 2 17.4%
  160. 160 Flan-T5-large 18.3%
  161. 161 Mixtral 8x7B 20.1%
  162. 162 Llama 3.2 1B 20.7%
  163. 163 OpenELM-3B-Instruct 24.8%
  164. 164 Qwen2.5-0.5B 25.2%
  165. 165 Gemma 1.1 2B IT 27.8%
  166. 166 falcon-7B-instruct 29.9%

Source: Official Vectara Hallucination Leaderboard (HHEM) leaderboard

#EntryScoreSettingDateSource
1 Finix-S1-32B 0.60% Official leaderboard
2 Gemini 2.0 Flash Feb 2025 0.70% Feb 2025 Official leaderboard
3 o3-mini 0.80% Jan 2025 Official leaderboard
4 Gemini 2.0 Pro 0.80% Feb 2025 Official leaderboard
5 Mockingbird-2-Echo 0.90% Official leaderboard
6 Gemini 2.5 Pro Mar 2025 1.10% Mar 2025 Official leaderboard
7 Kimi K2 Jul 2025 1.10% Jul 2025 Official leaderboard
8 Gemini 2.0 Flash-Lite 1.20% Feb 2025 Official leaderboard
9 GPT-4.5 1.20% Feb 2025 Official leaderboard
10 GLM-4-9B-Chat 1.30% Jun 2024 Official leaderboard
11 Gemini 2.0 Flash Dec 2024 1.30% Dec 2024 Official leaderboard
12 Gemini 2.5 Flash Apr 2025 1.30% Apr 2025 Official leaderboard
13 o1-mini 1.40% Sep 2024 Official leaderboard
14 GPT-5 1.40% high Aug 2025 Official leaderboard
15 GPT-4o Mar 2025 1.50% Mar 2025 Official leaderboard
16 Amazon Nova Micro 1.60% Dec 2024 Official leaderboard
17 GPT-4 Turbo Apr 2024 1.70% Apr 2024 Official leaderboard
18 GPT-4o mini 1.70% Jul 2024 Official leaderboard
19 GPT-4 Jun 2023 1.80% Jun 2023 Official leaderboard
20 Amazon Nova Lite 1.80% Dec 2024 Official leaderboard
21 Amazon Nova Pro 1.80% Dec 2024 Official leaderboard
22 Gemini 2.0 Flash Thinking Jan 2025 1.80% Jan 2025 Official leaderboard
23 GPT-3.5 Turbo Jan 2024 1.90% Jan 2024 Official leaderboard
24 Grok-2 Dec 2024 1.90% Dec 2024 Official leaderboard
25 GLM-4.5-Air 1.90% Jul 2025 Official leaderboard
26 GPT-4.1 2.00% Apr 2025 Official leaderboard
27 GPT-4.1 nano 2.00% Apr 2025 Official leaderboard
28 Grok 3 2.10% Apr 2025 Official leaderboard
29 GPT-4.1 mini 2.20% Apr 2025 Official leaderboard
30 Qwen3-14B 2.20% Apr 2025 Official leaderboard
31 Jamba 1.6 Large 2.30% Mar 2025 Official leaderboard
32 DeepSeek-V2.5 Sep 2024 2.40% Sep 2024 Official leaderboard
33 o1 2.40% Dec 2024 Official leaderboard
34 o1-pro 2.40% Mar 2025 Official leaderboard
35 gpt-oss-120b 2.40% Aug 2025 Official leaderboard
36 Orca-2-13b 2.50% Official leaderboard
37 Phi-3.5-MoE 2.50% Aug 2024 Official leaderboard
38 Neural-Chat-7B-v3-3 2.60% Official leaderboard
39 Gemini 2.5 Pro Jun 2025 2.60% Jun 2025 Official leaderboard
40 Qwen3-4B 2.70% Apr 2025 Official leaderboard
41 Qwen2.5-7B 2.80% Sep 2024 Official leaderboard
42 Gemma 3 12B 2.80% Mar 2025 Official leaderboard
43 Qwen3-32B 2.80% Apr 2025 Official leaderboard
44 Jamba 1.5 Mini 2.90% Aug 2024 Official leaderboard
45 Grok 2 Vision (1212) 2.90% Dec 2024 Official leaderboard
46 Qwen2.5-Max 2.90% Jan 2025 Official leaderboard
47 Gemini 2.5 Flash-Lite Jun 2025 2.90% Jun 2025 Official leaderboard
48 Snowflake Arctic Instruct 3.00% Apr 2024 Official leaderboard
49 Qwen2.5-32B 3.00% Sep 2024 Official leaderboard
50 Gemma 3 27B 3.00% Mar 2025 Official leaderboard
51 Qwen3-8B 3.00% Apr 2025 Official leaderboard
52 Phi-3-mini-128k-instruct 3.10% Apr 2024 Official leaderboard
53 Mistral Small 3 3.10% Jan 2025 Official leaderboard
54 GPT-5 mini 3.20% Aug 2025 Official leaderboard
55 o1-preview 3.30% Sep 2024 Official leaderboard
56 Grok-3 mini 3.30% Apr 2025 Official leaderboard
57 Gemini 1.5 Flash Sep 2024 3.40% Aug 2024 Official leaderboard
58 Phi-4 Mini 3.40% Feb 2025 Official leaderboard
59 Yi-1.5-34B 3.70% May 2024 Official leaderboard
60 Gemma 3 4B 3.70% Mar 2025 Official leaderboard
61 Qwen3-0.6B 3.70% Apr 2025 Official leaderboard
62 GLM-4.5 3.70% Aug 2025 Official leaderboard
63 gpt-oss-20b 3.70% Aug 2025 Official leaderboard
64 Qwen3-Next-80B-A3B 3.70% thinking Sep 2025 Official leaderboard
65 Qwen3-Max 3.80% Sep 2025 Official leaderboard
66 Llama 3.1-405B 3.90% Jul 2024 Official leaderboard
67 DeepSeek-V3 3.90% Dec 2024 Official leaderboard
68 phi-3-mini 3.8B 4.00% Apr 2024 Official leaderboard
69 Llama 3.3 70B Instruct 4.00% Dec 2024 Official leaderboard
70 InternLM3-8B-Instruct 4.00% Jan 2025 Official leaderboard
71 Llama 3-70B 4.10% Apr 2024 Official leaderboard
72 Phi-3.5-mini 4.10% Aug 2024 Official leaderboard
73 Mistral Large 2 Nov 2024 4.10% Nov 2024 Official leaderboard
74 Qwen2-VL-7B-Instruct 4.20% Aug 2024 Official leaderboard
75 Qwen2.5-14B 4.20% Sep 2024 Official leaderboard
76 Claude Opus 4.1 4.20% Aug 2025 Official leaderboard
77 Qwen2.5-72B 4.30% Sep 2024 Official leaderboard
78 Llama 3.2 90B 4.30% Sep 2024 Official leaderboard
79 GLM-4.6 4.30% Sep 2025 Official leaderboard
80 Claude 3.7 Sonnet 4.40% Feb 2025 Official leaderboard
81 Qwen3-1.7B 4.40% Apr 2025 Official leaderboard
82 Cohere Command A 4.50% Mar 2025 Official leaderboard
83 Claude Sonnet 4 4.50% May 2025 Official leaderboard
84 Claude Haiku 4.5 4.50% Oct 2025 Official leaderboard
85 Grok 4.60% Official leaderboard
86 Claude 3.5 Sonnet 4.60% Jun 2024 Official leaderboard
87 Jamba 1.6 Mini 4.60% Mar 2025 Official leaderboard
88 Llama 4 Maverick 4.60% Apr 2025 Official leaderboard
89 o4-mini 4.60% Apr 2025 Official leaderboard
90 DeepSeek-V3.2-Exp 4.60% Sep 2025 Official leaderboard
91 Mixtral 8x22B 4.70% Apr 2024 Official leaderboard
92 Qwen2-72B 4.70% Jun 2024 Official leaderboard
93 Phi-4 4.70% Dec 2024 Official leaderboard
94 Llama 4 Scout 4.70% Apr 2025 Official leaderboard
95 GPT-5 nano 4.70% Aug 2025 Official leaderboard
96 Claude Opus 4 4.80% May 2025 Official leaderboard
97 Grok 4 4.80% Jul 2025 Official leaderboard
98 Qwen3-30B-A3B-Thinking Jul 2025 4.80% Jul 2025 Official leaderboard
99 Command R (03-2024) 4.90% Mar 2024 Official leaderboard
100 Yi-1.5-9B-Chat 4.90% May 2024 Official leaderboard
101 Claude 3.5 Haiku 4.90% Oct 2024 Official leaderboard
102 Llama 3.1-70B 5.00% Jul 2024 Official leaderboard
103 Gemma 3 1B 5.30% Mar 2025 Official leaderboard
104 Llama 3.1-8B 5.40% Jul 2024 Official leaderboard
105 Command R+ 5.40% Aug 2024 Official leaderboard
106 DeepSeek-TNG-R1T2-Chimera 5.50% Official leaderboard
107 Llama 3.2 11B 5.50% Sep 2024 Official leaderboard
108 DeepSeek-V3.1 5.50% Aug 2025 Official leaderboard
109 Claude Sonnet 4.5 5.50% Sep 2025 Official leaderboard
110 Mistral Small 3.1 5.60% Mar 2025 Official leaderboard
111 Llama 2-70B 5.90% Jul 2023 Official leaderboard
112 Kimi K2 Sep 2025 6.20% Sep 2025 Official leaderboard
113 Granite-3.0-8B-Instruct 6.50% Oct 2024 Official leaderboard
114 granite-4.0-micro 6.50% Oct 2025 Official leaderboard
115 Mistral-Pixtral 6.60% Official leaderboard
116 Gemini 1.5 Pro Sep 2024 6.60% Sep 2024 Official leaderboard
117 Phi-2 6.70% Dec 2023 Official leaderboard
118 o3 6.80% Apr 2025 Official leaderboard
119 Gemma 2 2B IT 7.00% Jul 2024 Official leaderboard
120 Qwen2.5-3B 7.00% Sep 2024 Official leaderboard
121 Granite-4.0-h-tiny 7.30% Official leaderboard
122 Llama 3-8B 7.40% Apr 2024 Official leaderboard
123 Ministral 8B 7.50% Oct 2024 Official leaderboard
124 Gemini 1.0 Pro 7.70% Dec 2023 Official leaderboard
125 DeepSeek-R1 May 2025 7.70% May 2025 Official leaderboard
126 Yi-1.5-6B-Chat 7.90% May 2024 Official leaderboard
127 Llama 3.2 3B 7.90% Sep 2024 Official leaderboard
128 DeepSeek-V3 Mar 2025 8.00% Mar 2025 Official leaderboard
129 DBRX 8.30% Mar 2024 Official leaderboard
130 Qwen2-VL-2B-Instruct 8.30% Aug 2024 Official leaderboard
131 Ministral 3B 8.30% Oct 2024 Official leaderboard
132 Granite-4.0-h-micro 8.40% Official leaderboard
133 Aya Expanse 32B 8.50% Oct 2024 Official leaderboard
134 Mistral-Small2 8.60% Official leaderboard
135 Granite-3.1-8B-Instruct 8.60% Dec 2024 Official leaderboard
136 Granite 3.2 8B Instruct 8.70% Feb 2025 Official leaderboard
137 Granite-3.0-2B-Instruct 8.80% Oct 2024 Official leaderboard
138 Mistral 7B v0.3 9.50% Sep 2023 Official leaderboard
139 Claude 3 Opus 10.1% Feb 2024 Official leaderboard
140 Gemma 2 9B 10.1% Jun 2024 Official leaderboard
141 Llama 2-13B 10.5% Jul 2023 Official leaderboard
142 OLMo-2-13B-Instruct 10.8% Official leaderboard
143 OLMo-2-7B-Instruct 11.1% Official leaderboard
144 Mistral NeMo 11.2% Jul 2024 Official leaderboard
145 Llama 2-7B 11.3% Jul 2023 Official leaderboard
146 Mistral Medium 3.1 11.4% Aug 2025 Official leaderboard
147 WizardLM-2 8x22B 11.7% Apr 2024 Official leaderboard
148 Aya Expanse 8B 12.2% Oct 2024 Official leaderboard
149 Titan-Express 13.5% Official leaderboard
150 PaLM-2 14.1% Official leaderboard
151 DeepSeek-R1 14.3% Jan 2025 Official leaderboard
152 Gemma 7B 14.8% Feb 2024 Official leaderboard
153 Granite-3.1-2B-Instruct 15.7% Dec 2024 Official leaderboard
154 Qwen2.5-1.5B 15.8% Sep 2024 Official leaderboard
155 QwQ-32B-Preview 16.1% Nov 2024 Official leaderboard
156 Claude 3 Sonnet 16.3% Feb 2024 Official leaderboard
157 Granite-3.2-2B-Instruct 16.5% Official leaderboard
158 Gemma 1.1 7B IT 17% Apr 2024 Official leaderboard
159 Claude 2 17.4% Jul 2023 Official leaderboard
160 Flan-T5-large 18.3% Official leaderboard
161 Mixtral 8x7B 20.1% Dec 2023 Official leaderboard
162 Llama 3.2 1B 20.7% Sep 2024 Official leaderboard
163 OpenELM-3B-Instruct 24.8% Official leaderboard
164 Qwen2.5-0.5B 25.2% Sep 2024 Official leaderboard
165 Gemma 1.1 2B IT 27.8% Apr 2024 Official leaderboard
166 falcon-7B-instruct 29.9% Official leaderboard

Labs report scores for 1 model in their cards. We plot every source in the charts below.

More effort adds a median 4 points

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
o4-mini
low high
6.80%
Claude 3.7 Sonnet
thinking
4.50%
GPT-5
minimal high
4.90%

Source: Official Vectara Hallucination Leaderboard (HHEM) leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Gemma 1.1 2B IT 27.8%
Qwen2.5-0.5B 25.2%
Llama 3.2 1B 20.7%
Mixtral 8x7B 20.1%
Claude 2 17.4%
Gemma 1.1 7B IT 17%
Claude 3 Sonnet 16.3%
QwQ-32B-Preview 16.1%
Qwen2.5-1.5B 15.8%
Granite-3.1-2B-Instruct 15.7%
Gemma 7B 14.8%
DeepSeek-R1 14.3%
Aya Expanse 8B 12.2%
WizardLM-2 8x22B 11.7%
Mistral Medium 3.1 11.4%
Llama 2-7B 11.3%
Mistral NeMo 11.2%
Llama 2-13B 10.5%
Claude 3 Opus 10.1%
Gemma 2 9B 10.1%
Mistral 7B v0.3 9.50%
Gemini 1.5 Pro Sep 2024 9.10%
Granite-3.0-2B-Instruct 8.80%
Granite 3.2 8B Instruct 8.70%
Granite-3.1-8B-Instruct 8.60%
Aya Expanse 32B 8.50%
DBRX 8.30%
Qwen2-VL-2B-Instruct 8.30%
Ministral 3B 8.30%
DeepSeek-V3 Mar 2025 8.00%
Yi-1.5-6B-Chat 7.90%
Llama 3.2 3B 7.90%
Gemini 1.0 Pro 7.70%
DeepSeek-R1 May 2025 7.70%
Ministral 8B 7.50%
Llama 3-8B 7.40%
Gemma 2 2B IT 7.00%
Qwen2.5-3B 7.00%
o3 6.80%
o4-mini low 6.80%
Phi-2 6.70%
Gemini 1.5 Flash Sep 2024 6.60%
Gemini 1.5 Pro Sep 2024 6.60%
Granite-3.0-8B-Instruct 6.50%
granite-4.0-micro 6.50%
Kimi K2 Sep 2025 6.20%
Llama 2-70B 5.90%
Mistral Small 3.1 5.60%
Llama 3.2 11B 5.50%
DeepSeek-V3.1 5.50%
Claude Sonnet 4.5 5.50%
Llama 3.1-8B 5.40%
Command R+ 5.40%
Gemma 3 1B 5.30%
o4-mini high 5.10%
Llama 3.1-70B 5.00%
Command R (03-2024) 4.90%
Yi-1.5-9B-Chat 4.90%
Claude 3.5 Haiku 4.90%
GPT-5 minimal 4.90%
Claude Opus 4 4.80%
Grok 4 4.80%
Qwen3-30B-A3B-Thinking Jul 2025 4.80%
Mixtral 8x22B 4.70%
Qwen2-72B 4.70%
Phi-4 4.70%
Llama 4 Scout 4.70%
GPT-5 nano 4.70%
Claude 3.5 Sonnet 4.60%
Jamba 1.6 Mini 4.60%
Llama 4 Maverick 4.60%
o4-mini 4.60%
DeepSeek-V3.2-Exp 4.60%
Claude 3.7 Sonnet thinking 4.50%
Cohere Command A 4.50%
Claude Sonnet 4 4.50%
Claude Haiku 4.5 4.50%
Claude 3.7 Sonnet 4.40%
Qwen3-1.7B 4.40%
Qwen2.5-72B 4.30%
Llama 3.2 90B 4.30%
GLM-4.6 4.30%
Qwen2-VL-7B-Instruct 4.20%
Qwen2.5-14B 4.20%
Claude Opus 4.1 4.20%
Llama 3-70B 4.10%
Phi-3.5-mini 4.10%
Mistral Large 2 Nov 2024 4.10%
phi-3-mini 3.8B 4.00%
Llama 3.3 70B Instruct 4.00%
InternLM3-8B-Instruct 4.00%
Llama 3.1-405B 3.90%
DeepSeek-V3 3.90%
Qwen3-Max 3.80%
Yi-1.5-34B 3.70%
Gemma 3 4B 3.70%
Qwen3-0.6B 3.70%
GLM-4.5 3.70%
gpt-oss-20b 3.70%
Qwen3-Next-80B-A3B thinking 3.70%
Gemini 1.5 Flash Sep 2024 3.40%
Phi-4 Mini 3.40%
o1-preview 3.30%
Grok-3 mini 3.30%
GPT-5 mini 3.20%
Phi-3-mini-128k-instruct 3.10%
Mistral Small 3 3.10%
Snowflake Arctic Instruct 3.00%
Qwen2.5-32B 3.00%
Gemma 3 27B 3.00%
Qwen3-8B 3.00%
Jamba 1.5 Mini 2.90%
Grok 2 Vision (1212) 2.90%
Qwen2.5-Max 2.90%
Gemini 2.5 Flash-Lite Jun 2025 2.90%
Qwen2.5-7B 2.80%
Gemma 3 12B 2.80%
Qwen3-32B 2.80%
Qwen3-4B 2.70%
Gemini 2.5 Pro Jun 2025 2.60%
Phi-3.5-MoE 2.50%
DeepSeek-V2.5 Sep 2024 2.40%
o1 2.40%
o1-pro 2.40%
gpt-oss-120b 2.40%
Jamba 1.6 Large 2.30%
GPT-4.1 mini 2.20%
Qwen3-14B 2.20%
Grok 3 2.10%
GPT-4.1 2.00%
GPT-4.1 nano 2.00%
GPT-3.5 Turbo Jan 2024 1.90%
Grok-2 Dec 2024 1.90%
GLM-4.5-Air 1.90%
GPT-4 Jun 2023 1.80%
Amazon Nova Lite 1.80%
Amazon Nova Pro 1.80%
Gemini 2.0 Flash Thinking Jan 2025 1.80%
GPT-4 Turbo Apr 2024 1.70%
GPT-4o mini 1.70%
Amazon Nova Micro 1.60%
GPT-4o Mar 2025 1.50%
o1-mini 1.40%
GPT-5 high 1.40%
GLM-4-9B-Chat 1.30%
Gemini 2.0 Flash Dec 2024 1.30%
Gemini 2.5 Flash Apr 2025 1.30%
Gemini 2.0 Flash-Lite 1.20%
GPT-4.5 1.20%
Gemini 2.5 Pro Mar 2025 1.10%
Kimi K2 Jul 2025 1.10%
o3-mini 0.80%
Gemini 2.0 Pro 0.80%
Gemini 2.0 Flash Feb 2025 0.70%

5 models have held number one on the official leaderboard since its release in November 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Z.ai Google DeepMind
Own leaderboard Replaced by 2, Nov 2025
Gemini 2.0 Flash 0.70%

Source: Official Vectara Hallucination Leaderboard (HHEM) leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2023 2024 2025 Gemini 2.0 Flash GLM-4-9B-Chat GPT-4 0% 20% 40% 60% 80% 100% 2023 2024 2025 Gemini 2.0 Flash GPT-4

Sources: Official Vectara Hallucination Leaderboard (HHEM) leaderboard and labs’ model cards

Labs report it only in system cards

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark. Some system cards do, below.

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Jul 21, 2025 Kimi K2 technical report 1.1 HHEM v2.1 (1-Hallu.), hallucination rate 1.1% (Table 3) Hallucination

One lab claim next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 1 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
Kimi K2 Jul 2025 1.10% HHEM v2.1 (1-Hallu.), hallucination rate 1.1% (Table 3) 1.10% Leaderboard Same

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

OpenAI Z.ai Google DeepMind
Vectara Hallucination Leaderboard (2025 dataset) 7 leaders
GPT-5.4 Nano 3.10%
Vectara Hallucination Leaderboard (HHEM) Replaced by 2, Nov 2025
Gemini 2.0 Flash 0.70%
Vectara Hallucination Leaderboard (2025 dataset) 2 leaders
GPT-5.4 Nano 3.10%

Source: Official leaderboards

Replaced by Vectara Hallucination Leaderboard (2025 dataset)

More in Factuality

All 74 in Factuality
  • SimpleVQA In use Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. Qwen3.7-Plus 81.7%
  • HallusionBench In use Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. Keye-VL-2.0-30B-A3B 73.5%
  • SimpleQA Verified In use 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. Gemini 3.1 Pro 77.5%
  • Kimi Hallu Bench In use Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. GPT-6 Astra 4.20%
  • Internal adversarially-designed factuality evaluations In use Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. GPT-6 Luna 7.60%
  • AA-Omniscience v1.1 In use Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. Inkling 2 Small 16.1