Benchmark Almanac

Vectara

We track 4 benchmarks that Vectara made or helped make.

0 in use 4 with public code 3 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Vectara Hallucination Leaderboard (HHEM) Models summarize short documents. The HHEM-2.1-Open classifier flags summaries that add unsupported facts, giving a hallucination rate. Faithful to sources Nov 2023 0.70% System cards only
FaithJudge Models answer retrieval-augmented tasks from given passages. An LLM judge guided by human-annotated examples flags unfaithful answers, giving a hallucination rate. Faithful to sources May 2025 6.26% System cards only
Vectara Hallucination Leaderboard (2025 dataset) Models summarize over 7,700 private articles of up to 32K tokens using only facts in the text. Vectara's commercial HHEM model flags hallucinated summaries. Faithful to sources Nov 2025 3.10% Not in cards
Open RAG Benchmark RAG systems answer 3,045 queries over 1,000 arXiv PDFs with text, tables, images, and hard negatives. Questions that need search Apr 2025 Not in cards