ChatRAG Bench
Models answer multi-turn conversational questions over documents or retrieved context. The suite is built from 10 existing datasets such as Doc2Dial, QuAC, QReCC and CoQA.
Introduced with NVIDIA's ChatQA models. About 120 likes on Hugging Face.
- Made by
- NVIDIA
- Released
- January 2024
- Kind
- Finding facts in long inputs
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- All public
- Strongest model tested
-
Llama 3-70B #181 on the capability index - Last in a system card
- April 2025
- Last model added
- April 2025
- Scores refreshed
- November 12th, 2026
ChatQA-1.5-70B tops the leaderboard with 58.3
Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is GPT-4, at 53.9.
The newest model with a score here came out in August 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- Claude Fable 5.2
-
Gemini 4 Carbon - GPT-6 Astra
- Claude Opus 5.5
- and 16 more
- 1 ChatQA-1.5-70B 58.3
- 2 ChatQA-1.5-8B 55.2
- 3 ChatQA-1.0-70B 54.1
- 4 GPT-4-Turbo (snapshot unspecified) 54.0
- 5 GPT-4 Jun 2023 53.9
- 6 Llama 3-70B 52.5
- 7
Command R+ 50.9 - 8 ChatQA-1.0-7B 47.7
Source: Official ChatRAG Bench leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | ChatQA-1.5-70B | 58.3 | Official leaderboard | ||
| 2 | ChatQA-1.5-8B | 55.2 | Official leaderboard | ||
| 3 | ChatQA-1.0-70B | 54.1 | Official leaderboard | ||
| 4 | GPT-4-Turbo (snapshot unspecified) | 54.0 | Official leaderboard | ||
| 5 | GPT-4 Jun 2023 | 53.9 | Jun 2023 | Official leaderboard | |
| 6 | Llama 3-70B | 52.5 | Apr 2024 | Official leaderboard | |
| 7 | Command R+ | 50.9 | Aug 2024 | Official leaderboard | |
| 8 | ChatQA-1.0-7B | 47.7 | Official leaderboard |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official ChatRAG Bench leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official ChatRAG Bench leaderboard
Labs report it only in system cards
We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark. Some system cards do, below.
One system card reports it
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Apr 1, 2025 | Cohere Command A technical report | 72.9 | ChatRAGBench, average of ten datasets, LLM-jury correctness (Table 4) | Capabilities |
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
- LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy.
Inkling 2 Small 70.6% - GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. Gemini 3.7 Flash 89%
- CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.
Step 5 Preview 88.3%