Benchmark Almanac

AbstentionBench

o1LowReasoningAPI 77.7% tops its leaderboard Qwen2.5-32B 75.3% leads the models we track
Qwen2.5-32B 75.3% leads since June 2025
Only in system cards

Models face underspecified, ill-posed or unanswerable questions. Scored on recall of appropriate abstention.

OpenAI reports abstention recall in its deception evaluations.

Released
June 2025
Kind
Hallucination rates
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Strongest model tested
Muse Spark #53 on the capability index
Last in a system card
June 2026
Last model added
June 2026
Scores refreshed
November 12th, 2026

In the authors’ words

For Large Language Models (LLMs) to be reliably deployed in both everyday and high-stakes domains, knowing when not to answer is equally critical as answering correctly. Real-world user queries, which can be underspecified, ill-posed, or fundamentally unanswerable, require LLMs to reason about uncertainty and selectively abstain – i.e., refuse to answer definitively. However, abstention remains understudied, without a systematic evaluation framework for modern LLMs. In this work, we introduce AbstentionBench, a large-scale benchmark for holistically evaluating abstention across 20 diverse datasets, including questions with unknown answers, underspecification, false premises, subjective interpretations, and outdated information. Evaluating 20 frontier LLMs reveals abstention is an unsolved problem, and one where scaling models is of little use. While recent reasoning LLMs have shown impressive results in complex problem solving, surprisingly, we find that reasoning fine-tuning degrades abstention (by 24% on average), even for math and science domains on which reasoning models are explicitly trained. We find that while a carefully crafted system prompt can boost abstention in practice, it does not resolve models' fundamental inability to reason about uncertainty. We release AbstentionBench to foster research into advancing LLM reliability.

Abstract of AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (arXiv:2506.09038, CC0)
23
Entries on its own leaderboard, 5 models we track
2
System cards report it, from 2 labs

o1LowReasoningAPI tops the leaderboard with 77.7%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Qwen2.5-32B, at 75.3%.

Qwen2.5-32B leads with 75.3%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in January 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

23 entries. Scroll sideways for the rest.

  1. 77.7 o1LowReasoningAPI
  2. 76.5 GPT-4o (snapshot unspecified)
  3. 76.1 o1HighReasoningAPI
  4. 75.3 Qwen2.5-32B
  5. 74.8 o1
  6. 74.3 Gemini 1.5 Pro (snapshot unspecified)
  7. 73.9 Llama 3.1 405B Instruct
  8. 73.8 Llama 3.1 70B Tulu 3 DPO
  9. 73.0 Llama 3.1 70B Tulu 3 PPO RLVF
  10. 71.6 Llama 3.1 70B Instruct
  11. 71.4 Llama 3.3 70B Instruct
  12. 70.6 Llama 3.1 8B Instruct
  13. 70.3 Mistral 7B v0.3
  14. 67.6 Llama 3.1 70B Tulu 3 SFT
  15. 63.2 Llama 3.1 8B Tulu 3 DPO
  16. 62.6 Llama 3.1 8B Tulu 3 PPO RLVF
  17. 62.0 OLMo 7B
  18. 58.2 DeepSeek-R1-Distill-Llama-70B
  19. 57.2 Llama 3.1 8B Tulu 3 SFT
  20. 55.1 Llama 3.1 70B Base
  21. 54.9 S1.1 32B
  22. 53.3 Llama 3.1 8B Base
  23. 37.2 TinyLlamaChat
  1. 1 o1LowReasoningAPI macro F1; 4 datasets 77.7%
  2. 2 GPT-4o (snapshot unspecified) macro F1; 31 datasets 76.5%
  3. 3 o1HighReasoningAPI macro F1; 4 datasets 76.1%
  4. 4 Qwen2.5-32B macro F1; 31 datasets 75.3%
  5. 5 o1 macro F1; 31 datasets 74.8%
  6. 6 Gemini 1.5 Pro (snapshot unspecified) macro F1; 31 datasets 74.3%
  7. 7 Llama 3.1 405B Instruct macro F1; 30 datasets 73.9%
  8. 8 Llama 3.1 70B Tulu 3 DPO macro F1; 31 datasets; DPO 73.8%
  9. 9 Llama 3.1 70B Tulu 3 PPO RLVF macro F1; 30 datasets; PPO RLVF 73%
  10. 10 Llama 3.1 70B Instruct macro F1; 31 datasets; Instruct 71.6%
  11. 11 Llama 3.3 70B Instruct macro F1; 31 datasets 71.4%
  12. 12 Llama 3.1 8B Instruct macro F1; 31 datasets; Instruct 70.6%
  13. 13 Mistral 7B v0.3 macro F1; 30 datasets 70.3%
  14. 14 Llama 3.1 70B Tulu 3 SFT macro F1; 30 datasets; SFT 67.6%
  15. 15 Llama 3.1 8B Tulu 3 DPO macro F1; 31 datasets; DPO 63.2%
  16. 16 Llama 3.1 8B Tulu 3 PPO RLVF macro F1; 31 datasets; PPO RLVF 62.6%
  17. 17 OLMo 7B macro F1; 30 datasets 62%
  18. 18 DeepSeek-R1-Distill-Llama-70B macro F1; 31 datasets 58.2%
  19. 19 Llama 3.1 8B Tulu 3 SFT macro F1; 31 datasets; SFT 57.2%
  20. 20 Llama 3.1 70B Base macro F1; 30 datasets; Base 55.1%
  21. 21 S1.1 32B macro F1; 31 datasets 54.9%
  22. 22 Llama 3.1 8B Base macro F1; 30 datasets; Base 53.3%
  23. 23 TinyLlamaChat macro F1; 3 datasets 37.2%

Source: Official AbstentionBench leaderboard

Sources: Official AbstentionBench leaderboard and labs’ model cards

#EntryScoreSettingDateSource
1 o1LowReasoningAPI 77.7% macro F1; 4 datasets Official leaderboard
2 GPT-4o (snapshot unspecified) 76.5% macro F1; 31 datasets Official leaderboard
3 o1HighReasoningAPI 76.1% macro F1; 4 datasets Official leaderboard
4 Qwen2.5-32B 75.3% macro F1; 31 datasets Sep 2024 Official leaderboard
5 o1 74.8% macro F1; 31 datasets Dec 2024 Official leaderboard
6 Gemini 1.5 Pro (snapshot unspecified) 74.3% macro F1; 31 datasets Official leaderboard
7 Llama 3.1 405B Instruct 73.9% macro F1; 30 datasets Official leaderboard
8 Llama 3.1 70B Tulu 3 DPO 73.8% macro F1; 31 datasets; DPO Official leaderboard
9 Llama 3.1 70B Tulu 3 PPO RLVF 73% macro F1; 30 datasets; PPO RLVF Official leaderboard
10 Llama 3.1 70B Instruct 71.6% macro F1; 31 datasets; Instruct Official leaderboard
11 Llama 3.3 70B Instruct 71.4% macro F1; 31 datasets Dec 2024 Official leaderboard
12 Llama 3.1 8B Instruct 70.6% macro F1; 31 datasets; Instruct Official leaderboard
13 Mistral 7B v0.3 70.3% macro F1; 30 datasets Sep 2023 Official leaderboard
14 Llama 3.1 70B Tulu 3 SFT 67.6% macro F1; 30 datasets; SFT Official leaderboard
15 Llama 3.1 8B Tulu 3 DPO 63.2% macro F1; 31 datasets; DPO Official leaderboard
16 Llama 3.1 8B Tulu 3 PPO RLVF 62.6% macro F1; 31 datasets; PPO RLVF Official leaderboard
17 OLMo 7B 62% macro F1; 30 datasets Official leaderboard
18 DeepSeek-R1-Distill-Llama-70B 58.2% macro F1; 31 datasets Jan 2025 Official leaderboard
19 Llama 3.1 8B Tulu 3 SFT 57.2% macro F1; 31 datasets; SFT Official leaderboard
20 Llama 3.1 70B Base 55.1% macro F1; 30 datasets; Base Official leaderboard
21 S1.1 32B 54.9% macro F1; 31 datasets Official leaderboard
22 Llama 3.1 8B Base 53.3% macro F1; 30 datasets; Base Official leaderboard
23 TinyLlamaChat 37.2% macro F1; 3 datasets Official leaderboard
#ModelScoreSettingDateSource
1 Qwen2.5-32B 75.3% macro F1; 31 datasets Sep 2024 Official leaderboard
2 o1 74.8% macro F1; 31 datasets Dec 2024 Official leaderboard
3 Llama 3.3 70B Instruct 71.4% macro F1; 31 datasets Dec 2024 Official leaderboard
4 Mistral 7B v0.3 70.3% macro F1; 30 datasets Sep 2023 Official leaderboard
5 Muse Spark 59.1% Jun 2026 Lab model card
6 DeepSeek-R1-Distill-Llama-70B 58.2% macro F1; 31 datasets Jan 2025 Official leaderboard
7 GPT-5 53% Aug 2025 Lab model card

Labs report scores for 2 models in their cards. Switch the scores above to rank them together.

Number one over time Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen
Own leaderboard Not tested since Jun 2025
Qwen2.5-32B 75.3%

Source: Official AbstentionBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Qwen2.5-32B Mistral 7B v0.3 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Qwen2.5-32B

Sources: Official AbstentionBench leaderboard and labs’ model cards

Labs report it only in system cards

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark. Some system cards do, below.

2 system cards report it, from 2 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Jun 12, 2026 Muse Spark system card 59.1 AbstentionBench correct abstention rate, macro-average recall (Figure 44a) Hallucination
Aug 7, 2025 GPT-5 system card 53 AbstentionBench, recall (Table 9) Alignment

More in Factuality

All 74 in Factuality
  • SimpleVQA In use Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. Qwen3.7-Plus 81.7%
  • HallusionBench In use Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. Keye-VL-2.0-30B-A3B 73.5%
  • SimpleQA Verified In use 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. Gemini 3.1 Pro 77.5%
  • Kimi Hallu Bench In use Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. GPT-6 Astra 4.20%
  • Internal adversarially-designed factuality evaluations In use Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. GPT-6 Luna 7.60%
  • AA-Omniscience v1.1 In use Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. Inkling 2 Small 16.1