Benchmark Almanac

BullshitBench

Claude Opus 4.8 96.4% leads since May 2026
Not in model cards

Models get prompts built on nonsensical premises. A three-judge model panel grades whether each answer pushes back clearly, challenges partly or accepts the nonsense.

Repo has about 1,900 GitHub stars. The V1 question set stays unchanged after V2.

Made by
Peter Gostev
Released
February 2026
Kind
Hallucination rates
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Strongest model tested
GPT-6 Astra #3 on the capability index
Last model added
September 2026
Scores refreshed
November 12th, 2026
115
Entries on its own leaderboard, 107 models we track

Claude Opus 4.8 leads with 96.4% among models released by September 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in September 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

115 entries. Scroll sideways for the rest.

  1. 96.4 Claude Opus 4.8
  2. 94.5 Claude Sonnet 4.6
  3. 92.7 Claude Opus 4.6
  4. 90.9 Claude Opus 4.5
  5. 87.3 Claude Haiku 4.5
  6. 87.3 Qwen 3.8 Max
  7. 80.0 Claude Opus 4.7
  8. 76.4 Qwen 3.8 27B
  9. 71.7 Claude Fable 5.1
  10. 67.3 Grok 4.20 Multi-agent
  11. 65.5 Claude Sonnet 4.5
  12. 65.5 Qwen3.5 397B-A17B
  13. 65.5 Nemotron 3 Super
  14. 65.5 Claude Sonnet 5
  15. 63.6 Grok 4.6
  16. 61.8 Grok 4.20
  17. 61.8 Grok 4.3
  18. 60.0 Grok 4.5
  19. 60.0 Claude Opus 5
  20. 58.2 Claude 3.5 Haiku
  21. 58.2 Kimi K3
  22. 54.5 Llama 4 Maverick
  23. 53.7 Claude Opus 5.5
  24. 52.7 Claude 3.5 Sonnet
  25. 52.7 Claude 3.7 Sonnet
  26. 52.7 Qwen3.7-Max
  27. 52.7 Gemini 3.5 Flash-Lite
  28. 50.9 GLM-5.3
  29. 47.3 Kimi K2.5
  30. 47.3 MiniMax-M3
  31. 47.3 GPT-6 Astra
  32. 45.5 MiMo-V2.5-Pro
  33. 43.6 stealth/ox-alpha
  34. 41.8 Nemotron 3 Ultra
  35. 41.8 DeepSeek V4.1 Flash
  36. 41.8 GPT-6 Sol
  37. 40.0 openai/gpt-6-luna-pro
  38. 40.0 Claude Opus 4.1
  39. 40.0 DeepSeek V4 Flash 0731
  40. 40.0 Muse Spark 1.2
  41. 40.0 GPT-6 Luna
  42. 38.2 openai/gpt-6-sol-pro
  43. 36.4 Llama 4 Scout
  44. 36.4 GPT-5.1 Instant
  45. 36.4 GPT-5.2 Codex
  46. 34.5 Llama 3.1-8B
  47. 34.5 GPT-5.6 Terra
  48. 32.7 openrouter/hunter-alpha
  49. 32.7 Claude Opus 4
  50. 32.7 DeepSeek V4 Pro 0813
  51. 30.9 GPT-5-Codex
  52. 30.9 Gemini 3 Pro
  53. 30.9 MiMo-V2.5
  54. 29.1 openrouter/healer-alpha
  55. 29.1 GPT-5.1-Codex
  56. 27.3 GPT-5.2
  57. 27.3 Nemotron 3 Nano 30B-A3B
  58. 27.3 GPT-5.3 Instant
  59. 27.3 GPT-5.5
  60. 27.3 GPT-5.6 Sol
  61. 25.4 Claude Sonnet 4
  62. 25.4 GPT-5.1
  63. 25.4 Step 3.5 Flash
  64. 25.4 GPT-5.4
  65. 25.4 GPT-5.4 Mini
  66. 25.4 GPT-5.6 Luna
  67. 23.6 o3
  68. 23.6 Gemini 2.5 Pro
  69. 23.6 Laguna S 2.1
  70. 23.6 Gemini 3.7 Flash
  71. 21.8 GPT-5
  72. 21.8 Gemini 3.1 Pro
  73. 21.8 Gemma 4 31B IT
  74. 20.0 Qwen3-Coder-480B-A35B
  75. 20.0 Kimi K2
  76. 20.0 GLM-5
  77. 20.0 Trinity Large Thinking
  78. 18.2 Claude 3 Haiku
  79. 18.2 GLM-5.2
  80. 18.2 Gemini 3.6 Flash
  81. 16.4 GPT-4o
  82. 16.4 MiMo-V2-Flash
  83. 16.4 MiniMax-M2.5
  84. 16.4 GPT-5.4 Nano
  85. 16.4 DeepSeek-V4-Flash
  86. 14.6 baidu/ernie-4.5-vl-424b-a47b
  87. 14.6 openai/gpt-5.5-chat
  88. 14.6 GPT-4.1
  89. 14.6 GLM-4.5
  90. 14.6 GPT-5 Chat
  91. 14.6 Gemini 2.5 Flash
  92. 14.6 Grok 4.1 Fast
  93. 14.6 DeepSeek-V3.2
  94. 14.6 GPT-5.2 Instant
  95. 14.6 GPT-5.3 Codex
  96. 14.6 Gemma 4 26B A4B
  97. 14.6 DeepSeek-V4-Pro
  98. 12.7 Gemini 2.0 Flash
  99. 12.7 DeepSeek-R1
  100. 12.7 ERNIE-4.5-300B-A47B
  101. 12.7 INTELLECT-3
  102. 12.7 Tencent Hy3 preview
  103. 10.9 o4-mini
  104. 10.9 Doubao-Seed-1.6
  105. 10.9 Gemini 3 Flash
  106. 9.09 Gemini 3.1 Flash-Lite
  107. 9.09 Gemini 3.5 Flash
  108. 7.27 GPT-4o mini
  109. 7.27 Qwen3-235B-A22B
  110. 7.27 Jamba 1.7 Large
  111. 7.27 nvidia-nemotron-nano-9b-v2
  112. 5.45 deepseek/deepseek-chat
  113. 3.64 Gemma 3 27B
  114. 3.64 gpt-oss-120b
  115. 3.64 Mistral Large 3
  1. 1 Claude Opus 4.8 none 96.4%
  2. 2 Claude Sonnet 4.6 none 94.5%
  3. 3 Claude Opus 4.6 none 92.7%
  4. 4 Claude Opus 4.5 high 90.9%
  5. 5 Claude Haiku 4.5 high 87.3%
  6. 6 Qwen 3.8 Max minimal 87.3%
  7. 7 Claude Opus 4.7 none 80%
  8. 8 Qwen 3.8 27B none 76.4%
  9. 9 Claude Fable 5.1 low 71.7%
  10. 10 Grok 4.20 Multi-agent xhigh 67.3%
  11. 11 Claude Sonnet 4.5 high 65.5%
  12. 12 Qwen3.5 397B-A17B none 65.5%
  13. 13 Nemotron 3 Super xhigh 65.5%
  14. 14 Claude Sonnet 5 low 65.5%
  15. 15 Grok 4.6 minimal 63.6%
  16. 16 Grok 4.20 low 61.8%
  17. 17 Grok 4.3 minimal 61.8%
  18. 18 Grok 4.5 high 60%
  19. 19 Claude Opus 5 low 60%
  20. 20 Claude 3.5 Haiku 58.2%
  21. 21 Kimi K3 minimal 58.2%
  22. 22 Llama 4 Maverick 54.5%
  23. 23 Claude Opus 5.5 low 53.7%
  24. 24 Claude 3.5 Sonnet 52.7%
  25. 25 Claude 3.7 Sonnet thinking 52.7%
  26. 26 Qwen3.7-Max none 52.7%
  27. 27 Gemini 3.5 Flash-Lite minimal 52.7%
  28. 28 GLM-5.3 minimal 50.9%
  29. 29 Kimi K2.5 none 47.3%
  30. 30 MiniMax-M3 xhigh 47.3%
  31. 31 GPT-6 Astra max 47.3%
  32. 32 MiMo-V2.5-Pro xhigh 45.5%
  33. 33 stealth/ox-alpha low 43.6%
  34. 34 Nemotron 3 Ultra none 41.8%
  35. 35 DeepSeek V4.1 Flash max 41.8%
  36. 36 GPT-6 Sol max 41.8%
  37. 37 openai/gpt-6-luna-pro none 40%
  38. 38 Claude Opus 4.1 none 40%
  39. 39 DeepSeek V4 Flash 0731 xhigh 40%
  40. 40 Muse Spark 1.2 minimal 40%
  41. 41 GPT-6 Luna none 40%
  42. 42 openai/gpt-6-sol-pro none 38.2%
  43. 43 Llama 4 Scout 36.4%
  44. 44 GPT-5.1 Instant 36.4%
  45. 45 GPT-5.2 Codex low 36.4%
  46. 46 Llama 3.1-8B 34.5%
  47. 47 GPT-5.6 Terra max 34.5%
  48. 48 openrouter/hunter-alpha none 32.7%
  49. 49 Claude Opus 4 32.7%
  50. 50 DeepSeek V4 Pro 0813 max 32.7%
  51. 51 GPT-5-Codex 30.9%
  52. 52 Gemini 3 Pro low 30.9%
  53. 53 MiMo-V2.5 none 30.9%
  54. 54 openrouter/healer-alpha none 29.1%
  55. 55 GPT-5.1-Codex 29.1%
  56. 56 GPT-5.2 none 27.3%
  57. 57 Nemotron 3 Nano 30B-A3B none 27.3%
  58. 58 GPT-5.3 Instant 27.3%
  59. 59 GPT-5.5 none 27.3%
  60. 60 GPT-5.6 Sol max 27.3%
  61. 61 Claude Sonnet 4 none 25.4%
  62. 62 GPT-5.1 25.4%
  63. 63 Step 3.5 Flash minimal 25.4%
  64. 64 GPT-5.4 none 25.4%
  65. 65 GPT-5.4 Mini xhigh 25.4%
  66. 66 GPT-5.6 Luna low 25.4%
  67. 67 o3 low 23.6%
  68. 68 Gemini 2.5 Pro Jun 2025 23.6%
  69. 69 Laguna S 2.1 xhigh 23.6%
  70. 70 Gemini 3.7 Flash minimal 23.6%
  71. 71 GPT-5 21.8%
  72. 72 Gemini 3.1 Pro low 21.8%
  73. 73 Gemma 4 31B IT high 21.8%
  74. 74 Qwen3-Coder-480B-A35B 20%
  75. 75 Kimi K2 Sep 2025 20%
  76. 76 GLM-5 high 20%
  77. 77 Trinity Large Thinking xhigh 20%
  78. 78 Claude 3 Haiku 18.2%
  79. 79 GLM-5.2 xhigh 18.2%
  80. 80 Gemini 3.6 Flash minimal 18.2%
  81. 81 GPT-4o Aug 2024 16.4%
  82. 82 MiMo-V2-Flash none 16.4%
  83. 83 MiniMax-M2.5 high 16.4%
  84. 84 GPT-5.4 Nano none 16.4%
  85. 85 DeepSeek-V4-Flash none 16.4%
  86. 86 baidu/ernie-4.5-vl-424b-a47b xhigh 14.6%
  87. 87 openai/gpt-5.5-chat 14.6%
  88. 88 GPT-4.1 14.6%
  89. 89 GLM-4.5 xhigh 14.6%
  90. 90 GPT-5 Chat 14.6%
  91. 91 Gemini 2.5 Flash Sep 2025 14.6%
  92. 92 Grok 4.1 Fast high 14.6%
  93. 93 DeepSeek-V3.2 high 14.6%
  94. 94 GPT-5.2 Instant 14.6%
  95. 95 GPT-5.3 Codex low 14.6%
  96. 96 Gemma 4 26B A4B xhigh 14.6%
  97. 97 DeepSeek-V4-Pro none 14.6%
  98. 98 Gemini 2.0 Flash Feb 2025 12.7%
  99. 99 DeepSeek-R1 May 2025, none 12.7%
  100. 100 ERNIE-4.5-300B-A47B 12.7%
  101. 101 INTELLECT-3 low 12.7%
  102. 102 Tencent Hy3 preview xhigh 12.7%
  103. 103 o4-mini high 10.9%
  104. 104 Doubao-Seed-1.6 none 10.9%
  105. 105 Gemini 3 Flash high 10.9%
  106. 106 Gemini 3.1 Flash-Lite 9.09%
  107. 107 Gemini 3.5 Flash minimal 9.09%
  108. 108 GPT-4o mini 7.27%
  109. 109 Qwen3-235B-A22B none 7.27%
  110. 110 Jamba 1.7 Large 7.27%
  111. 111 nvidia-nemotron-nano-9b-v2 xhigh 7.27%
  112. 112 deepseek/deepseek-chat 5.45%
  113. 113 Gemma 3 27B 3.64%
  114. 114 gpt-oss-120b low 3.64%
  115. 115 Mistral Large 3 3.64%

Source: Official BullshitBench leaderboard

#EntryScoreSettingDateSource
1 Claude Opus 4.8 96.4% none May 2026 Official leaderboard
2 Claude Sonnet 4.6 94.5% none Feb 2026 Official leaderboard
3 Claude Opus 4.6 92.7% none Feb 2026 Official leaderboard
4 Claude Opus 4.5 90.9% high Nov 2025 Official leaderboard
5 Claude Haiku 4.5 87.3% high Oct 2025 Official leaderboard
6 Qwen 3.8 Max 87.3% minimal Aug 2026 Official leaderboard
7 Claude Opus 4.7 80% none Apr 2026 Official leaderboard
8 Qwen 3.8 27B 76.4% none Aug 2026 Official leaderboard
9 Claude Fable 5.1 71.7% low Sep 2026 Official leaderboard
10 Grok 4.20 Multi-agent 67.3% xhigh Mar 2026 Official leaderboard
11 Claude Sonnet 4.5 65.5% high Sep 2025 Official leaderboard
12 Qwen3.5 397B-A17B 65.5% none Feb 2026 Official leaderboard
13 Nemotron 3 Super 65.5% xhigh Mar 2026 Official leaderboard
14 Claude Sonnet 5 65.5% low Jun 2026 Official leaderboard
15 Grok 4.6 63.6% minimal Aug 2026 Official leaderboard
16 Grok 4.20 61.8% low Feb 2026 Official leaderboard
17 Grok 4.3 61.8% minimal Apr 2026 Official leaderboard
18 Grok 4.5 60% high Jul 2026 Official leaderboard
19 Claude Opus 5 60% low Jul 2026 Official leaderboard
20 Claude 3.5 Haiku 58.2% Oct 2024 Official leaderboard
21 Kimi K3 58.2% minimal Jul 2026 Official leaderboard
22 Llama 4 Maverick 54.5% Apr 2025 Official leaderboard
23 Claude Opus 5.5 53.7% low Sep 2026 Official leaderboard
24 Claude 3.5 Sonnet 52.7% Jun 2024 Official leaderboard
25 Claude 3.7 Sonnet 52.7% thinking Feb 2025 Official leaderboard
26 Qwen3.7-Max 52.7% none May 2026 Official leaderboard
27 Gemini 3.5 Flash-Lite 52.7% minimal Jul 2026 Official leaderboard
28 GLM-5.3 50.9% minimal Aug 2026 Official leaderboard
29 Kimi K2.5 47.3% none Jan 2026 Official leaderboard
30 MiniMax-M3 47.3% xhigh Jun 2026 Official leaderboard
31 GPT-6 Astra 47.3% max Sep 2026 Official leaderboard
32 MiMo-V2.5-Pro 45.5% xhigh Apr 2026 Official leaderboard
33 stealth/ox-alpha 43.6% low Official leaderboard
34 Nemotron 3 Ultra 41.8% none Jun 2026 Official leaderboard
35 DeepSeek V4.1 Flash 41.8% max Sep 2026 Official leaderboard
36 GPT-6 Sol 41.8% max Sep 2026 Official leaderboard
37 openai/gpt-6-luna-pro 40% none Official leaderboard
38 Claude Opus 4.1 40% none Aug 2025 Official leaderboard
39 DeepSeek V4 Flash 0731 40% xhigh Jul 2026 Official leaderboard
40 Muse Spark 1.2 40% minimal Aug 2026 Official leaderboard
41 GPT-6 Luna 40% none Sep 2026 Official leaderboard
42 openai/gpt-6-sol-pro 38.2% none Official leaderboard
43 Llama 4 Scout 36.4% Apr 2025 Official leaderboard
44 GPT-5.1 Instant 36.4% Nov 2025 Official leaderboard
45 GPT-5.2 Codex 36.4% low Dec 2025 Official leaderboard
46 Llama 3.1-8B 34.5% Jul 2024 Official leaderboard
47 GPT-5.6 Terra 34.5% max Jul 2026 Official leaderboard
48 openrouter/hunter-alpha 32.7% none Official leaderboard
49 Claude Opus 4 32.7% May 2025 Official leaderboard
50 DeepSeek V4 Pro 0813 32.7% max Aug 2026 Official leaderboard
51 GPT-5-Codex 30.9% Sep 2025 Official leaderboard
52 Gemini 3 Pro 30.9% low Nov 2025 Official leaderboard
53 MiMo-V2.5 30.9% none Apr 2026 Official leaderboard
54 openrouter/healer-alpha 29.1% none Official leaderboard
55 GPT-5.1-Codex 29.1% Nov 2025 Official leaderboard
56 GPT-5.2 27.3% none Dec 2025 Official leaderboard
57 Nemotron 3 Nano 30B-A3B 27.3% none Dec 2025 Official leaderboard
58 GPT-5.3 Instant 27.3% Mar 2026 Official leaderboard
59 GPT-5.5 27.3% none Apr 2026 Official leaderboard
60 GPT-5.6 Sol 27.3% max Jul 2026 Official leaderboard
61 Claude Sonnet 4 25.4% none May 2025 Official leaderboard
62 GPT-5.1 25.4% Nov 2025 Official leaderboard
63 Step 3.5 Flash 25.4% minimal Feb 2026 Official leaderboard
64 GPT-5.4 25.4% none Mar 2026 Official leaderboard
65 GPT-5.4 Mini 25.4% xhigh Mar 2026 Official leaderboard
66 GPT-5.6 Luna 25.4% low Jul 2026 Official leaderboard
67 o3 23.6% low Apr 2025 Official leaderboard
68 Gemini 2.5 Pro Jun 2025 23.6% Jun 2025 Official leaderboard
69 Laguna S 2.1 23.6% xhigh Jul 2026 Official leaderboard
70 Gemini 3.7 Flash 23.6% minimal Aug 2026 Official leaderboard
71 GPT-5 21.8% Aug 2025 Official leaderboard
72 Gemini 3.1 Pro 21.8% low Feb 2026 Official leaderboard
73 Gemma 4 31B IT 21.8% high Apr 2026 Official leaderboard
74 Qwen3-Coder-480B-A35B 20% Jul 2025 Official leaderboard
75 Kimi K2 Sep 2025 20% Sep 2025 Official leaderboard
76 GLM-5 20% high Feb 2026 Official leaderboard
77 Trinity Large Thinking 20% xhigh Apr 2026 Official leaderboard
78 Claude 3 Haiku 18.2% Mar 2024 Official leaderboard
79 GLM-5.2 18.2% xhigh Jun 2026 Official leaderboard
80 Gemini 3.6 Flash 18.2% minimal Jul 2026 Official leaderboard
81 GPT-4o Aug 2024 16.4% Aug 2024 Official leaderboard
82 MiMo-V2-Flash 16.4% none Dec 2025 Official leaderboard
83 MiniMax-M2.5 16.4% high Feb 2026 Official leaderboard
84 GPT-5.4 Nano 16.4% none Mar 2026 Official leaderboard
85 DeepSeek-V4-Flash 16.4% none Apr 2026 Official leaderboard
86 baidu/ernie-4.5-vl-424b-a47b 14.6% xhigh Official leaderboard
87 openai/gpt-5.5-chat 14.6% Official leaderboard
88 GPT-4.1 14.6% Apr 2025 Official leaderboard
89 GLM-4.5 14.6% xhigh Aug 2025 Official leaderboard
90 GPT-5 Chat 14.6% Aug 2025 Official leaderboard
91 Gemini 2.5 Flash Sep 2025 14.6% Sep 2025 Official leaderboard
92 Grok 4.1 Fast 14.6% high Nov 2025 Official leaderboard
93 DeepSeek-V3.2 14.6% high Dec 2025 Official leaderboard
94 GPT-5.2 Instant 14.6% Dec 2025 Official leaderboard
95 GPT-5.3 Codex 14.6% low Feb 2026 Official leaderboard
96 Gemma 4 26B A4B 14.6% xhigh Apr 2026 Official leaderboard
97 DeepSeek-V4-Pro 14.6% none Apr 2026 Official leaderboard
98 Gemini 2.0 Flash Feb 2025 12.7% Feb 2025 Official leaderboard
99 DeepSeek-R1 May 2025 12.7% none May 2025 Official leaderboard
100 ERNIE-4.5-300B-A47B 12.7% Jun 2025 Official leaderboard
101 INTELLECT-3 12.7% low Nov 2025 Official leaderboard
102 Tencent Hy3 preview 12.7% xhigh Apr 2026 Official leaderboard
103 o4-mini 10.9% high Apr 2025 Official leaderboard
104 Doubao-Seed-1.6 10.9% none Jun 2025 Official leaderboard
105 Gemini 3 Flash 10.9% high Dec 2025 Official leaderboard
106 Gemini 3.1 Flash-Lite 9.09% Mar 2026 Official leaderboard
107 Gemini 3.5 Flash 9.09% minimal May 2026 Official leaderboard
108 GPT-4o mini 7.27% Jul 2024 Official leaderboard
109 Qwen3-235B-A22B 7.27% none Apr 2025 Official leaderboard
110 Jamba 1.7 Large 7.27% Jul 2025 Official leaderboard
111 nvidia-nemotron-nano-9b-v2 7.27% xhigh Aug 2025 Official leaderboard
112 deepseek/deepseek-chat 5.45% Official leaderboard
113 Gemma 3 27B 3.64% Mar 2025 Official leaderboard
114 gpt-oss-120b 3.64% low Aug 2025 Official leaderboard
115 Mistral Large 3 3.64% Dec 2025 Official leaderboard

The best score, 96.4%, costs $0.01 a task

Runs on the cost frontier score higher than any run that's cheaper. Claude Sonnet 4.6 at none comes within 1.8 points of the best for $0.005 a task, 3 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Meta Anthropic OpenAI Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.00001 $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale Claude Opus 4.8, none Claude Haiku 4.5, high Claude Haiku 4.5, none Claude 3.5 Haiku GPT-6 Luna, none Llama 3.1-8B 0% 20% 40% 60% 80% 100% $0.00001 $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale Claude Opus 4.8, none Claude Haiku 4.5, high

Source: Official BullshitBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 4.8 none 96.4% $0.01 584
Claude Sonnet 4.6 none 94.5% $0.005 358
Claude Opus 4.6 none 92.7% $0.01 388
Claude Sonnet 4.6 high 92.7% $0.01 731
Claude Opus 4.8 xhigh 92.7% $0.02 592
Claude Opus 4.5 high 90.9% $0.02 679
Claude Opus 4.6 high 89.1% $0.02 632
Claude Haiku 4.5 high 87.3% $0.003 633
Qwen 3.8 Max minimal 87.3% $0.004 600
Claude Haiku 4.5 none 81.8% $0.002 300
Claude Opus 4.5 none 80% $0.007 283
Claude Opus 4.7 none 80% $0.01 558
Qwen 3.8 27B none 76.4% $0.002 722
Claude Fable 5.1 low 71.7% $0.04 788
Grok 4.20 Multi-agent xhigh 67.3% $0.37 48,621
Grok 4.20 Multi-agent low 67.3% $0.08 10,753
Claude Sonnet 4.5 high 65.5% $0.008 549
Qwen3.5 397B-A17B none 65.5% $0.003 719
Nemotron 3 Super xhigh 65.5% 1,924
Claude Sonnet 5 low 65.5% $0.007 645
Grok 4.6 minimal 63.6% $0.004 645
Grok 4.20 low 61.8% $0.01 1,565
Grok 4.3 minimal 61.8% $0.005 1,779
Claude Sonnet 5 max 61.8% $0.06 6,370
Claude Fable 5.1 max 61.1% $0.16 3,242
Claude Opus 4.7 max 60% $0.05 1,938
Grok 4.5 high 60% $0.008 1,354
Claude Opus 5 low 60% $0.03 1,318
Grok 4.6 max 60% $0.009 1,487
Claude 3.5 Haiku 58.2% $0.0004 103
Grok 4.20 xhigh 58.2% $0.009 1,550
Kimi K3 minimal 58.2% $0.03 2,194
Claude Opus 5 xhigh 58.2% $0.07 2,709
Grok 4.3 xhigh 56.4% $0.004 1,585
Llama 4 Maverick 54.5% $0.0004 403
Grok 4.5 low 54.5% $0.005 814
Claude Opus 5.5 low 53.7% $0.02 974
Claude 3.5 Sonnet 52.7% $0.003 101
Claude 3.7 Sonnet thinking 52.7% $0.008 539
Claude Sonnet 4.5 none 52.7% $0.005 320
Qwen3.5 397B-A17B high 52.7% $0.007 2,016
Qwen3.7-Max none 52.7% $0.006 748
Gemini 3.5 Flash-Lite minimal 52.7% $0.001 415
GLM-5.3 minimal 50.9% $0.002 434
GLM-5.3 max 50.9% $0.01 3,288
Gemini 3.5 Flash-Lite xhigh 49.1% $0.003 1,282
Kimi K2.5 none 47.3% $0.001 356
MiniMax-M3 xhigh 47.3% $0.001 968
GPT-6 Astra max 47.3% $0.14 2,805
Claude 3.7 Sonnet 45.5% $0.003 189
MiMo-V2.5-Pro xhigh 45.5% $0.002 737
MiniMax-M3 none 45.5% $0.0006 449
Claude Opus 5.5 max 44.4% $0.25 12,445
Kimi K3 xhigh 43.6% $0.03 1,919
Nemotron 3 Ultra none 41.8% $0.003 1,133
DeepSeek V4.1 Flash max 41.8% $0.004 3,829
DeepSeek V4.1 Flash low 41.8% $0.002 2,120
GPT-6 Sol max 41.8% $0.02 1,551
Claude Opus 4.1 none 40% $0.02 284
DeepSeek V4 Flash 0731 xhigh 40% $0.0005 1,717
Muse Spark 1.2 minimal 40% $0.004 980
GPT-6 Luna none 40% $0.0001 252
Nemotron 3 Super none 38.2% 1,392
Qwen3.7-Max xhigh 38.2% $0.02 2,182
Llama 4 Scout 36.4% $0.0002 475
GPT-5.1 Instant 36.4% $0.002 235
GPT-5.2 Codex low 36.4% $0.005 348
Llama 3.1-8B 34.5% $0.00002 315
Claude Opus 4.1 high 34.5% $0.04 497
GPT-5.6 Terra max 34.5% $0.07 4,987
GPT-6 Luna max 34.5% $0.002 3,271
Claude Opus 4 32.7% $0.02 292
Muse Spark 1.2 xhigh 32.7% $0.007 1,694
DeepSeek V4 Pro 0813 max 32.7% $0.006 3,343
GPT-6 Sol none 32.7% $0.002 206
GPT-5-Codex 30.9% $0.008 832
Gemini 3 Pro low 30.9% $0.01 937
Kimi K2.5 high 30.9% $0.003 1,246
MiMo-V2.5 none 30.9% $0.001 508
Nemotron 3 Ultra xhigh 30.9% $0.004 1,565
GPT-5.1-Codex 29.1% $0.004 393
GPT-5.2 Codex high 29.1% $0.02 1,466
MiMo-V2.5-Pro none 29.1% $0.001 363
GPT-5.6 Terra low 29.1% $0.01 858
GPT-6 Astra low 29.1% $0.02 389
GPT-5.2 none 27.3% $0.01 980
Nemotron 3 Nano 30B-A3B none 27.3% 1,365
GPT-5.3 Instant 27.3% $0.006 394
GPT-5.5 none 27.3% 717
GPT-5.6 Sol max 27.3% $0.06 2,029
Claude Sonnet 4 none 25.4% $0.004 287
GPT-5.1 25.4% $0.02 1,723
GPT-5.2 Codex xhigh 25.4% $0.03 2,312
Step 3.5 Flash minimal 25.4% $0.0006 1,917
GPT-5.4 none 25.4% $0.01 665
GPT-5.4 Mini xhigh 25.4% $0.03 6,645
GPT-5.4 Mini high 25.4% $0.006 1,408
GPT-5.6 Luna low 25.4% $0.004 624
GPT-5.6 Sol low 25.4% $0.01 477
DeepSeek V4 Pro 0813 low 25.4% $0.003 1,808
o3 low 23.6% $0.009 1,093
Claude Sonnet 4 high 23.6% $0.008 551
Gemini 2.5 Pro Jun 2025 23.6% $0.03 3,215
Gemini 3 Pro high 23.6% $0.03 2,096
GPT-5.2 high 23.6% $0.02 1,539
GPT-5.4 Mini none 23.6% $0.002 490
Laguna S 2.1 xhigh 23.6% $0.0002 782
Gemini 3.7 Flash minimal 23.6% $0.002 1,158
Gemini 3.7 Flash max 23.6% $0.004 2,208
GPT-5 21.8% $0.03 2,662
Gemini 3.1 Pro low 21.8% $0.02 1,455
GPT-5.4 xhigh 21.8% $0.08 5,371
Gemma 4 31B IT high 21.8% $0.0006 1,384
DeepSeek V4 Flash 0731 none 21.8% $0.0003 1,131
Qwen 3.8 Max max 21.8% $0.03 4,156
Qwen3-Coder-480B-A35B 20% $0.0004 265
Kimi K2 Sep 2025 20% $0.001 581
Step 3.5 Flash xhigh 20% $0.0006 2,016
GLM-5 high 20% $0.008 2,420
Trinity Large Thinking xhigh 20% $0.001 1,471
MiMo-V2.5 xhigh 20% $0.002 874
Laguna S 2.1 none 20% $0.00008 398
Qwen 3.8 27B max 20% $0.01 5,375
Claude 3 Haiku 18.2% $0.0003 246
Gemini 3.1 Pro high 18.2% $0.03 2,086
Trinity Large Thinking minimal 18.2% $0.001 1,456
GLM-5.2 xhigh 18.2% $0.01 2,173
GPT-5.6 Luna max 18.2% $0.03 4,526
Gemini 3.6 Flash minimal 18.2% $0.008 1,037
GPT-4o Aug 2024 16.4% $0.004 376
MiMo-V2-Flash none 16.4% $0.0002 574
GLM-5 none 16.4% $0.003 981
MiniMax-M2.5 high 16.4% $0.003 2,258
GPT-5.4 Nano none 16.4% $0.001 896
DeepSeek-V4-Flash none 16.4% $0.0003 1,125
Gemini 3.6 Flash xhigh 16.4% $0.02 2,218
GPT-4o Aug 2024 14.6% $0.004 384
GPT-4.1 14.6% $0.005 670
GLM-4.5 xhigh 14.6% $0.004 1,966
GPT-5 Chat 14.6% $0.006 576
Gemini 2.5 Flash Sep 2025 14.6% $0.004 1,712
Grok 4.1 Fast high 14.6% $0.0006 1,256
DeepSeek-V3.2 high 14.6% $0.0005 1,232
GPT-5.2 Instant 14.6% $0.01 799
GPT-5.3 Codex low 14.6% $0.008 541
GPT-5.4 Nano xhigh 14.6% $0.005 4,308
Gemma 4 26B A4B xhigh 14.6% $0.0007 1,683
DeepSeek-V4-Pro none 14.6% $0.004 1,154
Gemini 2.0 Flash Feb 2025 12.7% $0.0006 1,413
DeepSeek-R1 May 2025 none 12.7% $0.003 1,274
ERNIE-4.5-300B-A47B 12.7% $0.001 977
INTELLECT-3 low 12.7% $0.002 2,158
DeepSeek-V3.2 none 12.7% $0.0004 877
GPT-5.3 Codex high 12.7% $0.01 1,064
MiniMax-M2.5 low 12.7% $0.002 2,050
Tencent Hy3 preview xhigh 12.7% 2,446
Tencent Hy3 preview none 12.7% 1,073
DeepSeek-V4-Flash xhigh 12.7% $0.002 5,516
o4-mini high 10.9% $0.01 2,226
Doubao-Seed-1.6 none 10.9% $0.003 1,488
Grok 4.1 Fast none 10.9% $0.0004 852
INTELLECT-3 high 10.9% $0.003 2,295
Nemotron 3 Nano 30B-A3B xhigh 10.9% 2,702
Gemini 3 Flash high 10.9% $0.005 1,534
GPT-5.3 Codex xhigh 10.9% $0.03 2,360
GPT-5.4 Nano high 10.9% $0.002 1,478
Gemma 4 31B IT none 10.9% $0.0003 755
DeepSeek-V4-Pro xhigh 10.9% $0.007 2,120
GLM-5.2 none 10.9% $0.004 1,002
o4-mini low 9.09% $0.004 827
MiMo-V2-Flash high 9.09% $0.0005 1,768
Gemini 3.1 Flash-Lite 9.09% $0.001 887
Gemma 4 26B A4B none 9.09% $0.0004 866
Gemini 3.5 Flash minimal 9.09% $0.01 1,149
Gemini 3.5 Flash xhigh 9.09% $0.02 2,380
GPT-4o mini 7.27% $0.0003 471
Qwen3-235B-A22B none 7.27% $0.002 934
DeepSeek-R1 May 2025 xhigh 7.27% $0.004 1,396
Doubao-Seed-1.6 high 7.27% $0.005 2,566
Jamba 1.7 Large 7.27% $0.006 683
nvidia-nemotron-nano-9b-v2 xhigh 7.27% 1,483
nvidia-nemotron-nano-9b-v2 none 7.27% 1,413
Gemini 3 Flash none 7.27% $0.003 852
GPT-4o mini 5.45% $0.0003 457
Qwen3-235B-A22B xhigh 5.45% $0.003 1,877
Gemma 3 27B 3.64% $0.0005 1,729
GLM-4.5 none 3.64% $0.005 2,115
gpt-oss-120b low 3.64% $0.0009 2,257
gpt-oss-120b high 3.64% $0.002 4,128
Mistral Large 3 3.64% $0.003 1,739

4 models have held number one on the official leaderboard since its release in February 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard Passed 85% in Feb 2026, by Claude Opus 4.5
Claude Opus 4.8 96.4%

Source: Official BullshitBench leaderboard

Claude Haiku 4.5 on the official leaderboard passed 85% in October 2025

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Claude Opus 4.8 Claude Opus 4.5 Claude Sonnet 4.5 Claude 3.5 Haiku Claude 3.5 Sonnet Claude 3 Haiku 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Claude Opus 4.8 Claude Haiku 4.5

Source: Official BullshitBench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

Anthropic
BullshitBench V2 Passed 85% in Mar 2026, by Claude Sonnet 4.6
Claude Opus 4.8 95%
BullshitBench Passed 85% in Feb 2026, by Claude Opus 4.5
Claude Opus 4.8 96.4%
BullshitBench V2 Passed 85% in Mar 2026, by Claude Sonnet 4.6
Claude Opus 4.8 95%

Source: Official leaderboards

Replaced by BullshitBench V2

More in Factuality

All 74 in Factuality
  • SimpleVQA In use Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. Qwen3.7-Plus 81.7%
  • HallusionBench In use Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. Keye-VL-2.0-30B-A3B 73.5%
  • SimpleQA Verified In use 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. Gemini 3.1 Pro 77.5%
  • Kimi Hallu Bench In use Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. GPT-6 Astra 4.20%
  • Internal adversarially-designed factuality evaluations In use Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. GPT-6 Luna 7.60%
  • AA-Omniscience v1.1 In use Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. Inkling 2 Small 16.1