Benchmark Almanac

Misguided Attention

Grok-3 mini 73.7% leads since April 2025
Not in model cards

Models get slight variations of famous riddles, paradoxes and thought experiments. A model fails when it answers the well-known original instead of the problem as stated.

Version 0.3 (2025-01) added an evaluation subset with tagged releases. About 490 GitHub stars.

Released
May 2024
Kind
Abstract puzzles and logic
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Strongest model tested
o4-mini #94 on the capability index
Last model added
June 2025
Scores refreshed
November 12th, 2026
52
Entries on its own leaderboard, 45 models we track

Grok-3 mini leads with 73.7% among models released by June 2025

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in June 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

52 entries. Scroll sideways for the rest.

  1. 73.7 Grok-3 mini
  2. 68.6 Claude Sonnet 4
  3. 67.1 DeepSeek-R1
  4. 64.7 Grok 3 Mini beta (unlisted snapshot)
  5. 64.1 Claude Opus 4
  6. 63.5 Gemini 2.5 Flash
  7. 62.5 Gemini 2.5 Flash
  8. 58.8 Gemini 2.5 Pro
  9. 57.4 Llama-3.3-Nemotron-Super-49B-v1
  10. 57.4 Gemini 2.5 Pro
  11. 56.1 Claude 3.7 Sonnet
  12. 56.1 QwQ-32B
  13. 55.2 Gemini 2.5 Pro
  14. 55.1 DeepSeek-R1
  15. 55.1 Grok 3
  16. 53.5 DeepSeek-V3
  17. 53.5 GLM-Z1-32B-0414
  18. 51.7 Magistral Medium 1.0
  19. 50.6 Mistral Medium 3
  20. 49.4 o4-mini-high-long
  21. 49.0 Qwen3-235B-A22B
  22. 48.7 GPT-4.1 mini
  23. 48.4 DeepHermes-3-Mistral-24B-Pre
  24. 48.4 Phi-4-Reasoning-plus
  25. 47.4 optimus-alpha
  26. 46.8 o3-mini
  27. 44.2 GPT-4.1
  28. 43.6 Qwen3-30B-A3B
  29. 42.9 o4-mini
  30. 42.6 DeepSeek-R1-Distill-Llama-70B
  31. 40.7 quasar-alpha
  32. 37.2 Claude 3.5 Sonnet (October 2024)
  33. 36.5 MiniMax-Text-01
  34. 35.9 Llama 4 Scout
  35. 35.3 Llama 3.3 70B Instruct
  36. 34.6 Gemini 2.0 Pro
  37. 33.7 GPT-4o
  38. 32.1 GPT-4o
  39. 31.4 Hermes-3-Llama-3.1-70B
  40. 31.4 Qwen2.5-Max
  41. 30.1 Cohere Command A
  42. 29.5 Llama 4 Maverick
  43. 28.2 Gemini 2.0 Flash
  44. 27.9 Mistral Small 3
  45. 26.9 DeepSeek-V3
  46. 26.6 GPT-4.1 nano
  47. 26.0 Mistral Small 3.1
  48. 25.0 Jamba 1.6 Large
  49. 23.4 Gemma 3 27B
  50. 22.9 gemini-exp-1206
  51. 22.8 Gemma 3n E4B
  52. 19.2 OLMo 2 32B Instruct
  1. 1 Grok-3 mini thinking 73.7%
  2. 2 Claude Sonnet 4 thinking 68.6%
  3. 3 DeepSeek-R1 May 2025 67.1%
  4. 4 Grok 3 Mini beta (unlisted snapshot) thinking 64.7%
  5. 5 Claude Opus 4 64.1%
  6. 6 Gemini 2.5 Flash May 2025, thinking 63.5%
  7. 7 Gemini 2.5 Flash Jun 2025, high 62.5%
  8. 8 Gemini 2.5 Pro May 2025 58.8%
  9. 9 Llama-3.3-Nemotron-Super-49B-v1 thinking 57.4%
  10. 10 Gemini 2.5 Pro Jun 2025, low 57.4%
  11. 11 Claude 3.7 Sonnet thinking 56.1%
  12. 12 QwQ-32B 56.1%
  13. 13 Gemini 2.5 Pro Mar 2025 55.2%
  14. 14 DeepSeek-R1 55.1%
  15. 15 Grok 3 55.1%
  16. 16 DeepSeek-V3 Mar 2025 53.5%
  17. 17 GLM-Z1-32B-0414 53.5%
  18. 18 Magistral Medium 1.0 thinking 51.7%
  19. 19 Mistral Medium 3 50.6%
  20. 20 o4-mini-high-long 49.4%
  21. 21 Qwen3-235B-A22B thinking 49%
  22. 22 GPT-4.1 mini 48.7%
  23. 23 DeepHermes-3-Mistral-24B-Pre 48.4%
  24. 24 Phi-4-Reasoning-plus 48.4%
  25. 25 optimus-alpha 47.4%
  26. 26 o3-mini 46.8%
  27. 27 GPT-4.1 44.2%
  28. 28 Qwen3-30B-A3B thinking 43.6%
  29. 29 o4-mini high 42.9%
  30. 30 DeepSeek-R1-Distill-Llama-70B 42.6%
  31. 31 quasar-alpha 40.7%
  32. 32 Claude 3.5 Sonnet (October 2024) 37.2%
  33. 33 MiniMax-Text-01 36.5%
  34. 34 Llama 4 Scout 35.9%
  35. 35 Llama 3.3 70B Instruct 35.3%
  36. 36 Gemini 2.0 Pro 34.6%
  37. 37 GPT-4o Nov 2024 33.7%
  38. 38 GPT-4o Jan 2025 32.1%
  39. 39 Hermes-3-Llama-3.1-70B 31.4%
  40. 40 Qwen2.5-Max 31.4%
  41. 41 Cohere Command A 30.1%
  42. 42 Llama 4 Maverick 29.5%
  43. 43 Gemini 2.0 Flash Feb 2025 28.2%
  44. 44 Mistral Small 3 27.9%
  45. 45 DeepSeek-V3 26.9%
  46. 46 GPT-4.1 nano 26.6%
  47. 47 Mistral Small 3.1 26%
  48. 48 Jamba 1.6 Large 25%
  49. 49 Gemma 3 27B 23.4%
  50. 50 gemini-exp-1206 22.9%
  51. 51 Gemma 3n E4B 22.8%
  52. 52 OLMo 2 32B Instruct 19.2%

Source: Official Misguided Attention leaderboard

#EntryScoreSettingDateSource
1 Grok-3 mini 73.7% thinking Apr 2025 Official leaderboard
2 Claude Sonnet 4 68.6% thinking May 2025 Official leaderboard
3 DeepSeek-R1 May 2025 67.1% May 2025 Official leaderboard
4 Grok 3 Mini beta (unlisted snapshot) 64.7% thinking Official leaderboard
5 Claude Opus 4 64.1% May 2025 Official leaderboard
6 Gemini 2.5 Flash May 2025 63.5% thinking May 2025 Official leaderboard
7 Gemini 2.5 Flash Jun 2025 62.5% high Jun 2025 Official leaderboard
8 Gemini 2.5 Pro May 2025 58.8% May 2025 Official leaderboard
9 Llama-3.3-Nemotron-Super-49B-v1 57.4% thinking Mar 2025 Official leaderboard
10 Gemini 2.5 Pro Jun 2025 57.4% low Jun 2025 Official leaderboard
11 Claude 3.7 Sonnet 56.1% thinking Feb 2025 Official leaderboard
12 QwQ-32B 56.1% Mar 2025 Official leaderboard
13 Gemini 2.5 Pro Mar 2025 55.2% Mar 2025 Official leaderboard
14 DeepSeek-R1 55.1% Jan 2025 Official leaderboard
15 Grok 3 55.1% Apr 2025 Official leaderboard
16 DeepSeek-V3 Mar 2025 53.5% Mar 2025 Official leaderboard
17 GLM-Z1-32B-0414 53.5% Apr 2025 Official leaderboard
18 Magistral Medium 1.0 51.7% thinking Jun 2025 Official leaderboard
19 Mistral Medium 3 50.6% May 2025 Official leaderboard
20 o4-mini-high-long 49.4% Official leaderboard
21 Qwen3-235B-A22B 49% thinking Apr 2025 Official leaderboard
22 GPT-4.1 mini 48.7% Apr 2025 Official leaderboard
23 DeepHermes-3-Mistral-24B-Pre 48.4% Official leaderboard
24 Phi-4-Reasoning-plus 48.4% Apr 2025 Official leaderboard
25 optimus-alpha 47.4% Official leaderboard
26 o3-mini 46.8% Jan 2025 Official leaderboard
27 GPT-4.1 44.2% Apr 2025 Official leaderboard
28 Qwen3-30B-A3B 43.6% thinking Apr 2025 Official leaderboard
29 o4-mini 42.9% high Apr 2025 Official leaderboard
30 DeepSeek-R1-Distill-Llama-70B 42.6% Jan 2025 Official leaderboard
31 quasar-alpha 40.7% Official leaderboard
32 Claude 3.5 Sonnet (October 2024) 37.2% Oct 2024 Official leaderboard
33 MiniMax-Text-01 36.5% Jan 2025 Official leaderboard
34 Llama 4 Scout 35.9% Apr 2025 Official leaderboard
35 Llama 3.3 70B Instruct 35.3% Dec 2024 Official leaderboard
36 Gemini 2.0 Pro 34.6% Feb 2025 Official leaderboard
37 GPT-4o Nov 2024 33.7% Nov 2024 Official leaderboard
38 GPT-4o Jan 2025 32.1% Jan 2025 Official leaderboard
39 Hermes-3-Llama-3.1-70B 31.4% Official leaderboard
40 Qwen2.5-Max 31.4% Jan 2025 Official leaderboard
41 Cohere Command A 30.1% Mar 2025 Official leaderboard
42 Llama 4 Maverick 29.5% Apr 2025 Official leaderboard
43 Gemini 2.0 Flash Feb 2025 28.2% Feb 2025 Official leaderboard
44 Mistral Small 3 27.9% Jan 2025 Official leaderboard
45 DeepSeek-V3 26.9% Dec 2024 Official leaderboard
46 GPT-4.1 nano 26.6% Apr 2025 Official leaderboard
47 Mistral Small 3.1 26% Mar 2025 Official leaderboard
48 Jamba 1.6 Large 25% Mar 2025 Official leaderboard
49 Gemma 3 27B 23.4% Mar 2025 Official leaderboard
50 gemini-exp-1206 22.9% Official leaderboard
51 Gemma 3n E4B 22.8% Jun 2025 Official leaderboard
52 OLMo 2 32B Instruct 19.2% Mar 2025 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Claude Sonnet 4
thinking
68.6%
Gemini 2.5 Flash
thinking
63.5%
Gemini 2.5 Pro
low
57.4%
Claude 3.7 Sonnet
thinking
56.1%
Qwen3-235B-A22B
non-thinking thinking
49%
Qwen3-30B-A3B
non-thinking thinking
43.6%

Source: Official Misguided Attention leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Grok-3 mini thinking 73.7%
Claude Sonnet 4 thinking 68.6%
DeepSeek-R1 May 2025 67.1%
Claude Opus 4 64.1%
Gemini 2.5 Flash May 2025 thinking 63.5%
Gemini 2.5 Flash Jun 2025 high 62.5%
Gemini 2.5 Pro May 2025 58.8%
Llama-3.3-Nemotron-Super-49B-v1 thinking 57.4%
Gemini 2.5 Pro Jun 2025 low 57.4%
Gemini 2.5 Pro Jun 2025 high 57.4%
Claude 3.7 Sonnet thinking 56.1%
QwQ-32B 56.1%
Gemini 2.5 Pro Mar 2025 55.2%
DeepSeek-R1 55.1%
Grok 3 55.1%
DeepSeek-V3 Mar 2025 53.5%
GLM-Z1-32B-0414 53.5%
Magistral Medium 1.0 thinking 51.7%
Mistral Medium 3 50.6%
Qwen3-235B-A22B thinking 49%
GPT-4.1 mini 48.7%
Phi-4-Reasoning-plus 48.4%
o3-mini 46.8%
Claude Sonnet 4 46.5%
Claude 3.7 Sonnet 46.2%
GPT-4.1 44.2%
Qwen3-235B-A22B non-thinking 43.6%
Qwen3-30B-A3B thinking 43.6%
o4-mini high 42.9%
DeepSeek-R1-Distill-Llama-70B 42.6%
Gemini 2.5 Flash May 2025 41.3%
Claude 3.5 Sonnet (October 2024) 37.2%
MiniMax-Text-01 36.5%
Llama 4 Scout 35.9%
Llama 3.3 70B Instruct 35.3%
Gemini 2.0 Pro 34.6%
GPT-4o Nov 2024 33.7%
GPT-4o Jan 2025 32.1%
Qwen2.5-Max 31.4%
Qwen3-30B-A3B non-thinking 31.1%
Cohere Command A 30.1%
Llama 4 Maverick 29.5%
Gemini 2.0 Flash Feb 2025 28.2%
Mistral Small 3 27.9%
DeepSeek-V3 26.9%
GPT-4.1 nano 26.6%
Mistral Small 3.1 26%
Jamba 1.6 Large 25%
Gemma 3 27B 23.4%
Gemma 3n E4B 22.8%
OLMo 2 32B Instruct 19.2%

5 models have held number one on the official leaderboard since its release in May 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic DeepSeek NVIDIA xAI
Own leaderboard Not tested since Jun 2025
Grok-3 mini 73.7%

Source: Official Misguided Attention leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Grok-3 mini Llama-3.3-Nemotron-Super-49B-v1 Claude 3.5 Sonnet (October 2024) 0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Grok-3 mini Llama-3.3-Nemotron-Super-49B-v1

Source: Official Misguided Attention leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Science and reasoning

All 92 in Science and reasoning
  • GPQA Diamond In use 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. Gemini 4 Carbon 96.5%
  • Terminal-Bench-Science In use 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. GPT-6 Astra 68.1%
  • FrontierScience In use Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. Intern-S2-397B 87%
  • BioMysteryBench In use Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. Claude Fable 5.2 83.7%
  • HiPhO In use Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. Gemini 3 Pro 78.3%
  • LABBench2 In use 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. GPT-5.2 60.9%