Benchmark Almanac

MMLU-Redux

Qwen3.7-Max 95% leads since May 2026
Fading

Manually re-annotated subset of MMLU that fixes wrong labels. Multiple-choice accuracy.

Saturated. This first release has 3,000 questions in 30 subjects. The authors now recommend MMLU-Redux 2.0, and cards that write only MMLU-Redux do not say which version.

Released
June 2024
Kind
Exam questions
Level
University
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Strongest model tested
Qwen3.7-Max #49 on the capability index
Last in a launch post
June 2026, dropped after it saturated
Last model added
June 2026

In the authors’ words

Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.

Abstract of Are We Done with MMLU? (arXiv:2406.04127, CC0)
45
Models scored in lab cards
32
Launch posts list it, from 6 labs
25
System cards report it, from 6 labs

Qwen3.7-Max leads with 95% among models released by June 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in June 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

45 models. Scroll sideways for the rest.

  1. 95.0 Qwen3.7-Max
  2. 94.9 Qwen3.5 397B-A17B
  3. 94.5 Qwen 3.6 Plus
  4. 94.5 Qwen3.7-Plus
  5. 94.4 Kimi K2 Thinking
  6. 94.2 Qwen3.5-Omni-Plus
  7. 94.0 Qwen3.5-122B-A10B
  8. 93.8 Qwen3-235B-A22B-Thinking
  9. 93.7 DeepSeek-V3.1
  10. 93.7 Qwen3-VL-235B-A22B
  11. 93.5 Qwen3.6 27B
  12. 93.4 DeepSeek-R1
  13. 93.3 Qwen3.5-35B-A3B
  14. 93.3 Qwen 3.6 35B-A3B
  15. 93.2 Qwen3.5-27B
  16. 92.9 DeepSeek-R1
  17. 92.8 Qwen3-Max
  18. 92.7 Qwen3-235B-A22B
  19. 92.7 Kimi K2
  20. 92.5 Qwen3-Next-80B-A3B
  21. 92.3 EXAONE 4.0 32B
  22. 91.2 Qwen3-Coder-Next
  23. 91.1 Qwen3.5-9B
  24. 90.9 Qwen3-32B
  25. 90.8 DeepSeek-V4-Pro
  26. 89.5 Qwen3-30B-A3B
  27. 89.3 LongCat-Flash-Thinking
  28. 89.1 DeepSeek-V3
  29. 88.8 Qwen3.5-4B
  30. 88.6 Qwen3-14B
  31. 87.5 Qwen3-8B
  32. 86.8 Qwen2.5-72B
  33. 86.6 Qwen3-Omni-30B-A3B
  34. 86.3 Nemotron-Cascade 2 30B-A3B
  35. 85.9 Qwen2.5-VL-72B
  36. 83.9 Qwen2.5-32B
  37. 83.7 Qwen3-4B
  38. 81.7 Qwen-Turbo
  39. 80.7 Qwen2.5-14B-Instruct-1M
  40. 80.0 Qwen2.5-14B
  41. 79.6 Qwen3.5-2B
  42. 75.4 Qwen2.5-7B
  43. 73.9 Qwen3-1.7B
  44. 71.0 Qwen2.5-Omni-7B
  45. 59.5 Qwen3.5-0.8B
  1. 1 Qwen3.7-Max 95%
  2. 2 Qwen3.5 397B-A17B 94.9%
  3. 3 Qwen 3.6 Plus 94.5%
  4. 4 Qwen3.7-Plus 94.5%
  5. 5 Kimi K2 Thinking 94.4%
  6. 6 Qwen3.5-Omni-Plus 94.2%
  7. 7 Qwen3.5-122B-A10B 94%
  8. 8 Qwen3-235B-A22B-Thinking Jul 2025 93.8%
  9. 9 DeepSeek-V3.1 93.7%
  10. 10 Qwen3-VL-235B-A22B 93.7%
  11. 11 Qwen3.6 27B 93.5%
  12. 12 DeepSeek-R1 May 2025 93.4%
  13. 13 Qwen3.5-35B-A3B 93.3%
  14. 14 Qwen 3.6 35B-A3B 93.3%
  15. 15 Qwen3.5-27B 93.2%
  16. 16 DeepSeek-R1 92.9%
  17. 17 Qwen3-Max 92.8%
  18. 18 Qwen3-235B-A22B 92.7%
  19. 19 Kimi K2 Jul 2025 92.7%
  20. 20 Qwen3-Next-80B-A3B 92.5%
  21. 21 EXAONE 4.0 32B 92.3%
  22. 22 Qwen3-Coder-Next 91.2%
  23. 23 Qwen3.5-9B 91.1%
  24. 24 Qwen3-32B 90.9%
  25. 25 DeepSeek-V4-Pro 90.8%
  26. 26 Qwen3-30B-A3B 89.5%
  27. 27 LongCat-Flash-Thinking 89.3%
  28. 28 DeepSeek-V3 89.1%
  29. 29 Qwen3.5-4B 88.8%
  30. 30 Qwen3-14B 88.6%
  31. 31 Qwen3-8B 87.5%
  32. 32 Qwen2.5-72B 86.8%
  33. 33 Qwen3-Omni-30B-A3B 86.6%
  34. 34 Nemotron-Cascade 2 30B-A3B 86.3%
  35. 35 Qwen2.5-VL-72B 85.9%
  36. 36 Qwen2.5-32B 83.9%
  37. 37 Qwen3-4B 83.7%
  38. 38 Qwen-Turbo 81.7%
  39. 39 Qwen2.5-14B-Instruct-1M 80.7%
  40. 40 Qwen2.5-14B 80%
  41. 41 Qwen3.5-2B 79.6%
  42. 42 Qwen2.5-7B 75.4%
  43. 43 Qwen3-1.7B 73.9%
  44. 44 Qwen2.5-Omni-7B 71%
  45. 45 Qwen3.5-0.8B 59.5%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Qwen3.7-Max 95% May 2026 Lab model card
2 Qwen3.5 397B-A17B 94.9% Feb 2026 Lab model card
3 Qwen 3.6 Plus 94.5% Apr 2026 Lab model card
4 Qwen3.7-Plus 94.5% Jun 2026 Lab model card
5 Kimi K2 Thinking 94.4% Nov 2025 Lab model card
6 Qwen3.5-Omni-Plus 94.2% Mar 2026 Lab model card
7 Qwen3.5-122B-A10B 94% Feb 2026 Lab model card
8 Qwen3-235B-A22B-Thinking Jul 2025 93.8% Jul 2025 Lab model card
9 DeepSeek-V3.1 93.7% Aug 2025 Lab model card
10 Qwen3-VL-235B-A22B 93.7% Sep 2025 Lab model card
11 Qwen3.6 27B 93.5% Apr 2026 Lab model card
12 DeepSeek-R1 May 2025 93.4% May 2025 Lab model card
13 Qwen3.5-35B-A3B 93.3% Feb 2026 Lab model card
14 Qwen 3.6 35B-A3B 93.3% Apr 2026 Lab model card
15 Qwen3.5-27B 93.2% Feb 2026 Lab model card
16 DeepSeek-R1 92.9% Jan 2025 Lab model card
17 Qwen3-Max 92.8% Jan 2026 Lab model card
18 Qwen3-235B-A22B 92.7% Apr 2025 Lab model card
19 Kimi K2 Jul 2025 92.7% Jul 2025 Lab model card
20 Qwen3-Next-80B-A3B 92.5% Sep 2025 Lab model card
21 EXAONE 4.0 32B 92.3% Jul 2025 Lab model card
22 Qwen3-Coder-Next 91.2% Feb 2026 Lab model card
23 Qwen3.5-9B 91.1% Mar 2026 Lab model card
24 Qwen3-32B 90.9% May 2025 Lab model card
25 DeepSeek-V4-Pro 90.8% Apr 2026 Lab model card
26 Qwen3-30B-A3B 89.5% May 2025 Lab model card
27 LongCat-Flash-Thinking 89.3% Sep 2025 Lab model card
28 DeepSeek-V3 89.1% Dec 2024 Lab model card
29 Qwen3.5-4B 88.8% Mar 2026 Lab model card
30 Qwen3-14B 88.6% May 2025 Lab model card
31 Qwen3-8B 87.5% May 2025 Lab model card
32 Qwen2.5-72B 86.8% Sep 2024 Lab model card
33 Qwen3-Omni-30B-A3B 86.6% Sep 2025 Lab model card
34 Nemotron-Cascade 2 30B-A3B 86.3% Mar 2026 Lab model card
35 Qwen2.5-VL-72B 85.9% Feb 2025 Lab model card
36 Qwen2.5-32B 83.9% Dec 2024 Lab model card
37 Qwen3-4B 83.7% May 2025 Lab model card
38 Qwen-Turbo 81.7% Dec 2024 Lab model card
39 Qwen2.5-14B-Instruct-1M 80.7% Jan 2025 Lab model card
40 Qwen2.5-14B 80% Dec 2024 Lab model card
41 Qwen3.5-2B 79.6% Mar 2026 Lab model card
42 Qwen2.5-7B 75.4% Dec 2024 Lab model card
43 Qwen3-1.7B 73.9% May 2025 Lab model card
44 Qwen2.5-Omni-7B 71% Mar 2025 Lab model card
45 Qwen3.5-0.8B 59.5% Mar 2026 Lab model card

8 models have held number one in labs’ own numbers since its release in June 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen DeepSeek Moonshot AI
Lab claims Passed 85% in Sep 2024, by Qwen2.5-72B
Qwen3.7-Max 95%

Source: Labs’ model cards

Qwen’s own number for Qwen2.5-72B passed 85% in September 2024

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen3.7-Max Kimi K2 Thinking DeepSeek-R1 DeepSeek-R1 Qwen2.5-72B 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen3.7-Max DeepSeek-R1

Source: Labs’ model cards

Labs headline it less often now, last in June 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
DeepSeek
Moonshot AI
LG AI Research
Meituan
NVIDIA
DateLaunch postScoreAs the post writes it
Jun 1, 2026 Qwen3.7-Plus 94.5 MMLU-Redux
May 20, 2026 Qwen3.7-Max 95 MMLU-Redux
Apr 24, 2026 DeepSeek-V4-Pro 90.8 MMLU-Redux (EM)
Apr 22, 2026 Qwen3.6 27B 93.5 MMLU-Redux
Apr 15, 2026 Qwen 3.6 35B-A3B 93.3 MMLU-Redux
Apr 2, 2026 Qwen 3.6 Plus 94.5 MMLU-Redux
Mar 30, 2026 Qwen3.5-Omni-Plus 94.2 MMLU-Redux
Mar 19, 2026 Nemotron-Cascade 2 30B-A3B 86.3 MMLU-Redux
Mar 2, 2026 Qwen3.5-0.8B 59.5 MMLU-Redux (Thinking)
Mar 2, 2026 Qwen3.5-2B 79.6 MMLU-Redux (Thinking)
Mar 2, 2026 Qwen3.5-4B 88.8 MMLU-Redux
Mar 2, 2026 Qwen3.5-9B 91.1 MMLU-Redux
Feb 24, 2026 Qwen3.5-35B-A3B 93.3 MMLU-Redux
Feb 24, 2026 Qwen3.5-27B 93.2 MMLU-Redux
Feb 24, 2026 Qwen3.5-122B-A10B 94 MMLU-Redux
Feb 16, 2026 Qwen3.5 397B-A17B 94.9 MMLU-Redux
Jan 23, 2026 Qwen3-Max 92.8 MMLU-Redux
Nov 6, 2025 Kimi K2 Thinking 94.4 MMLU-Redux, no tools
Sep 23, 2025 Qwen3-VL-235B-A22B 93.7 MMLU-Redux
Sep 22, 2025 Qwen3-Omni-30B-A3B 86.6 MMLU-Redux
Sep 21, 2025 LongCat-Flash-Thinking 89.3 MMLU-Redux(acc)
Sep 11, 2025 Qwen3-Next-80B-A3B 92.5 MMLU-Redux
Aug 21, 2025 DeepSeek-V3.1 93.7 MMLU-Redux (EM)
Jul 25, 2025 Qwen3-235B-A22B-Thinking 93.8 MMLU-Redux
Jul 15, 2025 EXAONE 4.0 32B 92.3 MMLU-Redux (reasoning)
Jul 11, 2025 Kimi K2 92.7 MMLU-Redux, EM
May 28, 2025 DeepSeek-R1 93.4 MMLU-Redux (EM)
Apr 29, 2025 Qwen3-235B-A22B 92.7 MMLU-Redux
Mar 27, 2025 Qwen2.5-Omni-7B 71 MMLU-redux
Jan 20, 2025 DeepSeek-R1 92.9 MMLU-Redux (EM)
Dec 26, 2024 DeepSeek-V3 89.1 MMLU-Redux (EM)
Sep 19, 2024 Qwen2.5-72B 86.8 MMLU-redux

25 system cards report it, from 6 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Apr 17, 2026 Qwen3.5-Omni-Plus technical report 94.2 MMLU-Redux (Table 4) Capabilities
Mar 19, 2026 Nemotron-Cascade 2 30B-A3B technical report 86.3 MMLU-Redux (Table 1) Capabilities
Feb 28, 2026 Qwen3-Coder-Next technical report 91.2 MMLU-Redux (Table 8) Capabilities
Nov 26, 2025 Qwen3-VL-235B-A22B technical report 93.7 MMLU-Redux (Table 6) Capabilities
Sep 23, 2025 LongCat-Flash-Thinking technical report 89.3 MMLU-Redux(acc) (Table 2) Capabilities
Sep 22, 2025 Qwen3-Omni-30B-A3B technical report 86.6 MMLU-Redux, Instruct (Table 4). Thinking 88.8 (Table 5) Capabilities
Jul 21, 2025 Kimi K2 technical report 92.7 MMLU-Redux (EM) (Table 3) Capabilities
Jul 15, 2025 EXAONE 4.0 32B technical report 92.3 MMLU-Redux (Table 3, Reasoning). Non-reasoning 89.8 (Table 4) Capabilities
May 14, 2025 Qwen3-8B technical report 87.5 MMLU-Redux, thinking mode (Table 17). Non-thinking 79.5 (Table 18) Capabilities
May 14, 2025 Qwen3-4B technical report 83.7 MMLU-Redux, thinking mode (Table 17). Non-thinking 77.3 (Table 18) Capabilities
May 14, 2025 Qwen3-32B technical report 90.9 MMLU-Redux, thinking mode (Table 13). Non-thinking 85.7 (Table 14) Capabilities
May 14, 2025 Qwen3-30B-A3B technical report 89.5 MMLU-Redux, thinking mode (Table 15). Non-thinking 84.1 (Table 16) Capabilities
May 14, 2025 Qwen3-235B-A22B technical report 92.7 MMLU-Redux, thinking mode (Table 11). Non-thinking 89.2 (Table 12) Capabilities
May 14, 2025 Qwen3-14B technical report 88.6 MMLU-Redux, thinking mode (Table 15). Non-thinking 82.0 (Table 16) Capabilities
May 14, 2025 Qwen3-1.7B technical report 73.9 MMLU-Redux, thinking mode (Table 19). Non-thinking 64.4 (Table 20) Capabilities
Mar 26, 2025 Qwen2.5-Omni-7B technical report 71 MMLU-redux (Table 1) Capabilities
Feb 19, 2025 Qwen2.5-VL-72B technical report 85.9 MMLU-redux (Table 4) Capabilities
Jan 26, 2025 Qwen2.5-14B-Instruct-1M technical report 80.7 MMLU-redux (Table 6) Capabilities
Jan 20, 2025 DeepSeek-R1 technical report 92.9 MMLU-Redux (EM), DeepSeek-R1 (Table 8, same as final column of Table 3) Capabilities
Dec 26, 2024 DeepSeek-V3 technical report 89.1 MMLU-Redux (EM), DeepSeek-V3 (Table 6) Capabilities
Dec 19, 2024 Qwen2.5-7B technical report 75.4 MMLU-redux, Qwen2.5-7B-Instruct (Table 8) Capabilities
Dec 19, 2024 Qwen2.5-72B technical report 86.8 MMLU-redux, Qwen2.5-72B-Instruct (Table 6) Capabilities
Dec 19, 2024 Qwen2.5-32B technical report 83.9 MMLU-redux, Qwen2.5-32B-Instruct (Table 7) Capabilities
Dec 19, 2024 Qwen2.5-14B technical report 80 MMLU-redux, Qwen2.5-14B-Instruct (Table 7) Capabilities
Dec 19, 2024 Qwen-Turbo technical report 81.7 MMLU-redux, Qwen2.5-Turbo (Table 7) Capabilities

Versions

Replaced by MMLU-Redux 2.0

More in Knowledge

All 121 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • KMMLU-Pro In use Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. Solar Pro 4 79.2%
  • HLE-Verified In use Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Seed2.0 Pro 73.6%