Benchmark Almanac

CMMLU

Hunyuan-TurboS 90.8% leads since February 2025
Qwen2-72B 89.7% leads since June 2024
Hunyuan-TurboS 90.8% leads since February 2025
Fading

The official leaderboard hasn’t scored a model released after Qwen2.5-72B (September 2024). Labs report their own numbers for 11 newer models, such as Hunyuan-TurboS at 90.8%.

Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy.

Saturated. HF dataset haonan-li/cmmlu now redirects to lmlmcat/cmmlu.

Released
June 2023
Kind
Exam questions
Level
University
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Measures
Capability
Language
Chinese
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
Strongest model tested
DeepSeek-V4-Pro #70 on the capability index
Last in a launch post
October 2026, dropped after it saturated
Last model added
October 2026
Scores refreshed
November 18th, 2026

In the authors’ words

As the capabilities of large language models (LLMs) continue to advance, evaluating their performance becomes increasingly crucial and challenging. This paper aims to bridge this gap by introducing CMMLU, a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. We conduct a thorough evaluation of 18 advanced multilingual- and Chinese-oriented LLMs, assessing their performance across different subjects and settings. The results reveal that most existing LLMs struggle to achieve an average accuracy of 50%, even when provided with in-context examples and chain-of-thought prompts, whereas the random baseline stands at 25%. This highlights significant room for improvement in LLMs. Additionally, we conduct extensive experiments to identify factors impacting the models' performance and propose directions for enhancing LLMs. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.

Abstract of CMMLU: Measuring massive multitask language understanding in Chinese (arXiv:2306.09212, CC0)
18
Models scored in lab cards
18
Launch posts list it, from 9 labs
8
System cards report it, from 6 labs

2 models share the top score of 90.8%

Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.

Qwen2-72B leads with 89.7% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.

Hunyuan-TurboS leads with 90.8%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

  1. 1 Hunyuan-TurboS 90.8%
  2. 2 DeepSeek-V4-Pro 90.8%
  3. 3 MiniCPM-V-4.7-35B-A3B 90.2%
  4. 4 ERNIE-4.5-300B-A47B 90.2%
  5. 5 InternVL3.5-241B-A28B 90.2%
  6. 6 Qwen2-72B 90.1%
  7. 7 Hunyuan-T1 90%
  8. 8 LongCat-Flash-Omni 89.4%
  9. 9 Ring-1T 89.1%
  10. 10 DeepSeek-V3 88.8%
  11. 11 LongCat-Flash-Chat 84.3%
  12. 12 Qwen1.5-72B 83.5%
  13. 13 InternLM3-8B-Instruct 83.1%
  14. 14 DeepSeek-V2 (MoE-236B, May 2024) 81.6%
  15. 15 Moonlight-16B-A3B 78.2%
  16. 16 Qwen-14B 71%
  17. 17 DeepSeek LLM 67B 67.8%
  18. 18 ChatGLM3-6B 67.5%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Hunyuan-TurboS 90.8% Feb 2025 Lab model card
2 DeepSeek-V4-Pro 90.8% Apr 2026 Lab model card
3 MiniCPM-V-4.7-35B-A3B 90.2% Oct 2026 Lab model card
4 ERNIE-4.5-300B-A47B 90.2% Jun 2025 Lab model card
5 InternVL3.5-241B-A28B 90.2% Aug 2025 Lab model card
6 Qwen2-72B 90.1% Jun 2024 Lab model card
7 Hunyuan-T1 90% Mar 2025 Lab model card
8 LongCat-Flash-Omni 89.4% Oct 2025 Lab model card
9 Ring-1T 89.1% Oct 2025 Lab model card
10 DeepSeek-V3 88.8% Dec 2024 Lab model card
11 LongCat-Flash-Chat 84.3% Aug 2025 Lab model card
12 Qwen1.5-72B 83.5% Feb 2024 Lab model card
13 InternLM3-8B-Instruct 83.1% Jan 2025 Lab model card
14 DeepSeek-V2 (MoE-236B, May 2024) 81.6% May 2024 Lab model card
15 Moonlight-16B-A3B 78.2% Feb 2025 Lab model card
16 Qwen-14B 71% Sep 2023 Lab model card
17 DeepSeek LLM 67B 67.8% Nov 2023 Lab model card
18 ChatGLM3-6B 67.5% Oct 2023 Lab model card
#ModelScoreSettingDateSource
1 Qwen2-72B 89.7% Five-shot Jun 2024 Official leaderboard
2 Qwen1.5-110B 88.3% Five-shot Apr 2024 Official leaderboard
3 Qwen2.5-72B 85.7% Five-shot Sep 2024 Official leaderboard
4 Qwen1.5-72B 83.5% Five-shot Feb 2024 Official leaderboard
5 Qwen1.5-32B 82.3% Five-shot Feb 2024 Official leaderboard
6 Qwen1.5-7B 72.5% Five-shot Feb 2024 Official leaderboard
7 Qwen-7B 58.7% Five-shot Sep 2023 Official leaderboard
8 chatglm2-6b 50% Zero-shot Jun 2023 Official leaderboard
9 Falcon-40B 41.5% Five-shot Mar 2023 Official leaderboard
10 ChatGLM-6B 40.8% Zero-shot Mar 2023 Official leaderboard
11 LLaMA-65B 39.8% Five-shot Feb 2023 Official leaderboard
#ModelScoreSettingDateSource
1 Hunyuan-TurboS 90.8% Feb 2025 Lab model card
2 DeepSeek-V4-Pro 90.8% Apr 2026 Lab model card
3 MiniCPM-V-4.7-35B-A3B 90.2% Oct 2026 Lab model card
4 ERNIE-4.5-300B-A47B 90.2% Jun 2025 Lab model card
5 InternVL3.5-241B-A28B 90.2% Aug 2025 Lab model card
6 Qwen2-72B 90.1% Jun 2024 Lab model card
7 Hunyuan-T1 90% Mar 2025 Lab model card
8 LongCat-Flash-Omni 89.4% Oct 2025 Lab model card
9 Ring-1T 89.1% Oct 2025 Lab model card
10 DeepSeek-V3 88.8% Dec 2024 Lab model card
11 Qwen1.5-110B 88.3% Five-shot Apr 2024 Official leaderboard
12 Qwen2.5-72B 85.7% Five-shot Sep 2024 Official leaderboard
13 LongCat-Flash-Chat 84.3% Aug 2025 Lab model card
14 Qwen1.5-72B 83.5% Five-shot Feb 2024 Official leaderboard
15 InternLM3-8B-Instruct 83.1% Jan 2025 Lab model card
16 Qwen1.5-32B 82.3% Five-shot Feb 2024 Official leaderboard
17 DeepSeek-V2 (MoE-236B, May 2024) 81.6% May 2024 Lab model card
18 Moonlight-16B-A3B 78.2% Feb 2025 Lab model card
19 Qwen1.5-7B 72.5% Five-shot Feb 2024 Official leaderboard
20 Qwen-14B 71% Sep 2023 Lab model card
21 DeepSeek LLM 67B 67.8% Nov 2023 Lab model card
22 ChatGLM3-6B 67.5% Oct 2023 Lab model card
23 Qwen-7B 58.7% Five-shot Sep 2023 Official leaderboard
24 chatglm2-6b 50% Zero-shot Jun 2023 Official leaderboard
25 Falcon-40B 41.5% Five-shot Mar 2023 Official leaderboard
26 ChatGLM-6B 40.8% Zero-shot Mar 2023 Official leaderboard
27 LLaMA-65B 39.8% Five-shot Feb 2023 Official leaderboard

The benchmark's own leaderboard has 47 entries, 11 of them models we track. Switch the scores above to rank them together.

Rankings by part

We read 5 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as STEM, Humanities and Social sciences. Qwen2-72B leads every one of them. Arrows show how far a model moved from its overall rank among the models we track.

STEM

11 models we track have a score here.

  1. 1 Qwen2-72B 82.8%
  2. 2 Qwen1.5-110B 81.6%
  3. 3 Qwen2.5-72B 80.7%
  4. 4 Qwen1.5-72B 76.8%
  5. 5 Qwen1.5-32B 76.3%
  6. 6 Qwen1.5-7B 63.6%
  7. 7 Qwen-7B 48.4%
  8. 8 chatglm2-6b 42.5%
  9. 9 LLaMA-65B ↑2 34.5%
  10. 10 Falcon-40B ↓1 33.3%

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Qwen2-72B
Zero-shot
89.7%
Qwen1.5-110B
Zero-shot
88.3%
Qwen2.5-72B
Zero-shot
85.7%
Qwen1.5-72B
Zero-shot
83.5%
Qwen1.5-32B
Zero-shot
82.3%
Qwen1.5-7B
Zero-shot
72.5%
Qwen-7B
Zero-shot
58.7%
chatglm2-6b
Five-shot
50%
Falcon-40B
Zero-shot
41.5%
ChatGLM-6B
Five-shot
40.8%
LLaMA-65B
Zero-shot
39.8%

Source: Official CMMLU leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Qwen2-72B Five-shot 89.7%
Qwen1.5-110B Five-shot 88.3%
Qwen1.5-110B Zero-shot 87.6%
Qwen2-72B Zero-shot 87.5%
Qwen2.5-72B Five-shot 85.7%
Qwen2.5-72B Zero-shot 84.7%
Qwen1.5-72B Five-shot 83.5%
Qwen1.5-32B Five-shot 82.3%
Qwen1.5-72B Zero-shot 81.8%
Qwen1.5-32B Zero-shot 81.5%
Qwen1.5-7B Five-shot 72.5%
Qwen1.5-7B Zero-shot 71%
Qwen-7B Five-shot 58.7%
Qwen-7B Zero-shot 57.6%
chatglm2-6b Zero-shot 50%
chatglm2-6b Five-shot 48.8%
Falcon-40B Five-shot 41.5%
ChatGLM-6B Zero-shot 40.8%
LLaMA-65B Five-shot 39.8%
Falcon-40B Zero-shot 38.5%
ChatGLM-6B Five-shot 37.5%
LLaMA-65B Zero-shot 34.9%

4 models have held number one in labs’ own numbers since its release in June 2023 6 models have held number one across all independent runs since its release in June 2023 7 models have held number one with claims included since its release in June 2023

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Technology Innovation Institute Other labs Qwen Tencent
Own leaderboard Passed 85% in Apr 2024, by Qwen1.5-110B
Qwen2-72B 89.7%
Lab claims Passed 85% in Jun 2024, by Qwen2-72B

Sources: Official CMMLU leaderboard and labs’ model cards

Qwen1.5-110B on the official leaderboard passed 85% in April 2024

That was 10 months after its release. By the same age, 13% of the benchmarks from 2023 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) Hunyuan-TurboS Qwen2-72B Qwen1.5-72B Qwen-14B chatglm2-6b Falcon-40B 0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) Hunyuan-TurboS Qwen1.5-72B

Sources: Official CMMLU leaderboard and labs’ model cards

Labs headline it less often now, last in October 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
Z.ai
DeepSeek
Shanghai AI Lab
Moonshot AI
Tencent
Baidu
Meituan
OpenBMB (ModelBest)
DateLaunch postScoreAs the post writes it
Oct 7, 2026 MiniCPM-V-4.7-35B-A3B 90.2 CMMLU
Apr 24, 2026 DeepSeek-V4-Pro 90.8 CMMLU (EM)
Oct 31, 2025 LongCat-Flash-Omni 89.4 CMMLU(acc)
Aug 29, 2025 LongCat-Flash-Chat 84.3 CMMLU(acc)
Jun 30, 2025 ERNIE-4.5-300B-A47B 90.2 CMMLU
Mar 21, 2025 Hunyuan-T1 90 CMMLU
Mar 16, 2025 ERNIE 4.5 CMMLU
Feb 27, 2025 Hunyuan-TurboS 90.8 CMMLU
Feb 22, 2025 Moonlight-16B-A3B 78.2 CMMLU
Jan 28, 2025 Qwen2.5-Max CMMLU
Jan 13, 2025 InternLM3-8B-Instruct 83.1 CMMLU(0-shot)
Dec 26, 2024 DeepSeek-V3 88.8 CMMLU (Acc.)
Jun 7, 2024 Qwen2-72B 90.1 CMMLU
May 7, 2024 DeepSeek-V2 (MoE-236B, May 2024) 81.6 CMMLU
Feb 5, 2024 Qwen1.5-72B 83.5 CMMLU
Nov 29, 2023 DeepSeek LLM 67B 67.8 CMMLU
Oct 27, 2023 ChatGLM3-6B 67.5 CMMLU
Sep 25, 2023 Qwen-14B 71 CMMLU (5-shot)

8 system cards report it, from 6 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 31, 2025 LongCat-Flash-Omni technical report 89.4 CMMLU (acc) (Table 13) Capabilities
Oct 21, 2025 Ring-1T technical report 89.1 CMMLU (Table 1) Capabilities
Sep 1, 2025 LongCat-Flash-Chat technical report 84.3 CMMLU (acc) (Table 3) Capabilities
Aug 25, 2025 InternVL3.5-241B-A28B technical report 90.2 CMMLU, OpenCompass (Table 14) Capabilities
Jun 29, 2025 ERNIE-4.5-300B-A47B technical report 90.2 CMMLU (Table 5) Capabilities
Jun 18, 2024 ChatGLM3-6B technical report 67.5 ChatGLM3-6B-Base, CMMLU (Table 1) Capabilities
May 7, 2024 DeepSeek-V2 (MoE-236B, May 2024) technical report 81.6 CMMLU 5-shot, DeepSeek-V2 Chat (RL) (Table 3) Capabilities
Jan 5, 2024 DeepSeek LLM 67B technical report 67.8 CMMLU 0-shot, DeepSeek 67B Chat (Table 6) Capabilities

2 lab claims next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 1 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
Qwen2-72B 90.1% CMMLU 89.7% Leaderboard, Five-shot +0.4 pts
Qwen1.5-72B 83.5% CMMLU 83.5% Leaderboard, Five-shot Same

More in Knowledge

All 332 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. Seed1.8 92.3%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%

Data audit

All 10 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 3 pass 2 notes
  • Pass Independent runs Scores from the official leaderboard (47 entries), launch posts (16 scores) and system cards (8 scores).
  • Pass Models scored 27 models we track, plus 36 other entries on its leaderboard.
  • Pass Newest score The newest is from October 2026, for MiniCPM-V-4.7-35B-A3B.
  • Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is DeepSeek-V4-Pro (#70).
  • Note Headroom Saturated. The best is 90.8%, past 85% since April 2024.
Leaderboard import 3 pass
  • Pass Importer We read the board on github.com: 22 rows for models we track and 36 other entries at the last read.
  • Pass Last read November 18th, 2026.
  • Pass Read worked The last read worked.
Consistency 2 pass
  • Pass Lab claims Both pairs agree within 10 points.
  • Pass Known issues No published audit found problems.

How we check