CMMLU
The official leaderboard hasn’t scored a model released after Qwen2.5-72B (September 2024). Labs report their own numbers for 11 newer models, such as Hunyuan-TurboS at 90.8%.
Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy.
Saturated. HF dataset haonan-li/cmmlu now redirects to lmlmcat/cmmlu.
- Made by
- MBZUAI + others (Li et al.)
- Released
- June 2023
- Kind
- Exam questions
- Level
- University
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Language
- Chinese
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
- In views
- Biology and medicine, Law and finance
- Strongest model tested
-
DeepSeek-V4-Pro #70 on the capability index
- Last in a launch post
- October 2026, dropped after it saturated
- Last model added
- October 2026
- Scores refreshed
- November 18th, 2026
In the authors’ words
As the capabilities of large language models (LLMs) continue to advance, evaluating their performance becomes increasingly crucial and challenging. This paper aims to bridge this gap by introducing CMMLU, a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. We conduct a thorough evaluation of 18 advanced multilingual- and Chinese-oriented LLMs, assessing their performance across different subjects and settings. The results reveal that most existing LLMs struggle to achieve an average accuracy of 50%, even when provided with in-context examples and chain-of-thought prompts, whereas the random baseline stands at 25%. This highlights significant room for improvement in LLMs. Additionally, we conduct extensive experiments to identify factors impacting the models' performance and propose directions for enhancing LLMs. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.
2 models share the top score of 90.8%
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
Qwen2-72B leads with 89.7% across all independent runs
Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.
Hunyuan-TurboS leads with 90.8%, claims included
Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.
- 90.8
Hunyuan-TurboS
- 90.8
DeepSeek-V4-Pro
- 90.2
MiniCPM-V-4.7-35B-A3B
- 90.2
ERNIE-4.5-300B-A47B
- 90.2
InternVL3.5-241B-A28B
- 90.1
Qwen2-72B
- 90.0
Hunyuan-T1
- 89.4
LongCat-Flash-Omni
- 89.1
Ring-1T
- 88.8
DeepSeek-V3
- 84.3
LongCat-Flash-Chat
- 83.5
Qwen1.5-72B
- 83.1
InternLM3-8B-Instruct
- 81.6
DeepSeek-V2 (MoE-236B, May 2024)
- 78.2 Moonlight-16B-A3B
- 71.0
Qwen-14B
- 67.8
DeepSeek LLM 67B
- 67.5 ChatGLM3-6B
- 1
Hunyuan-TurboS 90.8%
- 2
DeepSeek-V4-Pro 90.8%
- 3
MiniCPM-V-4.7-35B-A3B 90.2%
- 4
ERNIE-4.5-300B-A47B 90.2%
- 5
InternVL3.5-241B-A28B 90.2%
- 6
Qwen2-72B 90.1%
- 7
Hunyuan-T1 90%
- 8
LongCat-Flash-Omni 89.4%
- 9
Ring-1T 89.1%
- 10
DeepSeek-V3 88.8%
- 11
LongCat-Flash-Chat 84.3%
- 12
Qwen1.5-72B 83.5%
- 13
InternLM3-8B-Instruct 83.1%
- 14
DeepSeek-V2 (MoE-236B, May 2024) 81.6%
- 15 Moonlight-16B-A3B 78.2%
- 16
Qwen-14B 71%
- 17
DeepSeek LLM 67B 67.8%
- 18 ChatGLM3-6B 67.5%
Source: Labs’ model cards
- 1
Qwen2-72B Five-shot 89.7%
- 2
Qwen1.5-110B Five-shot 88.3%
- 3
Qwen2.5-72B Five-shot 85.7%
- 4
Qwen1.5-72B Five-shot 83.5%
- 5
Qwen1.5-32B Five-shot 82.3%
- 6
Qwen1.5-7B Five-shot 72.5%
- 7
Qwen-7B Five-shot 58.7%
- 8 chatglm2-6b Zero-shot 50%
- 9
Falcon-40B Five-shot 41.5%
- 10 ChatGLM-6B Zero-shot 40.8%
- 11
LLaMA-65B Five-shot 39.8%
Source: Official CMMLU leaderboard
- 90.8
Hunyuan-TurboS
- 90.8
DeepSeek-V4-Pro
- 90.2
MiniCPM-V-4.7-35B-A3B
- 90.2
ERNIE-4.5-300B-A47B
- 90.2
InternVL3.5-241B-A28B
- 90.1
Qwen2-72B
- 90.0
Hunyuan-T1
- 89.4
LongCat-Flash-Omni
- 89.1
Ring-1T
- 88.8
DeepSeek-V3
- 88.3
Qwen1.5-110B
- 85.7
Qwen2.5-72B
- 84.3
LongCat-Flash-Chat
- 83.5
Qwen1.5-72B
- 83.1
InternLM3-8B-Instruct
- 82.3
Qwen1.5-32B
- 81.6
DeepSeek-V2 (MoE-236B, May 2024)
- 78.2 Moonlight-16B-A3B
- 72.5
Qwen1.5-7B
- 71.0
Qwen-14B
- 67.8
DeepSeek LLM 67B
- 67.5 ChatGLM3-6B
- 58.7
Qwen-7B
- 50.0 chatglm2-6b
- 41.5
Falcon-40B
- 40.8 ChatGLM-6B
- 39.8
LLaMA-65B
- 1
Hunyuan-TurboS 90.8%
- 2
DeepSeek-V4-Pro 90.8%
- 3
MiniCPM-V-4.7-35B-A3B 90.2%
- 4
ERNIE-4.5-300B-A47B 90.2%
- 5
InternVL3.5-241B-A28B 90.2%
- 6
Qwen2-72B 90.1%
- 7
Hunyuan-T1 90%
- 8
LongCat-Flash-Omni 89.4%
- 9
Ring-1T 89.1%
- 10
DeepSeek-V3 88.8%
- 11
Qwen1.5-110B Five-shot 88.3%
- 12
Qwen2.5-72B Five-shot 85.7%
- 13
LongCat-Flash-Chat 84.3%
- 14
Qwen1.5-72B Five-shot 83.5%
- 15
InternLM3-8B-Instruct 83.1%
- 16
Qwen1.5-32B Five-shot 82.3%
- 17
DeepSeek-V2 (MoE-236B, May 2024) 81.6%
- 18 Moonlight-16B-A3B 78.2%
- 19
Qwen1.5-7B Five-shot 72.5%
- 20
Qwen-14B 71%
- 21
DeepSeek LLM 67B 67.8%
- 22 ChatGLM3-6B 67.5%
- 23
Qwen-7B Five-shot 58.7%
- 24 chatglm2-6b Zero-shot 50%
- 25
Falcon-40B Five-shot 41.5%
- 26 ChatGLM-6B Zero-shot 40.8%
- 27
LLaMA-65B Five-shot 39.8%
Sources: Labs’ model cards and official CMMLU leaderboard
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 90.8% | Feb 2025 | Lab model card | |
| 2 | | 90.8% | Apr 2026 | Lab model card | |
| 3 | | 90.2% | Oct 2026 | Lab model card | |
| 4 | | 90.2% | Jun 2025 | Lab model card | |
| 5 | | 90.2% | Aug 2025 | Lab model card | |
| 6 | | 90.1% | Jun 2024 | Lab model card | |
| 7 | | 90% | Mar 2025 | Lab model card | |
| 8 | | 89.4% | Oct 2025 | Lab model card | |
| 9 | | 89.1% | Oct 2025 | Lab model card | |
| 10 | | 88.8% | Dec 2024 | Lab model card | |
| 11 | | 84.3% | Aug 2025 | Lab model card | |
| 12 | | 83.5% | Feb 2024 | Lab model card | |
| 13 | | 83.1% | Jan 2025 | Lab model card | |
| 14 | | 81.6% | May 2024 | Lab model card | |
| 15 | Moonlight-16B-A3B | 78.2% | Feb 2025 | Lab model card | |
| 16 | | 71% | Sep 2023 | Lab model card | |
| 17 | | 67.8% | Nov 2023 | Lab model card | |
| 18 | ChatGLM3-6B | 67.5% | Oct 2023 | Lab model card |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 89.7% | Five-shot | Jun 2024 | Official leaderboard | |
| 2 | 88.3% | Five-shot | Apr 2024 | Official leaderboard | |
| 3 | 85.7% | Five-shot | Sep 2024 | Official leaderboard | |
| 4 | 83.5% | Five-shot | Feb 2024 | Official leaderboard | |
| 5 | 82.3% | Five-shot | Feb 2024 | Official leaderboard | |
| 6 | 72.5% | Five-shot | Feb 2024 | Official leaderboard | |
| 7 | 58.7% | Five-shot | Sep 2023 | Official leaderboard | |
| 8 | chatglm2-6b | 50% | Zero-shot | Jun 2023 | Official leaderboard |
| 9 | 41.5% | Five-shot | Mar 2023 | Official leaderboard | |
| 10 | ChatGLM-6B | 40.8% | Zero-shot | Mar 2023 | Official leaderboard |
| 11 | 39.8% | Five-shot | Feb 2023 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 90.8% | Feb 2025 | Lab model card | ||
| 2 | 90.8% | Apr 2026 | Lab model card | ||
| 3 | 90.2% | Oct 2026 | Lab model card | ||
| 4 | 90.2% | Jun 2025 | Lab model card | ||
| 5 | 90.2% | Aug 2025 | Lab model card | ||
| 6 | 90.1% | Jun 2024 | Lab model card | ||
| 7 | 90% | Mar 2025 | Lab model card | ||
| 8 | 89.4% | Oct 2025 | Lab model card | ||
| 9 | 89.1% | Oct 2025 | Lab model card | ||
| 10 | 88.8% | Dec 2024 | Lab model card | ||
| 11 | 88.3% | Five-shot | Apr 2024 | Official leaderboard | |
| 12 | 85.7% | Five-shot | Sep 2024 | Official leaderboard | |
| 13 | 84.3% | Aug 2025 | Lab model card | ||
| 14 | 83.5% | Five-shot | Feb 2024 | Official leaderboard | |
| 15 | 83.1% | Jan 2025 | Lab model card | ||
| 16 | 82.3% | Five-shot | Feb 2024 | Official leaderboard | |
| 17 | 81.6% | May 2024 | Lab model card | ||
| 18 | Moonlight-16B-A3B | 78.2% | Feb 2025 | Lab model card | |
| 19 | 72.5% | Five-shot | Feb 2024 | Official leaderboard | |
| 20 | 71% | Sep 2023 | Lab model card | ||
| 21 | 67.8% | Nov 2023 | Lab model card | ||
| 22 | ChatGLM3-6B | 67.5% | Oct 2023 | Lab model card | |
| 23 | 58.7% | Five-shot | Sep 2023 | Official leaderboard | |
| 24 | chatglm2-6b | 50% | Zero-shot | Jun 2023 | Official leaderboard |
| 25 | 41.5% | Five-shot | Mar 2023 | Official leaderboard | |
| 26 | ChatGLM-6B | 40.8% | Zero-shot | Mar 2023 | Official leaderboard |
| 27 | 39.8% | Five-shot | Feb 2023 | Official leaderboard |
The benchmark's own leaderboard has 47 entries, 11 of them models we track. Switch the scores above to rank them together.
Rankings by part
We read 5 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as STEM, Humanities and Social sciences. Qwen2-72B leads every one of them. Arrows show how far a model moved from its overall rank among the models we track.
STEM
11 models we track have a score here.
- 1
Qwen2-72B 82.8%
- 2
Qwen1.5-110B 81.6%
- 3
Qwen2.5-72B 80.7%
- 4
Qwen1.5-72B 76.8%
- 5
Qwen1.5-32B 76.3%
- 6
Qwen1.5-7B 63.6%
- 7
Qwen-7B 48.4%
- 8 chatglm2-6b 42.5%
- 9
LLaMA-65B ↑2 34.5%
- 10
Falcon-40B ↓1 33.3%
Humanities
11 models we track have a score here.
- 1
Qwen2-72B 93.8%
- 2
Qwen1.5-110B 92.4%
- 3
Qwen2.5-72B 88.4%
- 4
Qwen1.5-72B 88.4%
- 5
Qwen1.5-32B 86.3%
- 6
Qwen1.5-7B 76.4%
- 7
Qwen-7B 63.8%
- 8 chatglm2-6b 52.9%
- 9
Falcon-40B 43.5%
- 10 ChatGLM-6B 42.9%
Social sciences
11 models we track have a score here.
- 1
Qwen2-72B 90.4%
- 2
Qwen1.5-110B 89.1%
- 3
Qwen2.5-72B 86%
- 4
Qwen1.5-72B 84.2%
- 5
Qwen1.5-32B 83.4%
- 6
Qwen1.5-7B 74.7%
- 7
Qwen-7B 61.2%
- 8 chatglm2-6b 53.4%
- 9 ChatGLM-6B ↑1 44.8%
- 10
Falcon-40B ↓1 44.3%
Other
11 models we track have a score here.
- 1
Qwen2-72B 92.7%
- 2
Qwen1.5-110B 91.2%
- 3
Qwen2.5-72B 87.3%
- 4
Qwen1.5-72B 86.1%
- 5
Qwen1.5-32B 84.3%
- 6
Qwen1.5-7B 75.9%
- 7
Qwen-7B 62.1%
- 8 chatglm2-6b 52.2%
- 9
Falcon-40B 44.8%
- 10
LLaMA-65B ↑1 42.9%
China specific topics
11 models we track have a score here.
- 1
Qwen2-72B 90.6%
- 2
Qwen1.5-110B 89%
- 3
Qwen2.5-72B 88.9%
- 4
Qwen1.5-72B 83.8%
- 5
Qwen1.5-32B 82.8%
- 6
Qwen1.5-7B 73.4%
- 7
Qwen-7B 58.8%
- 8 chatglm2-6b 50.6%
- 9 ChatGLM-6B ↑1 41.9%
- 10
Falcon-40B ↓1 39.5%
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Source: Official CMMLU leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Five-shot | 89.7% | ||||
| Five-shot | 88.3% | ||||
| Zero-shot | 87.6% | ||||
| Zero-shot | 87.5% | ||||
| Five-shot | 85.7% | ||||
| Zero-shot | 84.7% | ||||
| Five-shot | 83.5% | ||||
| Five-shot | 82.3% | ||||
| Zero-shot | 81.8% | ||||
| Zero-shot | 81.5% | ||||
| Five-shot | 72.5% | ||||
| Zero-shot | 71% | ||||
| Five-shot | 58.7% | ||||
| Zero-shot | 57.6% | ||||
| chatglm2-6b | Zero-shot | 50% | |||
| chatglm2-6b | Five-shot | 48.8% | |||
| Five-shot | 41.5% | ||||
| ChatGLM-6B | Zero-shot | 40.8% | |||
| Five-shot | 39.8% | ||||
| Zero-shot | 38.5% | ||||
| ChatGLM-6B | Five-shot | 37.5% | |||
| Zero-shot | 34.9% |
4 models have held number one in labs’ own numbers since its release in June 2023 6 models have held number one across all independent runs since its release in June 2023 7 models have held number one with claims included since its release in June 2023
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
Sources: Official CMMLU leaderboard and labs’ model cards
Qwen1.5-110B on the official leaderboard passed 85% in April 2024
That was 10 months after its release. By the same age, 13% of the benchmarks from 2023 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.
Sources: Official CMMLU leaderboard and labs’ model cards
Labs headline it less often now, last in October 2026
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Oct 7, 2026 | 90.2 | CMMLU | |
| Apr 24, 2026 | 90.8 | CMMLU (EM) | |
| Oct 31, 2025 | 89.4 | CMMLU(acc) | |
| Aug 29, 2025 | 84.3 | CMMLU(acc) | |
| Jun 30, 2025 | 90.2 | CMMLU | |
| Mar 21, 2025 | 90 | CMMLU | |
| Mar 16, 2025 | CMMLU | ||
| Feb 27, 2025 | 90.8 | CMMLU | |
| Feb 22, 2025 | Moonlight-16B-A3B | 78.2 | CMMLU |
| Jan 28, 2025 | CMMLU | ||
| Jan 13, 2025 | 83.1 | CMMLU(0-shot) | |
| Dec 26, 2024 | 88.8 | CMMLU (Acc.) | |
| Jun 7, 2024 | 90.1 | CMMLU | |
| May 7, 2024 | 81.6 | CMMLU | |
| Feb 5, 2024 | 83.5 | CMMLU | |
| Nov 29, 2023 | 67.8 | CMMLU | |
| Oct 27, 2023 | ChatGLM3-6B | 67.5 | CMMLU |
| Sep 25, 2023 | 71 | CMMLU (5-shot) |
8 system cards report it, from 6 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Oct 31, 2025 | | 89.4 | CMMLU (acc) (Table 13) | Capabilities |
| Oct 21, 2025 | | 89.1 | CMMLU (Table 1) | Capabilities |
| Sep 1, 2025 | | 84.3 | CMMLU (acc) (Table 3) | Capabilities |
| Aug 25, 2025 | | 90.2 | CMMLU, OpenCompass (Table 14) | Capabilities |
| Jun 29, 2025 | | 90.2 | CMMLU (Table 5) | Capabilities |
| Jun 18, 2024 | ChatGLM3-6B technical report | 67.5 | ChatGLM3-6B-Base, CMMLU (Table 1) | Capabilities |
| May 7, 2024 | | 81.6 | CMMLU 5-shot, DeepSeek-V2 Chat (RL) (Table 3) | Capabilities |
| Jan 5, 2024 | | 67.8 | CMMLU 0-shot, DeepSeek 67B Chat (Table 6) | Capabilities |
2 lab claims next to the leaderboard
Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 1 of them match, which usually means one side copied the other. Every benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
All 10 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 3 pass 2 notes
- Pass Independent runs Scores from the official leaderboard (47 entries), launch posts (16 scores) and system cards (8 scores).
- Pass Models scored 27 models we track, plus 36 other entries on its leaderboard.
- Pass Newest score The newest is from October 2026, for MiniCPM-V-4.7-35B-A3B.
- Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is DeepSeek-V4-Pro (#70).
- Note Headroom Saturated. The best is 90.8%, past 85% since April 2024.
Leaderboard import 3 pass
- Pass Importer We read the board on github.com: 22 rows for models we track and 36 other entries at the last read.
- Pass Last read November 18th, 2026.
- Pass Read worked The last read worked.
Consistency 2 pass
- Pass Lab claims Both pairs agree within 10 points.
- Pass Known issues No published audit found problems.