Qwen2.5-72B

Qwen2.5-72B ranks first on 10 of the 128 benchmarks with five or more models scored.

Qwen Released September 19th, 2024 Family Qwen2.5 The lab's model card Compare with others

139

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Qwen Sep 2024

Qwen2.5-72B

Open weights Flagship

127.2 #165 of 256

Epoch Capabilities Index

Self-Reported Index 127.7 #294 of 404

GPQA Diamond 49.1% #157 /229
OSWorld 5.00% #8 /8
KoBALT 37% #2 /12

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
Open Arabic LLM Leaderboard v1	69.1%	#1 of 55	Official leaderboard
ArabicMMLU	71.4%	#1 of 39	Official leaderboard
ArbMMLU-HT	72.2%	#1 of 39	Official leaderboard
Open Arabic LLM Leaderboard v2	72.4%	#1 of 39	Official leaderboard
QIMMA	68.5%	#1 of 25	Official leaderboard
MT-Bench	9.35	#1 of 23	Lab model card
AlGhafa	78.2%	#2 of 39	Official leaderboard
MadinahQA	65%	#2 of 39	Official leaderboard
MMLU-CF	71.6%	#2 of 37	Official leaderboard
MuSR	47.7%	#6 of 98	Official leaderboard
MLogiQA	58.4%	#2 of 17	Official leaderboard
RULER	95.1%	#5 of 42	Lab model card
Okapi MMLU	80%	#1 of 8	Lab model card
KoBALT	37%	#2 of 12	Official leaderboard
Alyah	74.6%	#2 of 10	Official leaderboard
LLM-AggreFact	75.6%	#2 of 10	Official leaderboard
WinoGrande	83.9%	#10 of 47	Lab model card
Global PIQA	78.3%	#11 of 44	Official leaderboard
AMMLU	72.4%	#1 of 4	Lab model card
IndoMMLU	69.3%	#1 of 4	Lab model card
JMMLU	80.6%	#1 of 4	Lab model card
Multilingual IFEval	87%	#1 of 4	Lab model card
TurkishMMLU	76.1%	#1 of 4	Lab model card
Chinese SimpleQA	50.2%	#7 of 27	Official leaderboard
CMMLU	85.7%	#3 of 11	Official leaderboard
BIG-Bench Hard	86.3%	#20 of 73	Lab model card
MGSM	88.2%	#16 of 57	Lab model card
ALRAGE	78.8%	#11 of 39	Official leaderboard
PIQA	82.6%	#18 of 58	Claim in DeepSeek-V3 Technical Report, via Epoch AI
EQ-Bench v2	79.0	#27 of 85	Official leaderboard
HellaSwag	87.6%	#17 of 53	Lab model card
LongBench-Chat	8.72	#2 of 6	Lab model card
Multilingual Exams	78.7%	#1 of 3	Lab model card
Multilingual Understanding	89.6%	#1 of 3	Lab model card
MMLU	86.1%	#48 of 131	Lab model card
BLEnD	32.5%	#2 of 5	Lab model card
SwitzerlandQA	70%	#3 of 7	Official leaderboard
Arena-Hard-Auto	81.2%	#18 of 39	Lab model card
Vectara Hallucination Leaderboard (HHEM)	4.30%	#73 of 148	Official leaderboard
TruthfulQA	60.4%	#15 of 30	Lab model card
Flores	39.0	#4 of 8	Lab model card
KMMLU	62%	#3 of 6	Lab model card
PenguinScrolls	81.8%	#1 of 2	Official leaderboard
AraGen	48.9%	#22 of 42	Official leaderboard
ARC-Challenge	72.4%	#29 of 49	Lab model card
SuperGPQA	40.8%	#35 of 59	Official leaderboard
BuzzBench	45.9	#12 of 20	Official leaderboard
AlignBench	8.16	#9 of 15	Lab model card
TableBench	48.8%	#6 of 10	Official leaderboard
LLM Confabulation Leaderboard	19.1%	#36 of 58	Official leaderboard
FaithJudge	20.7%	#23 of 37	Official leaderboard
LongBench v2	43.5%	#21 of 33	Official leaderboard
HELM Arabic	76.9	#18 of 28	Official leaderboard
IFEval	84.1%	#89 of 135	Lab model card
LMArena (Elo)	1,303	#230 of 335	Official leaderboard
Encyclo-K	25%	#16 of 23	Official leaderboard
TriviaQA	71.9%	#27 of 38	Claim in DeepSeek-V3 Technical Report, via Epoch AI
MMLU-Redux	86.8%	#32 of 45	Lab model card
MMLU-Pro	71.1%	#120 of 167	Lab model card
WritingBench	57.8	#33 of 45	Official leaderboard
KINA	23%	#31 of 37	Official leaderboard
EQ-Bench 3	896	#68 of 78	Official leaderboard
Arena-Hard v2.0	10.1%	#23 of 23	Official leaderboard
MARS-Bench	29.2%	#15 of 15	Official leaderboard
MultiLoKo	16.4%	#7 of 7	Official leaderboard
MMLU-stem	82.7%	#2 of 2	Lab model card

Math and science

Benchmark	Score	Rank	Source
GSM8K	95.8%	#9 of 92	Lab model card
MATH	83.1%	#17 of 99	Lab model card
PHYSICS	28.7%	#5 of 20	Official leaderboard
KOR-Bench	52.2%	#9 of 33	Official leaderboard
CNMO 2024	15.9%	#2 of 7	Official leaderboard
ZebraLogic	26.6%	#16 of 46	Official leaderboard
MATH Level 5	63.2%	#41 of 94	Epoch AI run
U-MATH	51.2%	#15 of 30	Official leaderboard
IneqMath	2.50%	#21 of 34	Official leaderboard
GPQA Diamond	49.1%	#157 of 229	Epoch AI run
GPQA Main	42.6%	#43 of 60	Official leaderboard
DTBench	62.9%	#130 of 180	Official leaderboard
ForecastBench	0.15	#62 of 82	Official leaderboard
OTIS Mock AIME 2024-2025	8.06%	#165 of 212	Epoch AI run
GSM8K Platinum Initial 268	97.4%	#16 of 19	Official leaderboard
PolyMath	16.9%	#36 of 38	Official leaderboard
AutoLogi	66%	#2 of 2	Official leaderboard
TheoremQA	42.4%	#2 of 2	Lab model card

Coding

Benchmark	Score	Rank	Source
MBPP	88.2%	#4 of 71	Lab model card
BigCodeBench	50.8%	#18 of 69	Official leaderboard
HumanEval	86.6%	#34 of 101	Lab model card
LiveCodeBench	55.5%	#25 of 54	Lab model card
BigCodeBench-Hard	25.4%	#45 of 96	Official leaderboard
FullStackBench	56.9%	#5 of 10	Official leaderboard
MultiPL-E	75.1%	#10 of 18	Lab model card
AutoCodeBench	34.3%	#26 of 34	Official leaderboard
Multi-SWE-bench mini	6.00%	#4 of 5	Official leaderboard
METR Time Horizon	5.8 min	#27 of 31	Official leaderboard
ArtifactsBench (Gemini 2.5 Pro 0506 judge)	34.5%	#16 of 18	Official leaderboard
OIBench	1.80%	#8 of 9	Official leaderboard
Multi-SWE-bench	2.57%	#10 of 11	Official leaderboard
WeirdML	16%	#105 of 112	Official leaderboard

Agents and tools

Benchmark	Score	Rank	Source
WebWalkerQA	33.3%	#1 of 5	Official leaderboard
Galileo Agent Leaderboard v2	51%	#7 of 21	Official leaderboard
AceBench	74.7%	#6 of 16	Official leaderboard
ComplexFuncBench	40.1%	#5 of 11	Official leaderboard
AgentBench FC	40.8%	#7 of 12	Official leaderboard
C3-Bench	6.25%	#18 of 30	Official leaderboard
When2Call	32.8%	#6 of 10	Official leaderboard
Galileo Agent Leaderboard	0.82	#20 of 29	Official leaderboard
Step Race	1	#45 of 59	Official leaderboard
BALROG	16.2%	#26 of 33	Official leaderboard
SnakeBench	18	#155 of 188	Official leaderboard
LLM Chess	-672	#117 of 123	Official leaderboard
BrowseComp-ZH	6.60%	#9 of 9	Official leaderboard
OSWorld	5.00%	#8 of 8	OS World Website, via Epoch AI

Professional work

Benchmark	Score	Rank	Source
BRIDGE	51.0	#4 of 39	Official leaderboard
FinEval	69.4%	#1 of 7	Official leaderboard
BizFinBench	67.7%	#3 of 13	Official leaderboard
MedQA	75.3%	#18 of 71	Official leaderboard
LegalBench	79.4%	#91 of 138	Official leaderboard
TheAgentCompany	5.70%	#5 of 7	Official leaderboard
BizFinBench v2	35.9%	#7 of 9	Official leaderboard

Vision, audio and robotics

Benchmark	Score	Rank	Source
Ego3D-Bench	7.50	#3 of 11	Official leaderboard
VRBench	53.5%	#10 of 19	Official leaderboard
SpatialViz-Bench	33.3%	#6 of 8	Claim in arxiv.org, via Epoch AI

Safety and security

Benchmark	Score	Rank	Source
AraTrust	89.2%	#2 of 39	Official leaderboard
SimpleSafetyTests	100%	#6 of 75	Official leaderboard
Anthropic Red Team	99.6%	#16 of 75	Official leaderboard
HELM Safety	93.1	#31 of 75	Official leaderboard
AIR-Bench 2024	59%	#53 of 82	Official leaderboard
nocot-bench (NCRI)	63.5	#126 of 185	Official leaderboard
BaxBench	10.4%	#27 of 34	Official leaderboard
AutoBaxBench	6.67%	#9 of 11	Official leaderboard
FORTRESS	56.4%	#66 of 73	Official leaderboard

Other

Benchmark	Score	Rank	Source
Open LLM Leaderboard v2	48.0	#1 of 99	Official leaderboard
HELM Lite	74.5	#18 of 71	Official leaderboard
LiveBench	52.3%	#18 of 27	Lab model card
HELM Capabilities	59.9	#42 of 60	Official leaderboard
Political Science LLM Benchmark	64.8%	#32 of 42	Official leaderboard
ACCoRD	42.5	#128 of 152	Official leaderboard
Conceptual Reasoning Index	24.2	#129 of 151	Official leaderboard
LMCA	13.4%	#127 of 147	Official leaderboard

Its lab's numbers put it at 127.7, in line with its ECI

The Self-Reported Index fits one score to the 20 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 125.1 to 130.2. How we compute it.

Its lab's numbers next to 4 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark	Lab says	Board says	Gap
IFEval	84.1% IFeval strict-prompt	80.6% Leaderboard, strict, share of instructions followed	+3.5 pts
Arena-Hard-Auto	81.2% Arena-Hard	78% Leaderboard	+3.2 pts
MMLU-Pro	71.1% MMLU-Pro	71.6% Leaderboard	−0.5 pts
GPQA Diamond	49% GPQA	49.1% Epoch AI run	Same

Qwen's models over time

Qwen's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Qwen2.5-LLM: Extending the boundary of LLMs (blog) Sep 19, 2024

Its system card

Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.

Qwen2.5 Technical Report (arXiv 2412.15115) Dec 19, 2024

Capabilities

MMLU-Pro 71.1
MMLU-Redux 86.8
LiveBench 52.3
GPQA Diamond 49
MATH 83.1
GSM8K 95.8
HumanEval 86.6
MBPP 88.2
MultiPL-E 75.1
LiveCodeBench 55.5
IFEval 84.1
Arena-Hard-Auto 81.2
MT-Bench 9.35
Multilingual IFEval 87
AMMLU 72.4
JMMLU 80.6
KMMLU 62
IndoMMLU 69.3
TurkishMMLU 76.1
Okapi MMLU 80
MGSM 88.2
BLEnD 32.5
Qwen in-house English automatic evaluation
Qwen in-house Chinese automatic evaluation
RULER 95.1
LV-Eval
LongBench-Chat 8.72