Qwen2.5-14B

Qwen2.5-14B places in the top three on WebWalkerQA.

Qwen Released September 19th, 2024 Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Qwen Sep 2024

Qwen2.5-14B

Open weights

124.6 #316 of 404

Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.

KOR-Bench 49.4% #12 /33
MolecularIQ 6.63% #11 /17
When2Call 36.2% #4 /10

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
ArabicMMLU	68.5%	#4 of 39	Official leaderboard
MadinahQA	59.9%	#5 of 39	Official leaderboard
MMLU	79.9%	#11 of 85	Claim in Phi-4 Technical Report, via Epoch AI
AlGhafa	77.2%	#6 of 39	Official leaderboard
ArbMMLU-HT	61.7%	#6 of 39	Official leaderboard
Open Arabic LLM Leaderboard v2	68.5%	#6 of 39	Official leaderboard
MuSR	45%	#16 of 98	Official leaderboard
Open Arabic LLM Leaderboard v1	53.4%	#9 of 55	Official leaderboard
MMLU-CF	67%	#9 of 37	Official leaderboard
ALRAGE	78.9%	#10 of 39	Official leaderboard
MLogiQA	53.5%	#5 of 17	Official leaderboard
EQ-Bench v2	79.2	#26 of 85	Official leaderboard
RULER	91.4%	#18 of 42	Lab model card
MT-Bench	8.88	#10 of 23	Lab model card
Ko-IFEval (Upstage)	37.6%	#10 of 21	Official leaderboard
Vectara Hallucination Leaderboard (HHEM)	4.20%	#71 of 148	Official leaderboard
MGSM	82.3%	#31 of 57	Lab model card
Chinese SimpleQA	37.2%	#15 of 27	Official leaderboard
Okapi MMLU	72.1%	#5 of 8	Lab model card
SuperGPQA	35.1%	#41 of 59	Official leaderboard
Arena-Hard-Auto	68.3%	#28 of 39	Lab model card
AMMLU	66.8%	#3 of 4	Lab model card
IndoMMLU	65.1%	#3 of 4	Lab model card
JMMLU	72.8%	#3 of 4	Lab model card
Multilingual IFEval	77.1%	#3 of 4	Lab model card
TurkishMMLU	66.8%	#3 of 4	Lab model card
IFEval	81%	#102 of 135	Lab model card
FaithJudge	29%	#28 of 37	Official leaderboard
BLEnD	27%	#4 of 5	Lab model card
KMMLU	59.7%	#5 of 6	Lab model card
LongBench-Chat	8.04	#5 of 6	Lab model card
MMLU-Pro	63.7%	#141 of 167	Lab model card
MMLU-Redux	80%	#40 of 45	Lab model card

Math and science

Benchmark	Score	Rank	Source
GSM8K	94.8%	#14 of 92	Lab model card
MATH	80%	#27 of 99	Lab model card
KOR-Bench	49.4%	#12 of 33	Official leaderboard
PHYSICS	19.6%	#8 of 20	Official leaderboard
GPQA Diamond	45.5%	#258 of 302	Lab model card

Coding

Benchmark	Score	Rank	Source
MBPP	82%	#16 of 71	Lab model card
BigCodeBench	46%	#32 of 69	Official leaderboard
HumanEval	83.5%	#47 of 101	Lab model card
BigCodeBench-Hard	20.9%	#54 of 96	Official leaderboard
LiveCodeBench	42.6%	#33 of 54	Lab model card
MultiPL-E	72.8%	#12 of 18	Lab model card

Agents and tools

Benchmark	Score	Rank	Source
When2Call	36.2%	#4 of 10	Official leaderboard
WebWalkerQA	27.5%	#2 of 5	Official leaderboard
C3-Bench	7.81%	#15 of 30	Official leaderboard
AgentBench FC	27.2%	#12 of 12	Official leaderboard

Safety and security

Benchmark	Score		Rank	Source
AraTrust	86.5%		#8 of 39	Official leaderboard
MolecularIQ	6.63%		#11 of 17	Official leaderboard

Other

Benchmark	Score		Rank	Source
Open LLM Leaderboard v2	31.9		#21 of 99	Official leaderboard
LiveBench	44.4%		#24 of 27	Lab model card

Its lab's numbers put it at 124.6, number 316 of 404

The Self-Reported Index fits one score to the 14 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 122.8 to 126.5. How we compute it.

Its lab's number next to a leaderboard

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark	Lab says	Board says	Gap
MMLU-Pro	63.7% MMLU-Pro, Qwen2.5-14B-Instruct (Table 7)	63.7% Leaderboard	Same

Its system card

Every result in the longer documents the lab published with it. 27 of them aren't in the launch post.

Qwen2.5 Technical Report (arXiv 2412.15115) Dec 19, 2024

Capabilities

MMLU-Pro 63.7
MMLU-Redux 80
LiveBench 44.4
GPQA Diamond 45.5
MATH 80
GSM8K 94.8
HumanEval 83.5
MBPP 82
MultiPL-E 72.8
LiveCodeBench 42.6
IFEval 81
Arena-Hard-Auto 68.3
MT-Bench 8.88
Multilingual IFEval 77.1
AMMLU 66.8
JMMLU 72.8
KMMLU 59.7
IndoMMLU 65.1
TurkishMMLU 66.8
Okapi MMLU 72.1
MGSM 82.3
BLEnD 27
Qwen in-house English automatic evaluation
Qwen in-house Chinese automatic evaluation
RULER 91.4
LV-Eval
LongBench-Chat 8.04