Qwen1.5-72B

Qwen1.5-72B.

Qwen Released February 4th, 2024 Family Qwen1.5 The lab's model card Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Qwen Feb 2024

Qwen1.5-72B

Open weights Flagship

108.5 #380 of 404

Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.

GPQA Diamond 28.8% #210 /229
BigCodeBench-Hard 12.2% #79 /96
Open Arabic LLM Leaderboard v1 48.3% #17 /55

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
EQ-Bench v2	82.8	#12 of 85	Official leaderboard
MixEval	84.1%	#10 of 34	Official leaderboard
Open Arabic LLM Leaderboard v1	48.3%	#17 of 55	Official leaderboard
CFBench	60.6%	#6 of 19	Official leaderboard
CulturalBench	44.7%	#7 of 15	Official leaderboard
WildBench	43.5	#22 of 47	Official leaderboard
MMLU-CF	59.8%	#18 of 37	Official leaderboard
RepoQA	67%	#12 of 22	Official leaderboard
Arena-Hard-Auto	36.1%	#22 of 38	Official leaderboard
ThaiExam	53.8%	#14 of 24	Official leaderboard
MMLU-Pro	52.6%	#129 of 212	Official leaderboard
CMMLU	83.5%	#11 of 17	Lab model card
MT-Bench	8.61	#15 of 23	Lab model card
MMLU	77.5%	#88 of 131	Lab model card
C-Eval	84.1%	#33 of 46	Lab model card
BIG-Bench Hard	65.5%	#53 of 73	Lab model card
LMArena (Elo)	1,234	#270 of 335	Official leaderboard
MGSM	61.7%	#49 of 57	Lab model card
Flores	35.6	#7 of 8	Lab model card
AlpacaEval 2.0	27.2%	#15 of 16	Lab model card
RewardBench	67.2%	#46 of 48	Official leaderboard
RULER	55.7%	#15 of 15	Official leaderboard
Multilingual Exams	66.3%	#3 of 3	Lab model card
Multilingual Understanding	78.2%	#3 of 3	Lab model card
L-Eval	78.1%	#1 of 1	Lab model card

Math and science

Benchmark	Score	Rank	Source
GSM8K	79.5%	#66 of 92	Lab model card
MATH	34.1%	#84 of 99	Lab model card
GPQA Diamond	28.8%	#210 of 229	Epoch AI run

Coding

Benchmark	Score	Rank	Source
McEval	35.8%	#2 of 4	Official leaderboard
MBPP+	61.6%	#19 of 35	Official leaderboard
HumanEval+	59.1%	#27 of 46	Official leaderboard
BigCodeBench	36.8%	#51 of 69	Official leaderboard
BigCodeBench-Hard	12.2%	#79 of 96	Official leaderboard
MBPP	53.4%	#59 of 71	Lab model card
HumanEval	41.5%	#89 of 101	Lab model card

Agents and tools

Benchmark	Score	Rank	Source
GTA	13.3%	#6 of 15	Official leaderboard
Code Interpreter Benchmark Math	73.1%	#1 of 2	Lab model card
T-Eval	76.7%	#1 of 2	Lab model card
Chinese Tool-Use Benchmark	95.1%	#3 of 4	Lab model card

Professional work

Benchmark	Score		Rank	Source
MedQA	67%		#34 of 71	Official leaderboard

Vision, audio and robotics

Benchmark	Score		Rank	Source
SGP-Bench	46.6%		#14 of 22	Official leaderboard

Safety and security

Benchmark	Score	Rank	Source
SALAD-Bench	93.5%	#5 of 13	Official leaderboard
SimpleSafetyTests	99%	#31 of 75	Official leaderboard
OR-Bench	46.9%	#14 of 28	Official leaderboard
Anthropic Red Team	99%	#40 of 75	Official leaderboard
HELM Safety	88.6	#54 of 75	Official leaderboard
SORRY-Bench	35.9%	#18 of 23	Official leaderboard
AIR-Bench 2024	48.6%	#67 of 82	Official leaderboard
nocot-bench (NCRI)	49.7	#166 of 185	Official leaderboard

Other

Benchmark	Score		Rank	Source
HELM Lite	60.8		#28 of 71	Official leaderboard

Its lab's numbers put it at 108.5, number 380 of 404

The Self-Reported Index fits one score to the 10 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 105.4 to 111.7. How we compute it.

C-Eval 84.1
CMMLU 83.5
GSM8K 79.5
MMLU 77.5
BIG-Bench Hard 65.5
MGSM 61.7
MBPP 53.4
HumanEval 41.5
MATH 34.1
AlpacaEval 2.0 27.2

Its lab's numbers next to 4 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark	Lab says	Board says	Gap
HumanEval	41.5% HumanEval	68.3% Leaderboard	−27 pts
MBPP	53.4% MBPP	72.5% Leaderboard, EvalPlus subset, original tests	−19 pts
AlpacaEval 2.0	27.2% AlpacaEval 2.0 (Win Rate)	36.6% Leaderboard, verified	−9.4 pts
CMMLU	83.5% CMMLU	83.5% Leaderboard, Five-shot	Same

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Introducing Qwen1.5 (blog) Feb 5, 2024

MMLU 77.5
C-Eval 84.1
GSM8K 79.5
MATH 34.1
HumanEval 41.5
MBPP 53.4
BBH 65.5
CMMLU 83.5
MT-Bench (Avg. Score) 8.61
AlpacaEval 2.0 (Win Rate) 27.2
Multilingual Exams (MMLU-multi, M3Exams) 66.3
Multilingual Understanding (BELEBELE, XWinograd, XCOPA, PAWS-X, XStoryCloze) 78.2
Multilingual Math (MGSM, 8-shot) 61.7
Multilingual Translation (Flores, BLEU) 35.6
L-Eval (Avg.) 78.1
RGB English: Noise 0.8 81.67 / Rejection 1.0 48.67 / Integration 0.4 61.00 / Counterfactual 28.00
T-Eval English (Overall) 76.7
Tool-Use Benchmark (Tool Selection Acc.) 95.1
Code Interpreter Benchmark (Accuracy of Code Execution Results, Math) 73.1