Qwen-7B

Qwen-7B places in the top three on CFLUE Knowledge.

Qwen Released September 28th, 2023 Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Qwen Sep 2023

Qwen-7B

Open weights

89.2 #231 of 256

Epoch Capabilities Index

Self-Reported Index 94.5 #399 of 404

OlympicArena 4.34% #25 /25
MathBench 27.7% #18 /21
CyberMetric 54.1% #6 /10

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
ARC-Challenge	75.3%	#21 of 75	Claim in Qwen Technical Report, via Epoch AI
CMMLU	58.7%	#7 of 11	Official leaderboard
BoolQ	76.4%	#42 of 61	Claim in Qwen Technical Report, via Epoch AI
PIQA	77.9%	#42 of 58	Claim in Qwen Technical Report, via Epoch AI
LAMBADA	67.9%	#23 of 26	Claim in Qwen Technical Report, via Epoch AI
BIG-Bench Hard	46.7%	#65 of 73	Lab model card
EQ-Bench v2	50.1	#77 of 85	Official leaderboard
MMLU	57%	#120 of 131	Lab model card
C-Eval	59.7%	#44 of 46	Lab model card
Qwen Chinese human evaluation	28.4%	#2 of 2	Lab model card

Math and science

Benchmark	Score	Rank	Source
MathBench	27.7%	#18 of 21	Official leaderboard
GSM8K	54.1%	#85 of 92	Lab model card
MATH-500	6.80%	#70 of 70	Lab model card
OlympicArena	4.34%	#25 of 25	Official leaderboard
Math23K	51.2%	#3 of 3	Lab model card
Math401	57.4%	#2 of 2	Lab model card

Coding

Benchmark	Score	Rank	Source
HumanEval	37.2%	#93 of 101	Lab model card
MBPP	35.8%	#66 of 71	Lab model card
HumanEvalPack	24.2%	#3 of 3	Lab model card

Agents and tools

Benchmark	Score	Rank	Source
T-Eval	59.5%	#4 of 12	Official leaderboard
Chinese Tool-Use Benchmark	98%	#2 of 4	Lab model card
Code Interpreter Benchmark	70.2%	#2 of 3	Lab model card
HuggingFace Agent Benchmark	87%	#2 of 3	Lab model card

Professional work

Benchmark	Score		Rank	Source
CFLUE Knowledge	43.6%		#2 of 8	Official leaderboard

Safety and security

Benchmark	Score		Rank	Source
CyberMetric	54.1%		#6 of 10	Official leaderboard
SuperCLUE-Safety	78.6		#6 of 7	Official leaderboard

Its lab's numbers put it at 94.5, 5.3 points above its ECI

The Self-Reported Index fits one score to the 7 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 91.1 to 98.0. How we compute it.

C-Eval 59.7
MMLU 57
GSM8K 54.1
BIG-Bench Hard 46.7
HumanEval 37.2
MBPP 35.8
MATH-500 6.8

Qwen's models over time

Qwen's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its system card

Every result in the longer documents the lab published with it. 14 of them aren't in the launch post.

Qwen Technical Report (arXiv 2309.16609) Sep 28, 2023