Qwen2.5-Coder (7B)

Qwen2.5-Coder (7B).

Qwen Released September 18th, 2024 Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Qwen Sep 2024

Open weights

107.7 #206 of 256

Epoch Capabilities Index

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Benchmark	Score	Rank	Source
MMLU	68%	#31 of 85	Claim in Qwen2.5-Coder Technical Report, via Epoch AI
ARC-Challenge	60.9%	#33 of 75	Claim in Qwen2.5-Coder Technical Report, via Epoch AI
Open Arabic LLM Leaderboard v1	43.3%	#30 of 55	Official leaderboard
WinoGrande	72.9%	#42 of 76	Claim in Qwen2.5-Coder Technical Report, via Epoch AI
HellaSwag	76.8%	#35 of 63	Claim in Qwen2.5-Coder Technical Report, via Epoch AI
Ko-IFEval (Upstage)	29.6%	#17 of 21	Official leaderboard
MuSR	34.5%	#90 of 98	Official leaderboard

Benchmark	Score		Rank	Source
GSM8K	86.7%		#12 of 70	Claim in Qwen2.5-Coder Technical Report, via Epoch AI

Benchmark	Score	Rank	Source
BigCodeBench	44.6%	#37 of 69	Official leaderboard
BigCodeBench-Hard	20.3%	#58 of 96	Official leaderboard
AutoCodeBench	22.5%	#31 of 34	Official leaderboard
LiveSQLBench	6.33%	#34 of 36	Official leaderboard

Benchmark	Score		Rank	Source
AceBench	48.9%		#13 of 16	Official leaderboard

Benchmark	Score		Rank	Source
Open LLM Leaderboard v2	19.2		#57 of 99	Official leaderboard

Qwen's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)