Benchmark Almanac

Qwen2.5-32B

Qwen2.5-32B ranks first on AbstentionBench.

Qwen Released September 17th, 2024 Compare with others
63
Benchmarks with a score
1
Ranked first, out of five or more models
7
In the top three
Qwen Sep 2024

Qwen2.5-32B

Open weights

127.7 #292 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • GPQA Diamond 46.1% #173 /229
  • KoBALT 30% #5 /12
  • KOR-Bench 54.7% #6 /33

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
MuSR 49.8% #4 of 98 Official leaderboard
AlGhafa 78.1% #3 of 39 Official leaderboard
MT-Bench 9.20 #2 of 23 Lab model card
Open Arabic LLM Leaderboard v2 70% #4 of 39 Official leaderboard
Open Arabic LLM Leaderboard v1 54.2% #6 of 55 Official leaderboard
ArabicMMLU 68% #5 of 39 Official leaderboard
ArbMMLU-HT 65.7% #5 of 39 Official leaderboard
MMLU-CF 69.7% #5 of 37 Official leaderboard
MLogiQA 57.4% #3 of 17 Official leaderboard
MadinahQA 57.1% #7 of 39 Official leaderboard
AbstentionBench 75.3% #1 of 5 Official leaderboard
ALRAGE 79.2% #8 of 39 Official leaderboard
EQ-Bench v2 79.9 #25 of 85 Official leaderboard
Vectara Hallucination Leaderboard (HHEM) 3.00% #45 of 148 Official leaderboard
RULER 92.9% #13 of 42 Lab model card
MGSM 87.2% #20 of 57 Lab model card
KoBALT 30% #5 of 12 Official leaderboard
Chinese SimpleQA 41.1% #12 of 27 Official leaderboard
Okapi MMLU 77.2% #4 of 8 Lab model card
LongBench-Chat 8.70 #3 of 6 Lab model card
AMMLU 70.4% #2 of 4 Lab model card
IndoMMLU 66.4% #2 of 4 Lab model card
JMMLU 76.5% #2 of 4 Lab model card
Multilingual IFEval 82.7% #2 of 4 Lab model card
TurkishMMLU 72.4% #2 of 4 Lab model card
FaithJudge 19.2% #21 of 37 Official leaderboard
Arena-Hard-Auto 74.5% #23 of 39 Lab model card
BLEnD 27.9% #3 of 5 Lab model card
SuperGPQA 38.8% #38 of 59 Official leaderboard
KMMLU 60.8% #4 of 6 Lab model card
MMLU-Pro 69% #125 of 167 Lab model card
IFEval 79.5% #106 of 135 Lab model card
MMLU-Redux 83.9% #36 of 45 Lab model card

Math and science

BenchmarkScoreRankSource
GSM8K 95.9% #7 of 92 Lab model card
MATH 83.1% #18 of 99 Lab model card
KOR-Bench 54.7% #6 of 33 Official leaderboard
PHYSICS 27.6% #6 of 20 Official leaderboard
ZebraLogic 26.1% #17 of 46 Official leaderboard
U-MATH 52.4% #14 of 30 Official leaderboard
MATH Level 5 56.1% #46 of 94 Epoch AI run
ChemBench 52.9% #16 of 24 Official leaderboard
GPQA Diamond 46.1% #173 of 229 Epoch AI run
OTIS Mock AIME 2024-2025 7.36% #170 of 212 Epoch AI run

Coding

BenchmarkScoreRankSource
MBPP 84% #12 of 71 Lab model card
HumanEval 88.4% #22 of 101 Lab model card
BigCodeBench 48.6% #27 of 69 Official leaderboard
MultiPL-E 75.4% #9 of 18 Lab model card
BigCodeBench-Hard 24.6% #49 of 96 Official leaderboard
LiveCodeBench 51.2% #29 of 54 Lab model card
ContextBench 7.60% #8 of 9 Official leaderboard
Aider-Edit 54.1% #31 of 34 Official leaderboard

Agents and tools

BenchmarkScoreRankSource
When2Call 32.9% #5 of 10 Official leaderboard
WebWalkerQA 26% #3 of 5 Official leaderboard
C3-Bench 5.86% #20 of 30 Official leaderboard
AgentBench FC 37.2% #10 of 12 Official leaderboard

Professional work

BenchmarkScoreRankSource
MultiFinBen 12.7 #7 of 9 Official leaderboard

Vision, audio and robotics

BenchmarkScoreRankSource
OCR-Reasoning 26.5% #7 of 15 Official leaderboard
Ego3D-Bench 13.7 #11 of 11 Official leaderboard

Safety and security

BenchmarkScoreRankSource
AraTrust 86.7% #7 of 39 Official leaderboard
nocot-bench (NCRI) 61.1 #135 of 185 Official leaderboard

Other

BenchmarkScoreRankSource
Open LLM Leaderboard v2 38.0 #8 of 99 Official leaderboard
Political Science LLM Benchmark 64.9% #31 of 42 Official leaderboard
LiveBench 50.7% #20 of 27 Lab model card

Its lab's numbers put it at 127.7, number 292 of 404

The Self-Reported Index fits one score to the 14 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 125.7 to 129.7. How we compute it.

Its lab's numbers next to 2 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
GPQA Diamond 49.5% GPQA, variant not stated, Qwen2.5-32B-Instruct (Table 7) 46.1% Epoch AI run +3.4 pts
MMLU-Pro 69% MMLU-Pro, Qwen2.5-32B-Instruct (Table 7) 69.2% Leaderboard −0.2 pts

Its system card

Every result in the longer documents the lab published with it. 27 of them aren't in the launch post.

Qwen2.5 Technical Report (arXiv 2412.15115) Dec 19, 2024

Capabilities