Benchmark Almanac

Qwen2.5-7B

Qwen2.5-7B places in the top three on MadinahQA.

Qwen Released September 19th, 2024 Compare with others
81
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Qwen Sep 2024

Qwen2.5-7B

Open weights

119.2 #349 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • GPQA Diamond 35.5% #196 /229
  • KoBALT 19% #8 /12
  • KOR-Bench 38.6% #22 /33

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
MadinahQA 62.7% #3 of 39 Official leaderboard
MuSR 44.2% #22 of 98 Official leaderboard
ArabicMMLU 61.4% #9 of 39 Official leaderboard
Vectara Hallucination Leaderboard (HHEM) 2.80% #37 of 148 Official leaderboard
AlGhafa 72.2% #10 of 39 Official leaderboard
Open Arabic LLM Leaderboard v2 61.1% #11 of 39 Official leaderboard
Open Arabic LLM Leaderboard v1 48.6% #16 of 55 Official leaderboard
ArbMMLU-HT 51.7% #12 of 39 Official leaderboard
ALRAGE 77.4% #13 of 39 Official leaderboard
MMLU-CF 61.3% #17 of 37 Official leaderboard
MLogiQA 45.9% #8 of 17 Official leaderboard
MT-Bench 8.75 #12 of 23 Lab model card
EQ-Bench v2 69.2 #54 of 85 Official leaderboard
Ko-IFEval (Upstage) 33.2% #14 of 21 Official leaderboard
KoBALT 19% #8 of 12 Official leaderboard
HELMET 20.9% #11 of 16 Official leaderboard
RULER 85.4% #29 of 42 Lab model card
Chinese SimpleQA 27.9% #19 of 27 Official leaderboard
SuperGPQA 28.8% #44 of 59 Official leaderboard
MGSM 66.1% #45 of 57 Lab model card
LongBench v2 35.6% #27 of 33 Official leaderboard
FaithJudge 38.5% #31 of 37 Official leaderboard
Arena-Hard-Auto 52% #33 of 39 Lab model card
WritingBench 49.5 #39 of 45 Official leaderboard
Okapi MMLU 67% #7 of 8 Lab model card
MMLU-Pro 56.3% #149 of 167 Lab model card
BuzzBench 31.4 #18 of 20 Official leaderboard
IFEval 71.2% #123 of 135 Lab model card
HELM Arabic 66.0 #26 of 28 Official leaderboard
MMLU-Redux 75.4% #42 of 45 Lab model card
KMMLU 46.6% #6 of 6 Lab model card
LongBench-Chat 7.42 #6 of 6 Lab model card
BLEnD 23.7% #5 of 5 Lab model card
AMMLU 59.8% #4 of 4 Lab model card
IndoMMLU 56.4% #4 of 4 Lab model card
JMMLU 61.9% #4 of 4 Lab model card
Multilingual IFEval 74.9% #4 of 4 Lab model card
TurkishMMLU 54.3% #4 of 4 Lab model card

Math and science

BenchmarkScoreRankSource
PHYSICS 20.4% #7 of 20 Official leaderboard
MATH 75.5% #37 of 99 Lab model card
GSM8K 91.6% #36 of 92 Lab model card
IneqMath 3.00% #19 of 34 Official leaderboard
KOR-Bench 38.6% #22 of 33 Official leaderboard
U-MATH 43.3% #20 of 30 Official leaderboard
ZebraLogic 12% #31 of 46 Official leaderboard
GPQA Diamond 35.5% #196 of 229 Epoch AI run
GPQA Main 34.1% #52 of 60 Official leaderboard
OTIS Mock AIME 2024-2025 2.50% #194 of 212 Epoch AI run
DTBench 47.7% #173 of 180 Official leaderboard

Coding

BenchmarkScoreRankSource
MBPP 79.2% #26 of 71 Lab model card
HumanEval 84.8% #42 of 101 Lab model card
BigCodeBench 41.9% #45 of 69 Official leaderboard
MultiPL-E 70.4% #13 of 18 Lab model card
BigCodeBench-Hard 14.2% #72 of 96 Official leaderboard
LiveCodeBench 28.7% #45 of 54 Lab model card

Agents and tools

BenchmarkScoreRankSource
AceBench 54.8% #11 of 16 Official leaderboard
When2Call 32% #7 of 10 Official leaderboard
ComplexFuncBench 5.00% #8 of 11 Official leaderboard
LLM-WikiRace 0% #16 of 21 Official leaderboard
C3-Bench 4.30% #24 of 30 Official leaderboard
WebWalkerQA 19.9% #4 of 5 Official leaderboard
BALROG 7.80% #31 of 33 Official leaderboard

Professional work

BenchmarkScoreRankSource
MedQA 60% #51 of 71 Official leaderboard
LegalBench 69.6% #117 of 138 Official leaderboard
LEXam 16.7% #28 of 30 Official leaderboard

Vision, audio and robotics

BenchmarkScoreRankSource
Ego3D-Bench 9.50 #9 of 11 Official leaderboard

Safety and security

BenchmarkScoreRankSource
AraTrust 80.7% #13 of 39 Official leaderboard
Anthropic Red Team 98.5% #48 of 75 Official leaderboard
HELM Safety 89.9 #49 of 75 Official leaderboard
MolecularIQ 5.07% #13 of 17 Official leaderboard
SimpleSafetyTests 96% #62 of 75 Official leaderboard
AIR-Bench 2024 47% #68 of 82 Official leaderboard
BaxBench 2.22% #31 of 34 Official leaderboard
AutoBaxBench 2.41% #11 of 11 Official leaderboard

Other

BenchmarkScoreRankSource
Open LLM Leaderboard v2 26.0 #39 of 99 Official leaderboard
HELM Lite 48.8 #41 of 71 Official leaderboard
HELM Capabilities 52.9 #52 of 60 Official leaderboard
LMCA 6.42% #139 of 147 Official leaderboard
ACCoRD 27.3 #144 of 152 Official leaderboard
Conceptual Reasoning Index 11.9 #145 of 151 Official leaderboard
LiveBench 35.9% #26 of 27 Lab model card

Its lab's numbers put it at 119.2, number 349 of 404

The Self-Reported Index fits one score to the 14 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 116.9 to 121.6. How we compute it.

Its lab's numbers next to 3 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
MMLU-Pro 56.3% MMLU-Pro, Qwen2.5-7B-Instruct (Table 8) 45% Leaderboard +11 pts
IFEval 71.2% IFEval, Qwen2.5-7B-Instruct (Table 8) 74.1% Leaderboard, strict, share of instructions followed −2.9 pts
GPQA Diamond 36.4% GPQA, variant not stated, Qwen2.5-7B-Instruct (Table 8) 35.5% Epoch AI run +0.9 pts

Its system card

Every result in the longer documents the lab published with it. 27 of them aren't in the launch post.

Qwen2.5 Technical Report (arXiv 2412.15115) Dec 19, 2024

Capabilities