Benchmark Almanac

Qwen2.5-14B-Instruct-1M

Qwen2.5-14B-Instruct-1M ranks first on LongBench-Chat.

Qwen Released January 27th, 2025 Family Qwen2.5-1M The lab's model card Compare with others
21
Benchmarks with a score
1
Ranked first, out of five or more models
2
In the top three
Qwen Jan 2025

Qwen2.5-14B-Instruct-1M

Open weights Small

124.3 #317 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • Open Arabic LLM Leaderboard v1 58.8% #3 /55
  • MMLU-Pro 63.3% #142 /167
  • IFEval 84.3% #87 /135

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
Open Arabic LLM Leaderboard v1 58.8% #3 of 55 Official leaderboard
RULER 95.7% #4 of 42 Lab model card
LongBench-Chat 8.76 #1 of 6 Lab model card
MT-Bench 8.89 #9 of 23 Lab model card
MuSR 41.8% #44 of 98 Official leaderboard
Stick to Your Role! 0.59 #8 of 15 Official leaderboard
IFEval 84.3% #87 of 135 Lab model card
Arena-Hard-Auto 70.2% #27 of 39 Lab model card
MMLU-Pro 63.3% #142 of 167 Lab model card
MMLU-Redux 80.7% #39 of 45 Lab model card
Passkey Retrieval 100% #2 of 2 Lab model card
LV-Eval 49.8% #1 of 1 Lab model card

Math and science

BenchmarkScoreRankSource
GSM8K 94.8% #16 of 92 Lab model card
MATH 79.5% #29 of 99 Lab model card
GPQA Diamond 39.9% #275 of 302 Lab model card

Coding

BenchmarkScoreRankSource
HumanEval 88.4% #23 of 101 Lab model card
MBPP 80.2% #21 of 71 Lab model card
MultiPL-E 77.1% #6 of 18 Lab model card
LiveCodeBench 38.6% #36 of 54 Lab model card

Other

BenchmarkScoreRankSource
Open LLM Leaderboard v2 41.6 #4 of 99 Official leaderboard
LiveBench 44.6% #23 of 27 Lab model card

Its lab's numbers put it at 124.3, number 317 of 404

The Self-Reported Index fits one score to the 12 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 121.8 to 126.9. How we compute it.

Its lab's number next to a leaderboard

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
RULER 95.7% RULER Avg. (4K to 128K) 95.7% Leaderboard Same

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Its system card

Every result in the longer documents the lab published with it. 13 of them aren't in the launch post.