Benchmark Almanac

Qwen-Turbo

Qwen-Turbo places in the top three on Needle In A Haystack.

Qwen Released November 1st, 2024 Compare with others
19
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Qwen Nov 2024

Qwen-Turbo

Closed weights

124.2 #318 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • GPQA Diamond 41.8% #181 /229
  • MMLU-Pro 64.5% #138 /167
  • IFEval 76.3% #115 /135

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
Needle In A Haystack 100% #2 of 9 Lab model card
RULER 93.1% #12 of 42 Lab model card
MT-Bench 8.81 #11 of 23 Lab model card
LongBench-Chat 8.34 #4 of 6 Lab model card
Arena-Hard-Auto 67.1% #29 of 39 Lab model card
MMLU-Pro 64.5% #138 of 167 Lab model card
MMLU-Redux 81.7% #38 of 45 Lab model card
IFEval 76.3% #115 of 135 Lab model card

Math and science

BenchmarkScoreRankSource
GSM8K 93.8% #21 of 92 Lab model card
MATH 81.1% #24 of 99 Lab model card
MATH Level 5 56.2% #45 of 94 Epoch AI run
GPQA Diamond 41.8% #181 of 229 Epoch AI run
OTIS Mock AIME 2024-2025 6.11% #179 of 212 Epoch AI run

Coding

BenchmarkScoreRankSource
MBPP 82.8% #15 of 71 Lab model card
HumanEval 86.6% #35 of 101 Lab model card
MultiPL-E 73.7% #11 of 18 Lab model card
LiveCodeBench 37.8% #38 of 54 Lab model card

Agents and tools

BenchmarkScoreRankSource
SnakeBench 26 #62 of 188 Official leaderboard

Other

BenchmarkScoreRankSource
LiveBench 42.3% #25 of 27 Lab model card

Its lab's numbers put it at 124.2, number 318 of 404

The Self-Reported Index fits one score to the 13 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 121.9 to 126.6. How we compute it.

Its lab's number next to a leaderboard

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
GPQA Diamond 42.3% GPQA, variant not stated, Qwen2.5-Turbo (Table 7) 41.8% Epoch AI run +0.5 pts

Its system card

Every result in the longer documents the lab published with it. 19 of them aren't in the launch post.