Qwen-14B ranks first on 2 of the 22 benchmarks with five or more models scored.
Qwen Sep 2023 Open weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MT-Bench-101 | 7.82 | #1 of 6 | Official leaderboard | |
| ARC-Challenge | 84.4% | #14 of 75 | Claim in Qwen Technical Report, via Epoch AI | |
| BoolQ | 86.2% | #14 of 61 | Claim in Qwen Technical Report, via Epoch AI | |
| CriticEval | 5.35 | #6 of 12 | Official leaderboard | |
| Qwen Chinese human evaluation | 30.7% | #1 of 2 | Lab model card | |
| PIQA | 79.9% | #37 of 58 | Claim in Qwen Technical Report, via Epoch AI | |
| EQ-Bench v2 | 63.5 | #61 of 85 | Official leaderboard | |
| AlpacaEval 2.0 | 12.4% | #37 of 47 | Official leaderboard | |
| LAMBADA | 71.1% | #21 of 26 | Claim in Qwen Technical Report, via Epoch AI | |
| MMLU | 66.5% | #108 of 131 | Lab model card | |
| BIG-Bench Hard | 58.7% | #61 of 73 | Lab model card | |
| C-Eval | 71.7% | #40 of 46 | Lab model card | |
| CMMLU | 71% | #15 of 17 | Lab model card | |
| LMArena (Elo) | 1,140 | #312 of 335 | Official leaderboard | |
| VCSUM | 17.3 | #1 of 1 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SuperCLUE-Math6 | 49.0 | #2 of 4 | Official leaderboard | |
| Math401 | 70.1% | #1 of 2 | Lab model card | |
| Math23K | 67% | #2 of 3 | Lab model card | |
| MathBench | 39.5% | #15 of 21 | Official leaderboard | |
| GSM8K | 60.1% | #80 of 92 | Lab model card | |
| MATH | 24.8% | #90 of 99 | Lab model card | |
| MATH-500 | 18.4% | #69 of 70 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEvalPack | 33.7% | #2 of 3 | Lab model card | |
| HumanEval | 43.9% | #88 of 101 | Lab model card | |
| MBPP | 46.4% | #63 of 71 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| T-Eval | 66.3% | #3 of 12 | Official leaderboard | |
| Chinese Tool-Use Benchmark | 98% | #1 of 4 | Lab model card | |
| Code Interpreter Benchmark | 81.7% | #1 of 3 | Lab model card | |
| HuggingFace Agent Benchmark | 93.5% | #1 of 3 | Lab model card | |
| Code Interpreter Benchmark General | 65.5% | #1 of 2 | Official leaderboard | |
| Code Interpreter Benchmark Math | 58.4% | #2 of 2 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CFLUE Knowledge | 53.8% | #1 of 8 | Official leaderboard | |
| CFLUE Application | 44.2 | #1 of 4 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SALAD-Bench | 95.3% | #4 of 13 | Official leaderboard |
The Self-Reported Index fits one score to the 9 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 97.4 to 103.5. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap |
|---|---|---|---|
| Code Interpreter Benchmark Math | 58.4% Code Interpreter benchmark, Accuracy of Code Execution Results (Math) | 58.4% Leaderboard, version 20231206 | Same |
Qwen's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 6 of them aren't in the launch post.