Qwen2-72B ranks first on 2 of the 68 benchmarks with five or more models scored.
Qwen Jun 2024 Open weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AlGhafa | 78.6% | #1 of 39 | Official leaderboard | |
| Open Arabic LLM Leaderboard v1 | 63.6% | #2 of 55 | Official leaderboard | |
| ArbMMLU-HT | 68.1% | #2 of 39 | Official leaderboard | |
| Open Arabic LLM Leaderboard v2 | 70.1% | #3 of 39 | Official leaderboard | |
| MuSR | 47% | #8 of 98 | Official leaderboard | |
| MadinahQA | 61.4% | #4 of 39 | Official leaderboard | |
| CFBench | 70.5% | #2 of 19 | Official leaderboard | |
| ArabicMMLU | 66.5% | #6 of 39 | Official leaderboard | |
| AlpacaEval 2.0 | 38.1% | #8 of 47 | Official leaderboard | |
| WinoGrande | 85.1% | #8 of 47 | Lab model card | |
| ThaiExam | 63.8% | #5 of 24 | Official leaderboard | |
| EQ-Bench v2 | 81.3 | #18 of 85 | Official leaderboard | |
| MT-Bench | 9.12 | #5 of 23 | Lab model card | |
| CMMLU | 90.1% | #5 of 17 | Lab model card | |
| HellaSwag | 87.6% | #16 of 53 | Lab model card | |
| CulturalBench | 48.7% | #5 of 15 | Official leaderboard | |
| WildBench | 46.4 | #18 of 47 | Official leaderboard | |
| MMLU-CF | 63.7% | #15 of 37 | Official leaderboard | |
| BIG-Bench Hard | 82.4% | #31 of 73 | Lab model card | |
| ToRR | 45.4 | #6 of 14 | Official leaderboard | |
| MGSM | 86.6% | #25 of 57 | Lab model card | |
| ALRAGE | 73.4% | #18 of 39 | Official leaderboard | |
| MMLU | 82.3% | #62 of 131 | Lab model card | |
| MixEval | 86.7% | #1 of 2 | Lab model card | |
| AlignBench | 8.27 | #8 of 15 | Lab model card | |
| Vectara Hallucination Leaderboard (HHEM) | 4.70% | #87 of 148 | Official leaderboard | |
| TruthfulQA | 54.8% | #18 of 30 | Lab model card | |
| Flores | 37.8 | #5 of 8 | Lab model card | |
| ARC-Challenge | 68.9% | #32 of 49 | Lab model card | |
| RULER | 85.9% | #10 of 15 | Official leaderboard | |
| Multilingual Exams | 76.6% | #2 of 3 | Lab model card | |
| Multilingual Understanding | 80.7% | #2 of 3 | Lab model card | |
| MMMLU | 78% | #49 of 71 | Lab model card | |
| C-Eval | 83.8% | #34 of 46 | Lab model card | |
| KINA | 24.9% | #28 of 37 | Official leaderboard | |
| LMArena (Elo) | 1,262 | #257 of 335 | Official leaderboard | |
| Multilingual human evaluation | 3.93 | #4 of 5 | Official leaderboard | |
| IFEval | 77.6% | #112 of 135 | Lab model card | |
| MMLU-Pro | 64.4% | #139 of 167 | Lab model card | |
| Arena-Hard-Auto | 48.1% | #34 of 39 | Lab model card | |
| Qwen in-house Chinese automatic evaluation | 69.6% | #1 of 1 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MathBench | 67% | #2 of 21 | Official leaderboard | |
| OlympicArena | 34.1% | #5 of 25 | Official leaderboard | |
| GSM8K | 93.2% | #23 of 92 | Lab model card | |
| ZebraLogic | 21.4% | #19 of 46 | Official leaderboard | |
| KOR-Bench | 47% | #15 of 33 | Official leaderboard | |
| TheoremQA | 44.4% | #1 of 2 | Lab model card | |
| MATH | 69% | #50 of 99 | Lab model card | |
| MATH Level 5 | 39.1% | #64 of 94 | Epoch AI run | |
| GPQA Diamond | 40.8% | #183 of 229 | Epoch AI run |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 80.2% | #19 of 71 | Lab model card | |
| HumanEval | 86% | #37 of 101 | Lab model card | |
| BigCodeBench | 46.3% | #31 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 20.6% | #56 of 96 | Official leaderboard | |
| EvalPlus | 79% | #4 of 6 | Lab model card | |
| LiveCodeBench | 35.7% | #39 of 54 | Lab model card | |
| SciCode | 1.50% | #13 of 18 | Official leaderboard | |
| Aider-Edit | 55.6% | #26 of 34 | Official leaderboard | |
| MultiPL-E | 69.2% | #14 of 18 | Lab model card | |
| METR Time Horizon | 2.5 min | #28 of 31 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SuperCLUE-Agent | 74.0 | #2 of 7 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 74.5% | #20 of 71 | Official leaderboard | |
| TheAgentCompany | 1.10% | #7 of 7 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SGP-Bench | 53.7% | #10 of 22 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AraTrust | 89.2% | #1 of 39 | Official leaderboard | |
| IHEval | 47.8% | #2 of 10 | Official leaderboard | |
| HELM Safety | 93.3 | #29 of 75 | Official leaderboard | |
| Anthropic Red Team | 99.1% | #35 of 75 | Official leaderboard | |
| SimpleSafetyTests | 98.5% | #43 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 62.1% | #50 of 82 | Official leaderboard | |
| nocot-bench (NCRI) | 61.0 | #136 of 185 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Open LLM Leaderboard v2 | 43.6 | #3 of 99 | Official leaderboard | |
| HELM Lite | 77.0 | #17 of 71 | Official leaderboard | |
| Qwen in-house English automatic evaluation | 72.9% | #1 of 1 | Lab model card |
The Self-Reported Index fits one score to the 20 capability results Qwen reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 118.6 to 123.4. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap |
|---|---|---|---|
| GPQA Diamond | 42.4% GPQA | 40.8% Epoch AI run | +1.6 pts |
| Arena-Hard-Auto | 48.1% Arena-Hard | 46.9% Leaderboard | +1.2 pts |
| CMMLU | 90.1% CMMLU | 89.7% Leaderboard, Five-shot | +0.4 pts |
| MMLU-Pro | 64.4% MMLU-Pro | 64.4% Leaderboard | Same |
| Multilingual human evaluation | 3.93 Human multilingual evaluation (Average, 1-5) | 3.93 Leaderboard | Same |
Qwen's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 7 of them aren't in the launch post.