Qwen Technical Report (arXiv 2309.16609) Sep 28, 2023
Capabilities
- MMLU 43.9
- C-Eval 50.7
- GSM8K 27.8
- BIG-Bench Hard 27.1
- HumanEval 14.6
- Chinese Tool-Use Benchmark 92
- Code Interpreter Benchmark 26.8
- HuggingFace Agent Benchmark 85.2
Qwen-1_8B
Qwen-1_8B.
Released November 30th, 2023 Compare with others
14
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Unknown lab Nov 2023
Qwen-1_8B
59.9
#253 of 256
Epoch Capabilities Index
Self-Reported Index 85.1 #403 of 404
- CFLUE Knowledge 38.7% #5 /8
- C-Eval 50.7% #45 /46
- MMLU 43.9% #128 /131
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ARC-Challenge | 53.2% | #42 of 75 | Claim in Qwen Technical Report, via Epoch AI | |
| BoolQ | 68% | #50 of 61 | Claim in Qwen Technical Report, via Epoch AI | |
| PIQA | 73.3% | #54 of 58 | Claim in Qwen Technical Report, via Epoch AI | |
| EQ-Bench v2 | 30.0 | #81 of 85 | Official leaderboard | |
| LAMBADA | 58.4% | #25 of 26 | Claim in Qwen Technical Report, via Epoch AI | |
| MMLU | 43.9% | #128 of 131 | Lab model card | |
| C-Eval | 50.7% | #45 of 46 | Lab model card | |
| BIG-Bench Hard | 27.1% | #72 of 73 | Lab model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 27.8% | #90 of 92 | Lab model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 14.6% | #100 of 101 | Lab model card |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Chinese Tool-Use Benchmark | 92% | #4 of 4 | Lab model card | |
| Code Interpreter Benchmark | 26.8% | #3 of 3 | Lab model card | |
| HuggingFace Agent Benchmark | 85.2% | #3 of 3 | Lab model card |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CFLUE Knowledge | 38.7% | #5 of 8 | Official leaderboard |
Its lab's numbers put it at 85.1, 25.2 points above its ECI
The Self-Reported Index fits one score to the 5 capability results the lab reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 80.7 to 89.5. How we compute it.
- C-Eval 50.7
- MMLU 43.9
- GSM8K 27.8
- BIG-Bench Hard 27.1
- HumanEval 14.6
The lab's models over time
's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its system card
Every result in the longer documents the lab published with it. 8 of them aren't in the launch post.