DeepSeek-V2 model card May 7, 2024
- MMLU 77.8
- BBH 79.7
- C-Eval 78
- CMMLU 81.6
- HumanEval 81.1
- MBPP 72
- LiveCodeBench (0901-0401) 32.5
- GSM8K 92.2
- Math 53.9
- Needle In A Haystack (NIAH)
- AlpacaEval 2.0
- MTBench
- Alignbench (总分) 7.91
DeepSeek-V2 (MoE-236B, May 2024)
DeepSeek-V2 (MoE-236B, May 2024) ranks first on Natural Questions.
31
Benchmarks with a score
1
Ranked first, out of five or more models
2
In the top three
DeepSeek May 2024 DeepSeek-V2 (MoE-236B, May 2024)
Open weights Flagship
120.9
#182 of 256
Epoch Capabilities Index
Self-Reported Index 120.0 #346 of 404
- MathBench 59.9% #4 /21
- Natural Questions 53.4% #1 /20
- IFEval 63.8% #128 /135
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Natural Questions | 53.4% | #1 of 20 | Lab model card | |
| WinoGrande | 86.3% | #5 of 76 | Claim in DeepSeek-V3 Technical Report, via Epoch AI | |
| HellaSwag | 87.1% | #5 of 63 | Claim in DeepSeek-V3 Technical Report, via Epoch AI | |
| ARC-Easy | 98.1% | #2 of 23 | Lab model card | |
| PIQA | 83.9% | #8 of 58 | Claim in DeepSeek-V3 Technical Report, via Epoch AI | |
| TriviaQA | 86.7% | #7 of 40 | Lab model card | |
| WildBench | 50.0 | #9 of 47 | Official leaderboard | |
| RepoQA | 83.4% | #6 of 22 | Official leaderboard | |
| ARC-Challenge | 92.3% | #14 of 49 | Lab model card | |
| MT-Bench | 8.97 | #7 of 23 | Lab model card | |
| MixEval | 83.7% | #12 of 34 | Official leaderboard | |
| AGIEval | 61.4% | #20 of 40 | Lab model card | |
| CLUEWSC | 89.9% | #4 of 8 | Lab model card | |
| BIG-Bench Hard | 79.7% | #37 of 73 | Lab model card | |
| MMLU-Pro | 54.8% | #127 of 212 | Official leaderboard | |
| MMLU | 77.8% | #86 of 131 | Lab model card | |
| AlignBench | 7.91 | #11 of 15 | Lab model card | |
| CMMLU | 81.6% | #13 of 17 | Lab model card | |
| C-Eval | 78% | #36 of 46 | Lab model card | |
| AlpacaEval 2.0 | 38.9% | #14 of 16 | Lab model card | |
| IFEval | 63.8% | #128 of 135 | Lab model card |
Math and science
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 81.1% | #53 of 101 | Lab model card | |
| CRUXEval | 61.5% | #5 of 9 | Lab model card | |
| MBPP | 72% | #42 of 71 | Lab model card | |
| LiveCodeBench | 32.5% | #41 of 54 | Lab model card |
Its lab's numbers put it at 120.0, in line with its ECI
The Self-Reported Index fits one score to the 18 capability results DeepSeek reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 117.4 to 122.7. How we compute it.
- ARC-Easy 98.1
- ARC-Challenge 92.3
- GSM8K 92.2
- CLUEWSC 89.9
- TriviaQA 86.7
- CMMLU 81.6
- HumanEval 81.1
- BIG-Bench Hard 79.7
- C-Eval 78
- MMLU 77.8
- MBPP 72
- IFEval 63.8
- CRUXEval 61.5
- AGIEval 61.4
- MATH 53.9
- Natural Questions 53.4
- AlpacaEval 2.0 38.9
- LiveCodeBench 32.5
DeepSeek's models over time
DeepSeek's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 10 of them aren't in the launch post.
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv 2405.04434) May 7, 2024
Capabilities
- TriviaQA 86.7
- Natural Questions 53.4
- MMLU 77.8
- ARC-Easy 98.1
- ARC-Challenge 92.3
- BIG-Bench Hard 79.7
- AGIEval 61.4
- IFEval 63.8
- HumanEval 81.1
- MBPP 72
- CRUXEval 61.5
- LiveCodeBench 32.5
- GSM8K 92.2
- MATH 53.9
- CMATH 81.9
- CLUEWSC 89.9
- C-Eval 78
- CMMLU 81.6
- MT-Bench 8.97
- AlpacaEval 2.0 38.9
- AlignBench 7.91
- SuperCLUE-Math6 83.3