DeepSeek-V2.5 places in the top three on 2 benchmarks.
DeepSeek Sep 2024 Sep 2024 snapshot Open weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Chinese SimpleQA | 55.7% | #4 of 27 | Official leaderboard | |
| EQ-Bench v2 | 82.0 | #17 of 85 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 2.40% | #30 of 148 | Official leaderboard | |
| MT-Bench | 9.02 | #6 of 23 | Lab model card | |
| MMLU-Pro | 65.8% | #98 of 212 | Official leaderboard | |
| Arena-Hard-Auto | 76.2% | #21 of 39 | Lab model card | |
| AlpacaEval 2.0 | 50.5% | #10 of 16 | Lab model card | |
| LMArena (Elo) | 1,307 | #222 of 335 | Official leaderboard | |
| AlignBench | 8.04 | #10 of 15 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 89% | #17 of 101 | Lab model card | |
| HumanEval+ | 83.5% | #8 of 46 | Official leaderboard | |
| MBPP+ | 74.1% | #7 of 35 | Official leaderboard | |
| MBPP | 87.6% | #9 of 35 | Official leaderboard | |
| MHPP | 40% | #9 of 23 | Official leaderboard | |
| BigCodeBench-Hard | 26.1% | #39 of 96 | Official leaderboard | |
| Aider-Edit | 72.2% | #3 of 5 | Lab model card | |
| HumanEval Multi | 73.8% | #3 of 5 | Lab model card | |
| LiveCodeBench | 41.8% | #34 of 54 | Lab model card | |
| SWE-bench Verified | 16.8% | #155 of 155 | Lab model card | |
| DS-Arena-Code | 63.1% | #1 of 1 | Lab model card | |
| DS-FIM-Eval | 78.3% | #1 of 1 | Lab model card |
The Self-Reported Index fits one score to the 5 capability results DeepSeek reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 119.1 to 127.1. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap |
|---|---|---|---|
| HumanEval | 89% HumanEval python | 90.2% Leaderboard | −1.2 pts |
| Aider-Edit | 72.2% Aider | 72.2% Leaderboard, aider, diff edits | Same |
| DS-Arena-Code | 63.1% DS-Arena-Code (in-house) | 63.1% Leaderboard | Same |
| DS-FIM-Eval | 78.3% DS-FIM-Eval (in-house) | 78.3% Leaderboard | Same |
The benchmarks the lab chose to headline, with the scores it gave.
