DeepSeek LLM 67B places in the top three on 3 benchmarks.
DeepSeek Nov 2023 Open weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MuSR | 50.6% | #2 of 98 | Official leaderboard | |
| Natural Questions | 47% | #3 of 20 | Lab model card | |
| CriticEval | 5.84 | #3 of 12 | Official leaderboard | |
| Chinese SimpleQA | 47% | #10 of 27 | Official leaderboard | |
| OpenBookQA | 63.2% | #9 of 24 | Lab model card | |
| TriviaQA | 81.5% | #16 of 40 | Lab model card | |
| PIQA | 82.6% | #20 of 37 | Lab model card | |
| ARC-Easy | 81.6% | #13 of 23 | Lab model card | |
| HellaSwag | 84% | #31 of 53 | Lab model card | |
| WinoGrande | 76% | #28 of 47 | Lab model card | |
| BIG-Bench Hard | 71.7% | #44 of 73 | Lab model card | |
| AlpacaEval 2.0 | 17.8% | #31 of 47 | Official leaderboard | |
| RACE-M | 70.9% | #2 of 3 | Lab model card | |
| ARC-Challenge | 64.1% | #35 of 49 | Lab model card | |
| MGSM | 74% | #41 of 57 | Lab model card | |
| AGIEval | 46.4% | #29 of 40 | Lab model card | |
| MMLU | 71.1% | #99 of 131 | Lab model card | |
| MT-Bench | 8.35 | #18 of 23 | Lab model card | |
| C3 | 77% | #5 of 6 | Lab model card | |
| DROP | 71.9% | #35 of 41 | Lab model card | |
| RACE-H | 56% | #6 of 7 | Lab model card | |
| LMArena (Elo) | 1,185 | #290 of 335 | Official leaderboard | |
| C-Eval | 65.2% | #43 of 46 | Lab model card | |
| CMMLU | 67.8% | #16 of 17 | Lab model card | |
| IFEval | 55.5% | #132 of 135 | Lab model card | |
| AlignBench | 6.43 | #15 of 15 | Lab model card | |
| CLUEWSC | 60% | #8 of 8 | Lab model card | |
| CCPM | 84.9% | #2 of 2 | Lab model card | |
| CHID | 72.6% | #1 of 1 | Lab model card | |
| ChineseQA | 85.1% | #1 of 1 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 84.1% | #59 of 92 | Lab model card | |
| MathBench | 40.3% | #14 of 21 | Official leaderboard | |
| MATH | 32.6% | #85 of 99 | Lab model card | |
| MATH Level 5 | 6.39% | #90 of 94 | Epoch AI run | |
| GPQA Diamond | 24.6% | #221 of 229 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 0.83% | #205 of 212 | Epoch AI run | |
| CMATH | 80.3% | #6 of 6 | Lab model card | |
| Hungarian National High-School Exam | 58% | #2 of 2 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 73.8% | #67 of 101 | Lab model card | |
| MBPP | 61.4% | #55 of 71 | Lab model card | |
| HumanEval Multi | 53.3% | #5 of 5 | Lab model card | |
| LeetCode Weekly Contest | 17.5% | #1 of 1 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GTA | 9.51% | #10 of 15 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 62.8% | #45 of 71 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Anthropic Red Team | 99.4% | #22 of 75 | Official leaderboard | |
| Do-Not-Answer | 97.8% | #1 of 2 | Lab model card | |
| HELM Safety | 87.3 | #59 of 75 | Official leaderboard | |
| SimpleSafetyTests | 96.8% | #59 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 50.5% | #66 of 82 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Open LLM Leaderboard v2 | 27.3 | #32 of 99 | Official leaderboard | |
| HELM Lite | 48.8 | #40 of 71 | Official leaderboard |
The Self-Reported Index fits one score to the 21 capability results DeepSeek reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 108.5 to 113.2. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 23 of them aren't in the launch post.