DeepSeek-V3

DeepSeek-V3 ranks first on 3 of the 76 benchmarks with five or more models scored.

DeepSeek Released December 26th, 2024 Family DeepSeek-V3 The lab's model card Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

DeepSeek Dec 2024

DeepSeek-V3

Open weights Flagship

131.7 #154 of 256

Epoch Capabilities Index

Self-Reported Index 129.5 #278 of 404

LongBench v2 48.7% #26 /35
GPQA Diamond 56.5% #142 /229
FrontierMath 1.72% #50 /55

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
ARC-Easy	98.9%	#1 of 23	Lab model card
DROP	91.6%	#4 of 41	Lab model card
ARC-Challenge	95.3%	#6 of 49	Lab model card
AGIEval	79.6%	#7 of 40	Lab model card
BIG-Bench Hard	87.5%	#13 of 73	Lab model card
MMLU	88.5%	#24 of 131	Lab model card
AlpacaEval 2.0	70%	#3 of 16	Lab model card
WinoGrande	84.9%	#9 of 47	Lab model card
Natural Questions	40%	#4 of 20	Lab model card
HellaSwag	88.9%	#12 of 53	Lab model card
PIQA	84.7%	#9 of 37	Lab model card
TriviaQA	82.9%	#14 of 40	Lab model card
CLUEWSC	90.9%	#3 of 8	Lab model card
MMLU-Pro	81.2%	#69 of 167	Lab model card
Vectara Hallucination Leaderboard (HHEM)	3.90%	#63 of 148	Official leaderboard
Arena-Hard-Auto	85.5%	#17 of 39	Lab model card
SuperGPQA	47.4%	#27 of 59	Official leaderboard
CCPM	92%	#1 of 2	Lab model card
RewardBench	87%	#1 of 2	Lab model card
LMArena (Elo)	1,358	#170 of 335	Official leaderboard
SimpleQA	24.9%	#31 of 60	Lab model card
IFEval	86.1%	#71 of 135	Lab model card
CMMLU	88.8%	#9 of 17	Lab model card
SEAL Precise Instruction Following	82.3	#11 of 18	Official leaderboard
MMLU-Redux	89.1%	#28 of 45	Lab model card
MGSM	79.8%	#36 of 57	Lab model card
MMMLU	79.4%	#45 of 71	Lab model card
C-Eval	86.5%	#30 of 46	Lab model card
C3	78.6%	#4 of 6	Lab model card
WritingBench	59.2	#31 of 45	Official leaderboard
LongBench v2	48.7%	#26 of 35	Lab model card
MARS-Bench	33.2%	#12 of 15	Official leaderboard
Chinese SimpleQA	64.8%	#11 of 13	Lab model card
FRAMES	73.3%	#9 of 9	Lab model card
RACE-H	51.3%	#7 of 7	Lab model card
RACE-M	67.1%	#3 of 3	Lab model card
CMRC	76.3%	#1 of 1	Lab model card

Math and science

Benchmark	Score	Rank	Source
ZebraLogic	42.1%	#6 of 46	Official leaderboard
U-MATH	62.6%	#8 of 30	Official leaderboard
MATH Level 5	64.8%	#39 of 94	Epoch AI run
GSM8K	89.3%	#44 of 92	Lab model card
CMATH	90.7%	#3 of 6	Lab model card
MATH	61.6%	#55 of 99	Lab model card
GPQA Diamond	56.5%	#142 of 229	Epoch AI run
MATH-500	90.2%	#52 of 70	Lab model card
OTIS Mock AIME 2024-2025	15.8%	#159 of 212	Epoch AI run
AIME 2024	59.4%	#70 of 88	Lab model card
PolyMath	20.4%	#32 of 38	Official leaderboard
Misguided Attention	26.9%	#39 of 45	Official leaderboard
FrontierMath	1.72%	#50 of 55	Official leaderboard
SimpleBench	18.9%	#101 of 106	Official leaderboard
CNMO 2024	43.2%	#7 of 7	Lab model card

Coding

Benchmark	Score	Rank	Source
BigCodeBench	56.1%	#2 of 69	Official leaderboard
BigCodeBench-Hard	34.5%	#5 of 96	Official leaderboard
HumanEval+	86.6%	#6 of 46	Official leaderboard
Aider-Edit	79.7%	#1 of 5	Lab model card
HumanEval Multi	82.6%	#1 of 5	Lab model card
MBPP+	73%	#10 of 35	Official leaderboard
SciCode	3.10%	#7 of 18	Official leaderboard
CRUXEval	69.8%	#4 of 9	Lab model card
MBPP	75.4%	#32 of 71	Lab model card
LiveCodeBench	49.2%	#30 of 54	Lab model card
SWE-bench Lite	30.7%	#9 of 15	Official leaderboard
METR Time Horizon	20 min	#19 of 31	Official leaderboard
SEAL Coding	985	#17 of 26	Official leaderboard
Aider Polyglot	49.6%	#21 of 30	Lab model card
Multi-SWE-bench	4.72%	#8 of 11	Official leaderboard
HumanEval	65.2%	#77 of 101	Lab model card
LiveCodeBench v5	31.7%	#17 of 21	Official leaderboard
BigO(Bench)	27.1%	#16 of 18	Official leaderboard
SWE-bench Verified	42%	#141 of 155	Lab model card
LiveCodeBench v6	27.2%	#20 of 21	Official leaderboard

Agents and tools

Benchmark	Score	Rank	Source
AceBench	74.8%	#5 of 16	Official leaderboard
ToolComp	62.5%	#12 of 32	Official leaderboard
Elimination Game	4	#29 of 56	Official leaderboard
LisanBench	372.3	#74 of 133	Official leaderboard
Step Race	1	#46 of 59	Official leaderboard
Factorio Learning Environment	48585.0	#5 of 6	Official leaderboard
LLM Chess	-823	#123 of 123	Official leaderboard

Safety and security

Benchmark	Score		Rank	Source
BaxBench	24.7%		#20 of 34	Official leaderboard

Other

Benchmark	Score		Rank	Source
Pile-test	0.55		#1 of 1	Lab model card

Its lab's numbers put it at 129.5, 2.2 points below its ECI

The Self-Reported Index fits one score to the 36 capability results DeepSeek reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 127.8 to 131.1. How we compute it.

Its lab's numbers next to 5 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark	Lab says	Board says	Gap
HumanEval	65.2% HumanEval (Pass@1)	91.5% Leaderboard	−26 pts
LiveCodeBench	49.2% LiveCodeBench	61.7% Leaderboard, 2023-05-01 to 2024-10-01 (exclusive)	−13 pts
MBPP	75.4% MBPP (Pass@1)	87.6% Leaderboard, EvalPlus subset, original tests	−12 pts
GPQA Diamond	68.4% GPQA	56.5% Epoch AI run	+12 pts
CNMO 2024	43.2% CNMO 2024 (Pass@1)	43.2% Leaderboard	Same