Claude 3 Sonnet

Claude 3 Sonnet ranks first on Honesty human feedback win rate (Anthropic).

Anthropic Released February 29th, 2024 Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Anthropic Feb 2024

Claude 3 Sonnet

Closed weights

119.5 #186 of 256

Epoch Capabilities Index

Self-Reported Index 118.5 #354 of 404

ZeroBench 1.00% #69 /76
GPQA Diamond 40.6% #184 /229
IHEval 30.7% #5 /10

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
Honesty human feedback win rate (Anthropic)	69%	#1 of 6	Lab model card
HellaSwag	89%	#10 of 53	Lab model card
ARC-Challenge	93.2%	#11 of 49	Lab model card
RepoQA	87.4%	#5 of 22	Official leaderboard
TurkishMMLU	67.3%	#2 of 8	Official leaderboard
AlpacaEval 2.0	34.9%	#12 of 47	Official leaderboard
EQ-Bench v2	80.5	#23 of 85	Official leaderboard
RACE-H	88.8%	#2 of 7	Lab model card
ThaiExam	57.9%	#9 of 24	Official leaderboard
BIG-Bench Hard	82.9%	#30 of 73	Lab model card
WildBench	45.2	#20 of 47	Official leaderboard
QuALITY	85.9%	#3 of 7	Lab model card
MixEval	81.7%	#15 of 34	Official leaderboard
Arena-Hard-Auto	46.8%	#19 of 38	Official leaderboard
MGSM	83.5%	#29 of 57	Lab model card
MMLU	81.5%	#68 of 131	Lab model card
MMLU-Pro	56.8%	#119 of 212	Official leaderboard
WinoGrande	75.1%	#31 of 47	Lab model card
Summary of a Haystack	41.2%	#2 of 3	Official leaderboard
RewardBench	74.6%	#34 of 48	Official leaderboard
LMArena (Elo)	1,281	#246 of 335	Official leaderboard
MMBench-CN v1.1	63.5%	#132 of 178	Official leaderboard
Okapi MMLU	69%	#6 of 8	Lab model card
LSAT	158.3	#3 of 4	Lab model card
AlignBench	6.71	#6 of 7	Official leaderboard
Vectara Hallucination Leaderboard (HHEM)	16.3%	#142 of 148	Official leaderboard
Needle In A Haystack	95.4%	#9 of 9	Lab model card

Math and science

Benchmark	Score	Rank	Source
GSM8K	92.3%	#30 of 92	Lab model card
OlympicArena	25.5%	#10 of 25	Official leaderboard
ZebraLogic	18.7%	#22 of 46	Official leaderboard
GSM1k	93.3%	#8 of 15	Official leaderboard
AMC 8 2023	54.0	#2 of 3	Lab model card
GPQA Extended	38%	#2 of 3	Lab model card
GPQA Main	38.5%	#5 of 7	Lab model card
GPQA Diamond	40.6%	#184 of 229	Epoch AI run
MATH	40.5%	#81 of 99	Lab model card
MATH Level 5	18.2%	#78 of 94	Epoch AI run
DTBench	53.6%	#155 of 180	Official leaderboard
OTIS Mock AIME 2024-2025	2.50%	#192 of 212	Epoch AI run
AMC 10 2023	24.0	#3 of 3	Lab model card
AMC 12 2023	27.0	#3 of 3	Lab model card

Coding

Benchmark	Score	Rank	Source
MBPP	79.4%	#24 of 71	Lab model card
MBPP+	69.3%	#14 of 35	Official leaderboard
NaturalCodeBench	38.9%	#2 of 5	Official leaderboard
BigCodeBench	48.3%	#28 of 69	Official leaderboard
HumanEval+	64%	#23 of 46	Official leaderboard
SciCode	1.50%	#9 of 18	Official leaderboard
BigCodeBench-Hard	23.4%	#52 of 96	Official leaderboard
APPS	55.9%	#2 of 3	Lab model card
HumanEval	73%	#68 of 101	Lab model card
Aider-Edit	54.9%	#28 of 34	Official leaderboard
SEAL Coding	879	#22 of 26	Official leaderboard
WeirdML	10.2%	#110 of 112	Official leaderboard
USACO	2.61%	#1 of 1	Official leaderboard

Agents and tools

Benchmark	Score	Rank	Source
ToolComp	54.2%	#19 of 32	Official leaderboard
LLM Chess	-322	#103 of 123	Official leaderboard
CRMArena	17.3%	#8 of 9	Official leaderboard
LLM Colosseum	1,315	#3 of 3	Official leaderboard

Professional work

Benchmark	Score	Rank	Source
PubMedQA	78.3%	#1 of 3	Lab model card
MedQA	68.4%	#30 of 71	Official leaderboard
MBE (Multistate Bar Examination)	71%	#3 of 4	Lab model card

Vision, audio and robotics

Benchmark	Score	Rank	Source
CharXiv (DQ)	73.7%	#20 of 67	Official leaderboard
WildVision Arena	1,044	#3 of 10	Official leaderboard
WildVision	50.0	#5 of 14	Official leaderboard
SGP-Bench	56.5%	#8 of 22	Official leaderboard
AI2D	88.7%	#25 of 52	Lab model card
Image2Struct	49.1	#9 of 18	Official leaderboard
MM-Vet	51.7%	#83 of 164	Official leaderboard
CharXiv	32.2%	#39 of 77	Official leaderboard
MMMU	53.1%	#76 of 136	Official leaderboard
MTVQA-Test	21.1%	#11 of 18	Official leaderboard
CoreCognition	52.5%	#77 of 119	Official leaderboard
MMStar	44.2%	#129 of 173	Official leaderboard
CCBench (MMBench Chinese Culture)	27.8%	#119 of 152	Official leaderboard
MMBench	67.8%	#122 of 155	Official leaderboard
VHELM	29.8	#33 of 41	Official leaderboard
DocVQA	89.5%	#29 of 36	Lab model card
ChartQA	81.1%	#30 of 37	Lab model card
MMBench-EN v1.1	64.2%	#149 of 178	Official leaderboard
ZeroBench	1.00%	#69 of 76	Official leaderboard
Vision Arena	1,017	#127 of 136	Official leaderboard
MathVista	47.9%	#68 of 70	Lab model card
SEED-Bench-2-Plus	27.1%	#134 of 136	Official leaderboard

Safety and security

Benchmark	Score	Rank	Source
SimpleSafetyTests	100%	#2 of 75	Official leaderboard
Anthropic Red Team	99.8%	#7 of 75	Official leaderboard
AIR-Bench 2024	84.7%	#14 of 82	Official leaderboard
SORRY-Bench	10.7%	#5 of 23	Official leaderboard
Harmlessness human feedback win rate (Anthropic)	87%	#1 of 4	Lab model card
HELM Safety	94.3	#27 of 75	Official leaderboard
Situational Awareness Dataset (SAD)	50%	#7 of 17	Official leaderboard
IHEval	30.7%	#5 of 10	Official leaderboard
Multimodal policy red-teaming (Anthropic)	99.2%	#1 of 2	Lab model card
AgentDojo	26.7%	#15 of 20	Official leaderboard
PrivacyLens	37.3%	#3 of 4	Official leaderboard
AgentHarm	20.7%	#7 of 8	Official leaderboard
OR-Bench	94.5%	#25 of 28	Official leaderboard

Other

Benchmark	Score		Rank	Source
OpenVLM Leaderboard	53.5		#114 of 173	Official leaderboard
HELM Lite	37.6		#52 of 71	Official leaderboard

Its lab's numbers put it at 118.5, in line with its ECI

The Self-Reported Index fits one score to the 18 capability results Anthropic reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 115.9 to 121.0. How we compute it.

Needle In A Haystack 95.4
ARC-Challenge 93.2
GSM8K 92.3
DocVQA 89.5
HellaSwag 89
AI2D 88.7
MGSM 83.5
BIG-Bench Hard 82.9
MMLU 81.5
ChartQA 81.1
MBPP 79.4
WinoGrande 75.1
HumanEval 73
Okapi MMLU 69
MMMU 53.1
MathVista 47.9
MATH 40.5
GPQA Diamond 40.4

Its lab's numbers next to 5 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark	Lab says	Board says	Gap
MBPP	79.4% MBPP, Pass@1 (Table 1)	83.6% Leaderboard, EvalPlus subset, original tests	−4.2 pts
HumanEval	73% HumanEval, 0-shot (Table 1)	70.7% Leaderboard	+2.3 pts
GPQA Diamond	40.4% GPQA (Diamond), 0-shot CoT, mean of 10 rollouts. 5-shot CoT 42.9, Maj@32 5-shot CoT 46.3 (Table 1, Table 8)	40.6% Epoch AI run	−0.2 pts
MathVista	47.9% MathVista (testmini), 0-shot CoT (Table 3)	47.9% Leaderboard	Same
MMMU	53.1% MMMU (val), Overall, 0-shot. Per-discipline values also given (Table 3)	53.1% Leaderboard	Same

Anthropic's models over time

Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its system card

Every result in the longer documents the lab published with it. 41 of them aren't in the launch post.