Claude 3 Opus

Claude 3 Opus ranks first on 13 of the 127 benchmarks with five or more models scored.

Anthropic Released February 29th, 2024 Family Claude 3 The lab's model card Compare with others

145

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Anthropic Feb 2024

Claude 3 Opus

Closed weights Flagship

125.5 #172 of 256

Epoch Capabilities Index

Self-Reported Index 127.8 #291 of 404

ZeroBench 0% #72 /76
GPQA Diamond 47.2% #170 /229
Cybench 10% #10 /16

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
HellaSwag	95.4%	#2 of 53	Lab model card
RepoQA	90.6%	#1 of 22	Official leaderboard
ARC-Challenge	96.4%	#3 of 49	Lab model card
WinoGrande	88.5%	#3 of 47	Lab model card
ThaiExam	68.6%	#2 of 24	Official leaderboard
MixEval	88.1%	#3 of 34	Official leaderboard
TurkishMMLU	79.1%	#1 of 8	Official leaderboard
AlpacaEval 2.0	40.5%	#6 of 47	Official leaderboard
QuALITY	90.5%	#1 of 7	Lab model card
RACE-H	92.9%	#1 of 7	Lab model card
MGSM	90.7%	#9 of 57	Lab model card
EQ-Bench v2	82.2	#14 of 85	Official leaderboard
WildBench	52.1	#8 of 47	Official leaderboard
Multilingual human evaluation	4.15	#1 of 5	Official leaderboard
MMLU	88.2%	#27 of 131	Lab model card
BIG-Bench Hard	86.8%	#16 of 73	Lab model card
Okapi MMLU	79.1%	#2 of 8	Lab model card
100Q-Hard (Anthropic)	46.5%	#2 of 7	Lab model card
LLM-AggreFact	74.8%	#3 of 10	Official leaderboard
GRE Verbal	166.0	#2 of 6	Lab model card
Summary of a Haystack	42.5%	#1 of 3	Official leaderboard
MMLU-Pro	68.5%	#90 of 212	Official leaderboard
AlignBench	7.53	#3 of 7	Official leaderboard
Needle In A Haystack	99.4%	#4 of 9	Lab model card
Arena-Hard-Auto	60.4%	#17 of 38	Official leaderboard
LSAT	161.0	#2 of 4	Lab model card
DROP	83.1%	#21 of 41	Lab model card
RewardBench	80.1%	#29 of 48	Official leaderboard
LMArena (Elo)	1,322	#208 of 335	Official leaderboard
SimpleQA	23.5%	#10 of 16	Official leaderboard
GRE Analytical Writing	5.00	#2 of 3	Lab model card
AidanBench	908.0	#10 of 13	Official leaderboard
LLM Confabulation Leaderboard	22.7%	#45 of 58	Official leaderboard
RewardBench 2	57.4%	#34 of 42	Official leaderboard
MMBench-CN v1.1	58.1%	#148 of 178	Official leaderboard
SEAL Precise Instruction Following	80.1	#15 of 18	Official leaderboard
Vectara Hallucination Leaderboard (HHEM)	10.1%	#129 of 148	Official leaderboard
SimpleQA Verified	12.6%	#89 of 94	Epoch AI run
SEAL Multilingual	919	#18 of 19	Official leaderboard
Multi-factual (Anthropic)	62.8%	#1 of 1	Lab model card

Math and science

Benchmark	Score	Rank	Source
GSM8K	95%	#13 of 92	Lab model card
MathBench	63%	#3 of 21	Official leaderboard
GPQA Main	49.1%	#1 of 7	Lab model card
GSM1k	95.2%	#4 of 15	Official leaderboard
ZebraLogic	27%	#15 of 46	Official leaderboard
AMC 10 2023	72.0	#1 of 3	Lab model card
AMC 12 2023	63.0	#1 of 3	Lab model card
AMC 8 2023	84.0	#1 of 3	Lab model card
GPQA Extended	48.8%	#1 of 3	Lab model card
MATH	73.7%	#40 of 99	Lab model card
GRE Quantitative	159.0	#3 of 6	Lab model card
OlympiadBench	7.65%	#3 of 6	Official leaderboard
ChemBench	57%	#15 of 24	Official leaderboard
ForecastBench	0.14	#54 of 82	Official leaderboard
MATH Level 5	37.5%	#65 of 94	Epoch AI run
GPQA Diamond	47.2%	#170 of 229	Epoch AI run
DTBench	61.6%	#136 of 180	Official leaderboard
OTIS Mock AIME 2024-2025	4.72%	#182 of 212	Epoch AI run
SimpleBench	23.5%	#93 of 106	Official leaderboard

Coding

Benchmark	Score	Rank	Source
ChartMimic	60.3	#1 of 14	Official leaderboard
CRUXEval	82%	#1 of 11	Official leaderboard
MBPP	86.4%	#7 of 71	Lab model card
NaturalCodeBench	48.3%	#1 of 5	Official leaderboard
BigCodeBench	51.5%	#16 of 69	Official leaderboard
MBPP+	73.3%	#9 of 35	Official leaderboard
HumanEval+	77.4%	#14 of 46	Official leaderboard
APPS	70.2%	#1 of 3	Lab model card
Design2Code	87.0	#2 of 5	Official leaderboard
HumanEval	84.9%	#41 of 101	Lab model card
BigCodeBench-Hard	26%	#40 of 96	Official leaderboard
Aider-Edit	68.4%	#15 of 34	Official leaderboard
SciCode	1.50%	#8 of 18	Official leaderboard
SWE-bench	10.5%	#6 of 9	Official leaderboard
SEAL Coding	959	#19 of 26	Official leaderboard
METR Time Horizon	7.1 min	#24 of 31	Official leaderboard
SWE-bench Lite	11.7%	#12 of 15	Official leaderboard
METR Time Horizon 1.1	4.0 min	#22 of 26	Official leaderboard
WeirdML	19.2%	#102 of 112	Official leaderboard
SWE-bench Verified	15.8%	#55 of 60	Official leaderboard

Agents and tools

Benchmark	Score	Rank	Source
GTA	23.4%	#4 of 15	Official leaderboard
AgentBench	3.11	#2 of 5	Official leaderboard
VisualAgentBench	21.9%	#2 of 5	Official leaderboard
Berkeley Function Calling Leaderboard (BFCL)	57%	#32 of 67	Official leaderboard
Spider2-V	8.10%	#3 of 5	Official leaderboard
LisanBench	249.0	#84 of 133	Official leaderboard
ToolComp	52.8%	#21 of 32	Official leaderboard
LLM Chess	-146	#89 of 123	Official leaderboard
Chess Puzzles	4.00%	#90 of 93	Epoch AI run
CRAB	19.6%	#3 of 3	Official leaderboard

Professional work

Benchmark	Score	Rank	Source
MedQA	77.5%	#11 of 71	Official leaderboard
MBE (Multistate Bar Examination)	85%	#1 of 4	Lab model card
GMAI-MMBench	32.4%	#12 of 13	Official leaderboard
PubMedQA	75.8%	#3 of 3	Lab model card

Vision, audio and robotics

Benchmark	Score	Rank	Source
WildVision Arena	1,100	#1 of 10	Official leaderboard
MMDU	62.6	#1 of 5	Official leaderboard
WildVision	62.0	#3 of 14	Official leaderboard
MTVQA-Test	25.7%	#4 of 18	Official leaderboard
TOMATO	28.2%	#8 of 22	Official leaderboard
BLINK	44.1%	#4 of 11	Official leaderboard
CharXiv (DQ)	71.5%	#25 of 67	Official leaderboard
Image2Struct	62.3	#7 of 18	Official leaderboard
CoreCognition	56.7%	#49 of 119	Official leaderboard
MMMU	59.4%	#60 of 136	Official leaderboard
MM-Vet	51.7%	#82 of 164	Official leaderboard
AI2D	88.1%	#28 of 52	Lab model card
CharXiv	30.2%	#44 of 77	Official leaderboard
MMStar	45.7%	#123 of 173	Official leaderboard
MATH-Vision (MathVision)	27.1%	#78 of 105	Official leaderboard
VHELM	31.9	#32 of 41	Official leaderboard
SEED-Bench-2	42%	#4 of 5	Official leaderboard
CCBench (MMBench Chinese Culture)	26.3%	#123 of 152	Official leaderboard
DocVQA	89.3%	#30 of 36	Lab model card
ChartQA	80.8%	#32 of 37	Lab model card
VISTA	27.8%	#47 of 54	Official leaderboard
Vision Arena	1,062	#119 of 136	Official leaderboard
MMBench	63.3%	#137 of 155	Official leaderboard
SEED-Bench	40.9%	#8 of 9	Official leaderboard
MMBench-EN v1.1	60.1%	#160 of 178	Official leaderboard
MathVista	50.5%	#66 of 70	Lab model card
ZeroBench	0%	#72 of 76	Official leaderboard
MLVU	25%	#9 of 9	Official leaderboard

Safety and security

Benchmark	Score	Rank	Source
SimpleSafetyTests	100%	#1 of 75	Official leaderboard
Anthropic Red Team	99.8%	#8 of 75	Official leaderboard
SORRY-Bench	7.27%	#3 of 23	Official leaderboard
BrowserART	11%	#1 of 6	Official leaderboard
HELM Safety	96.8	#13 of 75	Official leaderboard
AIR-Bench 2024	84.4%	#16 of 82	Official leaderboard
Situational Awareness Dataset (SAD)	53.4%	#4 of 17	Official leaderboard
XSTest	9.00%	#5 of 16	Lab model card
LAB-Bench ProtocolQA	52%	#1 of 3	Official leaderboard
SEAL Adversarial Robustness	13.0	#3 of 8	Official leaderboard
AgentDojo	11.3%	#9 of 20	Official leaderboard
Cybench	10%	#10 of 16	Official leaderboard
LAB-Bench CloningScenarios	27%	#2 of 3	Official leaderboard
LAB-Bench FigQA	24%	#2 of 3	Official leaderboard
LAB-Bench LitQA2	6.00%	#2 of 3	Official leaderboard
AgentHarm	14.4%	#6 of 8	Official leaderboard
OR-Bench	91%	#24 of 28	Official leaderboard
LAB-Bench	29.6%	#3 of 3	Official leaderboard
Multimodal policy red-teaming (Anthropic)	97.9%	#2 of 2	Lab model card

Other

Benchmark	Score	Rank	Source
HELM Lite	68.3	#24 of 71	Official leaderboard
OpenVLM Leaderboard	54.4	#110 of 173	Official leaderboard
LMCA	17%	#120 of 147	Official leaderboard
Conceptual Reasoning Index	24.0	#131 of 151	Official leaderboard
ACCoRD	33.1	#141 of 152	Official leaderboard

Its lab's numbers put it at 127.8, 2.3 points above its ECI

The Self-Reported Index fits one score to the 19 capability results Anthropic reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 125.2 to 130.4. How we compute it.

Needle In A Haystack 99.4
ARC-Challenge 96.4
HellaSwag 95.4
GSM8K 95
MGSM 90.7
DocVQA 89.3
WinoGrande 88.5
MMLU 88.2
AI2D 88.1
BIG-Bench Hard 86.8
MBPP 86.4
HumanEval 84.9
DROP 83.1
ChartQA 80.8
Okapi MMLU 79.1
MATH 73.7
GPQA Diamond 59.5
MMMU 59.4
MathVista 50.5

Its lab's numbers next to 5 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark	Lab says	Board says	Gap
GPQA Diamond	59.5% GPQA (Diamond) (Maj@32 5-shot CoT)	47.2% Epoch AI run	+12 pts
MBPP	86.4% MBPP (Pass@1)	89.4% Leaderboard, EvalPlus subset, original tests	−3.0 pts
HumanEval	84.9% HumanEval (0-shot)	82.9% Leaderboard	+2.0 pts
MathVista	50.5% MathVista (testmini) CoT	50.5% Leaderboard	Same
MMMU	59.4% MMMU (val) Overall	59.4% Leaderboard	Same

Anthropic's models over time

Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

The Claude 3 Model Family: Opus, Sonnet, Haiku Mar 4, 2024

MMLU (5-shot CoT) 88.2
MATH (Maj@32 4-shot) 73.7
GSM8K (0-shot CoT) 95
HumanEval (0-shot) 84.9
GPQA (Diamond) (Maj@32 5-shot CoT) 59.5
MGSM (0-shot) 90.7
DROP (F1 Score, 3-shot) 83.1
BIG-Bench-Hard (3-shot CoT) 86.8
ARC-Challenge (25-shot) 96.4
HellaSwag (10-shot) 95.4
PubMedQA (5-shot) 75.8
WinoGrande (5-shot) 88.5
RACE-H (5-shot) 92.9
APPS (0-shot) 70.2
MBPP (Pass@1) 86.4
MMMU (val) Overall 59.4
DocVQA (test, ANLS score) 89.3
MathVista (testmini) CoT 50.5
AI2D (test) 88.1
ChartQA (test, relaxed accuracy) CoT 80.8

Its system card

Every result in the longer documents the lab published with it. 23 of them aren't in the launch post.

Claude 3 Opus

Claude 3 Opus

Where it ranks

Knowledge and language

Math and science

Coding

Agents and tools

Professional work

Vision, audio and robotics

Safety and security

Other

Its lab's numbers put it at 127.8, 2.3 points above its ECI

Its lab's numbers next to 5 leaderboards

Anthropic's models over time

Its launch post

Its system card

Capabilities

Hallucination

Safety

Bias

Cyber risk

Autonomy