Benchmark Almanac

Claude 3 Haiku

Claude 3 Haiku ranks first on Anthropic Red Team.

Anthropic Released March 7th, 2024 Compare with others
110
Benchmarks with a score
1
Ranked first, out of five or more models
4
In the top three
Anthropic Mar 2024

Claude 3 Haiku

Closed weights

117.7 #190 of 256
Epoch Capabilities Index
Self-Reported Index 115.9 #361 of 404
  • ZeroBench 0% #73 /76
  • GPQA Diamond 36.3% #194 /229
  • IHEval 27.9% #7 /10

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
RepoQA 81.8% #8 of 22 Official leaderboard
ARC-Challenge 89.2% #21 of 49 Lab model card
HellaSwag 85.9% #24 of 53 Lab model card
TurkishMMLU 65.4% #4 of 8 Official leaderboard
Arena-Hard-Auto 41.5% #20 of 38 Official leaderboard
ThaiExam 54% #13 of 24 Official leaderboard
RACE-H 87% #4 of 7 Lab model card
WildBench 40.3 #27 of 47 Official leaderboard
BIG-Bench Hard 73.7% #42 of 73 Lab model card
MixEval 79.7% #22 of 34 Official leaderboard
BullshitBench 18.2% #73 of 107 Official leaderboard
MGSM 75.1% #39 of 57 Lab model card
MMLU 76.7% #90 of 131 Lab model card
WinoGrande 74.2% #33 of 47 Lab model card
EQ-Bench v2 63.6 #60 of 85 Official leaderboard
MMLU-Pro 42.3% #155 of 212 Official leaderboard
LMArena (Elo) 1,262 #258 of 335 Official leaderboard
RewardBench 72.9% #38 of 48 Official leaderboard
QuALITY 80.2% #6 of 7 Lab model card
MMBench-CN v1.1 56.2% #155 of 178 Official leaderboard
BullshitBench V2 10% #113 of 129 Official leaderboard
Needle In A Haystack 95.9% #8 of 9 Lab model card
Creative Writing v3 48.3 #109 of 120 Official leaderboard
LLM Confabulation Leaderboard 34.2% #55 of 58 Official leaderboard
RewardBench 2 37.1% #42 of 42 Official leaderboard
JudgeBench 33.1% #11 of 11 Official leaderboard
Okapi MMLU 65.2% #8 of 8 Lab model card
AlignBench 6.38 #7 of 7 Official leaderboard
LSAT 156.3 #4 of 4 Lab model card
Summary of a Haystack 26.8% #3 of 3 Official leaderboard

Math and science

BenchmarkScoreRankSource
GSM8K 88.9% #46 of 92 Lab model card
ZebraLogic 14.3% #27 of 46 Official leaderboard
AMC 10 2023 54.0 #2 of 3 Lab model card
AMC 12 2023 48.0 #2 of 3 Lab model card
ChemBench 50% #18 of 24 Official leaderboard
MATH 40.9% #80 of 99 Lab model card
GPQA Diamond 36.3% #194 of 229 Epoch AI run
GPQA Main 35.2% #6 of 7 Lab model card
MATH Level 5 14.9% #82 of 94 Epoch AI run
DTBench 50.1% #164 of 180 Official leaderboard
OTIS Mock AIME 2024-2025 1.81% #199 of 212 Epoch AI run
ForecastBench 0.21 #81 of 82 Official leaderboard
AMC 8 2023 36.0 #3 of 3 Lab model card
GPQA Extended 34.7% #3 of 3 Lab model card

Coding

BenchmarkScoreRankSource
MBPP 80.4% #18 of 71 Lab model card
HumanEval+ 68.9% #20 of 46 Official leaderboard
MBPP+ 68.8% #16 of 35 Official leaderboard
BigCodeBench 44.8% #36 of 69 Official leaderboard
NaturalCodeBench 36.2% #3 of 5 Official leaderboard
HumanEval 75.9% #61 of 101 Lab model card
MHPP 24.8% #14 of 23 Official leaderboard
BigCodeBench-Hard 18.3% #64 of 96 Official leaderboard
SWT-bench 2.50% #6 of 7 Official leaderboard
Aider-Edit 47.4% #32 of 34 Official leaderboard
WeirdML 9.84% #111 of 112 Official leaderboard
LiveCodeBench 23.1% #21 of 21 Official leaderboard
LiveCodeBench v5 18.5% #21 of 21 Official leaderboard
LiveCodeBench v6 20.2% #21 of 21 Official leaderboard
APPS 54.8% #3 of 3 Lab model card

Agents and tools

BenchmarkScoreRankSource
LLM Colosseum 1,365 #2 of 3 Official leaderboard
LisanBench 135.7 #104 of 133 Official leaderboard

Professional work

BenchmarkScoreRankSource
MedQA 70.2% #24 of 71 Official leaderboard
PubMedQA 76% #2 of 3 Lab model card
CadEval 12% #13 of 13 Official leaderboard
MBE (Multistate Bar Examination) 64% #4 of 4 Lab model card

Vision, audio and robotics

BenchmarkScoreRankSource
WildVision Arena 1,002 #4 of 10 Official leaderboard
CharXiv (DQ) 65.1% #28 of 67 Official leaderboard
WildVision 37.8 #6 of 14 Official leaderboard
SEED-Bench-2-Plus 61.7% #69 of 136 Official leaderboard
CharXiv 31.8% #40 of 77 Official leaderboard
ScienceQA 72% #9 of 17 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
TOMATO 26.2% #12 of 22 Official leaderboard
SGP-Bench 48.6% #13 of 22 Official leaderboard
MM-Vet 46.4% #99 of 164 Official leaderboard
MMMU 50.2% #84 of 136 Official leaderboard
AI2D 86.7% #35 of 52 Lab model card
CoreCognition 51.8% #82 of 119 Official leaderboard
ChartQA 81.7% #26 of 37 Lab model card
CCBench (MMBench Chinese Culture) 24.5% #126 of 152 Official leaderboard
DocVQA 88.8% #31 of 36 Lab model card
VHELM 27.1 #36 of 41 Official leaderboard
MMStar 38.1% #154 of 173 Official leaderboard
MMBench 60.7% #142 of 155 Official leaderboard
MMBench-EN v1.1 58% #164 of 178 Official leaderboard
Vision Arena 1,000 #129 of 136 Official leaderboard
ZeroBench 0% #73 of 76 Official leaderboard
MathVista 46.4% #69 of 70 Lab model card

Safety and security

BenchmarkScoreRankSource
Anthropic Red Team 99.96% #1 of 75 Official leaderboard
SimpleSafetyTests 100% #3 of 75 Official leaderboard
SORRY-Bench 7.95% #4 of 23 Official leaderboard
AIR-Bench 2024 82.7% #18 of 82 Official leaderboard
LAB-Bench 35.2% #1 of 3 Official leaderboard
LAB-Bench LitQA2 27% #1 of 3 Official leaderboard
AgentDojo 9.06% #7 of 20 Official leaderboard
AgentHarm 6.60% #3 of 8 Official leaderboard
Situational Awareness Dataset (SAD) 45.9% #11 of 17 Official leaderboard
IHEval 27.9% #7 of 10 Official leaderboard
HELM Safety 87.8 #56 of 75 Official leaderboard
nocot-bench (NCRI) 57.3 #149 of 185 Official leaderboard
OR-Bench 96.3% #27 of 28 Official leaderboard
NYU CTF Bench 4.00% #7 of 7 Official leaderboard
PrivacyLens 39% #4 of 4 Official leaderboard
LAB-Bench CloningScenarios 26% #3 of 3 Official leaderboard
LAB-Bench FigQA 23% #3 of 3 Official leaderboard
LAB-Bench ProtocolQA 45% #3 of 3 Official leaderboard

Other

BenchmarkScoreRankSource
OpenVLM Leaderboard 50.6 #123 of 173 Official leaderboard
HELM Lite 26.3 #64 of 71 Official leaderboard
LMCA 8.84% #135 of 147 Official leaderboard
Conceptual Reasoning Index 11.6 #146 of 151 Official leaderboard
ACCoRD 14.7 #149 of 152 Official leaderboard

Its lab's numbers put it at 115.9, 1.8 points below its ECI

The Self-Reported Index fits one score to the 18 capability results Anthropic reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 113.7 to 118.2. How we compute it.

Its lab's numbers next to 5 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
GPQA Diamond 33.3% GPQA (Diamond), 0-shot CoT, mean of 10 rollouts. 5-shot CoT 36.4, Maj@32 5-shot CoT 40.1 (Table 1, Table 8) 36.3% Epoch AI run −3.0 pts
HumanEval 75.9% HumanEval, 0-shot (Table 1) 76.8% Leaderboard −0.9 pts
MBPP 80.4% MBPP, Pass@1 (Table 1) 80.2% Leaderboard, EvalPlus subset, original tests +0.2 pts
MathVista 46.4% MathVista (testmini), 0-shot CoT (Table 3) 46.4% Leaderboard Same
MMMU 50.2% MMMU (val), Overall, 0-shot. Per-discipline values also given (Table 3) 50.2% Leaderboard Same

Anthropic's models over time

Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.

50 100 150 2023 2024 2025 2026 2027 Claude Fable 5.2 Claude Haiku 5.5 Claude Sonnet 4.6 Claude Haiku 4.5 Claude 3.7 Sonnet Claude 3.5 Haiku Claude 3.5 Sonnet (October 2024) 50 100 150 2023 2024 2025 2026 2027 Claude Fable 5.2 Claude Haiku 5.5

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its system card

Every result in the longer documents the lab published with it. 37 of them aren't in the launch post.

The Claude 3 Model Family: Opus, Sonnet, Haiku Mar 4, 2024

Capabilities

Hallucination

Safety

Bias