Claude 3.5 Haiku places in the top three on 3 benchmarks.
Closed weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BullshitBench | 58.2% | #20 of 107 | Official leaderboard | |
| BIG-Bench Hard | 86.6% | #17 of 73 | Lab model card | |
| BuzzBench | 53.3 | #7 of 20 | Official leaderboard | |
| BullshitBench V2 | 50% | #49 of 129 | Official leaderboard | |
| Phare | 69.1% | #24 of 62 | Official leaderboard | |
| MGSM | 85.6% | #26 of 57 | Lab model card | |
| FACTS Grounding | 78.1% | #16 of 34 | Official leaderboard | |
| MMLU | 80.9% | #71 of 131 | Lab model card | |
| IFEval | 85.9% | #75 of 135 | Lab model card | |
| LMArena (Elo) | 1,325 | #202 of 335 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 4.90% | #96 of 148 | Official leaderboard | |
| ToRR | 42.7 | #10 of 14 | Official leaderboard | |
| ECLeKTic | 10.1% | #26 of 35 | Official leaderboard | |
| Answer Factuality (Perplexity) | 71.7% | #3 of 4 | Official leaderboard | |
| MMLU-Pro | 65% | #136 of 167 | Lab model card | |
| Creative Writing v3 | 54.1 | #99 of 120 | Official leaderboard | |
| AidanBench | 904.0 | #11 of 13 | Official leaderboard | |
| WritingBench | 49.1 | #40 of 45 | Official leaderboard | |
| LLM Confabulation Leaderboard | 36.7% | #56 of 58 | Official leaderboard | |
| Global MMLU Lite | 61.1% | #42 of 43 | Official leaderboard | |
| Honesty human feedback win rate (Anthropic) | 42% | #6 of 6 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MATH | 69.2% | #49 of 99 | Lab model card | |
| ZebraLogic | 18.7% | #23 of 46 | Official leaderboard | |
| MATH Level 5 | 46.4% | #56 of 94 | Epoch AI run | |
| PhyX | 13.6% | #6 of 10 | Official leaderboard | |
| GPQA Diamond | 38.1% | #189 of 229 | Epoch AI run | |
| DTBench | 56.7% | #149 of 180 | Official leaderboard | |
| GPQA Main | 36.3% | #51 of 60 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 4.31% | #186 of 212 | Epoch AI run | |
| AstaBench | 21.4% | #12 of 13 | Official leaderboard | |
| MATH-500 | 64.2% | #36 of 38 | Official leaderboard | |
| FrontierMath | 0.34% | #54 of 55 | Official leaderboard | |
| AIME 2024 | 5.30% | #88 of 88 | Lab model card | |
| GSM8K Platinum Initial 268 | 96.3% | #19 of 19 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench | 52.5% | #10 of 69 | Official leaderboard | |
| Aider-Edit | 75.2% | #5 of 34 | Official leaderboard | |
| BigCodeBench-Hard | 30.1% | #20 of 96 | Official leaderboard | |
| HumanEval | 88.1% | #25 of 101 | Lab model card | |
| SWE-PolyBench | 9.86% | #3 of 6 | Official leaderboard | |
| Internal agentic coding evaluation (Anthropic) | 74% | #2 of 3 | Lab model card | |
| Aider Polyglot | 28% | #31 of 46 | Official leaderboard | |
| SWE-bench Verified | 40.6% | #41 of 60 | Official leaderboard | |
| WeirdML | 30.7% | #98 of 112 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ComplexFuncBench | 45.8% | #4 of 11 | Official leaderboard | |
| Berkeley Function Calling Leaderboard (BFCL) | 53.7% | #39 of 67 | Official leaderboard | |
| LLM Chess | -120 | #80 of 123 | Official leaderboard | |
| BALROG | 19.3% | #22 of 33 | Official leaderboard | |
| LisanBench | 144.0 | #101 of 133 | Official leaderboard | |
| Elimination Game | 3 | #45 of 56 | Official leaderboard | |
| Galileo Agent Leaderboard | 0.77 | #24 of 29 | Official leaderboard | |
| Step Race | 1 | #58 of 59 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 72.2% | #22 of 71 | Official leaderboard | |
| CadEval | 32% | #9 of 13 | Official leaderboard | |
| FailureSensorIQ | 57.6% | #18 of 24 | Official leaderboard | |
| Vals Finance Agent v1 | 14.3% | #36 of 47 | Official leaderboard | |
| CorpFin v2 | 50.8% | #95 of 119 | Official leaderboard | |
| LegalBench | 70.3% | #114 of 138 | Official leaderboard | |
| TaxEval v2 | 57.4% | #121 of 129 | Official leaderboard | |
| ProfBench | 21.2% | #72 of 72 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MicroVQA | 47.1% | #12 of 19 | Official leaderboard | |
| Vision Arena | 1,127 | #108 of 136 | Official leaderboard | |
| GeoBench | 34% | #48 of 48 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| XSTest | 2.00% | #3 of 16 | Lab model card | |
| Gray Swan Agent Red Teaming (ART) benchmark | 2.44% | #2 of 8 | Official leaderboard | |
| WildChat non-toxic incorrect refusals (Anthropic) | 4.80% | #1 of 3 | Lab model card | |
| KORA | 54% | #15 of 42 | Official leaderboard | |
| Lakera B3 (Backbone Breaker Benchmark) | 53.9% | #19 of 35 | Official leaderboard | |
| FORTRESS | 30.4% | #48 of 73 | Official leaderboard | |
| Harmlessness human feedback win rate (Anthropic) | 43% | #4 of 4 | Lab model card | |
| WildChat toxic refusals (Anthropic) | 88% | #3 of 3 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 53.1 | #34 of 71 | Official leaderboard | |
| HELM Capabilities | 54.9 | #51 of 60 | Official leaderboard |
The Self-Reported Index fits one score to the 10 capability results Anthropic reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 116.4 to 122.8. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap |
|---|---|---|---|
| IFEval | 85.9% IFEval (Table 8) | 79.2% Leaderboard, strict, share of instructions followed | +6.7 pts |
| GPQA Diamond | 41.6% GPQA (Diamond), 0-shot CoT (Table 8) | 38.1% Epoch AI run, Oct 2024 | +3.5 pts |
| MMLU-Pro | 65% MMLU Pro, 0-shot CoT (Table 8) | 62.1% Leaderboard | +2.9 pts |
| SWE-bench Verified | 40.6% SWE-bench Verified, % of problems which pass all tests, pass@1 (Table 2) | 40.6% Leaderboard, Tools | Same |
Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Every result in the longer documents the lab published with it. 19 of them aren't in the launch post.