- OSWorld (screenshot only, 50 steps) Overall 22
- SWE-bench Verified (% of problems which pass all tests) 49
- Mean of TAU-bench Retail (pass^1) / TAU-bench Airline (pass^1) 57.6
- MMMU (validation) 70.4
- MathVista (testmini) 70.7
- AI2D (test) 95.3
- ChartQA (test, relaxed accuracy) 90.8
- DocVQA (test, ANLS score) 94.2
- GPQA (Diamond) (0-shot CoT) 65
- MMLU (5-shot CoT) 90.5
- MMLU Pro (0-shot CoT) 78
- MATH (0-shot CoT) 78.3
- HumanEval (0-shot) 93.7
- MGSM (0-shot CoT) 92.5
- DROP (F1 Score, 3-shot) 88.3
- BIG-Bench Hard (3-shot CoT) 93.2
- AIME 2024 (Maj@64 0-shot CoT) 27.6
- IFEval 90.2
Claude 3.5 Sonnet (October 2024)
Claude 3.5 Sonnet (October 2024) ranks first on 7 of the 95 benchmarks with five or more models scored.
102
Benchmarks with a score
7
Ranked first, out of five or more models
18
In the top three
Anthropic Oct 2024
Claude 3.5 Sonnet (October 2024)
Closed weights
133.4
#149 of 256
Epoch Capabilities Index
Self-Reported Index 126.8 #304 of 404
- Humanity's Last Exam 4.08% #45 /47
- ZeroBench 3.00% #55 /76
- GPQA Diamond 55.3% #148 /229
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BIG-Bench Hard | 93.2% | #2 of 73 | Lab model card | |
| MGSM | 92.5% | #3 of 57 | Lab model card | |
| ToRR | 49.9 | #1 of 14 | Official leaderboard | |
| MMLU | 90.5% | #10 of 131 | Lab model card | |
| Arena-Hard-Auto | 85.2% | #3 of 38 | Official leaderboard | |
| MMBench-CN v1.1 | 82.5% | #29 of 178 | Official leaderboard | |
| IFEval | 90.2% | #28 of 135 | Lab model card | |
| DROP | 88.3% | #10 of 41 | Lab model card | |
| SuperGPQA | 48.2% | #25 of 59 | Official leaderboard | |
| LMArena (Elo) | 1,374 | #158 of 335 | Official leaderboard | |
| Creation-MMBench | -15.46 | #9 of 19 | Official leaderboard | |
| MMLU-Pro | 78% | #92 of 167 | Lab model card | |
| LongBench v2 | 46.7% | #19 of 33 | Official leaderboard | |
| AgentIF | 56.6% | #7 of 12 | Official leaderboard | |
| LLM Confabulation Leaderboard | 19.9% | #37 of 58 | Official leaderboard | |
| Arena-Hard v2.0 | 33% | #16 of 23 | Official leaderboard | |
| Honesty human feedback win rate (Anthropic) | 55% | #5 of 6 | Lab model card | |
| Humanity's Last Exam | 4.08% | #45 of 47 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ZebraLogic | 36.2% | #7 of 46 | Official leaderboard | |
| MATH | 78.3% | #30 of 99 | Lab model card | |
| CNMO 2024 | 13.1% | #3 of 7 | Official leaderboard | |
| GPQA Main | 56.5% | #28 of 60 | Official leaderboard | |
| MATH Level 5 | 57% | #44 of 94 | Epoch AI run | |
| GSM8K Platinum Initial 268 | 98.9% | #10 of 19 | Official leaderboard | |
| Misguided Attention | 37.2% | #27 of 45 | Official leaderboard | |
| GPQA Diamond | 55.3% | #148 of 229 | Epoch AI run | |
| DTBench | 67.8% | #119 of 180 | Official leaderboard | |
| SimpleBench | 41.4% | #71 of 106 | Official leaderboard | |
| TSP-Opt | 13.3% | #8 of 11 | Official leaderboard | |
| GSM8K-Platinum | 99.1% | #6 of 8 | Official leaderboard | |
| U-MATH | 38.7% | #23 of 30 | Official leaderboard | |
| FrontierMath | 2.07% | #54 of 70 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 8.47% | #164 of 212 | Epoch AI run | |
| FrontierMath Tier 4 | 0% | #47 of 55 | Epoch AI run | |
| AIME 2024 | 27.6% | #82 of 88 | Lab model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Aider-Edit | 84.2% | #1 of 34 | Official leaderboard | |
| HumanEval | 93.7% | #6 of 101 | Lab model card | |
| HumanEval-V | 36.8% | #2 of 24 | Official leaderboard | |
| SWE-PolyBench | 14.1% | #1 of 6 | Official leaderboard | |
| MHPP | 48.1% | #4 of 23 | Official leaderboard | |
| SciCode | 4.60% | #4 of 18 | Official leaderboard | |
| Multi-SWE-bench | 7.97% | #3 of 11 | Official leaderboard | |
| Internal agentic coding evaluation (Anthropic) | 78% | #1 of 3 | Lab model card | |
| METR Time Horizon | 30 min | #15 of 31 | Official leaderboard | |
| METR Time Horizon 1.1 | 21 min | #16 of 26 | Official leaderboard | |
| LiveCodeBench v5 | 36.5% | #16 of 21 | Official leaderboard | |
| LiveCodeBench v6 | 36.4% | #16 of 21 | Official leaderboard | |
| SWE-bench Multimodal | 25.3% | #7 of 9 | Official leaderboard | |
| GSO-Bench | 4.60% | #24 of 30 | Official leaderboard | |
| LiveCodeBench | 54.1% | #17 of 21 | Official leaderboard | |
| SWE-bench Verified | 49% | #133 of 155 | Lab model card | |
| ArtifactsBench | 39.9% | #19 of 20 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ComplexFuncBench | 61% | #1 of 11 | Official leaderboard | |
| Factorio Learning Environment | 293206.0 | #1 of 6 | Official leaderboard | |
| Elimination Game | 5 | #14 of 56 | Official leaderboard | |
| Berkeley Function Calling Leaderboard (BFCL) | 56.9% | #33 of 67 | Official leaderboard | |
| tau-bench Airline | 46% | #1 of 2 | Official leaderboard | |
| tau-bench Retail | 69.2% | #1 of 2 | Official leaderboard | |
| LisanBench | 459.7 | #69 of 133 | Official leaderboard | |
| Step Race | 2 | #34 of 59 | Official leaderboard | |
| LLM Chess | -99 | #74 of 123 | Official leaderboard | |
| tau-bench | 57.6% | #15 of 23 | Lab model card | |
| Galileo Agent Leaderboard | 0.80 | #23 of 29 | Official leaderboard | |
| OSWorld | 14.9% | #11 of 13 | Lab model card | |
| OSWorld-Verified | 22% | #41 of 41 | Lab model card |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 85.9% | #2 of 71 | Official leaderboard | |
| MedHELM | 54.2 | #6 of 13 | Official leaderboard | |
| CadEval | 48% | #7 of 13 | Official leaderboard | |
| Vending-Bench | $2,218 | #6 of 9 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AI2D | 95.3% | #2 of 52 | Lab model card | |
| MEGA-Bench | 54.3% | #1 of 22 | Official leaderboard | |
| ChartQA | 90.8% | #3 of 37 | Lab model card | |
| MM-Vet | 70.1% | #20 of 164 | Official leaderboard | |
| KRETA | 80.5% | #3 of 17 | Official leaderboard | |
| MMStar | 65.1% | #31 of 173 | Official leaderboard | |
| Video-MMMU | 65.8% | #5 of 24 | Official leaderboard | |
| MMVU | 65.2% | #6 of 28 | Official leaderboard | |
| MMBench | 82.6% | #35 of 155 | Official leaderboard | |
| MMBench-EN v1.1 | 80.9% | #47 of 178 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 56.1% | #46 of 152 | Official leaderboard | |
| DocVQA | 94.2% | #16 of 36 | Lab model card | |
| Image2Struct | 60.5 | #8 of 18 | Official leaderboard | |
| VHELM | 49.7 | #24 of 41 | Official leaderboard | |
| MMMU | 70.4% | #61 of 103 | Lab model card | |
| MathVista | 70.7% | #45 of 70 | Lab model card | |
| VISTA | 38.7% | #35 of 54 | Official leaderboard | |
| Vision Arena | 1,161 | #94 of 136 | Official leaderboard | |
| ZeroBench | 3.00% | #55 of 76 | Official leaderboard | |
| VPCT | 33% | #26 of 27 | VPCT leaderboard, via Epoch AI | |
| CompassArena Multimodal | 1,026 | #8 of 8 | Official leaderboard | |
| CC-OCR | 62.7% | #3 of 3 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AIR-Bench 2024 | 90.8% | #2 of 82 | Official leaderboard | |
| AgentDojo | 1.11% | #1 of 20 | Official leaderboard | |
| XSTest | 4.30% | #4 of 16 | Lab model card | |
| Human Pathogen Capabilities Test | 49.9% | #47 of 86 | Official leaderboard | |
| WildChat non-toxic incorrect refusals (Anthropic) | 5.30% | #2 of 3 | Lab model card | |
| WildChat toxic refusals (Anthropic) | 89.2% | #2 of 3 | Lab model card | |
| Molecular Biology Capabilities Test | 42.9% | #65 of 90 | Official leaderboard | |
| Harmlessness human feedback win rate (Anthropic) | 48% | #3 of 4 | Lab model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 84.6 | #9 of 71 | Official leaderboard | |
| OpenVLM Leaderboard | 70.6 | #33 of 173 | Official leaderboard | |
| HELM Capabilities | 65.3 | #32 of 60 | Official leaderboard |
Its lab's numbers put it at 126.8, 6.6 points below its ECI
The Self-Reported Index fits one score to the 19 capability results Anthropic reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 124.2 to 129.4. How we compute it.
- AI2D 95.3
- DocVQA 94.2
- HumanEval 93.7
- BIG-Bench Hard 93.2
- MGSM 92.5
- ChartQA 90.8
- MMLU 90.5
- IFEval 90.2
- DROP 88.3
- MATH 78.3
- MMLU-Pro 78
- MathVista 70.7
- MMMU 70.4
- GPQA Diamond 65
- tau-bench 57.6
- SWE-bench Verified 49
- AIME 2024 27.6
- OSWorld-Verified 22
- OSWorld 14.9
Its lab's numbers next to 3 leaderboards
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Anthropic's models over time
Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 8 of them aren't in the launch post.
Capabilities
- OSWorld 14.9
- SWE-bench Verified 49
- tau-bench
- Internal agentic coding evaluation (Anthropic) 78
- MMMU 70.4
- MathVista 70.7
- AI2D 95.3
- ChartQA 90.8
- DocVQA 94.2
- GPQA Diamond 65
- MMLU 90.5
- MMLU-Pro 78
- MATH 78.3
- HumanEval 93.7
- MGSM 92.5
- DROP 88.3
- BIG-Bench Hard 93.2
- AIME 2024 16
- IFEval 90.2
- Human feedback win rates (Anthropic)