Model Card and Evaluations for Claude Models Jul 11, 2023
- Codex P@1 (0-shot) 71.2
- GSM8k (0-shot CoT) 88
- MMLU (5-shot CoT) 78.5
- TriviaQA (5-shot) 87.5
- QuALITY (5-shot) 83.2
- ARC-Challenge (5-shot) 91
- RACE-H (5-shot) 88.3
Claude 2
Claude 2 places in the top three on 2 benchmarks.
42
Benchmarks with a score
0
Ranked first, out of five or more models
2
In the top three
Anthropic Jul 2023
Claude 2
Closed weights Flagship
117.1
#191 of 256
Epoch Capabilities Index
Self-Reported Index 121.2 #338 of 404
- GPQA Diamond 34.7% #197 /229
- Human Pathogen Capabilities Test 23.6% #82 /86
- ChemBench 47.4% #20 /24
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TriviaQA | 87.5% | #6 of 40 | Lab model card | |
| TruthfulQA | 69% | #7 of 30 | Lab model card | |
| GRE Analytical Writing | 5.00 | #1 of 3 | Lab model card | |
| ARC-Challenge | 91% | #18 of 49 | Lab model card | |
| AlpacaEval 2.0 | 28.2% | #18 of 47 | Official leaderboard | |
| RACE-H | 88.3% | #3 of 7 | Lab model card | |
| AlpacaEval | 91.4% | #9 of 19 | Official leaderboard | |
| EQ-Bench v2 | 72.9 | #46 of 85 | Official leaderboard | |
| QuALITY | 83.2% | #4 of 7 | Lab model card | |
| MMLU | 78.5% | #83 of 131 | Lab model card | |
| Arena-Hard-Auto | 24% | #28 of 38 | Official leaderboard | |
| GRE Verbal | 165.0 | #5 of 6 | Lab model card | |
| Vectara Hallucination Leaderboard (HHEM) | 17.4% | #144 of 148 | Official leaderboard | |
| RTE | 54.1% | #2 of 2 | Official leaderboard | |
| InfiniteBench/En.MC | 62.9% | #1 of 1 | Official leaderboard | |
| InfiniteBench/En.QA | 12.0 | #1 of 1 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 88% | #51 of 92 | Lab model card | |
| ChemBench | 47.4% | #20 of 24 | Official leaderboard | |
| GRE Quantitative | 154.0 | #5 of 6 | Lab model card | |
| GPQA Diamond | 34.7% | #197 of 229 | Epoch AI run | |
| DTBench | 51.9% | #159 of 180 | Official leaderboard | |
| MATH Level 5 | 11.7% | #84 of 94 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 2.50% | #191 of 212 | Epoch AI run | |
| Hungarian National High-School Exam | 55% | #2 of 2 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 61.6% | #26 of 46 | Official leaderboard | |
| HumanEval | 71.2% | #74 of 101 | Lab model card | |
| Bird-SQL | 49% | #10 of 13 | Official leaderboard | |
| SWE-bench | 1.96% | #7 of 9 | Official leaderboard | |
| SWE-bench Lite | 3.00% | #13 of 15 | Official leaderboard | |
| SWE-bench Verified | 4.40% | #58 of 60 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AgentBoard | 48.9% | #1 of 4 | Official leaderboard | |
| AgentBench | 2.49 | #4 of 5 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBE (Multistate Bar Examination) | 76.5% | #2 of 4 | Lab model card | |
| MedQA | 65.2% | #39 of 71 | Official leaderboard | |
| USMLE Step 1 (sample questions) | 68.9% | #1 of 1 | Lab model card | |
| USMLE Step 2 (sample questions) | 63.3% | #1 of 1 | Lab model card | |
| USMLE Step 3 (sample questions) | 67.2% | #1 of 1 | Lab model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SORRY-Bench | 7.05% | #2 of 23 | Official leaderboard | |
| Human Pathogen Capabilities Test | 23.6% | #82 of 86 | Official leaderboard | |
| Combined HHH evals (Anthropic) | 86% | #1 of 1 | Lab model card | |
| Harmfulness scores on held-out prompts (Anthropic) | 1.20% | #1 of 1 | Lab model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 48.9 | #39 of 71 | Official leaderboard |
Its lab's numbers put it at 121.2, 4.1 points above its ECI
The Self-Reported Index fits one score to the 6 capability results Anthropic reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 116.8 to 125.5. How we compute it.
- ARC-Challenge 91
- GSM8K 88
- TriviaQA 87.5
- MMLU 78.5
- HumanEval 71.2
- TruthfulQA 69
Its lab's number next to a leaderboard
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Anthropic's models over time
Anthropic's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 13 of them aren't in the launch post.
Model Card and Evaluations for Claude Models Jul 11, 2023
Capabilities
- FLORES-200
- HumanEval 71.2
- GSM8K 88
- MMLU 78.5
- TriviaQA 87.5
- QuALITY 83.2
- ARC-Challenge 91
- RACE-H 88.3
- GRE Verbal 165
- GRE Quantitative 154
- GRE Analytical Writing 5
- MBE (Multistate Bar Examination) 76.5
- USMLE Step 1 (sample questions) 68.9
- USMLE Step 2 (sample questions) 63.3
- USMLE Step 3 (sample questions) 67.2
Hallucination
- TruthfulQA 69