Claude 2.1
Claude 2.1 ranks first on SORRY-Bench.
17
Benchmarks with a score
1
Ranked first, out of five or more models
1
In the top three
Anthropic Nov 2023
Claude 2.1
Closed weights
No Epoch Capabilities Index score yet
- GPQA Diamond 33% #202 /229
- Situational Awareness Dataset (SAD) 46.2% #10 /17
- LLM-AggreFact 71.1% #10 /10
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AlpacaEval 2.0 | 25.3% | #21 of 47 | Official leaderboard | |
| EQ-Bench v2 | 74.0 | #39 of 85 | Official leaderboard | |
| LongBench-Chat | 8.03 | #1 of 2 | Official leaderboard | |
| AlpacaEval | 87.1% | #12 of 19 | Official leaderboard | |
| Arena-Hard-Auto | 22.8% | #30 of 38 | Official leaderboard | |
| LLM-AggreFact | 71.1% | #10 of 10 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GPQA Diamond | 33% | #202 of 229 | Epoch AI run | |
| DTBench | 51% | #161 of 180 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 1.94% | #196 of 212 | Epoch AI run | |
| ForecastBench | 0.17 | #77 of 82 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| NaturalCodeBench | 34.4% | #4 of 5 | Official leaderboard | |
| WeirdML | 7.06% | #112 of 112 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 64.4% | #41 of 71 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SORRY-Bench | 5.23% | #1 of 23 | Official leaderboard | |
| Situational Awareness Dataset (SAD) | 46.2% | #10 of 17 | Official leaderboard | |
| OR-Bench | 99.8% | #28 of 28 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 43.7 | #46 of 71 | Official leaderboard |