OLMo-2-0325-32B-Instruct model card (Hugging Face) Mar 13, 2025
- AlpacaEval 2 LC 42.8
- BBH 70.6
- DROP 78
- GSM8k 87.6
- IFEval 85.6
- MATH 49.7
- MMLU 77.3
- Safety (Ai2 safety suite average) 85.9
- PopQA 37.5
- TruthQA 73.2
OLMo 2 32B Instruct
OLMo 2 32B Instruct places in the top three on 2 benchmarks.
24
Benchmarks with a score
0
Ranked first, out of five or more models
2
In the top three
Ai2 Mar 2025 OLMo 2 32B Instruct
Open weights Flagship
116.5
#359 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
- PopQA 37.5% #2 /7
- GPQA Main 28.7% #59 /60
- Misguided Attention 19.2% #45 /45
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TruthfulQA | 73.2% | #4 of 30 | Lab model card | |
| PopQA | 37.5% | #2 of 7 | Lab model card | |
| AGIEval | 68.3% | #15 of 40 | Lab model card | |
| IFEval | 85.6% | #77 of 135 | Lab model card | |
| BIG-Bench Hard | 70.6% | #47 of 73 | Lab model card | |
| MMLU | 77.3% | #89 of 131 | Lab model card | |
| DROP | 78% | #30 of 41 | Lab model card | |
| AlpacaEval 2.0 | 42.8% | #12 of 16 | Lab model card | |
| LMArena (Elo) | 1,252 | #262 of 335 | Official leaderboard | |
| SwitzerlandQA | 62.2% | #6 of 7 | Official leaderboard | |
| MMLU-Pro | 52.7% | #153 of 167 | Lab model card | |
| IFEval-OOD | 33.1% | #1 of 1 | Lab model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 87.6% | #53 of 92 | Lab model card | |
| MATH | 49.7% | #70 of 99 | Lab model card | |
| DeepMind Mathematics | 34.7% | #3 of 4 | Lab model card | |
| GPQA Diamond | 35.9% | #282 of 302 | Lab model card | |
| GPQA Main | 28.7% | #59 of 60 | Official leaderboard | |
| Misguided Attention | 19.2% | #45 of 45 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SnakeBench | 18 | #150 of 188 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Anthropic Red Team | 99.3% | #26 of 75 | Official leaderboard | |
| Tulu 3 Safety Eval | 85.9% | #3 of 6 | Lab model card | |
| SimpleSafetyTests | 98% | #47 of 75 | Official leaderboard | |
| HELM Safety | 89.7 | #51 of 75 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Capabilities | 47.5 | #56 of 60 | Official leaderboard |
Its lab's numbers put it at 116.5, number 359 of 404
The Self-Reported Index fits one score to the 11 capability results Ai2 reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 113.9 to 119.0. How we compute it.
- GSM8K 87.6
- IFEval 85.6
- DROP 78
- MMLU 77.3
- TruthfulQA 73.2
- BIG-Bench Hard 70.6
- AGIEval 68.3
- MMLU-Pro 52.7
- MATH 49.7
- AlpacaEval 2.0 42.8
- GPQA Diamond 35.9
Its lab's number next to a leaderboard
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.
2 OLMo 2 Furious Dec 31, 2024
Capabilities
- AlpacaEval 2.0 42.8
- BIG-Bench Hard 70.6
- DROP 78
- GSM8K 87.6
- IFEval 85.6
- MATH 49.7
- MMLU 77.3
- PopQA 37.5
- TruthfulQA 73.2
- AGIEval 68.3
- DeepMind Mathematics 34.7
- GPQA Diamond 35.9
- IFEval-OOD 33.1
- MMLU-Pro 52.7
Safety
- Tulu 3 Safety Eval 85.9