Benchmark Almanac

OLMo 2 1B Instruct

OLMo 2 1B Instruct ranks first on Tulu 3 Safety Eval.

Ai2 Released April 29th, 2025 Family OLMo 2 The lab's model card Compare with others
10
Benchmarks with a score
1
Ranked first, out of five or more models
1
In the top three
Ai2 Apr 2025

OLMo 2 1B Instruct

Open weights Small

94.7 #398 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • PopQA 12.9% #7 /7
  • IFEval 70.1% #125 /135
  • MMLU 40% #130 /131

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
TruthfulQA 48.7% #23 of 30 Lab model card
IFEval 70.1% #125 of 135 Lab model card
BIG-Bench Hard 35% #70 of 73 Lab model card
MMLU 40% #130 of 131 Lab model card
DROP 34.6% #41 of 41 Lab model card
AlpacaEval 2.0 9.10% #16 of 16 Lab model card
PopQA 12.9% #7 of 7 Lab model card

Math and science

BenchmarkScoreRankSource
GSM8K 68.3% #75 of 92 Lab model card
MATH 20.7% #93 of 99 Lab model card

Safety and security

BenchmarkScoreRankSource
Tulu 3 Safety Eval 87.6% #1 of 6 Lab model card

Its lab's numbers put it at 94.7, number 398 of 404

The Self-Reported Index fits one score to the 8 capability results Ai2 reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 89.9 to 99.6. How we compute it.

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Its system card

Every result in the longer documents the lab published with it.