Benchmark Almanac

Tulu 3 Safety Eval

OLMo 2 1B Instruct 87.6% leads since April 2025
Fading

Models answer harmful, jailbreak and benign prompts from several safety benchmarks in Ai2's safety-eval suite. A WildGuard classifier judges the outputs, and the score is an average.

An aggregate used in the Tulu 3 release. The member benchmarks include WildGuardTest, HarmBench and XSTest. Released month taken from the Tulu 3 release.

Made by
Ai2
Released
November 2024
Kind
Jailbreaks and prompt injection
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Last in a launch post
April 2026, dropped after it saturated
Last model added
April 2026
6
Models scored in lab cards
6
Launch posts list it, from 2 labs
5
System cards report it, from 1 lab

OLMo 2 1B Instruct leads with 87.6% among models released by April 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in April 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 87.6 OLMo 2 1B Instruct
  2. 86.7 Tulu 3 405B
  3. 85.9 OLMo 2 32B Instruct
  4. 83.6 Olmo 3.1 32B Think
  5. 78.2 Granite 4.1 30B
  6. 68.8 Olmo 3 32B Think
  1. 1 OLMo 2 1B Instruct 87.6%
  2. 2 Tulu 3 405B 86.7%
  3. 3 OLMo 2 32B Instruct 85.9%
  4. 4 Olmo 3.1 32B Think 83.6%
  5. 5 Granite 4.1 30B 78.2%
  6. 6 Olmo 3 32B Think 68.8%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 OLMo 2 1B Instruct 87.6% Apr 2025 Lab model card
2 Tulu 3 405B 86.7% Jan 2025 Lab model card
3 OLMo 2 32B Instruct 85.9% Mar 2025 Lab model card
4 Olmo 3.1 32B Think 83.6% Dec 2025 Lab model card
5 Granite 4.1 30B 78.2% Apr 2026 Lab model card
6 Olmo 3 32B Think 68.8% Nov 2025 Lab model card

2 models have held number one in labs’ own numbers since its release in November 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Ai2
Lab claims Passed 85% in Nov 2024, by Tulu 3 405B
OLMo 2 1B Instruct 87.6%

Source: Labs’ model cards

Ai2’s own number for Tulu 3 405B passed 85% in November 2024

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) OLMo 2 1B Instruct Tulu 3 405B 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) OLMo 2 1B Instruct

Source: Labs’ model cards

Labs headline it less often now, last in April 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Ai2
IBM
DateLaunch postScoreAs the post writes it
Apr 29, 2026 Granite 4.1 30B 78.2 Tulu3 Safety Eval Avg
Dec 12, 2025 Olmo 3.1 32B Think 83.6 Safety (Ai2 safety suite average)
Nov 20, 2025 Olmo 3 32B Think 68.8 Safety (Ai2 safety suite average)
Apr 29, 2025 OLMo 2 1B Instruct 87.6 Safety (Ai2 safety suite average)
Mar 13, 2025 OLMo 2 32B Instruct 85.9 Safety (Ai2 safety suite average)
Jan 30, 2025 Tulu 3 405B 86.7 Safety (6 task avg.)

5 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Dec 15, 2025 Olmo 3 32B Think technical report 68.8 Safety, average of the Ai2 safety suite, Final Think 3.0 column (Table 14) Safety
Dec 15, 2025 Olmo 3.1 32B Think technical report 83.6 Safety, average of the Ai2 safety suite (Table 14) Safety
Oct 8, 2025 OLMo 2 1B Instruct technical report 87.6 Safety (Table 7, Table 23) Safety
Dec 31, 2024 OLMo 2 32B Instruct technical report 85.9 Safety, Tulu 3 Safety average (Tables 7 and 16) Safety
Nov 22, 2024 Tulu 3 405B technical report 86.7 Safety (6 task avg.), RLVR (Table 4) Safety

More in Safety and alignment

All 465 in Safety and alignment
  • FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
  • StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Inkling 98.6%
  • Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
  • DistressBench In use Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. Muse Spark 1.2 88.4%
  • Lakera B3 (Backbone Breaker Benchmark) In use Crowdsourced prompt-injection attacks hit backbone LLMs inside agent threat snapshots at three defense levels. The score is attack resistance, 100 minus the board's vulnerability percentage. Claude Opus 4.5 89%
  • IHEval In use Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction. GPT-4o 70%