HarmBench
TML-Interaction-Small 99% leads since May 2026 Models answer harmful behavior requests, including copyright and contextual cases, with and without automated red-team attacks. A classifier judges whether the response complies.
Scores depend on the attack method and judge model, so labs differ.
- Released
- February 2024
- Kind
- Jailbreaks and prompt injection
- Level
- Everyday
- Scored
- Graded by a model
- Answers
- Open-ended
- Tasks
- All public
- Strongest model tested
-
DeepSeek-R1 #129 on the capability index - Last in a launch post
- May 2026, dropped after it saturated
- Last model added
- May 2026
In the authors’ words
Automated red teaming holds substantial promise for uncovering and mitigating the risks associated with the malicious use of large language models (LLMs), yet the field lacks a standardized evaluation framework to rigorously assess new methods. To address this issue, we introduce HarmBench, a standardized evaluation framework for automated red teaming. We identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. Using HarmBench, we conduct a large-scale comparison of 18 red teaming methods and 33 target LLMs and defenses, yielding novel insights. We also introduce a highly efficient adversarial training method that greatly enhances LLM robustness across a wide range of attacks, demonstrating how HarmBench enables codevelopment of attacks and defenses. We open source HarmBench at https://github.com/centerforaisafety/HarmBench.
TML-Interaction-Small leads with 99% among models released by May 2026
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in May 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- Claude Fable 5.2
-
Gemini 4 Carbon - GPT-6 Astra
- Claude Opus 5.5
- and 16 more
- 1 TML-Interaction-Small 99%
- 2
Olmo 3.1 32B Think 89.7% - 3 Olmo 3 32B Think 73.5%
- 4 DeepSeek-R1 35%
- 5
Apertus 70B 31.9%
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | TML-Interaction-Small | 99% | May 2026 | Lab model card | |
| 2 | Olmo 3.1 32B Think | 89.7% | Dec 2025 | Lab model card | |
| 3 | Olmo 3 32B Think | 73.5% | Dec 2025 | Lab model card | |
| 4 | DeepSeek-R1 | 35% | Jan 2025 | Lab model card | |
| 5 | Apertus 70B | 31.9% | Sep 2025 | Lab model card |
3 models have held number one in labs’ own numbers since its release in February 2024
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Labs’ model cards
Ai2’s own number for Olmo 3.1 32B Think passed 85% in December 2025
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Labs’ model cards
Labs headline it less often now, last in May 2026
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| May 11, 2026 | TML-Interaction-Small | 99 | HarmBench refusal rate, text |
5 system cards report it, from 4 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Jan 5, 2026 |  Falcon H1R 7B technical report | HarmBench safe rate, CoT+Answer, three subsets with no overall: contextual 98.0, copyright 100.0, standard 99.0 (Table 10) | Safety | |
| Dec 15, 2025 | Olmo 3 32B Think technical report | 73.5 | HarmBench, Final Think 3.0 column (Table 54) | Safety |
| Dec 15, 2025 | Olmo 3.1 32B Think technical report | 89.7 | HarmBench, Final Think 3.1 column (Table 54) | Safety |
| Sep 17, 2025 | Apertus 70B technical report | 31.9 | HarmBench, lower is better (Table 26). Direct Request 10.3, Human Jailbreaks 36.2 | Safety |
| Jan 20, 2025 | DeepSeek-R1 technical report | 35 | HarmBench* safety score for the pure model without the risk control system. 89.3 with the risk control system, GPT-4o judge (Table 9). Table 9 average 85.9 pure, 95.0 with risk control | Safety |
More in Safety and alignment
- FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
- StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Inkling 98.6%
- Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
- DistressBench In use Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics.
Muse Spark 1.2 88.4% - Lakera B3 (Backbone Breaker Benchmark) In use Crowdsourced prompt-injection attacks hit backbone LLMs inside agent threat snapshots at three defense levels. The score is attack resistance, 100 minus the board's vulnerability percentage. Claude Opus 4.5 89%
- IHEval In use Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction. GPT-4o 70%