Benchmark Almanac

StrongREJECT

Inkling 98.6% leads since July 2026
In use

Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is.

HF copy linked by Mistral is a third-party mirror (walledai).

Released
February 2024
Kind
Jailbreaks and prompt injection
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Measures
Propensity
Behavior
Harmful use
Language
English
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
In views
Content safety
Strongest model tested
GPT-5.2 #51 on the capability index
Last in a launch post
November 2026
Last model added
November 2026

In the authors’ words

Most jailbreak papers claim the jailbreaks they propose are highly effective, often boasting near-100% attack success rates. However, it is perhaps more common than not for jailbreak developers to substantially exaggerate the effectiveness of their jailbreaks. We suggest this problem arises because jailbreak researchers lack a standard, high-quality benchmark for evaluating jailbreak performance, leaving researchers to create their own. To create a benchmark, researchers must choose a dataset of forbidden prompts to which a victim model will respond, along with an evaluation method that scores the harmfulness of the victim model's responses. We show that existing benchmarks suffer from significant shortcomings and introduce the StrongREJECT benchmark to address these issues. StrongREJECT's dataset contains prompts that victim models must answer with specific, harmful information, while its automated evaluator measures the extent to which a response gives useful information to forbidden prompts. In doing so, the StrongREJECT evaluator achieves state-of-the-art agreement with human judgments of jailbreak effectiveness. Notably, we find that existing evaluation methods significantly overstate jailbreak effectiveness compared to human judgments and the StrongREJECT evaluator. We describe a surprising and novel phenomenon that explains this discrepancy: jailbreaks bypassing a victim model's safety fine-tuning tend to reduce its capabilities. Together, our findings underscore the need for researchers to use a high-quality benchmark, such as StrongREJECT, when developing new jailbreak attacks. We release the StrongREJECT code and data at https://strong-reject.readthedocs.io/en/latest/.

Abstract of A StrongREJECT for Empty Jailbreaks (arXiv:2402.10260, CC0)
20
Models scored in lab cards
4
Launch posts list it, from 1 lab
31
System cards report it, from 5 labs

Inkling leads with 98.6%

Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.

  1. 1 Inkling 98.6%
  2. 2 Inkling-Small 98.4%
  3. 3 codex-1 98%
  4. 4 Inkling 2 Small 97.8%
  5. 5 Inkling 2 Mini 97.8%
  6. 6 GPT-5.2 97.5%
  7. 7 o3 97%
  8. 8 GPT-5.1 96.7%
  9. 9 GPT-5.1-Codex-Max 96.7%
  10. 10 o4-mini 96%
  11. 11 GPT-5.2 Codex 93.3%
  12. 12 OpenAI deep research 93%
  13. 13 Olmo 3.1 32B Think 90.5%
  14. 14 o1-preview 84%
  15. 15 o1-mini 83%
  16. 16 Olmo 3 32B Think 80.8%
  17. 17 o3-mini 73%
  18. 18 o1 72%
  19. 19 OpenAI Computer-Using Agent (CUA) 66%
  20. 20 GPT-4.5 34%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Inkling 98.6% Jul 2026 Lab model card
2 Inkling-Small 98.4% Jul 2026 Lab model card
3 codex-1 98% May 2025 Lab model card
4 Inkling 2 Small 97.8% Nov 2026 Lab model card
5 Inkling 2 Mini 97.8% Nov 2026 Lab model card
6 GPT-5.2 97.5% Dec 2025 Lab model card
7 o3 97% Apr 2025 Lab model card
8 GPT-5.1 96.7% Nov 2025 Lab model card
9 GPT-5.1-Codex-Max 96.7% Nov 2025 Lab model card
10 o4-mini 96% Apr 2025 Lab model card
11 GPT-5.2 Codex 93.3% Dec 2025 Lab model card
12 OpenAI deep research 93% Feb 2025 Lab model card
13 Olmo 3.1 32B Think 90.5% Dec 2025 Lab model card
14 o1-preview 84% Sep 2024 Lab model card
15 o1-mini 83% Sep 2024 Lab model card
16 Olmo 3 32B Think 80.8% Dec 2025 Lab model card
17 o3-mini 73% Jan 2025 Lab model card
18 o1 72% Dec 2024 Lab model card
19 OpenAI Computer-Using Agent (CUA) 66% Jan 2025 Lab model card
20 GPT-4.5 34% Feb 2025 Lab model card

5 models have held number one in labs’ own numbers since its release in February 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Thinking Machines
Lab claims Passed 85% in Feb 2025, by OpenAI deep research
Inkling 98.6%

Source: Labs’ model cards

OpenAI’s own number for OpenAI deep research passed 85% in February 2025

That was 12 months after its release. By the same age, 21% of the benchmarks from 2024 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Inkling codex-1 OpenAI deep research o1-preview 0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Inkling OpenAI deep research

Source: Labs’ model cards

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out. OpenAI accounts for 20 of the 30 releases that named it since 2024.

Thinking Machines
DateLaunch postScoreAs the post writes it
Nov 20, 2026 Inkling 2 Mini 97.8 StrongREJECT (none)
Nov 5, 2026 Inkling 2 Small 97.8 StrongREJECT (none)
Jul 30, 2026 Inkling-Small 98.4 StrongREJECT
Jul 15, 2026 Inkling 98.6 StrongREJECT

31 system cards report it, from 5 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 20, 2026 Inkling 2 Mini model card 97.8 StrongREJECT (none) Safety
Sep 21, 2026 Grok 4.7 model card StrongReject, compliance (lower is better), high (8.1 table) Robustness
Aug 12, 2026 Grok 4.6 model card StrongReject, compliance (lower is better), high (9.1 table) Robustness
Jul 30, 2026 Inkling-Small model card 98.4 StrongREJECT (5. Evaluations table) Safety
Jul 15, 2026 Inkling model card 98.6 StrongREJECT (5. Evaluations table, effort 0.99) Safety
Dec 18, 2025 GPT-5.2 Codex system card addendum 93.3 StrongReject, not_unsafe 0.933 (Table 2) Robustness
Dec 15, 2025 Olmo 3 32B Think technical report 80.8 StrongReject, reversed so higher is safer, Final Think 3.0 column (Table 54) Safety
Dec 15, 2025 Olmo 3.1 32B Think technical report 90.5 StrongReject, reversed so higher is safer, Final Think 3.1 column (Table 54) Safety
Dec 11, 2025 GPT-5.2 system card addendum 97.5 StrongReject filtered, not_unsafe, gpt-5.2-thinking (Table 2) Robustness
Nov 18, 2025 GPT-5.1-Codex-Max system card 96.7 StrongReject, not_unsafe 0.967 (Table 2) Robustness
Nov 12, 2025 GPT-5.1 system card addendum 96.7 StrongReject, not_unsafe, gpt-5.1-thinking (Table 2) Robustness
Oct 27, 2025 GPT-5 system card addendum Jailbreak evaluations (StrongReject), not_unsafe, 4 categories, no overall score (Table 2) Robustness
Sep 15, 2025 GPT-5-Codex system card addendum StrongReject, not_unsafe, 4 categories from 0.992 to 0.997, no overall score (Table 2) Robustness
Aug 7, 2025 GPT-5 system card Jailbreak evaluations (StrongReject), not_unsafe, 4 categories, no overall score (Table 5) Robustness
Aug 7, 2025 GPT-5 nano system card StrongReject, gpt-5-thinking-nano, 4 categories, no overall score (Table 25) Robustness
Aug 7, 2025 GPT-5 mini system card StrongReject, gpt-5-thinking-mini, 4 categories, no overall score (Table 25) Robustness
Aug 5, 2025 gpt-oss-20b model card Jailbreak evaluations (StrongReject), not_unsafe, 4 categories, no overall score (Table 6) Robustness
Aug 5, 2025 gpt-oss-120b model card Jailbreak evaluations (StrongReject), not_unsafe, 4 categories, no overall score (Table 6) Robustness
Jul 17, 2025 ChatGPT agent system card StrongReject, not_unsafe, 4 categories from 0.987 to 0.996, no overall score (Table 3) Robustness
May 22, 2025 Claude Sonnet 4 system card StrongREJECT best score (jailbreak success rate, lower is better), extended thinking 2.24%. Top 3 average extended 1.38%, best score standard 6.71%, top 3 average standard 2.66% (Table 2.7.A) Robustness
May 22, 2025 Claude Opus 4 system card StrongREJECT best score (jailbreak success rate, lower is better), extended thinking 2.24%. Top 3 average extended 1.17%, best score standard 18.21%, top 3 average standard 7.14% (Table 2.7.A) Robustness
May 16, 2025 codex-1 system card addendum 98 StrongReject (goodness@0.1), 0.98 (Table 2) Robustness
Apr 16, 2025 o4-mini system card 96 StrongReject, not_unsafe (Table 3) Robustness
Apr 16, 2025 o3 system card 97 StrongReject, not_unsafe (Table 3) Robustness
Feb 27, 2025 GPT-4.5 system card 34 StrongReject goodness@0.1 (Table 3) Robustness
Feb 25, 2025 OpenAI deep research system card 93 StrongReject, not_unsafe accuracy over the full jailbreak set 0.93 (Table 5) Robustness
Jan 31, 2025 o3-mini system card 73 StrongReject goodness@0.1 (Table 2) Robustness
Jan 23, 2025 OpenAI Computer-Using Agent (CUA) system card 66 StrongREJECT, goodness@0.1 0.66 (Table 3) Robustness
Dec 5, 2024 o1 system card 72 StrongReject, goodness@0.1 (Table 23) Robustness
Sep 12, 2024 o1-preview system card 84 StrongReject, goodness@0.1 (Table 17) Robustness
Sep 12, 2024 o1-mini system card 83 StrongReject, goodness@0.1 (Table 17) Robustness

More in Safety and alignment

All 636 in Safety and alignment
  • FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
  • Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
  • Covert misbehavior (Thinking Machines) In use Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Inkling 2 Mini 8.30%
  • Destructive actions (Thinking Machines) In use Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Inkling 2 Small 4.70%
  • Misbehavior (Thinking Machines) In use Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Inkling 2 Mini 9.80%
  • DistressBench In use Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. Muse Spark 1.2 88.4%

Data audit

All 5 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Note Independent runs Only labs’ own numbers, so no one else has checked them. Scores from launch posts (4 scores) and system cards (19 scores).
  • Pass Models scored 20 models we track.
  • Pass Newest score The newest is from November 2026, for Inkling 2 Mini.
  • Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is GPT-5.2 (#51).
  • Note Headroom Saturated. The best is 98.6%, past 85% since February 2025.
Leaderboard import 1 note
  • Note Importer No public leaderboard to read.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check