Benchmark Almanac

HELM Safety

GPT-5 nano 98.6 leads since August 2025
Not in model cards

Stanford CRFM runs five safety benchmarks over six risk categories, such as violence, fraud and discrimination, and reports one leaderboard.

Version 1.0 evaluated 24 models at launch.

Made by
Stanford CRFM
Released
November 2024
Kind
Refusing harmful requests
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Strongest model tested
Gemini 3 Pro #55 on the capability index
Last model added
November 2025
Scores refreshed
November 12th, 2026
81
Entries on its own leaderboard, 75 models we track

GPT-5 nano leads with 98.6 among models released by November 2025

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in November 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

81 entries. Scroll sideways for the rest.

  1. 98.6 GPT-5 nano
  2. 98.2 o3
  3. 98.1 gpt-oss-120b
  4. 98.1 Claude Sonnet 4
  5. 98.1 GPT-5
  6. 98.0 GPT-5 mini
  7. 98.0 Kimi K2
  8. 97.7 Claude 3.5 Sonnet
  9. 97.6 o1
  10. 97.3 o4-mini
  11. 97.1 Claude Sonnet 4.5
  12. 96.9 Claude Opus 4
  13. 96.8 Claude 3 Opus
  14. 96.8 GPT-5.1
  15. 96.5 GPT-4.5
  16. 96.4 gpt-oss-20b
  17. 96.3 GPT-4.1
  18. 96.2 o3-mini
  19. 96.1 GPT-4 Turbo
  20. 95.5 o1-mini
  21. 95.5 Claude Haiku 4.5
  22. 94.9 GPT-4.1 mini
  23. 94.8 Qwen3-235B-A22B-Instruct
  24. 94.8 palmyra-fin
  25. 94.8 Qwen3-Next-80B-A3B
  26. 94.6 GPT-4o
  27. 94.5 Claude 3.7 Sonnet
  28. 94.3 Claude 3 Sonnet
  29. 93.8 GPT-4.1 nano
  30. 93.3 Qwen2-72B
  31. 93.3 Palmyra X4
  32. 93.1 Qwen2.5-72B
  33. 93.0 GPT-4o mini
  34. 92.7 Gemini 1.5 Flash
  35. 92.6 Palmyra X5
  36. 92.5 Gemini 3 Pro
  37. 92.5 Gemini 1.5 Pro
  38. 91.4 Gemini 2.5 Pro
  39. 91.4 Granite 4.0 H Small
  40. 91.2 Gemini 2.5 Flash
  41. 91.0 Gemini 2.5 Flash-Lite
  42. 91.0 Gemini 2.0 Flash
  43. 90.8 Gemini 2.0 Flash-Lite
  44. 90.6 Llama 4 Maverick
  45. 90.5 Gemini 2.0 Pro
  46. 90.5 Granite 3.3 8B Instruct
  47. 90.2 Grok-3 mini
  48. 90.1 GLM-4.5-Air
  49. 90.0 Qwen3-235B-A22B
  50. 89.9 Qwen2.5-7B
  51. 89.7 Llama 3.1-405B
  52. 89.7 OLMo 2 32B Instruct
  53. 89.5 Llama 3-70B
  54. 89.4 DeepSeek-R1
  55. 88.6 Qwen1.5-72B
  56. 88.6 Llama 3-8B
  57. 87.8 Claude 3 Haiku
  58. 87.3 Llama 4 Scout
  59. 87.3 granite-4.0-micro
  60. 87.3 DeepSeek LLM 67B
  61. 87.2 DeepSeek-V3
  62. 87.0 marin-8b-instruct
  63. 86.2 Llama 3.1-8B
  64. 86.0 Command R+
  65. 85.7 palmyra-med
  66. 85.5 Grok 3
  67. 85.3 GPT-3.5 Turbo
  68. 85.1 Mixtral 8x22B
  69. 84.7 Mistral Small 3
  70. 84.5 Llama 3.1-70B
  71. 83.6 Grok 4
  72. 83.5 GPT-3.5 Turbo
  73. 81.4 Mixtral 8x7B
  74. 81.4 GPT-3.5 Turbo
  75. 80.9 Command R (03-2024)
  76. 77.3 olmo-2-1124-13b-instruct
  77. 73.0 Mistral 7B v0.3
  78. 72.9 olmo-2-1124-7b-instruct
  79. 70.1 olmoe-1b-7b-0125-instruct
  80. 62.8 DBRX
  81. 52.5 Mistral-7B-Instruct-v0.1
  1. 1 GPT-5 nano 98.6
  2. 2 o3 98.2
  3. 3 gpt-oss-120b 98.1
  4. 4 Claude Sonnet 4 thinking 10k 98.1
  5. 5 GPT-5 98.1
  6. 6 GPT-5 mini 98.0
  7. 7 Kimi K2 Jul 2025 98.0
  8. 8 Claude 3.5 Sonnet 97.7
  9. 9 o1 97.6
  10. 10 o4-mini 97.3
  11. 11 Claude Sonnet 4.5 97.1
  12. 12 Claude Opus 4 thinking 10k 96.9
  13. 13 Claude 3 Opus 96.8
  14. 14 GPT-5.1 96.8
  15. 15 GPT-4.5 96.5
  16. 16 gpt-oss-20b 96.4
  17. 17 GPT-4.1 96.3
  18. 18 o3-mini 96.2
  19. 19 GPT-4 Turbo Apr 2024 96.1
  20. 20 o1-mini 95.5
  21. 21 Claude Haiku 4.5 95.5
  22. 22 GPT-4.1 mini 94.9
  23. 23 Qwen3-235B-A22B-Instruct Jul 2025 94.8
  24. 24 palmyra-fin 94.8
  25. 25 Qwen3-Next-80B-A3B thinking 94.8
  26. 26 GPT-4o May 2024 94.6
  27. 27 Claude 3.7 Sonnet 94.5
  28. 28 Claude 3 Sonnet 94.3
  29. 29 GPT-4.1 nano 93.8
  30. 30 Qwen2-72B 93.3
  31. 31 Palmyra X4 93.3
  32. 32 Qwen2.5-72B 93.1
  33. 33 GPT-4o mini 93.0
  34. 34 Gemini 1.5 Flash May 2024 92.7
  35. 35 Palmyra X5 92.6
  36. 36 Gemini 3 Pro 92.5
  37. 37 Gemini 1.5 Pro May 2024 92.5
  38. 38 Gemini 2.5 Pro Jun 2025 91.4
  39. 39 Granite 4.0 H Small with guardian 91.4
  40. 40 Gemini 2.5 Flash Sep 2025 91.2
  41. 41 Gemini 2.5 Flash-Lite Sep 2025 91.0
  42. 42 Gemini 2.0 Flash Feb 2025 91.0
  43. 43 Gemini 2.0 Flash-Lite 90.8
  44. 44 Llama 4 Maverick 90.6
  45. 45 Gemini 2.0 Pro 90.5
  46. 46 Granite 3.3 8B Instruct 90.5
  47. 47 Grok-3 mini 90.2
  48. 48 GLM-4.5-Air 90.1
  49. 49 Qwen3-235B-A22B 90.0
  50. 50 Qwen2.5-7B 89.9
  51. 51 Llama 3.1-405B 89.7
  52. 52 OLMo 2 32B Instruct 89.7
  53. 53 Llama 3-70B 89.5
  54. 54 DeepSeek-R1 May 2025 89.4
  55. 55 Qwen1.5-72B 88.6
  56. 56 Llama 3-8B 88.6
  57. 57 Claude 3 Haiku 87.8
  58. 58 Llama 4 Scout 87.3
  59. 59 granite-4.0-micro with guardian 87.3
  60. 60 DeepSeek LLM 67B 87.3
  61. 61 DeepSeek-V3 Mar 2025 87.2
  62. 62 marin-8b-instruct 87.0
  63. 63 Llama 3.1-8B 86.2
  64. 64 Command R+ 86.0
  65. 65 palmyra-med 85.7
  66. 66 Grok 3 85.5
  67. 67 GPT-3.5 Turbo Jun 2023 85.3
  68. 68 Mixtral 8x22B 85.1
  69. 69 Mistral Small 3 84.7
  70. 70 Llama 3.1-70B 84.5
  71. 71 Grok 4 83.6
  72. 72 GPT-3.5 Turbo Nov 2023 83.5
  73. 73 Mixtral 8x7B 81.4
  74. 74 GPT-3.5 Turbo Jan 2024 81.4
  75. 75 Command R (03-2024) 80.9
  76. 76 olmo-2-1124-13b-instruct 77.3
  77. 77 Mistral 7B v0.3 73.0
  78. 78 olmo-2-1124-7b-instruct 72.9
  79. 79 olmoe-1b-7b-0125-instruct 70.1
  80. 80 DBRX 62.8
  81. 81 Mistral-7B-Instruct-v0.1 52.5

Source: Official HELM Safety leaderboard

#EntryScoreSettingDateSource
1 GPT-5 nano 98.6 Aug 2025 Official leaderboard
2 o3 98.2 Apr 2025 Official leaderboard
3 gpt-oss-120b 98.1 Aug 2025 Official leaderboard
4 Claude Sonnet 4 98.1 thinking 10k May 2025 Official leaderboard
5 GPT-5 98.1 Aug 2025 Official leaderboard
6 GPT-5 mini 98.0 Aug 2025 Official leaderboard
7 Kimi K2 Jul 2025 98.0 Jul 2025 Official leaderboard
8 Claude 3.5 Sonnet 97.7 Jun 2024 Official leaderboard
9 o1 97.6 Dec 2024 Official leaderboard
10 o4-mini 97.3 Apr 2025 Official leaderboard
11 Claude Sonnet 4.5 97.1 Sep 2025 Official leaderboard
12 Claude Opus 4 96.9 thinking 10k May 2025 Official leaderboard
13 Claude 3 Opus 96.8 Feb 2024 Official leaderboard
14 GPT-5.1 96.8 Nov 2025 Official leaderboard
15 GPT-4.5 96.5 Feb 2025 Official leaderboard
16 gpt-oss-20b 96.4 Aug 2025 Official leaderboard
17 GPT-4.1 96.3 Apr 2025 Official leaderboard
18 o3-mini 96.2 Jan 2025 Official leaderboard
19 GPT-4 Turbo Apr 2024 96.1 Apr 2024 Official leaderboard
20 o1-mini 95.5 Sep 2024 Official leaderboard
21 Claude Haiku 4.5 95.5 Oct 2025 Official leaderboard
22 GPT-4.1 mini 94.9 Apr 2025 Official leaderboard
23 Qwen3-235B-A22B-Instruct Jul 2025 94.8 Jul 2025 Official leaderboard
24 palmyra-fin 94.8 Official leaderboard
25 Qwen3-Next-80B-A3B 94.8 thinking Sep 2025 Official leaderboard
26 GPT-4o May 2024 94.6 May 2024 Official leaderboard
27 Claude 3.7 Sonnet 94.5 Feb 2025 Official leaderboard
28 Claude 3 Sonnet 94.3 Feb 2024 Official leaderboard
29 GPT-4.1 nano 93.8 Apr 2025 Official leaderboard
30 Qwen2-72B 93.3 Jun 2024 Official leaderboard
31 Palmyra X4 93.3 Oct 2024 Official leaderboard
32 Qwen2.5-72B 93.1 Sep 2024 Official leaderboard
33 GPT-4o mini 93.0 Jul 2024 Official leaderboard
34 Gemini 1.5 Flash May 2024 92.7 May 2024 Official leaderboard
35 Palmyra X5 92.6 Apr 2025 Official leaderboard
36 Gemini 3 Pro 92.5 Nov 2025 Official leaderboard
37 Gemini 1.5 Pro May 2024 92.5 May 2024 Official leaderboard
38 Gemini 2.5 Pro Jun 2025 91.4 Jun 2025 Official leaderboard
39 Granite 4.0 H Small 91.4 with guardian Oct 2025 Official leaderboard
40 Gemini 2.5 Flash Sep 2025 91.2 Sep 2025 Official leaderboard
41 Gemini 2.5 Flash-Lite Sep 2025 91.0 Sep 2025 Official leaderboard
42 Gemini 2.0 Flash Feb 2025 91.0 Feb 2025 Official leaderboard
43 Gemini 2.0 Flash-Lite 90.8 Feb 2025 Official leaderboard
44 Llama 4 Maverick 90.6 Apr 2025 Official leaderboard
45 Gemini 2.0 Pro 90.5 Feb 2025 Official leaderboard
46 Granite 3.3 8B Instruct 90.5 Apr 2025 Official leaderboard
47 Grok-3 mini 90.2 Apr 2025 Official leaderboard
48 GLM-4.5-Air 90.1 Jul 2025 Official leaderboard
49 Qwen3-235B-A22B 90.0 Apr 2025 Official leaderboard
50 Qwen2.5-7B 89.9 Sep 2024 Official leaderboard
51 Llama 3.1-405B 89.7 Jul 2024 Official leaderboard
52 OLMo 2 32B Instruct 89.7 Mar 2025 Official leaderboard
53 Llama 3-70B 89.5 Apr 2024 Official leaderboard
54 DeepSeek-R1 May 2025 89.4 May 2025 Official leaderboard
55 Qwen1.5-72B 88.6 Feb 2024 Official leaderboard
56 Llama 3-8B 88.6 Apr 2024 Official leaderboard
57 Claude 3 Haiku 87.8 Mar 2024 Official leaderboard
58 Llama 4 Scout 87.3 Apr 2025 Official leaderboard
59 granite-4.0-micro 87.3 with guardian Oct 2025 Official leaderboard
60 DeepSeek LLM 67B 87.3 Nov 2023 Official leaderboard
61 DeepSeek-V3 Mar 2025 87.2 Mar 2025 Official leaderboard
62 marin-8b-instruct 87.0 Official leaderboard
63 Llama 3.1-8B 86.2 Jul 2024 Official leaderboard
64 Command R+ 86.0 Aug 2024 Official leaderboard
65 palmyra-med 85.7 Official leaderboard
66 Grok 3 85.5 Apr 2025 Official leaderboard
67 GPT-3.5 Turbo Jun 2023 85.3 Jun 2023 Official leaderboard
68 Mixtral 8x22B 85.1 Apr 2024 Official leaderboard
69 Mistral Small 3 84.7 Jan 2025 Official leaderboard
70 Llama 3.1-70B 84.5 Jul 2024 Official leaderboard
71 Grok 4 83.6 Jul 2025 Official leaderboard
72 GPT-3.5 Turbo Nov 2023 83.5 Nov 2023 Official leaderboard
73 Mixtral 8x7B 81.4 Dec 2023 Official leaderboard
74 GPT-3.5 Turbo Jan 2024 81.4 Jan 2024 Official leaderboard
75 Command R (03-2024) 80.9 Mar 2024 Official leaderboard
76 olmo-2-1124-13b-instruct 77.3 Official leaderboard
77 Mistral 7B v0.3 73.0 Sep 2023 Official leaderboard
78 olmo-2-1124-7b-instruct 72.9 Official leaderboard
79 olmoe-1b-7b-0125-instruct 70.1 Official leaderboard
80 DBRX 62.8 Mar 2024 Official leaderboard
81 Mistral-7B-Instruct-v0.1 52.5 Sep 2023 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Claude Sonnet 4
thinking 10k
98.1
Claude Opus 4
thinking 10k
96.9
Granite 4.0 H Small
with guardi…
91.4
DeepSeek-R1
reasoning h…
89.4
granite-4.0-micro
with guardi…
87.3

Source: Official HELM Safety leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-5 nano 98.6 257
o3 98.2 146
gpt-oss-120b 98.1 270
Claude Sonnet 4 thinking 10k 98.1 111
GPT-5 98.1 208
GPT-5 mini 98.0 291
Kimi K2 Jul 2025 98.0 136
Claude 3.5 Sonnet 97.7 85
o1 97.6 108
Claude Sonnet 4 97.4 113
o4-mini 97.3 110
Claude Sonnet 4.5 97.1 120
Claude Opus 4 thinking 10k 96.9 115
Claude Opus 4 96.8 113
Claude 3 Opus 96.8 93
GPT-5.1 96.8 223
GPT-4.5 96.5 59
gpt-oss-20b 96.4 538
GPT-4.1 96.3 82
o3-mini 96.2 91
GPT-4 Turbo Apr 2024 96.1 106
o1-mini 95.5 117
Claude Haiku 4.5 95.5 110
GPT-4.1 mini 94.9 70
Qwen3-235B-A22B-Instruct Jul 2025 94.8 208
Qwen3-Next-80B-A3B thinking 94.8
GPT-4o May 2024 94.6 108
Claude 3.7 Sonnet 94.5 118
Claude 3 Sonnet 94.3 70
GPT-4.1 nano 93.8 59
Qwen2-72B 93.3 153
Palmyra X4 93.3 178
Qwen2.5-72B 93.1 172
GPT-4o mini 93.0 83
Gemini 1.5 Flash May 2024 92.7
Palmyra X5 92.6
Gemini 3 Pro 92.5
Gemini 1.5 Pro May 2024 92.5
Gemini 2.5 Pro Jun 2025 91.4
Granite 4.0 H Small with guardian 91.4 108
Gemini 2.5 Flash Sep 2025 91.2
Granite 4.0 H Small 91.2 202
Gemini 2.5 Flash-Lite Sep 2025 91.0
Gemini 2.0 Flash Feb 2025 91.0
Gemini 2.0 Flash-Lite 90.8
Llama 4 Maverick 90.6 119
Gemini 2.0 Pro 90.5
Granite 3.3 8B Instruct 90.5 104
Grok-3 mini 90.2 465
GLM-4.5-Air 90.1 1,192
Qwen3-235B-A22B 90.0 854
Qwen2.5-7B 89.9 185
Llama 3.1-405B 89.7 139
OLMo 2 32B Instruct 89.7 159
Llama 3-70B 89.5 140
DeepSeek-R1 May 2025 89.4
Qwen1.5-72B 88.6 169
Llama 3-8B 88.6 108
Claude 3 Haiku 87.8 82
Llama 4 Scout 87.3 161
granite-4.0-micro with guardian 87.3 150
DeepSeek LLM 67B 87.3 108
DeepSeek-V3 Mar 2025 87.2
DeepSeek-R1 May 2025 reasoning hidden 86.8
DeepSeek-R1 May 2025 86.5
Llama 3.1-8B 86.2 131
Command R+ 86.0
Grok 3 85.5 420
granite-4.0-micro 85.5 186
GPT-3.5 Turbo Jun 2023 85.3 93
Mixtral 8x22B 85.1 177
Mistral Small 3 84.7 254
Llama 3.1-70B 84.5 183
Grok 4 83.6 597
GPT-3.5 Turbo Nov 2023 83.5 66
Mixtral 8x7B 81.4 183
GPT-3.5 Turbo Jan 2024 81.4 84
Command R (03-2024) 80.9
Mistral 7B v0.3 73.0 231
DBRX 62.8 217
Mistral-7B-Instruct-v0.1 52.5 176

3 models have held number one on the official leaderboard since its release in November 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic OpenAI
Own leaderboard Not tested since Nov 2025
GPT-5 nano 98.6

Source: Official HELM Safety leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
50.0 60.0 70.0 80.0 90.0 100.0 2023 2024 2025 2026 50.0 60.0 70.0 80.0 90.0 100.0 2023 2024 2025 2026

Source: Official HELM Safety leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Safety and alignment

All 465 in Safety and alignment
  • FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
  • StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Inkling 98.6%
  • Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
  • DistressBench In use Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. Muse Spark 1.2 88.4%
  • Lakera B3 (Backbone Breaker Benchmark) In use Crowdsourced prompt-injection attacks hit backbone LLMs inside agent threat snapshots at three defense levels. The score is attack resistance, 100 minus the board's vulnerability percentage. Claude Opus 4.5 89%
  • IHEval In use Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction. GPT-4o 70%