Benchmark Almanac

S-Eval

gpt-oss-120b 88.8% leads since August 2025
Not in model cards

Models respond to Chinese and English harmful prompts and attack variants. A trained safety judge measures the safe-response fraction and attack success rate.

The released benchmark contains 20,000 base risk prompts and 200,000 attack prompts. Safety score and attack success rate have opposite directions.

Released
May 2024
Kind
Refusing harmful requests
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Measures
Propensity
Behavior
Harmful use
Language
English, Chinese
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
In views
Content safety
Strongest model tested
DeepSeek-V3.2 #91 on the capability index
Last model added
December 2025
Scores refreshed
November 20th, 2026

In the authors’ words

Generative large language models (LLMs) have revolutionized natural language processing with their transformative and emergent capabilities. However, recent evidence indicates that LLMs can produce harmful content that violates social norms, raising significant concerns regarding the safety and ethical ramifications of deploying these advanced models. Thus, it is both critical and imperative to perform a rigorous and comprehensive safety evaluation of LLMs before deployment. Despite this need, owing to the extensiveness of LLM generation space, it still lacks a unified and standardized risk taxonomy to systematically reflect the LLM content safety, as well as automated safety assessment techniques to explore the potential risk efficiently. To bridge the striking gap, we propose S-Eval, a novel LLM-based automated Safety Evaluation framework with a newly defined comprehensive risk taxonomy. S-Eval incorporates two key components, i.e., an expert testing LLM Mt and a novel safety critique LLM Mc. Mt is responsible for automatically generating test cases in accordance with the proposed risk taxonomy. Mc can provide quantitative and explainable safety evaluations for better risk awareness of LLMs. In contrast to prior works, S-Eval is efficient and effective in test generation and safety evaluation. Moreover, S-Eval can be flexibly configured and adapted to the rapid evolution of LLMs and accompanying new safety threats, test generation methods and safety critique methods thanks to the LLM-based architecture. S-Eval has been deployed in our industrial partner for the automated safety evaluation of multiple LLMs serving millions of users, demonstrating its effectiveness in real-world scenarios. Our benchmark is publicly available at https://github.com/IS2Lab/S-Eval.

Abstract of S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models (arXiv:2405.14191, CC0)
36
Entries on its own leaderboard, 19 models we track

gpt-oss-120b leads with 88.8% among models released by December 2025 gpt-oss-120b leads with 88.8% among models released by December 2025

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. The board reports each language separately, so this is the plain mean of Chinese and English base risk safety scores. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in December 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 gpt-oss-120b 88.8%
  2. 2 gpt-oss-20b 85.7%
  3. 3 ErnieBot-4.0 83.7%
  4. 4 DeepSeek-V3.2 thinking 82.8%
  5. 5 Qwen3-235B-A22B-Thinking Jul 2025 80.8%
  6. 6 Qwen3-235B-A22B-Instruct Jul 2025 79.9%
  7. 7 Qwen3-Max-0923 79%
  8. 8 Qwen-72B-Chat 72.3%
  9. 9 Baichuan2-13B-Chat 72%
  10. 10 Qwen-14B 70%
  11. 11 DeepSeek-V3 (snapshot unspecified) 70%
  12. 12 Qwen2.5-32B-Instruct 65.2%
  13. 13 Qwen2.5-14B-Instruct 64.1%
  14. 14 Qwen-7B-Chat 59.5%
  15. 15 Qwen2.5-Max 59%
  16. 16 Mistral Small 3.2 59%
  17. 17 GPT-4-Turbo (snapshot unspecified) 58.9%
  18. 18 ChatGLM3-6B 58.7%
  19. 19 Gemma-7B-it 55.7%
  20. 20 Gemma-2B-it 54.9%
  21. 21 Doubao-Seed-1.6 thinking 54.4%
  22. 22 Doubao-1.5-Pro-32k 54%
  23. 23 Qwen-1.8B-Chat 54%
  24. 24 GPT-4o (snapshot unspecified) 53%
  25. 25 Qwen3-235B-A22B thinking 52.1%
  26. 26 Qwen3-14B thinking 52%
  27. 27 Gemini 1.0 Pro 47.9%
  28. 28 Yi-34B-Chat 43%
  29. 29 DeepSeek-R1 (snapshot unspecified) 38.5%
  30. 30 QwQ-32B 37.7%
  31. 31 Gemini-2.5-Pro (snapshot unspecified) 37.6%
  32. 32 Gemma 3 27B 34.8%
  33. 33 DeepSeek-R1-Distill-Qwen-14B 34.1%
  34. 34 DeepSeek-R1-Distill-Qwen-32B 34.1%
  35. 35 Qwen3-0.6B (Instruct) 24.6%
  36. 36 Qwen3-0.6B thinking 20.1%

Source: Official S-Eval leaderboard

#EntryScoreSettingDateSource
1 gpt-oss-120b 88.8% Aug 2025 Official leaderboard
2 gpt-oss-20b 85.7% Aug 2025 Official leaderboard
3 ErnieBot-4.0 83.7% Official leaderboard
4 DeepSeek-V3.2 82.8% thinking Dec 2025 Official leaderboard
5 Qwen3-235B-A22B-Thinking Jul 2025 80.8% Jul 2025 Official leaderboard
6 Qwen3-235B-A22B-Instruct Jul 2025 79.9% Jul 2025 Official leaderboard
7 Qwen3-Max-0923 79% Official leaderboard
8 Qwen-72B-Chat 72.3% Official leaderboard
9 Baichuan2-13B-Chat 72% Sep 2023 Official leaderboard
10 Qwen-14B 70% Sep 2023 Official leaderboard
11 DeepSeek-V3 (snapshot unspecified) 70% Official leaderboard
12 Qwen2.5-32B-Instruct 65.2% Official leaderboard
13 Qwen2.5-14B-Instruct 64.1% Official leaderboard
14 Qwen-7B-Chat 59.5% Official leaderboard
15 Qwen2.5-Max 59% Jan 2025 Official leaderboard
16 Mistral Small 3.2 59% Jun 2025 Official leaderboard
17 GPT-4-Turbo (snapshot unspecified) 58.9% Official leaderboard
18 ChatGLM3-6B 58.7% Oct 2023 Official leaderboard
19 Gemma-7B-it 55.7% Official leaderboard
20 Gemma-2B-it 54.9% Official leaderboard
21 Doubao-Seed-1.6 54.4% thinking Jun 2025 Official leaderboard
22 Doubao-1.5-Pro-32k 54% Official leaderboard
23 Qwen-1.8B-Chat 54% Official leaderboard
24 GPT-4o (snapshot unspecified) 53% Official leaderboard
25 Qwen3-235B-A22B 52.1% thinking Apr 2025 Official leaderboard
26 Qwen3-14B 52% thinking Apr 2025 Official leaderboard
27 Gemini 1.0 Pro 47.9% Dec 2023 Official leaderboard
28 Yi-34B-Chat 43% Official leaderboard
29 DeepSeek-R1 (snapshot unspecified) 38.5% Official leaderboard
30 QwQ-32B 37.7% Mar 2025 Official leaderboard
31 Gemini-2.5-Pro (snapshot unspecified) 37.6% Official leaderboard
32 Gemma 3 27B 34.8% Mar 2025 Official leaderboard
33 DeepSeek-R1-Distill-Qwen-14B 34.1% Jan 2025 Official leaderboard
34 DeepSeek-R1-Distill-Qwen-32B 34.1% Jan 2025 Official leaderboard
35 Qwen3-0.6B (Instruct) 24.6% Official leaderboard
36 Qwen3-0.6B 20.1% thinking Apr 2025 Official leaderboard
#ModelScoreSettingDateSource
1 gpt-oss-120b 88.8% Aug 2025 Official leaderboard
2 gpt-oss-20b 85.7% Aug 2025 Official leaderboard
3 DeepSeek-V3.2 82.8% thinking Dec 2025 Official leaderboard
4 Qwen3-235B-A22B-Thinking Jul 2025 80.8% Jul 2025 Official leaderboard
5 Qwen3-235B-A22B-Instruct Jul 2025 79.9% Jul 2025 Official leaderboard
6 Baichuan2-13B-Chat 72% Sep 2023 Official leaderboard
7 Qwen-14B 70% Sep 2023 Official leaderboard
8 Qwen2.5-Max 59% Jan 2025 Official leaderboard
9 Mistral Small 3.2 59% Jun 2025 Official leaderboard
10 ChatGLM3-6B 58.7% Oct 2023 Official leaderboard
11 Doubao-Seed-1.6 54.4% thinking Jun 2025 Official leaderboard
12 Qwen3-235B-A22B 52.1% thinking Apr 2025 Official leaderboard
13 Qwen3-14B 52% thinking Apr 2025 Official leaderboard
14 Gemini 1.0 Pro 47.9% Dec 2023 Official leaderboard
15 QwQ-32B 37.7% Mar 2025 Official leaderboard
16 Gemma 3 27B 34.8% Mar 2025 Official leaderboard
17 DeepSeek-R1-Distill-Qwen-14B 34.1% Jan 2025 Official leaderboard
18 DeepSeek-R1-Distill-Qwen-32B 34.1% Jan 2025 Official leaderboard
19 Qwen3-0.6B 20.1% thinking Apr 2025 Official leaderboard

Rankings by part

We read 42 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as Chinese, Chinese: 违法犯罪 and Chinese: 仇恨言论. gpt-oss-120b leads 13 of them, and 7 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.

Chinese

19 models we track have a score here.

  1. 1 gpt-oss-120b 91.2%
  2. 2 DeepSeek-V3.2 ↑1 89%
  3. 3 gpt-oss-20b ↓1 88.4%
  4. 4 Qwen3-235B-A22B-Instruct Jul 2025 ↑1 84.3%
  5. 5 Qwen3-235B-A22B-Thinking Jul 2025 ↓1 83.8%
  6. 6 Doubao-Seed-1.6 ↑5 66.8%
  7. 7 Baichuan2-13B-Chat ↓1 66.6%
  8. 8 Qwen-14B ↓1 66.5%
  9. 9 Qwen2.5-Max ↓1 64.3%
  10. 10 Qwen3-14B ↑3 62.2%

3 models have held number one on the official leaderboard since its release in May 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Other labs Qwen OpenAI
Own leaderboard Passed 85% in Aug 2025, by gpt-oss-120b
gpt-oss-120b 88.8%

Source: Official S-Eval leaderboard

Gpt-oss-120b on the official leaderboard passed 85% in August 2025

That was 15 months after its release. By the same age, 27% of the benchmarks from 2024 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) gpt-oss-120b Qwen3-235B-A22B-Thinking Baichuan2-13B-Chat 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) gpt-oss-120b Qwen3-235B-A22B-Thinking

Source: Official S-Eval leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Safety and alignment

All 636 in Safety and alignment
  • FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
  • StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Inkling 98.6%
  • Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
  • Covert misbehavior (Thinking Machines) In use Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Inkling 2 Mini 8.30%
  • Destructive actions (Thinking Machines) In use Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Inkling 2 Small 4.70%
  • Misbehavior (Thinking Machines) In use Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Inkling 2 Mini 9.80%

Data audit

All 8 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the official leaderboard (36 entries).
  • Pass Models scored 19 models we track, plus 17 other entries on its leaderboard.
  • Note Newest score The newest came 11 months ago, in December 2025, for DeepSeek-V3.2. That’s expected for a saturated test.
  • Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is DeepSeek-V3.2 (#91).
  • Note Headroom Saturated. The best is 88.8%, past 85% since August 2025.
Leaderboard import 3 pass
  • Pass Importer We read the board on s.alibaba.com: 21 rows for models we track and 17 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check