Benchmark Almanac

StereoSet

gpt2_ModelNSP_GPT2LM 71.9 tops its leaderboard
Not in model cards

Models choose between stereotypical, anti-stereotypical and unrelated completions in four domains: gender, profession, race and religion. Scores combine language modeling and stereotype preference.

The ideal stereotype score is 50.

Released
April 2020
Kind
Bias, fairness and politics
Level
Everyday
Scored
Checked automatically
Answers
Multiple choice
Tasks
Public, plus a private test set

In the authors’ words

A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or Asians are bad drivers. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases. In order to assess the adverse effects of these models, it is important to quantify the bias captured in them. Existing literature on quantifying bias evaluates pretrained language models on a small set of artificially constructed bias-assessing sentences. We present StereoSet, a large-scale natural dataset in English to measure stereotypical biases in four domains: gender, profession, race, and religion. We evaluate popular models like BERT, GPT-2, RoBERTa, and XLNet on our dataset and show that these models exhibit strong stereotypical biases. We also present a leaderboard with a hidden test set to track the bias of future language models at https://stereoset.mit.edu

Abstract of StereoSet: Measuring stereotypical bias in pretrained language models (arXiv:2004.09456, CC0)
11
Entries on its own leaderboard, 0 models we track

gpt2_ModelNSP_GPT2LM tops the leaderboard with 71.9

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

  1. 71.9 gpt2_ModelNSP_GPT2LM
  2. 71.5 gpt2-medium_ModelNSP_GPT2LM
  3. 71.2 xlnet-large-cased_ModelNSP_XLNetLM
  4. 69.4 bert-base-cased_BertNextSentence_BertLM
  5. 69.2 bert-large-cased_BertNextSentence_BertLM
  6. 69.0 EnsembleModel_
  7. 68.8 roberta-base_ModelNSP_RoBERTaLM
  8. 67.8 gpt2-large_ModelNSP_GPT2LM
  9. 67.4 roberta-large_ModelNSP_RoBERTaLM
  10. 61.6 xlnet-base-cased_ModelNSP_XLNetLM
  11. 49.6 SentimentModel
  1. 1 gpt2_ModelNSP_GPT2LM 71.9
  2. 2 gpt2-medium_ModelNSP_GPT2LM 71.5
  3. 3 xlnet-large-cased_ModelNSP_XLNetLM 71.2
  4. 4 bert-base-cased_BertNextSentence_BertLM 69.4
  5. 5 bert-large-cased_BertNextSentence_BertLM 69.2
  6. 6 EnsembleModel_ 69.0
  7. 7 roberta-base_ModelNSP_RoBERTaLM 68.8
  8. 8 gpt2-large_ModelNSP_GPT2LM 67.8
  9. 9 roberta-large_ModelNSP_RoBERTaLM 67.4
  10. 10 xlnet-base-cased_ModelNSP_XLNetLM 61.6
  11. 11 SentimentModel 49.6

Source: Official StereoSet leaderboard

#EntryScoreSettingDateSource
1 gpt2_ModelNSP_GPT2LM 71.9 Official leaderboard
2 gpt2-medium_ModelNSP_GPT2LM 71.5 Official leaderboard
3 xlnet-large-cased_ModelNSP_XLNetLM 71.2 Official leaderboard
4 bert-base-cased_BertNextSentence_BertLM 69.4 Official leaderboard
5 bert-large-cased_BertNextSentence_BertLM 69.2 Official leaderboard
6 EnsembleModel_ 69.0 Official leaderboard
7 roberta-base_ModelNSP_RoBERTaLM 68.8 Official leaderboard
8 gpt2-large_ModelNSP_GPT2LM 67.8 Official leaderboard
9 roberta-large_ModelNSP_RoBERTaLM 67.4 Official leaderboard
10 xlnet-base-cased_ModelNSP_XLNetLM 61.6 Official leaderboard
11 SentimentModel 49.6 Official leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Safety and alignment

All 465 in Safety and alignment
  • FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
  • StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Inkling 98.6%
  • Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
  • DistressBench In use Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. Muse Spark 1.2 88.4%
  • Lakera B3 (Backbone Breaker Benchmark) In use Crowdsourced prompt-injection attacks hit backbone LLMs inside agent threat snapshots at three defense levels. The score is attack resistance, 100 minus the board's vulnerability percentage. Claude Opus 4.5 89%
  • IHEval In use Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction. GPT-4o 70%