Benchmark Almanac

SuperGLUE

Hairuo 91.4% tops its leaderboard PaLM (540B) 90.4% leads the models we track
PaLM (540B) 90.4% leads since April 2022
Not in model cards

Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score.

Saturated, with models above the human baseline since 2021.

Released
May 2019
Kind
Language reasoning
Level
Everyday
Scored
Checked automatically
Answers
Multiple choice
Tasks
Public, plus a private test set
Measures
Capability
Language
English
In and out
Text → Text
Interaction
Single turn
Items from
Another benchmark
Refresh
One fixed set
Run by
A host
Last model added
April 2022
Scores refreshed
November 19th, 2026

In the authors’ words

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

Abstract of SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (arXiv:1905.00537, CC0)
26
Entries on its own leaderboard, 3 models we track

Hairuo tops the leaderboard with 91.4% PaLM (540B) leads with 90.4% among models released by April 2022

Best score per entry from the benchmark's own leaderboard, read on November 19th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is PaLM (540B), at 90.4%. This view shows only models we track, each at its best.

PaLM (540B) leads with 90.4%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

The newest model with a score here came out in April 2022, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Hairuo ✓ 91.4%
  2. 2 Vega v2 ✓ 91.3%
  3. 3 ST-MoE-32B ✓ 91.2%
  4. 4 Turing NLR v5 ✓ 90.9%
  5. 5 ERNIE 3.0 ✓ 90.6%
  6. 6 PaLM (540B) ✓ 90.4%
  7. 7 T5 + UDG, Single Model (Google Brain) ✓ 90.4%
  8. 8 DeBERTa / TuringNLRv4 ✓ 90.3%
  9. 9 SuperGLUE Human Baselines ✓ 89.8%
  10. 10 T5-11B ✓ 89.3%
  11. 11 Frozen T5 1.1 + SPoT ✓ 89.2%
  12. 12 NEZHA-Plus ✓ 86.7%
  13. 13 PAI Albert ✓ 86.1%
  14. 14 RoBERTa-iCETS ✓ 86%
  15. 15 RoBERTa (ensemble) ✓ 85.9%
  16. 16 RoBERTa-mtl-adv ✓ 85.7%
  17. 17 RoBERTa ✓ 84.5%
  18. 18 AILabs Team, Transformers ✓ 82.6%
  19. 19 FSL++(ALBERT)-Few-Shot(32 Examples) ✓ 77.7%
  20. 20 Text to Text PETL ✓ 77%
  21. 21 ADAPET (ALBERT) - few-shot ✓ 76%
  22. 22 iPET (ALBERT) - Few-Shot (32 Examples) ✓ 75.4%
  23. 23 Bort (Alexa AI) ✓ 74.1%
  24. 24 BERT-mtl ✓ 73.5%
  25. 25 GPT-3 175B (davinci) ✓ 71.8%
  26. 26 BERT++ ✓ 71.5%

Source: Official SuperGLUE leaderboard

Source: Official SuperGLUE leaderboard

Sources: Official SuperGLUE leaderboard and another board, via Epoch AI (CC BY 4.0)

#EntryScoreSettingReleasedRun onLinkSource
1 Hairuo ✓ 91.4% Aug 2024 Official leaderboard
2 Vega v2 ✓ 91.3% Oct 2022 Official leaderboard
3 ST-MoE-32B ✓ 91.2% Feb 2022 Official leaderboard
4 Turing NLR v5 ✓ 90.9% Dec 2021 Official leaderboard
5 ERNIE 3.0 ✓ 90.6% Jul 2021 Official leaderboard
6 PaLM (540B) ✓ 90.4% Apr 2022 Mar 2022 Official leaderboard
7 T5 + UDG, Single Model (Google Brain) ✓ 90.4% Mar 2021 Official leaderboard
8 DeBERTa / TuringNLRv4 ✓ 90.3% Jan 2021 github.com Official leaderboard
9 SuperGLUE Human Baselines ✓ 89.8% May 2019 Official leaderboard
10 T5-11B ✓ 89.3% Oct 2019 Jan 2020 github.com Official leaderboard
11 Frozen T5 1.1 + SPoT ✓ 89.2% Feb 2022 Official leaderboard
12 NEZHA-Plus ✓ 86.7% Sep 2020 Official leaderboard
13 PAI Albert ✓ 86.1% Aug 2020 Official leaderboard
14 RoBERTa-iCETS ✓ 86% Mar 2021 Official leaderboard
15 RoBERTa (ensemble) ✓ 85.9% Jul 2020 Official leaderboard
16 RoBERTa-mtl-adv ✓ 85.7% Jan 2020 Official leaderboard
17 RoBERTa ✓ 84.5% Aug 2019 github.com Official leaderboard
18 AILabs Team, Transformers ✓ 82.6% Mar 2021 Official leaderboard
19 FSL++(ALBERT)-Few-Shot(32 Examples) ✓ 77.7% Oct 2022 Official leaderboard
20 Text to Text PETL ✓ 77% Apr 2022 github.com Official leaderboard
21 ADAPET (ALBERT) - few-shot ✓ 76% Mar 2021 github.com Official leaderboard
22 iPET (ALBERT) - Few-Shot (32 Examples) ✓ 75.4% Sep 2020 github.com Official leaderboard
23 Bort (Alexa AI) ✓ 74.1% Oct 2020 github.com Official leaderboard
24 BERT-mtl ✓ 73.5% Oct 2019 Official leaderboard
25 GPT-3 175B (davinci) ✓ 71.8% May 2020 May 2020 github.com Official leaderboard
26 BERT++ ✓ 71.5% Jul 2019 Official leaderboard
#ModelScoreSettingReleasedRun onLinkSource
1 PaLM (540B) ✓ 90.4% Apr 2022 Mar 2022 Official leaderboard
2 T5-11B ✓ 89.3% Oct 2019 Jan 2020 github.com Official leaderboard
3 GPT-3 175B (davinci) ✓ 71.8% May 2020 May 2020 github.com Official leaderboard
#ModelScoreSettingDateSource
1 PaLM (540B) 90.4% Apr 2022 Official leaderboard
2 T5-11B 89.3% Oct 2019 Official leaderboard
3 T5-3B 86.4% Oct 2019 Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI
4 Switch-Large 84.7% Claim in Switch Transformers: Scaling to Trillion Parameter Models…, via Epoch AI
5 T5-Large 82.7% Claim in Switch Transformers: Scaling to Trillion Parameter Models…, via Epoch AI
6 T5-Base 76.2% Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI
7 Switch-Base 73.3% Claim in Switch Transformers: Scaling to Trillion Parameter Models…, via Epoch AI
8 GPT-3 175B (davinci) 71.8% May 2020 Official leaderboard
9 InstructGPT 175B 71.8% Jan 2022 Claim in Language Models are Few-Shot Learners, via Epoch AI
10 T5-Small 63.3% Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI

Epoch AI collected claims for 8 models from labs’ reports and papers. Switch the scores above to rank them together.

Number one over time Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind
Own leaderboard Passed 85% in Oct 2019, by T5-11B
PaLM (540B) 90.4%

Source: Official SuperGLUE leaderboard

T5-11B on the official leaderboard passed 85% in October 2019

That was 5 months after its release. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard A claim Epoch AI collected from a report or paper Best so far
0% 20% 40% 60% 80% 100% 2020 2021 2022 Near the ceiling (85%) PaLM (540B) T5-11B 0% 20% 40% 60% 80% 100% 2020 2021 2022 Near the ceiling (85%) PaLM (540B) T5-11B

Sources: Official SuperGLUE leaderboard and claims collected by Epoch AI (CC BY 4.0)

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Knowledge

All 332 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. Seed1.8 92.3%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%

Data audit

All 8 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access Public tasks, plus a private test set.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the official leaderboard (26 entries) and claims Epoch AI collected (8 models).
  • Pass Models scored 10 models we track, plus 23 other entries on its leaderboard.
  • Note Newest score The newest came 4 years ago, in April 2022, for PaLM (540B). That’s expected for a saturated test.
  • Note Frontier models None of its models are on the capabilities index.
  • Note Headroom Saturated. The best is 90.4%, past 85% since October 2019.
Leaderboard import 3 pass
  • Pass Importer We read the board on super.gluebenchmark.com: 3 rows for models we track and 23 other entries at the last read.
  • Pass Last read November 19th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check