SuperGLUE
Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score.
Saturated, with models above the human baseline since 2021.
- Made by
- NYU, Facebook AI, University of Washington, and DeepMind
- Released
- May 2019
- Kind
- Language reasoning
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- Public, plus a private test set
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Another benchmark
- Refresh
- One fixed set
- Run by
- A host
- Last model added
- April 2022
- Scores refreshed
- November 19th, 2026
In the authors’ words
In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.
Hairuo tops the leaderboard with 91.4% PaLM (540B) leads with 90.4% among models released by April 2022
Best score per entry from the benchmark's own leaderboard, read on November 19th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is PaLM (540B), at 90.4%. This view shows only models we track, each at its best.
PaLM (540B) leads with 90.4%, claims included
Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.
The newest model with a score here came out in April 2022, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 91.4 Hairuo
- 91.3 Vega v2
- 91.2 ST-MoE-32B
- 90.9 Turing NLR v5
- 90.6 ERNIE 3.0
- 90.4
PaLM (540B)
- 90.4 T5 + UDG, Single Model (Google Brain)
- 90.3 DeBERTa / TuringNLRv4
- 89.8 SuperGLUE Human Baselines
- 89.3
T5-11B
- 89.2 Frozen T5 1.1 + SPoT
- 86.7 NEZHA-Plus
- 86.1 PAI Albert
- 86.0 RoBERTa-iCETS
- 85.9 RoBERTa (ensemble)
- 85.7 RoBERTa-mtl-adv
- 84.5 RoBERTa
- 82.6 AILabs Team, Transformers
- 77.7 FSL++(ALBERT)-Few-Shot(32 Examples)
- 77.0 Text to Text PETL
- 76.0 ADAPET (ALBERT) - few-shot
- 75.4 iPET (ALBERT) - Few-Shot (32 Examples)
- 74.1 Bort (Alexa AI)
- 73.5 BERT-mtl
- 71.8 GPT-3 175B (davinci)
- 71.5 BERT++
- 1 Hairuo ✓ 91.4%
- 2 Vega v2 ✓ 91.3%
- 3 ST-MoE-32B ✓ 91.2%
- 4 Turing NLR v5 ✓ 90.9%
- 5 ERNIE 3.0 ✓ 90.6%
- 6
PaLM (540B) ✓ 90.4%
- 7 T5 + UDG, Single Model (Google Brain) ✓ 90.4%
- 8 DeBERTa / TuringNLRv4 ✓ 90.3%
- 9 SuperGLUE Human Baselines ✓ 89.8%
- 10
T5-11B ✓ 89.3%
- 11 Frozen T5 1.1 + SPoT ✓ 89.2%
- 12 NEZHA-Plus ✓ 86.7%
- 13 PAI Albert ✓ 86.1%
- 14 RoBERTa-iCETS ✓ 86%
- 15 RoBERTa (ensemble) ✓ 85.9%
- 16 RoBERTa-mtl-adv ✓ 85.7%
- 17 RoBERTa ✓ 84.5%
- 18 AILabs Team, Transformers ✓ 82.6%
- 19 FSL++(ALBERT)-Few-Shot(32 Examples) ✓ 77.7%
- 20 Text to Text PETL ✓ 77%
- 21 ADAPET (ALBERT) - few-shot ✓ 76%
- 22 iPET (ALBERT) - Few-Shot (32 Examples) ✓ 75.4%
- 23 Bort (Alexa AI) ✓ 74.1%
- 24 BERT-mtl ✓ 73.5%
- 25 GPT-3 175B (davinci) ✓ 71.8%
- 26 BERT++ ✓ 71.5%
Source: Official SuperGLUE leaderboard
- 1
PaLM (540B) ✓ 90.4%
- 2
T5-11B ✓ 89.3%
- 3 GPT-3 175B (davinci) ✓ 71.8%
Source: Official SuperGLUE leaderboard
- 1
PaLM (540B) ✓ 90.4%
- 2
T5-11B ✓ 89.3%
- 3
T5-3B 86.4%
- 4 Switch-Large 84.7%
- 5 T5-Large 82.7%
- 6 T5-Base 76.2%
- 7 Switch-Base 73.3%
- 8 GPT-3 175B (davinci) ✓ 71.8%
- 9 InstructGPT 175B 71.8%
- 10 T5-Small 63.3%
Sources: Official SuperGLUE leaderboard and another board, via Epoch AI (CC BY 4.0)
| # | Entry | Score | Setting | Released | Run on | Link | Source |
|---|---|---|---|---|---|---|---|
| 1 | Hairuo ✓ | 91.4% | Aug 2024 | Official leaderboard | |||
| 2 | Vega v2 ✓ | 91.3% | Oct 2022 | Official leaderboard | |||
| 3 | ST-MoE-32B ✓ | 91.2% | Feb 2022 | Official leaderboard | |||
| 4 | Turing NLR v5 ✓ | 90.9% | Dec 2021 | Official leaderboard | |||
| 5 | ERNIE 3.0 ✓ | 90.6% | Jul 2021 | Official leaderboard | |||
| 6 | | 90.4% | Apr 2022 | Mar 2022 | Official leaderboard | ||
| 7 | T5 + UDG, Single Model (Google Brain) ✓ | 90.4% | Mar 2021 | Official leaderboard | |||
| 8 | DeBERTa / TuringNLRv4 ✓ | 90.3% | Jan 2021 | github.com | Official leaderboard | ||
| 9 | SuperGLUE Human Baselines ✓ | 89.8% | May 2019 | Official leaderboard | |||
| 10 | | 89.3% | Oct 2019 | Jan 2020 | github.com | Official leaderboard | |
| 11 | Frozen T5 1.1 + SPoT ✓ | 89.2% | Feb 2022 | Official leaderboard | |||
| 12 | NEZHA-Plus ✓ | 86.7% | Sep 2020 | Official leaderboard | |||
| 13 | PAI Albert ✓ | 86.1% | Aug 2020 | Official leaderboard | |||
| 14 | RoBERTa-iCETS ✓ | 86% | Mar 2021 | Official leaderboard | |||
| 15 | RoBERTa (ensemble) ✓ | 85.9% | Jul 2020 | Official leaderboard | |||
| 16 | RoBERTa-mtl-adv ✓ | 85.7% | Jan 2020 | Official leaderboard | |||
| 17 | RoBERTa ✓ | 84.5% | Aug 2019 | github.com | Official leaderboard | ||
| 18 | AILabs Team, Transformers ✓ | 82.6% | Mar 2021 | Official leaderboard | |||
| 19 | FSL++(ALBERT)-Few-Shot(32 Examples) ✓ | 77.7% | Oct 2022 | Official leaderboard | |||
| 20 | Text to Text PETL ✓ | 77% | Apr 2022 | github.com | Official leaderboard | ||
| 21 | ADAPET (ALBERT) - few-shot ✓ | 76% | Mar 2021 | github.com | Official leaderboard | ||
| 22 | iPET (ALBERT) - Few-Shot (32 Examples) ✓ | 75.4% | Sep 2020 | github.com | Official leaderboard | ||
| 23 | Bort (Alexa AI) ✓ | 74.1% | Oct 2020 | github.com | Official leaderboard | ||
| 24 | BERT-mtl ✓ | 73.5% | Oct 2019 | Official leaderboard | |||
| 25 | GPT-3 175B (davinci) ✓ | 71.8% | May 2020 | May 2020 | github.com | Official leaderboard | |
| 26 | BERT++ ✓ | 71.5% | Jul 2019 | Official leaderboard |
| # | Model | Score | Setting | Released | Run on | Link | Source |
|---|---|---|---|---|---|---|---|
| 1 | | 90.4% | Apr 2022 | Mar 2022 | Official leaderboard | ||
| 2 | | 89.3% | Oct 2019 | Jan 2020 | github.com | Official leaderboard | |
| 3 | GPT-3 175B (davinci) ✓ | 71.8% | May 2020 | May 2020 | github.com | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 90.4% | Apr 2022 | Official leaderboard | ||
| 2 | 89.3% | Oct 2019 | Official leaderboard | ||
| 3 | 86.4% | Oct 2019 | Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI | ||
| 4 | Switch-Large | 84.7% | Claim in Switch Transformers: Scaling to Trillion Parameter Models…, via Epoch AI | ||
| 5 | T5-Large | 82.7% | Claim in Switch Transformers: Scaling to Trillion Parameter Models…, via Epoch AI | ||
| 6 | T5-Base | 76.2% | Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI | ||
| 7 | Switch-Base | 73.3% | Claim in Switch Transformers: Scaling to Trillion Parameter Models…, via Epoch AI | ||
| 8 | GPT-3 175B (davinci) | 71.8% | May 2020 | Official leaderboard | |
| 9 | InstructGPT 175B | 71.8% | Jan 2022 | Claim in Language Models are Few-Shot Learners, via Epoch AI | |
| 10 | T5-Small | 63.3% | Claim in Exploring the Limits of Transfer Learning with a Unified…, via Epoch AI |
Epoch AI collected claims for 8 models from labs’ reports and papers. Switch the scores above to rank them together.
Number one over time Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official SuperGLUE leaderboard
T5-11B on the official leaderboard passed 85% in October 2019
That was 5 months after its release. All the scores we have, one dot each, so a model tested at several settings appears more than once.
Sources: Official SuperGLUE leaderboard and claims collected by Epoch AI (CC BY 4.0)
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access Public tasks, plus a private test set.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the official leaderboard (26 entries) and claims Epoch AI collected (8 models).
- Pass Models scored 10 models we track, plus 23 other entries on its leaderboard.
- Note Newest score The newest came 4 years ago, in April 2022, for PaLM (540B). That’s expected for a saturated test.
- Note Frontier models None of its models are on the capabilities index.
- Note Headroom Saturated. The best is 90.4%, past 85% since October 2019.
Leaderboard import 3 pass
- Pass Importer We read the board on super.gluebenchmark.com: 3 rows for models we track and 23 other entries at the last read.
- Pass Last read November 19th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.