WikiText-2
Language models predict the next token in held-out Wikipedia articles. Lower perplexity means the model gave more probability to the tokens that actually appeared.
The September 2016 paper reports test perplexity of 80.8 for Pointer Sentinel-LSTM. On October 5, 2026, Kimi-Linear 48B scored 6.41 in BF16, 6.54 in INT4 and 6.63 in NVFP4 on matched windows.
- Made by
- Stephen Merity, Caiming Xiong, James Bradbury and Richard Socher
- Released
- September 2016
- Kind
- Other
- Level
- University
- Scored
- Checked automatically
- Answers
- Prediction
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Real work and use
- Refresh
- One fixed set
- Run by
- Anyone
Pointer Sentinel-LSTM (medium) tops the leaderboard with 72.4
Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 1 Pointer Sentinel-LSTM (medium) 72.4
- 2 Zilly et al. (2016) - Variational RHN 72.8
- 3 Gal (2015) - Variational LSTM (large, untied) 77.9
- 4 Gal (2015) - Variational LSTM (medium, untied) 81.9
- 5 Zaremba et al. (2014) - LSTM (large) 82.2
- 6 Zoneout + Variational LSTM (medium) 84.4
- 7 Zaremba et al. (2014) - LSTM (medium) 86.2
Source: The WikiText-2 paper
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Pointer Sentinel-LSTM (medium) | 72.4 | Official leaderboard | ||
| 2 | Zilly et al. (2016) - Variational RHN | 72.8 | Official leaderboard | ||
| 3 | Gal (2015) - Variational LSTM (large, untied) | 77.9 | Official leaderboard | ||
| 4 | Gal (2015) - Variational LSTM (medium, untied) | 81.9 | Official leaderboard | ||
| 5 | Zaremba et al. (2014) - LSTM (large) | 82.2 | Official leaderboard | ||
| 6 | Zoneout + Variational LSTM (medium) | 84.4 | Official leaderboard | ||
| 7 | Zaremba et al. (2014) - LSTM (medium) | 86.2 | Official leaderboard |
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Other
- Artificial Analysis Intelligence Index v5 In use Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. Claude Fable 5.2 64.6
- Personal Superintelligence Index (Meta) In use Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO.
- Vals Science Index In use Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. GPT-6.2 Astra 68.4%
- Decision Index Public v0.3 In use Decision models return probabilities over labeled options in public tasks covering knowledge, language, retrieval, tools, and arts. A chance-corrected weighted aggregate combines the task scores. Granite Switch 4.1 30B 51.3
- SAGE v1.1 (Vals) In use Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading. Claude Opus 5.5 91.3%
- MTEB Fading Massive Text Embedding Benchmark: retrieval, clustering, classification across languages and modalities.
Qwen3-Embedding-8B 72.2%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the benchmark’s paper (7 entries).
- Pass Models scored 7 entries on its leaderboard, none a model we track.
- Note Newest score No scores yet.
- Note Frontier models No models scored yet.
- Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
- Pass Importer We read the board on arxiv.org: 7 entries at the last read, none a model we track.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.