Benchmark Almanac

LAMBADA

PaLM 2-L 86.9%
Megatron-Turing NLG 530B 87.2% leads since October 2021
Dropped

Models predict the last word of narrative passages where humans can guess it only from the whole passage. Scored by accuracy.

Released
June 2016
Kind
Reading comprehension
Level
Everyday
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Strongest model tested
PaLM 2-L #195 on the capability index
Last in a launch post
May 2023, dropped after it saturated
Last model added
November 2023
Scores refreshed
November 12th, 2026

In the authors’ words

We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text.

Abstract of The LAMBADA dataset: Word prediction requiring a broad discourse context (arXiv:1606.06031, CC0)
1
Model scored in lab cards
1
Launch post lists it, from 1 lab

PaLM 2-L leads with 86.9%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Megatron-Turing NLG 530B leads with 87.2%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

  1. 86.9 PaLM 2-L
  1. 1 PaLM 2-L 86.9%

Source: Labs’ model cards

27 models. Scroll sideways for the rest.

  1. 87.2 Megatron-Turing NLG 530B
  2. 86.9 PaLM 2-L
  3. 86.4 InstructGPT 175B
  4. 79.8 Falcon-180B
  5. 78.9 Llama 2-70B
  6. 78.5 Inflection-1
  7. 77.9 PaLM (540B)
  8. 77.7 LLaMA-65B
  9. 77.4 Chinchilla
  10. 77.3 Falcon-40B
  11. 77.2 LLaMA-33B
  12. 76.5 Llama 2-13B
  13. 75.2 LLaMA-13B
  14. 74.9 Falcon-7B
  15. 74.5 Gopher (280B)
  16. 74.0 Baichuan2-13B
  17. 73.3 LLaMA-7B
  18. 73.3 Llama 2-7B
  19. 73.3 Baichuan 2-7B
  20. 71.8 internlm-20b
  21. 71.3 Stable Beluga 2
  22. 71.1 Qwen-14B
  23. 70.0 MPT-7B
  24. 67.9 Qwen-7B
  25. 67.0 internlm-7b
  26. 58.4 Qwen-1_8B
  27. 54.3 chatglm2-6b
  1. 1 Megatron-Turing NLG 530B 87.2%
  2. 2 PaLM 2-L 86.9%
  3. 3 InstructGPT 175B 86.4%
  4. 4 Falcon-180B 79.8%
  5. 5 Llama 2-70B 78.9%
  6. 6 Inflection-1 78.5%
  7. 7 PaLM (540B) 77.9%
  8. 8 LLaMA-65B 77.7%
  9. 9 Chinchilla 77.4%
  10. 10 Falcon-40B 77.3%
  11. 11 LLaMA-33B 77.2%
  12. 12 Llama 2-13B 76.5%
  13. 13 LLaMA-13B 75.2%
  14. 14 Falcon-7B 74.9%
  15. 15 Gopher (280B) 74.5%
  16. 16 Baichuan2-13B 74%
  17. 17 LLaMA-7B 73.3%
  18. 18 Llama 2-7B 73.3%
  19. 19 Baichuan 2-7B 73.3%
  20. 20 internlm-20b 71.8%
  21. 21 Stable Beluga 2 71.3%
  22. 22 Qwen-14B 71.1%
  23. 23 MPT-7B 70%
  24. 24 Qwen-7B 67.9%
  25. 25 internlm-7b 67%
  26. 26 Qwen-1_8B 58.4%
  27. 27 chatglm2-6b 54.3%

Sources: Another board, via Epoch AI (CC BY 4.0) and labs’ model cards

#ModelScoreSettingDateSource
1 PaLM 2-L 86.9% May 2023 Lab model card
#ModelScoreSettingDateSource
1 Megatron-Turing NLG 530B 87.2% Oct 2021 Claim in Using DeepSpeed and Megatron to Train Megatron-Turing NLG…, via Epoch AI
2 PaLM 2-L 86.9% May 2023 Lab model card
3 InstructGPT 175B 86.4% Jan 2022 Claim in GLaM: Efficient Scaling of Language Models with Mixture-o…, via Epoch AI
4 Falcon-180B 79.8% Sep 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
5 Llama 2-70B 78.9% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
6 Inflection-1 78.5% Jun 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
7 PaLM (540B) 77.9% Apr 2022 Claim in The Falcon Series of Open Language Models, via Epoch AI
8 LLaMA-65B 77.7% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
9 Chinchilla 77.4% Mar 2022 Claim in The Falcon Series of Open Language Models, via Epoch AI
10 Falcon-40B 77.3% Mar 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
11 LLaMA-33B 77.2% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
12 Llama 2-13B 76.5% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
13 LLaMA-13B 75.2% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
14 Falcon-7B 74.9% Apr 2023 Claim in The Falcon Series of Open Language Models, via Epoch AI
15 Gopher (280B) 74.5% Dec 2021 Claim in Using DeepSpeed and Megatron to Train Megatron-Turing NLG…, via Epoch AI
16 Baichuan2-13B 74% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
17 LLaMA-7B 73.3% Feb 2023 Claim in Qwen Technical Report, via Epoch AI
18 Llama 2-7B 73.3% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
19 Baichuan 2-7B 73.3% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
20 internlm-20b 71.8% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
21 Stable Beluga 2 71.3% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
22 Qwen-14B 71.1% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
23 MPT-7B 70% May 2023 Claim in Qwen Technical Report, via Epoch AI
24 Qwen-7B 67.9% Sep 2023 Claim in Qwen Technical Report, via Epoch AI
25 internlm-7b 67% Jul 2023 Claim in Qwen Technical Report, via Epoch AI
26 Qwen-1_8B 58.4% Nov 2023 Claim in Qwen Technical Report, via Epoch AI
27 chatglm2-6b 54.3% Jun 2023 Claim in Qwen Technical Report, via Epoch AI

Epoch AI collected claims for 26 models from labs’ reports and papers. Switch the scores above to rank them together.

A claim for Megatron-Turing NLG 530B that Epoch AI collected passed 85% in October 2021

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card A claim Epoch AI collected from a report or paper
0% 20% 40% 60% 80% 100% 2022 2023 2024 Near the ceiling (85%) Megatron-Turing NLG 530B 0% 20% 40% 60% 80% 100% 2022 2023 2024 Near the ceiling (85%) Megatron-Turing NLG 530B

Sources: Labs’ model cards and claims collected by Epoch AI (CC BY 4.0)

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Google DeepMind
DateLaunch postScoreAs the post writes it
May 10, 2023 PaLM 2-L 86.9 LAMBADA, 1-shot

More in Knowledge

All 121 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • KMMLU-Pro In use Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. Solar Pro 4 79.2%
  • HLE-Verified In use Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Seed2.0 Pro 73.6%