SQuAD

Llama 3.1-405B 89.3% leads since July 2024

Dropped

Models extract answer spans from Wikipedia paragraphs for over 100,000 crowdsourced questions. Scored by exact match and F1.

Saturated, with models above human performance since 2018. SQuAD 2.0 added unanswerable questions.

Made by: Stanford NLP
Released: June 2016
Kind: Reading comprehension
Level: Everyday
Scored: Checked automatically
Answers: Short answer
Tasks: Public, plus a private test set

Strongest model tested: Llama 3.1-405B #162 on the capability index
Last in a launch post: April 2024, dropped after it saturated
Last model added: July 2024

In the authors’ words

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com

Abstract of SQuAD: 100,000+ Questions for Machine Comprehension of Text (arXiv:1606.05250, CC0)

Models scored in lab cards

Launch post lists it, from 1 lab

System cards report it, from 1 lab

Llama 3.1-405B leads with 89.3% among models released by July 2024

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

The newest model with a score here came out in July 2024, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

1 Llama 3.1-405B 89.3%
2 Llama 3-70B 85.6%
3 Llama 3.1-70B 81.8%
4 Llama 3.1-8B 77%
5 Llama 3-8B 76.4%

Source: Labs’ model cards

#	Model	Score	Date	Source
1	Llama 3.1-405B	89.3%	Jul 2024	Lab model card
2	Llama 3-70B	85.6%	Apr 2024	Lab model card
3	Llama 3.1-70B	81.8%	Jul 2024	Lab model card
4	Llama 3.1-8B	77%	Jul 2024	Lab model card
5	Llama 3-8B	76.4%	Apr 2024	Lab model card

2 models have held number one in labs’ own numbers since 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

Meta’s own number for Llama 3-70B passed 85% in April 2024

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far

Source: Labs’ model cards

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

5 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
Jul 23, 2024	Llama 3.1-8B model card	77	SQuAD, 1-shot, em, pretrained (Base pretrained models table)	Capabilities
Jul 23, 2024	Llama 3.1-70B model card	81.8	SQuAD, 1-shot, em, pretrained (Base pretrained models table)	Capabilities
Jul 23, 2024	Llama 3.1-405B model card	89.3	SQuAD, 1-shot, em, pretrained (Base pretrained models table)	Capabilities
Apr 18, 2024	Llama 3-8B model card	76.4	SQuAD (1-shot), pretrained (Base pretrained models table)	Capabilities
Apr 18, 2024	Llama 3-70B model card	85.6	SQuAD (1-shot), pretrained (Base pretrained models table)	Capabilities

Versions

Replaced by SQuAD v2