MS MARCO Question Answering
Models answer real Bing queries using retrieved web passages. BLEU and ROUGE-L compare generated answers with human-written references.
This record covers the question-answering task. Passage ranking, conversational search, and other MS MARCO tasks have different protocols.
- Made by
- Microsoft
- Released
- November 2016
- Kind
- Reading comprehension
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Open-ended
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Real work and use
- Refresh
- One fixed set
- Run by
- Anyone
In the authors’ words
We introduce a large scale MAchine Reading COmprehension dataset, which we name MS MARCO. The dataset comprises of 1,010,916 anonymized questions—sampled from Bing's search query logs—each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages—extracted from 3,563,535 web documents retrieved by Bing—that provide the information necessary for curating the natural language answers. A question in the MS MARCO dataset may have multiple answers or no answers at all. Using this dataset, we propose three different tasks with varying levels of difficulty: (i) predict if a question is answerable given a set of context passages, and extract and synthesize the answer as a human would (ii) generate a well-formed answer (if possible) based on the context passages that can be understood with the question and passage context, and finally (iii) rank a set of retrieved passages given a question. The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering. We believe that the scale and the real-world nature of this dataset makes it attractive for benchmarking machine reading comprehension and question-answering models.
Multi-doc Enriched BERT tops the leaderboard with 0.54
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 0.54 Multi-doc Enriched BERT
- 0.53 Selector+Combine-Content-Generator QA Model
- 0.52 Masque Q&A Style
- 0.52 LM+Generator
- 0.52 Deep Cascade QA
- 0.52 PALM
- 0.52 Unnamed
- 0.52 VNET
- 0.51 MultiLM QnA Model
- 0.51 LNET
- 0.51 BERT Encoded T-NET
- 0.50 BERT+ Multi-Pointer-Generator
- 0.50 Selector+Combine-Content-Generator NL Model
- 0.49 CompLM
- 0.49 REAG
- 0.49 MUSST-NLG
- 0.49 Masque NLGEN Style
- 0.48 Communicating BERT
- 0.48 roberta_T_tlcd_18k
- 0.48 MultiLM NLGen Model
- 0.48 MDC-Generator
- 0.47 BERT + Transfer
- 0.47 Bert Based Multi-task
- 0.47 T-RoBERTa-wf-BERTbaseA-120k
- 0.47 BERT-SS-K1-100k
- 0.47 T-RoBERTa-wf-BERTbaseA-80k
- 0.47 Multi-passage QA Model
- 0.46 BERT-RGLM
- 0.45 SNET + CES2S
- 0.45 ranking+nlg
- 0.44 ranker-reader
- 0.44 Extraction-net
- 0.44 SNET
- 0.44 BIDAF+ELMo+SofterMax
- 0.43 DNET
- 0.43 KIGN-QA
- 0.43 MaRCo-da-GAAMA
- 0.42 Reader-Writer
- 0.42 Masque2 (single / NLG Style)
- 0.41 BERT+Multi-Loss
- 0.41 REAG(based on PALM)
- 0.41 RGLM
- 0.40 SNET+seq2seq
- 0.39 SSK3+BERTBaseAnswerGenerator
- 0.39 MP-MRC BERT
- 0.39 MP-MRC BERT-base
- 0.38 MUSST
- 0.34 fj-net(single)
- 0.34 MNet-Base(Single) NLGEN
- 0.34 fj-reader(single)
- 0.34 Generation with latent retrieval per answer
- 0.33 MDCG-Base
- 0.33 MDCC-Base
- 0.33 Generation with latent retrieval Baseline 2
- 0.33 MDCC
- 0.30 Generation with latent retrieval Baseline 1
- 0.30 MultiTask+DataAug+Unlikelihood
- 0.30 lightNLP+BiDAF
- 0.30 MUSST-QA
- 0.29 Pretrained seq2seq model
- 0.29 roberta_T_tlx_90k
- 0.28 BIDAF+seq2seq
- 0.24 BiDaF Baseline(Implemented By MSMARCO Team)
- 0.20 TrioNLP + BiDAF
- 0.15 BiDAF + LSTM
- 1 Multi-doc Enriched BERT QnA v2.1; submitted June 20th, 2019 0.54
- 2 Selector+Combine-Content-Generator QA Model QnA v2.1; submitted March 19th, 2019 0.53
- 3 Masque Q&A Style QnA v2.1; submitted January 3rd, 2019 0.52
- 4 LM+Generator QnA v2.1; submitted November 25th,2019 0.52
- 5 Deep Cascade QA QnA v2.1; submitted December 12th, 2018 0.52
- 6 PALM QnA v2.1; submitted December 9th,2019 0.52
- 7 Unnamed QnA v2.1; submitted December 9th,2019 0.52
- 8 VNET QnA v2.1; submitted November 8th, 2018 0.52
- 9 MultiLM QnA Model QnA v2.1; submitted December 2nd, 2019 0.51
- 10 LNET QnA v2.1; submitted April 8th, 2020 0.51
- 11 BERT Encoded T-NET QnA v2.1; submitted July 12th, 2019 0.51
- 12 BERT+ Multi-Pointer-Generator QnA v2.1; submitted June 11th, 2019 0.50
- 13 Selector+Combine-Content-Generator NL Model QnA v2.1; submitted March 11th, 2019 0.50
- 14 CompLM QnA v2.1; submitted December 2nd, 2019 0.49
- 15 REAG QnA v2.1; submitted March 27th, 2020 0.49
- 16 MUSST-NLG QnA v2.1; submitted May 15th, 2020 0.49
- 17 Masque NLGEN Style QnA v2.1; submitted January 3rd, 2019 0.49
- 18 Communicating BERT QnA v2.1; submitted October 4th, 2019 0.48
- 19 roberta_T_tlcd_18k QnA v2.1; submitted May 14th, 2020 0.48
- 20 MultiLM NLGen Model QnA v2.1; submitted December 2nd, 2019 0.48
- 21 MDC-Generator QnA v2.1; submitted April 23rd, 2020 0.48
- 22 BERT + Transfer QnA v2.1; submitted October 16th, 2019 0.47
- 23 Bert Based Multi-task QnA v2.1; submitted June 26th, 2019 0.47
- 24 T-RoBERTa-wf-BERTbaseA-120k QnA v2.1; submitted February 13th, 2020 0.47
- 25 BERT-SS-K1-100k QnA v2.1; submitted January 26th, 2020 0.47
- 26 T-RoBERTa-wf-BERTbaseA-80k QnA v2.1; submitted February 21st, 2020 0.47
- 27 Multi-passage QA Model QnA v2.1; submitted October 21st, 2020 0.47
- 28 BERT-RGLM QnA v2.1; submitted April 22nd, 2020 0.46
- 29 SNET + CES2S QnA v2.1; submitted July 24th, 2018 0.45
- 30 ranking+nlg QnA v2.1; submitted October 9th, 2019 0.45
- 31 ranker-reader QnA v2.1; submitted May 15th, 2019 0.44
- 32 Extraction-net QnA v2.1; submitted October 20th, 2018 0.44
- 33 SNET QnA v2.1; submitted August 30th, 2018 0.44
- 34 BIDAF+ELMo+SofterMax QnA v2.1; submitted November 16th, 2018 0.44
- 35 DNET QnA v2.1; submitted August 1st, 2018 0.43
- 36 KIGN-QA QnA v2.1; submitted April 22nd, 2019 0.43
- 37 MaRCo-da-GAAMA QnA v2.1; submitted April 7th, 2020 0.43
- 38 Reader-Writer QnA v2.1; submitted September 16th, 2018 0.42
- 39 Masque2 (single / NLG Style) QnA v2.1; submitted October 22nd, 2020 0.42
- 40 BERT+Multi-Loss QnA v2.1; submitted November 4th, 2019 0.41
- 41 REAG(based on PALM) QnA v2.1; submitted June 1st,2020 0.41
- 42 RGLM QnA v2.1; submitted May 5th, 2020 0.41
- 43 SNET+seq2seq QnA v2.1; submitted June 1st, 2018 0.40
- 44 SSK3+BERTBaseAnswerGenerator QnA v2.1; submitted Jan 21st, 2020 0.39
- 45 MP-MRC BERT QnA v2.1; submitted Aug 27th, 2020 0.39
- 46 MP-MRC BERT-base QnA v2.1; submitted Sep 4th, 2020 0.39
- 47 MUSST QnA v2.1; submitted March 31st, 2020 0.38
- 48 fj-net(single) QnA v2.1; submitted August 3rd, 2020 0.34
- 49 MNet-Base(Single) NLGEN QnA v2.1; submitted July 8th, 2020 0.34
- 50 fj-reader(single) QnA v2.1; submitted July 28th, 2020 0.34
- 51 Generation with latent retrieval per answer QnA v2.1; submitted May 11th, 2020 0.34
- 52 MDCG-Base QnA v2.1; submitted June 8th, 2020 0.33
- 53 MDCC-Base QnA v2.1; submitted June 10th, 2020 0.33
- 54 Generation with latent retrieval Baseline 2 QnA v2.1; submitted May 11th, 2020 0.33
- 55 MDCC QnA v2.1; submitted June 10th, 2020 0.33
- 56 Generation with latent retrieval Baseline 1 QnA v2.1; submitted May 11th, 2020 0.30
- 57 MultiTask+DataAug+Unlikelihood QnA v2.1; submitted June 3rd, 2020 0.30
- 58 lightNLP+BiDAF QnA v2.1; submitted February 1st, 2019 0.30
- 59 MUSST-QA QnA v2.1; submitted June 1st, 2020 0.30
- 60 Pretrained seq2seq model QnA v2.1; submitted September 10th, 2020 0.29
- 61 roberta_T_tlx_90k QnA v2.1; submitted July 29th, 2020 0.29
- 62 BIDAF+seq2seq QnA v2.1; submitted May 29th, 2018 0.28
- 63 BiDaF Baseline(Implemented By MSMARCO Team) QnA v2.1; submitted April 23th, 2018 0.24
- 64 TrioNLP + BiDAF QnA v2.1; submitted September 23rd, 2018 0.20
- 65 BiDAF + LSTM QnA v2.1; submitted January 15th,2019 0.15
Source: Official MS MARCO Question Answering leaderboard
| # | Entry | Score | Setting | Submitted by | Date | Source |
|---|---|---|---|---|---|---|
| 1 | Multi-doc Enriched BERT | 0.54 | QnA v2.1; submitted June 20th, 2019 | Ming Yan of Alibaba Damo NLP | Jun 2019 | Official leaderboard |
| 2 | Selector+Combine-Content-Generator QA Model | 0.53 | QnA v2.1; submitted March 19th, 2019 | Shengjie Qian of Caiyun xiaoyi AI and BUPT | Mar 2019 | Official leaderboard |
| 3 | Masque Q&A Style | 0.52 | QnA v2.1; submitted January 3rd, 2019 | NTT Media Intelligence Laboratories [ Nishida et al. '19 ] | Jan 2019 | Official leaderboard |
| 4 | LM+Generator | 0.52 | QnA v2.1; submitted November 25th,2019 | Alibaba Damo NLP | Nov 2019 | Official leaderboard |
| 5 | Deep Cascade QA | 0.52 | QnA v2.1; submitted December 12th, 2018 | Ming Yan of Alibaba Damo NLP [ Yan et al. '18 ] | Dec 2018 | Official leaderboard |
| 6 | PALM | 0.52 | QnA v2.1; submitted December 9th,2019 | Alibaba Damo NLP | Dec 2019 | Official leaderboard |
| 7 | Unnamed | 0.52 | QnA v2.1; submitted December 9th,2019 | anonymous | Dec 2019 | Official leaderboard |
| 8 | VNET | 0.52 | QnA v2.1; submitted November 8th, 2018 | Baidu NLP [ Wang et al. '18 ] | Nov 2018 | Official leaderboard |
| 9 | MultiLM QnA Model | 0.51 | QnA v2.1; submitted December 2nd, 2019 | anonymous | Dec 2019 | Official leaderboard |
| 10 | LNET | 0.51 | QnA v2.1; submitted April 8th, 2020 | S.L. Liu of NEUKG | Apr 2020 | Official leaderboard |
| 11 | BERT Encoded T-NET | 0.51 | QnA v2.1; submitted July 12th, 2019 | Y. Zhang, C. Wang, X.L. Chen | Jul 2019 | Official leaderboard |
| 12 | BERT+ Multi-Pointer-Generator | 0.50 | QnA v2.1; submitted June 11th, 2019 | Tongjun Li of the ColorfulClouds Tech and BUPT | Jun 2019 | Official leaderboard |
| 13 | Selector+Combine-Content-Generator NL Model | 0.50 | QnA v2.1; submitted March 11th, 2019 | Shengjie Qian of Caiyun xiaoyi AI and BUPT | Mar 2019 | Official leaderboard |
| 14 | CompLM | 0.49 | QnA v2.1; submitted December 2nd, 2019 | Alibaba Damo NLP | Dec 2019 | Official leaderboard |
| 15 | REAG | 0.49 | QnA v2.1; submitted March 27th, 2020 | Anonymous | Mar 2020 | Official leaderboard |
| 16 | MUSST-NLG | 0.49 | QnA v2.1; submitted May 15th, 2020 | Anonymous | May 2020 | Official leaderboard |
| 17 | Masque NLGEN Style | 0.49 | QnA v2.1; submitted January 3rd, 2019 | NTT Media Intelligence Laboratories [ Nishida et al. '19 ] | Jan 2019 | Official leaderboard |
| 18 | Communicating BERT | 0.48 | QnA v2.1; submitted October 4th, 2019 | Xuan Liang of RIDLL from the University of Technology Sydney | Oct 2019 | Official leaderboard |
| 19 | roberta_T_tlcd_18k | 0.48 | QnA v2.1; submitted May 14th, 2020 | Anonymous | May 2020 | Official leaderboard |
| 20 | MultiLM NLGen Model | 0.48 | QnA v2.1; submitted December 2nd, 2019 | anonymous | Dec 2019 | Official leaderboard |
| 21 | MDC-Generator | 0.48 | QnA v2.1; submitted April 23rd, 2020 | Ssk-nlp | Apr 2020 | Official leaderboard |
| 22 | BERT + Transfer | 0.47 | QnA v2.1; submitted October 16th, 2019 | anonymous | Oct 2019 | Official leaderboard |
| 23 | Bert Based Multi-task | 0.47 | QnA v2.1; submitted June 26th, 2019 | ZhangY & WangC | Jun 2019 | Official leaderboard |
| 24 | T-RoBERTa-wf-BERTbaseA-120k | 0.47 | QnA v2.1; submitted February 13th, 2020 | Anonymous | Feb 2020 | Official leaderboard |
| 25 | BERT-SS-K1-100k | 0.47 | QnA v2.1; submitted January 26th, 2020 | Anonymous | Jan 2020 | Official leaderboard |
| 26 | T-RoBERTa-wf-BERTbaseA-80k | 0.47 | QnA v2.1; submitted February 21st, 2020 | Anonymous | Feb 2020 | Official leaderboard |
| 27 | Multi-passage QA Model | 0.47 | QnA v2.1; submitted October 21st, 2020 | SudaNLP | Oct 2020 | Official leaderboard |
| 28 | BERT-RGLM | 0.46 | QnA v2.1; submitted April 22nd, 2020 | Anonymous | Apr 2020 | Official leaderboard |
| 29 | SNET + CES2S | 0.45 | QnA v2.1; submitted July 24th, 2018 | Bo Shao of SYSU University | Jul 2018 | Official leaderboard |
| 30 | ranking+nlg | 0.45 | QnA v2.1; submitted October 9th, 2019 | anonymous | Oct 2019 | Official leaderboard |
| 31 | ranker-reader | 0.44 | QnA v2.1; submitted May 15th, 2019 | RCZoo of UCAS | May 2019 | Official leaderboard |
| 32 | Extraction-net | 0.44 | QnA v2.1; submitted October 20th, 2018 | zlsh80826 | Oct 2018 | Official leaderboard |
| 33 | SNET | 0.44 | QnA v2.1; submitted August 30th, 2018 | JY Zhao | Aug 2018 | Official leaderboard |
| 34 | BIDAF+ELMo+SofterMax | 0.44 | QnA v2.1; submitted November 16th, 2018 | Wang Changbao | Nov 2018 | Official leaderboard |
| 35 | DNET | 0.43 | QnA v2.1; submitted August 1st, 2018 | QA Geeks | Aug 2018 | Official leaderboard |
| 36 | KIGN-QA | 0.43 | QnA v2.1; submitted April 22nd, 2019 | Chenliang Li | Apr 2019 | Official leaderboard |
| 37 | MaRCo-da-GAAMA | 0.43 | QnA v2.1; submitted April 7th, 2020 | IBM Research AI Multilingual NLP Group | Apr 2020 | Official leaderboard |
| 38 | Reader-Writer | 0.42 | QnA v2.1; submitted September 16th, 2018 | Microsoft Business Applications Group AI Research | Sep 2018 | Official leaderboard |
| 39 | Masque2 (single / NLG Style) | 0.42 | QnA v2.1; submitted October 22nd, 2020 | NTT Media Intelligence Laboratories | Oct 2020 | Official leaderboard |
| 40 | BERT+Multi-Loss | 0.41 | QnA v2.1; submitted November 4th, 2019 | S.L. Liu of NEUKG | Nov 2019 | Official leaderboard |
| 41 | REAG(based on PALM) | 0.41 | QnA v2.1; submitted June 1st,2020 | anonymous | Jun 2020 | Official leaderboard |
| 42 | RGLM | 0.41 | QnA v2.1; submitted May 5th, 2020 | anonymous | May 2020 | Official leaderboard |
| 43 | SNET+seq2seq | 0.40 | QnA v2.1; submitted June 1st, 2018 | Yihan Ni of the CAS Key Lab of Web Data Science and Technology, ICT, CAS | Jun 2018 | Official leaderboard |
| 44 | SSK3+BERTBaseAnswerGenerator | 0.39 | QnA v2.1; submitted Jan 21st, 2020 | anonymous | Jan 2020 | Official leaderboard |
| 45 | MP-MRC BERT | 0.39 | QnA v2.1; submitted Aug 27th, 2020 | H.Y. Zhang | Aug 2020 | Official leaderboard |
| 46 | MP-MRC BERT-base | 0.39 | QnA v2.1; submitted Sep 4th, 2020 | H.Y. Zhang | Sep 2020 | Official leaderboard |
| 47 | MUSST | 0.38 | QnA v2.1; submitted March 31st, 2020 | anonymous | Mar 2020 | Official leaderboard |
| 48 | fj-net(single) | 0.34 | QnA v2.1; submitted August 3rd, 2020 | yzm nlp group | Aug 2020 | Official leaderboard |
| 49 | MNet-Base(Single) NLGEN | 0.34 | QnA v2.1; submitted July 8th, 2020 | fuii of iDW | Jul 2020 | Official leaderboard |
| 50 | fj-reader(single) | 0.34 | QnA v2.1; submitted July 28th, 2020 | yzm nlp group | Jul 2020 | Official leaderboard |
| 51 | Generation with latent retrieval per answer | 0.34 | QnA v2.1; submitted May 11th, 2020 | anonymous | May 2020 | Official leaderboard |
| 52 | MDCG-Base | 0.33 | QnA v2.1; submitted June 8th, 2020 | ssk-nlp | Jun 2020 | Official leaderboard |
| 53 | MDCC-Base | 0.33 | QnA v2.1; submitted June 10th, 2020 | ssk-nlp | Jun 2020 | Official leaderboard |
| 54 | Generation with latent retrieval Baseline 2 | 0.33 | QnA v2.1; submitted May 11th, 2020 | anonymous | May 2020 | Official leaderboard |
| 55 | MDCC | 0.33 | QnA v2.1; submitted June 10th, 2020 | ssk-nlp | Jun 2020 | Official leaderboard |
| 56 | Generation with latent retrieval Baseline 1 | 0.30 | QnA v2.1; submitted May 11th, 2020 | anonymous | May 2020 | Official leaderboard |
| 57 | MultiTask+DataAug+Unlikelihood | 0.30 | QnA v2.1; submitted June 3rd, 2020 | UvA | Jun 2020 | Official leaderboard |
| 58 | lightNLP+BiDAF | 0.30 | QnA v2.1; submitted February 1st, 2019 | Enliple AI | Feb 2019 | Official leaderboard |
| 59 | MUSST-QA | 0.30 | QnA v2.1; submitted June 1st, 2020 | Anonymous | Jun 2020 | Official leaderboard |
| 60 | Pretrained seq2seq model | 0.29 | QnA v2.1; submitted September 10th, 2020 | BDEG | Sep 2020 | Official leaderboard |
| 61 | roberta_T_tlx_90k | 0.29 | QnA v2.1; submitted July 29th, 2020 | Anonymous | Jul 2020 | Official leaderboard |
| 62 | BIDAF+seq2seq | 0.28 | QnA v2.1; submitted May 29th, 2018 | Yihan Ni of the CAS Key Lab of Web Data Science and Technology, ICT, CAS | May 2018 | Official leaderboard |
| 63 | BiDaF Baseline(Implemented By MSMARCO Team) | 0.24 | QnA v2.1; submitted April 23th, 2018 | Allen Institute for AI & University of Washington [ Seo et al. '16 ] | Apr 2018 | Official leaderboard |
| 64 | TrioNLP + BiDAF | 0.20 | QnA v2.1; submitted September 23rd, 2018 | Trio.AI of the CCNU | Sep 2018 | Official leaderboard |
| 65 | BiDAF + LSTM | 0.15 | QnA v2.1; submitted January 15th,2019 | Meefly | Jan 2019 | Official leaderboard |
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the official leaderboard (65 entries).
- Pass Models scored 65 entries on its leaderboard, none a model we track.
- Note Newest score No scores yet.
- Note Frontier models No models scored yet.
- Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
- Pass Importer We read the board on microsoft.github.io: 65 entries at the last read, none a model we track.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.