Benchmark Almanac

MS MARCO Question Answering

Multi-doc Enriched BERT 0.54 tops its leaderboard
Not in model cards

Models answer real Bing queries using retrieved web passages. BLEU and ROUGE-L compare generated answers with human-written references.

This record covers the question-answering task. Passage ranking, conversational search, and other MS MARCO tasks have different protocols.

Made by
Microsoft
Released
November 2016
Kind
Reading comprehension
Level
Everyday
Scored
Checked automatically
Answers
Open-ended
Tasks
All public
Measures
Capability
Language
English
In and out
Text → Text
Interaction
Single turn
Items from
Real work and use
Refresh
One fixed set
Run by
Anyone

In the authors’ words

We introduce a large scale MAchine Reading COmprehension dataset, which we name MS MARCO. The dataset comprises of 1,010,916 anonymized questions—sampled from Bing's search query logs—each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages—extracted from 3,563,535 web documents retrieved by Bing—that provide the information necessary for curating the natural language answers. A question in the MS MARCO dataset may have multiple answers or no answers at all. Using this dataset, we propose three different tasks with varying levels of difficulty: (i) predict if a question is answerable given a set of context passages, and extract and synthesize the answer as a human would (ii) generate a well-formed answer (if possible) based on the context passages that can be understood with the question and passage context, and finally (iii) rank a set of retrieved passages given a question. The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering. We believe that the scale and the real-world nature of this dataset makes it attractive for benchmarking machine reading comprehension and question-answering models.

Abstract of MS MARCO: A Human Generated MAchine Reading COmprehension Dataset (arXiv:1611.09268, CC0)
65
Entries on its own leaderboard, 0 models we track

Multi-doc Enriched BERT tops the leaderboard with 0.54

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

65 entries. Scroll sideways for the rest.

  1. 0.54 Multi-doc Enriched BERT
  2. 0.53 Selector+Combine-Content-Generator QA Model
  3. 0.52 Masque Q&A Style
  4. 0.52 LM+Generator
  5. 0.52 Deep Cascade QA
  6. 0.52 PALM
  7. 0.52 Unnamed
  8. 0.52 VNET
  9. 0.51 MultiLM QnA Model
  10. 0.51 LNET
  11. 0.51 BERT Encoded T-NET
  12. 0.50 BERT+ Multi-Pointer-Generator
  13. 0.50 Selector+Combine-Content-Generator NL Model
  14. 0.49 CompLM
  15. 0.49 REAG
  16. 0.49 MUSST-NLG
  17. 0.49 Masque NLGEN Style
  18. 0.48 Communicating BERT
  19. 0.48 roberta_T_tlcd_18k
  20. 0.48 MultiLM NLGen Model
  21. 0.48 MDC-Generator
  22. 0.47 BERT + Transfer
  23. 0.47 Bert Based Multi-task
  24. 0.47 T-RoBERTa-wf-BERTbaseA-120k
  25. 0.47 BERT-SS-K1-100k
  26. 0.47 T-RoBERTa-wf-BERTbaseA-80k
  27. 0.47 Multi-passage QA Model
  28. 0.46 BERT-RGLM
  29. 0.45 SNET + CES2S
  30. 0.45 ranking+nlg
  31. 0.44 ranker-reader
  32. 0.44 Extraction-net
  33. 0.44 SNET
  34. 0.44 BIDAF+ELMo+SofterMax
  35. 0.43 DNET
  36. 0.43 KIGN-QA
  37. 0.43 MaRCo-da-GAAMA
  38. 0.42 Reader-Writer
  39. 0.42 Masque2 (single / NLG Style)
  40. 0.41 BERT+Multi-Loss
  41. 0.41 REAG(based on PALM)
  42. 0.41 RGLM
  43. 0.40 SNET+seq2seq
  44. 0.39 SSK3+BERTBaseAnswerGenerator
  45. 0.39 MP-MRC BERT
  46. 0.39 MP-MRC BERT-base
  47. 0.38 MUSST
  48. 0.34 fj-net(single)
  49. 0.34 MNet-Base(Single) NLGEN
  50. 0.34 fj-reader(single)
  51. 0.34 Generation with latent retrieval per answer
  52. 0.33 MDCG-Base
  53. 0.33 MDCC-Base
  54. 0.33 Generation with latent retrieval Baseline 2
  55. 0.33 MDCC
  56. 0.30 Generation with latent retrieval Baseline 1
  57. 0.30 MultiTask+DataAug+Unlikelihood
  58. 0.30 lightNLP+BiDAF
  59. 0.30 MUSST-QA
  60. 0.29 Pretrained seq2seq model
  61. 0.29 roberta_T_tlx_90k
  62. 0.28 BIDAF+seq2seq
  63. 0.24 BiDaF Baseline(Implemented By MSMARCO Team)
  64. 0.20 TrioNLP + BiDAF
  65. 0.15 BiDAF + LSTM
  1. 1 Multi-doc Enriched BERT QnA v2.1; submitted June 20th, 2019 0.54
  2. 2 Selector+Combine-Content-Generator QA Model QnA v2.1; submitted March 19th, 2019 0.53
  3. 3 Masque Q&A Style QnA v2.1; submitted January 3rd, 2019 0.52
  4. 4 LM+Generator QnA v2.1; submitted November 25th,2019 0.52
  5. 5 Deep Cascade QA QnA v2.1; submitted December 12th, 2018 0.52
  6. 6 PALM QnA v2.1; submitted December 9th,2019 0.52
  7. 7 Unnamed QnA v2.1; submitted December 9th,2019 0.52
  8. 8 VNET QnA v2.1; submitted November 8th, 2018 0.52
  9. 9 MultiLM QnA Model QnA v2.1; submitted December 2nd, 2019 0.51
  10. 10 LNET QnA v2.1; submitted April 8th, 2020 0.51
  11. 11 BERT Encoded T-NET QnA v2.1; submitted July 12th, 2019 0.51
  12. 12 BERT+ Multi-Pointer-Generator QnA v2.1; submitted June 11th, 2019 0.50
  13. 13 Selector+Combine-Content-Generator NL Model QnA v2.1; submitted March 11th, 2019 0.50
  14. 14 CompLM QnA v2.1; submitted December 2nd, 2019 0.49
  15. 15 REAG QnA v2.1; submitted March 27th, 2020 0.49
  16. 16 MUSST-NLG QnA v2.1; submitted May 15th, 2020 0.49
  17. 17 Masque NLGEN Style QnA v2.1; submitted January 3rd, 2019 0.49
  18. 18 Communicating BERT QnA v2.1; submitted October 4th, 2019 0.48
  19. 19 roberta_T_tlcd_18k QnA v2.1; submitted May 14th, 2020 0.48
  20. 20 MultiLM NLGen Model QnA v2.1; submitted December 2nd, 2019 0.48
  21. 21 MDC-Generator QnA v2.1; submitted April 23rd, 2020 0.48
  22. 22 BERT + Transfer QnA v2.1; submitted October 16th, 2019 0.47
  23. 23 Bert Based Multi-task QnA v2.1; submitted June 26th, 2019 0.47
  24. 24 T-RoBERTa-wf-BERTbaseA-120k QnA v2.1; submitted February 13th, 2020 0.47
  25. 25 BERT-SS-K1-100k QnA v2.1; submitted January 26th, 2020 0.47
  26. 26 T-RoBERTa-wf-BERTbaseA-80k QnA v2.1; submitted February 21st, 2020 0.47
  27. 27 Multi-passage QA Model QnA v2.1; submitted October 21st, 2020 0.47
  28. 28 BERT-RGLM QnA v2.1; submitted April 22nd, 2020 0.46
  29. 29 SNET + CES2S QnA v2.1; submitted July 24th, 2018 0.45
  30. 30 ranking+nlg QnA v2.1; submitted October 9th, 2019 0.45
  31. 31 ranker-reader QnA v2.1; submitted May 15th, 2019 0.44
  32. 32 Extraction-net QnA v2.1; submitted October 20th, 2018 0.44
  33. 33 SNET QnA v2.1; submitted August 30th, 2018 0.44
  34. 34 BIDAF+ELMo+SofterMax QnA v2.1; submitted November 16th, 2018 0.44
  35. 35 DNET QnA v2.1; submitted August 1st, 2018 0.43
  36. 36 KIGN-QA QnA v2.1; submitted April 22nd, 2019 0.43
  37. 37 MaRCo-da-GAAMA QnA v2.1; submitted April 7th, 2020 0.43
  38. 38 Reader-Writer QnA v2.1; submitted September 16th, 2018 0.42
  39. 39 Masque2 (single / NLG Style) QnA v2.1; submitted October 22nd, 2020 0.42
  40. 40 BERT+Multi-Loss QnA v2.1; submitted November 4th, 2019 0.41
  41. 41 REAG(based on PALM) QnA v2.1; submitted June 1st,2020 0.41
  42. 42 RGLM QnA v2.1; submitted May 5th, 2020 0.41
  43. 43 SNET+seq2seq QnA v2.1; submitted June 1st, 2018 0.40
  44. 44 SSK3+BERTBaseAnswerGenerator QnA v2.1; submitted Jan 21st, 2020 0.39
  45. 45 MP-MRC BERT QnA v2.1; submitted Aug 27th, 2020 0.39
  46. 46 MP-MRC BERT-base QnA v2.1; submitted Sep 4th, 2020 0.39
  47. 47 MUSST QnA v2.1; submitted March 31st, 2020 0.38
  48. 48 fj-net(single) QnA v2.1; submitted August 3rd, 2020 0.34
  49. 49 MNet-Base(Single) NLGEN QnA v2.1; submitted July 8th, 2020 0.34
  50. 50 fj-reader(single) QnA v2.1; submitted July 28th, 2020 0.34
  51. 51 Generation with latent retrieval per answer QnA v2.1; submitted May 11th, 2020 0.34
  52. 52 MDCG-Base QnA v2.1; submitted June 8th, 2020 0.33
  53. 53 MDCC-Base QnA v2.1; submitted June 10th, 2020 0.33
  54. 54 Generation with latent retrieval Baseline 2 QnA v2.1; submitted May 11th, 2020 0.33
  55. 55 MDCC QnA v2.1; submitted June 10th, 2020 0.33
  56. 56 Generation with latent retrieval Baseline 1 QnA v2.1; submitted May 11th, 2020 0.30
  57. 57 MultiTask+DataAug+Unlikelihood QnA v2.1; submitted June 3rd, 2020 0.30
  58. 58 lightNLP+BiDAF QnA v2.1; submitted February 1st, 2019 0.30
  59. 59 MUSST-QA QnA v2.1; submitted June 1st, 2020 0.30
  60. 60 Pretrained seq2seq model QnA v2.1; submitted September 10th, 2020 0.29
  61. 61 roberta_T_tlx_90k QnA v2.1; submitted July 29th, 2020 0.29
  62. 62 BIDAF+seq2seq QnA v2.1; submitted May 29th, 2018 0.28
  63. 63 BiDaF Baseline(Implemented By MSMARCO Team) QnA v2.1; submitted April 23th, 2018 0.24
  64. 64 TrioNLP + BiDAF QnA v2.1; submitted September 23rd, 2018 0.20
  65. 65 BiDAF + LSTM QnA v2.1; submitted January 15th,2019 0.15

Source: Official MS MARCO Question Answering leaderboard

#EntryScoreSettingSubmitted byDateSource
1 Multi-doc Enriched BERT 0.54 QnA v2.1; submitted June 20th, 2019 Ming Yan of Alibaba Damo NLP Jun 2019 Official leaderboard
2 Selector+Combine-Content-Generator QA Model 0.53 QnA v2.1; submitted March 19th, 2019 Shengjie Qian of Caiyun xiaoyi AI and BUPT Mar 2019 Official leaderboard
3 Masque Q&A Style 0.52 QnA v2.1; submitted January 3rd, 2019 NTT Media Intelligence Laboratories [ Nishida et al. '19 ] Jan 2019 Official leaderboard
4 LM+Generator 0.52 QnA v2.1; submitted November 25th,2019 Alibaba Damo NLP Nov 2019 Official leaderboard
5 Deep Cascade QA 0.52 QnA v2.1; submitted December 12th, 2018 Ming Yan of Alibaba Damo NLP [ Yan et al. '18 ] Dec 2018 Official leaderboard
6 PALM 0.52 QnA v2.1; submitted December 9th,2019 Alibaba Damo NLP Dec 2019 Official leaderboard
7 Unnamed 0.52 QnA v2.1; submitted December 9th,2019 anonymous Dec 2019 Official leaderboard
8 VNET 0.52 QnA v2.1; submitted November 8th, 2018 Baidu NLP [ Wang et al. '18 ] Nov 2018 Official leaderboard
9 MultiLM QnA Model 0.51 QnA v2.1; submitted December 2nd, 2019 anonymous Dec 2019 Official leaderboard
10 LNET 0.51 QnA v2.1; submitted April 8th, 2020 S.L. Liu of NEUKG Apr 2020 Official leaderboard
11 BERT Encoded T-NET 0.51 QnA v2.1; submitted July 12th, 2019 Y. Zhang, C. Wang, X.L. Chen Jul 2019 Official leaderboard
12 BERT+ Multi-Pointer-Generator 0.50 QnA v2.1; submitted June 11th, 2019 Tongjun Li of the ColorfulClouds Tech and BUPT Jun 2019 Official leaderboard
13 Selector+Combine-Content-Generator NL Model 0.50 QnA v2.1; submitted March 11th, 2019 Shengjie Qian of Caiyun xiaoyi AI and BUPT Mar 2019 Official leaderboard
14 CompLM 0.49 QnA v2.1; submitted December 2nd, 2019 Alibaba Damo NLP Dec 2019 Official leaderboard
15 REAG 0.49 QnA v2.1; submitted March 27th, 2020 Anonymous Mar 2020 Official leaderboard
16 MUSST-NLG 0.49 QnA v2.1; submitted May 15th, 2020 Anonymous May 2020 Official leaderboard
17 Masque NLGEN Style 0.49 QnA v2.1; submitted January 3rd, 2019 NTT Media Intelligence Laboratories [ Nishida et al. '19 ] Jan 2019 Official leaderboard
18 Communicating BERT 0.48 QnA v2.1; submitted October 4th, 2019 Xuan Liang of RIDLL from the University of Technology Sydney Oct 2019 Official leaderboard
19 roberta_T_tlcd_18k 0.48 QnA v2.1; submitted May 14th, 2020 Anonymous May 2020 Official leaderboard
20 MultiLM NLGen Model 0.48 QnA v2.1; submitted December 2nd, 2019 anonymous Dec 2019 Official leaderboard
21 MDC-Generator 0.48 QnA v2.1; submitted April 23rd, 2020 Ssk-nlp Apr 2020 Official leaderboard
22 BERT + Transfer 0.47 QnA v2.1; submitted October 16th, 2019 anonymous Oct 2019 Official leaderboard
23 Bert Based Multi-task 0.47 QnA v2.1; submitted June 26th, 2019 ZhangY & WangC Jun 2019 Official leaderboard
24 T-RoBERTa-wf-BERTbaseA-120k 0.47 QnA v2.1; submitted February 13th, 2020 Anonymous Feb 2020 Official leaderboard
25 BERT-SS-K1-100k 0.47 QnA v2.1; submitted January 26th, 2020 Anonymous Jan 2020 Official leaderboard
26 T-RoBERTa-wf-BERTbaseA-80k 0.47 QnA v2.1; submitted February 21st, 2020 Anonymous Feb 2020 Official leaderboard
27 Multi-passage QA Model 0.47 QnA v2.1; submitted October 21st, 2020 SudaNLP Oct 2020 Official leaderboard
28 BERT-RGLM 0.46 QnA v2.1; submitted April 22nd, 2020 Anonymous Apr 2020 Official leaderboard
29 SNET + CES2S 0.45 QnA v2.1; submitted July 24th, 2018 Bo Shao of SYSU University Jul 2018 Official leaderboard
30 ranking+nlg 0.45 QnA v2.1; submitted October 9th, 2019 anonymous Oct 2019 Official leaderboard
31 ranker-reader 0.44 QnA v2.1; submitted May 15th, 2019 RCZoo of UCAS May 2019 Official leaderboard
32 Extraction-net 0.44 QnA v2.1; submitted October 20th, 2018 zlsh80826 Oct 2018 Official leaderboard
33 SNET 0.44 QnA v2.1; submitted August 30th, 2018 JY Zhao Aug 2018 Official leaderboard
34 BIDAF+ELMo+SofterMax 0.44 QnA v2.1; submitted November 16th, 2018 Wang Changbao Nov 2018 Official leaderboard
35 DNET 0.43 QnA v2.1; submitted August 1st, 2018 QA Geeks Aug 2018 Official leaderboard
36 KIGN-QA 0.43 QnA v2.1; submitted April 22nd, 2019 Chenliang Li Apr 2019 Official leaderboard
37 MaRCo-da-GAAMA 0.43 QnA v2.1; submitted April 7th, 2020 IBM Research AI Multilingual NLP Group Apr 2020 Official leaderboard
38 Reader-Writer 0.42 QnA v2.1; submitted September 16th, 2018 Microsoft Business Applications Group AI Research Sep 2018 Official leaderboard
39 Masque2 (single / NLG Style) 0.42 QnA v2.1; submitted October 22nd, 2020 NTT Media Intelligence Laboratories Oct 2020 Official leaderboard
40 BERT+Multi-Loss 0.41 QnA v2.1; submitted November 4th, 2019 S.L. Liu of NEUKG Nov 2019 Official leaderboard
41 REAG(based on PALM) 0.41 QnA v2.1; submitted June 1st,2020 anonymous Jun 2020 Official leaderboard
42 RGLM 0.41 QnA v2.1; submitted May 5th, 2020 anonymous May 2020 Official leaderboard
43 SNET+seq2seq 0.40 QnA v2.1; submitted June 1st, 2018 Yihan Ni of the CAS Key Lab of Web Data Science and Technology, ICT, CAS Jun 2018 Official leaderboard
44 SSK3+BERTBaseAnswerGenerator 0.39 QnA v2.1; submitted Jan 21st, 2020 anonymous Jan 2020 Official leaderboard
45 MP-MRC BERT 0.39 QnA v2.1; submitted Aug 27th, 2020 H.Y. Zhang Aug 2020 Official leaderboard
46 MP-MRC BERT-base 0.39 QnA v2.1; submitted Sep 4th, 2020 H.Y. Zhang Sep 2020 Official leaderboard
47 MUSST 0.38 QnA v2.1; submitted March 31st, 2020 anonymous Mar 2020 Official leaderboard
48 fj-net(single) 0.34 QnA v2.1; submitted August 3rd, 2020 yzm nlp group Aug 2020 Official leaderboard
49 MNet-Base(Single) NLGEN 0.34 QnA v2.1; submitted July 8th, 2020 fuii of iDW Jul 2020 Official leaderboard
50 fj-reader(single) 0.34 QnA v2.1; submitted July 28th, 2020 yzm nlp group Jul 2020 Official leaderboard
51 Generation with latent retrieval per answer 0.34 QnA v2.1; submitted May 11th, 2020 anonymous May 2020 Official leaderboard
52 MDCG-Base 0.33 QnA v2.1; submitted June 8th, 2020 ssk-nlp Jun 2020 Official leaderboard
53 MDCC-Base 0.33 QnA v2.1; submitted June 10th, 2020 ssk-nlp Jun 2020 Official leaderboard
54 Generation with latent retrieval Baseline 2 0.33 QnA v2.1; submitted May 11th, 2020 anonymous May 2020 Official leaderboard
55 MDCC 0.33 QnA v2.1; submitted June 10th, 2020 ssk-nlp Jun 2020 Official leaderboard
56 Generation with latent retrieval Baseline 1 0.30 QnA v2.1; submitted May 11th, 2020 anonymous May 2020 Official leaderboard
57 MultiTask+DataAug+Unlikelihood 0.30 QnA v2.1; submitted June 3rd, 2020 UvA Jun 2020 Official leaderboard
58 lightNLP+BiDAF 0.30 QnA v2.1; submitted February 1st, 2019 Enliple AI Feb 2019 Official leaderboard
59 MUSST-QA 0.30 QnA v2.1; submitted June 1st, 2020 Anonymous Jun 2020 Official leaderboard
60 Pretrained seq2seq model 0.29 QnA v2.1; submitted September 10th, 2020 BDEG Sep 2020 Official leaderboard
61 roberta_T_tlx_90k 0.29 QnA v2.1; submitted July 29th, 2020 Anonymous Jul 2020 Official leaderboard
62 BIDAF+seq2seq 0.28 QnA v2.1; submitted May 29th, 2018 Yihan Ni of the CAS Key Lab of Web Data Science and Technology, ICT, CAS May 2018 Official leaderboard
63 BiDaF Baseline(Implemented By MSMARCO Team) 0.24 QnA v2.1; submitted April 23th, 2018 Allen Institute for AI & University of Washington [ Seo et al. '16 ] Apr 2018 Official leaderboard
64 TrioNLP + BiDAF 0.20 QnA v2.1; submitted September 23rd, 2018 Trio.AI of the CCNU Sep 2018 Official leaderboard
65 BiDAF + LSTM 0.15 QnA v2.1; submitted January 15th,2019 Meefly Jan 2019 Official leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Knowledge

All 332 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. Seed1.8 92.3%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%

Data audit

All 8 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the official leaderboard (65 entries).
  • Pass Models scored 65 entries on its leaderboard, none a model we track.
  • Note Newest score No scores yet.
  • Note Frontier models No models scored yet.
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on microsoft.github.io: 65 entries at the last read, none a model we track.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check