NevIR
Models distinguish documents whose meanings differ because one passage negates a statement.
Each pair has queries that favor opposite passages.
- Made by
- Orion Weller and colleagues
- Released
- May 2023
- Kind
- Questions that need search
- Level
- School
- Scored
- Checked automatically
- Answers
- Prediction
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
MonoT5 3B tops the leaderboard with 50.6%
Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 50.6 MonoT5 3B
- 45.8 MonoT5 large
- 34.9 MonoT5 base (default)
- 34.1 qnli-electra-base
- 30.2 nli-deberta-v3-base
- 27.7 MonoT5 small
- 26.3 RocketQA v1
- 24.9 stsb-roberta-large
- 22.4 RocketQA v2
- 19.7 ColBERTv1
- 13.0 ColBERTv2
- 11.1 multi-qa-mpnet-base-dot-v1
- 8.70 msmarco-distilbert-cos-v5
- 8.70 SPLADEv2 self-distill
- 8.10 all-mpnet-base-v2
- 8.00 nq-distilbert-base-v1
- 8.00 SPLADEv2 ensemble-distill
- 7.70 coCondenser
- 6.90 msmarco-bert-base-dot-v5
- 6.80 DPR
- 2.00 TF-IDF
- 1 MonoT5 3B Cross-encoder 50.6%
- 2 MonoT5 large Cross-encoder 45.8%
- 3 MonoT5 base (default) Cross-encoder 34.9%
- 4 qnli-electra-base Cross-encoder 34.1%
- 5 nli-deberta-v3-base Cross-encoder 30.2%
- 6 MonoT5 small Cross-encoder 27.7%
- 7 RocketQA v1 Cross-encoder 26.3%
- 8 stsb-roberta-large Cross-encoder 24.9%
- 9 RocketQA v2 Cross-encoder 22.4%
- 10 ColBERTv1 Late interaction 19.7%
- 11 ColBERTv2 Late interaction 13%
- 12 multi-qa-mpnet-base-dot-v1 Bi-encoder 11.1%
- 13 msmarco-distilbert-cos-v5 Bi-encoder 8.70%
- 14 SPLADEv2 self-distill Sparse 8.70%
- 15 all-mpnet-base-v2 Bi-encoder 8.10%
- 16 nq-distilbert-base-v1 Bi-encoder 8.00%
- 17 SPLADEv2 ensemble-distill Sparse 8.00%
- 18 coCondenser Bi-encoder 7.70%
- 19 msmarco-bert-base-dot-v5 Bi-encoder 6.90%
- 20 DPR Bi-encoder 6.80%
- 21 TF-IDF Sparse 2.00%
Source: The NevIR paper
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | MonoT5 3B 2.9B | 50.6% | Cross-encoder | Official leaderboard | |
| 2 | MonoT5 large 0.7B | 45.8% | Cross-encoder | Official leaderboard | |
| 3 | MonoT5 base (default) 0.2B | 34.9% | Cross-encoder | Official leaderboard | |
| 4 | qnli-electra-base 0.1B | 34.1% | Cross-encoder | Official leaderboard | |
| 5 | nli-deberta-v3-base 0.2B | 30.2% | Cross-encoder | Official leaderboard | |
| 6 | MonoT5 small 0.1B | 27.7% | Cross-encoder | Official leaderboard | |
| 7 | RocketQA v1 0.3B | 26.3% | Cross-encoder | Official leaderboard | |
| 8 | stsb-roberta-large 0.4B | 24.9% | Cross-encoder | Official leaderboard | |
| 9 | RocketQA v2 0.1B | 22.4% | Cross-encoder | Official leaderboard | |
| 10 | ColBERTv1 0.1B | 19.7% | Late interaction | Official leaderboard | |
| 11 | ColBERTv2 0.1B | 13% | Late interaction | Official leaderboard | |
| 12 | multi-qa-mpnet-base-dot-v1 0.1B | 11.1% | Bi-encoder | Official leaderboard | |
| 13 | msmarco-distilbert-cos-v5 0.1B | 8.70% | Bi-encoder | Official leaderboard | |
| 14 | SPLADEv2 self-distill 0.1B | 8.70% | Sparse | Official leaderboard | |
| 15 | all-mpnet-base-v2 0.1B | 8.10% | Bi-encoder | Official leaderboard | |
| 16 | nq-distilbert-base-v1 0.1B | 8.00% | Bi-encoder | Official leaderboard | |
| 17 | SPLADEv2 ensemble-distill 0.1B | 8.00% | Sparse | Official leaderboard | |
| 18 | coCondenser 0.1B | 7.70% | Bi-encoder | Official leaderboard | |
| 19 | msmarco-bert-base-dot-v5 0.1B | 6.90% | Bi-encoder | Official leaderboard | |
| 20 | DPR 0.2B | 6.80% | Bi-encoder | Official leaderboard | |
| 21 | TF-IDF | 2.00% | Sparse | Official leaderboard |
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the benchmark’s paper (21 entries).
- Pass Models scored 21 entries on its leaderboard, none a model we track.
- Note Newest score No scores yet.
- Note Frontier models No models scored yet.
- Note Headroom No scores yet.
Leaderboard import 3 pass
- Pass Importer We read the board on aclanthology.org: 21 entries at the last read, none a model we track.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.