Benchmark Almanac

AIR-Bench 24.05

CMIT_retriever_test_1202_No 0.74 tops its leaderboard
Not in model cards
Version 24.05

Retrievers find documents or passages that answer questions. This release covers 69 datasets across 13 languages.

The official repository has 168 stars and links the results for this release.

Released
October 2024
Kind
Questions that need search
Level
University
Scored
Checked automatically
Answers
Prediction
Tasks
All public
Measures
Capability
Language
Many languages
In and out
Text → Text
Interaction
Single turn
Run by
Anyone
Based on
AIR-Bench 2024
In views
Many languages
111
Entries in its paper, 0 models we track

CMIT_retriever_test_1202_No tops the leaderboard with 0.74

Best score per entry from the benchmark's paper, read on November 21st, 2026. A paper’s table doesn’t add newer models. The paper reports no overall score, so this is the QA nDCG@10 column, which it puts before Long-Doc Recall@10. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

111 entries. Scroll sideways for the rest.

  1. 0.74 CMIT_retriever_test_1202_No
  2. 0.74 CMIT_retriever_test_1202_NoReranker
  3. 0.73 CMIT_retriever_test_1113
  4. 0.68 INF_test_20241230
  5. 0.68 inf-retriever-v1
  6. 0.66 inf-retriever-v1-1.5b
  7. 0.61 dense_250616_test_a
  8. 0.60 dense_250728_test_esm
  9. 0.60 dense_0513_md
  10. 0.60 dense_0306_test_f
  11. 0.60 dense_250616_test_r
  12. 0.59 dense_0506_multi_dev
  13. 0.59 dense_0305_test
  14. 0.59 dense_250728_test_es
  15. 0.59 BM25
  16. 0.58 dense_250728_test_e
  17. 0.58 dense_0513_mx
  18. 0.58 0606_q3_dev
  19. 0.57 bge-en-icl_few-shot
  20. 0.56 dense_0427_test_t
  21. 0.56 test_qwen3_emb_8b
  22. 0.56 dense_0506_multi
  23. 0.56 dense_0427_test_d
  24. 0.56 dense_250616_test
  25. 0.55 dense_0425_test
  26. 0.55 dense_0423
  27. 0.55 dense_250526_test_3
  28. 0.55 dense_0424_test
  29. 0.55 dense_0306_test
  30. 0.55 dense_0425_test_old
  31. 0.54 bge-en-icl_zero-shot
  32. 0.54 bge-multilingual-gemma2
  33. 0.54 dense_0430_multi_f
  34. 0.53 dense_0417
  35. 0.53 NV-Embed-v2
  36. 0.53 dense_250617_test_m
  37. 0.53 SFR-Embedding-Mistral
  38. 0.53 dense_250617_test
  39. 0.52 dense_0427_test_f
  40. 0.52 dense_250526_test_2
  41. 0.52 u_test_0610
  42. 0.51 bge-m3
  43. 0.51 test-leaderboard-2025-11-07
  44. 0.51 multilingual-e5-large-instruct
  45. 0.51 NV-Embed-v1
  46. 0.51 repllama-v1-7b-lora-passage
  47. 0.51 SFR-Embedding-2_R
  48. 0.50 gte-Qwen2-7B-instruct
  49. 0.50 dense_0214_test
  50. 0.50 TEST_0418
  51. 0.50 test_qwen3_emb_0.6b
  52. 0.50 Linq-Embed-Mistral
  53. 0.50 TEST_0417
  54. 0.49 TRY_EMB_TEST
  55. 0.49 dense_0411_test
  56. 0.49 dense_0430_fr_d
  57. 0.49 0506_fr_dev
  58. 0.49 TRY_EMB_1117
  59. 0.49 KaLM-Embedding-V2.5
  60. 0.49 TRY_EMB_1116
  61. 0.48 jina-embeddings-v3
  62. 0.48 dense_250611_test
  63. 0.48 e5-mistral-7b-instruct
  64. 0.48 test_bixse_qwen2
  65. 0.48 dense_0403_test
  66. 0.47 gte-multilingual-base
  67. 0.47 multilingual-e5-large
  68. 0.47 dense_250728_test_as
  69. 0.47 dense_250728_test_asm
  70. 0.46 e5-large-v2
  71. 0.46 gte-large-en-v1.5
  72. 0.45 gte-large
  73. 0.45 bge-large-en-v1.5
  74. 0.45 anon4
  75. 0.45 gte-base
  76. 0.45 gte-Qwen2-1.5B-instruct
  77. 0.44 multilingual-e5-base
  78. 0.44 dense_250728_test_a
  79. 0.44 gtr-t5-xxl
  80. 0.44 e5-base-v2
  81. 0.44 qa_healthcare_zh_v0_xxxx
  82. 0.43 gtr-t5-xl
  83. 0.43 gte-small
  84. 0.43 Qwen-embedding-M3.3.9step10k-D250505InfoNCE-C1.1.1
  85. 0.43 e5-small-v2
  86. 0.43 bge-base-zh-v1.5
  87. 0.43 bge-base-en-v1.5
  88. 0.43 gtr-t5-large
  89. 0.42 bge-small-en-v1.5
  90. 0.42 CMIT_retriever_test_1204_NoReranker
  91. 0.42 dense_0409_test
  92. 0.41 multilingual-e5-small
  93. 0.41 bge-large-zh-v1.5
  94. 0.40 gtr-t5-base
  95. 0.39 CMIT_leaderboard_1219
  96. 0.39 gte-base-zh
  97. 0.36 bge-small-zh-v1.5
  98. 0.35 IT_retriever_simple_1217_NoReranker
  99. 0.35 gte-large-zh
  100. 0.34 CMIT_retriever_test_1219
  101. 0.34 CMIT_retriever_1219
  102. 0.34 IT_retriever_V3
  103. 0.34 gte-small-zh
  104. 0.34 CMIT_retriever_test_1205_NoReranker
  105. 0.33 aari1995/gte-german-rag-sts-all
  106. 0.32 IT_retriever_V2
  107. 0.26 IT_retriever_1219
  108. 0.24 bce-embedding-base_v1
  109. 0.20 CMIT_retriever_test_1203_NoReranker
  110. 0.19 anon5
  111. 0.09 anon3
  1. 1 CMIT_retriever_test_1202_No 0.74
  2. 2 CMIT_retriever_test_1202_NoReranker 0.74
  3. 3 CMIT_retriever_test_1113 0.73
  4. 4 INF_test_20241230 0.68
  5. 5 inf-retriever-v1 0.68
  6. 6 inf-retriever-v1-1.5b 0.66
  7. 7 dense_250616_test_a 0.61
  8. 8 dense_250728_test_esm 0.60
  9. 9 dense_0513_md 0.60
  10. 10 dense_0306_test_f 0.60
  11. 11 dense_250616_test_r 0.60
  12. 12 dense_0506_multi_dev 0.59
  13. 13 dense_0305_test 0.59
  14. 14 dense_250728_test_es 0.59
  15. 15 BM25 0.59
  16. 16 dense_250728_test_e 0.58
  17. 17 dense_0513_mx 0.58
  18. 18 0606_q3_dev 0.58
  19. 19 bge-en-icl_few-shot 0.57
  20. 20 dense_0427_test_t 0.56
  21. 21 test_qwen3_emb_8b 0.56
  22. 22 dense_0506_multi 0.56
  23. 23 dense_0427_test_d 0.56
  24. 24 dense_250616_test 0.56
  25. 25 dense_0425_test 0.55
  26. 26 dense_0423 0.55
  27. 27 dense_250526_test_3 0.55
  28. 28 dense_0424_test 0.55
  29. 29 dense_0306_test 0.55
  30. 30 dense_0425_test_old 0.55
  31. 31 bge-en-icl_zero-shot 0.54
  32. 32 bge-multilingual-gemma2 0.54
  33. 33 dense_0430_multi_f 0.54
  34. 34 dense_0417 0.53
  35. 35 NV-Embed-v2 0.53
  36. 36 dense_250617_test_m 0.53
  37. 37 SFR-Embedding-Mistral 0.53
  38. 38 dense_250617_test 0.53
  39. 39 dense_0427_test_f 0.52
  40. 40 dense_250526_test_2 0.52
  41. 41 u_test_0610 0.52
  42. 42 bge-m3 0.51
  43. 43 test-leaderboard-2025-11-07 0.51
  44. 44 multilingual-e5-large-instruct 0.51
  45. 45 NV-Embed-v1 0.51
  46. 46 repllama-v1-7b-lora-passage 0.51
  47. 47 SFR-Embedding-2_R 0.51
  48. 48 gte-Qwen2-7B-instruct 0.50
  49. 49 dense_0214_test 0.50
  50. 50 TEST_0418 0.50
  51. 51 test_qwen3_emb_0.6b 0.50
  52. 52 Linq-Embed-Mistral 0.50
  53. 53 TEST_0417 0.50
  54. 54 TRY_EMB_TEST 0.49
  55. 55 dense_0411_test 0.49
  56. 56 dense_0430_fr_d 0.49
  57. 57 0506_fr_dev 0.49
  58. 58 TRY_EMB_1117 0.49
  59. 59 KaLM-Embedding-V2.5 0.49
  60. 60 TRY_EMB_1116 0.49
  61. 61 jina-embeddings-v3 0.48
  62. 62 dense_250611_test 0.48
  63. 63 e5-mistral-7b-instruct 0.48
  64. 64 test_bixse_qwen2 0.48
  65. 65 dense_0403_test 0.48
  66. 66 gte-multilingual-base 0.47
  67. 67 multilingual-e5-large 0.47
  68. 68 dense_250728_test_as 0.47
  69. 69 dense_250728_test_asm 0.47
  70. 70 e5-large-v2 0.46
  71. 71 gte-large-en-v1.5 0.46
  72. 72 gte-large 0.45
  73. 73 bge-large-en-v1.5 0.45
  74. 74 anon4 0.45
  75. 75 gte-base 0.45
  76. 76 gte-Qwen2-1.5B-instruct 0.45
  77. 77 multilingual-e5-base 0.44
  78. 78 dense_250728_test_a 0.44
  79. 79 gtr-t5-xxl 0.44
  80. 80 e5-base-v2 0.44
  81. 81 qa_healthcare_zh_v0_xxxx 0.44
  82. 82 gtr-t5-xl 0.43
  83. 83 gte-small 0.43
  84. 84 Qwen-embedding-M3.3.9step10k-D250505InfoNCE-C1.1.1 0.43
  85. 85 e5-small-v2 0.43
  86. 86 bge-base-zh-v1.5 0.43
  87. 87 bge-base-en-v1.5 0.43
  88. 88 gtr-t5-large 0.43
  89. 89 bge-small-en-v1.5 0.42
  90. 90 CMIT_retriever_test_1204_NoReranker 0.42
  91. 91 dense_0409_test 0.42
  92. 92 multilingual-e5-small 0.41
  93. 93 bge-large-zh-v1.5 0.41
  94. 94 gtr-t5-base 0.40
  95. 95 CMIT_leaderboard_1219 0.39
  96. 96 gte-base-zh 0.39
  97. 97 bge-small-zh-v1.5 0.36
  98. 98 IT_retriever_simple_1217_NoReranker 0.35
  99. 99 gte-large-zh 0.35
  100. 100 CMIT_retriever_test_1219 0.34
  101. 101 CMIT_retriever_1219 0.34
  102. 102 IT_retriever_V3 0.34
  103. 103 gte-small-zh 0.34
  104. 104 CMIT_retriever_test_1205_NoReranker 0.34
  105. 105 aari1995/gte-german-rag-sts-all 0.33
  106. 106 IT_retriever_V2 0.32
  107. 107 IT_retriever_1219 0.26
  108. 108 bce-embedding-base_v1 0.24
  109. 109 CMIT_retriever_test_1203_NoReranker 0.20
  110. 110 anon5 0.19
  111. 111 anon3 0.09

Source: The AIR-Bench 24.05 paper

#EntryScoreSettingDateSource
1 CMIT_retriever_test_1202_No 0.74 NoReranker Dec 2025 Official leaderboard
2 CMIT_retriever_test_1202_NoReranker 0.74 NoReranker Dec 2025 Official leaderboard
3 CMIT_retriever_test_1113 0.73 NoReranker Nov 2025 Official leaderboard
4 INF_test_20241230 0.68 bge-reranker-v2-m3 Jan 2025 Official leaderboard
5 inf-retriever-v1 0.68 bge-reranker-v2-m3 Jan 2025 Official leaderboard
6 inf-retriever-v1-1.5b 0.66 bge-reranker-v2-m3 Feb 2025 Official leaderboard
7 dense_250616_test_a 0.61 NoReranker Jun 2025 Official leaderboard
8 dense_250728_test_esm 0.60 NoReranker Jul 2025 Official leaderboard
9 dense_0513_md 0.60 NoReranker May 2025 Official leaderboard
10 dense_0306_test_f 0.60 NoReranker Mar 2025 Official leaderboard
11 dense_250616_test_r 0.60 NoReranker Jun 2025 Official leaderboard
12 dense_0506_multi_dev 0.59 NoReranker May 2025 Official leaderboard
13 dense_0305_test 0.59 NoReranker Mar 2025 Official leaderboard
14 dense_250728_test_es 0.59 NoReranker Jul 2025 Official leaderboard
15 BM25 0.59 bge-reranker-v2-m3 Dec 2024 Official leaderboard
16 dense_250728_test_e 0.58 NoReranker Jul 2025 Official leaderboard
17 dense_0513_mx 0.58 NoReranker May 2025 Official leaderboard
18 0606_q3_dev 0.58 NoReranker Jun 2025 Official leaderboard
19 bge-en-icl_few-shot 0.57 NoReranker Dec 2024 Official leaderboard
20 dense_0427_test_t 0.56 NoReranker Apr 2025 Official leaderboard
21 test_qwen3_emb_8b 0.56 NoReranker Jun 2025 Official leaderboard
22 dense_0506_multi 0.56 NoReranker May 2025 Official leaderboard
23 dense_0427_test_d 0.56 NoReranker Apr 2025 Official leaderboard
24 dense_250616_test 0.56 NoReranker Jun 2025 Official leaderboard
25 dense_0425_test 0.55 NoReranker Apr 2025 Official leaderboard
26 dense_0423 0.55 NoReranker Apr 2025 Official leaderboard
27 dense_250526_test_3 0.55 NoReranker May 2025 Official leaderboard
28 dense_0424_test 0.55 NoReranker Apr 2025 Official leaderboard
29 dense_0306_test 0.55 NoReranker Mar 2025 Official leaderboard
30 dense_0425_test_old 0.55 NoReranker Apr 2025 Official leaderboard
31 bge-en-icl_zero-shot 0.54 NoReranker Dec 2024 Official leaderboard
32 bge-multilingual-gemma2 0.54 NoReranker Dec 2024 Official leaderboard
33 dense_0430_multi_f 0.54 NoReranker Apr 2025 Official leaderboard
34 dense_0417 0.53 NoReranker Apr 2025 Official leaderboard
35 NV-Embed-v2 0.53 NoReranker Dec 2024 Official leaderboard
36 dense_250617_test_m 0.53 NoReranker Jun 2025 Official leaderboard
37 SFR-Embedding-Mistral 0.53 NoReranker Dec 2024 Official leaderboard
38 dense_250617_test 0.53 NoReranker Jun 2025 Official leaderboard
39 dense_0427_test_f 0.52 NoReranker Apr 2025 Official leaderboard
40 dense_250526_test_2 0.52 NoReranker May 2025 Official leaderboard
41 u_test_0610 0.52 NoReranker Jun 2025 Official leaderboard
42 bge-m3 0.51 NoReranker Dec 2024 Official leaderboard
43 test-leaderboard-2025-11-07 0.51 NoReranker Nov 2025 Official leaderboard
44 multilingual-e5-large-instruct 0.51 NoReranker Dec 2024 Official leaderboard
45 NV-Embed-v1 0.51 NoReranker Dec 2024 Official leaderboard
46 repllama-v1-7b-lora-passage 0.51 NoReranker Dec 2024 Official leaderboard
47 SFR-Embedding-2_R 0.51 NoReranker Dec 2024 Official leaderboard
48 gte-Qwen2-7B-instruct 0.50 NoReranker Dec 2024 Official leaderboard
49 dense_0214_test 0.50 NoReranker Feb 2025 Official leaderboard
50 TEST_0418 0.50 NoReranker Apr 2025 Official leaderboard
51 test_qwen3_emb_0.6b 0.50 NoReranker Jul 2025 Official leaderboard
52 Linq-Embed-Mistral 0.50 NoReranker Dec 2024 Official leaderboard
53 TEST_0417 0.50 NoReranker Apr 2025 Official leaderboard
54 TRY_EMB_TEST 0.49 NoReranker Nov 2025 Official leaderboard
55 dense_0411_test 0.49 NoReranker Apr 2025 Official leaderboard
56 dense_0430_fr_d 0.49 NoReranker Apr 2025 Official leaderboard
57 0506_fr_dev 0.49 NoReranker May 2025 Official leaderboard
58 TRY_EMB_1117 0.49 NoReranker Nov 2025 Official leaderboard
59 KaLM-Embedding-V2.5 0.49 NoReranker Jan 2026 Official leaderboard
60 TRY_EMB_1116 0.49 NoReranker Nov 2025 Official leaderboard
61 jina-embeddings-v3 0.48 NoReranker Oct 2024 Official leaderboard
62 dense_250611_test 0.48 NoReranker Jun 2025 Official leaderboard
63 e5-mistral-7b-instruct 0.48 NoReranker Dec 2024 Official leaderboard
64 test_bixse_qwen2 0.48 NoReranker Nov 2024 Official leaderboard
65 dense_0403_test 0.48 NoReranker Apr 2025 Official leaderboard
66 gte-multilingual-base 0.47 NoReranker Dec 2024 Official leaderboard
67 multilingual-e5-large 0.47 NoReranker Dec 2024 Official leaderboard
68 dense_250728_test_as 0.47 NoReranker Jul 2025 Official leaderboard
69 dense_250728_test_asm 0.47 NoReranker Jul 2025 Official leaderboard
70 e5-large-v2 0.46 NoReranker Dec 2024 Official leaderboard
71 gte-large-en-v1.5 0.46 NoReranker Dec 2024 Official leaderboard
72 gte-large 0.45 NoReranker Dec 2024 Official leaderboard
73 bge-large-en-v1.5 0.45 NoReranker Dec 2024 Official leaderboard
74 anon4 0.45 NoReranker Sep 2025 Official leaderboard
75 gte-base 0.45 NoReranker Dec 2024 Official leaderboard
76 gte-Qwen2-1.5B-instruct 0.45 NoReranker Dec 2024 Official leaderboard
77 multilingual-e5-base 0.44 NoReranker Dec 2024 Official leaderboard
78 dense_250728_test_a 0.44 NoReranker Jul 2025 Official leaderboard
79 gtr-t5-xxl 0.44 NoReranker Dec 2024 Official leaderboard
80 e5-base-v2 0.44 NoReranker Dec 2024 Official leaderboard
81 qa_healthcare_zh_v0_xxxx 0.44 NoReranker Apr 2025 Official leaderboard
82 gtr-t5-xl 0.43 NoReranker Dec 2024 Official leaderboard
83 gte-small 0.43 NoReranker Dec 2024 Official leaderboard
84 Qwen-embedding-M3.3.9step10k-D250505InfoNCE-C1.1.1 0.43 bge-reranker-v2-m3 May 2025 Official leaderboard
85 e5-small-v2 0.43 NoReranker Dec 2024 Official leaderboard
86 bge-base-zh-v1.5 0.43 NoReranker Dec 2024 Official leaderboard
87 bge-base-en-v1.5 0.43 NoReranker Dec 2024 Official leaderboard
88 gtr-t5-large 0.43 NoReranker Dec 2024 Official leaderboard
89 bge-small-en-v1.5 0.42 NoReranker Dec 2024 Official leaderboard
90 CMIT_retriever_test_1204_NoReranker 0.42 NoReranker Dec 2025 Official leaderboard
91 dense_0409_test 0.42 NoReranker Apr 2025 Official leaderboard
92 multilingual-e5-small 0.41 NoReranker Dec 2024 Official leaderboard
93 bge-large-zh-v1.5 0.41 NoReranker Dec 2024 Official leaderboard
94 gtr-t5-base 0.40 NoReranker Dec 2024 Official leaderboard
95 CMIT_leaderboard_1219 0.39 NoReranker Dec 2025 Official leaderboard
96 gte-base-zh 0.39 NoReranker Dec 2024 Official leaderboard
97 bge-small-zh-v1.5 0.36 NoReranker Dec 2024 Official leaderboard
98 IT_retriever_simple_1217_NoReranker 0.35 NoReranker Dec 2025 Official leaderboard
99 gte-large-zh 0.35 NoReranker Dec 2024 Official leaderboard
100 CMIT_retriever_test_1219 0.34 NoReranker Dec 2025 Official leaderboard
101 CMIT_retriever_1219 0.34 NoReranker Dec 2025 Official leaderboard
102 IT_retriever_V3 0.34 NoReranker Dec 2025 Official leaderboard
103 gte-small-zh 0.34 NoReranker Dec 2024 Official leaderboard
104 CMIT_retriever_test_1205_NoReranker 0.34 NoReranker Dec 2025 Official leaderboard
105 aari1995/gte-german-rag-sts-all 0.33 NoReranker Sep 2025 Official leaderboard
106 IT_retriever_V2 0.32 NoReranker Dec 2025 Official leaderboard
107 IT_retriever_1219 0.26 NoReranker Dec 2025 Official leaderboard
108 bce-embedding-base_v1 0.24 NoReranker Dec 2024 Official leaderboard
109 CMIT_retriever_test_1203_NoReranker 0.20 NoReranker Dec 2025 Official leaderboard
110 anon5 0.19 NoReranker Sep 2025 Official leaderboard
111 anon3 0.09 NoReranker Sep 2025 Official leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

Versions

More in Knowledge

All 332 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. Seed1.8 92.3%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%

Data audit

All 8 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the benchmark’s paper (111 entries).
  • Pass Models scored 111 entries on its leaderboard, none a model we track.
  • Note Newest score No scores yet.
  • Note Frontier models No models scored yet.
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on huggingface.co: 111 entries at the last read, none a model we track.
  • Pass Last read November 21st, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check