Benchmark Almanac

ArxivRollBench

Kimi K2.6 70.8% leads since April 2026
Not in model cards

Models reconstruct passages from recently published scientific papers. They order fragments, fill gaps and choose what comes next.

Made by
Zi Liang and colleagues
Released
July 2025
Kind
Reading comprehension
Level
University
Scored
Checked automatically
Answers
Multiple choice
Tasks
Public, plus a private test set
Measures
Capability
Language
English
In and out
Text → Text
Interaction
Single turn
Items from
Real work and use
Run by
A host
In views
Kept fresh
Strongest model tested
GPT-5.5 #21 on the capability index
Last model added
May 2026
Scores refreshed
November 20th, 2026
39
Entries on its own leaderboard, 31 models we track

Kimi K2.6 leads with 70.8% among models released by May 2026 Kimi K2.6 leads with 70.8% among models released by May 2026

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in May 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Kimi K2.6 2026a, 50-item task samples 70.8%
  2. 2 Gemini 3.5 Flash 2026a, 50-item task samples 55.2%
  3. 3 GPT-5.5 2026a, 50-item task samples 54.4%
  4. 4 Claude Opus 4.7 2026a, 50-item task samples 53.4%
  5. 5 DeepSeek-V4-Pro 2026a, 50-item task samples 53.3%
  6. 6 Grok 4.3 2026a, 50-item task samples 52%
  7. 7 Claude Sonnet 4.6 2026a, 50-item task samples 48.3%
  8. 8 Kimi K2.5 2026a, 50-item task samples 46%
  9. 9 DeepSeek-V4-Flash 2026a, 50-item task samples 45.3%
  10. 10 DeepSeek-R1 May 2025, 2026a, 50-item task samples 43.2%
  11. 11 Claude Opus 4.6 2026a, 50-item task samples 42.9%
  12. 12 GPT-5.3 Codex 2026a, 50-item task samples 42.1%
  13. 13 Claude Haiku 4.5 2026a, 50-item task samples 40.8%
  14. 14 GPT-5.4 2026a, 50-item task samples 39.3%
  15. 15 Qwen3-14B 2026a, full tasks 38.5%
  16. 16 deepseek/deepseek-chat-v3-0324 2026a, 50-item task samples 37.6%
  17. 17 GPT-5.4 Mini 2026a, 50-item task samples 35.5%
  18. 18 Grok 4.20 2026a, 50-item task samples 34.5%
  19. 19 DeepSeek-V3.1 2026a, 50-item task samples 33.9%
  20. 20 DeepSeek-V3.2 2026a, 50-item task samples 31.9%
  21. 21 Qwen/Qwen2.5-7B-Instruct 2026a, full tasks 30%
  22. 22 Qwen2-7B-Instruct 2026a, full tasks 29.5%
  23. 23 Qwen3-8B 2026a, full tasks 29.3%
  24. 24 Qwen2.5-7B 2026a, full tasks 28.2%
  25. 25 meta-llama/Llama-3.1-8B-Instruct 2026a, full tasks 27.7%
  26. 26 Qwen3-4B 2026a, full tasks 27.4%
  27. 27 mistralai/Mistral-7B-Instruct-v0.3 2026a, full tasks 26.2%
  28. 28 Mistral-7B-Instruct-v0.1 2026a, full tasks 25.8%
  29. 29 Falcon3-10B-Instruct 2026a, full tasks 23.8%
  30. 30 Mistral-7B-Instruct-v0.2 2026a, full tasks 23.7%
  31. 31 Llama 3.2 3B 2026a, full tasks 23.4%
  32. 32 meta-llama/Llama-3.2-3B-Instruct 2026a, full tasks 23.2%
  33. 33 Llama 3.2 1B 2026a, full tasks 22.1%
  34. 34 Llama 3.1-8B 2026a, full tasks 21.9%
  35. 35 Qwen/Qwen2.5-Math-7B 2026a, full tasks 21.6%
  36. 36 Phi-4-Reasoning-plus 2026a, full tasks 21.2%
  37. 37 meta-llama/Llama-3.2-1B-Instruct 2026a, full tasks 12.6%
  38. 38 Phi-4-reasoning 2026a, full tasks 8.25%
  39. 39 Qwen/Qwen2.5-Math-7B-Instruct 2026a, full tasks 7.55%

Source: Official ArxivRollBench leaderboard

Source: Official ArxivRollBench leaderboard

#EntryScoreSettingDateSource
1 Kimi K2.6 70.8% 2026a, 50-item task samples Apr 2026 Official leaderboard
2 Gemini 3.5 Flash 55.2% 2026a, 50-item task samples May 2026 Official leaderboard
3 GPT-5.5 54.4% 2026a, 50-item task samples Apr 2026 Official leaderboard
4 Claude Opus 4.7 53.4% 2026a, 50-item task samples Apr 2026 Official leaderboard
5 DeepSeek-V4-Pro 53.3% 2026a, 50-item task samples Apr 2026 Official leaderboard
6 Grok 4.3 52% 2026a, 50-item task samples Apr 2026 Official leaderboard
7 Claude Sonnet 4.6 48.3% 2026a, 50-item task samples Feb 2026 Official leaderboard
8 Kimi K2.5 46% 2026a, 50-item task samples Jan 2026 Official leaderboard
9 DeepSeek-V4-Flash 45.3% 2026a, 50-item task samples Apr 2026 Official leaderboard
10 DeepSeek-R1 May 2025 43.2% 2026a, 50-item task samples May 2025 Official leaderboard
11 Claude Opus 4.6 42.9% 2026a, 50-item task samples Feb 2026 Official leaderboard
12 GPT-5.3 Codex 42.1% 2026a, 50-item task samples Feb 2026 Official leaderboard
13 Claude Haiku 4.5 40.8% 2026a, 50-item task samples Oct 2025 Official leaderboard
14 GPT-5.4 39.3% 2026a, 50-item task samples Mar 2026 Official leaderboard
15 Qwen3-14B 38.5% 2026a, full tasks Apr 2025 Official leaderboard
16 deepseek/deepseek-chat-v3-0324 37.6% 2026a, 50-item task samples Official leaderboard
17 GPT-5.4 Mini 35.5% 2026a, 50-item task samples Mar 2026 Official leaderboard
18 Grok 4.20 34.5% 2026a, 50-item task samples Feb 2026 Official leaderboard
19 DeepSeek-V3.1 33.9% 2026a, 50-item task samples Aug 2025 Official leaderboard
20 DeepSeek-V3.2 31.9% 2026a, 50-item task samples Dec 2025 Official leaderboard
21 Qwen/Qwen2.5-7B-Instruct 30% 2026a, full tasks Official leaderboard
22 Qwen2-7B-Instruct 29.5% 2026a, full tasks Jun 2024 Official leaderboard
23 Qwen3-8B 29.3% 2026a, full tasks Apr 2025 Official leaderboard
24 Qwen2.5-7B 28.2% 2026a, full tasks Sep 2024 Official leaderboard
25 meta-llama/Llama-3.1-8B-Instruct 27.7% 2026a, full tasks Official leaderboard
26 Qwen3-4B 27.4% 2026a, full tasks Apr 2025 Official leaderboard
27 mistralai/Mistral-7B-Instruct-v0.3 26.2% 2026a, full tasks Official leaderboard
28 Mistral-7B-Instruct-v0.1 25.8% 2026a, full tasks Sep 2023 Official leaderboard
29 Falcon3-10B-Instruct 23.8% 2026a, full tasks Dec 2024 Official leaderboard
30 Mistral-7B-Instruct-v0.2 23.7% 2026a, full tasks Dec 2023 Official leaderboard
31 Llama 3.2 3B 23.4% 2026a, full tasks Sep 2024 Official leaderboard
32 meta-llama/Llama-3.2-3B-Instruct 23.2% 2026a, full tasks Official leaderboard
33 Llama 3.2 1B 22.1% 2026a, full tasks Sep 2024 Official leaderboard
34 Llama 3.1-8B 21.9% 2026a, full tasks Jul 2024 Official leaderboard
35 Qwen/Qwen2.5-Math-7B 21.6% 2026a, full tasks Official leaderboard
36 Phi-4-Reasoning-plus 21.2% 2026a, full tasks Apr 2025 Official leaderboard
37 meta-llama/Llama-3.2-1B-Instruct 12.6% 2026a, full tasks Official leaderboard
38 Phi-4-reasoning 8.25% 2026a, full tasks Apr 2025 Official leaderboard
39 Qwen/Qwen2.5-Math-7B-Instruct 7.55% 2026a, full tasks Official leaderboard
#ModelScoreSettingDateSource
1 Kimi K2.6 70.8% 2026a, 50-item task samples Apr 2026 Official leaderboard
2 Gemini 3.5 Flash 55.2% 2026a, 50-item task samples May 2026 Official leaderboard
3 GPT-5.5 54.4% 2026a, 50-item task samples Apr 2026 Official leaderboard
4 Claude Opus 4.7 53.4% 2026a, 50-item task samples Apr 2026 Official leaderboard
5 DeepSeek-V4-Pro 53.3% 2026a, 50-item task samples Apr 2026 Official leaderboard
6 Grok 4.3 52% 2026a, 50-item task samples Apr 2026 Official leaderboard
7 Claude Sonnet 4.6 48.3% 2026a, 50-item task samples Feb 2026 Official leaderboard
8 Kimi K2.5 46% 2026a, 50-item task samples Jan 2026 Official leaderboard
9 DeepSeek-V4-Flash 45.3% 2026a, 50-item task samples Apr 2026 Official leaderboard
10 DeepSeek-R1 May 2025 43.2% 2026a, 50-item task samples May 2025 Official leaderboard
11 Claude Opus 4.6 42.9% 2026a, 50-item task samples Feb 2026 Official leaderboard
12 GPT-5.3 Codex 42.1% 2026a, 50-item task samples Feb 2026 Official leaderboard
13 Claude Haiku 4.5 40.8% 2026a, 50-item task samples Oct 2025 Official leaderboard
14 GPT-5.4 39.3% 2026a, 50-item task samples Mar 2026 Official leaderboard
15 Qwen3-14B 38.5% 2026a, full tasks Apr 2025 Official leaderboard
16 GPT-5.4 Mini 35.5% 2026a, 50-item task samples Mar 2026 Official leaderboard
17 Grok 4.20 34.5% 2026a, 50-item task samples Feb 2026 Official leaderboard
18 DeepSeek-V3.1 33.9% 2026a, 50-item task samples Aug 2025 Official leaderboard
19 DeepSeek-V3.2 31.9% 2026a, 50-item task samples Dec 2025 Official leaderboard
20 Qwen2-7B-Instruct 29.5% 2026a, full tasks Jun 2024 Official leaderboard
21 Qwen3-8B 29.3% 2026a, full tasks Apr 2025 Official leaderboard
22 Qwen2.5-7B 28.2% 2026a, full tasks Sep 2024 Official leaderboard
23 Qwen3-4B 27.4% 2026a, full tasks Apr 2025 Official leaderboard
24 Mistral-7B-Instruct-v0.1 25.8% 2026a, full tasks Sep 2023 Official leaderboard
25 Falcon3-10B-Instruct 23.8% 2026a, full tasks Dec 2024 Official leaderboard
26 Mistral-7B-Instruct-v0.2 23.7% 2026a, full tasks Dec 2023 Official leaderboard
27 Llama 3.2 3B 23.4% 2026a, full tasks Sep 2024 Official leaderboard
28 Llama 3.2 1B 22.1% 2026a, full tasks Sep 2024 Official leaderboard
29 Llama 3.1-8B 21.9% 2026a, full tasks Jul 2024 Official leaderboard
30 Phi-4-Reasoning-plus 21.2% 2026a, full tasks Apr 2025 Official leaderboard
31 Phi-4-reasoning 8.25% 2026a, full tasks Apr 2025 Official leaderboard

Rankings by part

We read 24 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as cs: Cloze, cs: Prediction and cs: Sequencing. Kimi K2.6 leads 16 of them, and 4 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.

cs: Cloze

31 models we track have a score here.

  1. 1 Kimi K2.6 50%
  2. 2 Mistral-7B-Instruct-v0.1 ↑22 28.3%
  3. 3 Qwen2-7B-Instruct ↑17 25.4%
  4. 4 Qwen2.5-7B ↑18 25.3%
  5. 5 DeepSeek-R1 May 2025 ↑5 25%
  6. 6 DeepSeek-V4-Flash ↑3 25%
  7. 7 Qwen3-4B ↑16 24.6%
  8. 8 Qwen3-14B ↑7 24.2%
  9. 9 Llama 3.2 3B ↑18 24.1%
  10. 10 Qwen3-8B ↑11 24%

5 models have held number one on the official leaderboard since its release in July 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

DeepSeek Moonshot AI Anthropic
Own leaderboard Not tested since May 2026
Kimi K2.6 70.8%

Source: Official ArxivRollBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Kimi K2.6 Claude Opus 4.7 Claude Sonnet 4.6 DeepSeek-R1 Qwen3-14B Qwen2-7B-Instruct Mistral-7B-Instruct-v0.1 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Kimi K2.6 Claude Opus 4.7

Source: Official ArxivRollBench leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Knowledge

All 332 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. Seed1.8 92.3%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%

Data audit

9 of 10 checks pass, 1 warning. Look first at the Newest score check. The newest came 6 months ago, in May 2026, for Gemini 3.5 Flash. Every benchmark

Catalog record 2 pass
  • Pass Task access Public tasks, plus a private test set.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 3 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (39 entries).
  • Pass Models scored 31 models we track, plus 8 other entries on its leaderboard.
  • Warning Newest score The newest came 6 months ago, in May 2026, for Gemini 3.5 Flash.
  • Note Frontier models None of the top 10 has a score. The strongest tested is GPT-5.5 (#21).
  • Pass Headroom The best is 70.8%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on arxivroll.moreoverai.com: 31 rows for models we track and 8 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check