Benchmark Almanac

RULER

Nemotron-Cascade 2 30B-A3B 99% leads since March 2026
Qwen2.5-14B-Instruct-1M 95.7% leads since January 2025
Nemotron-Cascade 2 30B-A3B 99% leads since March 2026
In use

Synthetic retrieval, multi-hop tracing, aggregation and QA tasks at configurable lengths. Measures effective context size.

Synthetic. Near-saturated for frontier models at 128K.

Made by
NVIDIA
Released
April 2024
Kind
Finding facts in long inputs
Level
Everyday
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Strongest model tested
Nemotron 3 Ultra #84 on the capability index
Last in a launch post
October 2026
Last model added
October 2026
Scores refreshed
November 12th, 2026

In the authors’ words

The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs.

Abstract of RULER: What's the Real Context Size of Your Long-Context Language Models? (arXiv:2404.06654, CC0)
42
Models scored in lab cards
16
Launch posts list it, from 9 labs
37
System cards report it, from 10 labs

Nemotron-Cascade 2 30B-A3B leads with 99%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Qwen2.5-14B-Instruct-1M leads with 95.7% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.

Nemotron-Cascade 2 30B-A3B leads with 99%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

42 models. Scroll sideways for the rest.

  1. 99.0 Nemotron-Cascade 2 30B-A3B
  2. 96.4 Gemma 4 31B IT
  3. 96.0 Mistral Medium 3
  4. 95.7 Qwen2.5-14B-Instruct-1M
  5. 95.1 Qwen2.5-72B
  6. 95.0 Cohere Command A
  7. 95.0 Qwen3-235B-A22B
  8. 94.7 Nemotron 3 Ultra
  9. 94.6 Qwen3-14B
  10. 94.6 seed-oss-36b-instruct
  11. 93.7 Qwen3-32B
  12. 93.1 Qwen-Turbo
  13. 92.9 Qwen2.5-32B
  14. 92.5 Nemotron 3 Ultra 550B-A55B Base
  15. 91.8 Nemotron 3 Super
  16. 91.6 Qwen3-30B-A3B
  17. 91.5 Nemotron 3.5 Lightning 30B-A3B
  18. 91.4 Qwen2.5-14B
  19. 91.1 Nemotron 3 Nano 4B
  20. 91.0 MiniMax-Text-01
  21. 89.8 Gemma 4 26B A4B
  22. 89.1 Qwen3-8B
  23. 88.2 EXAONE 4.0 32B
  24. 87.9 InternLM3-8B-Instruct
  25. 87.1 Phi-3.5-MoE
  26. 86.6 Gemma 4 E4B
  27. 86.3 Nemotron 3 Nano 30B-A3B
  28. 85.8 Nemotron 3.5 Bolt E4B
  29. 85.4 Qwen2.5-7B
  30. 85.2 Qwen3-4B
  31. 84.1 Phi-3.5-mini
  32. 81.4 Granite 4.2 30B
  33. 81.2 Mistral Small 3.1
  34. 78.9 nvidia-nemotron-nano-9b-v2
  35. 76.7 Hunyuan-A13B-Instruct
  36. 75.4 Nemotron 3 Nano 30B-A3B Base
  37. 72.1 Nemotron Nano 12B v2 VL
  38. 70.4 Gemma 4 E2B
  39. 66.0 Gemma 3 27B
  40. 65.6 LFM3-1.7B
  41. 57.1 Gemma 3 12B
  42. 46.8 Gemma 3 4B
  1. 1 Nemotron-Cascade 2 30B-A3B 99%
  2. 2 Gemma 4 31B IT 96.4%
  3. 3 Mistral Medium 3 96%
  4. 4 Qwen2.5-14B-Instruct-1M 95.7%
  5. 5 Qwen2.5-72B 95.1%
  6. 6 Cohere Command A 95%
  7. 7 Qwen3-235B-A22B 95%
  8. 8 Nemotron 3 Ultra 94.7%
  9. 9 Qwen3-14B 94.6%
  10. 10 seed-oss-36b-instruct 94.6%
  11. 11 Qwen3-32B 93.7%
  12. 12 Qwen-Turbo 93.1%
  13. 13 Qwen2.5-32B 92.9%
  14. 14 Nemotron 3 Ultra 550B-A55B Base 92.5%
  15. 15 Nemotron 3 Super 91.8%
  16. 16 Qwen3-30B-A3B 91.6%
  17. 17 Nemotron 3.5 Lightning 30B-A3B 91.5%
  18. 18 Qwen2.5-14B 91.4%
  19. 19 Nemotron 3 Nano 4B 91.1%
  20. 20 MiniMax-Text-01 91%
  21. 21 Gemma 4 26B A4B 89.8%
  22. 22 Qwen3-8B 89.1%
  23. 23 EXAONE 4.0 32B 88.2%
  24. 24 InternLM3-8B-Instruct 87.9%
  25. 25 Phi-3.5-MoE 87.1%
  26. 26 Gemma 4 E4B 86.6%
  27. 27 Nemotron 3 Nano 30B-A3B 86.3%
  28. 28 Nemotron 3.5 Bolt E4B 85.8%
  29. 29 Qwen2.5-7B 85.4%
  30. 30 Qwen3-4B 85.2%
  31. 31 Phi-3.5-mini 84.1%
  32. 32 Granite 4.2 30B 81.4%
  33. 33 Mistral Small 3.1 81.2%
  34. 34 nvidia-nemotron-nano-9b-v2 78.9%
  35. 35 Hunyuan-A13B-Instruct 76.7%
  36. 36 Nemotron 3 Nano 30B-A3B Base 75.4%
  37. 37 Nemotron Nano 12B v2 VL 72.1%
  38. 38 Gemma 4 E2B 70.4%
  39. 39 Gemma 3 27B 66%
  40. 40 LFM3-1.7B 65.6%
  41. 41 Gemma 3 12B 57.1%
  42. 42 Gemma 3 4B 46.8%

Source: Labs’ model cards

Source: Official RULER leaderboard

49 models. Scroll sideways for the rest.

  1. 99.0 Nemotron-Cascade 2 30B-A3B
  2. 96.4 Gemma 4 31B IT
  3. 96.0 Mistral Medium 3
  4. 95.7 Qwen2.5-14B-Instruct-1M
  5. 95.1 Qwen2.5-72B
  6. 95.0 Cohere Command A
  7. 95.0 Qwen3-235B-A22B
  8. 94.7 Nemotron 3 Ultra
  9. 94.6 Qwen3-14B
  10. 94.6 seed-oss-36b-instruct
  11. 93.7 Qwen3-32B
  12. 93.1 Qwen-Turbo
  13. 93.1 EXAONE 4.0 32B
  14. 92.9 Qwen2.5-32B
  15. 92.5 Nemotron 3 Ultra 550B-A55B Base
  16. 91.8 Nemotron 3 Super
  17. 91.6 Qwen3-30B-A3B
  18. 91.5 Nemotron 3.5 Lightning 30B-A3B
  19. 91.4 Qwen2.5-14B
  20. 91.1 Nemotron 3 Nano 4B
  21. 91.0 MiniMax-Text-01
  22. 89.8 Gemma 4 26B A4B
  23. 89.1 Qwen3-8B
  24. 88.9 c4ai-command-r-08-2024
  25. 88.3 Command R (03-2024)
  26. 87.9 InternLM3-8B-Instruct
  27. 87.1 Phi-3.5-MoE
  28. 86.6 Gemma 4 E4B
  29. 86.3 Nemotron 3 Nano 30B-A3B
  30. 85.9 Qwen2-72B
  31. 85.8 Nemotron 3.5 Bolt E4B
  32. 85.4 Qwen2.5-7B
  33. 85.2 Qwen3-4B
  34. 84.8 Phi-3-mini-128k-instruct
  35. 84.1 Phi-3.5-mini
  36. 81.5 Phi-3-medium-128k-instruct
  37. 81.4 Granite 4.2 30B
  38. 81.2 Mistral Small 3.1
  39. 78.9 nvidia-nemotron-nano-9b-v2
  40. 76.7 Hunyuan-A13B-Instruct
  41. 75.4 Nemotron 3 Nano 30B-A3B Base
  42. 72.1 Nemotron Nano 12B v2 VL
  43. 70.4 Gemma 4 E2B
  44. 68.4 Mistral-7B-Instruct-v0.2
  45. 66.0 Gemma 3 27B
  46. 65.6 LFM3-1.7B
  47. 57.1 Gemma 3 12B
  48. 55.7 Qwen1.5-72B
  49. 46.8 Gemma 3 4B
  1. 1 Nemotron-Cascade 2 30B-A3B 99%
  2. 2 Gemma 4 31B IT 96.4%
  3. 3 Mistral Medium 3 96%
  4. 4 Qwen2.5-14B-Instruct-1M 95.7%
  5. 5 Qwen2.5-72B 95.1%
  6. 6 Cohere Command A 95%
  7. 7 Qwen3-235B-A22B 95%
  8. 8 Nemotron 3 Ultra 94.7%
  9. 9 Qwen3-14B 94.6%
  10. 10 seed-oss-36b-instruct 94.6%
  11. 11 Qwen3-32B 93.7%
  12. 12 Qwen-Turbo 93.1%
  13. 13 EXAONE 4.0 32B 93.1%
  14. 14 Qwen2.5-32B 92.9%
  15. 15 Nemotron 3 Ultra 550B-A55B Base 92.5%
  16. 16 Nemotron 3 Super 91.8%
  17. 17 Qwen3-30B-A3B 91.6%
  18. 18 Nemotron 3.5 Lightning 30B-A3B 91.5%
  19. 19 Qwen2.5-14B 91.4%
  20. 20 Nemotron 3 Nano 4B 91.1%
  21. 21 MiniMax-Text-01 91%
  22. 22 Gemma 4 26B A4B 89.8%
  23. 23 Qwen3-8B 89.1%
  24. 24 c4ai-command-r-08-2024 88.9%
  25. 25 Command R (03-2024) 88.3%
  26. 26 InternLM3-8B-Instruct 87.9%
  27. 27 Phi-3.5-MoE 87.1%
  28. 28 Gemma 4 E4B 86.6%
  29. 29 Nemotron 3 Nano 30B-A3B 86.3%
  30. 30 Qwen2-72B 85.9%
  31. 31 Nemotron 3.5 Bolt E4B 85.8%
  32. 32 Qwen2.5-7B 85.4%
  33. 33 Qwen3-4B 85.2%
  34. 34 Phi-3-mini-128k-instruct 84.8%
  35. 35 Phi-3.5-mini 84.1%
  36. 36 Phi-3-medium-128k-instruct 81.5%
  37. 37 Granite 4.2 30B 81.4%
  38. 38 Mistral Small 3.1 81.2%
  39. 39 nvidia-nemotron-nano-9b-v2 78.9%
  40. 40 Hunyuan-A13B-Instruct 76.7%
  41. 41 Nemotron 3 Nano 30B-A3B Base 75.4%
  42. 42 Nemotron Nano 12B v2 VL 72.1%
  43. 43 Gemma 4 E2B 70.4%
  44. 44 Mistral-7B-Instruct-v0.2 68.4%
  45. 45 Gemma 3 27B 66%
  46. 46 LFM3-1.7B 65.6%
  47. 47 Gemma 3 12B 57.1%
  48. 48 Qwen1.5-72B 55.7%
  49. 49 Gemma 3 4B 46.8%

Sources: Labs’ model cards and official RULER leaderboard

#ModelScoreSettingDateSource
1 Nemotron-Cascade 2 30B-A3B 99% Mar 2026 Lab model card
2 Gemma 4 31B IT 96.4% Jul 2026 Lab model card
3 Mistral Medium 3 96% May 2025 Lab model card
4 Qwen2.5-14B-Instruct-1M 95.7% Jan 2025 Lab model card
5 Qwen2.5-72B 95.1% Dec 2024 Lab model card
6 Cohere Command A 95% Apr 2025 Lab model card
7 Qwen3-235B-A22B 95% May 2025 Lab model card
8 Nemotron 3 Ultra 94.7% Jun 2026 Lab model card
9 Qwen3-14B 94.6% May 2025 Lab model card
10 seed-oss-36b-instruct 94.6% Aug 2025 Lab model card
11 Qwen3-32B 93.7% May 2025 Lab model card
12 Qwen-Turbo 93.1% Dec 2024 Lab model card
13 Qwen2.5-32B 92.9% Dec 2024 Lab model card
14 Nemotron 3 Ultra 550B-A55B Base 92.5% Jun 2026 Lab model card
15 Nemotron 3 Super 91.8% Mar 2026 Lab model card
16 Qwen3-30B-A3B 91.6% May 2025 Lab model card
17 Nemotron 3.5 Lightning 30B-A3B 91.5% Oct 2026 Lab model card
18 Qwen2.5-14B 91.4% Dec 2024 Lab model card
19 Nemotron 3 Nano 4B 91.1% Mar 2026 Lab model card
20 MiniMax-Text-01 91% Jan 2025 Lab model card
21 Gemma 4 26B A4B 89.8% Jul 2026 Lab model card
22 Qwen3-8B 89.1% May 2025 Lab model card
23 EXAONE 4.0 32B 88.2% Jul 2025 Lab model card
24 InternLM3-8B-Instruct 87.9% Jan 2025 Lab model card
25 Phi-3.5-MoE 87.1% Aug 2024 Lab model card
26 Gemma 4 E4B 86.6% Jul 2026 Lab model card
27 Nemotron 3 Nano 30B-A3B 86.3% Dec 2025 Lab model card
28 Nemotron 3.5 Bolt E4B 85.8% Oct 2026 Lab model card
29 Qwen2.5-7B 85.4% Dec 2024 Lab model card
30 Qwen3-4B 85.2% May 2025 Lab model card
31 Phi-3.5-mini 84.1% Aug 2024 Lab model card
32 Granite 4.2 30B 81.4% Aug 2026 Lab model card
33 Mistral Small 3.1 81.2% Mar 2025 Lab model card
34 nvidia-nemotron-nano-9b-v2 78.9% Aug 2025 Lab model card
35 Hunyuan-A13B-Instruct 76.7% Jun 2025 Lab model card
36 Nemotron 3 Nano 30B-A3B Base 75.4% Dec 2025 Lab model card
37 Nemotron Nano 12B v2 VL 72.1% Nov 2025 Lab model card
38 Gemma 4 E2B 70.4% Jul 2026 Lab model card
39 Gemma 3 27B 66% Mar 2025 Lab model card
40 LFM3-1.7B 65.6% Oct 2026 Lab model card
41 Gemma 3 12B 57.1% Mar 2025 Lab model card
42 Gemma 3 4B 46.8% Mar 2025 Lab model card
#ModelScoreSettingDateSource
1 Qwen2.5-14B-Instruct-1M 95.7% Jan 2025 Official leaderboard
2 Qwen3-235B-A22B 95% Apr 2025 Official leaderboard
3 Qwen3-14B 94.6% Apr 2025 Official leaderboard
4 Qwen3-32B 93.7% Apr 2025 Official leaderboard
5 EXAONE 4.0 32B 93.1% Jul 2025 Official leaderboard
6 Qwen3-30B-A3B 91.6% Apr 2025 Official leaderboard
7 Qwen3-8B 89.1% Apr 2025 Official leaderboard
8 c4ai-command-r-08-2024 88.9% Aug 2024 Official leaderboard
9 Command R (03-2024) 88.3% Mar 2024 Official leaderboard
10 Qwen2-72B 85.9% Jun 2024 Official leaderboard
11 Qwen3-4B 85.2% Apr 2025 Official leaderboard
12 Phi-3-mini-128k-instruct 84.8% Apr 2024 Official leaderboard
13 Phi-3-medium-128k-instruct 81.5% May 2024 Official leaderboard
14 Mistral-7B-Instruct-v0.2 68.4% Dec 2023 Official leaderboard
15 Qwen1.5-72B 55.7% Feb 2024 Official leaderboard
#ModelScoreSettingDateSource
1 Nemotron-Cascade 2 30B-A3B 99% Mar 2026 Lab model card
2 Gemma 4 31B IT 96.4% Jul 2026 Lab model card
3 Mistral Medium 3 96% May 2025 Lab model card
4 Qwen2.5-14B-Instruct-1M 95.7% Jan 2025 Official leaderboard
5 Qwen2.5-72B 95.1% Dec 2024 Lab model card
6 Cohere Command A 95% Apr 2025 Lab model card
7 Qwen3-235B-A22B 95% Apr 2025 Official leaderboard
8 Nemotron 3 Ultra 94.7% Jun 2026 Lab model card
9 Qwen3-14B 94.6% Apr 2025 Official leaderboard
10 seed-oss-36b-instruct 94.6% Aug 2025 Lab model card
11 Qwen3-32B 93.7% Apr 2025 Official leaderboard
12 Qwen-Turbo 93.1% Dec 2024 Lab model card
13 EXAONE 4.0 32B 93.1% Jul 2025 Official leaderboard
14 Qwen2.5-32B 92.9% Dec 2024 Lab model card
15 Nemotron 3 Ultra 550B-A55B Base 92.5% Jun 2026 Lab model card
16 Nemotron 3 Super 91.8% Mar 2026 Lab model card
17 Qwen3-30B-A3B 91.6% Apr 2025 Official leaderboard
18 Nemotron 3.5 Lightning 30B-A3B 91.5% Oct 2026 Lab model card
19 Qwen2.5-14B 91.4% Dec 2024 Lab model card
20 Nemotron 3 Nano 4B 91.1% Mar 2026 Lab model card
21 MiniMax-Text-01 91% Jan 2025 Lab model card
22 Gemma 4 26B A4B 89.8% Jul 2026 Lab model card
23 Qwen3-8B 89.1% Apr 2025 Official leaderboard
24 c4ai-command-r-08-2024 88.9% Aug 2024 Official leaderboard
25 Command R (03-2024) 88.3% Mar 2024 Official leaderboard
26 InternLM3-8B-Instruct 87.9% Jan 2025 Lab model card
27 Phi-3.5-MoE 87.1% Aug 2024 Lab model card
28 Gemma 4 E4B 86.6% Jul 2026 Lab model card
29 Nemotron 3 Nano 30B-A3B 86.3% Dec 2025 Lab model card
30 Qwen2-72B 85.9% Jun 2024 Official leaderboard
31 Nemotron 3.5 Bolt E4B 85.8% Oct 2026 Lab model card
32 Qwen2.5-7B 85.4% Dec 2024 Lab model card
33 Qwen3-4B 85.2% Apr 2025 Official leaderboard
34 Phi-3-mini-128k-instruct 84.8% Apr 2024 Official leaderboard
35 Phi-3.5-mini 84.1% Aug 2024 Lab model card
36 Phi-3-medium-128k-instruct 81.5% May 2024 Official leaderboard
37 Granite 4.2 30B 81.4% Aug 2026 Lab model card
38 Mistral Small 3.1 81.2% Mar 2025 Lab model card
39 nvidia-nemotron-nano-9b-v2 78.9% Aug 2025 Lab model card
40 Hunyuan-A13B-Instruct 76.7% Jun 2025 Lab model card
41 Nemotron 3 Nano 30B-A3B Base 75.4% Dec 2025 Lab model card
42 Nemotron Nano 12B v2 VL 72.1% Nov 2025 Lab model card
43 Gemma 4 E2B 70.4% Jul 2026 Lab model card
44 Mistral-7B-Instruct-v0.2 68.4% Dec 2023 Official leaderboard
45 Gemma 3 27B 66% Mar 2025 Lab model card
46 LFM3-1.7B 65.6% Oct 2026 Lab model card
47 Gemma 3 12B 57.1% Mar 2025 Lab model card
48 Qwen1.5-72B 55.7% Feb 2024 Official leaderboard
49 Gemma 3 4B 46.8% Mar 2025 Lab model card

The benchmark's own leaderboard has 44 entries, 15 of them models we track. Switch the scores above to rank them together.

5 models have held number one in labs’ own numbers since its release in April 2024 3 models have held number one across all independent runs since its release in April 2024 6 models have held number one with claims included since its release in April 2024

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Cohere Other labs Qwen Microsoft Mistral AI NVIDIA
Own leaderboard Passed 85% in Apr 2024, by Command R (03-2024)
Qwen2.5-14B-Instruct-1M 95.7%
Lab claims Passed 85% in Aug 2024, by Phi-3.5-MoE
Nemotron-Cascade 2 30B-A3B 99%

Sources: Official RULER leaderboard and labs’ model cards

Command R (03-2024) on the official leaderboard passed 85% in March 2024

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Nemotron-Cascade 2 30B-A3B Mistral Medium 3 c4ai-command-r-08-2024 Mistral-7B-Instruct-v0.2 0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Nemotron-Cascade 2 30B-A3B c4ai-command-r-08-2024

Sources: Official RULER leaderboard and labs’ model cards

3 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Shanghai AI Lab
MiniMax
Qwen
Mistral AI
LG AI Research
NVIDIA
ByteDance Seed
IBM
Liquid AI
DateLaunch postScoreAs the post writes it
Oct 27, 2026 LFM3-1.7B 65.6 RULER 128K
Oct 23, 2026 Nemotron 3.5 Bolt E4B 85.8 RULER (64K, non-reasoning)
Aug 25, 2026 Granite 4.2 30B 81.4 RULER 128K (64K 89.96)
Jun 4, 2026 Nemotron 3 Ultra 94.7 RULER (1M)
Mar 19, 2026 Nemotron-Cascade 2 30B-A3B 99 NIAH@1M (RULER Subset)
Mar 16, 2026 Nemotron 3 Nano 4B 91.1 RULER (128k, reasoning off)
Mar 11, 2026 Nemotron 3 Super 91.8 RULER @ 1M
Dec 15, 2025 Nemotron 3 Nano 30B-A3B 86.3 RULER-100@1M
Aug 20, 2025 seed-oss-36b-instruct 94.6 RULER (128K)
Aug 18, 2025 nvidia-nemotron-nano-9b-v2 78.9 RULER (128K)
Jul 15, 2025 EXAONE 4.0 32B 88.2 RULER (non-reasoning)
May 7, 2025 Mistral Medium 3 96 Best of RULER 32K / RULER 128K
Mar 17, 2025 Mistral Small 3.1 81.2 RULER 128K (RULER 32K 93.96)
Jan 27, 2025 Qwen2.5-14B-Instruct-1M 95.7 RULER Avg. (4K to 128K)
Jan 15, 2025 MiniMax-Text-01 91 Ruler (1M)
Jan 13, 2025 InternLM3-8B-Instruct 87.9 RULER(4-128K Average)

37 system cards report it, from 10 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 23, 2026 Nemotron 3.5 Bolt E4B technical report 85.8 RULER (64K), non-reasoning, released Q4_K_M imatrix checkpoint (Table 5). BF16 86.99 (Table 3) Capabilities
Oct 15, 2026 Nemotron 3.5 Lightning 30B-A3B technical report 91.5 RULER-v1 (1M), BF16 acc. (Table 22) Capabilities
Jul 24, 2026 Gemma 4 E4B technical report 86.6 RULER accuracy, 128k, without thinking (Table 9) Capabilities
Jul 24, 2026 Gemma 4 E2B technical report 70.4 RULER accuracy, 128k, without thinking (Table 9) Capabilities
Jul 24, 2026 Gemma 4 31B IT technical report 96.4 RULER accuracy, 128k, without thinking (Table 9). 32k: 96.8 Capabilities
Jul 24, 2026 Gemma 4 26B A4B technical report 89.8 RULER accuracy, 128k, without thinking (Table 9). 32k: 97.3 Capabilities
Jun 9, 2026 Nemotron 3 Ultra technical report 94.7 RULER (1M) (Table 10) Capabilities
Jun 4, 2026 Nemotron 3 Ultra 550B-A55B Base model card 92.5 RULER 128K, 0-shot (64K 95.30, 256K 86.22, 512K 84.54, 1M 76.83) Capabilities
Mar 19, 2026 Nemotron-Cascade 2 30B-A3B technical report 99 NIAH@1M (RULER Subset), four NIAH variants at 1M tokens (Table 1 and A.5) Capabilities
Mar 11, 2026 Nemotron 3 Super technical report 91.6 RULER 1M (Table 5). 256k 96.83, 512k 95.22, and 128k 97.04 in Table 8 BF16 Capabilities
Dec 23, 2025 Nemotron 3 Nano 30B-A3B technical report 86.3 RULER-100 @ 1M, reasoning off (Table 3). 256k 92.92, 512K 91.25 Capabilities
Dec 23, 2025 Nemotron 3 Nano 30B-A3B Base technical report 75.4 RULER, 256K 0-shot accuracy, 64K 87.50 and 128K 82.92, base model Table 2 Capabilities
Nov 6, 2025 Nemotron Nano 12B v2 VL technical report 72.1 RULER, reasoning on, SFT stage 4, length not stated (Table 6) Capabilities
Sep 30, 2025 Code World Model (CWM) technical report RULER, no overall score: 32k 84.3, 128k 69.7 (Table 14) Capabilities
Sep 17, 2025 Apertus 70B technical report RULER by context length with no overall (Table 23): 4k 94.8, 8k 89.9, 16k 85.7, 32k 81.9, 64k 67.3 Capabilities
Aug 20, 2025 nvidia-nemotron-nano-9b-v2 technical report 78.9 RULER 128k, NVIDIA-Nemotron-Nano-9B-v2 (Figure 1) Capabilities
Jul 15, 2025 EXAONE 4.0 32B technical report 88.2 RULER, 128K, Non-reasoning only (Table 4). 4K 96.26 to 64K 91.73 in Table 10 Capabilities
Jun 26, 2025 Hunyuan-A13B-Instruct technical report 76.7 RULER QA sub-task Avg., 0-8K 78.7, 8K-32K 75.3, 32K-64K 78.0, 64K-128K 73.9 (Table 6) Capabilities
May 14, 2025 Qwen3-8B technical report 89.1 RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 77.4. Thinking mode Avg. 84.4 with 8192-token thinking budget (Table 23) Capabilities
May 14, 2025 Qwen3-4B technical report 85.2 RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 66.0. Thinking mode Avg. 83.5 with 8192-token thinking budget (Table 23) Capabilities
May 14, 2025 Qwen3-32B technical report 93.7 RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 85.6. Thinking mode Avg. 91.0 with 8192-token thinking budget (Table 23) Capabilities
May 14, 2025 Qwen3-30B-A3B technical report 91.6 RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 79.2. Thinking mode Avg. 86.6 with 8192-token thinking budget (Table 23) Capabilities
May 14, 2025 Qwen3-235B-A22B technical report 95 RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 90.6. Thinking mode Avg. 92.2 with 8192-token thinking budget (Table 23) Capabilities
May 14, 2025 Qwen3-14B technical report 94.6 RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 85.1. Thinking mode Avg. 90.1 with 8192-token thinking budget (Table 23) Capabilities
Apr 1, 2025 Cohere Command A technical report 95 RULER average up to 128k (Table 21). 128k 90.0, 256k 84.6 Capabilities
Mar 25, 2025 Gemma 3 4B technical report 46.8 RULER 128K, IT (Table 15). RULER 32K IT: 61.4 Capabilities
Mar 25, 2025 Gemma 3 27B technical report 66 RULER 128K, IT (Table 15). RULER 32K IT: 91.1 Capabilities
Mar 25, 2025 Gemma 3 12B technical report 57.1 RULER 128K, IT (Table 15). RULER 32K IT: 80.3 Capabilities
Jan 26, 2025 Qwen2.5-14B-Instruct-1M technical report 95.7 RULER Avg. (4K to 128K) (Table 4). 4K 97.5, 8K 97.1, 16K 94.6, 32K 94.9, 64K 94.9, 128K 92.2 Capabilities
Jan 14, 2025 MiniMax-Text-01 technical report 91 Ruler, 1M column, 0.910 (Table 9). 128k 0.947, 64k 0.943 Capabilities
Dec 19, 2024 Qwen2.5-7B technical report 85.4 RULER Avg. 4K to 128K, with YARN+DCA, 128K 55.1 (Table 16) Capabilities
Dec 19, 2024 Qwen2.5-72B technical report 95.1 RULER Avg. 4K to 128K, with YARN+DCA, 128K 88.4 (Table 16) Capabilities
Dec 19, 2024 Qwen2.5-32B technical report 92.9 RULER Avg. 4K to 128K, with YARN+DCA, 128K 82.0 (Table 16) Capabilities
Dec 19, 2024 Qwen2.5-14B technical report 91.4 RULER Avg. 4K to 128K, with YARN+DCA, 128K 78.1 (Table 16) Capabilities
Dec 19, 2024 Qwen-Turbo technical report 93.1 RULER Avg. 4K to 128K, 128K 84.5 (Table 16) Capabilities
Aug 30, 2024 Phi-3.5-MoE technical report 87.1 RULER, average over 4k to 128k (Table 2), 128k alone 64.2 Capabilities
Aug 30, 2024 Phi-3.5-mini technical report 84.1 RULER, average over 4k to 128k (Table 2), 128k alone 63.6 Capabilities

8 lab claims next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 7 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
EXAONE 4.0 32B 88.2% RULER (non-reasoning) 93.1% Leaderboard −4.9 pts
Qwen2.5-14B-Instruct-1M 95.7% RULER Avg. (4K to 128K) 95.7% Leaderboard Same
Qwen3-14B 94.6% RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 85.1. Thinking mode Avg. 90.1 with 8192-token thinking budget (Table 23) 94.6% Leaderboard Same
Qwen3-235B-A22B 95% RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 90.6. Thinking mode Avg. 92.2 with 8192-token thinking budget (Table 23) 95% Leaderboard Same
Qwen3-30B-A3B 91.6% RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 79.2. Thinking mode Avg. 86.6 with 8192-token thinking budget (Table 23) 91.6% Leaderboard Same
Qwen3-32B 93.7% RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 85.6. Thinking mode Avg. 91.0 with 8192-token thinking budget (Table 23) 93.7% Leaderboard Same
Qwen3-4B 85.2% RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 66.0. Thinking mode Avg. 83.5 with 8192-token thinking budget (Table 23) 85.2% Leaderboard Same
Qwen3-8B 89.1% RULER Avg. (4K to 128K, YaRN x4), non-thinking mode, 128K 77.4. Thinking mode Avg. 84.4 with 8192-token thinking budget (Table 23) 89.1% Leaderboard Same

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

Microsoft Qwen Mistral AI NVIDIA OpenAI
RULERv2 Passed 85% in Oct 2025, by GPT-4.1
GPT-4.1 89.2%
RULER Passed 85% in Aug 2024, by Phi-3.5-MoE
Nemotron-Cascade 2 30B-A3B 99%
RULERv2 Passed 85% in Oct 2025, by GPT-4.1
GPT-4.1 89.2%

Sources: Official leaderboards and labs’ model cards

Replaced by RULERv2

More in Long context

All 79 in Long context
  • MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
  • LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Inkling 2 Small 70.6%
  • GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
  • OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. Gemini 3.7 Flash 89%
  • CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
  • AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key. Step 5 Preview 88.3%