Benchmark Almanac

AA-LCR 1.1

Step 5 Preview 88.3% leads since September 2026
In use
Version 1.1

Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.

Version 1.1 corrects 16 answer keys and adds a judge system prompt. Questions and documents are unchanged, but the maker says 1.0 and 1.1 scores are not directly comparable.

Released
September 2026
Kind
Reasoning over long documents
Level
University
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Measures
Capability
Language
English
In and out
Text → Text
Interaction
Single turn
Run by
Anyone
Last in a launch post
November 2026
Last model added
November 2026
7
Models scored in lab cards
6
Launch posts list it, from 6 labs
2
System cards report it, from 2 labs

Step 5 Preview leads with 88.3%

Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.

  1. 1 Step 5 Preview 88.3%
  2. 2 Solar Mini 4 83.3%
  3. 3 Inkling 2 Small 83.3%
  4. 4 Inkling 2 Mini 80%
  5. 5 Mercury 3 77.7%
  6. 6 Nemotron 3.5 Super VL 68.6%
  7. 7 Spark-X2.6-4B 67.3%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Step 5 Preview 88.3% Sep 2026 Lab model card
2 Solar Mini 4 83.3% Oct 2026 Lab model card
3 Inkling 2 Small 83.3% Nov 2026 Lab model card
4 Inkling 2 Mini 80% Nov 2026 Lab model card
5 Mercury 3 77.7% Oct 2026 Lab model card
6 Nemotron 3.5 Super VL 68.6% Oct 2026 Lab model card
7 Spark-X2.6-4B 67.3% Oct 2026 Lab model card

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

StepFun
Lab claims Passed 85% in Sep 2026, by Step 5 Preview

Source: Labs’ model cards

StepFun’s own number for Step 5 Preview passed 85% in September 2026

Models released before it came out already scored that high. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
Reported by the lab in its model card
0% 20% 40% 60% 80% 100% 2027 Near the ceiling (85%) Step 5 Preview 0% 20% 40% 60% 80% 100% 2027 Near the ceiling (85%) Step 5 Preview

Source: Labs’ model cards

6 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

StepFun
Upstage
NVIDIA
Inception Labs
iFlytek
Thinking Machines
DateLaunch postScoreAs the post writes it
Nov 5, 2026 Inkling 2 Small 83.3 AA-LCR v1.1
Oct 22, 2026 Spark-X2.6-4B 67.3 AA-LCRv1.1
Oct 19, 2026 Mercury 3 77.7 AA-LCR v1.1
Oct 15, 2026 Nemotron 3.5 Super VL 68.6 AA-LCR v1.1
Oct 1, 2026 Solar Mini 4 83.3 AA-LCR (Artificial Analysis score quoted by Upstage)
Sep 20, 2026 Step 5 Preview 88.3 AA-LCR v1.1

2 system cards report it, from 2 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 20, 2026 Inkling 2 Mini model card 80 AA-LCR v1.1 Capabilities
Oct 15, 2026 Nemotron 3.5 Super VL technical report 68.6 AA-LCR v1.1, mean of 16 (Table 10) Capabilities

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

MiniMax Qwen Moonshot AI Mistral AI Tencent Meta StepFun
AA-LCR 1.1 Passed 85% in Sep 2026, by Step 5 Preview
AA-LCR 1.1 Passed 85% in Sep 2026, by Step 5 Preview

Source: Labs’ model cards

More in Long context

All 115 in Long context
  • MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
  • LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Inkling 2 Small 70.6%
  • GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
  • OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. Gemini 3.7 Flash 89%
  • CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
  • LOCA-Bench In use Agentic tasks whose environment state is scaled to grow context length while task semantics stay fixed. Success rate versus context length. GPT-5.2 21.3%

Data audit

All 5 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Note Independent runs Only labs’ own numbers, so no one else has checked them. Scores from launch posts (6 scores) and system cards (2 scores).
  • Pass Models scored 7 models we track.
  • Pass Newest score The newest is from November 2026, for Inkling 2 Mini.
  • Note Frontier models None of its models are on the capabilities index.
  • Note Headroom Saturated. The best is 88.3%, past 85% since September 2026.
Leaderboard import 1 note
  • Note Importer Artificial Analysis allows reuse of only its index numbers, so we link to its board instead.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check