AA-LCR (Artificial Analysis Long Context Reasoning)
Version 1.0
Questions requiring reasoning across multiple real documents totaling about 100K tokens. Answers judged by an LLM.
Version 1.0.0 stays available at dataset revision bdae010. Version 1.1 (September 2026) fixed 16 answer keys, so scores differ.
- Made by
- Artificial Analysis
- Released
- August 2025
- Kind
- Reasoning over long documents
- Level
- University
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Real work and use
- Refresh
- New versions replace it
- Run by
- Anyone
- Strongest model tested
- Kimi K3 #25 on the capability index
- Last in a launch post
- October 2026
- Last model added
- October 2026
Muse Glimmer leads with 80%
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
- 80.0
Muse Glimmer
- 79.3 Beam 1.0
- 74.7 Kimi K3
- 73.4
Tencent Hy3
- 72.0
MiniMax-M2.7
- 71.2
Mistral Small 4
- 71.0
Solar Pro 4
- 70.0 Kimi K2.5
- 69.5
MiniMax-M2.5
- 68.7
Qwen3-Max-2026-01-23
- 68.7
Qwen3.5 397B-A17B
- 68.3
Qwen 3.6 Plus
- 68.0 Mercury 2.5
- 66.9
Qwen3.5-122B-A10B
- 66.3
Tencent Hy3 preview
- 66.1
Qwen3.5-27B
- 65.4
Nemotron 3 Ultra
- 64.1
Nemotron 3.5 Bolt 74B-A9B
- 63.9
Step 3.7 Flash
- 63.0
Qwen3.5-9B
- 62.3
Solar Open 2
- 62.0
MiniMax-M2.1
- 61.0
MiniMax-M2
- 58.5
Qwen3.5-35B-A3B
- 58.3
Nemotron 3 Super
- 57.0
Qwen3.5-4B
- 57.0
Qwen3.5-Omni-Plus
- 56.2
K-EXAONE 2.0
- 53.5
K-EXAONE
- 52.8 LFM3-1.7B
- 52.4
Nemotron 3.5 Lightning 30B-A3B
- 50.6
EXAONE 4.5 33B
- 41.0
Nemotron 3 Nano Omni 30B-A3B
- 39.1
Nemotron-Cascade 2 30B-A3B
- 35.9
Nemotron 3 Nano 30B-A3B
- 35.0
Solar Open 100B
- 25.6
Qwen3.5-2B
- 4.70
Qwen3.5-0.8B
- 1
Muse Glimmer 80%
- 2 Beam 1.0 79.3%
- 3 Kimi K3 74.7%
- 4
Tencent Hy3 73.4%
- 5
MiniMax-M2.7 72%
- 6
Mistral Small 4 71.2%
- 7
Solar Pro 4 71%
- 8 Kimi K2.5 70%
- 9
MiniMax-M2.5 69.5%
- 10
Qwen3-Max-2026-01-23 68.7%
- 11
Qwen3.5 397B-A17B 68.7%
- 12
Qwen 3.6 Plus 68.3%
- 13 Mercury 2.5 68%
- 14
Qwen3.5-122B-A10B 66.9%
- 15
Tencent Hy3 preview 66.3%
- 16
Qwen3.5-27B 66.1%
- 17
Nemotron 3 Ultra 65.4%
- 18
Nemotron 3.5 Bolt 74B-A9B 64.1%
- 19
Step 3.7 Flash 63.9%
- 20
Qwen3.5-9B 63%
- 21
Solar Open 2 62.3%
- 22
MiniMax-M2.1 62%
- 23
MiniMax-M2 61%
- 24
Qwen3.5-35B-A3B 58.5%
- 25
Nemotron 3 Super 58.3%
- 26
Qwen3.5-4B 57%
- 27
Qwen3.5-Omni-Plus 57%
- 28
K-EXAONE 2.0 56.2%
- 29
K-EXAONE 53.5%
- 30 LFM3-1.7B 52.8%
- 31
Nemotron 3.5 Lightning 30B-A3B 52.4%
- 32
EXAONE 4.5 33B 50.6%
- 33
Nemotron 3 Nano Omni 30B-A3B 41%
- 34
Nemotron-Cascade 2 30B-A3B 39.1%
- 35
Nemotron 3 Nano 30B-A3B 35.9%
- 36
Solar Open 100B 35%
- 37
Qwen3.5-2B 25.6%
- 38
Qwen3.5-0.8B 4.70%
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 80% | Aug 2026 | Lab model card | |
| 2 | Beam 1.0 | 79.3% | Oct 2026 | Lab model card | |
| 3 | Kimi K3 | 74.7% | Jul 2026 | Lab model card | |
| 4 | | 73.4% | Jul 2026 | Lab model card | |
| 5 | | 72% | May 2026 | Lab model card | |
| 6 | | 71.2% | Mar 2026 | Lab model card | |
| 7 | | 71% | Aug 2026 | Lab model card | |
| 8 | Kimi K2.5 | 70% | Jan 2026 | Lab model card | |
| 9 | | 69.5% | Feb 2026 | Lab model card | |
| 10 | | 68.7% | Jan 2026 | Lab model card | |
| 11 | | 68.7% | Feb 2026 | Lab model card | |
| 12 | | 68.3% | Apr 2026 | Lab model card | |
| 13 | Mercury 2.5 | 68% | Sep 2026 | Lab model card | |
| 14 | | 66.9% | Feb 2026 | Lab model card | |
| 15 | | 66.3% | Apr 2026 | Lab model card | |
| 16 | | 66.1% | Feb 2026 | Lab model card | |
| 17 | | 65.4% | Jun 2026 | Lab model card | |
| 18 | | 64.1% | Oct 2026 | Lab model card | |
| 19 | | 63.9% | May 2026 | Lab model card | |
| 20 | | 63% | Mar 2026 | Lab model card | |
| 21 | | 62.3% | Jul 2026 | Lab model card | |
| 22 | | 62% | Dec 2025 | Lab model card | |
| 23 | | 61% | Oct 2025 | Lab model card | |
| 24 | | 58.5% | Feb 2026 | Lab model card | |
| 25 | | 58.3% | Mar 2026 | Lab model card | |
| 26 | | 57% | Mar 2026 | Lab model card | |
| 27 | | 57% | Mar 2026 | Lab model card | |
| 28 | | 56.2% | Jul 2026 | Lab model card | |
| 29 | | 53.5% | Dec 2025 | Lab model card | |
| 30 | LFM3-1.7B | 52.8% | Oct 2026 | Lab model card | |
| 31 | | 52.4% | Oct 2026 | Lab model card | |
| 32 | | 50.6% | Apr 2026 | Lab model card | |
| 33 | | 41% | Apr 2026 | Lab model card | |
| 34 | | 39.1% | Mar 2026 | Lab model card | |
| 35 | | 35.9% | Dec 2025 | Lab model card | |
| 36 | | 35% | Jan 2026 | Lab model card | |
| 37 | | 25.6% | Mar 2026 | Lab model card | |
| 38 | | 4.70% | Mar 2026 | Lab model card |
9 models have held number one in labs’ own numbers since its release in August 2025
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Labs’ model cards
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Labs’ model cards
9 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Oct 27, 2026 | LFM3-1.7B | 52.8 | AA-LCR |
| Oct 15, 2026 | 64.1 | AA-LCR | |
| Oct 5, 2026 | Beam 1.0 | 79.3 | AA-LCR |
| Sep 8, 2026 | Mercury 2.5 | 68 | AA-LCR (version not stated) |
| Aug 11, 2026 | 52 | AA-LCR | |
| Aug 11, 2026 | 71 | AA-LCR (Artificial Analysis score quoted by Upstage) | |
| Aug 10, 2026 | 80 | AA-LCR | |
| Jul 31, 2026 | 56.2 | AA-LCR | |
| Jul 22, 2026 | 62.3 | AA-LCR | |
| Jul 16, 2026 | Kimi K3 | 74.7 | AA-LCR |
| Jul 2, 2026 | 73.4 | AA-LCR | |
| Jun 4, 2026 | 65.4 | AA-LCR | |
| May 29, 2026 | 63.9 | AA-LCR (avg@16/acc) | |
| Apr 23, 2026 | 66.3 | AA-LCR | |
| Apr 9, 2026 | 50.6 | AA-LCR | |
| Apr 2, 2026 | 68.3 | AA-LCR | |
| Mar 30, 2026 | 57 | AA-LCR | |
| Mar 19, 2026 | 39.1 | AA-LCR | |
| Mar 16, 2026 | 71.2 | LCR [Mistral Small 4 - High] (text: 'AA LCR ... 0.72') | |
| Mar 11, 2026 | 58.3 | AA-LCR | |
| Mar 2, 2026 | 4.7 | AA-LCR | |
| Mar 2, 2026 | 25.6 | AA-LCR | |
| Mar 2, 2026 | 57 | AA-LCR | |
| Mar 2, 2026 | 63 | AA-LCR | |
| Feb 24, 2026 | 58.5 | AA-LCR | |
| Feb 24, 2026 | 66.1 | AA-LCR | |
| Feb 24, 2026 | 66.9 | AA-LCR | |
| Feb 16, 2026 | 68.7 | AA-LCR | |
| Feb 12, 2026 | 69.5 | AA-LCR | |
| Jan 27, 2026 | Kimi K2.5 | 70 | AA-LCR 70.0 (evaluation table) |
| Jan 23, 2026 | 68.7 | AA-LCR | |
| Jan 5, 2026 | 35 | AA-LCR | |
| Dec 31, 2025 | 53.5 | AA-LCR | |
| Dec 23, 2025 | 62 | AA-LCR | |
| Dec 15, 2025 | 35.9 | AA-LCR | |
| Oct 27, 2025 | 61 | AA-LCR |
18 system cards report it, from 7 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Oct 19, 2026 | | 80 | AA-LCR, high (Tables 2 and 5) | Capabilities |
| Oct 15, 2026 | | 52.4 | AA-LCR, BF16 acc. (Table 22) | Capabilities |
| Oct 15, 2026 | | 64.1 | AA-LCR, released 24GB checkpoint (Table 7). BF16 67.38 | Capabilities |
| Aug 5, 2026 | | 56.2 | AA-LCR (Table 6, Reasoning mode) | Capabilities |
| Jul 27, 2026 | Kimi K3 technical report | 74.7 | AA-LCR, cited from Artificial Analysis (Table 2) | Capabilities |
| Jul 22, 2026 | | 62.3 | AA-LCR (Table 3) | Capabilities |
| Jun 9, 2026 | | 65.4 | AA-LCR (Table 10) | Capabilities |
| May 26, 2026 | | 61 | AA-LCR, MiniMax-M2 bar, printed label (Figure 9) | Capabilities |
| May 26, 2026 | | 72 | AA-LCR, AA Index v4.0 protocol (Table 4) | Capabilities |
| May 26, 2026 | | 65 | AA-LCR, AA Index v4.0 protocol (Table 4) | Capabilities |
| Apr 27, 2026 | | 41 | AA-LCR (Table 10) | Capabilities |
| Apr 17, 2026 | | 57 | AA-LCR (Table 4) | Capabilities |
| Apr 9, 2026 | | 50.6 | AA-LCR (Table 3, Reasoning mode) | Capabilities |
| Mar 19, 2026 | | 39.1 | AA-LCR (Table 1) | Capabilities |
| Mar 11, 2026 | | 58.3 | AA-LCR (Table 5) | Capabilities |
| Jan 11, 2026 | | 35 | AA-LCR (Table 6) | Capabilities |
| Jan 5, 2026 | | 53.5 | AA-LCR (Table 3, Reasoning mode). Non-reasoning 45.2 (Table 4) | Capabilities |
| Dec 23, 2025 | | 35.9 | AA-LCR, reasoning on (Table 3) | Capabilities |
One logged change that moves scores
Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.
- Questions corrected November 26th, 2025
Artificial Analysis fixed question-string encoding in the AA-LCR dataset. Evaluations using older revisions can present corrupted question text, so comparisons should identify the dataset revision. Source
Versions
We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.
Source: Labs’ model cards
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens.
Muse Spark 1.3 98.5%
- LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy.
Inkling 2 Small 70.6%
- GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin.
Gemini 3.7 Flash 89%
- CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- LOCA-Bench In use Agentic tasks whose environment state is scaled to grow context length while task semantics stay fixed. Success rate versus context length. GPT-5.2 21.3%
Data audit
5 of 6 checks pass, 1 warning. Look first at the Confidence check. Likely. Some facts rest on secondary sources. Every benchmark
Catalog record 1 warning 1 pass
- Pass Task access All tasks are public.
- Warning Confidence Likely. Some facts rest on secondary sources.
Scores 3 pass 2 notes
- Note Independent runs Only labs’ own numbers, so no one else has checked them. Scores from launch posts (36 scores) and system cards (18 scores).
- Pass Models scored 38 models we track.
- Pass Newest score The newest is from October 2026, for LFM3-1.7B.
- Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is Kimi K3 (#25).
- Pass Headroom The best is 80%, under 85%, so it still separates models.
Leaderboard import 1 note
- Note Importer Artificial Analysis allows reuse of only its index numbers, so we link to its board instead.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.