Passkey Retrieval
Mixtral 8x7B 100% Models find a random passkey hidden at a random position in a long filler prompt. Scored by retrieval accuracy.
Synthetic test from the Landmark Attention paper. Strong long-context models usually score near 100%.
- Made by
- EPFL
- Released
- May 2023
- Kind
- Finding facts in long inputs
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- All public
- Strongest model tested
- Mixtral 8x7B #198 on the capability index
- Last in a launch post
- January 2025, dropped after it saturated
- Last model added
- January 2025
2 models share the top score of 100%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Mixtral 8x7B | 100% | Jan 2024 | Lab model card | |
| 2 |  Qwen2.5-14B-Instruct-1M | 100% | Jan 2025 | Lab model card |
Mistral AI’s own number for Mixtral 8x7B passed 85% in January 2024
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Labs’ model cards
Labs dropped it from launch posts once the best score passed 85%
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Jan 27, 2025 | Qwen2.5-14B-Instruct-1M | Passkey Retrieval at 1M tokens (chart only) |
2 system cards report it, from 2 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Jan 26, 2025 | Qwen2.5-14B-Instruct-1M technical report | 100 | Passkey Retrieval with documents up to 1M tokens, perfect accuracy stated in text (Figure 1, section 6.1) | Capabilities |
| Jan 8, 2024 | Mixtral 8x7B technical report | 100 | Passkey retrieval, 100% at all context lengths and positions (Section 3.2, Figure 4 Left) | Capabilities |
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens.
Muse Spark 1.3 98.5% - LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy.
Inkling 2 Small 70.6% - GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin.
Gemini 3.7 Flash 89% - CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.
Step 5 Preview 88.3%