Needle In A Haystack
GLM-4 100% leads since January 2024
Dropped Models retrieve a planted fact from long filler text at varied context lengths and insertion depths. Scored by retrieval accuracy.
Most long-context models score near 100%, and labs run their own variants, so scores are not comparable.
- Made by
- Greg Kamradt
- Released
- November 2023
- Kind
- Finding facts in long inputs
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- All public
- Strongest model tested
- Claude 3.5 Sonnet #157 on the capability index
- Last in a launch post
- December 2024, dropped after it saturated
- Last model added
- January 2025
9
Models scored in lab cards
4
Launch posts list it, from 3 labs
11
System cards report it, from 5 labs
2 models share the top score of 100% among models released by November 2024
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in November 2024, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- Claude Fable 5.2
-
Gemini 4 Carbon - GPT-6 Astra
- Claude Opus 5.5
- and 16 more
- 1 GLM-4 100%
- 2
Qwen-Turbo 100% - 3 Claude 3.5 Sonnet 99.7%
- 4 Claude 3 Opus 99.4%
- 5
Llama 3.1-8B 98.8% - 6 Llama 3.1-405B 98.1%
- 7 Llama 3.1-70B 97.5%
- 8 Claude 3 Haiku 95.9%
- 9 Claude 3 Sonnet 95.4%
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GLM-4 | 100% | Jan 2024 | Lab model card | |
| 2 | Qwen-Turbo | 100% | Dec 2024 | Lab model card | |
| 3 | Claude 3.5 Sonnet | 99.7% | Jun 2024 | Lab model card | |
| 4 | Claude 3 Opus | 99.4% | Mar 2024 | Lab model card | |
| 5 | Llama 3.1-8B | 98.8% | Nov 2024 | Lab model card | |
| 6 | Llama 3.1-405B | 98.1% | Jul 2024 | Lab model card | |
| 7 | Llama 3.1-70B | 97.5% | Nov 2024 | Lab model card | |
| 8 | Claude 3 Haiku | 95.9% | Mar 2024 | Lab model card | |
| 9 | Claude 3 Sonnet | 95.4% | Mar 2024 | Lab model card |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Z.ai
Source: Labs’ model cards
Z.ai’s own number for GLM-4 passed 85% in January 2024
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Reported by the lab in its model card
Source: Labs’ model cards
Labs dropped it from launch posts once the best score passed 85%
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Dec 26, 2024 |  DeepSeek-V3 | Needle In A Haystack (NIAH) | |
| Jul 23, 2024 | Llama 3.1-405B | 98.1 | NIH/Multi-needle |
| May 7, 2024 | DeepSeek-V2 (MoE-236B, May 2024) | Needle In A Haystack (NIAH) | |
| Jan 16, 2024 | GLM-4 | 100 | 大海捞针 (128K) |
11 system cards report it, from 5 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Jan 14, 2025 |  MiniMax-Text-01 technical report | 4M vanilla Needle-In-A-Haystack pressure test, heatmap without printed values (Figure 14). Text says the model handles up to 4M tokens | Capabilities | |
| Dec 26, 2024 | DeepSeek-V3 technical report | Needle In A Haystack up to 128K after SFT, heatmap only (Figure 8) | Capabilities | |
| Dec 19, 2024 | Qwen-Turbo technical report | 100 | Passkey Retrieval at 1M tokens, 100% accuracy (5.2.4 text, Figure 2) | Capabilities |
| Nov 23, 2024 | Llama 3.1-8B technical report | 98.8 | NIH/Multi-needle, average recall of 2 of 4 needles up to 128K, post-trained (Table 2, Table 21) | Capabilities |
| Nov 23, 2024 | Llama 3.1-70B technical report | 97.5 | NIH/Multi-needle, average recall of 2 of 4 needles up to 128K, post-trained (Table 2, Table 21) | Capabilities |
| Nov 23, 2024 | Llama 3.1-405B technical report | 98.1 | NIH/Multi-needle, average recall of 2 of 4 needles up to 128K, post-trained (Table 2, Table 21) | Capabilities |
| Jul 15, 2024 | Qwen2-72B technical report | Needle in a Haystack, 8K to 128K with YARN beyond 32K, heatmap only (Figure 1) | Capabilities | |
| Jun 20, 2024 | Claude 3.5 Sonnet model card | 99.7 | Needle In A Haystack (Anthropic version, modified prompt), average recall over all context lengths up to 200k. 200k context length 99.7 (Table 5) | Capabilities |
| Mar 4, 2024 | Claude 3 Sonnet model card | 95.4 | Needle In A Haystack (Anthropic version, 30 needles, two haystacks), average recall over all context lengths up to 200k. 200k context length 91.4 (Table 7) | Capabilities |
| Mar 4, 2024 | Claude 3 Opus model card | 99.4 | Needle In A Haystack (Anthropic version, 30 needles, two haystacks), average recall over all context lengths up to 200k. 200k context length 98.3 (Table 7) | Capabilities |
| Mar 4, 2024 | Claude 3 Haiku model card | 95.9 | Needle In A Haystack (Anthropic version, 30 needles, two haystacks), average recall over all context lengths up to 200k. 200k context length 91.9 (Table 7) | Capabilities |
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
- LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy.
Inkling 2 Small 70.6% - GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. Gemini 3.7 Flash 89%
- CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.
Step 5 Preview 88.3%