Benchmark Almanac

DeepSearchQA

Kimi K3 95% leads since July 2026
GPT-5 Pro 65.2% leads since January 2026
Kimi K3 95% leads since July 2026
In use

The official leaderboard hasn’t scored a model released after GPT-5.4 (March 2026). Labs report their own numbers for 20 newer models, such as Kimi K3 at 95%.

Questions that need multi-step search and synthesis. Final answers graded against references.

Official grading needs the gemini-2.5-flash autorater and prompt. Other judges shift results. Only Kaggle starter code, no GitHub repo.

Made by
Google DeepMind
Released
January 2026
Kind
Deep research
Level
Expert
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Measures
Capability
Language
English
In and out
Text → Actions, Text
Interaction
Agentic
Environment
Browser
Refresh
One fixed set
Run by
Anyone
In views
Agents
Strongest model tested
Claude Opus 5 #11 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 18th, 2026

In the authors’ words

We introduce DeepSearchQA, a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks across 17 different fields. Unlike traditional benchmarks that target single answer retrieval or broad-spectrum factuality, DeepSearchQA features a dataset of challenging, handcrafted tasks designed to evaluate an agent's ability to execute complex search plans to generate exhaustive answer lists. This shift in design explicitly tests three critical, yet under-evaluated capabilities: 1) systematic collation of fragmented information from disparate sources, 2) de-duplication and entity resolution to ensure precision, and 3) the ability to reason about stopping criteria within an open-ended search space. Each task is structured as a causal chain, where discovering information for one step is dependent on the successful completion of the previous one, stressing long-horizon planning and context retention. All tasks are grounded in the open web with objectively verifiable answer sets. Our comprehensive evaluation of state-of-the-art agent architectures reveals significant performance limitations: even the most advanced models struggle to balance high recall with precision. We observe distinct failure modes ranging from premature stopping (under-retrieval) to hedging behaviors, where agents cast an overly wide net of low-confidence answers to artificially boost recall. These findings highlight critical headroom in current agent designs and position DeepSearchQA as an essential diagnostic tool for driving future research toward more robust, deep-research capabilities.

Abstract of DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents (arXiv:2601.20975, CC0)
24
Models scored in lab cards
16
Launch posts list it, from 10 labs
13
System cards report it, from 6 labs

2 models share the top score of 95%

Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.

GPT-5 Pro leads with 65.2% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.

Kimi K3 leads with 95%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

  1. 1 Kimi K3 95%
  2. 2 Claude Opus 5 max effort 95%
  3. 3 Claude Opus 4.8 max effort 93.1%
  4. 4 Step 3.7 Flash 92.8%
  5. 5 Kimi K2.6 92.5%
  6. 6 Claude Opus 4.6 max effort 91.3%
  7. 7 Tencent Hy3 91%
  8. 8 Inkling 2 Small 90.9%
  9. 9 Muse Spark 1.3 90.3%
  10. 10 Claude Sonnet 4.6 max effort 89.2%
  11. 11 Claude Opus 4.7 max effort 89.1%
  12. 12 MiMo-V2-Pro 86.7%
  13. 13 Inkling 2 Mini 85.3%
  14. 14 Muse Spark 1.1 84.9%
  15. 15 Grok 4.6 81.6%
  16. 16 Beam 1.0 80.1%
  17. 17 Seed2.0 Pro 77.4%
  18. 18 ERNIE 5.1 77.3%
  19. 19 Kimi K2.5 77.1%
  20. 20 Muse Spark 74.8%
  21. 21 Muse Glimmer 74.6%
  22. 22 Mercury 3 71.3%
  23. 23 Grok 4.5 38.4%
  24. 24 Mercury 2.5 34%

Source: Labs’ model cards

Sources: Labs’ model cards and official DeepSearchQA leaderboard

#ModelScoreSettingDateSource
1 Kimi K3 95% Jul 2026 Lab model card
2 Claude Opus 5 95% max effort Jul 2026 Lab model card
3 Claude Opus 4.8 93.1% max effort May 2026 Lab model card
4 Step 3.7 Flash 92.8% May 2026 Lab model card
5 Kimi K2.6 92.5% Apr 2026 Lab model card
6 Claude Opus 4.6 91.3% max effort Feb 2026 Lab model card
7 Tencent Hy3 91% Jul 2026 Lab model card
8 Inkling 2 Small 90.9% Nov 2026 Lab model card
9 Muse Spark 1.3 90.3% Sep 2026 Lab model card
10 Claude Sonnet 4.6 89.2% max effort Feb 2026 Lab model card
11 Claude Opus 4.7 89.1% max effort Apr 2026 Lab model card
12 MiMo-V2-Pro 86.7% Mar 2026 Lab model card
13 Inkling 2 Mini 85.3% Nov 2026 Lab model card
14 Muse Spark 1.1 84.9% Jul 2026 Lab model card
15 Grok 4.6 81.6% Aug 2026 Lab model card
16 Beam 1.0 80.1% Oct 2026 Lab model card
17 Seed2.0 Pro 77.4% Feb 2026 Lab model card
18 ERNIE 5.1 77.3% May 2026 Lab model card
19 Kimi K2.5 77.1% Jan 2026 Lab model card
20 Muse Spark 74.8% Apr 2026 Lab model card
21 Muse Glimmer 74.6% Aug 2026 Lab model card
22 Mercury 3 71.3% Oct 2026 Lab model card
23 Grok 4.5 38.4% Jul 2026 Lab model card
24 Mercury 2.5 34% Sep 2026 Lab model card
#ModelScoreSettingDateSource
1 GPT-5 Pro 65.2% Oct 2025 Official leaderboard
2 GPT-5.4 63.7% Mar 2026 Official leaderboard
3 Gemini 3.1 Pro 60.2% Feb 2026 Official leaderboard
4 GPT-5 59.4% Aug 2025 Official leaderboard
5 OpenAI deep research 44.2% Feb 2025 Official leaderboard
6 o4-mini-deep-research 40.4% Jun 2025 Official leaderboard
7 Gemini 2.5 Flash Jun 2025 25.9% Jun 2025 Official leaderboard
8 Gemini 2.5 Pro Jun 2025 24.6% Jun 2025 Official leaderboard
9 Claude Opus 4.5 24% thinking Nov 2025 Official leaderboard
10 Claude Sonnet 4.5 16% thinking Sep 2025 Official leaderboard
11 Claude Haiku 4.5 12.8% thinking Oct 2025 Official leaderboard
#ModelScoreSettingDateSource
1 Kimi K3 95% Jul 2026 Lab model card
2 Claude Opus 5 95% max effort Jul 2026 Lab model card
3 Claude Opus 4.8 93.1% max effort May 2026 Lab model card
4 Step 3.7 Flash 92.8% May 2026 Lab model card
5 Kimi K2.6 92.5% Apr 2026 Lab model card
6 Claude Opus 4.6 91.3% max effort Feb 2026 Lab model card
7 Tencent Hy3 91% Jul 2026 Lab model card
8 Inkling 2 Small 90.9% Nov 2026 Lab model card
9 Muse Spark 1.3 90.3% Sep 2026 Lab model card
10 Claude Sonnet 4.6 89.2% max effort Feb 2026 Lab model card
11 Claude Opus 4.7 89.1% max effort Apr 2026 Lab model card
12 MiMo-V2-Pro 86.7% Mar 2026 Lab model card
13 Inkling 2 Mini 85.3% Nov 2026 Lab model card
14 Muse Spark 1.1 84.9% Jul 2026 Lab model card
15 Grok 4.6 81.6% Aug 2026 Lab model card
16 Beam 1.0 80.1% Oct 2026 Lab model card
17 Seed2.0 Pro 77.4% Feb 2026 Lab model card
18 ERNIE 5.1 77.3% May 2026 Lab model card
19 Kimi K2.5 77.1% Jan 2026 Lab model card
20 Muse Spark 74.8% Apr 2026 Lab model card
21 Muse Glimmer 74.6% Aug 2026 Lab model card
22 Mercury 3 71.3% Oct 2026 Lab model card
23 GPT-5 Pro 65.2% Oct 2025 Official leaderboard
24 GPT-5.4 63.7% Mar 2026 Official leaderboard
25 Gemini 3.1 Pro 60.2% Feb 2026 Official leaderboard
26 GPT-5 59.4% Aug 2025 Official leaderboard
27 OpenAI deep research 44.2% Feb 2025 Official leaderboard
28 o4-mini-deep-research 40.4% Jun 2025 Official leaderboard
29 Grok 4.5 38.4% Jul 2026 Lab model card
30 Mercury 2.5 34% Sep 2026 Lab model card
31 Gemini 2.5 Flash Jun 2025 25.9% Jun 2025 Official leaderboard
32 Gemini 2.5 Pro Jun 2025 24.6% Jun 2025 Official leaderboard
33 Claude Opus 4.5 24% thinking Nov 2025 Official leaderboard
34 Claude Sonnet 4.5 16% thinking Sep 2025 Official leaderboard
35 Claude Haiku 4.5 12.8% thinking Oct 2025 Official leaderboard

The benchmark's own leaderboard has 12 entries, 11 of them models we track. Switch the scores above to rank them together.

5 models have held number one in labs’ own numbers since its release in January 2026 Number one over time 6 models have held number one with claims included since its release in January 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Moonshot AI Anthropic
Own leaderboard Not tested since Mar 2026
GPT-5 Pro 65.2%
Lab claims Passed 85% in Feb 2026, by Claude Opus 4.6
Kimi K3 95%

Sources: Official DeepSearchQA leaderboard and labs’ model cards

Anthropic’s own number for Claude Opus 4.6 passed 85% in February 2026

That was 1 month after its release. By the same age, 20% of the benchmarks from 2026 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Kimi K3 Claude Opus 4.6 Kimi K2.5 GPT-5 Pro GPT-5 OpenAI deep research 0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Kimi K3 Kimi K2.5

Sources: Official DeepSearchQA leaderboard and labs’ model cards

6 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Moonshot AI
ByteDance Seed
Xiaomi Corp
Meta
Baidu
StepFun
Tencent
Inception Labs
Reflection
Thinking Machines
DateLaunch postScoreAs the post writes it
Nov 20, 2026 Inkling 2 Mini 85.3 DeepSearchQA with compaction
Nov 5, 2026 Inkling 2 Small 90.9 DeepSearchQA, 90.9%
Oct 19, 2026 Mercury 3 71.3 DeepSearchQA
Oct 5, 2026 Beam 1.0 80.1 DeepSearchQA w/context management
Sep 8, 2026 Mercury 2.5 34 DSQA (@10 tool calls)
Sep 2, 2026 Muse Spark 1.3 90.3 DeepSearchQA (Agentic browsing)
Aug 10, 2026 Muse Glimmer 74.6 DeepSearch QA
Jul 16, 2026 Kimi K3 95 DeepSearchQA (F1)
Jul 2, 2026 Tencent Hy3 91 DeepSearchQA
May 29, 2026 Step 3.7 Flash 92.8 deepsearchQA (F1), acc 81.7
May 9, 2026 ERNIE 5.1 77.3 DeepSearchQA (F1)
Apr 20, 2026 Kimi K2.6 92.5 DeepSearchQA (f1-score)
Apr 8, 2026 Muse Spark 74.8 DeepSearchQA (Agentic Search)
Mar 18, 2026 MiMo-V2-Pro 86.7 DeepSearch QA-F1
Feb 14, 2026 Seed2.0 Pro 77.4 DeepSearchQA
Jan 27, 2026 Kimi K2.5 77.1 DeepSearchQA

13 system cards report it, from 6 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 20, 2026 Inkling 2 Mini model card 85.3 DeepSearchQA with compaction Capabilities
Oct 19, 2026 Muse Glimmer technical report 74.6 DeepSearchQA, high, 75.3 xhigh (Tables 2 and 5) Capabilities
Aug 12, 2026 Grok 4.6 model card 81.6 DeepSearchQA, internal implementation, accuracy, high (6.2) Capabilities
Jul 27, 2026 Kimi K3 technical report 95 DeepSearchQA (F1) (Table 2) Capabilities
Jul 24, 2026 Claude Opus 5 system card 95 DeepSearchQA mean F1 at max effort, 980k-token budget, printed on chart (8.10.3, Figure 8.10.3.A) Capabilities
Jul 14, 2026 Grok 4.5 model card 38.4 DeepSearchQA, internal implementation, accuracy, high (6.2) Capabilities
Jul 9, 2026 Muse Spark 1.1 system card 84.9 DeepSearchQA (Figure 44) Capabilities
May 28, 2026 Claude Opus 4.8 system card 93.1 DeepSearchQA F1, max effort, 1M-token budget, no compaction (fully correct 84.8%) (Table 8.10.3.B) Capabilities
Apr 16, 2026 Claude Opus 4.7 system card 89.1 DeepSearchQA F1, adaptive thinking at max effort, compaction up to 10M tokens. Fully correct 80.7 (Table 8.8.3.B) Capabilities
Feb 17, 2026 Claude Sonnet 4.6 system card 89.2 DeepSearchQA F1, single agent, adaptive thinking, max effort, up to 10M tokens; multi-agent 91.1 (Figure 2.20.3.A, 2.20.3.1) Capabilities
Feb 14, 2026 Seed2.0 Pro model card 77.4 DeepSearchQA (Table 11) Capabilities
Feb 5, 2026 Claude Opus 4.6 system card 91.3 DeepSearchQA F1, best single-agent configuration, adaptive thinking, max effort, compaction (2.21.3.2, Figure 2.21.3.1.A); multi-agent 92.5 Capabilities
Jan 30, 2026 Kimi K2.5 technical report 77.1 DeepSearchQA (Table 4) Capabilities

More in Tool use

All 265 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Neon 56.3%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%

Data audit

All 9 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 3 pass 2 notes
  • Pass Independent runs Scores from the official leaderboard (12 entries), launch posts (16 scores) and system cards (13 scores).
  • Pass Models scored 35 models we track, plus 1 other entry on its leaderboard.
  • Pass Newest score The newest is from November 2026, for Inkling 2 Mini.
  • Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is Claude Opus 5 (#11).
  • Note Headroom Saturated. The best is 95%, past 85% since February 2026.
Leaderboard import 3 pass
  • Pass Importer We read the board on kaggle.com: 11 rows for models we track and 1 other entry at the last read.
  • Pass Last read November 18th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check