Benchmark Almanac

MADQA

Gemini 3 Pro 85.2% leads since March 2026
Qwen 3.8 Max 91.8% leads since August 2026
In use

Agents search 800 PDF documents to answer 2,250 human-written questions. Scoring covers answer accuracy and how much search effort the agent spends.

Released
March 2026
Kind
Documents, charts and text in images
Level
Everyday
Scored
Graded by a model
Answers
Agent task
Tasks
Public, plus a private test set
Strongest model tested
Qwen 3.8 Max #29 on the capability index
Last in a launch post
August 2026
Last model added
August 2026
Scores refreshed
November 12th, 2026

In the authors’ words

Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning.

Abstract of Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections (arXiv:2603.12180, CC0)
18
Models scored on its own leaderboard
1
Launch post lists it, from 1 lab

Gemini 3 Pro leads with 85.2% among models released by December 2025

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best.

Qwen 3.8 Max leads with 91.8%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in December 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

18 models. Scroll sideways for the rest.

  1. 85.2 Gemini 3 Pro
  2. 80.6 Claude Sonnet 4.5
  3. 77.7 GPT-5
  4. 73.1 Gemini 2.5 Pro
  5. 71.8 Gemini 2.5 Flash
  6. 68.2 Claude Haiku 4.5
  7. 67.8 GPT-5.2
  8. 66.9 GPT-5 mini
  9. 66.1 GLM-4.6V
  10. 60.3 Qwen3-VL-235B-A22B
  11. 60.0 GPT-4.1
  12. 58.8 Qwen3-VL-32B-Thinking
  13. 58.2 GPT-5 nano
  14. 53.0 GPT-4.1 mini
  15. 48.6 GPT-4o
  16. 47.4 Qwen3-VL-8B-Thinking
  17. 46.0 GLM-4.6V-Flash
  18. 19.5 GPT-4.1 nano
  1. 1 Gemini 3 Pro preview 85.2%
  2. 2 Claude Sonnet 4.5 80.6%
  3. 3 GPT-5 77.7%
  4. 4 Gemini 2.5 Pro Jun 2025 73.1%
  5. 5 Gemini 2.5 Flash Jun 2025 71.8%
  6. 6 Claude Haiku 4.5 68.2%
  7. 7 GPT-5.2 67.8%
  8. 8 GPT-5 mini 66.9%
  9. 9 GLM-4.6V 66.1%
  10. 10 Qwen3-VL-235B-A22B thinking 60.3%
  11. 11 GPT-4.1 60%
  12. 12 Qwen3-VL-32B-Thinking thinking 58.8%
  13. 13 GPT-5 nano 58.2%
  14. 14 GPT-4.1 mini 53%
  15. 15 GPT-4o Aug 2024 48.6%
  16. 16 Qwen3-VL-8B-Thinking thinking 47.4%
  17. 17 GLM-4.6V-Flash 46%
  18. 18 GPT-4.1 nano 19.5%

Source: Official MADQA leaderboard

19 models. Scroll sideways for the rest.

  1. 91.8 Qwen 3.8 Max
  2. 85.2 Gemini 3 Pro
  3. 80.6 Claude Sonnet 4.5
  4. 77.7 GPT-5
  5. 73.1 Gemini 2.5 Pro
  6. 71.8 Gemini 2.5 Flash
  7. 68.2 Claude Haiku 4.5
  8. 67.8 GPT-5.2
  9. 66.9 GPT-5 mini
  10. 66.1 GLM-4.6V
  11. 60.3 Qwen3-VL-235B-A22B
  12. 60.0 GPT-4.1
  13. 58.8 Qwen3-VL-32B-Thinking
  14. 58.2 GPT-5 nano
  15. 53.0 GPT-4.1 mini
  16. 48.6 GPT-4o
  17. 47.4 Qwen3-VL-8B-Thinking
  18. 46.0 GLM-4.6V-Flash
  19. 19.5 GPT-4.1 nano
  1. 1 Qwen 3.8 Max 91.8%
  2. 2 Gemini 3 Pro preview 85.2%
  3. 3 Claude Sonnet 4.5 80.6%
  4. 4 GPT-5 77.7%
  5. 5 Gemini 2.5 Pro Jun 2025 73.1%
  6. 6 Gemini 2.5 Flash Jun 2025 71.8%
  7. 7 Claude Haiku 4.5 68.2%
  8. 8 GPT-5.2 67.8%
  9. 9 GPT-5 mini 66.9%
  10. 10 GLM-4.6V 66.1%
  11. 11 Qwen3-VL-235B-A22B thinking 60.3%
  12. 12 GPT-4.1 60%
  13. 13 Qwen3-VL-32B-Thinking thinking 58.8%
  14. 14 GPT-5 nano 58.2%
  15. 15 GPT-4.1 mini 53%
  16. 16 GPT-4o Aug 2024 48.6%
  17. 17 Qwen3-VL-8B-Thinking thinking 47.4%
  18. 18 GLM-4.6V-Flash 46%
  19. 19 GPT-4.1 nano 19.5%

Sources: Labs’ model cards and official MADQA leaderboard

#ModelScoreSettingDateSource
1 Gemini 3 Pro 85.2% preview, BM25 Search Tool Nov 2025 Official leaderboard
2 Claude Sonnet 4.5 80.6% BM25 Search Tool Sep 2025 Official leaderboard
3 GPT-5 77.7% BM25 Search Tool Aug 2025 Official leaderboard
4 Gemini 2.5 Pro Jun 2025 73.1% File Search Jun 2025 Official leaderboard
5 Gemini 2.5 Flash Jun 2025 71.8% File Search Jun 2025 Official leaderboard
6 Claude Haiku 4.5 68.2% BM25 Search Tool Oct 2025 Official leaderboard
7 GPT-5.2 67.8% BM25 Search Tool Dec 2025 Official leaderboard
8 GPT-5 mini 66.9% BM25 Search Tool Aug 2025 Official leaderboard
9 GLM-4.6V 66.1% BM25 Search Tool Dec 2025 Official leaderboard
10 Qwen3-VL-235B-A22B 60.3% thinking, BM25 Search Tool Sep 2025 Official leaderboard
11 GPT-4.1 60% BM25 Search Tool Apr 2025 Official leaderboard
12 Qwen3-VL-32B-Thinking 58.8% thinking, BM25 Search Tool Oct 2025 Official leaderboard
13 GPT-5 nano 58.2% BM25 Search Tool Aug 2025 Official leaderboard
14 GPT-4.1 mini 53% BM25 Search Tool Apr 2025 Official leaderboard
15 GPT-4o Aug 2024 48.6% HEAVEN Retrieval Aug 2024 Official leaderboard
16 Qwen3-VL-8B-Thinking 47.4% thinking, BM25 Search Tool Oct 2025 Official leaderboard
17 GLM-4.6V-Flash 46% BM25 Search Tool Dec 2025 Official leaderboard
18 GPT-4.1 nano 19.5% BM25 Search Tool Apr 2025 Official leaderboard
#ModelScoreSettingDateSource
1 Qwen 3.8 Max 91.8% Aug 2026 Lab model card
2 Gemini 3 Pro 85.2% preview, BM25 Search Tool Nov 2025 Official leaderboard
3 Claude Sonnet 4.5 80.6% BM25 Search Tool Sep 2025 Official leaderboard
4 GPT-5 77.7% BM25 Search Tool Aug 2025 Official leaderboard
5 Gemini 2.5 Pro Jun 2025 73.1% File Search Jun 2025 Official leaderboard
6 Gemini 2.5 Flash Jun 2025 71.8% File Search Jun 2025 Official leaderboard
7 Claude Haiku 4.5 68.2% BM25 Search Tool Oct 2025 Official leaderboard
8 GPT-5.2 67.8% BM25 Search Tool Dec 2025 Official leaderboard
9 GPT-5 mini 66.9% BM25 Search Tool Aug 2025 Official leaderboard
10 GLM-4.6V 66.1% BM25 Search Tool Dec 2025 Official leaderboard
11 Qwen3-VL-235B-A22B 60.3% thinking, BM25 Search Tool Sep 2025 Official leaderboard
12 GPT-4.1 60% BM25 Search Tool Apr 2025 Official leaderboard
13 Qwen3-VL-32B-Thinking 58.8% thinking, BM25 Search Tool Oct 2025 Official leaderboard
14 GPT-5 nano 58.2% BM25 Search Tool Aug 2025 Official leaderboard
15 GPT-4.1 mini 53% BM25 Search Tool Apr 2025 Official leaderboard
16 GPT-4o Aug 2024 48.6% HEAVEN Retrieval Aug 2024 Official leaderboard
17 Qwen3-VL-8B-Thinking 47.4% thinking, BM25 Search Tool Oct 2025 Official leaderboard
18 GLM-4.6V-Flash 46% BM25 Search Tool Dec 2025 Official leaderboard
19 GPT-4.1 nano 19.5% BM25 Search Tool Apr 2025 Official leaderboard

Labs report scores for 1 model in their cards. Switch the scores above to rank them together.

Accuracy vs Attribution

Each point is one submission: LLM-judged answer accuracy against how well it cites the right pages (page F1). The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 80% 100% 0%20%40%60%80%100% Attribution (Page F1) Accuracy (LLM judge) GPT-4o: Accuracy (LLM judge) 48.6%, attribution (page f1) 43.2% GPT-4.1: Accuracy (LLM judge) 60%, attribution (page f1) 64.1% GPT-4.1 mini: Accuracy (LLM judge) 53%, attribution (page f1) 59.9% GPT-4.1 nano: Accuracy (LLM judge) 19.5%, attribution (page f1) 27.6% Gemini 2.5 Pro: Accuracy (LLM judge) 60.1%, attribution (page f1) 60.3% Gemini 2.5 Pro: Accuracy (LLM judge) 73.1%, attribution (page f1) 60.9% Gemini 2.5 Flash: Accuracy (LLM judge) 58.5%, attribution (page f1) 61% Gemini 2.5 Flash: Accuracy (LLM judge) 71.8%, attribution (page f1) 52.2% GPT-5: Accuracy (LLM judge) 77.7%, attribution (page f1) 74.2% GPT-5: Accuracy (LLM judge) 49.6%, attribution (page f1) 29.3% GPT-5 mini: Accuracy (LLM judge) 66.9%, attribution (page f1) 67.6% GPT-5 mini: Accuracy (LLM judge) 48.5%, attribution (page f1) 29% GPT-5 nano: Accuracy (LLM judge) 58.2%, attribution (page f1) 60.9% Qwen3-VL-235B-A22B (thinking): Accuracy (LLM judge) 60.3%, attribution (page f1) 58.6% Claude Sonnet 4.5: Accuracy (LLM judge) 80.6%, attribution (page f1) 79.1% Claude Haiku 4.5: Accuracy (LLM judge) 68.2%, attribution (page f1) 72% Qwen3-VL-8B-Thinking (thinking): Accuracy (LLM judge) 47.4%, attribution (page f1) 47.6% Qwen3-VL-32B-Thinking (thinking): Accuracy (LLM judge) 58.8%, attribution (page f1) 54.7% Gemini 3 Pro (preview): Accuracy (LLM judge) 78.6%, attribution (page f1) 70.1% Gemini 3 Pro: Accuracy (LLM judge) 82.2%, attribution (page f1) 78.5% GLM-4.6V: Accuracy (LLM judge) 66.1%, attribution (page f1) 65.9% GLM-4.6V-Flash: Accuracy (LLM judge) 46%, attribution (page f1) 28.9% GPT-5.2: Accuracy (LLM judge) 67.8%, attribution (page f1) 67.6% GPT-5.2: Accuracy (LLM judge) 50%, attribution (page f1) 28.5% GPT-5.2: Accuracy (LLM judge) 52.9%, attribution (page f1) 48.4% GPT-5.2: Accuracy (LLM judge) 47.5%, attribution (page f1) 46.4% Gemini 3 Pro (preview): Accuracy (LLM judge) 85.2%, attribution (page f1) 80.4% Gemini 3 Pro (preview) 0% 20% 40% 60% 80% 100% 0%20%40%60%80%100% Attribution (Page F1) Accuracy (LLM judge) GPT-4o: Accuracy (LLM judge) 48.6%, attribution (page f1) 43.2% GPT-4.1: Accuracy (LLM judge) 60%, attribution (page f1) 64.1% GPT-4.1 mini: Accuracy (LLM judge) 53%, attribution (page f1) 59.9% GPT-4.1 nano: Accuracy (LLM judge) 19.5%, attribution (page f1) 27.6% Gemini 2.5 Pro: Accuracy (LLM judge) 60.1%, attribution (page f1) 60.3% Gemini 2.5 Pro: Accuracy (LLM judge) 73.1%, attribution (page f1) 60.9% Gemini 2.5 Flash: Accuracy (LLM judge) 58.5%, attribution (page f1) 61% Gemini 2.5 Flash: Accuracy (LLM judge) 71.8%, attribution (page f1) 52.2% GPT-5: Accuracy (LLM judge) 77.7%, attribution (page f1) 74.2% GPT-5: Accuracy (LLM judge) 49.6%, attribution (page f1) 29.3% GPT-5 mini: Accuracy (LLM judge) 66.9%, attribution (page f1) 67.6% GPT-5 mini: Accuracy (LLM judge) 48.5%, attribution (page f1) 29% GPT-5 nano: Accuracy (LLM judge) 58.2%, attribution (page f1) 60.9% Qwen3-VL-235B-A22B (thinking): Accuracy (LLM judge) 60.3%, attribution (page f1) 58.6% Claude Sonnet 4.5: Accuracy (LLM judge) 80.6%, attribution (page f1) 79.1% Claude Haiku 4.5: Accuracy (LLM judge) 68.2%, attribution (page f1) 72% Qwen3-VL-8B-Thinking (thinking): Accuracy (LLM judge) 47.4%, attribution (page f1) 47.6% Qwen3-VL-32B-Thinking (thinking): Accuracy (LLM judge) 58.8%, attribution (page f1) 54.7% Gemini 3 Pro (preview): Accuracy (LLM judge) 78.6%, attribution (page f1) 70.1% Gemini 3 Pro: Accuracy (LLM judge) 82.2%, attribution (page f1) 78.5% GLM-4.6V: Accuracy (LLM judge) 66.1%, attribution (page f1) 65.9% GLM-4.6V-Flash: Accuracy (LLM judge) 46%, attribution (page f1) 28.9% GPT-5.2: Accuracy (LLM judge) 67.8%, attribution (page f1) 67.6% GPT-5.2: Accuracy (LLM judge) 50%, attribution (page f1) 28.5% GPT-5.2: Accuracy (LLM judge) 52.9%, attribution (page f1) 48.4% GPT-5.2: Accuracy (LLM judge) 47.5%, attribution (page f1) 46.4% Gemini 3 Pro (preview): Accuracy (LLM judge) 85.2%, attribution (page f1) 80.4% Gemini 3 Pro (preview)
ModelSettingAccuracy (LLM judge)Attribution (Page F1)
Gemini 3 Pro preview, BM25 Search Tool 85.2% 80.4%
Gemini 3 Pro BM25 Search Tool 82.2% 78.5%
Claude Sonnet 4.5 BM25 Search Tool 80.6% 79.1%
Gemini 3 Pro preview, File Search 78.6% 70.1%
GPT-5 BM25 Search Tool 77.7% 74.2%
Gemini 2.5 Pro Jun 2025 File Search 73.1% 60.9%
Gemini 2.5 Flash Jun 2025 File Search 71.8% 52.2%
Claude Haiku 4.5 BM25 Search Tool 68.2% 72%
GPT-5.2 BM25 Search Tool 67.8% 67.6%
GPT-5 mini BM25 Search Tool 66.9% 67.6%
GLM-4.6V BM25 Search Tool 66.1% 65.9%
Qwen3-VL-235B-A22B thinking, BM25 Search Tool 60.3% 58.6%
Gemini 2.5 Pro Jun 2025 BM25 Search Tool 60.1% 60.3%
GPT-4.1 BM25 Search Tool 60% 64.1%
Qwen3-VL-32B-Thinking thinking, BM25 Search Tool 58.8% 54.7%
Gemini 2.5 Flash Jun 2025 BM25 Search Tool 58.5% 61%
GPT-5 nano BM25 Search Tool 58.2% 60.9%
GPT-4.1 mini BM25 Search Tool 53% 59.9%
GPT-5.2 HEAVEN Retrieval 52.9% 48.4%
GPT-5.2 File Search 50% 28.5%
GPT-5 File Search 49.6% 29.3%
GPT-4o Aug 2024 HEAVEN Retrieval 48.6% 43.2%
GPT-5 mini File Search 48.5% 29%
GPT-5.2 HEAVEN Retrieval 47.6% 46.4%
Qwen3-VL-8B-Thinking thinking, BM25 Search Tool 47.4% 47.6%
GLM-4.6V-Flash BM25 Search Tool 46% 28.9%
GPT-4.1 nano BM25 Search Tool 19.5% 27.6%

Number one over time 2 models have held number one with claims included since its release in March 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind
Own leaderboard Passed 85% in Mar 2026, by Gemini 3 Pro
Gemini 3 Pro 85.2%

Source: Official MADQA leaderboard

Gemini 3 Pro on the official leaderboard passed 85% in November 2025

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen 3.8 Max Gemini 3 Pro Claude Sonnet 4.5 Gemini 2.5 Pro GPT-4.1 GPT-4o 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen 3.8 Max Gemini 3 Pro

Sources: Official MADQA leaderboard and labs’ model cards

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
DateLaunch postScoreAs the post writes it
Aug 3, 2026 Qwen 3.8 Max 91.8 MADQA

More in Vision

All 281 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. GPT-5.1 85.4%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%
  • ChartQA In use Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. Command A Vision 90.9%