Benchmark Almanac

MCP Atlas

Gemini 4 Argon 90.7% leads since September 2026
Gemini 4 Argon 90.7% leads since September 2026
In use

Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks.

500 public tasks of 1,000. Rest held back. Labs use different judge models, so self-reported scores vary.

Made by
Scale AI
Released
January 2026
Kind
Apps and MCP servers
Level
University
Scored
Graded by a model
Answers
Agent task
Tasks
Public, plus a private test set
Strongest model tested
Claude Opus 5.5 #4 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 12th, 2026

In the authors’ words

The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evaluations fall short along three key axes: realistic multi-step workflows with cross-server orchestration, breadth across authentic MCP servers rather than mocks, and structured, reproducible claim-level scoring disentangled from agent verbosity or style. We introduce MCP-Atlas, a benchmark for measuring tool-use competency against production MCP servers. MCP-Atlas contains 1,000 natural-language tasks written and verified by human experts spanning 36 real MCP servers and 220 tools. Prompts do not specify servers, tools, or parameters, requiring agents to identify relevant tools among semantically plausible distractors and to compose multi-step, cross-server workflows. Each task is scored with a claim-level rubric, where final answers are scored against atomic factual claims grounded in tool outputs. This answer-centric scoring permits valid alternative tool-call trajectories to receive credit. We pair this with an 11-category diagnostic taxonomy that disentangles tool-call failures from cognitive failures in task understanding, synthesis, parsing, and stopping. Evaluating 20 frontier models from six providers under matched task-level conditions, we find pass rates up to 82.2% at a 0.75 claim coverage threshold and a clear three-tier performance structure. Automated diagnostics show that 63.3% of diagnosed failures are cognitive rather than tool-call related. Notably, several high-performing models fail after successful tool execution due to premature stopping or incorrect synthesis. We release the task schema, containerized harness, claim evaluator, and a 500-task public split, while reserving a 500-task private split to preserve leaderboard integrity. The code is at https://github.com/scaleapi/mcp-atlas.

Abstract of MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (arXiv:2602.00933, CC0)
37
Models scored on its own leaderboard
43
Launch posts list it, from 20 labs
21
System cards report it, from 11 labs

Gemini 4 Argon leads with 90.7%

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best.

Gemini 4 Argon leads with 90.7%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

37 models. Scroll sideways for the rest.

  1. 90.7 Gemini 4 Argon
  2. 88.1 Muse Spark 1.1
  3. 87.2 Claude Fable 5.1
  4. 85.8 Claude Opus 5
  5. 85.4 Claude Opus 5.5
  6. 85.4 Inkling 2 Small
  7. 84.5 Qwen3.8-2.4T-A95B
  8. 84.2 GLM-5.3
  9. 83.6 Gemini 3.5 Flash
  10. 83.3 Claude Fable 5
  11. 82.3 Kimi K3
  12. 82.2 Muse Spark
  13. 82.2 Claude Opus 4.8
  14. 81.8 GPT-5.6 Sol
  15. 79.2 Inkling-Small
  16. 79.1 Claude Opus 4.7
  17. 78.2 Gemini 3.1 Pro
  18. 77.8 GLM-5.2
  19. 76.8 Claude Opus 4.6
  20. 76.0 Inkling
  21. 75.6 GLM-5.1
  22. 75.3 GPT-5.5
  23. 70.6 GPT-5.4
  24. 70.3 Gemini 3 Pro
  25. 69.8 Claude Opus 4.5
  26. 69.5 Claude Sonnet 4.6
  27. 67.6 GPT-5.2
  28. 64.4 Kimi K2.5
  29. 63.1 Nemotron 3 Ultra
  30. 62.0 Gemini 3 Flash
  31. 59.5 Claude Sonnet 4.5
  32. 58.1 GLM-4.7
  33. 57.1 Gemini 3.1 Flash-Lite
  34. 56.7 GPT-5.4 Mini
  35. 50.1 GPT-5.1
  36. 44.5 o3-pro
  37. 40.2 Claude Haiku 4.5
  1. 1 Gemini 4 Argon 90.7%
  2. 2 Muse Spark 1.1 88.1%
  3. 3 Claude Fable 5.1 87.2%
  4. 4 Claude Opus 5 xhigh 85.8%
  5. 5 Claude Opus 5.5 85.4%
  6. 6 Inkling 2 Small 85.4%
  7. 7 Qwen3.8-2.4T-A95B xhigh 84.5%
  8. 8 GLM-5.3 84.2%
  9. 9 Gemini 3.5 Flash high 83.6%
  10. 10 Claude Fable 5 83.3%
  11. 11 Kimi K3 max 82.3%
  12. 12 Muse Spark 82.2%
  13. 13 Claude Opus 4.8 max 82.2%
  14. 14 GPT-5.6 Sol 81.8%
  15. 15 Inkling-Small 79.2%
  16. 16 Claude Opus 4.7 max 79.1%
  17. 17 Gemini 3.1 Pro high 78.2%
  18. 18 GLM-5.2 77.8%
  19. 19 Claude Opus 4.6 max 76.8%
  20. 20 Inkling xhigh 76%
  21. 21 GLM-5.1 75.6%
  22. 22 GPT-5.5 xhigh 75.3%
  23. 23 GPT-5.4 xhigh 70.6%
  24. 24 Gemini 3 Pro 70.3%
  25. 25 Claude Opus 4.5 high 69.8%
  26. 26 Claude Sonnet 4.6 69.5%
  27. 27 GPT-5.2 xhigh 67.6%
  28. 28 Kimi K2.5 64.4%
  29. 29 Nemotron 3 Ultra thinking 63.1%
  30. 30 Gemini 3 Flash 62%
  31. 31 Claude Sonnet 4.5 thinking 59.5%
  32. 32 GLM-4.7 58.1%
  33. 33 Gemini 3.1 Flash-Lite high 57.1%
  34. 34 GPT-5.4 Mini xhigh 56.7%
  35. 35 GPT-5.1 high 50.1%
  36. 36 o3-pro 44.5%
  37. 37 Claude Haiku 4.5 40.2%

Source: Official MCP Atlas leaderboard

63 models. Scroll sideways for the rest.

  1. 90.7 Gemini 4 Argon
  2. 90.3 Muse Spark 1.2
  3. 89.2 Muse Spark Open
  4. 88.1 Muse Spark 1.1
  5. 87.2 Claude Fable 5.1
  6. 85.8 Claude Opus 5
  7. 85.6 Step 5 Preview
  8. 85.4 Claude Opus 5.5
  9. 85.4 Inkling 2 Small
  10. 84.5 Qwen3.8-2.4T-A95B
  11. 84.2 Kimi K3
  12. 84.2 GLM-5.3
  13. 83.8 Seed2.1 Pro
  14. 83.7 Tencent Hy4 preview
  15. 83.6 Gemini 3.5 Flash
  16. 83.3 Claude Fable 5
  17. 83.1 LongCat-2.5
  18. 83.0 Ling-3.1-flash
  19. 82.2 Muse Spark
  20. 82.2 Claude Opus 4.8
  21. 81.8 GPT-5.6 Sol
  22. 79.6 Inkling-Small
  23. 79.1 Claude Opus 4.7
  24. 79.1 Tencent Hy3
  25. 78.7 Beam 1.0
  26. 78.2 Gemini 3.1 Pro
  27. 77.8 GLM-5.2
  28. 76.8 Claude Opus 4.6
  29. 76.4 Qwen3.7-Max
  30. 76.0 Kimi K2.7 Code
  31. 76.0 Inkling
  32. 75.6 GLM-5.1
  33. 75.5 Muse Glimmer
  34. 75.3 GPT-5.5
  35. 74.2 MiniMax-M3
  36. 74.1 Qwen 3.6 Plus
  37. 73.6 DeepSeek-V4-Pro
  38. 73.2 Qwen3.7-Plus
  39. 70.6 GPT-5.4
  40. 70.3 Gemini 3 Pro
  41. 69.8 Claude Opus 4.5
  42. 69.5 Claude Sonnet 4.6
  43. 69.0 DeepSeek-V4-Flash
  44. 67.8 GLM-5
  45. 67.6 GPT-5.2
  46. 64.4 Kimi K2.5
  47. 63.1 Nemotron 3 Ultra
  48. 62.8 Qwen 3.6 35B-A3B
  49. 62.4 Spark-X2.6-4B
  50. 62.0 Gemini 3 Flash
  51. 61.4 Solar Pro 4
  52. 59.5 Claude Sonnet 4.5
  53. 58.2 Solar Open 2
  54. 58.1 GLM-4.7
  55. 57.7 GPT-5.4 Mini
  56. 57.1 Gemini 3.1 Flash-Lite
  57. 56.1 GPT-5.4 Nano
  58. 52.2 Nemotron 3.5 Bolt 74B-A9B
  59. 50.1 GPT-5.1
  60. 44.5 o3-pro
  61. 40.2 Claude Haiku 4.5
  62. 32.6 LFM3-1.7B
  63. 24.6 Nova 2 Lite
  1. 1 Gemini 4 Argon 90.7%
  2. 2 Muse Spark 1.2 90.3%
  3. 3 Muse Spark Open 89.2%
  4. 4 Muse Spark 1.1 88.1%
  5. 5 Claude Fable 5.1 87.2%
  6. 6 Claude Opus 5 xhigh 85.8%
  7. 7 Step 5 Preview 85.6%
  8. 8 Claude Opus 5.5 85.4%
  9. 9 Inkling 2 Small 85.4%
  10. 10 Qwen3.8-2.4T-A95B xhigh 84.5%
  11. 11 Kimi K3 84.2%
  12. 12 GLM-5.3 84.2%
  13. 13 Seed2.1 Pro 83.8%
  14. 14 Tencent Hy4 preview 83.7%
  15. 15 Gemini 3.5 Flash high 83.6%
  16. 16 Claude Fable 5 83.3%
  17. 17 LongCat-2.5 83.1%
  18. 18 Ling-3.1-flash 83%
  19. 19 Muse Spark 82.2%
  20. 20 Claude Opus 4.8 max 82.2%
  21. 21 GPT-5.6 Sol 81.8%
  22. 22 Inkling-Small 79.6%
  23. 23 Claude Opus 4.7 max 79.1%
  24. 24 Tencent Hy3 79.1%
  25. 25 Beam 1.0 78.7%
  26. 26 Gemini 3.1 Pro high 78.2%
  27. 27 GLM-5.2 77.8%
  28. 28 Claude Opus 4.6 max 76.8%
  29. 29 Qwen3.7-Max 76.4%
  30. 30 Kimi K2.7 Code 76%
  31. 31 Inkling xhigh 76%
  32. 32 GLM-5.1 75.6%
  33. 33 Muse Glimmer 75.5%
  34. 34 GPT-5.5 xhigh 75.3%
  35. 35 MiniMax-M3 74.2%
  36. 36 Qwen 3.6 Plus 74.1%
  37. 37 DeepSeek-V4-Pro 73.6%
  38. 38 Qwen3.7-Plus 73.2%
  39. 39 GPT-5.4 xhigh 70.6%
  40. 40 Gemini 3 Pro 70.3%
  41. 41 Claude Opus 4.5 high 69.8%
  42. 42 Claude Sonnet 4.6 69.5%
  43. 43 DeepSeek-V4-Flash 69%
  44. 44 GLM-5 67.8%
  45. 45 GPT-5.2 xhigh 67.6%
  46. 46 Kimi K2.5 64.4%
  47. 47 Nemotron 3 Ultra thinking 63.1%
  48. 48 Qwen 3.6 35B-A3B 62.8%
  49. 49 Spark-X2.6-4B 62.4%
  50. 50 Gemini 3 Flash 62%
  51. 51 Solar Pro 4 61.4%
  52. 52 Claude Sonnet 4.5 thinking 59.5%
  53. 53 Solar Open 2 58.2%
  54. 54 GLM-4.7 58.1%
  55. 55 GPT-5.4 Mini 57.7%
  56. 56 Gemini 3.1 Flash-Lite high 57.1%
  57. 57 GPT-5.4 Nano 56.1%
  58. 58 Nemotron 3.5 Bolt 74B-A9B 52.2%
  59. 59 GPT-5.1 high 50.1%
  60. 60 o3-pro 44.5%
  61. 61 Claude Haiku 4.5 40.2%
  62. 62 LFM3-1.7B 32.6%
  63. 63 Nova 2 Lite 24.6%

Sources: Official MCP Atlas leaderboard and labs’ model cards

#ModelScoreSettingDateSource
1 Gemini 4 Argon 90.7% Sep 2026 Official leaderboard
2 Muse Spark 1.1 88.1% Jul 2026 Official leaderboard
3 Claude Fable 5.1 87.2% Sep 2026 Official leaderboard
4 Claude Opus 5 85.8% xhigh Jul 2026 Official leaderboard
5 Claude Opus 5.5 85.4% Sep 2026 Official leaderboard
6 Inkling 2 Small 85.4% Nov 2026 Official leaderboard
7 Qwen3.8-2.4T-A95B 84.5% xhigh Aug 2026 Official leaderboard
8 GLM-5.3 84.2% Aug 2026 Official leaderboard
9 Gemini 3.5 Flash 83.6% high May 2026 Official leaderboard
10 Claude Fable 5 83.3% Jun 2026 Official leaderboard
11 Kimi K3 82.3% max Jul 2026 Official leaderboard
12 Muse Spark 82.2% Apr 2026 Official leaderboard
13 Claude Opus 4.8 82.2% max May 2026 Official leaderboard
14 GPT-5.6 Sol 81.8% Jul 2026 Official leaderboard
15 Inkling-Small 79.2% Jul 2026 Official leaderboard
16 Claude Opus 4.7 79.1% max Apr 2026 Official leaderboard
17 Gemini 3.1 Pro 78.2% high Feb 2026 Official leaderboard
18 GLM-5.2 77.8% Jun 2026 Official leaderboard
19 Claude Opus 4.6 76.8% max Feb 2026 Official leaderboard
20 Inkling 76% xhigh Jul 2026 Official leaderboard
21 GLM-5.1 75.6% Apr 2026 Official leaderboard
22 GPT-5.5 75.3% xhigh Apr 2026 Official leaderboard
23 GPT-5.4 70.6% xhigh Mar 2026 Official leaderboard
24 Gemini 3 Pro 70.3% Nov 2025 Official leaderboard
25 Claude Opus 4.5 69.8% high Nov 2025 Official leaderboard
26 Claude Sonnet 4.6 69.5% Feb 2026 Official leaderboard
27 GPT-5.2 67.6% xhigh Dec 2025 Official leaderboard
28 Kimi K2.5 64.4% Jan 2026 Official leaderboard
29 Nemotron 3 Ultra 63.1% thinking Jun 2026 Official leaderboard
30 Gemini 3 Flash 62% Dec 2025 Official leaderboard
31 Claude Sonnet 4.5 59.5% thinking Sep 2025 Official leaderboard
32 GLM-4.7 58.1% Dec 2025 Official leaderboard
33 Gemini 3.1 Flash-Lite 57.1% high Mar 2026 Official leaderboard
34 GPT-5.4 Mini 56.7% xhigh Mar 2026 Official leaderboard
35 GPT-5.1 50.1% high Nov 2025 Official leaderboard
36 o3-pro 44.5% Jun 2025 Official leaderboard
37 Claude Haiku 4.5 40.2% Oct 2025 Official leaderboard
#ModelScoreSettingDateSource
1 Gemini 4 Argon 90.7% Sep 2026 Official leaderboard
2 Muse Spark 1.2 90.3% Aug 2026 Lab model card
3 Muse Spark Open 89.2% Oct 2026 Lab model card
4 Muse Spark 1.1 88.1% Jul 2026 Official leaderboard
5 Claude Fable 5.1 87.2% Sep 2026 Official leaderboard
6 Claude Opus 5 85.8% xhigh Jul 2026 Official leaderboard
7 Step 5 Preview 85.6% Sep 2026 Lab model card
8 Claude Opus 5.5 85.4% Sep 2026 Official leaderboard
9 Inkling 2 Small 85.4% Nov 2026 Official leaderboard
10 Qwen3.8-2.4T-A95B 84.5% xhigh Aug 2026 Official leaderboard
11 Kimi K3 84.2% Jul 2026 Lab model card
12 GLM-5.3 84.2% Aug 2026 Official leaderboard
13 Seed2.1 Pro 83.8% Jun 2026 Lab model card
14 Tencent Hy4 preview 83.7% Aug 2026 Lab model card
15 Gemini 3.5 Flash 83.6% high May 2026 Official leaderboard
16 Claude Fable 5 83.3% Jun 2026 Official leaderboard
17 LongCat-2.5 83.1% Nov 2026 Lab model card
18 Ling-3.1-flash 83% Oct 2026 Lab model card
19 Muse Spark 82.2% Apr 2026 Official leaderboard
20 Claude Opus 4.8 82.2% max May 2026 Official leaderboard
21 GPT-5.6 Sol 81.8% Jul 2026 Official leaderboard
22 Inkling-Small 79.6% Jul 2026 Lab model card
23 Claude Opus 4.7 79.1% max Apr 2026 Official leaderboard
24 Tencent Hy3 79.1% Jul 2026 Lab model card
25 Beam 1.0 78.7% Oct 2026 Lab model card
26 Gemini 3.1 Pro 78.2% high Feb 2026 Official leaderboard
27 GLM-5.2 77.8% Jun 2026 Official leaderboard
28 Claude Opus 4.6 76.8% max Feb 2026 Official leaderboard
29 Qwen3.7-Max 76.4% May 2026 Lab model card
30 Kimi K2.7 Code 76% Jun 2026 Lab model card
31 Inkling 76% xhigh Jul 2026 Official leaderboard
32 GLM-5.1 75.6% Apr 2026 Official leaderboard
33 Muse Glimmer 75.5% Aug 2026 Lab model card
34 GPT-5.5 75.3% xhigh Apr 2026 Official leaderboard
35 MiniMax-M3 74.2% Jun 2026 Lab model card
36 Qwen 3.6 Plus 74.1% Apr 2026 Lab model card
37 DeepSeek-V4-Pro 73.6% Apr 2026 Lab model card
38 Qwen3.7-Plus 73.2% Jun 2026 Lab model card
39 GPT-5.4 70.6% xhigh Mar 2026 Official leaderboard
40 Gemini 3 Pro 70.3% Nov 2025 Official leaderboard
41 Claude Opus 4.5 69.8% high Nov 2025 Official leaderboard
42 Claude Sonnet 4.6 69.5% Feb 2026 Official leaderboard
43 DeepSeek-V4-Flash 69% Apr 2026 Lab model card
44 GLM-5 67.8% Feb 2026 Lab model card
45 GPT-5.2 67.6% xhigh Dec 2025 Official leaderboard
46 Kimi K2.5 64.4% Jan 2026 Official leaderboard
47 Nemotron 3 Ultra 63.1% thinking Jun 2026 Official leaderboard
48 Qwen 3.6 35B-A3B 62.8% Apr 2026 Lab model card
49 Spark-X2.6-4B 62.4% Oct 2026 Lab model card
50 Gemini 3 Flash 62% Dec 2025 Official leaderboard
51 Solar Pro 4 61.4% Aug 2026 Lab model card
52 Claude Sonnet 4.5 59.5% thinking Sep 2025 Official leaderboard
53 Solar Open 2 58.2% Jul 2026 Lab model card
54 GLM-4.7 58.1% Dec 2025 Official leaderboard
55 GPT-5.4 Mini 57.7% Mar 2026 Lab model card
56 Gemini 3.1 Flash-Lite 57.1% high Mar 2026 Official leaderboard
57 GPT-5.4 Nano 56.1% Mar 2026 Lab model card
58 Nemotron 3.5 Bolt 74B-A9B 52.2% Oct 2026 Lab model card
59 GPT-5.1 50.1% high Nov 2025 Official leaderboard
60 o3-pro 44.5% Jun 2025 Official leaderboard
61 Claude Haiku 4.5 40.2% Oct 2025 Official leaderboard
62 LFM3-1.7B 32.6% Oct 2026 Lab model card
63 Nova 2 Lite 24.6% Dec 2025 Lab model card

Labs report scores for 46 models in their cards. Switch the scores above to rank them together.

7 models have held number one on the official leaderboard since its release in January 2026 9 models have held number one with claims included since its release in January 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Google DeepMind Anthropic Meta Z.ai Qwen ByteDance Seed
Own leaderboard Passed 85% in Jul 2026, by Muse Spark 1.1
Gemini 4 Argon 90.7%
Lab claims Passed 85% in Jul 2026, by Muse Spark 1.1
Muse Spark 1.2 90.3%

Sources: Official MCP Atlas leaderboard and labs’ model cards

Muse Spark 1.1 on the official leaderboard passed 85% in July 2026

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Gemini 4 Argon Seed2.1 Pro Gemini 3.1 Pro Gemini 3 Pro Claude Sonnet 4.5 o3-pro 0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Gemini 4 Argon Seed2.1 Pro

Sources: Official MCP Atlas leaderboard and labs’ model cards

15 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Anthropic
Amazon
Google DeepMind
Z.ai
OpenAI
Qwen
DeepSeek
MiniMax
Moonshot AI
Tencent
Meta
Thinking Machines
Upstage
StepFun
Reflection
Ant Group
NVIDIA
iFlytek
Liquid AI
Meituan
DateLaunch postScoreAs the post writes it
Nov 5, 2026 Inkling 2 Small 85 MCP Atlas (Public; 85.4 on Full)
Nov 2, 2026 LongCat-2.5 83.1 MCP Atlas
Oct 27, 2026 LFM3-1.7B 32.6 MCP-Atlas
Oct 22, 2026 Spark-X2.6-4B 62.4 MCP-Atlas
Oct 15, 2026 Muse Spark Open 89.2 MCP-Atlas
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B 52.2 MCP-Atlas (500 public tasks)
Oct 13, 2026 Ling-3.1-flash 83 Mcp-altas
Oct 5, 2026 Beam 1.0 78.7 MCP Atlas
Sep 20, 2026 Step 5 Preview 85.6 MCP-Atlas
Aug 28, 2026 Tencent Hy4 preview 83.7 MCP-Atlas (public)
Aug 11, 2026 Solar Pro 4 61.4 MCP-Atlas* (in-house)
Aug 10, 2026 Muse Glimmer 75.5 MCP Atlas (Public)
Aug 5, 2026 Muse Spark 1.2 90.3 MCP-Atlas, 90.3%
Jul 30, 2026 Inkling-Small 79.6 MCP Atlas public 79.6 / all 79.2
Jul 22, 2026 Solar Open 2 58.2 MCP-Atlas
Jul 16, 2026 Kimi K3 84.2 MCP-Atlas
Jul 15, 2026 Inkling 76 MCP Atlas
Jul 9, 2026 Muse Spark 1.1 88.1 MCP Atlas (Scaled tool use)
Jul 2, 2026 Tencent Hy3 79.1 MCP atlas (public)
Jun 16, 2026 GLM-5.2 76.8 MCP-Atlas (Public Set)
Jun 12, 2026 Kimi K2.7 Code 76 MCP Atlas
Jun 1, 2026 Qwen3.7-Plus 73.2 MCP-Atlas (public set)
Jun 1, 2026 MiniMax-M3 74.2 MCP Atlas
May 28, 2026 Claude Opus 4.8 82.2 MCP-Atlas
May 20, 2026 Qwen3.7-Max 76.4 MCP-Atlas
May 19, 2026 Gemini 3.5 Flash 83.6 MCP Atlas
Apr 24, 2026 DeepSeek-V4-Flash 69 MCPAtlas (Pass@1), V4-Flash Max
Apr 24, 2026 DeepSeek-V4-Pro 73.6 MCPAtlas Public (Pass@1)
Apr 23, 2026 GPT-5.5 75.3 MCP Atlas
Apr 16, 2026 Claude Opus 4.7 77.3 Scaled tool use MCP-Atlas
Apr 15, 2026 Qwen 3.6 35B-A3B 62.8 MCP-Atlas (public set)
Apr 7, 2026 GLM-5.1 71.8 MCP-Atlas (Public Set)
Apr 2, 2026 Qwen 3.6 Plus 74.1 MCP-Atlas
Mar 17, 2026 GPT-5.4 Nano 56.1 MCP Atlas
Mar 17, 2026 GPT-5.4 Mini 57.7 MCP-Atlas
Mar 5, 2026 GPT-5.4 67.2 MCP Atlas
Feb 19, 2026 Gemini 3.1 Pro 69.2 MCP Atlas
Feb 17, 2026 Claude Sonnet 4.6 61.3 Scaled tool use MCP-Atlas
Feb 11, 2026 GLM-5 67.8 MCP-Atlas (Public Set)
Feb 5, 2026 Claude Opus 4.6 59.5 Scaled tool use MCP Atlas
Dec 17, 2025 Gemini 3 Flash 57.4 MCP Atlas
Dec 2, 2025 Nova 2 Lite 24.6 MCP Atlas pass rate (blind eval)
Nov 24, 2025 Claude Opus 4.5 62.3 Scaled tool use MCP Atlas

21 system cards report it, from 11 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 19, 2026 Muse Glimmer technical report 75.5 MCPAtlas, 500 public tasks, high, 77.4 xhigh (Tables 2 and 5) Capabilities
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B technical report 52.2 MCP-Atlas, public 500 tasks, reasoning on, released 24GB checkpoint (Table 7 and Section 8). BF16 55.60 Capabilities
Jul 30, 2026 Inkling-Small model card 79.6 MCP Atlas public 79.6, all 79.2 (5. Evaluations table) Capabilities
Jul 27, 2026 Kimi K3 technical report 84.2 MCP-Atlas, 500-task public subset, Gemini 3.1 Pro judge (Table 2) Capabilities
Jul 24, 2026 Claude Opus 5 system card 85.8 MCP-Atlas pass rate (mean claim coverage 89.1%) (8.13.2) Capabilities
Jul 22, 2026 Solar Open 2 technical report 58.2 MCP-Atlas (Table 3) Capabilities
Jul 15, 2026 Inkling model card 76 MCP Atlas (5. Evaluations table, effort 0.99) Capabilities
Jul 9, 2026 Muse Spark 1.1 system card 88.1 MCP Atlas (Figure 44) Capabilities
Jun 23, 2026 Seed2.1 Pro model card 83.8 MCP-Atlas, Pass@1 (Table 2) Capabilities
Jun 9, 2026 Claude Fable 5 system card 83.3 MCP-Atlas pass rate (8.17.5) Capabilities
May 28, 2026 Claude Opus 4.8 system card 82.2 MCP-Atlas pass rate, run by Scale AI with a 100-tool-call budget, mean claim coverage 86.2% (Table 8.1.A, 8.13.4) Capabilities
May 19, 2026 Gemini 3.5 Flash model card 83.6 MCP Atlas (results table, p.4) Capabilities
Apr 26, 2026 DeepSeek-V4-Pro technical report 73.6 MCPAtlas Public (Pass@1), DS-V4-Pro Max (Table 6). High mode 74.2 (Table 7) Capabilities
Apr 26, 2026 DeepSeek-V4-Flash technical report 69 MCPAtlas Public (Pass@1), DeepSeek-V4-Flash Max (Table 7) Capabilities
Apr 16, 2026 Claude Opus 4.7 system card 77.3 MCP-Atlas pass rate, max effort, refreshed harness, run by Scale AI. Extended config 79.5 (Table 8.1.A, 8.10.3) Capabilities
Feb 19, 2026 Gemini 3.1 Pro model card 69.2 MCP Atlas (results table, p.4) Capabilities
Feb 17, 2026 GLM-5 technical report 67.8 MCP-Atlas (Public Set), Gemini 3 Pro judge, 10-minute timeout (Table 7) Capabilities
Feb 17, 2026 Claude Sonnet 4.6 system card 61.3 MCP-Atlas, max effort (Table 2.1.A, 2.14) Capabilities
Feb 5, 2026 Claude Opus 4.6 system card 59.5 MCP-Atlas, max effort (Table 2.3.A, 2.16); 62.7 at high effort Capabilities
Dec 2, 2025 Nova 2 Lite technical report 24.6 MCP Atlas pass rate, blind eval (Table 2) Capabilities
Nov 24, 2025 Claude Opus 4.5 system card 62.3 MCP Atlas, without extended thinking (Table 2.3.A, 2.12) Capabilities

The lab's number comes out ahead in 3 of 13 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 7 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
Claude Opus 4.6 59.5% Scaled tool use MCP Atlas 76.8% Leaderboard, max −17 pts
Gemini 3.1 Pro 69.2% MCP Atlas 78.2% Leaderboard, high −9.0 pts
Claude Sonnet 4.6 61.3% Scaled tool use MCP-Atlas 69.5% Leaderboard −8.2 pts
Claude Opus 4.5 62.3% Scaled tool use MCP Atlas 69.8% Leaderboard, high −7.5 pts
Gemini 3 Flash 57.4% MCP Atlas 62% Leaderboard −4.6 pts
GLM-5.1 71.8% MCP-Atlas (Public Set) 75.6% Leaderboard −3.8 pts
GPT-5.4 67.2% MCP Atlas 70.6% Leaderboard, xhigh −3.4 pts
Kimi K3 84.2% MCP-Atlas 82.3% Leaderboard, max +1.9 pts
Claude Opus 4.7 77.3% Scaled tool use MCP-Atlas 79.1% Leaderboard, max −1.8 pts
GPT-5.4 Mini 57.7% MCP-Atlas 56.7% Leaderboard, xhigh +1.0 pts
GLM-5.2 76.8% MCP-Atlas (Public Set) 77.8% Leaderboard −1.0 pts
Inkling 2 Small 85% MCP Atlas (Public; 85.4 on Full) 85.4% Leaderboard −0.4 pts
Inkling-Small 79.6% MCP Atlas public 79.6 / all 79.2 79.2% Leaderboard +0.4 pts
GPT-5.5 75.3% MCP Atlas 75.3% Leaderboard, xhigh Same
Gemini 3.5 Flash 83.6% MCP Atlas 83.6% Leaderboard, high Same
Claude Opus 4.8 82.2% MCP-Atlas 82.2% Leaderboard, max Same
Muse Spark 1.1 88.1% MCP Atlas (Scaled tool use) 88.1% Leaderboard Same
Inkling 76% MCP Atlas 76% Leaderboard, xhigh Same
Claude Fable 5 83.3% MCP-Atlas pass rate (8.17.5) 83.3% Leaderboard Same
Claude Opus 5 85.8% MCP-Atlas pass rate (mean claim coverage 89.1%) (8.13.2) 85.8% Leaderboard, xhigh Same

2 logged changes that move scores

Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.

  1. Harness changed October 23rd, 2026

    The maintainers enforced per-task tool restrictions during execution and stopped advertising disabled tools. Earlier runs could invoke tools outside the allowed set. Source

  2. Harness changed July 4th, 2026

    The maintainers replaced the Python harness with a TypeScript agent loop and configurable run controls. Scores from the old and new harnesses require matching execution settings. Source

More in Tool use

All 144 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%
  • Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%