Benchmark Almanac

MCP Atlas

Gemini 4 Argon 90.7% leads since September 2026
Gemini 4 Argon 90.7% leads since September 2026
In use

The official leaderboard hasn’t scored a model released after Inkling 2 Small (November 2026). Labs report their own numbers for one newer model: Inkling 2 Mini at 82.7%.

Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks.

500 public tasks of 1,000. Rest held back. Labs use different judge models, so self-reported scores vary.

Made by
Scale AI
Released
November 2025
Announced on X
5K views, 32 likes
Kind
Apps and MCP servers
Level
University
Scored
Graded by a model
Answers
Agent task
Tasks
Public, plus a private test set
Measures
Capability
Language
English
In and out
Text → Actions, Text
Interaction
Agentic
Environment
APIs and tools
Refresh
One fixed set
Run by
A host
In views
Agents
Strongest model tested
Claude Opus 5.5 #5 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 19th, 2026

In the authors’ words

The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evaluations fall short along three key axes: realistic multi-step workflows with cross-server orchestration, breadth across authentic MCP servers rather than mocks, and structured, reproducible claim-level scoring disentangled from agent verbosity or style. We introduce MCP-Atlas, a benchmark for measuring tool-use competency against production MCP servers. MCP-Atlas contains 1,000 natural-language tasks written and verified by human experts spanning 36 real MCP servers and 220 tools. Prompts do not specify servers, tools, or parameters, requiring agents to identify relevant tools among semantically plausible distractors and to compose multi-step, cross-server workflows. Each task is scored with a claim-level rubric, where final answers are scored against atomic factual claims grounded in tool outputs. This answer-centric scoring permits valid alternative tool-call trajectories to receive credit. We pair this with an 11-category diagnostic taxonomy that disentangles tool-call failures from cognitive failures in task understanding, synthesis, parsing, and stopping. Evaluating 20 frontier models from six providers under matched task-level conditions, we find pass rates up to 82.2% at a 0.75 claim coverage threshold and a clear three-tier performance structure. Automated diagnostics show that 63.3% of diagnosed failures are cognitive rather than tool-call related. Notably, several high-performing models fail after successful tool execution due to premature stopping or incorrect synthesis. We release the task schema, containerized harness, claim evaluator, and a 500-task public split, while reserving a 500-task private split to preserve leaderboard integrity. The code is at https://github.com/scaleapi/mcp-atlas.

Abstract of MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (arXiv:2602.00933, CC0)
37
Models scored on its own leaderboard
44
Launch posts list it, from 20 labs
22
System cards report it, from 11 labs

Gemini 4 Argon leads with 90.7%

Best score per model from the benchmark's own leaderboard, read on November 19th, 2026. Where a model ran with several settings or harnesses, we keep its best.

Gemini 4 Argon leads with 90.7%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

  1. 1 Gemini 4 Argon 90.7%
  2. 2 Muse Spark 1.1 88.1%
  3. 3 Claude Fable 5.1 87.2%
  4. 4 Claude Opus 5 xhigh 85.8%
  5. 5 Claude Opus 5.5 85.4%
  6. 6 Inkling 2 Small 85.4%
  7. 7 Qwen3.8-2.4T-A95B xhigh 84.5%
  8. 8 GLM-5.3 84.2%
  9. 9 Gemini 3.5 Flash high 83.6%
  10. 10 Claude Fable 5 83.3%
  11. 11 Kimi K3 max 82.3%
  12. 12 Muse Spark 82.2%
  13. 13 Claude Opus 4.8 max 82.2%
  14. 14 GPT-5.6 Sol 81.8%
  15. 15 Inkling-Small 79.2%
  16. 16 Claude Opus 4.7 max 79.1%
  17. 17 Gemini 3.1 Pro high 78.2%
  18. 18 GLM-5.2 77.8%
  19. 19 Claude Opus 4.6 max 76.8%
  20. 20 Inkling xhigh 76%
  21. 21 GLM-5.1 75.6%
  22. 22 GPT-5.5 xhigh 75.3%
  23. 23 GPT-5.4 xhigh 70.6%
  24. 24 Gemini 3 Pro 70.3%
  25. 25 Claude Opus 4.5 high 69.8%
  26. 26 Claude Sonnet 4.6 69.5%
  27. 27 GPT-5.2 xhigh 67.6%
  28. 28 Kimi K2.5 64.4%
  29. 29 Nemotron 3 Ultra thinking 63.1%
  30. 30 Gemini 3 Flash 62%
  31. 31 Claude Sonnet 4.5 thinking 59.5%
  32. 32 GLM-4.7 58.1%
  33. 33 Gemini 3.1 Flash-Lite high 57.1%
  34. 34 GPT-5.4 Mini xhigh 56.7%
  35. 35 GPT-5.1 high 50.1%
  36. 36 o3-pro 44.5%
  37. 37 Claude Haiku 4.5 40.2%

Source: Official MCP Atlas leaderboard

  1. 1 Gemini 4 Argon 90.7%
  2. 2 Muse Spark 1.2 90.3%
  3. 3 Muse Spark Open 89.2%
  4. 4 Muse Spark 1.1 88.1%
  5. 5 Claude Fable 5.1 87.2%
  6. 6 Claude Opus 5 xhigh 85.8%
  7. 7 Step 5 Preview 85.6%
  8. 8 Claude Opus 5.5 85.4%
  9. 9 Inkling 2 Small 85.4%
  10. 10 Qwen3.8-2.4T-A95B xhigh 84.5%
  11. 11 Kimi K3 84.2%
  12. 12 GLM-5.3 84.2%
  13. 13 Seed2.1 Pro 83.8%
  14. 14 Tencent Hy4 preview 83.7%
  15. 15 Gemini 3.5 Flash high 83.6%
  16. 16 Claude Fable 5 83.3%
  17. 17 LongCat-2.5 83.1%
  18. 18 Ling-3.1-flash 83%
  19. 19 Inkling 2 Mini 82.7%
  20. 20 Muse Spark 82.2%
  21. 21 Claude Opus 4.8 max 82.2%
  22. 22 GPT-5.6 Sol 81.8%
  23. 23 Inkling-Small 79.6%
  24. 24 Claude Opus 4.7 max 79.1%
  25. 25 Tencent Hy3 79.1%
  26. 26 Beam 1.0 78.7%
  27. 27 Gemini 3.1 Pro high 78.2%
  28. 28 GLM-5.2 77.8%
  29. 29 Claude Opus 4.6 max 76.8%
  30. 30 Qwen3.7-Max 76.4%
  31. 31 Kimi K2.7 Code 76%
  32. 32 Inkling xhigh 76%
  33. 33 GLM-5.1 75.6%
  34. 34 Muse Glimmer 75.5%
  35. 35 GPT-5.5 xhigh 75.3%
  36. 36 MiniMax-M3 74.2%
  37. 37 Qwen 3.6 Plus 74.1%
  38. 38 DeepSeek-V4-Pro 73.6%
  39. 39 Qwen3.7-Plus 73.2%
  40. 40 GPT-5.4 xhigh 70.6%
  41. 41 Gemini 3 Pro 70.3%
  42. 42 Claude Opus 4.5 high 69.8%
  43. 43 Claude Sonnet 4.6 69.5%
  44. 44 DeepSeek-V4-Flash 69%
  45. 45 GLM-5 67.8%
  46. 46 GPT-5.2 xhigh 67.6%
  47. 47 Kimi K2.5 64.4%
  48. 48 Nemotron 3 Ultra thinking 63.1%
  49. 49 Qwen 3.6 35B-A3B 62.8%
  50. 50 Spark-X2.6-4B 62.4%
  51. 51 Gemini 3 Flash 62%
  52. 52 Solar Pro 4 61.4%
  53. 53 Claude Sonnet 4.5 thinking 59.5%
  54. 54 Solar Open 2 58.2%
  55. 55 GLM-4.7 58.1%
  56. 56 GPT-5.4 Mini 57.7%
  57. 57 Gemini 3.1 Flash-Lite high 57.1%
  58. 58 GPT-5.4 Nano 56.1%
  59. 59 Nemotron 3.5 Bolt 74B-A9B 52.2%
  60. 60 GPT-5.1 high 50.1%
  61. 61 o3-pro 44.5%
  62. 62 Claude Haiku 4.5 40.2%
  63. 63 LFM3-1.7B 32.6%
  64. 64 Nova 2 Lite 24.6%

Sources: Official MCP Atlas leaderboard and labs’ model cards

#ModelScore ±SettingDateSource
1 Gemini 4 Argon 90.7% ±1.8 Sep 2026 Official leaderboard
2 Muse Spark 1.1 88.1% ±1.9 Jul 2026 Official leaderboard
3 Claude Fable 5.1 87.2% ±2 Sep 2026 Official leaderboard
4 Claude Opus 5 85.8% ±2.1 xhigh Jul 2026 Official leaderboard
5 Claude Opus 5.5 85.4% ±2.2 Sep 2026 Official leaderboard
6 Inkling 2 Small 85.4% ±2.2 Nov 2026 Official leaderboard
7 Qwen3.8-2.4T-A95B 84.5% ±2.3 xhigh Aug 2026 Official leaderboard
8 GLM-5.3 84.2% ±2.1 Aug 2026 Official leaderboard
9 Gemini 3.5 Flash 83.6% ±2.3 high May 2026 Official leaderboard
10 Claude Fable 5 83.3% ±2.3 Jun 2026 Official leaderboard
11 Kimi K3 82.3% ±2.4 max Jul 2026 Official leaderboard
12 Muse Spark 82.2% ±2.3 Apr 2026 Official leaderboard
13 Claude Opus 4.8 82.2% ±2.4 max May 2026 Official leaderboard
14 GPT-5.6 Sol 81.8% ±2.4 Jul 2026 Official leaderboard
15 Inkling-Small 79.2% ±2.5 Jul 2026 Official leaderboard
16 Claude Opus 4.7 79.1% ±2.5 max Apr 2026 Official leaderboard
17 Gemini 3.1 Pro 78.2% ±2.5 high Feb 2026 Official leaderboard
18 GLM-5.2 77.8% ±2.6 Jun 2026 Official leaderboard
19 Claude Opus 4.6 76.8% ±2.7 max Feb 2026 Official leaderboard
20 Inkling 76% ±2.6 xhigh Jul 2026 Official leaderboard
21 GLM-5.1 75.6% ±2.7 Apr 2026 Official leaderboard
22 GPT-5.5 75.3% ±2.7 xhigh Apr 2026 Official leaderboard
23 GPT-5.4 70.6% ±2.8 xhigh Mar 2026 Official leaderboard
24 Gemini 3 Pro 70.3% ±2.8 Nov 2025 Official leaderboard
25 Claude Opus 4.5 69.8% ±2.9 high Nov 2025 Official leaderboard
26 Claude Sonnet 4.6 69.5% ±2.9 Feb 2026 Official leaderboard
27 GPT-5.2 67.6% ±2.9 xhigh Dec 2025 Official leaderboard
28 Kimi K2.5 64.4% ±3 Jan 2026 Official leaderboard
29 Nemotron 3 Ultra 63.1% ±3 thinking Jun 2026 Official leaderboard
30 Gemini 3 Flash 62% ±3 Dec 2025 Official leaderboard
31 Claude Sonnet 4.5 59.5% ±3.1 thinking Sep 2025 Official leaderboard
32 GLM-4.7 58.1% ±3 Dec 2025 Official leaderboard
33 Gemini 3.1 Flash-Lite 57.1% ±3 high Mar 2026 Official leaderboard
34 GPT-5.4 Mini 56.7% ±3.1 xhigh Mar 2026 Official leaderboard
35 GPT-5.1 50.1% ±3.1 high Nov 2025 Official leaderboard
36 o3-pro 44.5% ±3.1 Jun 2025 Official leaderboard
37 Claude Haiku 4.5 40.2% ±3 Oct 2025 Official leaderboard
#ModelScoreSettingDateSource
1 Gemini 4 Argon 90.7% Sep 2026 Official leaderboard
2 Muse Spark 1.2 90.3% Aug 2026 Lab model card
3 Muse Spark Open 89.2% Oct 2026 Lab model card
4 Muse Spark 1.1 88.1% Jul 2026 Official leaderboard
5 Claude Fable 5.1 87.2% Sep 2026 Official leaderboard
6 Claude Opus 5 85.8% xhigh Jul 2026 Official leaderboard
7 Step 5 Preview 85.6% Sep 2026 Lab model card
8 Claude Opus 5.5 85.4% Sep 2026 Official leaderboard
9 Inkling 2 Small 85.4% Nov 2026 Official leaderboard
10 Qwen3.8-2.4T-A95B 84.5% xhigh Aug 2026 Official leaderboard
11 Kimi K3 84.2% Jul 2026 Lab model card
12 GLM-5.3 84.2% Aug 2026 Official leaderboard
13 Seed2.1 Pro 83.8% Jun 2026 Lab model card
14 Tencent Hy4 preview 83.7% Aug 2026 Lab model card
15 Gemini 3.5 Flash 83.6% high May 2026 Official leaderboard
16 Claude Fable 5 83.3% Jun 2026 Official leaderboard
17 LongCat-2.5 83.1% Nov 2026 Lab model card
18 Ling-3.1-flash 83% Oct 2026 Lab model card
19 Inkling 2 Mini 82.7% Nov 2026 Lab model card
20 Muse Spark 82.2% Apr 2026 Official leaderboard
21 Claude Opus 4.8 82.2% max May 2026 Official leaderboard
22 GPT-5.6 Sol 81.8% Jul 2026 Official leaderboard
23 Inkling-Small 79.6% Jul 2026 Lab model card
24 Claude Opus 4.7 79.1% max Apr 2026 Official leaderboard
25 Tencent Hy3 79.1% Jul 2026 Lab model card
26 Beam 1.0 78.7% Oct 2026 Lab model card
27 Gemini 3.1 Pro 78.2% high Feb 2026 Official leaderboard
28 GLM-5.2 77.8% Jun 2026 Official leaderboard
29 Claude Opus 4.6 76.8% max Feb 2026 Official leaderboard
30 Qwen3.7-Max 76.4% May 2026 Lab model card
31 Kimi K2.7 Code 76% Jun 2026 Lab model card
32 Inkling 76% xhigh Jul 2026 Official leaderboard
33 GLM-5.1 75.6% Apr 2026 Official leaderboard
34 Muse Glimmer 75.5% Aug 2026 Lab model card
35 GPT-5.5 75.3% xhigh Apr 2026 Official leaderboard
36 MiniMax-M3 74.2% Jun 2026 Lab model card
37 Qwen 3.6 Plus 74.1% Apr 2026 Lab model card
38 DeepSeek-V4-Pro 73.6% Apr 2026 Lab model card
39 Qwen3.7-Plus 73.2% Jun 2026 Lab model card
40 GPT-5.4 70.6% xhigh Mar 2026 Official leaderboard
41 Gemini 3 Pro 70.3% Nov 2025 Official leaderboard
42 Claude Opus 4.5 69.8% high Nov 2025 Official leaderboard
43 Claude Sonnet 4.6 69.5% Feb 2026 Official leaderboard
44 DeepSeek-V4-Flash 69% Apr 2026 Lab model card
45 GLM-5 67.8% Feb 2026 Lab model card
46 GPT-5.2 67.6% xhigh Dec 2025 Official leaderboard
47 Kimi K2.5 64.4% Jan 2026 Official leaderboard
48 Nemotron 3 Ultra 63.1% thinking Jun 2026 Official leaderboard
49 Qwen 3.6 35B-A3B 62.8% Apr 2026 Lab model card
50 Spark-X2.6-4B 62.4% Oct 2026 Lab model card
51 Gemini 3 Flash 62% Dec 2025 Official leaderboard
52 Solar Pro 4 61.4% Aug 2026 Lab model card
53 Claude Sonnet 4.5 59.5% thinking Sep 2025 Official leaderboard
54 Solar Open 2 58.2% Jul 2026 Lab model card
55 GLM-4.7 58.1% Dec 2025 Official leaderboard
56 GPT-5.4 Mini 57.7% Mar 2026 Lab model card
57 Gemini 3.1 Flash-Lite 57.1% high Mar 2026 Official leaderboard
58 GPT-5.4 Nano 56.1% Mar 2026 Lab model card
59 Nemotron 3.5 Bolt 74B-A9B 52.2% Oct 2026 Lab model card
60 GPT-5.1 50.1% high Nov 2025 Official leaderboard
61 o3-pro 44.5% Jun 2025 Official leaderboard
62 Claude Haiku 4.5 40.2% Oct 2025 Official leaderboard
63 LFM3-1.7B 32.6% Oct 2026 Lab model card
64 Nova 2 Lite 24.6% Dec 2025 Lab model card

Labs report scores for 47 models in their cards. Switch the scores above to rank them together.

8 models have held number one on the official leaderboard since its release in November 2025 10 models have held number one with claims included since its release in November 2025

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Anthropic Google DeepMind Meta Z.ai Qwen ByteDance Seed
Own leaderboard Passed 85% in Jul 2026, by Muse Spark 1.1
Lab claims Passed 85% in Jul 2026, by Muse Spark 1.1

Sources: Official MCP Atlas leaderboard and labs’ model cards

Muse Spark 1.1 on the official leaderboard passed 85% in July 2026

That was 8 months after its release. By the same age, 16% of the benchmarks from 2025 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Gemini 4 Argon Seed2.1 Pro Gemini 3.1 Pro Gemini 3 Pro Claude Sonnet 4.5 o3-pro 0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Gemini 4 Argon Seed2.1 Pro

Sources: Official MCP Atlas leaderboard and labs’ model cards

15 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Anthropic
Amazon
Google DeepMind
Z.ai
OpenAI
Qwen
DeepSeek
MiniMax
Moonshot AI
Tencent
Meta
Thinking Machines
Upstage
StepFun
Reflection
Ant Group
NVIDIA
iFlytek
Liquid AI
Meituan
DateLaunch postScoreAs the post writes it
Nov 20, 2026 Inkling 2 Mini 82.7 MCP Atlas (Public)
Nov 5, 2026 Inkling 2 Small 85 MCP Atlas (Public; 85.4 on Full)
Nov 2, 2026 LongCat-2.5 83.1 MCP Atlas
Oct 27, 2026 LFM3-1.7B 32.6 MCP-Atlas
Oct 22, 2026 Spark-X2.6-4B 62.4 MCP-Atlas
Oct 15, 2026 Muse Spark Open 89.2 MCP-Atlas
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B 52.2 MCP-Atlas (500 public tasks)
Oct 13, 2026 Ling-3.1-flash 83 Mcp-altas
Oct 5, 2026 Beam 1.0 78.7 MCP Atlas
Sep 20, 2026 Step 5 Preview 85.6 MCP-Atlas
Aug 28, 2026 Tencent Hy4 preview 83.7 MCP-Atlas (public)
Aug 11, 2026 Solar Pro 4 61.4 MCP-Atlas* (in-house)
Aug 10, 2026 Muse Glimmer 75.5 MCP Atlas (Public)
Aug 5, 2026 Muse Spark 1.2 90.3 MCP-Atlas, 90.3%
Jul 30, 2026 Inkling-Small 79.6 MCP Atlas public 79.6 / all 79.2
Jul 22, 2026 Solar Open 2 58.2 MCP-Atlas
Jul 16, 2026 Kimi K3 84.2 MCP-Atlas
Jul 15, 2026 Inkling 76 MCP Atlas
Jul 9, 2026 Muse Spark 1.1 88.1 MCP Atlas (Scaled tool use)
Jul 2, 2026 Tencent Hy3 79.1 MCP atlas (public)
Jun 16, 2026 GLM-5.2 76.8 MCP-Atlas (Public Set)
Jun 12, 2026 Kimi K2.7 Code 76 MCP Atlas
Jun 1, 2026 Qwen3.7-Plus 73.2 MCP-Atlas (public set)
Jun 1, 2026 MiniMax-M3 74.2 MCP Atlas
May 28, 2026 Claude Opus 4.8 82.2 MCP-Atlas
May 20, 2026 Qwen3.7-Max 76.4 MCP-Atlas
May 19, 2026 Gemini 3.5 Flash 83.6 MCP Atlas
Apr 24, 2026 DeepSeek-V4-Flash 69 MCPAtlas (Pass@1), V4-Flash Max
Apr 24, 2026 DeepSeek-V4-Pro 73.6 MCPAtlas Public (Pass@1)
Apr 23, 2026 GPT-5.5 75.3 MCP Atlas
Apr 16, 2026 Claude Opus 4.7 77.3 Scaled tool use MCP-Atlas
Apr 15, 2026 Qwen 3.6 35B-A3B 62.8 MCP-Atlas (public set)
Apr 7, 2026 GLM-5.1 71.8 MCP-Atlas (Public Set)
Apr 2, 2026 Qwen 3.6 Plus 74.1 MCP-Atlas
Mar 17, 2026 GPT-5.4 Nano 56.1 MCP Atlas
Mar 17, 2026 GPT-5.4 Mini 57.7 MCP-Atlas
Mar 5, 2026 GPT-5.4 67.2 MCP Atlas
Feb 19, 2026 Gemini 3.1 Pro 69.2 MCP Atlas
Feb 17, 2026 Claude Sonnet 4.6 61.3 Scaled tool use MCP-Atlas
Feb 11, 2026 GLM-5 67.8 MCP-Atlas (Public Set)
Feb 5, 2026 Claude Opus 4.6 59.5 Scaled tool use MCP Atlas
Dec 17, 2025 Gemini 3 Flash 57.4 MCP Atlas
Dec 2, 2025 Nova 2 Lite 24.6 MCP Atlas pass rate (blind eval)
Nov 24, 2025 Claude Opus 4.5 62.3 Scaled tool use MCP Atlas

22 system cards report it, from 11 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 20, 2026 Inkling 2 Mini model card 82.7 MCP Atlas (Public) Capabilities
Oct 19, 2026 Muse Glimmer technical report 75.5 MCPAtlas, 500 public tasks, high, 77.4 xhigh (Tables 2 and 5) Capabilities
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B technical report 52.2 MCP-Atlas, public 500 tasks, reasoning on, released 24GB checkpoint (Table 7 and Section 8). BF16 55.60 Capabilities
Jul 30, 2026 Inkling-Small model card 79.6 MCP Atlas public 79.6, all 79.2 (5. Evaluations table) Capabilities
Jul 27, 2026 Kimi K3 technical report 84.2 MCP-Atlas, 500-task public subset, Gemini 3.1 Pro judge (Table 2) Capabilities
Jul 24, 2026 Claude Opus 5 system card 85.8 MCP-Atlas pass rate (mean claim coverage 89.1%) (8.13.2) Capabilities
Jul 22, 2026 Solar Open 2 technical report 58.2 MCP-Atlas (Table 3) Capabilities
Jul 15, 2026 Inkling model card 76 MCP Atlas (5. Evaluations table, effort 0.99) Capabilities
Jul 9, 2026 Muse Spark 1.1 system card 88.1 MCP Atlas (Figure 44) Capabilities
Jun 23, 2026 Seed2.1 Pro model card 83.8 MCP-Atlas, Pass@1 (Table 2) Capabilities
Jun 9, 2026 Claude Fable 5 system card 83.3 MCP-Atlas pass rate (8.17.5) Capabilities
May 28, 2026 Claude Opus 4.8 system card 82.2 MCP-Atlas pass rate, run by Scale AI with a 100-tool-call budget, mean claim coverage 86.2% (Table 8.1.A, 8.13.4) Capabilities
May 19, 2026 Gemini 3.5 Flash model card 83.6 MCP Atlas (results table, p.4) Capabilities
Apr 26, 2026 DeepSeek-V4-Pro technical report 73.6 MCPAtlas Public (Pass@1), DS-V4-Pro Max (Table 6). High mode 74.2 (Table 7) Capabilities
Apr 26, 2026 DeepSeek-V4-Flash technical report 69 MCPAtlas Public (Pass@1), DeepSeek-V4-Flash Max (Table 7) Capabilities
Apr 16, 2026 Claude Opus 4.7 system card 77.3 MCP-Atlas pass rate, max effort, refreshed harness, run by Scale AI. Extended config 79.5 (Table 8.1.A, 8.10.3) Capabilities
Feb 19, 2026 Gemini 3.1 Pro model card 69.2 MCP Atlas (results table, p.4) Capabilities
Feb 17, 2026 GLM-5 technical report 67.8 MCP-Atlas (Public Set), Gemini 3 Pro judge, 10-minute timeout (Table 7) Capabilities
Feb 17, 2026 Claude Sonnet 4.6 system card 61.3 MCP-Atlas, max effort (Table 2.1.A, 2.14) Capabilities
Feb 5, 2026 Claude Opus 4.6 system card 59.5 MCP-Atlas, max effort (Table 2.3.A, 2.16); 62.7 at high effort Capabilities
Dec 2, 2025 Nova 2 Lite technical report 24.6 MCP Atlas pass rate, blind eval (Table 2) Capabilities
Nov 24, 2025 Claude Opus 4.5 system card 62.3 MCP Atlas, without extended thinking (Table 2.3.A, 2.12) Capabilities

The lab's number comes out ahead in 3 of 13 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 7 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap What differs
Claude Opus 4.6 59.5% Scaled tool use MCP Atlas 76.8% Leaderboard, max −17 pts Anthropic reported its February evaluation. Scale later changed the judge, added tool retries, replaced the 20 turn limit with 100 tool calls, and rescored models.
Gemini 3.1 Pro 69.2% MCP Atlas 78.2% Leaderboard, high −9.0 pts
Claude Sonnet 4.6 61.3% Scaled tool use MCP-Atlas 69.5% Leaderboard −8.2 pts
Claude Opus 4.5 62.3% Scaled tool use MCP Atlas 69.8% Leaderboard, high −7.5 pts
Gemini 3 Flash 57.4% MCP Atlas 62% Leaderboard −4.6 pts
GLM-5.1 71.8% MCP-Atlas (Public Set) 75.6% Leaderboard −3.8 pts
GPT-5.4 67.2% MCP Atlas 70.6% Leaderboard, xhigh −3.4 pts
Kimi K3 84.2% MCP-Atlas 82.3% Leaderboard, max +1.9 pts
Claude Opus 4.7 77.3% Scaled tool use MCP-Atlas 79.1% Leaderboard, max −1.8 pts
GPT-5.4 Mini 57.7% MCP-Atlas 56.7% Leaderboard, xhigh +1.0 pts
GLM-5.2 76.8% MCP-Atlas (Public Set) 77.8% Leaderboard −1.0 pts
Inkling 2 Small 85% MCP Atlas (Public; 85.4 on Full) 85.4% Leaderboard −0.4 pts
Inkling-Small 79.6% MCP Atlas public 79.6 / all 79.2 79.2% Leaderboard +0.4 pts
GPT-5.5 75.3% MCP Atlas 75.3% Leaderboard, xhigh Same
Gemini 3.5 Flash 83.6% MCP Atlas 83.6% Leaderboard, high Same
Claude Opus 4.8 82.2% MCP-Atlas 82.2% Leaderboard, max Same
Muse Spark 1.1 88.1% MCP Atlas (Scaled tool use) 88.1% Leaderboard Same
Inkling 76% MCP Atlas 76% Leaderboard, xhigh Same
Claude Fable 5 83.3% MCP-Atlas pass rate (8.17.5) 83.3% Leaderboard Same
Claude Opus 5 85.8% MCP-Atlas pass rate (mean claim coverage 89.1%) (8.13.2) 85.8% Leaderboard, xhigh Same

2 logged changes that move scores

Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.

  1. Harness changed October 23rd, 2026

    The maintainers enforced per-task tool restrictions during execution and stopped advertising disabled tools. Earlier runs could invoke tools outside the allowed set. Source

  2. Harness changed July 4th, 2026

    The maintainers replaced the Python harness with a TypeScript agent loop and configurable run controls. Scores from the old and new harnesses require matching execution settings. Source

More in Tool use

All 265 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Neon 56.3%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%
  • Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%

Data audit

All 10 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access Public tasks, plus a private test set.
  • Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (37 entries), launch posts (44 scores) and system cards (22 scores).
  • Pass Models scored 64 models we track.
  • Pass Newest score The newest is from November 2026, for Inkling 2 Mini.
  • Pass Frontier models 3 of the top 10 have a score, led by Claude Opus 5.5 (#5).
  • Note Headroom Saturated. The best is 90.7%, past 85% since July 2026.
Leaderboard import 3 pass
  • Pass Importer We read the board on labs.scale.com: 37 rows for models we track at the last read.
  • Pass Last read November 19th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims 1 large gap of 20 pairs, each explained in our notes.
  • Pass Known issues No published audit found problems.

How we check