Benchmark Almanac

Toolathlon-Verified

GLM-5.3-Flash 78.4% leads since August 2026
Claude Opus 5 80.6% leads since July 2026
In use

108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state.

A checked subset of Toolathlon tasks.

Made by
HKUST NLP
Released
June 2026
Kind
Apps and MCP servers
Level
University
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Strongest model tested
Claude Opus 5.5 #4 on the capability index
Last in a launch post
October 2026
Last model added
October 2026
Scores refreshed
November 12th, 2026
25
Models scored on its own leaderboard
17
Launch posts list it, from 10 labs
9
System cards report it, from 5 labs

GLM-5.3-Flash leads with 78.4% among models released by August 2026

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best.

Claude Opus 5 leads with 80.6%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in August 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

25 models. Scroll sideways for the rest.

  1. 78.4 GLM-5.3-Flash
  2. 76.5 Kimi K3
  3. 76.2 Claude Opus 4.8
  4. 75.9 Muse Spark 1.2
  5. 75.6 Muse Spark 1.1
  6. 74.4 DeepSeek V4 Pro 0813
  7. 73.5 GPT-5.5
  8. 71.6 Claude Sonnet 5
  9. 70.7 DeepSeek V4 Flash 0731
  10. 67.3 Gemini 3.5 Flash
  11. 61.1 Gemini 3.1 Pro
  12. 59.9 GLM-5.2
  13. 58.0 Kimi K2.6
  14. 58.0 Kimi K2.7 Code
  15. 57.1 Gemini 3.5 Flash-Lite
  16. 56.8 Tencent Hy3
  17. 55.9 DeepSeek-V4-Pro
  18. 54.4 Inkling-Small
  19. 50.9 DeepSeek-V4-Flash
  20. 49.1 MiMo-V2.5
  21. 47.5 MiniMax-M2.7
  22. 45.5 Inkling
  23. 40.7 Qwen3.5 397B-A17B
  24. 34.3 Nemotron 3 Ultra
  25. 33.0 Kimi K2.5
  1. 1 GLM-5.3-Flash max 78.4%
  2. 2 Kimi K3 max 76.5%
  3. 3 Claude Opus 4.8 max 76.2%
  4. 4 Muse Spark 1.2 xhigh 75.9%
  5. 5 Muse Spark 1.1 xhigh 75.6%
  6. 6 DeepSeek V4 Pro 0813 max 74.4%
  7. 7 GPT-5.5 xhigh 73.5%
  8. 8 Claude Sonnet 5 max 71.6%
  9. 9 DeepSeek V4 Flash 0731 max 70.7%
  10. 10 Gemini 3.5 Flash high 67.3%
  11. 11 Gemini 3.1 Pro high 61.1%
  12. 12 GLM-5.2 max 59.9%
  13. 13 Kimi K2.6 58%
  14. 14 Kimi K2.7 Code 58%
  15. 15 Gemini 3.5 Flash-Lite high 57.1%
  16. 16 Tencent Hy3 high 56.8%
  17. 17 DeepSeek-V4-Pro max 55.9%
  18. 18 Inkling-Small xhigh 54.4%
  19. 19 DeepSeek-V4-Flash max 50.9%
  20. 20 MiMo-V2.5 49.1%
  21. 21 MiniMax-M2.7 47.5%
  22. 22 Inkling xhigh 45.5%
  23. 23 Qwen3.5 397B-A17B 40.7%
  24. 24 Nemotron 3 Ultra 34.3%
  25. 25 Kimi K2.5 33%

Source: Official Toolathlon-Verified leaderboard

39 models. Scroll sideways for the rest.

  1. 80.6 Claude Opus 5
  2. 78.4 GLM-5.3-Flash
  3. 77.8 Claude Fable 5.1
  4. 77.8 Claude Opus 5.5
  5. 77.8 Claude Sonnet 5.5
  6. 76.9 MiMo-V2.6 Pro
  7. 76.5 Kimi K3
  8. 76.2 Claude Opus 4.8
  9. 75.9 Muse Spark 1.2
  10. 75.9 DeepSeek-V4-Flash-Vision-Exp
  11. 75.6 Muse Spark 1.1
  12. 74.4 DeepSeek V4 Pro 0813
  13. 74.1 Tencent Hy4 preview
  14. 74.1 Step 5 Preview
  15. 73.5 GPT-5.5
  16. 73.5 Qwen3.8 Flash
  17. 73.3 Qwen3.8 Max (0902)
  18. 73.0 GLM-5.3
  19. 72.8 Muse Spark Open
  20. 72.5 Qwen 3.8 Max
  21. 71.6 Claude Sonnet 5
  22. 70.7 DeepSeek V4 Flash 0731
  23. 67.3 Gemini 3.5 Flash
  24. 61.1 Gemini 3.1 Pro
  25. 59.9 GLM-5.2
  26. 58.0 Kimi K2.6
  27. 58.0 Kimi K2.7 Code
  28. 57.1 Gemini 3.5 Flash-Lite
  29. 56.8 Tencent Hy3
  30. 55.9 DeepSeek-V4-Pro
  31. 54.4 Inkling-Small
  32. 50.9 DeepSeek-V4-Flash
  33. 49.7 Laguna S 2.1
  34. 49.1 MiMo-V2.5
  35. 47.5 MiniMax-M2.7
  36. 45.5 Inkling
  37. 40.7 Qwen3.5 397B-A17B
  38. 34.3 Nemotron 3 Ultra
  39. 33.0 Kimi K2.5
  1. 1 Claude Opus 5 80.6%
  2. 2 GLM-5.3-Flash max 78.4%
  3. 3 Claude Fable 5.1 77.8%
  4. 4 Claude Opus 5.5 77.8%
  5. 5 Claude Sonnet 5.5 77.8%
  6. 6 MiMo-V2.6 Pro 76.9%
  7. 7 Kimi K3 max 76.5%
  8. 8 Claude Opus 4.8 max 76.2%
  9. 9 Muse Spark 1.2 xhigh 75.9%
  10. 10 DeepSeek-V4-Flash-Vision-Exp 75.9%
  11. 11 Muse Spark 1.1 xhigh 75.6%
  12. 12 DeepSeek V4 Pro 0813 max 74.4%
  13. 13 Tencent Hy4 preview 74.1%
  14. 14 Step 5 Preview 74.1%
  15. 15 GPT-5.5 xhigh 73.5%
  16. 16 Qwen3.8 Flash 73.5%
  17. 17 Qwen3.8 Max (0902) 73.3%
  18. 18 GLM-5.3 73%
  19. 19 Muse Spark Open 72.8%
  20. 20 Qwen 3.8 Max 72.5%
  21. 21 Claude Sonnet 5 max 71.6%
  22. 22 DeepSeek V4 Flash 0731 max 70.7%
  23. 23 Gemini 3.5 Flash high 67.3%
  24. 24 Gemini 3.1 Pro high 61.1%
  25. 25 GLM-5.2 max 59.9%
  26. 26 Kimi K2.6 58%
  27. 27 Kimi K2.7 Code 58%
  28. 28 Gemini 3.5 Flash-Lite high 57.1%
  29. 29 Tencent Hy3 high 56.8%
  30. 30 DeepSeek-V4-Pro max 55.9%
  31. 31 Inkling-Small xhigh 54.4%
  32. 32 DeepSeek-V4-Flash max 50.9%
  33. 33 Laguna S 2.1 49.7%
  34. 34 MiMo-V2.5 49.1%
  35. 35 MiniMax-M2.7 47.5%
  36. 36 Inkling xhigh 45.5%
  37. 37 Qwen3.5 397B-A17B 40.7%
  38. 38 Nemotron 3 Ultra 34.3%
  39. 39 Kimi K2.5 33%

Sources: Labs’ model cards and official Toolathlon-Verified leaderboard

#ModelScoreSettingDateSource
1 GLM-5.3-Flash 78.4% max Aug 2026 Official leaderboard
2 Kimi K3 76.5% max Jul 2026 Official leaderboard
3 Claude Opus 4.8 76.2% max May 2026 Official leaderboard
4 Muse Spark 1.2 75.9% xhigh Aug 2026 Official leaderboard
5 Muse Spark 1.1 75.6% xhigh Jul 2026 Official leaderboard
6 DeepSeek V4 Pro 0813 74.4% max Aug 2026 Official leaderboard
7 GPT-5.5 73.5% xhigh Apr 2026 Official leaderboard
8 Claude Sonnet 5 71.6% max Jun 2026 Official leaderboard
9 DeepSeek V4 Flash 0731 70.7% max Jul 2026 Official leaderboard
10 Gemini 3.5 Flash 67.3% high May 2026 Official leaderboard
11 Gemini 3.1 Pro 61.1% high Feb 2026 Official leaderboard
12 GLM-5.2 59.9% max Jun 2026 Official leaderboard
13 Kimi K2.6 58% Apr 2026 Official leaderboard
14 Kimi K2.7 Code 58% Jun 2026 Official leaderboard
15 Gemini 3.5 Flash-Lite 57.1% high Jul 2026 Official leaderboard
16 Tencent Hy3 56.8% high Jul 2026 Official leaderboard
17 DeepSeek-V4-Pro 55.9% max Apr 2026 Official leaderboard
18 Inkling-Small 54.4% xhigh Jul 2026 Official leaderboard
19 DeepSeek-V4-Flash 50.9% max Apr 2026 Official leaderboard
20 MiMo-V2.5 49.1% Apr 2026 Official leaderboard
21 MiniMax-M2.7 47.5% Mar 2026 Official leaderboard
22 Inkling 45.5% xhigh Jul 2026 Official leaderboard
23 Qwen3.5 397B-A17B 40.7% Feb 2026 Official leaderboard
24 Nemotron 3 Ultra 34.3% Jun 2026 Official leaderboard
25 Kimi K2.5 33% Jan 2026 Official leaderboard
#ModelScoreSettingDateSource
1 Claude Opus 5 80.6% Jul 2026 Lab model card
2 GLM-5.3-Flash 78.4% max Aug 2026 Official leaderboard
3 Claude Fable 5.1 77.8% Sep 2026 Lab model card
4 Claude Opus 5.5 77.8% Sep 2026 Lab model card
5 Claude Sonnet 5.5 77.8% Sep 2026 Lab model card
6 MiMo-V2.6 Pro 76.9% Sep 2026 Lab model card
7 Kimi K3 76.5% max Jul 2026 Official leaderboard
8 Claude Opus 4.8 76.2% max May 2026 Official leaderboard
9 Muse Spark 1.2 75.9% xhigh Aug 2026 Official leaderboard
10 DeepSeek-V4-Flash-Vision-Exp 75.9% Aug 2026 Lab model card
11 Muse Spark 1.1 75.6% xhigh Jul 2026 Official leaderboard
12 DeepSeek V4 Pro 0813 74.4% max Aug 2026 Official leaderboard
13 Tencent Hy4 preview 74.1% Aug 2026 Lab model card
14 Step 5 Preview 74.1% Sep 2026 Lab model card
15 GPT-5.5 73.5% xhigh Apr 2026 Official leaderboard
16 Qwen3.8 Flash 73.5% Aug 2026 Lab model card
17 Qwen3.8 Max (0902) 73.3% Sep 2026 Lab model card
18 GLM-5.3 73% Aug 2026 Lab model card
19 Muse Spark Open 72.8% Oct 2026 Lab model card
20 Qwen 3.8 Max 72.5% Aug 2026 Lab model card
21 Claude Sonnet 5 71.6% max Jun 2026 Official leaderboard
22 DeepSeek V4 Flash 0731 70.7% max Jul 2026 Official leaderboard
23 Gemini 3.5 Flash 67.3% high May 2026 Official leaderboard
24 Gemini 3.1 Pro 61.1% high Feb 2026 Official leaderboard
25 GLM-5.2 59.9% max Jun 2026 Official leaderboard
26 Kimi K2.6 58% Apr 2026 Official leaderboard
27 Kimi K2.7 Code 58% Jun 2026 Official leaderboard
28 Gemini 3.5 Flash-Lite 57.1% high Jul 2026 Official leaderboard
29 Tencent Hy3 56.8% high Jul 2026 Official leaderboard
30 DeepSeek-V4-Pro 55.9% max Apr 2026 Official leaderboard
31 Inkling-Small 54.4% xhigh Jul 2026 Official leaderboard
32 DeepSeek-V4-Flash 50.9% max Apr 2026 Official leaderboard
33 Laguna S 2.1 49.7% Jul 2026 Lab model card
34 MiMo-V2.5 49.1% Apr 2026 Official leaderboard
35 MiniMax-M2.7 47.5% Mar 2026 Official leaderboard
36 Inkling 45.5% xhigh Jul 2026 Official leaderboard
37 Qwen3.5 397B-A17B 40.7% Feb 2026 Official leaderboard
38 Nemotron 3 Ultra 34.3% Jun 2026 Official leaderboard
39 Kimi K2.5 33% Jan 2026 Official leaderboard

Labs report scores for 21 models in their cards. Switch the scores above to rank them together.

3 models have held number one on the official leaderboard since its release in June 2026 3 models have held number one with claims included since its release in June 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Anthropic Moonshot AI Z.ai Meta
Own leaderboard Not tested since Aug 2026
GLM-5.3-Flash 78.4%
Lab claims 3 leaders
Claude Opus 5 80.6%

Sources: Official Toolathlon-Verified leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 5 Claude Opus 4.8 Gemini 3.1 Pro Qwen3.5 397B-A17B Kimi K2.5 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 5 GPT-5.5

Sources: Official Toolathlon-Verified leaderboard and labs’ model cards

10 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Meta
Thinking Machines
Moonshot AI
Poolside
DeepSeek
Qwen
Z.ai
Tencent
StepFun
Xiaomi Corp
DateLaunch postScoreAs the post writes it
Oct 15, 2026 Muse Spark Open 72.8 Toolathlon Verified
Sep 22, 2026 MiMo-V2.6 Pro 76.9 Toolathlon-Verified
Sep 20, 2026 Step 5 Preview 74.1 Toolathlon-Verified
Sep 2, 2026 Qwen3.8 Max (0902) 73.3 Real-world tool use, Toolathlon Verified (Pass@1)
Aug 28, 2026 Tencent Hy4 preview 74.1 Toolathlon-Verified
Aug 26, 2026 Qwen3.8 Flash 73.5 Real-world tool use Toolathlon Verified (Pass@1)
Aug 26, 2026 GLM-5.3-Flash 78.4 Toolathlon Verified
Aug 21, 2026 DeepSeek-V4-Flash-Vision-Exp 75.9 Toolathlon-Verified
Aug 14, 2026 GLM-5.3 73 Toolathlon Verified
Aug 13, 2026 DeepSeek V4 Pro 0813 74.1 Toolathlon-Verified
Aug 3, 2026 Qwen 3.8 Max 72.5 Toolathlon Verified (Pass@1)
Jul 31, 2026 DeepSeek V4 Flash 0731 70.3 Toolathlon-Verified
Jul 30, 2026 Inkling-Small 54.4 Toolathlon Verified
Jul 21, 2026 Laguna S 2.1 49.7 Toolathlon Verified
Jul 16, 2026 Kimi K3 76.5 Toolathlon-Verified
Jul 15, 2026 Inkling 45.5 Toolathlon Verified
Jul 9, 2026 Muse Spark 1.1 75.6 Toolathlon-Verified (Personal tool use)

9 system cards report it, from 5 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Sep 28, 2026 Claude Sonnet 5.5 system card 77.8 Toolathlon-Verified Pass@1 over three trials (Pass@3 85.2, Pass^3 68.5) (Table 8.14.5.A) Capabilities
Sep 22, 2026 MiMo-V2.6 Pro technical report 76.9 Toolathlon-Verified (Table 3) Capabilities
Sep 22, 2026 Claude Opus 5.5 system card 77.8 Toolathlon-Verified Pass@1 over three trials, max effort, production classifiers on (Pass@3 82.4, Pass^3 72.2) (Table 8.14.5.A) Capabilities
Sep 1, 2026 Claude Fable 5.1 system card 77.8 Toolathlon Verified Pass@1 over 3 trials, max effort, production classifiers with Opus 4.8 fallback; Pass@3 81.5, Pass^3 73.1 (Table 8.15.5.A) Capabilities
Jul 30, 2026 Inkling-Small model card 54.4 Toolathlon Verified (5. Evaluations table) Capabilities
Jul 27, 2026 Kimi K3 technical report 76.5 Toolathlon-Verified, cited from the official leaderboard as of 2026-07-24 (Table 2) Capabilities
Jul 24, 2026 Claude Opus 5 system card 80.6 Toolathlon-Verified Pass@1 over three trials, max effort, internal harness, no safety classifiers (Pass@3 87.0, Pass^3 73.1) (Table 8.13.6.A) Capabilities
Jul 15, 2026 Inkling model card 45.5 Toolathlon Verified (5. Evaluations table, effort 0.99) Capabilities
Jul 9, 2026 Muse Spark 1.1 system card 75.6 Toolathlon-Verified (Figure 44) Capabilities

The lab's number comes out ahead in 0 of 2 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 5 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
DeepSeek V4 Flash 0731 70.3% Toolathlon-Verified 70.7% Leaderboard, max −0.4 pts
DeepSeek V4 Pro 0813 74.1% Toolathlon-Verified 74.4% Leaderboard, max −0.3 pts
Muse Spark 1.1 75.6% Toolathlon-Verified (Personal tool use) 75.6% Leaderboard, xhigh Same
Inkling 45.5% Toolathlon Verified 45.5% Leaderboard, xhigh Same
Kimi K3 76.5% Toolathlon-Verified 76.5% Leaderboard, max Same
Inkling-Small 54.4% Toolathlon Verified 54.4% Leaderboard, xhigh Same
GLM-5.3-Flash 78.4% Toolathlon Verified 78.4% Leaderboard, max Same

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

Anthropic OpenAI Google DeepMind Moonshot AI Z.ai
Toolathlon-Verified Not tested since Aug 2026
GLM-5.3-Flash 78.4%
Toolathlon Replaced by Toolathlon-Verified, Jun 2026
Gemini 3.5 Flash 56.5%
Toolathlon-Verified Not tested since Aug 2026
GLM-5.3-Flash 78.4%

Source: Official leaderboards

Replaces Toolathlon

More in Tool use

All 144 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%
  • Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%