Benchmark Almanac

AutomationBench

Gemini 4 Carbon 53.9% leads since October 2026
Kimi K3.1 Pro 73.3% leads since October 2026
In use 1 known issue

Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state.

Gemini cards report the private set from Zapier's leaderboard. Zapier ships 1.0.x patch releases, such as 1.0.6 on July 31st, 2026, that fix task fairness and make private tasks harder. It then reruns all models, so the board stays one record.

Made by
Zapier
Released
April 2026
Kind
Apps and MCP servers
Level
University
Scored
Checked automatically
Answers
Agent task
Tasks
Public, plus a private test set
Strongest model tested
Claude Fable 5.2 #1 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 13th, 2026

In the authors’ words

Existing AI benchmarks for software automation rarely combine cross-application coordination, autonomous API discovery, and policy adherence. Real business workflows demand all three: a single task may span a CRM, inbox, calendar, and messaging platform - requiring the agent to find the right endpoints, follow a policy document, and write correct data to each system. To address this gap, we introduce AutomationBench, a benchmark for evaluating AI agents on cross-application workflow orchestration via REST APIs. Drawing on real workflow patterns from Zapier's platform, tasks span Sales, Marketing, Operations, Support, Finance, and HR domains. Agents must discover relevant endpoints themselves, follow layered business rules, and navigate environments with irrelevant and sometimes misleading records. Grading is programmatic and end-state only: whether the correct data ended up in the right systems. Even the best frontier models currently score below 10%. AutomationBench provides a challenging, realistic measure of where current models stand relative to the agentic capabilities businesses actually need.

Abstract of AutomationBench (arXiv:2604.18934, CC0)

Known issues

A published audit found a problem in AutomationBench. We summarize each finding in our words and link its source. The scores on this page don't include any fixes. See all audits

  1. Unclear tasks Broken grader Fixed in a copy

    Prime Intellect reports that AutomationBench describes API request bodies in prose instead of JSON Schema, so models had to guess parameters. It also reports graders that enforced requirements the tasks did not state.

    After the fixes, Claude Opus 5.5 at max effort scores 92.15% (70.83% strict) and DeepSeek V4.1-Flash 1017 scores 91.61% (69.50% strict) on the public set.

    the actions the models can discover with the search tool describe their request bodies in prose rather than in JSON Schema

    Learnings from Auditing Public Evaluations by Prime Intellect, November 3rd, 2026

43
Entries on its own leaderboard, 42 models we track
47
Launch posts list it, from 20 labs
16
System cards report it, from 6 labs

Gemini 4 Carbon leads with 53.9%

Best score per entry from the benchmark's own leaderboard, read on November 13th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

Kimi K3.1 Pro leads with 73.3%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

43 entries. Scroll sideways for the rest.

  1. 53.9 Gemini 4 Carbon
  2. 51.5 Claude Fable 5.2
  3. 51.3 Gemini 4 Argon
  4. 51.1 Gemini 4 Hydrogen
  5. 44.8 Claude Sonnet 5.5
  6. 42.5 Claude Opus 5.5
  7. 41.4 GPT-6 Astra
  8. 33.2 GPT-6 Sol
  9. 31.4 Claude Fable 5.1 (with Opus 5 Fallback)
  10. 30.4 Gemini 3.7 Flash
  11. 29.7 Gemini 3.8 Flash
  12. 28.8 GPT-5.6 Sol
  13. 26.9 Claude Opus 5
  14. 23.6 GPT-5.6 Terra
  15. 22.7 Kimi K3
  16. 22.4 Claude Fable 5.1
  17. 20.7 Muse Spark 1.3
  18. 20.7 GPT-6 Luna
  19. 18.1 DeepSeek-V4-Flash
  20. 17.1 Claude Fable 5
  21. 17.1 GPT-5.6 Luna
  22. 17.1 Gemini 3.6 Flash
  23. 16.9 Claude Opus 4.8
  24. 16.6 GPT-5.5
  25. 13.4 Claude Opus 4.7
  26. 13.4 Gemini 3.5 Flash
  27. 12.0 Claude Haiku 5.5
  28. 11.6 GPT-5.4
  29. 10.7 Claude Sonnet 5
  30. 8.68 Gemini 3.1 Pro
  31. 7.76 GLM-5.2
  32. 6.20 GLM-5.1
  33. 5.33 Claude Sonnet 4.6
  34. 4.87 MiniMax-M3
  35. 4.30 Qwen 3.6 Plus
  36. 3.96 Gemini 3.5 Flash-Lite
  37. 3.70 Qwen3.7-Plus
  38. 3.20 Kimi K2.7 Code
  39. 2.59 Kimi K2.6
  40. 1.70 Gemma 4 31B IT
  41. 0.46 Claude Haiku 4.5
  42. 0.46 MiniMax-M2.7
  43. 0.15 gpt-oss-120b
  1. 1 Gemini 4 Carbon high 53.9%
  2. 2 Claude Fable 5.2 max 51.5%
  3. 3 Gemini 4 Argon high 51.3%
  4. 4 Gemini 4 Hydrogen high 51.1%
  5. 5 Claude Sonnet 5.5 max 44.8%
  6. 6 Claude Opus 5.5 max 42.5%
  7. 7 GPT-6 Astra max 41.4%
  8. 8 GPT-6 Sol xhigh 33.2%
  9. 9 Claude Fable 5.1 (with Opus 5 Fallback) 31.4%
  10. 10 Gemini 3.7 Flash high 30.4%
  11. 11 Gemini 3.8 Flash medium 29.7%
  12. 12 GPT-5.6 Sol max 28.8%
  13. 13 Claude Opus 5 max 26.9%
  14. 14 GPT-5.6 Terra max 23.6%
  15. 15 Kimi K3 22.7%
  16. 16 Claude Fable 5.1 max 22.4%
  17. 17 Muse Spark 1.3 max 20.7%
  18. 18 GPT-6 Luna max 20.7%
  19. 19 DeepSeek-V4-Flash max 18.1%
  20. 20 Claude Fable 5 max 17.1%
  21. 21 GPT-5.6 Luna max 17.1%
  22. 22 Gemini 3.6 Flash high 17.1%
  23. 23 Claude Opus 4.8 xhigh 16.9%
  24. 24 GPT-5.5 xhigh 16.6%
  25. 25 Claude Opus 4.7 max 13.4%
  26. 26 Gemini 3.5 Flash high 13.4%
  27. 27 Claude Haiku 5.5 max 12%
  28. 28 GPT-5.4 xhigh 11.6%
  29. 29 Claude Sonnet 5 max 10.7%
  30. 30 Gemini 3.1 Pro high 8.68%
  31. 31 GLM-5.2 max 7.76%
  32. 32 GLM-5.1 max 6.20%
  33. 33 Claude Sonnet 4.6 max 5.33%
  34. 34 MiniMax-M3 max 4.87%
  35. 35 Qwen 3.6 Plus high 4.30%
  36. 36 Gemini 3.5 Flash-Lite medium 3.96%
  37. 37 Qwen3.7-Plus 3.70%
  38. 38 Kimi K2.7 Code 3.20%
  39. 39 Kimi K2.6 2.59%
  40. 40 Gemma 4 31B IT max 1.70%
  41. 41 Claude Haiku 4.5 0.46%
  42. 42 MiniMax-M2.7 high 0.46%
  43. 43 gpt-oss-120b high 0.15%

Source: Official AutomationBench leaderboard

70 models. Scroll sideways for the rest.

  1. 73.3 Kimi K3.1 Pro
  2. 62.8 MiniMax-M3.1-Flash
  3. 60.5 DeepSeek V4.1 Flash 1017
  4. 59.9 Mistral Large 4
  5. 56.2 Tencent Hy4
  6. 54.8 DeepSeek V4.1 Flash
  7. 53.9 Gemini 4 Carbon
  8. 53.1 MiMo-V2.6 Pro
  9. 52.5 Ling-3.1-flash
  10. 51.8 LongCat-2.5
  11. 51.5 Claude Fable 5.2
  12. 51.3 Gemini 4 Argon
  13. 51.1 Gemini 4 Hydrogen
  14. 50.8 Qwen3.8 Max (0902)
  15. 49.6 Muse Spark 1.3
  16. 48.8 GLM-5.3-Flash
  17. 48.2 GLM-5.3
  18. 45.6 Inkling 2 Small
  19. 45.4 Muse Wonder Full
  20. 44.8 Claude Sonnet 5.5
  21. 44.0 Step 5 Preview
  22. 42.5 Claude Opus 5.5
  23. 41.4 GPT-6 Astra
  24. 41.3 LFM3-1.7B
  25. 39.1 Muse Wonder
  26. 37.8 Muse Ember
  27. 37.0 Beam 1.0
  28. 36.1 GPT-6.1 Sol
  29. 33.2 GPT-6 Sol
  30. 32.4 Nemotron 3.5 Super VL
  31. 32.1 Tencent Hy4 preview
  32. 31.8 DeepSeek V4 Pro 0813
  33. 31.4 Claude Fable 5.1
  34. 30.8 Kimi K3
  35. 30.4 Gemini 3.7 Flash
  36. 29.7 Gemini 3.8 Flash
  37. 28.8 GPT-5.6 Sol
  38. 27.7 LFM2.5-24B-A2B
  39. 27.3 Qwen 3.8 Max
  40. 26.9 Claude Opus 5
  41. 25.7 DeepSeek-V4-Flash-Vision-Exp
  42. 25.1 DeepSeek V4 Flash 0731
  43. 23.6 GPT-5.6 Terra
  44. 22.3 Solar Mini 4
  45. 20.7 GPT-6 Luna
  46. 18.1 DeepSeek-V4-Flash
  47. 17.4 Claude Fable 5
  48. 17.1 GPT-5.6 Luna
  49. 17.1 Gemini 3.6 Flash
  50. 16.9 Claude Opus 4.8
  51. 16.6 GPT-5.5
  52. 13.5 Claude Sonnet 5
  53. 13.4 Claude Opus 4.7
  54. 13.4 Gemini 3.5 Flash
  55. 12.0 Claude Haiku 5.5
  56. 11.6 GPT-5.4
  57. 8.68 Gemini 3.1 Pro
  58. 7.76 GLM-5.2
  59. 6.20 GLM-5.1
  60. 5.33 Claude Sonnet 4.6
  61. 4.87 MiniMax-M3
  62. 4.30 Qwen 3.6 Plus
  63. 3.96 Gemini 3.5 Flash-Lite
  64. 3.70 Qwen3.7-Plus
  65. 3.20 Kimi K2.7 Code
  66. 2.59 Kimi K2.6
  67. 1.70 Gemma 4 31B IT
  68. 0.46 Claude Haiku 4.5
  69. 0.46 MiniMax-M2.7
  70. 0.15 gpt-oss-120b
  1. 1 Kimi K3.1 Pro 73.3%
  2. 2 MiniMax-M3.1-Flash 62.8%
  3. 3 DeepSeek V4.1 Flash 1017 60.5%
  4. 4 Mistral Large 4 59.9%
  5. 5 Tencent Hy4 56.2%
  6. 6 DeepSeek V4.1 Flash 54.8%
  7. 7 Gemini 4 Carbon 53.9%
  8. 8 MiMo-V2.6 Pro 53.1%
  9. 9 Ling-3.1-flash 52.5%
  10. 10 LongCat-2.5 51.8%
  11. 11 Claude Fable 5.2 max 51.5%
  12. 12 Gemini 4 Argon 51.3%
  13. 13 Gemini 4 Hydrogen high 51.1%
  14. 14 Qwen3.8 Max (0902) 50.8%
  15. 15 Muse Spark 1.3 49.6%
  16. 16 GLM-5.3-Flash 48.8%
  17. 17 GLM-5.3 48.2%
  18. 18 Inkling 2 Small 45.6%
  19. 19 Muse Wonder Full 45.4%
  20. 20 Claude Sonnet 5.5 max 44.8%
  21. 21 Step 5 Preview 44%
  22. 22 Claude Opus 5.5 max 42.5%
  23. 23 GPT-6 Astra max 41.4%
  24. 24 LFM3-1.7B 41.3%
  25. 25 Muse Wonder 39.1%
  26. 26 Muse Ember 37.8%
  27. 27 Beam 1.0 37%
  28. 28 GPT-6.1 Sol 36.1%
  29. 29 GPT-6 Sol xhigh 33.2%
  30. 30 Nemotron 3.5 Super VL 32.4%
  31. 31 Tencent Hy4 preview 32.1%
  32. 32 DeepSeek V4 Pro 0813 31.8%
  33. 33 Claude Fable 5.1 31.4%
  34. 34 Kimi K3 30.8%
  35. 35 Gemini 3.7 Flash high 30.4%
  36. 36 Gemini 3.8 Flash medium 29.7%
  37. 37 GPT-5.6 Sol max 28.8%
  38. 38 LFM2.5-24B-A2B 27.7%
  39. 39 Qwen 3.8 Max 27.3%
  40. 40 Claude Opus 5 max 26.9%
  41. 41 DeepSeek-V4-Flash-Vision-Exp 25.7%
  42. 42 DeepSeek V4 Flash 0731 25.1%
  43. 43 GPT-5.6 Terra max 23.6%
  44. 44 Solar Mini 4 22.3%
  45. 45 GPT-6 Luna max 20.7%
  46. 46 DeepSeek-V4-Flash max 18.1%
  47. 47 Claude Fable 5 17.4%
  48. 48 GPT-5.6 Luna max 17.1%
  49. 49 Gemini 3.6 Flash high 17.1%
  50. 50 Claude Opus 4.8 xhigh 16.9%
  51. 51 GPT-5.5 xhigh 16.6%
  52. 52 Claude Sonnet 5 13.5%
  53. 53 Claude Opus 4.7 max 13.4%
  54. 54 Gemini 3.5 Flash high 13.4%
  55. 55 Claude Haiku 5.5 max 12%
  56. 56 GPT-5.4 xhigh 11.6%
  57. 57 Gemini 3.1 Pro high 8.68%
  58. 58 GLM-5.2 max 7.76%
  59. 59 GLM-5.1 max 6.20%
  60. 60 Claude Sonnet 4.6 max 5.33%
  61. 61 MiniMax-M3 max 4.87%
  62. 62 Qwen 3.6 Plus high 4.30%
  63. 63 Gemini 3.5 Flash-Lite medium 3.96%
  64. 64 Qwen3.7-Plus 3.70%
  65. 65 Kimi K2.7 Code 3.20%
  66. 66 Kimi K2.6 2.59%
  67. 67 Gemma 4 31B IT max 1.70%
  68. 68 Claude Haiku 4.5 0.46%
  69. 69 MiniMax-M2.7 high 0.46%
  70. 70 gpt-oss-120b high 0.15%

Sources: Labs’ model cards and official AutomationBench leaderboard

#EntryScoreSettingDateSource
1 Gemini 4 Carbon 53.9% high Oct 2026 Official leaderboard
2 Claude Fable 5.2 51.5% max, default fallbacks Nov 2026 Official leaderboard
3 Gemini 4 Argon 51.3% high Sep 2026 Official leaderboard
4 Gemini 4 Hydrogen 51.1% high Nov 2026 Official leaderboard
5 Claude Sonnet 5.5 44.8% max, default fallbacks Sep 2026 Official leaderboard
6 Claude Opus 5.5 42.5% max, default fallbacks Sep 2026 Official leaderboard
7 GPT-6 Astra 41.4% max Sep 2026 Official leaderboard
8 GPT-6 Sol 33.2% xhigh Sep 2026 Official leaderboard
9 Claude Fable 5.1 (with Opus 5 Fallback) 31.4% Official leaderboard
10 Gemini 3.7 Flash 30.4% high Aug 2026 Official leaderboard
11 Gemini 3.8 Flash 29.7% medium Sep 2026 Official leaderboard
12 GPT-5.6 Sol 28.8% max Jul 2026 Official leaderboard
13 Claude Opus 5 26.9% max Jul 2026 Official leaderboard
14 GPT-5.6 Terra 23.6% max Jul 2026 Official leaderboard
15 Kimi K3 22.7% Jul 2026 Official leaderboard
16 Claude Fable 5.1 22.4% max Sep 2026 Official leaderboard
17 Muse Spark 1.3 20.7% max Sep 2026 Official leaderboard
18 GPT-6 Luna 20.7% max Sep 2026 Official leaderboard
19 DeepSeek-V4-Flash 18.1% max Apr 2026 Official leaderboard
20 Claude Fable 5 17.1% max Jun 2026 Official leaderboard
21 GPT-5.6 Luna 17.1% max Jul 2026 Official leaderboard
22 Gemini 3.6 Flash 17.1% high Jul 2026 Official leaderboard
23 Claude Opus 4.8 16.9% xhigh May 2026 Official leaderboard
24 GPT-5.5 16.6% xhigh Apr 2026 Official leaderboard
25 Claude Opus 4.7 13.4% max Apr 2026 Official leaderboard
26 Gemini 3.5 Flash 13.4% high May 2026 Official leaderboard
27 Claude Haiku 5.5 12% max, default fallbacks Oct 2026 Official leaderboard
28 GPT-5.4 11.6% xhigh Mar 2026 Official leaderboard
29 Claude Sonnet 5 10.7% max Jun 2026 Official leaderboard
30 Gemini 3.1 Pro 8.68% high Feb 2026 Official leaderboard
31 GLM-5.2 7.76% max Jun 2026 Official leaderboard
32 GLM-5.1 6.20% max Apr 2026 Official leaderboard
33 Claude Sonnet 4.6 5.33% max Feb 2026 Official leaderboard
34 MiniMax-M3 4.87% max Jun 2026 Official leaderboard
35 Qwen 3.6 Plus 4.30% high Mar 2026 Official leaderboard
36 Gemini 3.5 Flash-Lite 3.96% medium Jul 2026 Official leaderboard
37 Qwen3.7-Plus 3.70% Jun 2026 Official leaderboard
38 Kimi K2.7 Code 3.20% Jun 2026 Official leaderboard
39 Kimi K2.6 2.59% Apr 2026 Official leaderboard
40 Gemma 4 31B IT 1.70% max Apr 2026 Official leaderboard
41 Claude Haiku 4.5 0.46% Oct 2025 Official leaderboard
42 MiniMax-M2.7 0.46% high Mar 2026 Official leaderboard
43 gpt-oss-120b 0.15% high Aug 2025 Official leaderboard
#ModelScoreSettingDateSource
1 Kimi K3.1 Pro 73.3% Oct 2026 Lab model card
2 MiniMax-M3.1-Flash 62.8% Oct 2026 Lab model card
3 DeepSeek V4.1 Flash 1017 60.5% Oct 2026 Lab model card
4 Mistral Large 4 59.9% Oct 2026 Lab model card
5 Tencent Hy4 56.2% Nov 2026 Lab model card
6 DeepSeek V4.1 Flash 54.8% Sep 2026 Lab model card
7 Gemini 4 Carbon 53.9% Oct 2026 Lab model card
8 MiMo-V2.6 Pro 53.1% Sep 2026 Lab model card
9 Ling-3.1-flash 52.5% Oct 2026 Lab model card
10 LongCat-2.5 51.8% Nov 2026 Lab model card
11 Claude Fable 5.2 51.5% max, default fallbacks Nov 2026 Official leaderboard
12 Gemini 4 Argon 51.3% Oct 2026 Lab model card
13 Gemini 4 Hydrogen 51.1% high Nov 2026 Official leaderboard
14 Qwen3.8 Max (0902) 50.8% Sep 2026 Lab model card
15 Muse Spark 1.3 49.6% Sep 2026 Lab model card
16 GLM-5.3-Flash 48.8% Aug 2026 Lab model card
17 GLM-5.3 48.2% Aug 2026 Lab model card
18 Inkling 2 Small 45.6% Nov 2026 Lab model card
19 Muse Wonder Full 45.4% Oct 2026 Lab model card
20 Claude Sonnet 5.5 44.8% max, default fallbacks Sep 2026 Official leaderboard
21 Step 5 Preview 44% Sep 2026 Lab model card
22 Claude Opus 5.5 42.5% max, default fallbacks Sep 2026 Official leaderboard
23 GPT-6 Astra 41.4% max Sep 2026 Official leaderboard
24 LFM3-1.7B 41.3% Oct 2026 Lab model card
25 Muse Wonder 39.1% Oct 2026 Lab model card
26 Muse Ember 37.8% Nov 2026 Lab model card
27 Beam 1.0 37% Oct 2026 Lab model card
28 GPT-6.1 Sol 36.1% Sep 2026 Lab model card
29 GPT-6 Sol 33.2% xhigh Sep 2026 Official leaderboard
30 Nemotron 3.5 Super VL 32.4% Oct 2026 Lab model card
31 Tencent Hy4 preview 32.1% Aug 2026 Lab model card
32 DeepSeek V4 Pro 0813 31.8% Aug 2026 Lab model card
33 Claude Fable 5.1 31.4% Sep 2026 Lab model card
34 Kimi K3 30.8% Jul 2026 Lab model card
35 Gemini 3.7 Flash 30.4% high Aug 2026 Official leaderboard
36 Gemini 3.8 Flash 29.7% medium Sep 2026 Official leaderboard
37 GPT-5.6 Sol 28.8% max Jul 2026 Official leaderboard
38 LFM2.5-24B-A2B 27.7% Oct 2026 Lab model card
39 Qwen 3.8 Max 27.3% Aug 2026 Lab model card
40 Claude Opus 5 26.9% max Jul 2026 Official leaderboard
41 DeepSeek-V4-Flash-Vision-Exp 25.7% Aug 2026 Lab model card
42 DeepSeek V4 Flash 0731 25.1% Jul 2026 Lab model card
43 GPT-5.6 Terra 23.6% max Jul 2026 Official leaderboard
44 Solar Mini 4 22.3% Oct 2026 Lab model card
45 GPT-6 Luna 20.7% max Sep 2026 Official leaderboard
46 DeepSeek-V4-Flash 18.1% max Apr 2026 Official leaderboard
47 Claude Fable 5 17.4% Jun 2026 Lab model card
48 GPT-5.6 Luna 17.1% max Jul 2026 Official leaderboard
49 Gemini 3.6 Flash 17.1% high Jul 2026 Official leaderboard
50 Claude Opus 4.8 16.9% xhigh May 2026 Official leaderboard
51 GPT-5.5 16.6% xhigh Apr 2026 Official leaderboard
52 Claude Sonnet 5 13.5% Jun 2026 Lab model card
53 Claude Opus 4.7 13.4% max Apr 2026 Official leaderboard
54 Gemini 3.5 Flash 13.4% high May 2026 Official leaderboard
55 Claude Haiku 5.5 12% max, default fallbacks Oct 2026 Official leaderboard
56 GPT-5.4 11.6% xhigh Mar 2026 Official leaderboard
57 Gemini 3.1 Pro 8.68% high Feb 2026 Official leaderboard
58 GLM-5.2 7.76% max Jun 2026 Official leaderboard
59 GLM-5.1 6.20% max Apr 2026 Official leaderboard
60 Claude Sonnet 4.6 5.33% max Feb 2026 Official leaderboard
61 MiniMax-M3 4.87% max Jun 2026 Official leaderboard
62 Qwen 3.6 Plus 4.30% high Mar 2026 Official leaderboard
63 Gemini 3.5 Flash-Lite 3.96% medium Jul 2026 Official leaderboard
64 Qwen3.7-Plus 3.70% Jun 2026 Official leaderboard
65 Kimi K2.7 Code 3.20% Jun 2026 Official leaderboard
66 Kimi K2.6 2.59% Apr 2026 Official leaderboard
67 Gemma 4 31B IT 1.70% max Apr 2026 Official leaderboard
68 Claude Haiku 4.5 0.46% Oct 2025 Official leaderboard
69 MiniMax-M2.7 0.46% high Mar 2026 Official leaderboard
70 gpt-oss-120b 0.15% high Aug 2025 Official leaderboard

Labs report scores for 47 models in their cards. Switch the scores above to rank them together.

The best score, 53.9%, costs $2.12 a task

Runs on the cost frontier score higher than any run that's cheaper. Gemini 4 Hydrogen at high comes within 2.7 points of the best for $0.82 a task, 3 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI Google DeepMind Cost frontier One model at each effort level
0% 20% 40% 60% $0.01 $0.10 $1.00 $10 Cost per task, log scale Gemini 4 Carbon, high Gemini 4 Hydrogen, high Gemini 4 Hydrogen, medium GPT-6 Sol, xhigh Gemini 4 Hydrogen, low GPT-6 Luna, max GPT-6 Luna, high 0% 20% 40% 60% $0.01 $0.10 $1.00 $10 Cost per task, log scale Gemini 4 Carbon, high Gemini 4 Hydrogen, medium

Source: Official AutomationBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Gemini 4 Carbon high 53.9% $2.12
Gemini 4 Carbon medium 53% $1.65
Claude Fable 5.2 max default fallbacks 51.5% $4.64
Gemini 4 Argon high 51.3% $1.70
Gemini 4 Hydrogen high 51.1% $0.82
Gemini 4 Argon medium 50.1% $1.54
Claude Fable 5.2 xhigh default fallbacks 48% $2.29
Gemini 4 Hydrogen medium 46.7% $0.41
Claude Sonnet 5.5 max default fallbacks 44.8% $1.14
Claude Fable 5.2 high default fallbacks 44.3% $1.69
Claude Opus 5.5 max default fallbacks 42.5% $1.44
GPT-6 Astra max 41.4% $1.73
Claude Fable 5.2 medium default fallbacks 40.2% $1.35
Claude Fable 5.2 low default fallbacks 39.7% $1.30
GPT-6 Astra xhigh 39% $1.50
GPT-6 Astra high 37.1% $1.44
Claude Sonnet 5.5 xhigh default fallbacks 36.8% $0.48
Claude Opus 5.5 xhigh default fallbacks 35.8% $0.89
GPT-6 Astra medium 34.1% $1.27
GPT-6 Sol xhigh 33.2% $0.27
Claude Opus 5.5 high default fallbacks 33% $0.71
GPT-6 Sol max 32% $0.34
Gemini 4 Carbon low 31.5% $1.02
GPT-6 Sol high 31.2% $0.24
Gemini 4 Hydrogen low 30.6% $0.18
Gemini 3.7 Flash high 30.4% $0.61
GPT-6 Astra low 30.3% $1.08
Gemini 3.8 Flash medium 29.7% $0.55
Gemini 3.8 Flash high 29.7% $0.62
Claude Opus 5.5 medium default fallbacks 29.5% $0.65
Claude Sonnet 5.5 high default fallbacks 29.4% $0.36
GPT-5.6 Sol max 28.8% $0.67
Claude Opus 5 max 26.9% $3.05
GPT-6 Sol medium 26.9% $0.21
GPT-5.6 Sol xhigh 26.3% $0.74
Claude Opus 5 xhigh 25.3% $1.15
GPT-6 Astra none 25.3% $1.28
GPT-5.6 Sol high 24.8% $0.64
Claude Opus 5.5 low default fallbacks 24.2% $0.51
Claude Opus 5 medium 23.9% $0.89
GPT-5.6 Terra max 23.6% $0.51
Claude Sonnet 5.5 medium default fallbacks 23.3% $0.31
Kimi K3 22.7% $0.43
Claude Fable 5.1 max 22.4% $2.45
Claude Fable 5.1 xhigh 21.3% $2.14
GPT-6 Sol low 21.2% $0.19
Muse Spark 1.3 max 20.7% $1.39
GPT-6 Luna max 20.7% $0.04
Claude Opus 5 high 20.6% $1.03
Claude Opus 5 low 20.4% $0.75
GPT-5.6 Sol medium 19.6% $0.57
Claude Sonnet 5.5 between tools default fallbacks 18.6% $0.30
Claude Sonnet 5.5 low default fallbacks 18.3% $0.28
DeepSeek-V4-Flash max 18.1% $0.04
Claude Fable 5 max 17.1% $2.03
GPT-5.6 Luna max 17.1% $0.07
Gemini 3.6 Flash high 17.1% $0.58
Claude Opus 4.8 xhigh 16.9% $1.91
GPT-5.5 xhigh 16.6% $1.72
Claude Opus 4.8 max 16.4% $2.36
Gemini 3.6 Flash medium 16.4% $0.55
Claude Opus 4.8 high 16.1% $1.48
Claude Fable 5 high 16.1% $1.65
Gemini 3.6 Flash low 15.7% $0.48
Claude Fable 5 xhigh 15.2% $1.80
GPT-5.6 Terra xhigh 14.8% $0.37
GPT-6 Luna high 14.5% $0.02
Claude Opus 4.8 medium 14% $1.44
Claude Fable 5 medium 13.7% $1.52
Claude Opus 4.8 low 13.6% $1.20
Claude Opus 4.7 max 13.4% $1.52
Gemini 3.5 Flash high 13.4% $0.42
GPT-5.6 Luna xhigh 12.9% $0.06
GPT-5.5 high 12.8% $0.94
GPT-6 Luna xhigh 12.6% $0.02
Claude Fable 5 low 12.5% $1.41
Claude Haiku 5.5 max default fallbacks 12% $0.10
GPT-5.6 Sol low 11.7% $0.42
GPT-5.6 Terra high 11.7% $0.29
GPT-5.4 xhigh 11.6% $1.15
Gemini 3.5 Flash medium 11.4% $0.38
Claude Sonnet 5 max 10.7% $0.88
Claude Haiku 5.5 high default fallbacks 10.5% $0.03
GPT-5.4 high 10.1% $0.62
Claude Sonnet 5 high 9.89% $0.56
Claude Haiku 5.5 xhigh default fallbacks 9.89% $0.04
Claude Opus 4.7 xhigh 9.74% $1.26
GPT-6 Luna medium 9.44% $0.02
Claude Opus 4.7 high 9.13% $1.08
GPT-5.6 Luna high 9.13% $0.05
GPT-6 Sol none 9.10% $0.25
GPT-5.5 medium 8.98% $0.75
Gemini 3.5 Flash low 8.98% $0.30
Gemini 3.1 Pro high 8.68% $0.30
Claude Haiku 5.5 medium default fallbacks 8.52% $0.02
GPT-5.6 Terra medium 8.07% $0.22
GLM-5.2 max 7.76% $0.22
Claude Sonnet 5 xhigh 7.61% $0.67
GPT-5.6 Terra low 7.46% $0.19
Gemini 3.1 Pro medium 7.00% $0.25
Claude Opus 4.7 medium 6.85% $0.92
GLM-5.1 max 6.20% $0.23
Claude Haiku 5.5 low default fallbacks 6.09% $0.02
Claude Sonnet 5 medium 5.78% $0.41
Claude Opus 4.7 low 5.48% $0.78
Claude Sonnet 4.6 max 5.33% $0.90
GPT-5.4 medium 5.33% $0.36
Claude Sonnet 4.6 high 5.02% $0.84
MiniMax-M3 max 4.87% $0.07
Gemini 3.1 Pro low 4.57% $0.28
Qwen 3.6 Plus high 4.30% $0.08
GPT-5.6 Luna medium 4.26% $0.02
Gemini 3.5 Flash-Lite medium 3.96% $0.12
GPT-5.6 Sol 3.81% $0.41
Qwen3.7-Plus 3.70% $0.05
Claude Sonnet 4.6 medium 3.65% $0.77
Claude Sonnet 5 low 3.65% $0.26
Gemini 3.5 Flash-Lite high 3.35% $0.13
Kimi K2.7 Code 3.20% $0.24
Claude Sonnet 4.6 low 2.89% $0.61
Kimi K2.6 2.59% $0.28
GPT-5.5 low 2.59% $0.36
GPT-5.6 Luna low 1.83% $0.01
Gemma 4 31B IT max 1.70% $0.09
GPT-5.4 low 1.52% $0.13
Gemini 3.5 Flash minimal 1.22% $0.18
GPT-6 Luna low 1.22% $0.01
GPT-5.5 1.07% $0.20
GPT-5.6 Terra 0.76% $0.12
Claude Haiku 4.5 0.46% $0.07
MiniMax-M2.7 high 0.46% $0.10
GPT-6 Luna none 0.30% $0.01
gpt-oss-120b high 0.15% $0.01
GPT-5.4 0.15% $0.07
GPT-5.6 Luna 0.15% $0.02
Gemini 3.5 Flash-Lite low 0% $0.06

11 models have held number one on the official leaderboard since its release in April 2026 15 models have held number one with claims included since its release in April 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Anthropic DeepSeek Google DeepMind Moonshot AI Z.ai Qwen Mistral AI MiniMax
Own leaderboard 11 leaders
Gemini 4 Carbon 53.9%
Lab claims 13 leaders
Kimi K3.1 Pro 73.3%

Sources: Official AutomationBench leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Kimi K3.1 Pro MiniMax-M3.1-Flash DeepSeek V4.1 Flash GLM-5.3-Flash DeepSeek V4 Pro 0813 DeepSeek-V4-Flash Claude Opus 4.7 0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Kimi K3.1 Pro MiniMax-M3.1-Flash

Sources: Official AutomationBench leaderboard and labs’ model cards

20 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Anthropic
OpenAI
Moonshot AI
DeepSeek
Qwen
Google DeepMind
Z.ai
Tencent
Meta
StepFun
Xiaomi Corp
Upstage
Reflection
Mistral AI
Ant Group
Liquid AI
NVIDIA
MiniMax
Meituan
Thinking Machines
DateLaunch postScoreAs the post writes it
Nov 9, 2026 Muse Ember 37.8 AutomationBench-AA, Muse Ember max reasoning
Nov 9, 2026 Claude Fable 5.2 51.4 Business workflows AutomationBench, max effort, default fallbacks (launch table and Table 8.1.A)
Nov 5, 2026 Inkling 2 Small 45.6 AutomationBench v1.0.6, 45.6%
Nov 4, 2026 Tencent Hy4 56.2 AutomationBench v1.0.6
Nov 2, 2026 LongCat-2.5 51.8 AutomationBench (v1.0.6)
Oct 28, 2026 Gemini 4 Carbon 53.9 AutomationBench (Zapier)
Oct 27, 2026 LFM3-1.7B 41.3 AutomationBench (public)
Oct 24, 2026 Kimi K3.1 Pro 73.3 Automation Bench-AA
Oct 19, 2026 MiniMax-M3.1-Flash 62.8 Automation-Bench v1.0.6
Oct 17, 2026 DeepSeek V4.1 Flash 1017 60.5 Automation-Bench v1.0.6
Oct 15, 2026 Nemotron 3.5 Super VL 32.4 AutomationBench-AA [offline]
Oct 13, 2026 LFM2.5-24B-A2B 27.7 AutomationBench
Oct 13, 2026 Ling-3.1-flash 52.5 Automationbench (Public)
Oct 13, 2026 Gemini 4 Argon 51.3 AutomationBench (Score)
Oct 12, 2026 Muse Wonder 39.1 AutomationBench-AA
Oct 12, 2026 Muse Wonder Full 45.4 AutomationBench-AA
Oct 6, 2026 Mistral Large 4 59.9 Agentic behaviour: Artificial Analysis - AutomationBench
Oct 5, 2026 Beam 1.0 37 AutomationBench public
Oct 1, 2026 Solar Mini 4 22.3 AutomationBench-AA (Artificial Analysis score quoted by Upstage)
Sep 29, 2026 GPT-6.1 Sol 36.1 AutomationBench 1.0.6 (GPT-6.1 Sol Max)
Sep 28, 2026 Claude Sonnet 5.5 44.7 AutomationBench
Sep 22, 2026 GPT-6 Luna 20.7 AutomationBench 1.0.6 (GPT-6 Luna max)
Sep 22, 2026 MiMo-V2.6 Pro 53.1 AutomationBench v1.0.6
Sep 22, 2026 Claude Opus 5.5 40 Business workflows AutomationBench
Sep 22, 2026 GPT-6 Sol 33.2 AutomationBench 1.0.6 (GPT-6 Sol xhigh)
Sep 20, 2026 Step 5 Preview 44 AutomationBench (public); AutomationBench-AA 51.0
Sep 10, 2026 DeepSeek V4.1 Flash 54.8 AutomationBench (Pass@1)
Sep 3, 2026 GPT-6 Astra 41.4 AutomationBench
Sep 2, 2026 Qwen3.8 Max (0902) 50.8 Agentic SaaS workflow, Automation Bench v1.0.6 (Pass@1)
Sep 2, 2026 Muse Spark 1.3 49.6 AutomationBench (E2E business workflows)
Sep 1, 2026 Claude Fable 5.1 31.4 Business workflows AutomationBench
Aug 28, 2026 Tencent Hy4 preview 32.1 AutomationBench (v1.0.6)
Aug 26, 2026 GLM-5.3-Flash 48.8 AutomationBench (v1.0.6)
Aug 21, 2026 DeepSeek-V4-Flash-Vision-Exp 25.7 AutomationBench (Public)
Aug 14, 2026 GLM-5.3 48.2 AutomationBench (v1.0.6)
Aug 13, 2026 DeepSeek V4 Pro 0813 31.8 AutomationBench (Public)
Aug 13, 2026 Gemini 3.7 Flash 30.4 AutomationBench (Private set)
Aug 3, 2026 Qwen 3.8 Max 27.3 Automation-Bench (Pass@1)
Jul 31, 2026 DeepSeek V4 Flash 0731 25.1 AutomationBench Public
Jul 24, 2026 Claude Opus 5 26 Business workflows AutomationBench
Jul 16, 2026 Kimi K3 30.8 AutomationBench
Jul 9, 2026 GPT-5.6 Luna 14.9 AutomationBench
Jul 9, 2026 GPT-5.6 Terra 15.2 AutomationBench
Jul 9, 2026 GPT-5.6 Sol 18.1 AutomationBench (Sol, max)
Jun 30, 2026 Claude Sonnet 5 13.5 AutomationBench
Jun 9, 2026 Claude Fable 5 17.4 Tool use AutomationBench
May 28, 2026 Claude Opus 4.8 15.5 Automation Bench

16 system cards report it, from 6 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 9, 2026 Muse Ember technical report 37.8 AutomationBench-AA, Muse Ember max reasoning Capabilities
Nov 9, 2026 Claude Fable 5.2 system card 51.4 AutomationBench, Zapier private 1.0.6 set, 657 tasks per effort level, max effort, default API fallbacks. Scores from October 8, 2026 run (Table 8.1.A and Section 8.14.5) Capabilities
Oct 13, 2026 Gemini 4 Argon model card 51.3 AutomationBench, Score (results table, p.5) Capabilities
Oct 12, 2026 Muse Wonder model card 39.1 AutomationBench-AA strict pass@1 from Artificial Analysis, Muse Wonder (max) (capabilities table image, page 3) Capabilities
Oct 12, 2026 Muse Wonder Full model card 45.4 AutomationBench-AA strict pass@1 from Artificial Analysis, Muse Wonder Full (max) (capabilities table image, page 3) Capabilities
Sep 28, 2026 Claude Sonnet 5.5 system card 44.7 AutomationBench at max effort with default fallbacks, run by Zapier (Table 8.1.A, 8.14.6) Capabilities
Sep 22, 2026 MiMo-V2.6 Pro technical report 53.1 AutomationBench v1.0.6 (Table 3) Capabilities
Sep 22, 2026 Claude Opus 5.5 system card 40 AutomationBench, private held-out leaderboard set, max effort, run by Zapier (Table 8.1.A, 8.14.6) Capabilities
Sep 10, 2026 DeepSeek V4.1 Flash technical report 54.8 Automation-Bench (Pass@1), public set of AutomationBench v1.0.6 (Table 3) Capabilities
Sep 1, 2026 Claude Fable 5.1 system card 31.4 AutomationBench, private held-out leaderboard set, max effort (Table 8.1.A, 8.15.6) Capabilities
Aug 13, 2026 Gemini 3.7 Flash model card 30.4 AutomationBench, private set (results table, p.5) Capabilities
Jul 27, 2026 Kimi K3 technical report 30.8 AutomationBench, 600-task public subset (Table 2) Capabilities
Jul 24, 2026 Claude Opus 5 system card 26 AutomationBench, private held-out leaderboard set, max effort (24% at medium) (Table 8.1.A, 8.13.7) Capabilities
Jun 30, 2026 Claude Sonnet 5 system card 13.5 AutomationBench, private held-out leaderboard set, max effort, run by Zapier (Table 8.1.A, 8.11.6) Capabilities
Jun 9, 2026 Claude Fable 5 system card 17.4 AutomationBench, private held-out leaderboard set, max effort, measured by Zapier (Table 8.1.A, 8.17.9) Capabilities
May 28, 2026 Claude Opus 4.8 system card 15.5 AutomationBench, Zapier private held-out leaderboard set, max effort (Table 8.1.A, 8.13.8) Capabilities

The lab's number comes out ahead in 5 of 11 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 8 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
Muse Spark 1.3 49.6% AutomationBench (E2E business workflows) 20.7% Leaderboard, max +29 pts
GPT-5.6 Sol 18.1% AutomationBench (Sol, max) 28.8% Leaderboard, max −11 pts
Claude Fable 5.1 31.4% Business workflows AutomationBench 22.4% Leaderboard, max +9.0 pts
GPT-5.6 Terra 15.2% AutomationBench 23.6% Leaderboard, max −8.4 pts
Kimi K3 30.8% AutomationBench 22.7% Leaderboard +8.1 pts
Claude Sonnet 5 13.5% AutomationBench 10.7% Leaderboard, max +2.8 pts
Claude Opus 5.5 40% Business workflows AutomationBench 42.5% Leaderboard, max, default fallbacks −2.5 pts
GPT-5.6 Luna 14.9% AutomationBench 17.1% Leaderboard, max −2.2 pts
Claude Opus 4.8 15.5% Automation Bench 16.9% Leaderboard, xhigh −1.4 pts
Claude Opus 5 26% Business workflows AutomationBench 26.9% Leaderboard, max −0.9 pts
Claude Fable 5 17.4% Tool use AutomationBench 17.1% Leaderboard, max +0.3 pts
Claude Fable 5.2 51.4% Business workflows AutomationBench, max effort, default fallbacks (launch table and Table 8.1.A) 51.5% Leaderboard, max, default fallbacks Same
Claude Sonnet 5.5 44.7% AutomationBench 44.8% Leaderboard, max, default fallbacks Same
Gemini 3.7 Flash 30.4% AutomationBench (Private set) 30.4% Leaderboard, high Same
Gemini 4 Carbon 53.9% AutomationBench (Zapier) 53.9% Leaderboard, high Same
Gemini 4 Argon 51.3% AutomationBench (Score) 51.3% Leaderboard, high Same
GPT-6 Astra 41.4% AutomationBench 41.4% Leaderboard, max Same
GPT-6 Sol 33.2% AutomationBench 1.0.6 (GPT-6 Sol xhigh) 33.2% Leaderboard, xhigh Same
GPT-6 Luna 20.7% AutomationBench 1.0.6 (GPT-6 Luna max) 20.7% Leaderboard, max Same

2 logged changes that move scores

Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.

  1. Tasks changed July 31st, 2026

    The maintainers corrected task discoverability and overly strict checks in patch 1.0.6, increased private-task difficulty, and refreshed official runs. Earlier scores use different task requirements. Source

  2. Tasks changed July 16th, 2026

    The maintainers expanded and revised tasks in patch 1.0.5 to improve fairness and task requirements. Scores from earlier task snapshots may not be comparable. Source

More in Tool use

All 144 in Tool use
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%
  • Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%