Benchmark Almanac

Galileo Agent Leaderboard v2

GPT-4.1 62% leads since July 2025
Not in model cards

Agents handle simulated multi-turn customer support conversations in banking, healthcare, insurance, investment and telecom, with synthetic personas and domain tools. Scores are Action Completion and Tool Selection Quality.

The board reports Action Completion. Version 2 replaced the sampled public datasets of version 1 with synthetic enterprise scenarios.

Made by
Galileo
Released
July 2025
Kind
Customer service
Level
Everyday
Scored
Graded by a model
Answers
Agent task
Tasks
All public
Strongest model tested
Grok 4 #85 on the capability index
Last model added
August 2025
Scores refreshed
November 12th, 2026
22
Entries on its own leaderboard, 21 models we track

GPT-4.1 leads with 62% among models released by August 2025

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in August 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

22 entries. Scroll sideways for the rest.

  1. 62.0 GPT-4.1
  2. 61.0 Mistral Medium 3.1
  3. 56.0 GPT-4.1 mini
  4. 55.0 Claude Sonnet 4
  5. 53.0 Kimi K2
  6. 53.0 Qwen3-235B-A22B-Instruct
  7. 51.0 Qwen2.5-72B
  8. 47.0 Gemini 2.5 Flash-Lite
  9. 44.0 GLM-4.5-Air
  10. 43.0 Gemini 2.5 Pro
  11. 42.0 Grok 4
  12. 40.0 DeepSeek-V3
  13. 38.0 GPT-4.1 nano
  14. 38.0 Gemini 2.5 Flash
  15. 34.0 Qwen3-235B-A22B-Thinking
  16. 32.0 Magistral Medium 1.0
  17. 29.0 Amazon Nova Pro
  18. 26.0 Mistral Small 3.2
  19. 20.0 Llama 3.3 70B Instruct
  20. 16.0 caller
  21. 16.0 Amazon Nova Lite
  22. 16.0 Magistral Small 1.0
  1. 1 GPT-4.1 62%
  2. 2 Mistral Medium 3.1 61%
  3. 3 GPT-4.1 mini 56%
  4. 4 Claude Sonnet 4 55%
  5. 5 Kimi K2 Jul 2025 53%
  6. 6 Qwen3-235B-A22B-Instruct Jul 2025 53%
  7. 7 Qwen2.5-72B 51%
  8. 8 Gemini 2.5 Flash-Lite Jun 2025 47%
  9. 9 GLM-4.5-Air 44%
  10. 10 Gemini 2.5 Pro Jun 2025 43%
  11. 11 Grok 4 42%
  12. 12 DeepSeek-V3 Mar 2025 40%
  13. 13 GPT-4.1 nano 38%
  14. 14 Gemini 2.5 Flash Jun 2025 38%
  15. 15 Qwen3-235B-A22B-Thinking Jul 2025 34%
  16. 16 Magistral Medium 1.0 32%
  17. 17 Amazon Nova Pro 29%
  18. 18 Mistral Small 3.2 26%
  19. 19 Llama 3.3 70B Instruct 20%
  20. 20 caller 16%
  21. 21 Amazon Nova Lite 16%
  22. 22 Magistral Small 1.0 16%

Source: Official Galileo Agent Leaderboard v2 leaderboard

#EntryScoreSettingDateSource
1 GPT-4.1 62% Apr 2025 Official leaderboard
2 Mistral Medium 3.1 61% Aug 2025 Official leaderboard
3 GPT-4.1 mini 56% Apr 2025 Official leaderboard
4 Claude Sonnet 4 55% May 2025 Official leaderboard
5 Kimi K2 Jul 2025 53% Jul 2025 Official leaderboard
6 Qwen3-235B-A22B-Instruct Jul 2025 53% Jul 2025 Official leaderboard
7 Qwen2.5-72B 51% Sep 2024 Official leaderboard
8 Gemini 2.5 Flash-Lite Jun 2025 47% Jun 2025 Official leaderboard
9 GLM-4.5-Air 44% Jul 2025 Official leaderboard
10 Gemini 2.5 Pro Jun 2025 43% Jun 2025 Official leaderboard
11 Grok 4 42% Jul 2025 Official leaderboard
12 DeepSeek-V3 Mar 2025 40% Mar 2025 Official leaderboard
13 GPT-4.1 nano 38% Apr 2025 Official leaderboard
14 Gemini 2.5 Flash Jun 2025 38% Jun 2025 Official leaderboard
15 Qwen3-235B-A22B-Thinking Jul 2025 34% Jul 2025 Official leaderboard
16 Magistral Medium 1.0 32% Jun 2025 Official leaderboard
17 Amazon Nova Pro 29% Dec 2024 Official leaderboard
18 Mistral Small 3.2 26% Jun 2025 Official leaderboard
19 Llama 3.3 70B Instruct 20% Dec 2024 Official leaderboard
20 caller 16% Official leaderboard
21 Amazon Nova Lite 16% Dec 2024 Official leaderboard
22 Magistral Small 1.0 16% Jun 2025 Official leaderboard

Speed vs Accuracy Trade-off

Average session duration of one conversation against Action Completion. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0 100 200 300 400 0%20%40%60%80% Action Completion Avg session duration (s) Qwen2.5-72B: Avg session duration 34.7 s, action completion 51% Amazon Nova Lite: Avg session duration 20.3 s, action completion 16% Amazon Nova Pro: Avg session duration 28 s, action completion 29% Llama 3.3 70B Instruct: Avg session duration 19.9 s, action completion 20% DeepSeek-V3: Avg session duration 60 s, action completion 40% GPT-4.1 mini: Avg session duration 26 s, action completion 56% GPT-4.1 nano: Avg session duration 12.4 s, action completion 38% Claude Sonnet 4: Avg session duration 66.6 s, action completion 55% Gemini 2.5 Pro: Avg session duration 126 s, action completion 43% Magistral Medium 1.0: Avg session duration 33 s, action completion 32% Magistral Small 1.0: Avg session duration 17.4 s, action completion 16% Gemini 2.5 Flash: Avg session duration 39.8 s, action completion 38% Mistral Small 3.2: Avg session duration 35.7 s, action completion 26% Grok 4: Avg session duration 226 s, action completion 42% Kimi K2: Avg session duration 164 s, action completion 53% GLM-4.5-Air: Avg session duration 69.2 s, action completion 44% Qwen3-235B-A22B-Instruct: Avg session duration 238 s, action completion 53% Qwen3-235B-A22B-Thinking: Avg session duration 302 s, action completion 34% Mistral Medium 3.1: Avg session duration 37.5 s, action completion 61% caller: Avg session duration 25.7 s, action completion 16%. Not in our model list GPT-4.1: Avg session duration 24.3 s, action completion 62% Gemini 2.5 Flash-Lite: Avg session duration 9.8 s, action completion 47% GPT-4.1Gemini 2.5 Flash-Lite 0 100 200 300 400 0%20%40%60%80% Action Completion Avg session duration (s) Qwen2.5-72B: Avg session duration 34.7 s, action completion 51% Amazon Nova Lite: Avg session duration 20.3 s, action completion 16% Amazon Nova Pro: Avg session duration 28 s, action completion 29% Llama 3.3 70B Instruct: Avg session duration 19.9 s, action completion 20% DeepSeek-V3: Avg session duration 60 s, action completion 40% GPT-4.1 mini: Avg session duration 26 s, action completion 56% GPT-4.1 nano: Avg session duration 12.4 s, action completion 38% Claude Sonnet 4: Avg session duration 66.6 s, action completion 55% Gemini 2.5 Pro: Avg session duration 126 s, action completion 43% Magistral Medium 1.0: Avg session duration 33 s, action completion 32% Magistral Small 1.0: Avg session duration 17.4 s, action completion 16% Gemini 2.5 Flash: Avg session duration 39.8 s, action completion 38% Mistral Small 3.2: Avg session duration 35.7 s, action completion 26% Grok 4: Avg session duration 226 s, action completion 42% Kimi K2: Avg session duration 164 s, action completion 53% GLM-4.5-Air: Avg session duration 69.2 s, action completion 44% Qwen3-235B-A22B-Instruct: Avg session duration 238 s, action completion 53% Qwen3-235B-A22B-Thinking: Avg session duration 302 s, action completion 34% Mistral Medium 3.1: Avg session duration 37.5 s, action completion 61% caller: Avg session duration 25.7 s, action completion 16%. Not in our model list GPT-4.1: Avg session duration 24.3 s, action completion 62% Gemini 2.5 Flash-Lite: Avg session duration 9.8 s, action completion 47% GPT-4.1Gemini 2.5 Flash-Lite
ModelSettingAvg session durationAction Completion
Gemini 2.5 Flash-Lite Jun 2025 9.8 s 47%
GPT-4.1 nano 12.36 s 38%
Magistral Small 1.0 17.42 s 16%
Llama 3.3 70B Instruct 19.92 s 20%
Amazon Nova Lite 20.26 s 16%
GPT-4.1 24.32 s 62%
caller 25.66 s 16%
GPT-4.1 mini 26 s 56%
Amazon Nova Pro 27.96 s 29%
Magistral Medium 1.0 32.96 s 32%
Qwen2.5-72B 34.68 s 51%
Mistral Small 3.2 35.69 s 26%
Mistral Medium 3.1 37.45 s 61%
Gemini 2.5 Flash Jun 2025 39.84 s 38%
DeepSeek-V3 Mar 2025 59.97 s 40%
Claude Sonnet 4 66.6 s 55%
GLM-4.5-Air 69.17 s 44%
Gemini 2.5 Pro Jun 2025 125.85 s 43%
Kimi K2 Jul 2025 163.62 s 53%
Grok 4 225.94 s 42%
Qwen3-235B-A22B-Instruct Jul 2025 238.02 s 53%
Qwen3-235B-A22B-Thinking Jul 2025 302.24 s 34%

The best score, 62%, costs $0.07 a task

Runs on the cost frontier score higher than any run that's cheaper. Mistral Medium 3.1 comes within 1 points of the best for $0.02 a task, 3 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Amazon OpenAI Google DeepMind Qwen Mistral AI Cost frontier One model at each effort level
0% 20% 40% 60% 80% $0.001 $0.01 $0.10 $1.00 Cost per task, log scale GPT-4.1 Mistral Medium 3.1 GPT-4.1 mini Qwen3-235B-A22B-Instruct Gemini 2.5 Flash-Lite GPT-4.1 nano Amazon Nova Lite 0% 20% 40% 60% 80% $0.001 $0.01 $0.10 $1.00 Cost per task, log scale GPT-4.1 Mistral Medium 3.1

Source: Official Galileo Agent Leaderboard v2 leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-4.1 62% $0.07
Mistral Medium 3.1 61% $0.02
GPT-4.1 mini 56% $0.01
Claude Sonnet 4 55% $0.15
Kimi K2 Jul 2025 53% $0.04
Qwen3-235B-A22B-Instruct Jul 2025 53% $0.007
Qwen2.5-72B 51% $0.04
Gemini 2.5 Flash-Lite Jun 2025 47% $0.004
GLM-4.5-Air 44% $0.02
Gemini 2.5 Pro Jun 2025 43% $0.14
Grok 4 42% $0.24
DeepSeek-V3 Mar 2025 40% $0.01
GPT-4.1 nano 38% $0.004
Gemini 2.5 Flash Jun 2025 38% $0.03
Qwen3-235B-A22B-Thinking Jul 2025 34% $0.06
Magistral Medium 1.0 32% $0.12
Amazon Nova Pro 29% $0.04
Mistral Small 3.2 26% $0.005
Llama 3.3 70B Instruct 20% $0.06
Amazon Nova Lite 16% $0.003
Magistral Small 1.0 16% $0.03

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Not tested since Aug 2025
GPT-4.1 62%

Source: Official Galileo Agent Leaderboard v2 leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) GPT-4.1 Qwen2.5-72B 0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) GPT-4.1 Qwen2.5-72B

Source: Official Galileo Agent Leaderboard v2 leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

Versions

Replaces Galileo Agent Leaderboard

More in Tool use

All 144 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%