Benchmark Almanac

τ³-bench Banking

Qwen 3.8 Max 55.1% leads since August 2026
Qwen 3.8 Max 55.1% leads since August 2026
In use

Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state.

A separate domain of τ³-bench. Its overall score covers retail, airline and telecom only.

Made by
Sierra
Released
March 2026
Kind
Customer service
Level
Everyday
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Strongest model tested
Claude Opus 5 #10 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 12th, 2026
26
Entries on its own leaderboard, 25 models we track
20
Launch posts list it, from 12 labs
11
System cards report it, from 7 labs

Qwen 3.8 Max leads with 55.1% among models released by August 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

Qwen 3.8 Max leads with 55.1%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in August 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

26 entries. Scroll sideways for the rest.

  1. 55.1 Qwen 3.8 Max
  2. 48.7 Claude Opus 5
  3. 47.9 Grok 4.5
  4. 46.9 GPT-5.6 Sol
  5. 44.6 GPT-5.5
  6. 40.5 Muse Spark 1.1
  7. 40.2 Claude Opus 4.7
  8. 39.7 Claude Opus 4.8
  9. 39.7 Claude Fable 5
  10. 39.4 GPT-5.4
  11. 37.1 GLM-5.2
  12. 37.1 Kimi K3
  13. 32.2 GPT-5.2
  14. 27.3 Gemini 3 Flash
  15. 27.3 Claude Opus 4.6
  16. 26.0 Gemini 3.1 Pro
  17. 25.3 Claude Sonnet 4.5
  18. 25.0 Inkling
  19. 24.7 Claude Opus 4.5
  20. 18.0 Gemini 3 Pro
  21. 18.0 Grok 4.2
  22. 15.7 Grok 4 Fast
  23. 13.7 Gemini 2.5 Pro
  24. 13.1 Grok 4.1 Fast
  25. 9.79 GLM-5
  26. 9.79 Qwen3.5 397B-A17B
  1. 1 Qwen 3.8 Max xhigh 55.1%
  2. 2 Claude Opus 5 max 48.7%
  3. 3 Grok 4.5 high 47.9%
  4. 4 GPT-5.6 Sol xhigh 46.9%
  5. 5 GPT-5.5 xhigh 44.6%
  6. 6 Muse Spark 1.1 xhigh 40.5%
  7. 7 Claude Opus 4.7 max 40.2%
  8. 8 Claude Opus 4.8 max 39.7%
  9. 9 Claude Fable 5 max 39.7%
  10. 10 GPT-5.4 xhigh 39.4%
  11. 11 GLM-5.2 xhigh 37.1%
  12. 12 Kimi K3 max 37.1%
  13. 13 GPT-5.2 high 32.2%
  14. 14 Gemini 3 Flash high 27.3%
  15. 15 Claude Opus 4.6 max 27.3%
  16. 16 Gemini 3.1 Pro high 26%
  17. 17 Claude Sonnet 4.5 enabled 25.3%
  18. 18 Inkling max 25%
  19. 19 Claude Opus 4.5 high 24.7%
  20. 20 Gemini 3 Pro high 18%
  21. 21 Grok 4.2 high 18%
  22. 22 Grok 4 Fast high 15.7%
  23. 23 Gemini 2.5 Pro Jun 2025, high 13.7%
  24. 24 Grok 4.1 Fast high 13.1%
  25. 25 GLM-5 enabled 9.79%
  26. 26 Qwen3.5 397B-A17B enabled 9.79%

Source: Official τ³-bench Banking leaderboard

44 models. Scroll sideways for the rest.

  1. 55.1 Qwen 3.8 Max
  2. 48.7 Claude Opus 5
  3. 47.9 Grok 4.5
  4. 47.8 Ling-3.1-flash
  5. 47.2 Solar Mini 4
  6. 46.9 GPT-5.6 Sol
  7. 44.6 GPT-5.5
  8. 42.5 Step 5 Preview
  9. 40.5 Muse Spark 1.1
  10. 40.4 Inkling 2 Small
  11. 40.2 Claude Opus 4.7
  12. 39.7 Claude Opus 4.8
  13. 39.7 Claude Fable 5
  14. 39.4 GPT-5.4
  15. 38.0 Beam 1.0
  16. 37.1 GLM-5.2
  17. 37.1 Kimi K3
  18. 35.1 Gemini 3.8 Live Extended Thinking
  19. 32.2 GPT-5.2
  20. 32.0 GPT-Live-1
  21. 30.2 Mercury 3
  22. 27.3 Gemini 3 Flash
  23. 27.3 Claude Opus 4.6
  24. 26.0 Gemini 3.1 Pro
  25. 25.3 Claude Sonnet 4.5
  26. 25.0 Inkling
  27. 24.7 Claude Opus 4.5
  28. 23.5 Muse Glimmer
  29. 23.0 Solar Pro 4
  30. 22.6 Nemotron 3 Ultra
  31. 19.6 Solar Open 2
  32. 18.9 Nemotron 3.5 Super VL
  33. 18.0 Gemini 3 Pro
  34. 17.0 Nemotron 3.5 Bolt 74B-A9B
  35. 15.7 Grok 4 Fast
  36. 15.5 Inkling-Small
  37. 14.2 K-EXAONE 2.0
  38. 13.7 Gemini 2.5 Pro
  39. 13.1 Grok 4.1 Fast
  40. 9.90 Nemotron 3.5 Lightning 30B-A3B
  41. 9.79 GLM-5
  42. 9.79 Qwen3.5 397B-A17B
  43. 8.04 LFM2.5-24B-A2B
  44. 5.67 LFM2.5-2.6B
  1. 1 Qwen 3.8 Max xhigh 55.1%
  2. 2 Claude Opus 5 max 48.7%
  3. 3 Grok 4.5 high 47.9%
  4. 4 Ling-3.1-flash 47.8%
  5. 5 Solar Mini 4 47.2%
  6. 6 GPT-5.6 Sol xhigh 46.9%
  7. 7 GPT-5.5 xhigh 44.6%
  8. 8 Step 5 Preview 42.5%
  9. 9 Muse Spark 1.1 xhigh 40.5%
  10. 10 Inkling 2 Small 40.4%
  11. 11 Claude Opus 4.7 max 40.2%
  12. 12 Claude Opus 4.8 max 39.7%
  13. 13 Claude Fable 5 max 39.7%
  14. 14 GPT-5.4 xhigh 39.4%
  15. 15 Beam 1.0 38%
  16. 16 GLM-5.2 xhigh 37.1%
  17. 17 Kimi K3 max 37.1%
  18. 18 Gemini 3.8 Live Extended Thinking 35.1%
  19. 19 GPT-5.2 high 32.2%
  20. 20 GPT-Live-1 32%
  21. 21 Mercury 3 30.2%
  22. 22 Gemini 3 Flash high 27.3%
  23. 23 Claude Opus 4.6 max 27.3%
  24. 24 Gemini 3.1 Pro high 26%
  25. 25 Claude Sonnet 4.5 enabled 25.3%
  26. 26 Inkling max 25%
  27. 27 Claude Opus 4.5 high 24.7%
  28. 28 Muse Glimmer 23.5%
  29. 29 Solar Pro 4 23%
  30. 30 Nemotron 3 Ultra 22.6%
  31. 31 Solar Open 2 19.6%
  32. 32 Nemotron 3.5 Super VL 18.9%
  33. 33 Gemini 3 Pro high 18%
  34. 34 Nemotron 3.5 Bolt 74B-A9B 17%
  35. 35 Grok 4 Fast high 15.7%
  36. 36 Inkling-Small 15.5%
  37. 37 K-EXAONE 2.0 14.2%
  38. 38 Gemini 2.5 Pro Jun 2025, high 13.7%
  39. 39 Grok 4.1 Fast high 13.1%
  40. 40 Nemotron 3.5 Lightning 30B-A3B 9.90%
  41. 41 GLM-5 enabled 9.79%
  42. 42 Qwen3.5 397B-A17B enabled 9.79%
  43. 43 LFM2.5-24B-A2B 8.04%
  44. 44 LFM2.5-2.6B 5.67%

Sources: Official τ³-bench Banking leaderboard and labs’ model cards

#EntryScoreSettingDateSource
1 Qwen 3.8 Max 55.1% xhigh Aug 2026 Official leaderboard
2 Claude Opus 5 48.7% max Jul 2026 Official leaderboard
3 Grok 4.5 47.9% high Jul 2026 Official leaderboard
4 GPT-5.6 Sol 46.9% xhigh Jul 2026 Official leaderboard
5 GPT-5.5 44.6% xhigh Apr 2026 Official leaderboard
6 Muse Spark 1.1 40.5% xhigh Jul 2026 Official leaderboard
7 Claude Opus 4.7 40.2% max Apr 2026 Official leaderboard
8 Claude Opus 4.8 39.7% max May 2026 Official leaderboard
9 Claude Fable 5 39.7% max Jun 2026 Official leaderboard
10 GPT-5.4 39.4% xhigh Mar 2026 Official leaderboard
11 GLM-5.2 37.1% xhigh Jun 2026 Official leaderboard
12 Kimi K3 37.1% max Jul 2026 Official leaderboard
13 GPT-5.2 32.2% high Dec 2025 Official leaderboard
14 Gemini 3 Flash 27.3% high Dec 2025 Official leaderboard
15 Claude Opus 4.6 27.3% max Feb 2026 Official leaderboard
16 Gemini 3.1 Pro 26% high Feb 2026 Official leaderboard
17 Claude Sonnet 4.5 25.3% enabled Sep 2025 Official leaderboard
18 Inkling 25% max Jul 2026 Official leaderboard
19 Claude Opus 4.5 24.7% high Nov 2025 Official leaderboard
20 Gemini 3 Pro 18% high Nov 2025 Official leaderboard
21 Grok 4.2 18% high May 2026 Official leaderboard
22 Grok 4 Fast 15.7% high Sep 2025 Official leaderboard
23 Gemini 2.5 Pro Jun 2025 13.7% high Jun 2025 Official leaderboard
24 Grok 4.1 Fast 13.1% high Nov 2025 Official leaderboard
25 GLM-5 9.79% enabled Feb 2026 Official leaderboard
26 Qwen3.5 397B-A17B 9.79% enabled Feb 2026 Official leaderboard
#ModelScoreSettingDateSource
1 Qwen 3.8 Max 55.1% xhigh Aug 2026 Official leaderboard
2 Claude Opus 5 48.7% max Jul 2026 Official leaderboard
3 Grok 4.5 47.9% high Jul 2026 Official leaderboard
4 Ling-3.1-flash 47.8% Oct 2026 Lab model card
5 Solar Mini 4 47.2% Oct 2026 Lab model card
6 GPT-5.6 Sol 46.9% xhigh Jul 2026 Official leaderboard
7 GPT-5.5 44.6% xhigh Apr 2026 Official leaderboard
8 Step 5 Preview 42.5% Sep 2026 Lab model card
9 Muse Spark 1.1 40.5% xhigh Jul 2026 Official leaderboard
10 Inkling 2 Small 40.4% Nov 2026 Lab model card
11 Claude Opus 4.7 40.2% max Apr 2026 Official leaderboard
12 Claude Opus 4.8 39.7% max May 2026 Official leaderboard
13 Claude Fable 5 39.7% max Jun 2026 Official leaderboard
14 GPT-5.4 39.4% xhigh Mar 2026 Official leaderboard
15 Beam 1.0 38% Oct 2026 Lab model card
16 GLM-5.2 37.1% xhigh Jun 2026 Official leaderboard
17 Kimi K3 37.1% max Jul 2026 Official leaderboard
18 Gemini 3.8 Live Extended Thinking 35.1% Sep 2026 Lab model card
19 GPT-5.2 32.2% high Dec 2025 Official leaderboard
20 GPT-Live-1 32% Jul 2026 Lab model card
21 Mercury 3 30.2% Oct 2026 Lab model card
22 Gemini 3 Flash 27.3% high Dec 2025 Official leaderboard
23 Claude Opus 4.6 27.3% max Feb 2026 Official leaderboard
24 Gemini 3.1 Pro 26% high Feb 2026 Official leaderboard
25 Claude Sonnet 4.5 25.3% enabled Sep 2025 Official leaderboard
26 Inkling 25% max Jul 2026 Official leaderboard
27 Claude Opus 4.5 24.7% high Nov 2025 Official leaderboard
28 Muse Glimmer 23.5% Aug 2026 Lab model card
29 Solar Pro 4 23% Aug 2026 Lab model card
30 Nemotron 3 Ultra 22.6% Jun 2026 Lab model card
31 Solar Open 2 19.6% Jul 2026 Lab model card
32 Nemotron 3.5 Super VL 18.9% Oct 2026 Lab model card
33 Gemini 3 Pro 18% high Nov 2025 Official leaderboard
34 Nemotron 3.5 Bolt 74B-A9B 17% Oct 2026 Lab model card
35 Grok 4 Fast 15.7% high Sep 2025 Official leaderboard
36 Inkling-Small 15.5% Jul 2026 Lab model card
37 K-EXAONE 2.0 14.2% Jul 2026 Lab model card
38 Gemini 2.5 Pro Jun 2025 13.7% high Jun 2025 Official leaderboard
39 Grok 4.1 Fast 13.1% high Nov 2025 Official leaderboard
40 Nemotron 3.5 Lightning 30B-A3B 9.90% Oct 2026 Lab model card
41 GLM-5 9.79% enabled Feb 2026 Official leaderboard
42 Qwen3.5 397B-A17B 9.79% enabled Feb 2026 Official leaderboard
43 LFM2.5-24B-A2B 8.04% Oct 2026 Lab model card
44 LFM2.5-2.6B 5.67% Aug 2026 Lab model card

Labs report scores for 22 models in their cards. Switch the scores above to rank them together.

7 models have held number one on the official leaderboard since its release in March 2026 7 models have held number one with claims included since its release in March 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Anthropic xAI Qwen NVIDIA Moonshot AI Google DeepMind StepFun Upstage Ant Group
Own leaderboard Not tested since Aug 2026
Qwen 3.8 Max 55.1%
Lab claims 8 leaders
Ling-3.1-flash 47.8%

Sources: Official τ³-bench Banking leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Qwen 3.8 Max Claude Opus 5 Claude Opus 4.7 GPT-5.2 Claude Sonnet 4.5 Grok 4 Fast 0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Qwen 3.8 Max Claude Opus 5

Sources: Official τ³-bench Banking leaderboard and labs’ model cards

12 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

NVIDIA
OpenAI
Thinking Machines
Upstage
LG AI Research
Liquid AI
Meta
Google DeepMind
StepFun
Reflection
Ant Group
Inception Labs
DateLaunch postScoreAs the post writes it
Nov 5, 2026 Inkling 2 Small 40.4 Tau 3 Banking v1.0.1
Oct 19, 2026 Mercury 3 30.2 τ³-bench Banking
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B 17 Tau3 Banking
Oct 15, 2026 Nemotron 3.5 Super VL 18.9 tau3-bench banking
Oct 13, 2026 LFM2.5-24B-A2B 8.04 τ³-Bench Banking
Oct 13, 2026 Ling-3.1-flash 47.8 TAU3-Banking
Oct 5, 2026 Beam 1.0 38 tau3 banking
Oct 1, 2026 Solar Mini 4 47.2 τ³-Banking (a separate evaluation, printed as 47.2)
Sep 20, 2026 Step 5 Preview 42.5 τ³-Banking
Sep 15, 2026 Gemini 3.8 Live Extended Thinking 35.1 τ³-Banking Leaderboard (Sierra), voice setting, High
Aug 11, 2026 Nemotron 3.5 Lightning 30B-A3B 9.28 τ³-bench (Banking)
Aug 11, 2026 Solar Pro 4 23 τ³-Banking (Artificial Analysis score quoted by Upstage)
Aug 10, 2026 Muse Glimmer 23.5 τ3-Banking
Aug 4, 2026 LFM2.5-2.6B 5.67 τ³-Bench Banking
Jul 31, 2026 K-EXAONE 2.0 14.2 τ3-Banking
Jul 30, 2026 Inkling-Small 15.5 Tau 3 Banking
Jul 22, 2026 Solar Open 2 19.6 τ³ (banking)
Jul 15, 2026 Inkling 23.7 Tau 3 Banking (Artificial Analysis)
Jul 8, 2026 GPT-Live-1 32 Tau Banking (Voice) Knowledge, Pass@1, GPT-Live-1 paired with GPT-6 Astra at medium reasoning effort (asterisked in the chart)
Jun 4, 2026 Nemotron 3 Ultra 22.6 TauBench V3 Banking

11 system cards report it, from 7 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 19, 2026 Muse Glimmer technical report 23.5 τ3-Banking, high, 26.0 xhigh (Tables 2 and 5) Capabilities
Oct 15, 2026 Nemotron 3.5 Super VL technical report 18.9 Tau3 Banking (Table 10) Capabilities
Oct 15, 2026 Nemotron 3.5 Lightning 30B-A3B technical report 9.9 Tau3 Banking, BF16 acc. (Table 22) Capabilities
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B technical report 17 Tau3 Banking, released 24GB checkpoint (Table 7). BF16 17.01 Capabilities
Aug 5, 2026 K-EXAONE 2.0 technical report 14.2 tau3-Banking (Table 6, Reasoning mode) Capabilities
Jul 30, 2026 Inkling-Small model card 15.5 Tau 3 Banking (5. Evaluations table) Capabilities
Jul 27, 2026 Kimi K3 technical report 33.4 τ3-Banking, cited from Artificial Analysis (Table 2) Capabilities
Jul 22, 2026 Solar Open 2 technical report 19.6 τ3 (banking) (Table 3) Capabilities
Jul 15, 2026 Inkling model card 23.7 Tau 3 Banking (5. Evaluations table, effort 0.99) Capabilities
Jul 14, 2026 Grok 4.5 model card 32.6 τ³-banking (AA), accuracy, high (4.2) Capabilities
Jun 9, 2026 Nemotron 3 Ultra technical report 22.6 TauBench V3 Banking (Table 10) Capabilities

The lab's number comes out ahead in 0 of 3 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. Every benchmark.

Model Lab says Board says Gap What differs
Grok 4.5 32.6% τ³-banking (AA), accuracy, high (4.2) 47.9% Leaderboard, high −15 pts xAI quotes Artificial Analysis's run of the same tasks, with its own user simulator and judge. The board shows Sierra's own runs.
Kimi K3 33.4% τ3-Banking, cited from Artificial Analysis (Table 2) 37.1% Leaderboard, max −3.7 pts
Inkling 23.7% Tau 3 Banking (Artificial Analysis) 25% Leaderboard, max −1.3 pts

2 logged changes that move scores

Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.

  1. Scores recomputed July 22nd, 2026

    The maintainers regraded banking leaderboard trajectories under the corrected evaluation scheme. The revised results are not comparable with scores from the earlier grader. Source

  2. Harness changed May 13th, 2026

    The maintainers changed banking’s default retrieval configuration from BM25 to AllTools, combining sparse search, dense search, and shell access. Default runs now expose different tools. Source

More in Tool use

All 144 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%
  • Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%