Benchmark Almanac

When2Call

GPT-4 Turbo Apr 2024 64.6% leads since April 2025
GPT-4 Turbo Apr 2024 64.6% leads since April 2025
In use

Models decide when to call a tool, when to ask a follow-up question and when to say the tools cannot answer. Scored on multiple-choice and LLM-judged test sets.

The Nemotron 3.5 Bolt E4B card reports a multi-turn variant, When2Call Multi-Turn. The repo also has a training set.

Made by
NVIDIA
Released
April 2025
Kind
Function calling
Level
Everyday
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Strongest model tested
Qwen2.5-72B #165 on the capability index
Last in a launch post
October 2026
Last model added
October 2026
Scores refreshed
November 12th, 2026
24
Entries on its own leaderboard, 10 models we track
1
Launch post lists it, from 1 lab
1
System card reports it, from 1 lab

GPT-4 Turbo leads with 64.6% among models released by September 2024

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

GPT-4 Turbo leads with 64.6%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in September 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

24 entries. Scroll sideways for the rest.

  1. 64.6 GPT-4 Turbo
  2. 61.3 GPT-4o (snapshot unspecified)
  3. 52.9 GPT-4o mini
  4. 52.4 MNM 8B dataset-RPO
  5. 51.0 MNM 4B dataset-RPO
  6. 49.4 MNM 8B dataset-SFT
  7. 48.1 MNM 4B dataset-SFT
  8. 37.8 Llama 3.1-70B
  9. 36.2 Qwen2.5-14B
  10. 34.3 xLAM 8x22B R
  11. 32.9 xLAM 8x7B R
  12. 32.9 Qwen2.5-32B
  13. 32.8 Qwen2.5-72B
  14. 32.0 Qwen 2.5 0.5B Instruct
  15. 32.0 Qwen2.5-7B
  16. 31.9 MNM 8B SFT (baseline)
  17. 31.5 xLAM 7B FC-R
  18. 29.9 Qwen 2.5 1.5B Instruct
  19. 29.8 Qwen 2.5 3B Instruct
  20. 29.7 MNM 4B SFT (baseline)
  21. 25.6 xLAM 1B FC-R
  22. 21.7 Llama 3.2 1B
  23. 17.9 Llama 3.2 3B
  24. 16.6 Llama 3.1-8B
  1. 1 GPT-4 Turbo Apr 2024, closed models; generation evaluation 64.6%
  2. 2 GPT-4o (snapshot unspecified) closed models; generation evaluation 61.3%
  3. 3 GPT-4o mini closed models; generation evaluation 52.9%
  4. 4 MNM 8B dataset-RPO open models; normalized likelihood evaluation 52.4%
  5. 5 MNM 4B dataset-RPO open models; normalized likelihood evaluation 51%
  6. 6 MNM 8B dataset-SFT open models; normalized likelihood evaluation 49.4%
  7. 7 MNM 4B dataset-SFT open models; normalized likelihood evaluation 48.1%
  8. 8 Llama 3.1-70B open models; normalized likelihood evaluation 37.8%
  9. 9 Qwen2.5-14B open models; normalized likelihood evaluation 36.2%
  10. 10 xLAM 8x22B R open models; normalized likelihood evaluation 34.3%
  11. 11 xLAM 8x7B R open models; normalized likelihood evaluation 32.9%
  12. 12 Qwen2.5-32B open models; normalized likelihood evaluation 32.9%
  13. 13 Qwen2.5-72B open models; normalized likelihood evaluation 32.8%
  14. 14 Qwen 2.5 0.5B Instruct open models; normalized likelihood evaluation 32%
  15. 15 Qwen2.5-7B open models; normalized likelihood evaluation 32%
  16. 16 MNM 8B SFT (baseline) open models; normalized likelihood evaluation 31.9%
  17. 17 xLAM 7B FC-R open models; normalized likelihood evaluation 31.5%
  18. 18 Qwen 2.5 1.5B Instruct open models; normalized likelihood evaluation 29.9%
  19. 19 Qwen 2.5 3B Instruct open models; normalized likelihood evaluation 29.8%
  20. 20 MNM 4B SFT (baseline) open models; normalized likelihood evaluation 29.7%
  21. 21 xLAM 1B FC-R open models; normalized likelihood evaluation 25.6%
  22. 22 Llama 3.2 1B open models; normalized likelihood evaluation 21.7%
  23. 23 Llama 3.2 3B open models; normalized likelihood evaluation 17.9%
  24. 24 Llama 3.1-8B open models; normalized likelihood evaluation 16.6%

Source: Official When2Call leaderboard

  1. 64.6 GPT-4 Turbo
  2. 60.5 Nemotron 3.5 Bolt E4B
  3. 52.9 GPT-4o mini
  4. 37.8 Llama 3.1-70B
  5. 36.2 Qwen2.5-14B
  6. 32.9 Qwen2.5-32B
  7. 32.8 Qwen2.5-72B
  8. 32.0 Qwen2.5-7B
  9. 21.7 Llama 3.2 1B
  10. 17.9 Llama 3.2 3B
  11. 16.6 Llama 3.1-8B
  1. 1 GPT-4 Turbo Apr 2024, closed models; generation evaluation 64.6%
  2. 2 Nemotron 3.5 Bolt E4B 60.5%
  3. 3 GPT-4o mini closed models; generation evaluation 52.9%
  4. 4 Llama 3.1-70B open models; normalized likelihood evaluation 37.8%
  5. 5 Qwen2.5-14B open models; normalized likelihood evaluation 36.2%
  6. 6 Qwen2.5-32B open models; normalized likelihood evaluation 32.9%
  7. 7 Qwen2.5-72B open models; normalized likelihood evaluation 32.8%
  8. 8 Qwen2.5-7B open models; normalized likelihood evaluation 32%
  9. 9 Llama 3.2 1B open models; normalized likelihood evaluation 21.7%
  10. 10 Llama 3.2 3B open models; normalized likelihood evaluation 17.9%
  11. 11 Llama 3.1-8B open models; normalized likelihood evaluation 16.6%

Sources: Official When2Call leaderboard and labs’ model cards

#EntryScoreSettingDateSource
1 GPT-4 Turbo Apr 2024 64.6% closed models; generation evaluation Apr 2024 Official leaderboard
2 GPT-4o (snapshot unspecified) 61.3% closed models; generation evaluation Official leaderboard
3 GPT-4o mini 52.9% closed models; generation evaluation Jul 2024 Official leaderboard
4 MNM 8B dataset-RPO 52.4% open models; normalized likelihood evaluation Official leaderboard
5 MNM 4B dataset-RPO 51% open models; normalized likelihood evaluation Official leaderboard
6 MNM 8B dataset-SFT 49.4% open models; normalized likelihood evaluation Official leaderboard
7 MNM 4B dataset-SFT 48.1% open models; normalized likelihood evaluation Official leaderboard
8 Llama 3.1-70B 37.8% open models; normalized likelihood evaluation Jul 2024 Official leaderboard
9 Qwen2.5-14B 36.2% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
10 xLAM 8x22B R 34.3% open models; normalized likelihood evaluation Official leaderboard
11 xLAM 8x7B R 32.9% open models; normalized likelihood evaluation Official leaderboard
12 Qwen2.5-32B 32.9% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
13 Qwen2.5-72B 32.8% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
14 Qwen 2.5 0.5B Instruct 32% open models; normalized likelihood evaluation Official leaderboard
15 Qwen2.5-7B 32% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
16 MNM 8B SFT (baseline) 31.9% open models; normalized likelihood evaluation Official leaderboard
17 xLAM 7B FC-R 31.5% open models; normalized likelihood evaluation Official leaderboard
18 Qwen 2.5 1.5B Instruct 29.9% open models; normalized likelihood evaluation Official leaderboard
19 Qwen 2.5 3B Instruct 29.8% open models; normalized likelihood evaluation Official leaderboard
20 MNM 4B SFT (baseline) 29.7% open models; normalized likelihood evaluation Official leaderboard
21 xLAM 1B FC-R 25.6% open models; normalized likelihood evaluation Official leaderboard
22 Llama 3.2 1B 21.7% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
23 Llama 3.2 3B 17.9% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
24 Llama 3.1-8B 16.6% open models; normalized likelihood evaluation Jul 2024 Official leaderboard
#ModelScoreSettingDateSource
1 GPT-4 Turbo Apr 2024 64.6% closed models; generation evaluation Apr 2024 Official leaderboard
2 Nemotron 3.5 Bolt E4B 60.5% Oct 2026 Lab model card
3 GPT-4o mini 52.9% closed models; generation evaluation Jul 2024 Official leaderboard
4 Llama 3.1-70B 37.8% open models; normalized likelihood evaluation Jul 2024 Official leaderboard
5 Qwen2.5-14B 36.2% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
6 Qwen2.5-32B 32.9% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
7 Qwen2.5-72B 32.8% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
8 Qwen2.5-7B 32% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
9 Llama 3.2 1B 21.7% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
10 Llama 3.2 3B 17.9% open models; normalized likelihood evaluation Sep 2024 Official leaderboard
11 Llama 3.1-8B 16.6% open models; normalized likelihood evaluation Jul 2024 Official leaderboard

Labs report scores for 1 model in their cards. Switch the scores above to rank them together.

Number one over time Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Not tested since Apr 2025
GPT-4 Turbo 64.6%

Source: Official When2Call leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card
0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) GPT-4 Turbo 0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) GPT-4 Turbo

Sources: Official When2Call leaderboard and labs’ model cards

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

NVIDIA
DateLaunch postScoreAs the post writes it
Oct 23, 2026 Nemotron 3.5 Bolt E4B 60.5 When2Call Multi-Turn

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 23, 2026 Nemotron 3.5 Bolt E4B technical report 60.5 When2Call-MT, released Q4_K_M imatrix checkpoint (Table 5). BF16 63.36 (Table 3) Capabilities

More in Tool use

All 144 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%