TaskBench
Models decompose requests, choose tools and dependencies, and predict tool parameters. Text metrics score decomposition, while graph and parameter F1 scores measure the remaining stages.
The benchmark contains 28,271 examples. The star count belongs to the broader JARVIS repository, whose TaskBench section reports 17 models.
- Made by
- Zhejiang University, Microsoft Research Asia and Fudan University
- Released
- November 2023
- Kind
- Function calling
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Agent task
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
- Claude 2 #192 on the capability index
- Last model added
- January 2024
- Scores refreshed
- November 20th, 2026
In the authors’ words
In recent years, the remarkable progress of large language models (LLMs) has sparked interest in task automation, which involves decomposing complex tasks described by user instructions into sub-tasks and invoking external tools to execute them, playing a central role in autonomous agents. However, there is a lack of systematic and standardized benchmarks to promote the development of LLMs in task automation. To address this, we introduce TaskBench, a comprehensive framework to evaluate the capability of LLMs in task automation. Specifically, task automation can be divided into three critical stages: task decomposition, tool selection, and parameter prediction. To tackle the complexities inherent in these stages, we introduce the concept of Tool Graph to represent decomposed tasks and adopt a back-instruct method to generate high-quality user instructions. We propose TaskEval, a multi-faceted evaluation methodology that assesses LLM performance across these three stages. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation. Experimental results demonstrate that TaskBench effectively reflects the capabilities of various LLMs in task automation. It provides insights into model performance across different task complexities and domains, pushing the boundaries of what current models can achieve. TaskBench offers a scalable, adaptable, and reliable benchmark for advancing LLM-based autonomous agents.
Claude 2 leads with 93.5 among models released by January 2024 Claude 2 leads with 93.5 among models released by January 2024
Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
The newest model with a score here came out in January 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 1 Claude 2 Daily Life APIs 93.5
- 2
Gemini 1.0 Pro Daily Life APIs 90.8
- 3 codellama-13b Daily Life APIs 87.7
- 4 GPT-3.5 Turbo Jan 2024, Daily Life APIs 85.4
- 5
Mistral 7B v0.3 Daily Life APIs 80.5
- 6 text-davinci-003 Daily Life APIs 80.4
- 7 vicuna-13b-v1.5 Daily Life APIs 75.7
- 8 nous-hermes-13b Daily Life APIs 73.5
- 9 llama-2-13b-chat Daily Life APIs 55.8
Source: The TaskBench paper
- 1 Claude 2 Daily Life APIs 93.5
- 2
Gemini 1.0 Pro Daily Life APIs 90.8
- 3 GPT-3.5 Turbo Jan 2024, Daily Life APIs 85.4
- 4
Mistral 7B v0.3 Daily Life APIs 80.5
Source: The TaskBench paper
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude 2 | 93.5 | Daily Life APIs | Jul 2023 | Official leaderboard |
| 2 | | 90.8 | Daily Life APIs | Dec 2023 | Official leaderboard |
| 3 | codellama-13b | 87.7 | Daily Life APIs | Official leaderboard | |
| 4 | GPT-3.5 Turbo Jan 2024 | 85.4 | Daily Life APIs | Jan 2024 | Official leaderboard |
| 5 | | 80.5 | Daily Life APIs | Sep 2023 | Official leaderboard |
| 6 | text-davinci-003 | 80.4 | Daily Life APIs | Official leaderboard | |
| 7 | vicuna-13b-v1.5 | 75.7 | Daily Life APIs | Official leaderboard | |
| 8 | nous-hermes-13b | 73.5 | Daily Life APIs | Official leaderboard | |
| 9 | llama-2-13b-chat | 55.8 | Daily Life APIs | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude 2 | 93.5 | Daily Life APIs | Jul 2023 | Official leaderboard |
| 2 | | 90.8 | Daily Life APIs | Dec 2023 | Official leaderboard |
| 3 | GPT-3.5 Turbo Jan 2024 | 85.4 | Daily Life APIs | Jan 2024 | Official leaderboard |
| 4 | | 80.5 | Daily Life APIs | Sep 2023 | Official leaderboard |
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Source: The TaskBench paper
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Claude 2 | Daily Life APIs | 93.5 | |||
| Daily Life APIs | 90.8 | ||||
| GPT-3.5 Turbo Jan 2024 | Daily Life APIs | 85.4 | |||
| Multimedia Tools | 81.5 | ||||
| Claude 2 | Multimedia Tools | 80.9 | |||
| Daily Life APIs | 80.5 | ||||
| Claude 2 | Hugging Face Tools | 79.0 | |||
| Hugging Face Tools | 76.6 | ||||
| Multimedia Tools | 76.1 | ||||
| GPT-3.5 Turbo Jan 2024 | Multimedia Tools | 72.8 | |||
| GPT-3.5 Turbo Jan 2024 | Hugging Face Tools | 69.5 | |||
| Hugging Face Tools | 66.0 |
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Source: The TaskBench paper
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Claude 2 | Daily Life APIs | 93.5 | |||
| Daily Life APIs | 90.8 | ||||
| GPT-3.5 Turbo Jan 2024 | Daily Life APIs | 85.4 | |||
| Multimedia Tools | 81.5 | ||||
| Claude 2 | Multimedia Tools | 80.9 | |||
| Daily Life APIs | 80.5 | ||||
| Claude 2 | Hugging Face Tools | 79.0 | |||
| Hugging Face Tools | 76.6 | ||||
| Multimedia Tools | 76.1 | ||||
| GPT-3.5 Turbo Jan 2024 | Multimedia Tools | 72.8 | |||
| GPT-3.5 Turbo Jan 2024 | Hugging Face Tools | 69.5 | |||
| Hugging Face Tools | 66.0 |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: The TaskBench paper
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official TaskBench leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Tool use
- AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state.
Gemini 4 Neon 56.3%
- MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks.
Gemini 4 Argon 90.7%
- BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes.
Qwen3.7-Max 75%
- τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state.
Qwen 3.8 Max 55.1%
- Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
- DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%
Data audit
8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in January 2024, for GPT-3.5 Turbo. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
- Pass Independent runs Scores from the benchmark’s paper (9 entries).
- Pass Models scored 4 models we track, plus 5 other entries on its leaderboard.
- Warning Newest score The newest came 2 years ago, in January 2024, for GPT-3.5 Turbo.
- Note Frontier models None of the top 10 has a score. The strongest tested is Claude 2 (#192).
- Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
- Pass Importer We read the board on arxiv.org: 12 rows for models we track and 5 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.