Benchmark Almanac

TaskBench

Claude 2 93.5 leads since November 2023
Not in model cards

Models decompose requests, choose tools and dependencies, and predict tool parameters. Text metrics score decomposition, while graph and parameter F1 scores measure the remaining stages.

The benchmark contains 28,271 examples. The star count belongs to the broader JARVIS repository, whose TaskBench section reports 17 models.

Released
November 2023
Kind
Function calling
Level
Everyday
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Measures
Capability
Language
English
In and out
Text → Text, Code
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
Strongest model tested
Claude 2 #192 on the capability index
Last model added
January 2024
Scores refreshed
November 20th, 2026

In the authors’ words

In recent years, the remarkable progress of large language models (LLMs) has sparked interest in task automation, which involves decomposing complex tasks described by user instructions into sub-tasks and invoking external tools to execute them, playing a central role in autonomous agents. However, there is a lack of systematic and standardized benchmarks to promote the development of LLMs in task automation. To address this, we introduce TaskBench, a comprehensive framework to evaluate the capability of LLMs in task automation. Specifically, task automation can be divided into three critical stages: task decomposition, tool selection, and parameter prediction. To tackle the complexities inherent in these stages, we introduce the concept of Tool Graph to represent decomposed tasks and adopt a back-instruct method to generate high-quality user instructions. We propose TaskEval, a multi-faceted evaluation methodology that assesses LLM performance across these three stages. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation. Experimental results demonstrate that TaskBench effectively reflects the capabilities of various LLMs in task automation. It provides insights into model performance across different task complexities and domains, pushing the boundaries of what current models can achieve. TaskBench offers a scalable, adaptable, and reliable benchmark for advancing LLM-based autonomous agents.

Abstract of TaskBench: Benchmarking Large Language Models for Task Automation (arXiv:2311.18760, CC0)
9
Entries in its paper, 4 models we track

Claude 2 leads with 93.5 among models released by January 2024 Claude 2 leads with 93.5 among models released by January 2024

Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in January 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Claude 2 Daily Life APIs 93.5
  2. 2 Gemini 1.0 Pro Daily Life APIs 90.8
  3. 3 codellama-13b Daily Life APIs 87.7
  4. 4 GPT-3.5 Turbo Jan 2024, Daily Life APIs 85.4
  5. 5 Mistral 7B v0.3 Daily Life APIs 80.5
  6. 6 text-davinci-003 Daily Life APIs 80.4
  7. 7 vicuna-13b-v1.5 Daily Life APIs 75.7
  8. 8 nous-hermes-13b Daily Life APIs 73.5
  9. 9 llama-2-13b-chat Daily Life APIs 55.8

Source: The TaskBench paper

#EntryScoreSettingDateSource
1 Claude 2 93.5 Daily Life APIs Jul 2023 Official leaderboard
2 Gemini 1.0 Pro 90.8 Daily Life APIs Dec 2023 Official leaderboard
3 codellama-13b 87.7 Daily Life APIs Official leaderboard
4 GPT-3.5 Turbo Jan 2024 85.4 Daily Life APIs Jan 2024 Official leaderboard
5 Mistral 7B v0.3 80.5 Daily Life APIs Sep 2023 Official leaderboard
6 text-davinci-003 80.4 Daily Life APIs Official leaderboard
7 vicuna-13b-v1.5 75.7 Daily Life APIs Official leaderboard
8 nous-hermes-13b 73.5 Daily Life APIs Official leaderboard
9 llama-2-13b-chat 55.8 Daily Life APIs Official leaderboard
#ModelScoreSettingDateSource
1 Claude 2 93.5 Daily Life APIs Jul 2023 Official leaderboard
2 Gemini 1.0 Pro 90.8 Daily Life APIs Dec 2023 Official leaderboard
3 GPT-3.5 Turbo Jan 2024 85.4 Daily Life APIs Jan 2024 Official leaderboard
4 Mistral 7B v0.3 80.5 Daily Life APIs Sep 2023 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Claude 2
Hugging Fac… Daily Life …
93.5
Gemini 1.0 Pro
Hugging Fac… Multimedia … Daily Life …
90.8
GPT-3.5 Turbo
Hugging Fac… Multimedia … Daily Life …
85.4
Mistral 7B v0.3
Hugging Fac… Multimedia … Daily Life …
80.5

Source: The TaskBench paper

ModelEffortHarnessScoreCost per taskOutput tokens
Claude 2 Daily Life APIs 93.5
Gemini 1.0 Pro Daily Life APIs 90.8
GPT-3.5 Turbo Jan 2024 Daily Life APIs 85.4
Gemini 1.0 Pro Multimedia Tools 81.5
Claude 2 Multimedia Tools 80.9
Mistral 7B v0.3 Daily Life APIs 80.5
Claude 2 Hugging Face Tools 79.0
Gemini 1.0 Pro Hugging Face Tools 76.6
Mistral 7B v0.3 Multimedia Tools 76.1
GPT-3.5 Turbo Jan 2024 Multimedia Tools 72.8
GPT-3.5 Turbo Jan 2024 Hugging Face Tools 69.5
Mistral 7B v0.3 Hugging Face Tools 66.0

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Claude 2
Hugging Fac… Daily Life …
93.5
Gemini 1.0 Pro
Hugging Fac… Multimedia … Daily Life …
90.8
GPT-3.5 Turbo
Hugging Fac… Multimedia … Daily Life …
85.4
Mistral 7B v0.3
Hugging Fac… Multimedia … Daily Life …
80.5

Source: The TaskBench paper

ModelEffortHarnessScoreCost per taskOutput tokens
Claude 2 Daily Life APIs 93.5
Gemini 1.0 Pro Daily Life APIs 90.8
GPT-3.5 Turbo Jan 2024 Daily Life APIs 85.4
Gemini 1.0 Pro Multimedia Tools 81.5
Claude 2 Multimedia Tools 80.9
Mistral 7B v0.3 Daily Life APIs 80.5
Claude 2 Hugging Face Tools 79.0
Gemini 1.0 Pro Hugging Face Tools 76.6
Mistral 7B v0.3 Multimedia Tools 76.1
GPT-3.5 Turbo Jan 2024 Multimedia Tools 72.8
GPT-3.5 Turbo Jan 2024 Hugging Face Tools 69.5
Mistral 7B v0.3 Hugging Face Tools 66.0

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Benchmark paper Not tested since Jan 2024
Claude 2 93.5

Source: The TaskBench paper

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
In the benchmark’s paper
60.0 70.0 80.0 90.0 100.0 2024 60.0 70.0 80.0 90.0 100.0 2024

Source: Official TaskBench leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Tool use

All 265 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Neon 56.3%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%

Data audit

8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in January 2024, for GPT-3.5 Turbo. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
  • Pass Independent runs Scores from the benchmark’s paper (9 entries).
  • Pass Models scored 4 models we track, plus 5 other entries on its leaderboard.
  • Warning Newest score The newest came 2 years ago, in January 2024, for GPT-3.5 Turbo.
  • Note Frontier models None of the top 10 has a score. The strongest tested is Claude 2 (#192).
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on arxiv.org: 12 rows for models we track and 5 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check