Chinese Tool-Use Benchmark

Qwen-14B 98% leads since September 2023

Dropped

Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate.

Qwen replaced version 20230803 with version 20231206, so scores from the two versions differ.

Made by: Qwen Team, Alibaba Cloud
Released: August 2023
Kind: Function calling
Level: Everyday
Scored: Checked automatically
Answers: Short answer
Tasks: All public

Strongest model tested: Qwen-14B #213 on the capability index
Last in a launch post: February 2024, dropped after it saturated
Last model added: February 2024

Models scored in lab cards

Launch posts list it, from 1 lab

System cards report it, from 1 lab

2 models share the top score of 98% among models released by February 2024

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

The newest model with a score here came out in February 2024, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

1 Qwen-14B 98%
2 Qwen-7B 98%
3 Qwen1.5-72B 95.1%
4 Qwen-1_8B 92%

Source: Labs’ model cards

#	Model	Score	Date	Source
1	Qwen-14B	98%	Sep 2023	Lab model card
2	Qwen-7B	98%	Sep 2023	Lab model card
3	Qwen1.5-72B	95.1%	Feb 2024	Lab model card
4	Qwen-1_8B	92%	Sep 2023	Lab model card

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen

Lab claims Passed 85% in Sep 2023, by Qwen-14B

Qwen-14B 98%

Source: Labs’ model cards

Qwen’s own number for Qwen-14B passed 85% in September 2023

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card

Source: Labs’ model cards

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen

Date	Launch post	Score	As the post writes it
Feb 5, 2024	Qwen1.5-72B	95.1	Tool-Use Benchmark (Tool Selection Acc.)
Sep 25, 2023	Qwen-14B	98	Chinese Tool-Use Benchmark (ReAct), Tool Selection Acc.

3 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
Sep 28, 2023	Qwen-7B technical report	98	In-house Chinese ReAct tool-use benchmark, Tool Selection (Acc.). Tool Input Rouge-L 91, False Positive Error 7.3% (Table 6)	Capabilities
Sep 28, 2023	Qwen-14B technical report	98	In-house Chinese ReAct tool-use benchmark, Tool Selection (Acc.). Tool Input Rouge-L 93, False Positive Error 2.4% (Table 6)	Capabilities
Sep 28, 2023	Qwen-1_8B technical report	92	In-house Chinese ReAct tool-use benchmark, Tool Selection (Acc.). Tool Input Rouge-L 89, False Positive Error 19.3% (Table 6)	Capabilities

2 models share the top score of 98% among models released by February 2024

Number one over time

Qwen’s own number for Qwen-14B passed 85% in September 2023

Labs dropped it from launch posts once the best score passed 85%

3 system cards report it, from 1 lab

More in Tool use