Benchmark Almanac

Qwen

We track 53 benchmarks and 2 arenas that Qwen made or helped make. Its models are on its lab page.

14 in use 20 with public code 31 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. Competition problems with answers Apr 2025 86.5% In use
DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. General assistants Jan 2026 58.9% Fading
AutoLogi Models solve open-ended logic puzzles in English and Chinese. Generated verifier programs check each answer against the puzzle constraints. Abstract puzzles and logic Feb 2025 89.5% Dropped
CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Tasks from real jobs 75.9% In use
QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. General assistants Apr 2026 60.5% Fading
Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Chat and instructions in many languages Jul 2024 69.6% System cards only
Qwen in-house English automatic evaluation Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Other Jul 2024 72.9% System cards only
Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. Function calling Aug 2023 98% Dropped
Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. Chat and instructions in many languages Sep 2024 87% Dropped
QwenWebBench Models generate front-end code for web apps, games, SVG, and other categories in English and Chinese. Multimodal judgments produce Elo ratings. Web and app front ends Apr 2026 1,532 Fading
RecreationBench Agents explore a running reference app and rebuild it, across 250 tasks on five platforms. Programmatic tests and a VLM judge score the result. Desktop apps Sep 2026 58.1% In use
Code Interpreter Benchmark Models use a Python code interpreter for math, data visualization, and general tasks. Scoring covers result accuracy and the share of code that runs. Function calling Sep 2023 81.7% System cards only
Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. Knowledge in many languages Feb 2024 78.7% Dropped
Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. Reasoning in many languages Feb 2024 89.6% Dropped
AgentWorldBench Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions. Interactive world models Jun 2026 58.3 In use
QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. Web and app front ends May 2026 1,608 Fading
Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. Chat and instructions in many languages Jun 2024 4.15 Dropped
QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. Fixing real repositories 86.3% In use
QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. Web and app front ends May 2026 1,617 Fading
QwenWorldBench Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. Interactive world models May 2026 62.1% Fading
Code Interpreter Benchmark Math Models use a Python interpreter to solve math problems. The score is execution accuracy on the Math task set. Function calling Sep 2023 73.1% Dropped
Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. Secure code and review Oct 2026 25.1% In use
QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. Web and app front ends Aug 2026 69.9% In use
QwenVisualOffice Models handle visual office tasks in an internal Qwen test set. Qwen has not published the tasks or how they are scored. Documents, charts and text in images Aug 2026 44.6% In use
QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). Fixing real repositories 63.1% In use
QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. Web and app front ends 1,770 In use
QwenSVGBench Model writes SVG code for bilingual prompts. Rendered outputs judged by a multimodal model and ranked with Bradley-Terry Elo. Web and app front ends 1,758 In use
QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. Web and app front ends May 2026 1,884 In use
ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. Function calling Apr 2026 86.1% Fading
CounterFactQA (Qwen) Models get counterfactual questions and must recognize that the premise is false instead of giving a hallucinated answer. Hallucination rates May 2025 68.1% System cards only
LengthCtrl (Qwen) Models do creative writing tasks with length requirements. The score depends on the gap between output length and target length. Following precise instructions May 2025 73.5% System cards only
LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. Long videos Aug 2026 85.6% In use
MathVision-Wildphoto Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description. Visual math and diagrams Nov 2025 63.8% System cards only
QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. Knowledge in many languages Apr 2026 84% Fading
Template Following generalization (Qwen) Models follow system prompts and emit exactly formatted XML or JSON tool calls for five scaffolds derived from popular IDE and CLI coding agents. Function calling Feb 2026 92.7% System cards only
ThinkFollow (Qwen) Models hold multi-turn dialogues with randomly inserted think and no-think flags and must switch reasoning mode correctly. Multi-turn conversation May 2025 98.9% System cards only
ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. Function calling May 2025 85.5% System cards only
Work360 Agents complete professional tasks drawn from a map of 26 industries, 182 subfields and 721 specialties. Qwen in-house scoring (method unpublished). Tasks from real jobs Nov 2026 52.7 In use
Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. Speech recognition and translation Mar 2025 System cards only
Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. Jailbreaks and prompt injection Jul 2024 System cards only
E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. Running a business Aug 2026 14.3 Not in cards
OccuBench Agents do 100 professional task scenarios across 10 industries and 65 domains. Language world models simulate each domain's tools so tasks stay solvable. Tasks from real jobs Apr 2026 79.6% Not in cards
Qwen-Image-Bench Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images. Video generation May 2026 64.7 Not in cards
ProcessBench Models read step-by-step solutions to GSM8K, MATH, OlympiadBench and Omni-MATH problems and name the earliest wrong step, or say none is wrong. Scored by F1. Competition problems with answers Dec 2024 87.9% Not in cards
CodeElo Models solve Codeforces contest problems through the platform's judging system. The harness converts contest outcomes into human-comparable Elo ratings and percentiles. Competitive programming Jan 2025 1,578 Not in cards
Code Interpreter Benchmark General Models use a Python interpreter for general tasks. The score is the percentage of generated programs that execute. Function calling Sep 2023 65.5% Not in cards
EntireSongs Models transcribe complete English and Chinese songs with background music. Qwen reports transcription errors separately for each language. Speech recognition and translation Jan 2026 14.3% Not in cards
BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. General assistants Sep 2026 Not in cards
D2K-Bench Agents turn expert GPU-kernel designs into efficient Triton, CUDA or CUTLASS kernels across 26 tasks. Runtime timing and design-code scores measure implementation quality. Speed and optimization Oct 2026 Not in cards
Omnilingua-Bench Models handle long-audio transcription, multilingual long-audio question answering, multi-speaker attribution and complex spoken instructions. WER, diarization, accuracy and rubric pass rates score the tracks. Speech recognition and translation Sep 2026 Not in cards
Qwen-Music Cover Song Generation Models remake a reference melody using requested lyrics and style. The evaluation aligns the melodies and measures their average pitch difference in semitones. Audio understanding and voice Jul 2026 Not in cards
Qwen3 forced alignment benchmarks Models place word or character boundaries in audio. Qwen measures timestamp shifts against automatic and human reference alignments. Speech recognition and translation Jan 2026 Not in cards
Qwen3-ASR internal transcription benchmarks Models transcribe accented conversations, noisy speech, tongue twisters, and multilingual news. Qwen reports error rates for each recording condition. Speech recognition and translation Jan 2026 Not in cards

Arenas

Name Kind Released Best score Status
Qwen Chinese human evaluation Three annotators rank chat model answers to 300 Chinese instructions on knowledge, language, writing, math and coding. Results are win rates against GPT-3.5. Overall chat quality Sep 2023 30.7% System cards only
Qwen-Music Text-to-Music Generation Models turn English or Chinese lyrics and style instructions into complete songs. Music professionals compare anonymized recordings, and automatic judges score the same outputs. Audio understanding and voice Jul 2026 Not in cards