Qwen
We track 53 benchmarks and 2 arenas that Qwen made or helped make. Its models are on its lab page.
14 in use 20 with public code 31 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. | Competition problems with answers | Apr 2025 | | In use |
| DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. | General assistants | Jan 2026 | 58.9% | Fading |
| AutoLogi Models solve open-ended logic puzzles in English and Chinese. Generated verifier programs check each answer against the puzzle constraints. | Abstract puzzles and logic | Feb 2025 | 89.5% | Dropped |
| CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). | Tasks from real jobs | 75.9% | In use | |
| QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. | General assistants | Apr 2026 | 60.5% | Fading |
| Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Chat and instructions in many languages | Jul 2024 | | System cards only |
| Qwen in-house English automatic evaluation Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Other | Jul 2024 | | System cards only |
| Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. | Function calling | Aug 2023 | | Dropped |
| Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. | Chat and instructions in many languages | Sep 2024 | | Dropped |
| QwenWebBench Models generate front-end code for web apps, games, SVG, and other categories in English and Chinese. Multimodal judgments produce Elo ratings. | Web and app front ends | Apr 2026 | | Fading |
| RecreationBench Agents explore a running reference app and rebuild it, across 250 tasks on five platforms. Programmatic tests and a VLM judge score the result. | Desktop apps | Sep 2026 | 58.1% | In use |
| Code Interpreter Benchmark Models use a Python code interpreter for math, data visualization, and general tasks. Scoring covers result accuracy and the share of code that runs. | Function calling | Sep 2023 | | System cards only |
| Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. | Knowledge in many languages | Feb 2024 | | Dropped |
| Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. | Reasoning in many languages | Feb 2024 | | Dropped |
| AgentWorldBench Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions. | Interactive world models | Jun 2026 | 58.3 | In use |
| QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. | Web and app front ends | May 2026 | | Fading |
| Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. | Chat and instructions in many languages | Jun 2024 | 4.15 | Dropped |
| QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. | Fixing real repositories | 86.3% | In use | |
| QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. | Web and app front ends | May 2026 | 1,617 | Fading |
| QwenWorldBench Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. | Interactive world models | May 2026 | | Fading |
| Code Interpreter Benchmark Math Models use a Python interpreter to solve math problems. The score is execution accuracy on the Math task set. | Function calling | Sep 2023 | | Dropped |
| Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. | Secure code and review | Oct 2026 | 25.1% | In use |
| QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. | Web and app front ends | Aug 2026 | | In use |
| QwenVisualOffice Models handle visual office tasks in an internal Qwen test set. Qwen has not published the tasks or how they are scored. | Documents, charts and text in images | Aug 2026 | | In use |
| QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). | Fixing real repositories | 63.1% | In use | |
| QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. | Web and app front ends | 1,770 | In use | |
| QwenSVGBench Model writes SVG code for bilingual prompts. Rendered outputs judged by a multimodal model and ranked with Bradley-Terry Elo. | Web and app front ends | 1,758 | In use | |
| QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. | Web and app front ends | May 2026 | 1,884 | In use |
| ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. | Function calling | Apr 2026 | | Fading |
| CounterFactQA (Qwen) Models get counterfactual questions and must recognize that the premise is false instead of giving a hallucinated answer. | Hallucination rates | May 2025 | | System cards only |
| LengthCtrl (Qwen) Models do creative writing tasks with length requirements. The score depends on the gap between output length and target length. | Following precise instructions | May 2025 | | System cards only |
| LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. | Long videos | Aug 2026 | | In use |
| MathVision-Wildphoto Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description. | Visual math and diagrams | Nov 2025 | | System cards only |
| QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. | Knowledge in many languages | Apr 2026 | | Fading |
| Template Following generalization (Qwen) Models follow system prompts and emit exactly formatted XML or JSON tool calls for five scaffolds derived from popular IDE and CLI coding agents. | Function calling | Feb 2026 | | System cards only |
| ThinkFollow (Qwen) Models hold multi-turn dialogues with randomly inserted think and no-think flags and must switch reasoning mode correctly. | Multi-turn conversation | May 2025 | | System cards only |
| ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. | Function calling | May 2025 | | System cards only |
| Work360 Agents complete professional tasks drawn from a map of 26 industries, 182 subfields and 721 specialties. Qwen in-house scoring (method unpublished). | Tasks from real jobs | Nov 2026 | | In use |
| Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. | Speech recognition and translation | Mar 2025 | System cards only | |
| Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. | Jailbreaks and prompt injection | Jul 2024 | System cards only | |
| E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. | Running a business | Aug 2026 | 14.3 | Not in cards |
| OccuBench Agents do 100 professional task scenarios across 10 industries and 65 domains. Language world models simulate each domain's tools so tasks stay solvable. | Tasks from real jobs | Apr 2026 | 79.6% | Not in cards |
| Qwen-Image-Bench Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images. | Video generation | May 2026 | 64.7 | Not in cards |
| ProcessBench Models read step-by-step solutions to GSM8K, MATH, OlympiadBench and Omni-MATH problems and name the earliest wrong step, or say none is wrong. Scored by F1. | Competition problems with answers | Dec 2024 | 87.9% | Not in cards |
| CodeElo Models solve Codeforces contest problems through the platform's judging system. The harness converts contest outcomes into human-comparable Elo ratings and percentiles. | Competitive programming | Jan 2025 | 1,578 | Not in cards |
| Code Interpreter Benchmark General Models use a Python interpreter for general tasks. The score is the percentage of generated programs that execute. | Function calling | Sep 2023 | | Not in cards |
| EntireSongs Models transcribe complete English and Chinese songs with background music. Qwen reports transcription errors separately for each language. | Speech recognition and translation | Jan 2026 | | Not in cards |
| BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. | General assistants | Sep 2026 | Not in cards | |
| D2K-Bench Agents turn expert GPU-kernel designs into efficient Triton, CUDA or CUTLASS kernels across 26 tasks. Runtime timing and design-code scores measure implementation quality. | Speed and optimization | Oct 2026 | Not in cards | |
| Omnilingua-Bench Models handle long-audio transcription, multilingual long-audio question answering, multi-speaker attribution and complex spoken instructions. WER, diarization, accuracy and rubric pass rates score the tracks. | Speech recognition and translation | Sep 2026 | Not in cards | |
| Qwen-Music Cover Song Generation Models remake a reference melody using requested lyrics and style. The evaluation aligns the melodies and measures their average pitch difference in semitones. | Audio understanding and voice | Jul 2026 | Not in cards | |
| Qwen3 forced alignment benchmarks Models place word or character boundaries in audio. Qwen measures timestamp shifts against automatic and human reference alignments. | Speech recognition and translation | Jan 2026 | Not in cards | |
| Qwen3-ASR internal transcription benchmarks Models transcribe accented conversations, noisy speech, tongue twisters, and multilingual news. Qwen reports error rates for each recording condition. | Speech recognition and translation | Jan 2026 | Not in cards |
Arenas
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Qwen Chinese human evaluation Three annotators rank chat model answers to 300 Chinese instructions on knowledge, language, writing, math and coding. Results are win rates against GPT-3.5. | Overall chat quality | Sep 2023 | | System cards only |
| Qwen-Music Text-to-Music Generation Models turn English or Chinese lyrics and style instructions into complete songs. Music professionals compare anonymized recordings, and automatic judges score the same outputs. | Audio understanding and voice | Jul 2026 | Not in cards |