Alibaba Group
We track 95 benchmarks and 1 arena that Alibaba Group made or helped make. Its models are on its lab page.
7 in use 62 with public code 62 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. | Competition problems with answers | Apr 2025 | | In use |
| WritingBench Models complete 1,000 real-world writing requests across 6 domains and 100 subdomains. A judge model scores each reply on five query-specific criteria. | Creative writing | Mar 2025 | 83.9 | Fading |
| Chinese SimpleQA 3,000 short Chinese factual questions across six topics. Graded correct, incorrect or not attempted. | Short factual questions | Nov 2024 | | Fading |
| CC-OCR Models read text in 7,058 images across four tracks: scene text, multilingual text, document parsing, and key information extraction. Each track has its own metric. | Documents, charts and text in images | Dec 2024 | | Fading |
| Bird-SQL Models convert natural language questions into SQL over 95 large real databases across 37 domains. Scored by execution accuracy. | Data science and SQL | May 2023 | 72.5% | Fading |
| NOVA-63 Models answer questions written by native speakers in 14 languages across 63 academic disciplines. Scoring is accuracy. | Knowledge in many languages | Nov 2025 | | Fading |
| DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. | General assistants | Jan 2026 | 58.9% | Fading |
| AutoLogi Models solve open-ended logic puzzles in English and Chinese. Generated verifier programs check each answer against the puzzle constraints. | Abstract puzzles and logic | Feb 2025 | 89.5% | Dropped |
| MLogiQA Models answer LogiQA logical reasoning questions translated into 10 languages, 80 per language. Scored by accuracy. | Reasoning in many languages | Nov 2024 | | Dropped |
| QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. | General assistants | Apr 2026 | 60.5% | Fading |
| Omni-MATH Models solve 4,428 Olympiad-level competition math problems across 33 subdomains and 10 difficulty levels. A GPT-4 or Omni-Judge grader checks answers. | Competition problems with answers | Oct 2024 | 60.5% | Dropped |
| API-Bank Models make API calls in dialogues that need tools such as calendars and search. Scored on call accuracy. | Function calling | Apr 2023 | | System cards only |
| Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Chat and instructions in many languages | Jul 2024 | | System cards only |
| Qwen in-house English automatic evaluation Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Other | Jul 2024 | | System cards only |
| HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. | General visual questions | Aug 2025 | 16% | Fading |
| Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. | Function calling | Aug 2023 | | Dropped |
| MobileWorld Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. | Phones | Dec 2025 | 55.3% | In use |
| BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. | Deep research | Aug 2025 | | Fading |
| Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. | Reasoning in many languages | Feb 2024 | | Dropped |
| AgentWorldBench Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions. | Interactive world models | Jun 2026 | 58.3 | In use |
| QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. | Web and app front ends | May 2026 | | Fading |
| SecCodeBench Models write or patch functions in project scaffolds drawn from real Alibaba vulnerabilities. Code must pass functional tests, then proof-of-concept security tests. | Secure code and review | Jul 2025 | | Fading |
| Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. | Chat and instructions in many languages | Jun 2024 | 4.15 | Dropped |
| QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. | Web and app front ends | May 2026 | 1,617 | Fading |
| QwenWorldBench Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. | Interactive world models | May 2026 | | Fading |
| Math401 Models compute 401 arithmetic expressions covering addition, multiplication, exponents, logarithms and trigonometry. Scored by exact numeric accuracy. | Word problems | Apr 2023 | | System cards only |
| CV3-Eval Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality. | Speech recognition and translation | May 2025 | System cards only | |
| M3GIA Models answer cognition test questions built on five Cattell-Horn-Carroll intelligence factors in English, Chinese, French, Spanish, Portuguese and Korean. Data comes from native contexts. | General visual questions | Jun 2024 | | System cards only |
| Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. | Secure code and review | Oct 2026 | 25.1% | In use |
| MT-Bench-101 Models hold 1,388 multi-turn dialogues with 4,208 turns across 13 tasks from a three-tier ability taxonomy. GPT-4 scores each turn from 1 to 10. | Multi-turn conversation | Feb 2024 | | Dropped |
| QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. | Web and app front ends | Aug 2026 | | In use |
| QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. | Web and app front ends | May 2026 | 1,884 | In use |
| ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. | Function calling | Apr 2026 | | Fading |
| AIR-Bench (Audio Instruction Benchmark) Audio language models answer questions about speech, natural sounds and music. A foundation set has 19 tasks with about 19k single-choice questions, and a chat set has 2k open-ended questions scored by GPT-4. | Audio understanding and voice | Feb 2024 | | System cards only |
| AIR-Bench Foundation Models answer foundational audio understanding questions. The score is the official average accuracy across nineteen tasks. | Audio understanding and voice | Feb 2024 | | In use |
| CorpusQA Corpus-level questions over hundreds of documents up to 10M tokens needing aggregation and statistics. Programmatically guaranteed answers, accuracy. | Reasoning over long documents | Dec 2025 | | System cards only |
| CounterFactQA (Qwen) Models get counterfactual questions and must recognize that the premise is false instead of giving a hallucinated answer. | Hallucination rates | May 2025 | | System cards only |
| LengthCtrl (Qwen) Models do creative writing tasks with length requirements. The score depends on the gap between output length and target length. | Following precise instructions | May 2025 | | System cards only |
| QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. | Knowledge in many languages | Apr 2026 | | Fading |
| ThinkFollow (Qwen) Models hold multi-turn dialogues with randomly inserted think and no-think flags and must switch reasoning mode correctly. | Multi-turn conversation | May 2025 | | System cards only |
| ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. | Function calling | May 2025 | | System cards only |
| M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. | Knowledge in many languages | Jun 2023 | Dropped | |
| Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. | Jailbreaks and prompt injection | Jul 2024 | System cards only | |
| SecRespond Agents investigate compromised-host snapshots and write incident analyses and remediation plans. A model evaluator applies expert checklists separately to detection and response planning. | Finding and exploiting real bugs | Jul 2026 | 79% | Not in cards |
| Terminal-Bench Pro Agents solve 400 terminal tasks across eight domains. Harbor distributes the 200 public tasks. Alibaba keeps the other 200 for private evaluation. | Terminal tasks | Dec 2025 | 51% | Not in cards |
| MobilePA-Bench Agents complete 1,705 mobile planning tasks using tools, persistent memory, reusable skills, and specialist agents. Task-specific checks evaluate tool calls, state changes, and interaction behavior. Additional gates check required memory access and skill loading. | General assistants | Aug 2026 | 76.9% | Not in cards |
| E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. | Running a business | Aug 2026 | 14.3 | Not in cards |
| S-Eval Models respond to Chinese and English harmful prompts and attack variants. A trained safety judge measures the safe-response fraction and attack success rate. | Refusing harmful requests | May 2024 | 88.8% | Not in cards |
| IndustryBench Models answer 2,049 Chinese industrial procurement questions grounded in national standards and product records. Answers must respect regulated thresholds and safety clauses. | Other professions | May 2026 | | Not in cards |
| SPM-Bench Models interpret microscopy images and select all correct answers. The primary score gives partial credit and penalizes incorrect selections. | Lab and data work | Feb 2026 | 0.94 | Not in cards |
| TeachArena Agents make teaching decisions, tutor students through dialogue, and carry out learning management system tasks. The evaluator checks conversation turns, artifacts, and final system states. | Other professions | May 2026 | 0.80 | Not in cards |
| OfficePrism Agents resolve Chinese workplace tickets using one business system at a time. The evaluator checks outcomes across 420 tasks in seven business functions. | Documents and spreadsheets | Nov 2026 | 76.4% | Not in cards |
| BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. | CAD models | Oct 2026 | 79.9 | Not in cards |
| MTHv2 Models read Chinese Buddhist texts from scanned historical pages. The authors annotate characters, line transcriptions, reading order, and boundary lines. | Documents, charts and text in images | Jun 2020 | 86.4% | Not in cards |
| ProcessBench Models read step-by-step solutions to GSM8K, MATH, OlympiadBench and Omni-MATH problems and name the earliest wrong step, or say none is wrong. Scored by F1. | Competition problems with answers | Dec 2024 | 87.9% | Not in cards |
| ClinHallu Models answer medical image questions and produce intermediate reasoning. The evaluation reports answer accuracy and separate hallucination rates for visual recognition, knowledge recall, and reasoning integration against annotated reasoning traces. | Medicine | Jun 2026 | | Not in cards |
| CodeElo Models solve Codeforces contest problems through the platform's judging system. The harness converts contest outcomes into human-comparable Elo ratings and percentiles. | Competitive programming | Jan 2025 | 1,578 | Not in cards |
| Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. | Video understanding | Oct 2026 | | Not in cards |
| OSWorld-MCP Computer-use agents complete OSWorld desktop tasks while they can also call 158 MCP tools across 7 common applications. The benchmark checks whether agents pick a tool call or GUI actions at the right time. It reports task accuracy, tool invocation rate and average completion steps. | Desktop apps | Oct 2025 | 56.6% | Not in cards |
| CFLUE Models answer Chinese financial questions in two parts. The knowledge part has more than 38,000 multiple-choice questions from financial qualification exams, with explanations to test reasoning. The application part has more than 16,000 test items for classification, translation, relation extraction, reading comprehension and text generation. | Finance | May 2024 | | Not in cards |
| CFLUE Knowledge Models answer Chinese financial qualification questions. The score is answer accuracy across the knowledge test. | Finance | May 2024 | | Not in cards |
| MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. | Simulated companies | Jul 2026 | | Not in cards |
| PRISM-Bench (image generation) Models generate images from English or Chinese prompts. Vision-language judges score visual appeal and prompt alignment across seven tracks, including composition, lettering, and style. | Image generation and editing | Sep 2025 | 86.3 | Not in cards |
| OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. | Embodied reasoning | Sep 2026 | | Not in cards |
| Atrex-Bench Agents implement GPU kernels for 30 operators and 440 shapes sampled from production inference workloads. The evaluator checks compilation and numerical correctness, then scores performance against hardware roofline ceilings with weights from production GPU usage. | GPU kernels | Jun 2026 | 0.11 | Not in cards |
| PCA-Bench Models interpret visual scenes, explain their reasoning, and select actions for embodied tasks. GPT-4-0125 compares perception concepts, reasoning, and actions with annotated references. | Embodied reasoning | Feb 2024 | 0.70 | Not in cards |
| AACR-Bench Models review 200 real pull requests in many languages with full repository context. Their review comments are matched against expert-annotated issues. | Fixing real repositories | Jan 2026 | 16.1% | Not in cards |
| CutCraft Benchmark Models generate multi-shot videos under editing instructions. The evaluator checks transitions, shot structure, and the timing of audio relative to cuts. | Video generation | Sep 2026 | | Not in cards |
| Omni-InteractScope Explicit Request Models answer explicit requests during audio and video streams. The score is the published average across tasks T1 through T7. | Video understanding | Oct 2026 | | Not in cards |
| WebWalkerQA Agents start from a website's home page and traverse its subpages to find the facts that answer a question. Answers are judged against references. | Deep research | Jan 2025 | | Not in cards |
| CFLUE Application Models complete Chinese financial language tasks. The score is the published average of the application task metrics. | Finance | May 2024 | | Not in cards |
| HappyWorld Embodied The HappyWorld arena publishes a separate embodied vote pool and Elo ranking. | Interactive world models | Sep 2026 | | Not in cards |
| Omni-InteractScope Proactive Models provide timely proactive assistance during audio and video streams. The score is the published T8 assistance recall. | Video understanding | Oct 2026 | | Not in cards |
| Omnilingua-AudioMC Models follow requirements across spoken conversations in ten languages. The authors compare each model's spoken and text responses against the same 1,404 rubric items. | Speech recognition and translation | Nov 2026 | | Not in cards |
| VidScribe Models write, copy, preserve, or edit visible text in videos. The evaluator tracks spelling, glyph shapes, placement, and stability across frames. | Video generation | Sep 2026 | | Not in cards |
| Qwen-Audio-Agent Smart Cockpit Benchmark Agents operate simulated vehicle, navigation, music, and weather services. The short suite checks whole cases. The long suite checks tool use on each turn. | Function calling | Sep 2026 | | Not in cards |
| HappyWorld Spatial Reconstruction The HappyWorld arena publishes a separate spatial reconstruction vote pool and Elo ranking. | Interactive world models | Sep 2026 | 1,252 | Not in cards |
| LifeChoice Models select the decision a character makes in a novel. The authors score the percentage of correct choices. | Reading comprehension | Apr 2024 | 65.3% | Not in cards |
| NarrLV Models generate videos from prompts that describe changes in a scene, an object, or an action. A multimodal judge checks what appears and in what order. | Video generation | May 2025 | | Not in cards |
| StateBench (video continuation) Models continue a video while tracking what happened in earlier segments. A judge checks whether visible objects, hidden processes, and later state changes remain consistent. | Video generation | Sep 2026 | | Not in cards |
| Alibaba CTF Agents solve 87 capture-the-flag tasks in Alibaba’s collection of competition challenges from 2023 through 2026. | Capture-the-flag challenges | Aug 2026 | Not in cards | |
| AliMeeting Models identify speakers in Mandarin meetings. NVIDIA compares diarization error using headset mixes and distant microphones. | Audio understanding and voice | Feb 2022 | Not in cards | |
| AndroidWorld-Knowledge Agents attempt 48 AndroidWorld tasks across five apps. Each app provides guidance for its simplest task, and harder tasks test whether the agent can reuse it. | Phones | May 2025 | Not in cards | |
| BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. | General assistants | Sep 2026 | Not in cards | |
| CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. | Running a business | Aug 2026 | Not in cards | |
| D2K-Bench Agents turn expert GPU-kernel designs into efficient Triton, CUDA or CUTLASS kernels across 26 tasks. Runtime timing and design-code scores measure implementation quality. | Speed and optimization | Oct 2026 | Not in cards | |
| HappyWorld-Bench World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them. | Interactive world models | Sep 2026 | Not in cards | |
| Mobile-Eval Agents follow 33 instructions on Android. Some requests specify the app. Others describe a goal and leave the choice of app to the agent. | Phones | Jan 2024 | Not in cards | |
| Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. | Phones | Jan 2025 | Not in cards | |
| Mobile-Eval-v2 Agents follow basic and advanced instructions on real phones. The authors give 44 instructions in English on Android and 44 in Chinese on HarmonyOS. | Phones | Jun 2024 | Not in cards | |
| Omnilingua-Bench Models handle long-audio transcription, multilingual long-audio question answering, multi-speaker attribution and complex spoken instructions. WER, diarization, accuracy and rubric pass rates score the tracks. | Speech recognition and translation | Sep 2026 | Not in cards | |
| PPR10K Models retouch groups of portraits against three expert retouchers’ Adobe Camera Raw edits. The test measures accuracy on people and color consistency across each group. | Image generation and editing | Mar 2021 | Not in cards | |
| Qwen-Audio-Agent In-House Cockpit Benchmark Agents handle requests with cockpit tools. The evaluation measures whether they complete each task when they call tools directly, delegate, or choose between both routes. | Function calling | Sep 2026 | Not in cards | |
| ReasonVOS Models segment and track objects in videos from explicit referring queries and implicit reasoning queries. The score is overall J&F on ReVOS. | Video understanding | Jul 2024 | Not in cards | |
| WorldOlympiad Models generate long videos for gaming, robotics, and everyday scenes. The evaluator checks physical rules, reconstructed geometry, and instruction following across video segments. | Video world models | Jun 2026 | Not in cards |
Arenas
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Qwen Chinese human evaluation Three annotators rank chat model answers to 300 Chinese instructions on knowledge, language, writing, math and coding. Results are win rates against GPT-3.5. | Overall chat quality | Sep 2023 | | System cards only |