Benchmark Almanac

Alibaba Group

We track 95 benchmarks and 1 arena that Alibaba Group made or helped make. Its models are on its lab page.

7 in use 62 with public code 62 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. Competition problems with answers Apr 2025 86.5% In use
WritingBench Models complete 1,000 real-world writing requests across 6 domains and 100 subdomains. A judge model scores each reply on five query-specific criteria. Creative writing Mar 2025 83.9 Fading
Chinese SimpleQA 3,000 short Chinese factual questions across six topics. Graded correct, incorrect or not attempted. Short factual questions Nov 2024 86% Fading
CC-OCR Models read text in 7,058 images across four tracks: scene text, multilingual text, document parsing, and key information extraction. Each track has its own metric. Documents, charts and text in images Dec 2024 83.4% Fading
Bird-SQL Models convert natural language questions into SQL over 95 large real databases across 37 domains. Scored by execution accuracy. Data science and SQL May 2023 72.5% Fading
NOVA-63 Models answer questions written by native speakers in 14 languages across 63 academic disciplines. Scoring is accuracy. Knowledge in many languages Nov 2025 59.1% Fading
DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. General assistants Jan 2026 58.9% Fading
AutoLogi Models solve open-ended logic puzzles in English and Chinese. Generated verifier programs check each answer against the puzzle constraints. Abstract puzzles and logic Feb 2025 89.5% Dropped
MLogiQA Models answer LogiQA logical reasoning questions translated into 10 languages, 80 per language. Scored by accuracy. Reasoning in many languages Nov 2024 77.1% Dropped
QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. General assistants Apr 2026 60.5% Fading
Omni-MATH Models solve 4,428 Olympiad-level competition math problems across 33 subdomains and 10 difficulty levels. A GPT-4 or Omni-Judge grader checks answers. Competition problems with answers Oct 2024 60.5% Dropped
API-Bank Models make API calls in dialogues that need tools such as calendars and search. Scored on call accuracy. Function calling Apr 2023 92.3% System cards only
Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Chat and instructions in many languages Jul 2024 69.6% System cards only
Qwen in-house English automatic evaluation Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Other Jul 2024 72.9% System cards only
HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. General visual questions Aug 2025 16% Fading
Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. Function calling Aug 2023 98% Dropped
MobileWorld Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. Phones Dec 2025 55.3% In use
BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. Deep research Aug 2025 59% Fading
Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. Reasoning in many languages Feb 2024 89.6% Dropped
AgentWorldBench Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions. Interactive world models Jun 2026 58.3 In use
QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. Web and app front ends May 2026 1,608 Fading
SecCodeBench Models write or patch functions in project scaffolds drawn from real Alibaba vulnerabilities. Code must pass functional tests, then proof-of-concept security tests. Secure code and review Jul 2025 71.1% Fading
Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. Chat and instructions in many languages Jun 2024 4.15 Dropped
QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. Web and app front ends May 2026 1,617 Fading
QwenWorldBench Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. Interactive world models May 2026 62.1% Fading
Math401 Models compute 401 arithmetic expressions covering addition, multiplication, exponents, logarithms and trigonometry. Scored by exact numeric accuracy. Word problems Apr 2023 70.1% System cards only
CV3-Eval Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality. Speech recognition and translation May 2025 System cards only
M3GIA Models answer cognition test questions built on five Cattell-Horn-Carroll intelligence factors in English, Chinese, French, Spanish, Portuguese and Korean. Data comes from native contexts. General visual questions Jun 2024 62.4% System cards only
Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. Secure code and review Oct 2026 25.1% In use
MT-Bench-101 Models hold 1,388 multi-turn dialogues with 4,208 turns across 13 tasks from a three-tier ability taxonomy. GPT-4 scores each turn from 1 to 10. Multi-turn conversation Feb 2024 7.82 Dropped
QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. Web and app front ends Aug 2026 69.9% In use
QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. Web and app front ends May 2026 1,884 In use
ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. Function calling Apr 2026 86.1% Fading
AIR-Bench (Audio Instruction Benchmark) Audio language models answer questions about speech, natural sounds and music. A foundation set has 19 tasks with about 19k single-choice questions, and a chat set has 2k open-ended questions scored by GPT-4. Audio understanding and voice Feb 2024 6.98 System cards only
AIR-Bench Foundation Models answer foundational audio understanding questions. The score is the official average accuracy across nineteen tasks. Audio understanding and voice Feb 2024 67.3% In use
CorpusQA Corpus-level questions over hundreds of documents up to 10M tokens needing aggregation and statistics. Programmatically guaranteed answers, accuracy. Reasoning over long documents Dec 2025 82% System cards only
CounterFactQA (Qwen) Models get counterfactual questions and must recognize that the premise is false instead of giving a hallucinated answer. Hallucination rates May 2025 68.1% System cards only
LengthCtrl (Qwen) Models do creative writing tasks with length requirements. The score depends on the gap between output length and target length. Following precise instructions May 2025 73.5% System cards only
QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. Knowledge in many languages Apr 2026 84% Fading
ThinkFollow (Qwen) Models hold multi-turn dialogues with randomly inserted think and no-think flags and must switch reasoning mode correctly. Multi-turn conversation May 2025 98.9% System cards only
ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. Function calling May 2025 85.5% System cards only
M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. Knowledge in many languages Jun 2023 Dropped
Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. Jailbreaks and prompt injection Jul 2024 System cards only
SecRespond Agents investigate compromised-host snapshots and write incident analyses and remediation plans. A model evaluator applies expert checklists separately to detection and response planning. Finding and exploiting real bugs Jul 2026 79% Not in cards
Terminal-Bench Pro Agents solve 400 terminal tasks across eight domains. Harbor distributes the 200 public tasks. Alibaba keeps the other 200 for private evaluation. Terminal tasks Dec 2025 51% Not in cards
MobilePA-Bench Agents complete 1,705 mobile planning tasks using tools, persistent memory, reusable skills, and specialist agents. Task-specific checks evaluate tool calls, state changes, and interaction behavior. Additional gates check required memory access and skill loading. General assistants Aug 2026 76.9% Not in cards
E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. Running a business Aug 2026 14.3 Not in cards
S-Eval Models respond to Chinese and English harmful prompts and attack variants. A trained safety judge measures the safe-response fraction and attack success rate. Refusing harmful requests May 2024 88.8% Not in cards
IndustryBench Models answer 2,049 Chinese industrial procurement questions grounded in national standards and product records. Answers must respect regulated thresholds and safety clauses. Other professions May 2026 69.4% Not in cards
SPM-Bench Models interpret microscopy images and select all correct answers. The primary score gives partial credit and penalizes incorrect selections. Lab and data work Feb 2026 0.94 Not in cards
TeachArena Agents make teaching decisions, tutor students through dialogue, and carry out learning management system tasks. The evaluator checks conversation turns, artifacts, and final system states. Other professions May 2026 0.80 Not in cards
OfficePrism Agents resolve Chinese workplace tickets using one business system at a time. The evaluator checks outcomes across 420 tasks in seven business functions. Documents and spreadsheets Nov 2026 76.4% Not in cards
BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. CAD models Oct 2026 79.9 Not in cards
MTHv2 Models read Chinese Buddhist texts from scanned historical pages. The authors annotate characters, line transcriptions, reading order, and boundary lines. Documents, charts and text in images Jun 2020 86.4% Not in cards
ProcessBench Models read step-by-step solutions to GSM8K, MATH, OlympiadBench and Omni-MATH problems and name the earliest wrong step, or say none is wrong. Scored by F1. Competition problems with answers Dec 2024 87.9% Not in cards
ClinHallu Models answer medical image questions and produce intermediate reasoning. The evaluation reports answer accuracy and separate hallucination rates for visual recognition, knowledge recall, and reasoning integration against annotated reasoning traces. Medicine Jun 2026 80.1% Not in cards
CodeElo Models solve Codeforces contest problems through the platform's judging system. The harness converts contest outcomes into human-comparable Elo ratings and percentiles. Competitive programming Jan 2025 1,578 Not in cards
Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. Video understanding Oct 2026 20.2% Not in cards
OSWorld-MCP Computer-use agents complete OSWorld desktop tasks while they can also call 158 MCP tools across 7 common applications. The benchmark checks whether agents pick a tool call or GUI actions at the right time. It reports task accuracy, tool invocation rate and average completion steps. Desktop apps Oct 2025 56.6% Not in cards
CFLUE Models answer Chinese financial questions in two parts. The knowledge part has more than 38,000 multiple-choice questions from financial qualification exams, with explanations to test reasoning. The application part has more than 16,000 test items for classification, translation, relation extraction, reading comprehension and text generation. Finance May 2024 53.8% Not in cards
CFLUE Knowledge Models answer Chinese financial qualification questions. The score is answer accuracy across the knowledge test. Finance May 2024 53.8% Not in cards
MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. Simulated companies Jul 2026 59.5 Not in cards
PRISM-Bench (image generation) Models generate images from English or Chinese prompts. Vision-language judges score visual appeal and prompt alignment across seven tracks, including composition, lettering, and style. Image generation and editing Sep 2025 86.3 Not in cards
OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. Embodied reasoning Sep 2026 43.9% Not in cards
Atrex-Bench Agents implement GPU kernels for 30 operators and 440 shapes sampled from production inference workloads. The evaluator checks compilation and numerical correctness, then scores performance against hardware roofline ceilings with weights from production GPU usage. GPU kernels Jun 2026 0.11 Not in cards
PCA-Bench Models interpret visual scenes, explain their reasoning, and select actions for embodied tasks. GPT-4-0125 compares perception concepts, reasoning, and actions with annotated references. Embodied reasoning Feb 2024 0.70 Not in cards
AACR-Bench Models review 200 real pull requests in many languages with full repository context. Their review comments are matched against expert-annotated issues. Fixing real repositories Jan 2026 16.1% Not in cards
CutCraft Benchmark Models generate multi-shot videos under editing instructions. The evaluator checks transitions, shot structure, and the timing of audio relative to cuts. Video generation Sep 2026 0.64 Not in cards
Omni-InteractScope Explicit Request Models answer explicit requests during audio and video streams. The score is the published average across tasks T1 through T7. Video understanding Oct 2026 20.2% Not in cards
WebWalkerQA Agents start from a website's home page and traverse its subpages to find the facts that answer a question. Answers are judged against references. Deep research Jan 2025 33.3% Not in cards
CFLUE Application Models complete Chinese financial language tasks. The score is the published average of the application task metrics. Finance May 2024 44.2 Not in cards
HappyWorld Embodied The HappyWorld arena publishes a separate embodied vote pool and Elo ranking. Interactive world models Sep 2026 1,074 Not in cards
Omni-InteractScope Proactive Models provide timely proactive assistance during audio and video streams. The score is the published T8 assistance recall. Video understanding Oct 2026 46.4% Not in cards
Omnilingua-AudioMC Models follow requirements across spoken conversations in ten languages. The authors compare each model's spoken and text responses against the same 1,404 rubric items. Speech recognition and translation Nov 2026 64% Not in cards
VidScribe Models write, copy, preserve, or edit visible text in videos. The evaluator tracks spelling, glyph shapes, placement, and stability across frames. Video generation Sep 2026 79.0 Not in cards
Qwen-Audio-Agent Smart Cockpit Benchmark Agents operate simulated vehicle, navigation, music, and weather services. The short suite checks whole cases. The long suite checks tool use on each turn. Function calling Sep 2026 98.8% Not in cards
HappyWorld Spatial Reconstruction The HappyWorld arena publishes a separate spatial reconstruction vote pool and Elo ranking. Interactive world models Sep 2026 1,252 Not in cards
LifeChoice Models select the decision a character makes in a novel. The authors score the percentage of correct choices. Reading comprehension Apr 2024 65.3% Not in cards
NarrLV Models generate videos from prompts that describe changes in a scene, an object, or an action. A multimodal judge checks what appears and in what order. Video generation May 2025 67.1 Not in cards
StateBench (video continuation) Models continue a video while tracking what happened in earlier segments. A judge checks whether visible objects, hidden processes, and later state changes remain consistent. Video generation Sep 2026 32.5 Not in cards
Alibaba CTF Agents solve 87 capture-the-flag tasks in Alibaba’s collection of competition challenges from 2023 through 2026. Capture-the-flag challenges Aug 2026 Not in cards
AliMeeting Models identify speakers in Mandarin meetings. NVIDIA compares diarization error using headset mixes and distant microphones. Audio understanding and voice Feb 2022 Not in cards
AndroidWorld-Knowledge Agents attempt 48 AndroidWorld tasks across five apps. Each app provides guidance for its simplest task, and harder tasks test whether the agent can reuse it. Phones May 2025 Not in cards
BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. General assistants Sep 2026 Not in cards
CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. Running a business Aug 2026 Not in cards
D2K-Bench Agents turn expert GPU-kernel designs into efficient Triton, CUDA or CUTLASS kernels across 26 tasks. Runtime timing and design-code scores measure implementation quality. Speed and optimization Oct 2026 Not in cards
HappyWorld-Bench World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them. Interactive world models Sep 2026 Not in cards
Mobile-Eval Agents follow 33 instructions on Android. Some requests specify the app. Others describe a goal and leave the choice of app to the agent. Phones Jan 2024 Not in cards
Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. Phones Jan 2025 Not in cards
Mobile-Eval-v2 Agents follow basic and advanced instructions on real phones. The authors give 44 instructions in English on Android and 44 in Chinese on HarmonyOS. Phones Jun 2024 Not in cards
Omnilingua-Bench Models handle long-audio transcription, multilingual long-audio question answering, multi-speaker attribution and complex spoken instructions. WER, diarization, accuracy and rubric pass rates score the tracks. Speech recognition and translation Sep 2026 Not in cards
PPR10K Models retouch groups of portraits against three expert retouchers’ Adobe Camera Raw edits. The test measures accuracy on people and color consistency across each group. Image generation and editing Mar 2021 Not in cards
Qwen-Audio-Agent In-House Cockpit Benchmark Agents handle requests with cockpit tools. The evaluation measures whether they complete each task when they call tools directly, delegate, or choose between both routes. Function calling Sep 2026 Not in cards
ReasonVOS Models segment and track objects in videos from explicit referring queries and implicit reasoning queries. The score is overall J&F on ReVOS. Video understanding Jul 2024 Not in cards
WorldOlympiad Models generate long videos for gaming, robotics, and everyday scenes. The evaluator checks physical rules, reconstructed geometry, and instruction following across video segments. Video world models Jun 2026 Not in cards

Arenas

Name Kind Released Best score Status
Qwen Chinese human evaluation Three annotators rank chat model answers to 300 Chinese instructions on knowledge, language, writing, math and coding. Results are win rates against GPT-3.5. Overall chat quality Sep 2023 30.7% System cards only