Benchmark Almanac

Z.ai

We track 42 benchmarks and 1 arena that Z.ai made or helped make. Its models are on its lab page.

5 in use 18 with public code 16 with a leaderboard here

Arena leaderboards

Top five in each of its 1 arena with three or more rated models, the most rated first.

Benchmarks

Name Kind Released Best score Status
LVBench Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. Long videos Jun 2024 91.7% In use
LongBench v2 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Reasoning over long documents Dec 2024 70.6% In use
AlignBench Models answer 683 real Chinese user queries across eight categories. A GPT-4 judge rates each answer from 1 to 10 against a reference. Overall chat quality Nov 2023 8.94 Dropped
MotionBench Models answer questions about fine-grained motion in video clips across six motion question types. Scoring is accuracy. Video understanding Jan 2025 75.2% Fading
Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Web and app front ends Mar 2026 82.3% In use
LongBench-Chat Models follow real user instructions over contexts of 10K to 100K tokens. GPT-4 scores each answer from 1 to 10 against references. Reasoning over long documents Jan 2024 8.76 Dropped
ComplexFuncBench Models make function calls for 1,000 travel booking style samples that need several steps, user constraints, implicit parameter reasoning, long parameter values or 128K context. Function calling Jan 2025 61% Dropped
NaturalCodeBench Models solve 402 Python and Java problems taken from real user queries to online coding services in 6 domains. Semi-automatically built tests check each solution. Function-level problems May 2024 52.5% Dropped
CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass. Fixing real repositories Feb 2026 25.8% Fading
CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file. Fixing real repositories Feb 2026 72.2% Fading
CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app. Web and app front ends Feb 2026 74.8% Fading
Novel logical reasoning problems expert evaluation (Z.ai) Human experts score model answers to new multi-step logic problems with a fixed scoring standard for each question. Abstract puzzles and logic Aug 2025 62.0 System cards only
CVTG-2K Image models render text in images across regions and languages. CVTG-2K reports normalized edit distance, CLIPScore and word accuracy. Image generation and editing Jan 2026 0.91 Fading
LongText-Bench Image models render long text in images. The benchmark reports separate English and Chinese scores. Image generation and editing Jan 2026 0.97 Fading
OneIG-Bench Image models generate images from prompts with English and Chinese evaluation splits. Image generation and editing Jan 2026 0.53 Fading
TIIF-Bench Image models generate short and long images scored on overall image quality. Image generation and editing Jan 2026 81.0 Fading
Z.ai Code Bench Zhipu in-house coding benchmark used for GLM-5.3. Reports pass rate and output tokens per task. Method unpublished. Fixing real repositories Aug 2026 39.5% In use
CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step. Fixing real repositories Feb 2026 52.3% System cards only
GLM-4 All Tools information seeking (Z.ai) Models use a web browser to find information for user questions. The ChatGLM report gives accuracy but does not describe the test set. Questions that need search Jan 2024 78.1% System cards only
Challenging translation human evaluation (Z.ai) People blindly score translations of 100 real cases with slang, nicknames and memes on a 0 to 3 scale. Translation Aug 2025 1.71 System cards only
IF-Badcase Models follow 450 multi-constraint instructions drawn from real user failure reports. A checklist checks each constraint. Following precise instructions Feb 2026 83.2% System cards only
ImageMining Multimodal agents retrieve and reason over image information for image mining tasks. Deep research Apr 2026 30.7% Fading
LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. Long videos Aug 2026 85.6% In use
MENT-SNS Models translate English and Chinese sentences with slang, wordplay and idioms. A GPT-4.1 judge compares each translation with a baseline. Translation Feb 2026 1013.0 System cards only
MMSearch Multimodal agents search and answer questions that require visual and textual evidence. Deep research Apr 2026 72.9% Fading
Slide generation human evaluation (Z.ai) People compare HTML slide decks from two models on content, layout and visual quality. Metric is overall win rate. Taste and style Feb 2026 67.5% System cards only
ToolCall-Badcase Models choose a tool and its arguments for 200 cases drawn from real user failure reports. Each case has a ground-truth call. Function calling Feb 2026 95.8% System cards only
Translation Human Scoring (Z.ai) People score model translations. The GLM-5 report does not describe the test set or the scale. Translation Feb 2026 3.12 System cards only
ZClawBench Agents do end-to-end OpenClaw workflows such as environment setup, development, information retrieval, data analysis and content creation, drawn from real OpenClaw use. General assistants Mar 2026 57.6% Fading
ZMultiDialBench Human annotators score model replies to 141 multilingual dialogue prompts on a 1 to 10 scale with category criteria. Chat and instructions in many languages Feb 2026 8.57 System cards only
ZMultiTransBench Models translate 1,220 real Chinese source texts into seven languages. A GPT-4.1 judge compares each translation with a fixed baseline. Translation Feb 2026 1050.0 System cards only
Real-scenario prompts human evaluation (Z.ai) One human evaluator scores replies to 660 curated real user prompts in English, Chinese and other languages on a 0 to 10 scale. Overall chat quality Aug 2025 System cards only
ComplexBench Models answer 1,150 hand-written Chinese instructions that compose several constraints by chaining, selection and nesting. LLM evaluators with rules check each constraint, and scores follow the dependency structure. Following precise instructions Jul 2024 80% Not in cards
Vision2Web S2-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Web and app front ends Mar 2026 43.5% Not in cards
Vision2Web S3-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Web and app front ends Jun 2026 74.6% Not in cards
Vision2Web S1-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Web and app front ends Mar 2026 52.7% Not in cards
VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. Web browsing Aug 2024 36.2% Not in cards
Terminal-Bench 2.0 Verified (Z.ai) Agents do the Terminal-Bench 2.0 tasks after Z.ai fixed environments for 89 tasks and instructions for 11. A further 2026-08-18 round fixed tests and instructions in 6 more tasks. Terminal tasks Feb 2026 61.8% Not in cards
AndroidLab Mobile agents complete 138 tasks across nine apps on predefined Android virtual devices, in text-only and multimodal modes. Checks on the final device state score each task. Phones Oct 2024 31.2% Not in cards
AISE-Bench Agents answer academic information-seeking questions through multi-step knowledge-graph APIs and source-grounded responses. Planning, execution and answer-grounding metrics score each trajectory. Deep research Jul 2026 Not in cards
DataSciBench Agents complete open-ended data-science tasks that require data cleaning, analysis, visualization and modeling. Component checks and overall task success score the work. Data science and SQL Apr 2026 Not in cards
Video-MonetBench Models rank two videos generated for the same prompt. The test compares each ranking with a human preference, including ties. Video understanding Dec 2024 Not in cards