Benchmark Almanac

Tsinghua University

We track 64 benchmarks that Tsinghua University made or helped make. Its models are on its lab page.

3 in use 59 with public code 36 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
MedXpertQA-MM Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. Medicine Jan 2025 80.4% In use
OJBench Models write C++ or Python solutions to 232 competition problems from NOI and ICPC. Judge test cases score each solution by pass@1. Competitive programming Jun 2025 70% Fading
AlignBench Models answer 683 real Chinese user queries across eight categories. A GPT-4 judge rates each answer from 1 to 10 against a reference. Overall chat quality Nov 2023 8.94 Dropped
MotionBench Models answer questions about fine-grained motion in video clips across six motion question types. Scoring is accuracy. Video understanding Jan 2025 75.2% Fading
LV-Eval Models answer single-hop and multi-hop questions over contexts at five length levels from 16K to 256K tokens, with confusing facts inserted. Reasoning over long documents Feb 2024 49.8% Dropped
LongBench-Chat Models follow real user instructions over contexts of 10K to 100K tokens. GPT-4 scores each answer from 1 to 10 against references. Reasoning over long documents Jan 2024 8.76 Dropped
OlympiadBench Models solve bilingual olympiad-level math and physics problems in Chinese and English, many with images. Answers are checked against a key. Competition problems with answers Feb 2024 18% Dropped
ComplexFuncBench Models make function calls for 1,000 travel booking style samples that need several steps, user constraints, implicit parameter reasoning, long parameter values or 128K context. Function calling Jan 2025 61% Dropped
NaturalCodeBench Models solve 402 Python and Java problems taken from real user queries to online coding services in 6 domains. Semi-automatically built tests check each solution. Function-level problems May 2024 52.5% Dropped
MedXpertQA Models answer 4,460 expert-level medical exam questions across 17 specialties, in a text subset and an image subset. Scoring is multiple-choice accuracy. Medicine Jan 2025 52.6% Fading
AgentIF Models follow long system prompts from 50 real agentic applications, each with many constraints. Scored by constraint and instruction success rates. Following precise instructions May 2025 59.8% In use
CCPM Models pick which of four classical Chinese poetry lines matches a given modern Chinese translation. Scored by accuracy. Reading in many languages Jun 2021 92% Dropped
OpenRCA Agents find the root cause of 335 software failures from enterprise telemetry logs, metrics and traces. Scored on component, start time and reason. Data science and SQL 34.9% System cards only
SafetyBench Models answer 11,435 multiple choice questions on seven safety topics in Chinese and English, such as ethics, illegal activities and privacy. Refusing harmful requests Sep 2023 89.9% System cards only
ChartMimic Multimodal models write plotting code that reproduces 1,000 charts from scientific papers. Scores compare the rendered chart with the original. Web and app front ends Jun 2024 60.3 System cards only
ViVerBench Models judge whether visual outcomes in multimodal reasoning are correct, across 16 task categories such as boxes, points, counts, mazes, GUIs, charts and LaTeX. General visual questions Oct 2025 74.4% System cards only
CHID Models fill blanks in Chinese passages by choosing the right idiom from candidates. Scored by accuracy. Reading in many languages Jun 2019 72.6% System cards only
LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. Long videos Aug 2026 85.6% In use
SciTS Models understand and generate scientific time series in 43 tasks across 12 domains. Signals range from one to ten million points. Each task uses its own metric. Research science Sep 2025 System cards only
ElephantBench Models answer 1,094 closed-book questions about facts with multiple verified but conflicting accounts. A model judge assigns complete, partial, or failed recall. Short factual questions Aug 2026 52.4% Not in cards
AgentIF (2026-10 setting) Models follow the same AgentIF instructions in thinking and non-thinking modes with vendor decoding settings. A gpt-5.6-luna judge scores 3 runs per model. Following precise instructions Oct 2026 67.4% Not in cards
VideoGAIA Agents inspect selected video segments, search the web, and read pages before answering questions. Model judges check answers, and the harness reports accuracy by category and difficulty. Video understanding Aug 2026 58.3% Not in cards
WebVR Models recreate a webpage from a screen recording and supplied assets. A model judge scores the rendered page against visual and interaction criteria. Web and app front ends Mar 2026 79.1 Not in cards
ReSpaceBench Models recognize rooms across separate visits and track objects that move between visits. The authors test 32 models on synthetic and real homes. Embodied reasoning Nov 2026 50.5% Not in cards
Accelerate Bench Coding agents speed up 32 real scientific applications, such as GROMACS and OpenFOAM, on a fixed CPU allocation within 8 hours. A hidden verifier checks correctness, and Score@1 averages the share of runtime saved, with incorrect programs scoring zero. Speed and optimization Nov 2026 67.4% Not in cards
EvoCode-Bench Coding agents keep one project working across 26 tasks of 5 to 15 rounds each, while requirements change, pile up and conflict. Cumulative hidden tests check every round in a persistent container. Building whole projects May 2026 54% Not in cards
EvoCode-Bench v2 Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit. Building whole projects Jun 2026 59.1% Not in cards
ALIVE-Bench Agents interpret social cues and act in simulated conversations. The evaluator tracks what agents perceive, what they infer, and how their actions affect other characters. Simulated robots Nov 2026 80.4% Not in cards
ComplexBench Models answer 1,150 hand-written Chinese instructions that compose several constraints by chaining, selection and nesting. LLM evaluators with rules check each constraint, and scores follow the dependency structure. Following precise instructions Jul 2024 80% Not in cards
DeskCraft Desktop agents complete 538 Ubuntu tasks, including 152 tasks with later user clarifications or interruptions. Programmatic checks inspect final application state, saved files, and exported artifacts. Desktop apps Jun 2026 29.8% Not in cards
OmniScore-Bench Models read sheet music, locate symbols, and transcribe passages. The evaluator checks their responses against annotations for the relevant part of the score. Documents, charts and text in images Nov 2026 91.0 Not in cards
SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. Spatial reasoning and counting Jun 2026 14.4% Not in cards
SpatialWorld Digital Agents complete entertainment tasks in digital environments. The score is the published digital task success rate. Spatial reasoning and counting Jun 2026 39% Not in cards
SpatialWorld Physical Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories. Spatial reasoning and counting Jun 2026 14.4% Not in cards
Agent-SafetyBench LLM agents act in 349 tool environments over 2,000 test cases covering 8 safety risk categories and 10 failure modes. A scorer model rates each run safe or unsafe. Refusing harmful requests Dec 2024 59.8% Not in cards
LexPilot Models interview a user about a Chinese civil case and predict the ruling on each claim. Paired consultations reveal the effect of missing facts. Law Oct 2026 72.1% Not in cards
OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. Embodied reasoning Sep 2026 43.9% Not in cards
TritonBench Models write Triton GPU operators in two channels, one from real GitHub kernels (G) and one matching PyTorch operators (T). Execution checks correctness and speedup. GPU kernels Feb 2025 34.3% Not in cards
VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. Web browsing Aug 2024 36.2% Not in cards
WorkflowArena Agents write executable workflows that coordinate other agents on professional tasks. Hidden verifiers check the completed deliverables. Multi-agent games Nov 2026 37% Not in cards
AudioTrust Audio language models answer prompts that probe hallucination, robustness, speaker authentication, privacy, fairness, and safety. Automated evaluators, including language-model judges, compare responses with task-specific references and criteria. Audio understanding and voice May 2025 0.37 Not in cards
ORAV Bench Models combine reference images, sounds, and videos into a requested audiovisual scene. A judge compares pairs of outputs against each reference's assigned role. Video generation Sep 2026 0.70 Not in cards
AgenticDataBench Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios. Data science and SQL Jul 2026 48.8% Not in cards
WorldState-Bench Video models generate two clips from the same image and action. A hidden state changes which ending the rule requires. A vision model checks both clips. Video world models Oct 2026 32% Not in cards
SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. Simulated robots Dec 2025 57.8% Not in cards
AndroidLab Mobile agents complete 138 tasks across nine apps on predefined Android virtual devices, in text-only and multimodal modes. Checks on the final device state score each task. Phones Oct 2024 31.2% Not in cards
HandWorldBench Models generate videos of human or robotic hands moving objects. The benchmark combines 17 measures of hand motion, object state, and video quality. Video world models Nov 2026 52.2 Not in cards
StableToolBench Models solve user requests by calling tools from ToolBench's large set of real-world APIs. A virtual API server with a cache and API simulators stands in for live APIs, so results stay stable over time. GPT-4 judges whether each request was solved and which of two answers is better, giving a solvable pass rate and a win rate. Function calling Mar 2024 26.4% Not in cards
Accessible Structured Subtitle Generation 2026 Systems produce timed Chinese subtitles for deaf and hard-of-hearing viewers. The structured track also scores speaker labels, key information, and readability. Video captioning Feb 2026 Not in cards
AISE-Bench Agents answer academic information-seeking questions through multi-step knowledge-graph APIs and source-grounded responses. Planning, execution and answer-grounding metrics score each trajectory. Deep research Jul 2026 Not in cards
CPRet Models retrieve contest solutions, duplicate problems, or full problems from simplified descriptions. Four retrieval tasks score the ranking of relevant results with NDCG@10. Competitive programming May 2025 Not in cards
DataSciBench Agents complete open-ended data-science tasks that require data cleaning, analysis, visualization and modeling. Component checks and overall task success score the work. Data science and SQL Apr 2026 Not in cards
DroidTask Agents select actions for 158 Android tasks across 13 apps. The original completion score requires every action to match the recorded sequence. Phones Aug 2023 Not in cards
ExtremeDrive Driving models plan trajectories from road recordings. A simulator executes the plans and measures success during emergency maneuvers and ordinary driving. Simulated robots Nov 2026 Not in cards
LlamaTouch Agents carry out requests in Android apps. The evaluator checks whether each run visited the required app or system states. Phones Apr 2024 Not in cards
PhysCapBench Models caption videos of physical interactions. The benchmark checks which motions, contacts, and changes of state the captions describe across 1,500 clips. Video captioning Nov 2026 Not in cards
PiRhDy Accompaniment Suggestion Models rank 50 accompaniment phrases for a MIDI melody. Several tracks can belong with the melody. The authors report mean average precision. Audio understanding and voice Oct 2020 Not in cards
PiRhDy Melody Completion Models rank 50 possible continuations of a MIDI melody. One phrase follows the prompt in the original piece. The authors report mean average precision. Audio understanding and voice Oct 2020 Not in cards
RivaBench Models answer questions about stand-up routines and academic talks using speech and visual cues. They also classify clips as real or synthetic. Video understanding Feb 2025 Not in cards
SongBench Models generate songs from lyrics and a style prompt. A trained music judge scores seven aspects, including the melody, arrangement, vocals, and mix. Audio understanding and voice Mar 2026 Not in cards
STORAL Models choose which moral fits a Chinese or English story. Two generation tasks ask models to write a moral from a story or illustrate a moral. Reading comprehension Apr 2022 Not in cards
ToolBench Models plan and call real REST APIs from RapidAPI, over 16,000 of them, to answer instructions that need one or many tools. ToolEval judges pass rate and win rate. Function calling Jul 2023 Not in cards
TriWorldBench World models generate synchronized robot-view videos from head and wrist cameras. Nineteen signals measure cross-view agreement, task alignment, physical coherence, motion, temporal stability, and visual quality. Video world models Sep 2026 Not in cards
Video-MonetBench Models rank two videos generated for the same prompt. The test compares each ranking with a human preference, including ties. Video understanding Dec 2024 Not in cards