Benchmark Almanac

Shanghai AI Lab

We track 70 benchmarks that Shanghai AI Lab made or helped make. Its models are on its lab page.

9 in use 56 with public code 53 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
MMStar Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. General visual questions Mar 2024 85.9% In use
OmniDocBench v1.5 Models parse 1,355 PDF pages into text, tables and formulas. The Overall score averages text accuracy from edit distance, table TEDS and formula CDM. Documents, charts and text in images Sep 2025 94.6 In use
MMBench-EN v1.1 Models answer English multiple-choice image questions across 20 ability dimensions, using CircularEval with shuffled options. Scored by accuracy on the dev split. General visual questions Apr 2024 88.5% In use
MVBench Models answer multiple-choice questions across 20 temporal video tasks that a single frame cannot solve. Scored by accuracy. Video understanding Nov 2023 79% Fading
MathVerse Models solve diagram-based math problems shown in six versions that move information between the text and the image. It tests whether models read the diagram. Visual math and diagrams Mar 2024 89.7% Fading
MMBench Vision language models answer multiple-choice questions covering 20 perception and reasoning skills. CircularEval rotates the options. Scored by accuracy. General visual questions Jul 2023 88.8% Fading
VisuLogic Models answer 1,000 visual reasoning puzzles in six categories that cannot be solved by describing the image in words. Scored by accuracy. Spatial reasoning and counting Apr 2025 54.3% Fading
PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. Medicine May 2023 66.2% Fading
MMBench-CN v1.1 Models answer the Chinese translation of the MMBench image questions, using CircularEval with shuffled options. Scored by accuracy. Knowledge in many languages 88.1% Fading
MMSI-Bench Models answer 1,000 human-written questions that need spatial reasoning across several images. Images come from indoor scans, driving, egocentric video and robot data. Spatial reasoning and counting May 2025 43.1% Fading
Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. General visual questions Jun 2025 60.7% Fading
OmniDocBench Diverse PDF pages with layout, text, table and formula annotations. Parsing scored by edit distance and TEDS. Documents, charts and text in images Dec 2024 Fading
OmniDocBench v1.6 Models parse 1,651 PDF pages, including nested tables, dense formulas and unusual layouts, into text, tables and formulas. Scored by the v1.5 Overall formula. Documents, charts and text in images Apr 2026 97.0 In use
CRPE Models answer single-choice questions about objects and their relations in images, covering subject, predicate and object. Each question is asked again with options rotated (CircularEval). General visual questions Feb 2024 79.2% Dropped
HiPhO Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. Science olympiads Sep 2025 78.3% In use
MSEarth Models answer questions about figures from Earth science papers across the atmosphere, cryosphere, hydrosphere, lithosphere and biosphere. Labs report the multiple choice split. General visual questions May 2025 68.8% Fading
SALAD-Bench Models answer harmful questions organized in a three-level taxonomy, some with attack or defense changes and some multiple-choice. The MD-Judge evaluator labels responses as safe or unsafe. Refusing harmful requests Feb 2024 96.8% Fading
MMBench-Video Video models answer free-form questions about long, multi-shot YouTube videos. A GPT-4 judge scores the answers. Long videos Jun 2024 2.15 Dropped
SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. Web and app front ends Aug 2025 0.92 Fading
MMT-Bench Vision-language models answer 31,325 multiple-choice questions across many multimodal tasks such as recognition, localization, reasoning and planning. General visual questions Apr 2024 62.4% In use
SciReasoner evaluation suite Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks. Research science Sep 2025 63.3% Fading
T-Eval Models use tools step by step, and each sub-process is scored separately: instruction following, planning, reasoning, retrieval, understanding, and review. Function calling Dec 2023 86.4% Dropped
ProteinLMBench Models answer 944 hand-checked multiple choice questions on protein sequence, structure and function. Genomics and life sciences Jun 2024 57.9% Dropped
TimeLens-Bench Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU. Video understanding Dec 2025 62.9% Fading
VBench Video generation models make clips from prompts, and automatic checks score 16 dimensions of quality and prompt adherence. Video generation Nov 2023 84.2% Fading
CCBench (MMBench Chinese Culture) Vision language models answer 510 Chinese-language multiple-choice questions about images of Chinese culture, such as calligraphy, painting, food and historical figures. General visual questions Oct 2023 86.9% Dropped
SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. Research science Dec 2025 33.8% In use
UniGenBench Text-to-image models draw images for 600 prompts across 5 themes and 20 subthemes, in English and Chinese and in short and long forms. A multimodal model judge checks each image on 10 primary and 27 secondary semantic dimensions, such as attributes, actions, relationships, layout, text and logical reasoning. Image generation and editing Aug 2025 95.8% System cards only
MMIU Vision-language models answer 11,000 multiple-choice questions over sets of images. 52 tasks cover 7 kinds of multi-image relationship. General visual questions Aug 2024 27.8% System cards only
LinguaSafe Models face 45k safety prompts in 12 languages, from Hungarian to Malay. Direct and indirect assessments score harmful responses, and an oversensitivity test checks over-refusal. Refusing harmful requests Aug 2025 12.0 System cards only
MT-Bench-101 Models hold 1,388 multi-turn dialogues with 4,208 turns across 13 tasks from a three-tier ability taxonomy. GPT-4 scores each turn from 1 to 10. Multi-turn conversation Feb 2024 7.82 Dropped
TOMG-Bench Models generate molecules from natural language requests in three tasks: molecule editing, molecule optimization and customized generation. Each output is checked for validity and for meeting the requested properties. Genomics and life sciences Dec 2024 18.6% In use
AdvancedMathBench Models write proofs for 245 advanced problems at undergraduate and doctoral qualifying exam levels. A verifier trained on expert annotations grades each proof. Written proofs Jul 2026 58.1% In use
MMBench-GUI GUI agents are tested at four levels on Windows, macOS, Linux, iOS, Android and the web: content understanding, element grounding, task automation and task collaboration. Finding things on screen Jul 2025 81.5% System cards only
SciTS Models understand and generate scientific time series in 43 tasks across 12 domains. Signals range from one to ten million points. Each task uses its own metric. Research science Sep 2025 System cards only
VRBench Models answer multi-step reasoning questions about long narrative videos, which average 1.6 hours. The 1,010 videos come with 9,468 human-labeled question and answer pairs and 30,292 timestamped reasoning steps across seven reasoning types. Multiple-choice answers score the outcome, and a model judge scores the reasoning steps. Long videos Jun 2025 68.2% Not in cards
LEGO-Puzzles Vision-language models answer LEGO assembly questions, from basic spatial understanding to planning several assembly steps. The elementary set has 1,100 questions over 11 tasks. Spatial reasoning and counting Mar 2025 72% Not in cards
OPT-BENCH Agents iteratively improve solutions to 20 machine-learning tasks and 10 combinatorial optimization problems using execution feedback. Win counts, invalid-solution rates, ranks, and improvement rates measure optimization performance. Scientific code Jun 2025 0.69 Not in cards
OVO-S-Bench Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos. Video understanding Jun 2026 59.2% Not in cards
GMAI-MMBench Vision-language models answer medical multiple-choice questions over many imaging modalities, clinical tasks and departments, at several perceptual granularities. Medicine Aug 2024 43.4% Not in cards
MathIF Reasoning models solve 420 math problems from GSM8K, MATH-500, Minerva, Olympiad and AIME while obeying added verifiable instructions. Code checks the instructions and the answer. Following precise instructions May 2025 50.7% Not in cards
HelloBench Models generate long answers, summaries, conversations, text completions, and texts from writing instructions. Model judges apply task-specific checklists to the responses. Overall chat quality Sep 2024 48.5 Not in cards
MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. Medicine Jun 2026 69.2 Not in cards
ToolMaze Agents complete tool-use tasks while the runtime introduces explicit or hidden tool failures. A judge checks completion across four task structures and five perturbation settings. Metrics report task success, recovery rate, and recovery cost. Function calling Jun 2026 53% Not in cards
CaLM Models answer causal discovery, intervention, and counterfactual reasoning tasks in English and Chinese. Evaluation compares answers with reference labels to measure accuracy and robustness across causal targets and prompting methods. Language reasoning May 2024 45.3% Not in cards
DriveBench Vision-language models answer 20,498 questions about 19,200 driving frames, covering perception, prediction, planning and behavior. Inputs come in 17 settings: clean images, corrupted images and text only. The test checks whether answers rest on the image or on general knowledge. Embodied reasoning Jan 2025 48.0 Not in cards
MM-AlignBench Models answer open-ended questions about 252 human-annotated images. GPT-4o compares their answers with Claude-Sonnet-3 reference answers to assess alignment with human preferences. General visual questions Feb 2025 82.5% Not in cards
OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. Embodied reasoning Jul 2025 53.4% Not in cards
RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. Puzzles and chess Jun 2026 49.9% Not in cards
ChartX Vision-language models do 7 chart tasks, such as question answering, summarization and redrawing, over 18 chart types and 22 topics. Documents, charts and text in images Feb 2024 20.9 Not in cards
FLAMES Models respond to Chinese prompts about safety and values. Evaluators assess the responses against the benchmark's alignment criteria. Deception and alignment Nov 2023 43.2% Not in cards
MMDU Vision-language models hold long conversations about many images, up to 20 images and 27 turns per dialog. A GPT-4o judge scores the answers. General visual questions Jun 2024 62.6 Not in cards
PICABench Models edit images while preserving physical effects in optics, mechanics, and state transitions. A vision-language judge answers region-specific questions to measure physical correctness. Image generation and editing Oct 2025 61.9% Not in cards
OS-Critic Bench Models judge whether a proposed action advances a task using the screenshot and action history. The grader compares each judgment with a human label. Finding things on screen Dec 2025 68.2% Not in cards
DriveBench Behavior Models answer driving behavior questions. The score is the published GPT evaluation score for this task family. Embodied reasoning Jan 2025 54.6 Not in cards
DriveBench Perception Models answer driving perception questions. The score is the published GPT evaluation score for this task family. Embodied reasoning Jan 2025 35.2 Not in cards
DriveBench Planning Models answer driving planning questions. The score is the published GPT evaluation score for this task family. Embodied reasoning Jan 2025 57.0 Not in cards
DriveBench Prediction Models answer driving prediction questions. The score is the published GPT evaluation score for this task family. Embodied reasoning Jan 2025 48.9 Not in cards
PRMBench Models identify errors and redundant steps in mathematical solutions. PRMScore combines positive and negative F1 scores across simplicity, soundness, and sensitivity tests. Word problems Jan 2025 68.8 Not in cards
MM-NIAH Vision-language models answer questions about long documents of interleaved images and text. Tasks are retrieval, counting and reasoning over needles hidden in the document. Finding facts in long inputs Jun 2024 11.6% Not in cards
OmniMedVQA Models answer questions about 118,010 medical images across 12 imaging modalities. The authors measure accuracy against the correct answer option. Medicine Feb 2024 50.7% Not in cards
VBench image-to-video Models animate input images. Automatic checks score image conditioning and video quality across the complete image-to-video suite. Video generation Nov 2024 86.9% Not in cards
VBench-2.0 Video generation models are scored on intrinsic faithfulness, such as human fidelity, controllability, creativity, physics and commonsense, beyond pixel quality and basic prompt match. Video generation Mar 2025 66.7% Not in cards
CPRet Models retrieve contest solutions, duplicate problems, or full problems from simplified descriptions. Four retrieval tasks score the ranking of relevant results with NDCG@10. Competitive programming May 2025 Not in cards
EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. Simulated robots Jun 2026 Not in cards
MedBench 1.0 (OpenCompass) Models answer Chinese medical questions across clinical reasoning, language understanding, generation, knowledge, and ethics. Medicine Jan 2024 Not in cards
MedBench 2.0 (OpenCompass) Models answer Chinese medical questions. This release changes the evaluation of open responses. Medicine Jul 2024 Not in cards
MedBench 3.0 (OpenCompass) Models handle medical tasks across an expanded clinical evaluation suite. Shanghai AI Laboratory reports results for participating models. Medicine Mar 2025 Not in cards
MedBench 4.0 (OpenCompass) Models handle clinical scenarios in an evaluation that covers specialized models and medical applications. Medicine Nov 2025 Not in cards
MM-SafetyBench Multimodal models get 5,040 harmful text and image pairs in 13 scenarios, where a query-relevant image carries the harmful intent. Judges score attack success. Jailbreaks and prompt injection Nov 2023 Not in cards