Shanghai AI Lab
We track 70 benchmarks that Shanghai AI Lab made or helped make. Its models are on its lab page.
9 in use 56 with public code 53 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| MMStar Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. | General visual questions | Mar 2024 | | In use |
| OmniDocBench v1.5 Models parse 1,355 PDF pages into text, tables and formulas. The Overall score averages text accuracy from edit distance, table TEDS and formula CDM. | Documents, charts and text in images | Sep 2025 | 94.6 | In use |
| MMBench-EN v1.1 Models answer English multiple-choice image questions across 20 ability dimensions, using CircularEval with shuffled options. Scored by accuracy on the dev split. | General visual questions | Apr 2024 | | In use |
| MVBench Models answer multiple-choice questions across 20 temporal video tasks that a single frame cannot solve. Scored by accuracy. | Video understanding | Nov 2023 | | Fading |
| MathVerse Models solve diagram-based math problems shown in six versions that move information between the text and the image. It tests whether models read the diagram. | Visual math and diagrams | Mar 2024 | | Fading |
| MMBench Vision language models answer multiple-choice questions covering 20 perception and reasoning skills. CircularEval rotates the options. Scored by accuracy. | General visual questions | Jul 2023 | | Fading |
| VisuLogic Models answer 1,000 visual reasoning puzzles in six categories that cannot be solved by describing the image in words. Scored by accuracy. | Spatial reasoning and counting | Apr 2025 | | Fading |
| PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. | Medicine | May 2023 | | Fading |
| MMBench-CN v1.1 Models answer the Chinese translation of the MMBench image questions, using CircularEval with shuffled options. Scored by accuracy. | Knowledge in many languages | | Fading | |
| MMSI-Bench Models answer 1,000 human-written questions that need spatial reasoning across several images. Images come from indoor scans, driving, egocentric video and robot data. | Spatial reasoning and counting | May 2025 | | Fading |
| Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. | General visual questions | Jun 2025 | | Fading |
| OmniDocBench Diverse PDF pages with layout, text, table and formula annotations. Parsing scored by edit distance and TEDS. | Documents, charts and text in images | Dec 2024 | Fading | |
| OmniDocBench v1.6 Models parse 1,651 PDF pages, including nested tables, dense formulas and unusual layouts, into text, tables and formulas. Scored by the v1.5 Overall formula. | Documents, charts and text in images | Apr 2026 | 97.0 | In use |
| CRPE Models answer single-choice questions about objects and their relations in images, covering subject, predicate and object. Each question is asked again with options rotated (CircularEval). | General visual questions | Feb 2024 | | Dropped |
| HiPhO Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. | Science olympiads | Sep 2025 | | In use |
| MSEarth Models answer questions about figures from Earth science papers across the atmosphere, cryosphere, hydrosphere, lithosphere and biosphere. Labs report the multiple choice split. | General visual questions | May 2025 | | Fading |
| SALAD-Bench Models answer harmful questions organized in a three-level taxonomy, some with attack or defense changes and some multiple-choice. The MD-Judge evaluator labels responses as safe or unsafe. | Refusing harmful requests | Feb 2024 | 96.8% | Fading |
| MMBench-Video Video models answer free-form questions about long, multi-shot YouTube videos. A GPT-4 judge scores the answers. | Long videos | Jun 2024 | 2.15 | Dropped |
| SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. | Web and app front ends | Aug 2025 | 0.92 | Fading |
| MMT-Bench Vision-language models answer 31,325 multiple-choice questions across many multimodal tasks such as recognition, localization, reasoning and planning. | General visual questions | Apr 2024 | 62.4% | In use |
| SciReasoner evaluation suite Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks. | Research science | Sep 2025 | | Fading |
| T-Eval Models use tools step by step, and each sub-process is scored separately: instruction following, planning, reasoning, retrieval, understanding, and review. | Function calling | Dec 2023 | 86.4% | Dropped |
| ProteinLMBench Models answer 944 hand-checked multiple choice questions on protein sequence, structure and function. | Genomics and life sciences | Jun 2024 | 57.9% | Dropped |
| TimeLens-Bench Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU. | Video understanding | Dec 2025 | | Fading |
| VBench Video generation models make clips from prompts, and automatic checks score 16 dimensions of quality and prompt adherence. | Video generation | Nov 2023 | | Fading |
| CCBench (MMBench Chinese Culture) Vision language models answer 510 Chinese-language multiple-choice questions about images of Chinese culture, such as calligraphy, painting, food and historical figures. | General visual questions | Oct 2023 | | Dropped |
| SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. | Research science | Dec 2025 | | In use |
| UniGenBench Text-to-image models draw images for 600 prompts across 5 themes and 20 subthemes, in English and Chinese and in short and long forms. A multimodal model judge checks each image on 10 primary and 27 secondary semantic dimensions, such as attributes, actions, relationships, layout, text and logical reasoning. | Image generation and editing | Aug 2025 | 95.8% | System cards only |
| MMIU Vision-language models answer 11,000 multiple-choice questions over sets of images. 52 tasks cover 7 kinds of multi-image relationship. | General visual questions | Aug 2024 | | System cards only |
| LinguaSafe Models face 45k safety prompts in 12 languages, from Hungarian to Malay. Direct and indirect assessments score harmful responses, and an oversensitivity test checks over-refusal. | Refusing harmful requests | Aug 2025 | 12.0 | System cards only |
| MT-Bench-101 Models hold 1,388 multi-turn dialogues with 4,208 turns across 13 tasks from a three-tier ability taxonomy. GPT-4 scores each turn from 1 to 10. | Multi-turn conversation | Feb 2024 | | Dropped |
| TOMG-Bench Models generate molecules from natural language requests in three tasks: molecule editing, molecule optimization and customized generation. Each output is checked for validity and for meeting the requested properties. | Genomics and life sciences | Dec 2024 | 18.6% | In use |
| AdvancedMathBench Models write proofs for 245 advanced problems at undergraduate and doctoral qualifying exam levels. A verifier trained on expert annotations grades each proof. | Written proofs | Jul 2026 | | In use |
| MMBench-GUI GUI agents are tested at four levels on Windows, macOS, Linux, iOS, Android and the web: content understanding, element grounding, task automation and task collaboration. | Finding things on screen | Jul 2025 | | System cards only |
| SciTS Models understand and generate scientific time series in 43 tasks across 12 domains. Signals range from one to ten million points. Each task uses its own metric. | Research science | Sep 2025 | System cards only | |
| VRBench Models answer multi-step reasoning questions about long narrative videos, which average 1.6 hours. The 1,010 videos come with 9,468 human-labeled question and answer pairs and 30,292 timestamped reasoning steps across seven reasoning types. Multiple-choice answers score the outcome, and a model judge scores the reasoning steps. | Long videos | Jun 2025 | 68.2% | Not in cards |
| LEGO-Puzzles Vision-language models answer LEGO assembly questions, from basic spatial understanding to planning several assembly steps. The elementary set has 1,100 questions over 11 tasks. | Spatial reasoning and counting | Mar 2025 | 72% | Not in cards |
| OPT-BENCH Agents iteratively improve solutions to 20 machine-learning tasks and 10 combinatorial optimization problems using execution feedback. Win counts, invalid-solution rates, ranks, and improvement rates measure optimization performance. | Scientific code | Jun 2025 | | Not in cards |
| OVO-S-Bench Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos. | Video understanding | Jun 2026 | | Not in cards |
| GMAI-MMBench Vision-language models answer medical multiple-choice questions over many imaging modalities, clinical tasks and departments, at several perceptual granularities. | Medicine | Aug 2024 | | Not in cards |
| MathIF Reasoning models solve 420 math problems from GSM8K, MATH-500, Minerva, Olympiad and AIME while obeying added verifiable instructions. Code checks the instructions and the answer. | Following precise instructions | May 2025 | | Not in cards |
| HelloBench Models generate long answers, summaries, conversations, text completions, and texts from writing instructions. Model judges apply task-specific checklists to the responses. | Overall chat quality | Sep 2024 | 48.5 | Not in cards |
| MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. | Medicine | Jun 2026 | 69.2 | Not in cards |
| ToolMaze Agents complete tool-use tasks while the runtime introduces explicit or hidden tool failures. A judge checks completion across four task structures and five perturbation settings. Metrics report task success, recovery rate, and recovery cost. | Function calling | Jun 2026 | | Not in cards |
| CaLM Models answer causal discovery, intervention, and counterfactual reasoning tasks in English and Chinese. Evaluation compares answers with reference labels to measure accuracy and robustness across causal targets and prompting methods. | Language reasoning | May 2024 | 45.3% | Not in cards |
| DriveBench Vision-language models answer 20,498 questions about 19,200 driving frames, covering perception, prediction, planning and behavior. Inputs come in 17 settings: clean images, corrupted images and text only. The test checks whether answers rest on the image or on general knowledge. | Embodied reasoning | Jan 2025 | 48.0 | Not in cards |
| MM-AlignBench Models answer open-ended questions about 252 human-annotated images. GPT-4o compares their answers with Claude-Sonnet-3 reference answers to assess alignment with human preferences. | General visual questions | Feb 2025 | 82.5% | Not in cards |
| OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. | Embodied reasoning | Jul 2025 | 53.4% | Not in cards |
| RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. | Puzzles and chess | Jun 2026 | | Not in cards |
| ChartX Vision-language models do 7 chart tasks, such as question answering, summarization and redrawing, over 18 chart types and 22 topics. | Documents, charts and text in images | Feb 2024 | 20.9 | Not in cards |
| FLAMES Models respond to Chinese prompts about safety and values. Evaluators assess the responses against the benchmark's alignment criteria. | Deception and alignment | Nov 2023 | 43.2% | Not in cards |
| MMDU Vision-language models hold long conversations about many images, up to 20 images and 27 turns per dialog. A GPT-4o judge scores the answers. | General visual questions | Jun 2024 | 62.6 | Not in cards |
| PICABench Models edit images while preserving physical effects in optics, mechanics, and state transitions. A vision-language judge answers region-specific questions to measure physical correctness. | Image generation and editing | Oct 2025 | | Not in cards |
| OS-Critic Bench Models judge whether a proposed action advances a task using the screenshot and action history. The grader compares each judgment with a human label. | Finding things on screen | Dec 2025 | 68.2% | Not in cards |
| DriveBench Behavior Models answer driving behavior questions. The score is the published GPT evaluation score for this task family. | Embodied reasoning | Jan 2025 | | Not in cards |
| DriveBench Perception Models answer driving perception questions. The score is the published GPT evaluation score for this task family. | Embodied reasoning | Jan 2025 | 35.2 | Not in cards |
| DriveBench Planning Models answer driving planning questions. The score is the published GPT evaluation score for this task family. | Embodied reasoning | Jan 2025 | 57.0 | Not in cards |
| DriveBench Prediction Models answer driving prediction questions. The score is the published GPT evaluation score for this task family. | Embodied reasoning | Jan 2025 | | Not in cards |
| PRMBench Models identify errors and redundant steps in mathematical solutions. PRMScore combines positive and negative F1 scores across simplicity, soundness, and sensitivity tests. | Word problems | Jan 2025 | 68.8 | Not in cards |
| MM-NIAH Vision-language models answer questions about long documents of interleaved images and text. Tasks are retrieval, counting and reasoning over needles hidden in the document. | Finding facts in long inputs | Jun 2024 | | Not in cards |
| OmniMedVQA Models answer questions about 118,010 medical images across 12 imaging modalities. The authors measure accuracy against the correct answer option. | Medicine | Feb 2024 | | Not in cards |
| VBench image-to-video Models animate input images. Automatic checks score image conditioning and video quality across the complete image-to-video suite. | Video generation | Nov 2024 | | Not in cards |
| VBench-2.0 Video generation models are scored on intrinsic faithfulness, such as human fidelity, controllability, creativity, physics and commonsense, beyond pixel quality and basic prompt match. | Video generation | Mar 2025 | | Not in cards |
| CPRet Models retrieve contest solutions, duplicate problems, or full problems from simplified descriptions. Four retrieval tasks score the ranking of relevant results with NDCG@10. | Competitive programming | May 2025 | Not in cards | |
| EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. | Simulated robots | Jun 2026 | Not in cards | |
| MedBench 1.0 (OpenCompass) Models answer Chinese medical questions across clinical reasoning, language understanding, generation, knowledge, and ethics. | Medicine | Jan 2024 | Not in cards | |
| MedBench 2.0 (OpenCompass) Models answer Chinese medical questions. This release changes the evaluation of open responses. | Medicine | Jul 2024 | Not in cards | |
| MedBench 3.0 (OpenCompass) Models handle medical tasks across an expanded clinical evaluation suite. Shanghai AI Laboratory reports results for participating models. | Medicine | Mar 2025 | Not in cards | |
| MedBench 4.0 (OpenCompass) Models handle clinical scenarios in an evaluation that covers specialized models and medical applications. | Medicine | Nov 2025 | Not in cards | |
| MM-SafetyBench Multimodal models get 5,040 harmful text and image pairs in 13 scenarios, where a query-relevant image carries the harmful intent. Judges score attack success. | Jailbreaks and prompt injection | Nov 2023 | Not in cards |