| NeedleBench Models retrieve and reason over one or several facts inserted at different depths in bilingual long texts of set lengths. | Finding facts in long inputs | Jul 2024 | | System cards only |
| OpenVLM Leaderboard OpenCompass runs vision-language models through VLMEvalKit on a fixed set of image benchmarks, such as MMBench, MMStar and MathVista. The headline is the average score across that set. | Composite indices | Jan 2024 | 80.1 | Not in cards |
| Open LMM Reasoning Leaderboard Models solve multimodal reasoning tasks. OpenCompass ranks the models using the component results. | Composite indices | Dec 2024 | 61.6 | Not in cards |
| OpenLMM Spatial Leaderboard Models answer spatial reasoning tasks involving images. OpenCompass compares performance across the component evaluations. | Composite indices | Jun 2025 | 62.5 | Not in cards |
| OpenVLM Video Leaderboard Models complete video understanding tasks. OpenCompass compares models across video evaluation datasets. | Composite indices | Sep 2024 | 35.4 | Not in cards |
| InteractScience Models generate code for interactive scientific demonstrations. Evaluators test the program and assess its visual output. | Scientific code | Oct 2025 | 41.5 | Not in cards |
| OpenLMM Subjective Leaderboard Models generate open-ended answers to multimodal tasks. The board reports subjective evaluation results. | Composite indices | Apr 2025 | 76.6 | Not in cards |
| NeedleBench V2 Models retrieve one or several needles and reason over fictional multi-needle facts in English and Chinese haystacks from 4K to 1M tokens. The overall score is the plain average of three tasks. | Finding facts in long inputs | May 2025 | 86.3% | Not in cards |
| MathBench Models answer 3,709 math questions in five stages from basic arithmetic to college, in Chinese and English. Each stage also tests theory knowledge, and Circular Evaluation reshuffles answer options to check consistency. | Word problems | May 2024 | 70.9% | Not in cards |
| Creation-MMBench Models write creative pieces grounded in up to nine images plus a role, background and instructions, across 765 cases in 51 tasks. A judge scores quality and visual fact alignment against case-specific criteria. | Creative writing | Mar 2025 | 4.48 | Not in cards |
| Compass Academic Leaderboard Models answer academic reasoning tasks. OpenCompass publishes model rankings and predictions across the underlying evaluations. | Composite indices | Dec 2024 | 72.6% | Not in cards |
| GTA Agents answer 229 human-written queries that need 1 to 4 real deployed tools for perception, operation, logic and creativity, often with image inputs. Queries leave the tools and steps implicit. | Function calling | Jul 2024 | 46.6% | Not in cards |
| CompassRank OpenCompass ranks open and API language models on its own evaluation sets and publishes monthly rankings. Model makers submit a repository link or API endpoint to be tested. | Composite indices | Sep 2023 | 83.3 | Not in cards |
| CriticEval Models critique responses across four dimensions in nine task scenarios, giving both scalar scores and written critiques. GPT-4 grades written critiques against a large set of annotated reference critiques. | Overall chat quality | Feb 2024 | 7.86 | Not in cards |
| BotChat Two instances of a model sustain a conversation. Judges assess the resulting dialogue. | Multi-turn conversation | Oct 2023 | 73.2 | Not in cards |
| CNFinBench Models answer financial questions and carry out financial workflows. The benchmark includes tasks on expertise, autonomy, and integrity. | Finance | Dec 2025 | 83.7 | Not in cards |
| GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. | General assistants | Apr 2026 | 11.4% | Not in cards |
| VerifierBench Judge models decide whether proposed answers match references in math, science, and other subjects. | Other | Aug 2025 | 81.1% | Not in cards |
| CIBench Agents use code interpreters to analyze data. The evaluation checks stages of the interactive analysis. | Data science and SQL | Jul 2024 | 75.5% | Not in cards |
| LiveMathBench Models solve recent competition math problems, sampled many times. The G-Pass@k metric rewards both peak accuracy and stable answers across samples. | Competition problems with answers | Dec 2024 | 76.8% | Not in cards |
| ANAH Models detect unsupported claims in answers to English and Chinese questions. The benchmark provides annotations for individual claims. | Hallucination rates | May 2024 | 42.2% | Not in cards |
| OASIS (agent safety) Agents handle multi-turn requests with different levels of complexity. Evaluators measure the proportion of safe responses. | Deception and alignment | Nov 2025 | 95.2% | Not in cards |
| ATLAS (OpenCompass) Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge. | Science questions | Nov 2025 | 42.9% | Not in cards |
| CodeCompass Models solve competitive programming problems. Test cases check the generated programs. | Competitive programming | Jul 2025 | 40% | Not in cards |
| Ada-LEval Models put shuffled text segments in order and select the accepted answer from many candidate answers. The benchmark varies input length up to 128,000 tokens and scores outputs against the correct order or answer. | Reasoning over long documents | Apr 2024 | 46% | Not in cards |
| DevBench Acceptance Testing Models generate acceptance tests for repositories. The score is the published oracle-test percentage. | Building whole projects | Mar 2024 | 29.2% | Not in cards |
| DevBench Environment Setup Models prepare repository environments. The score is the percentage that pass the example usage test. | Building whole projects | Mar 2024 | 41.7% | Not in cards |
| DevBench Unit Testing Models generate unit tests for repositories. The score is the published oracle-test percentage. | Building whole projects | Mar 2024 | 36.5% | Not in cards |
| ReasonZoo Models solve tasks in logical reasoning, puzzles, and programming. The evaluation compares their answers with task solutions. | Abstract puzzles and logic | Aug 2025 | 61% | Not in cards |
| TextEdit Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity. | Documents, charts and text in images | Mar 2026 | 0.92 | Not in cards |
| DevBench Models work through the software lifecycle on 22 curated repositories in Python, C/C++, Java and JavaScript: design, environment setup, implementation, acceptance tests and unit tests. Tests score code and an LLM judge scores design. | Building whole projects | Mar 2024 | 41.7% | Not in cards |
| DevBench Implementation Models implement repository code. The score is the acceptance-test pass rate, with unit-test pass rate as an auxiliary metric. | Building whole projects | Mar 2024 | 7.10% | Not in cards |
| GAOKAO-Eval 2024 Models take Chinese national college entrance examinations. High school teachers grade the open responses. | Exam questions | Jun 2024 | 476.5 | Not in cards |
| JudgerBench Judge models assess candidate answers. The leaderboard reports English and Chinese results, accuracy, and correlation. | Other | Oct 2024 | 0.82 | Not in cards |
| TextEdit Real Models edit text in real images in the full TextEdit test set. The score is the published judged average. | Documents, charts and text in images | Mar 2026 | 0.90 | Not in cards |
| TextEdit Virtual Models edit text in virtual images in the full TextEdit test set. The score is the published judged average. | Documents, charts and text in images | Mar 2026 | 0.90 | Not in cards |
| JudgerBench V2 Judge models evaluate responses across tasks. OpenCompass reports results for generalist judges on the revised benchmark. | Other | Jul 2025 | 64.4 | Not in cards |
| ProSA Models answer equivalent instructions with different wording. PromptSensiScore measures how much model performance varies with the prompt. | Following precise instructions | Oct 2024 | 0.14 | Not in cards |
| CompassBench v1.3 Models answer Chinese and English tasks in coding, instruction following, knowledge, language, math and reasoning. OpenCompass publishes objective and subjective evaluation configurations. | Composite indices | Jul 2024 | | Not in cards |
| LCBench v0.1 Models solve English and Chinese LeetCode contest problems. Tests check the submitted programs. | Competitive programming | May 2024 | | Not in cards |
| MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. | Knowledge in many languages | Oct 2024 | | Not in cards |
| SWE-Bench Pro Verified Agents repair software repositories. OpenCompass corrects inconsistent tasks and strengthens tests in SWE-Bench Pro. | Fixing real repositories | Sep 2026 | | Not in cards |