Benchmark Almanac

OpenCompass

We track 42 benchmarks and 2 arenas that OpenCompass made or helped make.

0 in use 36 with public code 37 with a leaderboard here

Arena leaderboards

Top five in each of its 2 arenas with three or more rated models, the most rated first.

Benchmarks

Name Kind Released Best score Status
NeedleBench Models retrieve and reason over one or several facts inserted at different depths in bilingual long texts of set lengths. Finding facts in long inputs Jul 2024 System cards only
OpenVLM Leaderboard OpenCompass runs vision-language models through VLMEvalKit on a fixed set of image benchmarks, such as MMBench, MMStar and MathVista. The headline is the average score across that set. Composite indices Jan 2024 80.1 Not in cards
Open LMM Reasoning Leaderboard Models solve multimodal reasoning tasks. OpenCompass ranks the models using the component results. Composite indices Dec 2024 61.6 Not in cards
OpenLMM Spatial Leaderboard Models answer spatial reasoning tasks involving images. OpenCompass compares performance across the component evaluations. Composite indices Jun 2025 62.5 Not in cards
OpenVLM Video Leaderboard Models complete video understanding tasks. OpenCompass compares models across video evaluation datasets. Composite indices Sep 2024 35.4 Not in cards
InteractScience Models generate code for interactive scientific demonstrations. Evaluators test the program and assess its visual output. Scientific code Oct 2025 41.5 Not in cards
OpenLMM Subjective Leaderboard Models generate open-ended answers to multimodal tasks. The board reports subjective evaluation results. Composite indices Apr 2025 76.6 Not in cards
NeedleBench V2 Models retrieve one or several needles and reason over fictional multi-needle facts in English and Chinese haystacks from 4K to 1M tokens. The overall score is the plain average of three tasks. Finding facts in long inputs May 2025 86.3% Not in cards
MathBench Models answer 3,709 math questions in five stages from basic arithmetic to college, in Chinese and English. Each stage also tests theory knowledge, and Circular Evaluation reshuffles answer options to check consistency. Word problems May 2024 70.9% Not in cards
Creation-MMBench Models write creative pieces grounded in up to nine images plus a role, background and instructions, across 765 cases in 51 tasks. A judge scores quality and visual fact alignment against case-specific criteria. Creative writing Mar 2025 4.48 Not in cards
Compass Academic Leaderboard Models answer academic reasoning tasks. OpenCompass publishes model rankings and predictions across the underlying evaluations. Composite indices Dec 2024 72.6% Not in cards
GTA Agents answer 229 human-written queries that need 1 to 4 real deployed tools for perception, operation, logic and creativity, often with image inputs. Queries leave the tools and steps implicit. Function calling Jul 2024 46.6% Not in cards
CompassRank OpenCompass ranks open and API language models on its own evaluation sets and publishes monthly rankings. Model makers submit a repository link or API endpoint to be tested. Composite indices Sep 2023 83.3 Not in cards
CriticEval Models critique responses across four dimensions in nine task scenarios, giving both scalar scores and written critiques. GPT-4 grades written critiques against a large set of annotated reference critiques. Overall chat quality Feb 2024 7.86 Not in cards
BotChat Two instances of a model sustain a conversation. Judges assess the resulting dialogue. Multi-turn conversation Oct 2023 73.2 Not in cards
CNFinBench Models answer financial questions and carry out financial workflows. The benchmark includes tasks on expertise, autonomy, and integrity. Finance Dec 2025 83.7 Not in cards
GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. General assistants Apr 2026 11.4% Not in cards
VerifierBench Judge models decide whether proposed answers match references in math, science, and other subjects. Other Aug 2025 81.1% Not in cards
CIBench Agents use code interpreters to analyze data. The evaluation checks stages of the interactive analysis. Data science and SQL Jul 2024 75.5% Not in cards
LiveMathBench Models solve recent competition math problems, sampled many times. The G-Pass@k metric rewards both peak accuracy and stable answers across samples. Competition problems with answers Dec 2024 76.8% Not in cards
ANAH Models detect unsupported claims in answers to English and Chinese questions. The benchmark provides annotations for individual claims. Hallucination rates May 2024 42.2% Not in cards
OASIS (agent safety) Agents handle multi-turn requests with different levels of complexity. Evaluators measure the proportion of safe responses. Deception and alignment Nov 2025 95.2% Not in cards
ATLAS (OpenCompass) Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge. Science questions Nov 2025 42.9% Not in cards
CodeCompass Models solve competitive programming problems. Test cases check the generated programs. Competitive programming Jul 2025 40% Not in cards
Ada-LEval Models put shuffled text segments in order and select the accepted answer from many candidate answers. The benchmark varies input length up to 128,000 tokens and scores outputs against the correct order or answer. Reasoning over long documents Apr 2024 46% Not in cards
DevBench Acceptance Testing Models generate acceptance tests for repositories. The score is the published oracle-test percentage. Building whole projects Mar 2024 29.2% Not in cards
DevBench Environment Setup Models prepare repository environments. The score is the percentage that pass the example usage test. Building whole projects Mar 2024 41.7% Not in cards
DevBench Unit Testing Models generate unit tests for repositories. The score is the published oracle-test percentage. Building whole projects Mar 2024 36.5% Not in cards
ReasonZoo Models solve tasks in logical reasoning, puzzles, and programming. The evaluation compares their answers with task solutions. Abstract puzzles and logic Aug 2025 61% Not in cards
TextEdit Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity. Documents, charts and text in images Mar 2026 0.92 Not in cards
DevBench Models work through the software lifecycle on 22 curated repositories in Python, C/C++, Java and JavaScript: design, environment setup, implementation, acceptance tests and unit tests. Tests score code and an LLM judge scores design. Building whole projects Mar 2024 41.7% Not in cards
DevBench Implementation Models implement repository code. The score is the acceptance-test pass rate, with unit-test pass rate as an auxiliary metric. Building whole projects Mar 2024 7.10% Not in cards
GAOKAO-Eval 2024 Models take Chinese national college entrance examinations. High school teachers grade the open responses. Exam questions Jun 2024 476.5 Not in cards
JudgerBench Judge models assess candidate answers. The leaderboard reports English and Chinese results, accuracy, and correlation. Other Oct 2024 0.82 Not in cards
TextEdit Real Models edit text in real images in the full TextEdit test set. The score is the published judged average. Documents, charts and text in images Mar 2026 0.90 Not in cards
TextEdit Virtual Models edit text in virtual images in the full TextEdit test set. The score is the published judged average. Documents, charts and text in images Mar 2026 0.90 Not in cards
JudgerBench V2 Judge models evaluate responses across tasks. OpenCompass reports results for generalist judges on the revised benchmark. Other Jul 2025 64.4 Not in cards
ProSA Models answer equivalent instructions with different wording. PromptSensiScore measures how much model performance varies with the prompt. Following precise instructions Oct 2024 0.14 Not in cards
CompassBench v1.3 Models answer Chinese and English tasks in coding, instruction following, knowledge, language, math and reasoning. OpenCompass publishes objective and subjective evaluation configurations. Composite indices Jul 2024 Not in cards
LCBench v0.1 Models solve English and Chinese LeetCode contest problems. Tests check the submitted programs. Competitive programming May 2024 Not in cards
MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. Knowledge in many languages Oct 2024 Not in cards
SWE-Bench Pro Verified Agents repair software repositories. OpenCompass corrects inconsistent tasks and strengthens tests in SWE-Bench Pro. Fixing real repositories Sep 2026 Not in cards