Benchmark Almanac

Shanghai Artificial Intelligence Laboratory

We track 11 benchmarks that Shanghai Artificial Intelligence Laboratory made or helped make.

0 in use 9 with public code 8 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Argus Inspection Models inspect images for small details that change the sensible response to a question. A judge scores whether the response identifies the relevant detail and chooses an appropriate action. The score combines those judgments and assigns zero when a separate description fails to recognize the detail. General visual questions Jun 2025 0.44 Not in cards
BenchMAX Models follow instructions, write code, reason, use tools, translate, and retrieve information across 17 languages. Ten tasks use rule checks, model-judged wins, code tests, answer matching, or translation scores. Reasoning in many languages Feb 2025 85.2 Not in cards
SpaceDG-Bench Models answer spatial questions under clean images and nine forms of visual degradation. Choice and binary answers use accuracy, while numerical answers use relative-error thresholds. The test has 1,102 verified questions from scenes excluded from the training set. Spatial reasoning and counting May 2026 56.7% Not in cards
ARE-bench Agents continue an existing research project from a real user request. A judge checks the deliverable against the request and earlier workspace requirements. AI research tasks Nov 2026 66.2% Not in cards
OmniEarth-Bench Models interpret satellite images and observational data across Earth systems, answering questions about environmental processes and their interactions. Multiple-choice accuracy and model judgments score answers. Separate measures check localization, captions, and annotated reasoning steps. Research science May 2025 31.5% Not in cards
RQA-70K weather radar forecast evaluation Models compare forecast radar images or sequences with observations. They rate errors such as missed storms and false alarms and write assessments of forecast quality. Lab and data work Aug 2025 52.7% Not in cards
PRISM-Bench (audio and video generation) Models generate speech, music, or environmental sounds with video. A judge compares outputs with reference clips across 35 criteria, including synchronization and prompt following. Video generation May 2026 91.9 Not in cards
SeedBench (Seed Science) Models answer questions, summarize passages and interpret research on rice breeding in Chinese and English. The 11 task types use accuracy, macro F1 or ROUGE-L. Science questions Dec 2024 68.4 Not in cards
CauGym Models infer causal effects from descriptions and probabilities. Test questions vary the wording, add irrelevant facts, or remove information needed to answer. Language reasoning Feb 2026 Not in cards
OpenFinData Models answer Chinese financial questions, extract entities and values, and check calculations or data consistency. OpenCompass scores answer accuracy, using required-keyword matching for entity extraction. Finance Jan 2024 Not in cards
OpenHuEval Models answer Hungarian questions, interpret proverbs, and fill gaps in texts across five tasks. Factual questions and open responses use model judges, while matching and blank completion also use rules and text similarity. Reasoning in many languages Mar 2025 Not in cards