Benchmark Almanac

Renmin University of China

We track 12 benchmarks that Renmin University of China made or helped make.

1 in use 11 with public code 9 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
POPE Vision language models answer yes or no questions about whether objects appear in COCO images. Scored by accuracy or F1 on object hallucination. Hallucination rates May 2023 90.7% Fading
OmniGAIA Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data. Deep research Feb 2026 62.5% Fading
SpreadsheetBench 2 Agents complete 321 expert-annotated business spreadsheet workflows, such as financial modeling, debugging, and charting, in multi-sheet workbooks. Scoring is task accuracy. Documents and spreadsheets Jun 2026 34.9% In use
SpreadsheetBench Verified Models solve 400 real spreadsheet manipulation questions from Excel forums, an expert annotated subset of SpreadsheetBench. Each answer runs against several test spreadsheets. Documents and spreadsheets Dec 2025 79.1% Fading
OlymMATH Models solve Olympiad-level mathematics problems in English or Chinese. The evaluator checks final answers and reports pass@1 and consistency across repeated samples. Competition problems with answers Mar 2025 74.2% Not in cards
MathIF Reasoning models solve 420 math problems from GSM8K, MATH-500, Minerva, Olympiad and AIME while obeying added verifiable instructions. Code checks the instructions and the answer. Following precise instructions May 2025 50.7% Not in cards
DISBench Multimodal agents search raw visual histories over several steps to find target images from implicit context cues. General visual questions Feb 2026 28.7% Not in cards
UHGEval Models identify hallucinated Chinese text or continue a supplied passage. Classification uses accuracy, while generation uses BLEU-4, ROUGE-L, keyword precision, and BERTScore. Hallucination rates Nov 2023 0.70 Not in cards
AgentSWE Coding agents build, edit, or optimize agents across 25 software-engineering commissions. Frozen deliveries run on held-out cases, where execution gates and task-specific model-judged rubrics score behavior. Optimization tasks measure improvement against each source benchmark objective. Machine learning engineering Oct 2026 19.1 Not in cards
TriPhysBench Models answer quantitative questions about physical events in videos. Some questions require measuring an object or looking up its specifications. Embodied reasoning Nov 2026 50.4% Not in cards
CORAL Models retrieve passages, answer questions, and cite supporting evidence across conversations. Retrieval metrics, response-overlap metrics, model judgments, and citation precision and recall score different parts of the task. Questions that need search Oct 2024 16.3 Not in cards
SingMOS Models listen to Chinese and Japanese singing and predict how listeners rate its quality. The recordings include human voices and synthesized voices. Audio understanding and voice Jun 2024 Not in cards