Benchmark Almanac

Beijing Academy of Artificial Intelligence

We track 28 benchmarks and 2 arenas that Beijing Academy of Artificial Intelligence made or helped make. Its models are on its lab page.

2 in use 18 with public code 16 with a leaderboard here

Arena leaderboards

Top five in each of its 2 arenas with three or more rated models, the most rated first.

Benchmarks

Name Kind Released Best score Status
MLVU Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. Long videos Jun 2024 90.8% In use
RefSpatial-Bench Real images where the model points to an object or free space defined by multi-step spatial relations. Scored by mask hit. Embodied reasoning Jun 2025 73.6% Fading
MeasureBench Vision-language models read values from images of measuring instruments, such as gauges and rulers, in real and synthesized photos. Scored by matching the reading to the key. General visual questions Oct 2025 62.9% In use
Open FlagEval VLM Leaderboard Models answer questions about images. The leaderboard compares their scores across visual understanding tasks. General visual questions Feb 2025 70.7 Not in cards
Open Chinese LLM Leaderboard Models answer Chinese language tasks. The leaderboard compares model results across its evaluation suite. Knowledge in many languages May 2024 75.7 Not in cards
TRUE (Text Recognition and Understanding) Models recognize text in images and answer questions about its content. Documents, charts and text in images 81.6 Not in cards
ROME Models solve reasoning problems selected to limit contamination. Language reasoning Sep 2025 69.4 Not in cards
MeasureBench Real-world Models read values and units from real instrument images. The score is the published overall measurement accuracy. General visual questions Oct 2025 41.7% Not in cards
MeasureBench Synthetic Models read values and units from generated instrument images. The score is the published overall measurement accuracy. General visual questions Oct 2025 39.2% Not in cards
LiveOCRVQA Models read stylized text in images and answer questions about it. Documents, charts and text in images 80.5 Not in cards
EmbodiedVerse-Bench Models answer questions about spatial relationships and actions from visual observations. The leaderboard compares models across embodied reasoning tasks. Embodied reasoning 50.7 Not in cards
Video-SafetyBench Video-language models receive 2,264 synthesized videos paired with a harmful or a benign-looking query across 13 unsafe categories and 48 subcategories. An LLM judge scores harm with a confidence-weighted metric called RJScore. Jailbreaks and prompt injection May 2025 1.30 Not in cards
HalluDial Models detect hallucinations in dialogue. Hallucination rates Jun 2024 80.2 Not in cards
AgentSWE Coding agents build, edit, or optimize agents across 25 software-engineering commissions. Frozen deliveries run on held-out cases, where execution gates and task-specific model-judged rubrics score behavior. Optimization tasks measure improvement against each source benchmark objective. Machine learning engineering Oct 2026 19.1 Not in cards
CMMU Models answer Chinese multimodal questions across seven school subjects. General visual questions Jan 2024 30.9 Not in cards
ATOM-Bench Robot policies combine manipulation skills on single arm and dual arm platforms. Real robots Jun 2026 48.2% Not in cards
ChildMandarin Models transcribe Mandarin speech from children ages three to five or verify speakers. Speech recognition and translation Sep 2024 11.5% Not in cards
CORAL Models retrieve passages, answer questions, and cite supporting evidence across conversations. Retrieval metrics, response-overlap metrics, model judgments, and citation precision and recall score different parts of the task. Questions that need search Oct 2024 16.3 Not in cards
EmotionTalk Models infer emotions and sentiment from Chinese conversations. Audio understanding and voice May 2025 50.9 Not in cards
SeniorTalk Models transcribe conversations with older Mandarin speakers and identify who speaks. Speech recognition and translation Mar 2025 23.8% Not in cards
Carena-Hard-Auto Models answer Chinese prompts. A judge compares pairs of answers, and the evaluation uses a Bayesian ranking model. Chat and instructions in many languages Not in cards
Chinese Linguistics and Cognition Challenge Models answer open questions about Chinese language and cognition. Human graders or the publisher’s judge model assess the answers. Reasoning in many languages Not in cards
HM3D-MFMON Agents navigate across floors and find a sequence of objects. Simulated robots Aug 2026 Not in cards
Massive Visualized IR Benchmark Models retrieve screenshots that match a search query. The benchmark compares performance across four retrieval task groups. Documents, charts and text in images Feb 2025 Not in cards
RefSpatial-Expand-Bench Models point to an object or free space described by multi-step spatial relations, in more indoor scenes plus new outdoor streets and parking lots. Scored by mask hit. Embodied reasoning Oct 2025 Not in cards
RefSpatial-Expand-Bench Location Models solve spatial location questions in the RefSpatial expansion set. The score is the published overall accuracy. Embodied reasoning Oct 2025 Not in cards
RefSpatial-Expand-Bench Placement Models solve spatial placement questions in the RefSpatial expansion set. The score is the published overall accuracy. Embodied reasoning Oct 2025 Not in cards
TraceSpatial-Bench Models plan an object-centric 3D trace, an ordered list of metric 3D waypoints, from language instructions in real cluttered indoor scenes with depth. Embodied reasoning Dec 2025 Not in cards