Benchmark Almanac

Peking University

We track 44 benchmarks that Peking University made or helped make.

2 in use 39 with public code 28 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
TempCompass Video models answer questions about temporal aspects of short clips, such as action order, speed, direction and attribute change, in several question formats. Video understanding Mar 2024 89.6% Fading
NOVA-63 Models answer questions written by native speakers in 14 languages across 63 academic disciplines. Scoring is accuracy. Knowledge in many languages Nov 2025 59.1% Fading
SysBench Models follow 500 system messages across five user turns each. A model verifier checks constraints, instructions and whole sessions. Following precise instructions Aug 2024 81.2% Fading
Omni-MATH Models solve 4,428 Olympiad-level competition math problems across 33 subdomains and 10 difficulty levels. A GPT-4 or Omni-Judge grader checks answers. Competition problems with answers Oct 2024 60.5% Dropped
CFBench Models follow 1,000 real-world instructions with embedded constraints, across more than 200 scenarios and 50 NLP tasks. A judge checks each constraint and the overall requirement. Following precise instructions Aug 2024 71.8% Dropped
SaaS-Bench 106 long-horizon GUI workflows across 23 locally deployed real SaaS apps. Scored from app databases with weighted checkpoints and full-resolution rate. Web browsing May 2026 31.1% In use
WISE Text-to-image models draw 1,000 prompts that need world knowledge in culture, space, time and natural science. A vision model judge gives the WiScore. Image generation and editing Mar 2025 0.82 System cards only
ImgEdit-Bench Image editing models carry out edit instructions such as add, remove, replace, style transfer and background change. A GPT-4.1 judge scores each result from 1 to 5. Image generation and editing May 2025 4.12 System cards only
SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. Research science Oct 2026 51.2% In use
VideoHallucer Paired basic and hallucinated questions about videos detect intrinsic and extrinsic hallucinations. Scored by pair accuracy. Video understanding Jun 2024 59.6% Fading
Echo General AI Prediction Leaderboard Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models. Real-world events Mar 2026 1,018 Not in cards
Accelerate Bench Coding agents speed up 32 real scientific applications, such as GROMACS and OpenFOAM, on a fixed CPU allocation within 8 hours. A hidden verifier checks correctness, and Score@1 averages the share of runtime saved, with incorrect programs scoring zero. Speed and optimization Nov 2026 67.4% Not in cards
HWE-Bench v1.0 (RTL bug repair) Agents fix bugs in six open-source hardware projects. The benchmark reports the share of fixes that pass repository tests in simulation. Electronics Apr 2026 79.9% Not in cards
HelloBench Models generate long answers, summaries, conversations, text completions, and texts from writing instructions. Model judges apply task-specific checklists to the responses. Overall chat quality Sep 2024 48.5 Not in cards
WISE Verified Models generate images from the revised WISE prompts. The verified evaluation changes about 200 prompts and the judge, so original scores remain separate. Image generation and editing Apr 2026 0.82 Not in cards
OSWorld-MCP Computer-use agents complete OSWorld desktop tasks while they can also call 158 MCP tools across 7 common applications. The benchmark checks whether agents pick a tool call or GUI actions at the right time. It reports task accuracy, tool invocation rate and average completion steps. Desktop apps Oct 2025 56.6% Not in cards
HWE-Bench v1.1 (RTL bug repair) Agents fix hardware bugs in 417 repository tasks. The revised tests check whether each fix restores the required behavior. Electronics Nov 2026 99% Not in cards
OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. Embodied reasoning Sep 2026 43.9% Not in cards
RoughCut-Bench Models select and order video clips or mark speech to remove from a transcript. The evaluation compares cut plans with professional editors’ annotations. Video understanding Jul 2026 0.53 Not in cards
VibeVideoBench Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result. Video generation Oct 2026 4.11 Not in cards
DataPrep-Bench Models prepare training data across six domains, and a base model is fine-tuned on their output. Downstream results of that model give the score. Machine learning engineering May 2026 48.4 Not in cards
K12-Bench Models select curriculum concepts, prerequisite relationships, related knowledge, experimental evidence, and textbook locations. The 23,640 multi-select questions come from a textbook knowledge graph. Exact match and instance-level F1 compare selected options with graph-derived answers. Language reasoning May 2026 57.1% Not in cards
PCA-Bench Models interpret visual scenes, explain their reasoning, and select actions for embodied tasks. GPT-4-0125 compares perception concepts, reasoning, and actions with annotated references. Embodied reasoning Feb 2024 0.70 Not in cards
ProOPF-B Models translate operational requirements into executable optimal power flow programs. They change grid parameters, objectives, and constraints. The evaluator checks implementations against expert references. Scientific code Feb 2026 37.2% Not in cards
DataPrep-Bench Finance Models prepare training data for the finance domain. The score is the published downstream average after backbone training. Machine learning engineering May 2026 59.3 Not in cards
DataPrep-Bench General Models prepare training data for the general domain. The score is the published downstream average after backbone training. Machine learning engineering May 2026 78.2 Not in cards
DataPrep-Bench Law Models prepare training data for the law domain. The score is the published downstream average after backbone training. Machine learning engineering May 2026 76.4 Not in cards
DataPrep-Bench Math Models prepare training data for the math domain. The score is the published downstream average after backbone training. Machine learning engineering May 2026 25.5 Not in cards
DataPrep-Bench Medical Models prepare training data for the medical domain. The score is the published downstream average after backbone training. Machine learning engineering May 2026 40.5 Not in cards
DataPrep-Bench Science Models prepare training data for the science domain. The score is the published downstream average after backbone training. Machine learning engineering May 2026 26.8 Not in cards
CMI-RewardBench Models judge musical quality and how well recordings follow requests. Some requests combine text, lyrics, and reference audio. Scores measure agreement with human listeners. Audio understanding and voice Feb 2026 0.60 Not in cards
AndroidArena Agents use apps such as Gmail and Google Maps, sometimes switching apps or following extra constraints. GPT-4 judges success from the recorded actions and observations. Phones Feb 2024 12.7% Not in cards
ContinualSkillBench Agents work through five domains of 100 linked subtasks of rising difficulty, testing whether skill libraries they build in context actually help later tasks. Learning from the context Aug 2026 64.1% Not in cards
Duplex-MPE Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios. Speech recognition and translation Sep 2026 72.3% Not in cards
EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. Simulated robots Oct 2026 52% Not in cards
LongAV-Compass Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time. Video generation May 2026 0.90 Not in cards
RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. Simulated robots Mar 2026 10.4% Not in cards
ChemCoTBench Models interpret and edit molecular structures, optimize molecular properties and predict reaction outcomes. Each task has its own scoring rule. Science questions May 2025 Not in cards
Chinese Literature Named Entity Recognition and Relation Extraction Models mark named entities in Chinese literary passages and identify relationships between them. The authors score both tasks with F1. Reading in many languages Nov 2017 Not in cards
ChronoMagic-Bench Video models generate time-lapse scenes from 1,649 prompts covering biological, human-made, weather, and physical changes. Automatic metrics measure the amount of change and temporal coherence, with a separate GPT-4o assessment of change. Video generation Jun 2024 Not in cards
MindWorldBench Models animate a scene to show how a person would act given their beliefs or intentions. Paired prompts change mental states while holding the scene fixed. Video generation Sep 2026 Not in cards
PhysCapBench Models caption videos of physical interactions. The benchmark checks which motions, contacts, and changes of state the captions describe across 1,500 clips. Video captioning Nov 2026 Not in cards
TraceSpatial-Bench Models plan an object-centric 3D trace, an ordered list of metric 3D waypoints, from language instructions in real cluttered indoor scenes with depth. Embodied reasoning Dec 2025 Not in cards
TriWorldBench World models generate synchronized robot-view videos from head and wrist cameras. Nineteen signals measure cross-view agreement, task alignment, physical coherence, motion, temporal stability, and visual quality. Video world models Sep 2026 Not in cards