Benchmark Almanac

Yale University

We track 7 benchmarks that Yale University made or helped make.

0 in use 6 with public code 4 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
TOMATO Models answer 1,484 human-written questions about 1,417 videos that need visual temporal reasoning, such as action counts, direction, rotation and velocity. Scored by accuracy. Video understanding Oct 2024 79.5% Fading
Spider Models turn natural language questions into SQL over 200 multi-table databases from 138 domains. Test databases are unseen in training, and answers are scored by exact match or execution. Data science and SQL Sep 2018 83.7% Dropped
PHYSICS Models solve 1,297 PhD qualifying exam physics problems across six subfields, 298 of them with images. Answers are checked by symbolic equivalence. Science questions Mar 2025 59.9% Dropped
PHYSICS-Verified Corrected PHYSICS test problems with answer-level accuracy. The complete verified test set. Science questions Sep 2026 91.1% Not in cards
ChemCoTBench Models interpret and edit molecular structures, optimize molecular properties and predict reaction outcomes. Each task has its own scoring rule. Science questions May 2025 Not in cards
EBM-NLPmod Models extract participants, interventions, comparators, and outcomes from trial titles and methods sections. Annotators separate comparison treatments from interventions. Medicine Aug 2023 Not in cards
QMSum Models answer a query about a long meeting transcript with a written summary of the relevant parts. The 1,808 query and summary pairs cover 232 meetings from several domains. Scores compare outputs with reference summaries using ROUGE. Reasoning over long documents Apr 2021 Not in cards