Benchmark Almanac

MIT

We track 27 benchmarks that MIT made or helped make.

2 in use 20 with public code 14 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
LiveCodeBench v6 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. Competitive programming Apr 2025 93.5% In use
LiveCodeBench Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. Competitive programming Mar 2024 93.5% In use
LiveCodeBench v5 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. Competitive programming Jan 2025 81.2% Fading
ToxiGen Models continue ToxiGen prompts about 13 minority groups. Scored by the average toxicity of the generations. Bias, fairness and politics Mar 2022 29.8 System cards only
MedQA Models answer multiple-choice questions from medical licensing exams, usually the 12,723-question English USMLE set. Scoring is accuracy. Medicine Sep 2020 86.3% Dropped
ToxiGen (discriminative) Models label statements about 13 minority groups as toxic or neutral. Scored by accuracy, often split into toxic and neutral labels. Bias, fairness and politics Mar 2022 85.5% System cards only
MedQA Mainland China Models answer mainland China medical licensing exam questions in simplified Chinese. Scoring is accuracy. Medicine Sep 2020 84% System cards only
MedQA Taiwan Models answer Taiwan medical licensing exam questions in traditional Chinese. Scoring is accuracy. Medicine Sep 2020 91% System cards only
MedQA USMLE (5 options) Models answer USMLE-style medical licensing questions with five answer options. Scoring is accuracy. Medicine 86% System cards only
AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. Machine learning engineering Jun 2026 0.68 Not in cards
DiG-bench Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt. Puzzles and chess Aug 2026 71.4% Not in cards
EvoArena Agents work in environments whose knowledge, tools and task conditions change over time. The suite tests whether agent memory keeps up with the changes. Learning from the context Jun 2026 53.3% Not in cards
EvoArena Terminal-Bench-Evo Agents solve terminal tasks that change across versions. The score is the published accuracy across versioned tasks. Learning from the context Jun 2026 65.1% Not in cards
Baba Is AI Models see a puzzle and write a plan to reach the winning object. Some puzzles require them to push words together to change the rules. Puzzles and chess Jul 2024 43.9% Not in cards
ReactionBench Models rank a viewer’s reactions, match a reaction to one of four video clips, and explain which cue prompted it. Video understanding Oct 2026 0.51 Not in cards
EnigmaEval 1,184 multimodal puzzle-hunt puzzles (v2 dataset has 1,161). Model gives one exact answer per puzzle. Normal and Hard splits. General visual questions Feb 2025 39.3% Not in cards
OBLIQ-Bench Retrievers search large real corpora for documents that match a latent pattern, such as an implicit stance. The suite has five such oblique search problems. Questions that need search May 2026 0.12 Not in cards
LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. Simulated robots Oct 2025 53% Not in cards
Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. Medicine Jul 2021 Not in cards
EvoArena PersonaMem-Evo Agents solve questions about changing user preferences. The score is the published accuracy across versioned tasks. Learning from the context Jun 2026 Not in cards
EvoArena SWE-Chain-Evo Agents solve software tasks that change across versions. The score is the published accuracy across versioned tasks. Learning from the context Jun 2026 Not in cards
MIT-Adobe FiveK Models adjust photographic tone to match human edits. Five photographers each edited 5,000 raw photos in Lightroom. The original evaluation compares luminance with each reference edit. Apps and MCP servers Jun 2011 Not in cards
MUSIC Instrument Separation Models use images of musicians to isolate an instrument from a mixture of performances. The authors measure separation on mixtures of solo recordings, where each instrument has a reference track. Audio understanding and voice Apr 2018 Not in cards
MUSIC-21 Instrument Separation Models separate performances from a mixture using video of the musicians. When two musicians play the same instrument, their movements help identify which sound belongs to each. Audio understanding and voice Apr 2019 Not in cards
PubMed-PICO Models classify sentences in structured medical abstracts. Section headings supply the reference labels, including those for the study population, intervention, and outcome. Medicine Jun 2018 Not in cards
QuixBugs Models repair 40 classic algorithm programs, in Python and Java, that each contain a one-line bug. The bugs fall into 14 defect classes. A fix counts when the program passes the test cases. Function-level problems Jul 2017 Not in cards
StereoSet Models choose between stereotypical, anti-stereotypical and unrelated completions in four domains: gender, profession, race and religion. Scores combine language modeling and stereotype preference. Bias, fairness and politics Apr 2020 Not in cards