MIT
We track 27 benchmarks that MIT made or helped make.
2 in use 20 with public code 14 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| LiveCodeBench v6 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. | Competitive programming | Apr 2025 | | In use |
| LiveCodeBench Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. | Competitive programming | Mar 2024 | | In use |
| LiveCodeBench v5 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. | Competitive programming | Jan 2025 | | Fading |
| ToxiGen Models continue ToxiGen prompts about 13 minority groups. Scored by the average toxicity of the generations. | Bias, fairness and politics | Mar 2022 | | System cards only |
| MedQA Models answer multiple-choice questions from medical licensing exams, usually the 12,723-question English USMLE set. Scoring is accuracy. | Medicine | Sep 2020 | 86.3% | Dropped |
| ToxiGen (discriminative) Models label statements about 13 minority groups as toxic or neutral. Scored by accuracy, often split into toxic and neutral labels. | Bias, fairness and politics | Mar 2022 | | System cards only |
| MedQA Mainland China Models answer mainland China medical licensing exam questions in simplified Chinese. Scoring is accuracy. | Medicine | Sep 2020 | 84% | System cards only |
| MedQA Taiwan Models answer Taiwan medical licensing exam questions in traditional Chinese. Scoring is accuracy. | Medicine | Sep 2020 | 91% | System cards only |
| MedQA USMLE (5 options) Models answer USMLE-style medical licensing questions with five answer options. Scoring is accuracy. | Medicine | 86% | System cards only | |
| AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. | Machine learning engineering | Jun 2026 | 0.68 | Not in cards |
| DiG-bench Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt. | Puzzles and chess | Aug 2026 | 71.4% | Not in cards |
| EvoArena Agents work in environments whose knowledge, tools and task conditions change over time. The suite tests whether agent memory keeps up with the changes. | Learning from the context | Jun 2026 | 53.3% | Not in cards |
| EvoArena Terminal-Bench-Evo Agents solve terminal tasks that change across versions. The score is the published accuracy across versioned tasks. | Learning from the context | Jun 2026 | 65.1% | Not in cards |
| Baba Is AI Models see a puzzle and write a plan to reach the winning object. Some puzzles require them to push words together to change the rules. | Puzzles and chess | Jul 2024 | | Not in cards |
| ReactionBench Models rank a viewer’s reactions, match a reaction to one of four video clips, and explain which cue prompted it. | Video understanding | Oct 2026 | 0.51 | Not in cards |
| EnigmaEval 1,184 multimodal puzzle-hunt puzzles (v2 dataset has 1,161). Model gives one exact answer per puzzle. Normal and Hard splits. | General visual questions | Feb 2025 | 39.3% | Not in cards |
| OBLIQ-Bench Retrievers search large real corpora for documents that match a latent pattern, such as an implicit stance. The suite has five such oblique search problems. | Questions that need search | May 2026 | 0.12 | Not in cards |
| LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. | Simulated robots | Oct 2025 | 53% | Not in cards |
| Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. | Medicine | Jul 2021 | Not in cards | |
| EvoArena PersonaMem-Evo Agents solve questions about changing user preferences. The score is the published accuracy across versioned tasks. | Learning from the context | Jun 2026 | Not in cards | |
| EvoArena SWE-Chain-Evo Agents solve software tasks that change across versions. The score is the published accuracy across versioned tasks. | Learning from the context | Jun 2026 | Not in cards | |
| MIT-Adobe FiveK Models adjust photographic tone to match human edits. Five photographers each edited 5,000 raw photos in Lightroom. The original evaluation compares luminance with each reference edit. | Apps and MCP servers | Jun 2011 | Not in cards | |
| MUSIC Instrument Separation Models use images of musicians to isolate an instrument from a mixture of performances. The authors measure separation on mixtures of solo recordings, where each instrument has a reference track. | Audio understanding and voice | Apr 2018 | Not in cards | |
| MUSIC-21 Instrument Separation Models separate performances from a mixture using video of the musicians. When two musicians play the same instrument, their movements help identify which sound belongs to each. | Audio understanding and voice | Apr 2019 | Not in cards | |
| PubMed-PICO Models classify sentences in structured medical abstracts. Section headings supply the reference labels, including those for the study population, intervention, and outcome. | Medicine | Jun 2018 | Not in cards | |
| QuixBugs Models repair 40 classic algorithm programs, in Python and Java, that each contain a one-line bug. The bugs fall into 14 defect classes. A fix counts when the program passes the test cases. | Function-level problems | Jul 2017 | Not in cards | |
| StereoSet Models choose between stereotypical, anti-stereotypical and unrelated completions in four domains: gender, profession, race and religion. Scores combine language modeling and stereotype preference. | Bias, fairness and politics | Apr 2020 | Not in cards |