University of Washington
We track 33 benchmarks and 1 arena that University of Washington made or helped make.
1 in use 25 with public code 14 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| HellaSwag Commonsense sentence completion with adversarially filtered wrong endings. Multiple-choice accuracy. | Commonsense | May 2019 | | Fading |
| TriviaQA Trivia questions with evidence documents. Scored by exact match. | Trivia and open questions | May 2017 | | Fading |
| SIQA Models answer multiple-choice questions about people's motives, reactions, and emotions in everyday social situations. Scored by accuracy. | Commonsense | Apr 2019 | | Dropped |
| JobBench 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. | Tasks from real jobs | May 2026 | 61.4% | In use |
| QuAC Models answer free-form questions in information-seeking dialogs about a hidden Wikipedia section, across 100K questions. Scored by word-level F1. | Reading comprehension | Aug 2018 | | Dropped |
| PopQA Models answer about 14k entity questions built from Wikidata, spread from popular to rare entities. Scored by whether the answer matches a gold alias. | Trivia and open questions | Dec 2022 | | Fading |
| GenEval Text-to-image models draw prompts about object count, color, position and co-occurrence, and an object detector checks each image. Scored by the share of correct images. | Image generation and editing | Oct 2023 | | Fading |
| RealToxicityPrompts Models continue 100,000 web text prompts. Scored on the rate of toxic generations judged by a toxicity classifier. | Refusing harmful requests | Sep 2020 | | System cards only |
| WildJailbreak Models face adversarial jailbreak prompts built from in-the-wild tactics, plus benign prompts that look harmful. Evaluations score refusals of harmful requests and compliance with benign ones. | Jailbreaks and prompt injection | Jun 2024 | | Fading |
| MindCube Models answer 21,154 multiple choice questions about scenes seen from a few views. Questions test positions, orientations, and what-if movements. | Spatial reasoning and counting | Jun 2025 | | Fading |
| Where2Place 100 cluttered real photos with free-space placement instructions. The model points, scored correct if inside the ground-truth mask. | Embodied reasoning | Jun 2024 | | System cards only |
| OMEGA Models solve generated competition-style math problems that test exploratory, compositional and transformative generalization beyond training patterns. Answers are checked automatically. | Competition problems with answers | Jun 2025 | | Fading |
| RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. | Simulated robots | Jul 2025 | | Fading |
| CulturalBench Models answer 1,696 human-written and human-verified questions on cultural knowledge across 45 global regions and 17 topics. Scored by accuracy. | Knowledge in many languages | Oct 2024 | 62% | System cards only |
| WSC Models decide whether a pronoun refers to a given noun phrase in sentences that need commonsense to resolve. Scored by accuracy. | Commonsense | May 2019 | 86.9% | Dropped |
| SAT (Spatial Aptitude Training) Models answer static and dynamic spatial questions, such as how object relations change when the camera or an object moves. The real-image test set has 150 questions. | Spatial reasoning and counting | Dec 2024 | | System cards only |
| Bamboogle Models answer 125 two-hop questions written so that a search engine cannot answer them directly. Models usually search and compose the facts. | Questions that need search | Oct 2022 | | System cards only |
| MathQA Models answer multiple-choice math word problems drawn from AQuA. Each problem is annotated with an operation program. Scored by accuracy. | Word problems | May 2019 | | System cards only |
| STARE Models solve 4K spatial tasks such as 2D and 3D transformations, cube net folding, tangrams, perspective and temporal reasoning that people solve by visual simulation. | Spatial reasoning and counting | Jun 2025 | | System cards only |
| SuperGLUE Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score. | Language reasoning | May 2019 | | Not in cards |
| WeatherQA (severe weather) Models read weather maps to identify a threatened area and classify its severe weather potential. The second score includes only cases with the correct area. | Science questions | Jun 2024 | | Not in cards |
| MonitorBench Models work through 1,540 instances across 19 tasks in 7 categories, each built around a decision-critical factor such as a cue, a hidden goal or a required step. A monitor LLM reads the chain of thought, and the monitorability score measures how often it surfaces that factor. | Deception and alignment | Mar 2026 | 91.9% | Not in cards |
| HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. | Simulated robots | Jul 2026 | | Not in cards |
| Infrathon Agents optimize AI infrastructure under correctness and fidelity constraints. Hidden workloads measure performance gains, with additional checks for rule compliance. | Speed and optimization | Nov 2026 | 91.2 | Not in cards |
| Humor in AI: Cartoon Captioning Models write captions for New Yorker cartoons. Human judges and language models compare the captions with contest entries from different ranking groups. | Creative writing | Jun 2024 | 73.9% | Not in cards |
| Agentic Abstention Agents work in web shopping, terminal and search question tasks where some goals cannot be met. They must abstain at the right turn instead of acting on. | Hallucination rates | Jun 2026 | Not in cards | |
| ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. | Simulated robots | Dec 2019 | Not in cards | |
| Defects4J A collection of 854 real, reproducible bugs from 17 open-source Java projects, each with the buggy and fixed versions and tests that expose the bug. Program repair studies ask a model to fix each bug, and a fix counts when the project's tests pass. It also serves fault localization and test generation research. | Fixing real repositories | Jul 2014 | Not in cards | |
| GenEval 2 Models generate images from compositional prompts. A visual question answering model checks whether each requested object, attribute, count, and relation appears. | Image generation and editing | Dec 2025 | Not in cards | |
| GLUE Models predict labels or similarity scores across nine English language-understanding tasks. The overall score averages task scores. Tasks with several metrics first average those metrics. | Language reasoning | Apr 2018 | Not in cards | |
| MAWPS Models solve one-unknown arithmetic and algebra word problems drawn from several earlier collections. Each problem has an equation and a numeric answer, and the final answer is scored by exact match. | Word problems | Jun 2016 | Not in cards | |
| NaturalProofs Models retrieve or generate references needed for a mathematical theorem's proof. Ranking and sequence metrics compare predicted references with those used in the source proof. | Written proofs | Apr 2021 | Not in cards | |
| UW Clinical Record Section Chunker i2b2 Discharge Summary Corpus Models locate section boundaries in discharge summaries and label each section. Researchers compare the predicted labels and boundaries with human annotations. | Medicine | May 2012 | Not in cards |
Arenas
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Husky Hold'em Bench Models write Python poker bots and revise them after practice games. Six bots compete at each table, and the tournament tracks average winnings. | Multi-agent games | Jul 2025 | Not in cards |