Benchmark Almanac

University of Washington

We track 33 benchmarks and 1 arena that University of Washington made or helped make.

1 in use 25 with public code 14 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
HellaSwag Commonsense sentence completion with adversarially filtered wrong endings. Multiple-choice accuracy. Commonsense May 2019 97% Fading
TriviaQA Trivia questions with evidence documents. Scored by exact match. Trivia and open questions May 2017 91.8% Fading
SIQA Models answer multiple-choice questions about people's motives, reactions, and emotions in everyday social situations. Scored by accuracy. Commonsense Apr 2019 80.2% Dropped
JobBench 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. Tasks from real jobs May 2026 61.4% In use
QuAC Models answer free-form questions in information-seeking dialogs about a hidden Wikipedia section, across 100K questions. Scored by word-level F1. Reading comprehension Aug 2018 51.1% Dropped
PopQA Models answer about 14k entity questions built from Wikidata, spread from popular to rare entities. Scored by whether the answer matches a gold alias. Trivia and open questions Dec 2022 55.5% Fading
GenEval Text-to-image models draw prompts about object count, color, position and co-occurrence, and an object detector checks each image. Scored by the share of correct images. Image generation and editing Oct 2023 90.1% Fading
RealToxicityPrompts Models continue 100,000 web text prompts. Scored on the rate of toxic generations judged by a toxicity classifier. Refusing harmful requests Sep 2020 0.20% System cards only
WildJailbreak Models face adversarial jailbreak prompts built from in-the-wild tactics, plus benign prompts that look harmful. Evaluations score refusals of harmful requests and compliance with benign ones. Jailbreaks and prompt injection Jun 2024 98.5% Fading
MindCube Models answer 21,154 multiple choice questions about scenes seen from a few views. Questions test positions, orientations, and what-if movements. Spatial reasoning and counting Jun 2025 47.6% Fading
Where2Place 100 cluttered real photos with free-space placement instructions. The model points, scored correct if inside the ground-truth mask. Embodied reasoning Jun 2024 64% System cards only
OMEGA Models solve generated competition-style math problems that test exploratory, compositional and transformative generalization beyond training patterns. Answers are checked automatically. Competition problems with answers Jun 2025 53.4% Fading
RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. Simulated robots Jul 2025 44.3% Fading
CulturalBench Models answer 1,696 human-written and human-verified questions on cultural knowledge across 45 global regions and 17 topics. Scored by accuracy. Knowledge in many languages Oct 2024 62% System cards only
WSC Models decide whether a pronoun refers to a given noun phrase in sentences that need commonsense to resolve. Scored by accuracy. Commonsense May 2019 86.9% Dropped
SAT (Spatial Aptitude Training) Models answer static and dynamic spatial questions, such as how object relations change when the camera or an object moves. The real-image test set has 150 questions. Spatial reasoning and counting Dec 2024 76.7% System cards only
Bamboogle Models answer 125 two-hop questions written so that a search engine cannot answer them directly. Models usually search and compose the facts. Questions that need search Oct 2022 76% System cards only
MathQA Models answer multiple-choice math word problems drawn from AQuA. Each problem is annotated with an operation program. Scored by accuracy. Word problems May 2019 33.9% System cards only
STARE Models solve 4K spatial tasks such as 2D and 3D transformations, cube net folding, tangrams, perspective and temporal reasoning that people solve by visual simulation. Spatial reasoning and counting Jun 2025 61.8% System cards only
SuperGLUE Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score. Language reasoning May 2019 90.4% Not in cards
WeatherQA (severe weather) Models read weather maps to identify a threatened area and classify its severe weather potential. The second score includes only cases with the correct area. Science questions Jun 2024 62.7% Not in cards
MonitorBench Models work through 1,540 instances across 19 tasks in 7 categories, each built around a decision-critical factor such as a cue, a hidden goal or a required step. A monitor LLM reads the chain of thought, and the monitorability score measures how often it surfaces that factor. Deception and alignment Mar 2026 91.9% Not in cards
HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. Simulated robots Jul 2026 11.1% Not in cards
Infrathon Agents optimize AI infrastructure under correctness and fidelity constraints. Hidden workloads measure performance gains, with additional checks for rule compliance. Speed and optimization Nov 2026 91.2 Not in cards
Humor in AI: Cartoon Captioning Models write captions for New Yorker cartoons. Human judges and language models compare the captions with contest entries from different ranking groups. Creative writing Jun 2024 73.9% Not in cards
Agentic Abstention Agents work in web shopping, terminal and search question tasks where some goals cannot be met. They must abstain at the right turn instead of acting on. Hallucination rates Jun 2026 Not in cards
ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. Simulated robots Dec 2019 Not in cards
Defects4J A collection of 854 real, reproducible bugs from 17 open-source Java projects, each with the buggy and fixed versions and tests that expose the bug. Program repair studies ask a model to fix each bug, and a fix counts when the project's tests pass. It also serves fault localization and test generation research. Fixing real repositories Jul 2014 Not in cards
GenEval 2 Models generate images from compositional prompts. A visual question answering model checks whether each requested object, attribute, count, and relation appears. Image generation and editing Dec 2025 Not in cards
GLUE Models predict labels or similarity scores across nine English language-understanding tasks. The overall score averages task scores. Tasks with several metrics first average those metrics. Language reasoning Apr 2018 Not in cards
MAWPS Models solve one-unknown arithmetic and algebra word problems drawn from several earlier collections. Each problem has an equation and a numeric answer, and the final answer is scored by exact match. Word problems Jun 2016 Not in cards
NaturalProofs Models retrieve or generate references needed for a mathematical theorem's proof. Ranking and sequence metrics compare predicted references with those used in the source proof. Written proofs Apr 2021 Not in cards
UW Clinical Record Section Chunker i2b2 Discharge Summary Corpus Models locate section boundaries in discharge summaries and label each section. Researchers compare the predicted labels and boundaries with human annotations. Medicine May 2012 Not in cards

Arenas

Name Kind Released Best score Status
Husky Hold'em Bench Models write Python poker bots and revise them after practice games. Six bots compete at each table, and the tournament tracks average winnings. Multi-agent games Jul 2025 Not in cards