Benchmark Almanac

University of Illinois Urbana-Champaign

We track 31 benchmarks that University of Illinois Urbana-Champaign made or helped make.

1 in use 27 with public code 24 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
DynaMath Models answer visual math questions generated from 501 seed programs that vary values and graphs. Scored by average and worst-case accuracy across variants. Visual math and diagrams Oct 2024 88% Fading
SEC-bench Pro Agents hunt and reproduce security bugs in large software projects from limited information. The first release covers V8 and SpiderMonkey. Finding and exploiting real bugs May 2026 90.5% In use
MIRAGE-MMST Models answer growers’ questions using photographs and written descriptions. They identify the cause of a problem and recommend how to manage it. General visual questions Jun 2025 68.0 Not in cards
EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. Simulated robots Feb 2025 50.5% Not in cards
DrawingVQA Models answer 92 multiple-choice questions about 33 construction drawings. They read symbols, cross-reference details, and calculate quantities. Documents, charts and text in images Apr 2026 77.2% Not in cards
LMBuild Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. CAD models Oct 2026 66.5% Not in cards
EmbodiedBench EB-ALFRED Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups. Simulated robots Feb 2025 67.7% Not in cards
EmbodiedBench EB-Habitat Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage. Simulated robots Feb 2025 58.7% Not in cards
EmbodiedBench EB-Manipulation Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage. Simulated robots Feb 2025 28.5% Not in cards
EmbodiedBench EB-Navigation Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage. Simulated robots Feb 2025 53.7% Not in cards
AI AgriBench Models answer 416 agronomy questions from agricultural extension publications. Three judges compare each answer with an expert-reviewed reference across four metrics. Tasks from real jobs Jan 2026 98.1 Not in cards
AdaPlanBench Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear. General assistants Jun 2026 67.8% Not in cards
PlanBench-XL Tool-use agents solve 327 retail tasks over 1,665 tools that they must first retrieve, chaining calls to uncover evidence. An optional blocking mode simulates tool failures. Function calling Jun 2026 77.1% Not in cards
AI AgriBench Smallholders Models answer questions that smallholder farmers sent to advisory services. A judge scores the responses against answers from the advisors. Tasks from real jobs 80.9 Not in cards
CSTutorBench Models tutor students who use block-based programming to control a virtual robot. A rubric grades the feedback on 17 scenarios, including debugging and answer disclosure. Other professions Jul 2026 0.89 Not in cards
MonitorBench Models work through 1,540 instances across 19 tasks in 7 categories, each built around a decision-critical factor such as a cue, a hidden goal or a required step. A monitor LLM reads the chain of thought, and the monitorability score measures how often it surfaces that factor. Deception and alignment Mar 2026 91.9% Not in cards
AI AgriBench Post-Cutoff Models answer 146 questions from newer agricultural extension documents. Judges score their answers for accuracy, relevance, completeness and conciseness. Tasks from real jobs Jan 2026 98.1 Not in cards
OpenFrontiers-Bench Agents try to rediscover proofs of research math problems, most resolved during 2026, stated in Lean 4. Two independent checkers grade the formal proofs, with closed-book, frozen-retrieval and live-web settings. Formal proofs Oct 2026 42.2% Not in cards
ReactionBench Models rank a viewer’s reactions, match a reaction to one of four video clips, and explain which cue prompted it. Video understanding Oct 2026 0.51 Not in cards
VGI-Bench Video generation models solve 27 visual reasoning tasks with 810 instances through generated frames. Scoring checks that the process is valid, not only the final frame. Video world models Aug 2026 51% Not in cards
MIRAGE-MMMT Models decide whether to ask for more information or answer a grower during an agricultural consultation. They then write the question or answer. Multi-turn conversation Jun 2025 65.5% Not in cards
SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. Finding and exploiting real bugs Jun 2026 58.4% Not in cards
ProtocolLLM Models write SystemVerilog modules for four communication protocols. The authors check syntax, synthesis, and whether simulated signals meet protocol timing requirements. Electronics Jun 2025 77.8% Not in cards
RefuNQ Models answer factual questions and decide when to decline. The authors create unanswerable questions by replacing a noun in a real question with an invented term. Short factual questions Nov 2023 53.7% Not in cards
Fire360 Models interpret panoramic video from firefighter training drills. They answer questions, caption actions, assess procedures, locate equipment, and match damaged objects to clean examples. Video understanding Jun 2025 53.8% Not in cards
SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. Finding and exploiting real bugs Jun 2025 18% Not in cards
GTFS Retrieval Models query a Chicago bus schedule from four routes. They answer directly or generate Python code whose output is compared with a reference answer. Expert-written questions Jul 2023 Not in cards
GTFS Semantics Models answer multiple-choice questions about the General Transit Feed Specification. Questions cover transit data files, field definitions, and relationships between tables. Expert-written questions Jul 2023 Not in cards
Minecraft Builder Action Prediction Models use dialogue and the current block layout to predict which blocks a Minecraft builder will place or remove. Net action F1 measures agreement with human moves. Embodied reasoning Jul 2020 Not in cards
Mobile-Eval-E Agents complete 25 smartphone tasks across 15 apps. Human reviewers score the share of task requirements each agent meets. Phones Jan 2025 Not in cards
TailOR Models generate images and videos of regular, unconventional, and impossible tool interactions. Annotators score instruction adherence, interaction accuracy, physical realism, and perceptual quality. Physical prediction Jun 2026 Not in cards