Benchmark Almanac

University of Maryland

We track 6 benchmarks that University of Maryland made or helped make.

2 in use 6 with public code 5 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
HallusionBench Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. Hallucination rates Oct 2023 73.5% In use
MMAU 10K audio clips with questions on speech, sound and music needing expert knowledge and reasoning. Multiple-choice accuracy. Audio understanding and voice Oct 2024 82.2% In use
MMAU-Pro Models answer 5,305 expert-written questions about speech, sounds, music and their mixes, covering 49 listening skills. Some clips run up to 10 minutes. Audio understanding and voice Aug 2025 72.2% Fading
ColorBench Models answer image questions about color perception, color reasoning, and robustness to color changes. The evaluator checks the selected answers against a single correct option. General visual questions Apr 2025 59.6% Not in cards
MMAU v05.15.25 Models answer multiple-choice questions about speech, sound and music clips that need expert knowledge and reasoning. Scored by accuracy on test-mini or the hidden-answer test set. Audio understanding and voice May 2025 77.8% Not in cards
Mythos Models write a new story in an author’s style using earlier stories as examples. Judges compare the result with the author’s writing. Creative writing Feb 2025 Not in cards