Benchmark Almanac

MMMU team

We track 2 benchmarks that MMMU team made or helped make.

2 in use 2 with public code 2 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. General visual questions Nov 2023 86% In use
MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. General visual questions Sep 2024 84.6% In use