Benchmark Almanac

Carnegie Mellon University

We track 48 benchmarks and 2 arenas that Carnegie Mellon University made or helped make. Its models are on its lab page.

0 in use 38 with public code 34 with a leaderboard here

Arena leaderboards

Top five in each of its 1 arena with three or more rated models, the most rated first. We don't have a leaderboard yet for one other, listed below.

Music Arena

Users write a music prompt, listen to tracks from two text-to-music systems, and vote. Preferences, listening data and feedback feed a leaderboard.

  1. Lyria 3 Pro 1,394
  2. Stable Audio 3 Medium 1,333
  3. Lyria 3 Clip 1,003

Official leaderboard

Benchmarks

Name Kind Released Best score Status
WebArena Agents complete 812 realistic tasks on self-hosted shopping, forum, GitLab, CMS and map websites. Functional graders check the outcome. Web browsing Jul 2023 68% Fading
RACE-H Models answer multiple-choice reading questions from English exams for Chinese high school students. The test split has 3,498 questions, scored by accuracy. Reading comprehension Apr 2017 92.9% Dropped
RACE Models answer reading comprehension questions from Chinese middle and high school English exams. Reported here without a subset split. Reading comprehension Apr 2017 94.2% System cards only
VisualPuzzles Models solve visual puzzles that need algorithmic, analogical, deductive, inductive and spatial reasoning but little domain knowledge. Scored by accuracy. Spatial reasoning and counting Apr 2025 57% Fading
PubMedQA Models answer biomedical research questions with yes, no, or maybe, given a PubMed abstract without its conclusion. Scored by accuracy. Medicine Sep 2019 78.3% Dropped
OK-VQA Models answer over 14,000 visual questions that need outside knowledge beyond the image. Scoring is VQA accuracy against human answers. General visual questions May 2019 60.2% Dropped
RACE-M Models read passages from English exams for Chinese middle school students and answer multiple-choice questions. Scored by accuracy. Reading comprehension Apr 2017 77% Dropped
MAVERIX Models answer 2,556 questions about 700 videos that need both the video and its audio. Questions come in multiple-choice and open-ended forms. Video understanding Mar 2025 64.1% Fading
xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. General visual questions Oct 2024 41.4% System cards only
miniCTX-v2 Provers prove Lean theorems from real projects using long repository context, with theorems taken after a November 28, 2024 cutoff. Formal proofs Mar 2025 81.8% Dropped
HotpotQA Models answer 113k Wikipedia questions that need facts from several documents. The distractor setting supplies gold and distractor paragraphs. Scored by exact match or F1. Reading comprehension Sep 2018 92.1% System cards only
TheAgentCompany Agents do work tasks in a simulated software company, using a browser, code, and chat with simulated coworkers. Scored by full and partial checkpoint completion. Simulated companies Dec 2024 Dropped
xChatBench Vision language models answer 400 open-ended multimodal chat prompts in 7 languages. An LLM judge scores each answer against fine-grained human-written criteria. General visual questions Oct 2024 70.1 System cards only
FinGREAT Models identify accounting relationships in financial passages. The authors compare 44 models against human annotations across 483 passages. Finance Oct 2026 92.8% Not in cards
CMIMC 2025 Models answer 40 final-answer problems from the 2025 Carnegie Mellon Informatics and Mathematics Competition. MathArena scores the average accuracy over four attempts. Competition problems with answers May 2025 95.6% Not in cards
GameDevBench Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result. Building whole projects Feb 2026 70.6% Not in cards
PianoBench Models answer questions about piano performances from recordings or symbolic transcriptions. The authors compare accuracy with and without access to the music. Audio understanding and voice Nov 2026 93.2% Not in cards
GSM-Infinite Models solve generated arithmetic problems across different context lengths and numbers of reasoning steps. The benchmark varies how many irrelevant details appear alongside each problem. Reasoning over long documents Feb 2025 8534.9 Not in cards
SusVibes Coding agents rebuild a feature in a real open-source Python project from a plain-language request. Each task comes from a historical vulnerability fix, so the original human version was insecure. Hidden functional tests and security tests check the result. Scores report the share of solutions that work and the share that are also secure. Secure code and review Dec 2025 43.5% Not in cards
CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. Simulated robots Mar 2026 32.3% Not in cards
MyPCBench Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks. Desktop apps Jun 2026 62% Not in cards
IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. Research science Jul 2026 27.3% Not in cards
K-BrowseComp Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers. Knowledge in many languages Jun 2026 45.7% Not in cards
Kinayat Models interpret Egyptian Arabic idioms, select appropriate contexts and identify connotations. The evaluation includes separate generation and multiple-choice tasks. Reasoning in many languages Oct 2025 95.3% Not in cards
EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. Research science Sep 2026 29.8 Not in cards
NaturalBench Models answer 10,000 human-checked questions on natural images. Each question pairs with two images that need different answers, so a model cannot pass without looking. General visual questions Oct 2024 55.1% Not in cards
SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. Genomics and life sciences May 2026 40.2% Not in cards
VisualWebArena Multimodal web agents complete 910 visually grounded tasks on self-hosted classifieds, shopping and Reddit-style sites. Execution-based checks score each task. Web browsing Jan 2024 16.4% Not in cards
iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. Phones Jun 2026 51.9% Not in cards
Gym-Anything-RT Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time. Video games Oct 2026 9.00% Not in cards
AnyAppBench Agents repeat the same task in different Android apps, with and without suggested steps. Each app has a verifier that checks the same success condition. Phones Sep 2026 67.8% Not in cards
CUA-World-Long Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task. Desktop apps Apr 2026 27.5% Not in cards
DS-1000 Models write Python for 1,000 data science problems from StackOverflow across seven libraries such as NumPy and Pandas. Test cases and API constraints check each answer. Data science and SQL Nov 2022 53.9% Not in cards
FieldWorkArena V2.0 Agents inspect workplace images, videos, manuals, and logs to plan actions and report operational events. The evaluator compares their answers and actions with reference results. It combines correctness judgments with partial credit for numerical agreement. Tasks from real jobs Jun 2025 52% Not in cards
Per-DOC Models use a reader’s previous comparison to predict which of two new plots the reader prefers on each of five criteria. Taste and style Oct 2023 0.21% Not in cards
Per-MPST Models use a reviewer’s earlier movie ratings and reviews to predict their rating of a new plot. The authors measure correlation with human ratings. Taste and style Oct 2023 18.1 Not in cards
RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. Simulated robots Oct 2026 0.06 Not in cards
TurnBench Turn-taking systems listen to two-person conversations and must detect when a speaker's turn ends and when the other speaker interrupts. The test set has 116 English conversations in six interaction styles, each labeled by three annotators. Systems rank by recall, and a system whose false-positive rate exceeds 0.15 ranks below every qualifying system. Latency after each event is also reported. Speech recognition and translation Aug 2026 65.7% Not in cards
LLM-SRBench Models infer scientific equations from observations on 239 problems across four scientific domains. Symbolic accuracy checks whether the discovered equation recovers the underlying relationship. Research science Apr 2025 15.1% Not in cards
MobileAgentBench Agents complete 100 tasks across ten Android apps. The grader checks app state and accessibility events, so agents can take different paths. Phones Jun 2024 8.00% Not in cards
AdvBench Models get harmful requests, such as a request for malware, and should refuse them. The set has 520 harmful behaviors and 574 harmful strings. Most papers add an attack, such as an adversarial suffix, and report the attack success rate. Jailbreaks and prompt injection Jul 2023 Not in cards
Alharthi LibriTTS Naturalness Listening Test Models estimate how natural synthesized speech sounds. Ten listeners rated each sample on a five-point scale for comparison. Audio understanding and voice Oct 2023 Not in cards
IdeaTaste Agents propose research ideas from literature frozen at 2025. Evaluation compares their citations and ideas with 2026 publications and measures diversity. AI research tasks Nov 2026 Not in cards
PAM-Audio Models judge the quality of generated environmental sounds and how closely each recording matches its caption. The listening study used 100 AudioCaps captions. Audio understanding and voice Feb 2024 Not in cards
RefAV Models find driving scenarios described in natural language. They identify when each scenario occurs and track the objects involved. Detection and captioning Mar 2025 Not in cards
SingMOS Models listen to Chinese and Japanese singing and predict how listeners rate its quality. The recordings include human voices and synthesized voices. Audio understanding and voice Jun 2024 Not in cards
Soohak Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems. Research mathematics May 2026 Not in cards
XTREME Multilingual models are fine-tuned on English data and then tested on 9 tasks in 40 languages. The tasks cover sentence classification, structured prediction such as named entities, question answering and sentence retrieval. The benchmark measures how well skills transfer across languages. Reading in many languages Mar 2020 Not in cards

Arenas without a leaderboard here

Name Kind Released Best score Status
Music Arena Vocal Models generate music with vocals. Listeners compare anonymous tracks, and separate vocal preference votes determine the Arena score. Audio understanding and voice Jul 2025 1,107 Not in cards