Benchmark Almanac

University of Oxford

We track 17 benchmarks that University of Oxford made or helped make.

1 in use 9 with public code 8 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
TruthfulQA Models answer 817 questions across 38 categories that are written to elicit common misconceptions. Scoring checks for truthful answers, often in multiple-choice form. Short factual questions Sep 2021 88% Fading
HorizonMath Models attack 113 mostly unsolved problems in eight areas of mathematics and the mathematical sciences. A program checks each proposed closed form, construction or bound. Research mathematics Mar 2026 11.5% In use
NarrativeQA Models answer questions about books and movie scripts, either from a Wikipedia summary or from the full story. The questions need an understanding of the whole narrative rather than one matching passage. Answers are free text, scored against two reference answers with F1, ROUGE-L or BLEU. Reading comprehension Dec 2017 0.80 Not in cards
LLM-WikiRace Models play the Wikipedia game: starting from one article, they follow one hyperlink at a time to reach a target article. At each step the model sees only the links on the current page, and a game is lost after 30 steps. Three difficulty levels are set by the shortest path between the two pages. Scores are the share of games won at each level. Puzzles and chess Feb 2026 23% Not in cards
LingOly-TOO Models solve Linguistics Olympiad problems whose spelling patterns have been systematically changed while preserving the solution logic. Answers receive exact-match credit with no partial credit. The evaluation compares performance on original and obfuscated problems. Reasoning in many languages Mar 2025 46.7% Not in cards
VoxCeleb2 Models compare two interview recordings and decide whether the same person speaks in both. Audio understanding and voice Jun 2018 75.9% Not in cards
VCBench Models predict venture outcomes from founder profiles. The benchmark emphasizes precision through the F0.5 score. The hub adaptation substitutes synthetic profiles for the original data. Finance Sep 2025 33.7% Not in cards
CUSP Models forecast scientific progress across 4,760 events in many fields, using only knowledge from before a set cutoff date. Tasks ask whether an advance is feasible, how it would work, how to design a solution and when it will happen. The benchmark also compares events before and after each model's training cutoff. Real-world events May 2026 60.4% Not in cards
CMD-AD-Eval Models describe what happens on screen during gaps in movie dialogue. The authors compare the descriptions with timed human narration using text overlap and model judgments. Video captioning Apr 2024 Not in cards
DEBAGREEMENT Models classify whether a reply agrees with a comment, disagrees, or takes a neutral position. Human labels provide the reference answers. Reading comprehension Oct 2021 Not in cards
MAD-Eval Models write descriptions for scenes from ten movies. One set of references names characters. Another replaces their names with the word “SOMEONE.” Video captioning Mar 2023 Not in cards
OpenLex3D Models label points and retrieve objects in indoor 3D scans. Annotators distinguish an object’s synonyms from pictures on its surface and visually similar objects. Spatial reasoning and counting Mar 2025 Not in cards
Oxford Flowers 102 Models identify flower categories from photographs. The benchmark tests recognition across changes in shape, scale, and lighting. General visual questions Dec 2008 Not in cards
Oxford-IIIT Pet Dataset Models identify the breed of a cat or dog from a photograph. Meta measures breed classification across 37 categories. General visual questions Not in cards
TV-AD Models describe what viewers can see during clips from television episodes. The test pairs each clip with a timed reference description. Video captioning Jul 2024 Not in cards
VGGSound Models identify sound events in video clips where the sound source is visible. Audio understanding and voice Apr 2020 Not in cards
VoxCeleb1 Models identify speakers from recorded speech. NVIDIA also uses the recordings for speaker counting within AIR-Bench. Audio understanding and voice Jun 2017 Not in cards