Benchmark Almanac

Mila

We track 8 benchmarks that Mila made or helped make.

0 in use 4 with public code 5 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
HotpotQA Models answer 113k Wikipedia questions that need facts from several documents. The distractor setting supplies gold and distractor paragraphs. Scored by exact match or F1. Reading comprehension Sep 2018 92.1% System cards only
DataAnalyticsBench Agents query a synthetic data warehouse to answer business questions. A judge model checks answers against a rubric and an answer key. Running a business Oct 2026 43.0 Not in cards
AfroBench-Lite Models complete language understanding, question answering, reasoning, and generation tasks across 14 African languages. Seven datasets use task-specific metrics, including accuracy, F1, character-level translation scores, and exact match. Knowledge in many languages Jun 2025 77.7 Not in cards
BabyAI Agents follow compositional English instructions to navigate rooms, open doors, and move objects in a grid world. The environment checks whether each instruction was completed, and evaluation reports task success rates. Simulated robots Oct 2018 37.5% Not in cards
AgentRewardBench Models judge whether web-agent trajectories completed their tasks. Their judgments are compared with expert labels across 1,302 trajectories from five benchmarks, using precision, recall, and F1. Web browsing Apr 2025 82% Not in cards
WebLINX Agents follow a user's multi-turn chat instructions to navigate real websites. 100,000 interactions from 2,300 expert demonstrations give turn-level metrics. Web browsing Feb 2024 10.4 Not in cards
MatSci-NLP Models read text from materials science papers and complete seven tasks, including identifying synthesis actions and linking measurements to experiments. The authors used 99% of examples for testing. Lab and data work May 2023 Not in cards
OpenLex3D Models label points and retrieve objects in indoor 3D scans. Annotators distinguish an object’s synonyms from pictures on its surface and visually similar objects. Spatial reasoning and counting Mar 2025 Not in cards