Benchmark Almanac

University of Waterloo

We track 11 benchmarks that University of Waterloo made or helped make.

1 in use 9 with public code 6 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
BrowseComp-Plus Deep research agents answer 830 BrowseComp questions by searching a fixed corpus of about 100,000 human-verified documents. Deep research Aug 2025 70.1% In use
Mantis-Eval Models answer questions that need reasoning across several interleaved images. Scored by accuracy. General visual questions May 2024 51.1% Dropped
TheoremQA Models solve 800 university-level questions in math, physics, EE and CS, and finance that each need a specific theorem. Scored by answer accuracy. Science questions May 2023 44.4% Dropped
MMEB-V2 Multimodal embedding models are scored on 78 tasks over images, videos and visual documents. V2 adds visual document retrieval, video retrieval, temporal grounding, video classification and video QA. General visual questions May 2025 77.9% Fading
FaithJudge Models answer retrieval-augmented tasks from given passages. An LLM judge guided by human-annotated examples flags unfaithful answers, giving a hallucination rate. Faithful to sources May 2025 6.26% System cards only
CodeEditorBench Models edit code in four scenarios: debugging, translation, requirement switching and polishing. Unit tests check the edits, and polishing also measures speed and memory gains. Function-level problems Apr 2024 85.5% System cards only
AfroBench-Lite Models complete language understanding, question answering, reasoning, and generation tasks across 14 African languages. Seven datasets use task-specific metrics, including accuracy, F1, character-level translation scores, and exact match. Knowledge in many languages Jun 2025 77.7 Not in cards
FreshStack Retrieval systems find documentation that supports answers to technical questions. The benchmark measures how many required facts the retrieved passages cover. Questions that need search Apr 2025 0.41 Not in cards
Known-Unknown Questions Models distinguish questions with established answers from questions with no definitive answer. The authors score this distinction using F1 and equal error rate. Short factual questions May 2023 0.56 Not in cards
MMEB Multimodal embedding models are scored on classification, visual question answering, multimodal retrieval and visual grounding, with 16 evaluation datasets in and out of distribution. General visual questions Oct 2024 80.1% Not in cards
MMEB-V3 Omni-modality embedding models are scored on 190 tasks across text, image, video, audio, visual documents and agent-centric retrieval. General visual questions Apr 2026 32.1% Not in cards