Benchmark Almanac

M-A-P

We track 6 benchmarks that M-A-P made or helped make.

2 in use 6 with public code 4 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Expert-written questions Feb 2025 73.9% In use
NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. Building whole projects Dec 2025 68.2% In use
FormalMATH Provers write Lean 4 proofs for 5,560 formal statements from high-school olympiad problems to undergraduate theorems. The Lean checker scores each proof. Formal proofs May 2025 28.3% System cards only
TableBench Models answer 886 questions about real-world tables across 18 fields in four groups: fact checking, numerical reasoning, data analysis and visualization. Fact and numeric answers are scored by exact match, analysis answers by ROUGE-L, and chart code by pass@1. Language reasoning Aug 2024 61.7% Not in cards
CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. General visual questions Jan 2024 36.8% Not in cards
FormalMATH Lite Models prove statements in the FormalMATH Lite subset. The board reports proof success at the stated sampling budget. Formal proofs 61.9% Not in cards