M-A-P
We track 6 benchmarks that M-A-P made or helped make.
2 in use 6 with public code 4 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. | Expert-written questions | Feb 2025 | | In use |
| NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. | Building whole projects | Dec 2025 | | In use |
| FormalMATH Provers write Lean 4 proofs for 5,560 formal statements from high-school olympiad problems to undergraduate theorems. The Lean checker scores each proof. | Formal proofs | May 2025 | | System cards only |
| TableBench Models answer 886 questions about real-world tables across 18 fields in four groups: fact checking, numerical reasoning, data analysis and visualization. Fact and numeric answers are scored by exact match, analysis answers by ROUGE-L, and chart code by pass@1. | Language reasoning | Aug 2024 | 61.7% | Not in cards |
| CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. | General visual questions | Jan 2024 | | Not in cards |
| FormalMATH Lite Models prove statements in the FormalMATH Lite subset. The board reports proof success at the stated sampling budget. | Formal proofs | | Not in cards |