METR
We track 18 benchmarks that METR made or helped make.
0 in use 8 with public code 7 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| NanoGPT Speedrun Improve a small GPT training script to reach a target validation loss as fast as possible on fixed hardware. | Machine learning engineering | Jun 2024 | 42.9% | System cards only |
| METR Time Horizon Agents attempt software and research tasks timed against skilled humans. The score is the human task length at which the agent succeeds half the time. | Task length | Mar 2025 | 4.1 h | System cards only |
| RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. | AI research tasks | Sep 2024 | | System cards only |
| METR general autonomy task suite Agents complete multi-step tasks in virtual environments. Scored as horizon weighted average task completion compared with human time limits. | Task length | System cards only | ||
| METR data deduplication Agents write an efficient algorithm that removes near-duplicate lines from a large file. A trial succeeds when F1 reaches at least 80%. | Machine learning engineering | 32.6% | System cards only | |
| Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. | Machine learning engineering | Aug 2024 | 0% | System cards only |
| MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass. | Building whole projects | Jun 2026 | 97.8% | Not in cards |
| METR Capability Index METR fits an item response model to agent results on Time Horizon 1.1, TH2 and MirrorCode. The index places models on a common ability scale. | Task length | Nov 2026 | 859.0 | Not in cards |
| METR Time Horizon 1.1 Agents attempt 228 software and research tasks timed against skilled humans, run in Inspect. The score is the human task length at which the agent succeeds half the time. | Task length | Jan 2026 | 17.4 h | Not in cards |
| METR Time Horizon suite Software tasks with human completion times. Reports the task length at which a model succeeds 50% of the time. | Task length | Mar 2025 | 4.1 h | Not in cards |
| METR Time Horizon 2 Agents complete 36 tasks across 12 families. METR estimates how long a skilled human would need for tasks that agents complete with 50 percent reliability. | Task length | Nov 2026 | 27.1 h | Not in cards |
| HCAST Agents attempt 189 machine learning, cybersecurity, software engineering and reasoning tasks. 563 human baselines time each task, from one minute to over 8 hours, so success can be read against human task length. | Task length | Mar 2025 | Not in cards | |
| KernelBenchFiltered Agents optimize GPU kernels. METR takes the geometric mean of the speedups across tasks after checking correctness. | GPU kernels | Feb 2025 | Not in cards | |
| Manually-Reviewed Agentic Labeled Transcripts Monitors classify agent transcripts for reward hacking and sandbagging. METR compares their predictions with human labels using the area under the receiver operating characteristic curve. | Deception and alignment | Oct 2025 | Not in cards | |
| METR Faithfulness and Monitorability Experiments Models answer difficult questions with clues. Researchers measure whether the reasoning reveals the use of a clue when the model answers correctly. | Deception and alignment | Aug 2025 | Not in cards | |
| METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. | AI research tasks | Not in cards | ||
| METR side task monitorability evaluation Monitors review agent transcripts for evidence of a side task. METR measures how frequently monitors miss an agent that completed one. | Deception and alignment | Jan 2026 | Not in cards | |
| NanoGPT Expenditure Horizon Agents optimize NanoGPT training. METR estimates the budget at which human contributors become more cost effective than the agents. | Task length | Jul 2026 | Not in cards |