Benchmark Almanac

METR

We track 18 benchmarks that METR made or helped make.

0 in use 8 with public code 7 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
NanoGPT Speedrun Improve a small GPT training script to reach a target validation loss as fast as possible on fixed hardware. Machine learning engineering Jun 2024 42.9% System cards only
METR Time Horizon Agents attempt software and research tasks timed against skilled humans. The score is the human task length at which the agent succeeds half the time. Task length Mar 2025 4.1 h System cards only
RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. AI research tasks Sep 2024 1.27 System cards only
METR general autonomy task suite Agents complete multi-step tasks in virtual environments. Scored as horizon weighted average task completion compared with human time limits. Task length System cards only
METR data deduplication Agents write an efficient algorithm that removes near-duplicate lines from a large file. A trial succeeds when F1 reaches at least 80%. Machine learning engineering 32.6% System cards only
Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. Machine learning engineering Aug 2024 0% System cards only
MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass. Building whole projects Jun 2026 97.8% Not in cards
METR Capability Index METR fits an item response model to agent results on Time Horizon 1.1, TH2 and MirrorCode. The index places models on a common ability scale. Task length Nov 2026 859.0 Not in cards
METR Time Horizon 1.1 Agents attempt 228 software and research tasks timed against skilled humans, run in Inspect. The score is the human task length at which the agent succeeds half the time. Task length Jan 2026 17.4 h Not in cards
METR Time Horizon suite Software tasks with human completion times. Reports the task length at which a model succeeds 50% of the time. Task length Mar 2025 4.1 h Not in cards
METR Time Horizon 2 Agents complete 36 tasks across 12 families. METR estimates how long a skilled human would need for tasks that agents complete with 50 percent reliability. Task length Nov 2026 27.1 h Not in cards
HCAST Agents attempt 189 machine learning, cybersecurity, software engineering and reasoning tasks. 563 human baselines time each task, from one minute to over 8 hours, so success can be read against human task length. Task length Mar 2025 Not in cards
KernelBenchFiltered Agents optimize GPU kernels. METR takes the geometric mean of the speedups across tasks after checking correctness. GPU kernels Feb 2025 Not in cards
Manually-Reviewed Agentic Labeled Transcripts Monitors classify agent transcripts for reward hacking and sandbagging. METR compares their predictions with human labels using the area under the receiver operating characteristic curve. Deception and alignment Oct 2025 Not in cards
METR Faithfulness and Monitorability Experiments Models answer difficult questions with clues. Researchers measure whether the reasoning reveals the use of a clue when the model answers correctly. Deception and alignment Aug 2025 Not in cards
METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. AI research tasks Not in cards
METR side task monitorability evaluation Monitors review agent transcripts for evidence of a side task. METR measures how frequently monitors miss an agent that completed one. Deception and alignment Jan 2026 Not in cards
NanoGPT Expenditure Horizon Agents optimize NanoGPT training. METR estimates the budget at which human contributors become more cost effective than the agents. Task length Jul 2026 Not in cards