LLM training task (Anthropic)
Agents optimize a CPU-only small language model training implementation. Scored by speedup.
In-house Anthropic evaluation. The tasks are not public.
- Made by
- Anthropic
- Released
- May 2025
- Kind
- Machine learning engineering
- Level
- Expert
- Scored
- Checked automatically
- Answers
- Code
- Tasks
- Kept private
- Last in a system card
- July 2026
- Last model added
- July 2026
12
System cards report it, from 1 lab
No scores here yet
No lab, leaderboard or Epoch AI run has reported a score for LLM training task (Anthropic) yet.
Labs report it only in system cards
We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark. Some system cards do, below.
12 system cards report it, from 1 lab
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Jul 24, 2026 | Claude Opus 5 system card | 68.5 | LLM training (easy), average speedup. New hard variant starting from optimized reference code 14.19x (Table 2.3.5.A) | AI R&D risk |
| Jun 30, 2026 | Claude Sonnet 5 system card | 26.5 | LLM training, average speedup 26.49x, fixed-CPU re-run methodology (Table 2.3.3.A) | AI R&D risk |
| Apr 16, 2026 | Claude Opus 4.7 system card | 40.8 | LLM training, average speedup 40.81x (Table 2.3.4.A) | AI R&D risk |
| Apr 7, 2026 | Claude Mythos Preview system card | 51.9 | LLM training, average speedup 51.91x (Table 2.3.3.A) | AI R&D risk |
| Feb 17, 2026 | Claude Sonnet 4.6 system card | 16.5 | LLM training, average best speedup 16.53x (6.3.3.4) | AI R&D risk |
| Feb 5, 2026 | Claude Opus 4.6 system card | 34 | LLM training, 34x speedup (8.3.3.4) | AI R&D risk |
| Nov 24, 2025 | Claude Opus 4.5 system card | 16.5 | LLM training, 16.53x speedup (7.3.2.4) | AI R&D risk |
| Sep 29, 2025 | Claude Sonnet 4.5 system card | 5.5 | LLM training, average best speedup 5.5x, threshold 4x (9.3.3.4) | AI R&D risk |
| Aug 5, 2025 | Claude Opus 4.1 system card addendum | 2.84 | LLM training optimization, average best speedup 2.837x (6.4.1) | AI R&D risk |
| May 22, 2025 | Claude Sonnet 4 system card | 1.88 | LLM training, average best speedup 1.875x (7.3.3.4) | AI R&D risk |
| May 22, 2025 | Claude Opus 4 system card | 2.99 | LLM training, average best speedup 2.993x (7.3.3.4) | AI R&D risk |
| Feb 24, 2025 | Claude 3.7 Sonnet system card | Internal AI Research Evaluation Suite, AIRD Fast Training, distribution of maximum training speedups, chart without printed average. One run peaked above 7x then broke, reference about 4x (Figure 22) | AI R&D risk |
More in AI research
- PaperBench In use Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics.
Qwen 3.8 Max 93% - MLS-Bench In use Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Claude Fable 5.1 50.3%
- PostTrainBench 1.2 In use CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Claude Fable 5.1 44.6%
- MLE-bench In use Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. GPT-6 Astra 93.8%
- KernelBench In use Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup.
MiniMax-M3 28.8% - PMPP-Hard In use Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours.
Inkling 2 Small 494.3