MLE-bench
Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards.
OpenAI runs an internal Revised version with 72 problems. The public leaderboard in the repo README stopped taking submissions on April 24th, 2026. MLE-bench Lite is the 22-competition Low split.
- Made by
- OpenAI
- Released
- October 2024
- Kind
- Machine learning engineering
- Level
- Expert
- Scored
- Checked automatically
- Answers
- Agent task
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Agentic
- Environment
- Terminal
- Items from
- Exams and contests
- Refresh
- One fixed set
- Run by
- Anyone
- In views
- Agents
- Strongest model tested
- GPT-6.2 Astra #1 on the capability index
- Last in a launch post
- July 2026
- Last model added
- November 2026
In the authors’ words
We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup–OpenAI's o1-preview with AIDE scaffolding–achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training. We open-source our benchmark code (github.com/openai/mle-bench/) to facilitate future research in understanding the ML engineering capabilities of AI agents.
GPT-6.2 Astra leads with 96.1%
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
- 1 GPT-6.2 Astra 96.1%
- 2 GPT-6 Astra max effort 93.8%
- 3 GPT-6.2 Sol 91.4%
- 4
MiniMax-M2.7 66.6%
- 5
Gemini 3.6 Flash 63.9%
- 6
MiniMax-M2.5 51.5%
- 7
MiniMax-M2 40%
- 8
Gemini 3.5 Flash-Lite 39.2%
- 9 o1 14%
- 10 o3-mini 11%
- 11 OpenAI deep research 11%
- 12 GPT-4.5 11%
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-6.2 Astra | 96.1% | Nov 2026 | Lab model card | |
| 2 | GPT-6 Astra | 93.8% | max effort | Sep 2026 | Lab model card |
| 3 | GPT-6.2 Sol | 91.4% | Nov 2026 | Lab model card | |
| 4 | | 66.6% | Mar 2026 | Lab model card | |
| 5 | | 63.9% | Jul 2026 | Lab model card | |
| 6 | | 51.5% | May 2026 | Lab model card | |
| 7 | | 40% | May 2026 | Lab model card | |
| 8 | | 39.2% | Jul 2026 | Lab model card | |
| 9 | o1 | 14% | Dec 2024 | Lab model card | |
| 10 | o3-mini | 11% | Jan 2025 | Lab model card | |
| 11 | OpenAI deep research | 11% | Feb 2025 | Lab model card | |
| 12 | GPT-4.5 | 11% | Feb 2025 | Lab model card |
4 models have held number one in labs’ own numbers since its release in October 2024
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Labs’ model cards
OpenAI’s own number for GPT-6 Astra passed 85% in September 2026
That was 23 months after its release. By the same age, 36% of the benchmarks from 2024 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Labs’ model cards
One lab put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out. OpenAI accounts for 10 of the 15 releases that named it since 2024.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Jul 21, 2026 | 39.2 | MLE-Bench Machine Learning Engineering | |
| Jul 21, 2026 | 63.9 | MLE-Bench: Machine Learning Engineering | |
| Mar 18, 2026 | 66.6 | MLE-Bench lite (medal rate) |
18 system cards report it, from 3 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Nov 17, 2026 | GPT-6.2 Sol system card | 91.4 | MLE-Bench Revised (72 problems), mean reference percentile (Table 5) | AI R&D risk |
| Nov 17, 2026 | GPT-6.2 Astra system card | 96.1 | MLE-Bench Revised (72 problems), mean reference percentile (Table 5) | AI R&D risk |
| Sep 3, 2026 | GPT-6 Astra system card | 93.8 | MLE-Bench Revised (72 problems), mean reward, max effort, printed chart label (Figure 56) | AI R&D risk |
| Jul 21, 2026 | | 63.9 | MLE-Bench (results table, p.4) | Capabilities |
| Jul 21, 2026 | | 39.2 | MLE-Bench (results table, p.4) | Capabilities |
| Jul 9, 2026 | GPT-5.6 Terra system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| Jul 9, 2026 | GPT-5.6 Sol system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| Jul 9, 2026 | GPT-5.6 Luna system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| Jun 25, 2026 | GPT-5.6 Terra system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| Jun 25, 2026 | GPT-5.6 Sol system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| Jun 25, 2026 | GPT-5.6 Luna system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| May 26, 2026 | | 40 | MLE-Bench Lite, MiniMax-M2 bar, printed label (Figure 9) | AI R&D risk |
| May 26, 2026 | | 66.6 | MLE Bench Lite, medal rate, mean of 3 24-hour trials (22 competitions) (Table 4) | AI R&D risk |
| May 26, 2026 | | 51.5 | MLE Bench Lite, medal rate, mean of 3 24-hour trials (22 competitions) (Table 4) | AI R&D risk |
| Feb 27, 2025 | GPT-4.5 system card | 11 | MLE-Bench, 75 Kaggle competitions, AIDE agent, bronze medal rate, post-mitigation (4.6.4) | Autonomy |
| Feb 25, 2025 | OpenAI deep research system card | 11 | MLE-Bench, bronze pass@1, internal tool scaffold, post-mitigation with and without browsing (3.4.7) | Autonomy |
| Jan 31, 2025 | o3-mini system card | 11 | MLE-bench (AIDE), bronze pass@1, o3-mini (Post), pass@10 20%, printed chart label (5.7.4) | Autonomy |
| Dec 5, 2024 | o1 system card | 14 | MLE-bench (AIDE), bronze pass@1, o1 (post-mitigation) (bronze pass@10 24%), printed chart label (5.8.4) | AI R&D risk |
More in AI research
- PaperBench In use Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics.
Qwen 3.8 Max 93%
- MLS-Bench In use Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Claude Fable 5.1 50.3%
- PostTrainBench 1.2 In use CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Claude Fable 5.1 44.6%
- PMPP-Hard In use Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours. Kimi K3 0.71
- KernelBench In use Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup.
MiniMax-M3 28.8%
- OpenAI PRs Only in system cards Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. GPT-5.2 55%
Data audit
All 9 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 3 pass 2 notes
- Pass Independent runs Scores from the official leaderboard (26 entries), launch posts (3 scores) and system cards (12 scores).
- Pass Models scored 12 models we track, plus 26 other entries on its leaderboard.
- Pass Newest score The newest is from November 2026, for GPT-6.2 Astra.
- Note Frontier models 2 of the top 10 have a score, which is fine for a retired test. The strongest tested is GPT-6.2 Astra (#1).
- Note Headroom Saturated. The best is 96.1%, past 85% since September 2026.
Leaderboard import 3 pass
- Pass Importer We read the board on github.com: 27 rows for models we track at the last read.
- Pass Last read November 18th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.