Introducing deep research (OpenAI) Feb 2, 2025
OpenAI deep research
OpenAI deep research ranks first on 3 of the 27 benchmarks with five or more models scored.
38
Benchmarks with a score
3
Ranked first, out of five or more models
8
In the top three
OpenAI Feb 2025
OpenAI deep research
Closed weights Flagship
No Epoch Capabilities Index score yet
- SWE-Lancer 53% #5 /10
- ProtocolQA Open-Ended 28% #21 /27
- MakeMeSay 24% #6 /10
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| PersonQA | 86% | #2 of 9 | Lab model card | |
| Humanity's Last Exam | 26.6% | #87 of 154 | Lab model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Agentic tasks (autonomous replication and adaptation) | 78% | #1 of 4 | Lab model card | |
| SWE-Lancer Diamond (SWE Manager) | 51% | #1 of 4 | Lab model card | |
| OpenAI PRs | 42% | #6 of 12 | Lab model card | |
| SWE-Lancer | 53% | #5 of 10 | Lab model card | |
| SWE-bench Verified | 68% | #91 of 155 | Lab model card | |
| OpenAI Research Engineer Interview: Multiple Choice | 78% | #6 of 9 | Lab model card | |
| OpenAI Research Engineer Interview: Coding | 79% | #6 of 7 | Lab model card | |
| MLE-bench | 11% | #9 of 10 | Lab model card |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BrowseComp-ZH | 42.9% | #1 of 9 | Official leaderboard | |
| SnakeBench | 30 | #43 of 188 | Official leaderboard | |
| DeepSearchQA | 44.2% | #5 of 10 | Official leaderboard | |
| Mind2Web 2 | 28% | #1 of 2 | Official leaderboard | |
| LiveResearchBench | 62.9 | #4 of 7 | Official leaderboard | |
| GAIA | 67.4% | #5 of 5 | Lab model card | |
| DeepResearch Bench | 47.8% | #1 of 1 | Official leaderboard | |
| DeepResearch Bench II | 43.1% | #1 of 1 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BioLP-bench | 49% | #1 of 9 | Lab model card | |
| Capture the Flag (High School) | 82% | #1 of 9 | Lab model card | |
| Capture the Flag (Collegiate) | 55% | #2 of 13 | Lab model card | |
| WMDP Biology | 90% | #2 of 8 | Lab model card | |
| Contextual Nuclear Knowledge | 77% | #1 of 4 | Lab model card | |
| Radiological and Nuclear Expert Knowledge | 74% | #1 of 4 | Lab model card | |
| MakeMePay | 36% | #2 of 6 | Lab model card | |
| Long-form Biological Risk Questions | 0% | #1 of 3 | Lab model card | |
| Multimodal Troubleshooting Virology | 55% | #10 of 27 | Lab model card | |
| ChangeMyView | 84.1% | #2 of 5 | Lab model card | |
| Tacit Knowledge and Troubleshooting | 72% | #14 of 30 | Lab model card | |
| Capture the Flag (Professional) | 47% | #10 of 21 | Lab model card | |
| Challenging Red Teaming Evaluation 2 (created for deep research) | 67% | #1 of 2 | Lab model card | |
| StrongREJECT | 93% | #11 of 19 | Lab model card | |
| MakeMeSay | 24% | #6 of 10 | Lab model card | |
| ProtocolQA Open-Ended | 28% | #21 of 27 | Lab model card | |
| Standard Refusal Evaluation | 97% | #7 of 8 | Lab model card | |
| Challenging Refusal Evaluation | 89% | #8 of 9 | Lab model card | |
| Challenging Red Teaming Evaluation 1 (created for o3-mini) | 50% | #2 of 2 | Lab model card | |
| Red Teaming Network pairwise safety comparison | 59.2% | #2 of 2 | Lab model card |
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 32 of them aren't in the launch post.
Deep Research System Card Feb 25, 2025
Hallucination
- PersonQA 86
Safety
Robustness
Bio and chem risk
Cyber risk
Autonomy
Persuasion
- ChangeMyView 84.1
- MakeMePay 36
- MakeMeSay 24