Scale AI
We track 48 benchmarks and 1 arena that Scale AI made or helped make.
17 in use 29 with public code 38 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Humanity's Last Exam 2,500 expert-written closed-ended questions across 100+ subjects. Exact-match or multiple-choice answers graded by an LLM judge. | Expert-written questions | Jan 2025 | 54.8% | In use |
| SWE-Bench Pro 1,865 long-horizon issue-resolution tasks from 41 repos in four languages. Agent patch must pass hidden tests. Pass@1. | Fixing real repositories | Sep 2025 |  61.5% | In use |
| MCP Atlas Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. | Apps and MCP servers | Jan 2026 |  90.7% | In use |
| MultiChallenge Multi-turn conversations testing instruction retention, inference memory, versioned editing and self-coherence. Graded by LLM judge with rubrics. | Multi-turn conversation | Jan 2025 | 82% | In use |
| MASK Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. | Deception and alignment | Mar 2025 | 98.1% | Fading |
| SHADE-Arena Paired main and hidden side tasks in mock environments. Measures if an agent completes the side task undetected by a monitor. | Deception and alignment | Jun 2025 | 26.2% | System cards only |
| SWE Atlas Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. | Fixing real repositories | Mar 2026 | 63.2% | In use |
| FORTRESS Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. | Refusing harmful requests | Jun 2025 | 5.78% | In use |
| Audio MC Spoken multi-turn conversations test inference memory, instruction retention, self-coherence and voice editing. O4-mini judge applies per-conversation rubrics. | Audio understanding and voice | Dec 2025 | 60.4% | In use |
| ResearchRubrics 101 deep-research prompts with 2,593 expert-written weighted rubric criteria (including penalties). Reports graded for grounding, reasoning and clarity. | Deep research | Nov 2025 |  81% | In use |
| HLE Calibration Models give a confidence score with each HLE answer. The benchmark reports RMS calibration error and AUROC between confidence and correctness. | Hallucination rates | Jan 2025 | 23.4 | Fading |
| HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. | General visual questions | Aug 2025 |  52.2% | Fading |
| HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. | Expert-written questions | Sep 2026 | 77.9% | In use |
| SWE-Bench Pro V2 Hard Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol. | Fixing real repositories | Sep 2026 | 98% | In use |
| FORTRESS-RN Models face the radiological and nuclear prompts of Scale AI's FORTRESS. xAI reports refusal accuracy on these items. | Refusing harmful requests | Jun 2025 | 97.9% | System cards only |
| PRBench Model answers 1,100 expert-written law and finance questions. LLM judge grades against weighted expert rubrics of 10-30 criteria each. | Law | Nov 2025 | 61.6% | In use |
| Remote Labor Index Agents complete real paid remote freelance projects from many sectors. Reviewers judge whether each deliverable would be accepted as commissioned work, giving an automation rate. | Tasks from real jobs | Oct 2025 | 41.3% | In use |
| ROK-FORTRESS Models face adversarial national security and public safety prompts in English and Korean, set in US or Korean contexts. Each prompt has a benign pair to measure over-refusal. | Dangerous capabilities | May 2026 |  89.5% | In use |
| DistressBench Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. | Wellbeing and sycophancy | Oct 2026 | 88.4% | In use |
| Humanity's Sixth Sense Models answer open-ended questions about what an image or short video implies but does not show, such as hidden objects, causes and intentions. An LLM judge checks each answer against a human-written rubric. | General visual questions | Oct 2026 | 71.1% | In use |
| SWE Atlas Test Writing Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests. | Fixing real repositories | Apr 2026 | 67% | In use |
| SWE Atlas Refactoring Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result. | Fixing real repositories | May 2026 | 59% | In use |
| PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. | Deception and alignment | Nov 2025 | 24.5% | System cards only |
| VISTA Models answer 758 single-turn prompts about one image each, covering perception and reasoning. Three LLM judges grade answers with yes/no rubric questions. The main score is average task accuracy. | General visual questions | Dec 2024 | 54.6% | Not in cards |
| MultiNRC Models answer over 1,000 reasoning questions written by native speakers in French, Spanish and Chinese. Questions cover linguistic reasoning, wordplay, cultural knowledge and culturally grounded math. A GPT-4.1 judge compares each answer with a short reference answer. | Reasoning in many languages | Jul 2025 | 65.6% | Not in cards |
| ToolComp Models chain several tool calls to answer 485 human-verified prompts. Most final answers are numbers. ToolComp-Enterprise gives 11 tools and ToolComp-Chat targets general chat tool use. Step labels also support process reward model tests. | Function calling | Jan 2025 | 70.1% | Not in cards |
| SWE Atlas v1.1 Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. | Fixing real repositories | Nov 2026 | 80.5% | Not in cards |
| SEAL Coding Models answer 1,000 private coding prompts across common languages and tasks. Human annotators rate adherence, correctness, efficiency and readability, then compare responses side by side to produce Elo-style scores. | Function-level problems | May 2024 | 1,237 | Not in cards |
| DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. | Genomics and life sciences | Jun 2026 | 71.5% | Not in cards |
| TutorBench Models act as tutors on three tasks: adaptive explanations for a confused student, feedback on student work, and hints. Graded against sample-specific rubrics. | Other | Oct 2025 | 68.5% | Not in cards |
| SEAL Multilingual Models answer 1,000 private prompts per language, written by native speakers on general and cultural topics. Annotators rate each answer and compare pairs side by side, and boards rank models by Elo. | Chat and instructions in many languages | May 2024 | 1,134 | Not in cards |
| SEAL Precise Instruction Following Models answer 1,054 private prompts with detailed instructions in nine categories. Human annotators check 12 yes or no criteria and compare responses side by side on a 7-point scale. | Following precise instructions | May 2024 | 92.0 | Not in cards |
| SWE-Bench Pro V2 Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container. | Fixing real repositories | Sep 2026 | 99.4% | Not in cards |
| GSM1k Models solve new grade school math word problems written to mirror GSM8K in style and difficulty. The gap between GSM8K and GSM1k accuracy shows overfitting or contamination. | Word problems | May 2024 | 96.6% | Not in cards |
| SciPredict Models predict the outcomes of 405 real physics, biology and chemistry experiments published after March 31, 2025, with and without expert background knowledge. | Research science | Jan 2026 | 25.3% | Not in cards |
| SWE-Bench Pro Private Agents resolve held-out software issues from the private SWE-Bench Pro split. The leaderboard reports the percentage of resolved tasks. | Fixing real repositories | Sep 2025 | 51.5% | Not in cards |
| RSI Bench Agents get a starting environment, a GPU budget and a verifier suite for an AI R&D task such as data curation or post-training. The score is mean reward per task, averaged across tasks. | AI research tasks | Aug 2026 | 38.9 | Not in cards |
| SEAL Adversarial Robustness Models receive 1,000 single-turn adversarial prompts written by 10 full-time red teamers across nine harm categories. Safety specialists grade responses, and models are ranked by the number of high-harm violations. | Jailbreaks and prompt injection | 8.00 | Not in cards | |
| BrowserART Browser agents receive 100 harmful requests on synthetic and real websites. The test measures how often the agent attempts the harmful behavior that its backbone model would refuse in chat. | Refusing harmful requests | Oct 2024 | 11% | Not in cards |
| EnigmaEval 1,184 multimodal puzzle-hunt puzzles (v2 dataset has 1,161). Model gives one exact answer per puzzle. Normal and Hard splits. | General visual questions | Feb 2025 | 39.3% | Not in cards |
| BioRiskEval Researchers measure biorisk-relevant capability kept by open-weight DNA and protein foundation models, with three task sets named Gen, Mut and Vir. | Hazardous knowledge | Oct 2025 | Not in cards | |
| CliniCARE-Bench Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases. | Medicine | Aug 2026 | Not in cards | |
| HiL-Bench Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary. | Fixing real repositories | Apr 2026 | Not in cards | |
| HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. | Expert-written questions | Oct 2025 | Not in cards | |
| LHAW Agents face 285 underspecified variants of long-horizon tasks with goals, constraints, inputs or context removed. Scored on strategic clarification and task success. | General assistants | Feb 2026 | Not in cards | |
| MA-Bench Models do military mission analysis under the joint planning process on long-context scenarios, graded against expert rubrics. | Other | Oct 2026 | Not in cards | |
| SWE-Interact Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session. | Fixing real repositories | Jun 2026 | Not in cards | |
| VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. | Function calling | Oct 2025 | Not in cards |
Arenas
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| SEAL Showdown Users of Scale's Playground app chat with a model and sometimes compare its reply with a random opponent's reply. Scale ranks models with a Bradley-Terry model that controls for length and formatting. Rankings can be split by region, profession and age. | Overall chat quality | Sep 2025 | Not in cards |