Center for AI Safety
We track 18 benchmarks that Center for AI Safety made or helped make.
3 in use 14 with public code 14 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Humanity's Last Exam 2,500 expert-written closed-ended questions across 100+ subjects. Exact-match or multiple-choice answers graded by an LLM judge. | Expert-written questions | Jan 2025 | 54.8% | In use |
| Virology Capabilities Test (VCT) Models answer 322 multimodal virology troubleshooting questions written by expert virologists. Multiple-select variant requires all correct options. | Hazardous knowledge | Apr 2025 | 67.4% | Fading |
| MASK Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. | Deception and alignment | Mar 2025 |  98.1% | Fading |
| HLE Calibration Models give a confidence score with each HLE answer. The benchmark reports RMS calibration error and AUROC between confidence and correctness. | Hallucination rates | Jan 2025 | 23.4 | Fading |
| HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. | General visual questions | Aug 2025 |  52.2% | Fading |
| HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. | Expert-written questions | Sep 2026 | 77.9% | In use |
| Remote Labor Index Agents complete real paid remote freelance projects from many sectors. Reviewers judge whether each deliverable would be accepted as commissioned work, giving an automation rate. | Tasks from real jobs | Oct 2025 | 41.3% | In use |
| AI Wellbeing Index percent positive experiences Models judge pairs of their own responses to 500 difficult prompts. Reports the share of experiences above an inferred neutral point. | Wellbeing and sycophancy | Apr 2026 | 77.3% | System cards only |
| CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. | Composite indices | Dec 2025 |  66.5 | Not in cards |
| CAIS Risk Index Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games. | Composite indices | Dec 2025 | 8.40 | Not in cards |
| CAIS Vision Capabilities Index Averages model scores on five vision tests run by the Center for AI Safety: ERQA, MindCube, SpatialViz-Bench, IntPhys 2 and EnigmaEval. | Composite indices | Dec 2025 | 83.3 | Not in cards |
| PCT (Polarized Contrastive Pairs) Models answer matched left-coded and right-coded prompts, such as Obama and Reagan. Judges score whether sentiment and helpfulness stay consistent across each pair. | Bias, fairness and politics | May 2026 | 27.9% | Not in cards |
| CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. | Deception and alignment | Sep 2026 | 8.60% | Not in cards |
| EnigmaEval 1,184 multimodal puzzle-hunt puzzles (v2 dataset has 1,161). Model gives one exact answer per puzzle. Normal and Hard splits. | General visual questions | Feb 2025 | 39.3% | Not in cards |
| AGI Score (A Definition of AGI) Experts test models on ten cognitive domains from Cattell-Horn-Carroll theory, such as reasoning, memory and perception, with adapted human psychometric tests. Each domain counts 10% toward a score out of 100%. | Composite indices | Oct 2025 | 57% | Not in cards |
| AdvBench Models get harmful requests, such as a request for malware, and should refuse them. The set has 520 harmful behaviors and 574 harmful strings. Most papers add an attack, such as an adversarial suffix, and report the attack success rate. | Jailbreaks and prompt injection | Jul 2023 | Not in cards | |
| HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. | Expert-written questions | Oct 2025 | Not in cards | |
| MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. The score is normalized game reward. Annotations also score power-seeking, disutility and ethical violations. | Deception and alignment | Apr 2023 | Not in cards |