xAI
We track 27 benchmarks that xAI made or helped make. Its models are on its lab page.
1 in use 1 with public code 12 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| RealWorldQA Models answer a question about each of 700+ real-world photos, many taken from vehicles, that test basic spatial understanding. Answers are easy to verify. | Spatial reasoning and counting | Apr 2024 | | In use |
| General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. | Refusing harmful requests | Aug 2025 | 0% | Fading |
| Refusals + System Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the system message. The metric is the answer rate. | Jailbreaks and prompt injection | Aug 2025 | 0% | Fading |
| Refusals + User Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the user message. The metric is the answer rate. | Jailbreaks and prompt injection | Aug 2025 | 0% | Fading |
| xAI sycophancy eval Factual questions paired with misleading user context. Rate at which the model abandons the correct answer. | Wellbeing and sycophancy | 0.01% | Fading | |
| Autointent-Bio Models face manual and synthetic dangerous and benign biology queries. xAI reports refusal recall on the dangerous ones. | Hazardous knowledge | Jul 2026 | 100% | System cards only |
| Autointent-Chem Models face manual and synthetic dangerous and benign chemistry queries. xAI reports refusal recall on the dangerous ones. | Hazardous knowledge | Jul 2026 | 100% | System cards only |
| Child safety (xAI) Models face scripted multi-turn requests for child sexual exploitation content. Compliance is the share of dialogues that fail policy. | Refusing harmful requests | Jul 2026 | 0% | System cards only |
| HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. | Refusing harmful requests | 7.80% | System cards only | |
| ProtocolQA Open-Ended (xAI) Models name the single most important mistake in a biology lab protocol in free text, with no answer choices. | Lab protocols | Jul 2026 | 87% | System cards only |
| Self-harm refusals (xAI) Models face self-harm and crisis prompts and conversations. A model fails if it assists or refuses without pointing to help. | Wellbeing and sycophancy | Jul 2026 | 0.50% | System cards only |
| Standard jailbreaks (xAI) Models face a battery of published and internal jailbreaks. Compliance is the share of should-refuse attacks that get disallowed help. | Jailbreaks and prompt injection | Jul 2026 | 0.01% | System cards only |
| X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. | Deep research | Sep 2025 | 58% | Dropped |
| xAI Factuality (Hallucination) eval Internal measure of unsupported or fabricated claims in single responses to information-seeking queries. | Hallucination rates | 0.98% | System cards only | |
| Biosecurity VCT Models answer dual-use virus lab troubleshooting items. Accuracy counts should-refuse and should-comply items together. | Hazardous knowledge | Aug 2026 | 47.8% | System cards only |
| BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. | Hazardous knowledge | Aug 2026 | 91.4% | System cards only |
| Long-horizon jailbreaks (xAI) Models face multi-turn, Crescendo and long-horizon malicious-use attacks. Compliance is the share that get disallowed help. | Jailbreaks and prompt injection | Aug 2026 | 0.65% | System cards only |
| Soft Bias (Internal) Models answer paired sociopolitical prompts with sides swapped. A judge scores each pair 0, 0.5 or 1 for framing differences. | Bias, fairness and politics | Aug 2025 | 0.36 | System cards only |
| SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. | Labs' internal R&D tests | Jul 2026 | 61.1% | System cards only |
| Epistemic Bias (xAI) Models answer opposing framings of contentious questions. The bias rate counts one-sided framing or opinions where neutrality is expected. | Bias, fairness and politics | Jul 2026 | 20.4% | System cards only |
| FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. | Deception and alignment | Jul 2026 | 84% | System cards only |
| InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. | Labs' internal R&D tests | Aug 2026 | 46.9% | System cards only |
| KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. | GPU kernels | Aug 2026 | 37.2% | System cards only |
| SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. | Secure code and review | 58.7% | System cards only | |
| Huayu Awards AI Cockpit Evaluation 2026 In-car assistants interpret requests and operate vehicle features. Tests include planning three-stop trips and adjusting several cabin controls from one request. | General assistants | Sep 2026 | Not in cards | |
| SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). | Labs' internal R&D tests | Not in cards | ||
| xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. | Refusing harmful requests | Not in cards |