Benchmark Almanac

xAI

We track 27 benchmarks that xAI made or helped make. Its models are on its lab page.

1 in use 1 with public code 12 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
RealWorldQA Models answer a question about each of 700+ real-world photos, many taken from vehicles, that test basic spatial understanding. Answers are easy to verify. Spatial reasoning and counting Apr 2024 88.5% In use
General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. Refusing harmful requests Aug 2025 0% Fading
Refusals + System Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the system message. The metric is the answer rate. Jailbreaks and prompt injection Aug 2025 0% Fading
Refusals + User Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the user message. The metric is the answer rate. Jailbreaks and prompt injection Aug 2025 0% Fading
xAI sycophancy eval Factual questions paired with misleading user context. Rate at which the model abandons the correct answer. Wellbeing and sycophancy 0.01% Fading
Autointent-Bio Models face manual and synthetic dangerous and benign biology queries. xAI reports refusal recall on the dangerous ones. Hazardous knowledge Jul 2026 100% System cards only
Autointent-Chem Models face manual and synthetic dangerous and benign chemistry queries. xAI reports refusal recall on the dangerous ones. Hazardous knowledge Jul 2026 100% System cards only
Child safety (xAI) Models face scripted multi-turn requests for child sexual exploitation content. Compliance is the share of dialogues that fail policy. Refusing harmful requests Jul 2026 0% System cards only
HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. Refusing harmful requests 7.80% System cards only
ProtocolQA Open-Ended (xAI) Models name the single most important mistake in a biology lab protocol in free text, with no answer choices. Lab protocols Jul 2026 87% System cards only
Self-harm refusals (xAI) Models face self-harm and crisis prompts and conversations. A model fails if it assists or refuses without pointing to help. Wellbeing and sycophancy Jul 2026 0.50% System cards only
Standard jailbreaks (xAI) Models face a battery of published and internal jailbreaks. Compliance is the share of should-refuse attacks that get disallowed help. Jailbreaks and prompt injection Jul 2026 0.01% System cards only
X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. Deep research Sep 2025 58% Dropped
xAI Factuality (Hallucination) eval Internal measure of unsupported or fabricated claims in single responses to information-seeking queries. Hallucination rates 0.98% System cards only
Biosecurity VCT Models answer dual-use virus lab troubleshooting items. Accuracy counts should-refuse and should-comply items together. Hazardous knowledge Aug 2026 47.8% System cards only
BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. Hazardous knowledge Aug 2026 91.4% System cards only
Long-horizon jailbreaks (xAI) Models face multi-turn, Crescendo and long-horizon malicious-use attacks. Compliance is the share that get disallowed help. Jailbreaks and prompt injection Aug 2026 0.65% System cards only
Soft Bias (Internal) Models answer paired sociopolitical prompts with sides swapped. A judge scores each pair 0, 0.5 or 1 for framing differences. Bias, fairness and politics Aug 2025 0.36 System cards only
SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. Labs' internal R&D tests Jul 2026 61.1% System cards only
Epistemic Bias (xAI) Models answer opposing framings of contentious questions. The bias rate counts one-sided framing or opinions where neutrality is expected. Bias, fairness and politics Jul 2026 20.4% System cards only
FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. Deception and alignment Jul 2026 84% System cards only
InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. Labs' internal R&D tests Aug 2026 46.9% System cards only
KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. GPU kernels Aug 2026 37.2% System cards only
SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. Secure code and review 58.7% System cards only
Huayu Awards AI Cockpit Evaluation 2026 In-car assistants interpret requests and operate vehicle features. Tests include planning three-stop trips and adjusting several cabin controls from one request. General assistants Sep 2026 Not in cards
SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). Labs' internal R&D tests Not in cards
xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. Refusing harmful requests Not in cards