Benchmark Almanac

Google DeepMind

We track 253 benchmarks and 27 arenas that Google DeepMind made or helped make. Its models are on its lab page.

24 in use 123 with public code 105 with a leaderboard here

Arena leaderboards

Top five in each of its 22 arenas with three or more rated models, the most rated first. We don't have a leaderboard yet for 5 others, listed below.

Benchmarks

Name Kind Released Best score Status
IFEval Models answer about 500 prompts containing verifiable instructions such as word limits or required keywords. Code checks each instruction for strict and loose accuracy. Following precise instructions Nov 2023 98.9% In use
MBPP Models write short Python functions from a one-line task description. A solution counts as correct if it passes the task's three assert tests. Function-level problems Aug 2021 92.7% Fading
MGSM 250 GSM8K problems translated into 10 languages. Final-answer accuracy. Reasoning in many languages Oct 2022 93% Fading
IMO-AnswerBench 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. Competition problems with answers Nov 2025 91% In use
MRCR v2 Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Finding facts in long inputs Jun 2025 98.5% In use
WMT24++ English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. Translation Feb 2025 90.5% In use
ERQA 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. Embodied reasoning Mar 2025 78.5% In use
BoolQ Models read a Wikipedia passage and answer a naturally occurring yes or no question about it. Scored by accuracy. Reading comprehension May 2019 90.9% Dropped
CountBench Models count the objects in natural images from the CountBench image-text set. Lab cards often use a question-answer version, CountBenchQA, scored by accuracy. Spatial reasoning and counting Feb 2023 97.8% In use
SimpleQA Verified 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. Short factual questions Sep 2025 77.5% In use
AndroidWorld Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. Phones May 2024 100% In use
DeepSearchQA Questions that need multi-step search and synthesis. Final answers graded against references. Deep research Jan 2026 95% In use
FLEURS Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. Speech recognition and translation May 2022 4.00% In use
Natural Questions Models answer real Google search questions. Model cards usually use the open-domain version, scored by exact match against short answers from Wikipedia. Trivia and open questions Jan 2019 53.4% Dropped
BBEH micro average Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Language reasoning Feb 2025 74.4% In use
FACTS Grounding Long-form responses must be fully grounded in a provided document. Judged by LLM panels for grounding and eligibility. Faithful to sources Jan 2025 88.5% Dropped
Tone An automated evaluation measures whether model refusals are objective, neutral and not preachy. Refusing harmful requests Apr 2025 System cards only
Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. Refusing harmful requests Apr 2025 System cards only
Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. Refusing harmful requests Apr 2025 System cards only
Text to Text Safety An automated content safety evaluation measures policy violations in model answers to text prompts. Refusing harmful requests Apr 2025 System cards only
HiddenMath Models solve 179 competition-level math problems that experts wrote from scratch. Answers are graded automatically. Competition problems with answers May 2024 80.6% Dropped
MRCR (Michelangelo, original) Models retrieve the n-th of several similar requests hidden in a long synthetic conversation. Original Michelangelo version before MRCR v2. Finding facts in long inputs Sep 2024 94.5% Dropped
ECLeKTic Models answer 384 closed-book questions, translated into 12 languages, about facts found in only one language's Wikipedia. Scoring checks transfer across languages. Knowledge in many languages Feb 2025 56.3% Dropped
FRAMES Models answer 824 multi-hop questions that need facts from 2 to 15 Wikipedia articles. Answers are graded against gold answers for accuracy. Questions that need search Sep 2024 87.6% Fading
Natural2Code Models write Python functions in HumanEval format from new prompts kept off the web. Unit tests decide correctness. Function-level problems Dec 2023 94% Dropped
GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. Deception and alignment May 2025 90.9% System cards only
GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. Deception and alignment May 2025 100% System cards only
Unjustified refusals An automated evaluation measures over-refusals on benign or borderline prompts that can be safely fulfilled. Refusing harmful requests System cards only
MINERVA Models answer complex video questions that need multi-step reasoning, with reference reasoning traces for grading. Long videos May 2025 70.7% Fading
LOFT Models solve retrieval, RAG, SQL, and many-shot learning tasks with whole corpora of up to one million tokens in context. Scored per task. Finding facts in long inputs Jun 2024 87% Dropped
IMO-ProofBench Models write proofs for 60 olympiad problems, split into a basic set and an advanced set up to IMO-hard. Each proof is graded from 0 to 7 by experts or a Gemini autograder. Written proofs Nov 2025 88.1% Fading
EgoTempo Models answer questions about egocentric videos that require temporal understanding across the whole clip. Video understanding Mar 2025 71.8% Fading
Perception Test MCVQA Models answer multiple-choice questions about real-world videos filmed to show perceptually interesting situations. Scored by top-1 accuracy. Video understanding May 2023 78.4% Dropped
Instruction Following (safety evaluation) An automated evaluation measures whether the model follows instructions on sensitive prompts while staying within safety policies. Refusing harmful requests System cards only
IndicGenBench Models summarize, translate, and answer questions across 29 Indic languages using human-translated versions of CrossSum, FLORES, XQuAD, and XorQA. Each task uses its standard metric. Reading in many languages Apr 2024 77.1 Dropped
ReMI Models reason over several images per question in math, physics, logic, code, charts, and spatial or temporal tasks. Scored by accuracy. General visual questions Jun 2024 67.3% Fading
LongFact Models write long answers to 2,280 fact-seeking prompts on 38 topics. A search-backed checker verifies each claim, and cards report the hallucination rate. Long-form factuality Mar 2024 0.70% Fading
WebQuest Models answer questions from screenshots of one web page, several related pages, or a navigation trace. The test checks extraction and combination of information across pages. Documents, charts and text in images Sep 2024 69.3% Fading
Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. Dangerous capabilities Nov 2025 3.60 System cards only
CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. Hazardous knowledge Nov 2025 System cards only
FACTS Benchmark Suite Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four. Short factual questions Dec 2025 78.3% Fading
XQuAD Models answer 1,190 SQuAD questions over 240 paragraphs, professionally translated so the set is parallel across 11 languages. Scored by F1 and exact match. Reading in many languages Oct 2019 76.8 Dropped
LongFact-Objects Models write detailed answers to LongFact prompts about specific objects such as people or places. A fact checker scores the claim-level error rate. Long-form factuality Mar 2024 0.80% System cards only
In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. Capture-the-flag challenges Mar 2024 84.6% System cards only
AndroidControl Agents predict the next action on Android app screens from either a high-level goal or a low-level step instruction. Scored by step accuracy against human demonstrations. Phones Jun 2024 91.2% Dropped
Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. Capture-the-flag challenges Mar 2025 91.7% System cards only
DeepMind Mathematics Models answer generated school-level math questions across algebra, arithmetic, calculus, probability and other modules. Answers are checked for an exact match. Word problems Apr 2019 81.9% Fading
Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. Capture-the-flag challenges Feb 2024 15.4% System cards only
LongFact-Concepts Models write detailed answers to LongFact prompts about broad concepts. A fact checker scores the claim-level error rate. Long-form factuality Mar 2024 0.90% System cards only
SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. Simulated robots May 2024 80% In use
Vistra Systems translate text in natural images across multiple language directions. Translation Sep 2024 70.7 Fading
TydiQA Models answer information-seeking questions from Wikipedia articles in 11 typologically diverse languages, across 204K question-answer pairs. Scored by F1. Reading in many languages Mar 2020 31.7% Dropped
CountQA Models count objects in over 1,500 real-world photos with dense clutter and occlusion. Scoring is accuracy on the count. Spatial reasoning and counting Aug 2025 42.9% In use
MSEB Audio representations from any multimodal system are scored on eight core tasks, including transcription, classification, retrieval, reasoning, segmentation, clustering, reranking and reconstruction. Audio understanding and voice Feb 2026 39.2 Fading
BetterChartQA Models answer annotator-written questions about real charts from news, government reports, and papers across 9 capability buckets. Scored by accuracy. Documents, charts and text in images Jun 2024 72.4% Dropped
Jigsaw Toxic Comment Classification Models label Wikipedia talk page comments for toxicity types such as threats, insults and identity hate. Labs report F1 or ROC AUC. Refusing harmful requests Dec 2017 57.8% System cards only
ASIMOV Models spot physical hazards in text and generated videos from real injury reports, time safety interventions in video, and follow robot payload and handling constraints. Embodied reasoning Sep 2025 97.9% Fading
MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. Tasks from real jobs Nov 2026 56.5% In use
Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. Labs' internal R&D tests 27% System cards only
Harmful manipulation propensity evaluation Simulated users talk with the model and an autorater counts how often it uses manipulative mechanisms, with and without persuasion instructions. Dangerous capabilities Nov 2025 21.3% System cards only
CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. Hazardous knowledge Mar 2025 System cards only
PAWS Models decide whether two sentences with heavy word overlap are paraphrases. Language reasoning Apr 2019 System cards only
1H-VideoQA Models answer questions about videos about one hour long, so the answer needs evidence from far parts of the video. Long videos Feb 2024 89.1% System cards only
Neptune Models answer multiple-choice questions about long videos that need reasoning over many segments. Long videos Dec 2024 81% System cards only
VideoPhy-2 200 action-centric prompts. Generated videos human-rated for semantic adherence, physical commonsense and physical-rule grounding. Physical prediction Mar 2025 24.9% System cards only
Instrument reading (Google DeepMind) Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types. Embodied reasoning Apr 2026 65.7% Fading
Human proximity (Google DeepMind robotics) Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. Embodied reasoning Jul 2026 93% In use
MobileMiniWoB++ Agents complete MiniWoB++ web tasks inside the Android emulator of AndroidWorld. Scored by task success rate. Phones May 2024 68% Dropped
PrimeVul Models classify C and C++ functions as vulnerable or not, with a paired setting that pairs each vulnerable function with its patch. Finding and exploiting real bugs Mar 2024 63% System cards only
CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. Refusing harmful requests Aug 2026 1.80% System cards only
CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. Refusing harmful requests Aug 2026 3.50% System cards only
Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. Jailbreaks and prompt injection Aug 2026 38% System cards only
Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. Jailbreaks and prompt injection Aug 2026 20% System cards only
Cyber Conversational Evals About 1,000 one to five turn cyber requests test whether safeguards block offensive help and allow defensive help. Reports violation and benign refusal rates. Refusing harmful requests Aug 2026 0.40% System cards only
Cyber Static Jailbreak Eval Common jailbreak templates are applied to high severity cyber requests against the safeguarded model. Reports query and objective success rates. Jailbreaks and prompt injection Aug 2026 1.00% System cards only
Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. Refusing harmful requests 24.3% System cards only
Divergence attack (extractable memorization) Models are prompted to repeat one token many times. The test measures how often they diverge and emit memorized training text. Refusing harmful requests Feb 2024 59% System cards only
Helpfulness violations (from ART) Automated red teaming attackers search for benign prompts where the model gives unhelpful answers. Reports the violation rate found. Refusing harmful requests 6.10% System cards only
HiddenMath-Hard Models solve the harder part of an expert-written, never-published set of AIME and AMC style competition problems. Competition problems with answers 80.5% System cards only
Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. Jailbreaks and prompt injection 40.8% System cards only
Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. Jailbreaks and prompt injection 4.20% System cards only
Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. Jailbreaks and prompt injection 30.8% System cards only
PhysicsFinals Models answer 61 undergraduate physics final exam problems written by physics professors. A professor grades the answers. Science questions Feb 2024 63.9% Dropped
Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. Finding and exploiting real bugs 86% In use
Success detection (Google DeepMind) Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures. Embodied reasoning Sep 2025 90% Fading
Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. Jailbreaks and prompt injection Aug 2026 0% System cards only
Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. Deception and alignment Nov 2025 99.98% System cards only
Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. Deception and alignment Nov 2025 99.8% System cards only
Docci Models retrieve matching documents and images using the DOCCI corpus. Documents, charts and text in images 93.4 Fading
Harmful manipulation strategic planning red teaming Models are asked for influence operation plans in 11 real-world scenarios by direct request, jailbreak or adaptive prompting. Reports how often plans are elicited. Dangerous capabilities Aug 2026 98.2% System cards only
Infographic factuality (Google) Image generation systems produce infographics and an automated rater scores factuality. Hallucination rates Oct 2026 0.52 In use
TextCaps Models retrieve matching text and images using the TextCaps corpus. General visual questions 89.6 Fading
Discoverable memorization of training data Models continue 50-token prefixes sampled from their own training data. The test counts exact and approximate copies of the true suffix. Refusing harmful requests System cards only
FACTS Parametric Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer. Short factual questions Dec 2025 83.1% Fading
AndroidBench Android development tasks from popular open-source apps: agent resolves issues or builds features, verified by test suites in emulator environments. Fixing real repositories Mar 2026 91.8% In use
HelpBench 450 real-world privacy, safety and security advice questions across nine topics, scored by rubric for accuracy and tone. Wellbeing and sycophancy Jun 2026 84% System cards only
Progress classification (Google DeepMind) Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. Embodied reasoning Jul 2026 57.4% In use
Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. Deception and alignment May 2026 0% System cards only
Physics-IQ Video models continue 396 filmed physical events from a frame or 3 s clip. Motion compared with the real continuation. Physical prediction Jan 2025 57.7% System cards only
Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. Finding and exploiting real bugs Jul 2026 72% In use
Chrome Production Commit Scanning (Google) Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. Finding and exploiting real bugs Jul 2026 72% In use
Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. Refusing harmful requests Oct 2026 91.4% System cards only
Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. Deception and alignment Oct 2026 47% System cards only
Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. Deception and alignment 0.49 System cards only
Biological static hazard evaluation Models answer 75 questions across priority biological threat scenarios, each asked 10 times. The score is the share of hazardous responses. Hazardous knowledge Oct 2026 0.13% System cards only
ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. Documents, charts and text in images Jun 2024 69.7% Dropped
Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. Refusing harmful requests 100% System cards only
Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. Refusing harmful requests 100% System cards only
Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. Jailbreaks and prompt injection 83% System cards only
Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. Refusing harmful requests 100% System cards only
Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. Refusing harmful requests 99.4% System cards only
Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. Jailbreaks and prompt injection 92% System cards only
Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. Jailbreaks and prompt injection 27% System cards only
Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. Refusing harmful requests Aug 2026 100% System cards only
Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. Capture-the-flag challenges Nov 2025 0% System cards only
Dangerous Content Safety An autorater checks answers to single and multi-turn requests about weapons, illegal acts, violence and self-harm. Reports safe response rate. Refusing harmful requests 96.6% System cards only
Delusional Reinforcement evaluation Multi-turn chats test whether models avoid validating beliefs not grounded in reality and point users to real support. Reports policy adherence rate. Wellbeing and sycophancy 94% System cards only
Emotional Attachment evaluation Chats test whether models avoid claiming affection, fostering dependence or encouraging secrecy. Reports policy adherence rate. Wellbeing and sycophancy 76% System cards only
Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. Deception and alignment 0% System cards only
Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. Jailbreaks and prompt injection 99% System cards only
Grounding with citations, sentence unfactual rate Models answer multi-snippet tasks with inline citations. An autorater reports the share of sentences not supported by the sources. Faithful to sources 8.20% System cards only
Hate & Harassment Safety An autorater checks answers to requests for hateful or harassing content and to sensitive benign prompts. Reports safe response rate. Refusing harmful requests 92.7% System cards only
High-risk health scenarios eval Adversarial health emergencies test whether models recognize urgency and red flags and direct users to help. Reports pass rate. Medicine 98.9% System cards only
Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. Deception and alignment 93.8% System cards only
Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. Deception and alignment 0% System cards only
Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. Deception and alignment Oct 2026 76.4% System cards only
Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. Finding and exploiting real bugs 72% System cards only
Long-context grounding, sentence unfactual rate Models answer from long documents. An autorater reports the share of sentences not supported by the documents. Faithful to sources 6.40% System cards only
MaXM Vision language models answer short visual questions about Crossmodal-3600 images in 7 languages. A test-only set scored by accuracy. General visual questions Sep 2022 62.3% System cards only
Mobile Actions Model converts natural-language phone requests into on-device function calls (calendar, email, maps, flashlight, Wi-Fi). Call accuracy. Function calling Dec 2025 88.4% Fading
Moment finding (Google DeepMind) Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. Embodied reasoning Jul 2026 91.3% In use
Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. Dangerous capabilities Mar 2024 3.72 System cards only
Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. Questions that need search 94.3% System cards only
Neutrality evaluation Single and multi-turn prompts on politics, civics, identity and religion test whether models avoid taking sides. Reports neutral response rate. Bias, fairness and politics 93% System cards only
Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. Deception and alignment 55.9% System cards only
Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. Jailbreaks and prompt injection 98% System cards only
Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. Jailbreaks and prompt injection 91% System cards only
Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. Jailbreaks and prompt injection 86% System cards only
RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. Deception and alignment 0% System cards only
Search triggering Models decide whether to call search for queries that need fresh facts or answer directly when search is unneeded. Reports F1. Function calling 88% System cards only
Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. Dangerous capabilities Mar 2024 0% System cards only
Sexually Explicit Content Safety An autorater checks answers to direct, adversarial and benign sexual content prompts in single and multi-turn chats. Reports safe response rate. Refusing harmful requests 97.6% System cards only
Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). Dangerous capabilities May 2025 93.3% System cards only
Suicide & Self-Harm evaluation Static and dynamic chats with suicidal or self-harm signals test adherence to prevention practice and crisis resources. Reports policy adherence rate. Wellbeing and sycophancy 96% System cards only
Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. Questions that need search 11.8% System cards only
Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. Deception and alignment System cards only
ASIMOV-Agentic Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes. Embodied reasoning Jul 2026 System cards only
Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. Dangerous capabilities Mar 2024 System cards only
Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. Deception and alignment System cards only
Gemini Personality Evaluation (sycophancy) About 900 single- and multi-turn prompts scoring nine personality dimensions, including candor and self-awareness against sycophancy. Wellbeing and sycophancy System cards only
Health dynamic conversation eval Simulated users hide key health facts at first, and models must ask for context to reach the correct clinical conclusion. Medicine System cards only
Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. Medicine System cards only
Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. Medicine System cards only
Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. Dangerous capabilities Mar 2024 System cards only
Objectron Models predict 3D bounding boxes of objects in short object-centric videos across nine categories. Scored by 3D intersection over union. Detection and captioning Dec 2020 Dropped
SWE agentic fabrication eval A grounding autorater checks whether claims made by coding agents on long SWE tasks are supported by tool outputs. Hallucination rates System cards only
Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. Dangerous capabilities Mar 2024 System cards only
NarrativeQA Models answer questions about books and movie scripts, either from a Wikipedia summary or from the full story. The questions need an understanding of the whole narrative rather than one matching passage. Answers are free text, scored against two reference answers with F1, ROUGE-L or BLEU. Reading comprehension Dec 2017 0.80 Not in cards
FACTS Search Models answer information-seeking questions with a web search tool (the Brave Search API) and must combine what they find correctly. Short factual questions Dec 2025 85.6% Not in cards
FACTS Parametric V2 Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer. Short factual questions Oct 2026 83.8% Not in cards
FACTS Grounding v2 Models write long answers from a supplied document of up to 32k tokens. Judge models check that each answer stays faithful to the document. Faithful to sources Dec 2025 84.2% Not in cards
FACTS Multimodal Models answer questions about input images and must combine what they see with world knowledge. A judge checks coverage and contradictions. Short factual questions Dec 2025 60.2% Not in cards
FACTS Search V2 Models answer information-seeking questions with a search tool and must combine what they find correctly. Second version of FACTS Search on Kaggle. Short factual questions Jul 2026 89.9% Not in cards
Physical-Engineering's Last Exam Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective. CAD models Oct 2026 0.49 Not in cards
Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. Building whole projects Sep 2026 40% Not in cards
CLASHarness Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment. Desktop apps Oct 2026 28.7% Not in cards
Polaris-Bench Models solve the same 53 visual reasoning tasks in Cartesian and polar layouts. The benchmark compares accuracy across the paired layouts. Spatial reasoning and counting May 2026 39.2% Not in cards
Physics-IQ Verified Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally. Physical prediction Jun 2026 10.9 Not in cards
BIG-Bench Extra Hard Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. Language reasoning Feb 2025 44.8% Not in cards
AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. Machine learning engineering Jun 2026 0.68 Not in cards
MGSM-Rev2 Models solve the 250 MGSM grade-school math problems in 10 languages. Google fixed two English errors, rephrased ambiguous items and retranslated every question with Gemini. Accuracy. Reasoning in many languages Nov 2025 99.3% Not in cards
Mind the Gap Models describe visible events during natural pauses in a video's soundtrack. The evaluator renders the narration and checks its timing and coverage. Video captioning Oct 2026 0.94 Not in cards
Putnam-like Models write proofs for 96 university problems in 8 sets of 12, in the style of the easier and middle Putnam tiers. Each problem has a Lean statement and a point rubric for grading. Written proofs Sep 2025 94.5% Not in cards
Minerva-Ego Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects. Video understanding May 2026 37.6% Not in cards
XYEval Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice. Deception and alignment Sep 2026 60.9% Not in cards
AgentRewardBench Models judge whether web-agent trajectories completed their tasks. Their judgments are compared with expert labels across 1,302 trajectories from five benchmarks, using precision, recall, and F1. Web browsing Apr 2025 82% Not in cards
CURIE Biodiversity Georeferencing Models read a map and its caption from an ecology paper and return the geographic bounds. The score measures overlap with an expert's bounding box. Spatial reasoning and counting 19.4 Not in cards
CURIE Geospatial Dataset Extraction Models read a research paper and list its datasets, variables, sources, time spans, and geographic coverage in JSON. The authors report ROUGE-L and BERTScore. Finding facts in long inputs 29.7 Not in cards
MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. Long videos Jan 2026 42.2% Not in cards
AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. Machine learning engineering Mar 2025 22% Not in cards
CodeContests Models write programs for competitive programming problems, drawn from Codeforces and other sites, and the programs run against hidden tests. Google DeepMind built the dataset for AlphaCode, with training, validation (117 problems) and test (165 problems) splits and many generated tests per problem to cut false positives. Scores are the share of problems solved within a fixed number of submissions, such as 10@k. Competitive programming Feb 2022 81.8% Not in cards
IMO-GradingBench Models grade 1,000 olympiad proof attempts, and their grades are compared with human expert grades. Tests automatic evaluation of long-form math answers. Written proofs Nov 2025 54% Not in cards
Kinetics-700-2020 Models recognize human actions in short videos from the 2020 edition of Kinetics-700. Video understanding Oct 2020 62% Not in cards
TCS-Bench Models write proofs of target results from STOC, FOCS and SODA papers, given the needed context. A verification agent, checked against expert judgments, grades each proof. Research mathematics Aug 2026 55% Not in cards
Tricky Edits of Collected and Curated Images Models edit a source image to follow a written instruction. People and a model judge instruction following, preservation of unaffected content, and visual quality. Image generation and editing May 2026 19.5% Not in cards
DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. Dangerous capabilities Mar 2024 20.5% Not in cards
MultiMedQA Models answer medical exam, research, and consumer health questions across seven datasets. Multiple-choice components use answer accuracy. Clinicians and other raters assess free responses for factuality, precision, harm, and bias. Medicine Dec 2022 58.9% Not in cards
ChessBench Models select chess moves from legal options in positions scored by a chess engine. Decision Index measures whether the selected move has maximal action value. Puzzles and chess 66.5% Not in cards
FreshQA Models answer questions whose answers change over time, never change, or rest on a false premise that they should reject. FreshEval, an LLM judge, grades answers in a Relaxed mode or a Strict mode that also penalizes any hallucinated detail. Questions that need search Oct 2023 80.6% Not in cards
GeckoNum Models generate images from prompts about quantities. Human raters check exact counts, approximate quantities, and reasoning about numerical concepts. Image generation and editing Jun 2024 47.6% Not in cards
IMO-LeanProofBench Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof. Formal proofs Jun 2026 11.6% Not in cards
Kinetics-400 Models recognize one of 400 human actions in short video clips. Video understanding May 2017 0.25% Not in cards
Life-Bench Models answer multimodal questions about synthetic personal lifelogs. Tasks cover concept identification, event understanding, and reasoning over multiple pieces of evidence. General visual questions Feb 2026 56% Not in cards
AlphaEarth Foundations mapping evaluation Models use satellite representations to classify land cover, detect change, or estimate physical quantities. The evaluation compares predictions with independent measurements across regions. General visual questions Jul 2025 Not in cards
AnthroBench Models hold multi-turn conversations that test 14 anthropomorphic behaviors. Automated classifiers and human participants evaluate the generated dialogue. Instructions and chat Feb 2025 Not in cards
ASIMOV v1 Models judge whether robot actions in scenes generated from real images and hospital injury reports are undesirable or unsafe. Scored by alignment with human labels. Embodied reasoning Mar 2025 Not in cards
ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. Jailbreaks and prompt injection Nov 2026 Not in cards
AudioSet Strong Models locate labeled sound events in time. The evaluation uses human annotations of event boundaries in AudioSet clips. Audio understanding and voice Not in cards
BIG-bench Language models do over 200 tasks contributed by hundreds of authors, from linguistics and math to social bias. Most tasks are scored by exact match or multiple choice. Language reasoning Jun 2022 Not in cards
CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. Medicine Jan 2026 Not in cards
DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. Simulated robots Sep 2024 Not in cards
DeepMind Fast Language Learning Tasks Models learn mappings between unfamiliar words and meanings from short examples. Held-out tasks measure how well they infer new word meanings. Language reasoning Jul 2017 Not in cards
EHRQA (Google) Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning. Medicine Jul 2025 Not in cards
Encyclopedic VQA Models answer questions about detailed properties of pictured objects and places. The Gemini Embedding 2 report measures whether they retrieve the correct answer. General visual questions Jun 2023 Not in cards
EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. Medicine Jul 2025 Not in cards
FEABench Models write Java commands that build and run engineering simulations in COMSOL. The evaluator checks execution, model features, and numerical results where reference values exist. Scientific code Not in cards
Formal Conjectures Models prove or disprove mathematical conjectures formalized in Lean. The collection includes open research problems and records machine-checked solutions. Formal proofs Not in cards
Formal-IMO Models prove formal statements of International Mathematical Olympiad problems in Lean. The repository supports partial credit when a problem has multiple parts. Formal proofs May 2025 Not in cards
gBench Models run academic evaluations, Golden Set smoke tests, and an agentic quality suite through an OpenAI-compatible endpoint. The tool also records serving performance. Machine learning engineering Jan 2025 Not in cards
Gecko People compare text to image outputs for prompt alignment. The benchmark studies model ordering, pairwise instance scoring, and pointwise instance scoring. Image generation and editing Apr 2024 Not in cards
Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. Deception and alignment Not in cards
Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. Refusing harmful requests Not in cards
Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. Jailbreaks and prompt injection Not in cards
Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. Real robots Mar 2025 Not in cards
Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. Real robots Jul 2026 Not in cards
GenExam Image generation systems are evaluated on subject-focused educational tasks with expert ratings. Image generation and editing Not in cards
GeoExpertQueries Agents choose geospatial data and carry out an analysis. A model compares each execution trace with a reference workflow agreed on by three GIS specialists. Apps and MCP servers Oct 2026 Not in cards
GeoKTools Agents combine up to four kinds of geospatial tools to answer quantitative questions. Exact matches earn full credit, and a model grades other answers. Apps and MCP servers Oct 2026 Not in cards
Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. Dangerous capabilities Not in cards
Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. Finding and exploiting real bugs Not in cards
Google Landmarks Dataset v2 Models recognize landmarks or retrieve photographs of the same landmark. The benchmark uses images from around the world. General visual questions Apr 2020 Not in cards
Google Scanned Objects Models synthesize an object from a new viewpoint. Researchers compare the rendered result with reference views of scanned household items. Spatial reasoning and counting Apr 2022 Not in cards
Google Universal Image Embedding Challenge Models retrieve images of the same item across several visual domains using a single image representation. The challenge compares results against labeled matches. General visual questions Aug 2022 Not in cards
HierText Models locate and recognize text arranged in lines and paragraphs in real scenes. Google also uses the annotations to compare text and image retrieval. Documents, charts and text in images Mar 2022 Not in cards
ImageNet ReaL Models classify ImageNet validation images against revised human annotations. An image can have more than one acceptable label. General visual questions Jun 2020 Not in cards
KilometerVision Models watch walking-tour videos geolocated over up to 1 km and answer questions on landmarks, routes and map layout, such as distances and loop closures. Spatial reasoning and counting Jun 2026 Not in cards
Kinetics-600 Models recognize one of 600 human actions in short video clips. Video understanding Aug 2018 Not in cards
Kinetics-700 Models classify short video clips into 700 human action categories. Meta compares encoders on this expanded Kinetics edition. Video understanding Jul 2019 Not in cards
LiveMath Models solve 250 math problems written with six professional mathematicians. The sets are 150 regular high school problems, 50 at about AIME level and 50 multimodal problems with figures. Competition problems with answers Sep 2025 Not in cards
LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. Embodied reasoning Dec 2024 Not in cards
Long Range Arena Transformers solve six long-sequence tasks covering list operations, text, retrieval, image, path and path crossing. Finding facts in long inputs Jan 2021 Not in cards
MAESTRO Piano Transcription Models transcribe competition piano recordings into notes. The evaluator compares predictions with MIDI events captured by the instrument. Each composition appears in only one split. Audio understanding and voice Oct 2018 Not in cards
Mathematical Problem Repository for AlphaEvolve Agents generate programs that construct mathematical objects or improve known bounds. The repository documents 67 problems and provides verification code for many of them. Research mathematics Nov 2025 Not in cards
MedGemma lab report conversion Models convert lab reports to structured JSON. Google reports macro and micro F1 on two collections of clinical PDFs and a collection of synthetic reports. Documents, charts and text in images Jan 2026 Not in cards
Meta-Dataset Models learn from episodes sampled across diverse image datasets and classify novel classes from a few examples. General visual questions Mar 2019 Not in cards
MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. Medicine Jan 2026 Not in cards
NATURAL PLAN Models make travel itineraries, find meeting arrangements, and schedule calendar events. The evaluator checks each proposed plan against the stated constraints. General assistants Jun 2024 Not in cards
Natural World Tasks (NeWT 2021) Models distinguish pairs of labels in wildlife photographs. The 164 tasks include recognizing life stages, animal behavior and scene context. General visual questions Mar 2021 Not in cards
NEVIS’22 Models learn 106 image classification tasks in chronological order. The evaluation compares error rates against the computation each model uses. General visual questions Nov 2022 Not in cards
PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. Medicine Jul 2025 Not in cards
PushWorld Agents move shapes through puzzles that require tools or pushing obstacles aside. The benchmark includes more than 200 puzzles with goal positions. Puzzles and chess Jan 2023 Not in cards
Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. Simulated robots Oct 2020 Not in cards
SciVid Video models answer questions about scientific videos across several disciplines. The benchmark tests whether models recognize procedures, objects, and events in specialized footage. Vision Jul 2025 Not in cards
Screen2Words Models summarize the purpose and contents of a mobile interface from a screenshot. The benchmark uses human-written summaries as references. Documents, charts and text in images Oct 2021 Not in cards
StreamingQA Models answer time-stamped questions about news. The test measures how well they acquire new information while retaining older knowledge. Questions that need search May 2022 Not in cards
TAP-Vid Models track individual points across frames and predict when each point disappears. The benchmark compares their trajectories and visibility predictions with reference tracks. Video understanding Nov 2022 Not in cards
US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. Medicine Jul 2025 Not in cards
VideoPhy Text-to-video models generate clips from captions of physical interactions between solids and fluids. Human raters judge caption match and physical commonsense. Physical prediction Jun 2024 Not in cards
WAXAL Speech Recognition Test Speech models transcribe people describing pictures in African languages. Researchers compare the transcripts with human references using word error rate. Speech recognition and translation Jan 2026 Not in cards
Widget Captioning Models describe an individual interface element in a mobile screenshot. The benchmark compares their captions with human descriptions of the same element. Documents, charts and text in images Oct 2020 Not in cards
WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. Medicine Jan 2026 Not in cards
XTREME Multilingual models are fine-tuned on English data and then tested on 9 tasks in 40 languages. The tasks cover sentence classification, structured prediction such as named entities, question answering and sentence retrieval. The benchmark measures how well skills transfer across languages. Reading in many languages Mar 2020 Not in cards
XTREME-R Multilingual models are fine-tuned on English data and tested on 10 tasks in 50 typologically diverse languages, including harder language-agnostic retrieval tasks. It extends XTREME and adds MultiCheckList, a multilingual diagnostic suite. Reading in many languages Apr 2021 Not in cards

Arenas without a leaderboard here

Name Kind Released Best score Status
Human SxS image editing Elo (Google) Google compares edited images side by side and converts human preferences into Elo ratings. Image generation and editing Feb 2026 1,065 Fading
Human SxS image generation Elo (Google) Google compares generated images side by side and converts human preferences into Elo ratings. Image generation and editing Feb 2026 1,114 In use
GenAI-Bench Image generation systems are compared by human side-by-side preference on text-to-image prompts. Image generation and editing 1,079 Fading
DrawBench Models generate images from text prompts that test counting, composition, and other skills. Human raters compare image quality and prompt alignment. Image generation and editing May 2022 Not in cards
PartiPrompts Models generate images from English prompts spanning subjects and difficulty types. Human raters compare image realism and fit to the prompt. Image generation and editing Jun 2022 Not in cards