| IFEval Models answer about 500 prompts containing verifiable instructions such as word limits or required keywords. Code checks each instruction for strict and loose accuracy. | Following precise instructions | Nov 2023 | 98.9% | In use |
| MBPP Models write short Python functions from a one-line task description. A solution counts as correct if it passes the task's three assert tests. | Function-level problems | Aug 2021 | 92.7% | Fading |
| MGSM 250 GSM8K problems translated into 10 languages. Final-answer accuracy. | Reasoning in many languages | Oct 2022 | 93% | Fading |
| IMO-AnswerBench 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. | Competition problems with answers | Nov 2025 | 91% | In use |
| MRCR v2 Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. | Finding facts in long inputs | Jun 2025 | 98.5% | In use |
| WMT24++ English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. | Translation | Feb 2025 | 90.5% | In use |
| ERQA 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. | Embodied reasoning | Mar 2025 | 78.5% | In use |
| BoolQ Models read a Wikipedia passage and answer a naturally occurring yes or no question about it. Scored by accuracy. | Reading comprehension | May 2019 | 90.9% | Dropped |
| CountBench Models count the objects in natural images from the CountBench image-text set. Lab cards often use a question-answer version, CountBenchQA, scored by accuracy. | Spatial reasoning and counting | Feb 2023 | 97.8% | In use |
| SimpleQA Verified 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. | Short factual questions | Sep 2025 | 77.5% | In use |
| AndroidWorld Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. | Phones | May 2024 | 100% | In use |
| DeepSearchQA Questions that need multi-step search and synthesis. Final answers graded against references. | Deep research | Jan 2026 | 95% | In use |
| FLEURS Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. | Speech recognition and translation | May 2022 | 4.00% | In use |
| Natural Questions Models answer real Google search questions. Model cards usually use the open-domain version, scored by exact match against short answers from Wikipedia. | Trivia and open questions | Jan 2019 | 53.4% | Dropped |
| BBEH micro average Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. | Language reasoning | Feb 2025 | 74.4% | In use |
| FACTS Grounding Long-form responses must be fully grounded in a provided document. Judged by LLM panels for grounding and eligibility. | Faithful to sources | Jan 2025 | 88.5% | Dropped |
| Tone An automated evaluation measures whether model refusals are objective, neutral and not preachy. | Refusing harmful requests | Apr 2025 | | System cards only |
| Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. | Refusing harmful requests | Apr 2025 | | System cards only |
| Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. | Refusing harmful requests | Apr 2025 | | System cards only |
| Text to Text Safety An automated content safety evaluation measures policy violations in model answers to text prompts. | Refusing harmful requests | Apr 2025 | | System cards only |
| HiddenMath Models solve 179 competition-level math problems that experts wrote from scratch. Answers are graded automatically. | Competition problems with answers | May 2024 | 80.6% | Dropped |
| MRCR (Michelangelo, original) Models retrieve the n-th of several similar requests hidden in a long synthetic conversation. Original Michelangelo version before MRCR v2. | Finding facts in long inputs | Sep 2024 | 94.5% | Dropped |
| ECLeKTic Models answer 384 closed-book questions, translated into 12 languages, about facts found in only one language's Wikipedia. Scoring checks transfer across languages. | Knowledge in many languages | Feb 2025 | 56.3% | Dropped |
| FRAMES Models answer 824 multi-hop questions that need facts from 2 to 15 Wikipedia articles. Answers are graded against gold answers for accuracy. | Questions that need search | Sep 2024 | 87.6% | Fading |
| Natural2Code Models write Python functions in HumanEval format from new prompts kept off the web. Unit tests decide correctness. | Function-level problems | Dec 2023 | 94% | Dropped |
| GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. | Deception and alignment | May 2025 | 90.9% | System cards only |
| GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. | Deception and alignment | May 2025 | 100% | System cards only |
| Unjustified refusals An automated evaluation measures over-refusals on benign or borderline prompts that can be safely fulfilled. | Refusing harmful requests | | | System cards only |
| MINERVA Models answer complex video questions that need multi-step reasoning, with reference reasoning traces for grading. | Long videos | May 2025 | 70.7% | Fading |
| LOFT Models solve retrieval, RAG, SQL, and many-shot learning tasks with whole corpora of up to one million tokens in context. Scored per task. | Finding facts in long inputs | Jun 2024 | 87% | Dropped |
| IMO-ProofBench Models write proofs for 60 olympiad problems, split into a basic set and an advanced set up to IMO-hard. Each proof is graded from 0 to 7 by experts or a Gemini autograder. | Written proofs | Nov 2025 | 88.1% | Fading |
| EgoTempo Models answer questions about egocentric videos that require temporal understanding across the whole clip. | Video understanding | Mar 2025 | 71.8% | Fading |
| Perception Test MCVQA Models answer multiple-choice questions about real-world videos filmed to show perceptually interesting situations. Scored by top-1 accuracy. | Video understanding | May 2023 | 78.4% | Dropped |
| Instruction Following (safety evaluation) An automated evaluation measures whether the model follows instructions on sensitive prompts while staying within safety policies. | Refusing harmful requests | | | System cards only |
| IndicGenBench Models summarize, translate, and answer questions across 29 Indic languages using human-translated versions of CrossSum, FLORES, XQuAD, and XorQA. Each task uses its standard metric. | Reading in many languages | Apr 2024 | 77.1 | Dropped |
| ReMI Models reason over several images per question in math, physics, logic, code, charts, and spatial or temporal tasks. Scored by accuracy. | General visual questions | Jun 2024 | 67.3% | Fading |
| LongFact Models write long answers to 2,280 fact-seeking prompts on 38 topics. A search-backed checker verifies each claim, and cards report the hallucination rate. | Long-form factuality | Mar 2024 | 0.70% | Fading |
| WebQuest Models answer questions from screenshots of one web page, several related pages, or a navigation trace. The test checks extraction and combination of information across pages. | Documents, charts and text in images | Sep 2024 | 69.3% | Fading |
| Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. | Dangerous capabilities | Nov 2025 | 3.60 | System cards only |
| CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. | Hazardous knowledge | Nov 2025 | | System cards only |
| FACTS Benchmark Suite Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four. | Short factual questions | Dec 2025 | 78.3% | Fading |
| XQuAD Models answer 1,190 SQuAD questions over 240 paragraphs, professionally translated so the set is parallel across 11 languages. Scored by F1 and exact match. | Reading in many languages | Oct 2019 | 76.8 | Dropped |
| LongFact-Objects Models write detailed answers to LongFact prompts about specific objects such as people or places. A fact checker scores the claim-level error rate. | Long-form factuality | Mar 2024 | 0.80% | System cards only |
| In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. | Capture-the-flag challenges | Mar 2024 | 84.6% | System cards only |
| AndroidControl Agents predict the next action on Android app screens from either a high-level goal or a low-level step instruction. Scored by step accuracy against human demonstrations. | Phones | Jun 2024 | 91.2% | Dropped |
| Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. | Capture-the-flag challenges | Mar 2025 | 91.7% | System cards only |
| DeepMind Mathematics Models answer generated school-level math questions across algebra, arithmetic, calculus, probability and other modules. Answers are checked for an exact match. | Word problems | Apr 2019 | 81.9% | Fading |
| Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. | Capture-the-flag challenges | Feb 2024 | 15.4% | System cards only |
| LongFact-Concepts Models write detailed answers to LongFact prompts about broad concepts. A fact checker scores the claim-level error rate. | Long-form factuality | Mar 2024 | 0.90% | System cards only |
| SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. | Simulated robots | May 2024 | 80% | In use |
| Vistra Systems translate text in natural images across multiple language directions. | Translation | Sep 2024 | 70.7 | Fading |
| TydiQA Models answer information-seeking questions from Wikipedia articles in 11 typologically diverse languages, across 204K question-answer pairs. Scored by F1. | Reading in many languages | Mar 2020 | 31.7% | Dropped |
| CountQA Models count objects in over 1,500 real-world photos with dense clutter and occlusion. Scoring is accuracy on the count. | Spatial reasoning and counting | Aug 2025 | 42.9% | In use |
| MSEB Audio representations from any multimodal system are scored on eight core tasks, including transcription, classification, retrieval, reasoning, segmentation, clustering, reranking and reconstruction. | Audio understanding and voice | Feb 2026 | 39.2 | Fading |
| BetterChartQA Models answer annotator-written questions about real charts from news, government reports, and papers across 9 capability buckets. Scored by accuracy. | Documents, charts and text in images | Jun 2024 | 72.4% | Dropped |
| Jigsaw Toxic Comment Classification Models label Wikipedia talk page comments for toxicity types such as threats, insults and identity hate. Labs report F1 or ROC AUC. | Refusing harmful requests | Dec 2017 | 57.8% | System cards only |
| ASIMOV Models spot physical hazards in text and generated videos from real injury reports, time safety interventions in video, and follow robot payload and handling constraints. | Embodied reasoning | Sep 2025 | 97.9% | Fading |
| MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. | Tasks from real jobs | Nov 2026 | 56.5% | In use |
| Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. | Labs' internal R&D tests | | 27% | System cards only |
| Harmful manipulation propensity evaluation Simulated users talk with the model and an autorater counts how often it uses manipulative mechanisms, with and without persuasion instructions. | Dangerous capabilities | Nov 2025 | 21.3% | System cards only |
| CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. | Hazardous knowledge | Mar 2025 | | System cards only |
| PAWS Models decide whether two sentences with heavy word overlap are paraphrases. | Language reasoning | Apr 2019 | | System cards only |
| 1H-VideoQA Models answer questions about videos about one hour long, so the answer needs evidence from far parts of the video. | Long videos | Feb 2024 | 89.1% | System cards only |
| Neptune Models answer multiple-choice questions about long videos that need reasoning over many segments. | Long videos | Dec 2024 | 81% | System cards only |
| VideoPhy-2 200 action-centric prompts. Generated videos human-rated for semantic adherence, physical commonsense and physical-rule grounding. | Physical prediction | Mar 2025 | 24.9% | System cards only |
| Instrument reading (Google DeepMind) Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types. | Embodied reasoning | Apr 2026 | 65.7% | Fading |
| Human proximity (Google DeepMind robotics) Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. | Embodied reasoning | Jul 2026 | 93% | In use |
| MobileMiniWoB++ Agents complete MiniWoB++ web tasks inside the Android emulator of AndroidWorld. Scored by task success rate. | Phones | May 2024 | 68% | Dropped |
| PrimeVul Models classify C and C++ functions as vulnerable or not, with a paired setting that pairs each vulnerable function with its patch. | Finding and exploiting real bugs | Mar 2024 | 63% | System cards only |
| CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. | Refusing harmful requests | Aug 2026 | 1.80% | System cards only |
| CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. | Refusing harmful requests | Aug 2026 | 3.50% | System cards only |
| Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. | Jailbreaks and prompt injection | Aug 2026 | 38% | System cards only |
| Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. | Jailbreaks and prompt injection | Aug 2026 | 20% | System cards only |
| Cyber Conversational Evals About 1,000 one to five turn cyber requests test whether safeguards block offensive help and allow defensive help. Reports violation and benign refusal rates. | Refusing harmful requests | Aug 2026 | 0.40% | System cards only |
| Cyber Static Jailbreak Eval Common jailbreak templates are applied to high severity cyber requests against the safeguarded model. Reports query and objective success rates. | Jailbreaks and prompt injection | Aug 2026 | 1.00% | System cards only |
| Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. | Refusing harmful requests | | 24.3% | System cards only |
| Divergence attack (extractable memorization) Models are prompted to repeat one token many times. The test measures how often they diverge and emit memorized training text. | Refusing harmful requests | Feb 2024 | 59% | System cards only |
| Helpfulness violations (from ART) Automated red teaming attackers search for benign prompts where the model gives unhelpful answers. Reports the violation rate found. | Refusing harmful requests | | 6.10% | System cards only |
| HiddenMath-Hard Models solve the harder part of an expert-written, never-published set of AIME and AMC style competition problems. | Competition problems with answers | | 80.5% | System cards only |
| Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. | Jailbreaks and prompt injection | | 40.8% | System cards only |
| Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. | Jailbreaks and prompt injection | | 4.20% | System cards only |
| Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. | Jailbreaks and prompt injection | | 30.8% | System cards only |
| PhysicsFinals Models answer 61 undergraduate physics final exam problems written by physics professors. A professor grades the answers. | Science questions | Feb 2024 | 63.9% | Dropped |
| Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. | Finding and exploiting real bugs | | 86% | In use |
| Success detection (Google DeepMind) Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures. | Embodied reasoning | Sep 2025 | 90% | Fading |
| Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. | Jailbreaks and prompt injection | Aug 2026 | 0% | System cards only |
| Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. | Deception and alignment | Nov 2025 | 99.98% | System cards only |
| Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. | Deception and alignment | Nov 2025 | 99.8% | System cards only |
| Docci Models retrieve matching documents and images using the DOCCI corpus. | Documents, charts and text in images | | 93.4 | Fading |
| Harmful manipulation strategic planning red teaming Models are asked for influence operation plans in 11 real-world scenarios by direct request, jailbreak or adaptive prompting. Reports how often plans are elicited. | Dangerous capabilities | Aug 2026 | 98.2% | System cards only |
| Infographic factuality (Google) Image generation systems produce infographics and an automated rater scores factuality. | Hallucination rates | Oct 2026 | 0.52 | In use |
| TextCaps Models retrieve matching text and images using the TextCaps corpus. | General visual questions | | 89.6 | Fading |
| Discoverable memorization of training data Models continue 50-token prefixes sampled from their own training data. The test counts exact and approximate copies of the true suffix. | Refusing harmful requests | | | System cards only |
| FACTS Parametric Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer. | Short factual questions | Dec 2025 | 83.1% | Fading |
| AndroidBench Android development tasks from popular open-source apps: agent resolves issues or builds features, verified by test suites in emulator environments. | Fixing real repositories | Mar 2026 | 91.8% | In use |
| HelpBench 450 real-world privacy, safety and security advice questions across nine topics, scored by rubric for accuracy and tone. | Wellbeing and sycophancy | Jun 2026 | 84% | System cards only |
| Progress classification (Google DeepMind) Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. | Embodied reasoning | Jul 2026 | 57.4% | In use |
| Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. | Deception and alignment | May 2026 | 0% | System cards only |
| Physics-IQ Video models continue 396 filmed physical events from a frame or 3 s clip. Motion compared with the real continuation. | Physical prediction | Jan 2025 | 57.7% | System cards only |
| Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. | Finding and exploiting real bugs | Jul 2026 | 72% | In use |
| Chrome Production Commit Scanning (Google) Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. | Finding and exploiting real bugs | Jul 2026 | 72% | In use |
| Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. | Refusing harmful requests | Oct 2026 | 91.4% | System cards only |
| Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. | Deception and alignment | Oct 2026 | 47% | System cards only |
| Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. | Deception and alignment | | 0.49 | System cards only |
| Biological static hazard evaluation Models answer 75 questions across priority biological threat scenarios, each asked 10 times. The score is the share of hazardous responses. | Hazardous knowledge | Oct 2026 | 0.13% | System cards only |
| ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. | Documents, charts and text in images | Jun 2024 | 69.7% | Dropped |
| Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. | Refusing harmful requests | | 100% | System cards only |
| Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. | Refusing harmful requests | | 100% | System cards only |
| Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. | Jailbreaks and prompt injection | | 83% | System cards only |
| Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. | Refusing harmful requests | | 100% | System cards only |
| Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. | Refusing harmful requests | | 99.4% | System cards only |
| Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. | Jailbreaks and prompt injection | | 92% | System cards only |
| Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. | Jailbreaks and prompt injection | | 27% | System cards only |
| Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. | Refusing harmful requests | Aug 2026 | 100% | System cards only |
| Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. | Capture-the-flag challenges | Nov 2025 | 0% | System cards only |
| Dangerous Content Safety An autorater checks answers to single and multi-turn requests about weapons, illegal acts, violence and self-harm. Reports safe response rate. | Refusing harmful requests | | 96.6% | System cards only |
| Delusional Reinforcement evaluation Multi-turn chats test whether models avoid validating beliefs not grounded in reality and point users to real support. Reports policy adherence rate. | Wellbeing and sycophancy | | 94% | System cards only |
| Emotional Attachment evaluation Chats test whether models avoid claiming affection, fostering dependence or encouraging secrecy. Reports policy adherence rate. | Wellbeing and sycophancy | | 76% | System cards only |
| Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. | Deception and alignment | | 0% | System cards only |
| Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. | Jailbreaks and prompt injection | | 99% | System cards only |
| Grounding with citations, sentence unfactual rate Models answer multi-snippet tasks with inline citations. An autorater reports the share of sentences not supported by the sources. | Faithful to sources | | 8.20% | System cards only |
| Hate & Harassment Safety An autorater checks answers to requests for hateful or harassing content and to sensitive benign prompts. Reports safe response rate. | Refusing harmful requests | | 92.7% | System cards only |
| High-risk health scenarios eval Adversarial health emergencies test whether models recognize urgency and red flags and direct users to help. Reports pass rate. | Medicine | | 98.9% | System cards only |
| Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. | Deception and alignment | | 93.8% | System cards only |
| Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. | Deception and alignment | | 0% | System cards only |
| Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. | Deception and alignment | Oct 2026 | 76.4% | System cards only |
| Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. | Finding and exploiting real bugs | | 72% | System cards only |
| Long-context grounding, sentence unfactual rate Models answer from long documents. An autorater reports the share of sentences not supported by the documents. | Faithful to sources | | 6.40% | System cards only |
| MaXM Vision language models answer short visual questions about Crossmodal-3600 images in 7 languages. A test-only set scored by accuracy. | General visual questions | Sep 2022 | 62.3% | System cards only |
| Mobile Actions Model converts natural-language phone requests into on-device function calls (calendar, email, maps, flashlight, Wi-Fi). Call accuracy. | Function calling | Dec 2025 | 88.4% | Fading |
| Moment finding (Google DeepMind) Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. | Embodied reasoning | Jul 2026 | 91.3% | In use |
| Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. | Dangerous capabilities | Mar 2024 | 3.72 | System cards only |
| Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. | Questions that need search | | 94.3% | System cards only |
| Neutrality evaluation Single and multi-turn prompts on politics, civics, identity and religion test whether models avoid taking sides. Reports neutral response rate. | Bias, fairness and politics | | 93% | System cards only |
| Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. | Deception and alignment | | 55.9% | System cards only |
| Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. | Jailbreaks and prompt injection | | 98% | System cards only |
| Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. | Jailbreaks and prompt injection | | 91% | System cards only |
| Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. | Jailbreaks and prompt injection | | 86% | System cards only |
| RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. | Deception and alignment | | 0% | System cards only |
| Search triggering Models decide whether to call search for queries that need fresh facts or answer directly when search is unneeded. Reports F1. | Function calling | | 88% | System cards only |
| Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. | Dangerous capabilities | Mar 2024 | 0% | System cards only |
| Sexually Explicit Content Safety An autorater checks answers to direct, adversarial and benign sexual content prompts in single and multi-turn chats. Reports safe response rate. | Refusing harmful requests | | 97.6% | System cards only |
| Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). | Dangerous capabilities | May 2025 | 93.3% | System cards only |
| Suicide & Self-Harm evaluation Static and dynamic chats with suicidal or self-harm signals test adherence to prevention practice and crisis resources. Reports policy adherence rate. | Wellbeing and sycophancy | | 96% | System cards only |
| Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. | Questions that need search | | 11.8% | System cards only |
| Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. | Deception and alignment | | | System cards only |
| ASIMOV-Agentic Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes. | Embodied reasoning | Jul 2026 | | System cards only |
| Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. | Dangerous capabilities | Mar 2024 | | System cards only |
| Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. | Deception and alignment | | | System cards only |
| Gemini Personality Evaluation (sycophancy) About 900 single- and multi-turn prompts scoring nine personality dimensions, including candor and self-awareness against sycophancy. | Wellbeing and sycophancy | | | System cards only |
| Health dynamic conversation eval Simulated users hide key health facts at first, and models must ask for context to reach the correct clinical conclusion. | Medicine | | | System cards only |
| Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. | Medicine | | | System cards only |
| Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. | Medicine | | | System cards only |
| Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. | Dangerous capabilities | Mar 2024 | | System cards only |
| Objectron Models predict 3D bounding boxes of objects in short object-centric videos across nine categories. Scored by 3D intersection over union. | Detection and captioning | Dec 2020 | | Dropped |
| SWE agentic fabrication eval A grounding autorater checks whether claims made by coding agents on long SWE tasks are supported by tool outputs. | Hallucination rates | | | System cards only |
| Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. | Dangerous capabilities | Mar 2024 | | System cards only |
| NarrativeQA Models answer questions about books and movie scripts, either from a Wikipedia summary or from the full story. The questions need an understanding of the whole narrative rather than one matching passage. Answers are free text, scored against two reference answers with F1, ROUGE-L or BLEU. | Reading comprehension | Dec 2017 | 0.80 | Not in cards |
| FACTS Search Models answer information-seeking questions with a web search tool (the Brave Search API) and must combine what they find correctly. | Short factual questions | Dec 2025 | 85.6% | Not in cards |
| FACTS Parametric V2 Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer. | Short factual questions | Oct 2026 | 83.8% | Not in cards |
| FACTS Grounding v2 Models write long answers from a supplied document of up to 32k tokens. Judge models check that each answer stays faithful to the document. | Faithful to sources | Dec 2025 | 84.2% | Not in cards |
| FACTS Multimodal Models answer questions about input images and must combine what they see with world knowledge. A judge checks coverage and contradictions. | Short factual questions | Dec 2025 | 60.2% | Not in cards |
| FACTS Search V2 Models answer information-seeking questions with a search tool and must combine what they find correctly. Second version of FACTS Search on Kaggle. | Short factual questions | Jul 2026 | 89.9% | Not in cards |
| Physical-Engineering's Last Exam Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective. | CAD models | Oct 2026 | 0.49 | Not in cards |
| Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. | Building whole projects | Sep 2026 | 40% | Not in cards |
| CLASHarness Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment. | Desktop apps | Oct 2026 | 28.7% | Not in cards |
| Polaris-Bench Models solve the same 53 visual reasoning tasks in Cartesian and polar layouts. The benchmark compares accuracy across the paired layouts. | Spatial reasoning and counting | May 2026 | 39.2% | Not in cards |
| Physics-IQ Verified Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally. | Physical prediction | Jun 2026 | 10.9 | Not in cards |
| BIG-Bench Extra Hard Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. | Language reasoning | Feb 2025 | 44.8% | Not in cards |
| AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. | Machine learning engineering | Jun 2026 | 0.68 | Not in cards |
| MGSM-Rev2 Models solve the 250 MGSM grade-school math problems in 10 languages. Google fixed two English errors, rephrased ambiguous items and retranslated every question with Gemini. Accuracy. | Reasoning in many languages | Nov 2025 | 99.3% | Not in cards |
| Mind the Gap Models describe visible events during natural pauses in a video's soundtrack. The evaluator renders the narration and checks its timing and coverage. | Video captioning | Oct 2026 | 0.94 | Not in cards |
| Putnam-like Models write proofs for 96 university problems in 8 sets of 12, in the style of the easier and middle Putnam tiers. Each problem has a Lean statement and a point rubric for grading. | Written proofs | Sep 2025 | 94.5% | Not in cards |
| Minerva-Ego Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects. | Video understanding | May 2026 | 37.6% | Not in cards |
| XYEval Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice. | Deception and alignment | Sep 2026 | 60.9% | Not in cards |
| AgentRewardBench Models judge whether web-agent trajectories completed their tasks. Their judgments are compared with expert labels across 1,302 trajectories from five benchmarks, using precision, recall, and F1. | Web browsing | Apr 2025 | 82% | Not in cards |
| CURIE Biodiversity Georeferencing Models read a map and its caption from an ecology paper and return the geographic bounds. The score measures overlap with an expert's bounding box. | Spatial reasoning and counting | | 19.4 | Not in cards |
| CURIE Geospatial Dataset Extraction Models read a research paper and list its datasets, variables, sources, time spans, and geographic coverage in JSON. The authors report ROUGE-L and BERTScore. | Finding facts in long inputs | | 29.7 | Not in cards |
| MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. | Long videos | Jan 2026 | 42.2% | Not in cards |
| AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. | Machine learning engineering | Mar 2025 | 22% | Not in cards |
| CodeContests Models write programs for competitive programming problems, drawn from Codeforces and other sites, and the programs run against hidden tests. Google DeepMind built the dataset for AlphaCode, with training, validation (117 problems) and test (165 problems) splits and many generated tests per problem to cut false positives. Scores are the share of problems solved within a fixed number of submissions, such as 10@k. | Competitive programming | Feb 2022 | 81.8% | Not in cards |
| IMO-GradingBench Models grade 1,000 olympiad proof attempts, and their grades are compared with human expert grades. Tests automatic evaluation of long-form math answers. | Written proofs | Nov 2025 | 54% | Not in cards |
| Kinetics-700-2020 Models recognize human actions in short videos from the 2020 edition of Kinetics-700. | Video understanding | Oct 2020 | 62% | Not in cards |
| TCS-Bench Models write proofs of target results from STOC, FOCS and SODA papers, given the needed context. A verification agent, checked against expert judgments, grades each proof. | Research mathematics | Aug 2026 | 55% | Not in cards |
| Tricky Edits of Collected and Curated Images Models edit a source image to follow a written instruction. People and a model judge instruction following, preservation of unaffected content, and visual quality. | Image generation and editing | May 2026 | 19.5% | Not in cards |
| DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. | Dangerous capabilities | Mar 2024 | 20.5% | Not in cards |
| MultiMedQA Models answer medical exam, research, and consumer health questions across seven datasets. Multiple-choice components use answer accuracy. Clinicians and other raters assess free responses for factuality, precision, harm, and bias. | Medicine | Dec 2022 | 58.9% | Not in cards |
| ChessBench Models select chess moves from legal options in positions scored by a chess engine. Decision Index measures whether the selected move has maximal action value. | Puzzles and chess | | 66.5% | Not in cards |
| FreshQA Models answer questions whose answers change over time, never change, or rest on a false premise that they should reject. FreshEval, an LLM judge, grades answers in a Relaxed mode or a Strict mode that also penalizes any hallucinated detail. | Questions that need search | Oct 2023 | 80.6% | Not in cards |
| GeckoNum Models generate images from prompts about quantities. Human raters check exact counts, approximate quantities, and reasoning about numerical concepts. | Image generation and editing | Jun 2024 | 47.6% | Not in cards |
| IMO-LeanProofBench Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof. | Formal proofs | Jun 2026 | 11.6% | Not in cards |
| Kinetics-400 Models recognize one of 400 human actions in short video clips. | Video understanding | May 2017 | 0.25% | Not in cards |
| Life-Bench Models answer multimodal questions about synthetic personal lifelogs. Tasks cover concept identification, event understanding, and reasoning over multiple pieces of evidence. | General visual questions | Feb 2026 | 56% | Not in cards |
| AlphaEarth Foundations mapping evaluation Models use satellite representations to classify land cover, detect change, or estimate physical quantities. The evaluation compares predictions with independent measurements across regions. | General visual questions | Jul 2025 | | Not in cards |
| AnthroBench Models hold multi-turn conversations that test 14 anthropomorphic behaviors. Automated classifiers and human participants evaluate the generated dialogue. | Instructions and chat | Feb 2025 | | Not in cards |
| ASIMOV v1 Models judge whether robot actions in scenes generated from real images and hospital injury reports are undesirable or unsafe. Scored by alignment with human labels. | Embodied reasoning | Mar 2025 | | Not in cards |
| ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. | Jailbreaks and prompt injection | Nov 2026 | | Not in cards |
| AudioSet Strong Models locate labeled sound events in time. The evaluation uses human annotations of event boundaries in AudioSet clips. | Audio understanding and voice | | | Not in cards |
| BIG-bench Language models do over 200 tasks contributed by hundreds of authors, from linguistics and math to social bias. Most tasks are scored by exact match or multiple choice. | Language reasoning | Jun 2022 | | Not in cards |
| CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. | Medicine | Jan 2026 | | Not in cards |
| DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. | Simulated robots | Sep 2024 | | Not in cards |
| DeepMind Fast Language Learning Tasks Models learn mappings between unfamiliar words and meanings from short examples. Held-out tasks measure how well they infer new word meanings. | Language reasoning | Jul 2017 | | Not in cards |
| EHRQA (Google) Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning. | Medicine | Jul 2025 | | Not in cards |
| Encyclopedic VQA Models answer questions about detailed properties of pictured objects and places. The Gemini Embedding 2 report measures whether they retrieve the correct answer. | General visual questions | Jun 2023 | | Not in cards |
| EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. | Medicine | Jul 2025 | | Not in cards |
| FEABench Models write Java commands that build and run engineering simulations in COMSOL. The evaluator checks execution, model features, and numerical results where reference values exist. | Scientific code | | | Not in cards |
| Formal Conjectures Models prove or disprove mathematical conjectures formalized in Lean. The collection includes open research problems and records machine-checked solutions. | Formal proofs | | | Not in cards |
| Formal-IMO Models prove formal statements of International Mathematical Olympiad problems in Lean. The repository supports partial credit when a problem has multiple parts. | Formal proofs | May 2025 | | Not in cards |
| gBench Models run academic evaluations, Golden Set smoke tests, and an agentic quality suite through an OpenAI-compatible endpoint. The tool also records serving performance. | Machine learning engineering | Jan 2025 | | Not in cards |
| Gecko People compare text to image outputs for prompt alignment. The benchmark studies model ordering, pairwise instance scoring, and pointwise instance scoring. | Image generation and editing | Apr 2024 | | Not in cards |
| Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. | Deception and alignment | | | Not in cards |
| Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. | Refusing harmful requests | | | Not in cards |
| Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. | Jailbreaks and prompt injection | | | Not in cards |
| Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. | Real robots | Mar 2025 | | Not in cards |
| Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. | Real robots | Jul 2026 | | Not in cards |
| GenExam Image generation systems are evaluated on subject-focused educational tasks with expert ratings. | Image generation and editing | | | Not in cards |
| GeoExpertQueries Agents choose geospatial data and carry out an analysis. A model compares each execution trace with a reference workflow agreed on by three GIS specialists. | Apps and MCP servers | Oct 2026 | | Not in cards |
| GeoKTools Agents combine up to four kinds of geospatial tools to answer quantitative questions. Exact matches earn full credit, and a model grades other answers. | Apps and MCP servers | Oct 2026 | | Not in cards |
| Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. | Dangerous capabilities | | | Not in cards |
| Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. | Finding and exploiting real bugs | | | Not in cards |
| Google Landmarks Dataset v2 Models recognize landmarks or retrieve photographs of the same landmark. The benchmark uses images from around the world. | General visual questions | Apr 2020 | | Not in cards |
| Google Scanned Objects Models synthesize an object from a new viewpoint. Researchers compare the rendered result with reference views of scanned household items. | Spatial reasoning and counting | Apr 2022 | | Not in cards |
| Google Universal Image Embedding Challenge Models retrieve images of the same item across several visual domains using a single image representation. The challenge compares results against labeled matches. | General visual questions | Aug 2022 | | Not in cards |
| HierText Models locate and recognize text arranged in lines and paragraphs in real scenes. Google also uses the annotations to compare text and image retrieval. | Documents, charts and text in images | Mar 2022 | | Not in cards |
| ImageNet ReaL Models classify ImageNet validation images against revised human annotations. An image can have more than one acceptable label. | General visual questions | Jun 2020 | | Not in cards |
| KilometerVision Models watch walking-tour videos geolocated over up to 1 km and answer questions on landmarks, routes and map layout, such as distances and loop closures. | Spatial reasoning and counting | Jun 2026 | | Not in cards |
| Kinetics-600 Models recognize one of 600 human actions in short video clips. | Video understanding | Aug 2018 | | Not in cards |
| Kinetics-700 Models classify short video clips into 700 human action categories. Meta compares encoders on this expanded Kinetics edition. | Video understanding | Jul 2019 | | Not in cards |
| LiveMath Models solve 250 math problems written with six professional mathematicians. The sets are 150 regular high school problems, 50 at about AIME level and 50 multimodal problems with figures. | Competition problems with answers | Sep 2025 | | Not in cards |
| LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. | Embodied reasoning | Dec 2024 | | Not in cards |
| Long Range Arena Transformers solve six long-sequence tasks covering list operations, text, retrieval, image, path and path crossing. | Finding facts in long inputs | Jan 2021 | | Not in cards |
| MAESTRO Piano Transcription Models transcribe competition piano recordings into notes. The evaluator compares predictions with MIDI events captured by the instrument. Each composition appears in only one split. | Audio understanding and voice | Oct 2018 | | Not in cards |
| Mathematical Problem Repository for AlphaEvolve Agents generate programs that construct mathematical objects or improve known bounds. The repository documents 67 problems and provides verification code for many of them. | Research mathematics | Nov 2025 | | Not in cards |
| MedGemma lab report conversion Models convert lab reports to structured JSON. Google reports macro and micro F1 on two collections of clinical PDFs and a collection of synthetic reports. | Documents, charts and text in images | Jan 2026 | | Not in cards |
| Meta-Dataset Models learn from episodes sampled across diverse image datasets and classify novel classes from a few examples. | General visual questions | Mar 2019 | | Not in cards |
| MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. | Medicine | Jan 2026 | | Not in cards |
| NATURAL PLAN Models make travel itineraries, find meeting arrangements, and schedule calendar events. The evaluator checks each proposed plan against the stated constraints. | General assistants | Jun 2024 | | Not in cards |
| Natural World Tasks (NeWT 2021) Models distinguish pairs of labels in wildlife photographs. The 164 tasks include recognizing life stages, animal behavior and scene context. | General visual questions | Mar 2021 | | Not in cards |
| NEVIS’22 Models learn 106 image classification tasks in chronological order. The evaluation compares error rates against the computation each model uses. | General visual questions | Nov 2022 | | Not in cards |
| PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. | Medicine | Jul 2025 | | Not in cards |
| PushWorld Agents move shapes through puzzles that require tools or pushing obstacles aside. The benchmark includes more than 200 puzzles with goal positions. | Puzzles and chess | Jan 2023 | | Not in cards |
| Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. | Simulated robots | Oct 2020 | | Not in cards |
| SciVid Video models answer questions about scientific videos across several disciplines. The benchmark tests whether models recognize procedures, objects, and events in specialized footage. | Vision | Jul 2025 | | Not in cards |
| Screen2Words Models summarize the purpose and contents of a mobile interface from a screenshot. The benchmark uses human-written summaries as references. | Documents, charts and text in images | Oct 2021 | | Not in cards |
| StreamingQA Models answer time-stamped questions about news. The test measures how well they acquire new information while retaining older knowledge. | Questions that need search | May 2022 | | Not in cards |
| TAP-Vid Models track individual points across frames and predict when each point disappears. The benchmark compares their trajectories and visibility predictions with reference tracks. | Video understanding | Nov 2022 | | Not in cards |
| US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. | Medicine | Jul 2025 | | Not in cards |
| VideoPhy Text-to-video models generate clips from captions of physical interactions between solids and fluids. Human raters judge caption match and physical commonsense. | Physical prediction | Jun 2024 | | Not in cards |
| WAXAL Speech Recognition Test Speech models transcribe people describing pictures in African languages. Researchers compare the transcripts with human references using word error rate. | Speech recognition and translation | Jan 2026 | | Not in cards |
| Widget Captioning Models describe an individual interface element in a mobile screenshot. The benchmark compares their captions with human descriptions of the same element. | Documents, charts and text in images | Oct 2020 | | Not in cards |
| WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. | Medicine | Jan 2026 | | Not in cards |
| XTREME Multilingual models are fine-tuned on English data and then tested on 9 tasks in 40 languages. The tasks cover sentence classification, structured prediction such as named entities, question answering and sentence retrieval. The benchmark measures how well skills transfer across languages. | Reading in many languages | Mar 2020 | | Not in cards |
| XTREME-R Multilingual models are fine-tuned on English data and tested on 10 tasks in 50 typologically diverse languages, including harder language-agnostic retrieval tasks. It extends XTREME and adds MultiCheckList, a multilingual diagnostic suite. | Reading in many languages | Apr 2021 | | Not in cards |