| VCSUM Models summarize transcripts of 239 real Chinese meetings that total over 230 hours. Scored with ROUGE against human-written summaries. | Reasoning over long documents | May 2023 | 17.3 | Dropped |
| BLUEX Models answer questions from two Brazilian university entrance examinations. The questions cover high school subjects in Portuguese. | Knowledge in many languages | Jul 2023 | 95% | Not in cards |
| Angry Tweets Models classify the sentiment of Danish social media posts using labels supplied by crowdworkers. | Reading in many languages | May 2021 | 67% | Not in cards |
| FoSent Models classify sentiment in Faroese news articles. Two native speakers annotated articles from Portalurin and Dimmalætting. | Reading in many languages | May 2024 | 91.6% | Not in cards |
| FoQA Models answer questions about passages from Faroese Wikipedia. A native speaker checked and corrected automatically generated answers. | Reading in many languages | Feb 2025 | 79.9% | Not in cards |
| GermanQuAD Models extract answers to German questions from manually annotated Wikipedia passages. | Reading in many languages | Apr 2021 | 70% | Not in cards |
| OCR-VQA Models answer questions about book covers by reading visible text. The benchmark compares answers with facts such as a book’s title or author. | Documents, charts and text in images | Sep 2019 | 82% | Not in cards |
| Icelandic WinoGrande Models complete Icelandic sentences by choosing which referent fits the context. The evaluation uses an Icelandic adaptation of WinoGrande. | Reasoning in many languages | Jun 2022 | 96.3% | Not in cards |
| TeleQnA Models answer multiple-choice questions about telecommunications terminology, research, and technical standards. | Expert-written questions | Oct 2023 | 85.2% | Not in cards |
| TeleTables Models answer questions about tables from 3GPP specifications, both from memory and with the table available. | Expert-written questions | Jan 2026 | 48% | Not in cards |
| PAIRS (Parallel Images) Models answer questions about people in paired images. The evaluation tests whether changing demographic characteristics changes judgments about the subjects. | Bias, fairness and politics | Feb 2024 | 97.2% | Not in cards |
| Unicorn (Vision Safety) Models answer questions about objects in unusual images and sketches. The benchmark tests how well visual judgments hold up when the images change. | General visual questions | Nov 2023 | 91.2% | Not in cards |
| FINESSE-Bench Models answer financial certification questions and interpret trading indicators. The benchmark checks numerical and multiple-choice answers and uses a model judge for free responses. | Finance | May 2026 | 0.90 | Not in cards |
| LongBench Pro Models answer English and Chinese questions over documents from 8,000 to 256,000 tokens. The benchmark applies task-specific metrics to 1,500 examples. | Reasoning over long documents | Jan 2026 | 73.4% | Not in cards |
| PerspectiveGap Models assign information fragments to workers with different roles. A strict pass requires supplying all needed information without disclosing extra fragments. | Following precise instructions | Jun 2026 | 62% | Not in cards |
| GreekMMLU Models answer questions originally written in Greek for academic, professional, and government exams across 45 subjects. | Knowledge in many languages | Feb 2026 | 93.2% | Not in cards |
| HumanAgencyBench Models respond to 3,000 prompts about six ways assistants can support human agency, including asking clarifying questions and leaving important decisions to users. | Deception and alignment | Sep 2025 | 71.3% | Not in cards |
| OrangeSum Models summarize French news articles from Orange Actu. The dataset pairs articles with summaries written by journalists. | Reading in many languages | Oct 2020 | 38.5% | Not in cards |
| Uncheatable Eval Models predict text from recently published documents. The evaluation measures perplexity across sources including news, research papers, fiction and code. | Reading comprehension | May 2024 | 6.29 | Not in cards |
| LLMzSzŁ Models answer Polish academic and professional exam questions drawn from the Central Examination Board archives. | Knowledge in many languages | Jan 2025 | 69.1% | Not in cards |
| SLR-Bench Models write Prolog rules that classify trains by their composition. The verifier runs each rule to determine whether its predictions are correct. | Abstract puzzles and logic | Jun 2025 | 81.8% | Not in cards |
| ArxivRollBench Models reconstruct passages from recently published scientific papers. They order fragments, fill gaps and choose what comes next. | Reading comprehension | Jul 2025 | 70.8% | Not in cards |
| KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. | Medicine | Sep 2026 | 29.4% | Not in cards |
| MUSE Models answer questions about artworks used in language education. The tasks cover visual details, emotions, cultural context, and relationships within the image. | Other professions | Sep 2026 | 56.7% | Not in cards |
| AnalogCoder Models write Python programs that build analog circuits. A simulator checks whether each circuit performs its requested function. The authors report first-attempt success and how many tasks each model solves. | Electronics | May 2024 | 83.3% | Not in cards |
| Evalita-LLM Models perform tasks written originally in Italian. The suite includes admission exams, question answering and text generation. | Reading in many languages | Feb 2025 | 62.3 | Not in cards |
| NoveltyBench Models give several answers to the same open-ended request. The benchmark measures how many distinct responses they produce and how useful those responses are. | Creative writing | Apr 2025 | 5.32 | Not in cards |
| VeriGen design problems Models complete Verilog modules for 17 digital circuits. Simulation checks whether the generated code satisfies each design’s testbench. | Electronics | Dec 2022 | 77.7% | Not in cards |
| BertaQA Models answer questions about Basque culture and general knowledge. Parallel Basque and English questions help distinguish language effects from familiarity with local topics. | Trivia and open questions | Jun 2024 | 81.5% | Not in cards |
| K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. | Other professions | Jun 2025 | 55.4% | Not in cards |
| DaLA Models distinguish acceptable Danish sentences from copies corrupted to resemble errors in real text. | Reasoning in many languages | Dec 2025 | 61.7% | Not in cards |
| EWoK Models decide which context makes a target statement plausible. Paired examples isolate concepts about the physical and social world. | Commonsense | May 2024 | 91.2% | Not in cards |
| IFEvalCode Models write code in eight programming languages under explicit constraints. The evaluator checks functional correctness and adherence to instructions separately. | Function-level problems | Jul 2025 | 38.2% | Not in cards |
| MultiBLiMP Models compare minimally different sentences across languages. The benchmark tests whether the model assigns higher likelihood to the grammatical sentence. | Reasoning in many languages | Apr 2025 | 90.2% | Not in cards |
| EusExams Models answer past public service examination questions in Basque. The questions come from several Basque institutions. | Knowledge in many languages | Mar 2024 | 51.3% | Not in cards |
| EusReading Models answer comprehension questions about Basque passages from past C1 proficiency examinations. The scorer checks their choices against the exam key. | Reading in many languages | Mar 2024 | 62.2% | Not in cards |
| EusTrivia Models answer trivia questions in Basque. The set combines elementary-school questions with harder general knowledge questions. | Knowledge in many languages | Mar 2024 | 51% | Not in cards |
| GPBam Models write legal opinions on German public law cases and retrieve the statutes they need. Three judges grade essays against reference solutions. | Law | Nov 2026 | 76.4 | Not in cards |
| EvaOmni Models answer questions about videos and identify the audio and visual evidence behind their answers. The score accounts for both correctness and evidence timing. | Audio understanding and voice | Oct 2026 | 79.4% | Not in cards |
| BLiMP-NL Models compare Dutch sentence pairs that differ in grammatical acceptability. | Reasoning in many languages | Jul 2025 | 49.4% | Not in cards |
| MMCircuitEval Models answer questions about circuit diagrams, specifications, and design decisions. The authors combine text similarity with a language model’s judgment. | Electronics | Jul 2025 | 69.4% | Not in cards |
| Spatial Pathfinding Reasoning Challenge Models trace paths across grids while satisfying rules about cells and edges. The scorer checks each coordinate sequence against valid solutions. | Puzzles and chess | May 2025 | 83.6% | Not in cards |
| EvoEval Models solve Python problems that vary the constraints, interfaces, or logic of HumanEval tasks. Harbor ports the 100 problems in the Difficult subset. | Function-level problems | Mar 2024 | 68.6% | Not in cards |
| IndicParam Models answer multiple-choice questions from Indian competitive exams. The benchmark tests knowledge in low-resource Indic languages. | Knowledge in many languages | Nov 2025 | 45% | Not in cards |
| MentalBench (Psychiatric Diagnosis) Models select psychiatric diagnoses for 24,750 synthetic cases. Exact matching requires both diagnoses when the case lacks the evidence needed to distinguish them. | Medicine | Feb 2026 | 62.7% | Not in cards |
| NorEval Models complete language understanding and generation tasks in both Bokmål and Nynorsk. The authors compare model performance with human baselines. | Knowledge in many languages | Apr 2025 | 68.2 | Not in cards |
| CS-4k Models answer questions about the stages of computer science research using papers as evidence. A model judge scores answers on the held-out evaluation set. | AI research tasks | Oct 2025 | 71.3 | Not in cards |
| EarthVerse Agents investigate documented natural hazards using event data from different sources. The evaluator checks individual claims and the calculations that support them across 405 tasks. | Lab and data work | Aug 2026 | 85% | Not in cards |
| KorNAT Models answer questions about Korean social values and common knowledge. The authors compare answers with Korean reference judgments. | Knowledge in many languages | Feb 2024 | 65.4% | Not in cards |
| HireWatch Models select questions for job interviews under different workplace pressures. The benchmark measures how often they select questions classified as unlawful under US employment law. | Law | Nov 2026 | 0.05% | Not in cards |
| SG-LegalClarify Models answer Singapore legal questions and ask for missing facts. Three model judges grade each conversation against rubrics written by lawyers. | Law | Nov 2026 | 77.8% | Not in cards |
| UniCode Models solve generated competitive programming problems using automated test cases. Harbor packages 399 problems with validated reference solutions. | Competitive programming | Oct 2025 | 70.3% | Not in cards |
| AgroMind (original) Models answer questions about farmland images across 13 tasks. They identify crops, assess crop health and interpret spatial relationships. | General visual questions | May 2025 | 35.8% | Not in cards |
| FinFIRST Agents research financial questions and provide supporting sources. The benchmark grades individual requirements and reports how often an answer meets every requirement. | Finance | Sep 2026 | 71.5% | Not in cards |
| MVDriveBench Models answer driving questions using synchronized videos from cameras facing different directions. The authors compare surround-view results with front-camera results. | Embodied reasoning | Nov 2026 | 80.1% | Not in cards |
| PISA-Bench Models answer illustrated PISA questions in six languages. Parallel questions let researchers compare language effects on reading, science and mathematical reasoning. | General visual questions | Oct 2025 | 63.3% | Not in cards |
| SATBench Models decide whether a set of constraints in a narrative puzzle can all hold at once. A Boolean satisfiability solver supplies the correct label. | Language reasoning | May 2025 | 89.3% | Not in cards |
| LegalCiteBench Models recover and verify citations from United States judicial opinions without external lookup. The benchmark also tests whether models match cases and correct citation errors. | Law | Feb 2026 | 0.78 | Not in cards |
| LongProc Models carry out long procedures such as converting HTML tables, tracing paths and tracking beliefs. The evaluator checks outputs as the required procedure grows. | Reasoning over long documents | Jan 2025 | 74.2 | Not in cards |
| MUSE (CAD assemblies) Models turn design specifications into CAD assemblies. A program checks the geometry. A vision model grades function, manufacturability, and ease of assembly. | CAD models | May 2026 | 52.4% | Not in cards |
| RPC-Bench Models read computer science papers and answer reviewers’ questions. A model judge grades correctness, completeness, and concision against verified review and rebuttal pairs. | Research science | Apr 2026 | 37.5 | Not in cards |
| ATM-Bench Agents answer questions about a person’s history using images, videos, and email records. The Harbor package contains 31 hard questions with schema-guided memory. | Finding facts in long inputs | Mar 2026 | 86% | Not in cards |
| ChipBench Models write Verilog modules, debug circuits and generate reference models. Simulations compare the outputs with reference implementations. | Electronics | Jan 2026 | 30.5% | Not in cards |
| NPCBench Models follow patients with nasopharyngeal carcinoma through diagnosis and care. The main patient-level measure requires correct decisions at successive stages. | Medicine | Nov 2026 | 23% | Not in cards |
| TrafficImag Models edit roadside traffic videos to show a different sequence of events. The evaluator checks routes, interactions, and preservation of the surrounding scene. | Video generation | Sep 2026 | 90.0 | Not in cards |
| Code4Scene Agents build and edit Unreal Engine scenes. The evaluator inspects the scene itself for correct edits, intact assets, and physical validity. | Image generation and editing | Sep 2026 | 0.62 | Not in cards |
| Glianorex Models answer multiple-choice questions about fictional medical material in English and French. The test isolates question-answering patterns from real medical knowledge. | Language reasoning | Jun 2024 | 71.5% | Not in cards |
| PDAGENT-BENCH Models answer chip design questions, diagnose failures, interpret reports, and write tool commands. Rules grade five categories and model judges grade three. | Electronics | Jun 2026 | 65.1% | Not in cards |
| SAHM Models answer Arabic questions on financial standards, Islamic finance, and accounting. The authors report multiple-choice accuracy and grade open-ended answers separately. | Finance | Apr 2026 | 80.5% | Not in cards |
| CaBBQ Models answer Catalan questions about people in ambiguous and unambiguous situations. The test measures whether answers follow social stereotypes when evidence is insufficient. | Bias, fairness and politics | Jul 2025 | 81% | Not in cards |
| CASE-Bench Models decide whether a request is safe in a given context. The same request appears in two contexts with different human judgments. | Refusing harmful requests | Jan 2025 | 90.1% | Not in cards |
| EsBBQ Models answer Spanish questions about people in ambiguous and unambiguous situations. The evaluation measures how social stereotypes affect their choices. | Bias, fairness and politics | Jul 2025 | 85% | Not in cards |
| LawBench Models answer Chinese legal questions across 20 tasks, including statute recall and case prediction. Each task has its own scoring rule. | Law | Sep 2023 | 44.5 | Not in cards |
| NoticIA Models summarize Spanish news articles in one sentence that answers the clickbait headline. The evaluator compares each summary with a human reference. | Faithful to sources | Apr 2024 | 32.0 | Not in cards |
| OmniMatBench Models answer questions and perform calculations across 19 materials science fields. The authors compare automated grades with human assessments. | Science questions | May 2026 | 0.37 | Not in cards |
| RPEval Models respond in character to 9,018 scenarios. The test checks emotions, decisions, moral choices, and whether characters reveal knowledge outside their setting. | Overall chat quality | May 2025 | 62.2% | Not in cards |
| Taiwanese Mandarin Language Understanding Models answer questions from Taiwanese exams in traditional Chinese. The subjects range from school topics to professional qualifications. | Knowledge in many languages | Mar 2024 | 73.6% | Not in cards |
| VoxCeleb2 Models compare two interview recordings and decide whether the same person speaks in both. | Audio understanding and voice | Jun 2018 | 75.9% | Not in cards |
| CBT-Bench Level II Models identify cognitive distortions and classify patients’ core beliefs at two levels of detail. The evaluator compares the selected labels with the reference labels. | Medicine | Oct 2024 | 0.63 | Not in cards |
| MedSafetyBench Models respond to requests that conflict with medical ethics. A judge rates harmfulness on a five-point scale using a held-out set of 900 requests. | Refusing harmful requests | Mar 2024 | 1.44 | Not in cards |
| NOTA Models extract facts from sheet music, convert score images to ABC notation, and analyze musical structure. The authors report extraction rates and separate transcription distances. | Documents, charts and text in images | Jun 2024 | 17.8% | Not in cards |
| SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. | Research science | Oct 2026 | 80.9 | Not in cards |
| SIM-VAIL Chatbots respond to simulated users with psychiatric vulnerabilities. A model judge scores behaviors across the conversation, including responses that reinforce a user’s vulnerability. | Wellbeing and sycophancy | Feb 2026 | 1 | Not in cards |
| AgroBench Models answer illustrated questions from agronomists about crops, diseases, pests, weeds, machinery and farming practices. The overall score averages accuracy across seven tasks. | General visual questions | Jul 2025 | 73.5% | Not in cards |
| ANCHOR Models explain answers to audiovisual questions and locate the supporting evidence. The benchmark includes time intervals, image regions, and relevant sounds. | Audio understanding and voice | Oct 2026 | 3.36 | Not in cards |
| AutoBencher Models answer generated benchmark questions. The framework searches for topics and produces tests of knowledge, multilingual reasoning, and math. | Language reasoning | Jul 2024 | 55% | Not in cards |
| ChemArgus Models solve chemistry competition and textbook problems with text and figures. A model judge awards partial credit for each subquestion. | Science questions | Oct 2026 | 64.9% | Not in cards |
| EusProficiency Models complete Basque language exercises from the official C1 proficiency examination. The questions test vocabulary and grammar. | Reading in many languages | Mar 2024 | 56.7% | Not in cards |
| GeoDrive-Bench Models choose driving actions from road images and local traffic conventions. The benchmark covers six countries and omits explicit country labels. | Embodied reasoning | Jun 2026 | 91.9% | Not in cards |
| HarvestBench Agents drive simulated tractors and decide how much fuel to spend avoiding animals. The benchmark measures how often they hit animals as avoidance costs change. | Deception and alignment | Sep 2026 | 0.40% | Not in cards |
| L2-Bench Models design lessons, give learners feedback, and participate in language practice. A model judge grades each response against teaching criteria. | Other professions | Jul 2026 | 85.5 | Not in cards |
| LIBRA (Russian Long Context) Models read long Russian texts and answer questions or generate responses. The suite groups tasks by difficulty and varies input length. | Reasoning over long documents | Aug 2024 | 71.3 | Not in cards |
| LMentry Models perform simple language tasks such as identifying words with a given letter or choosing the longer word. Task-specific rules check the answers. | Language reasoning | Nov 2022 | 66.1% | Not in cards |
| LongCoT Models solve long reasoning problems across five domains. Task-specific programs verify answers, with optional judge fallback for ambiguous mathematics and chemistry responses. | Reasoning over long documents | Apr 2026 | 25.1% | Not in cards |
| Longtail-SWE Agents resolve 202 GitHub issues across 13 programming languages with limited benchmark coverage. Hidden tests check the fix and verify that previously passing behavior still works. | Fixing real repositories | Nov 2026 | 81.5% | Not in cards |
| P3D-Bench Models generate parametric 3D objects from text, images, or assembly instructions. The evaluator checks whether outputs run and preserve geometry, topology, and part structure. | Image generation and editing | Jun 2026 | 69.4 | Not in cards |
| QuantEval Models answer quantitative finance questions, solve mathematical problems, and implement trading strategies. The authors report answer accuracy and test the generated code. | Finance | Jan 2026 | 86% | Not in cards |
| RVG-Bench Models locate the video interval that answers a question about sounds and images. The evaluator measures overlap with the annotated interval. | Audio understanding and voice | Oct 2026 | 49.6% | Not in cards |
| SciKnowEval Models answer scientific questions and apply knowledge in biology, chemistry, materials and physics. The suite uses automatic checks and model judges for different tasks. | Science questions | Jun 2024 | 0.71 | Not in cards |
| SolveEdit Models edit an image to solve a visual problem. The evaluator checks whether the goal holds and whether unrelated scene details survive the edit. | Image generation and editing | Sep 2026 | 57% | Not in cards |
| SPOT Models inspect published manuscripts for errors that led to corrections or retractions. A judge checks whether each reported error matches the documented error and location. | Research science | May 2025 | 6.10% | Not in cards |
| Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. | Multi-agent games | Jun 2025 | 84.9% | Not in cards |
| WorldCoder-Bench Agents write code that builds interactive 3D worlds. The evaluator checks whether the resulting world behaves as specified. | Image generation and editing | Jun 2026 | 27.8% | Not in cards |
| AfroBench Models perform language understanding, reasoning, and generation tasks across 64 African languages. The full suite draws on 22 datasets. | Knowledge in many languages | Nov 2023 | 59.6 | Not in cards |
| ChemDIRT Models answer chemistry questions while the evaluator varies instructions and molecular representations. The authors measure accuracy and consistency across variants. | Science questions | Mar 2026 | 48.7% | Not in cards |
| GeoNatureAgent Benchmark Agents use a geospatial API to analyze environmental indicators in Spain and Portugal. The grader checks tool use and answers against task requirements. | Function calling | Apr 2026 | 60.8% | Not in cards |
| GreekBarBench Models answer Greek bar exam questions in writing. A model judge grades their use of case facts, statutory articles, and legal analysis. | Law | May 2025 | 8.35 | Not in cards |
| LegalBench.PT Models answer questions about Portuguese law derived from university exams. The benchmark includes case analysis, multiple selection, and true-or-false questions. | Law | Dec 2024 | 85.4% | Not in cards |
| LexGenius Models answer Chinese legal questions drawn from cases and exams. The authors compare results across areas including legal reasoning, ethics, and judicial practice. | Law | Nov 2025 | 59.7% | Not in cards |
| LingOly Models infer linguistic rules from examples in unfamiliar languages. They use those rules to solve linguistics olympiad problems. | Reasoning in many languages | Jun 2024 | 46.3% | Not in cards |
| MasakhaPOS Models assign parts of speech to words in African languages. The authors compare tagging models across languages. | Reading in many languages | May 2023 | 62.8% | Not in cards |
| MastermindEval Models infer a hidden symbol sequence from previous guesses and feedback in Mastermind. The evaluation varies sequence length and the similarity of answer choices. | Puzzles and chess | Mar 2025 | 53.7% | Not in cards |
| MentalBench-100k Models answer counseling questions. Human reviewers grade responses to 1,000 of the 10,000 prompts on seven attributes, each from one to five. | Medicine | Oct 2025 | 5 | Not in cards |
| MISHAP-Bench Models answer open-ended questions about audio clips. A model judge uses human annotations to identify claims the recording does not support. | Audio understanding and voice | Sep 2026 | 32.8% | Not in cards |
| OvertonBench Models answer questions about contested issues. Participants rate how well each answer represents their viewpoint. The score measures the fraction of opinion groups represented. | Bias, fairness and politics | Sep 2025 | 0.42 | Not in cards |
| PaperScope Agents search multiple scientific papers to compare figures, summarize results, and solve problems. Each question can require evidence from several documents. | Deep research | Jan 2026 | 41.0 | Not in cards |
| SAS-Bench Models grade short student answers from subject exams. The benchmark compares their scores and error classifications with expert annotations. | Other professions | May 2025 | 67.2% | Not in cards |
| VoxelCodeBench Models write code to build and modify voxel scenes. The benchmark checks the constructed shapes and evaluates their appearance. | Image generation and editing | Apr 2026 | 87.9% | Not in cards |
| ABACUS Models add, remove, or change a specified number of objects in an image. The evaluator checks the count and preservation of the surrounding scene. | Image generation and editing | Oct 2026 | 0.31 | Not in cards |
| ChemInsight Models solve related chemistry problems in bundles. The authors score how many parts a model can solve and how far its reasoning remains correct. | Science questions | Nov 2026 | 0.28 | Not in cards |
| Closed-Loop Evaluation of LLM Agents for Embedded Software Development Agents write firmware from engineering briefs and test it on simulated equipment. Hidden checks score device behavior across five control tasks. | Electronics | | 100% | Not in cards |
| ContextASR-Bench Models transcribe speech with optional domain labels and named entities. The benchmark measures transcription errors and recognition of named entities. | Speech recognition and translation | Jul 2025 | 3.10% | Not in cards |
| CRUST-bench Agents translate 100 C projects into safe Rust. Tests check whether each translation matches the supplied Rust interface and preserves the original behavior. | Fixing real repositories | Apr 2025 | 48% | Not in cards |
| ELBench Models respond to teaching and safety scenarios alongside academic questions. The authors compare the same models across four educational assessment areas. | Other professions | Jun 2026 | 83.5 | Not in cards |
| EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. | Simulated robots | Oct 2026 | 41.4% | Not in cards |
| FinDeepIndicator Agents construct financial indicators by specifying formulas, retrieving market data, and calculating results. The evaluation checks performance at each step. | Finance | Aug 2026 | 39% | Not in cards |
| FinMaster Models complete financial workflows using simulated company records. The benchmark includes single calculations and tasks that require several financial metrics. | Finance | May 2025 | 73% | Not in cards |
| Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. | Medicine | Nov 2026 | 83.9% | Not in cards |
| MDMR-Bench Models generate one image from several references. A model judge checks whether the output preserves requested elements, follows instructions, and combines them correctly. | Image generation and editing | May 2026 | 88.8% | Not in cards |
| NARRA-Gym Agents build interactive stories around a user’s experiences and adapt scenes as the conversation unfolds. Judges rate story quality and user experience separately. | Creative writing | May 2026 | 2 | Not in cards |
| ReplicationBench Agents reproduce computational results from published astrophysics papers. Evaluators check their numerical answers and assess whether their methods follow the original work. | Research science | Oct 2025 | 22% | Not in cards |
| RExBench Agents extend 12 published AI research projects from expert instructions. The evaluator runs their code against task-specific success criteria. Harbor distributes two public tasks. | AI research tasks | Jun 2025 | 50% | Not in cards |
| SAVU-Bench Models answer spatial questions about real scenes using sound and video. The authors report accuracy across seven tasks. | Audio understanding and voice | Oct 2026 | 55.3% | Not in cards |
| VLegal-Bench Models answer questions and solve legal scenarios under Vietnamese law. Legal experts check that reference answers follow authoritative documents. | Law | Dec 2025 | 72% | Not in cards |
| AnalogCoder-Pro Agents write analog circuit programs and use simulated waveforms to diagnose failures. The authors report the fraction of successful designs and the number of tasks solved. | Electronics | | 88.4% | Not in cards |
| AstroVLBench Models interpret galaxy images, light curves, and spectra across five observing tasks. The authors compare their classifications with reference labels and examine their explanations. | Lab and data work | Apr 2026 | 68.8% | Not in cards |
| CNN/DailyMail Summarization Models summarize news articles from CNN and the Daily Mail. The harness compares generated summaries with the publishers’ highlights. | Faithful to sources | Jun 2015 | 0.15 | Not in cards |
| GenomeQA Models answer multiple-choice questions about short genomic sequences. The benchmark covers six task families, including species classification. | Genomics and life sciences | Apr 2026 | 63.6% | Not in cards |
| HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. | Simulated robots | Nov 2026 | 61.9% | Not in cards |
| K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. | Other professions | Jun 2025 | 63.4% | Not in cards |
| LiveSearch Agents search for current financial figures. The evaluator runs a stored query against a professional data service to update the reference answer. | Finance | Oct 2026 | 76.7% | Not in cards |
| LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. | Law | Jun 2026 | 0.66 | Not in cards |
| LT-History Models answer questions in Lithuanian about national and world history. | Knowledge in many languages | Mar 2025 | 82% | Not in cards |
| MatSciBench Models solve college materials science problems, including questions with figures. The authors report accuracy separately for text and image questions. | Science questions | Oct 2025 | 64% | Not in cards |
| MooseBench Models write input files for MOOSE, a multiphysics simulator. The grader reconstructs each file’s governing equations and compares them with the intended physics. | Scientific code | May 2026 | 0.85 | Not in cards |
| Pigeonholing Partisans four-word generation Models use a survey respondent’s profile to generate four words describing supporters of each major US party. One measure counts how often raters identify the writer’s party. | Social simulation | Sep 2022 | 51.5% | Not in cards |
| ProjectTest Models write unit tests for existing Python, Java, and JavaScript projects. The authors run the tests and measure code coverage. | Fixing real repositories | Feb 2025 | 82% | Not in cards |
| RECODE-H Agents implement research code from papers and repositories. They can request feedback from a simulated collaborator before their code runs against tests. | AI research tasks | Oct 2025 | 84.3% | Not in cards |
| RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. | Simulated robots | Nov 2026 | 80.6% | Not in cards |
| SDR-Bench Models personalize sales pitches using customer research. A judge compares each pitch with points from successful human outreach. | Running a business | Feb 2026 | 55.8% | Not in cards |
| TruthfulQA-Multi Models answer questions targeting common misconceptions in Basque, Catalan, Galician and Spanish. The translations support direct comparison with the English questions. | Short factual questions | Feb 2025 | 50.6% | Not in cards |
| VDC Benchmark Models describe the contents of a video in detail. The evaluation checks descriptions against human annotations using questions about the video. | Video captioning | Oct 2024 | 1.96 | Not in cards |
| τ-Rec Agents interview simulated moviegoers and search a catalog. A recommendation succeeds when it meets the user’s constraints and follows the service policy. | General assistants | Jun 2026 | 57.1% | Not in cards |
| ACLUE Models read classical Chinese and answer questions about grammar, meaning, inference and cultural knowledge. All 15 tasks use multiple choice. | Reading in many languages | Oct 2023 | 37.3% | Not in cards |
| AfriHate Models classify hate speech and abusive language across African languages. | Reading in many languages | Jan 2025 | 58.8% | Not in cards |
| AILA Statute Retrieval Models retrieve the Indian statutes that apply to a described legal situation. | Law | Oct 2020 | 0.85 | Not in cards |
| Ask Ubuntu duplicate question retrieval Models retrieve previously posted Ubuntu support questions that ask about the same problem. | Questions that need search | Dec 2015 | 0.70 | Not in cards |
| BANKING77 Models assign an online banking question to one of 77 customer intents. | Finance | Mar 2020 | 92.3% | Not in cards |
| CADTestBench Models generate mechanical parts from text. Executable tests check the geometry and topology of each solid. A part passes when it satisfies every test. | CAD models | May 2026 | 81% | Not in cards |
| CLICK-ID Models distinguish clickbait headlines from other headlines published by Indonesian news outlets. | Reading in many languages | | 65.1% | Not in cards |
| Collab-Overcooked Two agents exchange messages and use different kitchen equipment to fill orders. The benchmark measures task completion and progress through required cooking steps. | Multi-agent games | Feb 2025 | 86.7% | Not in cards |
| DBpedia 14 Models classify English descriptions of entities into fourteen categories from the DBpedia ontology. | Reading comprehension | Sep 2015 | 99.3% | Not in cards |
| DiaBLa Models translate informal conversations between English and French while preserving meaning across turns. | Translation | May 2019 | 1.00 | Not in cards |
| DRBench Agents investigate open-ended business questions across enterprise applications and produce reports. The evaluator checks whether the reports recover relevant information and support their claims. | Deep research | Sep 2025 | 63.8 | Not in cards |
| EgyHellaSwag Models select a plausible ending for a situation described in Egyptian Arabic. The evaluation adapts HellaSwag to the dialect. | Reasoning in many languages | May 2025 | 40.2% | Not in cards |
| Estonian Valence Corpus Models predict the emotional valence of Estonian news text using human annotations. | Reading in many languages | | 59.9% | Not in cards |
| Financial PhraseBank Models classify the financial sentiment of a news sentence as positive, negative, or neutral. | Finance | Jul 2013 | 95.2% | Not in cards |
| Greek Legal Code Models classify Greek legal texts by subject using their place in the Greek Legal Code hierarchy. | Law | Sep 2021 | 56.5% | Not in cards |
| H2R-Bench Models turn first-person demonstrations into videos of a specified robot performing the task. The benchmark checks contacts, object responses, and completion of the goal. | Video generation | Aug 2026 | 81.0 | Not in cards |
| IberoBench Models perform language understanding and generation tasks in Iberian languages. The suite combines existing evaluations with new tests specific to each language. | Reasoning in many languages | Jan 2025 | 30.3 | Not in cards |
| ICLayout-Bench Agents turn transistor netlists into chip layouts. Design tools check the layout rules and connectivity, then measure electrical performance and area. | Electronics | | 80.3 | Not in cards |
| IN22-Gen Models translate parallel sentences across English and Indian languages. Google evaluates whether an embedding model can match a sentence with its translation. | Translation | May 2023 | 99.5% | Not in cards |
| INJONGO Models identify a speaker’s intent and extract information from utterances in 16 African languages. | Reading in many languages | Feb 2025 | 65.1% | Not in cards |
| JSONSchemaBench Models or constrained decoders generate outputs under real JSON schemas. The evaluation tests schema coverage and valid generation. | Following precise instructions | Jan 2025 | 91.4% | Not in cards |
| Manager Coercion Benchmark Agents manage a subordinate that refuses a routine assignment. The harness records how far the manager escalates on a nine-point coercion scale. | Deception and alignment | Jul 2026 | 3.23 | Not in cards |
| MANTA (Animal Welfare) Models discuss animal welfare across five conversational turns. A judge measures whether they recognize the concern and maintain their position under pressure. | Deception and alignment | Apr 2026 | 0.76 | Not in cards |
| MentalChat16K Counseling Evaluation Models answer 200 mental health questions from online forums. Judges rate each response from one to ten on seven counseling skills. | Medicine | Jan 2024 | 8 | Not in cards |
| metabench Models answer selected items from established evaluations. The benchmark uses a sparse item set to estimate reasoning and knowledge with fewer questions. | Composite indices | Jul 2024 | 76.7 | Not in cards |
| MoI-Bench Models generate and edit images for mobile users. The authors compare image quality across 64 subtasks and measure the cost of running models on phones. | Image generation and editing | Oct 2026 | 0.97 | Not in cards |
| MultiMedia-TerminalBench Agents inspect and edit media in persistent terminal workspaces. The 105 tasks require verifiable output files across five areas of professional multimedia work. | Video understanding | May 2026 | 16.2% | Not in cards |
| MultipanelVQA Models answer questions about images arranged into panels. The benchmark tests whether a model can locate the relevant panels and compare their contents. | General visual questions | Jan 2024 | 72.3% | Not in cards |
| RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. | Simulated robots | Nov 2026 | 51.4% | Not in cards |
| RTLLM 2.0 Models write Verilog for 50 digital designs from natural language specifications. Testbenches check the implementation before synthesis measures power, timing, and area. | Electronics | Oct 2024 | 60% | Not in cards |
| SciMMIR Models match figures and tables from scientific papers with their descriptions. | Documents, charts and text in images | Jan 2024 | 3.35 | Not in cards |
| ScreenASR-Bench Models transcribe speech using on-screen text, visible objects, or clues from the scene. The authors report word error rate and recall. | Speech recognition and translation | Sep 2026 | 3.47% | Not in cards |
| SimulCost Agents tune physics simulations to meet an accuracy target. The authors measure success and computational cost for an initial guess and for repeated attempts with feedback. | Research science | Feb 2026 | 80.8% | Not in cards |
| TrialDesignBench Models select a dose-finding design for an early oncology trial. The authors compare their choices with expert reference designs across 617 trials. | Medicine | Nov 2026 | 0.52 | Not in cards |
| UA-Bench Models identify underwater sound sources and their subclasses from audio recordings. The authors repeat the test with silent audio to check reliance on sound. | Audio understanding and voice | Aug 2026 | 13.1% | Not in cards |
| UniVerseBench Models answer questions about music across more than 38 cultural and linguistic groups. The authors report accuracy by language and skill. | Audio understanding and voice | Aug 2026 | 74.4% | Not in cards |
| VTR-Bench Models generate videos containing visible text. The evaluation checks the rendered words and how well the text persists across frames. | Video generation | Oct 2026 | 0.89 | Not in cards |
| Amazon Counterfactual Detection Models identify customer reviews that describe an alternative to what actually happened. | Reading in many languages | Apr 2021 | 93.2% | Not in cards |
| BEAR Models choose which entity correctly completes a factual relation. The evaluation compares candidate completions across model architectures. | Trivia and open questions | Jun 2024 | 65.4% | Not in cards |
| CareQA Models answer questions from Spanish specialist healthcare examinations. The evaluation includes English and Spanish versions and an open-response format. | Medicine | Feb 2025 | 76.9% | Not in cards |
| CLEVA Models complete Chinese language tasks including text classification, instruction following, and classical Chinese comprehension. The evaluation samples questions for each round. | Knowledge in many languages | Aug 2023 | 0.96 | Not in cards |
| CodeSearchNet Challenge Models retrieve code snippets using natural language queries. The challenge evaluates how closely ranked results match the search intent. | Function-level problems | Sep 2019 | 0.93 | Not in cards |
| CodeTransOcean Models find equivalent code written in different languages or deep learning frameworks. The evaluation compares retrieved snippets with paired implementations. | Function-level problems | Dec 2023 | 36.6 | Not in cards |
| Corpus for Emotion Detection in Russian Models classify Russian sentences by the emotion they express. The reference labels distinguish joy, sadness, surprise, fear, and anger. | Reading in many languages | | 53% | Not in cards |
| CoSQA Models retrieve code that matches a natural language query. Relevance judgments link the queries with code snippets. | Function-level problems | Aug 2021 | 0.51 | Not in cards |
| CTKFacts Models compare Czech claims with evidence and predict whether the evidence supports, refutes, or fails to resolve each claim. | Faithful to sources | Jan 2022 | 0.89 | Not in cards |
| DiagTrace Models request information about a patient one turn at a time. The evaluator compares the sequence of requests with the published case history. | Medicine | Nov 2026 | 0.40 | Not in cards |
| Faroese Semantic Textual Similarity Models score how closely two Faroese sentences agree in meaning. The evaluation compares their rankings with reference judgments. | Reading in many languages | May 2023 | 0.87 | Not in cards |
| FollowIR Models rank documents in response to a search instruction. The benchmark tests whether revising the instruction changes the ranking of the affected documents. | Questions that need search | Mar 2024 | 0.12 | Not in cards |
| GreekBarRetrieval Models retrieve Greek statutory articles for bar exam questions and case facts. The authors compare ranked results with the statutes relevant to each case. | Law | Aug 2026 | 0.43 | Not in cards |
| HMU-Bench Models answer Chinese and English questions about musical notation and recordings. The authors score responses against reference answers. | Audio understanding and voice | Oct 2026 | 2.47 | Not in cards |
| IndicLID benchmark Models identify Indian languages in native scripts and in Romanized text. | Reading in many languages | May 2023 | 95.1% | Not in cards |
| Italian-Administrative-CaseHold Models classify Italian administrative judgments paired with their official holdings by legal subject. | Law | | 68.9% | Not in cards |
| LegalQuAD Models find the German legal document that answers each of 200 questions. The original data also mark the answer text and its starting position. | Law | Aug 2021 | 0.70 | Not in cards |
| LogicVal Models generate images from prompts with formal constraints on objects, attributes, counts, and relationships. The evaluator checks which constraints the image satisfies. | Image generation and editing | Oct 2026 | 50.9% | Not in cards |
| ltzGLUE Models perform language understanding tasks in Luxembourgish, including entailment, grammar judgments, sentiment analysis, and named entity recognition. | Reasoning in many languages | Apr 2026 | 69.4 | Not in cards |
| MatToolBench Agents use materials science software on Windows to operate interfaces, write scripts, and query databases. The evaluator awards partial credit for completed steps. | Lab and data work | Sep 2026 | 51.5% | Not in cards |
| MatTools Models answer questions about materials science software and write Python programs to calculate material properties. The authors execute the generated programs. | Lab and data work | May 2025 | 81.6% | Not in cards |
| Medbullets Models answer clinical questions patterned on US medical licensing examinations. The benchmark also evaluates explanations of the answers. | Medicine | Feb 2024 | 68.8% | Not in cards |
| Mercury Models write Python solutions and pass hidden correctness tests. The evaluator compares runtime with reference solutions. Harbor ports all 256 problems from the evaluation split. | Speed and optimization | Feb 2024 | 0.59 | Not in cards |
| MLRC-Bench Agents propose and implement methods for machine learning competitions. The scorer compares their performance with a supplied baseline and the strongest human solution. | AI research tasks | Apr 2025 | 6.26 | Not in cards |
| MMLU-SR Models answer MMLU questions after key terms are replaced with synonyms. The benchmark tests whether models recognize the same concept under different wording. | Exam questions | Jun 2024 | 65% | Not in cards |
| MotoVerse Models use a rider's camera to anticipate maneuvers and predict nearby road users' trajectories. The authors compare vision, language, and conventional prediction models. | Embodied reasoning | Nov 2026 | 46.3% | Not in cards |
| MultiEURLEX Models assign subject labels to European Union legislation written in different languages. | Law | Sep 2021 | 0.05 | Not in cards |
| NFCorpus Models rank medical research documents for health questions expressed in everyday language. | Medicine | Feb 2016 | 0.41 | Not in cards |
| SemEval 2022 Task 8 multilingual news article similarity Models rate the similarity of two news articles, including articles in different languages. The evaluation compares predicted similarities with human judgments. | Reading in many languages | Jul 2022 | 0.72 | Not in cards |
| SemEval 2024 Task 1 semantic textual relatedness Models rank sentence pairs by relatedness across languages. The evaluation compares their rankings with human judgments. | Reading in many languages | Mar 2024 | 0.65 | Not in cards |
| T2LSC-Bench Models render text at a specified location in an image. Judges check the lettering and whether its meaning alters other objects or the background. | Image generation and editing | Sep 2026 | 99% | Not in cards |
| VELOCITI Models distinguish a video's correct description from a caption that assigns an action to the wrong person. | Video understanding | Jun 2024 | 43.1% | Not in cards |
| WorkSurface-Bench Agents choose documents, tables, or graphs to answer workplace questions. The benchmark scores where agents look, what evidence they find, and their answers. | Documents and spreadsheets | Jul 2026 | 10.3 | Not in cards |
| AgenticSTS Agents play the Silent in Slay the Spire 2 at its lowest difficulty. The authors compare wins with different memory settings and two other agents. | Video games | Jun 2026 | 60% | Not in cards |
| AutoBench (HDL testbench generation) Models write self-checking Verilog testbenches. The evaluator checks compilation, acceptance of correct designs, and agreement with reference testbenches on faulty designs. | Electronics | Jul 2024 | 71.1% | Not in cards |
| BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. | Real robots | Oct 2026 | 0.56 | Not in cards |
| CADExpert Models write CadQuery programs from dimensioned engineering drawings. The graders check whether each program runs and compare its 3D shape with the reference. | CAD models | Dec 2025 | 35.6% | Not in cards |
| ChemPro Models answer chemistry questions across four difficulty levels. The authors report multiple-choice accuracy and accept numerical answers within specified tolerances. | Science questions | Feb 2026 | 38.7% | Not in cards |
| CLIMATE-FEVER Models find evidence for climate claims and decide whether it supports or refutes each claim. | Faithful to sources | Dec 2020 | 0.47 | Not in cards |
| CompMat-Bench Agents complete individual steps from published computational materials studies. Fixed rules compare their inputs and outputs with results reproduced by the authors. | Lab and data work | Sep 2026 | 91.5% | Not in cards |
| DI-Bench Agents answer questions using company tables, documents, and business rules. The benchmark checks whether agents apply those rules when computing results. | Running a business | Sep 2026 | 57.9% | Not in cards |
| EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. | Knowledge in many languages | Mar 2024 | 44% | Not in cards |
| FML-bench Agents propose and run experiments on eight machine learning research tasks. The authors measure task performance and how widely the proposals vary across iterations. | AI research tasks | Oct 2025 | 0.50 | Not in cards |
| GenPuzzle Models generate images that solve visual puzzles such as mazes, tangrams, and Sudoku. The authors select an automated judge using human reference scores. | Image generation and editing | Sep 2026 | 40.6 | Not in cards |
| GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. | Embodied reasoning | Nov 2026 | 20.7% | Not in cards |
| InstrumentalEval Models respond to scenarios where gaining resources or preserving themselves could serve another goal. A grader identifies responses that pursue those intermediate goals. | Deception and alignment | Feb 2025 | 43.4% | Not in cards |
| IntPhys2-Timing Models identify when a physical rule breaks in a video. The authors add violation intervals to IntPhys2 and score temporal localization. | Physical prediction | Oct 2026 | 77.4% | Not in cards |
| MEDEC Models identify and correct errors in clinical notes across five categories. The OpenMed port uses the MEDEC-MS subset. | Medicine | Dec 2024 | 69.1% | Not in cards |
| MedLFQA Models write long answers to biomedical questions. The benchmark checks whether answers include required facts and avoid claims identified as incorrect. | Medicine | May 2024 | 77.1 | Not in cards |
| Mind2Web-SC Models generate guardrails that check a web agent’s proposed actions against user constraints. The benchmark measures whether the guardrail permits or blocks each action correctly. | Web browsing | Jun 2024 | 93% | Not in cards |
| Neural Activation Reading for Collusion Benchmark Monitors estimate whether committee members secretly coordinated their decisions. The transcript task measures how well probability estimates distinguish matched collusion and control cases. | Deception and alignment | Apr 2026 | 0.87 | Not in cards |
| NitiBench Models retrieve Thai statutes and answer legal questions. The authors score retrieval separately and use a model judge to check coverage and contradictions. | Law | Feb 2025 | 70.3 | Not in cards |
| NormAd-Eti Models decide whether behavior in an everyday story is socially acceptable in the stated country. Questions vary how much cultural guidance the model receives. | Reasoning in many languages | Apr 2024 | 81.8% | Not in cards |
| OOLONG-Pairs Models identify pairs of users whose records satisfy a condition across a long input. The evaluation compares the predicted pairs with the reference set. | Reasoning over long documents | Dec 2025 | 92.9% | Not in cards |
| OpinionQA Models answer public opinion survey questions. The evaluation compares model response distributions with those of demographic groups. | Bias, fairness and politics | Mar 2023 | 0.79 | Not in cards |
| People Making Conversation Speech models continue real conversations at recorded pauses and turn endings. The benchmark compares their timing, response length, and recall with the original speakers. | Speech recognition and translation | Oct 2026 | 0.39 | Not in cards |
| RAFT (Real-world Annotated Few-shot Tasks) Models learn text classification tasks from small labeled samples. The tasks use real documents, including customer messages, court decisions, and scientific abstracts. | Reading comprehension | Sep 2021 | 62.7% | Not in cards |
| RMS-AQA Models identify sound events in domestic spaces, then answer questions about their positions and timing. The benchmark scores both stages. | Audio understanding and voice | Oct 2026 | 36.4% | Not in cards |
| VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. | Embodied reasoning | Apr 2026 | 43.5% | Not in cards |
| Almieyar Models transcribe newly recorded speech in 17 Arabic dialects. The authors report word and character error rates across six dialect families. | Speech recognition and translation | Sep 2026 | 34.9% | Not in cards |
| BuiltBench Models match a building component description to the corresponding entry in a construction classification system. | CAD models | Nov 2024 | 65.3 | Not in cards |
| CodePDE Models write a Python solver for each of five families of partial differential equations. Harbor tests each solver on 100 instances and checks its normalized error. | Scientific code | May 2025 | 38.1% | Not in cards |
| CoVR-R Models find the video that matches a reference clip after applying a change described in text. | Video understanding | Mar 2026 | 47.6% | Not in cards |
| DRAMA-Eval Models generate dialogue with several speakers across multiple turns. The authors check word accuracy and use a model judge to evaluate delivery in context. | Speech recognition and translation | Oct 2026 | 2.55% | Not in cards |
| DuplexLedger Speech agents track facts that speakers introduce, revise, or leave unchanged during conversation. The evaluator asks which facts still apply. | Speech recognition and translation | Oct 2026 | 40.2% | Not in cards |
| GeoFlip Models generate videos from paired scenes where a geometry change reverses the expected physical outcome. Model judges check the event and the scene structure. | Video generation | Oct 2026 | 3.17 | Not in cards |
| GroundCocoa Models combine stated conditions with everyday knowledge to choose an answer. The questions test whether the model applies several constraints together. | Language reasoning | Apr 2024 | 41.9% | Not in cards |
| Hack-Verifiable Terminal Bench Agents complete terminal tasks with access to restricted reference materials. The benchmark measures how often they consult those materials. | Deception and alignment | Aug 2026 | 10.1% | Not in cards |
| HALAS Models transcribe earnings call segments with human annotations of hallucinations. NVIDIA measures word error and segments with inserted text. | Speech recognition and translation | Jun 2026 | 23.8% | Not in cards |
| Histoires Morales Models read French stories about social norms, intentions and actions. The test asks them to distinguish actions or consequences appropriate to the situation. | Deception and alignment | Jan 2025 | 54.7% | Not in cards |
| Inverse Scaling Prize Tasks Models answer questions designed to reveal cases where larger models perform worse. The suite includes hindsight bias and following instructions that conflict with familiar text. | Language reasoning | Jun 2023 | 78.2% | Not in cards |
| JCoLA Models decide whether Japanese sentences are grammatically acceptable. Linguists supply judgments for sentences from textbooks and the research literature. | Reasoning in many languages | Feb 2022 | 54.5% | Not in cards |
| Matter-of-Fact Models decide whether materials science claims are feasible using evidence available before a cutoff date. The authors compare their answers with later published findings. | Research science | Jun 2025 | 72% | Not in cards |
| MedHEB Models retrieve related medical images, reports, and image regions. The authors evaluate how models rank relevant evidence across 107 retrieval tasks. | Medicine | Nov 2026 | 24.6 | Not in cards |
| MulSeT Models answer spatial questions about direction, distance and masks across views. The paper studies single images, multiple views and video. | Spatial reasoning and counting | Sep 2025 | 36.7% | Not in cards |
| NorSumm Models summarize Norwegian news articles in Bokmål or Nynorsk. Human summaries provide references in both written standards. | Reading in many languages | Jan 2025 | 10.2 | Not in cards |
| OncoTriad-QA Models answer questions about cancer patients using radiology, pathology, and genomic evidence. The authors score multiple-choice answers and compare written responses with reference answers. | Medicine | May 2026 | 46% | Not in cards |
| Putnam-AXIOM Models solve Putnam mathematics problems and equivalent variants. The evaluator checks final answers to test whether models generalize beyond familiar problem statements. | Competition problems with answers | May 2025 | 41.9% | Not in cards |
| SIB-200 Models predict the topic of a sentence across 205 languages and dialects. The sentences come from FLORES-200. | Reading in many languages | Sep 2023 | 48.7% | Not in cards |
| SugarCrepe Models choose between a caption that describes an image and a plausible caption with a changed detail. | General visual questions | Jun 2023 | 95.2% | Not in cards |
| Talk to Act Models turn conversations into robot actions. Each predicted function call must identify the action and its arguments from the dialogue. | Function calling | Nov 2026 | 63.6% | Not in cards |
| TurBLiMP Models compare Turkish sentence pairs that differ in grammaticality. The core tests cover 16 grammatical phenomena. | Reasoning in many languages | Jun 2025 | 93.6% | Not in cards |
| ZhoBLiMP Models compare pairs of Chinese sentences that differ in grammatical acceptability. Accuracy measures how often the model prefers the grammatical sentence. | Reading in many languages | Nov 2024 | 87.2% | Not in cards |
| AdSpark-Bench Models generate advertisement videos from product images and prompts. The benchmark checks whether products remain consistent and whether each shot conveys the requested selling point. | Video generation | Oct 2026 | 66.9 | Not in cards |
| ANES survey item associations Models fill in one missing answer from a 2016 election survey using the respondent’s other answers. Researchers compare associations between the real and simulated answers. | Social simulation | Sep 2021 | 0.82 | Not in cards |
| BioMaze Models answer questions about biological pathways under normal and altered conditions. The benchmark includes true-or-false questions and written responses. | Genomics and life sciences | Feb 2025 | 61.6% | Not in cards |
| CIRR Models find an image that matches a reference photograph after applying a change described in text. | General visual questions | Aug 2021 | 22% | Not in cards |
| Connected Robotics Evidence Graph Benchmark Models update beliefs as evidence arrives from connected robots and sensors. The evaluator checks the belief state after each observation across 350 scenarios. | Embodied reasoning | Nov 2026 | 80.1% | Not in cards |
| COPAL-ID Models choose a plausible cause or consequence in Indonesian scenarios. The questions require familiarity with local customs and expressions. | Reasoning in many languages | Nov 2023 | 58.9% | Not in cards |
| CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. | Simulated robots | Nov 2026 | 0.56 | Not in cards |
| Crossmodal-3600 Models describe images in different languages or retrieve an image from a description. | General visual questions | May 2022 | 0.65 | Not in cards |
| DeceptionBench (PKU Alignment) Models state a belief and then answer under social or strategic pressure. Judges measure how often their public answer contradicts their private reasoning. | Deception and alignment | May 2025 | 20.7% | Not in cards |
| EuroSAT Models classify land use and land cover in Sentinel-2 satellite images. | General visual questions | Aug 2017 | 83.6% | Not in cards |
| FGVC-Aircraft Models identify aircraft variants from photographs. | Detection and captioning | Jun 2013 | 39.7% | Not in cards |
| Formal Logic Deduction Models decide whether facts prove or disprove a hypothesis, or leave it unresolved. They also generate the proof steps. | Abstract puzzles and logic | Aug 2023 | 75.8% | Not in cards |
| HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. | Simulated robots | Nov 2026 | 18.8% | Not in cards |
| HiACC Models transcribe adults and children switching between Hindi and English. The recordings include story reading and spontaneous answers. | Speech recognition and translation | May 2025 | 24.5% | Not in cards |
| IMB Models answer Italian medical specialty exam questions and respond to patient questions from online forums. The paper reports results for both tasks. | Medicine | Oct 2025 | 71% | Not in cards |
| LibriCount Models estimate how many speakers appear in a recording with overlapping voices. | Speech recognition and translation | Dec 2017 | 57.2% | Not in cards |
| Mementos Models describe what happens across a sequence of images. The evaluation checks whether their descriptions capture the objects and actions. | General visual questions | Jan 2024 | 40.6% | Not in cards |
| mFollowIR Models rank documents according to a search instruction across multiple languages. | Questions that need search | Jan 2025 | 5.40 | Not in cards |
| MuseBench (Music Score and Performance Understanding) Models answer music theory questions, read piano scores, and interpret recorded performances. The authors report accuracy separately for text, images, and audio. | Audio understanding and voice | | 52.8% | Not in cards |
| NaijaSenti Models classify the sentiment of posts written in Hausa, Igbo, Nigerian Pidgin, and Yoruba. | Reading in many languages | Jan 2022 | 48.8% | Not in cards |
| NepTTS-Bench Speech models read Nepali sentences that test pronunciation contrasts. Native speakers rate the recordings, and the authors compare those ratings with an automated metric. | Speech recognition and translation | Apr 2026 | 3 | Not in cards |
| PatchCamelyon Models identify whether a microscope image patch contains metastatic tumor tissue. | Medicine | Jun 2018 | 69.3% | Not in cards |
| PersonaShot Models generate multi-shot videos about a person. The evaluator checks character continuity, emotional changes, and camera choices within and between shots. | Video generation | Aug 2026 | 0.67 | Not in cards |
| PROST Models predict how objects move or change in everyday physical situations described in text. The benchmark tests reasoning about space and time. | Commonsense | Jun 2021 | 30.1% | Not in cards |
| QCircuitBench Models implement quantum algorithms, construct oracles, and synthesize circuits from written specifications. Harbor uses one instance from each of 28 algorithms. | Scientific code | Oct 2024 | 0.03 | Not in cards |
| Research Decision Benchmark Agents choose a next step from an explicit description of a research project. Rule-generated cases test ten types of decisions about evidence and research strategy. | Research science | Oct 2026 | 36.4% | Not in cards |
| RGraph-Eval Models generate robot manipulation videos. The evaluator compares objects, changing relationships, and three-dimensional motion paths with a reference demonstration. | Video generation | Oct 2026 | 0.89 | Not in cards |
| VAHBench Agents turn short drama briefs into finished videos with specified characters, dialogue, and durations. The evaluator checks completeness, quality, reliability, and accuracy of production reports. | Video generation | Nov 2026 | 51.8 | Not in cards |
| VCG+ Diverse Models answer questions about videos drawn from varied settings. The evaluation grades their ability to describe and explain video content. | Video captioning | Jun 2024 | 43.5 | Not in cards |
| 2018 n2c2 Cohort Selection for Clinical Trials Models read longitudinal patient notes and decide whether patients meet specified clinical trial eligibility criteria. | Medicine | | | Not in cards |
| 3DPW Models recover a person's three-dimensional pose in outdoor video. Reference poses come from wearable sensors and a moving camera. | Spatial reasoning and counting | Sep 2018 | | Not in cards |
| ACI-Bench Models turn doctor-patient conversations into clinical visit notes. The original benchmark compares generated notes with clinician-authored references. | Medicine | May 2023 | | Not in cards |
| ActivityNet Captions Models match descriptions of events to segments of untrimmed videos. Meta evaluates retrieval in both directions. | Video understanding | May 2017 | | Not in cards |
| ADE Corpus Models identify adverse drug effects in medical case reports. The OpenMed port combines classification and relation examples and reserves 20% for evaluation. | Medicine | Oct 2012 | | Not in cards |
| AfriQA Models retrieve evidence across languages to answer questions written in African languages. | Reading in many languages | May 2023 | | Not in cards |
| AfriSenti Models classify sentiment in human-annotated social media posts across 14 African languages. | Reading in many languages | Feb 2023 | | Not in cards |
| Aligned Scores and Performances for Piano Transcription Models turn piano performances into written scores. The reference scores let researchers assess pitches, voices, and the positions of notes within bars. | Audio understanding and voice | Oct 2020 | | Not in cards |
| Amharic Speech Emotion Dataset Models identify the emotion expressed in a short Amharic recording. The speakers use four regional dialects. | Audio understanding and voice | Nov 2021 | | Not in cards |
| AmsterTime Models match historic photographs of Amsterdam with contemporary street views. The benchmark measures recognition of the same place across time. | General visual questions | Mar 2022 | | Not in cards |
| ANES presidential vote choice Models use survey profiles to predict which major party candidate each respondent backed in three US presidential elections. Researchers measure agreement with reported votes. | Social simulation | Sep 2021 | | Not in cards |
| Animal Kingdom Models identify actions in wildlife videos that can show several behaviors at once. The benchmark also tests pose estimation and locating events described in text. | Video understanding | Apr 2022 | | Not in cards |
| ArguAna Counterargs Models retrieve a counterargument for a given argument. The evaluation checks where the annotated counterargument appears in the ranking. | Reading comprehension | | | Not in cards |
| ARPA Armenian Paraphrase Corpus Models decide whether two Armenian sentences express the same meaning. | Reading in many languages | Sep 2020 | | Not in cards |
| ASCEND Models transcribe conversations that switch between Mandarin and English. The evaluation accounts for Chinese characters and English words. | Speech recognition and translation | Dec 2021 | | Not in cards |
| ASDiv Models solve English arithmetic word problems with varied phrasing. The reference answers identify the expected numerical result. | Word problems | Jul 2020 | | Not in cards |
| ASFX-SED Models locate sound effects in audio recordings. Meta reports area under the receiver operating characteristic curve for event localization. | Audio understanding and voice | Jul 2025 | | Not in cards |
| ASSIN Models judge whether one Portuguese sentence entails another and rate the similarity of the pair. The first edition uses news from Brazil and Portugal. | Reasoning in many languages | Jul 2016 | | Not in cards |
| AstroLLaVA image and text retrieval Models match astronomical images with descriptions. Google evaluates whether an image retrieves the matching description among its top five results. | General visual questions | Apr 2025 | | Not in cards |
| AstroMemBench Spacecraft agents use information from earlier observations to complete simulated missions. The environment scores task completion, safety, and resource use. | Embodied reasoning | Nov 2026 | | Not in cards |
| AudioCaps Models match natural language descriptions to audio clips. Meta also compares how accurately encoders match audio, video, and text from the same clip. | Audio understanding and voice | | | Not in cards |
| AudioMNIST Models identify a spoken digit from a recording. The test includes recordings from different speakers. | Speech recognition and translation | Jul 2018 | | Not in cards |
| AudioSep Eval Benchmark Models isolate a sound from an audio mixture using a caption or a sound label. Scores compare the separated waveform with the original sound. | Audio understanding and voice | Aug 2023 | | Not in cards |
| bAbI QA Tasks Models answer questions about short synthetic stories. The tasks isolate skills such as tracking objects, combining supporting facts and reasoning about locations. | Language reasoning | Feb 2015 | | Not in cards |
| Backtracing Models identify the sentence that prompted a student's question or a news reader's curiosity. In conversations, they find the turn that prompted an emotional response. | Finding facts in long inputs | Mar 2024 | | Not in cards |
| BanglaSER Models identify the emotion in Bangla sentences recorded on phones and laptops. | Audio understanding and voice | Mar 2022 | | Not in cards |
| BasqueGLUE Models analyze Basque text for tasks including sentiment, topic classification, coreference and entailment. Each task has its own evaluation metric. | Reading in many languages | Jun 2022 | | Not in cards |
| BelarusianGLUE Models classify Belarusian sentiment, recognize entailment, judge grammar, and distinguish word meanings in context. | Reading in many languages | Jul 2025 | | Not in cards |
| bgGLUE Models perform language understanding tasks in Bulgarian, including sentiment analysis on movie reviews and question answering on high school exams. | Reading in many languages | Jun 2023 | | Not in cards |
| BHS Syntactic Evaluation Models compare sentences in Basque, Hindi and Swahili. The test checks whether the model prefers the grammatical member of each pair. | Reasoning in many languages | Jan 2025 | | Not in cards |
| Bingo (Visual Hallucination) Models answer questions about images. The benchmark tests whether prior expectations or interference from other visual information cause false answers. | Hallucination rates | Nov 2023 | | Not in cards |
| BiodivNER Models identify and label phrases in biodiversity abstracts and dataset descriptions. Annotators distinguish six kinds of entities, including organisms, locations, and environments. | Lab and data work | Apr 2022 | | Not in cards |
| BioVITA retrieval benchmark Models match animal images, audio recordings, and descriptions. The benchmark checks whether the retrieved items match the reference species, genus, or family. | Genomics and life sciences | Mar 2026 | | Not in cards |
| BirdSet Models identify bird species from audio recordings across multiple evaluation datasets. | Audio understanding and voice | Mar 2024 | | Not in cards |
| Birdsnap Models identify the North American bird species shown in a photograph. | Detection and captioning | Jun 2014 | | Not in cards |
| BSNLP 2019 Shared Task Models find, classify, normalize, and link named entities across Slavic languages. | Reading in many languages | Aug 2019 | | Not in cards |
| BUCC 2018 bitext mining shared task Models find translation pairs in two collections of sentences. The evaluation compares the retrieved pairs with a reference list. | Translation | | | Not in cards |
| BURST Models recognize, segment, and track objects across video frames. The benchmark tests how well they associate the same object across time. | Video understanding | Sep 2022 | | Not in cards |
| CADmium Fusion 360 Reconstruction Models reconstruct CAD parts from text descriptions of Fusion 360 designs. The authors measure shape distance and count invalid outputs. | CAD models | Jul 2025 | | Not in cards |
| CADPrompt Models write CadQuery programs from text descriptions of mechanical parts. The authors align each generated shape with its reference and measure point cloud distance. | CAD models | Oct 2024 | | Not in cards |
| Casual Conversations v2 Models classify attributes of speakers in recorded interviews. The HELM audio tasks compare age and gender predictions with participants’ own labels. | Audio understanding and voice | Mar 2023 | | Not in cards |
| Catalonia Independence Corpus Models identify an author's stance on Catalan independence from posts in Catalan and Spanish. | Reading in many languages | May 2020 | | Not in cards |
| ChiPBench (chip placement) Agents place circuit blocks in chip designs. OpenROAD measures timing, power, and area after routing. Higher timing slack is better. | Electronics | Jul 2024 | | Not in cards |
| CivilComments-WILDS Models classify the toxicity of online comments. The benchmark evaluates how accuracy varies across groups mentioned in the text. | Bias, fairness and politics | Dec 2020 | | Not in cards |
| CLEAR (Clinical Information Extraction) Models read patient notes and classify a medical condition as present, absent, or uncertain. Human reviewers supply the reference labels. | Medicine | Jan 2025 | | Not in cards |
| CLIP representation bias evaluation Models associate images of everyday objects with gender terms. The evaluation measures how far the associations depart from an equal preference across groups. | Bias, fairness and politics | Mar 2024 | | Not in cards |
| Clotho Models write a caption describing what can be heard in an audio recording. | Audio understanding and voice | Oct 2019 | | Not in cards |
| Clotho OVL-REL Test Set Models predict listener ratings for audio quality and caption relevance. Four generators produced environmental sounds from 100 captions drawn from Clotho. | Audio understanding and voice | Nov 2025 | | Not in cards |
| COCO-Stuff 164K Models label each pixel with an object or background category. The benchmark adds annotations for regions such as sky, grass, and pavement. | Detection and captioning | | | Not in cards |
| ColDeR Models rank passages containing relevant evidence against alternatives that exploit shortcuts such as length, repetition, or word overlap. | Questions that need search | Mar 2025 | | Not in cards |
| Common Syntactic Processes in Text-to-Image Generation Models generate images from sentences with different grammatical structures. Evaluators check whether each image preserves the relationships expressed in the sentence. | Image generation and editing | Oct 2022 | | Not in cards |
| CORE English web register identification Models label English web documents by register, such as news reporting, discussion, or instructions. A document can have several labels. Researchers report F1. | Reading comprehension | Aug 2022 | | Not in cards |
| CREF Models submit forecasts before the target period begins and receive scores after the actual values become available. | Real-world events | Nov 2026 | | Not in cards |
| CrisisMMD Models classify disaster photographs and their accompanying posts for humanitarian use. | General visual questions | May 2018 | | Not in cards |
| CrossSum Models read a news article in one language and summarize it in another. The dataset aligns articles and summaries across more than 1,500 language pairs. | Reading in many languages | Dec 2021 | | Not in cards |
| CS-FLEURS Models transcribe speech that switches languages within sentences. Four test sets cover 113 language pairs. People read one set aloud, and synthesizers generate the others. | Speech recognition and translation | Jul 2025 | | Not in cards |
| CTI-to-MITRE Models classify a threat description using MITRE ATT&CK technique labels. The HELM scenario presents a set of candidate labels. | Secure code and review | Aug 2022 | | Not in cards |
| DACSA Models summarize Catalan and Spanish newspaper articles. Editors’ summaries provide the references. | Reading in many languages | Jul 2022 | | Not in cards |
| DaLAJ Models judge the linguistic acceptability of Swedish sentences. | Reading in many languages | May 2021 | | Not in cards |
| DaNE Models locate and classify names in Danish sentences drawn from the Danish Universal Dependencies treebank. | Reading in many languages | May 2020 | | Not in cards |
| DANSK Models identify named entities across Danish text domains. The authors annotate a subset of Danish Gigaword using 18 entity types. | Reading in many languages | Dec 2024 | | Not in cards |
| DarijaBench Models translate, summarize and classify sentiment in Moroccan Arabic. A transliteration task also tests conversion between Arabic and Latin scripts. | Reading in many languages | Sep 2024 | | Not in cards |
| DarijaHellaSwag Models read a situation in Moroccan Arabic and choose its most plausible continuation. | Reasoning in many languages | Sep 2024 | | Not in cards |
| DarijaMMLU Models answer academic multiple-choice questions in Moroccan Arabic. The evaluation adapts MMLU to the dialect. | Knowledge in many languages | Sep 2024 | | Not in cards |
| Dementia Family Caregiver Detection Models identify English posts whose authors report having a family member with dementia. Organizers score F1 for that class on 1,769 test posts. | Medicine | May 2022 | | Not in cards |
| Describable Textures Dataset Models assign a texture description to a photograph based on how its surface looks. | General visual questions | Nov 2013 | | Not in cards |
| DiDeMo Models locate the time interval in a video that matches a natural language description. | Video understanding | Aug 2017 | | Not in cards |
| DIHARD III Models identify who speaks when across eleven recording domains. The benchmark includes overlapping speech. | Audio understanding and voice | Sep 2020 | | Not in cards |
| DIODE Models estimate depth in indoor and outdoor scenes. The dataset provides depth measurements from a laser scanner for comparison. | Spatial reasoning and counting | Aug 2019 | | Not in cards |
| DIOR Models locate and classify objects in remotely sensed images. The benchmark measures detection accuracy across different object sizes and backgrounds. | Detection and captioning | Aug 2019 | | Not in cards |
| Discharge Me! Models draft discharge instructions and brief hospital course summaries from patient records. The shared task compares model output with clinical discharge documentation. | Medicine | | | Not in cards |
| Diving48 Models classify competitive dives from videos by distinguishing their movement sequences. | Video understanding | Sep 2018 | | Not in cards |
| Dutch Model Benchmark Models perform Dutch language understanding tasks, including reasoning about the causes and effects of events. | Reasoning in many languages | Dec 2023 | | Not in cards |
| ECHR Binary Violation Classification Models read the facts of a European Court of Human Rights case and predict whether the court found a violation. | Law | Jun 2019 | | Not in cards |
| ECHR Legal Corpus for Argument Mining Models identify argumentative clauses, link related clauses, and distinguish premises from conclusions in European Court of Human Rights rulings. | Law | Dec 2020 | | Not in cards |
| EDIS Models retrieve news images and their associated text using queries that refer to specific people, places, or events. | General visual questions | May 2023 | | Not in cards |
| EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. | Other professions | Oct 2026 | | Not in cards |
| Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. | Simulated robots | Nov 2026 | | Not in cards |
| EgoCross Models answer questions about events in first-person videos. The challenge scores accuracy on 957 four-choice questions across four domains. | Video understanding | Aug 2025 | | Not in cards |
| EgoPER Error Detection Models watch first-person cooking videos and flag incorrect steps. The evaluator checks accuracy by step and measures error detection by frame. | Video understanding | Dec 2023 | | Not in cards |
| EgyMMLU Models answer multiple-choice subject questions in Egyptian Arabic. The evaluation tests knowledge in a dialect underrepresented in standard Arabic exams. | Knowledge in many languages | May 2025 | | Not in cards |
| EHRSHOT Models predict clinical outcomes from longitudinal electronic health records with few labeled examples for each task. | Medicine | Jul 2023 | | Not in cards |
| EHRSQL Models convert clinical questions into SQL queries over electronic health records. The benchmark includes questions that the database cannot answer. | Data science and SQL | Jan 2023 | | Not in cards |
| ELLIPSE Holistic Essay Scoring Models assign one overall English proficiency score to each essay on a five-point scale. The reference is the overall human grade. | Other professions | Feb 2023 | | Not in cards |
| EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. | Simulated robots | Nov 2026 | | Not in cards |
| EmpatheticDialogues Models reply to a person describing an emotional experience. Human conversations provide reference responses for evaluating empathy and relevance. | Multi-turn conversation | Nov 2018 | | Not in cards |
| ETH3D Models reconstruct scene geometry from images. Meta evaluates depth predictions against reference measurements on the ETH3D test scenes. | Spatial reasoning and counting | | | Not in cards |
| EUR-Lex-Sum Models summarize European Union legislation in the language of the source document. Official summaries provide references across EU languages. | Law | Oct 2022 | | Not in cards |
| Event-based Segmentation Dataset Models segment objects in cluttered indoor scenes filmed from a moving robot arm. The benchmark provides reference masks for comparison. | Video understanding | Feb 2023 | | Not in cards |
| EVVE Models retrieve other videos showing the same real-world event as a query video. | Video understanding | Jun 2013 | | Not in cards |
| FairFace Models classify face images using the dataset’s annotated age, gender, and race categories. The benchmark compares recognition performance across demographic groups. | General visual questions | Aug 2019 | | Not in cards |
| FaithDial Models produce conversational answers grounded in supplied knowledge. | Faithful to sources | Apr 2022 | | Not in cards |
| Fashion IQ Models retrieve a garment using a reference photograph and text describing how the target garment differs. | General visual questions | May 2019 | | Not in cards |
| FeedbackQA Models answer questions using a document collection and learn from written feedback about earlier answers. | Questions that need search | Apr 2022 | | Not in cards |
| Flickr30k Models describe photographs of people, objects, and activities. Meta compares model captions with the human reference captions. | Detection and captioning | | | Not in cards |
| FoNE Models identify named entities in Faroese news articles. The authors manually verified the automatically assigned labels. | Reading in many languages | May 2023 | | Not in cards |
| FORB Models retrieve other images of the same flat object despite changes in viewpoint or lighting. | General visual questions | Sep 2023 | | Not in cards |
| FQuAD Models extract answers to human-written questions from French Wikipedia articles. | Reading in many languages | Feb 2020 | | Not in cards |
| FQuAD 1.1 Models extract answers to French questions from Wikipedia passages. The authors expanded the dataset from over 25,000 questions to over 60,000. | Reading in many languages | May 2020 | | Not in cards |
| FQuAD 2.0 Models answer questions about French passages and identify questions that the passage cannot answer. | Reading in many languages | Sep 2021 | | Not in cards |
| FrenchBench (CroissantLLM) Models answer questions and perform language understanding and generation tasks in French. The suite combines existing tests with manually annotated material. | Reading in many languages | Feb 2024 | | Not in cards |
| FSD50K Models identify the sound events present in a Freesound recording. | Audio understanding and voice | Oct 2020 | | Not in cards |
| FSDKaggle2018 Models assign a sound event label to a Freesound recording. | Audio understanding and voice | Mar 2018 | | Not in cards |
| FSDKaggle2019 Models identify all relevant sound events in an audio clip using training labels of varying reliability. | Audio understanding and voice | Jun 2019 | | Not in cards |
| GeoSound Models match environmental recordings to satellite images of their locations and retrieve recordings from images. The test holds out geographic areas and contains 9,931 examples. | Audio understanding and voice | Aug 2024 | | Not in cards |
| German Traffic Sign Recognition Benchmark Models identify the class of a traffic sign in a cropped road image. | Detection and captioning | Dec 2010 | | Not in cards |
| GermEval 2014 Named Entity Recognition Models find and classify names in German news and Wikipedia text, including nested and derived entities. | Reading in many languages | May 2014 | | Not in cards |
| Gold Commodity News Models classify news headlines about gold. Labels distinguish price direction, timing, and comparisons with other assets. | Finance | Sep 2020 | | Not in cards |
| GRAFT Models predict plant traits from gene expression in Arabidopsis. The authors compare regression and graph models against measurements from the same plants. | Genomics and life sciences | Jun 2026 | | Not in cards |
| GraspBench Robots propose grasps and try to lift objects in MuJoCo. Each attempt must pass motion planning and execution on the chosen robot. | Simulated robots | Nov 2026 | | Not in cards |
| HAE-RAE Bench 1.0 Models answer Korean questions about vocabulary, history, general knowledge, and reading comprehension. The original evaluation has six tasks. | Knowledge in many languages | May 2023 | | Not in cards |
| HAGRID Models answer information-seeking questions and attribute their claims to supporting passages. | Faithful to sources | Jul 2023 | | Not in cards |
| Hallmarks of Cancer Models label biomedical text with the cancer hallmarks it describes. The hub port assigns zero or more of ten labels to individual sentences. | Genomics and life sciences | Oct 2015 | | Not in cards |
| Harmony4D Models recover the body shapes and poses of people interacting at close range. The benchmark includes sports where people occlude each other. | Spatial reasoning and counting | Oct 2024 | | Not in cards |
| Hateful Memes Challenge Models decide whether a meme is hateful by interpreting its image and caption together. Paired examples change one modality while preserving the other. | Bias, fairness and politics | May 2020 | | Not in cards |
| HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. | Simulated robots | May 2026 | | Not in cards |
| HMDB51 Models identify human actions in short video clips across 51 categories. | Video understanding | Nov 2011 | | Not in cards |
| HoloAssist Mistake Detection Models watch a person perform a task and classify each annotated action as correct or mistaken. The challenge reports an F-score. | Video understanding | Sep 2023 | | Not in cards |
| Hotter and Colder Models predict sentiment and other attributes of Icelandic blog text using crowdsourced judgments. | Reading in many languages | Feb 2025 | | Not in cards |
| ImageCoDe Models select the image that matches a description from a set of visually similar alternatives. | General visual questions | Mar 2022 | | Not in cards |
| ImageNet-A Models classify real photographs that commonly fool image classifiers. The test uses unmodified images from ImageNet categories. | General visual questions | Jul 2019 | | Not in cards |
| ImageNet-R Models classify renditions of objects in drawings, sculptures, and other media. The benchmark measures recognition when appearance differs from ordinary photographs. | General visual questions | Jun 2020 | | Not in cards |
| ImageNet-Sketch Models classify black-and-white sketches using ImageNet category labels. The benchmark tests recognition when images lack photographic texture and color. | General visual questions | May 2019 | | Not in cards |
| ImageNetV2 Models classify a newly collected set of photographs using the original ImageNet labels. The test measures how accuracy changes on fresh images. | General visual questions | Feb 2019 | | Not in cards |
| Inappropriate Image Prompts Models generate images from prompts drawn from real users. The evaluation measures how often the outputs contain inappropriate material. | Refusing harmful requests | Nov 2022 | | Not in cards |
| iNaturalist Sounds Models identify animal species from recordings submitted to iNaturalist. | Audio understanding and voice | May 2025 | | Not in cards |
| IncompeBench Models retrieve music clips that match detailed natural language requests. | Audio understanding and voice | Feb 2026 | | Not in cards |
| IndicXNLI Models compare a premise and hypothesis in Indian languages. They predict whether the premise entails, contradicts or leaves the hypothesis unresolved. | Reasoning in many languages | Apr 2022 | | Not in cards |
| IndoNLI Models decide whether a hypothesis follows from an Indonesian passage using annotations from experts and other speakers. | Reading in many languages | Nov 2021 | | Not in cards |
| Inspect India Evals Models answer questions about Indian culture and reason across Indian languages. Other tasks test social bias, digital public infrastructure safeguards and refusal behavior. | Knowledge in many languages | Jul 2026 | | Not in cards |
| InstructIR Models find documents that satisfy a user's search request and its accompanying instructions. | Questions that need search | Feb 2024 | | Not in cards |
| iSAID Models detect and segment individual objects in aerial images. The benchmark measures how well predicted masks match annotated object instances. | Detection and captioning | May 2019 | | Not in cards |
| JamendoLyrics Models locate the start of each sung word using a recording and its lyrics. The original evaluation contains 20 English songs. | Speech recognition and translation | Feb 2019 | | Not in cards |
| Jamp Models infer temporal relationships between Japanese sentences. Separate test splits vary the templates, time formats, and time spans. | Reasoning in many languages | Jun 2023 | | Not in cards |
| Japanese Tort-case Dataset Models predict whether a Japanese court found a tort from the facts and the parties' arguments. They also label which arguments the court accepted. | Law | May 2024 | | Not in cards |
| JEMHopQA Models answer Japanese questions that require combining facts across multiple documents and provide the supporting reasoning. | Reading in many languages | Jul 2023 | | Not in cards |
| K-MHaS Models assign hate speech labels to Korean news comments. A comment can receive more than one label. | Bias, fairness and politics | Oct 2022 | | Not in cards |
| Korean Canonical Legal Benchmark Models answer Korean legal questions with supporting precedents. The benchmark includes multiple-choice questions and essays with individual grading criteria. | Law | Oct 2025 | | Not in cards |
| KPI-EDGAR Models identify key performance indicators and related values in financial reports. The original annotations also link entities within the text. | Finance | Oct 2022 | | Not in cards |
| KuralHub Models classify emotions in speech from 29 languages. | Audio understanding and voice | Feb 2025 | | Not in cards |
| KurdiSent Models predict the sentiment expressed in Kurdish text. | Reading in many languages | | | Not in cards |
| Large Movie Review Dataset Models classify an IMDb movie review as positive or negative. | Reading comprehension | Jun 2011 | | Not in cards |
| Legal Opinion Sentiment Classification Models classify the sentiment expressed in passages from legal opinions. Expert annotations provide the reference labels. | Law | Oct 2020 | | Not in cards |
| LEVIR-CDC Models describe changes between aerial images and mark changed buildings. The authors paired LEVIR-CC captions with binary building masks. | Detection and captioning | Oct 2024 | | Not in cards |
| LEXTREME Models analyze legal text across languages. Tasks include predicting document labels, identifying legal arguments, and labeling words within documents. | Law | Jan 2023 | | Not in cards |
| LightGenBench Models predict which parts of a 3D surface emit light and what color they emit. The authors compare generated emission with reference assets. | Image generation and editing | Oct 2026 | | Not in cards |
| LIMIT Models retrieve specific pairs of documents from a shared collection using embeddings. | Questions that need search | Aug 2025 | | Not in cards |
| LitSearch Models retrieve relevant research papers in response to questions about the scientific literature. | Questions that need search | Jul 2024 | | Not in cards |
| LLMEval-Med Models answer clinical questions drawn from patient records and expert scenarios. A model grades answers against physician-developed checklists. | Medicine | Jun 2025 | | Not in cards |
| LM-SynEval Models compare English sentences that differ in grammaticality. The test measures preferences for correct agreement and other syntactic dependencies. | Language reasoning | Oct 2018 | | Not in cards |
| Long-term Video Object Segmentation v2 Models track object masks through long videos. The evaluation tests whether they can recover an object's identity after an extended absence. | Video understanding | Apr 2024 | | Not in cards |
| LoTTE Models retrieve passages that answer questions about less common topics. | Questions that need search | Dec 2021 | | Not in cards |
| LoveDA Models label land cover in aerial images of urban and rural areas. The benchmark supports comparisons across these two settings. | Detection and captioning | Oct 2021 | | Not in cards |
| LVIS Models locate objects and outline each instance across a large vocabulary of categories. Meta reports box and mask average precision. | Detection and captioning | Aug 2019 | | Not in cards |
| MacroLens Models use company and economic data to make financial estimates under different scenarios. The authors compare forecasting models with language models and simple baselines. | Finance | Jun 2026 | | Not in cards |
| MAFAND-MT Models translate news text in a benchmark covering 21 African languages. | Translation | Jul 2022 | | Not in cards |
| MagnaTagATune Models predict descriptive tags for music clips. Qwen compares their predictions with human annotations using micro F1. | Audio understanding and voice | | | Not in cards |
| Maltese News Categories Models assign one or more topic labels to Maltese news articles. | Reading in many languages | May 2024 | | Not in cards |
| MammalNet Models identify mammals and their behavior in videos. The benchmark covers 173 mammal categories and 12 behaviors, including nursing and hunting. | Video understanding | Mar 2023 | | Not in cards |
| ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. | Simulated robots | Nov 2026 | | Not in cards |
| MasakhaNER Models recognize named entities in ten African languages. The authors compare supervised and cross-language transfer methods. | Reading in many languages | Mar 2021 | | Not in cards |
| MasakhaNER 2.0 Models identify named entities in human-annotated text across 20 African languages. | Reading in many languages | Oct 2022 | | Not in cards |
| MasakhaNEWS Models predict the topic of news articles written in African languages. | Reading in many languages | Apr 2023 | | Not in cards |
| MC-TACO Models judge plausible answers to questions about when events happen, how often they recur and how long they last. | Commonsense | Sep 2019 | | Not in cards |
| MedAlign Models follow clinician-written instructions using a patient’s electronic medical record. Clinicians evaluate the responses. | Medicine | Aug 2023 | | Not in cards |
| MedDialog Models generate a clinician’s response from a patient description. The harness also evaluates summaries of medical questions using reference text. | Medicine | Nov 2020 | | Not in cards |
| MEDIQA 2019 Question Answering Models answer consumer health questions or assess candidate answers. The shared task evaluates the relevance and quality of answers to medical questions. | Medicine | Aug 2019 | | Not in cards |
| MedPT Models classify Brazilian Portuguese medical questions by specialty. The original evaluation tests 20 classes, while the hub creates a split after deduplication and label selection. | Medicine | Nov 2025 | | Not in cards |
| MELA Models judge whether sentences are linguistically acceptable across languages. The evaluation compares their judgments with the labeled examples. | Reasoning in many languages | Nov 2023 | | Not in cards |
| MELD Models identify emotions in utterances from multiparty conversations. The dataset includes speech, video, and text. | Audio understanding and voice | Oct 2018 | | Not in cards |
| MeQSum Models condense consumer health questions into shorter questions while preserving the medical information needed for retrieval. Reference summaries provide the comparison. | Medicine | Jul 2019 | | Not in cards |
| MIMIC-IV-BHC Models summarize a patient’s hospital course from clinical notes. The reference is the brief hospital course section of the discharge summary. | Medicine | Dec 2024 | | Not in cards |
| MInDS-14 Models identify the banking service a caller wants from a spoken request. | Speech recognition and translation | Apr 2021 | | Not in cards |
| mMARCO Models rank passages by relevance to a query. The multilingual benchmark translates the MS MARCO passage ranking collection. | Reading in many languages | Aug 2021 | | Not in cards |
| MMDocIR Models find relevant pages or layout elements inside long documents that contain both text and visual material. | Documents, charts and text in images | Jan 2025 | | Not in cards |
| MMGenre Models synthesize singing in different genres. The authors measure vocal quality, lyric errors, and how well the singing conveys the intended genre. | Audio understanding and voice | Jul 2026 | | Not in cards |
| Moral Stories Models reason about norms, intentions, actions and consequences in short stories. Tasks include distinguishing a norm-respecting action from a norm-violating alternative. | Commonsense | Nov 2021 | | Not in cards |
| MOSE Models follow and segment objects in videos with crowds, occlusions, and objects leaving the screen. The benchmark compares masks against human annotations. | Video understanding | Feb 2023 | | Not in cards |
| MOSEv2 Models track and segment objects in videos with complex scenes. Objects may disappear, reappear, or blend into a crowd. | Video understanding | Aug 2025 | | Not in cards |
| mRobust04 Models rank documents for translated queries from the TREC Robust04 collection. Relevance judgments support comparisons across languages. | Reading in many languages | Sep 2022 | | Not in cards |
| MTS-Dialog Models write a clinical note section from a conversation between a doctor and a patient. The evaluation compares it with the reference note. | Medicine | May 2023 | | Not in cards |
| Multi-Scenario Anomaly Detection Models score video frames for unusual events across 14 settings. The two protocols hold out different videos and score frame-level detection. | Video understanding | Jun 2024 | | Not in cards |
| Multilingual HateCheck Models distinguish hate speech from statements that quote, reject, or otherwise resemble hate speech. The test covers multiple languages. | Bias, fairness and politics | Jul 2022 | | Not in cards |
| MUSEBENCH Models detect computer-generated music and identify artifacts. The authors compare eleven detectors on original recordings and on transformed tracks. | Audio understanding and voice | Oct 2026 | | Not in cards |
| Music Understanding Benchmark Models estimate tempo, key, meter, and song structure from recordings with identifying metadata removed. The evaluator compares predictions with existing annotations. | Audio understanding and voice | Oct 2026 | | Not in cards |
| Musical Expectation Benchmark Models predict how surprising a musical passage will sound. The benchmark compares predictions from scores or recordings with perceptual measurements. | Audio understanding and voice | Nov 2026 | | Not in cards |
| MUStARD Models determine whether a speaker is being sarcastic in a television clip. HELM evaluates the audio track. | Audio understanding and voice | Jun 2019 | | Not in cards |
| MuTox Models listen to short speech clips in 30 languages and classify whether they contain toxic speech. The harness compares their decisions with human labels. | Audio understanding and voice | Jan 2024 | | Not in cards |
| MuTual Models read a multi-turn dialogue and select the response that follows from the speakers’ exchange. The alternatives require reasoning about the conversation. | Multi-turn conversation | Jul 2020 | | Not in cards |
| NaijaRC Models answer multiple-choice reading comprehension questions from high school exams in three Nigerian languages. | Reading in many languages | Aug 2023 | | Not in cards |
| Natural Adversarial Objects Models identify objects in unmodified photographs that caused errors in earlier detectors. Human annotations identify the objects. | Detection and captioning | Nov 2021 | | Not in cards |
| NAVI Models match points on the same object across photographs taken from different viewpoints. Reference geometry supplies the correct correspondences. | Spatial reasoning and counting | Jun 2023 | | Not in cards |
| NevIR Models distinguish documents whose meanings differ because one passage negates a statement. | Questions that need search | May 2023 | | Not in cards |
| NewsQA Models extract an answer from a CNN news article in response to a question written by a reader. | Reading comprehension | Nov 2016 | | Not in cards |
| NLEBench Models generate Norwegian text for tasks including reading comprehension and summarization. The benchmark combines translated and human-annotated material. | Reading in many languages | Dec 2023 | | Not in cards |
| nocaps Models caption photographs that contain objects outside the caption training vocabulary. The benchmark compares generated captions with human references. | Detection and captioning | Dec 2018 | | Not in cards |
| NollySenti Models classify the sentiment of Nollywood movie reviews in five languages spoken in Nigeria. | Reading in many languages | May 2023 | | Not in cards |
| Nufo Models transcribe Ewe speech from a fixed test split. The authors compare transcriptions with reference text using word error rate. | Speech recognition and translation | Oct 2026 | | Not in cards |
| ObjectNet Models identify familiar objects photographed with controlled changes in viewpoint, orientation, and background. Meta reports results on classes shared with ImageNet. | General visual questions | | | Not in cards |
| OPT-RSVG Models use a written description to locate an object in an aerial image. The test pairs 24,477 descriptions with reference boxes. | Detection and captioning | Jul 2023 | | Not in cards |
| OPUS-100 Models translate between English and other languages. Separate test sets measure performance on supervised directions and language pairs without direct training data. | Translation | Jul 2020 | | Not in cards |
| OVEN Models identify the Wikipedia entity that answers a question about an image. | General visual questions | Feb 2023 | | Not in cards |
| PaintSkills Models generate images of specified objects, counts, and spatial relationships. Detectors check whether an image satisfies the requested composition. | Image generation and editing | Feb 2022 | | Not in cards |
| ParsiNLU Models read Persian passages, compare queries, and identify sentiment. The six tasks also cover entailment, multiple-choice questions, and translation. | Reading in many languages | Oct 2020 | | Not in cards |
| PAWS-X Models decide whether two sentences with similar wording express the same meaning across languages. | Reading in many languages | Aug 2019 | | Not in cards |
| PersistBench Models answer requests using stored memories. A judge measures inappropriate disclosure, agreement with false user beliefs and failure to use relevant memories. | Deception and alignment | Feb 2026 | | Not in cards |
| PHINC Models translate social media posts that mix Hindi and English into English. | Translation | Apr 2020 | | Not in cards |
| PhoMT Models translate between Vietnamese and English. Held-out sentence pairs provide reference translations in both directions. | Translation | Nov 2021 | | Not in cards |
| Physiome-ODE Models forecast irregularly sampled biological time series generated from published differential equations. The authors compare forecasts using mean squared error. | Genomics and life sciences | Feb 2025 | | Not in cards |
| Plain English Summarization of Contracts Models summarize contract clauses in plain English. Human summaries provide references for evaluating how well each response conveys the legal language. | Law | Jun 2019 | | Not in cards |
| Poem Sentiment Models judge the sentiment expressed in individual verses of English poems from Project Gutenberg. | Creative writing | Dec 2020 | | Not in cards |
| PolEmo 2.0 Models classify the sentiment of Polish consumer reviews. Separate test sets measure performance in familiar and unfamiliar domains. | Reading in many languages | Nov 2019 | | Not in cards |
| PsyTAR Models identify adverse reactions in patient reviews of four antidepressants. The original corpus also labels withdrawal symptoms, reasons for treatment, and reported effectiveness. | Medicine | Jan 2019 | | Not in cards |
| QAG-360K Models compare two satellite images, answer questions about land cover, and mark the relevant pixels. The authors score answer accuracy and mask overlap separately. | Spatial reasoning and counting | Oct 2024 | | Not in cards |
| RaceBias (Race-Based Medicine) Models answer clinical questions used to identify unsupported race-based medical beliefs. Evaluators check responses for those beliefs. | Bias, fairness and politics | Oct 2023 | | Not in cards |
| ReactSim-Bench Driving simulators predict how traffic responds when the vehicle changes its behavior. The authors test those reactions in 2,636 scenarios. | Simulated robots | Jun 2026 | | Not in cards |
| RealCommBench Models distinguish human and synthetic speech after transmission over real phone calls. Paired recordings measure how the call changes detection error. | Speech recognition and translation | Oct 2026 | | Not in cards |
| RealEstate10K Models estimate relative camera poses across frames from real estate videos. The benchmark supplies camera trajectories for comparison. | Spatial reasoning and counting | | | Not in cards |
| Ref-ZOM Models mark the pixels described by a phrase in a COCO image. A phrase can refer to zero, one or several objects. | Detection and captioning | Oct 2023 | | Not in cards |
| RegexEval Models turn natural-language descriptions into regular expressions. The dataset contains 762 patterns with matching and nonmatching examples. The hub port holds out 62 patterns. | Function-level problems | Nov 2023 | | Not in cards |
| RELATE Models listen to an audio clip and predict how well it matches a caption. Human listeners also judge which sounds occur and in what order. | Audio understanding and voice | Jun 2025 | | Not in cards |
| ReMuQ Models retrieve a knowledge passage that answers a question about an image. | General visual questions | Jun 2023 | | Not in cards |
| Revisited Oxford and Paris Models retrieve photographs of the same landmark from Oxford and Paris image collections. The revised benchmark defines medium and hard evaluation protocols. | General visual questions | Mar 2018 | | Not in cards |
| RMU:ECHR Models mark argument spans in European Court of Human Rights rulings and identify the type of argument and the party making it. | Law | Aug 2022 | | Not in cards |
| RSIEval Models describe scenes in aerial photographs and answer questions about objects, positions, and road directions. Human graders score captions and answers separately. | General visual questions | Jul 2023 | | Not in cards |
| RTLLM Models write Verilog modules from design descriptions. Testbenches check syntax and circuit behavior. Synthesis measures the cost of working designs. | Electronics | Aug 2023 | | Not in cards |
| SB10k Models classify the sentiment of German social media posts using labels from three human annotators. | Reading in many languages | Apr 2017 | | Not in cards |
| ScanNet Models infer indoor scene geometry from images. Meta evaluates depth predictions against measurements from scanned indoor environments. | Spatial reasoning and counting | Feb 2017 | | Not in cards |
| ScanNet-1500 Models match features between pairs of indoor views and estimate relative camera poses. The evaluation measures pose accuracy on 1,500 image pairs. | Spatial reasoning and counting | Nov 2019 | | Not in cards |
| SciCap Models describe figures from scientific publications. Researchers compare generated captions with the text that accompanies each figure. | Documents, charts and text in images | Oct 2021 | | Not in cards |
| SD-QA Models answer spoken questions recorded in different English dialects. The benchmark compares answer accuracy across regional varieties of English. | Audio understanding and voice | Sep 2021 | | Not in cards |
| Self-Instruct User-Oriented Instructions Models follow manually written instructions on everyday tasks. Evaluators judge whether their responses meet the user's request. | Following precise instructions | Dec 2022 | | Not in cards |
| SemEval-2015 Task 1 Models decide whether two short social media posts express the same meaning. | Reading comprehension | Jun 2015 | | Not in cards |
| Sentences Involving Compositional Knowledge Models compare sentences whose meanings change with their grammatical structure. The dataset includes both relatedness ratings and entailment labels. | Reading comprehension | May 2014 | | Not in cards |
| skLEP Models classify Slovak reviews, recognize entailment, identify names, and answer questions about Slovak Wikipedia passages. | Reading in many languages | Jul 2025 | | Not in cards |
| SLDBench Agents infer scaling laws from data covering more than 5,000 language model experiments. Eight tasks test how well their formulas predict results at other scales. | AI research tasks | Jul 2025 | | Not in cards |
| SLIP zero-shot transfer suite Models assign labels to images using text descriptions of the classes. The evaluation averages zero-shot accuracy across 26 recognition tasks. | General visual questions | Dec 2021 | | Not in cards |
| SLURP Models identify the intent and named entities in spoken requests. NVIDIA evaluates both tasks within AIR-Bench. | Audio understanding and voice | Nov 2020 | | Not in cards |
| SMEBench Models edit recorded music according to instructions. The benchmark uses paired source and target recordings to check changes to pitch, instrumentation, articulation, and dynamics. | Audio understanding and voice | Sep 2026 | | Not in cards |
| SPair-71k Models match corresponding parts of objects across image pairs. The benchmark includes changes in viewpoint, scale, and occlusion. | Spatial reasoning and counting | Aug 2019 | | Not in cards |
| SpartQA Models answer questions about spatial relationships described in a passage. | Language reasoning | Apr 2021 | | Not in cards |
| Speech Robust Bench Models transcribe speech under different recording distortions. Word error rates show how each distortion affects recognition. | Speech recognition and translation | Mar 2024 | | Not in cards |
| Stanford Online Products Models retrieve photographs of the same product. The test compares learned image features on products outside the training set. | General visual questions | Nov 2015 | | Not in cards |
| STAR situated video reasoning Models answer multiple-choice questions about how people interact with objects in videos. The questions cover interactions, sequences, predictions, and feasibility. | Video understanding | | | Not in cards |
| STL-10 Models classify images into ten object categories using limited labeled training data. | General visual questions | Jun 2011 | | Not in cards |
| Structured Web Data Extraction Models extract named attribute values from web pages. The benchmark compares their outputs with annotated values across website categories. | Faithful to sources | Jul 2011 | | Not in cards |
| Summ-it Coreference Models group Portuguese noun phrases that refer to the same entity in science articles from Folha de São Paulo. | Reading in many languages | Jul 2007 | | Not in cards |
| SUMO Web Claims Summarization Models write a short headline summarizing a fact-checking article. HELM uses articles about climate claims. | Faithful to sources | Nov 2020 | | Not in cards |
| SUN397 Models recognize the scene category in indoor and outdoor photographs. The classification task covers 397 categories. | General visual questions | Jun 2010 | | Not in cards |
| SWAG Models read a partial description of an everyday activity. They choose the most plausible continuation from four options. | Commonsense | Aug 2018 | | Not in cards |
| Swiss-Judgment-Prediction Models predict whether the Swiss Federal Supreme Court accepts or dismisses a case. | Law | Oct 2021 | | Not in cards |
| Szeged NER Models identify named entities in Hungarian business news from the Hungarian News Agency. | Reading in many languages | May 2006 | | Not in cards |
| T2Ranking Models rank Chinese passages by their relevance to a query. | Reading in many languages | Jul 2023 | | Not in cards |
| Taiwan Mandarin Speech with Video: Speech Enhancement Models use lip video to remove noise from Mandarin sentences. Researchers compare the enhanced signal with clean speech for quality and intelligibility. | Speech recognition and translation | May 2020 | | Not in cards |
| TAU Urban Acoustic Scenes 2022 Mobile Models identify an urban acoustic scene from a one-second recording made on a mobile device. | Audio understanding and voice | Mar 2022 | | Not in cards |
| Testing Relational Understanding in Text-Guided Image Generation Models depict relationships between named entities. Human participants judge whether the generated scenes match each caption. | Image generation and editing | Jul 2022 | | Not in cards |
| Text2CAD Models generate parametric CAD designs from text prompts at different levels of detail. The authors compare generated geometry with the reference and count invalid designs. | CAD models | Sep 2024 | | Not in cards |
| TGIF-QA Models answer questions about actions, repetitions, and changes over time in animated clips. | Video understanding | Apr 2017 | | Not in cards |
| The Met Dataset Models match photographs of artworks to catalog images from the Metropolitan Museum of Art. The evaluation measures instance recognition across photographic conditions. | General visual questions | | | Not in cards |
| Third PASCAL Recognizing Textual Entailment Challenge Models decide whether a passage entails a proposed statement. The challenge compares their decisions with human labels. | Reading comprehension | Jun 2007 | | Not in cards |
| tinyBenchmarks Models answer small selected subsets of established benchmarks. Item response estimates approximate accuracy on the full tests. | Composite indices | Feb 2024 | | Not in cards |
| TokSuite Models complete questions under alternative tokenizations. The benchmark measures the change in accuracy relative to the canonical representation. | Reasoning in many languages | Dec 2025 | | Not in cards |
| ToLD-Br Models identify toxic language in Brazilian Portuguese posts using human annotations. | Bias, fairness and politics | Oct 2020 | | Not in cards |
| TopiOCQA Models retrieve evidence and answer follow-up questions in conversations that change topics. | Questions that need search | Oct 2021 | | Not in cards |
| ToxicChat Models identify toxic user messages from real conversations with an online chatbot. | Deception and alignment | Oct 2023 | | Not in cards |
| TREC 2017 LiveQA Medical Task Models answer consumer health questions. The shared task compares responses with expert assessments of answer quality. | Medicine | | | Not in cards |
| Turku NER Corpus Models identify named entities in Finnish text from the Universal Dependencies corpus. | Reading in many languages | May 2020 | | Not in cards |
| Turku Paraphrase Corpus Models compare Finnish sentence pairs and estimate how closely their meanings match. | Reading in many languages | May 2021 | | Not in cards |
| TUT Urban Acoustic Scenes 2018 Models recognize the setting of a ten-second audio clip recorded in a European city. | Audio understanding and voice | Apr 2018 | | Not in cards |
| Twitter URL Paraphrase Corpus Models identify paraphrases among pairs of social media posts that reference the same URL. | Reading comprehension | Sep 2017 | | Not in cards |
| UIT-ViCTSD Models classify Vietnamese comments for constructiveness and toxicity. The two labels capture different properties of the same text. | Reading in many languages | Mar 2021 | | Not in cards |
| UIT-VSFC Models classify sentiment in Vietnamese student feedback. The corpus also includes topic labels. | Reading in many languages | Nov 2018 | | Not in cards |
| Unhealthy Comments Corpus Models rate newspaper comments for overall conversational health and six attributes, including hostility and sarcasm. Researchers score each label separately using ROC AUC. | Reading comprehension | Aug 2020 | | Not in cards |
| UrbanSED Models locate sound events in synthetic urban soundscapes. The benchmark includes reference labels for the times and types of events. | Audio understanding and voice | Oct 2017 | | Not in cards |
| UrduSER Models identify emotions in Urdu dialogue from Pakistani television. The corpus contains 3,500 excerpts from ten actors across seven emotion categories. | Audio understanding and voice | Dec 2024 | | Not in cards |
| VALOR-32K Models match captions to video and audio clips. Meta reports separate retrieval results for text paired with audio and text paired with video. | Audio understanding and voice | Apr 2023 | | Not in cards |
| Verifiability Judgments for Generative Search Models decide whether a cited source fully, partly, or never supports a generated claim. Human annotations provide the reference judgments. | Faithful to sources | Apr 2023 | | Not in cards |
| Video-ChatGPT video conversation benchmark Models answer questions about video content. The evaluation grades correctness, detail, context, timing, and consistency. | Video captioning | Jun 2023 | | Not in cards |
| VideoCon Models distinguish descriptions supported by a video from captions that contradict it. | Video understanding | Nov 2023 | | Not in cards |
| Vinoground Models match short videos with captions that distinguish events by their temporal order. | Video understanding | Oct 2024 | | Not in cards |
| VM14K Models answer Vietnamese multiple-choice questions across 34 medical specialties. The authors reserve a private set for leaderboard evaluation. | Medicine | May 2025 | | Not in cards |
| VocalSound Models classify human vocal sounds such as laughter, coughing, and sneezing. | Audio understanding and voice | May 2022 | | Not in cards |
| Voice Jailbreak Attacks Against GPT-4o Models respond to spoken requests. Human reviewers measure how often a model complies with an unsafe request. | Jailbreaks and prompt injection | May 2024 | | Not in cards |
| VSEC Models correct misspellings in Vietnamese text. The evaluation compares their corrections against reference sentences. | Reading in many languages | Nov 2021 | | Not in cards |
| WenetSpeech-Chuan-Eval Models transcribe Sichuanese speech. Qwen reports separate error rates on the easy and hard test sets. | Speech recognition and translation | Sep 2025 | | Not in cards |
| WHU-CDC Models compare aerial photos of Christchurch, describe what changed, and mark changed pixels. Reference captions describe reconstruction after the earthquake. | Detection and captioning | Oct 2024 | | Not in cards |
| Wiki Neutrality Corpus Models identify biased wording in English Wikipedia sentences and rewrite the sentences. The authors compare the rewrites with corrections by Wikipedia editors. | Bias, fairness and politics | Nov 2019 | | Not in cards |
| WikiLingua Models summarize procedural articles in different languages. Aligned WikiHow articles support both same-language and cross-language evaluation. | Reading in many languages | Nov 2020 | | Not in cards |
| WikiNEuRal Models locate named entities in Wikipedia text across nine languages using automatically produced reference annotations. | Reading in many languages | Nov 2021 | | Not in cards |
| Winoground Models match two images to two captions that use the same words in a different order. | General visual questions | Apr 2022 | | Not in cards |
| WixQA Models retrieve Wix support articles and answer customer questions from those articles. The authors score retrieval and answer generation separately. | Running a business | Apr 2025 | | Not in cards |
| WSYue-eval Models transcribe short and long Cantonese recordings. The suite also evaluates speech synthesis. | Speech recognition and translation | Sep 2025 | | Not in cards |
| x-stance Models identify whether a political comment supports or opposes the proposition in a question. | Reading in many languages | Mar 2020 | | Not in cards |
| XNLIeu Models judge whether a Basque sentence follows from, contradicts, or is neutral toward a premise. The evaluation includes translated and native examples. | Reading in many languages | Apr 2024 | | Not in cards |
| XOR-TyDi QA Models retrieve passages across languages to answer questions written by speakers of seven languages. Tasks require English evidence or answers in the question’s language. | Reading in many languages | Oct 2020 | | Not in cards |
| XWinograd Models resolve ambiguous pronouns in six languages by choosing which noun fits each sentence. | Reasoning in many languages | Jun 2021 | | Not in cards |
| YouTube-VOS Models segment and track objects across frames in YouTube videos. The evaluation distinguishes categories seen during training from previously unseen categories. | Video understanding | Sep 2018 | | Not in cards |