Benchmark Almanac

colleagues

We track 599 benchmarks and 4 arenas that colleagues made or helped make.

0 in use 403 with public code 250 with a leaderboard here

Arena leaderboards

Top five in each of its 2 arenas with three or more rated models, the most rated first. We don't have a leaderboard yet for 2 others, listed below.

Benchmarks

Name Kind Released Best score Status
VCSUM Models summarize transcripts of 239 real Chinese meetings that total over 230 hours. Scored with ROUGE against human-written summaries. Reasoning over long documents May 2023 17.3 Dropped
BLUEX Models answer questions from two Brazilian university entrance examinations. The questions cover high school subjects in Portuguese. Knowledge in many languages Jul 2023 95% Not in cards
Angry Tweets Models classify the sentiment of Danish social media posts using labels supplied by crowdworkers. Reading in many languages May 2021 67% Not in cards
FoSent Models classify sentiment in Faroese news articles. Two native speakers annotated articles from Portalurin and Dimmalætting. Reading in many languages May 2024 91.6% Not in cards
FoQA Models answer questions about passages from Faroese Wikipedia. A native speaker checked and corrected automatically generated answers. Reading in many languages Feb 2025 79.9% Not in cards
GermanQuAD Models extract answers to German questions from manually annotated Wikipedia passages. Reading in many languages Apr 2021 70% Not in cards
OCR-VQA Models answer questions about book covers by reading visible text. The benchmark compares answers with facts such as a book’s title or author. Documents, charts and text in images Sep 2019 82% Not in cards
Icelandic WinoGrande Models complete Icelandic sentences by choosing which referent fits the context. The evaluation uses an Icelandic adaptation of WinoGrande. Reasoning in many languages Jun 2022 96.3% Not in cards
TeleQnA Models answer multiple-choice questions about telecommunications terminology, research, and technical standards. Expert-written questions Oct 2023 85.2% Not in cards
TeleTables Models answer questions about tables from 3GPP specifications, both from memory and with the table available. Expert-written questions Jan 2026 48% Not in cards
PAIRS (Parallel Images) Models answer questions about people in paired images. The evaluation tests whether changing demographic characteristics changes judgments about the subjects. Bias, fairness and politics Feb 2024 97.2% Not in cards
Unicorn (Vision Safety) Models answer questions about objects in unusual images and sketches. The benchmark tests how well visual judgments hold up when the images change. General visual questions Nov 2023 91.2% Not in cards
FINESSE-Bench Models answer financial certification questions and interpret trading indicators. The benchmark checks numerical and multiple-choice answers and uses a model judge for free responses. Finance May 2026 0.90 Not in cards
LongBench Pro Models answer English and Chinese questions over documents from 8,000 to 256,000 tokens. The benchmark applies task-specific metrics to 1,500 examples. Reasoning over long documents Jan 2026 73.4% Not in cards
PerspectiveGap Models assign information fragments to workers with different roles. A strict pass requires supplying all needed information without disclosing extra fragments. Following precise instructions Jun 2026 62% Not in cards
GreekMMLU Models answer questions originally written in Greek for academic, professional, and government exams across 45 subjects. Knowledge in many languages Feb 2026 93.2% Not in cards
HumanAgencyBench Models respond to 3,000 prompts about six ways assistants can support human agency, including asking clarifying questions and leaving important decisions to users. Deception and alignment Sep 2025 71.3% Not in cards
OrangeSum Models summarize French news articles from Orange Actu. The dataset pairs articles with summaries written by journalists. Reading in many languages Oct 2020 38.5% Not in cards
Uncheatable Eval Models predict text from recently published documents. The evaluation measures perplexity across sources including news, research papers, fiction and code. Reading comprehension May 2024 6.29 Not in cards
LLMzSzŁ Models answer Polish academic and professional exam questions drawn from the Central Examination Board archives. Knowledge in many languages Jan 2025 69.1% Not in cards
SLR-Bench Models write Prolog rules that classify trains by their composition. The verifier runs each rule to determine whether its predictions are correct. Abstract puzzles and logic Jun 2025 81.8% Not in cards
ArxivRollBench Models reconstruct passages from recently published scientific papers. They order fragments, fill gaps and choose what comes next. Reading comprehension Jul 2025 70.8% Not in cards
KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. Medicine Sep 2026 29.4% Not in cards
MUSE Models answer questions about artworks used in language education. The tasks cover visual details, emotions, cultural context, and relationships within the image. Other professions Sep 2026 56.7% Not in cards
AnalogCoder Models write Python programs that build analog circuits. A simulator checks whether each circuit performs its requested function. The authors report first-attempt success and how many tasks each model solves. Electronics May 2024 83.3% Not in cards
Evalita-LLM Models perform tasks written originally in Italian. The suite includes admission exams, question answering and text generation. Reading in many languages Feb 2025 62.3 Not in cards
NoveltyBench Models give several answers to the same open-ended request. The benchmark measures how many distinct responses they produce and how useful those responses are. Creative writing Apr 2025 5.32 Not in cards
VeriGen design problems Models complete Verilog modules for 17 digital circuits. Simulation checks whether the generated code satisfies each design’s testbench. Electronics Dec 2022 77.7% Not in cards
BertaQA Models answer questions about Basque culture and general knowledge. Parallel Basque and English questions help distinguish language effects from familiarity with local topics. Trivia and open questions Jun 2024 81.5% Not in cards
K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. Other professions Jun 2025 55.4% Not in cards
DaLA Models distinguish acceptable Danish sentences from copies corrupted to resemble errors in real text. Reasoning in many languages Dec 2025 61.7% Not in cards
EWoK Models decide which context makes a target statement plausible. Paired examples isolate concepts about the physical and social world. Commonsense May 2024 91.2% Not in cards
IFEvalCode Models write code in eight programming languages under explicit constraints. The evaluator checks functional correctness and adherence to instructions separately. Function-level problems Jul 2025 38.2% Not in cards
MultiBLiMP Models compare minimally different sentences across languages. The benchmark tests whether the model assigns higher likelihood to the grammatical sentence. Reasoning in many languages Apr 2025 90.2% Not in cards
EusExams Models answer past public service examination questions in Basque. The questions come from several Basque institutions. Knowledge in many languages Mar 2024 51.3% Not in cards
EusReading Models answer comprehension questions about Basque passages from past C1 proficiency examinations. The scorer checks their choices against the exam key. Reading in many languages Mar 2024 62.2% Not in cards
EusTrivia Models answer trivia questions in Basque. The set combines elementary-school questions with harder general knowledge questions. Knowledge in many languages Mar 2024 51% Not in cards
GPBam Models write legal opinions on German public law cases and retrieve the statutes they need. Three judges grade essays against reference solutions. Law Nov 2026 76.4 Not in cards
EvaOmni Models answer questions about videos and identify the audio and visual evidence behind their answers. The score accounts for both correctness and evidence timing. Audio understanding and voice Oct 2026 79.4% Not in cards
BLiMP-NL Models compare Dutch sentence pairs that differ in grammatical acceptability. Reasoning in many languages Jul 2025 49.4% Not in cards
MMCircuitEval Models answer questions about circuit diagrams, specifications, and design decisions. The authors combine text similarity with a language model’s judgment. Electronics Jul 2025 69.4% Not in cards
Spatial Pathfinding Reasoning Challenge Models trace paths across grids while satisfying rules about cells and edges. The scorer checks each coordinate sequence against valid solutions. Puzzles and chess May 2025 83.6% Not in cards
EvoEval Models solve Python problems that vary the constraints, interfaces, or logic of HumanEval tasks. Harbor ports the 100 problems in the Difficult subset. Function-level problems Mar 2024 68.6% Not in cards
IndicParam Models answer multiple-choice questions from Indian competitive exams. The benchmark tests knowledge in low-resource Indic languages. Knowledge in many languages Nov 2025 45% Not in cards
MentalBench (Psychiatric Diagnosis) Models select psychiatric diagnoses for 24,750 synthetic cases. Exact matching requires both diagnoses when the case lacks the evidence needed to distinguish them. Medicine Feb 2026 62.7% Not in cards
NorEval Models complete language understanding and generation tasks in both Bokmål and Nynorsk. The authors compare model performance with human baselines. Knowledge in many languages Apr 2025 68.2 Not in cards
CS-4k Models answer questions about the stages of computer science research using papers as evidence. A model judge scores answers on the held-out evaluation set. AI research tasks Oct 2025 71.3 Not in cards
EarthVerse Agents investigate documented natural hazards using event data from different sources. The evaluator checks individual claims and the calculations that support them across 405 tasks. Lab and data work Aug 2026 85% Not in cards
KorNAT Models answer questions about Korean social values and common knowledge. The authors compare answers with Korean reference judgments. Knowledge in many languages Feb 2024 65.4% Not in cards
HireWatch Models select questions for job interviews under different workplace pressures. The benchmark measures how often they select questions classified as unlawful under US employment law. Law Nov 2026 0.05% Not in cards
SG-LegalClarify Models answer Singapore legal questions and ask for missing facts. Three model judges grade each conversation against rubrics written by lawyers. Law Nov 2026 77.8% Not in cards
UniCode Models solve generated competitive programming problems using automated test cases. Harbor packages 399 problems with validated reference solutions. Competitive programming Oct 2025 70.3% Not in cards
AgroMind (original) Models answer questions about farmland images across 13 tasks. They identify crops, assess crop health and interpret spatial relationships. General visual questions May 2025 35.8% Not in cards
FinFIRST Agents research financial questions and provide supporting sources. The benchmark grades individual requirements and reports how often an answer meets every requirement. Finance Sep 2026 71.5% Not in cards
MVDriveBench Models answer driving questions using synchronized videos from cameras facing different directions. The authors compare surround-view results with front-camera results. Embodied reasoning Nov 2026 80.1% Not in cards
PISA-Bench Models answer illustrated PISA questions in six languages. Parallel questions let researchers compare language effects on reading, science and mathematical reasoning. General visual questions Oct 2025 63.3% Not in cards
SATBench Models decide whether a set of constraints in a narrative puzzle can all hold at once. A Boolean satisfiability solver supplies the correct label. Language reasoning May 2025 89.3% Not in cards
LegalCiteBench Models recover and verify citations from United States judicial opinions without external lookup. The benchmark also tests whether models match cases and correct citation errors. Law Feb 2026 0.78 Not in cards
LongProc Models carry out long procedures such as converting HTML tables, tracing paths and tracking beliefs. The evaluator checks outputs as the required procedure grows. Reasoning over long documents Jan 2025 74.2 Not in cards
MUSE (CAD assemblies) Models turn design specifications into CAD assemblies. A program checks the geometry. A vision model grades function, manufacturability, and ease of assembly. CAD models May 2026 52.4% Not in cards
RPC-Bench Models read computer science papers and answer reviewers’ questions. A model judge grades correctness, completeness, and concision against verified review and rebuttal pairs. Research science Apr 2026 37.5 Not in cards
ATM-Bench Agents answer questions about a person’s history using images, videos, and email records. The Harbor package contains 31 hard questions with schema-guided memory. Finding facts in long inputs Mar 2026 86% Not in cards
ChipBench Models write Verilog modules, debug circuits and generate reference models. Simulations compare the outputs with reference implementations. Electronics Jan 2026 30.5% Not in cards
NPCBench Models follow patients with nasopharyngeal carcinoma through diagnosis and care. The main patient-level measure requires correct decisions at successive stages. Medicine Nov 2026 23% Not in cards
TrafficImag Models edit roadside traffic videos to show a different sequence of events. The evaluator checks routes, interactions, and preservation of the surrounding scene. Video generation Sep 2026 90.0 Not in cards
Code4Scene Agents build and edit Unreal Engine scenes. The evaluator inspects the scene itself for correct edits, intact assets, and physical validity. Image generation and editing Sep 2026 0.62 Not in cards
Glianorex Models answer multiple-choice questions about fictional medical material in English and French. The test isolates question-answering patterns from real medical knowledge. Language reasoning Jun 2024 71.5% Not in cards
PDAGENT-BENCH Models answer chip design questions, diagnose failures, interpret reports, and write tool commands. Rules grade five categories and model judges grade three. Electronics Jun 2026 65.1% Not in cards
SAHM Models answer Arabic questions on financial standards, Islamic finance, and accounting. The authors report multiple-choice accuracy and grade open-ended answers separately. Finance Apr 2026 80.5% Not in cards
CaBBQ Models answer Catalan questions about people in ambiguous and unambiguous situations. The test measures whether answers follow social stereotypes when evidence is insufficient. Bias, fairness and politics Jul 2025 81% Not in cards
CASE-Bench Models decide whether a request is safe in a given context. The same request appears in two contexts with different human judgments. Refusing harmful requests Jan 2025 90.1% Not in cards
EsBBQ Models answer Spanish questions about people in ambiguous and unambiguous situations. The evaluation measures how social stereotypes affect their choices. Bias, fairness and politics Jul 2025 85% Not in cards
LawBench Models answer Chinese legal questions across 20 tasks, including statute recall and case prediction. Each task has its own scoring rule. Law Sep 2023 44.5 Not in cards
NoticIA Models summarize Spanish news articles in one sentence that answers the clickbait headline. The evaluator compares each summary with a human reference. Faithful to sources Apr 2024 32.0 Not in cards
OmniMatBench Models answer questions and perform calculations across 19 materials science fields. The authors compare automated grades with human assessments. Science questions May 2026 0.37 Not in cards
RPEval Models respond in character to 9,018 scenarios. The test checks emotions, decisions, moral choices, and whether characters reveal knowledge outside their setting. Overall chat quality May 2025 62.2% Not in cards
Taiwanese Mandarin Language Understanding Models answer questions from Taiwanese exams in traditional Chinese. The subjects range from school topics to professional qualifications. Knowledge in many languages Mar 2024 73.6% Not in cards
VoxCeleb2 Models compare two interview recordings and decide whether the same person speaks in both. Audio understanding and voice Jun 2018 75.9% Not in cards
CBT-Bench Level II Models identify cognitive distortions and classify patients’ core beliefs at two levels of detail. The evaluator compares the selected labels with the reference labels. Medicine Oct 2024 0.63 Not in cards
MedSafetyBench Models respond to requests that conflict with medical ethics. A judge rates harmfulness on a five-point scale using a held-out set of 900 requests. Refusing harmful requests Mar 2024 1.44 Not in cards
NOTA Models extract facts from sheet music, convert score images to ABC notation, and analyze musical structure. The authors report extraction rates and separate transcription distances. Documents, charts and text in images Jun 2024 17.8% Not in cards
SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. Research science Oct 2026 80.9 Not in cards
SIM-VAIL Chatbots respond to simulated users with psychiatric vulnerabilities. A model judge scores behaviors across the conversation, including responses that reinforce a user’s vulnerability. Wellbeing and sycophancy Feb 2026 1 Not in cards
AgroBench Models answer illustrated questions from agronomists about crops, diseases, pests, weeds, machinery and farming practices. The overall score averages accuracy across seven tasks. General visual questions Jul 2025 73.5% Not in cards
ANCHOR Models explain answers to audiovisual questions and locate the supporting evidence. The benchmark includes time intervals, image regions, and relevant sounds. Audio understanding and voice Oct 2026 3.36 Not in cards
AutoBencher Models answer generated benchmark questions. The framework searches for topics and produces tests of knowledge, multilingual reasoning, and math. Language reasoning Jul 2024 55% Not in cards
ChemArgus Models solve chemistry competition and textbook problems with text and figures. A model judge awards partial credit for each subquestion. Science questions Oct 2026 64.9% Not in cards
EusProficiency Models complete Basque language exercises from the official C1 proficiency examination. The questions test vocabulary and grammar. Reading in many languages Mar 2024 56.7% Not in cards
GeoDrive-Bench Models choose driving actions from road images and local traffic conventions. The benchmark covers six countries and omits explicit country labels. Embodied reasoning Jun 2026 91.9% Not in cards
HarvestBench Agents drive simulated tractors and decide how much fuel to spend avoiding animals. The benchmark measures how often they hit animals as avoidance costs change. Deception and alignment Sep 2026 0.40% Not in cards
L2-Bench Models design lessons, give learners feedback, and participate in language practice. A model judge grades each response against teaching criteria. Other professions Jul 2026 85.5 Not in cards
LIBRA (Russian Long Context) Models read long Russian texts and answer questions or generate responses. The suite groups tasks by difficulty and varies input length. Reasoning over long documents Aug 2024 71.3 Not in cards
LMentry Models perform simple language tasks such as identifying words with a given letter or choosing the longer word. Task-specific rules check the answers. Language reasoning Nov 2022 66.1% Not in cards
LongCoT Models solve long reasoning problems across five domains. Task-specific programs verify answers, with optional judge fallback for ambiguous mathematics and chemistry responses. Reasoning over long documents Apr 2026 25.1% Not in cards
Longtail-SWE Agents resolve 202 GitHub issues across 13 programming languages with limited benchmark coverage. Hidden tests check the fix and verify that previously passing behavior still works. Fixing real repositories Nov 2026 81.5% Not in cards
P3D-Bench Models generate parametric 3D objects from text, images, or assembly instructions. The evaluator checks whether outputs run and preserve geometry, topology, and part structure. Image generation and editing Jun 2026 69.4 Not in cards
QuantEval Models answer quantitative finance questions, solve mathematical problems, and implement trading strategies. The authors report answer accuracy and test the generated code. Finance Jan 2026 86% Not in cards
RVG-Bench Models locate the video interval that answers a question about sounds and images. The evaluator measures overlap with the annotated interval. Audio understanding and voice Oct 2026 49.6% Not in cards
SciKnowEval Models answer scientific questions and apply knowledge in biology, chemistry, materials and physics. The suite uses automatic checks and model judges for different tasks. Science questions Jun 2024 0.71 Not in cards
SolveEdit Models edit an image to solve a visual problem. The evaluator checks whether the goal holds and whether unrelated scene details survive the edit. Image generation and editing Sep 2026 57% Not in cards
SPOT Models inspect published manuscripts for errors that led to corrections or retractions. A judge checks whether each reported error matches the documented error and location. Research science May 2025 6.10% Not in cards
Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. Multi-agent games Jun 2025 84.9% Not in cards
WorldCoder-Bench Agents write code that builds interactive 3D worlds. The evaluator checks whether the resulting world behaves as specified. Image generation and editing Jun 2026 27.8% Not in cards
AfroBench Models perform language understanding, reasoning, and generation tasks across 64 African languages. The full suite draws on 22 datasets. Knowledge in many languages Nov 2023 59.6 Not in cards
ChemDIRT Models answer chemistry questions while the evaluator varies instructions and molecular representations. The authors measure accuracy and consistency across variants. Science questions Mar 2026 48.7% Not in cards
GeoNatureAgent Benchmark Agents use a geospatial API to analyze environmental indicators in Spain and Portugal. The grader checks tool use and answers against task requirements. Function calling Apr 2026 60.8% Not in cards
GreekBarBench Models answer Greek bar exam questions in writing. A model judge grades their use of case facts, statutory articles, and legal analysis. Law May 2025 8.35 Not in cards
LegalBench.PT Models answer questions about Portuguese law derived from university exams. The benchmark includes case analysis, multiple selection, and true-or-false questions. Law Dec 2024 85.4% Not in cards
LexGenius Models answer Chinese legal questions drawn from cases and exams. The authors compare results across areas including legal reasoning, ethics, and judicial practice. Law Nov 2025 59.7% Not in cards
LingOly Models infer linguistic rules from examples in unfamiliar languages. They use those rules to solve linguistics olympiad problems. Reasoning in many languages Jun 2024 46.3% Not in cards
MasakhaPOS Models assign parts of speech to words in African languages. The authors compare tagging models across languages. Reading in many languages May 2023 62.8% Not in cards
MastermindEval Models infer a hidden symbol sequence from previous guesses and feedback in Mastermind. The evaluation varies sequence length and the similarity of answer choices. Puzzles and chess Mar 2025 53.7% Not in cards
MentalBench-100k Models answer counseling questions. Human reviewers grade responses to 1,000 of the 10,000 prompts on seven attributes, each from one to five. Medicine Oct 2025 5 Not in cards
MISHAP-Bench Models answer open-ended questions about audio clips. A model judge uses human annotations to identify claims the recording does not support. Audio understanding and voice Sep 2026 32.8% Not in cards
OvertonBench Models answer questions about contested issues. Participants rate how well each answer represents their viewpoint. The score measures the fraction of opinion groups represented. Bias, fairness and politics Sep 2025 0.42 Not in cards
PaperScope Agents search multiple scientific papers to compare figures, summarize results, and solve problems. Each question can require evidence from several documents. Deep research Jan 2026 41.0 Not in cards
SAS-Bench Models grade short student answers from subject exams. The benchmark compares their scores and error classifications with expert annotations. Other professions May 2025 67.2% Not in cards
VoxelCodeBench Models write code to build and modify voxel scenes. The benchmark checks the constructed shapes and evaluates their appearance. Image generation and editing Apr 2026 87.9% Not in cards
ABACUS Models add, remove, or change a specified number of objects in an image. The evaluator checks the count and preservation of the surrounding scene. Image generation and editing Oct 2026 0.31 Not in cards
ChemInsight Models solve related chemistry problems in bundles. The authors score how many parts a model can solve and how far its reasoning remains correct. Science questions Nov 2026 0.28 Not in cards
Closed-Loop Evaluation of LLM Agents for Embedded Software Development Agents write firmware from engineering briefs and test it on simulated equipment. Hidden checks score device behavior across five control tasks. Electronics 100% Not in cards
ContextASR-Bench Models transcribe speech with optional domain labels and named entities. The benchmark measures transcription errors and recognition of named entities. Speech recognition and translation Jul 2025 3.10% Not in cards
CRUST-bench Agents translate 100 C projects into safe Rust. Tests check whether each translation matches the supplied Rust interface and preserves the original behavior. Fixing real repositories Apr 2025 48% Not in cards
ELBench Models respond to teaching and safety scenarios alongside academic questions. The authors compare the same models across four educational assessment areas. Other professions Jun 2026 83.5 Not in cards
EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. Simulated robots Oct 2026 41.4% Not in cards
FinDeepIndicator Agents construct financial indicators by specifying formulas, retrieving market data, and calculating results. The evaluation checks performance at each step. Finance Aug 2026 39% Not in cards
FinMaster Models complete financial workflows using simulated company records. The benchmark includes single calculations and tasks that require several financial metrics. Finance May 2025 73% Not in cards
Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. Medicine Nov 2026 83.9% Not in cards
MDMR-Bench Models generate one image from several references. A model judge checks whether the output preserves requested elements, follows instructions, and combines them correctly. Image generation and editing May 2026 88.8% Not in cards
NARRA-Gym Agents build interactive stories around a user’s experiences and adapt scenes as the conversation unfolds. Judges rate story quality and user experience separately. Creative writing May 2026 2 Not in cards
ReplicationBench Agents reproduce computational results from published astrophysics papers. Evaluators check their numerical answers and assess whether their methods follow the original work. Research science Oct 2025 22% Not in cards
RExBench Agents extend 12 published AI research projects from expert instructions. The evaluator runs their code against task-specific success criteria. Harbor distributes two public tasks. AI research tasks Jun 2025 50% Not in cards
SAVU-Bench Models answer spatial questions about real scenes using sound and video. The authors report accuracy across seven tasks. Audio understanding and voice Oct 2026 55.3% Not in cards
VLegal-Bench Models answer questions and solve legal scenarios under Vietnamese law. Legal experts check that reference answers follow authoritative documents. Law Dec 2025 72% Not in cards
AnalogCoder-Pro Agents write analog circuit programs and use simulated waveforms to diagnose failures. The authors report the fraction of successful designs and the number of tasks solved. Electronics 88.4% Not in cards
AstroVLBench Models interpret galaxy images, light curves, and spectra across five observing tasks. The authors compare their classifications with reference labels and examine their explanations. Lab and data work Apr 2026 68.8% Not in cards
CNN/DailyMail Summarization Models summarize news articles from CNN and the Daily Mail. The harness compares generated summaries with the publishers’ highlights. Faithful to sources Jun 2015 0.15 Not in cards
GenomeQA Models answer multiple-choice questions about short genomic sequences. The benchmark covers six task families, including species classification. Genomics and life sciences Apr 2026 63.6% Not in cards
HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. Simulated robots Nov 2026 61.9% Not in cards
K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. Other professions Jun 2025 63.4% Not in cards
LiveSearch Agents search for current financial figures. The evaluator runs a stored query against a professional data service to update the reference answer. Finance Oct 2026 76.7% Not in cards
LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. Law Jun 2026 0.66 Not in cards
LT-History Models answer questions in Lithuanian about national and world history. Knowledge in many languages Mar 2025 82% Not in cards
MatSciBench Models solve college materials science problems, including questions with figures. The authors report accuracy separately for text and image questions. Science questions Oct 2025 64% Not in cards
MooseBench Models write input files for MOOSE, a multiphysics simulator. The grader reconstructs each file’s governing equations and compares them with the intended physics. Scientific code May 2026 0.85 Not in cards
Pigeonholing Partisans four-word generation Models use a survey respondent’s profile to generate four words describing supporters of each major US party. One measure counts how often raters identify the writer’s party. Social simulation Sep 2022 51.5% Not in cards
ProjectTest Models write unit tests for existing Python, Java, and JavaScript projects. The authors run the tests and measure code coverage. Fixing real repositories Feb 2025 82% Not in cards
RECODE-H Agents implement research code from papers and repositories. They can request feedback from a simulated collaborator before their code runs against tests. AI research tasks Oct 2025 84.3% Not in cards
RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. Simulated robots Nov 2026 80.6% Not in cards
SDR-Bench Models personalize sales pitches using customer research. A judge compares each pitch with points from successful human outreach. Running a business Feb 2026 55.8% Not in cards
TruthfulQA-Multi Models answer questions targeting common misconceptions in Basque, Catalan, Galician and Spanish. The translations support direct comparison with the English questions. Short factual questions Feb 2025 50.6% Not in cards
VDC Benchmark Models describe the contents of a video in detail. The evaluation checks descriptions against human annotations using questions about the video. Video captioning Oct 2024 1.96 Not in cards
τ-Rec Agents interview simulated moviegoers and search a catalog. A recommendation succeeds when it meets the user’s constraints and follows the service policy. General assistants Jun 2026 57.1% Not in cards
ACLUE Models read classical Chinese and answer questions about grammar, meaning, inference and cultural knowledge. All 15 tasks use multiple choice. Reading in many languages Oct 2023 37.3% Not in cards
AfriHate Models classify hate speech and abusive language across African languages. Reading in many languages Jan 2025 58.8% Not in cards
AILA Statute Retrieval Models retrieve the Indian statutes that apply to a described legal situation. Law Oct 2020 0.85 Not in cards
Ask Ubuntu duplicate question retrieval Models retrieve previously posted Ubuntu support questions that ask about the same problem. Questions that need search Dec 2015 0.70 Not in cards
BANKING77 Models assign an online banking question to one of 77 customer intents. Finance Mar 2020 92.3% Not in cards
CADTestBench Models generate mechanical parts from text. Executable tests check the geometry and topology of each solid. A part passes when it satisfies every test. CAD models May 2026 81% Not in cards
CLICK-ID Models distinguish clickbait headlines from other headlines published by Indonesian news outlets. Reading in many languages 65.1% Not in cards
Collab-Overcooked Two agents exchange messages and use different kitchen equipment to fill orders. The benchmark measures task completion and progress through required cooking steps. Multi-agent games Feb 2025 86.7% Not in cards
DBpedia 14 Models classify English descriptions of entities into fourteen categories from the DBpedia ontology. Reading comprehension Sep 2015 99.3% Not in cards
DiaBLa Models translate informal conversations between English and French while preserving meaning across turns. Translation May 2019 1.00 Not in cards
DRBench Agents investigate open-ended business questions across enterprise applications and produce reports. The evaluator checks whether the reports recover relevant information and support their claims. Deep research Sep 2025 63.8 Not in cards
EgyHellaSwag Models select a plausible ending for a situation described in Egyptian Arabic. The evaluation adapts HellaSwag to the dialect. Reasoning in many languages May 2025 40.2% Not in cards
Estonian Valence Corpus Models predict the emotional valence of Estonian news text using human annotations. Reading in many languages 59.9% Not in cards
Financial PhraseBank Models classify the financial sentiment of a news sentence as positive, negative, or neutral. Finance Jul 2013 95.2% Not in cards
Greek Legal Code Models classify Greek legal texts by subject using their place in the Greek Legal Code hierarchy. Law Sep 2021 56.5% Not in cards
H2R-Bench Models turn first-person demonstrations into videos of a specified robot performing the task. The benchmark checks contacts, object responses, and completion of the goal. Video generation Aug 2026 81.0 Not in cards
IberoBench Models perform language understanding and generation tasks in Iberian languages. The suite combines existing evaluations with new tests specific to each language. Reasoning in many languages Jan 2025 30.3 Not in cards
ICLayout-Bench Agents turn transistor netlists into chip layouts. Design tools check the layout rules and connectivity, then measure electrical performance and area. Electronics 80.3 Not in cards
IN22-Gen Models translate parallel sentences across English and Indian languages. Google evaluates whether an embedding model can match a sentence with its translation. Translation May 2023 99.5% Not in cards
INJONGO Models identify a speaker’s intent and extract information from utterances in 16 African languages. Reading in many languages Feb 2025 65.1% Not in cards
JSONSchemaBench Models or constrained decoders generate outputs under real JSON schemas. The evaluation tests schema coverage and valid generation. Following precise instructions Jan 2025 91.4% Not in cards
Manager Coercion Benchmark Agents manage a subordinate that refuses a routine assignment. The harness records how far the manager escalates on a nine-point coercion scale. Deception and alignment Jul 2026 3.23 Not in cards
MANTA (Animal Welfare) Models discuss animal welfare across five conversational turns. A judge measures whether they recognize the concern and maintain their position under pressure. Deception and alignment Apr 2026 0.76 Not in cards
MentalChat16K Counseling Evaluation Models answer 200 mental health questions from online forums. Judges rate each response from one to ten on seven counseling skills. Medicine Jan 2024 8 Not in cards
metabench Models answer selected items from established evaluations. The benchmark uses a sparse item set to estimate reasoning and knowledge with fewer questions. Composite indices Jul 2024 76.7 Not in cards
MoI-Bench Models generate and edit images for mobile users. The authors compare image quality across 64 subtasks and measure the cost of running models on phones. Image generation and editing Oct 2026 0.97 Not in cards
MultiMedia-TerminalBench Agents inspect and edit media in persistent terminal workspaces. The 105 tasks require verifiable output files across five areas of professional multimedia work. Video understanding May 2026 16.2% Not in cards
MultipanelVQA Models answer questions about images arranged into panels. The benchmark tests whether a model can locate the relevant panels and compare their contents. General visual questions Jan 2024 72.3% Not in cards
RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. Simulated robots Nov 2026 51.4% Not in cards
RTLLM 2.0 Models write Verilog for 50 digital designs from natural language specifications. Testbenches check the implementation before synthesis measures power, timing, and area. Electronics Oct 2024 60% Not in cards
SciMMIR Models match figures and tables from scientific papers with their descriptions. Documents, charts and text in images Jan 2024 3.35 Not in cards
ScreenASR-Bench Models transcribe speech using on-screen text, visible objects, or clues from the scene. The authors report word error rate and recall. Speech recognition and translation Sep 2026 3.47% Not in cards
SimulCost Agents tune physics simulations to meet an accuracy target. The authors measure success and computational cost for an initial guess and for repeated attempts with feedback. Research science Feb 2026 80.8% Not in cards
TrialDesignBench Models select a dose-finding design for an early oncology trial. The authors compare their choices with expert reference designs across 617 trials. Medicine Nov 2026 0.52 Not in cards
UA-Bench Models identify underwater sound sources and their subclasses from audio recordings. The authors repeat the test with silent audio to check reliance on sound. Audio understanding and voice Aug 2026 13.1% Not in cards
UniVerseBench Models answer questions about music across more than 38 cultural and linguistic groups. The authors report accuracy by language and skill. Audio understanding and voice Aug 2026 74.4% Not in cards
VTR-Bench Models generate videos containing visible text. The evaluation checks the rendered words and how well the text persists across frames. Video generation Oct 2026 0.89 Not in cards
Amazon Counterfactual Detection Models identify customer reviews that describe an alternative to what actually happened. Reading in many languages Apr 2021 93.2% Not in cards
BEAR Models choose which entity correctly completes a factual relation. The evaluation compares candidate completions across model architectures. Trivia and open questions Jun 2024 65.4% Not in cards
CareQA Models answer questions from Spanish specialist healthcare examinations. The evaluation includes English and Spanish versions and an open-response format. Medicine Feb 2025 76.9% Not in cards
CLEVA Models complete Chinese language tasks including text classification, instruction following, and classical Chinese comprehension. The evaluation samples questions for each round. Knowledge in many languages Aug 2023 0.96 Not in cards
CodeSearchNet Challenge Models retrieve code snippets using natural language queries. The challenge evaluates how closely ranked results match the search intent. Function-level problems Sep 2019 0.93 Not in cards
CodeTransOcean Models find equivalent code written in different languages or deep learning frameworks. The evaluation compares retrieved snippets with paired implementations. Function-level problems Dec 2023 36.6 Not in cards
Corpus for Emotion Detection in Russian Models classify Russian sentences by the emotion they express. The reference labels distinguish joy, sadness, surprise, fear, and anger. Reading in many languages 53% Not in cards
CoSQA Models retrieve code that matches a natural language query. Relevance judgments link the queries with code snippets. Function-level problems Aug 2021 0.51 Not in cards
CTKFacts Models compare Czech claims with evidence and predict whether the evidence supports, refutes, or fails to resolve each claim. Faithful to sources Jan 2022 0.89 Not in cards
DiagTrace Models request information about a patient one turn at a time. The evaluator compares the sequence of requests with the published case history. Medicine Nov 2026 0.40 Not in cards
Faroese Semantic Textual Similarity Models score how closely two Faroese sentences agree in meaning. The evaluation compares their rankings with reference judgments. Reading in many languages May 2023 0.87 Not in cards
FollowIR Models rank documents in response to a search instruction. The benchmark tests whether revising the instruction changes the ranking of the affected documents. Questions that need search Mar 2024 0.12 Not in cards
GreekBarRetrieval Models retrieve Greek statutory articles for bar exam questions and case facts. The authors compare ranked results with the statutes relevant to each case. Law Aug 2026 0.43 Not in cards
HMU-Bench Models answer Chinese and English questions about musical notation and recordings. The authors score responses against reference answers. Audio understanding and voice Oct 2026 2.47 Not in cards
IndicLID benchmark Models identify Indian languages in native scripts and in Romanized text. Reading in many languages May 2023 95.1% Not in cards
Italian-Administrative-CaseHold Models classify Italian administrative judgments paired with their official holdings by legal subject. Law 68.9% Not in cards
LegalQuAD Models find the German legal document that answers each of 200 questions. The original data also mark the answer text and its starting position. Law Aug 2021 0.70 Not in cards
LogicVal Models generate images from prompts with formal constraints on objects, attributes, counts, and relationships. The evaluator checks which constraints the image satisfies. Image generation and editing Oct 2026 50.9% Not in cards
ltzGLUE Models perform language understanding tasks in Luxembourgish, including entailment, grammar judgments, sentiment analysis, and named entity recognition. Reasoning in many languages Apr 2026 69.4 Not in cards
MatToolBench Agents use materials science software on Windows to operate interfaces, write scripts, and query databases. The evaluator awards partial credit for completed steps. Lab and data work Sep 2026 51.5% Not in cards
MatTools Models answer questions about materials science software and write Python programs to calculate material properties. The authors execute the generated programs. Lab and data work May 2025 81.6% Not in cards
Medbullets Models answer clinical questions patterned on US medical licensing examinations. The benchmark also evaluates explanations of the answers. Medicine Feb 2024 68.8% Not in cards
Mercury Models write Python solutions and pass hidden correctness tests. The evaluator compares runtime with reference solutions. Harbor ports all 256 problems from the evaluation split. Speed and optimization Feb 2024 0.59 Not in cards
MLRC-Bench Agents propose and implement methods for machine learning competitions. The scorer compares their performance with a supplied baseline and the strongest human solution. AI research tasks Apr 2025 6.26 Not in cards
MMLU-SR Models answer MMLU questions after key terms are replaced with synonyms. The benchmark tests whether models recognize the same concept under different wording. Exam questions Jun 2024 65% Not in cards
MotoVerse Models use a rider's camera to anticipate maneuvers and predict nearby road users' trajectories. The authors compare vision, language, and conventional prediction models. Embodied reasoning Nov 2026 46.3% Not in cards
MultiEURLEX Models assign subject labels to European Union legislation written in different languages. Law Sep 2021 0.05 Not in cards
NFCorpus Models rank medical research documents for health questions expressed in everyday language. Medicine Feb 2016 0.41 Not in cards
SemEval 2022 Task 8 multilingual news article similarity Models rate the similarity of two news articles, including articles in different languages. The evaluation compares predicted similarities with human judgments. Reading in many languages Jul 2022 0.72 Not in cards
SemEval 2024 Task 1 semantic textual relatedness Models rank sentence pairs by relatedness across languages. The evaluation compares their rankings with human judgments. Reading in many languages Mar 2024 0.65 Not in cards
T2LSC-Bench Models render text at a specified location in an image. Judges check the lettering and whether its meaning alters other objects or the background. Image generation and editing Sep 2026 99% Not in cards
VELOCITI Models distinguish a video's correct description from a caption that assigns an action to the wrong person. Video understanding Jun 2024 43.1% Not in cards
WorkSurface-Bench Agents choose documents, tables, or graphs to answer workplace questions. The benchmark scores where agents look, what evidence they find, and their answers. Documents and spreadsheets Jul 2026 10.3 Not in cards
AgenticSTS Agents play the Silent in Slay the Spire 2 at its lowest difficulty. The authors compare wins with different memory settings and two other agents. Video games Jun 2026 60% Not in cards
AutoBench (HDL testbench generation) Models write self-checking Verilog testbenches. The evaluator checks compilation, acceptance of correct designs, and agreement with reference testbenches on faulty designs. Electronics Jul 2024 71.1% Not in cards
BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. Real robots Oct 2026 0.56 Not in cards
CADExpert Models write CadQuery programs from dimensioned engineering drawings. The graders check whether each program runs and compare its 3D shape with the reference. CAD models Dec 2025 35.6% Not in cards
ChemPro Models answer chemistry questions across four difficulty levels. The authors report multiple-choice accuracy and accept numerical answers within specified tolerances. Science questions Feb 2026 38.7% Not in cards
CLIMATE-FEVER Models find evidence for climate claims and decide whether it supports or refutes each claim. Faithful to sources Dec 2020 0.47 Not in cards
CompMat-Bench Agents complete individual steps from published computational materials studies. Fixed rules compare their inputs and outputs with results reproduced by the authors. Lab and data work Sep 2026 91.5% Not in cards
DI-Bench Agents answer questions using company tables, documents, and business rules. The benchmark checks whether agents apply those rules when computing results. Running a business Sep 2026 57.9% Not in cards
EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. Knowledge in many languages Mar 2024 44% Not in cards
FML-bench Agents propose and run experiments on eight machine learning research tasks. The authors measure task performance and how widely the proposals vary across iterations. AI research tasks Oct 2025 0.50 Not in cards
GenPuzzle Models generate images that solve visual puzzles such as mazes, tangrams, and Sudoku. The authors select an automated judge using human reference scores. Image generation and editing Sep 2026 40.6 Not in cards
GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. Embodied reasoning Nov 2026 20.7% Not in cards
InstrumentalEval Models respond to scenarios where gaining resources or preserving themselves could serve another goal. A grader identifies responses that pursue those intermediate goals. Deception and alignment Feb 2025 43.4% Not in cards
IntPhys2-Timing Models identify when a physical rule breaks in a video. The authors add violation intervals to IntPhys2 and score temporal localization. Physical prediction Oct 2026 77.4% Not in cards
MEDEC Models identify and correct errors in clinical notes across five categories. The OpenMed port uses the MEDEC-MS subset. Medicine Dec 2024 69.1% Not in cards
MedLFQA Models write long answers to biomedical questions. The benchmark checks whether answers include required facts and avoid claims identified as incorrect. Medicine May 2024 77.1 Not in cards
Mind2Web-SC Models generate guardrails that check a web agent’s proposed actions against user constraints. The benchmark measures whether the guardrail permits or blocks each action correctly. Web browsing Jun 2024 93% Not in cards
Neural Activation Reading for Collusion Benchmark Monitors estimate whether committee members secretly coordinated their decisions. The transcript task measures how well probability estimates distinguish matched collusion and control cases. Deception and alignment Apr 2026 0.87 Not in cards
NitiBench Models retrieve Thai statutes and answer legal questions. The authors score retrieval separately and use a model judge to check coverage and contradictions. Law Feb 2025 70.3 Not in cards
NormAd-Eti Models decide whether behavior in an everyday story is socially acceptable in the stated country. Questions vary how much cultural guidance the model receives. Reasoning in many languages Apr 2024 81.8% Not in cards
OOLONG-Pairs Models identify pairs of users whose records satisfy a condition across a long input. The evaluation compares the predicted pairs with the reference set. Reasoning over long documents Dec 2025 92.9% Not in cards
OpinionQA Models answer public opinion survey questions. The evaluation compares model response distributions with those of demographic groups. Bias, fairness and politics Mar 2023 0.79 Not in cards
People Making Conversation Speech models continue real conversations at recorded pauses and turn endings. The benchmark compares their timing, response length, and recall with the original speakers. Speech recognition and translation Oct 2026 0.39 Not in cards
RAFT (Real-world Annotated Few-shot Tasks) Models learn text classification tasks from small labeled samples. The tasks use real documents, including customer messages, court decisions, and scientific abstracts. Reading comprehension Sep 2021 62.7% Not in cards
RMS-AQA Models identify sound events in domestic spaces, then answer questions about their positions and timing. The benchmark scores both stages. Audio understanding and voice Oct 2026 36.4% Not in cards
VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. Embodied reasoning Apr 2026 43.5% Not in cards
Almieyar Models transcribe newly recorded speech in 17 Arabic dialects. The authors report word and character error rates across six dialect families. Speech recognition and translation Sep 2026 34.9% Not in cards
BuiltBench Models match a building component description to the corresponding entry in a construction classification system. CAD models Nov 2024 65.3 Not in cards
CodePDE Models write a Python solver for each of five families of partial differential equations. Harbor tests each solver on 100 instances and checks its normalized error. Scientific code May 2025 38.1% Not in cards
CoVR-R Models find the video that matches a reference clip after applying a change described in text. Video understanding Mar 2026 47.6% Not in cards
DRAMA-Eval Models generate dialogue with several speakers across multiple turns. The authors check word accuracy and use a model judge to evaluate delivery in context. Speech recognition and translation Oct 2026 2.55% Not in cards
DuplexLedger Speech agents track facts that speakers introduce, revise, or leave unchanged during conversation. The evaluator asks which facts still apply. Speech recognition and translation Oct 2026 40.2% Not in cards
GeoFlip Models generate videos from paired scenes where a geometry change reverses the expected physical outcome. Model judges check the event and the scene structure. Video generation Oct 2026 3.17 Not in cards
GroundCocoa Models combine stated conditions with everyday knowledge to choose an answer. The questions test whether the model applies several constraints together. Language reasoning Apr 2024 41.9% Not in cards
Hack-Verifiable Terminal Bench Agents complete terminal tasks with access to restricted reference materials. The benchmark measures how often they consult those materials. Deception and alignment Aug 2026 10.1% Not in cards
HALAS Models transcribe earnings call segments with human annotations of hallucinations. NVIDIA measures word error and segments with inserted text. Speech recognition and translation Jun 2026 23.8% Not in cards
Histoires Morales Models read French stories about social norms, intentions and actions. The test asks them to distinguish actions or consequences appropriate to the situation. Deception and alignment Jan 2025 54.7% Not in cards
Inverse Scaling Prize Tasks Models answer questions designed to reveal cases where larger models perform worse. The suite includes hindsight bias and following instructions that conflict with familiar text. Language reasoning Jun 2023 78.2% Not in cards
JCoLA Models decide whether Japanese sentences are grammatically acceptable. Linguists supply judgments for sentences from textbooks and the research literature. Reasoning in many languages Feb 2022 54.5% Not in cards
Matter-of-Fact Models decide whether materials science claims are feasible using evidence available before a cutoff date. The authors compare their answers with later published findings. Research science Jun 2025 72% Not in cards
MedHEB Models retrieve related medical images, reports, and image regions. The authors evaluate how models rank relevant evidence across 107 retrieval tasks. Medicine Nov 2026 24.6 Not in cards
MulSeT Models answer spatial questions about direction, distance and masks across views. The paper studies single images, multiple views and video. Spatial reasoning and counting Sep 2025 36.7% Not in cards
NorSumm Models summarize Norwegian news articles in Bokmål or Nynorsk. Human summaries provide references in both written standards. Reading in many languages Jan 2025 10.2 Not in cards
OncoTriad-QA Models answer questions about cancer patients using radiology, pathology, and genomic evidence. The authors score multiple-choice answers and compare written responses with reference answers. Medicine May 2026 46% Not in cards
Putnam-AXIOM Models solve Putnam mathematics problems and equivalent variants. The evaluator checks final answers to test whether models generalize beyond familiar problem statements. Competition problems with answers May 2025 41.9% Not in cards
SIB-200 Models predict the topic of a sentence across 205 languages and dialects. The sentences come from FLORES-200. Reading in many languages Sep 2023 48.7% Not in cards
SugarCrepe Models choose between a caption that describes an image and a plausible caption with a changed detail. General visual questions Jun 2023 95.2% Not in cards
Talk to Act Models turn conversations into robot actions. Each predicted function call must identify the action and its arguments from the dialogue. Function calling Nov 2026 63.6% Not in cards
TurBLiMP Models compare Turkish sentence pairs that differ in grammaticality. The core tests cover 16 grammatical phenomena. Reasoning in many languages Jun 2025 93.6% Not in cards
ZhoBLiMP Models compare pairs of Chinese sentences that differ in grammatical acceptability. Accuracy measures how often the model prefers the grammatical sentence. Reading in many languages Nov 2024 87.2% Not in cards
AdSpark-Bench Models generate advertisement videos from product images and prompts. The benchmark checks whether products remain consistent and whether each shot conveys the requested selling point. Video generation Oct 2026 66.9 Not in cards
ANES survey item associations Models fill in one missing answer from a 2016 election survey using the respondent’s other answers. Researchers compare associations between the real and simulated answers. Social simulation Sep 2021 0.82 Not in cards
BioMaze Models answer questions about biological pathways under normal and altered conditions. The benchmark includes true-or-false questions and written responses. Genomics and life sciences Feb 2025 61.6% Not in cards
CIRR Models find an image that matches a reference photograph after applying a change described in text. General visual questions Aug 2021 22% Not in cards
Connected Robotics Evidence Graph Benchmark Models update beliefs as evidence arrives from connected robots and sensors. The evaluator checks the belief state after each observation across 350 scenarios. Embodied reasoning Nov 2026 80.1% Not in cards
COPAL-ID Models choose a plausible cause or consequence in Indonesian scenarios. The questions require familiarity with local customs and expressions. Reasoning in many languages Nov 2023 58.9% Not in cards
CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. Simulated robots Nov 2026 0.56 Not in cards
Crossmodal-3600 Models describe images in different languages or retrieve an image from a description. General visual questions May 2022 0.65 Not in cards
DeceptionBench (PKU Alignment) Models state a belief and then answer under social or strategic pressure. Judges measure how often their public answer contradicts their private reasoning. Deception and alignment May 2025 20.7% Not in cards
EuroSAT Models classify land use and land cover in Sentinel-2 satellite images. General visual questions Aug 2017 83.6% Not in cards
FGVC-Aircraft Models identify aircraft variants from photographs. Detection and captioning Jun 2013 39.7% Not in cards
Formal Logic Deduction Models decide whether facts prove or disprove a hypothesis, or leave it unresolved. They also generate the proof steps. Abstract puzzles and logic Aug 2023 75.8% Not in cards
HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. Simulated robots Nov 2026 18.8% Not in cards
HiACC Models transcribe adults and children switching between Hindi and English. The recordings include story reading and spontaneous answers. Speech recognition and translation May 2025 24.5% Not in cards
IMB Models answer Italian medical specialty exam questions and respond to patient questions from online forums. The paper reports results for both tasks. Medicine Oct 2025 71% Not in cards
LibriCount Models estimate how many speakers appear in a recording with overlapping voices. Speech recognition and translation Dec 2017 57.2% Not in cards
Mementos Models describe what happens across a sequence of images. The evaluation checks whether their descriptions capture the objects and actions. General visual questions Jan 2024 40.6% Not in cards
mFollowIR Models rank documents according to a search instruction across multiple languages. Questions that need search Jan 2025 5.40 Not in cards
MuseBench (Music Score and Performance Understanding) Models answer music theory questions, read piano scores, and interpret recorded performances. The authors report accuracy separately for text, images, and audio. Audio understanding and voice 52.8% Not in cards
NaijaSenti Models classify the sentiment of posts written in Hausa, Igbo, Nigerian Pidgin, and Yoruba. Reading in many languages Jan 2022 48.8% Not in cards
NepTTS-Bench Speech models read Nepali sentences that test pronunciation contrasts. Native speakers rate the recordings, and the authors compare those ratings with an automated metric. Speech recognition and translation Apr 2026 3 Not in cards
PatchCamelyon Models identify whether a microscope image patch contains metastatic tumor tissue. Medicine Jun 2018 69.3% Not in cards
PersonaShot Models generate multi-shot videos about a person. The evaluator checks character continuity, emotional changes, and camera choices within and between shots. Video generation Aug 2026 0.67 Not in cards
PROST Models predict how objects move or change in everyday physical situations described in text. The benchmark tests reasoning about space and time. Commonsense Jun 2021 30.1% Not in cards
QCircuitBench Models implement quantum algorithms, construct oracles, and synthesize circuits from written specifications. Harbor uses one instance from each of 28 algorithms. Scientific code Oct 2024 0.03 Not in cards
Research Decision Benchmark Agents choose a next step from an explicit description of a research project. Rule-generated cases test ten types of decisions about evidence and research strategy. Research science Oct 2026 36.4% Not in cards
RGraph-Eval Models generate robot manipulation videos. The evaluator compares objects, changing relationships, and three-dimensional motion paths with a reference demonstration. Video generation Oct 2026 0.89 Not in cards
VAHBench Agents turn short drama briefs into finished videos with specified characters, dialogue, and durations. The evaluator checks completeness, quality, reliability, and accuracy of production reports. Video generation Nov 2026 51.8 Not in cards
VCG+ Diverse Models answer questions about videos drawn from varied settings. The evaluation grades their ability to describe and explain video content. Video captioning Jun 2024 43.5 Not in cards
2018 n2c2 Cohort Selection for Clinical Trials Models read longitudinal patient notes and decide whether patients meet specified clinical trial eligibility criteria. Medicine Not in cards
3DPW Models recover a person's three-dimensional pose in outdoor video. Reference poses come from wearable sensors and a moving camera. Spatial reasoning and counting Sep 2018 Not in cards
ACI-Bench Models turn doctor-patient conversations into clinical visit notes. The original benchmark compares generated notes with clinician-authored references. Medicine May 2023 Not in cards
ActivityNet Captions Models match descriptions of events to segments of untrimmed videos. Meta evaluates retrieval in both directions. Video understanding May 2017 Not in cards
ADE Corpus Models identify adverse drug effects in medical case reports. The OpenMed port combines classification and relation examples and reserves 20% for evaluation. Medicine Oct 2012 Not in cards
AfriQA Models retrieve evidence across languages to answer questions written in African languages. Reading in many languages May 2023 Not in cards
AfriSenti Models classify sentiment in human-annotated social media posts across 14 African languages. Reading in many languages Feb 2023 Not in cards
Aligned Scores and Performances for Piano Transcription Models turn piano performances into written scores. The reference scores let researchers assess pitches, voices, and the positions of notes within bars. Audio understanding and voice Oct 2020 Not in cards
Amharic Speech Emotion Dataset Models identify the emotion expressed in a short Amharic recording. The speakers use four regional dialects. Audio understanding and voice Nov 2021 Not in cards
AmsterTime Models match historic photographs of Amsterdam with contemporary street views. The benchmark measures recognition of the same place across time. General visual questions Mar 2022 Not in cards
ANES presidential vote choice Models use survey profiles to predict which major party candidate each respondent backed in three US presidential elections. Researchers measure agreement with reported votes. Social simulation Sep 2021 Not in cards
Animal Kingdom Models identify actions in wildlife videos that can show several behaviors at once. The benchmark also tests pose estimation and locating events described in text. Video understanding Apr 2022 Not in cards
ArguAna Counterargs Models retrieve a counterargument for a given argument. The evaluation checks where the annotated counterargument appears in the ranking. Reading comprehension Not in cards
ARPA Armenian Paraphrase Corpus Models decide whether two Armenian sentences express the same meaning. Reading in many languages Sep 2020 Not in cards
ASCEND Models transcribe conversations that switch between Mandarin and English. The evaluation accounts for Chinese characters and English words. Speech recognition and translation Dec 2021 Not in cards
ASDiv Models solve English arithmetic word problems with varied phrasing. The reference answers identify the expected numerical result. Word problems Jul 2020 Not in cards
ASFX-SED Models locate sound effects in audio recordings. Meta reports area under the receiver operating characteristic curve for event localization. Audio understanding and voice Jul 2025 Not in cards
ASSIN Models judge whether one Portuguese sentence entails another and rate the similarity of the pair. The first edition uses news from Brazil and Portugal. Reasoning in many languages Jul 2016 Not in cards
AstroLLaVA image and text retrieval Models match astronomical images with descriptions. Google evaluates whether an image retrieves the matching description among its top five results. General visual questions Apr 2025 Not in cards
AstroMemBench Spacecraft agents use information from earlier observations to complete simulated missions. The environment scores task completion, safety, and resource use. Embodied reasoning Nov 2026 Not in cards
AudioCaps Models match natural language descriptions to audio clips. Meta also compares how accurately encoders match audio, video, and text from the same clip. Audio understanding and voice Not in cards
AudioMNIST Models identify a spoken digit from a recording. The test includes recordings from different speakers. Speech recognition and translation Jul 2018 Not in cards
AudioSep Eval Benchmark Models isolate a sound from an audio mixture using a caption or a sound label. Scores compare the separated waveform with the original sound. Audio understanding and voice Aug 2023 Not in cards
bAbI QA Tasks Models answer questions about short synthetic stories. The tasks isolate skills such as tracking objects, combining supporting facts and reasoning about locations. Language reasoning Feb 2015 Not in cards
Backtracing Models identify the sentence that prompted a student's question or a news reader's curiosity. In conversations, they find the turn that prompted an emotional response. Finding facts in long inputs Mar 2024 Not in cards
BanglaSER Models identify the emotion in Bangla sentences recorded on phones and laptops. Audio understanding and voice Mar 2022 Not in cards
BasqueGLUE Models analyze Basque text for tasks including sentiment, topic classification, coreference and entailment. Each task has its own evaluation metric. Reading in many languages Jun 2022 Not in cards
BelarusianGLUE Models classify Belarusian sentiment, recognize entailment, judge grammar, and distinguish word meanings in context. Reading in many languages Jul 2025 Not in cards
bgGLUE Models perform language understanding tasks in Bulgarian, including sentiment analysis on movie reviews and question answering on high school exams. Reading in many languages Jun 2023 Not in cards
BHS Syntactic Evaluation Models compare sentences in Basque, Hindi and Swahili. The test checks whether the model prefers the grammatical member of each pair. Reasoning in many languages Jan 2025 Not in cards
Bingo (Visual Hallucination) Models answer questions about images. The benchmark tests whether prior expectations or interference from other visual information cause false answers. Hallucination rates Nov 2023 Not in cards
BiodivNER Models identify and label phrases in biodiversity abstracts and dataset descriptions. Annotators distinguish six kinds of entities, including organisms, locations, and environments. Lab and data work Apr 2022 Not in cards
BioVITA retrieval benchmark Models match animal images, audio recordings, and descriptions. The benchmark checks whether the retrieved items match the reference species, genus, or family. Genomics and life sciences Mar 2026 Not in cards
BirdSet Models identify bird species from audio recordings across multiple evaluation datasets. Audio understanding and voice Mar 2024 Not in cards
Birdsnap Models identify the North American bird species shown in a photograph. Detection and captioning Jun 2014 Not in cards
BSNLP 2019 Shared Task Models find, classify, normalize, and link named entities across Slavic languages. Reading in many languages Aug 2019 Not in cards
BUCC 2018 bitext mining shared task Models find translation pairs in two collections of sentences. The evaluation compares the retrieved pairs with a reference list. Translation Not in cards
BURST Models recognize, segment, and track objects across video frames. The benchmark tests how well they associate the same object across time. Video understanding Sep 2022 Not in cards
CADmium Fusion 360 Reconstruction Models reconstruct CAD parts from text descriptions of Fusion 360 designs. The authors measure shape distance and count invalid outputs. CAD models Jul 2025 Not in cards
CADPrompt Models write CadQuery programs from text descriptions of mechanical parts. The authors align each generated shape with its reference and measure point cloud distance. CAD models Oct 2024 Not in cards
Casual Conversations v2 Models classify attributes of speakers in recorded interviews. The HELM audio tasks compare age and gender predictions with participants’ own labels. Audio understanding and voice Mar 2023 Not in cards
Catalonia Independence Corpus Models identify an author's stance on Catalan independence from posts in Catalan and Spanish. Reading in many languages May 2020 Not in cards
ChiPBench (chip placement) Agents place circuit blocks in chip designs. OpenROAD measures timing, power, and area after routing. Higher timing slack is better. Electronics Jul 2024 Not in cards
CivilComments-WILDS Models classify the toxicity of online comments. The benchmark evaluates how accuracy varies across groups mentioned in the text. Bias, fairness and politics Dec 2020 Not in cards
CLEAR (Clinical Information Extraction) Models read patient notes and classify a medical condition as present, absent, or uncertain. Human reviewers supply the reference labels. Medicine Jan 2025 Not in cards
CLIP representation bias evaluation Models associate images of everyday objects with gender terms. The evaluation measures how far the associations depart from an equal preference across groups. Bias, fairness and politics Mar 2024 Not in cards
Clotho Models write a caption describing what can be heard in an audio recording. Audio understanding and voice Oct 2019 Not in cards
Clotho OVL-REL Test Set Models predict listener ratings for audio quality and caption relevance. Four generators produced environmental sounds from 100 captions drawn from Clotho. Audio understanding and voice Nov 2025 Not in cards
COCO-Stuff 164K Models label each pixel with an object or background category. The benchmark adds annotations for regions such as sky, grass, and pavement. Detection and captioning Not in cards
ColDeR Models rank passages containing relevant evidence against alternatives that exploit shortcuts such as length, repetition, or word overlap. Questions that need search Mar 2025 Not in cards
Common Syntactic Processes in Text-to-Image Generation Models generate images from sentences with different grammatical structures. Evaluators check whether each image preserves the relationships expressed in the sentence. Image generation and editing Oct 2022 Not in cards
CORE English web register identification Models label English web documents by register, such as news reporting, discussion, or instructions. A document can have several labels. Researchers report F1. Reading comprehension Aug 2022 Not in cards
CREF Models submit forecasts before the target period begins and receive scores after the actual values become available. Real-world events Nov 2026 Not in cards
CrisisMMD Models classify disaster photographs and their accompanying posts for humanitarian use. General visual questions May 2018 Not in cards
CrossSum Models read a news article in one language and summarize it in another. The dataset aligns articles and summaries across more than 1,500 language pairs. Reading in many languages Dec 2021 Not in cards
CS-FLEURS Models transcribe speech that switches languages within sentences. Four test sets cover 113 language pairs. People read one set aloud, and synthesizers generate the others. Speech recognition and translation Jul 2025 Not in cards
CTI-to-MITRE Models classify a threat description using MITRE ATT&CK technique labels. The HELM scenario presents a set of candidate labels. Secure code and review Aug 2022 Not in cards
DACSA Models summarize Catalan and Spanish newspaper articles. Editors’ summaries provide the references. Reading in many languages Jul 2022 Not in cards
DaLAJ Models judge the linguistic acceptability of Swedish sentences. Reading in many languages May 2021 Not in cards
DaNE Models locate and classify names in Danish sentences drawn from the Danish Universal Dependencies treebank. Reading in many languages May 2020 Not in cards
DANSK Models identify named entities across Danish text domains. The authors annotate a subset of Danish Gigaword using 18 entity types. Reading in many languages Dec 2024 Not in cards
DarijaBench Models translate, summarize and classify sentiment in Moroccan Arabic. A transliteration task also tests conversion between Arabic and Latin scripts. Reading in many languages Sep 2024 Not in cards
DarijaHellaSwag Models read a situation in Moroccan Arabic and choose its most plausible continuation. Reasoning in many languages Sep 2024 Not in cards
DarijaMMLU Models answer academic multiple-choice questions in Moroccan Arabic. The evaluation adapts MMLU to the dialect. Knowledge in many languages Sep 2024 Not in cards
Dementia Family Caregiver Detection Models identify English posts whose authors report having a family member with dementia. Organizers score F1 for that class on 1,769 test posts. Medicine May 2022 Not in cards
Describable Textures Dataset Models assign a texture description to a photograph based on how its surface looks. General visual questions Nov 2013 Not in cards
DiDeMo Models locate the time interval in a video that matches a natural language description. Video understanding Aug 2017 Not in cards
DIHARD III Models identify who speaks when across eleven recording domains. The benchmark includes overlapping speech. Audio understanding and voice Sep 2020 Not in cards
DIODE Models estimate depth in indoor and outdoor scenes. The dataset provides depth measurements from a laser scanner for comparison. Spatial reasoning and counting Aug 2019 Not in cards
DIOR Models locate and classify objects in remotely sensed images. The benchmark measures detection accuracy across different object sizes and backgrounds. Detection and captioning Aug 2019 Not in cards
Discharge Me! Models draft discharge instructions and brief hospital course summaries from patient records. The shared task compares model output with clinical discharge documentation. Medicine Not in cards
Diving48 Models classify competitive dives from videos by distinguishing their movement sequences. Video understanding Sep 2018 Not in cards
Dutch Model Benchmark Models perform Dutch language understanding tasks, including reasoning about the causes and effects of events. Reasoning in many languages Dec 2023 Not in cards
ECHR Binary Violation Classification Models read the facts of a European Court of Human Rights case and predict whether the court found a violation. Law Jun 2019 Not in cards
ECHR Legal Corpus for Argument Mining Models identify argumentative clauses, link related clauses, and distinguish premises from conclusions in European Court of Human Rights rulings. Law Dec 2020 Not in cards
EDIS Models retrieve news images and their associated text using queries that refer to specific people, places, or events. General visual questions May 2023 Not in cards
EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. Other professions Oct 2026 Not in cards
Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. Simulated robots Nov 2026 Not in cards
EgoCross Models answer questions about events in first-person videos. The challenge scores accuracy on 957 four-choice questions across four domains. Video understanding Aug 2025 Not in cards
EgoPER Error Detection Models watch first-person cooking videos and flag incorrect steps. The evaluator checks accuracy by step and measures error detection by frame. Video understanding Dec 2023 Not in cards
EgyMMLU Models answer multiple-choice subject questions in Egyptian Arabic. The evaluation tests knowledge in a dialect underrepresented in standard Arabic exams. Knowledge in many languages May 2025 Not in cards
EHRSHOT Models predict clinical outcomes from longitudinal electronic health records with few labeled examples for each task. Medicine Jul 2023 Not in cards
EHRSQL Models convert clinical questions into SQL queries over electronic health records. The benchmark includes questions that the database cannot answer. Data science and SQL Jan 2023 Not in cards
ELLIPSE Holistic Essay Scoring Models assign one overall English proficiency score to each essay on a five-point scale. The reference is the overall human grade. Other professions Feb 2023 Not in cards
EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. Simulated robots Nov 2026 Not in cards
EmpatheticDialogues Models reply to a person describing an emotional experience. Human conversations provide reference responses for evaluating empathy and relevance. Multi-turn conversation Nov 2018 Not in cards
ETH3D Models reconstruct scene geometry from images. Meta evaluates depth predictions against reference measurements on the ETH3D test scenes. Spatial reasoning and counting Not in cards
EUR-Lex-Sum Models summarize European Union legislation in the language of the source document. Official summaries provide references across EU languages. Law Oct 2022 Not in cards
Event-based Segmentation Dataset Models segment objects in cluttered indoor scenes filmed from a moving robot arm. The benchmark provides reference masks for comparison. Video understanding Feb 2023 Not in cards
EVVE Models retrieve other videos showing the same real-world event as a query video. Video understanding Jun 2013 Not in cards
FairFace Models classify face images using the dataset’s annotated age, gender, and race categories. The benchmark compares recognition performance across demographic groups. General visual questions Aug 2019 Not in cards
FaithDial Models produce conversational answers grounded in supplied knowledge. Faithful to sources Apr 2022 Not in cards
Fashion IQ Models retrieve a garment using a reference photograph and text describing how the target garment differs. General visual questions May 2019 Not in cards
FeedbackQA Models answer questions using a document collection and learn from written feedback about earlier answers. Questions that need search Apr 2022 Not in cards
Flickr30k Models describe photographs of people, objects, and activities. Meta compares model captions with the human reference captions. Detection and captioning Not in cards
FoNE Models identify named entities in Faroese news articles. The authors manually verified the automatically assigned labels. Reading in many languages May 2023 Not in cards
FORB Models retrieve other images of the same flat object despite changes in viewpoint or lighting. General visual questions Sep 2023 Not in cards
FQuAD Models extract answers to human-written questions from French Wikipedia articles. Reading in many languages Feb 2020 Not in cards
FQuAD 1.1 Models extract answers to French questions from Wikipedia passages. The authors expanded the dataset from over 25,000 questions to over 60,000. Reading in many languages May 2020 Not in cards
FQuAD 2.0 Models answer questions about French passages and identify questions that the passage cannot answer. Reading in many languages Sep 2021 Not in cards
FrenchBench (CroissantLLM) Models answer questions and perform language understanding and generation tasks in French. The suite combines existing tests with manually annotated material. Reading in many languages Feb 2024 Not in cards
FSD50K Models identify the sound events present in a Freesound recording. Audio understanding and voice Oct 2020 Not in cards
FSDKaggle2018 Models assign a sound event label to a Freesound recording. Audio understanding and voice Mar 2018 Not in cards
FSDKaggle2019 Models identify all relevant sound events in an audio clip using training labels of varying reliability. Audio understanding and voice Jun 2019 Not in cards
GeoSound Models match environmental recordings to satellite images of their locations and retrieve recordings from images. The test holds out geographic areas and contains 9,931 examples. Audio understanding and voice Aug 2024 Not in cards
German Traffic Sign Recognition Benchmark Models identify the class of a traffic sign in a cropped road image. Detection and captioning Dec 2010 Not in cards
GermEval 2014 Named Entity Recognition Models find and classify names in German news and Wikipedia text, including nested and derived entities. Reading in many languages May 2014 Not in cards
Gold Commodity News Models classify news headlines about gold. Labels distinguish price direction, timing, and comparisons with other assets. Finance Sep 2020 Not in cards
GRAFT Models predict plant traits from gene expression in Arabidopsis. The authors compare regression and graph models against measurements from the same plants. Genomics and life sciences Jun 2026 Not in cards
GraspBench Robots propose grasps and try to lift objects in MuJoCo. Each attempt must pass motion planning and execution on the chosen robot. Simulated robots Nov 2026 Not in cards
HAE-RAE Bench 1.0 Models answer Korean questions about vocabulary, history, general knowledge, and reading comprehension. The original evaluation has six tasks. Knowledge in many languages May 2023 Not in cards
HAGRID Models answer information-seeking questions and attribute their claims to supporting passages. Faithful to sources Jul 2023 Not in cards
Hallmarks of Cancer Models label biomedical text with the cancer hallmarks it describes. The hub port assigns zero or more of ten labels to individual sentences. Genomics and life sciences Oct 2015 Not in cards
Harmony4D Models recover the body shapes and poses of people interacting at close range. The benchmark includes sports where people occlude each other. Spatial reasoning and counting Oct 2024 Not in cards
Hateful Memes Challenge Models decide whether a meme is hateful by interpreting its image and caption together. Paired examples change one modality while preserving the other. Bias, fairness and politics May 2020 Not in cards
HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. Simulated robots May 2026 Not in cards
HMDB51 Models identify human actions in short video clips across 51 categories. Video understanding Nov 2011 Not in cards
HoloAssist Mistake Detection Models watch a person perform a task and classify each annotated action as correct or mistaken. The challenge reports an F-score. Video understanding Sep 2023 Not in cards
Hotter and Colder Models predict sentiment and other attributes of Icelandic blog text using crowdsourced judgments. Reading in many languages Feb 2025 Not in cards
ImageCoDe Models select the image that matches a description from a set of visually similar alternatives. General visual questions Mar 2022 Not in cards
ImageNet-A Models classify real photographs that commonly fool image classifiers. The test uses unmodified images from ImageNet categories. General visual questions Jul 2019 Not in cards
ImageNet-R Models classify renditions of objects in drawings, sculptures, and other media. The benchmark measures recognition when appearance differs from ordinary photographs. General visual questions Jun 2020 Not in cards
ImageNet-Sketch Models classify black-and-white sketches using ImageNet category labels. The benchmark tests recognition when images lack photographic texture and color. General visual questions May 2019 Not in cards
ImageNetV2 Models classify a newly collected set of photographs using the original ImageNet labels. The test measures how accuracy changes on fresh images. General visual questions Feb 2019 Not in cards
Inappropriate Image Prompts Models generate images from prompts drawn from real users. The evaluation measures how often the outputs contain inappropriate material. Refusing harmful requests Nov 2022 Not in cards
iNaturalist Sounds Models identify animal species from recordings submitted to iNaturalist. Audio understanding and voice May 2025 Not in cards
IncompeBench Models retrieve music clips that match detailed natural language requests. Audio understanding and voice Feb 2026 Not in cards
IndicXNLI Models compare a premise and hypothesis in Indian languages. They predict whether the premise entails, contradicts or leaves the hypothesis unresolved. Reasoning in many languages Apr 2022 Not in cards
IndoNLI Models decide whether a hypothesis follows from an Indonesian passage using annotations from experts and other speakers. Reading in many languages Nov 2021 Not in cards
Inspect India Evals Models answer questions about Indian culture and reason across Indian languages. Other tasks test social bias, digital public infrastructure safeguards and refusal behavior. Knowledge in many languages Jul 2026 Not in cards
InstructIR Models find documents that satisfy a user's search request and its accompanying instructions. Questions that need search Feb 2024 Not in cards
iSAID Models detect and segment individual objects in aerial images. The benchmark measures how well predicted masks match annotated object instances. Detection and captioning May 2019 Not in cards
JamendoLyrics Models locate the start of each sung word using a recording and its lyrics. The original evaluation contains 20 English songs. Speech recognition and translation Feb 2019 Not in cards
Jamp Models infer temporal relationships between Japanese sentences. Separate test splits vary the templates, time formats, and time spans. Reasoning in many languages Jun 2023 Not in cards
Japanese Tort-case Dataset Models predict whether a Japanese court found a tort from the facts and the parties' arguments. They also label which arguments the court accepted. Law May 2024 Not in cards
JEMHopQA Models answer Japanese questions that require combining facts across multiple documents and provide the supporting reasoning. Reading in many languages Jul 2023 Not in cards
K-MHaS Models assign hate speech labels to Korean news comments. A comment can receive more than one label. Bias, fairness and politics Oct 2022 Not in cards
Korean Canonical Legal Benchmark Models answer Korean legal questions with supporting precedents. The benchmark includes multiple-choice questions and essays with individual grading criteria. Law Oct 2025 Not in cards
KPI-EDGAR Models identify key performance indicators and related values in financial reports. The original annotations also link entities within the text. Finance Oct 2022 Not in cards
KuralHub Models classify emotions in speech from 29 languages. Audio understanding and voice Feb 2025 Not in cards
KurdiSent Models predict the sentiment expressed in Kurdish text. Reading in many languages Not in cards
Large Movie Review Dataset Models classify an IMDb movie review as positive or negative. Reading comprehension Jun 2011 Not in cards
Legal Opinion Sentiment Classification Models classify the sentiment expressed in passages from legal opinions. Expert annotations provide the reference labels. Law Oct 2020 Not in cards
LEVIR-CDC Models describe changes between aerial images and mark changed buildings. The authors paired LEVIR-CC captions with binary building masks. Detection and captioning Oct 2024 Not in cards
LEXTREME Models analyze legal text across languages. Tasks include predicting document labels, identifying legal arguments, and labeling words within documents. Law Jan 2023 Not in cards
LightGenBench Models predict which parts of a 3D surface emit light and what color they emit. The authors compare generated emission with reference assets. Image generation and editing Oct 2026 Not in cards
LIMIT Models retrieve specific pairs of documents from a shared collection using embeddings. Questions that need search Aug 2025 Not in cards
LitSearch Models retrieve relevant research papers in response to questions about the scientific literature. Questions that need search Jul 2024 Not in cards
LLMEval-Med Models answer clinical questions drawn from patient records and expert scenarios. A model grades answers against physician-developed checklists. Medicine Jun 2025 Not in cards
LM-SynEval Models compare English sentences that differ in grammaticality. The test measures preferences for correct agreement and other syntactic dependencies. Language reasoning Oct 2018 Not in cards
Long-term Video Object Segmentation v2 Models track object masks through long videos. The evaluation tests whether they can recover an object's identity after an extended absence. Video understanding Apr 2024 Not in cards
LoTTE Models retrieve passages that answer questions about less common topics. Questions that need search Dec 2021 Not in cards
LoveDA Models label land cover in aerial images of urban and rural areas. The benchmark supports comparisons across these two settings. Detection and captioning Oct 2021 Not in cards
LVIS Models locate objects and outline each instance across a large vocabulary of categories. Meta reports box and mask average precision. Detection and captioning Aug 2019 Not in cards
MacroLens Models use company and economic data to make financial estimates under different scenarios. The authors compare forecasting models with language models and simple baselines. Finance Jun 2026 Not in cards
MAFAND-MT Models translate news text in a benchmark covering 21 African languages. Translation Jul 2022 Not in cards
MagnaTagATune Models predict descriptive tags for music clips. Qwen compares their predictions with human annotations using micro F1. Audio understanding and voice Not in cards
Maltese News Categories Models assign one or more topic labels to Maltese news articles. Reading in many languages May 2024 Not in cards
MammalNet Models identify mammals and their behavior in videos. The benchmark covers 173 mammal categories and 12 behaviors, including nursing and hunting. Video understanding Mar 2023 Not in cards
ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. Simulated robots Nov 2026 Not in cards
MasakhaNER Models recognize named entities in ten African languages. The authors compare supervised and cross-language transfer methods. Reading in many languages Mar 2021 Not in cards
MasakhaNER 2.0 Models identify named entities in human-annotated text across 20 African languages. Reading in many languages Oct 2022 Not in cards
MasakhaNEWS Models predict the topic of news articles written in African languages. Reading in many languages Apr 2023 Not in cards
MC-TACO Models judge plausible answers to questions about when events happen, how often they recur and how long they last. Commonsense Sep 2019 Not in cards
MedAlign Models follow clinician-written instructions using a patient’s electronic medical record. Clinicians evaluate the responses. Medicine Aug 2023 Not in cards
MedDialog Models generate a clinician’s response from a patient description. The harness also evaluates summaries of medical questions using reference text. Medicine Nov 2020 Not in cards
MEDIQA 2019 Question Answering Models answer consumer health questions or assess candidate answers. The shared task evaluates the relevance and quality of answers to medical questions. Medicine Aug 2019 Not in cards
MedPT Models classify Brazilian Portuguese medical questions by specialty. The original evaluation tests 20 classes, while the hub creates a split after deduplication and label selection. Medicine Nov 2025 Not in cards
MELA Models judge whether sentences are linguistically acceptable across languages. The evaluation compares their judgments with the labeled examples. Reasoning in many languages Nov 2023 Not in cards
MELD Models identify emotions in utterances from multiparty conversations. The dataset includes speech, video, and text. Audio understanding and voice Oct 2018 Not in cards
MeQSum Models condense consumer health questions into shorter questions while preserving the medical information needed for retrieval. Reference summaries provide the comparison. Medicine Jul 2019 Not in cards
MIMIC-IV-BHC Models summarize a patient’s hospital course from clinical notes. The reference is the brief hospital course section of the discharge summary. Medicine Dec 2024 Not in cards
MInDS-14 Models identify the banking service a caller wants from a spoken request. Speech recognition and translation Apr 2021 Not in cards
mMARCO Models rank passages by relevance to a query. The multilingual benchmark translates the MS MARCO passage ranking collection. Reading in many languages Aug 2021 Not in cards
MMDocIR Models find relevant pages or layout elements inside long documents that contain both text and visual material. Documents, charts and text in images Jan 2025 Not in cards
MMGenre Models synthesize singing in different genres. The authors measure vocal quality, lyric errors, and how well the singing conveys the intended genre. Audio understanding and voice Jul 2026 Not in cards
Moral Stories Models reason about norms, intentions, actions and consequences in short stories. Tasks include distinguishing a norm-respecting action from a norm-violating alternative. Commonsense Nov 2021 Not in cards
MOSE Models follow and segment objects in videos with crowds, occlusions, and objects leaving the screen. The benchmark compares masks against human annotations. Video understanding Feb 2023 Not in cards
MOSEv2 Models track and segment objects in videos with complex scenes. Objects may disappear, reappear, or blend into a crowd. Video understanding Aug 2025 Not in cards
mRobust04 Models rank documents for translated queries from the TREC Robust04 collection. Relevance judgments support comparisons across languages. Reading in many languages Sep 2022 Not in cards
MTS-Dialog Models write a clinical note section from a conversation between a doctor and a patient. The evaluation compares it with the reference note. Medicine May 2023 Not in cards
Multi-Scenario Anomaly Detection Models score video frames for unusual events across 14 settings. The two protocols hold out different videos and score frame-level detection. Video understanding Jun 2024 Not in cards
Multilingual HateCheck Models distinguish hate speech from statements that quote, reject, or otherwise resemble hate speech. The test covers multiple languages. Bias, fairness and politics Jul 2022 Not in cards
MUSEBENCH Models detect computer-generated music and identify artifacts. The authors compare eleven detectors on original recordings and on transformed tracks. Audio understanding and voice Oct 2026 Not in cards
Music Understanding Benchmark Models estimate tempo, key, meter, and song structure from recordings with identifying metadata removed. The evaluator compares predictions with existing annotations. Audio understanding and voice Oct 2026 Not in cards
Musical Expectation Benchmark Models predict how surprising a musical passage will sound. The benchmark compares predictions from scores or recordings with perceptual measurements. Audio understanding and voice Nov 2026 Not in cards
MUStARD Models determine whether a speaker is being sarcastic in a television clip. HELM evaluates the audio track. Audio understanding and voice Jun 2019 Not in cards
MuTox Models listen to short speech clips in 30 languages and classify whether they contain toxic speech. The harness compares their decisions with human labels. Audio understanding and voice Jan 2024 Not in cards
MuTual Models read a multi-turn dialogue and select the response that follows from the speakers’ exchange. The alternatives require reasoning about the conversation. Multi-turn conversation Jul 2020 Not in cards
NaijaRC Models answer multiple-choice reading comprehension questions from high school exams in three Nigerian languages. Reading in many languages Aug 2023 Not in cards
Natural Adversarial Objects Models identify objects in unmodified photographs that caused errors in earlier detectors. Human annotations identify the objects. Detection and captioning Nov 2021 Not in cards
NAVI Models match points on the same object across photographs taken from different viewpoints. Reference geometry supplies the correct correspondences. Spatial reasoning and counting Jun 2023 Not in cards
NevIR Models distinguish documents whose meanings differ because one passage negates a statement. Questions that need search May 2023 Not in cards
NewsQA Models extract an answer from a CNN news article in response to a question written by a reader. Reading comprehension Nov 2016 Not in cards
NLEBench Models generate Norwegian text for tasks including reading comprehension and summarization. The benchmark combines translated and human-annotated material. Reading in many languages Dec 2023 Not in cards
nocaps Models caption photographs that contain objects outside the caption training vocabulary. The benchmark compares generated captions with human references. Detection and captioning Dec 2018 Not in cards
NollySenti Models classify the sentiment of Nollywood movie reviews in five languages spoken in Nigeria. Reading in many languages May 2023 Not in cards
Nufo Models transcribe Ewe speech from a fixed test split. The authors compare transcriptions with reference text using word error rate. Speech recognition and translation Oct 2026 Not in cards
ObjectNet Models identify familiar objects photographed with controlled changes in viewpoint, orientation, and background. Meta reports results on classes shared with ImageNet. General visual questions Not in cards
OPT-RSVG Models use a written description to locate an object in an aerial image. The test pairs 24,477 descriptions with reference boxes. Detection and captioning Jul 2023 Not in cards
OPUS-100 Models translate between English and other languages. Separate test sets measure performance on supervised directions and language pairs without direct training data. Translation Jul 2020 Not in cards
OVEN Models identify the Wikipedia entity that answers a question about an image. General visual questions Feb 2023 Not in cards
PaintSkills Models generate images of specified objects, counts, and spatial relationships. Detectors check whether an image satisfies the requested composition. Image generation and editing Feb 2022 Not in cards
ParsiNLU Models read Persian passages, compare queries, and identify sentiment. The six tasks also cover entailment, multiple-choice questions, and translation. Reading in many languages Oct 2020 Not in cards
PAWS-X Models decide whether two sentences with similar wording express the same meaning across languages. Reading in many languages Aug 2019 Not in cards
PersistBench Models answer requests using stored memories. A judge measures inappropriate disclosure, agreement with false user beliefs and failure to use relevant memories. Deception and alignment Feb 2026 Not in cards
PHINC Models translate social media posts that mix Hindi and English into English. Translation Apr 2020 Not in cards
PhoMT Models translate between Vietnamese and English. Held-out sentence pairs provide reference translations in both directions. Translation Nov 2021 Not in cards
Physiome-ODE Models forecast irregularly sampled biological time series generated from published differential equations. The authors compare forecasts using mean squared error. Genomics and life sciences Feb 2025 Not in cards
Plain English Summarization of Contracts Models summarize contract clauses in plain English. Human summaries provide references for evaluating how well each response conveys the legal language. Law Jun 2019 Not in cards
Poem Sentiment Models judge the sentiment expressed in individual verses of English poems from Project Gutenberg. Creative writing Dec 2020 Not in cards
PolEmo 2.0 Models classify the sentiment of Polish consumer reviews. Separate test sets measure performance in familiar and unfamiliar domains. Reading in many languages Nov 2019 Not in cards
PsyTAR Models identify adverse reactions in patient reviews of four antidepressants. The original corpus also labels withdrawal symptoms, reasons for treatment, and reported effectiveness. Medicine Jan 2019 Not in cards
QAG-360K Models compare two satellite images, answer questions about land cover, and mark the relevant pixels. The authors score answer accuracy and mask overlap separately. Spatial reasoning and counting Oct 2024 Not in cards
RaceBias (Race-Based Medicine) Models answer clinical questions used to identify unsupported race-based medical beliefs. Evaluators check responses for those beliefs. Bias, fairness and politics Oct 2023 Not in cards
ReactSim-Bench Driving simulators predict how traffic responds when the vehicle changes its behavior. The authors test those reactions in 2,636 scenarios. Simulated robots Jun 2026 Not in cards
RealCommBench Models distinguish human and synthetic speech after transmission over real phone calls. Paired recordings measure how the call changes detection error. Speech recognition and translation Oct 2026 Not in cards
RealEstate10K Models estimate relative camera poses across frames from real estate videos. The benchmark supplies camera trajectories for comparison. Spatial reasoning and counting Not in cards
Ref-ZOM Models mark the pixels described by a phrase in a COCO image. A phrase can refer to zero, one or several objects. Detection and captioning Oct 2023 Not in cards
RegexEval Models turn natural-language descriptions into regular expressions. The dataset contains 762 patterns with matching and nonmatching examples. The hub port holds out 62 patterns. Function-level problems Nov 2023 Not in cards
RELATE Models listen to an audio clip and predict how well it matches a caption. Human listeners also judge which sounds occur and in what order. Audio understanding and voice Jun 2025 Not in cards
ReMuQ Models retrieve a knowledge passage that answers a question about an image. General visual questions Jun 2023 Not in cards
Revisited Oxford and Paris Models retrieve photographs of the same landmark from Oxford and Paris image collections. The revised benchmark defines medium and hard evaluation protocols. General visual questions Mar 2018 Not in cards
RMU:ECHR Models mark argument spans in European Court of Human Rights rulings and identify the type of argument and the party making it. Law Aug 2022 Not in cards
RSIEval Models describe scenes in aerial photographs and answer questions about objects, positions, and road directions. Human graders score captions and answers separately. General visual questions Jul 2023 Not in cards
RTLLM Models write Verilog modules from design descriptions. Testbenches check syntax and circuit behavior. Synthesis measures the cost of working designs. Electronics Aug 2023 Not in cards
SB10k Models classify the sentiment of German social media posts using labels from three human annotators. Reading in many languages Apr 2017 Not in cards
ScanNet Models infer indoor scene geometry from images. Meta evaluates depth predictions against measurements from scanned indoor environments. Spatial reasoning and counting Feb 2017 Not in cards
ScanNet-1500 Models match features between pairs of indoor views and estimate relative camera poses. The evaluation measures pose accuracy on 1,500 image pairs. Spatial reasoning and counting Nov 2019 Not in cards
SciCap Models describe figures from scientific publications. Researchers compare generated captions with the text that accompanies each figure. Documents, charts and text in images Oct 2021 Not in cards
SD-QA Models answer spoken questions recorded in different English dialects. The benchmark compares answer accuracy across regional varieties of English. Audio understanding and voice Sep 2021 Not in cards
Self-Instruct User-Oriented Instructions Models follow manually written instructions on everyday tasks. Evaluators judge whether their responses meet the user's request. Following precise instructions Dec 2022 Not in cards
SemEval-2015 Task 1 Models decide whether two short social media posts express the same meaning. Reading comprehension Jun 2015 Not in cards
Sentences Involving Compositional Knowledge Models compare sentences whose meanings change with their grammatical structure. The dataset includes both relatedness ratings and entailment labels. Reading comprehension May 2014 Not in cards
skLEP Models classify Slovak reviews, recognize entailment, identify names, and answer questions about Slovak Wikipedia passages. Reading in many languages Jul 2025 Not in cards
SLDBench Agents infer scaling laws from data covering more than 5,000 language model experiments. Eight tasks test how well their formulas predict results at other scales. AI research tasks Jul 2025 Not in cards
SLIP zero-shot transfer suite Models assign labels to images using text descriptions of the classes. The evaluation averages zero-shot accuracy across 26 recognition tasks. General visual questions Dec 2021 Not in cards
SLURP Models identify the intent and named entities in spoken requests. NVIDIA evaluates both tasks within AIR-Bench. Audio understanding and voice Nov 2020 Not in cards
SMEBench Models edit recorded music according to instructions. The benchmark uses paired source and target recordings to check changes to pitch, instrumentation, articulation, and dynamics. Audio understanding and voice Sep 2026 Not in cards
SPair-71k Models match corresponding parts of objects across image pairs. The benchmark includes changes in viewpoint, scale, and occlusion. Spatial reasoning and counting Aug 2019 Not in cards
SpartQA Models answer questions about spatial relationships described in a passage. Language reasoning Apr 2021 Not in cards
Speech Robust Bench Models transcribe speech under different recording distortions. Word error rates show how each distortion affects recognition. Speech recognition and translation Mar 2024 Not in cards
Stanford Online Products Models retrieve photographs of the same product. The test compares learned image features on products outside the training set. General visual questions Nov 2015 Not in cards
STAR situated video reasoning Models answer multiple-choice questions about how people interact with objects in videos. The questions cover interactions, sequences, predictions, and feasibility. Video understanding Not in cards
STL-10 Models classify images into ten object categories using limited labeled training data. General visual questions Jun 2011 Not in cards
Structured Web Data Extraction Models extract named attribute values from web pages. The benchmark compares their outputs with annotated values across website categories. Faithful to sources Jul 2011 Not in cards
Summ-it Coreference Models group Portuguese noun phrases that refer to the same entity in science articles from Folha de São Paulo. Reading in many languages Jul 2007 Not in cards
SUMO Web Claims Summarization Models write a short headline summarizing a fact-checking article. HELM uses articles about climate claims. Faithful to sources Nov 2020 Not in cards
SUN397 Models recognize the scene category in indoor and outdoor photographs. The classification task covers 397 categories. General visual questions Jun 2010 Not in cards
SWAG Models read a partial description of an everyday activity. They choose the most plausible continuation from four options. Commonsense Aug 2018 Not in cards
Swiss-Judgment-Prediction Models predict whether the Swiss Federal Supreme Court accepts or dismisses a case. Law Oct 2021 Not in cards
Szeged NER Models identify named entities in Hungarian business news from the Hungarian News Agency. Reading in many languages May 2006 Not in cards
T2Ranking Models rank Chinese passages by their relevance to a query. Reading in many languages Jul 2023 Not in cards
Taiwan Mandarin Speech with Video: Speech Enhancement Models use lip video to remove noise from Mandarin sentences. Researchers compare the enhanced signal with clean speech for quality and intelligibility. Speech recognition and translation May 2020 Not in cards
TAU Urban Acoustic Scenes 2022 Mobile Models identify an urban acoustic scene from a one-second recording made on a mobile device. Audio understanding and voice Mar 2022 Not in cards
Testing Relational Understanding in Text-Guided Image Generation Models depict relationships between named entities. Human participants judge whether the generated scenes match each caption. Image generation and editing Jul 2022 Not in cards
Text2CAD Models generate parametric CAD designs from text prompts at different levels of detail. The authors compare generated geometry with the reference and count invalid designs. CAD models Sep 2024 Not in cards
TGIF-QA Models answer questions about actions, repetitions, and changes over time in animated clips. Video understanding Apr 2017 Not in cards
The Met Dataset Models match photographs of artworks to catalog images from the Metropolitan Museum of Art. The evaluation measures instance recognition across photographic conditions. General visual questions Not in cards
Third PASCAL Recognizing Textual Entailment Challenge Models decide whether a passage entails a proposed statement. The challenge compares their decisions with human labels. Reading comprehension Jun 2007 Not in cards
tinyBenchmarks Models answer small selected subsets of established benchmarks. Item response estimates approximate accuracy on the full tests. Composite indices Feb 2024 Not in cards
TokSuite Models complete questions under alternative tokenizations. The benchmark measures the change in accuracy relative to the canonical representation. Reasoning in many languages Dec 2025 Not in cards
ToLD-Br Models identify toxic language in Brazilian Portuguese posts using human annotations. Bias, fairness and politics Oct 2020 Not in cards
TopiOCQA Models retrieve evidence and answer follow-up questions in conversations that change topics. Questions that need search Oct 2021 Not in cards
ToxicChat Models identify toxic user messages from real conversations with an online chatbot. Deception and alignment Oct 2023 Not in cards
TREC 2017 LiveQA Medical Task Models answer consumer health questions. The shared task compares responses with expert assessments of answer quality. Medicine Not in cards
Turku NER Corpus Models identify named entities in Finnish text from the Universal Dependencies corpus. Reading in many languages May 2020 Not in cards
Turku Paraphrase Corpus Models compare Finnish sentence pairs and estimate how closely their meanings match. Reading in many languages May 2021 Not in cards
TUT Urban Acoustic Scenes 2018 Models recognize the setting of a ten-second audio clip recorded in a European city. Audio understanding and voice Apr 2018 Not in cards
Twitter URL Paraphrase Corpus Models identify paraphrases among pairs of social media posts that reference the same URL. Reading comprehension Sep 2017 Not in cards
UIT-ViCTSD Models classify Vietnamese comments for constructiveness and toxicity. The two labels capture different properties of the same text. Reading in many languages Mar 2021 Not in cards
UIT-VSFC Models classify sentiment in Vietnamese student feedback. The corpus also includes topic labels. Reading in many languages Nov 2018 Not in cards
Unhealthy Comments Corpus Models rate newspaper comments for overall conversational health and six attributes, including hostility and sarcasm. Researchers score each label separately using ROC AUC. Reading comprehension Aug 2020 Not in cards
UrbanSED Models locate sound events in synthetic urban soundscapes. The benchmark includes reference labels for the times and types of events. Audio understanding and voice Oct 2017 Not in cards
UrduSER Models identify emotions in Urdu dialogue from Pakistani television. The corpus contains 3,500 excerpts from ten actors across seven emotion categories. Audio understanding and voice Dec 2024 Not in cards
VALOR-32K Models match captions to video and audio clips. Meta reports separate retrieval results for text paired with audio and text paired with video. Audio understanding and voice Apr 2023 Not in cards
Verifiability Judgments for Generative Search Models decide whether a cited source fully, partly, or never supports a generated claim. Human annotations provide the reference judgments. Faithful to sources Apr 2023 Not in cards
Video-ChatGPT video conversation benchmark Models answer questions about video content. The evaluation grades correctness, detail, context, timing, and consistency. Video captioning Jun 2023 Not in cards
VideoCon Models distinguish descriptions supported by a video from captions that contradict it. Video understanding Nov 2023 Not in cards
Vinoground Models match short videos with captions that distinguish events by their temporal order. Video understanding Oct 2024 Not in cards
VM14K Models answer Vietnamese multiple-choice questions across 34 medical specialties. The authors reserve a private set for leaderboard evaluation. Medicine May 2025 Not in cards
VocalSound Models classify human vocal sounds such as laughter, coughing, and sneezing. Audio understanding and voice May 2022 Not in cards
Voice Jailbreak Attacks Against GPT-4o Models respond to spoken requests. Human reviewers measure how often a model complies with an unsafe request. Jailbreaks and prompt injection May 2024 Not in cards
VSEC Models correct misspellings in Vietnamese text. The evaluation compares their corrections against reference sentences. Reading in many languages Nov 2021 Not in cards
WenetSpeech-Chuan-Eval Models transcribe Sichuanese speech. Qwen reports separate error rates on the easy and hard test sets. Speech recognition and translation Sep 2025 Not in cards
WHU-CDC Models compare aerial photos of Christchurch, describe what changed, and mark changed pixels. Reference captions describe reconstruction after the earthquake. Detection and captioning Oct 2024 Not in cards
Wiki Neutrality Corpus Models identify biased wording in English Wikipedia sentences and rewrite the sentences. The authors compare the rewrites with corrections by Wikipedia editors. Bias, fairness and politics Nov 2019 Not in cards
WikiLingua Models summarize procedural articles in different languages. Aligned WikiHow articles support both same-language and cross-language evaluation. Reading in many languages Nov 2020 Not in cards
WikiNEuRal Models locate named entities in Wikipedia text across nine languages using automatically produced reference annotations. Reading in many languages Nov 2021 Not in cards
Winoground Models match two images to two captions that use the same words in a different order. General visual questions Apr 2022 Not in cards
WixQA Models retrieve Wix support articles and answer customer questions from those articles. The authors score retrieval and answer generation separately. Running a business Apr 2025 Not in cards
WSYue-eval Models transcribe short and long Cantonese recordings. The suite also evaluates speech synthesis. Speech recognition and translation Sep 2025 Not in cards
x-stance Models identify whether a political comment supports or opposes the proposition in a question. Reading in many languages Mar 2020 Not in cards
XNLIeu Models judge whether a Basque sentence follows from, contradicts, or is neutral toward a premise. The evaluation includes translated and native examples. Reading in many languages Apr 2024 Not in cards
XOR-TyDi QA Models retrieve passages across languages to answer questions written by speakers of seven languages. Tasks require English evidence or answers in the question’s language. Reading in many languages Oct 2020 Not in cards
XWinograd Models resolve ambiguous pronouns in six languages by choosing which noun fits each sentence. Reasoning in many languages Jun 2021 Not in cards
YouTube-VOS Models segment and track objects across frames in YouTube videos. The evaluation distinguishes categories seen during training from previously unseen categories. Video understanding Sep 2018 Not in cards

Arenas without a leaderboard here

Name Kind Released Best score Status
Economics Arena for Large Language Models Models choose numbers in beauty contests and place sealed bids in auctions. Researchers compare payoffs and measure how strategies change when players see previous rounds. Multi-agent games Jan 2024 0.95 Not in cards
3DGen-Bench People compare generated 3D objects in pairs. The authors use their judgments to evaluate automated scoring methods for text-to-3D and image-to-3D generation. Image generation and editing Mar 2025 Not in cards