Biology and medicine
Tasks that need biology or medical knowledge. We count 730 benchmarks here, 4 of them arenas. A benchmark belongs here when subject includes biological sciences, medical and health sciences, medical engineering, environmental biotechnology or industrial biotechnology.
How fast it grows
126 were released since July, more than the 114 in the first half of 2026.
How much is still open
No model has passed 85% on 250 of the 331 benchmarks here that score in percent, 76% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| 67AI Lab L2 nuclear medicine report structuring Models extract findings from 300 synthetic PET/CT and SPECT reports into a fixed form. They also flag conflicting values in 17 reports. | University | Nov 2026 | | Not in cards |
| 67AI Lab L3 nuclear medicine guideline questions Models answer questions using eight retrieved passages from nuclear medicine guidelines. They must abstain when the passages lack an answer. | University | Nov 2026 | 90.8% | Not in cards |
| AbMaze Agents make choices in a simulated antibody optimization experiment under a fixed budget. Recorded assay results determine normalized improvement on 200 test episodes. | Expert | Nov 2026 | Not in cards | |
| Bio-PPI Risk: Serous vs Endometrioid Models estimate a cancer subtype probability from tumor protein measurements and protein interactions. The evaluation measures discrimination and probability accuracy. | Expert | Nov 2026 | 0.22 | Not in cards |
| Bio-Probe Models give probabilities for 21 unpublished biology hypotheses before their results are known. Scored by Brier score against the outcomes. | Research frontier | Nov 2026 | System cards only | |
| BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. | Expert | Nov 2026 | 66.7% | Not in cards |
| BioSimBench Models predict which genes change after drugs, gene knockdowns, and immune signals across four prediction tracks containing 60, 60, 60, and 27 cases. The overall score averages perturbation-specific F1 across those tracks on a zero-to-one scale. | Expert | Nov 2026 | | Not in cards |
| DiagTrace Models request information about a patient one turn at a time. The evaluator compares the sequence of requests with the published case history. | Expert | Nov 2026 | 0.40 | Not in cards |
| EUREKA-Bench (OpenAI) Agents build predictive models for six biology research tasks. Scored as predictive performance with the best published predictor in each field set to 100. | Research frontier | Nov 2026 | 100.3 | System cards only |
| FrontierBioML 25 machine learning tasks in computational biology, across antibody development, protein engineering, structural biology, binder design and functional genomics. An agent gets real experimental data and a brief, works offline for up to six hours and submits a prediction script. A sealed grader runs the script on held-out data, such as new antibody families or new protein targets, and scores each task from 0 at a simple baseline to 100 for a perfect answer. | Expert | Nov 2026 | 40.3 | Not in cards |
| GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. | Expert | Nov 2026 | 16.4% | Not in cards |
| Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. | Expert | Nov 2026 | 83.9% | Not in cards |
| MedHEB Models retrieve related medical images, reports, and image regions. The authors evaluate how models rank relevant evidence across 107 retrieval tasks. | University | Nov 2026 | | Not in cards |
| MedPhysBench Models calculate doses and check medical physics worksheets for planted errors. A grader checks the units and accepts values within a specified tolerance. | Expert | Nov 2026 | 100% | Not in cards |
| NPCBench Models follow patients with nasopharyngeal carcinoma through diagnosis and care. The main patient-level measure requires correct decisions at successive stages. | Expert | Nov 2026 | 23% | Not in cards |
| OpenAI Epoch Capabilities Index (OECI) OpenAI's own capabilities index, fit with Epoch AI's method over 34 public and 35 internal benchmarks. | Research frontier | Nov 2026 | 170.1 | System cards only |
| PWS-Health Web research systems answer healthcare questions from clinical trials, medical literature, registries, drug labels, and healthcare workflows. | Expert | Nov 2026 | Not in cards | |
| Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. | Research frontier | Nov 2026 | 44.7 | Not in cards |
| Sequitur Research Lab Automation Agents run simulated wet-lab automation protocols. Scored out of 100 on how well each experimental phase is carried out, averaged over six runs. | Research frontier | Nov 2026 | 74.5 | System cards only |
| Suicide Memory Bench AI systems, each a model in its own app or coding harness, talk with 128 fictional personas over 29 chats. Five chats carry suicidal statements or warning signs of Suicide Crisis Syndrome, and a final request follows, such as a farewell note. Four sub-benchmarks cross explicit statements and warning signs with one chat or separate chats joined only by memory. A model judge rates each reply on a 0 to 4 safety scale. | Everyday | Nov 2026 | | Not in cards |
| TrialDesignBench Models select a dose-finding design for an early oncology trial. The authors compare their choices with expert reference designs across 617 trials. | Expert | Nov 2026 | 0.52 | Not in cards |
| WildlifeBench Wildlife Knowledge V1 Models answer 185 questions about wild species without search or tools. The overall score weights ten tasks equally, including taxonomy, conservation status, and geographic range. | University | Nov 2026 | | Not in cards |
| AI Professions Prompt Benchmark Models draft everyday work documents from fictional inputs: 100 prompts for tasks in 10 professions, such as accounting, law, healthcare administration and real estate, plus Excel and Microsoft 365, in English, Spanish, French and Portuguese. A judge model from another family checks 5 pass or fail criteria per prompt, fixed before any model ran. | University | Oct 2026 | 85.9% | Not in cards |
| Artificial Analysis Decision Index Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning. | Everyday | Oct 2026 | Not in cards | |
| Artificial Analysis Intelligence Index v5 Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. | Oct 2026 | 64.6 | In use | |
| Biological static hazard evaluation Models answer 75 questions across priority biological threat scenarios, each asked 10 times. The score is the share of hazardous responses. | Expert | Oct 2026 | | System cards only |
| CareManage-Bench Models respond to patient scenarios involving medication safety, care plans, and triage. The harness checks the responses against case specific criteria. | University | Oct 2026 | Not in cards | |
| CellCaution: Reasoning About Cell Composition Models reason about changes in cell populations and decide when the information supports a numerical claim about their composition. | University | Oct 2026 | | Not in cards |
| DistressBench Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. | Expert | Oct 2026 | | In use |
| EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. | University | Oct 2026 | Not in cards | |
| HealthAgentBench V2 Agents complete 75 healthcare tasks across nine categories in terminal environments. Task-specific verifiers check outputs against hidden labels, with model grading for radiology report correction. | Expert | Oct 2026 | 57.3% | Not in cards |
| KMMLU-Pro 2025 Models answer questions from the 2025 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. | University | Oct 2026 | Not in cards | |
| KMMLU-Pro 2026 Models answer questions from the 2026 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. | University | Oct 2026 | 78.1% | Not in cards |
| MedSafe-Dx v0.3 Models read synthetic primary-care intake summaries and decide whether to flag a dangerous condition. Rule-based labels determine weighted error costs, with missed serious cases costing seven times a false escalation or incorrect danger label. The score normalizes these costs against a fixed escalation baseline. | Expert | Oct 2026 | 73.5 | Not in cards |
| micro1 Medical Index Models are scored on the equal-weight average of micro1 medical benchmarks, currently RealmPathology, PharmaBench and WellnessBench. | Expert | Oct 2026 | 71% | Not in cards |
| MindEval2 Models support simulated patients from MindSim, a tool-using harness with detailed profiles. Psychologist-written rubrics (MHQR) replace Likert scales for scoring. | Expert | Oct 2026 | | Not in cards |
| Mitosis Bench Agents mark every dividing cell in a tumor image. Each mark must fall within 30 pixels of an expert label. The grader rejects extra or missing marks. | Expert | Oct 2026 | | Not in cards |
| OmicsBench v1 Agents analyze molecular data across eight benchmarks. LatchBio reports pass rates for 741 short tasks and 43 long tasks. | Expert | Oct 2026 | 40.8% | Not in cards |
| OncologyBench Separate agents run analyses of cancer research data, and a final agent makes a decision on detection, diagnosis or treatment. Strict scoring needs full marks on both parts. | Expert | Oct 2026 | 50.7% | Not in cards |
| PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. | Research frontier | Oct 2026 | 14% | Not in cards |
| PharmaBench (micro1) Agents write clinical trial documentation inside 8 synthetic drug programs, across 120 tasks written by biostatisticians, medical writers, safety physicians and regulatory staff. | Expert | Oct 2026 | 17.8% | Not in cards |
| RealmPathology Models interpret real anatomic pathology reports and must not claim a diagnosis, stage or treatment the report does not support. Rubrics have 772 positive and 299 penalty criteria. | Expert | Oct 2026 | 78% | Not in cards |
| RSNA Knee AUC10 Models label twelve knee findings from images, radiology reports, or both. The grader measures how well their predictions distinguish findings on ten reference studies. | Expert | Oct 2026 | 0.92 | Not in cards |
| RSNA Knee LLM Label Extraction Models extract twelve findings from knee MRI reports in a batch. The grader compares the labels with reference findings using macro ROC AUC. | University | Oct 2026 | 0.89 | Not in cards |
| SAFE-Teen Health advice Models respond to simulated teens in health advice scenarios. The score is the published conversation failure rate. | Everyday | Oct 2026 | 9.00% | Not in cards |
| SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. | Expert | Oct 2026 | 80.9 | Not in cards |
| ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. | Expert | Oct 2026 | 0.53 | Not in cards |
| SciTaste-Bench Models judge the research value of papers in six disciplines. The benchmark compares their judgments with a reference index built from seven dimensions. | Expert | Oct 2026 | | Not in cards |
| SDBench Diagnostic agents order tests, ask questions and commit diagnoses in sequential clinical cases. Scores combine diagnostic correctness with test cost. | Expert | Oct 2026 | Not in cards | |
| The Pushback Test Models advise a parent who questions urgent medical advice. The evaluation checks whether the model maintains the urgency required by NHS guidance. | University | Oct 2026 | 0.81 | Not in cards |
| Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. | Expert | Oct 2026 | 68.4% | In use |
| VIALS 2 Models interpret 161 professional life-science visual artifacts. The score is judged accuracy averaged across three rollouts. | Expert | Oct 2026 | 49.9% | Not in cards |
| WellnessBench Models answer young people asking about eating, sexual safety, substances and self harm, in clinician-written single-turn and pushback dialogues. A pass must be both safe and helpful. | Everyday | Oct 2026 | 66% | Not in cards |
| AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. | Expert | Sep 2026 | System cards only | |
| Advanced Screening Evasion Models propose strategies to evade DNA synthesis screening and design evading fragments. Scored from 0 to 1 on strategy quality and fragment evasion. | Research frontier | Sep 2026 | 0.73 | System cards only |
| Artificial Analysis Intelligence Index v4.2 Version 4.2 of Artificial Analysis's index, which added multi-hour office work and document tasks and dropped GPQA Diamond. | Sep 2026 | Not in cards | ||
| Artificial Analysis Intelligence Index v4.3 Version 4.3 of Artificial Analysis's index, eleven weighted parts led by knowledge work, office tasks and Terminal-Bench 4.0. | Sep 2026 | | In use | |
| Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. | Expert | Sep 2026 | 72.2% | System cards only |
| Biopharma Bench Agents take an employee seat inside one of 12 simulated biopharma companies and complete 71 professional assignments, such as a regulatory assessment, an out-of-specification investigation or an audit package. They must work out which records govern among contradictory files. A rubric grader checks 752 frozen criteria, with code checks for exact numbers. | Expert | Sep 2026 | 65.1% | Not in cards |
| BioSecBench-Function Agents recover biological function from experimental data. Scored by verifiable checks. One of three LatchBio BioSecBench evaluations. | Expert | Sep 2026 | 73.6% | System cards only |
| Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. | Research frontier | Sep 2026 | 0.47 | System cards only |
| DAYJOB: Healthcare Agents handle short requests from healthcare staff using charts, claims exports, policies and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Sep 2026 | 48.3% | Not in cards |
| De novo protein binder design (Anthropic) Agents get a target protein structure and 24 hours to design 30 miniprotein binders. The evaluation covers 15 targets. | Expert | Sep 2026 | 85.1% | System cards only |
| DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. | Expert | Sep 2026 | 40.0 | Not in cards |
| EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. | Research frontier | Sep 2026 | 29.8 | Not in cards |
| Fireworks Specialized Intelligence Index An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number. | Expert | Sep 2026 | Not in cards | |
| HealthBench-Psych v2 Models respond to 611 health conversations that clinicians selected for mental health relevance. Three model judges score each answer against its physician rubric. | Expert | Sep 2026 | 0.66 | Not in cards |
| HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. | Research frontier | Sep 2026 | 77.9% | In use |
| K-Bench Models respond across multi-turn mental-health risk conversations involving suicide, self-harm, domestic violence, substance misuse, and no-risk cases. | Expert | Sep 2026 | 99% | Not in cards |
| KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. | Expert | Sep 2026 | 29.4% | Not in cards |
| LifeSci Diamond Models write regulatory affairs deliverables for drugs, biologics and medical devices, such as change assessment memos, dossier inconsistency logs and agency letters. The 55 tasks span five families: reporting categories, Module 2/3 dossier consistency, missing evidence, submissions and correspondence, and device vigilance. A model judge grades each answer against a task rubric, and a task passes only when every rubric line is met and no penalty fires. | Expert | Sep 2026 | 20% | Not in cards |
| Medical Bench Agents read longitudinal dermatology charts and write clinical assessments. Expert-authored rubrics grade their findings and management decisions. | Expert | Sep 2026 | 88% | Not in cards |
| Medical Speech-to-Text Benchmark Speech models transcribe 1,513 English medical recordings. A common scorer measures word errors, medical-term errors, drug-name errors, and dosage matches. Separate tests measure batch response time and final streaming latency. | Expert | Sep 2026 | 1.32% | Not in cards |
| Medicinal Chemistry (Anthropic) Models see several real drug-discovery compounds and pick which had the highest or lowest measured ADME property. The set has 504 questions. | Expert | Sep 2026 | 65.3% | System cards only |
| MentalHealthBench 1,215 synthetic mental-health conversations with 5,262 clinician-authored rubric criteria. An LLM judge scores safety and guidance quality. | Expert | Sep 2026 | 57.3% | System cards only |
| MetagenomicsBench Agents analyze microbial communities from shotgun, amplicon, long-read and spatial metagenomics data. A grader checks the biological result. | Expert | Sep 2026 | 62.7% | Not in cards |
| Miðeind Icelandic Terminology Models explain Icelandic knitting and crochet terms and name local species from English and Latin names. A model grades definitions. A program checks species names. | University | Sep 2026 | 78% | Not in cards |
| Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. | Expert | Sep 2026 | 34% | System cards only |
| Neuro Reasoning (dwin2) Models answer generated questions about EEG experiment design and preprocessing. The evaluator computes a reference answer for each question. | University | Sep 2026 | Not in cards | |
| OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. | Expert | Sep 2026 | 73.7% | Not in cards |
| Periapical Lesions Bench Agents locate and grade lesions around tooth roots in panoramic dental X-rays. A perfect report matches every lesion and grade without adding extra boxes. | Expert | Sep 2026 | | Not in cards |
| Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. | Research frontier | Sep 2026 | 10.7 | System cards only |
| SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. | Expert | Sep 2026 | 45.3% | Not in cards |
| SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. | Research frontier | Sep 2026 | 0.46 | System cards only |
| Sonetra Norwegian SNOMED CT Term Recognition Models transcribe Norwegian sentences read by a synthetic voice. A term counts as correct when every word in the medical term matches. | University | Sep 2026 | 66.4% | Not in cards |
| TMMLU+ 1.1 Models answer traditional Chinese subject questions from a revised exam set. The revision corrects 197 answers and removes 539 questions from the original release. | University | Sep 2026 | 96.2% | Not in cards |
| TxBench-Oligo-Discovery Agents make oligonucleotide discovery decisions, such as ASO and siRNA design, from real data. A grader checks each decision. | Expert | Sep 2026 | 62.2% | Not in cards |
| VCT-v2 Models answer an updated set of expert virology troubleshooting questions from SecureBio. Scored as refusal-corrected accuracy. | Expert | Sep 2026 | 60.5% | System cards only |
| WearableQA Models answer 4,084 ten-option questions using longitudinal wearable measurements, blood biomarkers, and demographics from 200 users. Questions test computations and physiological interpretation across single or multiple signals. Accuracy compares the selected option with the answer key. | Expert | Sep 2026 | | Not in cards |
| Apodex Discovery Solver systems of model, harness and tools run long, verifiable investigations on high-value problems scouted from 561 industries in 16 sectors. | Research frontier | Aug 2026 | 2.80 | Not in cards |
| Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| Bio-VQA Models identify organisms, describe visible traits, judge spatial relationships, and recognize individual animals across photographs. The benchmark has 24 tasks. | University | Aug 2026 | | Not in cards |
| Biosecurity VCT Models answer dual-use virus lab troubleshooting items. Accuracy counts should-refuse and should-comply items together. | Expert | Aug 2026 | 47.8% | System cards only |
| BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. | Expert | Aug 2026 | 91.4% | System cards only |
| BixBench3 Agents rebuild the full computational analysis of 20 published biology papers from raw data on a cloud VM. Output tables are graded against the papers' own artifacts. | Research frontier | Aug 2026 | | Not in cards |
| CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. | Expert | Aug 2026 | | System cards only |
| CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. | Expert | Aug 2026 | | System cards only |
| Chartography 100 tasks on specialized professional charts (Kaplan-Meier, candlestick, Sankey, Bode). Answers graded against expert references. | Expert | Aug 2026 | 90.7% | In use |
| Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. | Expert | Aug 2026 | | System cards only |
| CliniCARE-Bench Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases. | Expert | Aug 2026 | 76.1% | Not in cards |
| FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. | Expert | Aug 2026 | 20.6% | Not in cards |
| GDPval Extended Models produce work products for occupations across nine GDP sectors. Mercor adds 132 new expert-written tasks to the 213 public ones and reports both scores. | Expert | Aug 2026 | 7.10% | Not in cards |
| Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. | Expert | Aug 2026 | | System cards only |
| HealthBench-Psych v1 Models respond to 610 health conversations that clinicians selected for mental health relevance. Three model judges score each answer against its physician rubric. | Expert | Aug 2026 | 0.63 | Not in cards |
| HealthBench-Psych-Hard Models answer mental health questions from conversations that also appear in HealthBench-Hard. Three model judges use the original physician rubrics. | Expert | Aug 2026 | 0.45 | Not in cards |
| HLE Extended Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores. | Expert | Aug 2026 | 77.3% | Not in cards |
| MedXpertQA MM Extended Models answer multimodal medical questions over clinical images and patient details. Mercor adds 51 new expert-written tasks to the 200 public ones and reports both scores. | Expert | Aug 2026 | 71.2% | Not in cards |
| MLCR-AA Models reason over synthetic medical claim files of about 25k to 64k tokens, as a claims reviewer would. Three judge models check completeness and accuracy. | Expert | Aug 2026 | Not in cards | |
| MMMU-Pro Extended Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores. | University | Aug 2026 | 61.3% | Not in cards |
| SYNERGY+ Models rank research records against a systematic review's eligibility criteria. Reviewers' final inclusion decisions supply the reference labels. | University | Aug 2026 | Not in cards | |
| Terminal-Bench-Science 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. | Expert | Aug 2026 | 68.1% | In use |
| Transluce Mental Health Evaluation Models hold multi-turn conversations with simulated users facing mental-health scenarios. Automated judges score 14 helpful and harmful behaviors. | Expert | Aug 2026 | Not in cards | |
| Tuesday Work Index A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday. | Expert | Aug 2026 | | Not in cards |
| TxBench-Antibody-Discovery Agents make antibody discovery and development decisions from real experimental data. A grader checks each decision. | Expert | Aug 2026 | 67.7% | Not in cards |
| VIALS Vision models answer 161 questions about lab images such as gel blots, plasmid maps, microscopy and flow cytometry plots. An LLM judge grades each answer against a key. | Expert | Aug 2026 | 65.8% | Not in cards |
| AI Security Leaderboard Models face systematic jailbreak attacks across chemical, biological, radiological, nuclear, explosive and cyber domains. Scores count universal jailbreaks and estimate attacker cost. | Expert | Jul 2026 | $60 | Not in cards |
| Autointent-Bio Models face manual and synthetic dangerous and benign biology queries. xAI reports refusal recall on the dangerous ones. | Jul 2026 | 100% | System cards only | |
| BioSecBench-Refusal Agents run 61 routine and 46 concealed-hazard biology analysis tasks. The score pairs red-team refusal with routine completion. | Expert | Jul 2026 | 62.4% | System cards only |
| BioSecBench-Surveillance Agents turn raw pathogen sequencing data into surveillance results. The endpoint pass rate excludes refused runs. | Expert | Jul 2026 | 60.8% | System cards only |
| IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. | Research frontier | Jul 2026 | 27.3% | Not in cards |
| MedChemBench (OpenAI) Models work through 99 medicinal chemistry problems drawn from recent papers and patents. Each task mixes prose with structures, assay tables and plots, then asks for the program’s next decision. | Expert | Jul 2026 | 83.8% | In use |
| OCT-Bench Models answer 10,076 questions about retinal scans, from anatomy to treatment and prognosis. The evaluator measures answer accuracy and counts invalid responses as incorrect. | Expert | Jul 2026 | 62% | Not in cards |
| PatientAgentBench Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria. | Expert | Jul 2026 | 4.25 | Not in cards |
| ProtocolQA Open-Ended (xAI) Models name the single most important mistake in a biology lab protocol in free text, with no answer choices. | Expert | Jul 2026 | 87% | System cards only |
| RadLE 2.0 Models diagnose single-image radiology cases with a confidence score from 0 to 4, or say they do not know. A confidence-weighted index rewards right answers and penalizes confident errors. | Expert | Jul 2026 | 758.0 | Not in cards |
| VariantBench Agents discover and interpret genetic variants from real sequencing analyses. A deterministic grader checks the result. | Expert | Jul 2026 | 52.8% | Not in cards |
| AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. | Research frontier | Jun 2026 | 0.59 | System cards only |
| ABC Bench Fragment Design Agents use biology tools to complete a computational genetic-construct design task. One task family of the Agentic Bio-Capabilities Benchmark. | Expert | Jun 2026 | | System cards only |
| ABC Bench Liquid-Handling Robot Agents use biology tools to plan liquid-handling robot steps for a construct-assembly task. One task family of the Agentic Bio-Capabilities Benchmark. | Expert | Jun 2026 | | System cards only |
| ABC Bench Screening Evasion Agents complete a computational task on nucleic-acid synthesis screening. One task family of the Agentic Bio-Capabilities Benchmark. | Expert | Jun 2026 | 82.9% | System cards only |
| AutoMedBench Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission. | Expert | Jun 2026 | 66.5 | Not in cards |
| Biology Model Refusal Evaluation Models answer severe, dual-use and benign biology prompts. Scored as safe rate on harmful prompts and not over-refusing benign ones. | Expert | Jun 2026 | 99.9% | System cards only |
| BioTIER v1 Refuse Models receive 501 dangerous biology prompts on select agents, dual-use research and catastrophe avoidance. The score is the share correctly refused. | Expert | Jun 2026 | | System cards only |
| ClinHallu Models answer medical image questions and produce intermediate reasoning. The evaluation reports answer accuracy and separate hallucination rates for visual recognition, knowledge recall, and reasoning integration against annotated reasoning traces. | Expert | Jun 2026 | | Not in cards |
| Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. | Expert | Jun 2026 | 0.20 | Not in cards |
| DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. | Expert | Jun 2026 | | Not in cards |
| EpiBench Agents analyze real epigenomics datasets in a sandbox with standard tools. A deterministic grader checks the biological result. | Expert | Jun 2026 | 39.6% | Not in cards |
| GeneBench Pro Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. | Expert | Jun 2026 | 31.5% | In use |
| GRAFT Models predict plant traits from gene expression in Arabidopsis. The authors compare regression and graph models against measurements from the same plants. | University | Jun 2026 | Not in cards | |
| HealthAgentBench Agents use terminal tools to complete 54 clinical and biomedical tasks across seven categories. Task-specific verifiers measure success, including a model judge for radiology report correction. | Expert | Jun 2026 | 55% | Not in cards |
| K9-Bench Models answer 4,744 questions about dog behavior in 907 videos. The evaluator compares each free-form answer with five options that the model never sees. | School | Jun 2026 | | Not in cards |
| KINA Models answer 899 knowledge questions across 261 fine-grained disciplines, each with ten lettered options. Scored by accuracy against the key. | Expert | Jun 2026 | | In use |
| LifeSciBench Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. | Expert | Jun 2026 | 61.3% | In use |
| MarineEVT Models answer 2,000 held-out questions about underwater events, including species interactions, movement, and timing. The authors report accuracy across five reasoning categories. | University | Jun 2026 | | Not in cards |
| MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. | Expert | Jun 2026 | 69.2 | Not in cards |
| NatureBench Coding agents work on 90 tasks distilled from Nature-family papers, each in its own container, and try to match or beat the published results. | Research frontier | Jun 2026 | 23.3% | Not in cards |
| Protein Design (Anthropic) Models generate novel protein sequences under design constraints and rank protein design libraries for experimental testing. | Expert | Jun 2026 | 42.5% | System cards only |
| PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. | Expert | Jun 2026 | | Not in cards |
| Real Point-of-Care Queries Clinical AI systems answer 620 questions from practicing physicians across 30 specialties. Specialists compare anonymous answers for accuracy, utility, source quality, completeness and verifiability. | Expert | Jun 2026 | -1.50% | Not in cards |
| scBench-Long Agents run long single-cell RNA-seq analyses that need sustained empirical work and biological reasoning. | Expert | Jun 2026 | 36.9% | Not in cards |
| scBench-Long (original study) Agents complete 21 tasks in long single-cell data analysis workflows. The authors compare their answers with results from the original analyses. | Expert | Jun 2026 | 25.4% | Not in cards |
| TxBench-Preclinical-Pharmacology Agents analyze preclinical pharmacology data and make development decisions. A grader checks each decision. | Expert | Jun 2026 | 68% | Not in cards |
| AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. | Expert | May 2026 | 0.80 | System cards only |
| BiomniBench Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer. | Expert | May 2026 | 63.9 | Not in cards |
| BioPhys-Bridge Models connect observations in biophysics papers to quantitative explanations and cite supporting evidence. The authors score the cited evidence against reference identifiers. | Expert | May 2026 | | Not in cards |
| Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. | Expert | May 2026 | System cards only | |
| CHI-Bench Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs. | Expert | May 2026 | 54.7% | Not in cards |
| CUSP Models forecast scientific progress across 4,760 events in many fields, using only knowledge from before a set cutoff date. Tasks ask whether an advance is feasible, how it would work, how to design a solution and when it will happen. The benchmark also compares events before and after each model's training cutoff. | Expert | May 2026 | 60.4% | Not in cards |
| DataPrep-Bench Medical Models prepare training data for the medical domain. The score is the published downstream average after backbone training. | Expert | May 2026 | 40.5 | Not in cards |
| ExpertEval Models answer cases in medicine, finance and law. Practicing professionals write the cases and supply weighted grading rubrics. | Expert | May 2026 | 66.9% | Not in cards |
| GenomeGym-BRCA1 Models estimate whether a documented BRCA1 variant causes loss of function. The grader compares each probability with a hidden functional label. | Expert | May 2026 | Not in cards | |
| Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. | University | May 2026 | 95.7% | In use |
| High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. | Expert | May 2026 | 4.10% | System cards only |
| K12-Bench Models select curriculum concepts, prerequisite relationships, related knowledge, experimental evidence, and textbook locations. The 23,640 multi-select questions come from a textbook knowledge graph. Exact match and instance-level F1 compare selected options with graph-derived answers. | School | May 2026 | | Not in cards |
| OncoTriad-QA Models answer questions about cancer patients using radiology, pathology, and genomic evidence. The authors score multiple-choice answers and compare written responses with reference answers. | Expert | May 2026 | 46% | Not in cards |
| OpenFARM Bioacoustics Models classify animal vocalizations across eight tasks, including pig call type and recording context. Each task uses labels from its source dataset. | Expert | May 2026 | Not in cards | |
| PhysicianBench 100 physician tasks performed by an agent inside an EHR environment. Scored by pass rate per attempt. | Expert | May 2026 | 50% | System cards only |
| ResearchClawBench Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs. | Research frontier | May 2026 | | Not in cards |
| SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. | Expert | May 2026 | 40.2% | Not in cards |
| SpatialBench-Long Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning. | Expert | May 2026 | 40.5% | Not in cards |
| SpatialBench-Long (original study) Agents complete 24 tasks in long spatial transcriptomics workflows. The authors compare their answers with results from the original analyses. | Expert | May 2026 | 11.1% | Not in cards |
| Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. | Expert | Apr 2026 | 39.3% | System cards only |
| BioPipelineBench Verified (Anthropic) Agents execute bioinformatics workflows with a bash tool. Only problems that passed a validation check by external reviewers are kept. | Expert | Apr 2026 | 83.6% | System cards only |
| Clinical SOAP Note Evaluation: Safety-First Benchmarking Models turn 300 synthetic doctor-patient dialogues into clinical SOAP notes. Three judges check claims against transcript evidence and score numeric fidelity, coverage, and writing quality. The composite weights safety at 50%, coverage at 30%, and general quality at 20%. | Expert | Apr 2026 | 4.72 | Not in cards |
| DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. | Research frontier | Apr 2026 | 16.5% | System cards only |
| GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. | Expert | Apr 2026 | 33.2% | Fading |
| GenomeQA Models answer multiple-choice questions about short genomic sequences. The benchmark covers six task families, including species classification. | University | Apr 2026 | | Not in cards |
| Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. | Research frontier | Apr 2026 | 7.60% | System cards only |
| HealthBench Professional 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. | Expert | Apr 2026 | 69.2% | In use |
| KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. | University | Apr 2026 | | Fading |
| Medical Chronology LLM Benchmark Models extract medical timelines from six synthetic document sets in three independent rounds. Checks compare extracted events with reference timelines and assess semantic fidelity, formatting, date order, and hallucinations. A weighted composite combines these checks with ROUGE and extraction F1. | Expert | Apr 2026 | 88.9 | Not in cards |
| Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. | Expert | Apr 2026 | System cards only | |
| Protocols (Anthropic life sciences) Models find and fix errors in molecular biology protocols and answer practical questions about applying real laboratory protocols. | Expert | Apr 2026 | 62% | System cards only |
| WASSCE Benchmark for Open Language Models Models answer questions based on the West African Senior School Certificate Examination. The harness compares models across mathematics, English, science, and social studies. | School | Apr 2026 | Not in cards | |
| $OneMillion-Bench Agent solves 400 expert-curated tasks in law, finance, industry, healthcare and science. Rubric grading of accuracy, logic, feasibility and compliance. | Expert | Mar 2026 | 42.1% | In use |
| BioTIER Safeguarded model configurations receive harmful and benign biology requests. Scored on refusing requests designated for refusal and allowing benign ones. | Expert | Mar 2026 | Not in cards | |
| BioTIER Permit Models receive 144 benign biology prompts. The score measures the percentage correctly answered. | Expert | Mar 2026 | 100% | System cards only |
| BioTIER Refuse Models receive 398 dangerous biology prompts. The score measures the percentage correctly refused. | Expert | Mar 2026 | 99.1% | System cards only |
| BioVITA retrieval benchmark Models match animal images, audio recordings, and descriptions. The benchmark checks whether the retrieved items match the reference species, genus, or family. | University | Mar 2026 | Not in cards | |
| BullshitBench V2 Models answer 100 nonsense questions across software, finance, legal, medical and physics, built with 13 techniques. A three-judge panel scores clear pushback. | Everyday | Mar 2026 | 95% | Not in cards |
| CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. | Mar 2026 | | System cards only | |
| BABE Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference. | Expert | Feb 2026 | 52.3% | Fading |
| Bio-Literature Search Level 2 Agents use search tools to compare abstracts from PubMedQA, including their sections, subject headings, and conclusions. Five task types require retrieving or combining evidence. | University | Feb 2026 | Not in cards | |
| BioMysteryBench Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. | Expert | Feb 2026 | 83.7% | In use |
| BioPipelineBench (Anthropic) Agents execute bioinformatics workflows such as long-read sequence analysis, metagenome assembly and chromatin profiling with a bash tool. | Expert | Feb 2026 | 87.7% | System cards only |
| Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. | Expert | Feb 2026 | 16.4 | System cards only |
| Disordered eating evaluations (Anthropic) Models handle conversations about eating disorders and disordered eating. Score is the appropriate response rate. | Everyday | Feb 2026 | System cards only | |
| GutBrainIE 2026 Models find entities in 80 gut and brain research abstracts, link them to biomedical concepts, and identify relationships between mentions and between concepts. | University | Feb 2026 | Not in cards | |
| HLE-Verified Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. | Expert | Feb 2026 | | In use |
| HLE-Verified (text-only Full) Models answer the text-only questions of Humanity's Last Exam after expert verification and revision. The Full set keeps the questions experts marked uncertain, and the score is pass@1 accuracy. | Expert | Feb 2026 | 46.8% | Not in cards |
| Korean Dental Licensing Examination 2024 and 2025 Models answer 642 Korean dental licensing questions from two new exams. The authors score accuracy and break it down by subject and image type. | Expert | Feb 2026 | 81.5% | Not in cards |
| KorMedMCQA-Mixed Models take combined text and image exams. The 2022 set contains 281 questions and the 2023 set contains 307. Each year has a separate score. | Expert | Feb 2026 | | Not in cards |
| KorMedMCQA-V Models answer Korean medical licensing questions using clinical images. The full test contains 1,534 questions and 2,043 images from 2012 through 2023. | Expert | Feb 2026 | | Not in cards |
| LABBench2 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. | Expert | Feb 2026 | 60.9% | In use |
| LABBench2 CloningQA Models answer molecular biology questions using supplied information or retrieval tools. | Expert | Feb 2026 | | Not in cards |
| LABBench2 FigQA2 Models read scientific figures and answer in open response. Served as images or as PDFs. The FigQA2 subtask of LABBench2. | Expert | Feb 2026 | 42.6% | Not in cards |
| LABBench2 LitQA3 Models answer open-response questions whose answers appear once in recent biology papers. The LitQA3 subtask of LABBench2. | Expert | Feb 2026 | 85.1% | Not in cards |
| LABBench2 PatentQA Models locate and interpret evidence in patent documents. | Expert | Feb 2026 | 90.9% | Not in cards |
| LABBench2 ProtocolQA2 Models find and fix errors planted in published lab protocols, answering in open response. The ProtocolQA2 subtask of LABBench2. | Expert | Feb 2026 | | Not in cards |
| LABBench2 SeqQA2 Models reason about DNA and protein sequences and answer in open response. The SeqQA2 subtask of LABBench2. | Expert | Feb 2026 | | Not in cards |
| LABBench2 TrialQA Models use clinical trial records to answer research questions. | Expert | Feb 2026 | 93.3% | Not in cards |
| MedCode Models read de-identified patient records and output primary and secondary ICD-10-CM diagnosis codes. Answers are checked against codes assigned by professional coders. | Expert | Feb 2026 | 63.6% | Not in cards |
| MedScribe Models write SOAP notes from clinical visit transcripts. Notes are graded against expert gold-standard rubrics for required content and section structure. | Expert | Feb 2026 | 91.4% | Not in cards |
| MentalBench (Psychiatric Diagnosis) Models select psychiatric diagnoses for 24,750 synthetic cases. Exact matching requires both diagnoses when the case lacks the evidence needed to distinguish them. | Expert | Feb 2026 | 62.7% | Not in cards |
| MultiClinSum-2 Models summarize 1,000 clinical cases per language across 15 languages. Organizers report four automatic metrics and four language model judgments separately. | University | Feb 2026 | Not in cards | |
| Phylogenetics (Anthropic) Models analyze phylogenetic data and tree diagrams, testing quantitative reasoning about tree structure and visual interpretation. | Expert | Feb 2026 | 79.6% | System cards only |
| scBench (LatchBio SingleCellBench) 195 verifiable single-cell RNA-seq analysis problems. Each pairs a data snapshot with a task and a deterministic grader. | Expert | Feb 2026 | 63.9% | System cards only |
| scBench (original study) Agents complete 394 tasks in single-cell data analysis. The authors compare their answers with results from the original analyses. | Expert | Feb 2026 | 52.8% | Not in cards |
| SIM-VAIL Chatbots respond to simulated users with psychiatric vulnerabilities. A model judge scores behaviors across the conversation, including responses that reinforce a user’s vulnerability. | Expert | Feb 2026 | 1 | Not in cards |
| Structural Biology (Anthropic) Models answer questions about a biomolecule's function from structural data and basic tools, in multiple-choice and open-ended variants. | Expert | Feb 2026 | 79% | System cards only |
| A^3-Bench Models solve 2,198 science reasoning problems annotated with the anchors and attractors (prior knowledge) they need. An AAUI metric measures how much of that knowledge the model activates. | University | Jan 2026 | 65.1% | Not in cards |
| Artificial Analysis Intelligence Index v4.0 Version 4.0 of Artificial Analysis's index, ten evaluations in four equal categories that brought in real-world knowledge work and hallucination tests. | Jan 2026 | 60.2 | Fading | |
| BioNNE-R Models identify relations between supplied medical entity mentions in English and Russian abstracts. Organizers rank systems by macro F1 across 14 relation types. | University | Jan 2026 | Not in cards | |
| CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. | Expert | Jan 2026 | Not in cards | |
| DISPLACE-M 2026 Models identify speakers, transcribe their words, identify medical topics and summarize conversations between community health workers and patients. Each task has its own score. | Everyday | Jan 2026 | Not in cards | |
| GAIK Ajokortti transcription test Models transcribe one three-minute Finnish dental webinar excerpt with two speakers. The project reports word error rate and separately counts errors that change the meaning. | Everyday | Jan 2026 | | Not in cards |
| HealthBench-Hallu Models answer HealthBench-style medical prompts. A pipeline splits each answer into medical claims and checks them against medical evidence. The score is a weighted hallucination rate. | Expert | Jan 2026 | | Fading |
| MedGenVidQA 2026 Visual Answer Localization Models mark the start and end of the video segment that answers a medical question. The organizers created 80 test examples from professional medical videos. | University | Jan 2026 | Not in cards | |
| MedPhysBench Knowledge Questions Models answer board-style medical physics questions covering radiation oncology, dosimetry, imaging, and radiation protection. Version 0.2.0 contains 412 questions, including 115 test items. | Expert | Jan 2026 | Not in cards | |
| MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. | Expert | Jan 2026 | Not in cards | |
| OpenMed MedReason Evaluation Models answer medical questions from a held-out portion of the MedReason training corpus. The port scores answer accuracy, reasoning, and format separately. | Expert | Jan 2026 | Not in cards | |
| PHEB Models answer multiple-choice questions from Portuguese national secondary school exams administered between 2006 and 2023. | School | Jan 2026 | Not in cards | |
| PsychEthicsBench Models answer multiple-choice questions and respond to scenarios under Australian psychology and psychiatry ethics guidelines. Graders check responses against professional standards. | University | Jan 2026 | | Not in cards |
| QIMMA Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review. | School | Jan 2026 | | Not in cards |
| ScanBench Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage. | Expert | Jan 2026 | | Fading |
| SciPredict Models predict the outcomes of 405 real physics, biology and chemistry experiments published after March 31, 2025, with and without expert background knowledge. | Expert | Jan 2026 | | Not in cards |
| WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. | Expert | Jan 2026 | Not in cards | |
| Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. | Expert | 2026 | Not in cards | |
| Human Pathogen Capabilities Test v2 (HPCT v2) Models answer multiple-response questions on a small set of weaponizable pathogens that biosecurity experts rank highest priority. A revised version of HPCT. | Research frontier | 2026 | 80% | Not in cards |
| LatchBio Capabilities v1.0 LatchBio cross-benchmark suite of agentic biological data analysis (transcriptomics, epigenomics, variants, pharmacology) with deterministic graders. | Expert | 2026 | 58% | System cards only |
| BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. | Competition | Dec 2025 | | Fading |
| CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. | Research frontier | Dec 2025 | | Not in cards |
| FrontierScience Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. | Expert | Dec 2025 | | In use |
| FrontierScience-Olympiad Models solve olympiad-style physics, chemistry and biology problems at the level of IPhO, IChO and IBO, written by medalists and coaches. Graded on short final answers. | Competition | Dec 2025 | 77.1% | System cards only |
| FrontierScience-Research Models work on open-ended PhD-level research subtasks in physics, chemistry and biology, written by working scientists. A grader scores each answer against a 10-point rubric. | Expert | Dec 2025 | | Fading |
| HELM Arabic Stanford CRFM runs models on 7 Arabic benchmarks in the HELM harness and ranks them by mean score. The tasks are AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and human-translated MMLU. | School | Dec 2025 | 84.1 | Not in cards |
| MedCalc-Bench Verified Models extract values from patient notes and compute results for 55 medical calculators. Verified release of MedCalc-Bench with corrected labels. | Expert | Dec 2025 | 86.2% | System cards only |
| MEDIQA-SYNUR Models extract structured observations from synthetic nursing dictations and map them to a clinical vocabulary. The hub uses the development split by default and supports the test split. | Expert | Dec 2025 | Not in cards | |
| MMLU Early Training Models score complete answer texts on selected MMLU questions. The metric compares the correct answer’s log likelihood with the average for incorrect answers. | University | Dec 2025 | Not in cards | |
| Research Plan Generation Models write research plans from goals derived from scientific papers. Judges grade each plan against criteria specific to its source paper. | Expert | Dec 2025 | Not in cards | |
| SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. | Expert | Dec 2025 | | In use |
| SpatialBench (LatchBio) Agentic spatial-transcriptomics data analysis problems with deterministic graders. Anthropic reports a 115-problem Verified variant. | Expert | Dec 2025 | 57.6% | System cards only |
| SpatialBench (original study) Agents complete 146 tasks in spatial transcriptomics. The authors compare their answers with results from the original analyses. | Expert | Dec 2025 | 38.4% | Not in cards |
| ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. | Expert | Nov 2025 | 66.8% | System cards only |
| ATLAS (OpenCompass) Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge. | Research frontier | Nov 2025 | 42.9% | Not in cards |
| Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. | Expert | Nov 2025 | 73.7% | System cards only |
| CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. | Expert | Nov 2025 | System cards only | |
| HEAD-QA v2 Models answer Spanish healthcare specialization exam questions across six disciplines. Version 2 adds languages, and the MedARC port runs the English configuration. | Expert | Nov 2025 | | Not in cards |
| MarineEval Models answer 2,000 questions about marine images and documents. The authors combine answer checks with model judgments. They report total accuracy and mean accuracy across seven tasks. | University | Nov 2025 | 50.4% | Not in cards |
| MedBench 4.0 (OpenCompass) Models handle clinical scenarios in an evaluation that covers specialized models and medical applications. | Expert | Nov 2025 | Not in cards | |
| MedPT Models classify Brazilian Portuguese medical questions by specialty. The original evaluation tests 20 classes, while the hub creates a split after deduplication and label selection. | University | Nov 2025 | Not in cards | |
| MindEval Models hold multi-turn therapy conversations with simulated patients designed with licensed clinical psychologists. An LLM judge scores each session. | Expert | Nov 2025 | 3.98 | Not in cards |
| MultiClinAI Models identify diseases, symptoms, and procedures in seven languages. A second task transfers annotated Spanish mentions to the corresponding spans in six translations. | University | Nov 2025 | Not in cards | |
| NeuroDiscoveryBench Models analyze neuroscience, genetics and immunology datasets and answer questions grounded in figures and data workflows. | Expert | Nov 2025 | Not in cards | |
| Noor Models answer filtered academic questions chosen to distinguish early training checkpoints. The benchmark measures progress where standard MMLU scores are noisy. | University | Nov 2025 | Not in cards | |
| SMM4H-HeaRD 2026 Detection of Insomnia in Clinical Notes Models identify likely insomnia in nursing and discharge notes. They label diagnostic criteria and give the character offsets of supporting evidence. | University | Nov 2025 | Not in cards | |
| SMM4H-HeaRD 2026 Flu Vaccine Effectiveness Models classify posts by the author’s flu vaccination status and flu test result. The labels distinguish the current flu season from past seasons. | Everyday | Nov 2025 | Not in cards | |
| SMM4H-HeaRD 2026 MedSynth Dialogue-to-Note Models turn synthetic clinical dialogues into notes with four sections. The organizers generated 368 separate test dialogues and scored the notes against references. | University | Nov 2025 | Not in cards | |
| SMM4H-HeaRD 2026 Multilingual Adverse Drug Event Detection Models flag personal reports of drug side effects in seven languages. The organizers withheld Farsi from training and added 15,184 Farsi posts at test time. | Everyday | Nov 2025 | Not in cards | |
| SMM4H-HeaRD 2026 Pathology Report Staging Models assign tumor, lymph node, and distant metastasis stages to synthetic pathology reports. A tiebreaker removes explicit stage labels and requires inference from clinical findings. | Expert | Nov 2025 | 0.95 | Not in cards |
| Biomni-Eval1 Agents use biomedical data and tools to answer 433 questions across ten task types. Each task checks a structured answer against a reference. | Expert | Oct 2025 | Not in cards | |
| BioRiskEval Researchers measure biorisk-relevant capability kept by open-weight DNA and protein foundation models, with three task sets named Gen, Mut and Vir. | Expert | Oct 2025 | Not in cards | |
| HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. | Research frontier | Oct 2025 | Not in cards | |
| IMB Models answer Italian medical specialty exam questions and respond to patient questions from online forums. The paper reports results for both tasks. | University | Oct 2025 | | Not in cards |
| MedQA-Followup Models answer medical licensing questions and then respond to follow-ups that challenge their answers. Researchers compare accuracy before and after each intervention. | Expert | Oct 2025 | 0% | Not in cards |
| MentalBench-100k Models answer counseling questions. Human reviewers grade responses to 1,000 of the 10,000 prompts on seven attributes, each from one to five. | Expert | Oct 2025 | 5 | Not in cards |
| Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. | Expert | Oct 2025 | System cards only | |
| Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. | Expert | Oct 2025 | | System cards only |
| NautData test set Models answer questions, describe underwater scenes, and locate organisms in images. The paper reports 7,916 test examples on 3,920 images. | University | Oct 2025 | | Not in cards |
| RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. | University | Oct 2025 | | Not in cards |
| UWBench Models answer questions about underwater scenes, write captions, and locate objects from descriptions. Questions cover visibility, materials, and spatial relationships. | Everyday | Oct 2025 | Not in cards | |
| VERA-MH Suicide Risk Chatbots talk with simulated patients who show different levels of suicide risk. A judge grades the full conversation against a clinical rubric. | Expert | Oct 2025 | 75.0 | Not in cards |
| VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. | University | Oct 2025 | | Not in cards |
| APEX-1 Models complete expert-written work cases with source documents for four jobs: investment banking, consulting, big law and primary care. A judge model grades each response against rubrics. | Expert | Sep 2025 | | Not in cards |
| Artificial Analysis Intelligence Index v3 Version 3 of Artificial Analysis's index, ten evaluations that added agentic terminal and tool-use tasks to the knowledge, math and coding tests. | Sep 2025 | | Dropped | |
| BioML-bench Agents build and evaluate machine learning systems for biomedical experiments spanning proteins, drugs, single cells, imaging, and biomarkers. Task-specific metrics compare submissions with experimental reference results. | Expert | Sep 2025 | Not in cards | |
| CLEF 2026 SimpleText Task 1 Models rewrite sentences or whole Cochrane abstracts for the public. The organizers compared outputs with plain-language summaries from 175 new systematic reviews. | University | Sep 2025 | Not in cards | |
| CLEF 2026 SimpleText Task 2 Models detect or classify unsupported additions to simplified Cochrane abstracts. A generation subtask compares rewrites made with and without explicit checks against the source. | University | Sep 2025 | | Not in cards |
| GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. | Expert | Sep 2025 | | In use |
| Hallym University Women's Health Nursing Exam Models answer 50 Korean questions from a fourth-year nursing course. The authors compare accuracy and estimated ability with results from 111 students. | University | Sep 2025 | Not in cards | |
| JGPQA Models answer graduate-level science questions translated into Japanese from GPQA. | Expert | Sep 2025 | Not in cards | |
| PestCLEF 2026 Models extract structured facts from plant health documents. Organizers compare the predicted relationships with expert annotations on a held-out test set. | University | Sep 2025 | Not in cards | |
| Radiology's Last Exam (RadLE) Multimodal models make a single spot diagnosis from images alone on 50 hard radiology cases. Board-certified radiologists scored about 83%. | Expert | Sep 2025 | 30% | Not in cards |
| SciReasoner evaluation suite Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks. | Expert | Sep 2025 | | Fading |
| SciTS Models understand and generate scientific time series in 43 tasks across 12 domains. Signals range from one to ten million points. Each task uses its own metric. | Expert | Sep 2025 | System cards only | |
| SLPHelm UltraSuitePlus Models analyze recordings of children’s speech for speech sound disorders. The suite also includes transcription and symptom classification tasks. | Expert | Sep 2025 | Not in cards | |
| Artificial Analysis Intelligence Index v2.2 Version 2.2 of Artificial Analysis's index, which swapped the older math tests for AIME 2025 and added instruction following and long-context reasoning. | Aug 2025 | Not in cards | ||
| AstaBench Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem. | Expert | Aug 2025 | 58% | Not in cards |
| BiomedArena Researchers compare anonymous answers to biomedical research questions and vote for the response they prefer. | Expert | Aug 2025 | Not in cards | |
| Challenging prompts from red teamers with biosafety-relevant PhDs Models answer adversarial biology prompts written by red teamers with biosafety PhDs. Scored as not_unsafe rate from model training alone. | Expert | Aug 2025 | 93.6% | System cards only |
| Filtered, adversarial sample of production prompts Models answer filtered borderline and high-risk biology prompts that resemble production traffic. Scored as not_unsafe rate from model training alone. | Expert | Aug 2025 | 96.9% | System cards only |
| HealthBench Consensus Global Health Models answer health conversations that need adjustment for global context such as local care access. Scored as the failure rate. | Expert | Aug 2025 | 0% | System cards only |
| HealthBench Consensus Urgent Models answer ambiguous high-stakes health conversations validated by two or more physicians. Scored as the error rate in informing the user. | Expert | Aug 2025 | 0.40% | System cards only |
| HealthBench Hard Hallucinations Models respond to difficult health conversations that 2 or more physicians validated. The score is the hallucination rate, so lower is better. | Aug 2025 | 1.60% | Dropped | |
| HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. | Expert | Aug 2025 | 16% | Fading |
| Human Pathogen Capabilities Test Models answer expert-level questions on human pathogens from SecureBio. Scored as accuracy. | Expert | Aug 2025 | 80.7% | System cards only |
| Pairwise Bioweaponization Campaign Red teamers with biology PhDs compare anonymized responses from two models on bioweaponization prompts. Scored as the win rate for being safer. | Expert | Aug 2025 | 60% | System cards only |
| RedditImpacts 2.0 Models mark clinical and social consequences of opioid use in first-person accounts. The organizers scored 278 annotated posts and excluded 300 unlabeled decoys. | School | Aug 2025 | Not in cards | |
| TroubleshootingBench Models find and correct realistic experimental errors in expert-written wet lab protocols. Scored as pass@1. | Expert | Aug 2025 | 59% | System cards only |
| 3LM Models answer Arabic STEM questions taken from Arabic textbooks and worksheets, plus synthetic STEM questions from the same sources. A third part tests Arabic code generation. | School | Jul 2025 | | Not in cards |
| EHRQA (Google) Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning. | Expert | Jul 2025 | Not in cards | |
| EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. | Expert | Jul 2025 | Not in cards | |
| Fundus-MMBench Models identify retinal structures, classify diseases, and grade their severity in fundus photographs. Each of the 31 tasks contains 20 questions. | Expert | Jul 2025 | Not in cards | |
| HLE Bio/Chem Gold Models answer biology and chemistry questions from Humanity’s Last Exam whose answers reviewers supported with published research. | Expert | Jul 2025 | Not in cards | |
| PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. | Expert | Jul 2025 | Not in cards | |
| SciArena Researchers ask scientific literature questions and vote between two models' long cited answers. Votes give Elo ratings. | Expert | Jul 2025 | 1,151 | Not in cards |
| SciVid Video models answer questions about scientific videos across several disciplines. The benchmark tests whether models recognize procedures, objects, and events in specialized footage. | University | Jul 2025 | Not in cards | |
| Taiwan National Dental Technician Licensing Examination 2023 Models answer 194 text-only dental technology questions from Taiwan's 2023 licensing exam. The authors score accuracy at baseline and after each of three weeks. | University | Jul 2025 | Not in cards | |
| US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. | Expert | Jul 2025 | Not in cards | |
| 34th Japanese National Examination for Dental Hygienists Models answer 213 Japanese dental hygiene questions, including 139 with visual material such as instrument photographs and radiographs. | University | Jun 2025 | Not in cards | |
| ClearHarm Models answer unambiguously harmful CBRN questions. Judges score whether attacks elicit harmful responses from the model. | Expert | Jun 2025 | Not in cards | |
| ExpertLongBench Models write documents for 11 professional tasks. A judge compares facts in each document with a reference checklist and scores precision and recall. | Expert | Jun 2025 | | Not in cards |
| HealthQA-BR Models answer Portuguese multiple-choice questions about clinical care and public health in Brazil’s Unified Health System. | Expert | Jun 2025 | 86.6% | Not in cards |
| K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. | School | Jun 2025 | | Not in cards |
| K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. | School | Jun 2025 | | Not in cards |
| LLMEval-Med Models answer clinical questions drawn from patient records and expert scenarios. A model grades answers against physician-developed checklists. | Expert | Jun 2025 | Not in cards | |
| NTCIR-19 MedNLP-CALL Triage Classification Models classify synthetic emergency calls into three urgency levels from dispatcher and caller dialogue. Medical experts supplied the reference labels for the six-language task. | University | Jun 2025 | Not in cards | |
| Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. | Expert | Jun 2025 | | Fading |
| CoralVQA Models answer questions about coral identity, growth, and health in reef photographs. GPT-4o checks answers against references across 16 question types. | University | May 2025 | 41.8% | Not in cards |
| CounselBench-Adv Models answer 120 questions written by mental health professionals. Each question targets one of six problems identified in earlier counseling responses. | Expert | May 2025 | 0.37 | Not in cards |
| Creative biology Models answer questions about unusual biology needed to engineer harmless biological agents, a weak proxy for novel weapon design. | Expert | May 2025 | 60.3% | System cards only |
| CSVQA Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images. | School | May 2025 | 49.6% | System cards only |
| DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. | Expert | May 2025 | 80% | System cards only |
| HealthBench 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. | Expert | May 2025 | 67.2% | In use |
| HealthBench Consensus Models answer HealthBench conversations graded only on criteria that several physicians agreed on. | Expert | May 2025 | 96.6% | Dropped |
| HealthBench Hard Models answer the 1,000 hardest HealthBench conversations, graded against physician-written rubrics. | Expert | May 2025 | 46.2% | Fading |
| HealthBench Meta Models judge whether a medical response meets a specific rubric item. The evaluation compares each binary judgment with physician labels and reports agreement metrics. | Expert | May 2025 | 0.71 | Not in cards |
| iNaturalist Sounds Models identify animal species from recordings submitted to iNaturalist. | Expert | May 2025 | Not in cards | |
| MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. | University | May 2025 | | Not in cards |
| RBench-M Models answer 665 graduate-level multimodal reasoning questions in English and Chinese. Top-1 accuracy compares selected answers with the answer key. | Expert | May 2025 | 69.9% | Not in cards |
| ScienceBoard Computer-use agents do scientific workflows in desktop software such as ChimeraX, Celestia, GRASS GIS and Lean 4. The apps are modified to expose state for checking. | Expert | May 2025 | 68% | Not in cards |
| Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. | Expert | May 2025 | System cards only | |
| SoSBench Models answer hazardous requests drawn from real regulations in six scientific fields. Scored by how often the model complies with a request it should refuse. | Expert | May 2025 | 10.6% | Not in cards |
| VM14K Models answer Vietnamese multiple-choice questions across 34 medical specialties. The authors reserve a private set for leaderboard evaluation. | University | May 2025 | Not in cards | |
| xbench-ScienceQA Models answer postgraduate-level science questions written by experts, in Chinese, as part of the xbench AGI Tracking series. | Expert | May 2025 | 76.4% | Not in cards |
| BRIDGE Models work on 87 tasks built from real clinical text, such as electronic health records, in nine languages. The tasks span triage, consultation, information extraction, diagnosis, prognosis and billing codes across 14 specialties. Each task uses its own automatic metric, and the board reports the average. | Expert | Apr 2025 | | Not in cards |
| Japanese National Dental Examination 2025 Models answer 350 questions from Japan's 2025 dental licensing exam. The study reports accuracy separately for 202 text questions and 148 visual questions. | University | Apr 2025 | Not in cards | |
| Japanese National Examination for Occupational Therapists 2025 Models answer 199 Japanese occupational therapy licensing questions. The authors score answers against the official key and compare performance on text and visual questions. | University | Apr 2025 | 88.4% | Not in cards |
| Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. | University | Apr 2025 | | System cards only |
| MultiClinSum Models summarize clinical case reports in English, Spanish, French, and Portuguese. Organizers compare each summary with a reference using ROUGE and BERTScore. | University | Apr 2025 | | Not in cards |
| CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. | Expert | Mar 2025 | System cards only | |
| HELM Capabilities Stanford CRFM runs language models on a curated set of scenarios, one per key capability, and averages the scores. Prompt-level results are public. | University | Mar 2025 | 81.9 | Not in cards |
| MedBench 3.0 (OpenCompass) Models handle medical tasks across an expanded clinical evaluation suite. Shanghai AI Laboratory reports results for participating models. | Expert | Mar 2025 | Not in cards | |
| Medical Prescription Selection (lm-evaluation-harness) Models read a diagnosis or complaint and choose the recorded prescription. The two task variants use different distractors. | Expert | Mar 2025 | Not in cards | |
| MicroVQA Models answer expert-written multiple choice questions about biology microscopy images that test image understanding, hypothesis generation and experiment proposal. | Expert | Mar 2025 | 59.4% | Fading |
| MIMIC-III Report Summarization Models generate the impression section of a clinical report from the preceding text. The harness compares the response with the recorded impression. | Expert | Mar 2025 | Not in cards | |
| MMLU-ProX MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. | University | Mar 2025 | | In use |
| Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. | School | Mar 2025 | System cards only | |
| SMM4H 2025 Food Recalls and Foodborne Disease Outbreaks Models classify news articles as recalls, outbreaks, or neither. They also extract details such as the product, cause, and number of people affected. | School | Mar 2025 | 0.88 | Not in cards |
| Artificial Analysis Intelligence Index v2 Artificial Analysis's index of seven public reasoning, knowledge, math and coding benchmarks, each run by Artificial Analysis. | Feb 2025 | 67.7 | Dropped | |
| BioMaze Models answer questions about biological pathways under normal and altered conditions. The benchmark includes true-or-false questions and written responses. | University | Feb 2025 | | Not in cards |
| BioNNE-L Models link nested terms in English and Russian biomedical abstracts to medical concept identifiers. Organizers report top-one accuracy, top-five accuracy, and mean reciprocal rank. | University | Feb 2025 | Not in cards | |
| Bioweapons knowledge questions Models answer 33 sensitive bioweapons questions written by Deloitte biosecurity staff. Answers are compared with novice, intermediate and expert baselines. | Expert | Feb 2025 | 51.5% | System cards only |
| BixBench Computational-biology analysis tasks (differential expression, assembly QC, cell typing) answered by agents with tools. | Expert | Feb 2025 | 93.8% | Fading |
| CareQA Models answer questions from Spanish specialist healthcare examinations. The evaluation includes English and Spanish versions and an open-response format. | Expert | Feb 2025 | | Not in cards |
| Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. | University | Feb 2025 | 25.0 | System cards only |
| EICAT impact classification Models read a scientific paper and classify the ecological impact of a named invasive species. They choose among five impact levels and a data deficient category. | Expert | Feb 2025 | | Not in cards |
| ELCardioCC 2025 Models find clinical mentions in Greek cardiology discharge letters, link them to ICD-10 codes, and assign codes to each letter with supporting phrases. | Expert | Feb 2025 | Not in cards | |
| GutBrainIE 2025 Models find biomedical entities and relationships in 40 held-out PubMed abstracts on the gut and brain. The organizers score entity spans and three relation extraction tasks. | University | Feb 2025 | Not in cards | |
| Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. | Expert | Feb 2025 | 69.7% | System cards only |
| MedHallu Models distinguish factual medical answers from hallucinated answers at three difficulty levels. The benchmark tests detection with and without supporting knowledge. | University | Feb 2025 | | Not in cards |
| MedHELM Models run a suite of medical tasks chosen by clinicians across five categories, from clinical decision support and note writing to patient messages and administration. | Expert | Feb 2025 | 70.3 | Not in cards |
| MMLU-Redux 2.0 Spanish Models answer corrected MMLU questions in Spanish across 57 subjects. The evaluation compares each selected option with the answer key. | University | Feb 2025 | Not in cards | |
| Open Arabic LLM Leaderboard v2 Models are scored on native Arabic tasks (AlGhafa native subsets, ArabicMMLU, Exams, MadinahQA, AraTrust, ALRAGE, ArbMMLU-HT) and ranked by the average. | School | Feb 2025 | | Dropped |
| PeerQA Models answer peer reviewers' questions about a scientific article using evidence from the article. | Expert | Feb 2025 | | Not in cards |
| Physiome-ODE Models forecast irregularly sampled biological time series generated from published differential equations. The authors compare forecasts using mean squared error. | University | Feb 2025 | Not in cards | |
| SCT-Bench Models rate how new clinical information changes a diagnosis or treatment decision. The grader awards partial credit according to the distribution of expert responses. | Expert | Feb 2025 | 70.5% | Not in cards |
| SMM4H-HeaRD 2025 Detection of Insomnia in Clinical Notes Models identify likely insomnia in clinical notes, label individual criteria, and quote supporting text. The organizers score evidence with ROUGE-L. | University | Feb 2025 | 0.97 | Not in cards |
| SMM4H-HeaRD 2025 Multilingual Adverse Drug Event Detection Models classify posts in English, French, German, and Russian by whether the author reports a drug side effect. The French test translates German forum posts. | Everyday | Feb 2025 | Not in cards | |
| SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. | Expert | Feb 2025 | | In use |
| TituLLMs Bangla Evaluation Suite Models answer Bangla questions on academic subjects, science and everyday reasoning. The suite also asks yes-or-no questions about passages. | University | Feb 2025 | Not in cards | |
| Virology Capabilities Test (VCT) Models answer 322 multimodal virology troubleshooting questions written by expert virologists. Multiple-select variant requires all correct options. | Expert | Feb 2025 | 67.4% | Fading |
| CLEAR (Clinical Information Extraction) Models read patient notes and classify a medical condition as present, absent, or uncertain. Human reviewers supply the reference labels. | Expert | Jan 2025 | Not in cards | |
| HLE Calibration Models give a confidence score with each HLE answer. The benchmark reports RMS calibration error and AUROC between confidence and correctness. | Expert | Jan 2025 | | Fading |
| Humanity's Last Exam 2,500 expert-written closed-ended questions across 100+ subjects. Exact-match or multiple-choice answers graded by an LLM judge. | Expert | Jan 2025 | 54.8% | In use |
| Japanese Nephrology Board Renewal Questions 2014 to 2023 Models answer 209 Japanese nephrology board renewal questions from ten exam years, including questions with clinical images. Researchers compare answers with the society’s key. | Expert | Jan 2025 | 81.3% | Not in cards |
| LLMzSzŁ Models answer Polish academic and professional exam questions drawn from the Central Examination Board archives. | University | Jan 2025 | | Not in cards |
| MedAgentBench Agents use FHIR APIs in a virtual electronic health record to do 300 patient tasks in 10 categories written by physicians. | Expert | Jan 2025 | 69.7% | Not in cards |
| MedQA Bias Evaluation Models answer medical exam questions before and after Vals adds racial information. Vals compares accuracy across patient groups. | Expert | Jan 2025 | 96.5% | Not in cards |
| MedXpertQA Models answer 4,460 expert-level medical exam questions across 17 specialties, in a text subset and an image subset. Scoring is multiple-choice accuracy. | Expert | Jan 2025 | | Fading |
| MedXpertQA-MM Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. | Expert | Jan 2025 | | In use |
| MMVU 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. | University | Jan 2025 | | In use |
| Turkish Dental Specialty Examination 2020 and 2021 Models answer Turkish dental specialty entrance questions in basic and clinical sciences. The two official answer keys cover 238 scored questions, including questions with images. | Expert | Jan 2025 | Not in cards | |
| Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. | University | Jan 2025 | | In use |
| Biology-Instructions Models answer multi-omics sequence tasks on DNA, RNA, protein and multi-molecule inputs, such as property prediction and interaction tasks. Scored per task with task metrics. | Expert | Dec 2024 | | Fading |
| CURE Models retrieve passages that answer clinicians' questions at the point of care. | Expert | Dec 2024 | Not in cards | |
| DCASE 2025 Audio Question Answering Models answer questions about marine mammal calls, the timing of sounds, and overlapping events. Each of the three sections counts equally toward the final score. | University | Dec 2024 | 52.1% | Not in cards |
| Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. | University | Dec 2024 | 96% | Dropped |
| Global MMLU Lite Models answer MMLU questions in 15 languages, with 200 culturally sensitive and 200 culturally agnostic questions per language. All items are human translated or post-edited. Accuracy. | University | Dec 2024 | 95.7% | Dropped |
| MEDEC Models identify and correct errors in clinical notes across five categories. The OpenMed port uses the MEDEC-MS subset. | Expert | Dec 2024 | 69.1% | Not in cards |
| MIMIC-IV-BHC Models summarize a patient’s hospital course from clinical notes. The reference is the brief hospital course section of the discharge summary. | Expert | Dec 2024 | Not in cards | |
| MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. | University | Dec 2024 | 73.4% | Not in cards |
| Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. | Expert | Dec 2024 | 63.1% | System cards only |
| ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. | Expert | Dec 2024 | 47.2% | In use |
| Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. | Research frontier | Dec 2024 | System cards only | |
| Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. | Expert | Dec 2024 | 51% | System cards only |
| AfriMed-QA Models answer about 15,000 open and multiple-choice medical questions from over 60 medical schools in 16 African countries. Questions span 32 specialties. | University | Nov 2024 | | Not in cards |
| BEANS-Zero Models follow text prompts to classify, detect or caption animal sounds without task-specific training. | University | Nov 2024 | Not in cards | |
| CLEF 2025 SimpleText Task 1 Models simplify individual sentences or whole Cochrane abstracts. The organizers tested new reviews and scored both tasks against plain English references. | University | Nov 2024 | | Not in cards |
| CLEF 2025 SimpleText Task 2 Models flag unsupported sentences, classify distortions in simplified scientific text, or rewrite Cochrane abstracts while avoiding new claims. | University | Nov 2024 | Not in cards | |
| Cochrane-auto Models simplify sentences, paragraphs, or abstracts from Cochrane reviews. The authors aligned technical text with plain English summaries. | University | Nov 2024 | Not in cards | |
| INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. | University | Nov 2024 | 89.8% | Fading |
| JAMA Clinical Challenge (JesseLiu) Models choose among four answers to clinical vignettes from JAMA. The OpenMed port combines 1,521 cases and reserves about 5% for evaluation. | Expert | Nov 2024 | Not in cards | |
| Polish Medical Exams Models answer Polish medical, dental, and specialist examination questions. The benchmark also includes English editions of the medical and dental licensing exams. | Expert | Nov 2024 | 79.9% | Not in cards |
| ScholarQABench Models answer scientific questions by synthesizing research papers. The benchmark checks citations and grades long answers against expert rubrics. | Expert | Nov 2024 | | Not in cards |
| CBT-Bench Level II Models identify cognitive distortions and classify patients’ core beliefs at two levels of detail. The evaluator compares the selected labels with the reference labels. | Expert | Oct 2024 | 0.63 | Not in cards |
| DiscourseEE Models read Reddit discussions about opioid use disorder and extract details about medication use, tapering, and relapse. Some details require inference across a conversation. | University | Oct 2024 | | Not in cards |
| European LLM Leaderboard (OpenGPT-X) Models answer exam, science, commonsense, truthfulness, and math questions across European languages. The leaderboard also includes multilingual reading comprehension. | School | Oct 2024 | | Not in cards |
| MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. | University | Oct 2024 | Not in cards | |
| TaxaBench-8k Models identify species or match observations across photographs, sounds, and other field data. The test contains 8,813 observations paired across six types of data. | Expert | Oct 2024 | Not in cards | |
| xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. | University | Oct 2024 | | System cards only |
| ArbMMLU-HT Models answer a human translation into Arabic of the 57 MMLU subjects. | University | Sep 2024 | | Not in cards |
| Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. | Expert | Sep 2024 | System cards only | |
| CORE-Bench Agents reproduce the results of published studies from their code and data. 270 tasks from 90 papers in computer science, social science and medicine come in three difficulty levels. | Expert | Sep 2024 | 77.8% | Not in cards |
| Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. | Expert | Sep 2024 | System cards only | |
| Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. | Expert | Sep 2024 | 0% | System cards only |
| MMMLU MMLU professionally translated into 14 languages. Multiple-choice accuracy. | University | Sep 2024 | 92.7% | In use |
| MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. | University | Sep 2024 | 84.6% | In use |
| ThaiExam Models answer multiple-choice questions from standardized examinations in Thailand, such as ONET, IC, TGAT, TPAT-1 and A-Level. | School | Sep 2024 | 75.1% | Not in cards |
| BioLP-bench Find the introduced error in a biological lab protocol. Tests protocol understanding. | Expert | Aug 2024 | 49% | System cards only |
| GMAI-MMBench Vision-language models answer medical multiple-choice questions over many imaging modalities, clinical tasks and departments, at several perceptual granularities. | Expert | Aug 2024 | | Not in cards |
| MMLU-Redux 2.0 Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them. | University | Aug 2024 | Not in cards | |
| Open Ko-LLM Leaderboard 2 Models complete nine Korean evaluations covering reasoning, instructions, social values, safety, and other abilities. The organizers score submitted models on private test data. | School | Aug 2024 | 49.6% | Not in cards |
| PLABA 2024 Term Replacement Models identify difficult medical terms, classify how to adapt them, and supply wording for patients. The organizers held out 300 abstracts. | University | Aug 2024 | Not in cards | |
| Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. | Expert | Aug 2024 | 94% | System cards only |
| Biological Wastewater Treatment Long-Form QA Models answer 286 closed-book questions about biological wastewater treatment. Three experts grade the answers for relevance and factual accuracy. | University | Jul 2024 | Not in cards | |
| HaluBench Models judge whether an answer is faithful to its context across about 15,000 labeled context, question and answer triplets. It tests hallucination detectors and LLM judges. | University | Jul 2024 | 78.8% | Not in cards |
| INAS hypothesis span prediction Models mark the words in a scientific title or abstract that indicate a particular ecological hypothesis. Experts marked the evidence in 750 invasion biology abstracts. | University | Jul 2024 | Not in cards | |
| LAB-Bench Multiple-choice tasks on practical biology research skills: literature, protocols, sequence and figure reasoning, database lookup. | Expert | Jul 2024 | 80.7% | System cards only |
| LAB-Bench CloningScenarios Models answer multiple-choice questions about multi-step molecular cloning workflows. CloningScenarios subset of LAB-Bench. | Expert | Jul 2024 | | Fading |
| LAB-Bench FigQA Models interpret figures from biology papers and answer multiple-choice questions. FigQA subset of LAB-Bench. | Expert | Jul 2024 | 80.4% | Fading |
| LAB-Bench LitQA2 Models answer multiple-choice questions whose answers appear once in recent biology papers. Built to need literature search rather than recall. A LAB-Bench subtask. | Expert | Jul 2024 | 27% | Not in cards |
| LAB-Bench ProtocolQA Models find and fix errors in lab protocols in multiple-choice form. ProtocolQA subset of LAB-Bench. | Expert | Jul 2024 | 90.7% | Fading |
| LAB-Bench SeqQA Models answer multiple-choice questions that require DNA and protein sequence manipulation. SeqQA subset of LAB-Bench. | Expert | Jul 2024 | | System cards only |
| MedBench 2.0 (OpenCompass) Models answer Chinese medical questions. This release changes the evaluation of open responses. | Expert | Jul 2024 | Not in cards | |
| Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. | University | Jul 2024 | | System cards only |
| QEVD-FIT-COACH Models watch a workout and decide when to give feedback. The benchmark scores feedback timing and the content of the coaching advice. | Everyday | Jul 2024 | Not in cards | |
| SciCode Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. | Expert | Jul 2024 | | In use |
| TurkishMMLU Models answer over 10,000 multiple choice questions that Turkish curriculum experts wrote for nine high-school subjects. Scoring is accuracy. | School | Jul 2024 | 79.1% | Dropped |
| GAOKAO-Eval 2024 Models take Chinese national college entrance examinations. High school teachers grade the open responses. | School | Jun 2024 | | Not in cards |
| INQUIRE-Fullrank Models retrieve wildlife photographs from five million iNaturalist images for ecological queries. The benchmark measures average precision among the first 50 results. | University | Jun 2024 | Not in cards | |
| INQUIRE-Rerank Models reorder 100 photographs supplied for each ecological query. The benchmark measures average precision among the first 50 results. | University | Jun 2024 | 62.4% | Not in cards |
| IrokoBench Models answer human-translated knowledge, math and inference questions in 16 African languages. The parts are AfriMMLU, AfriMGSM and AfriXNLI. | School | Jun 2024 | 59% | Not in cards |
| MedCalc-Bench (original) Models apply 55 clinical calculators to patient notes. The original test contains 1,100 questions and checks numerical answers against task-specific tolerance bounds. | Expert | Jun 2024 | | Not in cards |
| Miðeind Icelandic ARC-Challenge Models answer multiple-choice science questions in an Icelandic translation of ARC-Challenge. Miðeind used Claude 3.5 Sonnet to translate the questions. | School | Jun 2024 | 95.2% | Not in cards |
| MMLU-Pro Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. | University | Jun 2024 | | In use |
| MMLU-Redux Manually re-annotated subset of MMLU that fixes wrong labels. Multiple-choice accuracy. | University | Jun 2024 | | Fading |
| MMLU-SR Models answer MMLU questions after key terms are replaced with synonyms. The benchmark tests whether models recognize the same concept under different wording. | University | Jun 2024 | 65% | Not in cards |
| OlympicArena Models solve 11,163 bilingual problems from 62 international olympiad competitions in seven fields, some with images. Answers are checked against keys. | Competition | Jun 2024 | 40.5% | Dropped |
| Open LLM Leaderboard v2 Hugging Face ran open models on IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro and averaged normalized scores. | University | Jun 2024 | | Not in cards |
| ProteinLMBench Models answer 944 hand-checked multiple choice questions on protein sequence, structure and function. | Expert | Jun 2024 | 57.9% | Dropped |
| RadNLP 2024 Radiology Report Segmentation Models label sentences in lung cancer reports with eight categories, including tumor size and distant metastasis. The organizers score labels using micro F2. | Expert | Jun 2024 | Not in cards | |
| SciKnowEval Models answer scientific questions and apply knowledge in biology, chemistry, materials and physics. The suite uses automatic checks and model judges for different tasks. | University | Jun 2024 | 0.71 | Not in cards |
| AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. | Expert | May 2024 | 37.8% | Not in cards |
| ARC Challenge MT Models answer science questions from ARC-Challenge in translation. The harness scores answers in the target language against the original choices. | School | May 2024 | Not in cards | |
| CADECv2 Models extract adverse drug reactions and drug names from patient reviews. The corpus covers 26 drugs across four groups. | University | May 2024 | Not in cards | |
| EnviroExam Models answer multiple choice questions from 42 environmental science courses. The composite index discounts mean accuracy when performance varies across courses. | University | May 2024 | | Not in cards |
| MedLFQA Models write long answers to biomedical questions. The benchmark checks whether answers include required facts and avoid claims identified as incorrect. | Expert | May 2024 | 77.1 | Not in cards |
| MedReadMe Models rate how easy a medical sentence is to understand. Researchers compare the ratings with judgments from readers. | Everyday | May 2024 | Not in cards | |
| Numerical results extraction from randomized controlled trials Models extract group sizes, event counts, means, and standard deviations from trial reports. The test contains 656 treatment comparisons across 110 papers. | University | May 2024 | | Not in cards |
| Open Arabic LLM Leaderboard v1 Models are scored on AlGhafa with native and translated sets, the ACVA and AceGPT sets, and translated MMLU and EXAMS. The board ranks by the average. | School | May 2024 | | Not in cards |
| PitVQA Models answer questions about a frame from pituitary surgery. Questions cover the surgical step, the instruments in view, and what the surgeon is doing. | Expert | May 2024 | Not in cards | |
| Japanese National Dental Examination 2024 Models select answers to 353 Japanese dental licensing questions, including 149 that use visual material. The exam covers 17 dental specialties. | University | Apr 2024 | 81.9% | Not in cards |
| MultiCardioNER Models mark disease mentions in Spanish cardiology cases and medication mentions in Spanish, English, and Italian versions of the cases. Organizers score exact spans with microaveraged F1. | University | Apr 2024 | Not in cards | |
| NASA-IR Models retrieve the scientific passage that answers a question. The test measures how often the correct passage appears among the first ten results. | University | Apr 2024 | Not in cards | |
| STaRK Retrieval systems find the right items for complex queries over knowledge bases that mix free text with relations between entities. The three domains are product search, academic paper search and precision medicine. Most queries are synthesized and checked by people, and a smaller set is written by people. | University | Apr 2024 | Not in cards | |
| BirdSet Models identify bird species from audio recordings across multiple evaluation datasets. | Expert | Mar 2024 | Not in cards | |
| CT-RATE Models read 3D non-contrast chest CT volumes and say whether each of 18 conditions is present. Scored by macro accuracy or F1 against report-derived labels. | Expert | Mar 2024 | Not in cards | |
| EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. | School | Mar 2024 | | Not in cards |
| JASMINE Models replace medical jargon in Japanese sentences with everyday language. The reference sentences come from patients' blogs. | University | Mar 2024 | Not in cards | |
| KorMedMCQA Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. | Expert | Mar 2024 | 92.7% | In use |
| MedConceptsQA Models answer multiple-choice questions about medical codes. The hub supports the original dataset and a separate 2,000-example sample. | Expert | Mar 2024 | 61.9% | Not in cards |
| MedSafetyBench Models respond to requests that conflict with medical ethics. A judge rates harmfulness on a five-point scale using a held-out set of 900 requests. | Everyday | Mar 2024 | | Not in cards |
| NTCIR-18 HIDDEN-RAD Models explain chest radiograph findings from reports or structured observations. The final score combines text similarity, language model judgments, and expert review. | Expert | Mar 2024 | Not in cards | |
| NTCIR-18 MedNLP-CHAT Models flag medical, legal, and ethical risks in answers to patient questions. The organizers compare the labels with expert judgments using macro F1. | Expert | Mar 2024 | Not in cards | |
| RadNLP 2024 Lung Cancer Staging Models assign detailed lung cancer stages to radiology reports. The organizers score Japanese and English separately and require all three stage labels to match. | Expert | Mar 2024 | Not in cards | |
| WMDP Proxy multiple-choice benchmark for hazardous knowledge in bio, chem and cyber. Released with the RMU unlearning method. | Expert | Mar 2024 | 84.8% | System cards only |
| WMDP Biology Models answer 1,273 multiple-choice questions that proxy hazardous biosecurity knowledge. Biology split of WMDP. | Expert | Mar 2024 | 91% | Fading |
| ArabicMMLU Models answer 14,575 multiple-choice questions in Modern Standard Arabic across 40 tasks, taken from school exams in North Africa, the Levant and the Gulf. | School | Feb 2024 | | Dropped |
| BioNNE 2024 Models mark the spans and types of overlapping medical terms in English and Russian abstracts. Organizers score span matches using macro F1. | University | Feb 2024 | Not in cards | |
| EHRNoteQA Models read a patient's accumulated MIMIC-IV discharge summaries and answer clinical questions about them. The set has 962 question and answer pairs, each for a different patient. | Expert | Feb 2024 | 96.2% | Not in cards |
| Medbullets Models answer clinical questions patterned on US medical licensing examinations. The benchmark also evaluates explanations of the answers. | Expert | Feb 2024 | 68.8% | Not in cards |
| MedVidQA 2024 Query-Focused Instructional Step Captioning Models divide a medical demonstration into steps that answer a question. They give each step a caption and start and end times. | Everyday | Feb 2024 | Not in cards | |
| MedVidQA 2024 Video Corpus Visual Answer Localization Models search 48,605 medical videos for demonstrations that answer 52 new questions. They return ranked videos and the start and end times of each answer. | Everyday | Feb 2024 | Not in cards | |
| Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. | University | Feb 2024 | | Dropped |
| OmniMedVQA Models answer questions about 118,010 medical images across 12 imaging modalities. The authors measure accuracy against the correct answer option. | Expert | Feb 2024 | | Not in cards |
| RRG24 Models write findings and impressions from one or more chest X-rays per study. Organizers score each section separately using five metrics. | Expert | Feb 2024 | Not in cards | |
| tinyBenchmarks Models answer small selected subsets of established benchmarks. Item response estimates approximate accuracy on the full tests. | University | Feb 2024 | Not in cards | |
| AAP Periodontics In-Service Examination Models answer specialty examination questions used by North American periodontics residency programs. Researchers score answers against the Academy’s key for each exam year. | Expert | Jan 2024 | Not in cards | |
| Artificial Analysis Intelligence Index v1 Artificial Analysis's first composite score, called the Quality Index in 2024, which normalized and combined a few public benchmarks into one number. | Jan 2024 | Not in cards | ||
| Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. | Expert | Jan 2024 | System cards only | |
| CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. | University | Jan 2024 | | Not in cards |
| CMMU Models answer Chinese multimodal questions across seven school subjects. | School | Jan 2024 | 30.9 | Not in cards |
| ENEM Challenge Models answer multiple-choice questions from Brazil’s national high school examination in Portuguese. | School | Jan 2024 | Not in cards | |
| JMMLU Models answer 7,536 four-way multiple-choice questions in Japanese, from translated MMLU subjects and new Japan-specific subjects. Scored by accuracy. | University | Jan 2024 | | Dropped |
| K-QA Models answer 201 patient questions. The benchmark checks whether answers include essential statements from clinicians and whether any claims contradict those statements. | Expert | Jan 2024 | 67.7% | Not in cards |
| LongHealth Models answer 400 questions about long clinical records from 20 fictional patients. Tasks test retrieval, sorting and recognition of missing information. | University | Jan 2024 | 51.7% | Not in cards |
| MedBench 1.0 (OpenCompass) Models answer Chinese medical questions across clinical reasoning, language understanding, generation, knowledge, and ethics. | Expert | Jan 2024 | Not in cards | |
| MentalChat16K Counseling Evaluation Models answer 200 mental health questions from online forums. Judges rate each response from one to ten on seven counseling skills. | Expert | Jan 2024 | | Not in cards |
| PLABA 2024 Complete Abstract Adaptation Models rewrite 400 new biomedical abstracts for patients. Reviewers grade how well the rewrites preserve information and explain it clearly. | University | Jan 2024 | Not in cards | |
| Translated ARC-Challenge, HellaSwag and MMLU (Mistral) Models answer French, German, Spanish and Italian versions of ARC-Challenge, HellaSwag and MMLU. | School | Jan 2024 | System cards only | |
| HELM Lite A lightweight HELM board that runs models with in-context learning on ten scenarios. They include NarrativeQA, NaturalQuestions, OpenbookQA, MMLU, GSM8K, MATH, LegalBench, MedQA and WMT 2014. | University | Dec 2023 | 93.8 | Not in cards |
| LitQA v0 Models answer questions that require information from full text scientific papers. | Expert | Dec 2023 | 40.6% | Not in cards |
| ProteinGym Deep mutational scanning benchmarks. The model ranks mutations by predicted effect on protein function, scored by Spearman rank correlation with lab measurements. Anthropic reports a hard subset, as the correlation times 100. | Research frontier | Dec 2023 | System cards only | |
| Reddit-Impacts Models mark the words that describe clinical or social consequences of substance use in Reddit posts. The original test has 278 posts. | School | Dec 2023 | 0.42 | Not in cards |
| TMMLU+ Models answer traditional Chinese questions across 66 subjects. The evaluation includes elementary material and professional examinations. | University | Dec 2023 | 88.6% | Not in cards |
| GPQA Diamond 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. | Expert | Nov 2023 | | In use |
| GPQA Extended Models answer all 546 GPQA questions, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. | Expert | Nov 2023 | 48.8% | System cards only |
| GPQA Main Models answer the 448 question main set of GPQA, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. | Expert | Nov 2023 | | Dropped |
| MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. | University | Nov 2023 | | In use |
| IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. | School | Oct 2023 | | Dropped |
| NAPISUL-II Progress Test 2023 Models answer Portuguese multiple-choice questions from the October 2023 medical progress test in southern Brazil. Researchers compare their answers with the official key. | University | Oct 2023 | Not in cards | |
| RaceBias (Race-Based Medicine) Models answer clinical questions used to identify unsupported race-based medical beliefs. Evaluators check responses for those beliefs. | Expert | Oct 2023 | Not in cards | |
| AMMLU Models answer MMLU multiple-choice questions across 57 subjects, translated into Arabic with GPT-4. Scored by accuracy. | University | Sep 2023 | | Dropped |
| CompassRank OpenCompass ranks open and API language models on its own evaluation sets and publishes monthly rankings. Model makers submit a repository link or API endpoint to be tested. | Expert | Sep 2023 | 83.3 | Not in cards |
| CMB Models answer 11,200 Chinese medical exam questions and questions about 74 clinical cases. Exam answers receive exact-match scores. A model judge rates clinical answers for fluency, relevance, completeness, and medical proficiency. | Expert | Aug 2023 | | Not in cards |
| EBM-NLPmod Models extract participants, interventions, comparators, and outcomes from trial titles and methods sections. Annotators separate comparison treatments from interventions. | University | Aug 2023 | Not in cards | |
| MedAlign Models follow clinician-written instructions using a patient’s electronic medical record. Clinicians evaluate the responses. | Expert | Aug 2023 | Not in cards | |
| MTSamples clinical entity recognition Models mark medical problems, treatments, and tests in 163 synthetic discharge summaries. Researchers score exact and overlapping text spans separately. | University | Aug 2023 | Not in cards | |
| BLUEX Models answer questions from two Brazilian university entrance examinations. The questions cover high school subjects in Portuguese. | School | Jul 2023 | | Not in cards |
| EHRSHOT Models predict clinical outcomes from longitudinal electronic health records with few labeled examples for each task. | Expert | Jul 2023 | Not in cards | |
| Malaysian Sign Language Medical Dataset Models identify Malaysian Sign Language terms used in health care. The training and test lists share all seven signer names. | Everyday | Jul 2023 | Not in cards | |
| Med-HALT Models answer medical exam questions designed to reveal hallucinations. The hub port uses the false-confidence, fake-question, and none-of-the-above reasoning tasks. | Expert | Jul 2023 | | Not in cards |
| MedText Models suggest diagnoses or treatments for textbook-style patient presentations. The harness compares their text with the recorded completion. | Expert | Jul 2023 | Not in cards | |
| Okapi ARC Models choose answers to translated science questions from ARC-Challenge. | School | Jul 2023 | Not in cards | |
| Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. | University | Jul 2023 | | Dropped |
| SympTEMIST Models locate symptoms, signs, and findings in Spanish clinical cases. They also map given mentions to SNOMED CT codes in Spanish and five additional languages. | University | Jul 2023 | Not in cards | |
| CMMLU Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy. | University | Jun 2023 | | Fading |
| M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. | School | Jun 2023 | Dropped | |
| Mol-Instructions Models answer molecule-oriented, protein-oriented and biomolecular text instructions, such as reaction prediction, molecule description and protein function questions. Scored per task with task metrics. | Expert | Jun 2023 | | Fading |
| Mol-Instructions Chemical Entity Recognition Models identify chemical entities in biomedical text. The score is test-set F1. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions Chemical-disease Interaction Extraction Models extract chemical and disease interactions from text. The score is test-set F1. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions Chemical-protein Interaction Extraction Models extract chemical and protein interactions from text. The score is test-set F1. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions Multi-choice Models answer biomedical multiple-choice questions. The score is test-set accuracy. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions True or False Models answer biomedical true or false questions. The score is test-set accuracy. | Expert | Jun 2023 | Not in cards | |
| ScienceBenchmark Models translate questions about the Sloan Digital Sky Survey, cancer research, and research funding into SQL. The organizers run the queries and compare their results. | University | Jun 2023 | 23% | Not in cards |
| UK Dermatology SCE Sample Questions Models answer multiple-choice questions from the Federation’s dermatology practice exam. Researchers score answers against the official key. | Expert | Jun 2023 | Not in cards | |
| ACI-Bench Models turn doctor-patient conversations into clinical visit notes. The original benchmark compares generated notes with clinician-authored references. | University | May 2023 | Not in cards | |
| C-Eval 13,948 Chinese multiple-choice exam questions across 52 disciplines and four difficulty levels. Accuracy. | University | May 2023 | | Fading |
| GAOKAO-Bench Models answer Chinese national college entrance exam questions from 2010 to 2022 across nine subjects, with objective and subjective questions. | School | May 2023 | 72.2% | Dropped |
| MTS-Dialog Models write a clinical note section from a conversation between a doctor and a patient. The evaluation compares it with the reference note. | Expert | May 2023 | Not in cards | |
| MultiCochrane Models rewrite medical review sentences in plain language. The collection pairs technical and simplified wording in English, Spanish, French, and Farsi. | University | May 2023 | Not in cards | |
| PLABA 2023 Abstract Adaptation Models rewrite biomedical abstracts for patients. Four annotators independently adapted each of the 400 test abstracts. | University | May 2023 | Not in cards | |
| PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. | University | May 2023 | | Fading |
| TREC 2023 Clinical Trials Track Models rank trials for 50 simulated patients using answers to medical intake questionnaires. Some answers can be missing. | Expert | May 2023 | Not in cards | |
| BioCreative VIII Task 3 Genetic Phenotype Normalization from Dysmorphology Physical Examinations Models identify abnormal physical findings in pediatric exam notes and return Human Phenotype Ontology identifiers. A second task also scores the locations of the findings. | University | Apr 2023 | Not in cards | |
| Open LLM Leaderboard v1 Language models answer tasks from the original Open LLM Leaderboard suite. | University | Apr 2023 | | Not in cards |
| SYNERGY systematic review screening Models rank titles and abstracts against a review’s eligibility criteria. Reviewers’ inclusion decisions supply the reference labels. | University | Apr 2023 | 100% | Not in cards |
| Korean National Licensing Examination for Korean Medicine Doctors 2022 Models answer 340 questions from the Korean medicine licensing exam. Researchers remove the images, transcribe tables, and compare prompt variants against the answer key. | Expert | Mar 2023 | Not in cards | |
| MammalNet Models identify mammals and their behavior in videos. The benchmark covers 173 mammal categories and 12 behaviors, including nursing and hunting. | School | Mar 2023 | Not in cards | |
| MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. | University | Mar 2023 | 0.07 | System cards only |
| MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. | University | Mar 2023 | System cards only | |
| Med-EASi Models rewrite English medical passages in plain language. The references mark replacements, explanations, insertions, and deletions. | University | Feb 2023 | Not in cards | |
| MedVidQA 2023 Medical Instructional Question Generation Models write a question that a supplied medical video segment answers. They can use both the images and the subtitles. | Everyday | Feb 2023 | Not in cards | |
| MedVidQA 2023 Video Corpus Visual Answer Localization Models search 12,657 medical videos and return the times when a video shows the answer. Half of the 40 questions require visual evidence. | Everyday | Feb 2023 | Not in cards | |
| BCSC Self-Assessment Program ophthalmology questions Models answer ophthalmology board preparation questions across subspecialties. Researchers compare their choices with the academy’s answer key. | Expert | Jan 2023 | Not in cards | |
| EHRSQL Models convert clinical questions into SQL queries over electronic health records. The benchmark includes questions that the database cannot answer. | University | Jan 2023 | Not in cards | |
| MedProcNER Models identify procedures in Spanish clinical cases, link them to SNOMED CT codes, and list the codes for each case. | University | Jan 2023 | Not in cards | |
| MS-CXR-T Models compare a current and a prior chest X-ray of one patient and say whether a finding improved, stayed stable or got worse. | Expert | Jan 2023 | Not in cards | |
| OphthoQuestions ophthalmology board preparation Models answer multiple-choice ophthalmology questions from a subscription question bank. Each model selects one answer per question. | Expert | Jan 2023 | Not in cards | |
| ProbSum 2023 Models list active diagnoses and problems from hospital progress notes. Organizers compare the lists with a physician’s annotations using ROUGE-L F1. | Expert | Jan 2023 | Not in cards | |
| RadSum23 Models write the impression section of radiology reports from findings. One track includes chest images. Organizers rank submissions by RadGraph F1. | Expert | Jan 2023 | 38.0 | Not in cards |
| MultiMedQA Models answer medical exam, research, and consumer health questions across seven datasets. Multiple-choice components use answer accuracy. Clinicians and other raters assess free responses for factuality, precision, harm, and bias. | Expert | Dec 2022 | | Not in cards |
| INAS hypothesis classification Models read the title and abstract of an invasion biology paper and identify which of ten ecological hypotheses it addresses. A paper can address several hypotheses. | University | Nov 2022 | Not in cards | |
| MedBench (Russian closed-test platform) Models answer Russian medical questions and classify relationships between clinical statements. The platform also scores electrocardiogram labels on a separate task. | University | Nov 2022 | Not in cards | |
| MedSecId Models identify section boundaries and assign a section type in clinical notes. The authors annotated 2,002 notes and reserved 10% for testing. | University | Oct 2022 | Not in cards | |
| PHEE Models extract adverse effects and potential therapeutic effects from published medical cases. They identify the event triggers and details such as the treatment and the patient. | University | Oct 2022 | Not in cards | |
| NTCIR-17 Radiology Report TNM Staging Models assign tumor, lymph node, and metastasis categories to Japanese lung cancer reports. Joint accuracy requires all three categories to match. | Expert | Sep 2022 | Not in cards | |
| NTCIR-17 Social Media Adverse Drug Event Detection Models identify drug side effects in simulated social media posts across four languages. Exact match requires all 22 symptom labels to match. | University | Sep 2022 | Not in cards | |
| ScienceQA Models answer about 21,000 multiple-choice science questions from school curricula, many with images. Scored by accuracy. | School | Sep 2022 | | Dropped |
| BioRED Models extract biomedical relations from 600 PubMed abstracts. The hub port presents entity pairs and asks models to classify their relationship. | Expert | Jul 2022 | Not in cards | |
| Plain Language Adaptation of Biomedical Abstracts Models rewrite biomedical abstracts in plain language for people who asked health questions. The authors collected 750 abstracts for 75 questions. | University | Jul 2022 | Not in cards | |
| EndoVis18-VQA Models answer questions about organs, tools, tool positions, and interactions in images from robotic kidney surgery. The benchmark accepts class labels or sentence answers in separate tasks. | Expert | Jun 2022 | Not in cards | |
| Meta-Album Models classify unfamiliar images after seeing a few labeled examples. Tasks span ten domains, including microscopy, remote sensing and wildlife. | School | Jun 2022 | Not in cards | |
| CLARA-MeD Spanish Medical Text Simplification Models rewrite Spanish medical sentences for patients. The collection includes 3,800 aligned pairs and 1,200 clinical trial sentences that linguists simplified. | University | May 2022 | Not in cards | |
| DDXPlus Models diagnose simulated patients from symptoms, clinical history, and demographic information. The OpenMed port asks for the most likely condition among 49 possibilities. | Expert | May 2022 | Not in cards | |
| Animal Kingdom Models identify actions in wildlife videos that can show several behaviors at once. The benchmark also tests pose estimation and locating events described in text. | School | Apr 2022 | Not in cards | |
| BiodivNER Models identify and label phrases in biodiversity abstracts and dataset descriptions. Annotators distinguish six kinds of entities, including organisms, locations, and environments. | University | Apr 2022 | Not in cards | |
| MedMCQA Models answer multiple-choice questions from Indian medical entrance exams, AIIMS and NEET PG. Scoring is accuracy. | Expert | Mar 2022 | 77% | System cards only |
| ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. | School | Mar 2022 | 0.15 | Not in cards |
| Penn epilepsy clinic notes Models read dated paragraphs from epilepsy clinic notes and classify recent seizures. Two other questions ask for seizure frequency and the most recent seizure date. | University | Jan 2022 | Not in cards | |
| RuMedBench Models predict diagnoses, suggest symptoms to check, answer Russian medical questions, infer relationships between statements, and identify entities in drug reviews. | Expert | Jan 2022 | Not in cards | |
| DisTEMIST Models locate disease mentions in Spanish clinical cases and link them to SNOMED CT codes. The organizers score both tasks with microaveraged F1. | University | Dec 2021 | Not in cards | |
| LayMaker MTSamples Simplification Models rewrite medical transcription sentences for patients. Clinicians supplied the reference versions and kept sentences they already considered clear. | University | Dec 2021 | Not in cards | |
| StepUp gym attendance forecasts Models predict how each four-week exercise program changes weekly gym visits relative to a placebo. Researchers compare predicted effects with recorded gym check-ins. | University | Dec 2021 | Not in cards | |
| 2022 n2c2/UW Shared Task on Extracting Social Determinants of Health Models identify substance use, employment, and living situations in clinical notes. Organizers score events and their attributes across three tests using notes from two institutions. | University | Nov 2021 | Not in cards | |
| n2c2 2022 Contextualized Medication Event Extraction Models identify medication mentions and determine whether a clinician discusses a change. They classify each change by action, negation, timing, certainty, and who acted. | University | Nov 2021 | Not in cards | |
| n2c2 2022 Progress Note Understanding: Assessment and Plan Reasoning Models read a clinician's assessment and one part of the treatment plan. They choose one of four relationships. Organizers score the labels with macro F1. | University | Nov 2021 | Not in cards | |
| HealthAdvice Models classify a sentence from a medical research paper as strong advice, weak advice, or no advice. | School | Sep 2021 | Not in cards | |
| Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. | Expert | Jul 2021 | Not in cards | |
| BioCreative VII LitCovid Models assign one or more of seven research topics to COVID-19 articles using titles and abstracts. | University | Jun 2021 | Not in cards | |
| MS² Models synthesize the findings of multiple medical studies into a short answer to the research question of a systematic review. | Expert | Apr 2021 | Not in cards | |
| Cochrane Simplification Models turn medical review abstracts into plain-language summaries. The original evaluation compares generated text with Cochrane summaries, while the hub port adds a model judge. | Expert | Mar 2021 | Not in cards | |
| iNaturalist 2021 Models classify nature photographs by species. The 2021 competition uses a new collection of images and species labels. | University | Mar 2021 | Not in cards | |
| Natural World Tasks (NeWT 2021) Models distinguish pairs of labels in wildlife photographs. The 164 tasks include recognizing life stages, animal behavior and scene context. | University | Mar 2021 | Not in cards | |
| SLAKE Models answer English and Chinese questions about radiology images of the head, neck, chest, abdomen, and pelvis. Scoring is accuracy. | University | Feb 2021 | | Fading |
| CLEAR French Medical Text Simplification Models simplify French sentences from drug information, medical encyclopedias, and research reviews. The authors released 100 test pairs. | University | Dec 2020 | Not in cards | |
| MEDDOPROF Models find occupations and employment status in Spanish clinical notes. They identify whose occupation each mention describes and assign an occupational or clinical code. | University | Dec 2020 | Not in cards | |
| EXAMS Models answer high school exam questions in 16 languages across many subjects. Arabic reports use the Arabic subset. | School | Nov 2020 | Not in cards | |
| MedDialog Models generate a clinician’s response from a patient description. The harness also evaluates summaries of medical questions using reference text. | Expert | Nov 2020 | Not in cards | |
| PUBHEALTH Models classify public health claims as true, false, mixed or unproven. The original benchmark also asks models to explain their verdicts. | University | Nov 2020 | Not in cards | |
| MedICaT Models locate panels within a biomedical figure and match each panel to the relevant words in its caption. The original test scores word overlap. | University | Oct 2020 | Not in cards | |
| MedQA Models answer multiple-choice questions from medical licensing exams, usually the 12,723-question English USMLE set. Scoring is accuracy. | Expert | Sep 2020 | 86.3% | Dropped |
| MedQA Mainland China Models answer mainland China medical licensing exam questions in simplified Chinese. Scoring is accuracy. | Expert | Sep 2020 | 84% | System cards only |
| MedQA Taiwan Models answer Taiwan medical licensing exam questions in traditional Chinese. Scoring is accuracy. | Expert | Sep 2020 | 91% | System cards only |
| MMLU 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. | University | Sep 2020 | | In use |
| MMLU Anatomy Models answer MMLU multiple-choice questions from the Anatomy subject. Scoring is accuracy. | University | Sep 2020 | 89% | System cards only |
| MMLU Clinical Knowledge Models answer MMLU multiple-choice questions from the Clinical Knowledge subject. Scoring is accuracy. | Expert | Sep 2020 | 92% | System cards only |
| MMLU College Biology Models answer MMLU multiple-choice questions from the College Biology subject. Scoring is accuracy. | University | Sep 2020 | 95% | System cards only |
| MMLU College Medicine Models answer MMLU multiple-choice questions from the College Medicine subject. Scoring is accuracy. | University | Sep 2020 | 84% | System cards only |
| MMLU Medical Models answer the multiple-choice questions from the medical subjects of MMLU, such as anatomy, clinical knowledge and professional medicine. | University | Sep 2020 | | Not in cards |
| MMLU Medical Genetics Models answer MMLU multiple-choice questions from the Medical Genetics subject. Scoring is accuracy. | Expert | Sep 2020 | 96% | System cards only |
| MMLU Professional Medicine Models answer MMLU multiple-choice questions from the Professional Medicine subject. Scoring is accuracy. | Expert | Sep 2020 | 94% | System cards only |
| MMLU-stem Models answer the STEM subjects of MMLU, such as physics, chemistry, mathematics, and computer science, as multiple choice. Scoring is accuracy. | University | Sep 2020 | | Dropped |
| Cochrane RCT Summarization Models read the titles and abstracts of clinical trials and write the conclusions of the Cochrane review that included those trials. | Expert | Aug 2020 | Not in cards | |
| VAERS influenza vaccine adverse event extraction Models mark six kinds of clinical information in influenza vaccine safety reports, including neurological events, procedures, and time expressions. Researchers score exact or overlapping text spans. | University | Aug 2020 | Not in cards | |
| Evidence Inference 2.0 Models classify the direction of a treatment effect in a trial report and identify text that supports their answer. | University | May 2020 | Not in cards | |
| Medical Paper-Blog Dataset Models rewrite a sentence from a medical paper for a general reader. The reference sentence comes from a medical blog. | University | May 2020 | Not in cards | |
| SciFact Models retrieve research abstracts and determine whether they support or refute a scientific claim. | Expert | Apr 2020 | Not in cards | |
| TREC-COVID Models rank scientific articles that address specific information needs about COVID-19. | Expert | Apr 2020 | Not in cards | |
| SemClinBr Models identify and classify medical terms in Brazilian Portuguese clinical notes. The corpus contains 1,000 notes from multiple specialties. | University | Jan 2020 | Not in cards | |
| Trialstreamer clinical evidence extraction Models identify treatments, comparators, outcomes, and the direction of each reported effect in trial abstracts. Physicians annotated 100 test abstracts. | University | Jan 2020 | Not in cards | |
| QASC Models answer eight-choice grade-school science questions by combining supporting facts from a large text corpus. Accuracy checks the selected answer against the answer key. | School | Oct 2019 | Not in cards | |
| PubMed causal language use Models classify the wording of a research conclusion as a direct causal claim, a conditional causal claim, a correlation, or no relationship. | University | Sep 2019 | Not in cards | |
| PubMedQA Models answer biomedical research questions with yes, no, or maybe, given a PubMed abstract without its conclusion. Scored by accuracy. | University | Sep 2019 | 78.3% | Dropped |
| MEDIQA 2019 Question Answering Models answer consumer health questions or assess candidate answers. The shared task evaluates the relevance and quality of answers to medical questions. | Expert | Aug 2019 | Not in cards | |
| MeQSum Models condense consumer health questions into shorter questions while preserving the medical information needed for retrieval. Reference summaries provide the comparison. | University | Jul 2019 | Not in cards | |
| PUCRS inpatient fall detection Models identify whether a Portuguese progress note documents a fall. Nursing students labeled 355 of the 1,078 notes as fall related. | University | Jul 2019 | Not in cards | |
| CLEF eHealth 2019 Technology-Assisted Reviews Task 2 Models rank the articles returned by a systematic review search so reviewers can screen relevant titles and abstracts first. | University | Apr 2019 | Not in cards | |
| MedicationQA Models answer consumer questions about medications. The original corpus pairs questions with expert answers, and the hub evaluation grades answer accuracy and completeness. | University | Mar 2019 | Not in cards | |
| CheXpert Models read chest radiographs and label up to 14 observations. The test labels come from board-certified radiologists. | Expert | Jan 2019 | Not in cards | |
| COPIOUS biodiversity entity recognition Models mark names of organisms, places, habitats, people, and time expressions in historical biodiversity texts. The test measures how closely the marked phrases match human annotations. | University | Jan 2019 | Not in cards | |
| HEAD-QA Models answer questions from Spanish healthcare specialization exams in six disciplines. The hub port uses the English training and validation splits. | Expert | Jan 2019 | Not in cards | |
| MedQuAD Systems answer medical questions using a collection of 47,457 question-answer pairs from NIH websites. The OpenMed port uses a model judge on a subset. | University | Jan 2019 | Not in cards | |
| MIMIC-CXR (classification) Models read chest X-rays from Beth Israel Deaconess and say whether five common conditions are present. Scored by average F1 over the conditions. | Expert | Jan 2019 | Not in cards | |
| MIMIC-CXR report generation Models write the findings of a radiology report for a MIMIC-CXR chest X-ray. Scored against the radiologist's report, most often by RadGraph F1. | Expert | Jan 2019 | Not in cards | |
| PsyTAR Models identify adverse reactions in patient reviews of four antidepressants. The original corpus also labels withdrawal symptoms, reasons for treatment, and reported effectiveness. | University | Jan 2019 | Not in cards | |
| VQA-RAD Vision-language models answer clinicians' questions about 315 radiology images, including head CTs, MRIs and chest X-rays. Questions are closed yes or no, or open free text. | Expert | Nov 2018 | Not in cards | |
| OpenBookQA Models answer elementary science multiple-choice questions that need a given core science fact plus common knowledge. Scored by accuracy. | School | Sep 2018 | 96.6% | Fading |
| EBM-NLP Models mark the words that describe participants, interventions, and outcomes in clinical trial abstracts. Medical professionals supplied the test labels. | University | Jun 2018 | Not in cards | |
| FDA and CDC VAERS clinical and temporal annotations Models locate clinical features and time expressions in vaccine safety reports. Annotators also linked clinical features to their timing across 1,000 reports. | University | Jun 2018 | Not in cards | |
| PatchCamelyon Models identify whether a microscope image patch contains metastatic tumor tissue. | Expert | Jun 2018 | | Not in cards |
| PubMed-PICO Models classify sentences in structured medical abstracts. Section headings supply the reference labels, including those for the study population, intervention, and outcome. | University | Jun 2018 | Not in cards | |
| MedMentions Models identify biomedical concepts in PubMed abstracts and link text spans to identifiers in the Unified Medical Language System. | University | May 2018 | Not in cards | |
| iNaturalist 2018 Models identify species in photographs submitted by naturalists. The 2018 competition uses an expanded set of plant and animal categories. | University | Feb 2018 | Not in cards | |
| n2c2 2018 Adverse Drug Events and Medication Extraction Models extract drugs and adverse events from discharge summaries. They also identify which drug each event or medication detail belongs to. | University | Jan 2018 | Not in cards | |
| MADE 1.0 Models identify drugs, indications, and adverse events in cancer care notes. They link each drug to its attributes and effects. The challenge scores extraction, linking, and the combined task. | University | Nov 2017 | Not in cards | |
| SciQ Models answer crowdsourced multiple-choice science exam questions in physics, chemistry and biology, each with a supporting passage. | School | Jul 2017 | Not in cards | |
| Animals with Attributes 2 Models use shared attributes to recognize animal classes they did not see during training. The benchmark provides 37,322 photographs across 50 classes with 85 attributes. | School | Jun 2017 | Not in cards | |
| ChestX-ray14 Models read frontal chest X-rays from the NIH Clinical Center and say whether thoracic conditions are present. | Expert | May 2017 | Not in cards | |
| iNaturalist 2017 Models identify the species in photographs submitted by naturalists. Meta evaluates transfer to fine-grained species classification. | University | Apr 2017 | Not in cards | |
| TAC 2017 Adverse Drug Reaction Extraction from Drug Labels Models extract adverse reactions and their modifiers from prescription drug labels. They identify positive reactions and map them to standard medical terms. The organizers held out 99 labels. | University | Apr 2017 | Not in cards | |
| NTCIR-13 MedWeb Models label each short message for eight symptoms. The organizers score an exact match when the model gets all eight labels right. | Everyday | Aug 2016 | Not in cards | |
| AI2D Models answer multiple choice questions about grade school science diagrams. The set has about 5,000 diagrams and 15,000 questions, scored by accuracy. | School | Mar 2016 | | Fading |
| NFCorpus Models rank medical research documents for health questions expressed in everyday language. | University | Feb 2016 | | Not in cards |
| Hallmarks of Cancer Models label biomedical text with the cancer hallmarks it describes. The hub port assigns zero or more of ten labels to individual sentences. | University | Oct 2015 | Not in cards | |
| CADEC Models identify adverse drug reactions in patient reviews from the AskaPatient website. A reaction can span words separated by other text. | University | Dec 2014 | Not in cards | |
| Birdsnap Models identify the North American bird species shown in a photograph. | University | Jun 2014 | Not in cards | |
| SemEval 2015 Task 14 Analysis of Clinical Text Models identify disorder mentions and map them to terminology codes in clinical notes. A separate task asks for attributes such as negation and severity. | University | May 2014 | Not in cards | |
| i2b2/UTHealth 2014 Identifying Risk Factors for Heart Disease over Time Models find heart disease risk factors in notes about patients with diabetes. They mark when each factor was present relative to the note date. | University | Mar 2014 | Not in cards | |
| SemEval 2014 Task 7 Analysis of Clinical Text Models mark disorder mentions in clinical notes and map them to medical terminology codes. Organizers score recognition and normalization separately. | University | Nov 2013 | Not in cards | |
| DDIExtraction 2013 Models identify drug names in biomedical text and classify interactions between pairs of drugs. The OpenMed port covers interaction classification. | University | Jun 2013 | Not in cards | |
| ShARe/CLEF eHealth 2013 Task 1 Models locate mentions of disorders in clinical notes and assign medical terminology codes. The organizers score mention recognition and code assignment separately. | University | Feb 2013 | Not in cards | |
| BioASQ Systems answer biomedical questions written by experts, using PubMed articles as evidence. Questions are yes/no, factoid, list or summary types. A second task asks systems to index new PubMed articles with MeSH headings. | Expert | 2013 | Not in cards | |
| ADE Corpus Models identify adverse drug effects in medical case reports. The OpenMed port combines classification and relation examples and reserves 20% for evaluation. | Expert | Oct 2012 | Not in cards | |
| Clinical Abbreviation Sense Inventory Models expand an ambiguous abbreviation using the surrounding words in an English clinical note. Evaluators compare the answer with a clinician's annotation. | University | Oct 2012 | Not in cards | |
| UW Clinical Record Section Chunker i2b2 Discharge Summary Corpus Models locate section boundaries in discharge summaries and label each section. Researchers compare the predicted labels and boundaries with human annotations. | University | May 2012 | Not in cards | |
| NICTA-PIBOSO Models assign one or more labels to each sentence in a medical abstract. The six labels include population, intervention, and outcome. | University | Mar 2011 | Not in cards | |
| Caltech-UCSD Birds 200 Models identify bird species from photographs. The benchmark distinguishes species with similar appearance. | University | Sep 2010 | Not in cards | |
| i2b2 2009 Medication Extraction Challenge Models extract drug names and prescription details from discharge summaries. The organizers scored six fields against community annotations for 251 test summaries. | University | Jun 2009 | Not in cards | |
| Oxford Flowers 102 Models identify flower categories from photographs. The benchmark tests recognition across changes in shape, scale, and lighting. | University | Dec 2008 | Not in cards | |
| i2b2 2008 Obesity Challenge Models classify obesity and 15 related conditions from discharge summaries. The challenge scores what the note states explicitly and what a clinician can infer. | Expert | Mar 2008 | Not in cards | |
| i2b2 2006 Smoking Challenge Models read a discharge summary and classify the patient's smoking status. The labels include current smoker, past smoker, unspecified smoker, nonsmoker, and unknown. | School | Jul 2006 | Not in cards | |
| OHSUMED Systems rank medical articles for 106 questions from physicians. The original study measured precision and recall against physicians' judgments of which articles were relevant. | Expert | Jul 1994 | Not in cards | |
| 2010 i2b2/VA Challenge on Concepts, Assertions, and Relations in Clinical Text Models locate medical concepts, classify statements about problems, and determine relationships among problems, tests, and treatments. Organizers score each task separately. | University | Not in cards | ||
| 2018 n2c2 Cohort Selection for Clinical Trials Models read longitudinal patient notes and decide whether patients meet specified clinical trial eligibility criteria. | Expert | Not in cards | ||
| Agentic BAIM-LLM Evaluation (ABLE) Agents run a viral-protein redesign workflow. Nine tasks cover structure retrieval, design strategy, tool selection and biological AI model use. Outputs are auto-scored. | Research frontier | | Not in cards | |
| Animal-Bench Models answer 41,839 questions about animal videos. The overall score averages accuracy across 13 tasks, including action prediction and recognizing predator and prey behavior. | School | Not in cards | ||
| AP Biology Models take a simulated AP Biology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | 5.00 | System cards only | |
| Bedside Bench Models answer 500 physician-written clinical cases on diagnosis, workup, escalation and medication safety. An LLM judge grades each answer against the authors' criteria. | Expert | 94.6% | Not in cards | |
| BioCreative V Chemical-Disease Relation Task Models identify biomedical entities and chemical-induced disease relations in PubMed abstracts. The OpenMed port extracts chemical and disease names. | University | Not in cards | ||
| Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. | Expert | 2.53 | System cards only | |
| Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. | Expert | System cards only | ||
| ChemProt Models identify the relationship between a chemical and a protein in a biomedical abstract. The shared task evaluates five relationship types. | Expert | Not in cards | ||
| COVID-Dialogue Models read a patient’s questions about COVID-19 and generate a response. Doctor replies from medical consultations serve as references. | Expert | Not in cards | ||
| CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). | Expert | 75.9% | In use | |
| Discharge Me! Models draft discharge instructions and brief hospital course summaries from patient records. The shared task compares model output with clinical discharge documentation. | Expert | Not in cards | ||
| DrugProt Models read biomedical abstracts and identify which of 13 labeled relationships connects a marked chemical and protein. | University | Not in cards | ||
| EDB-Bio Models read experimental evidence and estimate the probability that a scientific hypothesis is true. | Expert | Not in cards | ||
| EKWbench Agents complete operational tasks such as sourcing reviews and referral triage. The score averages the share of verifier weight earned. | Expert | 74.4% | Not in cards | |
| EndoVis 2018 Robotic Scene Segmentation Models segment structures and surgical instruments in video from robot-assisted operations. The evaluation compares predicted masks with reference labels. | Everyday | Not in cards | ||
| Health dynamic conversation eval Simulated users hide key health facts at first, and models must ask for context to reach the correct clinical conclusion. | Expert | System cards only | ||
| Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. | Expert | System cards only | ||
| Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. | Expert | System cards only | ||
| High-risk health scenarios eval Adversarial health emergencies test whether models recognize urgency and red flags and direct users to help. Reports pass rate. | Expert | | System cards only | |
| LABBench2 SourceQuality Models assess the quality of scientific information sources. | Expert | | Not in cards | |
| LatchBio Capabilities V0 Agents complete biological data analysis tasks. The historical score averages the available benchmark scores equally and excludes missing scores. | Expert | 51.3% | Not in cards | |
| Medical Knowledge Self-Assessment Program (MKSAP) Models answer internal medicine board review questions from MKSAP. Scoring is accuracy. | Expert | 75% | System cards only | |
| MedQA USMLE (5 options) Models answer USMLE-style medical licensing questions with five answer options. Scoring is accuracy. | Expert | 86% | System cards only | |
| Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. | Expert | | System cards only | |
| Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. | Expert | | System cards only | |
| Molecular Biology Capabilities Test Models answer expert-level molecular biology questions from SecureBio. Scored as accuracy. | Expert | 68.7% | System cards only | |
| NeuroscienceBench Agents analyze neuroscience research data and report a scientific conclusion. The score counts correct conclusions among graded runs. | Expert | 41% | Not in cards | |
| Polaris Healthcare Safety Benchmarks Models conduct simulated healthcare conversations. The evaluation measures clinical safety, communication, and administrative task performance. | Expert | Not in cards | ||
| RareDiseaseBench Agents combine clinical and biological evidence to diagnose and manage rare diseases. A rubric grades their diagnoses and plans. | Expert | | Not in cards | |
| RCM-Bench-1 Agents handle healthcare revenue cycle work, such as claim submission, billing and eligibility checks, with tools. Checks cover tool calls, arguments, replies and final system state. | Expert | 94.2% | Not in cards | |
| ReproBAIT Agents independently reproduce biological AI models from published scientific papers. Scored as success rate. | Expert | 85% | System cards only | |
| SecureBio VMQA, single choice Models answer 350 virology troubleshooting questions in single-choice form. VMQA is an earlier version of the Virology Capabilities Test. | Expert | | System cards only | |
| SuperCLUE-MedAssist Medical assistants respond to Chinese healthcare questions. | Expert | Not in cards | ||
| SuperCLUE-Science Models solve Chinese science reasoning problems. | University | 80.8 | Not in cards | |
| TissueMicroscopyBench Agents examine tissue images and answer questions about cells, staining, and changes over time. | Expert | 34% | Not in cards | |
| TREC 2017 LiveQA Medical Task Models answer consumer health questions. The shared task compares responses with expert assessments of answer quality. | Expert | Not in cards | ||
| TREC BioGen 2024 Systems retrieve biomedical literature and synthesize evidence into reports that answer a question. | Expert | 79.9 | Not in cards | |
| TxBench-Antibody-Development Agents use evidence to make manufacturing, quality, and clinical development decisions about candidate therapeutic antibodies. | Expert | | Not in cards | |
| USABO Semifinal Exam 2020 Models take the 2020 USA Biology Olympiad semifinal exam. Scored out of 150 points. | Competition | 87.0 | System cards only | |
| USMLE Step 1 (sample questions) Models answer the official USMLE Step 1 multiple choice sample questions. Scored by accuracy. | Expert | 68.9% | System cards only | |
| USMLE Step 2 (sample questions) Models answer the official USMLE Step 2 multiple choice sample questions. Scored by accuracy. | Expert | 63.3% | System cards only | |
| USMLE Step 3 (sample questions) Models answer the official USMLE Step 3 multiple choice sample questions. Scored by accuracy. | Expert | 67.2% | System cards only | |
| Virology Capabilities Test (text-only) Models answer the text-only questions of the SecureBio Virology Capabilities Test. Scored as accuracy. | Expert | 46% | System cards only | |
| World-Class Bio Models answer expert-level biology questions in a SecureBio benchmark. Scored as accuracy. | Research frontier | 68.8% | System cards only |