Benchmark Almanac

Biology and medicine

Tasks that need biology or medical knowledge. We count 730 benchmarks here, 4 of them arenas. A benchmark belongs here when subject includes biological sciences, medical and health sciences, medical engineering, environmental biotechnology or industrial biotechnology.

How fast it grows

126 were released since July, more than the 114 in the first half of 2026.

  1. Earlier 173
  2. 2024 H1 62
  3. 2024 H2 55
  4. 2025 H1 75
  5. 2025 H2 79
  6. 2026 H1 114
  7. 2026 H2 126

How much is still open

No model has passed 85% on 250 of the 331 benchmarks here that score in percent, 76% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
67AI Lab L2 nuclear medicine report structuring Models extract findings from 300 synthetic PET/CT and SPECT reports into a fixed form. They also flag conflicting values in 17 reports. University Nov 2026 1.00 Not in cards
67AI Lab L3 nuclear medicine guideline questions Models answer questions using eight retrieved passages from nuclear medicine guidelines. They must abstain when the passages lack an answer. University Nov 2026 90.8% Not in cards
AbMaze Agents make choices in a simulated antibody optimization experiment under a fixed budget. Recorded assay results determine normalized improvement on 200 test episodes. Expert Nov 2026 Not in cards
Bio-PPI Risk: Serous vs Endometrioid Models estimate a cancer subtype probability from tumor protein measurements and protein interactions. The evaluation measures discrimination and probability accuracy. Expert Nov 2026 0.22 Not in cards
Bio-Probe Models give probabilities for 21 unpublished biology hypotheses before their results are known. Scored by Brier score against the outcomes. Research frontier Nov 2026 System cards only
BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. Expert Nov 2026 66.7% Not in cards
BioSimBench Models predict which genes change after drugs, gene knockdowns, and immune signals across four prediction tracks containing 60, 60, 60, and 27 cases. The overall score averages perturbation-specific F1 across those tracks on a zero-to-one scale. Expert Nov 2026 0.18 Not in cards
DiagTrace Models request information about a patient one turn at a time. The evaluator compares the sequence of requests with the published case history. Expert Nov 2026 0.40 Not in cards
EUREKA-Bench (OpenAI) Agents build predictive models for six biology research tasks. Scored as predictive performance with the best published predictor in each field set to 100. Research frontier Nov 2026 100.3 System cards only
FrontierBioML 25 machine learning tasks in computational biology, across antibody development, protein engineering, structural biology, binder design and functional genomics. An agent gets real experimental data and a brief, works offline for up to six hours and submits a prediction script. A sealed grader runs the script on held-out data, such as new antibody families or new protein targets, and scores each task from 0 at a simple baseline to 100 for a perfect answer. Expert Nov 2026 40.3 Not in cards
GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. Expert Nov 2026 16.4% Not in cards
Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. Expert Nov 2026 83.9% Not in cards
MedHEB Models retrieve related medical images, reports, and image regions. The authors evaluate how models rank relevant evidence across 107 retrieval tasks. University Nov 2026 24.6 Not in cards
MedPhysBench Models calculate doses and check medical physics worksheets for planted errors. A grader checks the units and accepts values within a specified tolerance. Expert Nov 2026 100% Not in cards
NPCBench Models follow patients with nasopharyngeal carcinoma through diagnosis and care. The main patient-level measure requires correct decisions at successive stages. Expert Nov 2026 23% Not in cards
OpenAI Epoch Capabilities Index (OECI) OpenAI's own capabilities index, fit with Epoch AI's method over 34 public and 35 internal benchmarks. Research frontier Nov 2026 170.1 System cards only
PWS-Health Web research systems answer healthcare questions from clinical trials, medical literature, registries, drug labels, and healthcare workflows. Expert Nov 2026 Not in cards
Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. Research frontier Nov 2026 44.7 Not in cards
Sequitur Research Lab Automation Agents run simulated wet-lab automation protocols. Scored out of 100 on how well each experimental phase is carried out, averaged over six runs. Research frontier Nov 2026 74.5 System cards only
Suicide Memory Bench AI systems, each a model in its own app or coding harness, talk with 128 fictional personas over 29 chats. Five chats carry suicidal statements or warning signs of Suicide Crisis Syndrome, and a final request follows, such as a farewell note. Four sub-benchmarks cross explicit statements and warning signs with one chat or separate chats joined only by memory. A model judge rates each reply on a 0 to 4 safety scale. Everyday Nov 2026 1.41 Not in cards
TrialDesignBench Models select a dose-finding design for an early oncology trial. The authors compare their choices with expert reference designs across 617 trials. Expert Nov 2026 0.52 Not in cards
WildlifeBench Wildlife Knowledge V1 Models answer 185 questions about wild species without search or tools. The overall score weights ten tasks equally, including taxonomy, conservation status, and geographic range. University Nov 2026 84.7% Not in cards
AI Professions Prompt Benchmark Models draft everyday work documents from fictional inputs: 100 prompts for tasks in 10 professions, such as accounting, law, healthcare administration and real estate, plus Excel and Microsoft 365, in English, Spanish, French and Portuguese. A judge model from another family checks 5 pass or fail criteria per prompt, fixed before any model ran. University Oct 2026 85.9% Not in cards
Artificial Analysis Decision Index Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning. Everyday Oct 2026 Not in cards
Artificial Analysis Intelligence Index v5 Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. Oct 2026 64.6 In use
Biological static hazard evaluation Models answer 75 questions across priority biological threat scenarios, each asked 10 times. The score is the share of hazardous responses. Expert Oct 2026 0.13% System cards only
CareManage-Bench Models respond to patient scenarios involving medication safety, care plans, and triage. The harness checks the responses against case specific criteria. University Oct 2026 Not in cards
CellCaution: Reasoning About Cell Composition Models reason about changes in cell populations and decide when the information supports a numerical claim about their composition. University Oct 2026 1.00 Not in cards
DistressBench Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. Expert Oct 2026 88.4% In use
EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. University Oct 2026 Not in cards
HealthAgentBench V2 Agents complete 75 healthcare tasks across nine categories in terminal environments. Task-specific verifiers check outputs against hidden labels, with model grading for radiology report correction. Expert Oct 2026 57.3% Not in cards
KMMLU-Pro 2025 Models answer questions from the 2025 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. University Oct 2026 Not in cards
KMMLU-Pro 2026 Models answer questions from the 2026 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. University Oct 2026 78.1% Not in cards
MedSafe-Dx v0.3 Models read synthetic primary-care intake summaries and decide whether to flag a dangerous condition. Rule-based labels determine weighted error costs, with missed serious cases costing seven times a false escalation or incorrect danger label. The score normalizes these costs against a fixed escalation baseline. Expert Oct 2026 73.5 Not in cards
micro1 Medical Index Models are scored on the equal-weight average of micro1 medical benchmarks, currently RealmPathology, PharmaBench and WellnessBench. Expert Oct 2026 71% Not in cards
MindEval2 Models support simulated patients from MindSim, a tool-using harness with detailed profiles. Psychologist-written rubrics (MHQR) replace Likert scales for scoring. Expert Oct 2026 4.08 Not in cards
Mitosis Bench Agents mark every dividing cell in a tumor image. Each mark must fall within 30 pixels of an expert label. The grader rejects extra or missing marks. Expert Oct 2026 32.4% Not in cards
OmicsBench v1 Agents analyze molecular data across eight benchmarks. LatchBio reports pass rates for 741 short tasks and 43 long tasks. Expert Oct 2026 40.8% Not in cards
OncologyBench Separate agents run analyses of cancer research data, and a final agent makes a decision on detection, diagnosis or treatment. Strict scoring needs full marks on both parts. Expert Oct 2026 50.7% Not in cards
PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. Research frontier Oct 2026 14% Not in cards
PharmaBench (micro1) Agents write clinical trial documentation inside 8 synthetic drug programs, across 120 tasks written by biostatisticians, medical writers, safety physicians and regulatory staff. Expert Oct 2026 17.8% Not in cards
RealmPathology Models interpret real anatomic pathology reports and must not claim a diagnosis, stage or treatment the report does not support. Rubrics have 772 positive and 299 penalty criteria. Expert Oct 2026 78% Not in cards
RSNA Knee AUC10 Models label twelve knee findings from images, radiology reports, or both. The grader measures how well their predictions distinguish findings on ten reference studies. Expert Oct 2026 0.92 Not in cards
RSNA Knee LLM Label Extraction Models extract twelve findings from knee MRI reports in a batch. The grader compares the labels with reference findings using macro ROC AUC. University Oct 2026 0.89 Not in cards
SAFE-Teen Health advice Models respond to simulated teens in health advice scenarios. The score is the published conversation failure rate. Everyday Oct 2026 9.00% Not in cards
SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. Expert Oct 2026 80.9 Not in cards
ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. Expert Oct 2026 0.53 Not in cards
SciTaste-Bench Models judge the research value of papers in six disciplines. The benchmark compares their judgments with a reference index built from seven dimensions. Expert Oct 2026 28.0 Not in cards
SDBench Diagnostic agents order tests, ask questions and commit diagnoses in sequential clinical cases. Scores combine diagnostic correctness with test cost. Expert Oct 2026 Not in cards
The Pushback Test Models advise a parent who questions urgent medical advice. The evaluation checks whether the model maintains the urgency required by NHS guidance. University Oct 2026 0.81 Not in cards
Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. Expert Oct 2026 68.4% In use
VIALS 2 Models interpret 161 professional life-science visual artifacts. The score is judged accuracy averaged across three rollouts. Expert Oct 2026 49.9% Not in cards
WellnessBench Models answer young people asking about eating, sexual safety, substances and self harm, in clinician-written single-turn and pushback dialogues. A pass must be both safe and helpful. Everyday Oct 2026 66% Not in cards
AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. Expert Sep 2026 System cards only
Advanced Screening Evasion Models propose strategies to evade DNA synthesis screening and design evading fragments. Scored from 0 to 1 on strategy quality and fragment evasion. Research frontier Sep 2026 0.73 System cards only
Artificial Analysis Intelligence Index v4.2 Version 4.2 of Artificial Analysis's index, which added multi-hour office work and document tasks and dropped GPQA Diamond. Sep 2026 Not in cards
Artificial Analysis Intelligence Index v4.3 Version 4.3 of Artificial Analysis's index, eleven weighted parts led by knowledge work, office tasks and Terminal-Bench 4.0. Sep 2026 56.0 In use
Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. Expert Sep 2026 72.2% System cards only
Biopharma Bench Agents take an employee seat inside one of 12 simulated biopharma companies and complete 71 professional assignments, such as a regulatory assessment, an out-of-specification investigation or an audit package. They must work out which records govern among contradictory files. A rubric grader checks 752 frozen criteria, with code checks for exact numbers. Expert Sep 2026 65.1% Not in cards
BioSecBench-Function Agents recover biological function from experimental data. Scored by verifiable checks. One of three LatchBio BioSecBench evaluations. Expert Sep 2026 73.6% System cards only
Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. Research frontier Sep 2026 0.47 System cards only
DAYJOB: Healthcare Agents handle short requests from healthcare staff using charts, claims exports, policies and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Expert Sep 2026 48.3% Not in cards
De novo protein binder design (Anthropic) Agents get a target protein structure and 24 hours to design 30 miniprotein binders. The evaluation covers 15 targets. Expert Sep 2026 85.1% System cards only
DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. Expert Sep 2026 40.0 Not in cards
EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. Research frontier Sep 2026 29.8 Not in cards
Fireworks Specialized Intelligence Index An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number. Expert Sep 2026 Not in cards
HealthBench-Psych v2 Models respond to 611 health conversations that clinicians selected for mental health relevance. Three model judges score each answer against its physician rubric. Expert Sep 2026 0.66 Not in cards
HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Research frontier Sep 2026 77.9% In use
K-Bench Models respond across multi-turn mental-health risk conversations involving suicide, self-harm, domestic violence, substance misuse, and no-risk cases. Expert Sep 2026 99% Not in cards
KlinikeBench Agents interview simulated patients, use clinical tools, and record diagnoses across 333 cases. A case passes only when the agent satisfies all four clinical criteria. Expert Sep 2026 29.4% Not in cards
LifeSci Diamond Models write regulatory affairs deliverables for drugs, biologics and medical devices, such as change assessment memos, dossier inconsistency logs and agency letters. The 55 tasks span five families: reporting categories, Module 2/3 dossier consistency, missing evidence, submissions and correspondence, and device vigilance. A model judge grades each answer against a task rubric, and a task passes only when every rubric line is met and no penalty fires. Expert Sep 2026 20% Not in cards
Medical Bench Agents read longitudinal dermatology charts and write clinical assessments. Expert-authored rubrics grade their findings and management decisions. Expert Sep 2026 88% Not in cards
Medical Speech-to-Text Benchmark Speech models transcribe 1,513 English medical recordings. A common scorer measures word errors, medical-term errors, drug-name errors, and dosage matches. Separate tests measure batch response time and final streaming latency. Expert Sep 2026 1.32% Not in cards
Medicinal Chemistry (Anthropic) Models see several real drug-discovery compounds and pick which had the highest or lowest measured ADME property. The set has 504 questions. Expert Sep 2026 65.3% System cards only
MentalHealthBench 1,215 synthetic mental-health conversations with 5,262 clinician-authored rubric criteria. An LLM judge scores safety and guidance quality. Expert Sep 2026 57.3% System cards only
MetagenomicsBench Agents analyze microbial communities from shotgun, amplicon, long-read and spatial metagenomics data. A grader checks the biological result. Expert Sep 2026 62.7% Not in cards
Miðeind Icelandic Terminology Models explain Icelandic knitting and crochet terms and name local species from English and Latin names. A model grades definitions. A program checks species names. University Sep 2026 78% Not in cards
Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. Expert Sep 2026 34% System cards only
Neuro Reasoning (dwin2) Models answer generated questions about EEG experiment design and preprocessing. The evaluator computes a reference answer for each question. University Sep 2026 Not in cards
OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. Expert Sep 2026 73.7% Not in cards
Periapical Lesions Bench Agents locate and grade lesions around tooth roots in panoramic dental X-rays. A perfect report matches every lesion and grade without adding extra boxes. Expert Sep 2026 18.7% Not in cards
Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. Research frontier Sep 2026 10.7 System cards only
SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. Expert Sep 2026 45.3% Not in cards
SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. Research frontier Sep 2026 0.46 System cards only
Sonetra Norwegian SNOMED CT Term Recognition Models transcribe Norwegian sentences read by a synthetic voice. A term counts as correct when every word in the medical term matches. University Sep 2026 66.4% Not in cards
TMMLU+ 1.1 Models answer traditional Chinese subject questions from a revised exam set. The revision corrects 197 answers and removes 539 questions from the original release. University Sep 2026 96.2% Not in cards
TxBench-Oligo-Discovery Agents make oligonucleotide discovery decisions, such as ASO and siRNA design, from real data. A grader checks each decision. Expert Sep 2026 62.2% Not in cards
VCT-v2 Models answer an updated set of expert virology troubleshooting questions from SecureBio. Scored as refusal-corrected accuracy. Expert Sep 2026 60.5% System cards only
WearableQA Models answer 4,084 ten-option questions using longitudinal wearable measurements, blood biomarkers, and demographics from 200 users. Questions test computations and physiological interpretation across single or multiple signals. Accuracy compares the selected option with the answer key. Expert Sep 2026 72.9% Not in cards
Apodex Discovery Solver systems of model, harness and tools run long, verifiable investigations on high-value problems scouted from 561 industries in 16 sectors. Research frontier Aug 2026 2.80 Not in cards
Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. Expert Aug 2026 0% System cards only
Bio-VQA Models identify organisms, describe visible traits, judge spatial relationships, and recognize individual animals across photographs. The benchmark has 24 tasks. University Aug 2026 0.91 Not in cards
Biosecurity VCT Models answer dual-use virus lab troubleshooting items. Accuracy counts should-refuse and should-comply items together. Expert Aug 2026 47.8% System cards only
BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. Expert Aug 2026 91.4% System cards only
BixBench3 Agents rebuild the full computational analysis of 20 published biology papers from raw data on a cloud VM. Output tables are graded against the papers' own artifacts. Research frontier Aug 2026 52.3 Not in cards
CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. Expert Aug 2026 1.80% System cards only
CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. Expert Aug 2026 3.50% System cards only
Chartography 100 tasks on specialized professional charts (Kaplan-Meier, candlestick, Sankey, Bode). Answers graded against expert references. Expert Aug 2026 90.7% In use
Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. Expert Aug 2026 38% System cards only
CliniCARE-Bench Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases. Expert Aug 2026 76.1% Not in cards
FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. Expert Aug 2026 20.6% Not in cards
GDPval Extended Models produce work products for occupations across nine GDP sectors. Mercor adds 132 new expert-written tasks to the 213 public ones and reports both scores. Expert Aug 2026 7.10% Not in cards
Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. Expert Aug 2026 4.00% System cards only
HealthBench-Psych v1 Models respond to 610 health conversations that clinicians selected for mental health relevance. Three model judges score each answer against its physician rubric. Expert Aug 2026 0.63 Not in cards
HealthBench-Psych-Hard Models answer mental health questions from conversations that also appear in HealthBench-Hard. Three model judges use the original physician rubrics. Expert Aug 2026 0.45 Not in cards
HLE Extended Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores. Expert Aug 2026 77.3% Not in cards
MedXpertQA MM Extended Models answer multimodal medical questions over clinical images and patient details. Mercor adds 51 new expert-written tasks to the 200 public ones and reports both scores. Expert Aug 2026 71.2% Not in cards
MLCR-AA Models reason over synthetic medical claim files of about 25k to 64k tokens, as a claims reviewer would. Three judge models check completeness and accuracy. Expert Aug 2026 Not in cards
MMMU-Pro Extended Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores. University Aug 2026 61.3% Not in cards
SYNERGY+ Models rank research records against a systematic review's eligibility criteria. Reviewers' final inclusion decisions supply the reference labels. University Aug 2026 Not in cards
Terminal-Bench-Science 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. Expert Aug 2026 68.1% In use
Transluce Mental Health Evaluation Models hold multi-turn conversations with simulated users facing mental-health scenarios. Automated judges score 14 helpful and harmful behaviors. Expert Aug 2026 Not in cards
Tuesday Work Index A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday. Expert Aug 2026 71.6 Not in cards
TxBench-Antibody-Discovery Agents make antibody discovery and development decisions from real experimental data. A grader checks each decision. Expert Aug 2026 67.7% Not in cards
VIALS Vision models answer 161 questions about lab images such as gel blots, plasmid maps, microscopy and flow cytometry plots. An LLM judge grades each answer against a key. Expert Aug 2026 65.8% Not in cards
AI Security Leaderboard Models face systematic jailbreak attacks across chemical, biological, radiological, nuclear, explosive and cyber domains. Scores count universal jailbreaks and estimate attacker cost. Expert Jul 2026 $60 Not in cards
Autointent-Bio Models face manual and synthetic dangerous and benign biology queries. xAI reports refusal recall on the dangerous ones. Jul 2026 100% System cards only
BioSecBench-Refusal Agents run 61 routine and 46 concealed-hazard biology analysis tasks. The score pairs red-team refusal with routine completion. Expert Jul 2026 62.4% System cards only
BioSecBench-Surveillance Agents turn raw pathogen sequencing data into surveillance results. The endpoint pass rate excludes refused runs. Expert Jul 2026 60.8% System cards only
IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. Research frontier Jul 2026 27.3% Not in cards
MedChemBench (OpenAI) Models work through 99 medicinal chemistry problems drawn from recent papers and patents. Each task mixes prose with structures, assay tables and plots, then asks for the program’s next decision. Expert Jul 2026 83.8% In use
OCT-Bench Models answer 10,076 questions about retinal scans, from anatomy to treatment and prognosis. The evaluator measures answer accuracy and counts invalid responses as incorrect. Expert Jul 2026 62% Not in cards
PatientAgentBench Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria. Expert Jul 2026 4.25 Not in cards
ProtocolQA Open-Ended (xAI) Models name the single most important mistake in a biology lab protocol in free text, with no answer choices. Expert Jul 2026 87% System cards only
RadLE 2.0 Models diagnose single-image radiology cases with a confidence score from 0 to 4, or say they do not know. A confidence-weighted index rewards right answers and penalizes confident errors. Expert Jul 2026 758.0 Not in cards
VariantBench Agents discover and interpret genetic variants from real sequencing analyses. A deterministic grader checks the result. Expert Jul 2026 52.8% Not in cards
AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. Research frontier Jun 2026 0.59 System cards only
ABC Bench Fragment Design Agents use biology tools to complete a computational genetic-construct design task. One task family of the Agentic Bio-Capabilities Benchmark. Expert Jun 2026 100% System cards only
ABC Bench Liquid-Handling Robot Agents use biology tools to plan liquid-handling robot steps for a construct-assembly task. One task family of the Agentic Bio-Capabilities Benchmark. Expert Jun 2026 100% System cards only
ABC Bench Screening Evasion Agents complete a computational task on nucleic-acid synthesis screening. One task family of the Agentic Bio-Capabilities Benchmark. Expert Jun 2026 82.9% System cards only
AutoMedBench Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission. Expert Jun 2026 66.5 Not in cards
Biology Model Refusal Evaluation Models answer severe, dual-use and benign biology prompts. Scored as safe rate on harmful prompts and not over-refusing benign ones. Expert Jun 2026 99.9% System cards only
BioTIER v1 Refuse Models receive 501 dangerous biology prompts on select agents, dual-use research and catastrophe avoidance. The score is the share correctly refused. Expert Jun 2026 98% System cards only
ClinHallu Models answer medical image questions and produce intermediate reasoning. The evaluation reports answer accuracy and separate hallucination rates for visual recognition, knowledge recall, and reasoning integration against annotated reasoning traces. Expert Jun 2026 80.1% Not in cards
Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. Expert Jun 2026 0.20 Not in cards
DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. Expert Jun 2026 71.5% Not in cards
EpiBench Agents analyze real epigenomics datasets in a sandbox with standard tools. A deterministic grader checks the biological result. Expert Jun 2026 39.6% Not in cards
GeneBench Pro Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. Expert Jun 2026 31.5% In use
GRAFT Models predict plant traits from gene expression in Arabidopsis. The authors compare regression and graph models against measurements from the same plants. University Jun 2026 Not in cards
HealthAgentBench Agents use terminal tools to complete 54 clinical and biomedical tasks across seven categories. Task-specific verifiers measure success, including a model judge for radiology report correction. Expert Jun 2026 55% Not in cards
K9-Bench Models answer 4,744 questions about dog behavior in 907 videos. The evaluator compares each free-form answer with five options that the model never sees. School Jun 2026 40.1% Not in cards
KINA Models answer 899 knowledge questions across 261 fine-grained disciplines, each with ten lettered options. Scored by accuracy against the key. Expert Jun 2026 53.2% In use
LifeSciBench Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. Expert Jun 2026 61.3% In use
MarineEVT Models answer 2,000 held-out questions about underwater events, including species interactions, movement, and timing. The authors report accuracy across five reasoning categories. University Jun 2026 43.7% Not in cards
MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. Expert Jun 2026 69.2 Not in cards
NatureBench Coding agents work on 90 tasks distilled from Nature-family papers, each in its own container, and try to match or beat the published results. Research frontier Jun 2026 23.3% Not in cards
Protein Design (Anthropic) Models generate novel protein sequences under design constraints and rank protein design libraries for experimental testing. Expert Jun 2026 42.5% System cards only
PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. Expert Jun 2026 95.3% Not in cards
Real Point-of-Care Queries Clinical AI systems answer 620 questions from practicing physicians across 30 specialties. Specialists compare anonymous answers for accuracy, utility, source quality, completeness and verifiability. Expert Jun 2026 -1.50% Not in cards
scBench-Long Agents run long single-cell RNA-seq analyses that need sustained empirical work and biological reasoning. Expert Jun 2026 36.9% Not in cards
scBench-Long (original study) Agents complete 21 tasks in long single-cell data analysis workflows. The authors compare their answers with results from the original analyses. Expert Jun 2026 25.4% Not in cards
TxBench-Preclinical-Pharmacology Agents analyze preclinical pharmacology data and make development decisions. A grader checks each decision. Expert Jun 2026 68% Not in cards
AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. Expert May 2026 0.80 System cards only
BiomniBench Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer. Expert May 2026 63.9 Not in cards
BioPhys-Bridge Models connect observations in biophysics papers to quantitative explanations and cite supporting evidence. The authors score the cited evidence against reference identifiers. Expert May 2026 0.36 Not in cards
Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. Expert May 2026 System cards only
CHI-Bench Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs. Expert May 2026 54.7% Not in cards
CUSP Models forecast scientific progress across 4,760 events in many fields, using only knowledge from before a set cutoff date. Tasks ask whether an advance is feasible, how it would work, how to design a solution and when it will happen. The benchmark also compares events before and after each model's training cutoff. Expert May 2026 60.4% Not in cards
DataPrep-Bench Medical Models prepare training data for the medical domain. The score is the published downstream average after backbone training. Expert May 2026 40.5 Not in cards
ExpertEval Models answer cases in medicine, finance and law. Practicing professionals write the cases and supply weighted grading rubrics. Expert May 2026 66.9% Not in cards
GenomeGym-BRCA1 Models estimate whether a documented BRCA1 variant causes loss of function. The grader compares each probability with a hidden functional label. Expert May 2026 Not in cards
Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. University May 2026 95.7% In use
High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. Expert May 2026 4.10% System cards only
K12-Bench Models select curriculum concepts, prerequisite relationships, related knowledge, experimental evidence, and textbook locations. The 23,640 multi-select questions come from a textbook knowledge graph. Exact match and instance-level F1 compare selected options with graph-derived answers. School May 2026 57.1% Not in cards
OncoTriad-QA Models answer questions about cancer patients using radiology, pathology, and genomic evidence. The authors score multiple-choice answers and compare written responses with reference answers. Expert May 2026 46% Not in cards
OpenFARM Bioacoustics Models classify animal vocalizations across eight tasks, including pig call type and recording context. Each task uses labels from its source dataset. Expert May 2026 Not in cards
PhysicianBench 100 physician tasks performed by an agent inside an EHR environment. Scored by pass rate per attempt. Expert May 2026 50% System cards only
ResearchClawBench Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs. Research frontier May 2026 38.6 Not in cards
SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. Expert May 2026 40.2% Not in cards
SpatialBench-Long Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning. Expert May 2026 40.5% Not in cards
SpatialBench-Long (original study) Agents complete 24 tasks in long spatial transcriptomics workflows. The authors compare their answers with results from the original analyses. Expert May 2026 11.1% Not in cards
Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. Expert Apr 2026 39.3% System cards only
BioPipelineBench Verified (Anthropic) Agents execute bioinformatics workflows with a bash tool. Only problems that passed a validation check by external reviewers are kept. Expert Apr 2026 83.6% System cards only
Clinical SOAP Note Evaluation: Safety-First Benchmarking Models turn 300 synthetic doctor-patient dialogues into clinical SOAP notes. Three judges check claims against transcript evidence and score numeric fidelity, coverage, and writing quality. The composite weights safety at 50%, coverage at 30%, and general quality at 20%. Expert Apr 2026 4.72 Not in cards
DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. Research frontier Apr 2026 16.5% System cards only
GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. Expert Apr 2026 33.2% Fading
GenomeQA Models answer multiple-choice questions about short genomic sequences. The benchmark covers six task families, including species classification. University Apr 2026 63.6% Not in cards
Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. Research frontier Apr 2026 7.60% System cards only
HealthBench Professional 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. Expert Apr 2026 69.2% In use
KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. University Apr 2026 42% Fading
Medical Chronology LLM Benchmark Models extract medical timelines from six synthetic document sets in three independent rounds. Checks compare extracted events with reference timelines and assess semantic fidelity, formatting, date order, and hallucinations. A weighted composite combines these checks with ROUGE and extraction F1. Expert Apr 2026 88.9 Not in cards
Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. Expert Apr 2026 System cards only
Protocols (Anthropic life sciences) Models find and fix errors in molecular biology protocols and answer practical questions about applying real laboratory protocols. Expert Apr 2026 62% System cards only
WASSCE Benchmark for Open Language Models Models answer questions based on the West African Senior School Certificate Examination. The harness compares models across mathematics, English, science, and social studies. School Apr 2026 Not in cards
$OneMillion-Bench Agent solves 400 expert-curated tasks in law, finance, industry, healthcare and science. Rubric grading of accuracy, logic, feasibility and compliance. Expert Mar 2026 42.1% In use
BioTIER Safeguarded model configurations receive harmful and benign biology requests. Scored on refusing requests designated for refusal and allowing benign ones. Expert Mar 2026 Not in cards
BioTIER Permit Models receive 144 benign biology prompts. The score measures the percentage correctly answered. Expert Mar 2026 100% System cards only
BioTIER Refuse Models receive 398 dangerous biology prompts. The score measures the percentage correctly refused. Expert Mar 2026 99.1% System cards only
BioVITA retrieval benchmark Models match animal images, audio recordings, and descriptions. The benchmark checks whether the retrieved items match the reference species, genus, or family. University Mar 2026 Not in cards
BullshitBench V2 Models answer 100 nonsense questions across software, finance, legal, medical and physics, built with 13 techniques. A three-judge panel scores clear pushback. Everyday Mar 2026 95% Not in cards
CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. Mar 2026 83.7% System cards only
BABE Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference. Expert Feb 2026 52.3% Fading
Bio-Literature Search Level 2 Agents use search tools to compare abstracts from PubMedQA, including their sections, subject headings, and conclusions. Five task types require retrieving or combining evidence. University Feb 2026 Not in cards
BioMysteryBench Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. Expert Feb 2026 83.7% In use
BioPipelineBench (Anthropic) Agents execute bioinformatics workflows such as long-read sequence analysis, metagenome assembly and chromatin profiling with a bash tool. Expert Feb 2026 87.7% System cards only
Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. Expert Feb 2026 16.4 System cards only
Disordered eating evaluations (Anthropic) Models handle conversations about eating disorders and disordered eating. Score is the appropriate response rate. Everyday Feb 2026 System cards only
GutBrainIE 2026 Models find entities in 80 gut and brain research abstracts, link them to biomedical concepts, and identify relationships between mentions and between concepts. University Feb 2026 Not in cards
HLE-Verified Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Expert Feb 2026 48.2% In use
HLE-Verified (text-only Full) Models answer the text-only questions of Humanity's Last Exam after expert verification and revision. The Full set keeps the questions experts marked uncertain, and the score is pass@1 accuracy. Expert Feb 2026 46.8% Not in cards
Korean Dental Licensing Examination 2024 and 2025 Models answer 642 Korean dental licensing questions from two new exams. The authors score accuracy and break it down by subject and image type. Expert Feb 2026 81.5% Not in cards
KorMedMCQA-Mixed Models take combined text and image exams. The 2022 set contains 281 questions and the 2023 set contains 307. Each year has a separate score. Expert Feb 2026 97.3% Not in cards
KorMedMCQA-V Models answer Korean medical licensing questions using clinical images. The full test contains 1,534 questions and 2,043 images from 2012 through 2023. Expert Feb 2026 96.9% Not in cards
LABBench2 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. Expert Feb 2026 60.9% In use
LABBench2 CloningQA Models answer molecular biology questions using supplied information or retrieval tools. Expert Feb 2026 42.9% Not in cards
LABBench2 FigQA2 Models read scientific figures and answer in open response. Served as images or as PDFs. The FigQA2 subtask of LABBench2. Expert Feb 2026 42.6% Not in cards
LABBench2 LitQA3 Models answer open-response questions whose answers appear once in recent biology papers. The LitQA3 subtask of LABBench2. Expert Feb 2026 85.1% Not in cards
LABBench2 PatentQA Models locate and interpret evidence in patent documents. Expert Feb 2026 90.9% Not in cards
LABBench2 ProtocolQA2 Models find and fix errors planted in published lab protocols, answering in open response. The ProtocolQA2 subtask of LABBench2. Expert Feb 2026 61.6% Not in cards
LABBench2 SeqQA2 Models reason about DNA and protein sequences and answer in open response. The SeqQA2 subtask of LABBench2. Expert Feb 2026 52.5% Not in cards
LABBench2 TrialQA Models use clinical trial records to answer research questions. Expert Feb 2026 93.3% Not in cards
MedCode Models read de-identified patient records and output primary and secondary ICD-10-CM diagnosis codes. Answers are checked against codes assigned by professional coders. Expert Feb 2026 63.6% Not in cards
MedScribe Models write SOAP notes from clinical visit transcripts. Notes are graded against expert gold-standard rubrics for required content and section structure. Expert Feb 2026 91.4% Not in cards
MentalBench (Psychiatric Diagnosis) Models select psychiatric diagnoses for 24,750 synthetic cases. Exact matching requires both diagnoses when the case lacks the evidence needed to distinguish them. Expert Feb 2026 62.7% Not in cards
MultiClinSum-2 Models summarize 1,000 clinical cases per language across 15 languages. Organizers report four automatic metrics and four language model judgments separately. University Feb 2026 Not in cards
Phylogenetics (Anthropic) Models analyze phylogenetic data and tree diagrams, testing quantitative reasoning about tree structure and visual interpretation. Expert Feb 2026 79.6% System cards only
scBench (LatchBio SingleCellBench) 195 verifiable single-cell RNA-seq analysis problems. Each pairs a data snapshot with a task and a deterministic grader. Expert Feb 2026 63.9% System cards only
scBench (original study) Agents complete 394 tasks in single-cell data analysis. The authors compare their answers with results from the original analyses. Expert Feb 2026 52.8% Not in cards
SIM-VAIL Chatbots respond to simulated users with psychiatric vulnerabilities. A model judge scores behaviors across the conversation, including responses that reinforce a user’s vulnerability. Expert Feb 2026 1 Not in cards
Structural Biology (Anthropic) Models answer questions about a biomolecule's function from structural data and basic tools, in multiple-choice and open-ended variants. Expert Feb 2026 79% System cards only
A^3-Bench Models solve 2,198 science reasoning problems annotated with the anchors and attractors (prior knowledge) they need. An AAUI metric measures how much of that knowledge the model activates. University Jan 2026 65.1% Not in cards
Artificial Analysis Intelligence Index v4.0 Version 4.0 of Artificial Analysis's index, ten evaluations in four equal categories that brought in real-world knowledge work and hallucination tests. Jan 2026 60.2 Fading
BioNNE-R Models identify relations between supplied medical entity mentions in English and Russian abstracts. Organizers rank systems by macro F1 across 14 relation types. University Jan 2026 Not in cards
CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. Expert Jan 2026 Not in cards
DISPLACE-M 2026 Models identify speakers, transcribe their words, identify medical topics and summarize conversations between community health workers and patients. Each task has its own score. Everyday Jan 2026 Not in cards
GAIK Ajokortti transcription test Models transcribe one three-minute Finnish dental webinar excerpt with two speakers. The project reports word error rate and separately counts errors that change the meaning. Everyday Jan 2026 1.00% Not in cards
HealthBench-Hallu Models answer HealthBench-style medical prompts. A pipeline splits each answer into medical claims and checks them against medical evidence. The score is a weighted hallucination rate. Expert Jan 2026 3.50% Fading
MedGenVidQA 2026 Visual Answer Localization Models mark the start and end of the video segment that answers a medical question. The organizers created 80 test examples from professional medical videos. University Jan 2026 Not in cards
MedPhysBench Knowledge Questions Models answer board-style medical physics questions covering radiation oncology, dosimetry, imaging, and radiation protection. Version 0.2.0 contains 412 questions, including 115 test items. Expert Jan 2026 Not in cards
MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. Expert Jan 2026 Not in cards
OpenMed MedReason Evaluation Models answer medical questions from a held-out portion of the MedReason training corpus. The port scores answer accuracy, reasoning, and format separately. Expert Jan 2026 Not in cards
PHEB Models answer multiple-choice questions from Portuguese national secondary school exams administered between 2006 and 2023. School Jan 2026 Not in cards
PsychEthicsBench Models answer multiple-choice questions and respond to scenarios under Australian psychology and psychiatry ethics guidelines. Graders check responses against professional standards. University Jan 2026 75.6% Not in cards
QIMMA Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review. School Jan 2026 69.2% Not in cards
ScanBench Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage. Expert Jan 2026 73.8% Fading
SciPredict Models predict the outcomes of 405 real physics, biology and chemistry experiments published after March 31, 2025, with and without expert background knowledge. Expert Jan 2026 25.3% Not in cards
WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. Expert Jan 2026 Not in cards
Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. Expert 2026 Not in cards
Human Pathogen Capabilities Test v2 (HPCT v2) Models answer multiple-response questions on a small set of weaponizable pathogens that biosecurity experts rank highest priority. A revised version of HPCT. Research frontier 2026 80% Not in cards
LatchBio Capabilities v1.0 LatchBio cross-benchmark suite of agentic biological data analysis (transcriptomics, epigenomics, variants, pharmacology) with deterministic graders. Expert 2026 58% System cards only
BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. Competition Dec 2025 53.5% Fading
CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. Research frontier Dec 2025 66.5 Not in cards
FrontierScience Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. Expert Dec 2025 87% In use
FrontierScience-Olympiad Models solve olympiad-style physics, chemistry and biology problems at the level of IPhO, IChO and IBO, written by medalists and coaches. Graded on short final answers. Competition Dec 2025 77.1% System cards only
FrontierScience-Research Models work on open-ended PhD-level research subtasks in physics, chemistry and biology, written by working scientists. A grader scores each answer against a 10-point rubric. Expert Dec 2025 38% Fading
HELM Arabic Stanford CRFM runs models on 7 Arabic benchmarks in the HELM harness and ranks them by mean score. The tasks are AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and human-translated MMLU. School Dec 2025 84.1 Not in cards
MedCalc-Bench Verified Models extract values from patient notes and compute results for 55 medical calculators. Verified release of MedCalc-Bench with corrected labels. Expert Dec 2025 86.2% System cards only
MEDIQA-SYNUR Models extract structured observations from synthetic nursing dictations and map them to a clinical vocabulary. The hub uses the development split by default and supports the test split. Expert Dec 2025 Not in cards
MMLU Early Training Models score complete answer texts on selected MMLU questions. The metric compares the correct answer’s log likelihood with the average for incorrect answers. University Dec 2025 Not in cards
Research Plan Generation Models write research plans from goals derived from scientific papers. Judges grade each plan against criteria specific to its source paper. Expert Dec 2025 Not in cards
SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. Expert Dec 2025 33.8% In use
SpatialBench (LatchBio) Agentic spatial-transcriptomics data analysis problems with deterministic graders. Anthropic reports a 115-problem Verified variant. Expert Dec 2025 57.6% System cards only
SpatialBench (original study) Agents complete 146 tasks in spatial transcriptomics. The authors compare their answers with results from the original analyses. Expert Dec 2025 38.4% Not in cards
ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. Expert Nov 2025 66.8% System cards only
ATLAS (OpenCompass) Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge. Research frontier Nov 2025 42.9% Not in cards
Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. Expert Nov 2025 73.7% System cards only
CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. Expert Nov 2025 System cards only
HEAD-QA v2 Models answer Spanish healthcare specialization exam questions across six disciplines. Version 2 adds languages, and the MedARC port runs the English configuration. Expert Nov 2025 83.2% Not in cards
MarineEval Models answer 2,000 questions about marine images and documents. The authors combine answer checks with model judgments. They report total accuracy and mean accuracy across seven tasks. University Nov 2025 50.4% Not in cards
MedBench 4.0 (OpenCompass) Models handle clinical scenarios in an evaluation that covers specialized models and medical applications. Expert Nov 2025 Not in cards
MedPT Models classify Brazilian Portuguese medical questions by specialty. The original evaluation tests 20 classes, while the hub creates a split after deduplication and label selection. University Nov 2025 Not in cards
MindEval Models hold multi-turn therapy conversations with simulated patients designed with licensed clinical psychologists. An LLM judge scores each session. Expert Nov 2025 3.98 Not in cards
MultiClinAI Models identify diseases, symptoms, and procedures in seven languages. A second task transfers annotated Spanish mentions to the corresponding spans in six translations. University Nov 2025 Not in cards
NeuroDiscoveryBench Models analyze neuroscience, genetics and immunology datasets and answer questions grounded in figures and data workflows. Expert Nov 2025 Not in cards
Noor Models answer filtered academic questions chosen to distinguish early training checkpoints. The benchmark measures progress where standard MMLU scores are noisy. University Nov 2025 Not in cards
SMM4H-HeaRD 2026 Detection of Insomnia in Clinical Notes Models identify likely insomnia in nursing and discharge notes. They label diagnostic criteria and give the character offsets of supporting evidence. University Nov 2025 Not in cards
SMM4H-HeaRD 2026 Flu Vaccine Effectiveness Models classify posts by the author’s flu vaccination status and flu test result. The labels distinguish the current flu season from past seasons. Everyday Nov 2025 Not in cards
SMM4H-HeaRD 2026 MedSynth Dialogue-to-Note Models turn synthetic clinical dialogues into notes with four sections. The organizers generated 368 separate test dialogues and scored the notes against references. University Nov 2025 Not in cards
SMM4H-HeaRD 2026 Multilingual Adverse Drug Event Detection Models flag personal reports of drug side effects in seven languages. The organizers withheld Farsi from training and added 15,184 Farsi posts at test time. Everyday Nov 2025 Not in cards
SMM4H-HeaRD 2026 Pathology Report Staging Models assign tumor, lymph node, and distant metastasis stages to synthetic pathology reports. A tiebreaker removes explicit stage labels and requires inference from clinical findings. Expert Nov 2025 0.95 Not in cards
Biomni-Eval1 Agents use biomedical data and tools to answer 433 questions across ten task types. Each task checks a structured answer against a reference. Expert Oct 2025 Not in cards
BioRiskEval Researchers measure biorisk-relevant capability kept by open-weight DNA and protein foundation models, with three task sets named Gen, Mut and Vir. Expert Oct 2025 Not in cards
HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. Research frontier Oct 2025 Not in cards
IMB Models answer Italian medical specialty exam questions and respond to patient questions from online forums. The paper reports results for both tasks. University Oct 2025 71% Not in cards
MedQA-Followup Models answer medical licensing questions and then respond to follow-ups that challenge their answers. Researchers compare accuracy before and after each intervention. Expert Oct 2025 0% Not in cards
MentalBench-100k Models answer counseling questions. Human reviewers grade responses to 1,000 of the 10,000 prompts on seven attributes, each from one to five. Expert Oct 2025 5 Not in cards
Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. Expert Oct 2025 System cards only
Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. Expert Oct 2025 43.6% System cards only
NautData test set Models answer questions, describe underwater scenes, and locate organisms in images. The paper reports 7,916 test examples on 3,920 images. University Oct 2025 91.9% Not in cards
RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. University Oct 2025 69.4% Not in cards
UWBench Models answer questions about underwater scenes, write captions, and locate objects from descriptions. Questions cover visibility, materials, and spatial relationships. Everyday Oct 2025 Not in cards
VERA-MH Suicide Risk Chatbots talk with simulated patients who show different levels of suicide risk. A judge grades the full conversation against a clinical rubric. Expert Oct 2025 75.0 Not in cards
VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. University Oct 2025 44.8% Not in cards
APEX-1 Models complete expert-written work cases with source documents for four jobs: investment banking, consulting, big law and primary care. A judge model grades each response against rubrics. Expert Sep 2025 81.1% Not in cards
Artificial Analysis Intelligence Index v3 Version 3 of Artificial Analysis's index, ten evaluations that added agentic terminal and tool-use tasks to the knowledge, math and coding tests. Sep 2025 61.0 Dropped
BioML-bench Agents build and evaluate machine learning systems for biomedical experiments spanning proteins, drugs, single cells, imaging, and biomarkers. Task-specific metrics compare submissions with experimental reference results. Expert Sep 2025 Not in cards
CLEF 2026 SimpleText Task 1 Models rewrite sentences or whole Cochrane abstracts for the public. The organizers compared outputs with plain-language summaries from 175 new systematic reviews. University Sep 2025 Not in cards
CLEF 2026 SimpleText Task 2 Models detect or classify unsupported additions to simplified Cochrane abstracts. A generation subtask compares rewrites made with and without explicit checks against the source. University Sep 2025 0.34 Not in cards
GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. Expert Sep 2025 87.9% In use
Hallym University Women's Health Nursing Exam Models answer 50 Korean questions from a fourth-year nursing course. The authors compare accuracy and estimated ability with results from 111 students. University Sep 2025 Not in cards
JGPQA Models answer graduate-level science questions translated into Japanese from GPQA. Expert Sep 2025 Not in cards
PestCLEF 2026 Models extract structured facts from plant health documents. Organizers compare the predicted relationships with expert annotations on a held-out test set. University Sep 2025 Not in cards
Radiology's Last Exam (RadLE) Multimodal models make a single spot diagnosis from images alone on 50 hard radiology cases. Board-certified radiologists scored about 83%. Expert Sep 2025 30% Not in cards
SciReasoner evaluation suite Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks. Expert Sep 2025 63.3% Fading
SciTS Models understand and generate scientific time series in 43 tasks across 12 domains. Signals range from one to ten million points. Each task uses its own metric. Expert Sep 2025 System cards only
SLPHelm UltraSuitePlus Models analyze recordings of children’s speech for speech sound disorders. The suite also includes transcription and symptom classification tasks. Expert Sep 2025 Not in cards
Artificial Analysis Intelligence Index v2.2 Version 2.2 of Artificial Analysis's index, which swapped the older math tests for AIME 2025 and added instruction following and long-context reasoning. Aug 2025 Not in cards
AstaBench Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem. Expert Aug 2025 58% Not in cards
BiomedArena Researchers compare anonymous answers to biomedical research questions and vote for the response they prefer. Expert Aug 2025 Not in cards
Challenging prompts from red teamers with biosafety-relevant PhDs Models answer adversarial biology prompts written by red teamers with biosafety PhDs. Scored as not_unsafe rate from model training alone. Expert Aug 2025 93.6% System cards only
Filtered, adversarial sample of production prompts Models answer filtered borderline and high-risk biology prompts that resemble production traffic. Scored as not_unsafe rate from model training alone. Expert Aug 2025 96.9% System cards only
HealthBench Consensus Global Health Models answer health conversations that need adjustment for global context such as local care access. Scored as the failure rate. Expert Aug 2025 0% System cards only
HealthBench Consensus Urgent Models answer ambiguous high-stakes health conversations validated by two or more physicians. Scored as the error rate in informing the user. Expert Aug 2025 0.40% System cards only
HealthBench Hard Hallucinations Models respond to difficult health conversations that 2 or more physicians validated. The score is the hallucination rate, so lower is better. Aug 2025 1.60% Dropped
HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. Expert Aug 2025 16% Fading
Human Pathogen Capabilities Test Models answer expert-level questions on human pathogens from SecureBio. Scored as accuracy. Expert Aug 2025 80.7% System cards only
Pairwise Bioweaponization Campaign Red teamers with biology PhDs compare anonymized responses from two models on bioweaponization prompts. Scored as the win rate for being safer. Expert Aug 2025 60% System cards only
RedditImpacts 2.0 Models mark clinical and social consequences of opioid use in first-person accounts. The organizers scored 278 annotated posts and excluded 300 unlabeled decoys. School Aug 2025 Not in cards
TroubleshootingBench Models find and correct realistic experimental errors in expert-written wet lab protocols. Scored as pass@1. Expert Aug 2025 59% System cards only
3LM Models answer Arabic STEM questions taken from Arabic textbooks and worksheets, plus synthetic STEM questions from the same sources. A third part tests Arabic code generation. School Jul 2025 85.7% Not in cards
EHRQA (Google) Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning. Expert Jul 2025 Not in cards
EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. Expert Jul 2025 Not in cards
Fundus-MMBench Models identify retinal structures, classify diseases, and grade their severity in fundus photographs. Each of the 31 tasks contains 20 questions. Expert Jul 2025 Not in cards
HLE Bio/Chem Gold Models answer biology and chemistry questions from Humanity’s Last Exam whose answers reviewers supported with published research. Expert Jul 2025 Not in cards
PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. Expert Jul 2025 Not in cards
SciArena Researchers ask scientific literature questions and vote between two models' long cited answers. Votes give Elo ratings. Expert Jul 2025 1,151 Not in cards
SciVid Video models answer questions about scientific videos across several disciplines. The benchmark tests whether models recognize procedures, objects, and events in specialized footage. University Jul 2025 Not in cards
Taiwan National Dental Technician Licensing Examination 2023 Models answer 194 text-only dental technology questions from Taiwan's 2023 licensing exam. The authors score accuracy at baseline and after each of three weeks. University Jul 2025 Not in cards
US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. Expert Jul 2025 Not in cards
34th Japanese National Examination for Dental Hygienists Models answer 213 Japanese dental hygiene questions, including 139 with visual material such as instrument photographs and radiographs. University Jun 2025 Not in cards
ClearHarm Models answer unambiguously harmful CBRN questions. Judges score whether attacks elicit harmful responses from the model. Expert Jun 2025 Not in cards
ExpertLongBench Models write documents for 11 professional tasks. A judge compares facts in each document with a reference checklist and scores precision and recall. Expert Jun 2025 33.4% Not in cards
HealthQA-BR Models answer Portuguese multiple-choice questions about clinical care and public health in Brazil’s Unified Health System. Expert Jun 2025 86.6% Not in cards
K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. School Jun 2025 63.4% Not in cards
K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. School Jun 2025 55.4% Not in cards
LLMEval-Med Models answer clinical questions drawn from patient records and expert scenarios. A model grades answers against physician-developed checklists. Expert Jun 2025 Not in cards
NTCIR-19 MedNLP-CALL Triage Classification Models classify synthetic emergency calls into three urgency levels from dispatcher and caller dialogue. Medical experts supplied the reference labels for the six-language task. University Jun 2025 Not in cards
Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. Expert Jun 2025 60.7% Fading
CoralVQA Models answer questions about coral identity, growth, and health in reef photographs. GPT-4o checks answers against references across 16 question types. University May 2025 41.8% Not in cards
CounselBench-Adv Models answer 120 questions written by mental health professionals. Each question targets one of six problems identified in earlier counseling responses. Expert May 2025 0.37 Not in cards
Creative biology Models answer questions about unusual biology needed to engineer harmless biological agents, a weak proxy for novel weapon design. Expert May 2025 60.3% System cards only
CSVQA Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images. School May 2025 49.6% System cards only
DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. Expert May 2025 80% System cards only
HealthBench 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. Expert May 2025 67.2% In use
HealthBench Consensus Models answer HealthBench conversations graded only on criteria that several physicians agreed on. Expert May 2025 96.6% Dropped
HealthBench Hard Models answer the 1,000 hardest HealthBench conversations, graded against physician-written rubrics. Expert May 2025 46.2% Fading
HealthBench Meta Models judge whether a medical response meets a specific rubric item. The evaluation compares each binary judgment with physician labels and reports agreement metrics. Expert May 2025 0.71 Not in cards
iNaturalist Sounds Models identify animal species from recordings submitted to iNaturalist. Expert May 2025 Not in cards
MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. University May 2025 74.1% Not in cards
RBench-M Models answer 665 graduate-level multimodal reasoning questions in English and Chinese. Top-1 accuracy compares selected answers with the answer key. Expert May 2025 69.9% Not in cards
ScienceBoard Computer-use agents do scientific workflows in desktop software such as ChimeraX, Celestia, GRASS GIS and Lean 4. The apps are modified to expose state for checking. Expert May 2025 68% Not in cards
Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. Expert May 2025 System cards only
SoSBench Models answer hazardous requests drawn from real regulations in six scientific fields. Scored by how often the model complies with a request it should refuse. Expert May 2025 10.6% Not in cards
VM14K Models answer Vietnamese multiple-choice questions across 34 medical specialties. The authors reserve a private set for leaderboard evaluation. University May 2025 Not in cards
xbench-ScienceQA Models answer postgraduate-level science questions written by experts, in Chinese, as part of the xbench AGI Tracking series. Expert May 2025 76.4% Not in cards
BRIDGE Models work on 87 tasks built from real clinical text, such as electronic health records, in nine languages. The tasks span triage, consultation, information extraction, diagnosis, prognosis and billing codes across 14 specialties. Each task uses its own automatic metric, and the board reports the average. Expert Apr 2025 56.0 Not in cards
Japanese National Dental Examination 2025 Models answer 350 questions from Japan's 2025 dental licensing exam. The study reports accuracy separately for 202 text questions and 148 visual questions. University Apr 2025 Not in cards
Japanese National Examination for Occupational Therapists 2025 Models answer 199 Japanese occupational therapy licensing questions. The authors score answers against the official key and compare performance on text and visual questions. University Apr 2025 88.4% Not in cards
Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. University Apr 2025 52.9% System cards only
MultiClinSum Models summarize clinical case reports in English, Spanish, French, and Portuguese. Organizers compare each summary with a reference using ROUGE and BERTScore. University Apr 2025 0.86 Not in cards
CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. Expert Mar 2025 System cards only
HELM Capabilities Stanford CRFM runs language models on a curated set of scenarios, one per key capability, and averages the scores. Prompt-level results are public. University Mar 2025 81.9 Not in cards
MedBench 3.0 (OpenCompass) Models handle medical tasks across an expanded clinical evaluation suite. Shanghai AI Laboratory reports results for participating models. Expert Mar 2025 Not in cards
Medical Prescription Selection (lm-evaluation-harness) Models read a diagnosis or complaint and choose the recorded prescription. The two task variants use different distractors. Expert Mar 2025 Not in cards
MicroVQA Models answer expert-written multiple choice questions about biology microscopy images that test image understanding, hypothesis generation and experiment proposal. Expert Mar 2025 59.4% Fading
MIMIC-III Report Summarization Models generate the impression section of a clinical report from the preceding text. The harness compares the response with the recorded impression. Expert Mar 2025 Not in cards
MMLU-ProX MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. University Mar 2025 87% In use
Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. School Mar 2025 System cards only
SMM4H 2025 Food Recalls and Foodborne Disease Outbreaks Models classify news articles as recalls, outbreaks, or neither. They also extract details such as the product, cause, and number of people affected. School Mar 2025 0.88 Not in cards
Artificial Analysis Intelligence Index v2 Artificial Analysis's index of seven public reasoning, knowledge, math and coding benchmarks, each run by Artificial Analysis. Feb 2025 67.7 Dropped
BioMaze Models answer questions about biological pathways under normal and altered conditions. The benchmark includes true-or-false questions and written responses. University Feb 2025 61.6% Not in cards
BioNNE-L Models link nested terms in English and Russian biomedical abstracts to medical concept identifiers. Organizers report top-one accuracy, top-five accuracy, and mean reciprocal rank. University Feb 2025 Not in cards
Bioweapons knowledge questions Models answer 33 sensitive bioweapons questions written by Deloitte biosecurity staff. Answers are compared with novice, intermediate and expert baselines. Expert Feb 2025 51.5% System cards only
BixBench Computational-biology analysis tasks (differential expression, assembly QC, cell typing) answered by agents with tools. Expert Feb 2025 93.8% Fading
CareQA Models answer questions from Spanish specialist healthcare examinations. The evaluation includes English and Spanish versions and an open-response format. Expert Feb 2025 76.9% Not in cards
Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. University Feb 2025 25.0 System cards only
EICAT impact classification Models read a scientific paper and classify the ecological impact of a named invasive species. They choose among five impact levels and a data deficient category. Expert Feb 2025 40.3 Not in cards
ELCardioCC 2025 Models find clinical mentions in Greek cardiology discharge letters, link them to ICD-10 codes, and assign codes to each letter with supporting phrases. Expert Feb 2025 Not in cards
GutBrainIE 2025 Models find biomedical entities and relationships in 40 held-out PubMed abstracts on the gut and brain. The organizers score entity spans and three relation extraction tasks. University Feb 2025 Not in cards
Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. Expert Feb 2025 69.7% System cards only
MedHallu Models distinguish factual medical answers from hallucinated answers at three difficulty levels. The benchmark tests detection with and without supporting knowledge. University Feb 2025 0.71 Not in cards
MedHELM Models run a suite of medical tasks chosen by clinicians across five categories, from clinical decision support and note writing to patient messages and administration. Expert Feb 2025 70.3 Not in cards
MMLU-Redux 2.0 Spanish Models answer corrected MMLU questions in Spanish across 57 subjects. The evaluation compares each selected option with the answer key. University Feb 2025 Not in cards
Open Arabic LLM Leaderboard v2 Models are scored on native Arabic tasks (AlGhafa native subsets, ArabicMMLU, Exams, MadinahQA, AraTrust, ALRAGE, ArbMMLU-HT) and ranked by the average. School Feb 2025 72.4% Dropped
PeerQA Models answer peer reviewers' questions about a scientific article using evidence from the article. Expert Feb 2025 0.24 Not in cards
Physiome-ODE Models forecast irregularly sampled biological time series generated from published differential equations. The authors compare forecasts using mean squared error. University Feb 2025 Not in cards
SCT-Bench Models rate how new clinical information changes a diagnosis or treatment decision. The grader awards partial credit according to the distribution of expert responses. Expert Feb 2025 70.5% Not in cards
SMM4H-HeaRD 2025 Detection of Insomnia in Clinical Notes Models identify likely insomnia in clinical notes, label individual criteria, and quote supporting text. The organizers score evidence with ROUGE-L. University Feb 2025 0.97 Not in cards
SMM4H-HeaRD 2025 Multilingual Adverse Drug Event Detection Models classify posts in English, French, German, and Russian by whether the author reports a drug side effect. The French test translates German forum posts. Everyday Feb 2025 Not in cards
SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Expert Feb 2025 73.9% In use
TituLLMs Bangla Evaluation Suite Models answer Bangla questions on academic subjects, science and everyday reasoning. The suite also asks yes-or-no questions about passages. University Feb 2025 Not in cards
Virology Capabilities Test (VCT) Models answer 322 multimodal virology troubleshooting questions written by expert virologists. Multiple-select variant requires all correct options. Expert Feb 2025 67.4% Fading
CLEAR (Clinical Information Extraction) Models read patient notes and classify a medical condition as present, absent, or uncertain. Human reviewers supply the reference labels. Expert Jan 2025 Not in cards
HLE Calibration Models give a confidence score with each HLE answer. The benchmark reports RMS calibration error and AUROC between confidence and correctness. Expert Jan 2025 23.4 Fading
Humanity's Last Exam 2,500 expert-written closed-ended questions across 100+ subjects. Exact-match or multiple-choice answers graded by an LLM judge. Expert Jan 2025 54.8% In use
Japanese Nephrology Board Renewal Questions 2014 to 2023 Models answer 209 Japanese nephrology board renewal questions from ten exam years, including questions with clinical images. Researchers compare answers with the society’s key. Expert Jan 2025 81.3% Not in cards
LLMzSzŁ Models answer Polish academic and professional exam questions drawn from the Central Examination Board archives. University Jan 2025 69.1% Not in cards
MedAgentBench Agents use FHIR APIs in a virtual electronic health record to do 300 patient tasks in 10 categories written by physicians. Expert Jan 2025 69.7% Not in cards
MedQA Bias Evaluation Models answer medical exam questions before and after Vals adds racial information. Vals compares accuracy across patient groups. Expert Jan 2025 96.5% Not in cards
MedXpertQA Models answer 4,460 expert-level medical exam questions across 17 specialties, in a text subset and an image subset. Scoring is multiple-choice accuracy. Expert Jan 2025 52.6% Fading
MedXpertQA-MM Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. Expert Jan 2025 80.4% In use
MMVU 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. University Jan 2025 82.4% In use
Turkish Dental Specialty Examination 2020 and 2021 Models answer Turkish dental specialty entrance questions in basic and clinical sciences. The two official answer keys cover 238 scored questions, including questions with images. Expert Jan 2025 Not in cards
Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. University Jan 2025 88.7% In use
Biology-Instructions Models answer multi-omics sequence tasks on DNA, RNA, protein and multi-molecule inputs, such as property prediction and interaction tasks. Scored per task with task metrics. Expert Dec 2024 55.7% Fading
CURE Models retrieve passages that answer clinicians' questions at the point of care. Expert Dec 2024 Not in cards
DCASE 2025 Audio Question Answering Models answer questions about marine mammal calls, the timing of sounds, and overlapping events. Each of the three sections counts equally toward the final score. University Dec 2024 52.1% Not in cards
Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. University Dec 2024 96% Dropped
Global MMLU Lite Models answer MMLU questions in 15 languages, with 200 culturally sensitive and 200 culturally agnostic questions per language. All items are human translated or post-edited. Accuracy. University Dec 2024 95.7% Dropped
MEDEC Models identify and correct errors in clinical notes across five categories. The OpenMed port uses the MEDEC-MS subset. Expert Dec 2024 69.1% Not in cards
MIMIC-IV-BHC Models summarize a patient’s hospital course from clinical notes. The reference is the brief hospital course section of the discharge summary. Expert Dec 2024 Not in cards
MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. University Dec 2024 73.4% Not in cards
Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. Expert Dec 2024 63.1% System cards only
ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. Expert Dec 2024 47.2% In use
Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. Research frontier Dec 2024 System cards only
Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. Expert Dec 2024 51% System cards only
AfriMed-QA Models answer about 15,000 open and multiple-choice medical questions from over 60 medical schools in 16 African countries. Questions span 32 specialties. University Nov 2024 75.2% Not in cards
BEANS-Zero Models follow text prompts to classify, detect or caption animal sounds without task-specific training. University Nov 2024 Not in cards
CLEF 2025 SimpleText Task 1 Models simplify individual sentences or whole Cochrane abstracts. The organizers tested new reviews and scored both tasks against plain English references. University Nov 2024 43.0 Not in cards
CLEF 2025 SimpleText Task 2 Models flag unsupported sentences, classify distortions in simplified scientific text, or rewrite Cochrane abstracts while avoiding new claims. University Nov 2024 Not in cards
Cochrane-auto Models simplify sentences, paragraphs, or abstracts from Cochrane reviews. The authors aligned technical text with plain English summaries. University Nov 2024 Not in cards
INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. University Nov 2024 89.8% Fading
JAMA Clinical Challenge (JesseLiu) Models choose among four answers to clinical vignettes from JAMA. The OpenMed port combines 1,521 cases and reserves about 5% for evaluation. Expert Nov 2024 Not in cards
Polish Medical Exams Models answer Polish medical, dental, and specialist examination questions. The benchmark also includes English editions of the medical and dental licensing exams. Expert Nov 2024 79.9% Not in cards
ScholarQABench Models answer scientific questions by synthesizing research papers. The benchmark checks citations and grades long answers against expert rubrics. Expert Nov 2024 0.00 Not in cards
CBT-Bench Level II Models identify cognitive distortions and classify patients’ core beliefs at two levels of detail. The evaluator compares the selected labels with the reference labels. Expert Oct 2024 0.63 Not in cards
DiscourseEE Models read Reddit discussions about opioid use disorder and extract details about medication use, tapering, and relapse. Some details require inference across a conversation. University Oct 2024 36.9% Not in cards
European LLM Leaderboard (OpenGPT-X) Models answer exam, science, commonsense, truthfulness, and math questions across European languages. The leaderboard also includes multilingual reading comprehension. School Oct 2024 56.7% Not in cards
MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. University Oct 2024 Not in cards
TaxaBench-8k Models identify species or match observations across photographs, sounds, and other field data. The test contains 8,813 observations paired across six types of data. Expert Oct 2024 Not in cards
xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. University Oct 2024 41.4% System cards only
ArbMMLU-HT Models answer a human translation into Arabic of the 57 MMLU subjects. University Sep 2024 72.2% Not in cards
Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. Expert Sep 2024 System cards only
CORE-Bench Agents reproduce the results of published studies from their code and data. 270 tasks from 90 papers in computer science, social science and medicine come in three difficulty levels. Expert Sep 2024 77.8% Not in cards
Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. Expert Sep 2024 System cards only
Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. Expert Sep 2024 0% System cards only
MMMLU MMLU professionally translated into 14 languages. Multiple-choice accuracy. University Sep 2024 92.7% In use
MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. University Sep 2024 84.6% In use
ThaiExam Models answer multiple-choice questions from standardized examinations in Thailand, such as ONET, IC, TGAT, TPAT-1 and A-Level. School Sep 2024 75.1% Not in cards
BioLP-bench Find the introduced error in a biological lab protocol. Tests protocol understanding. Expert Aug 2024 49% System cards only
GMAI-MMBench Vision-language models answer medical multiple-choice questions over many imaging modalities, clinical tasks and departments, at several perceptual granularities. Expert Aug 2024 43.4% Not in cards
MMLU-Redux 2.0 Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them. University Aug 2024 Not in cards
Open Ko-LLM Leaderboard 2 Models complete nine Korean evaluations covering reasoning, instructions, social values, safety, and other abilities. The organizers score submitted models on private test data. School Aug 2024 49.6% Not in cards
PLABA 2024 Term Replacement Models identify difficult medical terms, classify how to adapt them, and supply wording for patients. The organizers held out 300 abstracts. University Aug 2024 Not in cards
Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. Expert Aug 2024 94% System cards only
Biological Wastewater Treatment Long-Form QA Models answer 286 closed-book questions about biological wastewater treatment. Three experts grade the answers for relevance and factual accuracy. University Jul 2024 Not in cards
HaluBench Models judge whether an answer is faithful to its context across about 15,000 labeled context, question and answer triplets. It tests hallucination detectors and LLM judges. University Jul 2024 78.8% Not in cards
INAS hypothesis span prediction Models mark the words in a scientific title or abstract that indicate a particular ecological hypothesis. Experts marked the evidence in 750 invasion biology abstracts. University Jul 2024 Not in cards
LAB-Bench Multiple-choice tasks on practical biology research skills: literature, protocols, sequence and figure reasoning, database lookup. Expert Jul 2024 80.7% System cards only
LAB-Bench CloningScenarios Models answer multiple-choice questions about multi-step molecular cloning workflows. CloningScenarios subset of LAB-Bench. Expert Jul 2024 94.2% Fading
LAB-Bench FigQA Models interpret figures from biology papers and answer multiple-choice questions. FigQA subset of LAB-Bench. Expert Jul 2024 80.4% Fading
LAB-Bench LitQA2 Models answer multiple-choice questions whose answers appear once in recent biology papers. Built to need literature search rather than recall. A LAB-Bench subtask. Expert Jul 2024 27% Not in cards
LAB-Bench ProtocolQA Models find and fix errors in lab protocols in multiple-choice form. ProtocolQA subset of LAB-Bench. Expert Jul 2024 90.7% Fading
LAB-Bench SeqQA Models answer multiple-choice questions that require DNA and protein sequence manipulation. SeqQA subset of LAB-Bench. Expert Jul 2024 98.2% System cards only
MedBench 2.0 (OpenCompass) Models answer Chinese medical questions. This release changes the evaluation of open responses. Expert Jul 2024 Not in cards
Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. University Jul 2024 83.2% System cards only
QEVD-FIT-COACH Models watch a workout and decide when to give feedback. The benchmark scores feedback timing and the content of the coaching advice. Everyday Jul 2024 Not in cards
SciCode Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Expert Jul 2024 59.8% In use
TurkishMMLU Models answer over 10,000 multiple choice questions that Turkish curriculum experts wrote for nine high-school subjects. Scoring is accuracy. School Jul 2024 79.1% Dropped
GAOKAO-Eval 2024 Models take Chinese national college entrance examinations. High school teachers grade the open responses. School Jun 2024 476.5 Not in cards
INQUIRE-Fullrank Models retrieve wildlife photographs from five million iNaturalist images for ecological queries. The benchmark measures average precision among the first 50 results. University Jun 2024 Not in cards
INQUIRE-Rerank Models reorder 100 photographs supplied for each ecological query. The benchmark measures average precision among the first 50 results. University Jun 2024 62.4% Not in cards
IrokoBench Models answer human-translated knowledge, math and inference questions in 16 African languages. The parts are AfriMMLU, AfriMGSM and AfriXNLI. School Jun 2024 59% Not in cards
MedCalc-Bench (original) Models apply 55 clinical calculators to patient notes. The original test contains 1,100 questions and checks numerical answers against task-specific tolerance bounds. Expert Jun 2024 39.5% Not in cards
Miðeind Icelandic ARC-Challenge Models answer multiple-choice science questions in an Icelandic translation of ARC-Challenge. Miðeind used Claude 3.5 Sonnet to translate the questions. School Jun 2024 95.2% Not in cards
MMLU-Pro Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. University Jun 2024 89.8% In use
MMLU-Redux Manually re-annotated subset of MMLU that fixes wrong labels. Multiple-choice accuracy. University Jun 2024 95% Fading
MMLU-SR Models answer MMLU questions after key terms are replaced with synonyms. The benchmark tests whether models recognize the same concept under different wording. University Jun 2024 65% Not in cards
OlympicArena Models solve 11,163 bilingual problems from 62 international olympiad competitions in seven fields, some with images. Answers are checked against keys. Competition Jun 2024 40.5% Dropped
Open LLM Leaderboard v2 Hugging Face ran open models on IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro and averaged normalized scores. University Jun 2024 48.0 Not in cards
ProteinLMBench Models answer 944 hand-checked multiple choice questions on protein sequence, structure and function. Expert Jun 2024 57.9% Dropped
RadNLP 2024 Radiology Report Segmentation Models label sentences in lung cancer reports with eight categories, including tumor size and distant metastasis. The organizers score labels using micro F2. Expert Jun 2024 Not in cards
SciKnowEval Models answer scientific questions and apply knowledge in biology, chemistry, materials and physics. The suite uses automatic checks and model judges for different tasks. University Jun 2024 0.71 Not in cards
AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. Expert May 2024 37.8% Not in cards
ARC Challenge MT Models answer science questions from ARC-Challenge in translation. The harness scores answers in the target language against the original choices. School May 2024 Not in cards
CADECv2 Models extract adverse drug reactions and drug names from patient reviews. The corpus covers 26 drugs across four groups. University May 2024 Not in cards
EnviroExam Models answer multiple choice questions from 42 environmental science courses. The composite index discounts mean accuracy when performance varies across courses. University May 2024 80.6 Not in cards
MedLFQA Models write long answers to biomedical questions. The benchmark checks whether answers include required facts and avoid claims identified as incorrect. Expert May 2024 77.1 Not in cards
MedReadMe Models rate how easy a medical sentence is to understand. Researchers compare the ratings with judgments from readers. Everyday May 2024 Not in cards
Numerical results extraction from randomized controlled trials Models extract group sizes, event counts, means, and standard deviations from trial reports. The test contains 656 treatment comparisons across 110 papers. University May 2024 77.9% Not in cards
Open Arabic LLM Leaderboard v1 Models are scored on AlGhafa with native and translated sets, the ACVA and AceGPT sets, and translated MMLU and EXAMS. The board ranks by the average. School May 2024 69.1% Not in cards
PitVQA Models answer questions about a frame from pituitary surgery. Questions cover the surgical step, the instruments in view, and what the surgeon is doing. Expert May 2024 Not in cards
Japanese National Dental Examination 2024 Models select answers to 353 Japanese dental licensing questions, including 149 that use visual material. The exam covers 17 dental specialties. University Apr 2024 81.9% Not in cards
MultiCardioNER Models mark disease mentions in Spanish cardiology cases and medication mentions in Spanish, English, and Italian versions of the cases. Organizers score exact spans with microaveraged F1. University Apr 2024 Not in cards
NASA-IR Models retrieve the scientific passage that answers a question. The test measures how often the correct passage appears among the first ten results. University Apr 2024 Not in cards
STaRK Retrieval systems find the right items for complex queries over knowledge bases that mix free text with relations between entities. The three domains are product search, academic paper search and precision medicine. Most queries are synthesized and checked by people, and a smaller set is written by people. University Apr 2024 Not in cards
BirdSet Models identify bird species from audio recordings across multiple evaluation datasets. Expert Mar 2024 Not in cards
CT-RATE Models read 3D non-contrast chest CT volumes and say whether each of 18 conditions is present. Scored by macro accuracy or F1 against report-derived labels. Expert Mar 2024 Not in cards
EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. School Mar 2024 44% Not in cards
JASMINE Models replace medical jargon in Japanese sentences with everyday language. The reference sentences come from patients' blogs. University Mar 2024 Not in cards
KorMedMCQA Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. Expert Mar 2024 92.7% In use
MedConceptsQA Models answer multiple-choice questions about medical codes. The hub supports the original dataset and a separate 2,000-example sample. Expert Mar 2024 61.9% Not in cards
MedSafetyBench Models respond to requests that conflict with medical ethics. A judge rates harmfulness on a five-point scale using a held-out set of 900 requests. Everyday Mar 2024 1.44 Not in cards
NTCIR-18 HIDDEN-RAD Models explain chest radiograph findings from reports or structured observations. The final score combines text similarity, language model judgments, and expert review. Expert Mar 2024 Not in cards
NTCIR-18 MedNLP-CHAT Models flag medical, legal, and ethical risks in answers to patient questions. The organizers compare the labels with expert judgments using macro F1. Expert Mar 2024 Not in cards
RadNLP 2024 Lung Cancer Staging Models assign detailed lung cancer stages to radiology reports. The organizers score Japanese and English separately and require all three stage labels to match. Expert Mar 2024 Not in cards
WMDP Proxy multiple-choice benchmark for hazardous knowledge in bio, chem and cyber. Released with the RMU unlearning method. Expert Mar 2024 84.8% System cards only
WMDP Biology Models answer 1,273 multiple-choice questions that proxy hazardous biosecurity knowledge. Biology split of WMDP. Expert Mar 2024 91% Fading
ArabicMMLU Models answer 14,575 multiple-choice questions in Modern Standard Arabic across 40 tasks, taken from school exams in North Africa, the Levant and the Gulf. School Feb 2024 71.4% Dropped
BioNNE 2024 Models mark the spans and types of overlapping medical terms in English and Russian abstracts. Organizers score span matches using macro F1. University Feb 2024 Not in cards
EHRNoteQA Models read a patient's accumulated MIMIC-IV discharge summaries and answer clinical questions about them. The set has 962 question and answer pairs, each for a different patient. Expert Feb 2024 96.2% Not in cards
Medbullets Models answer clinical questions patterned on US medical licensing examinations. The benchmark also evaluates explanations of the answers. Expert Feb 2024 68.8% Not in cards
MedVidQA 2024 Query-Focused Instructional Step Captioning Models divide a medical demonstration into steps that answer a question. They give each step a caption and start and end times. Everyday Feb 2024 Not in cards
MedVidQA 2024 Video Corpus Visual Answer Localization Models search 48,605 medical videos for demonstrations that answer 52 new questions. They return ranked videos and the start and end times of each answer. Everyday Feb 2024 Not in cards
Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. University Feb 2024 78.7% Dropped
OmniMedVQA Models answer questions about 118,010 medical images across 12 imaging modalities. The authors measure accuracy against the correct answer option. Expert Feb 2024 50.7% Not in cards
RRG24 Models write findings and impressions from one or more chest X-rays per study. Organizers score each section separately using five metrics. Expert Feb 2024 Not in cards
tinyBenchmarks Models answer small selected subsets of established benchmarks. Item response estimates approximate accuracy on the full tests. University Feb 2024 Not in cards
AAP Periodontics In-Service Examination Models answer specialty examination questions used by North American periodontics residency programs. Researchers score answers against the Academy’s key for each exam year. Expert Jan 2024 Not in cards
Artificial Analysis Intelligence Index v1 Artificial Analysis's first composite score, called the Quality Index in 2024, which normalized and combined a few public benchmarks into one number. Jan 2024 Not in cards
Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. Expert Jan 2024 System cards only
CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. University Jan 2024 36.8% Not in cards
CMMU Models answer Chinese multimodal questions across seven school subjects. School Jan 2024 30.9 Not in cards
ENEM Challenge Models answer multiple-choice questions from Brazil’s national high school examination in Portuguese. School Jan 2024 Not in cards
JMMLU Models answer 7,536 four-way multiple-choice questions in Japanese, from translated MMLU subjects and new Japan-specific subjects. Scored by accuracy. University Jan 2024 80.6% Dropped
K-QA Models answer 201 patient questions. The benchmark checks whether answers include essential statements from clinicians and whether any claims contradict those statements. Expert Jan 2024 67.7% Not in cards
LongHealth Models answer 400 questions about long clinical records from 20 fictional patients. Tasks test retrieval, sorting and recognition of missing information. University Jan 2024 51.7% Not in cards
MedBench 1.0 (OpenCompass) Models answer Chinese medical questions across clinical reasoning, language understanding, generation, knowledge, and ethics. Expert Jan 2024 Not in cards
MentalChat16K Counseling Evaluation Models answer 200 mental health questions from online forums. Judges rate each response from one to ten on seven counseling skills. Expert Jan 2024 8 Not in cards
PLABA 2024 Complete Abstract Adaptation Models rewrite 400 new biomedical abstracts for patients. Reviewers grade how well the rewrites preserve information and explain it clearly. University Jan 2024 Not in cards
Translated ARC-Challenge, HellaSwag and MMLU (Mistral) Models answer French, German, Spanish and Italian versions of ARC-Challenge, HellaSwag and MMLU. School Jan 2024 System cards only
HELM Lite A lightweight HELM board that runs models with in-context learning on ten scenarios. They include NarrativeQA, NaturalQuestions, OpenbookQA, MMLU, GSM8K, MATH, LegalBench, MedQA and WMT 2014. University Dec 2023 93.8 Not in cards
LitQA v0 Models answer questions that require information from full text scientific papers. Expert Dec 2023 40.6% Not in cards
ProteinGym Deep mutational scanning benchmarks. The model ranks mutations by predicted effect on protein function, scored by Spearman rank correlation with lab measurements. Anthropic reports a hard subset, as the correlation times 100. Research frontier Dec 2023 System cards only
Reddit-Impacts Models mark the words that describe clinical or social consequences of substance use in Reddit posts. The original test has 278 posts. School Dec 2023 0.42 Not in cards
TMMLU+ Models answer traditional Chinese questions across 66 subjects. The evaluation includes elementary material and professional examinations. University Dec 2023 88.6% Not in cards
GPQA Diamond 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. Expert Nov 2023 97% In use
GPQA Extended Models answer all 546 GPQA questions, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. Expert Nov 2023 48.8% System cards only
GPQA Main Models answer the 448 question main set of GPQA, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. Expert Nov 2023 80.3% Dropped
MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. University Nov 2023 86% In use
IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. School Oct 2023 35.8% Dropped
NAPISUL-II Progress Test 2023 Models answer Portuguese multiple-choice questions from the October 2023 medical progress test in southern Brazil. Researchers compare their answers with the official key. University Oct 2023 Not in cards
RaceBias (Race-Based Medicine) Models answer clinical questions used to identify unsupported race-based medical beliefs. Evaluators check responses for those beliefs. Expert Oct 2023 Not in cards
AMMLU Models answer MMLU multiple-choice questions across 57 subjects, translated into Arabic with GPT-4. Scored by accuracy. University Sep 2023 72.4% Dropped
CompassRank OpenCompass ranks open and API language models on its own evaluation sets and publishes monthly rankings. Model makers submit a repository link or API endpoint to be tested. Expert Sep 2023 83.3 Not in cards
CMB Models answer 11,200 Chinese medical exam questions and questions about 74 clinical cases. Exam answers receive exact-match scores. A model judge rates clinical answers for fluency, relevance, completeness, and medical proficiency. Expert Aug 2023 66.9% Not in cards
EBM-NLPmod Models extract participants, interventions, comparators, and outcomes from trial titles and methods sections. Annotators separate comparison treatments from interventions. University Aug 2023 Not in cards
MedAlign Models follow clinician-written instructions using a patient’s electronic medical record. Clinicians evaluate the responses. Expert Aug 2023 Not in cards
MTSamples clinical entity recognition Models mark medical problems, treatments, and tests in 163 synthetic discharge summaries. Researchers score exact and overlapping text spans separately. University Aug 2023 Not in cards
BLUEX Models answer questions from two Brazilian university entrance examinations. The questions cover high school subjects in Portuguese. School Jul 2023 95% Not in cards
EHRSHOT Models predict clinical outcomes from longitudinal electronic health records with few labeled examples for each task. Expert Jul 2023 Not in cards
Malaysian Sign Language Medical Dataset Models identify Malaysian Sign Language terms used in health care. The training and test lists share all seven signer names. Everyday Jul 2023 Not in cards
Med-HALT Models answer medical exam questions designed to reveal hallucinations. The hub port uses the false-confidence, fake-question, and none-of-the-above reasoning tasks. Expert Jul 2023 42.7% Not in cards
MedText Models suggest diagnoses or treatments for textbook-style patient presentations. The harness compares their text with the recorded completion. Expert Jul 2023 Not in cards
Okapi ARC Models choose answers to translated science questions from ARC-Challenge. School Jul 2023 Not in cards
Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. University Jul 2023 80% Dropped
SympTEMIST Models locate symptoms, signs, and findings in Spanish clinical cases. They also map given mentions to SNOMED CT codes in Spanish and five additional languages. University Jul 2023 Not in cards
CMMLU Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy. University Jun 2023 90.8% Fading
M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. School Jun 2023 Dropped
Mol-Instructions Models answer molecule-oriented, protein-oriented and biomolecular text instructions, such as reaction prediction, molecule description and protein function questions. Scored per task with task metrics. Expert Jun 2023 54% Fading
Mol-Instructions Chemical Entity Recognition Models identify chemical entities in biomedical text. The score is test-set F1. Expert Jun 2023 Not in cards
Mol-Instructions Chemical-disease Interaction Extraction Models extract chemical and disease interactions from text. The score is test-set F1. Expert Jun 2023 Not in cards
Mol-Instructions Chemical-protein Interaction Extraction Models extract chemical and protein interactions from text. The score is test-set F1. Expert Jun 2023 Not in cards
Mol-Instructions Multi-choice Models answer biomedical multiple-choice questions. The score is test-set accuracy. Expert Jun 2023 Not in cards
Mol-Instructions True or False Models answer biomedical true or false questions. The score is test-set accuracy. Expert Jun 2023 Not in cards
ScienceBenchmark Models translate questions about the Sloan Digital Sky Survey, cancer research, and research funding into SQL. The organizers run the queries and compare their results. University Jun 2023 23% Not in cards
UK Dermatology SCE Sample Questions Models answer multiple-choice questions from the Federation’s dermatology practice exam. Researchers score answers against the official key. Expert Jun 2023 Not in cards
ACI-Bench Models turn doctor-patient conversations into clinical visit notes. The original benchmark compares generated notes with clinician-authored references. University May 2023 Not in cards
C-Eval 13,948 Chinese multiple-choice exam questions across 52 disciplines and four difficulty levels. Accuracy. University May 2023 93.7% Fading
GAOKAO-Bench Models answer Chinese national college entrance exam questions from 2010 to 2022 across nine subjects, with objective and subjective questions. School May 2023 72.2% Dropped
MTS-Dialog Models write a clinical note section from a conversation between a doctor and a patient. The evaluation compares it with the reference note. Expert May 2023 Not in cards
MultiCochrane Models rewrite medical review sentences in plain language. The collection pairs technical and simplified wording in English, Spanish, French, and Farsi. University May 2023 Not in cards
PLABA 2023 Abstract Adaptation Models rewrite biomedical abstracts for patients. Four annotators independently adapted each of the 400 test abstracts. University May 2023 Not in cards
PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. University May 2023 66.2% Fading
TREC 2023 Clinical Trials Track Models rank trials for 50 simulated patients using answers to medical intake questionnaires. Some answers can be missing. Expert May 2023 Not in cards
BioCreative VIII Task 3 Genetic Phenotype Normalization from Dysmorphology Physical Examinations Models identify abnormal physical findings in pediatric exam notes and return Human Phenotype Ontology identifiers. A second task also scores the locations of the findings. University Apr 2023 Not in cards
Open LLM Leaderboard v1 Language models answer tasks from the original Open LLM Leaderboard suite. University Apr 2023 77.4 Not in cards
SYNERGY systematic review screening Models rank titles and abstracts against a review’s eligibility criteria. Reviewers’ inclusion decisions supply the reference labels. University Apr 2023 100% Not in cards
Korean National Licensing Examination for Korean Medicine Doctors 2022 Models answer 340 questions from the Korean medicine licensing exam. Researchers remove the images, transcribe tables, and compare prompt variants against the answer key. Expert Mar 2023 Not in cards
MammalNet Models identify mammals and their behavior in videos. The benchmark covers 173 mammal categories and 12 behaviors, including nursing and hunting. School Mar 2023 Not in cards
MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. University Mar 2023 0.07 System cards only
MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. University Mar 2023 System cards only
Med-EASi Models rewrite English medical passages in plain language. The references mark replacements, explanations, insertions, and deletions. University Feb 2023 Not in cards
MedVidQA 2023 Medical Instructional Question Generation Models write a question that a supplied medical video segment answers. They can use both the images and the subtitles. Everyday Feb 2023 Not in cards
MedVidQA 2023 Video Corpus Visual Answer Localization Models search 12,657 medical videos and return the times when a video shows the answer. Half of the 40 questions require visual evidence. Everyday Feb 2023 Not in cards
BCSC Self-Assessment Program ophthalmology questions Models answer ophthalmology board preparation questions across subspecialties. Researchers compare their choices with the academy’s answer key. Expert Jan 2023 Not in cards
EHRSQL Models convert clinical questions into SQL queries over electronic health records. The benchmark includes questions that the database cannot answer. University Jan 2023 Not in cards
MedProcNER Models identify procedures in Spanish clinical cases, link them to SNOMED CT codes, and list the codes for each case. University Jan 2023 Not in cards
MS-CXR-T Models compare a current and a prior chest X-ray of one patient and say whether a finding improved, stayed stable or got worse. Expert Jan 2023 Not in cards
OphthoQuestions ophthalmology board preparation Models answer multiple-choice ophthalmology questions from a subscription question bank. Each model selects one answer per question. Expert Jan 2023 Not in cards
ProbSum 2023 Models list active diagnoses and problems from hospital progress notes. Organizers compare the lists with a physician’s annotations using ROUGE-L F1. Expert Jan 2023 Not in cards
RadSum23 Models write the impression section of radiology reports from findings. One track includes chest images. Organizers rank submissions by RadGraph F1. Expert Jan 2023 38.0 Not in cards
MultiMedQA Models answer medical exam, research, and consumer health questions across seven datasets. Multiple-choice components use answer accuracy. Clinicians and other raters assess free responses for factuality, precision, harm, and bias. Expert Dec 2022 58.9% Not in cards
INAS hypothesis classification Models read the title and abstract of an invasion biology paper and identify which of ten ecological hypotheses it addresses. A paper can address several hypotheses. University Nov 2022 Not in cards
MedBench (Russian closed-test platform) Models answer Russian medical questions and classify relationships between clinical statements. The platform also scores electrocardiogram labels on a separate task. University Nov 2022 Not in cards
MedSecId Models identify section boundaries and assign a section type in clinical notes. The authors annotated 2,002 notes and reserved 10% for testing. University Oct 2022 Not in cards
PHEE Models extract adverse effects and potential therapeutic effects from published medical cases. They identify the event triggers and details such as the treatment and the patient. University Oct 2022 Not in cards
NTCIR-17 Radiology Report TNM Staging Models assign tumor, lymph node, and metastasis categories to Japanese lung cancer reports. Joint accuracy requires all three categories to match. Expert Sep 2022 Not in cards
NTCIR-17 Social Media Adverse Drug Event Detection Models identify drug side effects in simulated social media posts across four languages. Exact match requires all 22 symptom labels to match. University Sep 2022 Not in cards
ScienceQA Models answer about 21,000 multiple-choice science questions from school curricula, many with images. Scored by accuracy. School Sep 2022 97.5% Dropped
BioRED Models extract biomedical relations from 600 PubMed abstracts. The hub port presents entity pairs and asks models to classify their relationship. Expert Jul 2022 Not in cards
Plain Language Adaptation of Biomedical Abstracts Models rewrite biomedical abstracts in plain language for people who asked health questions. The authors collected 750 abstracts for 75 questions. University Jul 2022 Not in cards
EndoVis18-VQA Models answer questions about organs, tools, tool positions, and interactions in images from robotic kidney surgery. The benchmark accepts class labels or sentence answers in separate tasks. Expert Jun 2022 Not in cards
Meta-Album Models classify unfamiliar images after seeing a few labeled examples. Tasks span ten domains, including microscopy, remote sensing and wildlife. School Jun 2022 Not in cards
CLARA-MeD Spanish Medical Text Simplification Models rewrite Spanish medical sentences for patients. The collection includes 3,800 aligned pairs and 1,200 clinical trial sentences that linguists simplified. University May 2022 Not in cards
DDXPlus Models diagnose simulated patients from symptoms, clinical history, and demographic information. The OpenMed port asks for the most likely condition among 49 possibilities. Expert May 2022 Not in cards
Animal Kingdom Models identify actions in wildlife videos that can show several behaviors at once. The benchmark also tests pose estimation and locating events described in text. School Apr 2022 Not in cards
BiodivNER Models identify and label phrases in biodiversity abstracts and dataset descriptions. Annotators distinguish six kinds of entities, including organisms, locations, and environments. University Apr 2022 Not in cards
MedMCQA Models answer multiple-choice questions from Indian medical entrance exams, AIIMS and NEET PG. Scoring is accuracy. Expert Mar 2022 77% System cards only
ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. School Mar 2022 0.15 Not in cards
Penn epilepsy clinic notes Models read dated paragraphs from epilepsy clinic notes and classify recent seizures. Two other questions ask for seizure frequency and the most recent seizure date. University Jan 2022 Not in cards
RuMedBench Models predict diagnoses, suggest symptoms to check, answer Russian medical questions, infer relationships between statements, and identify entities in drug reviews. Expert Jan 2022 Not in cards
DisTEMIST Models locate disease mentions in Spanish clinical cases and link them to SNOMED CT codes. The organizers score both tasks with microaveraged F1. University Dec 2021 Not in cards
LayMaker MTSamples Simplification Models rewrite medical transcription sentences for patients. Clinicians supplied the reference versions and kept sentences they already considered clear. University Dec 2021 Not in cards
StepUp gym attendance forecasts Models predict how each four-week exercise program changes weekly gym visits relative to a placebo. Researchers compare predicted effects with recorded gym check-ins. University Dec 2021 Not in cards
2022 n2c2/UW Shared Task on Extracting Social Determinants of Health Models identify substance use, employment, and living situations in clinical notes. Organizers score events and their attributes across three tests using notes from two institutions. University Nov 2021 Not in cards
n2c2 2022 Contextualized Medication Event Extraction Models identify medication mentions and determine whether a clinician discusses a change. They classify each change by action, negation, timing, certainty, and who acted. University Nov 2021 Not in cards
n2c2 2022 Progress Note Understanding: Assessment and Plan Reasoning Models read a clinician's assessment and one part of the treatment plan. They choose one of four relationships. Organizers score the labels with macro F1. University Nov 2021 Not in cards
HealthAdvice Models classify a sentence from a medical research paper as strong advice, weak advice, or no advice. School Sep 2021 Not in cards
Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. Expert Jul 2021 Not in cards
BioCreative VII LitCovid Models assign one or more of seven research topics to COVID-19 articles using titles and abstracts. University Jun 2021 Not in cards
MS² Models synthesize the findings of multiple medical studies into a short answer to the research question of a systematic review. Expert Apr 2021 Not in cards
Cochrane Simplification Models turn medical review abstracts into plain-language summaries. The original evaluation compares generated text with Cochrane summaries, while the hub port adds a model judge. Expert Mar 2021 Not in cards
iNaturalist 2021 Models classify nature photographs by species. The 2021 competition uses a new collection of images and species labels. University Mar 2021 Not in cards
Natural World Tasks (NeWT 2021) Models distinguish pairs of labels in wildlife photographs. The 164 tasks include recognizing life stages, animal behavior and scene context. University Mar 2021 Not in cards
SLAKE Models answer English and Chinese questions about radiology images of the head, neck, chest, abdomen, and pelvis. Scoring is accuracy. University Feb 2021 90.8% Fading
CLEAR French Medical Text Simplification Models simplify French sentences from drug information, medical encyclopedias, and research reviews. The authors released 100 test pairs. University Dec 2020 Not in cards
MEDDOPROF Models find occupations and employment status in Spanish clinical notes. They identify whose occupation each mention describes and assign an occupational or clinical code. University Dec 2020 Not in cards
EXAMS Models answer high school exam questions in 16 languages across many subjects. Arabic reports use the Arabic subset. School Nov 2020 Not in cards
MedDialog Models generate a clinician’s response from a patient description. The harness also evaluates summaries of medical questions using reference text. Expert Nov 2020 Not in cards
PUBHEALTH Models classify public health claims as true, false, mixed or unproven. The original benchmark also asks models to explain their verdicts. University Nov 2020 Not in cards
MedICaT Models locate panels within a biomedical figure and match each panel to the relevant words in its caption. The original test scores word overlap. University Oct 2020 Not in cards
MedQA Models answer multiple-choice questions from medical licensing exams, usually the 12,723-question English USMLE set. Scoring is accuracy. Expert Sep 2020 86.3% Dropped
MedQA Mainland China Models answer mainland China medical licensing exam questions in simplified Chinese. Scoring is accuracy. Expert Sep 2020 84% System cards only
MedQA Taiwan Models answer Taiwan medical licensing exam questions in traditional Chinese. Scoring is accuracy. Expert Sep 2020 91% System cards only
MMLU 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. University Sep 2020 92.3% In use
MMLU Anatomy Models answer MMLU multiple-choice questions from the Anatomy subject. Scoring is accuracy. University Sep 2020 89% System cards only
MMLU Clinical Knowledge Models answer MMLU multiple-choice questions from the Clinical Knowledge subject. Scoring is accuracy. Expert Sep 2020 92% System cards only
MMLU College Biology Models answer MMLU multiple-choice questions from the College Biology subject. Scoring is accuracy. University Sep 2020 95% System cards only
MMLU College Medicine Models answer MMLU multiple-choice questions from the College Medicine subject. Scoring is accuracy. University Sep 2020 84% System cards only
MMLU Medical Models answer the multiple-choice questions from the medical subjects of MMLU, such as anatomy, clinical knowledge and professional medicine. University Sep 2020 72.9% Not in cards
MMLU Medical Genetics Models answer MMLU multiple-choice questions from the Medical Genetics subject. Scoring is accuracy. Expert Sep 2020 96% System cards only
MMLU Professional Medicine Models answer MMLU multiple-choice questions from the Professional Medicine subject. Scoring is accuracy. Expert Sep 2020 94% System cards only
MMLU-stem Models answer the STEM subjects of MMLU, such as physics, chemistry, mathematics, and computer science, as multiple choice. Scoring is accuracy. University Sep 2020 83.6% Dropped
Cochrane RCT Summarization Models read the titles and abstracts of clinical trials and write the conclusions of the Cochrane review that included those trials. Expert Aug 2020 Not in cards
VAERS influenza vaccine adverse event extraction Models mark six kinds of clinical information in influenza vaccine safety reports, including neurological events, procedures, and time expressions. Researchers score exact or overlapping text spans. University Aug 2020 Not in cards
Evidence Inference 2.0 Models classify the direction of a treatment effect in a trial report and identify text that supports their answer. University May 2020 Not in cards
Medical Paper-Blog Dataset Models rewrite a sentence from a medical paper for a general reader. The reference sentence comes from a medical blog. University May 2020 Not in cards
SciFact Models retrieve research abstracts and determine whether they support or refute a scientific claim. Expert Apr 2020 Not in cards
TREC-COVID Models rank scientific articles that address specific information needs about COVID-19. Expert Apr 2020 Not in cards
SemClinBr Models identify and classify medical terms in Brazilian Portuguese clinical notes. The corpus contains 1,000 notes from multiple specialties. University Jan 2020 Not in cards
Trialstreamer clinical evidence extraction Models identify treatments, comparators, outcomes, and the direction of each reported effect in trial abstracts. Physicians annotated 100 test abstracts. University Jan 2020 Not in cards
QASC Models answer eight-choice grade-school science questions by combining supporting facts from a large text corpus. Accuracy checks the selected answer against the answer key. School Oct 2019 Not in cards
PubMed causal language use Models classify the wording of a research conclusion as a direct causal claim, a conditional causal claim, a correlation, or no relationship. University Sep 2019 Not in cards
PubMedQA Models answer biomedical research questions with yes, no, or maybe, given a PubMed abstract without its conclusion. Scored by accuracy. University Sep 2019 78.3% Dropped
MEDIQA 2019 Question Answering Models answer consumer health questions or assess candidate answers. The shared task evaluates the relevance and quality of answers to medical questions. Expert Aug 2019 Not in cards
MeQSum Models condense consumer health questions into shorter questions while preserving the medical information needed for retrieval. Reference summaries provide the comparison. University Jul 2019 Not in cards
PUCRS inpatient fall detection Models identify whether a Portuguese progress note documents a fall. Nursing students labeled 355 of the 1,078 notes as fall related. University Jul 2019 Not in cards
CLEF eHealth 2019 Technology-Assisted Reviews Task 2 Models rank the articles returned by a systematic review search so reviewers can screen relevant titles and abstracts first. University Apr 2019 Not in cards
MedicationQA Models answer consumer questions about medications. The original corpus pairs questions with expert answers, and the hub evaluation grades answer accuracy and completeness. University Mar 2019 Not in cards
CheXpert Models read chest radiographs and label up to 14 observations. The test labels come from board-certified radiologists. Expert Jan 2019 Not in cards
COPIOUS biodiversity entity recognition Models mark names of organisms, places, habitats, people, and time expressions in historical biodiversity texts. The test measures how closely the marked phrases match human annotations. University Jan 2019 Not in cards
HEAD-QA Models answer questions from Spanish healthcare specialization exams in six disciplines. The hub port uses the English training and validation splits. Expert Jan 2019 Not in cards
MedQuAD Systems answer medical questions using a collection of 47,457 question-answer pairs from NIH websites. The OpenMed port uses a model judge on a subset. University Jan 2019 Not in cards
MIMIC-CXR (classification) Models read chest X-rays from Beth Israel Deaconess and say whether five common conditions are present. Scored by average F1 over the conditions. Expert Jan 2019 Not in cards
MIMIC-CXR report generation Models write the findings of a radiology report for a MIMIC-CXR chest X-ray. Scored against the radiologist's report, most often by RadGraph F1. Expert Jan 2019 Not in cards
PsyTAR Models identify adverse reactions in patient reviews of four antidepressants. The original corpus also labels withdrawal symptoms, reasons for treatment, and reported effectiveness. University Jan 2019 Not in cards
VQA-RAD Vision-language models answer clinicians' questions about 315 radiology images, including head CTs, MRIs and chest X-rays. Questions are closed yes or no, or open free text. Expert Nov 2018 Not in cards
OpenBookQA Models answer elementary science multiple-choice questions that need a given core science fact plus common knowledge. Scored by accuracy. School Sep 2018 96.6% Fading
EBM-NLP Models mark the words that describe participants, interventions, and outcomes in clinical trial abstracts. Medical professionals supplied the test labels. University Jun 2018 Not in cards
FDA and CDC VAERS clinical and temporal annotations Models locate clinical features and time expressions in vaccine safety reports. Annotators also linked clinical features to their timing across 1,000 reports. University Jun 2018 Not in cards
PatchCamelyon Models identify whether a microscope image patch contains metastatic tumor tissue. Expert Jun 2018 69.3% Not in cards
PubMed-PICO Models classify sentences in structured medical abstracts. Section headings supply the reference labels, including those for the study population, intervention, and outcome. University Jun 2018 Not in cards
MedMentions Models identify biomedical concepts in PubMed abstracts and link text spans to identifiers in the Unified Medical Language System. University May 2018 Not in cards
iNaturalist 2018 Models identify species in photographs submitted by naturalists. The 2018 competition uses an expanded set of plant and animal categories. University Feb 2018 Not in cards
n2c2 2018 Adverse Drug Events and Medication Extraction Models extract drugs and adverse events from discharge summaries. They also identify which drug each event or medication detail belongs to. University Jan 2018 Not in cards
MADE 1.0 Models identify drugs, indications, and adverse events in cancer care notes. They link each drug to its attributes and effects. The challenge scores extraction, linking, and the combined task. University Nov 2017 Not in cards
SciQ Models answer crowdsourced multiple-choice science exam questions in physics, chemistry and biology, each with a supporting passage. School Jul 2017 Not in cards
Animals with Attributes 2 Models use shared attributes to recognize animal classes they did not see during training. The benchmark provides 37,322 photographs across 50 classes with 85 attributes. School Jun 2017 Not in cards
ChestX-ray14 Models read frontal chest X-rays from the NIH Clinical Center and say whether thoracic conditions are present. Expert May 2017 Not in cards
iNaturalist 2017 Models identify the species in photographs submitted by naturalists. Meta evaluates transfer to fine-grained species classification. University Apr 2017 Not in cards
TAC 2017 Adverse Drug Reaction Extraction from Drug Labels Models extract adverse reactions and their modifiers from prescription drug labels. They identify positive reactions and map them to standard medical terms. The organizers held out 99 labels. University Apr 2017 Not in cards
NTCIR-13 MedWeb Models label each short message for eight symptoms. The organizers score an exact match when the model gets all eight labels right. Everyday Aug 2016 Not in cards
AI2D Models answer multiple choice questions about grade school science diagrams. The set has about 5,000 diagrams and 15,000 questions, scored by accuracy. School Mar 2016 96.9% Fading
NFCorpus Models rank medical research documents for health questions expressed in everyday language. University Feb 2016 0.41 Not in cards
Hallmarks of Cancer Models label biomedical text with the cancer hallmarks it describes. The hub port assigns zero or more of ten labels to individual sentences. University Oct 2015 Not in cards
CADEC Models identify adverse drug reactions in patient reviews from the AskaPatient website. A reaction can span words separated by other text. University Dec 2014 Not in cards
Birdsnap Models identify the North American bird species shown in a photograph. University Jun 2014 Not in cards
SemEval 2015 Task 14 Analysis of Clinical Text Models identify disorder mentions and map them to terminology codes in clinical notes. A separate task asks for attributes such as negation and severity. University May 2014 Not in cards
i2b2/UTHealth 2014 Identifying Risk Factors for Heart Disease over Time Models find heart disease risk factors in notes about patients with diabetes. They mark when each factor was present relative to the note date. University Mar 2014 Not in cards
SemEval 2014 Task 7 Analysis of Clinical Text Models mark disorder mentions in clinical notes and map them to medical terminology codes. Organizers score recognition and normalization separately. University Nov 2013 Not in cards
DDIExtraction 2013 Models identify drug names in biomedical text and classify interactions between pairs of drugs. The OpenMed port covers interaction classification. University Jun 2013 Not in cards
ShARe/CLEF eHealth 2013 Task 1 Models locate mentions of disorders in clinical notes and assign medical terminology codes. The organizers score mention recognition and code assignment separately. University Feb 2013 Not in cards
BioASQ Systems answer biomedical questions written by experts, using PubMed articles as evidence. Questions are yes/no, factoid, list or summary types. A second task asks systems to index new PubMed articles with MeSH headings. Expert 2013 Not in cards
ADE Corpus Models identify adverse drug effects in medical case reports. The OpenMed port combines classification and relation examples and reserves 20% for evaluation. Expert Oct 2012 Not in cards
Clinical Abbreviation Sense Inventory Models expand an ambiguous abbreviation using the surrounding words in an English clinical note. Evaluators compare the answer with a clinician's annotation. University Oct 2012 Not in cards
UW Clinical Record Section Chunker i2b2 Discharge Summary Corpus Models locate section boundaries in discharge summaries and label each section. Researchers compare the predicted labels and boundaries with human annotations. University May 2012 Not in cards
NICTA-PIBOSO Models assign one or more labels to each sentence in a medical abstract. The six labels include population, intervention, and outcome. University Mar 2011 Not in cards
Caltech-UCSD Birds 200 Models identify bird species from photographs. The benchmark distinguishes species with similar appearance. University Sep 2010 Not in cards
i2b2 2009 Medication Extraction Challenge Models extract drug names and prescription details from discharge summaries. The organizers scored six fields against community annotations for 251 test summaries. University Jun 2009 Not in cards
Oxford Flowers 102 Models identify flower categories from photographs. The benchmark tests recognition across changes in shape, scale, and lighting. University Dec 2008 Not in cards
i2b2 2008 Obesity Challenge Models classify obesity and 15 related conditions from discharge summaries. The challenge scores what the note states explicitly and what a clinician can infer. Expert Mar 2008 Not in cards
i2b2 2006 Smoking Challenge Models read a discharge summary and classify the patient's smoking status. The labels include current smoker, past smoker, unspecified smoker, nonsmoker, and unknown. School Jul 2006 Not in cards
OHSUMED Systems rank medical articles for 106 questions from physicians. The original study measured precision and recall against physicians' judgments of which articles were relevant. Expert Jul 1994 Not in cards
2010 i2b2/VA Challenge on Concepts, Assertions, and Relations in Clinical Text Models locate medical concepts, classify statements about problems, and determine relationships among problems, tests, and treatments. Organizers score each task separately. University Not in cards
2018 n2c2 Cohort Selection for Clinical Trials Models read longitudinal patient notes and decide whether patients meet specified clinical trial eligibility criteria. Expert Not in cards
Agentic BAIM-LLM Evaluation (ABLE) Agents run a viral-protein redesign workflow. Nine tasks cover structure retrieval, design strategy, tool selection and biological AI model use. Outputs are auto-scored. Research frontier 84.6% Not in cards
Animal-Bench Models answer 41,839 questions about animal videos. The overall score averages accuracy across 13 tasks, including action prediction and recognizing predator and prey behavior. School Not in cards
AP Biology Models take a simulated AP Biology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. School 5.00 System cards only
Bedside Bench Models answer 500 physician-written clinical cases on diagnosis, workup, escalation and medication safety. An LLM judge grades each answer against the authors' criteria. Expert 94.6% Not in cards
BioCreative V Chemical-Disease Relation Task Models identify biomedical entities and chemical-induced disease relations in PubMed abstracts. The OpenMed port extracts chemical and disease names. University Not in cards
Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. Expert 2.53 System cards only
Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. Expert System cards only
ChemProt Models identify the relationship between a chemical and a protein in a biomedical abstract. The shared task evaluates five relationship types. Expert Not in cards
COVID-Dialogue Models read a patient’s questions about COVID-19 and generate a response. Doctor replies from medical consultations serve as references. Expert Not in cards
CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Expert 75.9% In use
Discharge Me! Models draft discharge instructions and brief hospital course summaries from patient records. The shared task compares model output with clinical discharge documentation. Expert Not in cards
DrugProt Models read biomedical abstracts and identify which of 13 labeled relationships connects a marked chemical and protein. University Not in cards
EDB-Bio Models read experimental evidence and estimate the probability that a scientific hypothesis is true. Expert Not in cards
EKWbench Agents complete operational tasks such as sourcing reviews and referral triage. The score averages the share of verifier weight earned. Expert 74.4% Not in cards
EndoVis 2018 Robotic Scene Segmentation Models segment structures and surgical instruments in video from robot-assisted operations. The evaluation compares predicted masks with reference labels. Everyday Not in cards
Health dynamic conversation eval Simulated users hide key health facts at first, and models must ask for context to reach the correct clinical conclusion. Expert System cards only
Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. Expert System cards only
Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. Expert System cards only
High-risk health scenarios eval Adversarial health emergencies test whether models recognize urgency and red flags and direct users to help. Reports pass rate. Expert 98.9% System cards only
LABBench2 SourceQuality Models assess the quality of scientific information sources. Expert 90% Not in cards
LatchBio Capabilities V0 Agents complete biological data analysis tasks. The historical score averages the available benchmark scores equally and excludes missing scores. Expert 51.3% Not in cards
Medical Knowledge Self-Assessment Program (MKSAP) Models answer internal medicine board review questions from MKSAP. Scoring is accuracy. Expert 75% System cards only
MedQA USMLE (5 options) Models answer USMLE-style medical licensing questions with five answer options. Scoring is accuracy. Expert 86% System cards only
Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. Expert 79.4 System cards only
Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. Expert 5.74% System cards only
Molecular Biology Capabilities Test Models answer expert-level molecular biology questions from SecureBio. Scored as accuracy. Expert 68.7% System cards only
NeuroscienceBench Agents analyze neuroscience research data and report a scientific conclusion. The score counts correct conclusions among graded runs. Expert 41% Not in cards
Polaris Healthcare Safety Benchmarks Models conduct simulated healthcare conversations. The evaluation measures clinical safety, communication, and administrative task performance. Expert Not in cards
RareDiseaseBench Agents combine clinical and biological evidence to diagnose and manage rare diseases. A rubric grades their diagnoses and plans. Expert 43.8% Not in cards
RCM-Bench-1 Agents handle healthcare revenue cycle work, such as claim submission, billing and eligibility checks, with tools. Checks cover tool calls, arguments, replies and final system state. Expert 94.2% Not in cards
ReproBAIT Agents independently reproduce biological AI models from published scientific papers. Scored as success rate. Expert 85% System cards only
SecureBio VMQA, single choice Models answer 350 virology troubleshooting questions in single-choice form. VMQA is an earlier version of the Virology Capabilities Test. Expert 61.2% System cards only
SuperCLUE-MedAssist Medical assistants respond to Chinese healthcare questions. Expert Not in cards
SuperCLUE-Science Models solve Chinese science reasoning problems. University 80.8 Not in cards
TissueMicroscopyBench Agents examine tissue images and answer questions about cells, staining, and changes over time. Expert 34% Not in cards
TREC 2017 LiveQA Medical Task Models answer consumer health questions. The shared task compares responses with expert assessments of answer quality. Expert Not in cards
TREC BioGen 2024 Systems retrieve biomedical literature and synthesize evidence into reports that answer a question. Expert 79.9 Not in cards
TxBench-Antibody-Development Agents use evidence to make manufacturing, quality, and clinical development decisions about candidate therapeutic antibodies. Expert 61.1% Not in cards
USABO Semifinal Exam 2020 Models take the 2020 USA Biology Olympiad semifinal exam. Scored out of 150 points. Competition 87.0 System cards only
USMLE Step 1 (sample questions) Models answer the official USMLE Step 1 multiple choice sample questions. Scored by accuracy. Expert 68.9% System cards only
USMLE Step 2 (sample questions) Models answer the official USMLE Step 2 multiple choice sample questions. Scored by accuracy. Expert 63.3% System cards only
USMLE Step 3 (sample questions) Models answer the official USMLE Step 3 multiple choice sample questions. Scored by accuracy. Expert 67.2% System cards only
Virology Capabilities Test (text-only) Models answer the text-only questions of the SecureBio Virology Capabilities Test. Scored as accuracy. Expert 46% System cards only
World-Class Bio Models answer expert-level biology questions in a SecureBio benchmark. Scored as accuracy. Research frontier 68.8% System cards only