Chemistry and materials
Tasks that need chemistry or materials science. We count 147 benchmarks here, 1 of them arenas. A benchmark belongs here when subject includes chemical sciences, chemical engineering or materials engineering.
How fast it grows
33 were released since July, more than the 22 in the first half of 2026.
How much is still open
No model has passed 85% on 67 of the 80 benchmarks here that score in percent, 84% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| ChemInsight Models solve related chemistry problems in bundles. The authors score how many parts a model can solve and how far its reasoning remains correct. | University | Nov 2026 | | Not in cards |
| Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. | Research frontier | Nov 2026 | 44.7 | Not in cards |
| ChemArgus Models solve chemistry competition and textbook problems with text and figures. A model judge awards partial credit for each subquestion. | Competition | Oct 2026 | 64.9% | Not in cards |
| DualChem Models answer chemistry questions written by experts. The researchers report answer accuracy and assess whether models recognize possible misuse. | University | Oct 2026 | 67.3% | Not in cards |
| EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. | University | Oct 2026 | Not in cards | |
| Into the Unknown Agents get a science paper and an unpublished follow-up direction from a DOE national lab scientist, then write the follow-up paper. Rubrics built from the scientist's conditions grade it, plus an integrity check. | Research frontier | Oct 2026 | 0% | Not in cards |
| PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. | Research frontier | Oct 2026 | 14% | Not in cards |
| SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. | Expert | Oct 2026 | 80.9 | Not in cards |
| ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. | Expert | Oct 2026 | 0.53 | Not in cards |
| SciTaste-Bench Models judge the research value of papers in six disciplines. The benchmark compares their judgments with a reference index built from seven dimensions. | Expert | Oct 2026 | | Not in cards |
| SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. | Expert | Oct 2026 | | In use |
| Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. | Expert | Oct 2026 | 68.4% | In use |
| CompMat-Bench Agents complete individual steps from published computational materials studies. Fixed rules compare their inputs and outputs with results reproduced by the authors. | Expert | Sep 2026 | 91.5% | Not in cards |
| EduFair-Bench Tutors teach the same simulated student under different demographic descriptions. A judge grades individual turns and whole conversations to measure changes in teaching quality. | University | Sep 2026 | Not in cards | |
| EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. | Research frontier | Sep 2026 | 29.8 | Not in cards |
| MANDOS Session 2 Models make process safety decisions across four private industrial scenarios. A model judge assesses each run. The leaderboard reports the share of passing runs. | Expert | Sep 2026 | 96.9% | Not in cards |
| MatToolBench Agents use materials science software on Windows to operate interfaces, write scripts, and query databases. The evaluator awards partial credit for completed steps. | University | Sep 2026 | 51.5% | Not in cards |
| Medicinal Chemistry (Anthropic) Models see several real drug-discovery compounds and pick which had the highest or lowest measured ADME property. The set has 504 questions. | Expert | Sep 2026 | 65.3% | System cards only |
| Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. | Expert | Sep 2026 | 34% | System cards only |
| OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. | Expert | Sep 2026 | 73.7% | Not in cards |
| SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. | Expert | Sep 2026 | 45.3% | Not in cards |
| AutoDS scientific machine learning cases Agents build predictive models for maize yield, polymer glass transition temperature, and supercomputer job outcomes. Each case supplies a train split and a held-out test. | Expert | Aug 2026 | Not in cards | |
| BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. | Expert | Aug 2026 | 91.4% | System cards only |
| CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. | Expert | Aug 2026 | | System cards only |
| CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. | Expert | Aug 2026 | | System cards only |
| Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. | Expert | Aug 2026 | | System cards only |
| FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. | Expert | Aug 2026 | 20.6% | Not in cards |
| Terminal-Bench-Science 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. | Expert | Aug 2026 | 68.1% | In use |
| AI Security Leaderboard Models face systematic jailbreak attacks across chemical, biological, radiological, nuclear, explosive and cyber domains. Scores count universal jailbreaks and estimate attacker cost. | Expert | Jul 2026 | $60 | Not in cards |
| Autointent-Chem Models face manual and synthetic dangerous and benign chemistry queries. xAI reports refusal recall on the dangerous ones. | Jul 2026 | 100% | System cards only | |
| IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. | Research frontier | Jul 2026 | 27.3% | Not in cards |
| MedChemBench (OpenAI) Models work through 99 medicinal chemistry problems drawn from recent papers and patents. Each task mixes prose with structures, assay tables and plots, then asks for the program’s next decision. | Expert | Jul 2026 | 83.8% | In use |
| Organic Chemistry V2 (Anthropic) Models predict structures from spectra, design multi-step syntheses, predict reaction products and read chemical structure images. V2 adds harder expert-recommended problems. | Expert | Jul 2026 | 69.2% | System cards only |
| Artificial Analysis Intelligence Index v4.1 Version 4.1 of Artificial Analysis's index, which moved to GDPval-AA v2, Terminal-Bench 2.1 and τ³-Banking and put the most weight on agents. | Jun 2026 | 61.0 | In use | |
| DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. | Expert | Jun 2026 | | Not in cards |
| MANDOS Run 003 Models make decisions during seven industrial incidents and four fictional scenarios. A model judge checks their decisions, including whether they seek expert advice. | Expert | Jun 2026 | 100% | Not in cards |
| TEM-VQA Models give short written answers about structures, image features and experimental design in transmission electron micrographs. | Expert | Jun 2026 | 40.4 | Not in cards |
| CUSP Models forecast scientific progress across 4,760 events in many fields, using only knowledge from before a set cutoff date. Tasks ask whether an advance is feasible, how it would work, how to design a solution and when it will happen. The benchmark also compares events before and after each model's training cutoff. | Expert | May 2026 | 60.4% | Not in cards |
| K12-Bench Models select curriculum concepts, prerequisite relationships, related knowledge, experimental evidence, and textbook locations. The 23,640 multi-select questions come from a textbook knowledge graph. Exact match and instance-level F1 compare selected options with graph-derived answers. | School | May 2026 | | Not in cards |
| OmniMatBench Models answer questions and perform calculations across 19 materials science fields. The authors compare automated grades with human assessments. | University | May 2026 | 0.37 | Not in cards |
| ResearchClawBench Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs. | Research frontier | May 2026 | | Not in cards |
| Science Sim Chemistry Models compare simulated compounds and choose whether to advance or reject them. The evaluator scores decisions against measured reaction outcomes and a budget. | University | May 2026 | Not in cards | |
| SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. | Expert | May 2026 | 40.2% | Not in cards |
| Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. | Expert | Apr 2026 | 39.3% | System cards only |
| LongCoT Models solve long reasoning problems across five domains. Task-specific programs verify answers, with optional judge fallback for ambiguous mathematics and chemistry responses. | Expert | Apr 2026 | 25.1% | Not in cards |
| Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. | Expert | Apr 2026 | System cards only | |
| ChemDIRT Models answer chemistry questions while the evaluator varies instructions and molecular representations. The authors measure accuracy and consistency across variants. | University | Mar 2026 | 48.7% | Not in cards |
| CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. | Mar 2026 | | System cards only | |
| ChemPro Models answer chemistry questions across four difficulty levels. The authors report multiple-choice accuracy and accept numerical answers within specified tolerances. | University | Feb 2026 | | Not in cards |
| Organic Chemistry (Anthropic) Models predict structures from spectra, design synthetic routes, predict reaction products and convert between IUPAC names, SMILES and structure images. | Expert | Feb 2026 | 86.2% | System cards only |
| SPM-Bench Models interpret microscopy images and select all correct answers. The primary score gives partial credit and penalizes incorrect selections. | Expert | Feb 2026 | 0.94 | Not in cards |
| A^3-Bench Models solve 2,198 science reasoning problems annotated with the anchors and attractors (prior knowledge) they need. An AAUI metric measures how much of that knowledge the model activates. | University | Jan 2026 | 65.1% | Not in cards |
| MolecularIQ Models answer questions about molecular graphs, such as counting, indexing and constrained generation, and a symbolic checker verifies each answer. | Expert | Jan 2026 | 47.5% | In use |
| SciPredict Models predict the outcomes of 405 real physics, biology and chemistry experiments published after March 31, 2025, with and without expert background knowledge. | Expert | Jan 2026 | | Not in cards |
| Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. | Expert | 2026 | Not in cards | |
| AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. | Expert | Dec 2025 | | Fading |
| AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages. | Expert | Dec 2025 | 44% | Not in cards |
| FrontierScience Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. | Expert | Dec 2025 | | In use |
| FrontierScience-Olympiad Models solve olympiad-style physics, chemistry and biology problems at the level of IPhO, IChO and IBO, written by medalists and coaches. Graded on short final answers. | Competition | Dec 2025 | 77.1% | System cards only |
| FrontierScience-Research Models work on open-ended PhD-level research subtasks in physics, chemistry and biology, written by working scientists. A grader scores each answer against a 10-point rubric. | Expert | Dec 2025 | | Fading |
| SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. | Expert | Dec 2025 | | In use |
| Solubility Optimization (anindya) Models choose chemical transformations to improve predicted solubility. The package includes a benchmark runner that measures improvement on evaluation examples. | University | Dec 2025 | Not in cards | |
| SUPERChem Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score. | Expert | Dec 2025 | | In use |
| ATLAS (OpenCompass) Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge. | Research frontier | Nov 2025 | 42.9% | Not in cards |
| CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. | Expert | Nov 2025 | System cards only | |
| BlueprintSymVL Models use an example symbol to find matching components in piping and instrumentation diagrams. They must give both the correct count and the component labels. | School | Oct 2025 | Not in cards | |
| MatSciBench Models solve college materials science problems, including questions with figures. The authors report accuracy separately for text and image questions. | University | Oct 2025 | 64% | Not in cards |
| ProfBench Model writes expert-level reports in physics, chemistry, finance and consulting. Calibrated LLM judges grade against PhD/MBA-written rubrics. | Expert | Oct 2025 | 66.2% | In use |
| Artificial Analysis Intelligence Index v3 Version 3 of Artificial Analysis's index, ten evaluations that added agentic terminal and tool-use tasks to the knowledge, math and coding tests. | Sep 2025 | | Dropped | |
| JGPQA Models answer graduate-level science questions translated into Japanese from GPQA. | Expert | Sep 2025 | Not in cards | |
| MatCha Models answer multiple-choice questions about material structure, shape and properties using images from materials characterization instruments. | Expert | Sep 2025 | | Not in cards |
| SciReasoner evaluation suite Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks. | Expert | Sep 2025 | | Fading |
| AstaBench Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem. | Expert | Aug 2025 | 58% | Not in cards |
| HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. | Expert | Aug 2025 | 16% | Fading |
| 3LM Models answer Arabic STEM questions taken from Arabic textbooks and worksheets, plus synthetic STEM questions from the same sources. A third part tests Arabic code generation. | School | Jul 2025 | | Not in cards |
| HLE Bio/Chem Gold Models answer biology and chemistry questions from Humanity’s Last Exam whose answers reviewers supported with published research. | Expert | Jul 2025 | Not in cards | |
| IChO 2025 Models solve the written theoretical problems of the 57th International Chemistry Olympiad, held in Dubai in July 2025. Answers are graded against the official marking scheme. | Competition | Jul 2025 | Fading | |
| SciArena Researchers ask scientific literature questions and vote between two models' long cited answers. Votes give Elo ratings. | Expert | Jul 2025 | 1,151 | Not in cards |
| ClearHarm Models answer unambiguously harmful CBRN questions. Judges score whether attacks elicit harmful responses from the model. | Expert | Jun 2025 | Not in cards | |
| ether0 benchmark Models answer chemistry problems from several task types. The benchmark checks their answers with automated reward functions. | Expert | Jun 2025 | Not in cards | |
| ExpertLongBench Models write documents for 11 professional tasks. A judge compares facts in each document with a reference checklist and scores precision and recall. | Expert | Jun 2025 | | Not in cards |
| K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. | School | Jun 2025 | | Not in cards |
| K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. | School | Jun 2025 | | Not in cards |
| Matter-of-Fact Models decide whether materials science claims are feasible using evidence available before a cutoff date. The authors compare their answers with later published findings. | Expert | Jun 2025 | 72% | Not in cards |
| Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. | Expert | Jun 2025 | | Fading |
| ChemCoTBench Models interpret and edit molecular structures, optimize molecular properties and predict reaction outcomes. Each task has its own scoring rule. | University | May 2025 | Not in cards | |
| CSVQA Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images. | School | May 2025 | 49.6% | System cards only |
| MatTools Models answer questions about materials science software and write Python programs to calculate material properties. The authors execute the generated programs. | University | May 2025 | | Not in cards |
| RBench-M Models answer 665 graduate-level multimodal reasoning questions in English and Chinese. Top-1 accuracy compares selected answers with the answer key. | Expert | May 2025 | 69.9% | Not in cards |
| SoSBench Models answer hazardous requests drawn from real regulations in six scientific fields. Scored by how often the model complies with a request it should refuse. | Expert | May 2025 | 10.6% | Not in cards |
| xbench-ScienceQA Models answer postgraduate-level science questions written by experts, in Chinese, as part of the xbench AGI Tracking series. | Expert | May 2025 | 76.4% | Not in cards |
| CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. | Expert | Mar 2025 | System cards only | |
| Artificial Analysis Intelligence Index v2 Artificial Analysis's index of seven public reasoning, knowledge, math and coding benchmarks, each run by Artificial Analysis. | Feb 2025 | 67.7 | Dropped | |
| Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. | University | Feb 2025 | 25.0 | System cards only |
| EMMA Models solve math, physics, chemistry and coding problems that need reasoning over text and images together, not each alone. Answers are checked against a key. | University | Jan 2025 | | Fading |
| Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. | Research frontier | Dec 2024 | System cards only | |
| TOMG-Bench Models generate molecules from natural language requests in three tasks: molecule editing, molecule optimization and customized generation. Each output is checked for validity and for meeting the requested properties. | Expert | Dec 2024 | 18.6% | In use |
| MaCBench Vision language models answer chemistry and materials questions from images. Tasks cover reading data, understanding experiments and interpreting results. | Expert | Nov 2024 | 72.6% | Not in cards |
| Open Ko-LLM Leaderboard 2 Models complete nine Korean evaluations covering reasoning, instructions, social values, safety, and other abilities. The organizers score submitted models on private test data. | School | Aug 2024 | 49.6% | Not in cards |
| SciCode Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. | Expert | Jul 2024 | | In use |
| ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. | University | Jun 2024 | | Dropped |
| GAOKAO-Eval 2024 Models take Chinese national college entrance examinations. High school teachers grade the open responses. | School | Jun 2024 | | Not in cards |
| MMLU-Pro Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. | University | Jun 2024 | | In use |
| OlympicArena Models solve 11,163 bilingual problems from 62 international olympiad competitions in seven fields, some with images. Answers are checked against keys. | Competition | Jun 2024 | 40.5% | Dropped |
| SciKnowEval Models answer scientific questions and apply knowledge in biology, chemistry, materials and physics. The suite uses automatic checks and model judges for different tasks. | University | Jun 2024 | 0.71 | Not in cards |
| ARC Challenge MT Models answer science questions from ARC-Challenge in translation. The harness scores answers in the target language against the original choices. | School | May 2024 | Not in cards | |
| ChemBench Models answer more than 2,700 chemistry questions on knowledge and reasoning. Results are compared with a group of human chemists. | Expert | Apr 2024 | 70.9% | Dropped |
| RiceChem Models grade long answers to four college chemistry questions. Each of the 1,264 answers has human judgments for the applicable rubric items. | University | Apr 2024 | | Not in cards |
| EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. | School | Mar 2024 | | Not in cards |
| WMDP Proxy multiple-choice benchmark for hazardous knowledge in bio, chem and cyber. Released with the RMU unlearning method. | Expert | Mar 2024 | 84.8% | System cards only |
| WMDP Chemistry Models answer 408 multiple-choice questions that proxy hazardous chemical security knowledge. Chemistry split of WMDP. | Expert | Mar 2024 | 90% | Fading |
| PNCExtract Models read a research paper and list its polymer nanocomposite samples. They link each sample to its matrix, filler, and composition by mass or volume. | University | Feb 2024 | Not in cards | |
| SMolInstruct Models solve 14 chemistry tasks such as name conversion, property prediction, molecule captioning, generation and reaction prediction. Each task uses its own metric on the test split. | Expert | Feb 2024 | | Fading |
| CMMU Models answer Chinese multimodal questions across seven school subjects. | School | Jan 2024 | 30.9 | Not in cards |
| GPQA Diamond 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. | Expert | Nov 2023 | | In use |
| GPQA Extended Models answer all 546 GPQA questions, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. | Expert | Nov 2023 | 48.8% | System cards only |
| GPQA Main Models answer the 448 question main set of GPQA, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. | Expert | Nov 2023 | | Dropped |
| SciBench Models solve open-ended college-level problems in mathematics, chemistry and physics. Most answers are numbers, which a script checks against the key. | University | Jul 2023 | | Not in cards |
| Mol-Instructions Models answer molecule-oriented, protein-oriented and biomolecular text instructions, such as reaction prediction, molecule description and protein function questions. Scored per task with task metrics. | Expert | Jun 2023 | | Fading |
| Mol-Instructions Chemical Entity Recognition Models identify chemical entities in biomedical text. The score is test-set F1. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions Chemical-disease Interaction Extraction Models extract chemical and disease interactions from text. The score is test-set F1. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions Chemical-protein Interaction Extraction Models extract chemical and protein interactions from text. The score is test-set F1. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions Description-guided Molecule Design Models generate molecules from descriptions. The score is exact-match accuracy on the molecule design test set. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions Forward Reaction Prediction Models predict reaction products from reactants. The score is exact-match accuracy on the forward reaction test set. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions Reagent Prediction Models predict reaction reagents. The score is exact-match accuracy on the reagent prediction test set. | Expert | Jun 2023 | Not in cards | |
| Mol-Instructions Retrosynthesis Models predict reactants for target products. The score is exact-match accuracy on the retrosynthesis test set. | Expert | Jun 2023 | Not in cards | |
| ChatGPT Chemistry Assistant MOF synthesis extraction Models extract 11 synthesis details from papers about metal-organic frameworks. Researchers check the reported materials, amounts, temperatures, and reaction times against their annotations. | University | May 2023 | Not in cards | |
| GAOKAO-Bench Models answer Chinese national college entrance exam questions from 2010 to 2022 across nine subjects, with objective and subjective questions. | School | May 2023 | 72.2% | Dropped |
| MatSci-NLP Models read text from materials science papers and complete seven tasks, including identifying synthesis actions and linking measurements to experiments. The authors used 99% of examples for testing. | University | May 2023 | Not in cards | |
| AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. | School | Apr 2023 | | Fading |
| LLM-NERRE materials extraction Models read materials papers and link dopants to host materials, guest species to metal-organic frameworks, and crystal structures to chemical formulas. | University | Dec 2022 | Not in cards | |
| DiSCoMaT Models extract material identifiers, constituents, percentages, and units from tables in materials science papers. The original test contains 737 tables from 326 papers. | University | Jun 2022 | Not in cards | |
| ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. | School | Mar 2022 | 0.15 | Not in cards |
| MatBERT gold nanoparticle entity recognition Models mark particle names and words describing their shapes in passages from gold nanoparticle papers. The evaluation scores both types of text span. | University | Oct 2021 | Not in cards | |
| ULSA synthesis action retrieval Models identify and classify actions in sentences about inorganic materials synthesis. The corpus contains 3,040 labeled sentences from 535 paragraphs. | University | Oct 2021 | Not in cards | |
| MMLU-stem Models answer the STEM subjects of MMLU, such as physics, chemistry, mathematics, and computer science, as multiple choice. Scoring is accuracy. | University | Sep 2020 | | Dropped |
| SOFC-Exp Models identify sentences that report fuel cell experiments, mark materials and measurements, and assign roles such as anode material or operating temperature. | University | Jun 2020 | Not in cards | |
| Matbench Models predict properties of inorganic materials in 13 supervised tasks built from DFT and experimental data. Scored with error or classification metrics per task. | Expert | May 2020 | | Fading |
| SC-CoMIcs Models read superconductivity abstracts and identify materials, properties, and measurements. The corpus also records relationships, including which dopant replaces an element. | University | May 2020 | Not in cards | |
| Materials Science Procedural Text Corpus Models identify operations in materials synthesis procedures and link each operation to its materials, instruments, and conditions. Experts annotated 230 procedures. | University | Nov 2018 | Not in cards | |
| SciQ Models answer crowdsourced multiple-choice science exam questions in physics, chemistry and biology, each with a supporting passage. | School | Jul 2017 | Not in cards | |
| AP Chemistry Models take a simulated AP Chemistry exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | 4.00 | System cards only | |
| BioCreative V Chemical-Disease Relation Task Models identify biomedical entities and chemical-induced disease relations in PubMed abstracts. The OpenMed port extracts chemical and disease names. | University | Not in cards | ||
| ChemProt Models identify the relationship between a chemical and a protein in a biomedical abstract. The shared task evaluates five relationship types. | Expert | Not in cards | ||
| Meta chemical-agent refusal evaluation Models receive high-risk chemical prompts and the benchmark measures how often they refuse. Covers development, procurement and production stages. | Expert | | System cards only | |
| SuperCLUE-Science Models solve Chinese science reasoning problems. | University | 80.8 | Not in cards | |
| USNCO Local Section Exam 2022 Models take the 2022 US National Chemistry Olympiad local section exam. Scored out of 60 questions. | Competition | 36.0 | System cards only |