Benchmark Almanac

Chemistry and materials

Tasks that need chemistry or materials science. We count 147 benchmarks here, 1 of them arenas. A benchmark belongs here when subject includes chemical sciences, chemical engineering or materials engineering.

How fast it grows

33 were released since July, more than the 22 in the first half of 2026.

  1. Earlier 27
  2. 2024 H1 14
  3. 2024 H2 5
  4. 2025 H1 17
  5. 2025 H2 23
  6. 2026 H1 22
  7. 2026 H2 33

How much is still open

No model has passed 85% on 67 of the 80 benchmarks here that score in percent, 84% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
ChemInsight Models solve related chemistry problems in bundles. The authors score how many parts a model can solve and how far its reasoning remains correct. University Nov 2026 0.28 Not in cards
Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. Research frontier Nov 2026 44.7 Not in cards
ChemArgus Models solve chemistry competition and textbook problems with text and figures. A model judge awards partial credit for each subquestion. Competition Oct 2026 64.9% Not in cards
DualChem Models answer chemistry questions written by experts. The researchers report answer accuracy and assess whether models recognize possible misuse. University Oct 2026 67.3% Not in cards
EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. University Oct 2026 Not in cards
Into the Unknown Agents get a science paper and an unpublished follow-up direction from a DOE national lab scientist, then write the follow-up paper. Rubrics built from the scientist's conditions grade it, plus an integrity check. Research frontier Oct 2026 0% Not in cards
PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. Research frontier Oct 2026 14% Not in cards
SCI-AGENT-EXAM Agents complete research tasks across 11 scientific fields. Programs check executable outputs, and a model judge grades written work. Expert Oct 2026 80.9 Not in cards
ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. Expert Oct 2026 0.53 Not in cards
SciTaste-Bench Models judge the research value of papers in six disciplines. The benchmark compares their judgments with a reference index built from seven dimensions. Expert Oct 2026 28.0 Not in cards
SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. Expert Oct 2026 51.2% In use
Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. Expert Oct 2026 68.4% In use
CompMat-Bench Agents complete individual steps from published computational materials studies. Fixed rules compare their inputs and outputs with results reproduced by the authors. Expert Sep 2026 91.5% Not in cards
EduFair-Bench Tutors teach the same simulated student under different demographic descriptions. A judge grades individual turns and whole conversations to measure changes in teaching quality. University Sep 2026 Not in cards
EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. Research frontier Sep 2026 29.8 Not in cards
MANDOS Session 2 Models make process safety decisions across four private industrial scenarios. A model judge assesses each run. The leaderboard reports the share of passing runs. Expert Sep 2026 96.9% Not in cards
MatToolBench Agents use materials science software on Windows to operate interfaces, write scripts, and query databases. The evaluator awards partial credit for completed steps. University Sep 2026 51.5% Not in cards
Medicinal Chemistry (Anthropic) Models see several real drug-discovery compounds and pick which had the highest or lowest measured ADME property. The set has 504 questions. Expert Sep 2026 65.3% System cards only
Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. Expert Sep 2026 34% System cards only
OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. Expert Sep 2026 73.7% Not in cards
SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. Expert Sep 2026 45.3% Not in cards
AutoDS scientific machine learning cases Agents build predictive models for maize yield, polymer glass transition temperature, and supercomputer job outcomes. Each case supplies a train split and a held-out test. Expert Aug 2026 Not in cards
BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. Expert Aug 2026 91.4% System cards only
CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. Expert Aug 2026 1.80% System cards only
CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. Expert Aug 2026 3.50% System cards only
Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. Expert Aug 2026 38% System cards only
FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. Expert Aug 2026 20.6% Not in cards
Terminal-Bench-Science 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. Expert Aug 2026 68.1% In use
AI Security Leaderboard Models face systematic jailbreak attacks across chemical, biological, radiological, nuclear, explosive and cyber domains. Scores count universal jailbreaks and estimate attacker cost. Expert Jul 2026 $60 Not in cards
Autointent-Chem Models face manual and synthetic dangerous and benign chemistry queries. xAI reports refusal recall on the dangerous ones. Jul 2026 100% System cards only
IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. Research frontier Jul 2026 27.3% Not in cards
MedChemBench (OpenAI) Models work through 99 medicinal chemistry problems drawn from recent papers and patents. Each task mixes prose with structures, assay tables and plots, then asks for the program’s next decision. Expert Jul 2026 83.8% In use
Organic Chemistry V2 (Anthropic) Models predict structures from spectra, design multi-step syntheses, predict reaction products and read chemical structure images. V2 adds harder expert-recommended problems. Expert Jul 2026 69.2% System cards only
Artificial Analysis Intelligence Index v4.1 Version 4.1 of Artificial Analysis's index, which moved to GDPval-AA v2, Terminal-Bench 2.1 and τ³-Banking and put the most weight on agents. Jun 2026 61.0 In use
DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. Expert Jun 2026 71.5% Not in cards
MANDOS Run 003 Models make decisions during seven industrial incidents and four fictional scenarios. A model judge checks their decisions, including whether they seek expert advice. Expert Jun 2026 100% Not in cards
TEM-VQA Models give short written answers about structures, image features and experimental design in transmission electron micrographs. Expert Jun 2026 40.4 Not in cards
CUSP Models forecast scientific progress across 4,760 events in many fields, using only knowledge from before a set cutoff date. Tasks ask whether an advance is feasible, how it would work, how to design a solution and when it will happen. The benchmark also compares events before and after each model's training cutoff. Expert May 2026 60.4% Not in cards
K12-Bench Models select curriculum concepts, prerequisite relationships, related knowledge, experimental evidence, and textbook locations. The 23,640 multi-select questions come from a textbook knowledge graph. Exact match and instance-level F1 compare selected options with graph-derived answers. School May 2026 57.1% Not in cards
OmniMatBench Models answer questions and perform calculations across 19 materials science fields. The authors compare automated grades with human assessments. University May 2026 0.37 Not in cards
ResearchClawBench Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs. Research frontier May 2026 38.6 Not in cards
Science Sim Chemistry Models compare simulated compounds and choose whether to advance or reject them. The evaluator scores decisions against measured reaction outcomes and a budget. University May 2026 Not in cards
SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. Expert May 2026 40.2% Not in cards
Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. Expert Apr 2026 39.3% System cards only
LongCoT Models solve long reasoning problems across five domains. Task-specific programs verify answers, with optional judge fallback for ambiguous mathematics and chemistry responses. Expert Apr 2026 25.1% Not in cards
Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. Expert Apr 2026 System cards only
ChemDIRT Models answer chemistry questions while the evaluator varies instructions and molecular representations. The authors measure accuracy and consistency across variants. University Mar 2026 48.7% Not in cards
CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. Mar 2026 83.7% System cards only
ChemPro Models answer chemistry questions across four difficulty levels. The authors report multiple-choice accuracy and accept numerical answers within specified tolerances. University Feb 2026 38.7% Not in cards
Organic Chemistry (Anthropic) Models predict structures from spectra, design synthetic routes, predict reaction products and convert between IUPAC names, SMILES and structure images. Expert Feb 2026 86.2% System cards only
SPM-Bench Models interpret microscopy images and select all correct answers. The primary score gives partial credit and penalizes incorrect selections. Expert Feb 2026 0.94 Not in cards
A^3-Bench Models solve 2,198 science reasoning problems annotated with the anchors and attractors (prior knowledge) they need. An AAUI metric measures how much of that knowledge the model activates. University Jan 2026 65.1% Not in cards
MolecularIQ Models answer questions about molecular graphs, such as counting, indexing and constrained generation, and a symbolic checker verifies each answer. Expert Jan 2026 47.5% In use
SciPredict Models predict the outcomes of 405 real physics, biology and chemistry experiments published after March 31, 2025, with and without expert background knowledge. Expert Jan 2026 25.3% Not in cards
Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. Expert 2026 Not in cards
AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. Expert Dec 2025 47.7% Fading
AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages. Expert Dec 2025 44% Not in cards
FrontierScience Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. Expert Dec 2025 87% In use
FrontierScience-Olympiad Models solve olympiad-style physics, chemistry and biology problems at the level of IPhO, IChO and IBO, written by medalists and coaches. Graded on short final answers. Competition Dec 2025 77.1% System cards only
FrontierScience-Research Models work on open-ended PhD-level research subtasks in physics, chemistry and biology, written by working scientists. A grader scores each answer against a 10-point rubric. Expert Dec 2025 38% Fading
SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. Expert Dec 2025 33.8% In use
Solubility Optimization (anindya) Models choose chemical transformations to improve predicted solubility. The package includes a benchmark runner that measures improvement on evaluation examples. University Dec 2025 Not in cards
SUPERChem Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score. Expert Dec 2025 66.4% In use
ATLAS (OpenCompass) Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge. Research frontier Nov 2025 42.9% Not in cards
CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. Expert Nov 2025 System cards only
BlueprintSymVL Models use an example symbol to find matching components in piping and instrumentation diagrams. They must give both the correct count and the component labels. School Oct 2025 Not in cards
MatSciBench Models solve college materials science problems, including questions with figures. The authors report accuracy separately for text and image questions. University Oct 2025 64% Not in cards
ProfBench Model writes expert-level reports in physics, chemistry, finance and consulting. Calibrated LLM judges grade against PhD/MBA-written rubrics. Expert Oct 2025 66.2% In use
Artificial Analysis Intelligence Index v3 Version 3 of Artificial Analysis's index, ten evaluations that added agentic terminal and tool-use tasks to the knowledge, math and coding tests. Sep 2025 61.0 Dropped
JGPQA Models answer graduate-level science questions translated into Japanese from GPQA. Expert Sep 2025 Not in cards
MatCha Models answer multiple-choice questions about material structure, shape and properties using images from materials characterization instruments. Expert Sep 2025 68.1% Not in cards
SciReasoner evaluation suite Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks. Expert Sep 2025 63.3% Fading
AstaBench Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem. Expert Aug 2025 58% Not in cards
HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. Expert Aug 2025 16% Fading
3LM Models answer Arabic STEM questions taken from Arabic textbooks and worksheets, plus synthetic STEM questions from the same sources. A third part tests Arabic code generation. School Jul 2025 85.7% Not in cards
HLE Bio/Chem Gold Models answer biology and chemistry questions from Humanity’s Last Exam whose answers reviewers supported with published research. Expert Jul 2025 Not in cards
IChO 2025 Models solve the written theoretical problems of the 57th International Chemistry Olympiad, held in Dubai in July 2025. Answers are graded against the official marking scheme. Competition Jul 2025 Fading
SciArena Researchers ask scientific literature questions and vote between two models' long cited answers. Votes give Elo ratings. Expert Jul 2025 1,151 Not in cards
ClearHarm Models answer unambiguously harmful CBRN questions. Judges score whether attacks elicit harmful responses from the model. Expert Jun 2025 Not in cards
ether0 benchmark Models answer chemistry problems from several task types. The benchmark checks their answers with automated reward functions. Expert Jun 2025 Not in cards
ExpertLongBench Models write documents for 11 professional tasks. A judge compares facts in each document with a reference checklist and scores precision and recall. Expert Jun 2025 33.4% Not in cards
K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. School Jun 2025 63.4% Not in cards
K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. School Jun 2025 55.4% Not in cards
Matter-of-Fact Models decide whether materials science claims are feasible using evidence available before a cutoff date. The authors compare their answers with later published findings. Expert Jun 2025 72% Not in cards
Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. Expert Jun 2025 60.7% Fading
ChemCoTBench Models interpret and edit molecular structures, optimize molecular properties and predict reaction outcomes. Each task has its own scoring rule. University May 2025 Not in cards
CSVQA Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images. School May 2025 49.6% System cards only
MatTools Models answer questions about materials science software and write Python programs to calculate material properties. The authors execute the generated programs. University May 2025 81.6% Not in cards
RBench-M Models answer 665 graduate-level multimodal reasoning questions in English and Chinese. Top-1 accuracy compares selected answers with the answer key. Expert May 2025 69.9% Not in cards
SoSBench Models answer hazardous requests drawn from real regulations in six scientific fields. Scored by how often the model complies with a request it should refuse. Expert May 2025 10.6% Not in cards
xbench-ScienceQA Models answer postgraduate-level science questions written by experts, in Chinese, as part of the xbench AGI Tracking series. Expert May 2025 76.4% Not in cards
CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. Expert Mar 2025 System cards only
Artificial Analysis Intelligence Index v2 Artificial Analysis's index of seven public reasoning, knowledge, math and coding benchmarks, each run by Artificial Analysis. Feb 2025 67.7 Dropped
Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. University Feb 2025 25.0 System cards only
EMMA Models solve math, physics, chemistry and coding problems that need reasoning over text and images together, not each alone. Answers are checked against a key. University Jan 2025 42.5% Fading
Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. Research frontier Dec 2024 System cards only
TOMG-Bench Models generate molecules from natural language requests in three tasks: molecule editing, molecule optimization and customized generation. Each output is checked for validity and for meeting the requested properties. Expert Dec 2024 18.6% In use
MaCBench Vision language models answer chemistry and materials questions from images. Tasks cover reading data, understanding experiments and interpreting results. Expert Nov 2024 72.6% Not in cards
Open Ko-LLM Leaderboard 2 Models complete nine Korean evaluations covering reasoning, instructions, social values, safety, and other abilities. The organizers score submitted models on private test data. School Aug 2024 49.6% Not in cards
SciCode Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Expert Jul 2024 59.8% In use
ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. University Jun 2024 69.7% Dropped
GAOKAO-Eval 2024 Models take Chinese national college entrance examinations. High school teachers grade the open responses. School Jun 2024 476.5 Not in cards
MMLU-Pro Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. University Jun 2024 89.8% In use
OlympicArena Models solve 11,163 bilingual problems from 62 international olympiad competitions in seven fields, some with images. Answers are checked against keys. Competition Jun 2024 40.5% Dropped
SciKnowEval Models answer scientific questions and apply knowledge in biology, chemistry, materials and physics. The suite uses automatic checks and model judges for different tasks. University Jun 2024 0.71 Not in cards
ARC Challenge MT Models answer science questions from ARC-Challenge in translation. The harness scores answers in the target language against the original choices. School May 2024 Not in cards
ChemBench Models answer more than 2,700 chemistry questions on knowledge and reasoning. Results are compared with a group of human chemists. Expert Apr 2024 70.9% Dropped
RiceChem Models grade long answers to four college chemistry questions. Each of the 1,264 answers has human judgments for the applicable rubric items. University Apr 2024 0.43 Not in cards
EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. School Mar 2024 44% Not in cards
WMDP Proxy multiple-choice benchmark for hazardous knowledge in bio, chem and cyber. Released with the RMU unlearning method. Expert Mar 2024 84.8% System cards only
WMDP Chemistry Models answer 408 multiple-choice questions that proxy hazardous chemical security knowledge. Chemistry split of WMDP. Expert Mar 2024 90% Fading
PNCExtract Models read a research paper and list its polymer nanocomposite samples. They link each sample to its matrix, filler, and composition by mass or volume. University Feb 2024 Not in cards
SMolInstruct Models solve 14 chemistry tasks such as name conversion, property prediction, molecule captioning, generation and reaction prediction. Each task uses its own metric on the test split. Expert Feb 2024 74.8% Fading
CMMU Models answer Chinese multimodal questions across seven school subjects. School Jan 2024 30.9 Not in cards
GPQA Diamond 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. Expert Nov 2023 97% In use
GPQA Extended Models answer all 546 GPQA questions, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. Expert Nov 2023 48.8% System cards only
GPQA Main Models answer the 448 question main set of GPQA, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. Expert Nov 2023 80.3% Dropped
SciBench Models solve open-ended college-level problems in mathematics, chemistry and physics. Most answers are numbers, which a script checks against the key. University Jul 2023 8.45% Not in cards
Mol-Instructions Models answer molecule-oriented, protein-oriented and biomolecular text instructions, such as reaction prediction, molecule description and protein function questions. Scored per task with task metrics. Expert Jun 2023 54% Fading
Mol-Instructions Chemical Entity Recognition Models identify chemical entities in biomedical text. The score is test-set F1. Expert Jun 2023 Not in cards
Mol-Instructions Chemical-disease Interaction Extraction Models extract chemical and disease interactions from text. The score is test-set F1. Expert Jun 2023 Not in cards
Mol-Instructions Chemical-protein Interaction Extraction Models extract chemical and protein interactions from text. The score is test-set F1. Expert Jun 2023 Not in cards
Mol-Instructions Description-guided Molecule Design Models generate molecules from descriptions. The score is exact-match accuracy on the molecule design test set. Expert Jun 2023 Not in cards
Mol-Instructions Forward Reaction Prediction Models predict reaction products from reactants. The score is exact-match accuracy on the forward reaction test set. Expert Jun 2023 Not in cards
Mol-Instructions Reagent Prediction Models predict reaction reagents. The score is exact-match accuracy on the reagent prediction test set. Expert Jun 2023 Not in cards
Mol-Instructions Retrosynthesis Models predict reactants for target products. The score is exact-match accuracy on the retrosynthesis test set. Expert Jun 2023 Not in cards
ChatGPT Chemistry Assistant MOF synthesis extraction Models extract 11 synthesis details from papers about metal-organic frameworks. Researchers check the reported materials, amounts, temperatures, and reaction times against their annotations. University May 2023 Not in cards
GAOKAO-Bench Models answer Chinese national college entrance exam questions from 2010 to 2022 across nine subjects, with objective and subjective questions. School May 2023 72.2% Dropped
MatSci-NLP Models read text from materials science papers and complete seven tasks, including identifying synthesis actions and linking measurements to experiments. The authors used 99% of examples for testing. University May 2023 Not in cards
AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. School Apr 2023 89.2% Fading
LLM-NERRE materials extraction Models read materials papers and link dopants to host materials, guest species to metal-organic frameworks, and crystal structures to chemical formulas. University Dec 2022 Not in cards
DiSCoMaT Models extract material identifiers, constituents, percentages, and units from tables in materials science papers. The original test contains 737 tables from 326 papers. University Jun 2022 Not in cards
ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. School Mar 2022 0.15 Not in cards
MatBERT gold nanoparticle entity recognition Models mark particle names and words describing their shapes in passages from gold nanoparticle papers. The evaluation scores both types of text span. University Oct 2021 Not in cards
ULSA synthesis action retrieval Models identify and classify actions in sentences about inorganic materials synthesis. The corpus contains 3,040 labeled sentences from 535 paragraphs. University Oct 2021 Not in cards
MMLU-stem Models answer the STEM subjects of MMLU, such as physics, chemistry, mathematics, and computer science, as multiple choice. Scoring is accuracy. University Sep 2020 83.6% Dropped
SOFC-Exp Models identify sentences that report fuel cell experiments, mark materials and measurements, and assign roles such as anode material or operating temperature. University Jun 2020 Not in cards
Matbench Models predict properties of inorganic materials in 13 supervised tasks built from DFT and experimental data. Scored with error or classification metrics per task. Expert May 2020 75.0 Fading
SC-CoMIcs Models read superconductivity abstracts and identify materials, properties, and measurements. The corpus also records relationships, including which dopant replaces an element. University May 2020 Not in cards
Materials Science Procedural Text Corpus Models identify operations in materials synthesis procedures and link each operation to its materials, instruments, and conditions. Experts annotated 230 procedures. University Nov 2018 Not in cards
SciQ Models answer crowdsourced multiple-choice science exam questions in physics, chemistry and biology, each with a supporting passage. School Jul 2017 Not in cards
AP Chemistry Models take a simulated AP Chemistry exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. School 4.00 System cards only
BioCreative V Chemical-Disease Relation Task Models identify biomedical entities and chemical-induced disease relations in PubMed abstracts. The OpenMed port extracts chemical and disease names. University Not in cards
ChemProt Models identify the relationship between a chemical and a protein in a biomedical abstract. The shared task evaluates five relationship types. Expert Not in cards
Meta chemical-agent refusal evaluation Models receive high-risk chemical prompts and the benchmark measures how often they refuse. Covers development, procurement and production stages. Expert 100% System cards only
SuperCLUE-Science Models solve Chinese science reasoning problems. University 80.8 Not in cards
USNCO Local Section Exam 2022 Models take the 2022 US National Chemistry Olympiad local section exam. Scored out of 60 questions. Competition 36.0 System cards only