Benchmark Almanac

National Institute of Standards

We track 9 benchmarks and 1 arena that National Institute of Standards made or helped make.

0 in use 0 with public code 0 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
2024 NIST GenAI Text-to-Text Discriminators Detectors estimate the probability that a machine wrote a summary, without seeing the source articles. NIST reports the area under the receiver operating characteristic curve. Other Apr 2024 Not in cards
MedVidQA 2023 Medical Instructional Question Generation Models write a question that a supplied medical video segment answers. They can use both the images and the subtitles. Medicine Feb 2023 Not in cards
MedVidQA 2023 Video Corpus Visual Answer Localization Models search 12,657 medical videos and return the times when a video shows the answer. Half of the 40 questions require visual evidence. Medicine Feb 2023 Not in cards
MedVidQA 2024 Query-Focused Instructional Step Captioning Models divide a medical demonstration into steps that answer a question. They give each step a caption and start and end times. Medicine Feb 2024 Not in cards
MedVidQA 2024 Video Corpus Visual Answer Localization Models search 48,605 medical videos for demonstrations that answer 52 new questions. They return ranked videos and the start and end times of each answer. Medicine Feb 2024 Not in cards
PLABA 2023 Abstract Adaptation Models rewrite biomedical abstracts for patients. Four annotators independently adapted each of the 400 test abstracts. Medicine May 2023 Not in cards
PLABA 2024 Complete Abstract Adaptation Models rewrite 400 new biomedical abstracts for patients. Reviewers grade how well the rewrites preserve information and explain it clearly. Medicine Jan 2024 Not in cards
PLABA 2024 Term Replacement Models identify difficult medical terms, classify how to adapt them, and supply wording for patients. The organizers held out 300 abstracts. Medicine Aug 2024 Not in cards
TREC-COVID Models rank scientific articles that address specific information needs about COVID-19. Questions that need search Apr 2020 Not in cards

Arenas

Name Kind Released Best score Status
2024 NIST GenAI Text-to-Text Generators Models summarize several articles for a given topic. NIST compares detector scores for generated and human summaries. Higher target Brier scores mean detectors more often miss generated summaries. Reasoning over long documents Apr 2024 Not in cards