Benchmark Almanac

Johns Hopkins University

We track 7 benchmarks that Johns Hopkins University made or helped make.

0 in use 5 with public code 3 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Winogender Models resolve pronouns in sentences about occupations. Accuracy shows whether gender stereotypes sway coreference. Bias, fairness and politics Apr 2018 79.2% System cards only
ProteinLMBench Models answer 944 hand-checked multiple choice questions on protein sequence, structure and function. Genomics and life sciences Jun 2024 57.9% Dropped
ReCoRD Models fill in a masked entity in a query about a news article, using the article as context. Scoring is exact match and token F1. Reading comprehension Oct 2018 93.8% Dropped
AL-QASIDA Models write in eight dialectal Arabic varieties. Scored on fidelity, understanding, quality and diglossia, including the ADI2 dialect identification score. Chat and instructions in many languages Dec 2024 System cards only
ManyIH-Bench Models resolve conflicts among instructions at up to twelve privilege levels. A response must satisfy all winning instructions and any applicable coding tests. Following precise instructions Apr 2026 60.9% Not in cards
HANS An entailment test built to catch three shallow shortcuts: lexical overlap, subsequence and constituent. Every premise and hypothesis pair shares words, but only some pairs are entailments. A model that relies on word overlap fails the non-entailment cases. Language reasoning Feb 2019 Not in cards
StyleInstructCaps Models describe a speaker's voice according to different instructions. The evaluator grades instruction following, accuracy about the recording, and consistency across the speaker's clips. Speech recognition and translation Oct 2026 Not in cards