Benchmark Almanac

EleutherAI

We track 8 benchmarks that EleutherAI made or helped make. Its models are on its lab page.

0 in use 5 with public code 0 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Pile-test Models predict text from the test split of the Pile, an 800GB corpus of 22 sources. Scored by bits per byte. Other Dec 2020 0.55 Dropped
Japanese LLM Leaderboard (lm-evaluation-harness) Models complete eight Japanese tasks covering reading, reasoning, math, and summarization. The harness groups the individual results. Reading in many languages Nov 2024 Not in cards
LAMBADA OpenAI Multilingual Models predict the last word in short narrative passages. The harness includes English and four machine-translated test sets. Reading in many languages Not in cards
Medical Prescription Selection (lm-evaluation-harness) Models read a diagnosis or complaint and choose the recorded prescription. The two task variants use different distractors. Medicine Mar 2025 Not in cards
MIMIC-III Report Summarization Models generate the impression section of a clinical report from the preceding text. The harness compares the response with the recorded impression. Medicine Mar 2025 Not in cards
MMLU Early Training Models score complete answer texts on selected MMLU questions. The metric compares the correct answer’s log likelihood with the average for incorrect answers. Exam questions Dec 2025 Not in cards
Simple Cooccurrence Bias Models complete occupation prompts with gendered words. The harness reports the percentage of prompts where the model prefers a male term. Bias, fairness and politics May 2020 Not in cards
Soohak Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems. Research mathematics May 2026 Not in cards