EleutherAI
We track 8 benchmarks that EleutherAI made or helped make. Its models are on its lab page.
0 in use 5 with public code 0 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Pile-test Models predict text from the test split of the Pile, an 800GB corpus of 22 sources. Scored by bits per byte. | Other | Dec 2020 | | Dropped |
| Japanese LLM Leaderboard (lm-evaluation-harness) Models complete eight Japanese tasks covering reading, reasoning, math, and summarization. The harness groups the individual results. | Reading in many languages | Nov 2024 | Not in cards | |
| LAMBADA OpenAI Multilingual Models predict the last word in short narrative passages. The harness includes English and four machine-translated test sets. | Reading in many languages | Not in cards | ||
| Medical Prescription Selection (lm-evaluation-harness) Models read a diagnosis or complaint and choose the recorded prescription. The two task variants use different distractors. | Medicine | Mar 2025 | Not in cards | |
| MIMIC-III Report Summarization Models generate the impression section of a clinical report from the preceding text. The harness compares the response with the recorded impression. | Medicine | Mar 2025 | Not in cards | |
| MMLU Early Training Models score complete answer texts on selected MMLU questions. The metric compares the correct answer’s log likelihood with the average for incorrect answers. | Exam questions | Dec 2025 | Not in cards | |
| Simple Cooccurrence Bias Models complete occupation prompts with gendered words. The harness reports the percentage of prompts where the model prefers a male term. | Bias, fairness and politics | May 2020 | Not in cards | |
| Soohak Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems. | Research mathematics | May 2026 | Not in cards |