Benchmark Almanac

Waseda University

We track 3 benchmarks that Waseda University made or helped make.

0 in use 3 with public code 1 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
JMMLU Models answer 7,536 four-way multiple-choice questions in Japanese, from translated MMLU subjects and new Japan-specific subjects. Scored by accuracy. Knowledge in many languages Jan 2024 80.6% Dropped
CORAL Models retrieve passages, answer questions, and cite supporting evidence across conversations. Retrieval metrics, response-overlap metrics, model judgments, and citation precision and recall score different parts of the task. Questions that need search Oct 2024 16.3 Not in cards
JGLUE Models classify Japanese sentences, compare sentence meanings, and answer reading comprehension and commonsense questions. The authors publish baseline scores for each task. Reading in many languages Mar 2022 Not in cards