Benchmark Almanac

HAE-RAE

We track 7 benchmarks that HAE-RAE made or helped make.

0 in use 1 with public code 5 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
KMMLU Models answer 35,030 Korean multiple-choice questions across 45 subjects, collected from original Korean exams. Scoring is accuracy. Knowledge in many languages Feb 2024 77% Fading
HRM8K Models solve 8,011 math problems given as English and Korean parallel pairs. It measures how much math reasoning drops when problems are in Korean. Word problems Jan 2025 63.4% Fading
HAE-RAE Bench Models answer Korean questions on vocabulary, history, general knowledge and reading comprehension. It tests cultural and contextual knowledge that translated benchmarks miss. Knowledge in many languages Sep 2023 86.2% Fading
HRM8K-KSM Models solve the 1,428 problems of the Korean School Math subset of HRM8K. Problems come from Korean olympiad and competition exams. Competition problems with answers Jan 2025 31.6% Fading
KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. General visual questions Apr 2026 42% Fading
KMMLU-Hard Models answer about 5,000 Korean multiple-choice questions from KMMLU that most models got wrong, across many expert subjects. Knowledge in many languages Jan 2024 30.5% Not in cards
CSATQA Models read Korean exam passages and choose among five answers. The harness aggregates results across six question groups. Reading in many languages Jul 2023 Not in cards