Benchmark Almanac

KAIST

We track 15 benchmarks that KAIST made or helped make.

2 in use 10 with public code 12 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
CLIcK Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Knowledge in many languages Mar 2024 90.7% In use
KorMedMCQA Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. Medicine Mar 2024 92.7% In use
BLEnD Models answer 52.6k questions about everyday culture in 16 countries and regions across 13 languages. Scoring is accuracy on short-answer and multiple-choice formats. Knowledge in many languages Jun 2024 66.3% Dropped
K-MetBench Models answer 1,774 questions from the Korean National Meteorological Engineer Examination under a zero-shot protocol. The benchmark reports answer accuracy across five subjects and separate visual and Korean-specific subsets. Expert rationales support an additional rubric score for reasoning. Science questions Apr 2026 93.7% Not in cards
MentalBench (Psychiatric Diagnosis) Models select psychiatric diagnoses for 24,750 synthetic cases. Exact matching requires both diagnoses when the case lacks the evidence needed to distinguish them. Medicine Feb 2026 62.7% Not in cards
WearableQA Models answer 4,084 ten-option questions using longitudinal wearable measurements, blood biomarkers, and demographics from 200 users. Questions test computations and physiological interpretation across single or multiple signals. Accuracy compares the selected option with the answer key. Medicine Sep 2026 72.9% Not in cards
ProgramDistill Agents inspect working web applications and restore missing features in editable copies. Replayed browser traces check whether patches recover the reference behavior. The paper evaluates partial repairs and full application reconstruction. Web and app front ends Sep 2026 84.3% Not in cards
FinRED Financial Safety Models answer prompts about financial risks. A finance-specific judge grades each response and reports the percentage that assist harmful conduct. Deception and alignment Jun 2026 0.15% Not in cards
AnyAppBench Agents repeat the same task in different Android apps, with and without suggested steps. Each app has a verifier that checks the same success condition. Phones Sep 2026 67.8% Not in cards
B-MoCA Agents use Android apps such as Instagram on phones with different languages, font sizes, and icon positions. Task-specific checks decide whether each attempt succeeded. Phones Apr 2024 47.7% Not in cards
EHRNoteQA Models read a patient's accumulated MIMIC-IV discharge summaries and answer clinical questions about them. The set has 962 question and answer pairs, each for a different patient. Medicine Feb 2024 96.2% Not in cards
ReTrace Computer agents recover from injected errors in desktop tasks. Replaying the remaining actions checks whether the recovered state produces the original successful outcome. Desktop apps Oct 2026 63.6% Not in cards
PerMemBench Memory systems for long-running agents decide what to store from years of interactions across many domains and user personas. Later questions check whether the system kept the context that matters for each user. The benchmark tests whether memory policies can adapt to the person instead of storing everything the same way. Learning from the context May 2026 0.82 Not in cards
AssurAI Models respond to Korean prompts across 35 risk categories. Model judges rate the safety of text, image, audio, and video outputs from 1 to 5. Deception and alignment Nov 2025 Not in cards
SoCRATES Models mediate disputes in 600 scenarios across eight domains. A model judge rates topic agreement to measure consensus gain, intervention timeliness, and intervention effectiveness. Multi-turn conversation Jun 2026 Not in cards