Benchmark Almanac

THUDM

We track 4 benchmarks that THUDM made or helped make.

0 in use 4 with public code 0 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
LongBench Models work through 21 English and Chinese long-input datasets covering QA, summarization, few-shot learning, synthetic tasks, and code. Each dataset has its own metric. Reasoning over long documents Aug 2023 81.1% Dropped
LongBench English Models answer questions and complete other long-context tasks in the original English headline table. The score is its published average on a percentage scale. Reasoning over long documents Aug 2023 48.1% System cards only
HumanEval-X Models generate programs or translate solutions among Python, C++, Java, JavaScript, and Go. The 820 human-written language variants include executable tests, and scoring uses pass@k. Function-level problems Mar 2023 9.78% Not in cards
LongBench Chinese Models answer questions and complete other long-context tasks in the original Chinese headline table. The score is its published average on a percentage scale. Reasoning over long documents Aug 2023 Not in cards