Benchmark Almanac

Saarland University

We track 3 benchmarks that Saarland University made or helped make.

0 in use 1 with public code 1 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
BenGER Models write German legal analyses for 596 exam-style cases and 531 doctrinal reasoning tasks. A rubric judge scores legal conclusions, reasoning, and presentation against expert references. Case scores map to the German 0-to-18 grading scale, with four points required to pass. Law May 2026 71.1 Not in cards
DFKI English-Russian Linguistic Test Suite at WMT25 Models translate English passages into idiomatic Russian. Rules match known translations. A linguist judges unresolved cases across 55 phenomena. The authors report item-weighted and category-weighted accuracy. Translation Jun 2025 89.5% Not in cards
Saarland MIDI-Audio Piano Music Models transcribe student piano performances. The Disklavier captured the reference notes as the musicians played. Audio understanding and voice Oct 2011 Not in cards