Benchmark Almanac

ETH Zurich

We track 12 benchmarks that ETH Zurich made or helped make.

1 in use 10 with public code 6 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Lakera B3 (Backbone Breaker Benchmark) Crowdsourced prompt-injection attacks hit backbone LLMs inside agent threat snapshots at three defense levels. The score is attack resistance, 100 minus the board's vulnerability percentage. Jailbreaks and prompt injection Oct 2025 89% In use
SWT-bench Agents write tests that reproduce real GitHub issues from SWE-bench repositories. Scored by the share of tests that fail before and pass after the fix. Fixing real repositories Jun 2024 89.1% Fading
OSI-Bench Spatial reasoning questions on outdoor pedestrian-view videos with metric 3D ground truth from synchronized stereo cameras, LiDAR and GPS. 8,736 questions range from relative direction to metric and motion estimates. Spatial reasoning and counting Dec 2025 30.6% System cards only
MathTutorBench Models solve math problems, diagnose student mistakes, ask Socratic questions, and give instructional support. Automatic answer checks, F1, and text similarity score the constrained tasks. A trained pedagogical reward model compares tutoring responses with teacher examples. Word problems Feb 2025 0.98 Not in cards
IMProofBench Agents with web search and SageMath write proofs for private PhD-level problems written by mathematicians. Experts grade the proofs, and final-answer subquestions are graded automatically. Research mathematics Sep 2025 49.3% Not in cards
CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. Finding and exploiting real bugs Jul 2026 34.2% Not in cards
AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. Machine learning engineering Mar 2025 22% Not in cards
Articulate3D Models locate movable parts and handles in scanned rooms. They predict how each part moves, including its axis and the location of its hinge. Spatial reasoning and counting Dec 2024 Not in cards
EMDB Models reconstruct human pose and body shape in everyday scenes. The benchmark compares joint positions with electromagnetic measurements. Spatial reasoning and counting Aug 2023 Not in cards
Food-101 Models identify the dish in a food photograph from 101 categories. General visual questions Sep 2014 Not in cards
MAUD Models classify provisions in merger agreements into expert-defined legal categories. The benchmark compares predictions with lawyer annotations and reports precision-recall area and micro and macro F1. Law Jan 2023 Not in cards
SceneFun3D Models identify movable parts in indoor scans, locate the part needed for a written task, and estimate how it moves. Spatial reasoning and counting Mar 2024 Not in cards