ETH Zurich
We track 12 benchmarks that ETH Zurich made or helped make.
1 in use 10 with public code 6 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Lakera B3 (Backbone Breaker Benchmark) Crowdsourced prompt-injection attacks hit backbone LLMs inside agent threat snapshots at three defense levels. The score is attack resistance, 100 minus the board's vulnerability percentage. | Jailbreaks and prompt injection | Oct 2025 | 89% | In use |
| SWT-bench Agents write tests that reproduce real GitHub issues from SWE-bench repositories. Scored by the share of tests that fail before and pass after the fix. | Fixing real repositories | Jun 2024 | | Fading |
| OSI-Bench Spatial reasoning questions on outdoor pedestrian-view videos with metric 3D ground truth from synchronized stereo cameras, LiDAR and GPS. 8,736 questions range from relative direction to metric and motion estimates. | Spatial reasoning and counting | Dec 2025 | | System cards only |
| MathTutorBench Models solve math problems, diagnose student mistakes, ask Socratic questions, and give instructional support. Automatic answer checks, F1, and text similarity score the constrained tasks. A trained pedagogical reward model compares tutoring responses with teacher examples. | Word problems | Feb 2025 | | Not in cards |
| IMProofBench Agents with web search and SageMath write proofs for private PhD-level problems written by mathematicians. Experts grade the proofs, and final-answer subquestions are graded automatically. | Research mathematics | Sep 2025 | 49.3% | Not in cards |
| CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. | Finding and exploiting real bugs | Jul 2026 | 34.2% | Not in cards |
| AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. | Machine learning engineering | Mar 2025 | 22% | Not in cards |
| Articulate3D Models locate movable parts and handles in scanned rooms. They predict how each part moves, including its axis and the location of its hinge. | Spatial reasoning and counting | Dec 2024 | Not in cards | |
| EMDB Models reconstruct human pose and body shape in everyday scenes. The benchmark compares joint positions with electromagnetic measurements. | Spatial reasoning and counting | Aug 2023 | Not in cards | |
| Food-101 Models identify the dish in a food photograph from 101 categories. | General visual questions | Sep 2014 | Not in cards | |
| MAUD Models classify provisions in merger agreements into expert-defined legal categories. The benchmark compares predictions with lawyer annotations and reports precision-recall area and micro and macro F1. | Law | Jan 2023 | Not in cards | |
| SceneFun3D Models identify movable parts in indoor scans, locate the part needed for a written task, and estimate how it moves. | Spatial reasoning and counting | Mar 2024 | Not in cards |