Benchmark Almanac

ETH Zurich SRI Lab

We track 5 benchmarks that ETH Zurich SRI Lab made or helped make.

2 in use 4 with public code 5 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
MathArena (Apex, IMO, USAMO) Evaluates models on newly released math competitions after release. Final-answer or human/LLM-judged proof scores. Competition problems with answers Feb 2025 81.3% In use
MathArena Apex Shortlist Models solve 47 hard final-answer problems that MathArena selected from 2025 competitions, where the best models scored about 50% at selection. Scored by accuracy over repeated attempts. Competition problems with answers Aug 2025 98.4% In use
BaxBench Models write backend servers from a specification. Each solution is checked for correctness by tests and for security by end to end exploits. Secure code and review Feb 2025 68.6% Not in cards
AutoBaxBench Models write backend servers for tasks that a pipeline generated. Tests and exploits are generated too. A larger task set built in the style of BaxBench. Secure code and review Dec 2025 88.6% Not in cards
BrokenMath Models are asked to prove false versions of 2025 competition problems. An LLM judge counts how often a model writes a proof instead of rejecting the statement. Written proofs Oct 2025 29% Not in cards