Benchmark Almanac

Stony Brook University

We track 4 benchmarks that Stony Brook University made or helped make.

0 in use 4 with public code 3 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
AppWorld Test-Challenge Agents complete 417 app tasks that require Amazon or Gmail APIs held out from training. The score is the percentage passing every task test. Apps and MCP servers Jul 2024 92.8% Not in cards
AppWorld Test-Normal Agents complete 168 everyday app tasks from familiar applications by calling APIs through code. The score is the percentage passing every task test. Apps and MCP servers Jul 2024 94% Not in cards
PRMBench Models identify errors and redundant steps in mathematical solutions. PRMScore combines positive and negative F1 scores across simplicity, soundness, and sensitivity tests. Word problems Jan 2025 68.8 Not in cards
AppWorld Agents write code against 457 APIs of 9 simulated everyday apps to finish 750 tasks for about 100 fictional users. State-based unit tests check the outcome and collateral damage. Apps and MCP servers Jul 2024 Not in cards