Benchmark Almanac

Abundant AI

We track 5 benchmarks that Abundant AI made or helped make.

2 in use 5 with public code 4 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
SWE-Marathon 20 realistic multi-hour software tasks (library reproductions, product clones, ML work). Verifiers combine hidden tests, parity and integrity checks. Building whole projects Jun 2026 29% In use
SWE-Marathon v1.1 Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials. Building whole projects Jul 2026 53.8% In use
Incident Arena Agents diagnose and repair incidents in running services. The score averages the repair success rate across 20 incidents. Terminal tasks Sep 2026 64.3% Not in cards
FrontierRSI Agents do 27 open-ended AI research and engineering tasks across pre-training, post-training and harnesses, inference serving and GPU kernels. Each trial is scored against the best valid result any model reached. AI research tasks Nov 2026 34.3% Not in cards
SWE-gen JS Agents fix issues drawn from 1,000 merged pull requests across 30 JavaScript and TypeScript repositories. Each task includes tests that fail before the reference fix. Fixing real repositories Jan 2026 Not in cards