Abundant AI
We track 5 benchmarks that Abundant AI made or helped make.
2 in use 5 with public code 4 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| SWE-Marathon 20 realistic multi-hour software tasks (library reproductions, product clones, ML work). Verifiers combine hidden tests, parity and integrity checks. | Building whole projects | Jun 2026 | 29% | In use |
| SWE-Marathon v1.1 Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials. | Building whole projects | Jul 2026 | | In use |
| Incident Arena Agents diagnose and repair incidents in running services. The score averages the repair success rate across 20 incidents. | Terminal tasks | Sep 2026 | 64.3% | Not in cards |
| FrontierRSI Agents do 27 open-ended AI research and engineering tasks across pre-training, post-training and harnesses, inference serving and GPU kernels. Each trial is scored against the best valid result any model reached. | AI research tasks | Nov 2026 | 34.3% | Not in cards |
| SWE-gen JS Agents fix issues drawn from 1,000 merged pull requests across 30 JavaScript and TypeScript repositories. Each task includes tests that fail before the reference fix. | Fixing real repositories | Jan 2026 | Not in cards |