Irregular
We track 4 benchmarks that Irregular made or helped make.
3 in use 0 with public code 2 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. | Capture-the-flag challenges | Dec 2025 | | In use |
| Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. | Capture-the-flag challenges | Dec 2025 | 90.9% | In use |
| FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. | Finding and exploiting real bugs | Jun 2026 | 38.1% | In use |
| Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). | Capture-the-flag challenges | | System cards only |