Benchmark Almanac

Irregular

We track 4 benchmarks that Irregular made or helped make.

3 in use 0 with public code 2 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Capture-the-flag challenges Dec 2025 100% In use
Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. Capture-the-flag challenges Dec 2025 90.9% In use
FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. Finding and exploiting real bugs Jun 2026 38.1% In use
Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). Capture-the-flag challenges 70% System cards only