Benchmark Almanac

US CAISI

We track 3 benchmarks that US CAISI made or helped make.

1 in use 1 with public code 2 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Gray Swan IPI Arena benchmark Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Jailbreaks and prompt injection Mar 2026 0% In use
CTF-Archive Diamond Agents solve 285 difficult capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. Capture-the-flag challenges May 2026 71% Not in cards
CTF-Archive Agents solve capture-the-flag tasks drawn from the pwn.college cybersecurity platform. Scores measure the share of tasks solved. Capture-the-flag challenges Sep 2025 Not in cards