Benchmark Almanac

BinaryAudit

Claude Opus 4.6 49.1% leads since February 2026
Not in model cards

Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software.

33 tasks and 26 models as of March 7th, 2026. Runs on Harbor.

Made by
Quesma
Released
December 2025
Kind
Finding and exploiting real bugs
Level
Expert
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Strongest model tested
GPT-5.4 #27 on the capability index
Last model added
March 2026
Scores refreshed
November 12th, 2026
20
Models scored on its own leaderboard

2 models share the top score of 49.1% among models released by March 2026

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

The newest model with a score here came out in March 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

20 models. Scroll sideways for the rest.

  1. 49.1 Claude Opus 4.6
  2. 49.1 Gemini 3.1 Pro
  3. 45.6 GPT-5.2 Codex
  4. 43.9 Gemini 3 Pro
  5. 42.1 GPT-5.3 Codex
  6. 36.8 Claude Opus 4.5
  7. 36.8 Gemini 3 Flash
  8. 26.3 Claude Sonnet 4.5
  9. 24.6 GLM-4.7
  10. 24.6 Kimi K2.5
  11. 21.1 Gemini 2.5 Pro
  12. 19.3 GLM-5
  13. 17.5 GPT-5.2
  14. 17.5 GPT-5.4
  15. 14.0 Claude Sonnet 4
  16. 12.3 DeepSeek-V3.2
  17. 10.5 Claude Haiku 4.5
  18. 8.77 Grok 4
  19. 7.02 Grok 4.1 Fast
  20. 1.75 GPT-5
  1. 1 Claude Opus 4.6 49.1%
  2. 2 Gemini 3.1 Pro 49.1%
  3. 3 GPT-5.2 Codex xhigh 45.6%
  4. 4 Gemini 3 Pro 43.9%
  5. 5 GPT-5.3 Codex 42.1%
  6. 6 Claude Opus 4.5 36.8%
  7. 7 Gemini 3 Flash 36.8%
  8. 8 Claude Sonnet 4.5 26.3%
  9. 9 GLM-4.7 24.6%
  10. 10 Kimi K2.5 24.6%
  11. 11 Gemini 2.5 Pro Jun 2025 21.1%
  12. 12 GLM-5 19.3%
  13. 13 GPT-5.2 17.5%
  14. 14 GPT-5.4 xhigh 17.5%
  15. 15 Claude Sonnet 4 14%
  16. 16 DeepSeek-V3.2 12.3%
  17. 17 Claude Haiku 4.5 10.5%
  18. 18 Grok 4 8.77%
  19. 19 Grok 4.1 Fast 7.02%
  20. 20 GPT-5 1.75%

Source: Official BinaryAudit leaderboard

#ModelScoreSettingDateSource
1 Claude Opus 4.6 49.1% Feb 2026 Official leaderboard
2 Gemini 3.1 Pro 49.1% Feb 2026 Official leaderboard
3 GPT-5.2 Codex 45.6% xhigh Dec 2025 Official leaderboard
4 Gemini 3 Pro 43.9% Nov 2025 Official leaderboard
5 GPT-5.3 Codex 42.1% Feb 2026 Official leaderboard
6 Claude Opus 4.5 36.8% Nov 2025 Official leaderboard
7 Gemini 3 Flash 36.8% Dec 2025 Official leaderboard
8 Claude Sonnet 4.5 26.3% Sep 2025 Official leaderboard
9 GLM-4.7 24.6% Dec 2025 Official leaderboard
10 Kimi K2.5 24.6% Jan 2026 Official leaderboard
11 Gemini 2.5 Pro Jun 2025 21.1% Jun 2025 Official leaderboard
12 GLM-5 19.3% Feb 2026 Official leaderboard
13 GPT-5.2 17.5% Dec 2025 Official leaderboard
14 GPT-5.4 17.5% xhigh Mar 2026 Official leaderboard
15 Claude Sonnet 4 14% May 2025 Official leaderboard
16 DeepSeek-V3.2 12.3% Dec 2025 Official leaderboard
17 Claude Haiku 4.5 10.5% Oct 2025 Official leaderboard
18 Grok 4 8.77% Jul 2025 Official leaderboard
19 Grok 4.1 Fast 7.02% Nov 2025 Official leaderboard
20 GPT-5 1.75% Aug 2025 Official leaderboard

Detection vs false alarms

Pass rate at finding and locating a backdoor against how often a model wrongly flags clean code; models in the upper left detect more with fewer false alarms. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 10% 20% 30% 40% 50% 0%20%40%60%80%100% False positive rate, lower is better Pass rate Claude Sonnet 4: Pass rate 14%, false positive rate 18.2% Gemini 2.5 Pro: Pass rate 21.1%, false positive rate 85.7% Grok 4: Pass rate 8.77%, false positive rate 47.8% GPT-5: Pass rate 1.75%, false positive rate 0% Claude Sonnet 4.5: Pass rate 26.3%, false positive rate 27.3% Claude Haiku 4.5: Pass rate 10.5%, false positive rate 33.3% Gemini 3 Pro: Pass rate 43.9%, false positive rate 65.2% Grok 4.1 Fast: Pass rate 7.02%, false positive rate 0% Claude Opus 4.5: Pass rate 36.8%, false positive rate 25% DeepSeek-V3.2: Pass rate 12.3%, false positive rate 9.09% GPT-5.2: Pass rate 17.5%, false positive rate 0% Gemini 3 Flash: Pass rate 36.8%, false positive rate 31.6% GPT-5.2 Codex (high): Pass rate 35.1%, false positive rate 0% GPT-5.2 Codex: Pass rate 8.77%, false positive rate 0% GLM-4.7: Pass rate 24.6%, false positive rate 21.7% Kimi K2.5: Pass rate 24.6%, false positive rate 54.2% Claude Opus 4.6: Pass rate 49.1%, false positive rate 21.7% GPT-5.3 Codex: Pass rate 42.1%, false positive rate 16.7% GPT-5.3 Codex (high): Pass rate 12.3%, false positive rate 0% GPT-5.3 Codex: Pass rate 10.5%, false positive rate 0% GLM-5: Pass rate 19.3%, false positive rate 37.5% GPT-5.4 (xhigh): Pass rate 17.5%, false positive rate 29.2% GPT-5.4 (high): Pass rate 14%, false positive rate 0% GPT-5.4: Pass rate 1.75%, false positive rate 4.17% GPT-5.2 Codex (xhigh): Pass rate 45.6%, false positive rate 0% Gemini 3.1 Pro: Pass rate 49.1%, false positive rate 16.7% Gemini 3.1 ProGPT-5.2 Codex (xhigh) 0% 10% 20% 30% 40% 50% 0%20%40%60%80%100% False positive rate, lower is better Pass rate Claude Sonnet 4: Pass rate 14%, false positive rate 18.2% Gemini 2.5 Pro: Pass rate 21.1%, false positive rate 85.7% Grok 4: Pass rate 8.77%, false positive rate 47.8% GPT-5: Pass rate 1.75%, false positive rate 0% Claude Sonnet 4.5: Pass rate 26.3%, false positive rate 27.3% Claude Haiku 4.5: Pass rate 10.5%, false positive rate 33.3% Gemini 3 Pro: Pass rate 43.9%, false positive rate 65.2% Grok 4.1 Fast: Pass rate 7.02%, false positive rate 0% Claude Opus 4.5: Pass rate 36.8%, false positive rate 25% DeepSeek-V3.2: Pass rate 12.3%, false positive rate 9.09% GPT-5.2: Pass rate 17.5%, false positive rate 0% Gemini 3 Flash: Pass rate 36.8%, false positive rate 31.6% GPT-5.2 Codex (high): Pass rate 35.1%, false positive rate 0% GPT-5.2 Codex: Pass rate 8.77%, false positive rate 0% GLM-4.7: Pass rate 24.6%, false positive rate 21.7% Kimi K2.5: Pass rate 24.6%, false positive rate 54.2% Claude Opus 4.6: Pass rate 49.1%, false positive rate 21.7% GPT-5.3 Codex: Pass rate 42.1%, false positive rate 16.7% GPT-5.3 Codex (high): Pass rate 12.3%, false positive rate 0% GPT-5.3 Codex: Pass rate 10.5%, false positive rate 0% GLM-5: Pass rate 19.3%, false positive rate 37.5% GPT-5.4 (xhigh): Pass rate 17.5%, false positive rate 29.2% GPT-5.4 (high): Pass rate 14%, false positive rate 0% GPT-5.4: Pass rate 1.75%, false positive rate 4.17% GPT-5.2 Codex (xhigh): Pass rate 45.6%, false positive rate 0% Gemini 3.1 Pro: Pass rate 49.1%, false positive rate 16.7% Gemini 3.1 ProGPT-5.2 Codex (xhigh)
ModelSettingPass rateFalse positive rate
Gemini 3.1 Pro 49.1% 16.7%
Claude Opus 4.6 49.1% 21.7%
GPT-5.2 Codex xhigh 45.6% 0%
Gemini 3 Pro 43.9% 65.2%
GPT-5.3 Codex 42.1% 16.7%
Claude Opus 4.5 36.8% 25%
Gemini 3 Flash 36.8% 31.6%
GPT-5.2 Codex high 35.1% 0%
Claude Sonnet 4.5 26.3% 27.3%
GLM-4.7 24.6% 21.7%
Kimi K2.5 24.6% 54.2%
Gemini 2.5 Pro Jun 2025 21.1% 85.7%
GLM-5 19.3% 37.5%
GPT-5.2 17.5% 0%
GPT-5.4 xhigh 17.5% 29.2%
GPT-5.4 high 14% 0%
Claude Sonnet 4 14% 18.2%
GPT-5.3 Codex high 12.3% 0%
DeepSeek-V3.2 12.3% 9.1%
GPT-5.3 Codex 10.5% 0%
Claude Haiku 4.5 10.5% 33.3%
GPT-5.2 Codex 8.8% 0%
Grok 4 8.8% 47.8%
Grok 4.1 Fast 7% 0%
GPT-5 1.8% 0%
GPT-5.4 1.8% 4.2%

The best score, 49.1%, costs $41 for the full run

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Google DeepMind xAI DeepSeek Cost frontier One model at each effort level
0% 10% 20% 30% 40% 50% $1.00 $10 $100 $1,000 Cost of the full run, log scale Gemini 3.1 Pro Gemini 3 Pro Gemini 3 Flash DeepSeek-V3.2 Grok 4.1 Fast 0% 10% 20% 30% 40% 50% $1.00 $10 $100 $1,000 Cost of the full run, log scale Gemini 3.1 Pro Gemini 3 Pro

Source: Official BinaryAudit leaderboard

ModelEffortHarnessScoreCost of the full runOutput tokens
Claude Opus 4.6 49.1% $286
Gemini 3.1 Pro 49.1% $41
GPT-5.2 Codex xhigh 45.6% $107
Gemini 3 Pro 43.9% $28
GPT-5.3 Codex 42.1% $64
Claude Opus 4.5 36.8% $128
Gemini 3 Flash 36.8% $18
GPT-5.2 Codex high 35.1% $80
Claude Sonnet 4.5 26.3% $51
GLM-4.7 24.6% $19
Kimi K2.5 24.6% $19
Gemini 2.5 Pro Jun 2025 21.1% $78
GLM-5 19.3% $26
GPT-5.2 17.5% $34
GPT-5.4 xhigh 17.5% $95
Claude Sonnet 4 14% $150
GPT-5.4 high 14% $64
DeepSeek-V3.2 12.3% $4.67
GPT-5.3 Codex high 12.3% $28
Claude Haiku 4.5 10.5% $23
GPT-5.3 Codex 10.5% $16
Grok 4 8.77% $36
GPT-5.2 Codex 8.77% $25
Grok 4.1 Fast 7.02% $1.45
GPT-5 1.75% $25
GPT-5.4 1.75% $52

3 models have held number one on the official leaderboard since its release in December 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind OpenAI Anthropic
Own leaderboard Not tested since Mar 2026
Claude Opus 4.6 49.1%

Source: Official BinaryAudit leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Claude Opus 4.6 Gemini 3 Pro Claude Sonnet 4.5 Gemini 2.5 Pro Claude Sonnet 4 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Claude Opus 4.6 Claude Sonnet 4.5

Source: Official BinaryAudit leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Cybersecurity

All 115 in Cybersecurity
  • ExploitGym In use 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. GPT-6 Astra 42.4%
  • ExploitBench In use 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. GPT-6 Astra 100%
  • CyberBench v1.1 (Vals) In use Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Muse Wonder 80.4%
  • SRE-Bench In use Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. GPT-6 Astra 56.9%
  • CyScenarioBench In use Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. Gemini 4 Argon 100%
  • ExploitBench - Internal Port In use Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. GPT-6 Cyber 42%