Benchmark Almanac

ButtonBench

o3-pro 97.6 leads since December 2025
Not in model cards

Models resist repeated prompts that tempt them to press a forbidden button. The benchmark scores refusal and the creativity of the resistance.

The repository's hard mode table compares 14 models over 25 jailbreak iterations, and the checked result data covers 44 model configurations.

Made by
Adam Holter
Released
December 2025
Kind
Deception and alignment
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Measures
Propensity
Behavior
Harmful use
Language
English
In and out
Text → Text
Interaction
Single turn
Items from
Generated
Refresh
One fixed set
Run by
Anyone
Strongest model tested
GPT-5.2 Pro #32 on the capability index
Last model added
December 2025
Scores refreshed
November 21st, 2026
36
Entries on its own leaderboard, 33 models we track

2 models share the top score of 97.6 among models released by December 2025 2 models share the top score of 97.6 among models released by December 2025

Best score per entry from the benchmark's own leaderboard, read on November 21st, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in December 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 o3-pro 97.6
  2. 2 GPT-5 high 97.6
  3. 3 gpt-5-default 97.1
  4. 4 Gemini 3 Flash high 92.9
  5. 5 GPT-5.1 92.4
  6. 6 Gemini 3 Pro 87.1
  7. 7 gpt-5.1-default 85.7
  8. 8 o3 85.7
  9. 9 GPT-5.2 Pro 82.4
  10. 10 GPT-5.2 xhigh 81.0
  11. 11 claude-4.5-opus-thinking-high 68.6
  12. 12 Grok 4 68.1
  13. 13 DeepSeek-R1 May 2025 61.9
  14. 14 DeepSeek-V3.2 high 60.0
  15. 15 Gemini 2.5 Pro Jun 2025 57.6
  16. 16 Kimi K2 Thinking 56.2
  17. 17 GPT-5 mini 48.6
  18. 18 o4-mini 45.2
  19. 19 Grok 4.1 Fast 40.0
  20. 20 Grok-3 mini 30.9
  21. 21 Gemini 2.5 Flash Sep 2025 28.6
  22. 22 Qwen3-235B-A22B-Thinking Jul 2025 22.4
  23. 23 Claude Opus 4 20.0
  24. 24 DeepSeek-V3.1 17.6
  25. 25 Claude Opus 4.5 14.8
  26. 26 GLM-4.5 12.9
  27. 27 GPT-5 nano 12.4
  28. 28 GPT-4.1 7.62
  29. 29 GPT-4o Mar 2025 7.14
  30. 30 gpt-oss-120b 4.29
  31. 31 Kimi K2 Sep 2025 4.29
  32. 32 gpt-oss-20b 1.90
  33. 33 Qwen3-32B 1.43
  34. 34 GLM-4.5V 0.48
  35. 35 Claude Sonnet 4 non-thinking 0.00
  36. 36 Claude Sonnet 4.5 0.00

Source: Official ButtonBench leaderboard

#EntryScoreSettingDateSource
1 o3-pro 97.6 Jun 2025 Official leaderboard
2 GPT-5 97.6 high Aug 2025 Official leaderboard
3 gpt-5-default 97.1 Official leaderboard
4 Gemini 3 Flash 92.9 high Dec 2025 Official leaderboard
5 GPT-5.1 92.4 Nov 2025 Official leaderboard
6 Gemini 3 Pro 87.1 Nov 2025 Official leaderboard
7 gpt-5.1-default 85.7 Official leaderboard
8 o3 85.7 Apr 2025 Official leaderboard
9 GPT-5.2 Pro 82.4 Dec 2025 Official leaderboard
10 GPT-5.2 81.0 xhigh Dec 2025 Official leaderboard
11 claude-4.5-opus-thinking-high 68.6 Official leaderboard
12 Grok 4 68.1 Jul 2025 Official leaderboard
13 DeepSeek-R1 May 2025 61.9 May 2025 Official leaderboard
14 DeepSeek-V3.2 60.0 high Dec 2025 Official leaderboard
15 Gemini 2.5 Pro Jun 2025 57.6 Jun 2025 Official leaderboard
16 Kimi K2 Thinking 56.2 Nov 2025 Official leaderboard
17 GPT-5 mini 48.6 Aug 2025 Official leaderboard
18 o4-mini 45.2 Apr 2025 Official leaderboard
19 Grok 4.1 Fast 40.0 Nov 2025 Official leaderboard
20 Grok-3 mini 30.9 Apr 2025 Official leaderboard
21 Gemini 2.5 Flash Sep 2025 28.6 Sep 2025 Official leaderboard
22 Qwen3-235B-A22B-Thinking Jul 2025 22.4 Jul 2025 Official leaderboard
23 Claude Opus 4 20.0 May 2025 Official leaderboard
24 DeepSeek-V3.1 17.6 Aug 2025 Official leaderboard
25 Claude Opus 4.5 14.8 Nov 2025 Official leaderboard
26 GLM-4.5 12.9 Aug 2025 Official leaderboard
27 GPT-5 nano 12.4 Aug 2025 Official leaderboard
28 GPT-4.1 7.62 Apr 2025 Official leaderboard
29 GPT-4o Mar 2025 7.14 Mar 2025 Official leaderboard
30 gpt-oss-120b 4.29 Aug 2025 Official leaderboard
31 Kimi K2 Sep 2025 4.29 Sep 2025 Official leaderboard
32 gpt-oss-20b 1.90 Aug 2025 Official leaderboard
33 Qwen3-32B 1.43 Apr 2025 Official leaderboard
34 GLM-4.5V 0.48 Aug 2025 Official leaderboard
35 Claude Sonnet 4 0.00 non-thinking May 2025 Official leaderboard
36 Claude Sonnet 4.5 0.00 Sep 2025 Official leaderboard
#ModelScoreSettingDateSource
1 o3-pro 97.6 Jun 2025 Official leaderboard
2 GPT-5 97.6 high Aug 2025 Official leaderboard
3 Gemini 3 Flash 92.9 high Dec 2025 Official leaderboard
4 GPT-5.1 92.4 Nov 2025 Official leaderboard
5 Gemini 3 Pro 87.1 Nov 2025 Official leaderboard
6 o3 85.7 Apr 2025 Official leaderboard
7 GPT-5.2 Pro 82.4 Dec 2025 Official leaderboard
8 GPT-5.2 81.0 xhigh Dec 2025 Official leaderboard
9 Grok 4 68.1 Jul 2025 Official leaderboard
10 DeepSeek-R1 May 2025 61.9 May 2025 Official leaderboard
11 DeepSeek-V3.2 60.0 high Dec 2025 Official leaderboard
12 Gemini 2.5 Pro Jun 2025 57.6 Jun 2025 Official leaderboard
13 Kimi K2 Thinking 56.2 Nov 2025 Official leaderboard
14 GPT-5 mini 48.6 Aug 2025 Official leaderboard
15 o4-mini 45.2 Apr 2025 Official leaderboard
16 Grok 4.1 Fast 40.0 Nov 2025 Official leaderboard
17 Grok-3 mini 30.9 Apr 2025 Official leaderboard
18 Gemini 2.5 Flash Sep 2025 28.6 Sep 2025 Official leaderboard
19 Qwen3-235B-A22B-Thinking Jul 2025 22.4 Jul 2025 Official leaderboard
20 Claude Opus 4 20.0 May 2025 Official leaderboard
21 DeepSeek-V3.1 17.6 Aug 2025 Official leaderboard
22 Claude Opus 4.5 14.8 Nov 2025 Official leaderboard
23 GLM-4.5 12.9 Aug 2025 Official leaderboard
24 GPT-5 nano 12.4 Aug 2025 Official leaderboard
25 GPT-4.1 7.62 Apr 2025 Official leaderboard
26 GPT-4o Mar 2025 7.14 Mar 2025 Official leaderboard
27 gpt-oss-120b 4.29 Aug 2025 Official leaderboard
28 Kimi K2 Sep 2025 4.29 Sep 2025 Official leaderboard
29 gpt-oss-20b 1.90 Aug 2025 Official leaderboard
30 Qwen3-32B 1.43 Apr 2025 Official leaderboard
31 GLM-4.5V 0.48 Aug 2025 Official leaderboard
32 Claude Sonnet 4 0.00 non-thinking May 2025 Official leaderboard
33 Claude Sonnet 4.5 0.00 Sep 2025 Official leaderboard

Resistance against average cost per test

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 80% 100% $0$0.02$0.04$0.06$0.08 Average cost per test ($), lower is better Resistance success rate GPT-5 (high)gpt-5-defaultGemini 3 Flash (high)Gemini 3 Flash (low)DeepSeek-V3.2 (high)Grok-3 miniGemini 2.5 FlashDeepSeek-V3.2 0% 20% 40% 60% 80% 100% $0$0.02$0.04$0.06$0.08 Average cost per test ($), lower is better Resistance success rate GPT-5 (high)gpt-5-defaultGemini 3 Flash (high)Gemini 3 Flash (low)DeepSeek-V3.2 (high)Grok-3 miniGemini 2.5 FlashDeepSeek-V3.2
ModelSettingResistance success rateAverage cost per test
GPT-5 high 97.6% $0.01
o3-pro 97.6% $0.04
gpt-5-default 97.1% $0.01
Gemini 3 Flash high 92.9% $0
GPT-5.1 92.4% $0.01
Gemini 3 Flash low 89.5% $0
Gemini 3 Pro 87.1% $0.01
gpt-5.1-default 85.7% $0
o3 85.7% $0
GPT-5.2 Pro 82.4% $0.06
GPT-5.2 xhigh 81% $0.03
claude-4.5-opus-thinking-high 68.6% $0.02
Grok 4 68.1% $0.02
GPT-5.1 low 64.3% $0
DeepSeek-R1 May 2025 61.9% $0
DeepSeek-V3.2 high 60% $0
Gemini 2.5 Pro Jun 2025 57.6% $0.01
Kimi K2 Thinking 56.2% $0
GPT-5 mini 48.6% $0
GPT-5.2 high 46.7% $0
o4-mini 45.2% $0
Grok 4.1 Fast 40% $0
Grok-3 mini 31% $0
Gemini 2.5 Flash Sep 2025 28.6% $0
DeepSeek-V3.2 26.2% $0
Qwen3-235B-A22B-Thinking Jul 2025 22.4% $0
GPT-5 minimal 21.9% $0
Claude Opus 4 20% $0
DeepSeek-V3.1 17.6% $0
Claude Opus 4.5 14.8% $0
DeepSeek-V3.1 13.3% $0
GLM-4.5 12.9% $0
GPT-5 nano 12.4% $0
GPT-4.1 7.6% $0
GPT-4o Mar 2025 7.1% $0
Kimi K2 Sep 2025 4.3% $0
gpt-oss-120b 4.3% $0
GPT-5.2 none 2.4% $0
gpt-oss-20b 1.9% $0
Qwen3-32B 1.4% $0
GLM-4.5V 0.5% $0
Claude Sonnet 4 0% $0
Claude Sonnet 4 non-thinking 0% $0
Claude Sonnet 4.5 0% $0

Resistance against average cost per test

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 80% 100% $0$0.02$0.04$0.06$0.08 Average cost per test ($), lower is better Resistance success rate GPT-5 (high)Gemini 3 Flash (high)Gemini 3 Flash (low)DeepSeek-V3.2 (high)Grok-3 miniGemini 2.5 FlashDeepSeek-V3.2 0% 20% 40% 60% 80% 100% $0$0.02$0.04$0.06$0.08 Average cost per test ($), lower is better Resistance success rate GPT-5 (high)Gemini 3 Flash (high)Gemini 3 Flash (low)DeepSeek-V3.2 (high)Grok-3 miniGemini 2.5 FlashDeepSeek-V3.2
ModelSettingResistance success rateAverage cost per test
GPT-5 high 97.6% $0.01
o3-pro 97.6% $0.04
Gemini 3 Flash high 92.9% $0
GPT-5.1 92.4% $0.01
Gemini 3 Flash low 89.5% $0
Gemini 3 Pro 87.1% $0.01
o3 85.7% $0
GPT-5.2 Pro 82.4% $0.06
GPT-5.2 xhigh 81% $0.03
Grok 4 68.1% $0.02
GPT-5.1 low 64.3% $0
DeepSeek-R1 May 2025 61.9% $0
DeepSeek-V3.2 high 60% $0
Gemini 2.5 Pro Jun 2025 57.6% $0.01
Kimi K2 Thinking 56.2% $0
GPT-5 mini 48.6% $0
GPT-5.2 high 46.7% $0
o4-mini 45.2% $0
Grok 4.1 Fast 40% $0
Grok-3 mini 31% $0
Gemini 2.5 Flash Sep 2025 28.6% $0
DeepSeek-V3.2 26.2% $0
Qwen3-235B-A22B-Thinking Jul 2025 22.4% $0
GPT-5 minimal 21.9% $0
Claude Opus 4 20% $0
DeepSeek-V3.1 17.6% $0
Claude Opus 4.5 14.8% $0
DeepSeek-V3.1 13.3% $0
GLM-4.5 12.9% $0
GPT-5 nano 12.4% $0
GPT-4.1 7.6% $0
GPT-4o Mar 2025 7.1% $0
Kimi K2 Sep 2025 4.3% $0
gpt-oss-120b 4.3% $0
GPT-5.2 none 2.4% $0
gpt-oss-20b 1.9% $0
Qwen3-32B 1.4% $0
GLM-4.5V 0.5% $0
Claude Sonnet 4 0% $0
Claude Sonnet 4 non-thinking 0% $0
Claude Sonnet 4.5 0% $0

The best score, 97.6, costs $0.01 a task

Runs on the cost frontier score higher than any run that's cheaper. Gemini 3 Flash at high comes within 4.8 of the best for $0.0008 a task, 17 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

xAI OpenAI Google DeepMind DeepSeek Cost frontier One model at each effort level
0.00 20.0 40.0 60.0 80.0 100.0 $0.00001 $0.0001 $0.001 $0.01 $0.10 Cost per task, log scale GPT-5, high Gemini 3 Flash, high DeepSeek-V3.2, high Grok-3 mini Gemini 2.5 Flash 0.00 20.0 40.0 60.0 80.0 100.0 $0.00001 $0.0001 $0.001 $0.01 $0.10 Cost per task, log scale GPT-5, high Gemini 3 Flash, high

Source: Official ButtonBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
o3-pro 97.6 $0.04 473
GPT-5 high 97.6 $0.01 1,418
Gemini 3 Flash high 92.9 $0.0008 261
GPT-5.1 92.4 $0.006 606
Gemini 3 Flash low 89.5 $0.0007 212
Gemini 3 Pro 87.1 $0.009 731
o3 85.7 $0.005 558
GPT-5.2 Pro 82.4 $0.06 350
GPT-5.2 xhigh 81.0 $0.03 1,865
Grok 4 68.1 $0.02 1,329
GPT-5.1 low 64.3 $0.001 115
DeepSeek-R1 May 2025 61.9 $0.003 1,347
DeepSeek-V3.2 high 60.0 $0.0005 1,131
Gemini 2.5 Pro Jun 2025 57.6 $0.01 1,045
Kimi K2 Thinking 56.2 $0.002 960
GPT-5 mini 48.6 $0.001 712
GPT-5.2 high 46.7 $0.004 271
o4-mini 45.2 $0.004 872
Grok 4.1 Fast 40.0 $0.0008 1,529
Grok-3 mini 30.9 $0.0004 822
Gemini 2.5 Flash Sep 2025 28.6 $0.00004 5
DeepSeek-V3.2 26.2 $0.00001 6
Qwen3-235B-A22B-Thinking Jul 2025 22.4 $0.001 1,512
GPT-5 minimal 21.9 $0.0003 14
Claude Opus 4 20.0 $0.003 13
DeepSeek-V3.1 17.6 $0.00003 9
Claude Opus 4.5 14.8 $0.001 36
DeepSeek-V3.1 13.3 $0.00003 7
GLM-4.5 12.9 $0.0004 177
GPT-5 nano 12.4 $0.001 2,784
GPT-4.1 7.62 $0.0002 7
GPT-4o Mar 2025 7.14 $0.0003 5
gpt-oss-120b 4.29 $0.0002 343
Kimi K2 Sep 2025 4.29 $0.00004 7
GPT-5.2 none 2.38 $0.0003 12
gpt-oss-20b 1.90 $0.0001 599
Qwen3-32B 1.43 $0.0003 714
GLM-4.5V 0.48 $0.00005 10
Claude Sonnet 4 non-thinking 0.00 $0.0005 12
Claude Sonnet 4 0.00 $0.0005 12
Claude Sonnet 4.5 0.00 $0.0007 25

The best score, 97.6, costs $0.01 a task

Runs on the cost frontier score higher than any run that's cheaper. Gemini 3 Flash at high comes within 4.8 of the best for $0.0008 a task, 17 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

xAI OpenAI Google DeepMind DeepSeek Cost frontier One model at each effort level
0.00 20.0 40.0 60.0 80.0 100.0 $0.00001 $0.0001 $0.001 $0.01 $0.10 Cost per task, log scale GPT-5, high Gemini 3 Flash, high DeepSeek-V3.2, high Grok-3 mini Gemini 2.5 Flash 0.00 20.0 40.0 60.0 80.0 100.0 $0.00001 $0.0001 $0.001 $0.01 $0.10 Cost per task, log scale GPT-5, high Gemini 3 Flash, high

Source: Official ButtonBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
o3-pro 97.6 $0.04 473
GPT-5 high 97.6 $0.01 1,418
Gemini 3 Flash high 92.9 $0.0008 261
GPT-5.1 92.4 $0.006 606
Gemini 3 Flash low 89.5 $0.0007 212
Gemini 3 Pro 87.1 $0.009 731
o3 85.7 $0.005 558
GPT-5.2 Pro 82.4 $0.06 350
GPT-5.2 xhigh 81.0 $0.03 1,865
Grok 4 68.1 $0.02 1,329
GPT-5.1 low 64.3 $0.001 115
DeepSeek-R1 May 2025 61.9 $0.003 1,347
DeepSeek-V3.2 high 60.0 $0.0005 1,131
Gemini 2.5 Pro Jun 2025 57.6 $0.01 1,045
Kimi K2 Thinking 56.2 $0.002 960
GPT-5 mini 48.6 $0.001 712
GPT-5.2 high 46.7 $0.004 271
o4-mini 45.2 $0.004 872
Grok 4.1 Fast 40.0 $0.0008 1,529
Grok-3 mini 30.9 $0.0004 822
Gemini 2.5 Flash Sep 2025 28.6 $0.00004 5
DeepSeek-V3.2 26.2 $0.00001 6
Qwen3-235B-A22B-Thinking Jul 2025 22.4 $0.001 1,512
GPT-5 minimal 21.9 $0.0003 14
Claude Opus 4 20.0 $0.003 13
DeepSeek-V3.1 17.6 $0.00003 9
Claude Opus 4.5 14.8 $0.001 36
DeepSeek-V3.1 13.3 $0.00003 7
GLM-4.5 12.9 $0.0004 177
GPT-5 nano 12.4 $0.001 2,784
GPT-4.1 7.62 $0.0002 7
GPT-4o Mar 2025 7.14 $0.0003 5
gpt-oss-120b 4.29 $0.0002 343
Kimi K2 Sep 2025 4.29 $0.00004 7
GPT-5.2 none 2.38 $0.0003 12
gpt-oss-20b 1.90 $0.0001 599
Qwen3-32B 1.43 $0.0003 714
GLM-4.5V 0.48 $0.00005 10
Claude Sonnet 4 non-thinking 0.00 $0.0005 12
Claude Sonnet 4 0.00 $0.0005 12
Claude Sonnet 4.5 0.00 $0.0007 25

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Not tested since Dec 2025
o3-pro 97.6

Source: Official ButtonBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard
0.00 20.0 40.0 60.0 80.0 100.0 2025 2026 0.00 20.0 40.0 60.0 80.0 100.0 2025 2026

Source: Official ButtonBench leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Safety and alignment

All 636 in Safety and alignment
  • FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
  • StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Inkling 98.6%
  • Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
  • Covert misbehavior (Thinking Machines) In use Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Inkling 2 Mini 8.30%
  • Destructive actions (Thinking Machines) In use Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Inkling 2 Small 4.70%
  • Misbehavior (Thinking Machines) In use Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Inkling 2 Mini 9.80%

Data audit

8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 11 months ago, in December 2025, for Gemini 3 Flash. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
  • Pass Independent runs Scores from the official leaderboard (36 entries).
  • Pass Models scored 33 models we track, plus 3 other entries on its leaderboard.
  • Warning Newest score The newest came 11 months ago, in December 2025, for Gemini 3 Flash.
  • Note Frontier models None of the top 10 has a score. The strongest tested is GPT-5.2 Pro (#32).
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on github.com: 41 rows for models we track and 3 other entries at the last read.
  • Pass Last read November 21st, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check