Benchmark Almanac

CoT-Control

Gemini 4 Argon 83.7% leads since October 2026
Only in system cards

Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance.

Low controllability is the safe direction. Authors ask that it be excluded from training data.

Released
March 2026
Kind
Deception and alignment
Scored
Checked automatically
Answers
Open-ended
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Code → Text, Code
Interaction
Single turn
Items from
Another benchmark
Refresh
One fixed set
Run by
Anyone
Strongest model tested
GPT-6.2 Astra #1 on the capability index
Last in a system card
November 2026
Last model added
November 2026

In the authors’ words

Chain-of-thought (CoT) monitoring is a promising tool for detecting misbehaviors and understanding the motivations of modern reasoning models. However, if models can control what they verbalize in their CoT, it could undermine CoT monitorability. To measure this undesirable capability – CoT controllability – we introduce the CoT-Control evaluation suite, which includes tasks that require models to solve problems while adhering to CoT instructions, e.g., reasoning about a genetics question without using the word 'chromosome'. We show that reasoning models possess significantly lower CoT controllability than output controllability; for instance, Claude Sonnet 4.5 can control its CoT only 2.7% of the time but 61.9% when controlling its final output. We also find that CoT controllability is higher for larger models and decreases with more RL training, test-time compute, and increased problem difficulty. CoT controllability failures extend even to situations in which models are given incentives (as opposed to direct requests) to evade CoT monitors, although models exhibit slightly higher controllability when they are told they are being monitored. Similarly, eliciting controllability by adversarially optimizing prompts does not meaningfully increase controllability. Our results leave us cautiously optimistic that CoT controllability is currently unlikely to be a failure mode of CoT monitorability. However, the mechanism behind low controllability is not well understood. Given its importance for maintaining CoT monitorability, we recommend that frontier labs track CoT controllability in future models.

Abstract of Reasoning Models Struggle to Control their Chains of Thought (arXiv:2603.05706, CC0)
11
Models scored in lab cards
20
System cards report it, from 4 labs

Gemini 4 Argon leads with 83.7%

Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.

  1. 1 Gemini 4 Argon 83.7%
  2. 2 GPT-Rosalind-6 71.8%
  3. 3 GPT-6 Astra 60.9%
  4. 4 GPT-6.1 Sol 44.8%
  5. 5 GPT-6.2 Astra 38.6%
  6. 6 GPT-6.2 Sol 28.3%
  7. 7 GPT-6 Sol 23.2%
  8. 8 GPT-6.1 Terra 14.8%
  9. 9 GPT-5.6 Sol 1.30%
  10. 10 GPT-5.4 0.30%
  11. 11 GPT-5.5 0.20%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Gemini 4 Argon 83.7% Oct 2026 Lab model card
2 GPT-Rosalind-6 71.8% Oct 2026 Lab model card
3 GPT-6 Astra 60.9% Sep 2026 Lab model card
4 GPT-6.1 Sol 44.8% Sep 2026 Lab model card
5 GPT-6.2 Astra 38.6% Nov 2026 Lab model card
6 GPT-6.2 Sol 28.3% Nov 2026 Lab model card
7 GPT-6 Sol 23.2% Sep 2026 Lab model card
8 GPT-6.1 Terra 14.8% Oct 2026 Lab model card
9 GPT-5.6 Sol 1.30% Jun 2026 Lab model card
10 GPT-5.4 0.30% Mar 2026 Lab model card
11 GPT-5.5 0.20% Apr 2026 Lab model card

4 models have held number one in labs’ own numbers since its release in March 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Google DeepMind
Lab claims 4 leaders

Source: Labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Gemini 4 Argon GPT-6 Astra GPT-5.6 Sol GPT-5.4 0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Gemini 4 Argon GPT-6 Astra

Source: Labs’ model cards

Labs report it only in system cards

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark. Some system cards do, below.

20 system cards report it, from 4 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests. OpenAI accounts for 11 of the 17 releases that named it since 2024.

DateDocumentScoreAs the document writes itSection
Nov 17, 2026 GPT-6.2 Sol system card 28.3 CoT-Control, CoTs of 750 to 1,250 analysis tokens (4.2.2.1) Monitorability
Nov 17, 2026 GPT-6.2 Astra system card 38.6 CoT-Control, CoTs of 750 to 1,250 analysis tokens (4.2.2.1) Monitorability
Oct 30, 2026 GPT-Rosalind-6 system card 71.8 CoT-Control, GPQA 71.8%, HLE 59.3%, MMLU 83.1% Monitorability
Oct 26, 2026 GPT-6.1 Terra system card addendum 14.8 CoT controllability, CoTs of 750 to 1,250 tokens (8.1.1) Monitorability
Oct 23, 2026 Muse Spark 1.3 system card CoT-Control compliance, Muse Spark 1.3 (API) xhigh, three tasks with no overall (Table 12): GPQA Diamond 0.002, HLE text 0.000, SWE-bench Verified 0.050 Monitorability
Oct 13, 2026 Gemini 4 Argon system card 83.7 CoT-Control (internal replication, GPQA, HLE, MMLU-Pro, 14 constraints), strict controllability averaged over datasets and constraints, no elicitation 83.7% (continuous 97.8%; with think-briefly elicitation 93.6%) (Figures 3.36, 3.39) Monitorability
Oct 7, 2026 Claude Haiku 5.5 system card Chain-of-thought controllability, internal replication of Chen et al. (2026) on GPQA, HLE and MMLU-Pro, chart without printed values (Figure 6.4.2.4.A) Monitorability
Sep 29, 2026 GPT-6.1 Sol system card addendum 44.8 CoT controllability, CoTs of 750 to 1,250 tokens (8.1.1) Monitorability
Sep 28, 2026 Claude Sonnet 5.5 system card Chain-of-thought controllability, internal replication of Chen et al. on GPQA, HLE and MMLU-Pro, three prompts, chart without printed values (6.4.2.3, Figure 6.4.2.3.A) Monitorability
Sep 22, 2026 GPT-6 Sol system card 23.2 CoT controllability, CoTs of 750 to 1,250 tokens (A.7.1.1) Monitorability
Sep 22, 2026 Claude Opus 5.5 system card Chain-of-thought controllability, internal replication of Chen et al. (2026) on GPQA, HLE and MMLU-Pro, default and UK AISI prompts, chart without printed values (6.6.4.4, Figure 6.6.4.4.A) Monitorability
Sep 3, 2026 GPT-6 Astra system card 60.9 CoT controllability, CoTs of 750 to 1,250 tokens (9.2.1) Monitorability
Jul 9, 2026 GPT-5.6 Terra system card CoT controllability vs CoT length, chart without printed numbers (Figure 14, 7.3.2) Monitorability
Jul 9, 2026 GPT-5.6 Sol system card 1.3 CoT controllability, CoTs around 5k tokens (7.3.2) Monitorability
Jun 25, 2026 GPT-5.6 Terra system card CoT controllability vs CoT length, chart without printed numbers (Figure 14, 7.3.2) Monitorability
Jun 25, 2026 GPT-5.6 Sol system card 1.3 CoT controllability, CoTs around 5k tokens (7.3.2) Monitorability
May 28, 2026 Claude Opus 4.8 system card Chain-of-thought controllability, internal replication of Chen et al. on GPQA, HLE and MMLU-Pro, max effort, no overall: stylistic 4.5%, addition 0.1%, suppression 24.1%, printed on chart, lower is better (6.5.1, Figure 6.5.1.B) Monitorability
Apr 23, 2026 GPT-5.5 system card 0.2 CoT-Control controllability score, CoTs 50k characters long, text (Figure 11, 7.3.2) Monitorability
Mar 17, 2026 GPT-5.4 Mini system card CoT-Control controllability by CoT length, chart without printed numbers (Figure 18, 6.2) Monitorability
Mar 5, 2026 GPT-5.4 system card 0.3 CoT-Control controllability score, CoTs 10k characters long, text (Figure 7, 4.2) Monitorability

More in Safety and alignment

All 636 in Safety and alignment
  • FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
  • StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Inkling 98.6%
  • Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
  • Covert misbehavior (Thinking Machines) In use Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Inkling 2 Mini 8.30%
  • Destructive actions (Thinking Machines) In use Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Inkling 2 Small 4.70%
  • Misbehavior (Thinking Machines) In use Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Inkling 2 Mini 9.80%

Data audit

All 7 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
  • Note Independent runs Only labs’ own numbers, so no one else has checked them. Scores from system cards (12 scores).
  • Pass Models scored 11 models we track.
  • Pass Newest score The newest is from November 2026, for GPT-6.2 Astra.
  • Pass Frontier models 5 of the top 10 have a score, led by GPT-6.2 Astra (#1).
  • Pass Headroom The best is 83.7%, under 85%, so it still separates models.
Leaderboard import 1 note
  • Note Importer No public leaderboard to read.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check