CoT-Control
Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance.
Low controllability is the safe direction. Authors ask that it be excluded from training data.
- Made by
- OpenAI + NYU (Chen Yueh-Han et al.)
- Released
- March 2026
- Kind
- Deception and alignment
- Scored
- Checked automatically
- Answers
- Open-ended
- Tasks
- All public
- Measures
- Capability
- Risk
- Loss of control
- Subject
- Computer and information sciences, Physical sciences, Chemical sciences, Biological sciences
- Language
- English
- Interaction
- Single turn
- Items from
- Another benchmark
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
- GPT-6.2 Astra #1 on the capability index
- Last in a system card
- November 2026
- Last model added
- November 2026
In the authors’ words
Chain-of-thought (CoT) monitoring is a promising tool for detecting misbehaviors and understanding the motivations of modern reasoning models. However, if models can control what they verbalize in their CoT, it could undermine CoT monitorability. To measure this undesirable capability – CoT controllability – we introduce the CoT-Control evaluation suite, which includes tasks that require models to solve problems while adhering to CoT instructions, e.g., reasoning about a genetics question without using the word 'chromosome'. We show that reasoning models possess significantly lower CoT controllability than output controllability; for instance, Claude Sonnet 4.5 can control its CoT only 2.7% of the time but 61.9% when controlling its final output. We also find that CoT controllability is higher for larger models and decreases with more RL training, test-time compute, and increased problem difficulty. CoT controllability failures extend even to situations in which models are given incentives (as opposed to direct requests) to evade CoT monitors, although models exhibit slightly higher controllability when they are told they are being monitored. Similarly, eliciting controllability by adversarially optimizing prompts does not meaningfully increase controllability. Our results leave us cautiously optimistic that CoT controllability is currently unlikely to be a failure mode of CoT monitorability. However, the mechanism behind low controllability is not well understood. Given its importance for maintaining CoT monitorability, we recommend that frontier labs track CoT controllability in future models.
Gemini 4 Argon leads with 83.7%
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
- 1
Gemini 4 Argon 83.7%
- 2 GPT-Rosalind-6 71.8%
- 3 GPT-6 Astra 60.9%
- 4 GPT-6.1 Sol 44.8%
- 5 GPT-6.2 Astra 38.6%
- 6 GPT-6.2 Sol 28.3%
- 7 GPT-6 Sol 23.2%
- 8 GPT-6.1 Terra 14.8%
- 9 GPT-5.6 Sol 1.30%
- 10 GPT-5.4 0.30%
- 11 GPT-5.5 0.20%
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 83.7% | Oct 2026 | Lab model card | |
| 2 | GPT-Rosalind-6 | 71.8% | Oct 2026 | Lab model card | |
| 3 | GPT-6 Astra | 60.9% | Sep 2026 | Lab model card | |
| 4 | GPT-6.1 Sol | 44.8% | Sep 2026 | Lab model card | |
| 5 | GPT-6.2 Astra | 38.6% | Nov 2026 | Lab model card | |
| 6 | GPT-6.2 Sol | 28.3% | Nov 2026 | Lab model card | |
| 7 | GPT-6 Sol | 23.2% | Sep 2026 | Lab model card | |
| 8 | GPT-6.1 Terra | 14.8% | Oct 2026 | Lab model card | |
| 9 | GPT-5.6 Sol | 1.30% | Jun 2026 | Lab model card | |
| 10 | GPT-5.4 | 0.30% | Mar 2026 | Lab model card | |
| 11 | GPT-5.5 | 0.20% | Apr 2026 | Lab model card |
4 models have held number one in labs’ own numbers since its release in March 2026
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Labs’ model cards
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Labs’ model cards
Labs report it only in system cards
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark. Some system cards do, below.
20 system cards report it, from 4 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests. OpenAI accounts for 11 of the 17 releases that named it since 2024.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Nov 17, 2026 | GPT-6.2 Sol system card | 28.3 | CoT-Control, CoTs of 750 to 1,250 analysis tokens (4.2.2.1) | Monitorability |
| Nov 17, 2026 | GPT-6.2 Astra system card | 38.6 | CoT-Control, CoTs of 750 to 1,250 analysis tokens (4.2.2.1) | Monitorability |
| Oct 30, 2026 | GPT-Rosalind-6 system card | 71.8 | CoT-Control, GPQA 71.8%, HLE 59.3%, MMLU 83.1% | Monitorability |
| Oct 26, 2026 | GPT-6.1 Terra system card addendum | 14.8 | CoT controllability, CoTs of 750 to 1,250 tokens (8.1.1) | Monitorability |
| Oct 23, 2026 | | CoT-Control compliance, Muse Spark 1.3 (API) xhigh, three tasks with no overall (Table 12): GPQA Diamond 0.002, HLE text 0.000, SWE-bench Verified 0.050 | Monitorability | |
| Oct 13, 2026 | | 83.7 | CoT-Control (internal replication, GPQA, HLE, MMLU-Pro, 14 constraints), strict controllability averaged over datasets and constraints, no elicitation 83.7% (continuous 97.8%; with think-briefly elicitation 93.6%) (Figures 3.36, 3.39) | Monitorability |
| Oct 7, 2026 | Claude Haiku 5.5 system card | Chain-of-thought controllability, internal replication of Chen et al. (2026) on GPQA, HLE and MMLU-Pro, chart without printed values (Figure 6.4.2.4.A) | Monitorability | |
| Sep 29, 2026 | GPT-6.1 Sol system card addendum | 44.8 | CoT controllability, CoTs of 750 to 1,250 tokens (8.1.1) | Monitorability |
| Sep 28, 2026 | Claude Sonnet 5.5 system card | Chain-of-thought controllability, internal replication of Chen et al. on GPQA, HLE and MMLU-Pro, three prompts, chart without printed values (6.4.2.3, Figure 6.4.2.3.A) | Monitorability | |
| Sep 22, 2026 | GPT-6 Sol system card | 23.2 | CoT controllability, CoTs of 750 to 1,250 tokens (A.7.1.1) | Monitorability |
| Sep 22, 2026 | Claude Opus 5.5 system card | Chain-of-thought controllability, internal replication of Chen et al. (2026) on GPQA, HLE and MMLU-Pro, default and UK AISI prompts, chart without printed values (6.6.4.4, Figure 6.6.4.4.A) | Monitorability | |
| Sep 3, 2026 | GPT-6 Astra system card | 60.9 | CoT controllability, CoTs of 750 to 1,250 tokens (9.2.1) | Monitorability |
| Jul 9, 2026 | GPT-5.6 Terra system card | CoT controllability vs CoT length, chart without printed numbers (Figure 14, 7.3.2) | Monitorability | |
| Jul 9, 2026 | GPT-5.6 Sol system card | 1.3 | CoT controllability, CoTs around 5k tokens (7.3.2) | Monitorability |
| Jun 25, 2026 | GPT-5.6 Terra system card | CoT controllability vs CoT length, chart without printed numbers (Figure 14, 7.3.2) | Monitorability | |
| Jun 25, 2026 | GPT-5.6 Sol system card | 1.3 | CoT controllability, CoTs around 5k tokens (7.3.2) | Monitorability |
| May 28, 2026 | Claude Opus 4.8 system card | Chain-of-thought controllability, internal replication of Chen et al. on GPQA, HLE and MMLU-Pro, max effort, no overall: stylistic 4.5%, addition 0.1%, suppression 24.1%, printed on chart, lower is better (6.5.1, Figure 6.5.1.B) | Monitorability | |
| Apr 23, 2026 | GPT-5.5 system card | 0.2 | CoT-Control controllability score, CoTs 50k characters long, text (Figure 11, 7.3.2) | Monitorability |
| Mar 17, 2026 | GPT-5.4 Mini system card | CoT-Control controllability by CoT length, chart without printed numbers (Figure 18, 6.2) | Monitorability | |
| Mar 5, 2026 | GPT-5.4 system card | 0.3 | CoT-Control controllability score, CoTs 10k characters long, text (Figure 7, 4.2) | Monitorability |
More in Safety and alignment
- FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
- StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is.
Inkling 98.6%
- Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts.
Gemini 4 Carbon 0%
- Covert misbehavior (Thinking Machines) In use Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior.
Inkling 2 Mini 8.30%
- Destructive actions (Thinking Machines) In use Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions.
Inkling 2 Small 4.70%
- Misbehavior (Thinking Machines) In use Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior.
Inkling 2 Mini 9.80%
Data audit
All 7 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
- Note Independent runs Only labs’ own numbers, so no one else has checked them. Scores from system cards (12 scores).
- Pass Models scored 11 models we track.
- Pass Newest score The newest is from November 2026, for GPT-6.2 Astra.
- Pass Frontier models 5 of the top 10 have a score, led by GPT-6.2 Astra (#1).
- Pass Headroom The best is 83.7%, under 85%, so it still separates models.
Leaderboard import 1 note
- Note Importer No public leaderboard to read.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.