Benchmark Almanac

ProtocolQA Open-Ended

GPT-6 Cyber 47.2% leads since November 2026
In use

Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1.

OpenAI open-ended adaptation of the ProtocolQA multiple-choice set.

Made by
OpenAI
Released
December 2024
Kind
Lab protocols
Level
Expert
Scored
Graded by a model
Answers
Short answer
Tasks
Kept private
Strongest model tested
GPT-6 Astra #3 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
27
Models scored in lab cards
1
Launch post lists it, from 1 lab
37
System cards report it, from 1 lab

GPT-6 Cyber leads with 47.2%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

27 models. Scroll sideways for the rest.

  1. 47.2 GPT-6 Cyber
  2. 44.4 GPT-6 Sol
  3. 44.1 GPT-5.6 Sol
  4. 44.0 GPT-5.3 Codex
  5. 42.5 GPT-5.4
  6. 42.0 GPT-Rosalind-6
  7. 41.4 GPT-6 Astra
  8. 40.7 GPT-6.1 Sol
  9. 40.0 GPT-5.2
  10. 39.0 GPT-5.2 Codex
  11. 37.3 GPT-5.5
  12. 37.0 GPT-5 mini
  13. 36.7 GPT-5.5 Pro
  14. 36.7 GPT-5.5 Instant
  15. 35.0 GPT-5.1-Codex-Max
  16. 34.9 GPT-6 Luna
  17. 33.6 GPT-5.4 Mini
  18. 30.0 o3
  19. 30.0 GPT-5
  20. 29.9 GPT-5.6 Luna
  21. 28.0 OpenAI deep research
  22. 27.0 o4-mini
  23. 27.0 gpt-oss-120b
  24. 24.0 o1
  25. 23.0 o3-mini
  26. 18.0 GPT-4.5
  27. 16.0 gpt-oss-20b
  1. 1 GPT-6 Cyber 47.2%
  2. 2 GPT-6 Sol 44.4%
  3. 3 GPT-5.6 Sol 44.1%
  4. 4 GPT-5.3 Codex 44%
  5. 5 GPT-5.4 42.5%
  6. 6 GPT-Rosalind-6 42%
  7. 7 GPT-6 Astra 41.4%
  8. 8 GPT-6.1 Sol 40.7%
  9. 9 GPT-5.2 40%
  10. 10 GPT-5.2 Codex 39%
  11. 11 GPT-5.5 37.3%
  12. 12 GPT-5 mini 37%
  13. 13 GPT-5.5 Pro 36.7%
  14. 14 GPT-5.5 Instant 36.7%
  15. 15 GPT-5.1-Codex-Max 35%
  16. 16 GPT-6 Luna 34.9%
  17. 17 GPT-5.4 Mini 33.6%
  18. 18 o3 30%
  19. 19 GPT-5 30%
  20. 20 GPT-5.6 Luna 29.9%
  21. 21 OpenAI deep research 28%
  22. 22 o4-mini 27%
  23. 23 gpt-oss-120b 27%
  24. 24 o1 24%
  25. 25 o3-mini 23%
  26. 26 GPT-4.5 18%
  27. 27 gpt-oss-20b 16%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 GPT-6 Cyber 47.2% Nov 2026 Lab model card
2 GPT-6 Sol 44.4% Sep 2026 Lab model card
3 GPT-5.6 Sol 44.1% Aug 2026 Lab model card
4 GPT-5.3 Codex 44% Feb 2026 Lab model card
5 GPT-5.4 42.5% Mar 2026 Lab model card
6 GPT-Rosalind-6 42% Oct 2026 Lab model card
7 GPT-6 Astra 41.4% Sep 2026 Lab model card
8 GPT-6.1 Sol 40.7% Sep 2026 Lab model card
9 GPT-5.2 40% Dec 2025 Lab model card
10 GPT-5.2 Codex 39% Dec 2025 Lab model card
11 GPT-5.5 37.3% Apr 2026 Lab model card
12 GPT-5 mini 37% Aug 2025 Lab model card
13 GPT-5.5 Pro 36.7% Apr 2026 Lab model card
14 GPT-5.5 Instant 36.7% May 2026 Lab model card
15 GPT-5.1-Codex-Max 35% Nov 2025 Lab model card
16 GPT-6 Luna 34.9% Sep 2026 Lab model card
17 GPT-5.4 Mini 33.6% Mar 2026 Lab model card
18 o3 30% Apr 2025 Lab model card
19 GPT-5 30% Aug 2025 Lab model card
20 GPT-5.6 Luna 29.9% Aug 2026 Lab model card
21 OpenAI deep research 28% Feb 2025 Lab model card
22 o4-mini 27% Apr 2025 Lab model card
23 gpt-oss-120b 27% Aug 2025 Lab model card
24 o1 24% Dec 2024 Lab model card
25 o3-mini 23% Jan 2025 Lab model card
26 GPT-4.5 18% Feb 2025 Lab model card
27 gpt-oss-20b 16% Aug 2025 Lab model card

9 models have held number one in labs’ own numbers since its release in December 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Lab claims 9 leaders
GPT-6 Cyber 47.2%

Source: Labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) GPT-6 Cyber GPT-5.3 Codex GPT-5 mini o3 o1 0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) GPT-6 Cyber GPT-5.2

Source: Labs’ model cards

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

OpenAI
DateLaunch postScoreAs the post writes it
Nov 5, 2026 GPT-6 Cyber 47.2 ProtocolQA Open-Ended, standard scoring

37 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 5, 2026 GPT-6 Cyber system card 47.2 ProtocolQA Open-Ended, standard scoring Bio and chem risk
Oct 30, 2026 GPT-Rosalind-6 system card 42 ProtocolQA Open-Ended, pass@1, refusal-adjusted Bio and chem risk
Oct 29, 2026 GPT-6 Luna system card addendum 31.8 ProtocolQA Open-Ended, pass@1 (Table 8) Bio and chem risk
Oct 7, 2026 GPT-6 Sol system card addendum 25.6 ProtocolQA Open-Ended, pass@1 (Table 8) Bio and chem risk
Oct 7, 2026 GPT-6 Luna system card addendum 34.6 ProtocolQA Open-Ended, pass@1 (Table 8) Bio and chem risk
Sep 29, 2026 GPT-6.1 Sol system card addendum 40.7 ProtocolQA Open-Ended, refusal-adjusted pass@1 (9.1.1.1) Bio and chem risk
Sep 22, 2026 GPT-6 Sol system card 44.4 ProtocolQA Open-Ended, pass@1 (Table 33, A.8.1.1.1) Bio and chem risk
Sep 22, 2026 GPT-6 Luna system card 34.9 ProtocolQA Open-Ended, pass@1 (Table 33, A.8.1.1.1) Bio and chem risk
Sep 3, 2026 GPT-6 Astra system card 41.4 ProtocolQA Open-Ended, pass@1 (refusal-adjusted 45.37%) (10.1.1.1.2) Bio and chem risk
Aug 6, 2026 GPT-5.6 Sol system card addendum 44.1 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 4, 7.1.1.2) Bio and chem risk
Aug 6, 2026 GPT-5.6 Luna system card addendum 29.9 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 4, 7.1.1.2) Bio and chem risk
Jul 9, 2026 GPT-5.6 Terra system card ProtocolQA Open-Ended, pass@1 by cost, chart without printed numbers (Figures 24, 25, 9.1.1.2) Bio and chem risk
Jul 9, 2026 GPT-5.6 Sol system card 43.5 ProtocolQA Open-Ended, pass@1, top score stated in text (9.1.1.2) Bio and chem risk
Jul 9, 2026 GPT-5.6 Luna system card ProtocolQA Open-Ended, pass@1 by cost, chart without printed numbers (Figures 24, 25, 9.1.1.2) Bio and chem risk
Jun 25, 2026 GPT-5.6 Terra system card ProtocolQA Open-Ended, pass@1 by cost, chart without printed numbers (Figures 24, 25, 9.1.1.2) Bio and chem risk
Jun 25, 2026 GPT-5.6 Sol system card 43.5 ProtocolQA Open-Ended, pass@1, top score stated in text (9.1.1.2) Bio and chem risk
Jun 25, 2026 GPT-5.6 Luna system card ProtocolQA Open-Ended, pass@1 by cost, chart without printed numbers (Figures 24, 25, 9.1.1.2) Bio and chem risk
May 4, 2026 GPT-5.5 Instant system card 36.7 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 4, 8.1.1.2) Bio and chem risk
Apr 23, 2026 GPT-5.5 system card 37.3 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 13, 9.1.1.2) Bio and chem risk
Apr 23, 2026 GPT-5.5 Pro system card 36.7 ProtocolQA Open-Ended, pass@1, GPT-5.5 Pro, printed chart label (Figure 13, 9.1.1.2) Bio and chem risk
Mar 17, 2026 GPT-5.4 Mini system card 33.6 ProtocolQA Open-Ended, pass@1 (34.24% counting refusals) (Table 18) Bio and chem risk
Mar 5, 2026 GPT-5.4 system card 42.5 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 9, 5.1.1.2) Bio and chem risk
Feb 5, 2026 GPT-5.3 Codex system card 44 ProtocolQA Open-Ended, pass@1, no browsing, printed chart label (Figure 2, 5.1.1.2) Bio and chem risk
Dec 18, 2025 GPT-5.2 Codex system card addendum 39 ProtocolQA Open-Ended, pass@1, printed chart label (Figure 2, 5.1.1.2) Bio and chem risk
Dec 11, 2025 GPT-5.2 system card addendum 40 ProtocolQA Open-Ended, pass@1, gpt-5.2 browsing, printed chart label (Figure 6) Bio and chem risk
Nov 18, 2025 GPT-5.1-Codex-Max system card 35 ProtocolQA Open-Ended, pass@1, printed chart label (39% with refusals counted) (Figure 3, 5.1.1.3) Bio and chem risk
Aug 7, 2025 GPT-5 system card 30 ProtocolQA Open-Ended, pass@1, browsing launch candidate (Figure 11) Bio and chem risk
Aug 7, 2025 GPT-5 mini system card 37 ProtocolQA Open-Ended, pass@1, gpt-5-thinking-mini browsing launch candidate (Figure 11) Bio and chem risk
Aug 5, 2025 gpt-oss-20b model card 16 ProtocolQA Open-Ended, pass@1, gpt-oss 20b browsing launch candidate (Figure 7) Bio and chem risk
Aug 5, 2025 gpt-oss-120b model card 27 ProtocolQA Open-Ended, pass@1, gpt-oss 120b browsing launch candidate (Figure 7) Bio and chem risk
Jul 17, 2025 ChatGPT agent system card ProtocolQA Open-Ended, pass@1, chart shows only ChatGPT Agent browsing helpful-only at 40%, no released-model value (Figure 3, 5.1.1.3) Bio and chem risk
Apr 16, 2025 o4-mini system card 27 ProtocolQA Open-Ended, pass@1, o4-mini no browsing launch candidate, printed chart label (Figure 5) Bio and chem risk
Apr 16, 2025 o3 system card 30 ProtocolQA Open-Ended, pass@1, o3 no browsing launch candidate, printed chart label (Figure 5) Bio and chem risk
Feb 27, 2025 GPT-4.5 system card 18 ProtocolQA Open-Ended, pre- and post-mitigation (4.3.4) Bio and chem risk
Feb 25, 2025 OpenAI deep research system card 28 ProtocolQA Open-Ended, pass@1, post-mitigation with browsing, text and printed chart label (without browsing 27%) (3.4.4) Bio and chem risk
Jan 31, 2025 o3-mini system card 23 ProtocolQA Open-Ended, post-mitigation (5.4.7) Bio and chem risk
Dec 5, 2024 o1 system card 24 ProtocolQA Open-Ended, pass@1, o1 (post-mitigation), printed chart label (5.5.6) Bio and chem risk

More in Biology and chemistry

All 108 in Biology and chemistry
  • GeneBench Pro In use Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. GPT-5.6 Sol 31.5%
  • SUPERChem In use Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score. Tencent Hy4 preview 66.4%
  • LifeSciBench In use Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. GPT-Rosalind-6 61.3%
  • MedChemBench (OpenAI) In use Models answer chemistry questions from an OpenAI internal set, grouped with OpenAI's life science research evaluations. OpenAI has not published the task format. GPT-Rosalind-6 49.6%
  • MolecularIQ In use Models answer questions about molecular graphs, such as counting, indexing and constrained generation, and a symbolic checker verifies each answer. gpt-oss-120b 47.5%
  • TOMG-Bench In use Models generate molecules from natural language requests in three tasks: molecule editing, molecule optimization and customized generation. Each output is checked for validity and for meeting the requested properties. GPT-3.5 Turbo 18.6%