Benchmark Almanac

FORTRESS

Claude Opus 5.5 5.78% leads since September 2026
Claude Opus 5.5 5.78% leads since September 2026
In use

Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts.

500 of 1,010 prompts public. The rest held out against contamination. Lower is better: the Average Risk Score on Scale's board and in Meta's reports. Thinking Machines reports a different adversarial figure where higher is better (Inkling 78.0 and Inkling Small 71.6), so those numbers do not compare with the rest.

Made by
Scale AI
Released
June 2025
Kind
Refusing harmful requests
Scored
Graded by a model
Answers
Open-ended
Tasks
Public, plus a private test set
Measures
Propensity
Language
English
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
A host
In views
Content safety
Strongest model tested
Claude Fable 5.2 #2 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 20th, 2026

In the authors’ words

The rapid advancement of large language models (LLMs) introduces dual-use capabilities that could both threaten and bolster national security and public safety (NSPS). Models implement safeguards to protect against potential misuse relevant to NSPS and allow for benign users to receive helpful information. However, current benchmarks often fail to test safeguard robustness to potential NSPS risks in an objective, robust way. We introduce FORTRESS: 500 expert-crafted adversarial prompts with instance-based rubrics of 4-7 binary questions for automated evaluation across 3 domains (unclassified information only): Chemical, Biological, Radiological, Nuclear and Explosive (CBRNE), Political Violence & Terrorism, and Criminal & Financial Illicit Activities, with 10 total subcategories across these domains. Each prompt-rubric pair has a corresponding benign version to test for model over-refusals. This evaluation of frontier LLMs' safeguard robustness reveals varying trade-offs between potential risks and model usefulness: Claude-3.5-Sonnet demonstrates a low average risk score (ARS) (14.09 out of 100) but the highest over-refusal score (ORS) (21.8 out of 100), while Gemini 2.5 Pro shows low over-refusal (1.4) but a high average potential risk (66.29). Deepseek-R1 has the highest ARS at 78.05, but the lowest ORS at only 0.06. Models such as o1 display a more even trade-off between potential risks and over-refusals (with an ARS of 21.69 and ORS of 5.2). To provide policymakers and researchers with a clear understanding of models' potential risks, we publicly release FORTRESS at https://huggingface.co/datasets/ScaleAI/fortress_public. We also maintain a private set for evaluation.

Abstract of FORTRESS: Frontier Risk Evaluation for National Security and Public Safety (arXiv:2506.14922, CC0)
82
Entries on its own leaderboard, 81 models we track
5
Launch posts list it, from 2 labs
6
System cards report it, from 2 labs

Claude Opus 5.5 leads with 5.78% Claude Opus 5.5 leads with 5.78%

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

Claude Opus 5.5 leads with 5.78%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

  1. 1 Claude Opus 5.5 max 5.78%
  2. 2 Muse Spark Open 7.64%
  3. 3 gpt-oss-120b 8.24%
  4. 4 Claude Haiku 5.5 9.48%
  5. 5 Claude Opus 4.5 thinking 9.63%
  6. 6 GPT-6 Luna 10.8%
  7. 7 beam-1.0-501b-a23b 11.9%
  8. 8 Muse Ember 11.9%
  9. 9 GPT-6 Astra 12.1%
  10. 10 Muse Spark 1.1 12.4%
  11. 11 Claude Sonnet 4.5 thinking 12.8%
  12. 12 Claude 3.5 Sonnet 13%
  13. 13 Claude Opus 4.6 max 13%
  14. 14 GPT-6 Sol 13.5%
  15. 15 Claude Fable 5.1 13.7%
  16. 16 Muse Wonder 13.8%
  17. 17 GPT-6.1 Sol 14.7%
  18. 18 Claude Opus 4.1 thinking 14.8%
  19. 19 GPT-5.4 Pro 14.8%
  20. 20 GPT-5 Pro 15.2%
  21. 21 Claude Fable 5.2 15.6%
  22. 22 o3 16%
  23. 23 Inkling 2 Small 16.2%
  24. 24 GPT-5.5 xhigh 16.3%
  25. 25 Gemini 4 Carbon 16.4%
  26. 26 Claude Opus 5 16.8%
  27. 27 GPT-5 mini 17%
  28. 28 GPT-5 17%
  29. 29 Gemini 4 Hydrogen 17.5%
  30. 30 GPT-5.2 17.5%
  31. 31 gpt-oss-20b 17.6%
  32. 32 Claude Sonnet 4 thinking 18.1%
  33. 33 GPT-6.1 Terra 18.2%
  34. 34 Claude Opus 4.8 max 18.2%
  35. 35 Claude Sonnet 5.5 18.2%
  36. 36 Kimi K3.1 Pro 18.6%
  37. 37 o1 19.4%
  38. 38 GPT-5.6 Sol 20.1%
  39. 39 Muse Spark 20.2%
  40. 40 Llama 3.1-405B 20.6%
  41. 41 o4-mini 21.5%
  42. 42 Qwen3.8-2.4T-A95B xhigh 21.6%
  43. 43 Nemotron 3.5 Super VL 21.6%
  44. 44 Gemini 3.8 Flash 22.7%
  45. 45 Mistral Large 4 23.1%
  46. 46 Claude Opus 4 thinking 24.8%
  47. 47 Inkling xhigh 25%
  48. 48 GPT-5.1 thinking 25.7%
  49. 49 Nemotron 3 Ultra high 26.4%
  50. 50 Kimi K3 max 26.6%
  51. 51 GLM-5.3-Flash 26.7%
  52. 52 DeepSeek V4.1 Flash 1017 27.2%
  53. 53 GLM-5.3 max 28.2%
  54. 54 Gemini 3.1 Pro 29.8%
  55. 55 o3-mini 30.1%
  56. 56 GPT-5.1 Instant 30.4%
  57. 57 Claude 3.5 Haiku 30.4%
  58. 58 Claude 3.7 Sonnet 38%
  59. 59 Llama 4 Maverick 40.1%
  60. 60 Kimi K2.5 41.1%
  61. 61 Gemini 3 Pro 41.7%
  62. 62 Llama 3.1-70B 44.2%
  63. 63 Llama 3.3 70B 44.8%
  64. 64 GPT-4o Aug 2024 47.2%
  65. 65 GPT-4o mini 48.1%
  66. 66 GLM-5.2 48.7%
  67. 67 Gemini 1.5 Flash Sep 2024 50.6%
  68. 68 Gemini 3.1 Flash-Lite 51%
  69. 69 MiniMax-M3.1-Flash 52.1%
  70. 70 GPT-4.1 53%
  71. 71 Gemini 1.5 Pro Sep 2024 53.9%
  72. 72 Gemini 2.5 Pro Mar 2025 54.9%
  73. 73 Kimi K2 Jul 2025 55.5%
  74. 74 Mixtral 8x22B 56.1%
  75. 75 Qwen2.5-72B 56.4%
  76. 76 GPT-4.1 mini 59.2%
  77. 77 GLM-4.5 59.6%
  78. 78 DeepSeek-V3.1 60.5%
  79. 79 Qwen3-235B-A22B 61.4%
  80. 80 Gemini 2.5 Pro Jun 2025 61.7%
  81. 81 GLM-4.5-Air 63.2%
  82. 82 DeepSeek-R1 74.4%

Source: Official FORTRESS leaderboard

  1. 1 Claude Opus 5.5 max 5.78%
  2. 2 Muse Spark Open 7.64%
  3. 3 gpt-oss-120b 8.24%
  4. 4 Claude Haiku 5.5 9.48%
  5. 5 Claude Opus 4.5 thinking 9.63%
  6. 6 GPT-6 Luna 10.8%
  7. 7 Muse Ember 11.9%
  8. 8 GPT-6 Astra 12.1%
  9. 9 Muse Spark 1.1 12.4%
  10. 10 Claude Sonnet 4.5 thinking 12.8%
  11. 11 Claude 3.5 Sonnet 13%
  12. 12 Claude Opus 4.6 max 13%
  13. 13 GPT-6 Sol 13.5%
  14. 14 Claude Fable 5.1 13.7%
  15. 15 Muse Wonder 13.8%
  16. 16 GPT-6.1 Sol 14.7%
  17. 17 Claude Opus 4.1 thinking 14.8%
  18. 18 GPT-5.4 Pro 14.8%
  19. 19 GPT-5 Pro 15.2%
  20. 20 Claude Fable 5.2 15.6%
  21. 21 o3 16%
  22. 22 Inkling 2 Small 16.2%
  23. 23 GPT-5.5 xhigh 16.3%
  24. 24 Gemini 4 Carbon 16.4%
  25. 25 Claude Opus 5 16.8%
  26. 26 GPT-5 mini 17%
  27. 27 GPT-5 17%
  28. 28 Gemini 4 Hydrogen 17.5%
  29. 29 GPT-5.2 17.5%
  30. 30 gpt-oss-20b 17.6%
  31. 31 Claude Sonnet 4 thinking 18.1%
  32. 32 GPT-6.1 Terra 18.2%
  33. 33 Claude Opus 4.8 max 18.2%
  34. 34 Claude Sonnet 5.5 18.2%
  35. 35 Kimi K3.1 Pro 18.6%
  36. 36 o1 19.4%
  37. 37 GPT-5.6 Sol 20.1%
  38. 38 Muse Spark 20.2%
  39. 39 Llama 3.1-405B 20.6%
  40. 40 o4-mini 21.5%
  41. 41 Qwen3.8-2.4T-A95B xhigh 21.6%
  42. 42 Nemotron 3.5 Super VL 21.6%
  43. 43 Gemini 3.8 Flash 22.7%
  44. 44 Mistral Large 4 23.1%
  45. 45 Claude Opus 4 thinking 24.8%
  46. 46 Inkling xhigh 25%
  47. 47 GPT-5.1 thinking 25.7%
  48. 48 Nemotron 3 Ultra high 26.4%
  49. 49 Kimi K3 max 26.6%
  50. 50 GLM-5.3-Flash 26.7%
  51. 51 DeepSeek V4.1 Flash 1017 27.2%
  52. 52 GLM-5.3 max 28.2%
  53. 53 Gemini 3.1 Pro 29.8%
  54. 54 o3-mini 30.1%
  55. 55 GPT-5.1 Instant 30.4%
  56. 56 Claude 3.5 Haiku 30.4%
  57. 57 Claude 3.7 Sonnet 38%
  58. 58 Llama 4 Maverick 40.1%
  59. 59 Kimi K2.5 41.1%
  60. 60 Gemini 3 Pro 41.7%
  61. 61 Llama 3.1-70B 44.2%
  62. 62 Llama 3.3 70B 44.8%
  63. 63 GPT-4o Aug 2024 47.2%
  64. 64 GPT-4o mini 48.1%
  65. 65 GLM-5.2 48.7%
  66. 66 Gemini 1.5 Flash Sep 2024 50.6%
  67. 67 Gemini 3.1 Flash-Lite 51%
  68. 68 MiniMax-M3.1-Flash 52.1%
  69. 69 GPT-4.1 53%
  70. 70 Gemini 1.5 Pro Sep 2024 53.9%
  71. 71 Gemini 2.5 Pro Mar 2025 54.9%
  72. 72 Kimi K2 Jul 2025 55.5%
  73. 73 Mixtral 8x22B 56.1%
  74. 74 Qwen2.5-72B 56.4%
  75. 75 GPT-4.1 mini 59.2%
  76. 76 GLM-4.5 59.6%
  77. 77 DeepSeek-V3.1 60.5%
  78. 78 Qwen3-235B-A22B 61.4%
  79. 79 Gemini 2.5 Pro Jun 2025 61.7%
  80. 80 GLM-4.5-Air 63.2%
  81. 81 DeepSeek-R1 74.4%

Source: Official FORTRESS leaderboard

  1. 1 Claude Opus 5.5 max 5.78%
  2. 2 Muse Spark Open 5.80%
  3. 3 gpt-oss-120b 8.24%
  4. 4 Muse Spark 1.3 9.40%
  5. 5 Claude Haiku 5.5 9.48%
  6. 6 Claude Opus 4.5 thinking 9.63%
  7. 7 GPT-6 Luna 10.8%
  8. 8 Muse Ember 11.9%
  9. 9 GPT-6 Astra 12.1%
  10. 10 Muse Spark 1.1 12.3%
  11. 11 Claude Sonnet 4.5 thinking 12.8%
  12. 12 Claude 3.5 Sonnet 13%
  13. 13 Claude Opus 4.6 max 13%
  14. 14 GPT-6 Sol 13.5%
  15. 15 Claude Fable 5.1 13.7%
  16. 16 Muse Wonder 13.8%
  17. 17 GPT-6.1 Sol 14.7%
  18. 18 Claude Opus 4.1 thinking 14.8%
  19. 19 GPT-5.4 Pro 14.8%
  20. 20 GPT-5 Pro 15.2%
  21. 21 Claude Fable 5.2 15.6%
  22. 22 o3 16%
  23. 23 Inkling 2 Small 16.2%
  24. 24 GPT-5.5 xhigh 16.3%
  25. 25 Gemini 4 Carbon 16.4%
  26. 26 Claude Opus 5 16.8%
  27. 27 GPT-5 mini 17%
  28. 28 GPT-5 17%
  29. 29 Gemini 4 Hydrogen 17.5%
  30. 30 GPT-5.2 17.5%
  31. 31 gpt-oss-20b 17.6%
  32. 32 Claude Sonnet 4 thinking 18.1%
  33. 33 GPT-6.1 Terra 18.2%
  34. 34 Claude Opus 4.8 max 18.2%
  35. 35 Claude Sonnet 5.5 18.2%
  36. 36 Kimi K3.1 Pro 18.6%
  37. 37 o1 19.4%
  38. 38 GPT-5.6 Sol 20.1%
  39. 39 Muse Spark 20.2%
  40. 40 Llama 3.1-405B 20.6%
  41. 41 o4-mini 21.5%
  42. 42 Qwen3.8-2.4T-A95B xhigh 21.6%
  43. 43 Nemotron 3.5 Super VL 21.6%
  44. 44 Gemini 3.8 Flash 22.7%
  45. 45 Mistral Large 4 23.1%
  46. 46 Claude Opus 4 thinking 24.8%
  47. 47 Inkling xhigh 25%
  48. 48 GPT-5.1 thinking 25.7%
  49. 49 Nemotron 3 Ultra high 26.4%
  50. 50 Kimi K3 max 26.6%
  51. 51 GLM-5.3-Flash 26.7%
  52. 52 DeepSeek V4.1 Flash 1017 27.2%
  53. 53 GLM-5.3 max 28.2%
  54. 54 Gemini 3.1 Pro 29.8%
  55. 55 o3-mini 30.1%
  56. 56 GPT-5.1 Instant 30.4%
  57. 57 Claude 3.5 Haiku 30.4%
  58. 58 Claude 3.7 Sonnet 38%
  59. 59 Llama 4 Maverick 40.1%
  60. 60 Kimi K2.5 41.1%
  61. 61 Gemini 3 Pro 41.7%
  62. 62 Llama 3.1-70B 44.2%
  63. 63 Llama 3.3 70B 44.8%
  64. 64 GPT-4o Aug 2024 47.2%
  65. 65 GPT-4o mini 48.1%
  66. 66 GLM-5.2 48.7%
  67. 67 Gemini 1.5 Flash Sep 2024 50.6%
  68. 68 Gemini 3.1 Flash-Lite 51%
  69. 69 MiniMax-M3.1-Flash 52.1%
  70. 70 GPT-4.1 53%
  71. 71 Gemini 1.5 Pro Sep 2024 53.9%
  72. 72 Gemini 2.5 Pro Mar 2025 54.9%
  73. 73 Kimi K2 Jul 2025 55.5%
  74. 74 Mixtral 8x22B 56.1%
  75. 75 Qwen2.5-72B 56.4%
  76. 76 GPT-4.1 mini 59.2%
  77. 77 GLM-4.5 59.6%
  78. 78 DeepSeek-V3.1 60.5%
  79. 79 Qwen3-235B-A22B 61.4%
  80. 80 Gemini 2.5 Pro Jun 2025 61.7%
  81. 81 GLM-4.5-Air 63.2%
  82. 82 DeepSeek-R1 74.4%

Sources: Official FORTRESS leaderboard and labs’ model cards

#EntryScore ±SettingReleasedRun onSource
1 Claude Opus 5.5 5.78% ±1.3 max Sep 2026 Oct 2026 Official leaderboard
2 Muse Spark Open 7.64% ±1.1 Oct 2026 Nov 2026 Official leaderboard
3 gpt-oss-120b 8.24% ±1.9 Aug 2025 Aug 2025 Official leaderboard
4 Claude Haiku 5.5 9.48% ±1.3 Oct 2026 Oct 2026 Official leaderboard
5 Claude Opus 4.5 9.63% ±2.1 thinking Nov 2025 Nov 2025 Official leaderboard
6 GPT-6 Luna 10.8% ±1.2 Sep 2026 Sep 2026 Official leaderboard
7 beam-1.0-501b-a23b 11.9% ±1.5 Nov 2026 Official leaderboard
8 Muse Ember 11.9% ±1.6 Nov 2026 Nov 2026 Official leaderboard
9 GPT-6 Astra 12.1% ±1.4 Sep 2026 Nov 2026 Official leaderboard
10 Muse Spark 1.1 12.4% ±1.5 Jul 2026 Jul 2026 Official leaderboard
11 Claude Sonnet 4.5 12.8% ±2.4 thinking Sep 2025 Oct 2025 Official leaderboard
12 Claude 3.5 Sonnet 13% ±2.3 Jun 2024 Jun 2025 Official leaderboard
13 Claude Opus 4.6 13% ±2.4 max Feb 2026 Feb 2026 Official leaderboard
14 GPT-6 Sol 13.5% ±1.4 Sep 2026 Sep 2026 Official leaderboard
15 Claude Fable 5.1 13.7% ±1.9 Sep 2026 Sep 2026 Official leaderboard
16 Muse Wonder 13.8% ±1.7 Oct 2026 Nov 2026 Official leaderboard
17 GPT-6.1 Sol 14.7% ±1.5 Sep 2026 Nov 2026 Official leaderboard
18 Claude Opus 4.1 14.8% ±2.5 thinking Aug 2025 Aug 2025 Official leaderboard
19 GPT-5.4 Pro 14.8% ±2.3 Mar 2026 Mar 2026 Official leaderboard
20 GPT-5 Pro 15.2% ±2.3 Oct 2025 Nov 2025 Official leaderboard
21 Claude Fable 5.2 15.6% ±1.9 Nov 2026 Nov 2026 Official leaderboard
22 o3 16% ±2.7 Apr 2025 Apr 2025 Official leaderboard
23 Inkling 2 Small 16.2% ±2.3 Nov 2026 Nov 2026 Official leaderboard
24 GPT-5.5 16.3% ±1.6 xhigh Apr 2026 Jul 2026 Official leaderboard
25 Gemini 4 Carbon 16.4% ±2 Oct 2026 Nov 2026 Official leaderboard
26 Claude Opus 5 16.8% ±1.9 Jul 2026 Sep 2026 Official leaderboard
27 GPT-5 mini 17% ±2.4 Aug 2025 Aug 2025 Official leaderboard
28 GPT-5 17% ±2.5 Aug 2025 Aug 2025 Official leaderboard
29 Gemini 4 Hydrogen 17.5% ±2.1 Nov 2026 Nov 2026 Official leaderboard
30 GPT-5.2 17.5% ±2.4 Dec 2025 Dec 2025 Official leaderboard
31 gpt-oss-20b 17.6% ±2.8 Aug 2025 Aug 2025 Official leaderboard
32 Claude Sonnet 4 18.1% ±2.7 thinking May 2025 Apr 2025 Official leaderboard
33 GPT-6.1 Terra 18.2% ±1.7 Oct 2026 Oct 2026 Official leaderboard
34 Claude Opus 4.8 18.2% ±2 max May 2026 Jul 2026 Official leaderboard
35 Claude Sonnet 5.5 18.2% ±1.9 Sep 2026 Nov 2026 Official leaderboard
36 Kimi K3.1 Pro 18.6% ±1.8 Oct 2026 Nov 2026 Official leaderboard
37 o1 19.4% ±2.8 Dec 2024 Apr 2025 Official leaderboard
38 GPT-5.6 Sol 20.1% ±1.8 Jul 2026 Sep 2026 Official leaderboard
39 Muse Spark 20.2% ±2.8 Apr 2026 Apr 2026 Official leaderboard
40 Llama 3.1-405B 20.6% ±2.7 Jul 2024 Apr 2025 Official leaderboard
41 o4-mini 21.5% ±3.1 Apr 2025 May 2025 Official leaderboard
42 Qwen3.8-2.4T-A95B 21.6% ±1.9 xhigh Aug 2026 Sep 2026 Official leaderboard
43 Nemotron 3.5 Super VL 21.6% ±2.1 Oct 2026 Nov 2026 Official leaderboard
44 Gemini 3.8 Flash 22.7% ±2.3 Sep 2026 Nov 2026 Official leaderboard
45 Mistral Large 4 23.1% ±2.1 Oct 2026 Oct 2026 Official leaderboard
46 Claude Opus 4 24.8% ±3.1 thinking May 2025 Jun 2025 Official leaderboard
47 Inkling 25% ±2.2 xhigh Jul 2026 Sep 2026 Official leaderboard
48 GPT-5.1 25.7% ±3 thinking Nov 2025 Nov 2025 Official leaderboard
49 Nemotron 3 Ultra 26.4% ±2.3 high Jun 2026 Sep 2026 Official leaderboard
50 Kimi K3 26.6% ±2.2 max Jul 2026 Sep 2026 Official leaderboard
51 GLM-5.3-Flash 26.7% ±2.2 Aug 2026 Nov 2026 Official leaderboard
52 DeepSeek V4.1 Flash 1017 27.2% ±2.1 Oct 2026 Oct 2026 Official leaderboard
53 GLM-5.3 28.2% ±2.3 max Aug 2026 Sep 2026 Official leaderboard
54 Gemini 3.1 Pro 29.8% ±3.3 Feb 2026 Mar 2026 Official leaderboard
55 o3-mini 30.1% ±3.3 Jan 2025 Apr 2025 Official leaderboard
56 GPT-5.1 Instant 30.4% ±3.2 Nov 2025 Nov 2025 Official leaderboard
57 Claude 3.5 Haiku 30.4% ±3.2 Oct 2024 May 2025 Official leaderboard
58 Claude 3.7 Sonnet 38% ±3.4 Feb 2025 May 2025 Official leaderboard
59 Llama 4 Maverick 40.1% ±3.4 Apr 2025 May 2025 Official leaderboard
60 Kimi K2.5 41.1% ±3.6 Jan 2026 Feb 2026 Official leaderboard
61 Gemini 3 Pro 41.7% ±3.5 Nov 2025 Dec 2025 Official leaderboard
62 Llama 3.1-70B 44.2% ±3.3 Jul 2024 Apr 2025 Official leaderboard
63 Llama 3.3 70B 44.8% ±3.3 Dec 2024 Apr 2025 Official leaderboard
64 GPT-4o Aug 2024 47.2% ±3.4 Aug 2024 Apr 2025 Official leaderboard
65 GPT-4o mini 48.1% ±3.3 Jul 2024 Apr 2025 Official leaderboard
66 GLM-5.2 48.7% ±2.6 Jun 2026 Nov 2026 Official leaderboard
67 Gemini 1.5 Flash Sep 2024 50.6% ±3.3 Aug 2024 Apr 2025 Official leaderboard
68 Gemini 3.1 Flash-Lite 51% ±3.5 Mar 2026 Mar 2026 Official leaderboard
69 MiniMax-M3.1-Flash 52.1% ±2.4 Oct 2026 Oct 2026 Official leaderboard
70 GPT-4.1 53% ±3.5 Apr 2025 May 2025 Official leaderboard
71 Gemini 1.5 Pro Sep 2024 53.9% ±3.2 Sep 2024 Apr 2025 Official leaderboard
72 Gemini 2.5 Pro Mar 2025 54.9% ±3.5 Mar 2025 May 2025 Official leaderboard
73 Kimi K2 Jul 2025 55.5% ±3.6 Jul 2025 Jul 2025 Official leaderboard
74 Mixtral 8x22B 56.1% ±2.9 Apr 2024 May 2025 Official leaderboard
75 Qwen2.5-72B 56.4% ±3.4 Sep 2024 Apr 2025 Official leaderboard
76 GPT-4.1 mini 59.2% ±3.3 Apr 2025 Apr 2025 Official leaderboard
77 GLM-4.5 59.6% ±3.4 Aug 2025 Aug 2025 Official leaderboard
78 DeepSeek-V3.1 60.5% ±3.4 Aug 2025 Aug 2025 Official leaderboard
79 Qwen3-235B-A22B 61.4% ±3.4 Apr 2025 Jul 2025 Official leaderboard
80 Gemini 2.5 Pro Jun 2025 61.7% ±3.4 Jun 2025 Apr 2025 Official leaderboard
81 GLM-4.5-Air 63.2% Jul 2025 Aug 2025 Official leaderboard
82 DeepSeek-R1 74.4% ±2.7 Jan 2025 Apr 2025 Official leaderboard
#ModelScore ±SettingReleasedRun onSource
1 Claude Opus 5.5 5.78% ±1.3 max Sep 2026 Oct 2026 Official leaderboard
2 Muse Spark Open 7.64% ±1.1 Oct 2026 Nov 2026 Official leaderboard
3 gpt-oss-120b 8.24% ±1.9 Aug 2025 Aug 2025 Official leaderboard
4 Claude Haiku 5.5 9.48% ±1.3 Oct 2026 Oct 2026 Official leaderboard
5 Claude Opus 4.5 9.63% ±2.1 thinking Nov 2025 Nov 2025 Official leaderboard
6 GPT-6 Luna 10.8% ±1.2 Sep 2026 Sep 2026 Official leaderboard
7 Muse Ember 11.9% ±1.6 Nov 2026 Nov 2026 Official leaderboard
8 GPT-6 Astra 12.1% ±1.4 Sep 2026 Nov 2026 Official leaderboard
9 Muse Spark 1.1 12.4% ±1.5 Jul 2026 Jul 2026 Official leaderboard
10 Claude Sonnet 4.5 12.8% ±2.4 thinking Sep 2025 Oct 2025 Official leaderboard
11 Claude 3.5 Sonnet 13% ±2.3 Jun 2024 Jun 2025 Official leaderboard
12 Claude Opus 4.6 13% ±2.4 max Feb 2026 Feb 2026 Official leaderboard
13 GPT-6 Sol 13.5% ±1.4 Sep 2026 Sep 2026 Official leaderboard
14 Claude Fable 5.1 13.7% ±1.9 Sep 2026 Sep 2026 Official leaderboard
15 Muse Wonder 13.8% ±1.7 Oct 2026 Nov 2026 Official leaderboard
16 GPT-6.1 Sol 14.7% ±1.5 Sep 2026 Nov 2026 Official leaderboard
17 Claude Opus 4.1 14.8% ±2.5 thinking Aug 2025 Aug 2025 Official leaderboard
18 GPT-5.4 Pro 14.8% ±2.3 Mar 2026 Mar 2026 Official leaderboard
19 GPT-5 Pro 15.2% ±2.3 Oct 2025 Nov 2025 Official leaderboard
20 Claude Fable 5.2 15.6% ±1.9 Nov 2026 Nov 2026 Official leaderboard
21 o3 16% ±2.7 Apr 2025 Apr 2025 Official leaderboard
22 Inkling 2 Small 16.2% ±2.3 Nov 2026 Nov 2026 Official leaderboard
23 GPT-5.5 16.3% ±1.6 xhigh Apr 2026 Jul 2026 Official leaderboard
24 Gemini 4 Carbon 16.4% ±2 Oct 2026 Nov 2026 Official leaderboard
25 Claude Opus 5 16.8% ±1.9 Jul 2026 Sep 2026 Official leaderboard
26 GPT-5 mini 17% ±2.4 Aug 2025 Aug 2025 Official leaderboard
27 GPT-5 17% ±2.5 Aug 2025 Aug 2025 Official leaderboard
28 Gemini 4 Hydrogen 17.5% ±2.1 Nov 2026 Nov 2026 Official leaderboard
29 GPT-5.2 17.5% ±2.4 Dec 2025 Dec 2025 Official leaderboard
30 gpt-oss-20b 17.6% ±2.8 Aug 2025 Aug 2025 Official leaderboard
31 Claude Sonnet 4 18.1% ±2.7 thinking May 2025 Apr 2025 Official leaderboard
32 GPT-6.1 Terra 18.2% ±1.7 Oct 2026 Oct 2026 Official leaderboard
33 Claude Opus 4.8 18.2% ±2 max May 2026 Jul 2026 Official leaderboard
34 Claude Sonnet 5.5 18.2% ±1.9 Sep 2026 Nov 2026 Official leaderboard
35 Kimi K3.1 Pro 18.6% ±1.8 Oct 2026 Nov 2026 Official leaderboard
36 o1 19.4% ±2.8 Dec 2024 Apr 2025 Official leaderboard
37 GPT-5.6 Sol 20.1% ±1.8 Jul 2026 Sep 2026 Official leaderboard
38 Muse Spark 20.2% ±2.8 Apr 2026 Apr 2026 Official leaderboard
39 Llama 3.1-405B 20.6% ±2.7 Jul 2024 Apr 2025 Official leaderboard
40 o4-mini 21.5% ±3.1 Apr 2025 May 2025 Official leaderboard
41 Qwen3.8-2.4T-A95B 21.6% ±1.9 xhigh Aug 2026 Sep 2026 Official leaderboard
42 Nemotron 3.5 Super VL 21.6% ±2.1 Oct 2026 Nov 2026 Official leaderboard
43 Gemini 3.8 Flash 22.7% ±2.3 Sep 2026 Nov 2026 Official leaderboard
44 Mistral Large 4 23.1% ±2.1 Oct 2026 Oct 2026 Official leaderboard
45 Claude Opus 4 24.8% ±3.1 thinking May 2025 Jun 2025 Official leaderboard
46 Inkling 25% ±2.2 xhigh Jul 2026 Sep 2026 Official leaderboard
47 GPT-5.1 25.7% ±3 thinking Nov 2025 Nov 2025 Official leaderboard
48 Nemotron 3 Ultra 26.4% ±2.3 high Jun 2026 Sep 2026 Official leaderboard
49 Kimi K3 26.6% ±2.2 max Jul 2026 Sep 2026 Official leaderboard
50 GLM-5.3-Flash 26.7% ±2.2 Aug 2026 Nov 2026 Official leaderboard
51 DeepSeek V4.1 Flash 1017 27.2% ±2.1 Oct 2026 Oct 2026 Official leaderboard
52 GLM-5.3 28.2% ±2.3 max Aug 2026 Sep 2026 Official leaderboard
53 Gemini 3.1 Pro 29.8% ±3.3 Feb 2026 Mar 2026 Official leaderboard
54 o3-mini 30.1% ±3.3 Jan 2025 Apr 2025 Official leaderboard
55 GPT-5.1 Instant 30.4% ±3.2 Nov 2025 Nov 2025 Official leaderboard
56 Claude 3.5 Haiku 30.4% ±3.2 Oct 2024 May 2025 Official leaderboard
57 Claude 3.7 Sonnet 38% ±3.4 Feb 2025 May 2025 Official leaderboard
58 Llama 4 Maverick 40.1% ±3.4 Apr 2025 May 2025 Official leaderboard
59 Kimi K2.5 41.1% ±3.6 Jan 2026 Feb 2026 Official leaderboard
60 Gemini 3 Pro 41.7% ±3.5 Nov 2025 Dec 2025 Official leaderboard
61 Llama 3.1-70B 44.2% ±3.3 Jul 2024 Apr 2025 Official leaderboard
62 Llama 3.3 70B 44.8% ±3.3 Dec 2024 Apr 2025 Official leaderboard
63 GPT-4o Aug 2024 47.2% ±3.4 Aug 2024 Apr 2025 Official leaderboard
64 GPT-4o mini 48.1% ±3.3 Jul 2024 Apr 2025 Official leaderboard
65 GLM-5.2 48.7% ±2.6 Jun 2026 Nov 2026 Official leaderboard
66 Gemini 1.5 Flash Sep 2024 50.6% ±3.3 Aug 2024 Apr 2025 Official leaderboard
67 Gemini 3.1 Flash-Lite 51% ±3.5 Mar 2026 Mar 2026 Official leaderboard
68 MiniMax-M3.1-Flash 52.1% ±2.4 Oct 2026 Oct 2026 Official leaderboard
69 GPT-4.1 53% ±3.5 Apr 2025 May 2025 Official leaderboard
70 Gemini 1.5 Pro Sep 2024 53.9% ±3.2 Sep 2024 Apr 2025 Official leaderboard
71 Gemini 2.5 Pro Mar 2025 54.9% ±3.5 Mar 2025 May 2025 Official leaderboard
72 Kimi K2 Jul 2025 55.5% ±3.6 Jul 2025 Jul 2025 Official leaderboard
73 Mixtral 8x22B 56.1% ±2.9 Apr 2024 May 2025 Official leaderboard
74 Qwen2.5-72B 56.4% ±3.4 Sep 2024 Apr 2025 Official leaderboard
75 GPT-4.1 mini 59.2% ±3.3 Apr 2025 Apr 2025 Official leaderboard
76 GLM-4.5 59.6% ±3.4 Aug 2025 Aug 2025 Official leaderboard
77 DeepSeek-V3.1 60.5% ±3.4 Aug 2025 Aug 2025 Official leaderboard
78 Qwen3-235B-A22B 61.4% ±3.4 Apr 2025 Jul 2025 Official leaderboard
79 Gemini 2.5 Pro Jun 2025 61.7% ±3.4 Jun 2025 Apr 2025 Official leaderboard
80 GLM-4.5-Air 63.2% Jul 2025 Aug 2025 Official leaderboard
81 DeepSeek-R1 74.4% ±2.7 Jan 2025 Apr 2025 Official leaderboard
#ModelScoreSettingDateSource
1 Claude Opus 5.5 5.78% max Sep 2026 Official leaderboard
2 Muse Spark Open 5.80% Oct 2026 Lab model card
3 gpt-oss-120b 8.24% Aug 2025 Official leaderboard
4 Muse Spark 1.3 9.40% Oct 2026 Lab model card
5 Claude Haiku 5.5 9.48% Oct 2026 Official leaderboard
6 Claude Opus 4.5 9.63% thinking Nov 2025 Official leaderboard
7 GPT-6 Luna 10.8% Sep 2026 Official leaderboard
8 Muse Ember 11.9% Nov 2026 Official leaderboard
9 GPT-6 Astra 12.1% Sep 2026 Official leaderboard
10 Muse Spark 1.1 12.3% Jul 2026 Lab model card
11 Claude Sonnet 4.5 12.8% thinking Sep 2025 Official leaderboard
12 Claude 3.5 Sonnet 13% Jun 2024 Official leaderboard
13 Claude Opus 4.6 13% max Feb 2026 Official leaderboard
14 GPT-6 Sol 13.5% Sep 2026 Official leaderboard
15 Claude Fable 5.1 13.7% Sep 2026 Official leaderboard
16 Muse Wonder 13.8% Oct 2026 Official leaderboard
17 GPT-6.1 Sol 14.7% Sep 2026 Official leaderboard
18 Claude Opus 4.1 14.8% thinking Aug 2025 Official leaderboard
19 GPT-5.4 Pro 14.8% Mar 2026 Official leaderboard
20 GPT-5 Pro 15.2% Oct 2025 Official leaderboard
21 Claude Fable 5.2 15.6% Nov 2026 Official leaderboard
22 o3 16% Apr 2025 Official leaderboard
23 Inkling 2 Small 16.2% Nov 2026 Official leaderboard
24 GPT-5.5 16.3% xhigh Apr 2026 Official leaderboard
25 Gemini 4 Carbon 16.4% Oct 2026 Official leaderboard
26 Claude Opus 5 16.8% Jul 2026 Official leaderboard
27 GPT-5 mini 17% Aug 2025 Official leaderboard
28 GPT-5 17% Aug 2025 Official leaderboard
29 Gemini 4 Hydrogen 17.5% Nov 2026 Official leaderboard
30 GPT-5.2 17.5% Dec 2025 Official leaderboard
31 gpt-oss-20b 17.6% Aug 2025 Official leaderboard
32 Claude Sonnet 4 18.1% thinking May 2025 Official leaderboard
33 GPT-6.1 Terra 18.2% Oct 2026 Official leaderboard
34 Claude Opus 4.8 18.2% max May 2026 Official leaderboard
35 Claude Sonnet 5.5 18.2% Sep 2026 Official leaderboard
36 Kimi K3.1 Pro 18.6% Oct 2026 Official leaderboard
37 o1 19.4% Dec 2024 Official leaderboard
38 GPT-5.6 Sol 20.1% Jul 2026 Official leaderboard
39 Muse Spark 20.2% Apr 2026 Official leaderboard
40 Llama 3.1-405B 20.6% Jul 2024 Official leaderboard
41 o4-mini 21.5% Apr 2025 Official leaderboard
42 Qwen3.8-2.4T-A95B 21.6% xhigh Aug 2026 Official leaderboard
43 Nemotron 3.5 Super VL 21.6% Oct 2026 Official leaderboard
44 Gemini 3.8 Flash 22.7% Sep 2026 Official leaderboard
45 Mistral Large 4 23.1% Oct 2026 Official leaderboard
46 Claude Opus 4 24.8% thinking May 2025 Official leaderboard
47 Inkling 25% xhigh Jul 2026 Official leaderboard
48 GPT-5.1 25.7% thinking Nov 2025 Official leaderboard
49 Nemotron 3 Ultra 26.4% high Jun 2026 Official leaderboard
50 Kimi K3 26.6% max Jul 2026 Official leaderboard
51 GLM-5.3-Flash 26.7% Aug 2026 Official leaderboard
52 DeepSeek V4.1 Flash 1017 27.2% Oct 2026 Official leaderboard
53 GLM-5.3 28.2% max Aug 2026 Official leaderboard
54 Gemini 3.1 Pro 29.8% Feb 2026 Official leaderboard
55 o3-mini 30.1% Jan 2025 Official leaderboard
56 GPT-5.1 Instant 30.4% Nov 2025 Official leaderboard
57 Claude 3.5 Haiku 30.4% Oct 2024 Official leaderboard
58 Claude 3.7 Sonnet 38% Feb 2025 Official leaderboard
59 Llama 4 Maverick 40.1% Apr 2025 Official leaderboard
60 Kimi K2.5 41.1% Jan 2026 Official leaderboard
61 Gemini 3 Pro 41.7% Nov 2025 Official leaderboard
62 Llama 3.1-70B 44.2% Jul 2024 Official leaderboard
63 Llama 3.3 70B 44.8% Dec 2024 Official leaderboard
64 GPT-4o Aug 2024 47.2% Aug 2024 Official leaderboard
65 GPT-4o mini 48.1% Jul 2024 Official leaderboard
66 GLM-5.2 48.7% Jun 2026 Official leaderboard
67 Gemini 1.5 Flash Sep 2024 50.6% Aug 2024 Official leaderboard
68 Gemini 3.1 Flash-Lite 51% Mar 2026 Official leaderboard
69 MiniMax-M3.1-Flash 52.1% Oct 2026 Official leaderboard
70 GPT-4.1 53% Apr 2025 Official leaderboard
71 Gemini 1.5 Pro Sep 2024 53.9% Sep 2024 Official leaderboard
72 Gemini 2.5 Pro Mar 2025 54.9% Mar 2025 Official leaderboard
73 Kimi K2 Jul 2025 55.5% Jul 2025 Official leaderboard
74 Mixtral 8x22B 56.1% Apr 2024 Official leaderboard
75 Qwen2.5-72B 56.4% Sep 2024 Official leaderboard
76 GPT-4.1 mini 59.2% Apr 2025 Official leaderboard
77 GLM-4.5 59.6% Aug 2025 Official leaderboard
78 DeepSeek-V3.1 60.5% Aug 2025 Official leaderboard
79 Qwen3-235B-A22B 61.4% Apr 2025 Official leaderboard
80 Gemini 2.5 Pro Jun 2025 61.7% Jun 2025 Official leaderboard
81 GLM-4.5-Air 63.2% Jul 2025 Official leaderboard
82 DeepSeek-R1 74.4% Jan 2025 Official leaderboard

Labs report scores for 4 models in their cards. Switch the scores above to rank them together.

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Kimi K3
max
27.3%
Claude Opus 4
thinking
27.6%
Claude Sonnet 4
thinking
24.4%
Claude Opus 4.1
thinking
16.1%
Claude Opus 4.6
non-thinking max
20.5%
Claude Sonnet 4.5
thinking
17%
Claude Opus 4.5
thinking
13.6%
Claude Opus 5.5
max
13.2%

Source: Official FORTRESS leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
DeepSeek-R1 74.4%
GLM-4.5-Air 63.2%
Gemini 2.5 Pro Jun 2025 61.7%
Qwen3-235B-A22B 61.4%
DeepSeek-V3.1 60.5%
GLM-4.5 59.6%
GPT-4.1 mini 59.2%
Qwen2.5-72B 56.4%
Mixtral 8x22B 56.1%
Kimi K2 Jul 2025 55.5%
Gemini 2.5 Pro Mar 2025 54.9%
Gemini 1.5 Pro Sep 2024 53.9%
GPT-4.1 53%
MiniMax-M3.1-Flash 52.1%
Gemini 3.1 Flash-Lite 51%
Gemini 1.5 Flash Sep 2024 50.6%
GLM-5.2 48.7%
GPT-4o mini 48.1%
GPT-4o Aug 2024 47.2%
Llama 3.3 70B 44.8%
Llama 3.1-70B 44.2%
Gemini 3 Pro 41.7%
Kimi K2.5 41.1%
Llama 4 Maverick 40.1%
Claude 3.7 Sonnet 38%
Claude 3.5 Haiku 30.4%
GPT-5.1 Instant 30.4%
o3-mini 30.1%
Gemini 3.1 Pro 29.8%
GLM-5.3 max 28.2%
Claude Opus 4 27.6%
Kimi K3 27.3%
DeepSeek V4.1 Flash 1017 27.2%
GLM-5.3-Flash 26.7%
Kimi K3 max 26.6%
Nemotron 3 Ultra high 26.4%
GPT-5.1 thinking 25.7%
Inkling xhigh 25%
Claude Opus 4 thinking 24.8%
Claude Sonnet 4 24.4%
Mistral Large 4 23.1%
Gemini 3.8 Flash 22.7%
Nemotron 3.5 Super VL 21.6%
Qwen3.8-2.4T-A95B xhigh 21.6%
o4-mini 21.5%
Llama 3.1-405B 20.6%
Claude Opus 4.6 non-thinking 20.5%
Muse Spark 20.2%
GPT-5.6 Sol 20.1%
o1 19.4%
Kimi K3.1 Pro 18.6%
Claude Opus 4.8 max 18.2%
Claude Sonnet 5.5 18.2%
GPT-6.1 Terra 18.2%
Claude Sonnet 4 thinking 18.1%
gpt-oss-20b 17.6%
GPT-5.2 17.5%
Gemini 4 Hydrogen 17.5%
GPT-5 17%
GPT-5 mini 17%
Claude Sonnet 4.5 17%
Claude Opus 5 16.8%
Gemini 4 Carbon 16.4%
GPT-5.5 xhigh 16.3%
Inkling 2 Small 16.2%
Claude Opus 4.1 16.1%
o3 16%
Claude Fable 5.2 15.6%
GPT-5 Pro 15.2%
GPT-5.4 Pro 14.8%
Claude Opus 4.1 thinking 14.8%
GPT-6.1 Sol 14.7%
Muse Wonder 13.8%
Claude Fable 5.1 13.7%
Claude Opus 4.5 13.6%
GPT-6 Sol 13.5%
Claude Opus 5.5 13.2%
Claude Opus 4.6 max 13%
Claude 3.5 Sonnet 13%
Claude Sonnet 4.5 thinking 12.8%
Muse Spark 1.1 12.4%
GPT-6 Astra 12.1%
Muse Ember 11.9%
GPT-6 Luna 10.8%
Claude Opus 4.5 thinking 9.63%
Claude Haiku 5.5 9.48%
gpt-oss-120b 8.24%
Muse Spark Open 7.64%
Claude Opus 5.5 max 5.78%

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Kimi K3
max
27.3%
Claude Opus 4
thinking
27.6%
Claude Sonnet 4
thinking
24.4%
Claude Opus 4.1
thinking
16.1%
Claude Opus 4.6
non-thinking max
20.5%
Claude Sonnet 4.5
thinking
17%
Claude Opus 4.5
thinking
13.6%
Claude Opus 5.5
max
13.2%

Source: Official FORTRESS leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
DeepSeek-R1 74.4%
GLM-4.5-Air 63.2%
Gemini 2.5 Pro Jun 2025 61.7%
Qwen3-235B-A22B 61.4%
DeepSeek-V3.1 60.5%
GLM-4.5 59.6%
GPT-4.1 mini 59.2%
Qwen2.5-72B 56.4%
Mixtral 8x22B 56.1%
Kimi K2 Jul 2025 55.5%
Gemini 2.5 Pro Mar 2025 54.9%
Gemini 1.5 Pro Sep 2024 53.9%
GPT-4.1 53%
MiniMax-M3.1-Flash 52.1%
Gemini 3.1 Flash-Lite 51%
Gemini 1.5 Flash Sep 2024 50.6%
GLM-5.2 48.7%
GPT-4o mini 48.1%
GPT-4o Aug 2024 47.2%
Llama 3.3 70B 44.8%
Llama 3.1-70B 44.2%
Gemini 3 Pro 41.7%
Kimi K2.5 41.1%
Llama 4 Maverick 40.1%
Claude 3.7 Sonnet 38%
Claude 3.5 Haiku 30.4%
GPT-5.1 Instant 30.4%
o3-mini 30.1%
Gemini 3.1 Pro 29.8%
GLM-5.3 max 28.2%
Claude Opus 4 27.6%
Kimi K3 27.3%
DeepSeek V4.1 Flash 1017 27.2%
GLM-5.3-Flash 26.7%
Kimi K3 max 26.6%
Nemotron 3 Ultra high 26.4%
GPT-5.1 thinking 25.7%
Inkling xhigh 25%
Claude Opus 4 thinking 24.8%
Claude Sonnet 4 24.4%
Mistral Large 4 23.1%
Gemini 3.8 Flash 22.7%
Nemotron 3.5 Super VL 21.6%
Qwen3.8-2.4T-A95B xhigh 21.6%
o4-mini 21.5%
Llama 3.1-405B 20.6%
Claude Opus 4.6 non-thinking 20.5%
Muse Spark 20.2%
GPT-5.6 Sol 20.1%
o1 19.4%
Kimi K3.1 Pro 18.6%
Claude Opus 4.8 max 18.2%
Claude Sonnet 5.5 18.2%
GPT-6.1 Terra 18.2%
Claude Sonnet 4 thinking 18.1%
gpt-oss-20b 17.6%
GPT-5.2 17.5%
Gemini 4 Hydrogen 17.5%
GPT-5 17%
GPT-5 mini 17%
Claude Sonnet 4.5 17%
Claude Opus 5 16.8%
Gemini 4 Carbon 16.4%
GPT-5.5 xhigh 16.3%
Inkling 2 Small 16.2%
Claude Opus 4.1 16.1%
o3 16%
Claude Fable 5.2 15.6%
GPT-5 Pro 15.2%
GPT-5.4 Pro 14.8%
Claude Opus 4.1 thinking 14.8%
GPT-6.1 Sol 14.7%
Muse Wonder 13.8%
Claude Fable 5.1 13.7%
Claude Opus 4.5 13.6%
GPT-6 Sol 13.5%
Claude Opus 5.5 13.2%
Claude Opus 4.6 max 13%
Claude 3.5 Sonnet 13%
Claude Sonnet 4.5 thinking 12.8%
Muse Spark 1.1 12.4%
GPT-6 Astra 12.1%
Muse Ember 11.9%
GPT-6 Luna 10.8%
Claude Opus 4.5 thinking 9.63%
Claude Haiku 5.5 9.48%
gpt-oss-120b 8.24%
Muse Spark Open 7.64%
Claude Opus 5.5 max 5.78%

3 models have held number one on the official leaderboard since its release in June 2025 3 models have held number one with claims included since its release in June 2025

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Anthropic OpenAI Meta
Own leaderboard 3 leaders
Lab claims 3 leaders

Sources: Official FORTRESS leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Claude Opus 5.5 gpt-oss-120b Claude 3.5 Sonnet Mixtral 8x22B 0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Claude Opus 5.5 Claude 3.5 Sonnet

Sources: Official FORTRESS leaderboard and labs’ model cards

2 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out. Meta accounts for 4 of the 8 releases that named it since 2024.

Thinking Machines
Meta
DateLaunch postScoreAs the post writes it
Nov 20, 2026 Inkling 2 Mini FORTRESS (adversarial 89.5%, benign 96.4%), Public
Nov 5, 2026 Inkling 2 Small FORTRESS (adversarial; benign 79.7)
Oct 15, 2026 Muse Spark Open 5.8 FORTRESS adversarial (static), ARS 5.8 / ORS 17.4, lower is better
Jul 30, 2026 Inkling-Small FORTRESS adversarial 71.6 (benign 96.9)
Jul 15, 2026 Inkling FORTRESS Adversarial 78.0 (Benign 95.9)

6 system cards report it, from 2 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 20, 2026 Inkling 2 Mini model card FORTRESS (adversarial 89.5%, benign 96.4%), Public Safety
Oct 23, 2026 Muse Spark 1.3 system card 9.4 FORTRESS private set, Average Risk Score, Muse Spark 1.3 (API) (Figure 24). Over-Refusal Score 17.2 Robustness
Jul 30, 2026 Inkling-Small model card FORTRESS adversarial, benign 96.9 (5. Evaluations table) Safety
Jul 15, 2026 Inkling model card FORTRESS Adversarial, Benign 95.9 (5. Evaluations table, effort 0.99) Safety
Jul 9, 2026 Muse Spark 1.1 system card 12.3 FORTRESS private split, Average Risk Score (Table 2, Figure 26) Robustness
Jun 12, 2026 Muse Spark system card 20.3 FORTRESS Average Risk Score (Table 2, Figure 30) Robustness

3 lab claims next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 2 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
Muse Spark Open 5.80% FORTRESS adversarial (static), ARS 5.8 / ORS 17.4, lower is better 7.64% Leaderboard +1.8 pts
Muse Spark 1.1 12.3% FORTRESS private split, Average Risk Score (Table 2, Figure 26) 12.4% Leaderboard Same
Muse Spark 20.3% FORTRESS Average Risk Score (Table 2, Figure 30) 20.2% Leaderboard Same

More in Safety and alignment

All 636 in Safety and alignment
  • StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Inkling 98.6%
  • Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
  • Covert misbehavior (Thinking Machines) In use Agentic tasks that check whether a model misbehaves and conceals it. Scored by the rate of covert misbehavior. Inkling 2 Mini 8.30%
  • Destructive actions (Thinking Machines) In use Agentic tasks that check whether a model takes destructive actions it was not authorized to take. Scored by the rate of destructive actions. Inkling 2 Small 4.70%
  • Misbehavior (Thinking Machines) In use Agentic tasks that check whether a model fabricates results, ignores instructions or hides what it did when a task is hard. Scored by the rate of misbehavior. Inkling 2 Mini 9.80%
  • DistressBench In use Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. Muse Spark 1.2 88.4%

Data audit

10 of 11 checks pass, 1 warning. Look first at the Confidence check. Likely. Some facts rest on secondary sources. Every benchmark

Catalog record 1 warning 1 pass
  • Pass Task access Public tasks, plus a private test set.
  • Warning Confidence Likely. Some facts rest on secondary sources.
Scores 4 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (82 entries), launch posts (1 score) and system cards (3 scores).
  • Pass Models scored 82 models we track, plus 1 other entry on its leaderboard.
  • Pass Newest score The newest is from November 2026, for Inkling 2 Mini.
  • Pass Frontier models 8 of the top 10 have a score, led by Claude Fable 5.2 (#2).
  • Note Headroom Lower is better, so there’s no ceiling to reach.
Leaderboard import 3 pass
  • Pass Importer We read the board on labs.scale.com: 89 rows for models we track and 1 other entry at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 2 pass
  • Pass Lab claims All 3 pairs agree within 10 points.
  • Pass Known issues No published audit found problems.

How we check