Benchmark Almanac

SO-Arena

Not in model cards

Models choose answers across 11 domains and five capability groups. Evaluation tests accuracy, stability under changed option sets, and recognition that the correct answer is missing.

The evaluation compares decision models with generative models. Accuracy and option-set diagnostics must retain their individual labels.

Released
November 2026
Kind
Language reasoning
Level
University
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Measures
Capability
Language
English
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
Strongest model tested
GPT-6 Sol #13 on the capability index
Last model added
September 2026
Scores refreshed
November 20th, 2026

In the authors’ words

System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.

Abstract of SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models (arXiv:2611.15777, CC0)
22
Entries on its own leaderboard, 2 models we track

GPT-6 Sol leads with 82.3% GPT-6 Sol leads with 82.3%

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

  1. 1 GPT-6 Sol Reasoning: Medium effort 82.3%
  2. 2 GPT-6 Luna Reasoning: Medium effort 79.7%
  3. 3 GLiDE Reasoning: No 70.2%
  4. 4 KnowLine-31B Reasoning: No 70.2%
  5. 5 Sieve-31B-v3 Reasoning: No 67.7%
  6. 6 Decider-27B Reasoning: No 66.7%
  7. 7 Kev-27B Reasoning: No 61.4%
  8. 8 Gemini Decision 1 61.3%
  9. 9 Jev API 1.14 60.4%
  10. 10 StartLux Instant Reasoning: No 59%
  11. 11 StartLux Auto Reasoning: Auto 58.6%
  12. 12 JevAny-27B Reasoning: No 58.4%
  13. 13 Clef-27B Reasoning: No 56.9%
  14. 14 H2O Lightning Reasoning: No 54.9%
  15. 15 OpenAI Decisions 53.6%
  16. 16 Decider Flash Reasoning: No 52%
  17. 17 Clef-Flash-9B Reasoning: No 51.6%
  18. 18 Jeva-4B Reasoning: No 51.6%
  19. 19 JevAny-4B Reasoning: No 50.9%
  20. 20 SII-4B Reasoning: No 48%
  21. 21 Kev-4B Reasoning: No 46.1%
  22. 22 Laya Reasoning: No 30.9%

Source: Official SO-Arena leaderboard

Source: Official SO-Arena leaderboard

#EntryScoreSettingDateSource
1 GPT-6 Sol 82.3% Reasoning: Medium effort Sep 2026 Official leaderboard
2 GPT-6 Luna 79.7% Reasoning: Medium effort Sep 2026 Official leaderboard
3 GLiDE open weights 70.2% Reasoning: No Official leaderboard
4 KnowLine-31B open weights 70.2% Reasoning: No Official leaderboard
5 Sieve-31B-v3 open weights 67.7% Reasoning: No Official leaderboard
6 Decider-27B open weights 66.7% Reasoning: No Official leaderboard
7 Kev-27B open weights 61.4% Reasoning: No Official leaderboard
8 Gemini Decision 1 closed weights 61.3% Official leaderboard
9 Jev API 1.14 closed weights 60.4% Official leaderboard
10 StartLux Instant open weights 59% Reasoning: No Official leaderboard
11 StartLux Auto closed weights 58.6% Reasoning: Auto Official leaderboard
12 JevAny-27B open weights 58.4% Reasoning: No Official leaderboard
13 Clef-27B open weights 56.9% Reasoning: No Official leaderboard
14 H2O Lightning open weights 54.9% Reasoning: No Official leaderboard
15 OpenAI Decisions closed weights 53.6% Official leaderboard
16 Decider Flash open weights 52% Reasoning: No Official leaderboard
17 Clef-Flash-9B open weights 51.6% Reasoning: No Official leaderboard
18 Jeva-4B open weights 51.6% Reasoning: No Official leaderboard
19 JevAny-4B open weights 50.9% Reasoning: No Official leaderboard
20 SII-4B open weights 48% Reasoning: No Official leaderboard
21 Kev-4B open weights 46.1% Reasoning: No Official leaderboard
22 Laya open weights 30.9% Reasoning: No Official leaderboard
#ModelScoreSettingDateSource
1 GPT-6 Sol 82.3% Reasoning: Medium effort Sep 2026 Official leaderboard
2 GPT-6 Luna 79.7% Reasoning: Medium effort Sep 2026 Official leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0% 20% 40% 60% 80% 100% Near the ceiling (85%) GPT-6 Sol 0% 20% 40% 60% 80% 100% Near the ceiling (85%) GPT-6 Sol

Source: Official SO-Arena leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Knowledge

All 332 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. Seed1.8 92.3%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%

Data audit

All 10 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (22 entries).
  • Pass Models scored 2 models we track, plus 20 other entries on its leaderboard.
  • Pass Newest score The newest is from September 2026, for GPT-6 Luna.
  • Note Frontier models None of the top 10 has a score. The strongest tested is GPT-6 Sol (#13).
  • Pass Headroom The best is 82.3%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on lichenlillc.github.io: 2 rows for models we track and 20 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check