SO-Arena
Models choose answers across 11 domains and five capability groups. Evaluation tests accuracy, stability under changed option sets, and recognition that the correct answer is missing.
The evaluation compares decision models with generative models. Accuracy and option-set diagnostics must retain their individual labels.
- Made by
- National University of Singapore and collaborators
- Released
- November 2026
- Kind
- Language reasoning
- Level
- University
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
- GPT-6 Sol #13 on the capability index
- Last model added
- September 2026
- Scores refreshed
- November 20th, 2026
In the authors’ words
System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.
GPT-6 Sol leads with 82.3% GPT-6 Sol leads with 82.3%
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
- 82.3 GPT-6 Sol
- 79.7 GPT-6 Luna
- 70.2 GLiDE
- 70.2 KnowLine-31B
- 67.7 Sieve-31B-v3
- 66.7 Decider-27B
- 61.4 Kev-27B
- 61.3 Gemini Decision 1
- 60.4 Jev API 1.14
- 59.0 StartLux Instant
- 58.6 StartLux Auto
- 58.4 JevAny-27B
- 56.9 Clef-27B
- 54.9 H2O Lightning
- 53.6 OpenAI Decisions
- 52.0 Decider Flash
- 51.6 Clef-Flash-9B
- 51.6 Jeva-4B
- 50.9 JevAny-4B
- 48.0 SII-4B
- 46.1 Kev-4B
- 30.9 Laya
- 1 GPT-6 Sol Reasoning: Medium effort 82.3%
- 2 GPT-6 Luna Reasoning: Medium effort 79.7%
- 3 GLiDE Reasoning: No 70.2%
- 4 KnowLine-31B Reasoning: No 70.2%
- 5 Sieve-31B-v3 Reasoning: No 67.7%
- 6 Decider-27B Reasoning: No 66.7%
- 7 Kev-27B Reasoning: No 61.4%
- 8 Gemini Decision 1 61.3%
- 9 Jev API 1.14 60.4%
- 10 StartLux Instant Reasoning: No 59%
- 11 StartLux Auto Reasoning: Auto 58.6%
- 12 JevAny-27B Reasoning: No 58.4%
- 13 Clef-27B Reasoning: No 56.9%
- 14 H2O Lightning Reasoning: No 54.9%
- 15 OpenAI Decisions 53.6%
- 16 Decider Flash Reasoning: No 52%
- 17 Clef-Flash-9B Reasoning: No 51.6%
- 18 Jeva-4B Reasoning: No 51.6%
- 19 JevAny-4B Reasoning: No 50.9%
- 20 SII-4B Reasoning: No 48%
- 21 Kev-4B Reasoning: No 46.1%
- 22 Laya Reasoning: No 30.9%
Source: Official SO-Arena leaderboard
Source: Official SO-Arena leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-6 Sol | 82.3% | Reasoning: Medium effort | Sep 2026 | Official leaderboard |
| 2 | GPT-6 Luna | 79.7% | Reasoning: Medium effort | Sep 2026 | Official leaderboard |
| 3 | GLiDE open weights | 70.2% | Reasoning: No | Official leaderboard | |
| 4 | KnowLine-31B open weights | 70.2% | Reasoning: No | Official leaderboard | |
| 5 | Sieve-31B-v3 open weights | 67.7% | Reasoning: No | Official leaderboard | |
| 6 | Decider-27B open weights | 66.7% | Reasoning: No | Official leaderboard | |
| 7 | Kev-27B open weights | 61.4% | Reasoning: No | Official leaderboard | |
| 8 | Gemini Decision 1 closed weights | 61.3% | Official leaderboard | ||
| 9 | Jev API 1.14 closed weights | 60.4% | Official leaderboard | ||
| 10 | StartLux Instant open weights | 59% | Reasoning: No | Official leaderboard | |
| 11 | StartLux Auto closed weights | 58.6% | Reasoning: Auto | Official leaderboard | |
| 12 | JevAny-27B open weights | 58.4% | Reasoning: No | Official leaderboard | |
| 13 | Clef-27B open weights | 56.9% | Reasoning: No | Official leaderboard | |
| 14 | H2O Lightning open weights | 54.9% | Reasoning: No | Official leaderboard | |
| 15 | OpenAI Decisions closed weights | 53.6% | Official leaderboard | ||
| 16 | Decider Flash open weights | 52% | Reasoning: No | Official leaderboard | |
| 17 | Clef-Flash-9B open weights | 51.6% | Reasoning: No | Official leaderboard | |
| 18 | Jeva-4B open weights | 51.6% | Reasoning: No | Official leaderboard | |
| 19 | JevAny-4B open weights | 50.9% | Reasoning: No | Official leaderboard | |
| 20 | SII-4B open weights | 48% | Reasoning: No | Official leaderboard | |
| 21 | Kev-4B open weights | 46.1% | Reasoning: No | Official leaderboard | |
| 22 | Laya open weights | 30.9% | Reasoning: No | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-6 Sol | 82.3% | Reasoning: Medium effort | Sep 2026 | Official leaderboard |
| 2 | GPT-6 Luna | 79.7% | Reasoning: Medium effort | Sep 2026 | Official leaderboard |
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official SO-Arena leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
All 10 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
- Pass Independent runs Scores from the official leaderboard (22 entries).
- Pass Models scored 2 models we track, plus 20 other entries on its leaderboard.
- Pass Newest score The newest is from September 2026, for GPT-6 Luna.
- Note Frontier models None of the top 10 has a score. The strongest tested is GPT-6 Sol (#13).
- Pass Headroom The best is 82.3%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on lichenlillc.github.io: 2 rows for models we track and 20 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.