Open CoT Leaderboard
Models answer logical reasoning questions with and without writing out their reasoning.
The board reports the best accuracy gain over the direct-answer baseline across a common set of prompting methods.
- Made by
- Logikon and Allen Institute for AI
- Released
- January 2024
- Kind
- Language reasoning
- Level
- University
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
-
Qwen2.5-72B #166 on the capability index
- Last model added
- January 2025
- Scores refreshed
- November 20th, 2026
DeepSeek-R1-Distill-Qwen-14B leads with 17.6% among models released by January 2025 DeepSeek-R1-Distill-Qwen-14B leads with 17.6% among models released by January 2025
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
The newest model with a score here came out in January 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 17.6
DeepSeek-R1-Distill-Qwen-14B
- 16.9
DeepSeek-R1-Distill-Qwen-32B
- 16.3 meta-llama/Meta-Llama-3.1-8B-Instruct
- 16.1 internlm/internlm2-chat-20b
- 16.1 deepseek-ai/DeepSeek-R1-Distill-Llama-8B
- 15.3
DeepSeek-R1-Distill-Llama-70B
- 14.8 meta-llama/Meta-Llama-3-8B-Instruct
- 14.5 Kukedlc/NeuralLLaMa-3-8b-ORPO-v0.3
- 14.3 mlabonne/Daredevil-8B-abliterated
- 14.1 OpenBuddy/openbuddy-mistral2-7b-v20.2-32k
- 14.0 meta-llama/Meta-Llama-3-70B-Instruct
- 14.0 arcee-ai/Llama-3.1-SuperNova-Lite
- 13.9
Llama-3.1-Nemotron-70B-Instruct
- 13.8 01-ai/Yi-1.5-34B-Chat
- 13.4 meta-llama/Meta-Llama-3.1-70B-Instruct
- 13.3 Kukedlc/NeuralLLaMa-3-8b-DT-v0.1
- 13.1 ai21labs/AI21-Jamba-1.5-Mini
- 13.1
Phi-3.5-MoE
- 12.8
SOLAR-10.7B-Instruct-v1.0
- 12.8 mistralai/Ministral-8B-Instruct-2410
- 12.6 CohereForAI/c4ai-command-r-plus-08-2024
- 12.6
DeepSeek-R1-Distill-Qwen-7B
- 12.4 Qwen/Qwen2.5-7B-Instruct
- 12.3 NousResearch/Hermes-3-Llama-3.1-70B
- 12.3 Qwen/Qwen2.5-14B-Instruct
- 12.3 arcee-ai/SuperNova-Medius
- 12.3
Aya Expanse 32B
- 11.9 internlm/internlm2-chat-7b
- 11.8 Qwen/Qwen2.5-32B-Instruct
- 11.8 OpenBuddy/openbuddy-yi1.5-9b-v21.1-32k
- 11.5 vicgalle/Configurable-Yi-1.5-9B-Chat
- 11.4
phi-3-small 7.4B
- 11.3 databricks/dbrx-instruct
- 11.2
Aya Expanse 8B
- 11.2
phi-3-mini 3.8B
- 11.1 DebateLabKIT/Llama-3.1-Argunaut-1-8B-SPIN
- 10.9 mistralai/Mistral-Small-Instruct-2409
- 10.8 mistralai/Mixtral-8x7B-Instruct-v0.1
- 10.8 OpenBuddy/openbuddy-llama3-8b-v21.1-8k
- 10.8 wenbopan/Faro-Yi-9B-DPO
- 10.3 Qwen/Qwen2.5-3B-Instruct
- 10.2 mistralai/Mixtral-8x22B-Instruct-v0.1
- 9.93 vicgalle/Configurable-Hermes-2-Pro-Llama-3-8B
- 9.75
Yi-1.5-9B-Chat
- 9.69 mistralai/Mistral-Nemo-Instruct-2407
- 9.56 CohereForAI/aya-23-35B
- 9.56
Qwen2.5-72B
- 9.44 01-ai/Yi-1.5-9B-Chat-16K
- 9.33 unsloth/Phi-3.5-mini-instruct
- 9.33
Llama 3-70B
- 9.06 princeton-nlp/gemma-2-9b-it-SimPO
- 8.57 openchat/openchat-3.5-0106
- 8.42 meta-llama/Llama-3.2-3B-Instruct
- 8.34
Tulu 2 DPO 70B
- 8.32 google/gemma-2-9b-it
- 8.16
Gemma 2 2B IT
- 8.11 allenai/OLMo-7B-0724-Instruct-hf
- 8.08
Phi-2
- 8.02 01-ai/Yi-34B-Chat
- 7.86
Qwen2-7B-Instruct
- 7.68 bunnycore/Mnemosyne-7B
- 7.59 microsoft/Phi-3.5-mini-instruct
- 7.52 allenai/tulu-2-70b
- 7.46 ChavyvAkvar/habib-DPO-v3
- 7.43 Salesforce/LLaMA-3-8B-SFR-Iterative-DPO-R
- 7.41
Mistral-7B-Instruct-v0.2
- 7.33 meta-llama/Llama-2-70b-chat-hf
- 7.27 Locutusque/Hercules-4.0-Mistral-v0.2-7B
- 7.20 internlm/internlm2-7b
- 7.17 Deci/DeciLM-7B-instruct
- 7.00 Locutusque/OpenCerebrum-1.0-7b-DPO
- 6.91 google/gemma-7b-it
- 6.89 ichigoberry/pandafish-2-7b-32k
- 6.81
Qwen2-72B
- 6.74
DeepSeek-R1-Distill-Qwen-1.5B
- 6.41
Mixtral 8x7B
- 6.35 mistralai/Mistral-7B-Instruct-v0.3
- 5.98 vicgalle/Configurable-Janus-7B
- 5.93
phi-3-medium 14B
- 5.82 internlm/internlm2-math-7b
- 5.80 ibm/merlinite-7b
- 5.78 google/gemma-2-27b-it
- 5.73 LeroyDyer/Mixtral_AI_CyberTron_DeepMind_III_UFT
- 5.50 internlm/internlm2-math-20b
- 5.47 cognitivecomputations/Llama-3-8B-Instruct-abliterated-v2
- 5.32 01-ai/Yi-1.5-9B-32K
- 5.29 allenai/tulu-2-7b
- 5.25 CohereForAI/aya-23-8B
- 5.22 NousResearch/Hermes-2-Pro-Mistral-7B
- 5.03 allenai/tulu-2-13b
- 5.03 teknium/OpenHermes-2.5-Mistral-7B
- 4.93 openbmb/Eurus-70b-sft
- 4.91 allenai/tulu-2-dpo-13b
- 4.87 Intel/neural-chat-7b-v3-1
- 4.70 0-hero/Matter-0.2-7B-DPO
- 4.60 DebateLabKIT/Llama-3.1-Argunaut-1-8B-SFT
- 4.48 Deci/DeciLM-7B
- 4.48
SmolLM2 1.7B Instruct
- 4.45 LeroyDyer/Mixtral_AI_Chat_1.0
- 4.14 allenai/tulu-2-dpo-7b
- 4.04 BSC-LT/salamandra-7b-instruct
- 3.89 01-ai/Yi-1.5-9B
- 3.88
Mistral 7B v0.1
- 3.81 google/gemma-2b-it
- 3.65 meta-llama/Llama-2-13b-chat-hf
- 3.65
Yi-34B
- 3.53 DebateLabKIT/Llama-3.3-Argunaut-1-70B-SFT
- 3.49 microsoft/Orca-2-7b
- 3.37 DebateLabKIT/Llama-3.3-Argunaut-1-70B-SPIN
- 3.30 cookinai/LlamaReflect-8B-CoT-safetensors
- 3.13 microsoft/Orca-2-13b
- 3.07 meta-llama/Llama-3.2-1B-Instruct
- 2.87 NousResearch/Nous-Hermes-llama-2-7b
- 2.54 wandb/gemma-2b-zephyr-dpo
- 2.47 Kukedlc/NeuralSynthesis-7b-v0.4-slerp
- 2.46 meta-llama/Llama-2-13b-hf
- 2.45
Zephyr 7B Beta
- 2.26 mlabonne/AlphaMonarch-7B
- 2.12 upstage/SOLAR-10.7B-v1.0
- 2.01 openbmb/Eurus-7b-kto
- 1.90 LeroyDyer/Mixtral_AI_Cyber_Boss
- 1.87 HuggingFaceTB/SmolLM-1.7B-Instruct
- 1.53
Gemma 2B
- 1.42 meta-llama/Llama-2-7b-hf
- 1.39 HuggingFaceTB/SmolLM-135M-Instruct
- 1.21
Yi 6B
- 1.11 databricks/dolly-v2-3b
- 0.68
Gemma 7B
- -0.06 terrycraddock/Reflection-Llama-3.1-8B
- -0.23 OpenBuddy/openbuddy-zen-3b-v21.1-32k
- -0.33 OpenBuddy/openbuddy-zen-3b-v21.2-32k
- -1.33
Llama 2-70B
- -2.63
Llama 3-8B
- 1
DeepSeek-R1-Distill-Qwen-14B bfloat16 17.6%
- 2
DeepSeek-R1-Distill-Qwen-32B bfloat16 16.9%
- 3 meta-llama/Meta-Llama-3.1-8B-Instruct bfloat16 16.3%
- 4 internlm/internlm2-chat-20b bfloat16 16.1%
- 5 deepseek-ai/DeepSeek-R1-Distill-Llama-8B bfloat16 16.1%
- 6
DeepSeek-R1-Distill-Llama-70B bfloat16 15.3%
- 7 meta-llama/Meta-Llama-3-8B-Instruct bfloat16 14.8%
- 8 Kukedlc/NeuralLLaMa-3-8b-ORPO-v0.3 float16 14.5%
- 9 mlabonne/Daredevil-8B-abliterated bfloat16 14.3%
- 10 OpenBuddy/openbuddy-mistral2-7b-v20.2-32k bfloat16 14.1%
- 11 meta-llama/Meta-Llama-3-70B-Instruct bfloat16 14%
- 12 arcee-ai/Llama-3.1-SuperNova-Lite bfloat16 14%
- 13
Llama-3.1-Nemotron-70B-Instruct bfloat16 13.9%
- 14 01-ai/Yi-1.5-34B-Chat bfloat16 13.8%
- 15 meta-llama/Meta-Llama-3.1-70B-Instruct bfloat16 13.4%
- 16 Kukedlc/NeuralLLaMa-3-8b-DT-v0.1 float16 13.3%
- 17 ai21labs/AI21-Jamba-1.5-Mini bfloat16 13.1%
- 18
Phi-3.5-MoE bfloat16 13.1%
- 19
SOLAR-10.7B-Instruct-v1.0 float16 12.8%
- 20 mistralai/Ministral-8B-Instruct-2410 bfloat16 12.8%
- 21 CohereForAI/c4ai-command-r-plus-08-2024 bfloat16 12.6%
- 22
DeepSeek-R1-Distill-Qwen-7B bfloat16 12.6%
- 23 Qwen/Qwen2.5-7B-Instruct bfloat16 12.4%
- 24 NousResearch/Hermes-3-Llama-3.1-70B bfloat16 12.3%
- 25 Qwen/Qwen2.5-14B-Instruct bfloat16 12.3%
- 26 arcee-ai/SuperNova-Medius bfloat16 12.3%
- 27
Aya Expanse 32B float16 12.3%
- 28 internlm/internlm2-chat-7b bfloat16 11.9%
- 29 Qwen/Qwen2.5-32B-Instruct bfloat16 11.8%
- 30 OpenBuddy/openbuddy-yi1.5-9b-v21.1-32k bfloat16 11.8%
- 31 vicgalle/Configurable-Yi-1.5-9B-Chat float16 11.5%
- 32
phi-3-small 7.4B bfloat16 11.4%
- 33 databricks/dbrx-instruct bfloat16 11.3%
- 34
Aya Expanse 8B float16 11.2%
- 35
phi-3-mini 3.8B bfloat16 11.2%
- 36 DebateLabKIT/Llama-3.1-Argunaut-1-8B-SPIN bfloat16 11.1%
- 37 mistralai/Mistral-Small-Instruct-2409 bfloat16 10.9%
- 38 mistralai/Mixtral-8x7B-Instruct-v0.1 bfloat16 10.8%
- 39 OpenBuddy/openbuddy-llama3-8b-v21.1-8k bfloat16 10.8%
- 40 wenbopan/Faro-Yi-9B-DPO float16 10.8%
- 41 Qwen/Qwen2.5-3B-Instruct bfloat16 10.3%
- 42 mistralai/Mixtral-8x22B-Instruct-v0.1 bfloat16 10.2%
- 43 vicgalle/Configurable-Hermes-2-Pro-Llama-3-8B float16 9.93%
- 44
Yi-1.5-9B-Chat bfloat16 9.75%
- 45 mistralai/Mistral-Nemo-Instruct-2407 bfloat16 9.69%
- 46 CohereForAI/aya-23-35B float16 9.56%
- 47
Qwen2.5-72B bfloat16 9.56%
- 48 01-ai/Yi-1.5-9B-Chat-16K bfloat16 9.44%
- 49 unsloth/Phi-3.5-mini-instruct bfloat16 9.33%
- 50
Llama 3-70B bfloat16 9.33%
- 51 princeton-nlp/gemma-2-9b-it-SimPO bfloat16 9.06%
- 52 openchat/openchat-3.5-0106 bfloat16 8.57%
- 53 meta-llama/Llama-3.2-3B-Instruct bfloat16 8.42%
- 54
Tulu 2 DPO 70B bfloat16 8.34%
- 55 google/gemma-2-9b-it bfloat16 8.32%
- 56
Gemma 2 2B IT bfloat16 8.16%
- 57 allenai/OLMo-7B-0724-Instruct-hf bfloat16 8.11%
- 58
Phi-2 float16 8.08%
- 59 01-ai/Yi-34B-Chat bfloat16 8.02%
- 60
Qwen2-7B-Instruct bfloat16 7.86%
- 61 bunnycore/Mnemosyne-7B bfloat16 7.68%
- 62 microsoft/Phi-3.5-mini-instruct bfloat16 7.59%
- 63 allenai/tulu-2-70b bfloat16 7.52%
- 64 ChavyvAkvar/habib-DPO-v3 float16 7.46%
- 65 Salesforce/LLaMA-3-8B-SFR-Iterative-DPO-R bfloat16 7.43%
- 66
Mistral-7B-Instruct-v0.2 float16 7.41%
- 67 meta-llama/Llama-2-70b-chat-hf float16 7.33%
- 68 Locutusque/Hercules-4.0-Mistral-v0.2-7B bfloat16 7.27%
- 69 internlm/internlm2-7b bfloat16 7.20%
- 70 Deci/DeciLM-7B-instruct float16 7.17%
- 71 Locutusque/OpenCerebrum-1.0-7b-DPO bfloat16 7.00%
- 72 google/gemma-7b-it bfloat16 6.91%
- 73 ichigoberry/pandafish-2-7b-32k bfloat16 6.89%
- 74
Qwen2-72B bfloat16 6.81%
- 75
DeepSeek-R1-Distill-Qwen-1.5B bfloat16 6.74%
- 76
Mixtral 8x7B bfloat16 6.41%
- 77 mistralai/Mistral-7B-Instruct-v0.3 bfloat16 6.35%
- 78 vicgalle/Configurable-Janus-7B float16 5.98%
- 79
phi-3-medium 14B bfloat16 5.93%
- 80 internlm/internlm2-math-7b bfloat16 5.82%
- 81 ibm/merlinite-7b float16 5.80%
- 82 google/gemma-2-27b-it bfloat16 5.78%
- 83 LeroyDyer/Mixtral_AI_CyberTron_DeepMind_III_UFT float16 5.73%
- 84 internlm/internlm2-math-20b bfloat16 5.50%
- 85 cognitivecomputations/Llama-3-8B-Instruct-abliterated-v2 bfloat16 5.47%
- 86 01-ai/Yi-1.5-9B-32K bfloat16 5.32%
- 87 allenai/tulu-2-7b float16 5.29%
- 88 CohereForAI/aya-23-8B float16 5.25%
- 89 NousResearch/Hermes-2-Pro-Mistral-7B bfloat16 5.22%
- 90 allenai/tulu-2-13b bfloat16 5.03%
- 91 teknium/OpenHermes-2.5-Mistral-7B bfloat16 5.03%
- 92 openbmb/Eurus-70b-sft float16 4.93%
- 93 allenai/tulu-2-dpo-13b bfloat16 4.91%
- 94 Intel/neural-chat-7b-v3-1 float16 4.87%
- 95 0-hero/Matter-0.2-7B-DPO bfloat16 4.70%
- 96 DebateLabKIT/Llama-3.1-Argunaut-1-8B-SFT bfloat16 4.60%
- 97 Deci/DeciLM-7B float16 4.48%
- 98
SmolLM2 1.7B Instruct bfloat16 4.48%
- 99 LeroyDyer/Mixtral_AI_Chat_1.0 float16 4.45%
- 100 allenai/tulu-2-dpo-7b bfloat16 4.14%
- 101 BSC-LT/salamandra-7b-instruct bfloat16 4.04%
- 102 01-ai/Yi-1.5-9B bfloat16 3.89%
- 103
Mistral 7B v0.1 float16 3.88%
- 104 google/gemma-2b-it bfloat16 3.81%
- 105 meta-llama/Llama-2-13b-chat-hf float16 3.65%
- 106
Yi-34B bfloat16 3.65%
- 107 DebateLabKIT/Llama-3.3-Argunaut-1-70B-SFT bfloat16 3.53%
- 108 microsoft/Orca-2-7b float16 3.49%
- 109 DebateLabKIT/Llama-3.3-Argunaut-1-70B-SPIN bfloat16 3.37%
- 110 cookinai/LlamaReflect-8B-CoT-safetensors bfloat16 3.30%
- 111 microsoft/Orca-2-13b float16 3.13%
- 112 meta-llama/Llama-3.2-1B-Instruct bfloat16 3.07%
- 113 NousResearch/Nous-Hermes-llama-2-7b bfloat16 2.87%
- 114 wandb/gemma-2b-zephyr-dpo bfloat16 2.54%
- 115 Kukedlc/NeuralSynthesis-7b-v0.4-slerp bfloat16 2.47%
- 116 meta-llama/Llama-2-13b-hf float16 2.46%
- 117
Zephyr 7B Beta bfloat16 2.45%
- 118 mlabonne/AlphaMonarch-7B bfloat16 2.26%
- 119 upstage/SOLAR-10.7B-v1.0 float16 2.12%
- 120 openbmb/Eurus-7b-kto bfloat16 2.01%
- 121 LeroyDyer/Mixtral_AI_Cyber_Boss float16 1.90%
- 122 HuggingFaceTB/SmolLM-1.7B-Instruct bfloat16 1.87%
- 123
Gemma 2B bfloat16 1.53%
- 124 meta-llama/Llama-2-7b-hf float16 1.42%
- 125 HuggingFaceTB/SmolLM-135M-Instruct bfloat16 1.39%
- 126
Yi 6B float16 1.21%
- 127 databricks/dolly-v2-3b bfloat16 1.11%
- 128
Gemma 7B bfloat16 0.68%
- 129 terrycraddock/Reflection-Llama-3.1-8B float16 -0.06%
- 130 OpenBuddy/openbuddy-zen-3b-v21.1-32k bfloat16 -0.23%
- 131 OpenBuddy/openbuddy-zen-3b-v21.2-32k bfloat16 -0.33%
- 132
Llama 2-70B bfloat16 -1.33%
- 133
Llama 3-8B bfloat16 -2.63%
Source: Official Open CoT Leaderboard leaderboard
- 17.6
DeepSeek-R1-Distill-Qwen-14B
- 16.9
DeepSeek-R1-Distill-Qwen-32B
- 15.3
DeepSeek-R1-Distill-Llama-70B
- 13.9
Llama-3.1-Nemotron-70B-Instruct
- 13.1
Phi-3.5-MoE
- 12.8
SOLAR-10.7B-Instruct-v1.0
- 12.6
DeepSeek-R1-Distill-Qwen-7B
- 12.3
Aya Expanse 32B
- 11.4
phi-3-small 7.4B
- 11.2
Aya Expanse 8B
- 11.2
phi-3-mini 3.8B
- 9.75
Yi-1.5-9B-Chat
- 9.56
Qwen2.5-72B
- 9.33
Llama 3-70B
- 8.34
Tulu 2 DPO 70B
- 8.16
Gemma 2 2B IT
- 8.08
Phi-2
- 7.86
Qwen2-7B-Instruct
- 7.41
Mistral-7B-Instruct-v0.2
- 6.81
Qwen2-72B
- 6.74
DeepSeek-R1-Distill-Qwen-1.5B
- 6.41
Mixtral 8x7B
- 5.93
phi-3-medium 14B
- 4.48
SmolLM2 1.7B Instruct
- 3.88
Mistral 7B v0.1
- 3.65
Yi-34B
- 2.45
Zephyr 7B Beta
- 1.53
Gemma 2B
- 1.21
Yi 6B
- 0.68
Gemma 7B
- -1.33
Llama 2-70B
- -2.63
Llama 3-8B
- 1
DeepSeek-R1-Distill-Qwen-14B bfloat16 17.6%
- 2
DeepSeek-R1-Distill-Qwen-32B bfloat16 16.9%
- 3
DeepSeek-R1-Distill-Llama-70B bfloat16 15.3%
- 4
Llama-3.1-Nemotron-70B-Instruct bfloat16 13.9%
- 5
Phi-3.5-MoE bfloat16 13.1%
- 6
SOLAR-10.7B-Instruct-v1.0 float16 12.8%
- 7
DeepSeek-R1-Distill-Qwen-7B bfloat16 12.6%
- 8
Aya Expanse 32B float16 12.3%
- 9
phi-3-small 7.4B bfloat16 11.4%
- 10
Aya Expanse 8B float16 11.2%
- 11
phi-3-mini 3.8B bfloat16 11.2%
- 12
Yi-1.5-9B-Chat bfloat16 9.75%
- 13
Qwen2.5-72B bfloat16 9.56%
- 14
Llama 3-70B bfloat16 9.33%
- 15
Tulu 2 DPO 70B bfloat16 8.34%
- 16
Gemma 2 2B IT bfloat16 8.16%
- 17
Phi-2 float16 8.08%
- 18
Qwen2-7B-Instruct bfloat16 7.86%
- 19
Mistral-7B-Instruct-v0.2 float16 7.41%
- 20
Qwen2-72B bfloat16 6.81%
- 21
DeepSeek-R1-Distill-Qwen-1.5B bfloat16 6.74%
- 22
Mixtral 8x7B bfloat16 6.41%
- 23
phi-3-medium 14B bfloat16 5.93%
- 24
SmolLM2 1.7B Instruct bfloat16 4.48%
- 25
Mistral 7B v0.1 float16 3.88%
- 26
Yi-34B bfloat16 3.65%
- 27
Zephyr 7B Beta bfloat16 2.45%
- 28
Gemma 2B bfloat16 1.53%
- 29
Yi 6B float16 1.21%
- 30
Gemma 7B bfloat16 0.68%
- 31
Llama 2-70B bfloat16 -1.33%
- 32
Llama 3-8B bfloat16 -2.63%
Source: Official Open CoT Leaderboard leaderboard
| # | Entry | Score | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | | 17.6% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 2 | | 16.9% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 3 | meta-llama/Meta-Llama-3.1-8B-Instruct 8B | 16.3% | bfloat16 | huggingface.co | Official leaderboard | |
| 4 | internlm/internlm2-chat-20b 19.9B | 16.1% | bfloat16 | huggingface.co | Official leaderboard | |
| 5 | deepseek-ai/DeepSeek-R1-Distill-Llama-8B 8B | 16.1% | bfloat16 | huggingface.co | Official leaderboard | |
| 6 | | 15.3% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 7 | meta-llama/Meta-Llama-3-8B-Instruct 8B | 14.8% | bfloat16 | huggingface.co | Official leaderboard | |
| 8 | Kukedlc/NeuralLLaMa-3-8b-ORPO-v0.3 8B | 14.5% | float16 | huggingface.co | Official leaderboard | |
| 9 | mlabonne/Daredevil-8B-abliterated 8B | 14.3% | bfloat16 | huggingface.co | Official leaderboard | |
| 10 | OpenBuddy/openbuddy-mistral2-7b-v20.2-32k 7.3B | 14.1% | bfloat16 | huggingface.co | Official leaderboard | |
| 11 | meta-llama/Meta-Llama-3-70B-Instruct 70.6B | 14% | bfloat16 | huggingface.co | Official leaderboard | |
| 12 | arcee-ai/Llama-3.1-SuperNova-Lite 8B | 14% | bfloat16 | huggingface.co | Official leaderboard | |
| 13 | | 13.9% | bfloat16 | Oct 2024 | huggingface.co | Official leaderboard |
| 14 | 01-ai/Yi-1.5-34B-Chat 34.4B | 13.8% | bfloat16 | huggingface.co | Official leaderboard | |
| 15 | meta-llama/Meta-Llama-3.1-70B-Instruct 70.6B | 13.4% | bfloat16 | huggingface.co | Official leaderboard | |
| 16 | Kukedlc/NeuralLLaMa-3-8b-DT-v0.1 8B | 13.3% | float16 | huggingface.co | Official leaderboard | |
| 17 | ai21labs/AI21-Jamba-1.5-Mini 51.6B | 13.1% | bfloat16 | huggingface.co | Official leaderboard | |
| 18 | | 13.1% | bfloat16 | Aug 2024 | huggingface.co | Official leaderboard |
| 19 | | 12.8% | float16 | Dec 2023 | huggingface.co | Official leaderboard |
| 20 | mistralai/Ministral-8B-Instruct-2410 8B | 12.8% | bfloat16 | huggingface.co | Official leaderboard | |
| 21 | CohereForAI/c4ai-command-r-plus-08-2024 103.8B | 12.6% | bfloat16 | huggingface.co | Official leaderboard | |
| 22 | | 12.6% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 23 | Qwen/Qwen2.5-7B-Instruct 7.6B | 12.4% | bfloat16 | huggingface.co | Official leaderboard | |
| 24 | NousResearch/Hermes-3-Llama-3.1-70B 70.6B | 12.3% | bfloat16 | huggingface.co | Official leaderboard | |
| 25 | Qwen/Qwen2.5-14B-Instruct 14.8B | 12.3% | bfloat16 | huggingface.co | Official leaderboard | |
| 26 | arcee-ai/SuperNova-Medius 14.8B | 12.3% | bfloat16 | huggingface.co | Official leaderboard | |
| 27 | | 12.3% | float16 | Oct 2024 | huggingface.co | Official leaderboard |
| 28 | internlm/internlm2-chat-7b 7.7B | 11.9% | bfloat16 | huggingface.co | Official leaderboard | |
| 29 | Qwen/Qwen2.5-32B-Instruct 32.8B | 11.8% | bfloat16 | huggingface.co | Official leaderboard | |
| 30 | OpenBuddy/openbuddy-yi1.5-9b-v21.1-32k 8.9B | 11.8% | bfloat16 | huggingface.co | Official leaderboard | |
| 31 | vicgalle/Configurable-Yi-1.5-9B-Chat 8.8B | 11.5% | float16 | huggingface.co | Official leaderboard | |
| 32 | | 11.4% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
| 33 | databricks/dbrx-instruct 131.6B | 11.3% | bfloat16 | huggingface.co | Official leaderboard | |
| 34 | | 11.2% | float16 | Oct 2024 | huggingface.co | Official leaderboard |
| 35 | | 11.2% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
| 36 | DebateLabKIT/Llama-3.1-Argunaut-1-8B-SPIN 8B | 11.1% | bfloat16 | huggingface.co | Official leaderboard | |
| 37 | mistralai/Mistral-Small-Instruct-2409 22.3B | 10.9% | bfloat16 | huggingface.co | Official leaderboard | |
| 38 | mistralai/Mixtral-8x7B-Instruct-v0.1 46.7B | 10.8% | bfloat16 | huggingface.co | Official leaderboard | |
| 39 | OpenBuddy/openbuddy-llama3-8b-v21.1-8k 8B | 10.8% | bfloat16 | huggingface.co | Official leaderboard | |
| 40 | wenbopan/Faro-Yi-9B-DPO 8.8B | 10.8% | float16 | huggingface.co | Official leaderboard | |
| 41 | Qwen/Qwen2.5-3B-Instruct 3.1B | 10.3% | bfloat16 | huggingface.co | Official leaderboard | |
| 42 | mistralai/Mixtral-8x22B-Instruct-v0.1 140.6B | 10.2% | bfloat16 | huggingface.co | Official leaderboard | |
| 43 | vicgalle/Configurable-Hermes-2-Pro-Llama-3-8B 8B | 9.93% | float16 | huggingface.co | Official leaderboard | |
| 44 | | 9.75% | bfloat16 | May 2024 | huggingface.co | Official leaderboard |
| 45 | mistralai/Mistral-Nemo-Instruct-2407 12.3B | 9.69% | bfloat16 | huggingface.co | Official leaderboard | |
| 46 | CohereForAI/aya-23-35B 35B | 9.56% | float16 | huggingface.co | Official leaderboard | |
| 47 | | 9.56% | bfloat16 | Sep 2024 | huggingface.co | Official leaderboard |
| 48 | 01-ai/Yi-1.5-9B-Chat-16K 8.8B | 9.44% | bfloat16 | huggingface.co | Official leaderboard | |
| 49 | unsloth/Phi-3.5-mini-instruct 3.8B | 9.33% | bfloat16 | huggingface.co | Official leaderboard | |
| 50 | | 9.33% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
| 51 | princeton-nlp/gemma-2-9b-it-SimPO 9.2B | 9.06% | bfloat16 | huggingface.co | Official leaderboard | |
| 52 | openchat/openchat-3.5-0106 7.2B | 8.57% | bfloat16 | huggingface.co | Official leaderboard | |
| 53 | meta-llama/Llama-3.2-3B-Instruct 3.2B | 8.42% | bfloat16 | huggingface.co | Official leaderboard | |
| 54 | | 8.34% | bfloat16 | Nov 2023 | huggingface.co | Official leaderboard |
| 55 | google/gemma-2-9b-it 9.2B | 8.32% | bfloat16 | huggingface.co | Official leaderboard | |
| 56 | | 8.16% | bfloat16 | Jul 2024 | huggingface.co | Official leaderboard |
| 57 | allenai/OLMo-7B-0724-Instruct-hf 6.9B | 8.11% | bfloat16 | huggingface.co | Official leaderboard | |
| 58 | | 8.08% | float16 | Dec 2023 | huggingface.co | Official leaderboard |
| 59 | 01-ai/Yi-34B-Chat 34.4B | 8.02% | bfloat16 | huggingface.co | Official leaderboard | |
| 60 | | 7.86% | bfloat16 | Jun 2024 | huggingface.co | Official leaderboard |
| 61 | bunnycore/Mnemosyne-7B 7.2B | 7.68% | bfloat16 | huggingface.co | Official leaderboard | |
| 62 | microsoft/Phi-3.5-mini-instruct 3.8B | 7.59% | bfloat16 | huggingface.co | Official leaderboard | |
| 63 | allenai/tulu-2-70b 70B | 7.52% | bfloat16 | huggingface.co | Official leaderboard | |
| 64 | ChavyvAkvar/habib-DPO-v3 7.2B | 7.46% | float16 | huggingface.co | Official leaderboard | |
| 65 | Salesforce/LLaMA-3-8B-SFR-Iterative-DPO-R 8B | 7.43% | bfloat16 | huggingface.co | Official leaderboard | |
| 66 | | 7.41% | float16 | Dec 2023 | huggingface.co | Official leaderboard |
| 67 | meta-llama/Llama-2-70b-chat-hf 69B | 7.33% | float16 | huggingface.co | Official leaderboard | |
| 68 | Locutusque/Hercules-4.0-Mistral-v0.2-7B 7.2B | 7.27% | bfloat16 | huggingface.co | Official leaderboard | |
| 69 | internlm/internlm2-7b 7B | 7.20% | bfloat16 | huggingface.co | Official leaderboard | |
| 70 | Deci/DeciLM-7B-instruct 7B | 7.17% | float16 | huggingface.co | Official leaderboard | |
| 71 | Locutusque/OpenCerebrum-1.0-7b-DPO 7.2B | 7.00% | bfloat16 | huggingface.co | Official leaderboard | |
| 72 | google/gemma-7b-it 8.5B | 6.91% | bfloat16 | huggingface.co | Official leaderboard | |
| 73 | ichigoberry/pandafish-2-7b-32k 7.2B | 6.89% | bfloat16 | huggingface.co | Official leaderboard | |
| 74 | | 6.81% | bfloat16 | Jun 2024 | huggingface.co | Official leaderboard |
| 75 | | 6.74% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 76 | | 6.41% | bfloat16 | Dec 2023 | huggingface.co | Official leaderboard |
| 77 | mistralai/Mistral-7B-Instruct-v0.3 7.3B | 6.35% | bfloat16 | huggingface.co | Official leaderboard | |
| 78 | vicgalle/Configurable-Janus-7B 7.2B | 5.98% | float16 | huggingface.co | Official leaderboard | |
| 79 | | 5.93% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
| 80 | internlm/internlm2-math-7b 7.7B | 5.82% | bfloat16 | huggingface.co | Official leaderboard | |
| 81 | ibm/merlinite-7b 7.2B | 5.80% | float16 | huggingface.co | Official leaderboard | |
| 82 | google/gemma-2-27b-it 27.2B | 5.78% | bfloat16 | huggingface.co | Official leaderboard | |
| 83 | LeroyDyer/Mixtral_AI_CyberTron_DeepMind_III_UFT 7.2B | 5.73% | float16 | huggingface.co | Official leaderboard | |
| 84 | internlm/internlm2-math-20b 19.9B | 5.50% | bfloat16 | huggingface.co | Official leaderboard | |
| 85 | cognitivecomputations/Llama-3-8B-Instruct-abliterated-v2 8B | 5.47% | bfloat16 | huggingface.co | Official leaderboard | |
| 86 | 01-ai/Yi-1.5-9B-32K 8.8B | 5.32% | bfloat16 | huggingface.co | Official leaderboard | |
| 87 | allenai/tulu-2-7b 7B | 5.29% | float16 | huggingface.co | Official leaderboard | |
| 88 | CohereForAI/aya-23-8B 8B | 5.25% | float16 | huggingface.co | Official leaderboard | |
| 89 | NousResearch/Hermes-2-Pro-Mistral-7B 7.2B | 5.22% | bfloat16 | huggingface.co | Official leaderboard | |
| 90 | allenai/tulu-2-13b 13B | 5.03% | bfloat16 | huggingface.co | Official leaderboard | |
| 91 | teknium/OpenHermes-2.5-Mistral-7B 7.2B | 5.03% | bfloat16 | huggingface.co | Official leaderboard | |
| 92 | openbmb/Eurus-70b-sft 69B | 4.93% | float16 | huggingface.co | Official leaderboard | |
| 93 | allenai/tulu-2-dpo-13b 13B | 4.91% | bfloat16 | huggingface.co | Official leaderboard | |
| 94 | Intel/neural-chat-7b-v3-1 7.2B | 4.87% | float16 | huggingface.co | Official leaderboard | |
| 95 | 0-hero/Matter-0.2-7B-DPO 7.2B | 4.70% | bfloat16 | huggingface.co | Official leaderboard | |
| 96 | DebateLabKIT/Llama-3.1-Argunaut-1-8B-SFT 8B | 4.60% | bfloat16 | huggingface.co | Official leaderboard | |
| 97 | Deci/DeciLM-7B 7B | 4.48% | float16 | huggingface.co | Official leaderboard | |
| 98 | | 4.48% | bfloat16 | Oct 2024 | huggingface.co | Official leaderboard |
| 99 | LeroyDyer/Mixtral_AI_Chat_1.0 7.2B | 4.45% | float16 | huggingface.co | Official leaderboard | |
| 100 | allenai/tulu-2-dpo-7b 7B | 4.14% | bfloat16 | huggingface.co | Official leaderboard | |
| 101 | BSC-LT/salamandra-7b-instruct 7.8B | 4.04% | bfloat16 | huggingface.co | Official leaderboard | |
| 102 | 01-ai/Yi-1.5-9B 8.8B | 3.89% | bfloat16 | huggingface.co | Official leaderboard | |
| 103 | | 3.88% | float16 | Sep 2023 | huggingface.co | Official leaderboard |
| 104 | google/gemma-2b-it 2.5B | 3.81% | bfloat16 | huggingface.co | Official leaderboard | |
| 105 | meta-llama/Llama-2-13b-chat-hf 13B | 3.65% | float16 | huggingface.co | Official leaderboard | |
| 106 | | 3.65% | bfloat16 | Nov 2023 | huggingface.co | Official leaderboard |
| 107 | DebateLabKIT/Llama-3.3-Argunaut-1-70B-SFT 70.6B | 3.53% | bfloat16 | huggingface.co | Official leaderboard | |
| 108 | microsoft/Orca-2-7b | 3.49% | float16 | huggingface.co | Official leaderboard | |
| 109 | DebateLabKIT/Llama-3.3-Argunaut-1-70B-SPIN 70.6B | 3.37% | bfloat16 | huggingface.co | Official leaderboard | |
| 110 | cookinai/LlamaReflect-8B-CoT-safetensors 8B | 3.30% | bfloat16 | huggingface.co | Official leaderboard | |
| 111 | microsoft/Orca-2-13b | 3.13% | float16 | huggingface.co | Official leaderboard | |
| 112 | meta-llama/Llama-3.2-1B-Instruct 1.2B | 3.07% | bfloat16 | huggingface.co | Official leaderboard | |
| 113 | NousResearch/Nous-Hermes-llama-2-7b 6.7B | 2.87% | bfloat16 | huggingface.co | Official leaderboard | |
| 114 | wandb/gemma-2b-zephyr-dpo 2.5B | 2.54% | bfloat16 | huggingface.co | Official leaderboard | |
| 115 | Kukedlc/NeuralSynthesis-7b-v0.4-slerp 7.2B | 2.47% | bfloat16 | huggingface.co | Official leaderboard | |
| 116 | meta-llama/Llama-2-13b-hf 13B | 2.46% | float16 | huggingface.co | Official leaderboard | |
| 117 | | 2.45% | bfloat16 | Oct 2023 | huggingface.co | Official leaderboard |
| 118 | mlabonne/AlphaMonarch-7B 7.2B | 2.26% | bfloat16 | huggingface.co | Official leaderboard | |
| 119 | upstage/SOLAR-10.7B-v1.0 10.7B | 2.12% | float16 | huggingface.co | Official leaderboard | |
| 120 | openbmb/Eurus-7b-kto 7.2B | 2.01% | bfloat16 | huggingface.co | Official leaderboard | |
| 121 | LeroyDyer/Mixtral_AI_Cyber_Boss 7.2B | 1.90% | float16 | huggingface.co | Official leaderboard | |
| 122 | HuggingFaceTB/SmolLM-1.7B-Instruct 1.7B | 1.87% | bfloat16 | huggingface.co | Official leaderboard | |
| 123 | | 1.53% | bfloat16 | Feb 2024 | huggingface.co | Official leaderboard |
| 124 | meta-llama/Llama-2-7b-hf 6.7B | 1.42% | float16 | huggingface.co | Official leaderboard | |
| 125 | HuggingFaceTB/SmolLM-135M-Instruct 0.1B | 1.39% | bfloat16 | huggingface.co | Official leaderboard | |
| 126 | | 1.21% | float16 | Nov 2023 | huggingface.co | Official leaderboard |
| 127 | databricks/dolly-v2-3b 3B | 1.11% | bfloat16 | huggingface.co | Official leaderboard | |
| 128 | | 0.68% | bfloat16 | Feb 2024 | huggingface.co | Official leaderboard |
| 129 | terrycraddock/Reflection-Llama-3.1-8B 8B | -0.06% | float16 | huggingface.co | Official leaderboard | |
| 130 | OpenBuddy/openbuddy-zen-3b-v21.1-32k 4.8B | -0.23% | bfloat16 | huggingface.co | Official leaderboard | |
| 131 | OpenBuddy/openbuddy-zen-3b-v21.2-32k 4.8B | -0.33% | bfloat16 | huggingface.co | Official leaderboard | |
| 132 | | -1.33% | bfloat16 | Jul 2023 | huggingface.co | Official leaderboard |
| 133 | | -2.63% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
| # | Model | Score | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | | 17.6% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 2 | | 16.9% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 3 | | 15.3% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 4 | | 13.9% | bfloat16 | Oct 2024 | huggingface.co | Official leaderboard |
| 5 | | 13.1% | bfloat16 | Aug 2024 | huggingface.co | Official leaderboard |
| 6 | | 12.8% | float16 | Dec 2023 | huggingface.co | Official leaderboard |
| 7 | | 12.6% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 8 | | 12.3% | float16 | Oct 2024 | huggingface.co | Official leaderboard |
| 9 | | 11.4% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
| 10 | | 11.2% | float16 | Oct 2024 | huggingface.co | Official leaderboard |
| 11 | | 11.2% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
| 12 | | 9.75% | bfloat16 | May 2024 | huggingface.co | Official leaderboard |
| 13 | | 9.56% | bfloat16 | Sep 2024 | huggingface.co | Official leaderboard |
| 14 | | 9.33% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
| 15 | | 8.34% | bfloat16 | Nov 2023 | huggingface.co | Official leaderboard |
| 16 | | 8.16% | bfloat16 | Jul 2024 | huggingface.co | Official leaderboard |
| 17 | | 8.08% | float16 | Dec 2023 | huggingface.co | Official leaderboard |
| 18 | | 7.86% | bfloat16 | Jun 2024 | huggingface.co | Official leaderboard |
| 19 | | 7.41% | float16 | Dec 2023 | huggingface.co | Official leaderboard |
| 20 | | 6.81% | bfloat16 | Jun 2024 | huggingface.co | Official leaderboard |
| 21 | | 6.74% | bfloat16 | Jan 2025 | huggingface.co | Official leaderboard |
| 22 | | 6.41% | bfloat16 | Dec 2023 | huggingface.co | Official leaderboard |
| 23 | | 5.93% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
| 24 | | 4.48% | bfloat16 | Oct 2024 | huggingface.co | Official leaderboard |
| 25 | | 3.88% | float16 | Sep 2023 | huggingface.co | Official leaderboard |
| 26 | | 3.65% | bfloat16 | Nov 2023 | huggingface.co | Official leaderboard |
| 27 | | 2.45% | bfloat16 | Oct 2023 | huggingface.co | Official leaderboard |
| 28 | | 1.53% | bfloat16 | Feb 2024 | huggingface.co | Official leaderboard |
| 29 | | 1.21% | float16 | Nov 2023 | huggingface.co | Official leaderboard |
| 30 | | 0.68% | bfloat16 | Feb 2024 | huggingface.co | Official leaderboard |
| 31 | | -1.33% | bfloat16 | Jul 2023 | huggingface.co | Official leaderboard |
| 32 | | -2.63% | bfloat16 | Apr 2024 | huggingface.co | Official leaderboard |
Rankings by part
We read 5 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as LogiQA Δ, LogiQA2 Δ and LSAT-ar Δ. DeepSeek-R1-Distill-Qwen-14B leads 1 of them, and 4 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.
LogiQA Δ
32 models we track have a score here.
- 1
DeepSeek-R1-Distill-Qwen-14B 13.7%
- 2
DeepSeek-R1-Distill-Llama-70B ↑1 12.6%
- 3
DeepSeek-R1-Distill-Qwen-7B ↑4 10.4%
- 4
DeepSeek-R1-Distill-Qwen-32B ↓2 10.1%
- 5
Phi-3.5-MoE 8.47%
- 6
Yi-1.5-9B-Chat ↑6 7.51%
- 7
Qwen2.5-72B ↑6 7.51%
- 8
Aya Expanse 8B ↑2 7.19%
- 9
Qwen2-7B-Instruct ↑9 6.55%
- 10
Phi-2 ↑7 6.23%
LogiQA2 Δ
32 models we track have a score here.
- 1
DeepSeek-R1-Distill-Qwen-32B ↑1 19%
- 2
DeepSeek-R1-Distill-Qwen-14B ↓1 18.7%
- 3
Phi-3.5-MoE ↑2 15.1%
- 4
SOLAR-10.7B-Instruct-v1.0 ↑2 15%
- 5
DeepSeek-R1-Distill-Llama-70B ↓2 14.3%
- 6
DeepSeek-R1-Distill-Qwen-7B ↑1 13.4%
- 7
phi-3-small 7.4B ↑2 12.8%
- 8
phi-3-mini 3.8B ↑3 12.5%
- 9
Llama-3.1-Nemotron-70B-Instruct ↓5 12.5%
- 10
Yi-1.5-9B-Chat ↑2 12.3%
LSAT-ar Δ
32 models we track have a score here.
- 1
DeepSeek-R1-Distill-Llama-70B ↑2 22.6%
- 2
DeepSeek-R1-Distill-Qwen-14B ↓1 17%
- 3
DeepSeek-R1-Distill-Qwen-32B ↓1 16.5%
- 4
Llama-3.1-Nemotron-70B-Instruct 8.70%
- 5
Llama 3-70B ↑9 8.26%
- 6
Phi-3.5-MoE ↓1 7.83%
- 7
SOLAR-10.7B-Instruct-v1.0 ↓1 6.96%
- 8
phi-3-mini 3.8B ↑3 6.52%
- 9
Qwen2.5-72B ↑4 6.52%
- 10
Mistral 7B v0.1 ↑15 6.09%
LSAT-lr Δ
32 models we track have a score here.
- 1
Llama-3.1-Nemotron-70B-Instruct ↑3 22%
- 2
DeepSeek-R1-Distill-Qwen-32B 20.4%
- 3
phi-3-small 7.4B ↑6 17.8%
- 4
DeepSeek-R1-Distill-Qwen-14B ↓3 17.6%
- 5
Phi-3.5-MoE 17.3%
- 6
phi-3-mini 3.8B ↑5 16.9%
- 7
Aya Expanse 8B ↑3 16.9%
- 8
Aya Expanse 32B 15.9%
- 9
DeepSeek-R1-Distill-Llama-70B ↓6 15.5%
- 10
Qwen2.5-72B ↑3 14.7%
LSAT-rc Δ
32 models we track have a score here.
- 1
Aya Expanse 32B ↑7 24.5%
- 2
SOLAR-10.7B-Instruct-v1.0 ↑4 22.3%
- 3
DeepSeek-R1-Distill-Qwen-14B ↓2 21.2%
- 4
DeepSeek-R1-Distill-Qwen-7B ↑3 20.8%
- 5
Llama-3.1-Nemotron-70B-Instruct ↓1 20.1%
- 6
phi-3-small 7.4B ↑3 19.7%
- 7
DeepSeek-R1-Distill-Qwen-32B ↓5 18.6%
- 8
Aya Expanse 8B ↑2 18.2%
- 9
Phi-3.5-MoE ↓4 16.7%
- 10
phi-3-mini 3.8B ↑1 14.9%
4 models have held number one on the official leaderboard since its release in January 2024
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official Open CoT Leaderboard leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official Open CoT Leaderboard leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
9 of 10 checks pass, 1 warning. Look first at the Newest score check. The newest came 22 months ago, in January 2025, for DeepSeek-R1-Distill-Llama-70B. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 1 warning 3 pass 1 note
- Pass Independent runs Scores from the official leaderboard (133 entries).
- Pass Models scored 32 models we track, plus 101 other entries on its leaderboard.
- Warning Newest score The newest came 22 months ago, in January 2025, for DeepSeek-R1-Distill-Llama-70B.
- Note Frontier models None of the top 10 has a score. The strongest tested is Qwen2.5-72B (#166).
- Pass Headroom The best is 17.6%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on huggingface.co: 32 rows for models we track and 101 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.