Benchmark Almanac

Nejumi LLM Leaderboard 3

o3 0.83 leads since April 2025
Not in model cards
Version 3

Models complete Japanese language and alignment evaluations. The organizers group results by use case and include safety evaluations.

The project announced a new scoring structure with this release.

Made by
Weights & Biases Japan
Released
July 2024
Kind
Reading in many languages
Level
School
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Measures
Capability
Language
Japanese
In and out
Text → Text
Interaction
Single turn
Run by
Anyone
Strongest model tested
o3 #84 on the capability index
Last model added
August 2025
Scores refreshed
November 20th, 2026
95
Entries on its own leaderboard, 58 models we track

o3 leads with 0.83 among models released by August 2025 o3 leads with 0.83 among models released by August 2025

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in August 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

95 entries. Scroll sideways for the rest.

  1. 0.83 o3
  2. 0.83 Claude Sonnet 4
  3. 0.82 o1
  4. 0.82 Claude 3.7 Sonnet
  5. 0.82 Claude 3.5 Sonnet
  6. 0.81 Claude Opus 4
  7. 0.81 Claude 3.5 Sonnet (October 2024)
  8. 0.80 GPT-4.5
  9. 0.80 o1-preview
  10. 0.80 o4-mini
  11. 0.79 chatgpt-4o-latest
  12. 0.79 GPT-4o
  13. 0.79 Grok 3
  14. 0.79 Gemini 2.5 Flash
  15. 0.79 Qwen2.5-Max
  16. 0.79 GPT-4.1
  17. 0.78 GPT-4o
  18. 0.78 Gemini 1.5 Pro
  19. 0.78 o3-mini
  20. 0.78 deepseek-chat
  21. 0.78 Gemini 2.0 Flash
  22. 0.77 GPT-4o
  23. 0.77 Gemini 2.5 Pro
  24. 0.77 gpt-oss-120b
  25. 0.77 DeepSeek-R1
  26. 0.77 Cohere Command A
  27. 0.76 Athene-V2-Chat
  28. 0.76 GPT-4 Turbo
  29. 0.76 Nexusflow/Athene-V2-Agent
  30. 0.76 Qwen2.5-72B
  31. 0.75 llm-jp/llm-jp-3.1-8x13b-instruct4
  32. 0.75 grok-beta
  33. 0.74 abeja/ABEJA-Qwen2.5-32b-Japanese-v0.1
  34. 0.74 MiniMax-Text-01
  35. 0.74 Gemini 1.5 Flash
  36. 0.74 Mistral Large 2
  37. 0.74 meta-llama/Meta-Llama-3.1-405B-Instruct-FP8
  38. 0.74 Qwen/Qwen2.5-32B-Instruct
  39. 0.73 Claude 3 Opus
  40. 0.73 Qwen2-72B
  41. 0.73 Amazon Nova Pro
  42. 0.73 gpt-oss-20b
  43. 0.73 plamo-2.0-prime
  44. 0.73 GPT-4o mini
  45. 0.73 AXCXEPT/EZO-Qwen2.5-72B-Instruct
  46. 0.73 Gemini 1.5 Pro
  47. 0.72 Gemma 3 27B
  48. 0.72 google/gemma-2-27b-it
  49. 0.72 Mistral Small 3.1
  50. 0.72 Mistral Large 2
  51. 0.72 GPT-4
  52. 0.71 llm-jp/llm-jp-3.1-13b-instruct4
  53. 0.71 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.3
  54. 0.71 Llama 4 Maverick
  55. 0.71 AXCXEPT/EZO-Qwen2.5-32B-Instruct
  56. 0.71 Gemini 1.5 Flash 8B
  57. 0.70 Llama 3.2 90B
  58. 0.70 plamo-1.0-prime
  59. 0.70 Claude 3 Sonnet
  60. 0.70 Gemini 1.5 Flash
  61. 0.70 Qwen/Qwen2.5-14B-Instruct
  62. 0.70 command-r-plus-08-2024
  63. 0.70 Llama 3.3 70B Instruct
  64. 0.70 cyberagent/Llama-3.1-70B-Japanese-Instruct-2407
  65. 0.70 Llama-3.1-Nemotron-70B-Instruct
  66. 0.69 Gemma 3 12B
  67. 0.69 meta-llama/Meta-Llama-3.1-70B-Instruct
  68. 0.69 Aya Expanse 32B
  69. 0.68 CohereForAI/c4ai-command-r-plus-08-2024
  70. 0.68 solar-pro-241126
  71. 0.68 Amazon Nova Micro
  72. 0.68 llm-jp/llm-jp-3-8x13b-instruct3
  73. 0.67 Claude 3 Haiku
  74. 0.67 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.1
  75. 0.67 ailab-rl/model-share/model_calm3_finetuning:v0
  76. 0.66 llm-jp/llm-jp-3.1-1.8b-instruct4
  77. 0.66 Amazon Nova Lite
  78. 0.66 meta-llama/Meta-Llama-3-70B-Instruct
  79. 0.66 Qwen/Qwen2.5-7B-Instruct
  80. 0.66 syn-karakuri-upstage-10b
  81. 0.65 c4ai-command-r-08-2024
  82. 0.65 tokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1
  83. 0.65 Llama 4 Scout
  84. 0.65 cyberagent/calm3-22b-chat
  85. 0.65 Phi-4
  86. 0.65 solar-mini
  87. 0.64 llm-jp/llm-jp-3-172b-instruct3
  88. 0.64 Aya Expanse 8B
  89. 0.63 QwQ-32B-Preview
  90. 0.63 turing-motors/Llama-3-heron-brain-70B-v0.3
  91. 0.63 llm-jp/llm-jp-3-172b-beta2-instruct2
  92. 0.63 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.2
  93. 0.62 Mistral NeMo
  94. 0.62 elyza/Llama-3-ELYZA-JP-8B
  95. 0.61 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.1
  1. 1 o3 0.83
  2. 2 Claude Sonnet 4 Extended Thinking 0.83
  3. 3 o1 0.82
  4. 4 Claude 3.7 Sonnet thinking 0.82
  5. 5 Claude 3.5 Sonnet 0.82
  6. 6 Claude Opus 4 Extended Thinking 0.81
  7. 7 Claude 3.5 Sonnet (October 2024) 0.81
  8. 8 GPT-4.5 0.80
  9. 9 o1-preview 0.80
  10. 10 o4-mini high 0.80
  11. 11 chatgpt-4o-latest 0.79
  12. 12 GPT-4o May 2024 0.79
  13. 13 Grok 3 0.79
  14. 14 Gemini 2.5 Flash Jun 2025 0.79
  15. 15 Qwen2.5-Max 0.79
  16. 16 GPT-4.1 0.79
  17. 17 GPT-4o Aug 2024 0.78
  18. 18 Gemini 1.5 Pro Sep 2024 0.78
  19. 19 o3-mini high 0.78
  20. 20 deepseek-chat 0.78
  21. 21 Gemini 2.0 Flash Dec 2024 0.78
  22. 22 GPT-4o Nov 2024 0.77
  23. 23 Gemini 2.5 Pro Jun 2025 0.77
  24. 24 gpt-oss-120b 0.77
  25. 25 DeepSeek-R1 0.77
  26. 26 Cohere Command A 0.77
  27. 27 Athene-V2-Chat 0.76
  28. 28 GPT-4 Turbo Apr 2024 0.76
  29. 29 Nexusflow/Athene-V2-Agent 0.76
  30. 30 Qwen2.5-72B 0.76
  31. 31 llm-jp/llm-jp-3.1-8x13b-instruct4 0.75
  32. 32 grok-beta 0.75
  33. 33 abeja/ABEJA-Qwen2.5-32b-Japanese-v0.1 0.74
  34. 34 MiniMax-Text-01 0.74
  35. 35 Gemini 1.5 Flash Sep 2024 0.74
  36. 36 Mistral Large 2 Nov 2024 0.74
  37. 37 meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 0.74
  38. 38 Qwen/Qwen2.5-32B-Instruct 0.74
  39. 39 Claude 3 Opus 0.73
  40. 40 Qwen2-72B 0.73
  41. 41 Amazon Nova Pro 0.73
  42. 42 gpt-oss-20b 0.73
  43. 43 plamo-2.0-prime 0.73
  44. 44 GPT-4o mini 0.73
  45. 45 AXCXEPT/EZO-Qwen2.5-72B-Instruct 0.73
  46. 46 Gemini 1.5 Pro May 2024 0.73
  47. 47 Gemma 3 27B 0.72
  48. 48 google/gemma-2-27b-it 0.72
  49. 49 Mistral Small 3.1 0.72
  50. 50 Mistral Large 2 Jul 2024 0.72
  51. 51 GPT-4 Jun 2023 0.72
  52. 52 llm-jp/llm-jp-3.1-13b-instruct4 0.71
  53. 53 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.3 0.71
  54. 54 Llama 4 Maverick 0.71
  55. 55 AXCXEPT/EZO-Qwen2.5-32B-Instruct 0.71
  56. 56 Gemini 1.5 Flash 8B 0.71
  57. 57 Llama 3.2 90B 0.70
  58. 58 plamo-1.0-prime 0.70
  59. 59 Claude 3 Sonnet 0.70
  60. 60 Gemini 1.5 Flash May 2024 0.70
  61. 61 Qwen/Qwen2.5-14B-Instruct 0.70
  62. 62 command-r-plus-08-2024 0.70
  63. 63 Llama 3.3 70B Instruct 0.70
  64. 64 cyberagent/Llama-3.1-70B-Japanese-Instruct-2407 0.70
  65. 65 Llama-3.1-Nemotron-70B-Instruct 0.70
  66. 66 Gemma 3 12B 0.69
  67. 67 meta-llama/Meta-Llama-3.1-70B-Instruct 0.69
  68. 68 Aya Expanse 32B 0.69
  69. 69 CohereForAI/c4ai-command-r-plus-08-2024 0.68
  70. 70 solar-pro-241126 0.68
  71. 71 Amazon Nova Micro 0.68
  72. 72 llm-jp/llm-jp-3-8x13b-instruct3 0.68
  73. 73 Claude 3 Haiku 0.67
  74. 74 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.1 0.67
  75. 75 ailab-rl/model-share/model_calm3_finetuning:v0 0.67
  76. 76 llm-jp/llm-jp-3.1-1.8b-instruct4 0.66
  77. 77 Amazon Nova Lite 0.66
  78. 78 meta-llama/Meta-Llama-3-70B-Instruct 0.66
  79. 79 Qwen/Qwen2.5-7B-Instruct 0.66
  80. 80 syn-karakuri-upstage-10b 0.66
  81. 81 c4ai-command-r-08-2024 0.65
  82. 82 tokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1 0.65
  83. 83 Llama 4 Scout 0.65
  84. 84 cyberagent/calm3-22b-chat 0.65
  85. 85 Phi-4 0.65
  86. 86 solar-mini 0.65
  87. 87 llm-jp/llm-jp-3-172b-instruct3 0.64
  88. 88 Aya Expanse 8B 0.64
  89. 89 QwQ-32B-Preview 0.63
  90. 90 turing-motors/Llama-3-heron-brain-70B-v0.3 0.63
  91. 91 llm-jp/llm-jp-3-172b-beta2-instruct2 0.63
  92. 92 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.2 0.63
  93. 93 Mistral NeMo 0.62
  94. 94 elyza/Llama-3-ELYZA-JP-8B 0.62
  95. 95 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.1 0.61

Source: Official Nejumi LLM Leaderboard 3 leaderboard

Source: Official Nejumi LLM Leaderboard 3 leaderboard

#EntryScoreSettingReleasedRun onLinkSource
1 o3 0.83 Apr 2025 Apr 2025 wandb.ai Official leaderboard
2 Claude Sonnet 4 0.83 Extended Thinking May 2025 Jun 2025 wandb.ai Official leaderboard
3 o1 0.82 Dec 2024 Dec 2024 wandb.ai Official leaderboard
4 Claude 3.7 Sonnet 0.82 thinking Feb 2025 Apr 2025 wandb.ai Official leaderboard
5 Claude 3.5 Sonnet 0.82 Jun 2024 Jul 2024 wandb.ai Official leaderboard
6 Claude Opus 4 0.81 Extended Thinking May 2025 Jun 2025 wandb.ai Official leaderboard
7 Claude 3.5 Sonnet (October 2024) 0.81 Oct 2024 Nov 2024 wandb.ai Official leaderboard
8 GPT-4.5 0.80 Feb 2025 Mar 2025 wandb.ai Official leaderboard
9 o1-preview 0.80 Sep 2024 Sep 2024 wandb.ai Official leaderboard
10 o4-mini 0.80 high Apr 2025 Jun 2025 wandb.ai Official leaderboard
11 chatgpt-4o-latest 0.79 Aug 2024 wandb.ai Official leaderboard
12 GPT-4o May 2024 0.79 May 2024 Jul 2024 wandb.ai Official leaderboard
13 Grok 3 0.79 Apr 2025 Apr 2025 wandb.ai Official leaderboard
14 Gemini 2.5 Flash Jun 2025 0.79 Jun 2025 Jun 2025 wandb.ai Official leaderboard
15 Qwen2.5-Max 0.79 Jan 2025 Feb 2025 wandb.ai Official leaderboard
16 GPT-4.1 0.79 Apr 2025 Apr 2025 wandb.ai Official leaderboard
17 GPT-4o Aug 2024 0.78 Aug 2024 Aug 2024 wandb.ai Official leaderboard
18 Gemini 1.5 Pro Sep 2024 0.78 Sep 2024 Nov 2024 wandb.ai Official leaderboard
19 o3-mini 0.78 high Jan 2025 Apr 2025 wandb.ai Official leaderboard
20 deepseek-chat 0.78 Dec 2024 wandb.ai Official leaderboard
21 Gemini 2.0 Flash Dec 2024 0.78 Dec 2024 Dec 2024 wandb.ai Official leaderboard
22 GPT-4o Nov 2024 0.77 Nov 2024 Nov 2024 wandb.ai Official leaderboard
23 Gemini 2.5 Pro Jun 2025 0.77 Jun 2025 Jun 2025 wandb.ai Official leaderboard
24 gpt-oss-120b 0.77 Aug 2025 Aug 2025 wandb.ai Official leaderboard
25 DeepSeek-R1 0.77 Jan 2025 Jan 2025 wandb.ai Official leaderboard
26 Cohere Command A 0.77 Mar 2025 Mar 2025 wandb.ai Official leaderboard
27 Athene-V2-Chat 0.76 Nov 2024 Nov 2024 wandb.ai Official leaderboard
28 GPT-4 Turbo Apr 2024 0.76 Apr 2024 Aug 2024 wandb.ai Official leaderboard
29 Nexusflow/Athene-V2-Agent 72.7B 0.76 Nov 2024 wandb.ai Official leaderboard
30 Qwen2.5-72B 0.76 Sep 2024 Sep 2024 wandb.ai Official leaderboard
31 llm-jp/llm-jp-3.1-8x13b-instruct4 73.3B 0.75 Jun 2025 wandb.ai Official leaderboard
32 grok-beta 0.75 Oct 2024 wandb.ai Official leaderboard
33 abeja/ABEJA-Qwen2.5-32b-Japanese-v0.1 32.8B 0.74 Jan 2025 wandb.ai Official leaderboard
34 MiniMax-Text-01 0.74 Jan 2025 Jan 2025 wandb.ai Official leaderboard
35 Gemini 1.5 Flash Sep 2024 0.74 Aug 2024 Nov 2024 wandb.ai Official leaderboard
36 Mistral Large 2 Nov 2024 0.74 Nov 2024 Nov 2024 wandb.ai Official leaderboard
37 meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 405.9B 0.74 Sep 2024 wandb.ai Official leaderboard
38 Qwen/Qwen2.5-32B-Instruct 32.8B 0.74 Sep 2024 wandb.ai Official leaderboard
39 Claude 3 Opus 0.73 Feb 2024 Sep 2024 wandb.ai Official leaderboard
40 Qwen2-72B 0.73 Jun 2024 Jul 2024 wandb.ai Official leaderboard
41 Amazon Nova Pro 0.73 Dec 2024 Dec 2024 wandb.ai Official leaderboard
42 gpt-oss-20b 0.73 Aug 2025 Aug 2025 wandb.ai Official leaderboard
43 plamo-2.0-prime 0.73 May 2025 wandb.ai Official leaderboard
44 GPT-4o mini 0.73 Jul 2024 Aug 2024 wandb.ai Official leaderboard
45 AXCXEPT/EZO-Qwen2.5-72B-Instruct 72.7B 0.73 Oct 2024 wandb.ai Official leaderboard
46 Gemini 1.5 Pro May 2024 0.73 May 2024 Aug 2024 wandb.ai Official leaderboard
47 Gemma 3 27B 0.72 Mar 2025 Apr 2025 wandb.ai Official leaderboard
48 google/gemma-2-27b-it 27.2B 0.72 Sep 2024 wandb.ai Official leaderboard
49 Mistral Small 3.1 0.72 Mar 2025 Apr 2025 wandb.ai Official leaderboard
50 Mistral Large 2 Jul 2024 0.72 Jul 2024 Aug 2024 wandb.ai Official leaderboard
51 GPT-4 Jun 2023 0.72 Jun 2023 Aug 2024 wandb.ai Official leaderboard
52 llm-jp/llm-jp-3.1-13b-instruct4 14B 0.71 Jun 2025 wandb.ai Official leaderboard
53 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.3 70.6B 0.71 Dec 2024 wandb.ai Official leaderboard
54 Llama 4 Maverick 0.71 Apr 2025 Apr 2025 wandb.ai Official leaderboard
55 AXCXEPT/EZO-Qwen2.5-32B-Instruct 32.8B 0.71 Oct 2024 wandb.ai Official leaderboard
56 Gemini 1.5 Flash 8B 0.71 Oct 2024 Dec 2024 wandb.ai Official leaderboard
57 Llama 3.2 90B 0.70 Sep 2024 Nov 2024 wandb.ai Official leaderboard
58 plamo-1.0-prime 0.70 Dec 2024 wandb.ai Official leaderboard
59 Claude 3 Sonnet 0.70 Feb 2024 Sep 2024 wandb.ai Official leaderboard
60 Gemini 1.5 Flash May 2024 0.70 May 2024 Aug 2024 wandb.ai Official leaderboard
61 Qwen/Qwen2.5-14B-Instruct 14.8B 0.70 Sep 2024 wandb.ai Official leaderboard
62 command-r-plus-08-2024 0.70 Sep 2024 wandb.ai Official leaderboard
63 Llama 3.3 70B Instruct 0.70 Dec 2024 Dec 2024 wandb.ai Official leaderboard
64 cyberagent/Llama-3.1-70B-Japanese-Instruct-2407 70.6B 0.70 Aug 2024 wandb.ai Official leaderboard
65 Llama-3.1-Nemotron-70B-Instruct 0.70 Oct 2024 Oct 2024 wandb.ai Official leaderboard
66 Gemma 3 12B 0.69 Mar 2025 Apr 2025 wandb.ai Official leaderboard
67 meta-llama/Meta-Llama-3.1-70B-Instruct 70.6B 0.69 Aug 2024 wandb.ai Official leaderboard
68 Aya Expanse 32B 0.69 Oct 2024 Oct 2024 wandb.ai Official leaderboard
69 CohereForAI/c4ai-command-r-plus-08-2024 103.8B 0.68 Sep 2024 wandb.ai Official leaderboard
70 solar-pro-241126 0.68 Dec 2024 wandb.ai Official leaderboard
71 Amazon Nova Micro 0.68 Dec 2024 Dec 2024 wandb.ai Official leaderboard
72 llm-jp/llm-jp-3-8x13b-instruct3 172.4B 0.68 Apr 2025 wandb.ai Official leaderboard
73 Claude 3 Haiku 0.67 Mar 2024 Aug 2024 wandb.ai Official leaderboard
74 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.1 70.6B 0.67 Nov 2024 wandb.ai Official leaderboard
75 ailab-rl/model-share/model_calm3_finetuning:v0 0.67 Jan 2025 wandb.ai Official leaderboard
76 llm-jp/llm-jp-3.1-1.8b-instruct4 1.8B 0.66 Jun 2025 wandb.ai Official leaderboard
77 Amazon Nova Lite 0.66 Dec 2024 Dec 2024 wandb.ai Official leaderboard
78 meta-llama/Meta-Llama-3-70B-Instruct 70.6B 0.66 Jul 2024 wandb.ai Official leaderboard
79 Qwen/Qwen2.5-7B-Instruct 7.6B 0.66 Sep 2024 wandb.ai Official leaderboard
80 syn-karakuri-upstage-10b 0.66 Mar 2025 wandb.ai Official leaderboard
81 c4ai-command-r-08-2024 0.65 Aug 2024 Sep 2024 wandb.ai Official leaderboard
82 tokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1 70.6B 0.65 Aug 2024 wandb.ai Official leaderboard
83 Llama 4 Scout 0.65 Apr 2025 Apr 2025 wandb.ai Official leaderboard
84 cyberagent/calm3-22b-chat 22.1B 0.65 Jul 2024 wandb.ai Official leaderboard
85 Phi-4 0.65 Dec 2024 Jan 2025 wandb.ai Official leaderboard
86 solar-mini 0.65 Feb 2025 wandb.ai Official leaderboard
87 llm-jp/llm-jp-3-172b-instruct3 172.4B 0.64 Apr 2025 wandb.ai Official leaderboard
88 Aya Expanse 8B 0.64 Oct 2024 Oct 2024 wandb.ai Official leaderboard
89 QwQ-32B-Preview 0.63 Nov 2024 Nov 2024 wandb.ai Official leaderboard
90 turing-motors/Llama-3-heron-brain-70B-v0.3 70.6B 0.63 Aug 2024 wandb.ai Official leaderboard
91 llm-jp/llm-jp-3-172b-beta2-instruct2 172.4B 0.63 Nov 2024 wandb.ai Official leaderboard
92 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.2 8B 0.63 Nov 2024 wandb.ai Official leaderboard
93 Mistral NeMo 0.62 Jul 2024 Aug 2024 wandb.ai Official leaderboard
94 elyza/Llama-3-ELYZA-JP-8B 8B 0.62 Jul 2024 wandb.ai Official leaderboard
95 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.1 8B 0.61 Nov 2024 wandb.ai Official leaderboard
#ModelScoreSettingReleasedRun onLinkSource
1 o3 0.83 Apr 2025 Apr 2025 wandb.ai Official leaderboard
2 Claude Sonnet 4 0.83 Extended Thinking May 2025 Jun 2025 wandb.ai Official leaderboard
3 o1 0.82 Dec 2024 Dec 2024 wandb.ai Official leaderboard
4 Claude 3.7 Sonnet 0.82 thinking Feb 2025 Apr 2025 wandb.ai Official leaderboard
5 Claude 3.5 Sonnet 0.82 Jun 2024 Jul 2024 wandb.ai Official leaderboard
6 Claude Opus 4 0.81 Extended Thinking May 2025 Jun 2025 wandb.ai Official leaderboard
7 Claude 3.5 Sonnet (October 2024) 0.81 Oct 2024 Nov 2024 wandb.ai Official leaderboard
8 GPT-4.5 0.80 Feb 2025 Mar 2025 wandb.ai Official leaderboard
9 o1-preview 0.80 Sep 2024 Sep 2024 wandb.ai Official leaderboard
10 o4-mini 0.80 high Apr 2025 Jun 2025 wandb.ai Official leaderboard
11 GPT-4o May 2024 0.79 May 2024 Jul 2024 wandb.ai Official leaderboard
12 Grok 3 0.79 Apr 2025 Apr 2025 wandb.ai Official leaderboard
13 Gemini 2.5 Flash Jun 2025 0.79 Jun 2025 Jun 2025 wandb.ai Official leaderboard
14 Qwen2.5-Max 0.79 Jan 2025 Feb 2025 wandb.ai Official leaderboard
15 GPT-4.1 0.79 Apr 2025 Apr 2025 wandb.ai Official leaderboard
16 GPT-4o Aug 2024 0.78 Aug 2024 Aug 2024 wandb.ai Official leaderboard
17 Gemini 1.5 Pro Sep 2024 0.78 Sep 2024 Nov 2024 wandb.ai Official leaderboard
18 o3-mini 0.78 high Jan 2025 Apr 2025 wandb.ai Official leaderboard
19 Gemini 2.0 Flash Dec 2024 0.78 Dec 2024 Dec 2024 wandb.ai Official leaderboard
20 GPT-4o Nov 2024 0.77 Nov 2024 Nov 2024 wandb.ai Official leaderboard
21 Gemini 2.5 Pro Jun 2025 0.77 Jun 2025 Jun 2025 wandb.ai Official leaderboard
22 gpt-oss-120b 0.77 Aug 2025 Aug 2025 wandb.ai Official leaderboard
23 DeepSeek-R1 0.77 Jan 2025 Jan 2025 wandb.ai Official leaderboard
24 Cohere Command A 0.77 Mar 2025 Mar 2025 wandb.ai Official leaderboard
25 Athene-V2-Chat 0.76 Nov 2024 Nov 2024 wandb.ai Official leaderboard
26 GPT-4 Turbo Apr 2024 0.76 Apr 2024 Aug 2024 wandb.ai Official leaderboard
27 Qwen2.5-72B 0.76 Sep 2024 Sep 2024 wandb.ai Official leaderboard
28 MiniMax-Text-01 0.74 Jan 2025 Jan 2025 wandb.ai Official leaderboard
29 Gemini 1.5 Flash Sep 2024 0.74 Aug 2024 Nov 2024 wandb.ai Official leaderboard
30 Mistral Large 2 Nov 2024 0.74 Nov 2024 Nov 2024 wandb.ai Official leaderboard
31 Claude 3 Opus 0.73 Feb 2024 Sep 2024 wandb.ai Official leaderboard
32 Qwen2-72B 0.73 Jun 2024 Jul 2024 wandb.ai Official leaderboard
33 Amazon Nova Pro 0.73 Dec 2024 Dec 2024 wandb.ai Official leaderboard
34 gpt-oss-20b 0.73 Aug 2025 Aug 2025 wandb.ai Official leaderboard
35 GPT-4o mini 0.73 Jul 2024 Aug 2024 wandb.ai Official leaderboard
36 Gemini 1.5 Pro May 2024 0.73 May 2024 Aug 2024 wandb.ai Official leaderboard
37 Gemma 3 27B 0.72 Mar 2025 Apr 2025 wandb.ai Official leaderboard
38 Mistral Small 3.1 0.72 Mar 2025 Apr 2025 wandb.ai Official leaderboard
39 Mistral Large 2 Jul 2024 0.72 Jul 2024 Aug 2024 wandb.ai Official leaderboard
40 GPT-4 Jun 2023 0.72 Jun 2023 Aug 2024 wandb.ai Official leaderboard
41 Llama 4 Maverick 0.71 Apr 2025 Apr 2025 wandb.ai Official leaderboard
42 Gemini 1.5 Flash 8B 0.71 Oct 2024 Dec 2024 wandb.ai Official leaderboard
43 Llama 3.2 90B 0.70 Sep 2024 Nov 2024 wandb.ai Official leaderboard
44 Claude 3 Sonnet 0.70 Feb 2024 Sep 2024 wandb.ai Official leaderboard
45 Gemini 1.5 Flash May 2024 0.70 May 2024 Aug 2024 wandb.ai Official leaderboard
46 Llama 3.3 70B Instruct 0.70 Dec 2024 Dec 2024 wandb.ai Official leaderboard
47 Llama-3.1-Nemotron-70B-Instruct 0.70 Oct 2024 Oct 2024 wandb.ai Official leaderboard
48 Gemma 3 12B 0.69 Mar 2025 Apr 2025 wandb.ai Official leaderboard
49 Aya Expanse 32B 0.69 Oct 2024 Oct 2024 wandb.ai Official leaderboard
50 Amazon Nova Micro 0.68 Dec 2024 Dec 2024 wandb.ai Official leaderboard
51 Claude 3 Haiku 0.67 Mar 2024 Aug 2024 wandb.ai Official leaderboard
52 Amazon Nova Lite 0.66 Dec 2024 Dec 2024 wandb.ai Official leaderboard
53 c4ai-command-r-08-2024 0.65 Aug 2024 Sep 2024 wandb.ai Official leaderboard
54 Llama 4 Scout 0.65 Apr 2025 Apr 2025 wandb.ai Official leaderboard
55 Phi-4 0.65 Dec 2024 Jan 2025 wandb.ai Official leaderboard
56 Aya Expanse 8B 0.64 Oct 2024 Oct 2024 wandb.ai Official leaderboard
57 QwQ-32B-Preview 0.63 Nov 2024 Nov 2024 wandb.ai Official leaderboard
58 Mistral NeMo 0.62 Jul 2024 Aug 2024 wandb.ai Official leaderboard

Rankings by part

We read 22 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as 汎用的言語性能(GLP)_AVG, アラインメント(ALT)_AVG and GLP_表現. o3 leads 7 of them, and 8 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.

汎用的言語性能(GLP)_AVG

58 models we track have a score here.

  1. 1 o3 0.81
  2. 2 o1 ↑1 0.80
  3. 3 o1-preview ↑6 0.79
  4. 4 o4-mini ↑6 0.78
  5. 5 Claude 3.7 Sonnet ↓1 0.78
  6. 6 GPT-4.1 ↑9 0.78
  7. 7 Claude Opus 4 ↓1 0.78
  8. 8 Grok 3 ↑4 0.77
  9. 9 o3-mini ↑9 0.77
  10. 10 Claude Sonnet 4 ↓8 0.77

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Claude Sonnet 4
Extended Th…
0.83
Claude 3.7 Sonnet
thinking
0.82
Claude Opus 4
Extended Th…
0.81

Source: Official Nejumi LLM Leaderboard 3 leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Claude Sonnet 4
Extended Th…
0.83
Claude 3.7 Sonnet
thinking
0.82
Claude Opus 4
Extended Th…
0.81

Source: Official Nejumi LLM Leaderboard 3 leaderboard

3 models have held number one on the official leaderboard since its release in July 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic OpenAI
Own leaderboard Replaced by 4, Aug 2025
o3 0.83

Source: Official Nejumi LLM Leaderboard 3 leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard
0.65 0.70 0.75 0.80 0.85 2023 2024 2025 0.65 0.70 0.75 0.80 0.85 2023 2024 2025

Source: Official Nejumi LLM Leaderboard 3 leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

Versions

More in Multilingual

All 545 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Claude Opus 5 95.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%

Data audit

All 8 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the official leaderboard (95 entries).
  • Pass Models scored 58 models we track, plus 37 other entries on its leaderboard.
  • Note Newest score The newest came 15 months ago, in August 2025, for gpt-oss-120b. That’s expected for a test replaced by Nejumi LLM Leaderboard 4.
  • Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is o3 (#84).
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on wandb.ai: 61 rows for models we track and 37 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check