Benchmark Almanac

NorEval

DeepSeek-V4-Flash-Base 68.2 leads since April 2026
Not in model cards

Models complete language understanding and generation tasks in both Bokmål and Nynorsk. The authors compare model performance with human baselines.

The authors release the datasets and more than 100 human-written prompts for the LM Evaluation Harness.

Released
April 2025
Kind
Knowledge in many languages
Level
School
Scored
Checked automatically
Answers
Open-ended
Tasks
All public
Measures
Capability
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
Strongest model tested
Qwen2.5-72B #166 on the capability index
Last model added
April 2026
Scores refreshed
November 20th, 2026
97
Entries on its own leaderboard, 19 models we track

DeepSeek-V4-Flash-Base leads with 68.2 among models released by April 2026 DeepSeek-V4-Flash-Base leads with 68.2 among models released by April 2026

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. The dashboard averages chance-adjusted task scores within categories, then weights each category equally. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in April 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

97 entries. Scroll sideways for the rest.

  1. 68.2 DeepSeek-V4-Flash-Base
  2. 67.2 Gemma4 31B (pretrained)
  3. 64.8 Nemotron 4 340B
  4. 64.0 Gemma3 27B (pretrained)
  5. 63.9 Gemma4 26B (4B active) (pretrained)
  6. 62.6 Qwen3.5-35B-A3B-Base
  7. 61.4 Llama4 Scout 109B (17B active)
  8. 61.4 GLM-4.5 Air (12B active)
  9. 61.1 Llama 3.1-70B
  10. 61.1 TildeOpen 30B 64k
  11. 60.9 Gemma4 12B (pretrained)
  12. 60.9 Gemma3 12B (pretrained)
  13. 59.6 NorOLMo 13B (stage 2, new mix)
  14. 59.4 NorwAI Mixtral 8x7B (47B)
  15. 59.2 Mistral 24B
  16. 58.5 GLM-4 32B
  17. 58.3 Apertus 70B
  18. 58.2 Gemma4 E4B (8B) (pretrained)
  19. 58.0 Qwen3.5-9B-Base
  20. 57.9 Mistral 24B (v2501)
  21. 57.8 NorMistral 11B Long
  22. 57.7 Prelude 9B
  23. 57.4 NorMistral 11B
  24. 57.3 Nemotron 3 Super 120B (12B active)
  25. 56.9 Qwen2.5-72B
  26. 56.3 Ministral 3 14B
  27. 56.1 Mixtral 8x22B (141B)
  28. 55.7 NorOLMo 13B
  29. 55.5 EuroLLM 1.5 9B
  30. 55.4 NorOLMo 13B (stage 2)
  31. 55.4 Bielik 11B v3
  32. 55.2 Qwen3-14B-Base
  33. 54.7 Apertus 8B Instruct
  34. 54.7 EuroLLM 22B
  35. 54.6 Qwen2.5-32B
  36. 54.4 Apertus 8B (stage 5)
  37. 53.6 AI Sweden Llama3 8B
  38. 53.2 Ministral3 8B
  39. 52.5 Mistral Nemo 12B
  40. 52.4 Qwen3-8B-Base
  41. 51.7 NorOLMo 13B (stage 1)
  42. 51.5 Apertus 8B (stage 4)
  43. 50.8 EuroLLM 9B
  44. 50.6 Mixtral 8x7B (47B)
  45. 50.6 Gemma3 4B (pretrained)
  46. 50.5 Nemotron 3 Nano 30B (3B active)
  47. 50.2 Viking 33B
  48. 49.5 Apertus 8B (stage 3)
  49. 49.5 Llama 3.1-8B
  50. 49.4 Tiny Aya 3B
  51. 49.3 Apertus 8B (stage 2)
  52. 49.0 DeepSeek V3 671B (37B active)
  53. 48.6 Gemma4 E2B (5B) (pretrained)
  54. 48.6 NorwAI Mistral 7B
  55. 48.4 Apertus 8B (stage 1)
  56. 48.4 OLMo3 32B
  57. 48.3 GPT-SW3 40B
  58. 48.0 Qwen3-4B-Base
  59. 47.6 Trinity Mini 26B (3B active)
  60. 47.0 Ministral3 3B
  61. 46.6 Marin 32B
  62. 46.5 NorMistral 7B
  63. 45.3 GPT-SW3 20B
  64. 45.1 Viking 13B
  65. 44.0 GPT-SW3 6.7B
  66. 42.6 Qwen3.5-2B-Base
  67. 42.1 Viking 7B
  68. 42.1 Llama 3.2 3B
  69. 42.1 Nemotron Nano 12B v2
  70. 41.7 Mistral 7B v0.1
  71. 41.2 NorMistral 7B (scratch)
  72. 40.1 NB-Llama 3.1 8B
  73. 39.7 Bielik 4.5B v3
  74. 39.6 Qwen3-1.7B-Base
  75. 39.4 OLMo2 13B
  76. 39.2 Marin 8B
  77. 38.1 NB-Llama 3.2 3B
  78. 37.5 GPT-SW3 1.3B
  79. 36.6 OLMo3 7B
  80. 36.4 Poro 34B
  81. 36.4 OLMo3 7B (stage 1)
  82. 35.6 Trinity Nano 6B (1B active)
  83. 35.3 OLMo2 13B (stage 1)
  84. 35.3 NB-GPT-J 6B v2
  85. 33.9 Qwen3.5-0.8B-Base
  86. 32.1 NB-GPT-J 6B
  87. 32.0 NorBERT4 1B
  88. 31.1 Gemma3 1B (pretrained)
  89. 30.3 EuroLLM 1.7B
  90. 29.4 DFM 7B
  91. 29.4 TildeOpen 30B
  92. 28.9 NB-Llama 3.2 1B
  93. 28.7 Bielik 1.5B v3
  94. 27.6 Llama 3.2 1B
  95. 20.5 OLMo 2 1B Instruct
  96. 20.4 NorwAI NorGPT 3B
  97. 8.93 NorwAI NorLlama 3B
  1. 1 DeepSeek-V4-Flash-Base 5-shot 68.2
  2. 2 Gemma4 31B (pretrained) 5-shot 67.2
  3. 3 Nemotron 4 340B 5-shot 64.8
  4. 4 Gemma3 27B (pretrained) 5-shot 64.0
  5. 5 Gemma4 26B (4B active) (pretrained) 5-shot 63.9
  6. 6 Qwen3.5-35B-A3B-Base 5-shot 62.6
  7. 7 Llama4 Scout 109B (17B active) 5-shot 61.4
  8. 8 GLM-4.5 Air (12B active) 5-shot 61.4
  9. 9 Llama 3.1-70B 5-shot 61.1
  10. 10 TildeOpen 30B 64k 5-shot 61.1
  11. 11 Gemma4 12B (pretrained) 5-shot 60.9
  12. 12 Gemma3 12B (pretrained) 5-shot 60.9
  13. 13 NorOLMo 13B (stage 2, new mix) 5-shot 59.6
  14. 14 NorwAI Mixtral 8x7B (47B) 5-shot 59.4
  15. 15 Mistral 24B 5-shot 59.2
  16. 16 GLM-4 32B 5-shot 58.5
  17. 17 Apertus 70B 5-shot 58.3
  18. 18 Gemma4 E4B (8B) (pretrained) 5-shot 58.2
  19. 19 Qwen3.5-9B-Base 5-shot 58.0
  20. 20 Mistral 24B (v2501) 5-shot 57.9
  21. 21 NorMistral 11B Long 5-shot 57.8
  22. 22 Prelude 9B 5-shot 57.7
  23. 23 NorMistral 11B 5-shot 57.4
  24. 24 Nemotron 3 Super 120B (12B active) 5-shot 57.3
  25. 25 Qwen2.5-72B 5-shot 56.9
  26. 26 Ministral 3 14B 5-shot 56.3
  27. 27 Mixtral 8x22B (141B) 5-shot 56.1
  28. 28 NorOLMo 13B 5-shot 55.7
  29. 29 EuroLLM 1.5 9B 5-shot 55.5
  30. 30 NorOLMo 13B (stage 2) 5-shot 55.4
  31. 31 Bielik 11B v3 5-shot 55.4
  32. 32 Qwen3-14B-Base 5-shot 55.2
  33. 33 Apertus 8B Instruct 5-shot 54.7
  34. 34 EuroLLM 22B 5-shot 54.7
  35. 35 Qwen2.5-32B 5-shot 54.6
  36. 36 Apertus 8B (stage 5) 5-shot 54.4
  37. 37 AI Sweden Llama3 8B 5-shot 53.6
  38. 38 Ministral3 8B 5-shot 53.2
  39. 39 Mistral Nemo 12B 5-shot 52.5
  40. 40 Qwen3-8B-Base 5-shot 52.4
  41. 41 NorOLMo 13B (stage 1) 5-shot 51.7
  42. 42 Apertus 8B (stage 4) 5-shot 51.5
  43. 43 EuroLLM 9B 5-shot 50.8
  44. 44 Mixtral 8x7B (47B) 5-shot 50.6
  45. 45 Gemma3 4B (pretrained) 5-shot 50.6
  46. 46 Nemotron 3 Nano 30B (3B active) 5-shot 50.5
  47. 47 Viking 33B 5-shot 50.2
  48. 48 Apertus 8B (stage 3) 5-shot 49.5
  49. 49 Llama 3.1-8B 5-shot 49.5
  50. 50 Tiny Aya 3B 5-shot 49.4
  51. 51 Apertus 8B (stage 2) 5-shot 49.3
  52. 52 DeepSeek V3 671B (37B active) 5-shot 49.0
  53. 53 Gemma4 E2B (5B) (pretrained) 5-shot 48.6
  54. 54 NorwAI Mistral 7B 5-shot 48.6
  55. 55 Apertus 8B (stage 1) 5-shot 48.4
  56. 56 OLMo3 32B 5-shot 48.4
  57. 57 GPT-SW3 40B 5-shot 48.3
  58. 58 Qwen3-4B-Base 5-shot 48.0
  59. 59 Trinity Mini 26B (3B active) 5-shot 47.6
  60. 60 Ministral3 3B 5-shot 47.0
  61. 61 Marin 32B 5-shot 46.6
  62. 62 NorMistral 7B 5-shot 46.5
  63. 63 GPT-SW3 20B 5-shot 45.3
  64. 64 Viking 13B 5-shot 45.1
  65. 65 GPT-SW3 6.7B 5-shot 44.0
  66. 66 Qwen3.5-2B-Base 5-shot 42.6
  67. 67 Viking 7B 5-shot 42.1
  68. 68 Llama 3.2 3B 5-shot 42.1
  69. 69 Nemotron Nano 12B v2 5-shot 42.1
  70. 70 Mistral 7B v0.1 5-shot 41.7
  71. 71 NorMistral 7B (scratch) 5-shot 41.2
  72. 72 NB-Llama 3.1 8B 5-shot 40.1
  73. 73 Bielik 4.5B v3 5-shot 39.7
  74. 74 Qwen3-1.7B-Base 5-shot 39.6
  75. 75 OLMo2 13B 5-shot 39.4
  76. 76 Marin 8B 5-shot 39.2
  77. 77 NB-Llama 3.2 3B 5-shot 38.1
  78. 78 GPT-SW3 1.3B 5-shot 37.5
  79. 79 OLMo3 7B 5-shot 36.6
  80. 80 Poro 34B 5-shot 36.4
  81. 81 OLMo3 7B (stage 1) 5-shot 36.4
  82. 82 Trinity Nano 6B (1B active) 5-shot 35.6
  83. 83 OLMo2 13B (stage 1) 5-shot 35.3
  84. 84 NB-GPT-J 6B v2 5-shot 35.3
  85. 85 Qwen3.5-0.8B-Base 5-shot 33.9
  86. 86 NB-GPT-J 6B 5-shot 32.1
  87. 87 NorBERT4 1B 5-shot 32.0
  88. 88 Gemma3 1B (pretrained) 5-shot 31.1
  89. 89 EuroLLM 1.7B 5-shot 30.3
  90. 90 DFM 7B 1-shot 29.4
  91. 91 TildeOpen 30B 1-shot 29.4
  92. 92 NB-Llama 3.2 1B 5-shot 28.9
  93. 93 Bielik 1.5B v3 5-shot 28.7
  94. 94 Llama 3.2 1B 5-shot 27.6
  95. 95 OLMo 2 1B Instruct 5-shot 20.5
  96. 96 NorwAI NorGPT 3B 0-shot 20.4
  97. 97 NorwAI NorLlama 3B 0-shot 8.93

Source: Official NorEval leaderboard

#EntryScore ±SettingDateSource
1 DeepSeek-V4-Flash-Base 68.2 66.9 to 69.1 5-shot Apr 2026 Official leaderboard
2 Gemma4 31B (pretrained) 31B 67.2 65.9 to 68.2 5-shot Official leaderboard
3 Nemotron 4 340B 340B 64.8 63.4 to 65.8 5-shot Official leaderboard
4 Gemma3 27B (pretrained) 27.4B 64.0 62.5 to 64.9 5-shot Official leaderboard
5 Gemma4 26B (4B active) (pretrained) 26.5B 63.9 62.5 to 64.9 5-shot Official leaderboard
6 Qwen3.5-35B-A3B-Base 62.6 61.2 to 63.6 5-shot Feb 2026 Official leaderboard
7 Llama4 Scout 109B (17B active) 108.7B 61.4 60 to 62.4 5-shot Official leaderboard
8 GLM-4.5 Air (12B active) 110.5B 61.4 59.9 to 62.4 5-shot Official leaderboard
9 Llama 3.1-70B 61.1 59.7 to 62.1 5-shot Jul 2024 Official leaderboard
10 TildeOpen 30B 64k 30.7B 61.1 59.6 to 62.1 5-shot Official leaderboard
11 Gemma4 12B (pretrained) 12B 60.9 59.4 to 61.9 5-shot Official leaderboard
12 Gemma3 12B (pretrained) 12.2B 60.9 59.4 to 61.9 5-shot Official leaderboard
13 NorOLMo 13B (stage 2, new mix) 13.7B 59.6 58.1 to 60.6 5-shot Official leaderboard
14 NorwAI Mixtral 8x7B (47B) 47B 59.4 57.9 to 60.5 5-shot Official leaderboard
15 Mistral 24B 23.6B 59.2 57.7 to 60.2 5-shot Official leaderboard
16 GLM-4 32B 32.6B 58.5 57 to 59.5 5-shot Official leaderboard
17 Apertus 70B 58.3 56.8 to 59.4 5-shot Sep 2025 Official leaderboard
18 Gemma4 E4B (8B) (pretrained) 8B 58.2 56.7 to 59.3 5-shot Official leaderboard
19 Qwen3.5-9B-Base 58.0 56.5 to 59 5-shot Feb 2026 Official leaderboard
20 Mistral 24B (v2501) 23.6B 57.9 56.4 to 58.9 5-shot Official leaderboard
21 NorMistral 11B Long 11.4B 57.8 56.2 to 58.8 5-shot Official leaderboard
22 Prelude 9B 9.1B 57.7 56.2 to 58.7 5-shot Official leaderboard
23 NorMistral 11B 11.4B 57.4 55.9 to 58.4 5-shot Official leaderboard
24 Nemotron 3 Super 120B (12B active) 123.6B 57.3 56 to 58.1 5-shot Official leaderboard
25 Qwen2.5-72B 56.9 55.5 to 58 5-shot Sep 2024 Official leaderboard
26 Ministral 3 14B 56.3 54.8 to 57.3 5-shot Dec 2025 Official leaderboard
27 Mixtral 8x22B (141B) 140.6B 56.1 54.6 to 57.1 5-shot Official leaderboard
28 NorOLMo 13B 13.7B 55.7 54.2 to 56.9 5-shot Official leaderboard
29 EuroLLM 1.5 9B 9.2B 55.5 54 to 56.6 5-shot Official leaderboard
30 NorOLMo 13B (stage 2) 13.7B 55.4 53.9 to 56.6 5-shot Official leaderboard
31 Bielik 11B v3 11.2B 55.4 53.8 to 56.5 5-shot Official leaderboard
32 Qwen3-14B-Base 55.2 53.8 to 56.3 5-shot Apr 2025 Official leaderboard
33 Apertus 8B Instruct 54.7 53.2 to 55.9 5-shot Sep 2025 Official leaderboard
34 EuroLLM 22B 22.6B 54.7 53.2 to 55.8 5-shot Official leaderboard
35 Qwen2.5-32B 54.6 53.2 to 55.7 5-shot Sep 2024 Official leaderboard
36 Apertus 8B (stage 5) 8.1B 54.4 52.9 to 55.5 5-shot Official leaderboard
37 AI Sweden Llama3 8B 8B 53.6 52.1 to 54.7 5-shot Official leaderboard
38 Ministral3 8B 8.4B 53.2 51.8 to 54.2 5-shot Official leaderboard
39 Mistral Nemo 12B 12.2B 52.5 51 to 53.6 5-shot Official leaderboard
40 Qwen3-8B-Base 52.4 50.9 to 53.5 5-shot Apr 2025 Official leaderboard
41 NorOLMo 13B (stage 1) 13.7B 51.7 50.1 to 52.8 5-shot Official leaderboard
42 Apertus 8B (stage 4) 8.1B 51.5 50 to 52.6 5-shot Official leaderboard
43 EuroLLM 9B 9.2B 50.8 49.3 to 52 5-shot Official leaderboard
44 Mixtral 8x7B (47B) 46.7B 50.6 49.1 to 51.7 5-shot Official leaderboard
45 Gemma3 4B (pretrained) 4.3B 50.6 49 to 51.7 5-shot Official leaderboard
46 Nemotron 3 Nano 30B (3B active) 31.6B 50.5 49.1 to 51.6 5-shot Official leaderboard
47 Viking 33B 33.1B 50.2 48.7 to 51.3 5-shot Official leaderboard
48 Apertus 8B (stage 3) 8.1B 49.5 48 to 50.7 5-shot Official leaderboard
49 Llama 3.1-8B 49.5 48 to 50.7 5-shot Jul 2024 Official leaderboard
50 Tiny Aya 3B 3.3B 49.4 47.9 to 50.4 5-shot Official leaderboard
51 Apertus 8B (stage 2) 8.1B 49.3 47.7 to 50.4 5-shot Official leaderboard
52 DeepSeek V3 671B (37B active) 671B 49.0 47.5 to 50 5-shot Official leaderboard
53 Gemma4 E2B (5B) (pretrained) 5.1B 48.6 47.1 to 49.8 5-shot Official leaderboard
54 NorwAI Mistral 7B 7.5B 48.6 47.3 to 49.6 5-shot Official leaderboard
55 Apertus 8B (stage 1) 8.1B 48.4 46.9 to 49.6 5-shot Official leaderboard
56 OLMo3 32B 32.3B 48.4 46.9 to 49.5 5-shot Official leaderboard
57 GPT-SW3 40B 40B 48.3 46.8 to 49.4 5-shot Official leaderboard
58 Qwen3-4B-Base 48.0 46.5 to 49.2 5-shot Apr 2025 Official leaderboard
59 Trinity Mini 26B (3B active) 26.1B 47.6 46 to 48.7 5-shot Official leaderboard
60 Ministral3 3B 3.4B 47.0 45.5 to 48.1 5-shot Official leaderboard
61 Marin 32B 32.5B 46.6 45.1 to 47.7 5-shot Official leaderboard
62 NorMistral 7B 7.2B 46.5 44.9 to 47.7 5-shot Official leaderboard
63 GPT-SW3 20B 20.8B 45.3 43.9 to 46.4 5-shot Official leaderboard
64 Viking 13B 14B 45.1 43.6 to 46.3 5-shot Official leaderboard
65 GPT-SW3 6.7B 6.7B 44.0 42.5 to 45.2 5-shot Official leaderboard
66 Qwen3.5-2B-Base 42.6 41 to 43.8 5-shot Feb 2026 Official leaderboard
67 Viking 7B 7.6B 42.1 40.6 to 43.3 5-shot Official leaderboard
68 Llama 3.2 3B 42.1 40.5 to 43.3 5-shot Sep 2024 Official leaderboard
69 Nemotron Nano 12B v2 12.3B 42.1 40.5 to 43.3 5-shot Official leaderboard
70 Mistral 7B v0.1 41.7 40.2 to 42.8 5-shot Sep 2023 Official leaderboard
71 NorMistral 7B (scratch) 7B 41.2 39.8 to 42.3 5-shot Official leaderboard
72 NB-Llama 3.1 8B 8B 40.1 38.7 to 41.1 5-shot Official leaderboard
73 Bielik 4.5B v3 4.8B 39.7 38.2 to 40.8 5-shot Official leaderboard
74 Qwen3-1.7B-Base 1.7B 39.6 38 to 40.7 5-shot Official leaderboard
75 OLMo2 13B 13.7B 39.4 37.9 to 40.6 5-shot Official leaderboard
76 Marin 8B 8B 39.2 37.7 to 40.4 5-shot Official leaderboard
77 NB-Llama 3.2 3B 3.2B 38.1 36.7 to 39.3 5-shot Official leaderboard
78 GPT-SW3 1.3B 1.4B 37.5 36.1 to 38.6 5-shot Official leaderboard
79 OLMo3 7B 7.3B 36.6 35.1 to 37.8 5-shot Official leaderboard
80 Poro 34B 34.2B 36.4 34.9 to 37.6 5-shot Official leaderboard
81 OLMo3 7B (stage 1) 7.3B 36.4 34.8 to 37.6 5-shot Official leaderboard
82 Trinity Nano 6B (1B active) 6.1B 35.6 34.1 to 36.7 5-shot Official leaderboard
83 OLMo2 13B (stage 1) 13.7B 35.3 33.8 to 36.4 5-shot Official leaderboard
84 NB-GPT-J 6B v2 6.1B 35.3 33.9 to 36.3 5-shot Official leaderboard
85 Qwen3.5-0.8B-Base 33.9 32.3 to 35.1 5-shot Feb 2026 Official leaderboard
86 NB-GPT-J 6B 6.1B 32.1 30.8 to 33.1 5-shot Official leaderboard
87 NorBERT4 1B 1B 32.0 30.6 to 33.1 5-shot Official leaderboard
88 Gemma3 1B (pretrained) 1B 31.1 29.6 to 32.2 5-shot Official leaderboard
89 EuroLLM 1.7B 1.7B 30.3 28.8 to 31.4 5-shot Official leaderboard
90 DFM 7B 7B 29.4 28 to 30.5 1-shot Official leaderboard
91 TildeOpen 30B 30.7B 29.4 28.1 to 30.4 1-shot Official leaderboard
92 NB-Llama 3.2 1B 1.2B 28.9 27.7 to 29.8 5-shot Official leaderboard
93 Bielik 1.5B v3 1.6B 28.7 27.2 to 29.9 5-shot Official leaderboard
94 Llama 3.2 1B 27.6 26.1 to 28.8 5-shot Sep 2024 Official leaderboard
95 OLMo 2 1B Instruct 20.5 19.1 to 21.6 5-shot Apr 2025 Official leaderboard
96 NorwAI NorGPT 3B 3.1B 20.4 19.3 to 21.4 0-shot Official leaderboard
97 NorwAI NorLlama 3B 2.8B 8.93 7.9 to 9.8 0-shot Official leaderboard
#ModelScore ±SettingDateSource
1 DeepSeek-V4-Flash-Base 68.2 66.9 to 69.1 5-shot Apr 2026 Official leaderboard
2 Qwen3.5-35B-A3B-Base 62.6 61.2 to 63.6 5-shot Feb 2026 Official leaderboard
3 Llama 3.1-70B 61.1 59.7 to 62.1 5-shot Jul 2024 Official leaderboard
4 Apertus 70B 58.3 56.8 to 59.4 5-shot Sep 2025 Official leaderboard
5 Qwen3.5-9B-Base 58.0 56.5 to 59 5-shot Feb 2026 Official leaderboard
6 Qwen2.5-72B 56.9 55.5 to 58 5-shot Sep 2024 Official leaderboard
7 Ministral 3 14B 56.3 54.8 to 57.3 5-shot Dec 2025 Official leaderboard
8 Qwen3-14B-Base 55.2 53.8 to 56.3 5-shot Apr 2025 Official leaderboard
9 Apertus 8B Instruct 54.7 53.2 to 55.9 5-shot Sep 2025 Official leaderboard
10 Qwen2.5-32B 54.6 53.2 to 55.7 5-shot Sep 2024 Official leaderboard
11 Qwen3-8B-Base 52.4 50.9 to 53.5 5-shot Apr 2025 Official leaderboard
12 Llama 3.1-8B 49.5 48 to 50.7 5-shot Jul 2024 Official leaderboard
13 Qwen3-4B-Base 48.0 46.5 to 49.2 5-shot Apr 2025 Official leaderboard
14 Qwen3.5-2B-Base 42.6 41 to 43.8 5-shot Feb 2026 Official leaderboard
15 Llama 3.2 3B 42.1 40.5 to 43.3 5-shot Sep 2024 Official leaderboard
16 Mistral 7B v0.1 41.7 40.2 to 42.8 5-shot Sep 2023 Official leaderboard
17 Qwen3.5-0.8B-Base 33.9 32.3 to 35.1 5-shot Feb 2026 Official leaderboard
18 Llama 3.2 1B 27.6 26.1 to 28.8 5-shot Sep 2024 Official leaderboard
19 OLMo 2 1B Instruct 20.5 19.1 to 21.6 5-shot Apr 2025 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
DeepSeek-V4-Flash-Base
0-shot 1-shot 5-shot
68.2
Qwen3.5-35B-A3B-Base
0-shot 1-shot 5-shot
62.6
Llama 3.1-70B
0-shot 1-shot 5-shot
61.1
Apertus 70B
0-shot 1-shot 5-shot
58.3
Qwen3.5-9B-Base
0-shot 1-shot 5-shot
58.0
Qwen2.5-72B
0-shot 1-shot 5-shot
56.9
Ministral 3 14B
0-shot 1-shot 5-shot
56.3
Qwen3-14B-Base
0-shot 1-shot 5-shot
55.2
Apertus 8B Instruct
0-shot 1-shot 5-shot
54.7
Qwen2.5-32B
0-shot 1-shot 5-shot
54.6
Qwen3-8B-Base
0-shot 1-shot 5-shot
52.4
Llama 3.1-8B
0-shot 1-shot 5-shot
49.5

Source: Official NorEval leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
DeepSeek-V4-Flash-Base 5-shot 68.2
DeepSeek-V4-Flash-Base 1-shot 63.3
Qwen3.5-35B-A3B-Base 5-shot 62.6
Llama 3.1-70B 5-shot 61.1
Apertus 70B 5-shot 58.3
Qwen3.5-9B-Base 5-shot 58.0
Qwen2.5-72B 5-shot 56.9
Qwen3.5-35B-A3B-Base 1-shot 56.8
Ministral 3 14B 5-shot 56.3
Llama 3.1-70B 1-shot 55.9
Qwen3-14B-Base 5-shot 55.2
Apertus 8B Instruct 5-shot 54.7
Qwen2.5-32B 5-shot 54.6
Qwen2.5-72B 1-shot 53.0
DeepSeek-V4-Flash-Base 0-shot 52.8
Qwen3-8B-Base 5-shot 52.4
Qwen3.5-9B-Base 1-shot 51.5
Apertus 70B 1-shot 51.3
Qwen2.5-32B 1-shot 50.3
Qwen3-14B-Base 1-shot 50.2
Llama 3.1-8B 5-shot 49.5
Ministral 3 14B 1-shot 49.4
Qwen2.5-72B 0-shot 49.0
Llama 3.1-70B 0-shot 48.9
Apertus 8B Instruct 1-shot 48.6
Qwen3.5-35B-A3B-Base 0-shot 48.1
Qwen3-4B-Base 5-shot 48.0
Qwen3-8B-Base 1-shot 47.1
Qwen2.5-32B 0-shot 45.2
Qwen3-14B-Base 0-shot 44.7
Qwen3-4B-Base 1-shot 43.8
Qwen3.5-2B-Base 5-shot 42.6
Llama 3.1-8B 1-shot 42.6
Llama 3.2 3B 5-shot 42.1
Apertus 70B 0-shot 41.9
Mistral 7B v0.1 5-shot 41.7
Qwen3.5-9B-Base 0-shot 41.4
Ministral 3 14B 0-shot 41.0
Qwen3-8B-Base 0-shot 40.9
Apertus 8B Instruct 0-shot 38.9
Qwen3-4B-Base 0-shot 38.2
Mistral 7B v0.1 1-shot 36.3
Qwen3.5-2B-Base 1-shot 36.0
Llama 3.2 3B 1-shot 34.9
Qwen3.5-0.8B-Base 5-shot 33.9
Llama 3.1-8B 0-shot 33.3
Llama 3.2 3B 0-shot 27.8
Llama 3.2 1B 5-shot 27.6
Qwen3.5-0.8B-Base 1-shot 26.5
Mistral 7B v0.1 0-shot 25.8
Qwen3.5-2B-Base 0-shot 25.5
Llama 3.2 1B 1-shot 23.4
OLMo 2 1B Instruct 5-shot 20.5
Llama 3.2 1B 0-shot 19.5
Qwen3.5-0.8B-Base 0-shot 17.2
OLMo 2 1B Instruct 1-shot 15.8
OLMo 2 1B Instruct 0-shot 12.3

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
DeepSeek-V4-Flash-Base
0-shot 1-shot 5-shot
68.2
Qwen3.5-35B-A3B-Base
0-shot 1-shot 5-shot
62.6
Llama 3.1-70B
0-shot 1-shot 5-shot
61.1
Apertus 70B
0-shot 1-shot 5-shot
58.3
Qwen3.5-9B-Base
0-shot 1-shot 5-shot
58.0
Qwen2.5-72B
0-shot 1-shot 5-shot
56.9
Ministral 3 14B
0-shot 1-shot 5-shot
56.3
Qwen3-14B-Base
0-shot 1-shot 5-shot
55.2
Apertus 8B Instruct
0-shot 1-shot 5-shot
54.7
Qwen2.5-32B
0-shot 1-shot 5-shot
54.6
Qwen3-8B-Base
0-shot 1-shot 5-shot
52.4
Llama 3.1-8B
0-shot 1-shot 5-shot
49.5

Source: Official NorEval leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
DeepSeek-V4-Flash-Base 5-shot 68.2
DeepSeek-V4-Flash-Base 1-shot 63.3
Qwen3.5-35B-A3B-Base 5-shot 62.6
Llama 3.1-70B 5-shot 61.1
Apertus 70B 5-shot 58.3
Qwen3.5-9B-Base 5-shot 58.0
Qwen2.5-72B 5-shot 56.9
Qwen3.5-35B-A3B-Base 1-shot 56.8
Ministral 3 14B 5-shot 56.3
Llama 3.1-70B 1-shot 55.9
Qwen3-14B-Base 5-shot 55.2
Apertus 8B Instruct 5-shot 54.7
Qwen2.5-32B 5-shot 54.6
Qwen2.5-72B 1-shot 53.0
DeepSeek-V4-Flash-Base 0-shot 52.8
Qwen3-8B-Base 5-shot 52.4
Qwen3.5-9B-Base 1-shot 51.5
Apertus 70B 1-shot 51.3
Qwen2.5-32B 1-shot 50.3
Qwen3-14B-Base 1-shot 50.2
Llama 3.1-8B 5-shot 49.5
Ministral 3 14B 1-shot 49.4
Qwen2.5-72B 0-shot 49.0
Llama 3.1-70B 0-shot 48.9
Apertus 8B Instruct 1-shot 48.6
Qwen3.5-35B-A3B-Base 0-shot 48.1
Qwen3-4B-Base 5-shot 48.0
Qwen3-8B-Base 1-shot 47.1
Qwen2.5-32B 0-shot 45.2
Qwen3-14B-Base 0-shot 44.7
Qwen3-4B-Base 1-shot 43.8
Qwen3.5-2B-Base 5-shot 42.6
Llama 3.1-8B 1-shot 42.6
Llama 3.2 3B 5-shot 42.1
Apertus 70B 0-shot 41.9
Mistral 7B v0.1 5-shot 41.7
Qwen3.5-9B-Base 0-shot 41.4
Ministral 3 14B 0-shot 41.0
Qwen3-8B-Base 0-shot 40.9
Apertus 8B Instruct 0-shot 38.9
Qwen3-4B-Base 0-shot 38.2
Mistral 7B v0.1 1-shot 36.3
Qwen3.5-2B-Base 1-shot 36.0
Llama 3.2 3B 1-shot 34.9
Qwen3.5-0.8B-Base 5-shot 33.9
Llama 3.1-8B 0-shot 33.3
Llama 3.2 3B 0-shot 27.8
Llama 3.2 1B 5-shot 27.6
Qwen3.5-0.8B-Base 1-shot 26.5
Mistral 7B v0.1 0-shot 25.8
Qwen3.5-2B-Base 0-shot 25.5
Llama 3.2 1B 1-shot 23.4
OLMo 2 1B Instruct 5-shot 20.5
Llama 3.2 1B 0-shot 19.5
Qwen3.5-0.8B-Base 0-shot 17.2
OLMo 2 1B Instruct 1-shot 15.8
OLMo 2 1B Instruct 0-shot 12.3

3 models have held number one on the official leaderboard since its release in April 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Meta Qwen DeepSeek
Own leaderboard Not tested since Apr 2026
DeepSeek-V4-Flash-Base 68.2

Source: Official NorEval leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard
0.00 20.0 40.0 60.0 80.0 2024 2025 2026 0.00 20.0 40.0 60.0 80.0 2024 2025 2026

Source: Official NorEval leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Multilingual

All 545 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Claude Opus 5 95.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%

Data audit

8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 7 months ago, in April 2026, for DeepSeek-V4-Flash-Base. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
  • Pass Independent runs Scores from the official leaderboard (97 entries).
  • Pass Models scored 19 models we track, plus 78 other entries on its leaderboard.
  • Warning Newest score The newest came 7 months ago, in April 2026, for DeepSeek-V4-Flash-Base.
  • Note Frontier models None of the top 10 has a score. The strongest tested is Qwen2.5-72B (#166).
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on ltgoslo.github.io: 57 rows for models we track and 78 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check