OrangeSum
Models summarize French news articles from Orange Actu. The dataset pairs articles with summaries written by journalists.
EuroEval compares generated summaries with references using chrF3++.
- Made by
- Moussa Kamal Eddine and colleagues
- Released
- October 2020
- Kind
- Reading in many languages
- Level
- School
- Scored
- Checked automatically
- Answers
- Open-ended
- Tasks
- All public
- Measures
- Capability
- Language
- French
- Interaction
- Single turn
- Items from
- Real work and use, Written by experts
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
-
Qwen3-32B #135 on the capability index
- Last model added
- March 2026
- Scores refreshed
- November 20th, 2026
Mistral Small 3.1 leads with 38.5% among models released by March 2026 Mistral Small 3.1 leads with 38.5% among models released by March 2026
Best score per entry from EuroEval’s leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
The newest model with a score here came out in March 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 38.5
Mistral Small 3.1
- 38.5 deepcogito/cogito-v2-preview-llama-70B
- 38.5 occiglot/occiglot-7b-eu5-instruct
- 38.5 utter-project/EuroLLM-9B-Instruct
- 38.4
Llama 3.3 70B Instruct
- 38.1 deepcogito/cogito-v1-preview-qwen-32B
- 38.0
Qwen2.5-72B
- 38.0
Qwen3-32B
- 37.9 meta-llama/Llama-3.1-8B-Instruct
- 37.9
Gemma 3 27B
- 37.9
Gemma 3 12B
- 37.9
Qwen3-14B
- 37.9
Mistral Small 3.2
- 37.8 HuggingFaceTB/SmolLM3-3B
- 37.8 deepcogito/cogito-v1-preview-llama-8B
- 37.8 granite-4.0-micro
- 37.5
Qwen3-8B
- 37.5 ibm-granite/granite-4.0-h-micro
- 37.4
Gemma 3n E4B
- 37.4 deepcogito/cogito-v1-preview-qwen-14B
- 37.4
Qwen3-4B
- 37.1 meta-llama/Llama-3.2-3B-Instruct
- 36.9
Gemma 3 4B
- 36.9 deepcogito/cogito-v1-preview-llama-3B
- 36.8
Qwen3-4B-Thinking-2507
- 36.5
Qwen3-1.7B
- 36.4
Voxtral Small
- 36.2
Olmo 3.1 32B Instruct
- 36.1 allenai/Olmo-3.1-32B-Instruct-DPO
- 36.0 utter-project/EuroLLM-22B-Instruct-2512
- 35.4 utter-project/EuroMoE-2.6B-A0.6B-Instruct-Preview
- 35.4 mii-llm/nesso-4B
- 35.3 yulan-team/YuLan-Mini-Instruct
- 35.2 mistralai/Ministral-3-8B-Reasoning-2512
- 35.2 LFM2-1.2B
- 35.1
Ministral 3 3B Reasoning
- 35.0 LiquidAI/LFM2-700M
- 34.8
Qwen3-0.6B
- 34.6 gpt-oss-20b
- 34.5 allenai/Olmo-3-7B-Instruct
- 34.4 allenai/Olmo-3-7B-Instruct-DPO
- 34.3 speakleash/Bielik-1.5B-v3.0-Instruct
- 34.3
Nemotron 3 Nano 4B
- 33.8 tiiuae/Falcon-H1-Tiny-Multilingual-100M-Instruct
- 33.1 LiquidAI/LFM2-350M
- 32.3 mistralai/Mistral-Small-3.1-24B-Base-2503
- 32.3
Gemma 3 270M
- 31.8
Gemma 3 27B PT
- 31.8 allenai/Olmo-3-1125-32B
- 31.8 HuggingFaceTB/SmolLM3-3B-Base
- 31.6
Llama 3.1-8B
- 31.5 allenai/Olmo-3-1025-7B
- 31.4 meta-llama/Llama-3.2-1B-Instruct
- 31.0
Gemma 3 1B
- 30.6
Llama 3.1-70B
- 29.7 allenai/Llama-3.1-Tulu-3-8B-SFT
- 29.3
Gemma 3 12B PT
- 29.1
Llama 3.2 1B
- 29.1 allenai/Olmo-3-7B-Instruct-SFT
- 28.8 BSC-LT/salamandra-7b-instruct
- 28.8 utter-project/EuroLLM-22B-2512
- 28.5 occiglot/occiglot-7b-eu5
- 28.5 utter-project/EuroLLM-9B
- 28.3 allenai/Olmo-3.1-32B-Instruct-SFT
- 27.9
Gemma 3 4B PT
- 27.9 PleIAs/Pleias-1.2b-Preview
- 26.8 HuggingFaceTB/SmolLM2-360M-Instruct
- 26.0 PleIAs/Pleias-3b-Preview
- 25.0
SmolLM2-360M
- 24.6 BSC-LT/salamandra-7b
- 24.3 tiiuae/Falcon-H1-Tiny-Multilingual-100M-Base
- 24.3 HuggingFaceTB/SmolLM2-135M-Instruct
- 24.0
Gemma 3 1B PT
- 23.7 utter-project/EuroMoE-2.6B-A0.6B-Preview
- 23.2 PleIAs/Pleias-350m-Preview
- 22.3
SmolLM2-135M
- 21.9
Gemma 3 270M Base
- 18.9 PleIAs/Baguettotron@refs/pr/6
- 10.6 PleIAs/Monad
- 1
Mistral Small 3.1 38.5%
- 2 deepcogito/cogito-v2-preview-llama-70B 38.5%
- 3 occiglot/occiglot-7b-eu5-instruct 38.5%
- 4 utter-project/EuroLLM-9B-Instruct 38.5%
- 5
Llama 3.3 70B Instruct 38.4%
- 6 deepcogito/cogito-v1-preview-qwen-32B 38.1%
- 7
Qwen2.5-72B 38%
- 8
Qwen3-32B no-thinking 38%
- 9 meta-llama/Llama-3.1-8B-Instruct 37.9%
- 10
Gemma 3 27B 37.9%
- 11
Gemma 3 12B 37.9%
- 12
Qwen3-14B no-thinking 37.9%
- 13
Mistral Small 3.2 37.9%
- 14 HuggingFaceTB/SmolLM3-3B no-thinking 37.8%
- 15 deepcogito/cogito-v1-preview-llama-8B 37.8%
- 16 granite-4.0-micro 37.8%
- 17
Qwen3-8B no-thinking 37.5%
- 18 ibm-granite/granite-4.0-h-micro 37.5%
- 19
Gemma 3n E4B 37.4%
- 20 deepcogito/cogito-v1-preview-qwen-14B 37.4%
- 21
Qwen3-4B no-thinking 37.4%
- 22 meta-llama/Llama-3.2-3B-Instruct 37.1%
- 23
Gemma 3 4B 36.9%
- 24 deepcogito/cogito-v1-preview-llama-3B 36.9%
- 25
Qwen3-4B-Thinking-2507 36.8%
- 26
Qwen3-1.7B no-thinking 36.5%
- 27
Voxtral Small 36.4%
- 28
Olmo 3.1 32B Instruct 36.2%
- 29 allenai/Olmo-3.1-32B-Instruct-DPO 36.1%
- 30 utter-project/EuroLLM-22B-Instruct-2512 36%
- 31 utter-project/EuroMoE-2.6B-A0.6B-Instruct-Preview 35.4%
- 32 mii-llm/nesso-4B 35.4%
- 33 yulan-team/YuLan-Mini-Instruct 35.3%
- 34 mistralai/Ministral-3-8B-Reasoning-2512 35.2%
- 35 LFM2-1.2B 35.2%
- 36
Ministral 3 3B Reasoning 35.1%
- 37 LiquidAI/LFM2-700M 35%
- 38
Qwen3-0.6B no-thinking 34.8%
- 39 gpt-oss-20b low 34.6%
- 40 allenai/Olmo-3-7B-Instruct 34.5%
- 41 allenai/Olmo-3-7B-Instruct-DPO 34.4%
- 42 speakleash/Bielik-1.5B-v3.0-Instruct 34.3%
- 43
Nemotron 3 Nano 4B 34.3%
- 44 tiiuae/Falcon-H1-Tiny-Multilingual-100M-Instruct 33.8%
- 45 LiquidAI/LFM2-350M 33.1%
- 46 mistralai/Mistral-Small-3.1-24B-Base-2503 32.3%
- 47
Gemma 3 270M 32.3%
- 48
Gemma 3 27B PT 31.8%
- 49 allenai/Olmo-3-1125-32B 31.8%
- 50 HuggingFaceTB/SmolLM3-3B-Base 31.8%
- 51
Llama 3.1-8B 31.6%
- 52 allenai/Olmo-3-1025-7B 31.5%
- 53 meta-llama/Llama-3.2-1B-Instruct 31.4%
- 54
Gemma 3 1B 31%
- 55
Llama 3.1-70B 30.6%
- 56 allenai/Llama-3.1-Tulu-3-8B-SFT 29.7%
- 57
Gemma 3 12B PT 29.3%
- 58
Llama 3.2 1B 29.1%
- 59 allenai/Olmo-3-7B-Instruct-SFT 29.1%
- 60 BSC-LT/salamandra-7b-instruct 28.8%
- 61 utter-project/EuroLLM-22B-2512 28.8%
- 62 occiglot/occiglot-7b-eu5 28.5%
- 63 utter-project/EuroLLM-9B 28.5%
- 64 allenai/Olmo-3.1-32B-Instruct-SFT 28.3%
- 65
Gemma 3 4B PT 27.9%
- 66 PleIAs/Pleias-1.2b-Preview 27.9%
- 67 HuggingFaceTB/SmolLM2-360M-Instruct 26.8%
- 68 PleIAs/Pleias-3b-Preview 26%
- 69
SmolLM2-360M 25%
- 70 BSC-LT/salamandra-7b 24.6%
- 71 tiiuae/Falcon-H1-Tiny-Multilingual-100M-Base 24.3%
- 72 HuggingFaceTB/SmolLM2-135M-Instruct 24.3%
- 73
Gemma 3 1B PT 24%
- 74 utter-project/EuroMoE-2.6B-A0.6B-Preview 23.7%
- 75 PleIAs/Pleias-350m-Preview 23.2%
- 76
SmolLM2-135M 22.3%
- 77
Gemma 3 270M Base 21.9%
- 78 PleIAs/Baguettotron@refs/pr/6 no-thinking 18.9%
- 79 PleIAs/Monad 10.6%
Source: EuroEval’s leaderboard
- 38.5
Mistral Small 3.1
- 38.4
Llama 3.3 70B Instruct
- 38.0
Qwen2.5-72B
- 38.0
Qwen3-32B
- 37.9
Gemma 3 27B
- 37.9
Gemma 3 12B
- 37.9
Qwen3-14B
- 37.9
Mistral Small 3.2
- 37.8 granite-4.0-micro
- 37.5
Qwen3-8B
- 37.4
Gemma 3n E4B
- 37.4
Qwen3-4B
- 36.9
Gemma 3 4B
- 36.8
Qwen3-4B-Thinking-2507
- 36.5
Qwen3-1.7B
- 36.4
Voxtral Small
- 36.2
Olmo 3.1 32B Instruct
- 35.2 LFM2-1.2B
- 35.1
Ministral 3 3B Reasoning
- 34.8
Qwen3-0.6B
- 34.6 gpt-oss-20b
- 34.3
Nemotron 3 Nano 4B
- 32.3
Gemma 3 270M
- 31.8
Gemma 3 27B PT
- 31.6
Llama 3.1-8B
- 31.0
Gemma 3 1B
- 30.6
Llama 3.1-70B
- 29.3
Gemma 3 12B PT
- 29.1
Llama 3.2 1B
- 27.9
Gemma 3 4B PT
- 25.0
SmolLM2-360M
- 24.0
Gemma 3 1B PT
- 22.3
SmolLM2-135M
- 21.9
Gemma 3 270M Base
- 1
Mistral Small 3.1 38.5%
- 2
Llama 3.3 70B Instruct 38.4%
- 3
Qwen2.5-72B 38%
- 4
Qwen3-32B no-thinking 38%
- 5
Gemma 3 27B 37.9%
- 6
Gemma 3 12B 37.9%
- 7
Qwen3-14B no-thinking 37.9%
- 8
Mistral Small 3.2 37.9%
- 9 granite-4.0-micro 37.8%
- 10
Qwen3-8B no-thinking 37.5%
- 11
Gemma 3n E4B 37.4%
- 12
Qwen3-4B no-thinking 37.4%
- 13
Gemma 3 4B 36.9%
- 14
Qwen3-4B-Thinking-2507 36.8%
- 15
Qwen3-1.7B no-thinking 36.5%
- 16
Voxtral Small 36.4%
- 17
Olmo 3.1 32B Instruct 36.2%
- 18 LFM2-1.2B 35.2%
- 19
Ministral 3 3B Reasoning 35.1%
- 20
Qwen3-0.6B no-thinking 34.8%
- 21 gpt-oss-20b low 34.6%
- 22
Nemotron 3 Nano 4B 34.3%
- 23
Gemma 3 270M 32.3%
- 24
Gemma 3 27B PT 31.8%
- 25
Llama 3.1-8B 31.6%
- 26
Gemma 3 1B 31%
- 27
Llama 3.1-70B 30.6%
- 28
Gemma 3 12B PT 29.3%
- 29
Llama 3.2 1B 29.1%
- 30
Gemma 3 4B PT 27.9%
- 31
SmolLM2-360M 25%
- 32
Gemma 3 1B PT 24%
- 33
SmolLM2-135M 22.3%
- 34
Gemma 3 270M Base 21.9%
Source: EuroEval’s leaderboard
| # | Entry | Score ± | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | | 38.5% ±0.1 | Mar 2025 | hf.co | Official leaderboard | |
| 2 | deepcogito/cogito-v2-preview-llama-70B open weights | 38.5% ±0.1 | hf.co | Official leaderboard | ||
| 3 | occiglot/occiglot-7b-eu5-instruct open weights | 38.5% ±0.2 | hf.co | Official leaderboard | ||
| 4 | utter-project/EuroLLM-9B-Instruct open weights | 38.5% ±0.1 | hf.co | Official leaderboard | ||
| 5 | | 38.4% ±0.2 | Dec 2024 | hf.co | Official leaderboard | |
| 6 | deepcogito/cogito-v1-preview-qwen-32B open weights | 38.1% ±0.1 | hf.co | Official leaderboard | ||
| 7 | | 38% ±0.1 | Sep 2024 | hf.co | Official leaderboard | |
| 8 | | 38% ±0.1 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 9 | meta-llama/Llama-3.1-8B-Instruct open weights | 37.9% ±0.2 | hf.co | Official leaderboard | ||
| 10 | | 37.9% ±0.1 | Mar 2025 | hf.co | Official leaderboard | |
| 11 | | 37.9% ±0.1 | Mar 2025 | hf.co | Official leaderboard | |
| 12 | | 37.9% ±0.1 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 13 | | 37.9% ±0.2 | Jun 2025 | hf.co | Official leaderboard | |
| 14 | HuggingFaceTB/SmolLM3-3B open weights | 37.8% ±0.1 | no-thinking | hf.co | Official leaderboard | |
| 15 | deepcogito/cogito-v1-preview-llama-8B open weights | 37.8% ±0.1 | hf.co | Official leaderboard | ||
| 16 | granite-4.0-micro | 37.8% ±0.2 | Oct 2025 | hf.co | Official leaderboard | |
| 17 | | 37.5% ±0.2 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 18 | ibm-granite/granite-4.0-h-micro open weights | 37.5% ±0.1 | hf.co | Official leaderboard | ||
| 19 | | 37.4% ±0.1 | Jun 2025 | hf.co | Official leaderboard | |
| 20 | deepcogito/cogito-v1-preview-qwen-14B open weights | 37.4% ±0.2 | hf.co | Official leaderboard | ||
| 21 | | 37.4% ±0.1 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 22 | meta-llama/Llama-3.2-3B-Instruct open weights | 37.1% ±0.1 | hf.co | Official leaderboard | ||
| 23 | | 36.9% ±0.1 | Mar 2025 | hf.co | Official leaderboard | |
| 24 | deepcogito/cogito-v1-preview-llama-3B open weights | 36.9% ±0.1 | hf.co | Official leaderboard | ||
| 25 | | 36.8% ±0.2 | Aug 2025 | hf.co | Official leaderboard | |
| 26 | | 36.5% ±0.1 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 27 | | 36.4% ±0.8 | Jul 2025 | hf.co | Official leaderboard | |
| 28 | | 36.2% ±0.2 | Dec 2025 | hf.co | Official leaderboard | |
| 29 | allenai/Olmo-3.1-32B-Instruct-DPO open weights | 36.1% ±0.1 | hf.co | Official leaderboard | ||
| 30 | utter-project/EuroLLM-22B-Instruct-2512 open weights | 36% ±0.9 | hf.co | Official leaderboard | ||
| 31 | utter-project/EuroMoE-2.6B-A0.6B-Instruct-Preview open weights | 35.4% ±0.3 | hf.co | Official leaderboard | ||
| 32 | mii-llm/nesso-4B open weights | 35.4% ±0.6 | hf.co | Official leaderboard | ||
| 33 | yulan-team/YuLan-Mini-Instruct open weights | 35.3% ±0.1 | hf.co | Official leaderboard | ||
| 34 | mistralai/Ministral-3-8B-Reasoning-2512 open weights | 35.2% ±1 | hf.co | Official leaderboard | ||
| 35 | LFM2-1.2B | 35.2% ±0.1 | Jul 2025 | hf.co | Official leaderboard | |
| 36 | | 35.1% ±0.1 | Dec 2025 | hf.co | Official leaderboard | |
| 37 | LiquidAI/LFM2-700M open weights | 35% ±0.2 | hf.co | Official leaderboard | ||
| 38 | | 34.8% ±0.2 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 39 | gpt-oss-20b | 34.6% ±0.1 | low | Aug 2025 | hf.co | Official leaderboard |
| 40 | allenai/Olmo-3-7B-Instruct open weights | 34.5% ±0.1 | hf.co | Official leaderboard | ||
| 41 | allenai/Olmo-3-7B-Instruct-DPO open weights | 34.4% ±0.1 | hf.co | Official leaderboard | ||
| 42 | speakleash/Bielik-1.5B-v3.0-Instruct open weights | 34.3% ±0.4 | hf.co | Official leaderboard | ||
| 43 | | 34.3% ±0.1 | Mar 2026 | hf.co | Official leaderboard | |
| 44 | tiiuae/Falcon-H1-Tiny-Multilingual-100M-Instruct open weights | 33.8% ±0.2 | hf.co | Official leaderboard | ||
| 45 | LiquidAI/LFM2-350M open weights | 33.1% ±0.1 | hf.co | Official leaderboard | ||
| 46 | mistralai/Mistral-Small-3.1-24B-Base-2503 open weights | 32.3% ±1.8 | hf.co | Official leaderboard | ||
| 47 | | 32.3% ±0.4 | Aug 2025 | hf.co | Official leaderboard | |
| 48 | | 31.8% ±0.9 | Mar 2025 | hf.co | Official leaderboard | |
| 49 | allenai/Olmo-3-1125-32B open weights | 31.8% ±0.8 | hf.co | Official leaderboard | ||
| 50 | HuggingFaceTB/SmolLM3-3B-Base open weights | 31.8% ±1.6 | hf.co | Official leaderboard | ||
| 51 | | 31.6% ±1.3 | Jul 2024 | hf.co | Official leaderboard | |
| 52 | allenai/Olmo-3-1025-7B open weights | 31.5% ±1.1 | hf.co | Official leaderboard | ||
| 53 | meta-llama/Llama-3.2-1B-Instruct open weights | 31.4% ±0.7 | hf.co | Official leaderboard | ||
| 54 | | 31% ±1.4 | Mar 2025 | hf.co | Official leaderboard | |
| 55 | | 30.6% ±1.6 | Jul 2024 | hf.co | Official leaderboard | |
| 56 | allenai/Llama-3.1-Tulu-3-8B-SFT open weights | 29.7% ±1.4 | hf.co | Official leaderboard | ||
| 57 | | 29.3% ±1.2 | Mar 2025 | hf.co | Official leaderboard | |
| 58 | | 29.1% ±0.6 | Sep 2024 | hf.co | Official leaderboard | |
| 59 | allenai/Olmo-3-7B-Instruct-SFT open weights | 29.1% ±1.2 | hf.co | Official leaderboard | ||
| 60 | BSC-LT/salamandra-7b-instruct open weights | 28.8% ±0.4 | hf.co | Official leaderboard | ||
| 61 | utter-project/EuroLLM-22B-2512 open weights | 28.8% ±1.2 | hf.co | Official leaderboard | ||
| 62 | occiglot/occiglot-7b-eu5 open weights | 28.5% ±1.5 | hf.co | Official leaderboard | ||
| 63 | utter-project/EuroLLM-9B open weights | 28.5% ±1.4 | hf.co | Official leaderboard | ||
| 64 | allenai/Olmo-3.1-32B-Instruct-SFT open weights | 28.3% ±0.9 | hf.co | Official leaderboard | ||
| 65 | | 27.9% ±1.2 | Mar 2025 | hf.co | Official leaderboard | |
| 66 | PleIAs/Pleias-1.2b-Preview open weights | 27.9% ±0.9 | hf.co | Official leaderboard | ||
| 67 | HuggingFaceTB/SmolLM2-360M-Instruct open weights | 26.8% ±1.8 | hf.co | Official leaderboard | ||
| 68 | PleIAs/Pleias-3b-Preview open weights | 26% ±1.5 | hf.co | Official leaderboard | ||
| 69 | | 25% ±1.1 | Oct 2024 | hf.co | Official leaderboard | |
| 70 | BSC-LT/salamandra-7b open weights | 24.6% ±0.7 | hf.co | Official leaderboard | ||
| 71 | tiiuae/Falcon-H1-Tiny-Multilingual-100M-Base open weights | 24.3% ±0.4 | hf.co | Official leaderboard | ||
| 72 | HuggingFaceTB/SmolLM2-135M-Instruct open weights | 24.3% ±0.9 | hf.co | Official leaderboard | ||
| 73 | | 24% ±1 | Mar 2025 | hf.co | Official leaderboard | |
| 74 | utter-project/EuroMoE-2.6B-A0.6B-Preview open weights | 23.7% ±1.1 | hf.co | Official leaderboard | ||
| 75 | PleIAs/Pleias-350m-Preview open weights | 23.2% ±0.6 | hf.co | Official leaderboard | ||
| 76 | | 22.3% ±0.9 | Oct 2024 | hf.co | Official leaderboard | |
| 77 | | 21.9% ±0.8 | Aug 2025 | hf.co | Official leaderboard | |
| 78 | PleIAs/Baguettotron@refs/pr/6 open weights | 18.9% ±0.3 | no-thinking | hf.co | Official leaderboard | |
| 79 | PleIAs/Monad open weights | 10.6% ±0.2 | hf.co | Official leaderboard |
| # | Model | Score ± | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | | 38.5% ±0.1 | Mar 2025 | hf.co | Official leaderboard | |
| 2 | | 38.4% ±0.2 | Dec 2024 | hf.co | Official leaderboard | |
| 3 | | 38% ±0.1 | Sep 2024 | hf.co | Official leaderboard | |
| 4 | | 38% ±0.1 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 5 | | 37.9% ±0.1 | Mar 2025 | hf.co | Official leaderboard | |
| 6 | | 37.9% ±0.1 | Mar 2025 | hf.co | Official leaderboard | |
| 7 | | 37.9% ±0.1 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 8 | | 37.9% ±0.2 | Jun 2025 | hf.co | Official leaderboard | |
| 9 | granite-4.0-micro | 37.8% ±0.2 | Oct 2025 | hf.co | Official leaderboard | |
| 10 | | 37.5% ±0.2 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 11 | | 37.4% ±0.1 | Jun 2025 | hf.co | Official leaderboard | |
| 12 | | 37.4% ±0.1 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 13 | | 36.9% ±0.1 | Mar 2025 | hf.co | Official leaderboard | |
| 14 | | 36.8% ±0.2 | Aug 2025 | hf.co | Official leaderboard | |
| 15 | | 36.5% ±0.1 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 16 | | 36.4% ±0.8 | Jul 2025 | hf.co | Official leaderboard | |
| 17 | | 36.2% ±0.2 | Dec 2025 | hf.co | Official leaderboard | |
| 18 | LFM2-1.2B | 35.2% ±0.1 | Jul 2025 | hf.co | Official leaderboard | |
| 19 | | 35.1% ±0.1 | Dec 2025 | hf.co | Official leaderboard | |
| 20 | | 34.8% ±0.2 | no-thinking | Apr 2025 | hf.co | Official leaderboard |
| 21 | gpt-oss-20b | 34.6% ±0.1 | low | Aug 2025 | hf.co | Official leaderboard |
| 22 | | 34.3% ±0.1 | Mar 2026 | hf.co | Official leaderboard | |
| 23 | | 32.3% ±0.4 | Aug 2025 | hf.co | Official leaderboard | |
| 24 | | 31.8% ±0.9 | Mar 2025 | hf.co | Official leaderboard | |
| 25 | | 31.6% ±1.3 | Jul 2024 | hf.co | Official leaderboard | |
| 26 | | 31% ±1.4 | Mar 2025 | hf.co | Official leaderboard | |
| 27 | | 30.6% ±1.6 | Jul 2024 | hf.co | Official leaderboard | |
| 28 | | 29.3% ±1.2 | Mar 2025 | hf.co | Official leaderboard | |
| 29 | | 29.1% ±0.6 | Sep 2024 | hf.co | Official leaderboard | |
| 30 | | 27.9% ±1.2 | Mar 2025 | hf.co | Official leaderboard | |
| 31 | | 25% ±1.1 | Oct 2024 | hf.co | Official leaderboard | |
| 32 | | 24% ±1 | Mar 2025 | hf.co | Official leaderboard | |
| 33 | | 22.3% ±0.9 | Oct 2024 | hf.co | Official leaderboard | |
| 34 | | 21.9% ±0.8 | Aug 2025 | hf.co | Official leaderboard |
4 models have held number one on EuroEval’s leaderboard since 2023
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: EuroEval’s leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: EuroEval’s leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
K-EXAONE 90.5%
- MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language.
Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7%
- Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Claude Opus 5 95.7%
- MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%
Data audit
9 of 10 checks pass, 1 warning. Look first at the Newest score check. The newest came 8 months ago, in March 2026, for Nemotron 3 Nano 4B. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 1 warning 3 pass 1 note
- Pass Independent runs Scores from the official leaderboard (79 entries).
- Pass Models scored 34 models we track, plus 45 other entries on its leaderboard.
- Warning Newest score The newest came 8 months ago, in March 2026, for Nemotron 3 Nano 4B.
- Note Frontier models None of the top 10 has a score. The strongest tested is Qwen3-32B (#135).
- Pass Headroom The best is 38.5%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on EuroEval: 36 rows for models we track and 45 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.