MultiBLiMP
Models compare minimally different sentences across languages. The benchmark tests whether the model assigns higher likelihood to the grammatical sentence.
- Made by
- Jaap Jumelet and colleagues
- Released
- April 2025
- Kind
- Reasoning in many languages
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Subject
- Languages and literature
- Language
- Many languages
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
- In views
- Many languages
- Strongest model tested
-
Qwen3-14B #139 on the capability index
- Last model added
- April 2025
- Scores refreshed
- November 20th, 2026
Gemma 3 27B PT leads with 90.2% among models released by April 2025 Gemma 3 27B PT leads with 90.2% among models released by April 2025
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
The newest model with a score here came out in April 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 90.2
Gemma 3 27B PT
- 90.2
Llama 3-70B
- 89.3 Llama3-70B-it
- 88.8
Gemma 3 12B PT
- 87.4
Qwen3-14B-Base
- 86.9
Llama 3-8B
- 86.4
Aya Expanse 32B
- 85.8
Gemma 3 4B PT
- 85.8 EuroLLM-9B
- 85.8
Gemma 3 27B
- 85.8
Qwen3-8B-Base
- 85.6 Llama3-8B-it
- 85.4 EuroLLM-9B-it
- 85.3
Qwen3-14B
- 84.9
Llama-3.1-Tulu-3-8B
- 83.7
Qwen3-4B-Base
- 83.6
Llama 3.2 3B
- 83.1
Qwen3-8B
- 82.9
Gemma 3 12B
- 82.7
OLMo 2 32B Instruct
- 81.9 EuroLLM-1.7B
- 81.5 EuroLLM-1.7B-it
- 80.8 Qwen3-1.7B (base)
- 80.7
Qwen3-4B
- 80.4
Aya Expanse 8B
- 80.1
Llama 3.2 1B
- 79.9 OLMo-2-13B
- 78.4 Llama3-3B-it
- 77.6 OLMo-2-7B
- 77.3
Qwen3-1.7B
- 77.1
Gemma 3 4B
- 75.8 Qwen3-0.6B (base)
- 75.6 OLMo-2-13B-it
- 74.8 OLMo-2-7B-it
- 73.7 Llama3-1B-it
- 71.8
Qwen3-0.6B
- 71.0
OLMo 2 1B Instruct
- 63.4 gpt2-xl
- 55.5
Gemma 3 1B
- 55.3
Gemma 3 1B PT
- 1
Gemma 3 27B PT 90.2%
- 2
Llama 3-70B 90.2%
- 3 Llama3-70B-it 89.3%
- 4
Gemma 3 12B PT 88.8%
- 5
Qwen3-14B-Base 87.4%
- 6
Llama 3-8B 86.9%
- 7
Aya Expanse 32B 86.4%
- 8
Gemma 3 4B PT 85.8%
- 9 EuroLLM-9B 85.8%
- 10
Gemma 3 27B 85.8%
- 11
Qwen3-8B-Base 85.8%
- 12 Llama3-8B-it 85.6%
- 13 EuroLLM-9B-it 85.4%
- 14
Qwen3-14B 85.3%
- 15
Llama-3.1-Tulu-3-8B 84.9%
- 16
Qwen3-4B-Base 83.7%
- 17
Llama 3.2 3B 83.6%
- 18
Qwen3-8B 83.1%
- 19
Gemma 3 12B 82.9%
- 20
OLMo 2 32B Instruct 82.7%
- 21 EuroLLM-1.7B 81.9%
- 22 EuroLLM-1.7B-it 81.5%
- 23 Qwen3-1.7B (base) 80.8%
- 24
Qwen3-4B 80.7%
- 25
Aya Expanse 8B 80.4%
- 26
Llama 3.2 1B 80.1%
- 27 OLMo-2-13B 79.9%
- 28 Llama3-3B-it 78.4%
- 29 OLMo-2-7B 77.6%
- 30
Qwen3-1.7B 77.3%
- 31
Gemma 3 4B 77.1%
- 32 Qwen3-0.6B (base) 75.8%
- 33 OLMo-2-13B-it 75.6%
- 34 OLMo-2-7B-it 74.8%
- 35 Llama3-1B-it 73.7%
- 36
Qwen3-0.6B 71.8%
- 37
OLMo 2 1B Instruct 71%
- 38 gpt2-xl 63.4%
- 39
Gemma 3 1B 55.5%
- 40
Gemma 3 1B PT 55.3%
Source: Official MultiBLiMP leaderboard
- 90.2
Gemma 3 27B PT
- 90.2
Llama 3-70B
- 88.8
Gemma 3 12B PT
- 87.4
Qwen3-14B-Base
- 86.9
Llama 3-8B
- 86.4
Aya Expanse 32B
- 85.8
Gemma 3 4B PT
- 85.8
Gemma 3 27B
- 85.8
Qwen3-8B-Base
- 85.3
Qwen3-14B
- 84.9
Llama-3.1-Tulu-3-8B
- 83.7
Qwen3-4B-Base
- 83.6
Llama 3.2 3B
- 83.1
Qwen3-8B
- 82.9
Gemma 3 12B
- 82.7
OLMo 2 32B Instruct
- 80.7
Qwen3-4B
- 80.4
Aya Expanse 8B
- 80.1
Llama 3.2 1B
- 77.3
Qwen3-1.7B
- 77.1
Gemma 3 4B
- 71.8
Qwen3-0.6B
- 71.0
OLMo 2 1B Instruct
- 55.5
Gemma 3 1B
- 55.3
Gemma 3 1B PT
- 1
Gemma 3 27B PT 90.2%
- 2
Llama 3-70B 90.2%
- 3
Gemma 3 12B PT 88.8%
- 4
Qwen3-14B-Base 87.4%
- 5
Llama 3-8B 86.9%
- 6
Aya Expanse 32B 86.4%
- 7
Gemma 3 4B PT 85.8%
- 8
Gemma 3 27B 85.8%
- 9
Qwen3-8B-Base 85.8%
- 10
Qwen3-14B 85.3%
- 11
Llama-3.1-Tulu-3-8B 84.9%
- 12
Qwen3-4B-Base 83.7%
- 13
Llama 3.2 3B 83.6%
- 14
Qwen3-8B 83.1%
- 15
Gemma 3 12B 82.9%
- 16
OLMo 2 32B Instruct 82.7%
- 17
Qwen3-4B 80.7%
- 18
Aya Expanse 8B 80.4%
- 19
Llama 3.2 1B 80.1%
- 20
Qwen3-1.7B 77.3%
- 21
Gemma 3 4B 77.1%
- 22
Qwen3-0.6B 71.8%
- 23
OLMo 2 1B Instruct 71%
- 24
Gemma 3 1B 55.5%
- 25
Gemma 3 1B PT 55.3%
Source: Official MultiBLiMP leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 90.2% | Mar 2025 | Official leaderboard | |
| 2 | | 90.2% | Apr 2024 | Official leaderboard | |
| 3 | Llama3-70B-it | 89.3% | Official leaderboard | ||
| 4 | | 88.8% | Mar 2025 | Official leaderboard | |
| 5 | | 87.4% | Apr 2025 | Official leaderboard | |
| 6 | | 86.9% | Apr 2024 | Official leaderboard | |
| 7 | | 86.4% | Oct 2024 | Official leaderboard | |
| 8 | | 85.8% | Mar 2025 | Official leaderboard | |
| 9 | EuroLLM-9B | 85.8% | Official leaderboard | ||
| 10 | | 85.8% | Mar 2025 | Official leaderboard | |
| 11 | | 85.8% | Apr 2025 | Official leaderboard | |
| 12 | Llama3-8B-it | 85.6% | Official leaderboard | ||
| 13 | EuroLLM-9B-it | 85.4% | Official leaderboard | ||
| 14 | | 85.3% | Apr 2025 | Official leaderboard | |
| 15 | | 84.9% | Nov 2024 | Official leaderboard | |
| 16 | | 83.7% | Apr 2025 | Official leaderboard | |
| 17 | | 83.6% | Sep 2024 | Official leaderboard | |
| 18 | | 83.1% | Apr 2025 | Official leaderboard | |
| 19 | | 82.9% | Mar 2025 | Official leaderboard | |
| 20 | | 82.7% | Mar 2025 | Official leaderboard | |
| 21 | EuroLLM-1.7B | 81.9% | Official leaderboard | ||
| 22 | EuroLLM-1.7B-it | 81.5% | Official leaderboard | ||
| 23 | Qwen3-1.7B (base) | 80.8% | Official leaderboard | ||
| 24 | | 80.7% | Apr 2025 | Official leaderboard | |
| 25 | | 80.4% | Oct 2024 | Official leaderboard | |
| 26 | | 80.1% | Sep 2024 | Official leaderboard | |
| 27 | OLMo-2-13B | 79.9% | Official leaderboard | ||
| 28 | Llama3-3B-it | 78.4% | Official leaderboard | ||
| 29 | OLMo-2-7B | 77.6% | Official leaderboard | ||
| 30 | | 77.3% | Apr 2025 | Official leaderboard | |
| 31 | | 77.1% | Mar 2025 | Official leaderboard | |
| 32 | Qwen3-0.6B (base) | 75.8% | Official leaderboard | ||
| 33 | OLMo-2-13B-it | 75.6% | Official leaderboard | ||
| 34 | OLMo-2-7B-it | 74.8% | Official leaderboard | ||
| 35 | Llama3-1B-it | 73.7% | Official leaderboard | ||
| 36 | | 71.8% | Apr 2025 | Official leaderboard | |
| 37 | | 71% | Apr 2025 | Official leaderboard | |
| 38 | gpt2-xl | 63.4% | Official leaderboard | ||
| 39 | | 55.5% | Mar 2025 | Official leaderboard | |
| 40 | | 55.3% | Mar 2025 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 90.2% | Mar 2025 | Official leaderboard | |
| 2 | | 90.2% | Apr 2024 | Official leaderboard | |
| 3 | | 88.8% | Mar 2025 | Official leaderboard | |
| 4 | | 87.4% | Apr 2025 | Official leaderboard | |
| 5 | | 86.9% | Apr 2024 | Official leaderboard | |
| 6 | | 86.4% | Oct 2024 | Official leaderboard | |
| 7 | | 85.8% | Mar 2025 | Official leaderboard | |
| 8 | | 85.8% | Mar 2025 | Official leaderboard | |
| 9 | | 85.8% | Apr 2025 | Official leaderboard | |
| 10 | | 85.3% | Apr 2025 | Official leaderboard | |
| 11 | | 84.9% | Nov 2024 | Official leaderboard | |
| 12 | | 83.7% | Apr 2025 | Official leaderboard | |
| 13 | | 83.6% | Sep 2024 | Official leaderboard | |
| 14 | | 83.1% | Apr 2025 | Official leaderboard | |
| 15 | | 82.9% | Mar 2025 | Official leaderboard | |
| 16 | | 82.7% | Mar 2025 | Official leaderboard | |
| 17 | | 80.7% | Apr 2025 | Official leaderboard | |
| 18 | | 80.4% | Oct 2024 | Official leaderboard | |
| 19 | | 80.1% | Sep 2024 | Official leaderboard | |
| 20 | | 77.3% | Apr 2025 | Official leaderboard | |
| 21 | | 77.1% | Mar 2025 | Official leaderboard | |
| 22 | | 71.8% | Apr 2025 | Official leaderboard | |
| 23 | | 71% | Apr 2025 | Official leaderboard | |
| 24 | | 55.5% | Mar 2025 | Official leaderboard | |
| 25 | | 55.3% | Mar 2025 | Official leaderboard |
Rankings by part
We read 107 parts from the benchmark’s own leaderboard in 2 groups: language and phenomenon. Each ranks models on one part of the test. Llama 3-70B leads 34 of them, and 19 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.
Old Russian
25 models we track have a score here.
- 1
Llama 3-70B ↑1 80%
- 2
Gemma 3 27B PT ↓1 78.6%
- 3
Aya Expanse 32B ↑3 77.6%
- 4
Llama 3-8B ↑1 76.9%
- 5
Gemma 3 12B PT ↓2 74.9%
- 6
Qwen3-14B-Base ↓2 74.7%
- 7
Qwen3-8B-Base ↑2 73.4%
- 8
Llama-3.1-Tulu-3-8B ↑3 73.1%
- 9
Qwen3-14B ↑1 72.7%
- 10
Gemma 3 27B ↓2 72.5%
Church Slavic
25 models we track have a score here.
- 1
Llama 3-70B ↑1 68.1%
- 2
Gemma 3 27B PT ↓1 66.3%
- 3
Qwen3-14B-Base ↑1 64.8%
- 4
Aya Expanse 32B ↑2 64.2%
- 5
OLMo 2 32B Instruct ↑11 64%
- 6
Gemma 3 12B PT ↓3 63.8%
- 7
Llama 3.2 3B ↑6 63.8%
- 8
Llama 3-8B ↓3 63.7%
- 9
Qwen3-8B-Base 63.2%
- 10
Qwen3-4B-Base ↑2 63.1%
Karelian
25 models we track have a score here.
- 1
Gemma 3 12B PT ↑2 69.6%
- 2
Gemma 3 27B ↑6 68.8%
- 3
Gemma 3 27B PT ↓2 68.5%
- 4
Llama 3-70B ↓2 68.1%
- 5
Llama 3-8B 65.8%
- 6
Gemma 3 12B ↑9 63.1%
- 7
Gemma 3 4B PT 61.9%
- 8
Aya Expanse 32B ↓2 61.2%
- 9
Qwen3-4B-Base ↑3 60.4%
- 10
OLMo 2 1B Instruct ↑13 60.4%
Gothic
25 models we track have a score here.
- 1
Llama 3-70B ↑1 75.4%
- 2
Llama 3-8B ↑3 68.3%
- 3
Llama-3.1-Tulu-3-8B ↑8 66.9%
- 4
OLMo 2 32B Instruct ↑12 63.7%
- 5
Aya Expanse 32B ↑1 63.4%
- 6
Llama 3.2 3B ↑7 62.4%
- 7
Qwen3-14B-Base ↓3 61.2%
- 8
Qwen3-8B-Base ↑1 60.5%
- 9
OLMo 2 1B Instruct ↑14 60.3%
- 10
Qwen3-4B-Base ↑2 60.2%
Classical Armenian
25 models we track have a score here.
- 1
Gemma 3 27B PT 76.9%
- 2
Llama 3-70B 75.4%
- 3
Gemma 3 12B PT 73.2%
- 4
Qwen3-14B ↑6 72%
- 5
Qwen3-14B-Base ↓1 72%
- 6
Gemma 3 27B ↑2 70.5%
- 7
Llama 3-8B ↓2 70%
- 8
Aya Expanse 32B ↓2 69.6%
- 9
Qwen3-8B-Base 68.6%
- 10
Qwen3-4B-Base ↑2 67.9%
German
25 models we track have a score here.
- 1
Llama 3-70B ↑1 99%
- 2
Aya Expanse 32B ↑4 98.9%
- 3
Gemma 3 27B PT ↓2 98.8%
- 4
Gemma 3 12B PT ↓1 98.7%
- 5
Qwen3-8B-Base ↑4 98.5%
- 6
Aya Expanse 8B ↑12 98.4%
- 7
Qwen3-14B-Base ↓3 98.1%
- 8
Gemma 3 4B PT ↓1 98%
- 9
Llama 3-8B ↓4 98%
- 10
Llama 3.2 3B ↑3 97.5%
Ukrainian
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 98.1%
- 2
Gemma 3 12B PT ↑1 97.8%
- 3
Gemma 3 27B PT ↓2 97.8%
- 4
Llama 3-70B ↓2 97.3%
- 5
Aya Expanse 8B ↑13 96.8%
- 6
Qwen3-14B-Base ↓2 96.4%
- 7
Gemma 3 4B PT 95.7%
- 8
Llama 3-8B ↓3 94.5%
- 9
Gemma 3 27B ↓1 94.4%
- 10
Qwen3-8B-Base ↓1 94.4%
Hungarian
25 models we track have a score here.
- 1
Llama 3-70B ↑1 99.2%
- 2
Gemma 3 27B PT ↓1 99.1%
- 3
Gemma 3 12B PT 98.7%
- 4
Llama 3-8B ↑1 97.3%
- 5
Gemma 3 4B PT ↑2 97.2%
- 6
Qwen3-14B-Base ↓2 96.5%
- 7
Qwen3-8B-Base ↑2 96.2%
- 8
Llama-3.1-Tulu-3-8B ↑3 96%
- 9
Llama 3.2 3B ↑4 95.4%
- 10
Gemma 3 27B ↓2 94.7%
Portuguese
25 models we track have a score here.
- 1
Llama 3-70B ↑1 98%
- 2
Qwen3-14B-Base ↑2 97.8%
- 3
Aya Expanse 32B ↑3 97.7%
- 4
Gemma 3 12B PT ↓1 97.2%
- 5
Gemma 3 27B PT ↓4 97.2%
- 6
Qwen3-8B-Base ↑3 97.1%
- 7
Llama 3-8B ↓2 96.9%
- 8
Qwen3-14B ↑2 96.7%
- 9
Qwen3-4B-Base ↑3 96.5%
- 10
Aya Expanse 8B ↑8 96.4%
Bulgarian
25 models we track have a score here.
- 1
Gemma 3 27B PT 99.2%
- 2
Gemma 3 12B PT ↑1 97.8%
- 3
Gemma 3 4B PT ↑4 96.8%
- 4
Qwen3-14B-Base 96.1%
- 5
Llama 3-70B ↓3 96%
- 6
Gemma 3 27B ↑2 95.8%
- 7
Qwen3-8B-Base ↑2 95.6%
- 8
Qwen3-14B ↑2 94.8%
- 9
Gemma 3 12B ↑6 93.9%
- 10
Llama 3-8B ↓5 93.6%
Italian
25 models we track have a score here.
- 1
Llama 3-70B ↑1 97.7%
- 2
Aya Expanse 32B ↑4 97.3%
- 3
Qwen3-14B-Base ↑1 97.2%
- 4
Gemma 3 27B PT ↓3 97%
- 5
Gemma 3 12B PT ↓2 96.7%
- 6
Qwen3-8B-Base ↑3 96.6%
- 7
Llama 3-8B ↓2 96.2%
- 8
Aya Expanse 8B ↑10 96%
- 9
Llama-3.1-Tulu-3-8B ↑2 95.9%
- 10
Gemma 3 4B PT ↓3 95.8%
Faroese
25 models we track have a score here.
- 1
Gemma 3 27B PT 94.8%
- 2
Gemma 3 27B ↑6 89.7%
- 3
Llama 3-70B ↓1 88.4%
- 4
Gemma 3 12B PT ↓1 87.9%
- 5
Gemma 3 12B ↑10 83.6%
- 6
OLMo 2 32B Instruct ↑10 83.2%
- 7
Gemma 3 4B PT 81.9%
- 8
Llama-3.1-Tulu-3-8B ↑3 80.2%
- 9
Llama 3-8B ↓4 79.7%
- 10
Qwen3-14B-Base ↓6 79.7%
Russian
25 models we track have a score here.
- 1
Llama 3-70B ↑1 98.6%
- 2
Gemma 3 27B PT ↓1 98.5%
- 3
Aya Expanse 32B ↑3 98.2%
- 4
Gemma 3 12B PT ↓1 98.1%
- 5
Qwen3-14B-Base ↓1 97.8%
- 6
Qwen3-8B-Base ↑3 97.4%
- 7
Llama 3-8B ↓2 97.4%
- 8
Gemma 3 4B PT ↓1 96.9%
- 9
Aya Expanse 8B ↑9 96.7%
- 10
Gemma 3 27B ↓2 96.6%
Ancient Greek (to 1453)
25 models we track have a score here.
- 1
OLMo 2 32B Instruct ↑15 91.2%
- 2
Llama 3-70B 90.8%
- 3
Aya Expanse 32B ↑3 87.7%
- 4
Gemma 3 27B PT ↓3 87.7%
- 5
Qwen3-14B-Base ↓1 87%
- 6
Llama 3-8B ↓1 86.8%
- 7
Gemma 3 12B PT ↓4 85.7%
- 8
Llama-3.1-Tulu-3-8B ↑3 85.4%
- 9
Qwen3-14B ↑1 84.7%
- 10
Qwen3-8B-Base ↓1 83.9%
Veps
25 models we track have a score here.
- 1
Llama 3-70B ↑1 70.6%
- 2
Llama 3-8B ↑3 69.5%
- 3
Llama 3.2 3B ↑10 64.2%
- 4
Llama-3.1-Tulu-3-8B ↑7 64.2%
- 5
Gemma 3 27B PT ↓4 62%
- 6
Gemma 3 27B ↑2 61%
- 7
Llama 3.2 1B ↑12 58.8%
- 8
Gemma 3 12B PT ↓5 58.8%
- 9
Aya Expanse 32B ↓3 58.3%
- 10
Aya Expanse 8B ↑8 56.1%
Northern Kurdish
25 models we track have a score here.
- 1
Llama 3-70B ↑1 91.2%
- 2
Gemma 3 27B PT ↓1 90.6%
- 3
Gemma 3 12B PT 89.9%
- 4
Llama 3-8B ↑1 85.7%
- 5
Gemma 3 27B ↑3 84.7%
- 6
Llama-3.1-Tulu-3-8B ↑5 82.9%
- 7
Gemma 3 12B ↑8 82.9%
- 8
Gemma 3 4B PT ↓1 79.2%
- 9
Qwen3-14B-Base ↓5 78.7%
- 10
Llama 3.2 3B ↑3 78.5%
Hebrew
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 90.9%
- 2
Gemma 3 27B PT ↓1 90.7%
- 3
Llama 3-70B ↓1 90%
- 4
Llama 3-8B ↑1 88.1%
- 5
Gemma 3 12B PT ↓2 87.2%
- 6
Gemma 3 4B PT ↑1 86.2%
- 7
Aya Expanse 8B ↑11 85.9%
- 8
Llama-3.1-Tulu-3-8B ↑3 85.7%
- 9
Llama 3.2 3B ↑4 84.2%
- 10
Gemma 3 27B ↓2 84.1%
Spanish
25 models we track have a score here.
- 1
Llama 3-70B ↑1 98.2%
- 2
Aya Expanse 32B ↑4 98%
- 3
Qwen3-14B-Base ↑1 97.9%
- 4
Llama 3-8B ↑1 97.8%
- 5
Aya Expanse 8B ↑13 97.8%
- 6
Gemma 3 27B PT ↓5 97.7%
- 7
Gemma 3 12B PT ↓4 97.6%
- 8
Qwen3-4B-Base ↑4 97.1%
- 9
Llama-3.1-Tulu-3-8B ↑2 97.1%
- 10
Qwen3-8B-Base ↓1 97%
Northern Sami
25 models we track have a score here.
- 1
Gemma 3 27B PT 72.5%
- 2
Llama 3-70B 71.7%
- 3
Gemma 3 12B PT 70%
- 4
Gemma 3 27B ↑4 69.4%
- 5
Llama 3-8B 68.2%
- 6
Aya Expanse 32B 67.5%
- 7
Llama-3.1-Tulu-3-8B ↑4 67.3%
- 8
Qwen3-14B-Base ↓4 67.2%
- 9
Llama 3.2 3B ↑4 67%
- 10
Gemma 3 12B ↑5 67%
Buriat
25 models we track have a score here.
- 1
Gemma 3 27B PT 77.7%
- 2
Llama 3-70B 73.8%
- 3
Qwen3-4B-Base ↑9 72.8%
- 4
OLMo 2 1B Instruct ↑19 72.8%
- 5
Qwen3-4B ↑12 72.8%
- 6
Gemma 3 4B PT ↑1 71.8%
- 7
Aya Expanse 8B ↑11 70.9%
- 8
Gemma 3 27B 70.9%
- 9
Gemma 3 12B PT ↓6 70.9%
- 10
Gemma 3 1B PT ↑15 70.9%
Persian
25 models we track have a score here.
- 1
Gemma 3 27B PT 97.5%
- 2
Gemma 3 12B PT ↑1 97.2%
- 3
Llama 3-70B ↓1 97%
- 4
Aya Expanse 32B ↑2 96.8%
- 5
Aya Expanse 8B ↑13 95.3%
- 6
Gemma 3 27B ↑2 95.1%
- 7
Gemma 3 4B PT 95.1%
- 8
Llama 3-8B ↓3 94.8%
- 9
Llama-3.1-Tulu-3-8B ↑2 94%
- 10
Gemma 3 12B ↑5 93.9%
Slovenian
25 models we track have a score here.
- 1
Gemma 3 27B PT 94.2%
- 2
Gemma 3 12B PT ↑1 93%
- 3
Llama 3-70B ↓1 91.3%
- 4
Gemma 3 4B PT ↑3 90.1%
- 5
Qwen3-14B-Base ↓1 89.2%
- 6
Gemma 3 27B ↑2 88.7%
- 7
Qwen3-8B-Base ↑2 87.8%
- 8
Qwen3-14B ↑2 87.2%
- 9
Llama 3-8B ↓4 86.4%
- 10
Gemma 3 12B ↑5 86%
Czech
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 97%
- 2
Gemma 3 27B PT ↓1 96.2%
- 3
Llama 3-70B ↓1 95.7%
- 4
Gemma 3 12B PT ↓1 95.2%
- 5
Aya Expanse 8B ↑13 94%
- 6
Qwen3-14B-Base ↓2 92.4%
- 7
Llama 3-8B ↓2 92.1%
- 8
Gemma 3 4B PT ↓1 92%
- 9
Gemma 3 27B ↓1 91.5%
- 10
Qwen3-8B-Base ↓1 91%
Kangri
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 79.1%
- 2
Qwen3-14B-Base ↑2 79.1%
- 3
Qwen3-8B-Base ↑6 79.1%
- 4
Llama 3-8B ↑1 77.9%
- 5
Gemma 3 4B PT ↑2 77.9%
- 6
Llama 3.2 3B ↑7 76.7%
- 7
Gemma 3 12B ↑8 76.7%
- 8
Gemma 3 12B PT ↓5 76.7%
- 9
Qwen3-8B ↑5 76.7%
- 10
Aya Expanse 8B ↑8 75.6%
Komi-Zyrian
25 models we track have a score here.
- 1
Llama 3-70B ↑1 77.2%
- 2
Gemma 3 27B PT ↓1 69.4%
- 3
Llama 3-8B ↑2 68.4%
- 4
Aya Expanse 32B ↑2 66.9%
- 5
Gemma 3 12B PT ↓2 66.3%
- 6
Aya Expanse 8B ↑12 61.9%
- 7
Llama 3.2 3B ↑6 61.6%
- 8
Llama-3.1-Tulu-3-8B ↑3 61.3%
- 9
Llama 3.2 1B ↑10 60.3%
- 10
Gemma 3 27B ↓2 60%
Nhengatu
25 models we track have a score here.
- 1
Llama 3.2 1B ↑18 64.9%
- 2
Llama 3.2 3B ↑11 63.3%
- 3
Llama 3-70B ↓1 63.1%
- 4
Qwen3-14B-Base 63.1%
- 5
Aya Expanse 32B ↑1 62.1%
- 6
Qwen3-4B-Base ↑6 61.7%
- 7
OLMo 2 1B Instruct ↑16 61.7%
- 8
OLMo 2 32B Instruct ↑8 61%
- 9
Qwen3-14B ↑1 60.8%
- 10
Qwen3-8B-Base ↓1 60.7%
Estonian
25 models we track have a score here.
- 1
Gemma 3 27B PT 95.4%
- 2
Gemma 3 12B PT ↑1 92.7%
- 3
Gemma 3 27B ↑5 89.3%
- 4
Gemma 3 4B PT ↑3 87.8%
- 5
Llama 3-70B ↓3 86.7%
- 6
Gemma 3 12B ↑9 85.1%
- 7
Qwen3-14B-Base ↓3 83.2%
- 8
Qwen3-8B-Base ↑1 80.4%
- 9
Llama 3-8B ↓4 79.6%
- 10
Qwen3-14B 79.5%
Abkhazian
25 models we track have a score here.
- 1
OLMo 2 1B Instruct ↑22 80%
- 2
Gemma 3 27B PT ↓1 77.5%
- 3
Llama 3-70B ↓1 70%
- 4
Llama 3.2 1B ↑15 70%
- 5
Gemma 3 27B ↑3 70%
- 6
OLMo 2 32B Instruct ↑10 70%
- 7
Qwen3-8B ↑7 70%
- 8
Llama 3.2 3B ↑5 67.5%
- 9
Aya Expanse 32B ↓3 67.5%
- 10
Llama 3-8B ↓5 65%
Latin
25 models we track have a score here.
- 1
Llama 3-70B ↑1 92.1%
- 2
Gemma 3 27B PT ↓1 87.6%
- 3
Llama 3-8B ↑2 87%
- 4
Aya Expanse 32B ↑2 85.1%
- 5
Qwen3-14B-Base ↓1 84.9%
- 6
OLMo 2 32B Instruct ↑10 84.7%
- 7
Llama-3.1-Tulu-3-8B ↑4 84.5%
- 8
Gemma 3 12B PT ↓5 84.3%
- 9
Gemma 3 27B ↓1 81.6%
- 10
Qwen3-14B 81.4%
K'iche'
25 models we track have a score here.
- 1
Qwen3-0.6B ↑21 74%
- 2
Gemma 3 1B PT ↑23 72.5%
- 3
Qwen3-4B-Base ↑9 71.8%
- 4
OLMo 2 1B Instruct ↑19 71.8%
- 5
Llama 3.2 3B ↑8 71%
- 6
Aya Expanse 32B 71%
- 7
Gemma 3 12B ↑8 71%
- 8
Qwen3-14B-Base ↓4 71%
- 9
Qwen3-8B ↑5 71%
- 10
Qwen3-8B-Base ↓1 71%
Finnish
25 models we track have a score here.
- 1
Gemma 3 27B PT 96.4%
- 2
Gemma 3 12B PT ↑1 96.3%
- 3
Gemma 3 4B PT ↑4 94.6%
- 4
Llama 3-70B ↓2 94.5%
- 5
Gemma 3 27B ↑3 93.9%
- 6
OLMo 2 32B Instruct ↑10 91.5%
- 7
Llama 3-8B ↓2 91.4%
- 8
Gemma 3 12B ↑7 91.4%
- 9
Qwen3-14B-Base ↓5 90.2%
- 10
Llama-3.1-Tulu-3-8B ↑1 89.8%
Standard Arabic
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 96%
- 2
Gemma 3 27B PT ↓1 95.4%
- 3
Aya Expanse 8B ↑15 94.7%
- 4
Qwen3-8B-Base ↑5 94.7%
- 5
Llama 3-70B ↓3 94.5%
- 6
Gemma 3 12B PT ↓3 94.2%
- 7
Qwen3-14B-Base ↓3 93.8%
- 8
Gemma 3 4B PT ↓1 92.9%
- 9
Qwen3-4B-Base ↑3 92%
- 10
Qwen3-8B ↑4 92%
Armenian
25 models we track have a score here.
- 1
Llama 3-70B ↑1 98.4%
- 2
Gemma 3 27B PT ↓1 97.9%
- 3
Gemma 3 12B PT 97.3%
- 4
Qwen3-14B-Base 96.5%
- 5
Qwen3-8B-Base ↑4 96.5%
- 6
Llama 3-8B ↓1 96.5%
- 7
Llama 3.2 3B ↑6 95.1%
- 8
Gemma 3 4B PT ↓1 95.1%
- 9
Qwen3-14B ↑1 94.8%
- 10
Qwen3-8B ↑4 94.8%
Low German
25 models we track have a score here.
- 1
Llama 3-70B ↑1 73.2%
- 2
Aya Expanse 32B ↑4 72.2%
- 3
Llama 3-8B ↑2 71.6%
- 4
Gemma 3 27B PT ↓3 71.1%
- 5
Llama 3.2 1B ↑14 69.7%
- 6
Llama-3.1-Tulu-3-8B ↑5 68.9%
- 7
Llama 3.2 3B ↑6 68.9%
- 8
Qwen3-14B ↑2 68.8%
- 9
Qwen3-14B-Base ↓5 68.6%
- 10
Qwen3-4B-Base ↑2 68.4%
Western Armenian
25 models we track have a score here.
- 1
Llama 3-70B ↑1 96.7%
- 2
Gemma 3 27B PT ↓1 96.6%
- 3
Gemma 3 12B PT 94.6%
- 4
Llama 3-8B ↑1 94.1%
- 5
Qwen3-14B-Base ↓1 93.4%
- 6
Qwen3-8B-Base ↑3 92.9%
- 7
Qwen3-14B ↑3 91%
- 8
Llama-3.1-Tulu-3-8B ↑3 90.5%
- 9
Gemma 3 4B PT ↓2 90.4%
- 10
Gemma 3 27B ↓2 89.9%
Tamil
25 models we track have a score here.
- 1
Llama 3-70B ↑1 98.7%
- 2
Aya Expanse 8B ↑16 98.4%
- 3
Aya Expanse 32B ↑3 98.2%
- 4
Llama 3-8B ↑1 97.9%
- 5
Gemma 3 27B PT ↓4 97.9%
- 6
Gemma 3 27B ↑2 97.6%
- 7
Qwen3-14B-Base ↓3 97.6%
- 8
Qwen3-4B-Base ↑4 97.6%
- 9
Gemma 3 12B PT ↓6 97.1%
- 10
Qwen3-14B 97.1%
Sanskrit
25 models we track have a score here.
- 1
Llama 3-70B ↑1 81.5%
- 2
OLMo 2 32B Instruct ↑14 78.4%
- 3
Qwen3-14B-Base ↑1 76.9%
- 4
Llama 3-8B ↑1 76.2%
- 5
Qwen3-14B ↑5 74.7%
- 6
Gemma 3 27B PT ↓5 74.2%
- 7
Llama-3.1-Tulu-3-8B ↑4 74.1%
- 8
Qwen3-8B-Base ↑1 73.4%
- 9
Llama 3.2 3B ↑4 72%
- 10
Qwen3-4B-Base ↑2 71.3%
Middle French (ca. 1400-1600)
25 models we track have a score here.
- 1
Llama 3-70B ↑1 98.3%
- 2
Gemma 3 12B PT ↑1 97.3%
- 3
Llama 3-8B ↑2 96.9%
- 4
Llama-3.1-Tulu-3-8B ↑7 96.9%
- 5
OLMo 2 32B Instruct ↑11 96.9%
- 6
Llama 3.2 3B ↑7 96.6%
- 7
Qwen3-14B-Base ↓3 96.6%
- 8
Qwen3-8B-Base ↑1 96.3%
- 9
Gemma 3 27B PT ↓8 95.6%
- 10
Qwen3-8B ↑4 95.6%
French
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 99.5%
- 2
Gemma 3 27B PT ↓1 99.1%
- 3
Qwen3-8B-Base ↑6 99.1%
- 4
Llama 3-70B ↓2 99%
- 5
Qwen3-14B-Base ↓1 99%
- 6
Aya Expanse 8B ↑12 98.9%
- 7
Llama 3-8B ↓2 98.7%
- 8
Gemma 3 12B PT ↓5 98.6%
- 9
OLMo 2 32B Instruct ↑7 98.6%
- 10
Qwen3-4B-Base ↑2 98.6%
Hindi
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 99.6%
- 2
Aya Expanse 8B ↑16 99.4%
- 3
Llama 3-70B ↓1 99.3%
- 4
Llama 3-8B ↑1 99.3%
- 5
Llama-3.1-Tulu-3-8B ↑6 99.2%
- 6
Gemma 3 27B PT ↓5 99.2%
- 7
Qwen3-14B-Base ↓3 99%
- 8
Qwen3-8B-Base ↑1 98.6%
- 9
Gemma 3 27B ↓1 98.5%
- 10
Gemma 3 12B PT ↓7 98.5%
Lithuanian
25 models we track have a score here.
- 1
Gemma 3 12B PT ↑2 98.2%
- 2
Gemma 3 27B PT ↓1 98.1%
- 3
Qwen3-14B-Base ↑1 97.1%
- 4
Gemma 3 4B PT ↑3 97%
- 5
Llama 3-70B ↓3 96.6%
- 6
Qwen3-14B ↑4 96.3%
- 7
Gemma 3 27B ↑1 95.5%
- 8
Qwen3-8B-Base ↑1 95.3%
- 9
Qwen3-4B-Base ↑3 94.2%
- 10
Qwen3-8B ↑4 93.8%
Marathi
25 models we track have a score here.
- 1
Gemma 3 27B PT 90.2%
- 2
Gemma 3 27B ↑6 88.3%
- 3
Qwen3-14B-Base ↑1 88.3%
- 4
Gemma 3 12B PT ↓1 88%
- 5
Qwen3-8B-Base ↑4 87.8%
- 6
Qwen3-8B ↑8 86.7%
- 7
Qwen3-14B ↑3 86.3%
- 8
OLMo 2 32B Instruct ↑8 85.9%
- 9
Llama 3-70B ↓7 85.4%
- 10
Gemma 3 12B ↑5 83.3%
Wolof
25 models we track have a score here.
- 1
Qwen3-14B-Base ↑3 87.2%
- 2
Llama 3-70B 86.1%
- 3
Qwen3-0.6B ↑19 86.1%
- 4
Qwen3-1.7B ↑16 85.1%
- 5
Llama 3-8B 85%
- 6
Gemma 3 27B PT ↓5 85%
- 7
Qwen3-14B ↑3 84.7%
- 8
Qwen3-4B ↑9 84.4%
- 9
OLMo 2 32B Instruct ↑7 83.8%
- 10
OLMo 2 1B Instruct ↑13 83.7%
Serbo-Croatian
25 models we track have a score here.
- 1
Gemma 3 27B PT 98.3%
- 2
Gemma 3 12B PT ↑1 98.2%
- 3
Gemma 3 4B PT ↑4 96%
- 4
Llama 3-70B ↓2 96%
- 5
Gemma 3 27B ↑3 95.5%
- 6
Llama 3-8B ↓1 94%
- 7
Qwen3-14B-Base ↓3 93.9%
- 8
Gemma 3 12B ↑7 93%
- 9
Llama-3.1-Tulu-3-8B ↑2 92.3%
- 10
Qwen3-8B-Base ↓1 92.3%
Slovak
25 models we track have a score here.
- 1
Gemma 3 27B PT 95.7%
- 2
Gemma 3 12B PT ↑1 95.6%
- 3
Gemma 3 4B PT ↑4 92.7%
- 4
Gemma 3 27B ↑4 92%
- 5
Llama 3-70B ↓3 91.8%
- 6
Qwen3-14B-Base ↓2 90.3%
- 7
Aya Expanse 32B ↓1 90.2%
- 8
Gemma 3 12B ↑7 88.9%
- 9
Qwen3-14B ↑1 87.9%
- 10
Qwen3-8B-Base ↓1 87.8%
Galician
25 models we track have a score here.
- 1
Gemma 3 12B PT ↑2 94.7%
- 2
Gemma 3 27B PT ↓1 94.7%
- 3
Llama 3-70B ↓1 92.3%
- 4
Gemma 3 4B PT ↑3 91.8%
- 5
Qwen3-14B-Base ↓1 90.8%
- 6
Llama 3-8B ↓1 89.9%
- 7
Aya Expanse 32B ↓1 89.6%
- 8
Qwen3-8B-Base ↑1 89.6%
- 9
Gemma 3 27B ↓1 88.8%
- 10
Qwen3-14B 88.7%
Old French (842-ca. 1400)
25 models we track have a score here.
- 1
Llama 3-70B ↑1 81.8%
- 2
Llama 3-8B ↑3 78.3%
- 3
OLMo 2 32B Instruct ↑13 76.7%
- 4
Aya Expanse 32B ↑2 76.3%
- 5
Qwen3-14B-Base ↓1 75.8%
- 6
Llama-3.1-Tulu-3-8B ↑5 75.7%
- 7
Qwen3-8B-Base ↑2 75.1%
- 8
Llama 3.2 3B ↑5 74.6%
- 9
Gemma 3 27B PT ↓8 73.7%
- 10
Qwen3-14B 73.1%
Dutch
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 98%
- 2
Llama 3-70B 97.7%
- 3
Gemma 3 27B PT ↓2 97.6%
- 4
Gemma 3 12B PT ↓1 97.5%
- 5
Gemma 3 4B PT ↑2 96.3%
- 6
Llama 3-8B ↓1 96.1%
- 7
Aya Expanse 8B ↑11 95.5%
- 8
Llama-3.1-Tulu-3-8B ↑3 95.2%
- 9
Qwen3-8B-Base 94.6%
- 10
Qwen3-14B-Base ↓6 94.5%
Hittite
25 models we track have a score here.
- 1
Gemma 3 1B PT ↑24 68%
- 2
Llama 3-70B 66%
- 3
Qwen3-8B-Base ↑6 66%
- 4
OLMo 2 1B Instruct ↑19 64%
- 5
Llama 3-8B 62%
- 6
Llama-3.1-Tulu-3-8B ↑5 62%
- 7
Gemma 3 1B ↑17 62%
- 8
OLMo 2 32B Instruct ↑8 62%
- 9
Qwen3-0.6B ↑13 62%
- 10
Llama 3.2 3B ↑3 60%
Latvian
25 models we track have a score here.
- 1
Gemma 3 27B PT 95.1%
- 2
Gemma 3 12B PT ↑1 93.8%
- 3
Qwen3-14B-Base ↑1 92.1%
- 4
Gemma 3 27B ↑4 90.6%
- 5
Gemma 3 4B PT ↑2 90.4%
- 6
Llama 3-70B ↓4 89.2%
- 7
Qwen3-8B-Base ↑2 89%
- 8
Qwen3-14B ↑2 88.7%
- 9
Gemma 3 12B ↑6 87.8%
- 10
Qwen3-4B-Base ↑2 86.7%
Romanian
25 models we track have a score here.
- 1
Gemma 3 27B PT 97.7%
- 2
Llama 3-70B 97.4%
- 3
Aya Expanse 32B ↑3 97.4%
- 4
Gemma 3 12B PT ↓1 97.3%
- 5
Qwen3-14B-Base ↓1 96.2%
- 6
Aya Expanse 8B ↑12 96.1%
- 7
Gemma 3 4B PT 95.7%
- 8
Llama 3-8B ↓3 95.4%
- 9
Gemma 3 27B ↓1 94.7%
- 10
Qwen3-14B 94.7%
Karo (Ethiopia)
25 models we track have a score here.
- 1
Gemma 3 4B ↑20 56.7%
- 2
Aya Expanse 8B ↑16 52.5%
- 3
Gemma 3 27B PT ↓2 50%
- 4
Qwen3-4B-Base ↑8 50%
- 5
OLMo 2 1B Instruct ↑18 49.2%
- 6
Llama 3.2 3B ↑7 48.3%
- 7
Gemma 3 4B PT 48.3%
- 8
Qwen3-0.6B ↑14 47.5%
- 9
Qwen3-4B ↑8 47.5%
- 10
Llama 3.2 1B ↑9 46.7%
Tektiteko
25 models we track have a score here.
- 1
Gemma 3 1B ↑23 47.8%
- 2
Gemma 3 1B PT ↑23 47.8%
- 3
Qwen3-8B-Base ↑6 47.8%
- 4
Qwen3-0.6B ↑18 47.8%
- 5
Gemma 3 12B PT ↓2 46.4%
- 6
Qwen3-14B-Base ↓2 46.4%
- 7
Aya Expanse 32B ↓1 44.9%
- 8
OLMo 2 32B Instruct ↑8 44.9%
- 9
Qwen3-14B ↑1 44.9%
- 10
Qwen3-4B-Base ↑2 44.9%
Ligurian
25 models we track have a score here.
- 1
Gemma 3 27B PT 81.5%
- 2
Gemma 3 12B PT ↑1 80.7%
- 3
Aya Expanse 32B ↑3 80.3%
- 4
Gemma 3 27B ↑4 79.9%
- 5
Qwen3-14B ↑5 79.9%
- 6
Qwen3-14B-Base ↓2 78.3%
- 7
Llama 3-70B ↓5 77.2%
- 8
Qwen3-1.7B ↑12 76.8%
- 9
Qwen3-8B ↑5 74%
- 10
Llama 3.2 1B ↑9 73.2%
Icelandic
25 models we track have a score here.
- 1
Gemma 3 27B PT 96%
- 2
Gemma 3 12B PT ↑1 94.6%
- 3
Llama 3-70B ↓1 91%
- 4
Gemma 3 27B ↑4 90.6%
- 5
Gemma 3 4B PT ↑2 90.3%
- 6
Qwen3-14B-Base ↓2 87.5%
- 7
Gemma 3 12B ↑8 86.7%
- 8
Llama 3-8B ↓3 85.3%
- 9
Qwen3-8B-Base 83.9%
- 10
Llama-3.1-Tulu-3-8B ↑1 82.9%
Uighur
25 models we track have a score here.
- 1
Gemma 3 27B PT 80.9%
- 2
Llama 3-70B 79.4%
- 3
Gemma 3 12B PT 78.9%
- 4
Qwen3-14B ↑6 77.8%
- 5
Gemma 3 27B ↑3 77.6%
- 6
Qwen3-14B-Base ↓2 76.9%
- 7
Qwen3-8B-Base ↑2 76.6%
- 8
Aya Expanse 32B ↓2 76.4%
- 9
Llama 3-8B ↓4 75.1%
- 10
Gemma 3 4B PT ↓3 75.1%
Belarusian
25 models we track have a score here.
- 1
Gemma 3 27B PT 96.9%
- 2
Gemma 3 12B PT ↑1 95.2%
- 3
Gemma 3 4B PT ↑4 93.4%
- 4
Llama 3-70B ↓2 93.1%
- 5
Gemma 3 27B ↑3 92.3%
- 6
Llama 3-8B ↓1 89.2%
- 7
Qwen3-14B-Base ↓3 87.9%
- 8
Gemma 3 12B ↑7 87.6%
- 9
Llama-3.1-Tulu-3-8B ↑2 86.5%
- 10
Qwen3-8B-Base ↓1 85.5%
Ancient Hebrew
25 models we track have a score here.
- 1
Gemma 3 27B PT 92.9%
- 2
Llama 3-70B 91.8%
- 3
Gemma 3 12B PT 91.6%
- 4
Aya Expanse 32B ↑2 90.9%
- 5
OLMo 2 32B Instruct ↑11 90%
- 6
Llama 3-8B ↓1 86.9%
- 7
Qwen3-14B-Base ↓3 86%
- 8
Qwen3-14B ↑2 83.7%
- 9
Gemma 3 27B ↓1 83.1%
- 10
Llama-3.1-Tulu-3-8B ↑1 82.8%
Zacatlán-Ahuacatlán-Tepetzintla Nahuatl
25 models we track have a score here.
- 1
Gemma 3 1B ↑23 65.8%
- 2
Gemma 3 1B PT ↑23 57.9%
- 3
Llama 3-70B ↓1 55.3%
- 4
Aya Expanse 8B ↑14 55.3%
- 5
Gemma 3 12B PT ↓2 55.3%
- 6
OLMo 2 32B Instruct ↑10 55.3%
- 7
Qwen3-8B ↑7 55.3%
- 8
Llama 3-8B ↓3 52.6%
- 9
Gemma 3 4B PT ↓2 52.6%
- 10
Qwen3-14B-Base ↓6 52.6%
Breton
25 models we track have a score here.
- 1
Gemma 3 27B PT 97.3%
- 2
Gemma 3 12B PT ↑1 96.9%
- 3
Llama 3-70B ↓1 95%
- 4
Llama 3-8B ↑1 94.6%
- 5
Gemma 3 4B PT ↑2 94.6%
- 6
Aya Expanse 32B 94.2%
- 7
Gemma 3 27B ↑1 93.5%
- 8
Llama-3.1-Tulu-3-8B ↑3 92.7%
- 9
Gemma 3 12B ↑6 92.7%
- 10
Llama 3.2 3B ↑3 92.3%
Akuntsu
25 models we track have a score here.
- 1
Gemma 3 1B PT ↑24 57.1%
- 2
Aya Expanse 8B ↑16 50%
- 3
Gemma 3 4B ↑18 50%
- 4
Llama 3-70B ↓2 42.9%
- 5
Gemma 3 1B ↑19 42.9%
- 6
Qwen3-8B ↑8 42.9%
- 7
Llama 3-8B ↓2 35.7%
- 8
Gemma 3 27B PT ↓7 35.7%
- 9
Gemma 3 4B PT ↓2 35.7%
- 10
OLMo 2 32B Instruct ↑6 35.7%
Welsh
25 models we track have a score here.
- 1
Llama 3-70B ↑1 94%
- 2
Aya Expanse 32B ↑4 91.8%
- 3
Gemma 3 27B PT ↓2 90.3%
- 4
Llama 3-8B ↑1 90.2%
- 5
OLMo 2 32B Instruct ↑11 90%
- 6
Qwen3-14B ↑4 88.3%
- 7
Gemma 3 12B PT ↓4 88.2%
- 8
Gemma 3 27B 88.1%
- 9
Llama-3.1-Tulu-3-8B ↑2 87.7%
- 10
Qwen3-14B-Base ↓6 86.6%
Modern Greek (1453-)
25 models we track have a score here.
- 1
Gemma 3 27B PT 99.5%
- 2
Llama 3-70B 99.3%
- 3
Aya Expanse 32B ↑3 99.2%
- 4
Qwen3-14B-Base 99.2%
- 5
Gemma 3 12B PT ↓2 99%
- 6
Gemma 3 4B PT ↑1 98.9%
- 7
Qwen3-8B-Base ↑2 98.4%
- 8
Llama 3-8B ↓3 97.9%
- 9
Aya Expanse 8B ↑9 97.9%
- 10
Qwen3-14B 97.9%
Moksha
25 models we track have a score here.
- 1
OLMo 2 1B Instruct ↑22 61%
- 2
Qwen3-4B ↑15 59.8%
- 3
Gemma 3 27B ↑5 58.5%
- 4
Llama 3-8B ↑1 56.1%
- 5
Llama 3.2 3B ↑8 56.1%
- 6
Aya Expanse 32B 56.1%
- 7
Gemma 3 27B PT ↓6 56.1%
- 8
Gemma 3 12B PT ↓5 54.9%
- 9
Gemma 3 4B PT ↓2 54.9%
- 10
Qwen3-14B-Base ↓6 54.9%
Catalan
25 models we track have a score here.
- 1
Gemma 3 27B PT 96.4%
- 2
Gemma 3 12B PT ↑1 96.2%
- 3
Llama 3-70B ↓1 96.1%
- 4
Qwen3-14B-Base 95.8%
- 5
Aya Expanse 32B ↑1 95.1%
- 6
Llama 3-8B ↓1 94.8%
- 7
Qwen3-8B-Base ↑2 94.8%
- 8
Gemma 3 4B PT ↓1 94.7%
- 9
Llama-3.1-Tulu-3-8B ↑2 94.4%
- 10
Qwen3-4B-Base ↑2 94.2%
Irish
25 models we track have a score here.
- 1
Llama-3.1-Tulu-3-8B ↑10 89.3%
- 2
OLMo 2 32B Instruct ↑14 85.7%
- 3
Llama 3-8B ↑2 82.1%
- 4
Llama 3.2 1B ↑15 82.1%
- 5
Aya Expanse 32B ↑1 82.1%
- 6
Aya Expanse 8B ↑12 82.1%
- 7
Llama 3-70B ↓5 78.6%
- 8
Gemma 3 12B ↑7 78.6%
- 9
Qwen3-14B ↑1 78.6%
- 10
Qwen3-0.6B ↑12 78.6%
Gheg Albanian
25 models we track have a score here.
- 1
Gemma 3 27B PT 80.4%
- 2
Aya Expanse 32B ↑4 76.8%
- 3
Llama 3-70B ↓1 76.4%
- 4
Gemma 3 12B PT ↓1 74.9%
- 5
Qwen3-14B-Base ↓1 74.2%
- 6
Qwen3-4B-Base ↑6 74.2%
- 7
Gemma 3 27B ↑1 74%
- 8
Gemma 3 12B ↑7 73.7%
- 9
OLMo 2 32B Instruct ↑7 73.1%
- 10
OLMo 2 1B Instruct ↑13 73.1%
Livvi
25 models we track have a score here.
- 1
Llama 3-70B ↑1 86.3%
- 2
Gemma 3 27B PT ↓1 83.2%
- 3
Llama 3-8B ↑2 79.5%
- 4
Gemma 3 12B PT ↓1 76.3%
- 5
Llama-3.1-Tulu-3-8B ↑6 75.3%
- 6
Qwen3-14B-Base ↓2 71.1%
- 7
Llama 3.2 3B ↑6 70.5%
- 8
Gemma 3 27B 70.5%
- 9
Gemma 3 4B PT ↓2 70%
- 10
Qwen3-4B-Base ↑2 68.9%
Erzya
25 models we track have a score here.
- 1
OLMo 2 32B Instruct ↑15 57.1%
- 2
Qwen3-14B-Base ↑2 57.1%
- 3
OLMo 2 1B Instruct ↑20 57.1%
- 4
Llama 3-70B ↓2 56.9%
- 5
Llama 3.2 1B ↑14 56.9%
- 6
Llama-3.1-Tulu-3-8B ↑5 55.6%
- 7
Qwen3-8B-Base ↑2 55.6%
- 8
Qwen3-1.7B ↑12 55.6%
- 9
Qwen3-4B ↑8 55%
- 10
Qwen3-8B ↑4 54.5%
Upper Sorbian
25 models we track have a score here.
- 1
Gemma 3 27B PT 80.6%
- 2
Llama 3-70B 79.6%
- 3
Gemma 3 27B ↑5 78.5%
- 4
Aya Expanse 32B ↑2 75.3%
- 5
Gemma 3 12B PT ↓2 75.3%
- 6
Llama 3-8B ↓1 71.5%
- 7
Gemma 3 4B PT 67.7%
- 8
Gemma 3 12B ↑7 67.2%
- 9
Qwen3-14B-Base ↓5 66.7%
- 10
Llama 3.2 3B ↑3 66.1%
Albanian
25 models we track have a score here.
- 1
Gemma 3 27B PT 96.7%
- 2
Gemma 3 12B PT ↑1 95.9%
- 3
Gemma 3 27B ↑5 92.2%
- 4
Gemma 3 4B PT ↑3 91.4%
- 5
Gemma 3 12B ↑10 90.9%
- 6
Qwen3-14B-Base ↓2 90.5%
- 7
Qwen3-8B-Base ↑2 90.1%
- 8
Llama 3-70B ↓6 88.5%
- 9
Qwen3-8B ↑5 87.7%
- 10
Qwen3-4B-Base ↑2 86.4%
Skolt Sami
25 models we track have a score here.
- 1
Qwen3-8B ↑13 82.9%
- 2
Qwen3-8B-Base ↑7 79.8%
- 3
Llama 3.2 3B ↑10 78.7%
- 4
Qwen3-14B ↑6 78.7%
- 5
Qwen3-14B-Base ↓1 78.7%
- 6
Llama 3-8B ↓1 78.3%
- 7
Aya Expanse 32B ↓1 78.3%
- 8
Qwen3-4B ↑9 77.9%
- 9
Llama-3.1-Tulu-3-8B ↑2 77.6%
- 10
Qwen3-0.6B ↑12 77.6%
Kirghiz
25 models we track have a score here.
- 1
Llama 3-70B ↑1 96.8%
- 2
Gemma 3 12B PT ↑1 96.8%
- 3
Gemma 3 27B PT ↓2 96.8%
- 4
Gemma 3 4B PT ↑3 96.8%
- 5
Qwen3-14B-Base ↓1 96.2%
- 6
Qwen3-8B-Base ↑3 95.7%
- 7
Llama 3-8B ↓2 95.1%
- 8
Qwen3-0.6B ↑14 94.1%
- 9
Qwen3-14B ↑1 93.5%
- 10
Qwen3-4B-Base ↑2 93.5%
Urubú-Kaapor
25 models we track have a score here.
- 1
Gemma 3 1B ↑23 61.5%
- 2
Aya Expanse 8B ↑16 53.8%
- 3
Qwen3-14B-Base ↑1 53.8%
- 4
Qwen3-8B-Base ↑5 53.8%
- 5
Llama 3-70B ↓3 46.2%
- 6
Llama 3-8B ↓1 46.2%
- 7
Llama 3.2 1B ↑12 46.2%
- 8
Llama 3.2 3B ↑5 46.2%
- 9
Aya Expanse 32B ↓3 46.2%
- 10
Llama-3.1-Tulu-3-8B ↑1 46.2%
Nigerian Pidgin
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 100%
- 2
Gemma 3 12B PT ↑1 100%
- 3
Gemma 3 27B PT ↓2 100%
- 4
OLMo 2 32B Instruct ↑12 100%
- 5
Qwen3-14B-Base ↓1 100%
- 6
Qwen3-4B-Base ↑6 100%
- 7
Qwen3-8B-Base ↑2 100%
- 8
OLMo 2 1B Instruct ↑15 100%
- 9
Qwen3-4B ↑8 100%
- 10
Llama 3-70B ↓8 96.2%
Kazakh
25 models we track have a score here.
- 1
Gemma 3 12B PT ↑2 95.4%
- 2
Qwen3-8B-Base ↑7 95.4%
- 3
Llama 3-70B ↓1 94.8%
- 4
Gemma 3 27B PT ↓3 94.8%
- 5
Qwen3-14B-Base ↓1 94.2%
- 6
Qwen3-4B-Base ↑6 94.2%
- 7
Gemma 3 12B ↑8 93.6%
- 8
Gemma 3 4B PT ↓1 93.6%
- 9
Qwen3-14B ↑1 93.6%
- 10
Qwen3-8B ↑4 93.6%
Amharic
25 models we track have a score here.
- 1
Llama 3-8B ↑4 100%
- 2
Aya Expanse 32B ↑4 99.1%
- 3
Llama 3-70B ↓1 98.2%
- 4
Aya Expanse 8B ↑14 97.3%
- 5
Llama 3.2 3B ↑8 96.4%
- 6
Gemma 3 12B PT ↓3 96.4%
- 7
Gemma 3 27B PT ↓6 96.4%
- 8
OLMo 2 32B Instruct ↑8 96.4%
- 9
Qwen3-8B ↑5 96.4%
- 10
Qwen3-8B-Base ↓1 96.4%
Basque
25 models we track have a score here.
- 1
Gemma 3 27B PT 97.1%
- 2
Qwen3-8B-Base ↑7 96.3%
- 3
Gemma 3 12B PT 96%
- 4
Llama 3.2 3B ↑9 95.6%
- 5
Qwen3-14B ↑5 95.6%
- 6
Llama 3-70B ↓4 95.2%
- 7
Llama 3-8B ↓2 94.1%
- 8
Gemma 3 12B ↑7 94.1%
- 9
Qwen3-14B-Base ↓5 94.1%
- 10
Qwen3-8B ↑4 93.8%
Turkish
25 models we track have a score here.
- 1
Gemma 3 27B PT 97.2%
- 2
Gemma 3 12B PT ↑1 96.6%
- 3
Gemma 3 27B ↑5 94.6%
- 4
Llama 3-70B ↓2 94.1%
- 5
Aya Expanse 32B ↑1 93.9%
- 6
Gemma 3 4B PT ↑1 93.7%
- 7
Aya Expanse 8B ↑11 92.9%
- 8
Gemma 3 12B ↑7 92.9%
- 9
Qwen3-14B-Base ↓5 92.2%
- 10
Llama 3-8B ↓5 89.3%
Polish
25 models we track have a score here.
- 1
Gemma 3 27B PT 98.8%
- 2
Gemma 3 12B PT ↑1 97.9%
- 3
Aya Expanse 32B ↑3 97.6%
- 4
Llama 3-70B ↓2 96.7%
- 5
Gemma 3 4B PT ↑2 95.6%
- 6
Gemma 3 27B ↑2 95.1%
- 7
Qwen3-14B-Base ↓3 94.3%
- 8
Aya Expanse 8B ↑10 94.2%
- 9
Llama 3-8B ↓4 93.9%
- 10
Qwen3-14B 93%
Yakut
25 models we track have a score here.
- 1
Gemma 3 27B PT 88.2%
- 2
Llama 3-70B 82.6%
- 3
Gemma 3 12B PT 79.9%
- 4
Llama 3-8B ↑1 78.5%
- 5
Qwen3-8B ↑9 78.5%
- 6
OLMo 2 32B Instruct ↑10 77.8%
- 7
Qwen3-1.7B ↑13 77.8%
- 8
Gemma 3 1B ↑16 77.1%
- 9
Gemma 3 1B PT ↑16 77.1%
- 10
Aya Expanse 32B ↓4 76.4%
Highland Puebla Nahuatl
25 models we track have a score here.
- 1
Qwen3-8B-Base ↑8 73.4%
- 2
Aya Expanse 32B ↑4 70.5%
- 3
Qwen3-8B ↑11 69.6%
- 4
Llama 3-8B ↑1 69.1%
- 5
OLMo 2 1B Instruct ↑18 69.1%
- 6
Qwen3-14B ↑4 68.6%
- 7
Llama 3-70B ↓5 67.2%
- 8
OLMo 2 32B Instruct ↑8 67.2%
- 9
Qwen3-4B-Base ↑3 67.2%
- 10
Llama 3.2 3B ↑3 66.2%
English
25 models we track have a score here.
- 1
Llama 3.2 3B ↑12 99.5%
- 2
OLMo 2 32B Instruct ↑14 99.5%
- 3
Qwen3-4B-Base ↑9 99.5%
- 4
Llama 3-8B ↑1 99.4%
- 5
Llama 3-70B ↓3 99%
- 6
Llama 3.2 1B ↑13 99%
- 7
Aya Expanse 8B ↑11 99%
- 8
Gemma 3 12B PT ↓5 99%
- 9
Qwen3-8B-Base 98.7%
- 10
OLMo 2 1B Instruct ↑13 98.7%
Bhojpuri
25 models we track have a score here.
- 1
Llama 3-8B ↑4 85.3%
- 2
Llama 3-70B 82.4%
- 3
Aya Expanse 32B ↑3 82.4%
- 4
Gemma 3 27B PT ↓3 82.4%
- 5
Gemma 3 27B ↑3 79.4%
- 6
Llama 3.2 3B ↑7 76.5%
- 7
Gemma 3 12B PT ↓4 76.5%
- 8
Qwen3-14B ↑2 73.5%
- 9
Llama-3.1-Tulu-3-8B ↑2 70.6%
- 10
Gemma 3 4B PT ↓3 70.6%
Tupinambá
25 models we track have a score here.
- 1
Gemma 3 12B ↑14 33.3%
- 2
Qwen3-8B ↑12 33.3%
- 3
Aya Expanse 8B ↑15 22.2%
- 4
Gemma 3 1B ↑20 22.2%
- 5
Gemma 3 1B PT ↑20 22.2%
- 6
Gemma 3 4B ↑15 22.2%
- 7
Llama 3-70B ↓5 11.1%
- 8
Gemma 3 27B 11.1%
- 9
Gemma 3 12B PT ↓6 11.1%
- 10
Gemma 3 27B PT ↓9 11.1%
Bengali
25 models we track have a score here.
- 1
Llama-3.1-Tulu-3-8B ↑10 100%
- 2
Llama 3-70B 95.2%
- 3
Llama 3.2 1B ↑16 95.2%
- 4
Aya Expanse 32B ↑2 95.2%
- 5
Gemma 3 12B PT ↓2 95.2%
- 6
Gemma 3 27B PT ↓5 95.2%
- 7
Gemma 3 4B PT 95.2%
- 8
Qwen3-14B ↑2 95.2%
- 9
Qwen3-14B-Base ↓5 95.2%
- 10
Qwen3-4B-Base ↑2 95.2%
Komi-Permyak
25 models we track have a score here.
- 1
Qwen3-8B-Base ↑8 69.8%
- 2
Llama 3-8B ↑3 60.5%
- 3
Gemma 3 27B ↑5 60.5%
- 4
Gemma 3 27B PT ↓3 60.5%
- 5
Gemma 3 4B ↑16 60.5%
- 6
Gemma 3 12B PT ↓3 55.8%
- 7
Aya Expanse 32B ↓1 53.5%
- 8
Qwen3-14B-Base ↓4 53.5%
- 9
Qwen3-4B-Base ↑3 53.5%
- 10
Qwen3-1.7B ↑10 53.5%
Ottoman Turkish (1500-1928)
25 models we track have a score here.
- 1
Qwen3-14B-Base ↑3 98%
- 2
Llama 3-70B 97%
- 3
Gemma 3 12B PT 97%
- 4
Qwen3-14B ↑6 97%
- 5
Llama 3-8B 96%
- 6
Gemma 3 27B PT ↓5 96%
- 7
Qwen3-8B-Base ↑2 96%
- 8
Llama 3.2 3B ↑5 94.9%
- 9
Aya Expanse 8B ↑9 94.9%
- 10
Llama-3.1-Tulu-3-8B ↑1 94.9%
Macedonian
25 models we track have a score here.
- 1
Llama 3-70B ↑1 100%
- 2
Gemma 3 27B PT ↓1 97.4%
- 3
Llama 3-8B ↑2 94.9%
- 4
Gemma 3 12B PT ↓1 94.9%
- 5
Qwen3-14B-Base ↓1 94.9%
- 6
Qwen3-4B-Base ↑6 94.9%
- 7
Llama 3.2 3B ↑6 92.3%
- 8
Aya Expanse 32B ↓2 92.3%
- 9
Gemma 3 27B ↓1 92.3%
- 10
Qwen3-4B ↑7 92.3%
Georgian
25 models we track have a score here.
- 1
Qwen3-14B ↑9 98%
- 2
Llama 3-8B ↑3 97.5%
- 3
Qwen3-8B ↑11 97.1%
- 4
Llama 3-70B ↓2 96.6%
- 5
Llama-3.1-Tulu-3-8B ↑6 96.1%
- 6
Gemma 3 4B PT ↑1 96.1%
- 7
Aya Expanse 32B ↓1 95.6%
- 8
Qwen3-14B-Base ↓4 95.6%
- 9
Qwen3-4B-Base ↑3 95.6%
- 10
Qwen3-8B-Base ↓1 95.1%
Warlpiri
25 models we track have a score here.
- 1
Gemma 3 12B PT ↑2 41.7%
- 2
Llama 3.2 1B ↑17 33.3%
- 3
Qwen3-8B ↑11 33.3%
- 4
Llama 3-8B ↑1 25%
- 5
Llama 3.2 3B ↑8 25%
- 6
Llama-3.1-Tulu-3-8B ↑5 25%
- 7
Gemma 3 1B PT ↑18 25%
- 8
Gemma 3 27B PT ↓7 25%
- 9
Gemma 3 4B PT ↓2 25%
- 10
OLMo 2 32B Instruct ↑6 25%
Uzbek
25 models we track have a score here.
- 1
Llama 3-70B ↑1 100%
- 2
Gemma 3 27B ↑6 100%
- 3
Gemma 3 12B PT 100%
- 4
Gemma 3 27B PT ↓3 100%
- 5
Gemma 3 4B PT ↑2 100%
- 6
Qwen3-8B-Base ↑3 100%
- 7
Gemma 3 12B ↑8 98%
- 8
Gemma 3 4B ↑13 98%
- 9
Qwen3-14B-Base ↓5 98%
- 10
Qwen3-8B ↑4 98%
Phrygian
25 models we track have a score here.
- 1
Aya Expanse 8B ↑17 94%
- 2
Llama 3-8B ↑3 92%
- 3
Llama 3.2 3B ↑10 92%
- 4
Aya Expanse 32B ↑2 92%
- 5
Llama-3.1-Tulu-3-8B ↑6 92%
- 6
Llama 3-70B ↓4 90%
- 7
Gemma 3 12B ↑8 90%
- 8
Qwen3-8B-Base ↑1 90%
- 9
Qwen3-0.6B ↑13 90%
- 10
Qwen3-4B-Base ↑2 88%
Apurinã
25 models we track have a score here.
- 1
Llama 3-70B ↑1 96.4%
- 2
Llama 3-8B ↑3 96.4%
- 3
Llama 3.2 1B ↑16 96.4%
- 4
Llama-3.1-Tulu-3-8B ↑7 96.4%
- 5
Gemma 3 12B PT ↓2 96.4%
- 6
OLMo 2 32B Instruct ↑10 96.4%
- 7
Qwen3-14B ↑3 96.4%
- 8
Qwen3-14B-Base ↓4 96.4%
- 9
Qwen3-4B-Base ↑3 96.4%
- 10
Qwen3-8B ↑4 96.4%
Borôro
25 models we track have a score here.
- 1
Aya Expanse 32B ↑5 67.6%
- 2
Qwen3-8B ↑12 67.6%
- 3
OLMo 2 1B Instruct ↑20 67.6%
- 4
Llama 3.2 3B ↑9 67.2%
- 5
Qwen3-14B-Base ↓1 67.2%
- 6
Qwen3-8B-Base ↑3 66.8%
- 7
Qwen3-1.7B ↑13 66.8%
- 8
Llama 3-8B ↓3 66%
- 9
Qwen3-0.6B ↑13 65.6%
- 10
Llama-3.1-Tulu-3-8B ↑1 65.1%
Scottish Gaelic
25 models we track have a score here.
- 1
Llama 3-70B ↑1 98.5%
- 2
OLMo 2 32B Instruct ↑14 98.5%
- 3
Qwen3-8B-Base ↑6 98.5%
- 4
OLMo 2 1B Instruct ↑19 98.5%
- 5
Llama 3.2 1B ↑14 97%
- 6
Gemma 3 12B PT ↓3 97%
- 7
Gemma 3 27B PT ↓6 97%
- 8
Llama-3.1-Tulu-3-8B ↑3 95.5%
- 9
Gemma 3 27B ↓1 95.5%
- 10
Qwen3-8B ↑4 95.5%
Swedish
25 models we track have a score here.
- 1
Llama 3-70B ↑1 100%
- 2
Llama 3-8B ↑3 100%
- 3
Aya Expanse 32B ↑3 100%
- 4
Gemma 3 12B PT ↓1 100%
- 5
Gemma 3 27B PT ↓4 100%
- 6
Gemma 3 4B PT ↑1 100%
- 7
Qwen3-14B-Base ↓3 100%
- 8
Qwen3-8B-Base ↑1 100%
- 9
Llama 3.2 1B ↑10 99.5%
- 10
Llama 3.2 3B ↑3 99.5%
Urdu
25 models we track have a score here.
- 1
Llama 3-70B ↑1 97.5%
- 2
Gemma 3 12B PT ↑1 97.5%
- 3
Gemma 3 27B PT ↓2 97.5%
- 4
Qwen3-14B ↑6 97.1%
- 5
Qwen3-14B-Base ↓1 96.9%
- 6
Llama 3-8B ↓1 96.7%
- 7
Gemma 3 27B ↑1 96.7%
- 8
OLMo 2 32B Instruct ↑8 96.5%
- 9
Gemma 3 4B PT ↓2 96.4%
- 10
Qwen3-8B-Base ↓1 96.4%
Egyptian (Ancient)
25 models we track have a score here.
- 1
Gemma 3 1B ↑23 59.1%
- 2
Gemma 3 1B PT ↑23 59.1%
- 3
Qwen3-14B-Base ↑1 54.5%
- 4
OLMo 2 1B Instruct ↑19 54.5%
- 5
Llama 3-8B 50%
- 6
Aya Expanse 32B 50%
- 7
Llama-3.1-Tulu-3-8B ↑4 50%
- 8
Gemma 3 27B PT ↓7 50%
- 9
Gemma 3 4B ↑12 50%
- 10
Qwen3-14B 50%
Danish
25 models we track have a score here.
- 1
Llama 3-70B ↑1 100%
- 2
Llama 3-8B ↑3 100%
- 3
Llama 3.2 3B ↑10 100%
- 4
Aya Expanse 32B ↑2 100%
- 5
Gemma 3 27B ↑3 100%
- 6
Gemma 3 12B PT ↓3 100%
- 7
Gemma 3 27B PT ↓6 100%
- 8
Gemma 3 4B PT ↓1 100%
- 9
Qwen3-14B ↑1 100%
- 10
Llama 3.2 1B ↑9 98%
Gujarati
25 models we track have a score here.
- 1
Llama 3-70B ↑1 100%
- 2
Llama 3-8B ↑3 100%
- 3
Llama 3.2 1B ↑16 100%
- 4
Llama 3.2 3B ↑9 100%
- 5
Aya Expanse 32B ↑1 100%
- 6
Aya Expanse 8B ↑12 100%
- 7
Llama-3.1-Tulu-3-8B ↑4 100%
- 8
Gemma 3 27B 100%
- 9
Gemma 3 12B ↑6 100%
- 10
Gemma 3 12B PT ↓7 100%
Subject-participle number
25 models we track have a score here.
- 1
Gemma 3 27B PT 92.8%
- 2
Llama 3-70B 92.1%
- 3
Gemma 3 12B PT 91.1%
- 4
Qwen3-14B-Base 90.2%
- 5
Llama 3-8B 89.4%
- 6
Gemma 3 4B PT ↑1 88.8%
- 7
Qwen3-8B-Base ↑2 88.7%
- 8
Gemma 3 27B 88.7%
- 9
Qwen3-14B ↑1 87.8%
- 10
Aya Expanse 32B ↓4 87.4%
Subject-verb person
25 models we track have a score here.
- 1
Llama 3-70B ↑1 91.2%
- 2
Gemma 3 27B PT ↓1 91.1%
- 3
Gemma 3 12B PT 90.2%
- 4
Qwen3-14B-Base 88.2%
- 5
Llama 3-8B 87.8%
- 6
Aya Expanse 32B 87.8%
- 7
Gemma 3 4B PT 87.2%
- 8
Gemma 3 27B 87%
- 9
Qwen3-8B-Base 86.8%
- 10
Qwen3-14B 86.4%
Subject-participle gender
25 models we track have a score here.
- 1
Gemma 3 27B PT 91.7%
- 2
Llama 3-70B 91.2%
- 3
Gemma 3 12B PT 90.3%
- 4
Qwen3-14B-Base 88.7%
- 5
Llama 3-8B 88%
- 6
Aya Expanse 32B 87%
- 7
Gemma 3 27B ↑1 86.7%
- 8
Qwen3-14B ↑2 86.6%
- 9
Qwen3-8B-Base 86.4%
- 10
Gemma 3 4B PT ↓3 86%
Subject-verb number
25 models we track have a score here.
- 1
Llama 3-70B ↑1 87.4%
- 2
Gemma 3 27B PT ↓1 87.2%
- 3
Gemma 3 12B PT 85.4%
- 4
Qwen3-14B-Base 84.6%
- 5
Llama 3-8B 84.2%
- 6
Qwen3-8B-Base ↑3 83.2%
- 7
Aya Expanse 32B ↓1 82.9%
- 8
Gemma 3 4B PT ↓1 82.8%
- 9
Gemma 3 27B ↓1 82.7%
- 10
Qwen3-14B 82.2%
Subject-verb gender
25 models we track have a score here.
- 1
Gemma 3 27B PT 91.3%
- 2
Llama 3-70B 91.1%
- 3
Gemma 3 12B PT 89.6%
- 4
Qwen3-14B-Base 88.4%
- 5
Aya Expanse 32B ↑1 88.1%
- 6
Llama 3-8B ↓1 88%
- 7
Qwen3-14B ↑3 86.4%
- 8
Gemma 3 27B 86.3%
- 9
Qwen3-8B-Base 86.2%
- 10
Gemma 3 4B PT ↓3 86%
Subject-participle person
25 models we track have a score here.
- 1
Llama 3-70B ↑1 97.5%
- 2
Gemma 3 27B PT ↓1 96.6%
- 3
Gemma 3 27B ↑5 95.8%
- 4
Gemma 3 12B PT ↓1 95.5%
- 5
Aya Expanse 32B ↑1 95.1%
- 6
Llama-3.1-Tulu-3-8B ↑5 94.9%
- 7
Llama 3-8B ↓2 94%
- 8
Aya Expanse 8B ↑10 93.3%
- 9
Gemma 3 4B PT ↓2 93.3%
- 10
Gemma 3 12B ↑5 93%
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official MultiBLiMP leaderboard
Llama 3-70B on the official leaderboard passed 85% in April 2024
Models released before it came out already scored that high. All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official MultiBLiMP leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
K-EXAONE 90.5%
- MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language.
Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7%
- Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Claude Opus 5 95.7%
- MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the official leaderboard (40 entries).
- Pass Models scored 25 models we track, plus 15 other entries on its leaderboard.
- Note Newest score The newest came 19 months ago, in April 2025, for OLMo 2 1B Instruct. That’s expected for a saturated test.
- Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is Qwen3-14B (#139).
- Note Headroom Saturated. The best is 90.2%, past 85% since April 2024.
Leaderboard import 3 pass
- Pass Importer We read the board on github.com: 27 rows for models we track and 15 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.