Benchmark Almanac

MultiBLiMP

Gemma 3 27B PT 90.2% leads since April 2025
Not in model cards

Models compare minimally different sentences across languages. The benchmark tests whether the model assigns higher likelihood to the grammatical sentence.

Made by
Jaap Jumelet and colleagues
Released
April 2025
Kind
Reasoning in many languages
Level
Everyday
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Measures
Capability
Language
Many languages
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
In views
Many languages
Strongest model tested
Qwen3-14B #139 on the capability index
Last model added
April 2025
Scores refreshed
November 20th, 2026
40
Entries on its own leaderboard, 25 models we track

Gemma 3 27B PT leads with 90.2% among models released by April 2025 Gemma 3 27B PT leads with 90.2% among models released by April 2025

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in April 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Gemma 3 27B PT 90.2%
  2. 2 Llama 3-70B 90.2%
  3. 3 Llama3-70B-it 89.3%
  4. 4 Gemma 3 12B PT 88.8%
  5. 5 Qwen3-14B-Base 87.4%
  6. 6 Llama 3-8B 86.9%
  7. 7 Aya Expanse 32B 86.4%
  8. 8 Gemma 3 4B PT 85.8%
  9. 9 EuroLLM-9B 85.8%
  10. 10 Gemma 3 27B 85.8%
  11. 11 Qwen3-8B-Base 85.8%
  12. 12 Llama3-8B-it 85.6%
  13. 13 EuroLLM-9B-it 85.4%
  14. 14 Qwen3-14B 85.3%
  15. 15 Llama-3.1-Tulu-3-8B 84.9%
  16. 16 Qwen3-4B-Base 83.7%
  17. 17 Llama 3.2 3B 83.6%
  18. 18 Qwen3-8B 83.1%
  19. 19 Gemma 3 12B 82.9%
  20. 20 OLMo 2 32B Instruct 82.7%
  21. 21 EuroLLM-1.7B 81.9%
  22. 22 EuroLLM-1.7B-it 81.5%
  23. 23 Qwen3-1.7B (base) 80.8%
  24. 24 Qwen3-4B 80.7%
  25. 25 Aya Expanse 8B 80.4%
  26. 26 Llama 3.2 1B 80.1%
  27. 27 OLMo-2-13B 79.9%
  28. 28 Llama3-3B-it 78.4%
  29. 29 OLMo-2-7B 77.6%
  30. 30 Qwen3-1.7B 77.3%
  31. 31 Gemma 3 4B 77.1%
  32. 32 Qwen3-0.6B (base) 75.8%
  33. 33 OLMo-2-13B-it 75.6%
  34. 34 OLMo-2-7B-it 74.8%
  35. 35 Llama3-1B-it 73.7%
  36. 36 Qwen3-0.6B 71.8%
  37. 37 OLMo 2 1B Instruct 71%
  38. 38 gpt2-xl 63.4%
  39. 39 Gemma 3 1B 55.5%
  40. 40 Gemma 3 1B PT 55.3%

Source: Official MultiBLiMP leaderboard

#EntryScoreSettingDateSource
1 Gemma 3 27B PT 90.2% Mar 2025 Official leaderboard
2 Llama 3-70B 90.2% Apr 2024 Official leaderboard
3 Llama3-70B-it 89.3% Official leaderboard
4 Gemma 3 12B PT 88.8% Mar 2025 Official leaderboard
5 Qwen3-14B-Base 87.4% Apr 2025 Official leaderboard
6 Llama 3-8B 86.9% Apr 2024 Official leaderboard
7 Aya Expanse 32B 86.4% Oct 2024 Official leaderboard
8 Gemma 3 4B PT 85.8% Mar 2025 Official leaderboard
9 EuroLLM-9B 85.8% Official leaderboard
10 Gemma 3 27B 85.8% Mar 2025 Official leaderboard
11 Qwen3-8B-Base 85.8% Apr 2025 Official leaderboard
12 Llama3-8B-it 85.6% Official leaderboard
13 EuroLLM-9B-it 85.4% Official leaderboard
14 Qwen3-14B 85.3% Apr 2025 Official leaderboard
15 Llama-3.1-Tulu-3-8B 84.9% Nov 2024 Official leaderboard
16 Qwen3-4B-Base 83.7% Apr 2025 Official leaderboard
17 Llama 3.2 3B 83.6% Sep 2024 Official leaderboard
18 Qwen3-8B 83.1% Apr 2025 Official leaderboard
19 Gemma 3 12B 82.9% Mar 2025 Official leaderboard
20 OLMo 2 32B Instruct 82.7% Mar 2025 Official leaderboard
21 EuroLLM-1.7B 81.9% Official leaderboard
22 EuroLLM-1.7B-it 81.5% Official leaderboard
23 Qwen3-1.7B (base) 80.8% Official leaderboard
24 Qwen3-4B 80.7% Apr 2025 Official leaderboard
25 Aya Expanse 8B 80.4% Oct 2024 Official leaderboard
26 Llama 3.2 1B 80.1% Sep 2024 Official leaderboard
27 OLMo-2-13B 79.9% Official leaderboard
28 Llama3-3B-it 78.4% Official leaderboard
29 OLMo-2-7B 77.6% Official leaderboard
30 Qwen3-1.7B 77.3% Apr 2025 Official leaderboard
31 Gemma 3 4B 77.1% Mar 2025 Official leaderboard
32 Qwen3-0.6B (base) 75.8% Official leaderboard
33 OLMo-2-13B-it 75.6% Official leaderboard
34 OLMo-2-7B-it 74.8% Official leaderboard
35 Llama3-1B-it 73.7% Official leaderboard
36 Qwen3-0.6B 71.8% Apr 2025 Official leaderboard
37 OLMo 2 1B Instruct 71% Apr 2025 Official leaderboard
38 gpt2-xl 63.4% Official leaderboard
39 Gemma 3 1B 55.5% Mar 2025 Official leaderboard
40 Gemma 3 1B PT 55.3% Mar 2025 Official leaderboard
#ModelScoreSettingDateSource
1 Gemma 3 27B PT 90.2% Mar 2025 Official leaderboard
2 Llama 3-70B 90.2% Apr 2024 Official leaderboard
3 Gemma 3 12B PT 88.8% Mar 2025 Official leaderboard
4 Qwen3-14B-Base 87.4% Apr 2025 Official leaderboard
5 Llama 3-8B 86.9% Apr 2024 Official leaderboard
6 Aya Expanse 32B 86.4% Oct 2024 Official leaderboard
7 Gemma 3 4B PT 85.8% Mar 2025 Official leaderboard
8 Gemma 3 27B 85.8% Mar 2025 Official leaderboard
9 Qwen3-8B-Base 85.8% Apr 2025 Official leaderboard
10 Qwen3-14B 85.3% Apr 2025 Official leaderboard
11 Llama-3.1-Tulu-3-8B 84.9% Nov 2024 Official leaderboard
12 Qwen3-4B-Base 83.7% Apr 2025 Official leaderboard
13 Llama 3.2 3B 83.6% Sep 2024 Official leaderboard
14 Qwen3-8B 83.1% Apr 2025 Official leaderboard
15 Gemma 3 12B 82.9% Mar 2025 Official leaderboard
16 OLMo 2 32B Instruct 82.7% Mar 2025 Official leaderboard
17 Qwen3-4B 80.7% Apr 2025 Official leaderboard
18 Aya Expanse 8B 80.4% Oct 2024 Official leaderboard
19 Llama 3.2 1B 80.1% Sep 2024 Official leaderboard
20 Qwen3-1.7B 77.3% Apr 2025 Official leaderboard
21 Gemma 3 4B 77.1% Mar 2025 Official leaderboard
22 Qwen3-0.6B 71.8% Apr 2025 Official leaderboard
23 OLMo 2 1B Instruct 71% Apr 2025 Official leaderboard
24 Gemma 3 1B 55.5% Mar 2025 Official leaderboard
25 Gemma 3 1B PT 55.3% Mar 2025 Official leaderboard

Rankings by part

We read 107 parts from the benchmark’s own leaderboard in 2 groups: language and phenomenon. Each ranks models on one part of the test. Llama 3-70B leads 34 of them, and 19 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.

Old Russian

25 models we track have a score here.

  1. 1 Llama 3-70B ↑1 80%
  2. 2 Gemma 3 27B PT ↓1 78.6%
  3. 3 Aya Expanse 32B ↑3 77.6%
  4. 4 Llama 3-8B ↑1 76.9%
  5. 5 Gemma 3 12B PT ↓2 74.9%
  6. 6 Qwen3-14B-Base ↓2 74.7%
  7. 7 Qwen3-8B-Base ↑2 73.4%
  8. 8 Llama-3.1-Tulu-3-8B ↑3 73.1%
  9. 9 Qwen3-14B ↑1 72.7%
  10. 10 Gemma 3 27B ↓2 72.5%

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind
Own leaderboard Passed 85% in Apr 2025, by Gemma 3 27B PT

Source: Official MultiBLiMP leaderboard

Llama 3-70B on the official leaderboard passed 85% in April 2024

Models released before it came out already scored that high. All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 2025 Near the ceiling (85%) Gemma 3 27B PT Llama 3-70B 0% 20% 40% 60% 80% 100% 2024 2025 Near the ceiling (85%) Gemma 3 27B PT Llama 3-70B

Source: Official MultiBLiMP leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Multilingual

All 545 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Claude Opus 5 95.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%

Data audit

All 8 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the official leaderboard (40 entries).
  • Pass Models scored 25 models we track, plus 15 other entries on its leaderboard.
  • Note Newest score The newest came 19 months ago, in April 2025, for OLMo 2 1B Instruct. That’s expected for a saturated test.
  • Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is Qwen3-14B (#139).
  • Note Headroom Saturated. The best is 90.2%, past 85% since April 2024.
Leaderboard import 3 pass
  • Pass Importer We read the board on github.com: 27 rows for models we track and 15 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check