Flores
Gemma 3 27B 48.8 leads since March 2025 Models translate 3,001 English Wikipedia sentences that professionals translated into 101 languages. Scored with BLEU or chrF against references.
FLORES-200 extended it to 200 languages in 2022, and the Open Language Data Initiative now maintains it.
- Made by
- Meta AI
- Released
- June 2021
- Kind
- Translation
- Level
- University
- Scored
- Checked automatically
- Answers
- Open-ended
- Tasks
- All public
- Strongest model tested
- Gemma 3 27B #164 on the capability index
- Last in a launch post
- March 2025, dropped for a newer version
- Last model added
- May 2025
In the authors’ words
One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the FLORES-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are multilingually aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.
Gemma 3 27B leads with 48.8 among models released by March 2025
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in March 2025, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- Claude Fable 5.2
- Gemini 4 Carbon
- GPT-6 Astra
- Claude Opus 5.5
- and 16 more
- 1 Gemma 3 27B 48.8
- 2 Gemma 3 12B 46.0
- 3 Gemma 3 4B 39.2
- 4
Qwen2.5-72B 39.0 - 5 Qwen2-72B 37.8
- 6
Aya Vision 32B 37.8 - 7 Qwen1.5-72B 35.6
- 8 Gemma 3 1B 29.5
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Gemma 3 27B | 48.8 | Mar 2025 | Lab model card | |
| 2 | Gemma 3 12B | 46.0 | Mar 2025 | Lab model card | |
| 3 | Gemma 3 4B | 39.2 | Mar 2025 | Lab model card | |
| 4 | Qwen2.5-72B | 39.0 | Sep 2024 | Lab model card | |
| 5 | Qwen2-72B | 37.8 | Jun 2024 | Lab model card | |
| 6 | Aya Vision 32B | 37.8 | May 2025 | Lab model card | |
| 7 | Qwen1.5-72B | 35.6 | Feb 2024 | Lab model card | |
| 8 | Gemma 3 1B | 29.5 | Mar 2025 | Lab model card |
4 models have held number one in labs’ own numbers since 2023
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Labs’ model cards
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Labs’ model cards
Labs moved on to FLORES-200
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Mar 12, 2025 | Gemma 3 27B | 48.8 | FloRes (PT) |
| Feb 17, 2025 |  Mistral Saba | Arabic-Centric FLORES-101 (chart, metric not stated) | |
| Sep 19, 2024 | Qwen2.5-72B | 39 | Multi-Translation |
| Jun 7, 2024 | Qwen2-72B | 37.8 | Multi-Translation |
| Feb 5, 2024 | Qwen1.5-72B | 35.6 | Multilingual Translation (Flores, BLEU) |
5 system cards report it, from 2 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| May 13, 2025 | Aya Vision 32B technical report | 37.8 | FLORES, En to X, spBLEU, 23 languages (Table 5) | Capabilities |
| Mar 25, 2025 | Gemma 3 4B technical report | 39.2 | FLoRes, chrF, PT, 1-shot (Table 13) | Capabilities |
| Mar 25, 2025 | Gemma 3 27B technical report | 48.8 | FLoRes, chrF, PT, 1-shot (Table 13) | Capabilities |
| Mar 25, 2025 | Gemma 3 1B technical report | 29.5 | FLoRes, chrF, PT, 1-shot (Table 13) | Capabilities |
| Mar 25, 2025 | Gemma 3 12B technical report | 46 | FLoRes, chrF, PT, 1-shot (Table 13) | Capabilities |
Versions
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
K-EXAONE 90.5% - MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7% - Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy.
Nemotron 3 Ultra 550B-A55B Base 90.1% - MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%