WMT24++
K-EXAONE 90.5% leads since December 2025 English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
Cohere added internal Irish, Maltese and Serbian variants.
- Made by
- Released
- February 2025
- Kind
- Translation
- Level
- University
- Scored
- Graded by a model
- Answers
- Open-ended
- Tasks
- All public
- Strongest model tested
-
Qwen3.7-Max #49 on the capability index - Last in a launch post
- October 2026
- Last model added
- October 2026
In the authors’ words
As large language models (LLM) become more and more capable in languages other than English, it is important to collect benchmark datasets in order to evaluate their multilingual performance, including on tasks like machine translation (MT). In this work, we extend the WMT24 dataset to cover 55 languages by collecting new human-written references and post-edits for 46 new languages and dialects in addition to post-edits of the references in 8 out of 9 languages in the original WMT24 dataset. The dataset covers four domains: literary, news, social, and speech. We benchmark a variety of MT providers and LLMs on the collected dataset using automatic metrics and find that LLMs are the best-performing MT systems in all 55 languages. These results should be confirmed using a human-based evaluation, which we leave for future work.
15
Models scored in lab cards
25
Launch posts list it, from 6 labs
19
System cards report it, from 6 labs
K-EXAONE leads with 90.5%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 90.5 K-EXAONE
- 86.7
Nemotron 3 Super - 86.2 Nemotron 3 Nano 30B-A3B
- 85.8
Hunyuan-MT-7B - 85.8 Qwen3.7-Max
- 84.8 Nemotron 3.5 Super VL
- 84.4
TranslateGemma 27B - 84.3 Qwen 3.6 Plus
- 84.1 Nemotron-Cascade 2 30B-A3B
- 83.8
Command A Translate - 83.7 Nemotron 3 Ultra
- 83.5 TranslateGemma 12B
- 81.0 Command A+
- 80.1 TranslateGemma 4B
- 78.9 Qwen3.5 397B-A17B
- 1 K-EXAONE 90.5%
- 2 Nemotron 3 Super 86.7%
- 3 Nemotron 3 Nano 30B-A3B 86.2%
- 4 Hunyuan-MT-7B 85.8%
- 5 Qwen3.7-Max 85.8%
- 6 Nemotron 3.5 Super VL 84.8%
- 7 TranslateGemma 27B 84.4%
- 8 Qwen 3.6 Plus 84.3%
- 9 Nemotron-Cascade 2 30B-A3B 84.1%
- 10 Command A Translate 83.8%
- 11 Nemotron 3 Ultra 83.7%
- 12 TranslateGemma 12B 83.5%
- 13 Command A+ 81%
- 14 TranslateGemma 4B 80.1%
- 15 Qwen3.5 397B-A17B 78.9%
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | K-EXAONE | 90.5% | Dec 2025 | Lab model card | |
| 2 | Nemotron 3 Super | 86.7% | Mar 2026 | Lab model card | |
| 3 | Nemotron 3 Nano 30B-A3B | 86.2% | Dec 2025 | Lab model card | |
| 4 | Hunyuan-MT-7B | 85.8% | Sep 2025 | Lab model card | |
| 5 | Qwen3.7-Max | 85.8% | May 2026 | Lab model card | |
| 6 | Nemotron 3.5 Super VL | 84.8% | Oct 2026 | Lab model card | |
| 7 | TranslateGemma 27B | 84.4% | Jan 2026 | Lab model card | |
| 8 | Qwen 3.6 Plus | 84.3% | Apr 2026 | Lab model card | |
| 9 | Nemotron-Cascade 2 30B-A3B | 84.1% | Mar 2026 | Lab model card | |
| 10 | Command A Translate | 83.8% | Aug 2025 | Lab model card | |
| 11 | Nemotron 3 Ultra | 83.7% | Jun 2026 | Lab model card | |
| 12 | TranslateGemma 12B | 83.5% | Jan 2026 | Lab model card | |
| 13 | Command A+ | 81% | May 2026 | Lab model card | |
| 14 | TranslateGemma 4B | 80.1% | Jan 2026 | Lab model card | |
| 15 | Qwen3.5 397B-A17B | 78.9% | Feb 2026 | Lab model card |
4 models have held number one in labs’ own numbers since its release in February 2025
Lab claims count from the day the card came out, independent runs from the model’s release.
Cohere Tencent NVIDIA LG AI Research
Source: Labs’ model cards
Tencent’s own number for Hunyuan-MT-7B passed 85% in September 2025
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Reported by the lab in its model card
Best so far
Source: Labs’ model cards
2 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Oct 15, 2026 | Nemotron 3.5 Super VL | 84.8 | WMT24++ (COMET) |
| Jun 4, 2026 | Nemotron 3 Ultra | 83.7 | WMT24++ (en→xx) |
| Jun 1, 2026 | Qwen3.7-Plus | 84.6 | WMT24++ (XCOMET-XXL, 55 languages) |
| May 20, 2026 | Command A+ | 81 | WMT24++ (xCOMET-XL) |
| May 20, 2026 | Qwen3.7-Max | 85.8 | WMT24++ |
| Apr 2, 2026 | Qwen 3.6 Plus | 84.3 | WMT24++ |
| Mar 19, 2026 | Nemotron-Cascade 2 30B-A3B | 84.1 | WMT24++ (en -> xx) |
| Mar 11, 2026 | Nemotron 3 Super | 86.7 | WMT24++ (en→xx) |
| Mar 2, 2026 | Qwen3.5-0.8B | 27.2 | WMT24++ |
| Mar 2, 2026 | Qwen3.5-2B | 45.8 | WMT24++ |
| Mar 2, 2026 | Qwen3.5-4B | 66.6 | WMT24++ |
| Mar 2, 2026 | Qwen3.5-9B | 72.6 | WMT24++ |
| Feb 24, 2026 | Qwen3.5-35B-A3B | 76.3 | WMT24++ |
| Feb 24, 2026 | Qwen3.5-27B | 77.6 | WMT24++ |
| Feb 24, 2026 | Qwen3.5-122B-A10B | 78.3 | WMT24++ (XCOMET-XXL, 55 languages) |
| Feb 16, 2026 | Qwen3.5 397B-A17B | 78.9 | WMT24++ |
| Jan 15, 2026 | TranslateGemma 4B | 80.1 | TranslateGemma 4B, COMET-22, WMT24++ 55-language average |
| Jan 15, 2026 | TranslateGemma 12B | 83.5 | TranslateGemma 12B, COMET-22, WMT24++ 55-language average |
| Jan 15, 2026 | TranslateGemma 27B | 84.4 | TranslateGemma 27B, COMET-22, WMT24++ 55-language average |
| Dec 31, 2025 | K-EXAONE | 90.5 | WMT24++ |
| Dec 15, 2025 | Nemotron 3 Nano 30B-A3B | 86.2 | WMT24++ (en->xx) |
| Sep 1, 2025 | Hunyuan-MT-7B | 85.8 | WMT24pp (XCOMET-XXL) |
| Aug 28, 2025 | Command A Translate | 83.8 | WMT24++ xCometXL, average over 23 languages, English to L2 |
| Jun 26, 2025 | Gemma 3n E4B | 50.1 | WMT24++ (ChrF) 0-shot (IT) |
| Mar 12, 2025 | Gemma 3 27B | 55.7 | WMT24++ (ChrF) (PT); IT 0-shot = 53.4 |
19 system cards report it, from 6 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Oct 23, 2026 | Nemotron 3.5 Bolt E4B technical report | 74.4 | WMT24++ English to X xCOMET, average of 36 languages, BF16 non-reasoning. chrF 52.5, X to English xCOMET 83.4 (Table 6) | Capabilities |
| Oct 19, 2026 |  Muse Glimmer technical report | 53.4 | WMT24++, chrF++, high. X to eng 55.7, eng to Y 51.0 (Tables 5 and 9) | Capabilities |
| Oct 15, 2026 | Nemotron 3.5 Super VL technical report | 84.8 | WMT24pp COMET (XCOMET-XXL, en to 9 languages) (Table 10) | Capabilities |
| Sep 12, 2026 | North Small Translate 1.0 technical report | 80.6 | WMT24++ xCOMET-XL (Table 1 and Section 3.2). Agentic 80.7 | Capabilities |
| Aug 5, 2026 | K-EXAONE 2.0 technical report | WMT24++ per-language EN to XX and XX to EN scores for nine languages, no overall (Table 9) | Capabilities | |
| Jun 9, 2026 | Nemotron 3 Ultra technical report | 83.7 | WMT24++ (en to xx) (Table 10) | Capabilities |
| Apr 9, 2026 | EXAONE 4.5 33B technical report | 91.5 | WMT24++ (ko, de, es, ja, vi), metric not stated (Table 3, Reasoning mode) | Capabilities |
| Mar 19, 2026 | Nemotron-Cascade 2 30B-A3B technical report | 84.1 | WMT24++ (en -> xx), XCOMET-XXL, 5 language pairs (Table 1 and A.7) | Capabilities |
| Mar 11, 2026 | Nemotron 3 Super technical report | 86.7 | WMT24++ (en to xx) (Table 5) | Capabilities |
| Jan 21, 2026 | TranslateGemma 27B technical report | 84.4 | TranslateGemma 27B, COMET-22, WMT24++ 55-language average | Other |
| Jan 5, 2026 | K-EXAONE technical report | 90.5 | WMT24++ (ko, de, es, ja, vi), GPT-5-mini judge score 0 to 100 (Table 3, Reasoning mode). Per direction in Table 6. Non-reasoning 88.0 | Capabilities |
| Dec 23, 2025 | Nemotron 3 Nano 30B-A3B technical report | 86.2 | WMT24++ (en to xx) (Table 3) | Capabilities |
| Sep 26, 2025 | Command A Translate technical report | 83.9 | WMT24++ xComet-XL Avg over 23 languages, English to each (Tables 1 and 2). The launch blog prints 83.8 | Capabilities |
| Sep 5, 2025 | Hunyuan-MT-7B technical report | 85.8 | WMT24pp XCOMET-XXL, 29 English-centric pairs (Table 4). CometKiwi 80.61 | Capabilities |
| Jul 15, 2025 | EXAONE 4.0 32B technical report | 90.7 | WMT24++ (ES), EN and ES pair, GPT-4.1 judge score 0 to 100, Non-reasoning only (Table 4) | Capabilities |
| Mar 25, 2025 | Gemma 3 4B technical report | 46.8 | WMT24++, chrF, IT, 0-shot (Table 18) | Capabilities |
| Mar 25, 2025 | Gemma 3 27B technical report | 53.4 | WMT24++, chrF, IT, 0-shot (Table 18) | Capabilities |
| Mar 25, 2025 | Gemma 3 1B technical report | 35.9 | WMT24++, chrF, IT, 0-shot (Table 18) | Capabilities |
| Mar 25, 2025 | Gemma 3 12B technical report | 51.6 | WMT24++, chrF, IT, 0-shot (Table 18) | Capabilities |
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7% - Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Nemotron 3 Ultra 550B-A55B Base 90.1%
- MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%
- Alyah In use Models answer 1,173 four-option questions written by native Emirati speakers on greetings, idioms, figurative language, heritage and poetry in Emirati Arabic dialect. Gemma 3 27B 74.7%