Benchmark Almanac

BOUQuET

OMT-LLama-8B 35.2 tops its leaderboard Gemma 4 31B IT 35.0 leads the models we track
Muse Glimmer 36.5 leads since October 2026
Only in system cards

Models translate handcrafted multi-way paragraphs from several non-English source languages across many registers and domains. Scored with chrF++ against human references.

The paper describes a collaborative initiative to extend the dataset to more languages through crowd sourcing.

Released
February 2025
Kind
Translation
Level
Everyday
Scored
Checked automatically
Answers
Open-ended
Tasks
All public
Strongest model tested
Gemma 4 31B IT #100 on the capability index
Last in a system card
October 2026
Last model added
October 2026
Scores refreshed
November 12th, 2026

In the authors’ words

BOUQuET is a multi-way, multicentric and multi-register/domain dataset and benchmark, and a broader collaborative initiative. This dataset is handcrafted in 8 non-English languages. Each of these source languages are representative of the most widely spoken ones and therefore they have the potential to serve as pivot languages that will enable more accurate translations. The dataset is multicentric to enforce representation of multilingual language features. In addition, the dataset goes beyond the sentence level, as it is organized in paragraphs of various lengths. Compared with related machine translation datasets, we show that BOUQuET has a broader representation of domains while simplifying the translation task for non-experts. Therefore, BOUQuET is specially suitable for crowd-source extension for which we are launching a call aiming at collecting a multi-way parallel corpus covering any written language.

Abstract of BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation (arXiv:2502.04314, CC0)
45
Entries on its own leaderboard, 18 models we track
1
System card reports it, from 1 lab

OMT-LLama-8B tops the leaderboard with 35.2

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Gemma 4 31B IT, at 35.0.

Muse Glimmer leads with 36.5, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in April 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

45 entries. Scroll sideways for the rest.

  1. 35.2 OMT-LLama-8B
  2. 35.0 Gemma 4 31B IT
  3. 34.2 OMT-NLLB
  4. 34.1 Muse-Glimmer-30B
  5. 33.1 gemma-4-26B-A4B-it
  6. 32.5 OMT-LLama-3B
  7. 31.8 nllb-200-3.3B
  8. 31.4 Qwen3.5-35B-A3B
  9. 31.3 gpt-oss-120b
  10. 31.2 nllb-200-1.3B
  11. 31.2 Qwen3.5-27B
  12. 30.6 Llama 3.3 70B Instruct
  13. 30.1 nllb-200-distilled-600M
  14. 29.5 Gemma 3 27B
  15. 29.2 gemma-3-12b-it
  16. 29.0 OMT-LLama-1B
  17. 28.6 Gemma 4 E4B
  18. 28.5 gpt-oss-20b
  19. 28.4 Qwen3.5-9B
  20. 28.1 madlad400-10b-mt
  21. 26.4 Tower-Plus-72B
  22. 26.2 Qwen3-32B
  23. 26.1 Aya Expanse 32B
  24. 26.0 Gemma 4 E2B
  25. 24.7 Qwen3.5-4B
  26. 24.2 gemma-3-4b-it
  27. 23.4 Ministral-3-14B-Instruct-2512
  28. 23.4 Tower-Plus-9B
  29. 23.3 Llama-3.1-8B-Instruct
  30. 22.3 Qwen3-8B
  31. 22.3 EuroLLM-22B-Instruct-2512
  32. 21.7 Aya Expanse 8B
  33. 21.5 Hunyuan-MT-7B
  34. 21.5 Tiny Aya Global
  35. 21.1 Qwen2.5-7B-Instruct
  36. 20.1 EuroLLM-9B-Instruct
  37. 19.2 Qwen3.5-2B
  38. 19.0 salamandraTA-7b-instruct
  39. 18.8 Olmo-3-7B-Instruct
  40. 18.6 Llama-3.2-3B-Instruct
  41. 17.3 TowerInstruct-7B-v0.2
  42. 16.4 gemma-3-1b-it
  43. 14.5 Marco-MT-Algharb
  44. 13.5 Llama-3.2-1B-Instruct
  45. 10.4 aya-101
  1. 1 OMT-LLama-8B 35.2
  2. 2 Gemma 4 31B IT 35.0
  3. 3 OMT-NLLB 34.2
  4. 4 Muse-Glimmer-30B reasoning high 34.1
  5. 5 gemma-4-26B-A4B-it 33.1
  6. 6 OMT-LLama-3B 32.5
  7. 7 nllb-200-3.3B 31.8
  8. 8 Qwen3.5-35B-A3B 31.4
  9. 9 gpt-oss-120b 31.3
  10. 10 nllb-200-1.3B 31.2
  11. 11 Qwen3.5-27B 31.2
  12. 12 Llama 3.3 70B Instruct 30.6
  13. 13 nllb-200-distilled-600M 30.1
  14. 14 Gemma 3 27B 29.5
  15. 15 gemma-3-12b-it 29.2
  16. 16 OMT-LLama-1B 29.0
  17. 17 Gemma 4 E4B 28.6
  18. 18 gpt-oss-20b 28.5
  19. 19 Qwen3.5-9B 28.4
  20. 20 madlad400-10b-mt 28.1
  21. 21 Tower-Plus-72B 26.4
  22. 22 Qwen3-32B 26.2
  23. 23 Aya Expanse 32B 26.1
  24. 24 Gemma 4 E2B 26.0
  25. 25 Qwen3.5-4B 24.7
  26. 26 gemma-3-4b-it 24.2
  27. 27 Ministral-3-14B-Instruct-2512 23.4
  28. 28 Tower-Plus-9B 23.4
  29. 29 Llama-3.1-8B-Instruct 23.3
  30. 30 Qwen3-8B 22.3
  31. 31 EuroLLM-22B-Instruct-2512 22.3
  32. 32 Aya Expanse 8B 21.7
  33. 33 Hunyuan-MT-7B 21.5
  34. 34 Tiny Aya Global 21.5
  35. 35 Qwen2.5-7B-Instruct 21.1
  36. 36 EuroLLM-9B-Instruct 20.1
  37. 37 Qwen3.5-2B 19.2
  38. 38 salamandraTA-7b-instruct 19.0
  39. 39 Olmo-3-7B-Instruct 18.8
  40. 40 Llama-3.2-3B-Instruct 18.6
  41. 41 TowerInstruct-7B-v0.2 17.3
  42. 42 gemma-3-1b-it 16.4
  43. 43 Marco-MT-Algharb 14.5
  44. 44 Llama-3.2-1B-Instruct 13.5
  45. 45 aya-101 10.4

Source: Official BOUQuET leaderboard

19 models. Scroll sideways for the rest.

  1. 36.5 Muse Glimmer
  2. 35.0 Gemma 4 31B IT
  3. 31.4 Qwen3.5-35B-A3B
  4. 31.3 gpt-oss-120b
  5. 31.2 Qwen3.5-27B
  6. 30.6 Llama 3.3 70B Instruct
  7. 29.5 Gemma 3 27B
  8. 28.6 Gemma 4 E4B
  9. 28.5 gpt-oss-20b
  10. 28.4 Qwen3.5-9B
  11. 26.2 Qwen3-32B
  12. 26.1 Aya Expanse 32B
  13. 26.0 Gemma 4 E2B
  14. 24.7 Qwen3.5-4B
  15. 22.3 Qwen3-8B
  16. 21.7 Aya Expanse 8B
  17. 21.5 Hunyuan-MT-7B
  18. 21.5 Tiny Aya Global
  19. 19.2 Qwen3.5-2B
  1. 1 Muse Glimmer 36.5
  2. 2 Gemma 4 31B IT 35.0
  3. 3 Qwen3.5-35B-A3B 31.4
  4. 4 gpt-oss-120b 31.3
  5. 5 Qwen3.5-27B 31.2
  6. 6 Llama 3.3 70B Instruct 30.6
  7. 7 Gemma 3 27B 29.5
  8. 8 Gemma 4 E4B 28.6
  9. 9 gpt-oss-20b 28.5
  10. 10 Qwen3.5-9B 28.4
  11. 11 Qwen3-32B 26.2
  12. 12 Aya Expanse 32B 26.1
  13. 13 Gemma 4 E2B 26.0
  14. 14 Qwen3.5-4B 24.7
  15. 15 Qwen3-8B 22.3
  16. 16 Aya Expanse 8B 21.7
  17. 17 Hunyuan-MT-7B 21.5
  18. 18 Tiny Aya Global 21.5
  19. 19 Qwen3.5-2B 19.2

Sources: Labs’ model cards and official BOUQuET leaderboard

#EntryScoreSettingDateSource
1 OMT-LLama-8B 35.2 Official leaderboard
2 Gemma 4 31B IT 35.0 Apr 2026 Official leaderboard
3 OMT-NLLB 34.2 Official leaderboard
4 Muse-Glimmer-30B 34.1 reasoning high Official leaderboard
5 gemma-4-26B-A4B-it 33.1 Official leaderboard
6 OMT-LLama-3B 32.5 Official leaderboard
7 nllb-200-3.3B 31.8 Official leaderboard
8 Qwen3.5-35B-A3B 31.4 Feb 2026 Official leaderboard
9 gpt-oss-120b 31.3 Aug 2025 Official leaderboard
10 nllb-200-1.3B 31.2 Official leaderboard
11 Qwen3.5-27B 31.2 Feb 2026 Official leaderboard
12 Llama 3.3 70B Instruct 30.6 Dec 2024 Official leaderboard
13 nllb-200-distilled-600M 30.1 Official leaderboard
14 Gemma 3 27B 29.5 Mar 2025 Official leaderboard
15 gemma-3-12b-it 29.2 Official leaderboard
16 OMT-LLama-1B 29.0 Official leaderboard
17 Gemma 4 E4B 28.6 Mar 2026 Official leaderboard
18 gpt-oss-20b 28.5 Aug 2025 Official leaderboard
19 Qwen3.5-9B 28.4 Feb 2026 Official leaderboard
20 madlad400-10b-mt 28.1 Official leaderboard
21 Tower-Plus-72B 26.4 Official leaderboard
22 Qwen3-32B 26.2 Apr 2025 Official leaderboard
23 Aya Expanse 32B 26.1 Oct 2024 Official leaderboard
24 Gemma 4 E2B 26.0 Mar 2026 Official leaderboard
25 Qwen3.5-4B 24.7 Feb 2026 Official leaderboard
26 gemma-3-4b-it 24.2 Official leaderboard
27 Ministral-3-14B-Instruct-2512 23.4 Official leaderboard
28 Tower-Plus-9B 23.4 Official leaderboard
29 Llama-3.1-8B-Instruct 23.3 Official leaderboard
30 Qwen3-8B 22.3 Apr 2025 Official leaderboard
31 EuroLLM-22B-Instruct-2512 22.3 Official leaderboard
32 Aya Expanse 8B 21.7 Oct 2024 Official leaderboard
33 Hunyuan-MT-7B 21.5 Sep 2025 Official leaderboard
34 Tiny Aya Global 21.5 Feb 2026 Official leaderboard
35 Qwen2.5-7B-Instruct 21.1 Official leaderboard
36 EuroLLM-9B-Instruct 20.1 Official leaderboard
37 Qwen3.5-2B 19.2 Feb 2026 Official leaderboard
38 salamandraTA-7b-instruct 19.0 Official leaderboard
39 Olmo-3-7B-Instruct 18.8 Official leaderboard
40 Llama-3.2-3B-Instruct 18.6 Official leaderboard
41 TowerInstruct-7B-v0.2 17.3 Official leaderboard
42 gemma-3-1b-it 16.4 Official leaderboard
43 Marco-MT-Algharb 14.5 Official leaderboard
44 Llama-3.2-1B-Instruct 13.5 Official leaderboard
45 aya-101 10.4 Official leaderboard
#ModelScoreSettingDateSource
1 Muse Glimmer 36.5 Oct 2026 Lab model card
2 Gemma 4 31B IT 35.0 Apr 2026 Official leaderboard
3 Qwen3.5-35B-A3B 31.4 Feb 2026 Official leaderboard
4 gpt-oss-120b 31.3 Aug 2025 Official leaderboard
5 Qwen3.5-27B 31.2 Feb 2026 Official leaderboard
6 Llama 3.3 70B Instruct 30.6 Dec 2024 Official leaderboard
7 Gemma 3 27B 29.5 Mar 2025 Official leaderboard
8 Gemma 4 E4B 28.6 Mar 2026 Official leaderboard
9 gpt-oss-20b 28.5 Aug 2025 Official leaderboard
10 Qwen3.5-9B 28.4 Feb 2026 Official leaderboard
11 Qwen3-32B 26.2 Apr 2025 Official leaderboard
12 Aya Expanse 32B 26.1 Oct 2024 Official leaderboard
13 Gemma 4 E2B 26.0 Mar 2026 Official leaderboard
14 Qwen3.5-4B 24.7 Feb 2026 Official leaderboard
15 Qwen3-8B 22.3 Apr 2025 Official leaderboard
16 Aya Expanse 8B 21.7 Oct 2024 Official leaderboard
17 Hunyuan-MT-7B 21.5 Sep 2025 Official leaderboard
18 Tiny Aya Global 21.5 Feb 2026 Official leaderboard
19 Qwen3.5-2B 19.2 Feb 2026 Official leaderboard

Labs report scores for 1 model in their cards. Switch the scores above to rank them together.

4 models have held number one on the official leaderboard since its release in February 2025 5 models have held number one with claims included since its release in February 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Meta OpenAI Qwen Google DeepMind
Own leaderboard Not tested since Apr 2026
Gemma 4 31B IT 35.0

Source: Official BOUQuET leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card
15.0 20.0 25.0 30.0 35.0 40.0 2025 2026 15.0 20.0 25.0 30.0 35.0 40.0 2025 2026

Sources: Official BOUQuET leaderboard and labs’ model cards

Labs report it only in system cards

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark. Some system cards do, below.

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 19, 2026 Muse Glimmer technical report 36.5 BOUQuET, chrF++, high. X to eng 41.4, eng to Y 31.7 (Tables 5 and 9) Capabilities

More in Multilingual

All 120 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Nemotron 3 Ultra 550B-A55B Base 90.1%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%