Benchmark Almanac

Multilingual human evaluation

Claude 3 Opus 4.15 leads since June 2024
Dropped

Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages.

This is an in-house evaluation, and Qwen never released the prompts.

Released
June 2024
Kind
Chat and instructions in many languages
Level
Everyday
Scored
Graded by people
Answers
Open-ended
Tasks
Kept private
Strongest model tested
GPT-4o May 2024 #161 on the capability index
Last in a launch post
June 2024, dropped before it saturated
Last model added
July 2024
Scores refreshed
November 12th, 2026
6
Entries on its own leaderboard, 5 models we track
1
Launch post lists it, from 1 lab
1
System card reports it, from 1 lab

Claude 3 Opus leads with 4.15 among models released by June 2024

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in June 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 4.15 Claude 3 Opus
  2. 4.09 GPT-4o
  3. 3.98 GPT-4 Turbo
  4. 3.93 Qwen2-72B
  5. 3.71 GPT-4
  6. 3.16 GPT-3.5-Turbo-1106
  1. 1 Claude 3 Opus 4.15
  2. 2 GPT-4o May 2024 4.09
  3. 3 GPT-4 Turbo Apr 2024 3.98
  4. 4 Qwen2-72B 3.93
  5. 5 GPT-4 Jun 2023 3.71
  6. 6 GPT-3.5-Turbo-1106 3.16

Source: Official Multilingual human evaluation leaderboard

#EntryScoreSettingDateSource
1 Claude 3 Opus 4.15 Feb 2024 Official leaderboard
2 GPT-4o May 2024 4.09 May 2024 Official leaderboard
3 GPT-4 Turbo Apr 2024 3.98 Apr 2024 Official leaderboard
4 Qwen2-72B 3.93 Jun 2024 Official leaderboard
5 GPT-4 Jun 2023 3.71 Jun 2023 Official leaderboard
6 GPT-3.5-Turbo-1106 3.16 Official leaderboard

Labs report scores for 1 model in their cards. We plot every source in the charts below.

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard Not tested since Jun 2024
Claude 3 Opus 4.15

Source: Official Multilingual human evaluation leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card
3.70 3.80 3.90 4.00 4.10 4.20 2023 2024 3.70 3.80 3.90 4.00 4.10 4.20 2023 2024

Sources: Official Multilingual human evaluation leaderboard and labs’ model cards

No launch post has listed it since June 2024

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
DateLaunch postScoreAs the post writes it
Jun 7, 2024 Qwen2-72B 3.93 Human multilingual evaluation (Average, 1-5)

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Jul 15, 2024 Qwen2-72B technical report 3.93 Multilingual human evaluation, average over 10 languages, scale 1 to 5 (Table 13) Capabilities

One lab claim next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. Every benchmark.

Model Lab says Board says Gap
Qwen2-72B 3.93 Human multilingual evaluation (Average, 1-5) 3.93 Leaderboard Same

More in Multilingual

All 120 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Nemotron 3 Ultra 550B-A55B Base 90.1%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%