Benchmark Almanac

MMMLU

Claude Mythos Preview 92.7% leads since April 2026
In use

MMLU professionally translated into 14 languages. Multiple-choice accuracy.

Made by
OpenAI
Released
September 2024
Kind
Knowledge in many languages
Level
University
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Strongest model tested
Claude Opus 4.7 #30 on the capability index
Last in a launch post
July 2026
Last model added
August 2026
71
Models scored in lab cards
57
Launch posts list it, from 13 labs
42
System cards report it, from 9 labs

Claude Mythos Preview leads with 92.7% among models released by July 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in July 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

71 models. Scroll sideways for the rest.

  1. 92.7 Claude Mythos Preview
  2. 92.6 Gemini 3.1 Pro
  3. 91.8 Gemini 3 Pro
  4. 91.8 Gemini 3 Flash
  5. 91.5 Claude Opus 4.7
  6. 91.1 Claude Opus 4.6
  7. 90.8 Claude Opus 4.5
  8. 90.3 DeepSeek-V4-Pro
  9. 90.3 Qwen3.7-Max
  10. 89.5 Claude Opus 4.1
  11. 89.5 Qwen 3.6 Plus
  12. 89.3 Claude Sonnet 4.6
  13. 89.1 Claude Sonnet 4.5
  14. 89.0 Qwen3.7-Plus
  15. 88.9 Gemini 3.1 Flash-Lite
  16. 88.8 o3
  17. 88.8 Claude Opus 4
  18. 88.5 Qwen3.5 397B-A17B
  19. 88.4 Gemma 4 31B IT
  20. 88.1 Seed2.0 Pro
  21. 86.7 Qwen3.5-122B-A10B
  22. 86.6 K-EXAONE 2.0
  23. 86.5 Claude Sonnet 4
  24. 86.3 Gemma 4 26B A4B
  25. 86.2 Cogito v2.1
  26. 86.1 Claude 3.7 Sonnet
  27. 85.9 Qwen3.5-27B
  28. 85.8 Llama 4 Behemoth
  29. 85.7 K-EXAONE
  30. 85.6 EXAONE 4.0 32B
  31. 85.5 Mistral Large 3
  32. 85.4 EXAONE 4.5 33B
  33. 85.2 o4-mini
  34. 85.2 Qwen3.5-35B-A3B
  35. 85.1 GPT-4.5
  36. 84.3 Qwen3-235B-A22B
  37. 83.4 Gemma 4 12B
  38. 83.0 Claude Haiku 4.5
  39. 82.4 Cogito v2 Preview 671B MoE
  40. 81.5 DiffusionGemma 26B A4B
  41. 81.3 gpt-oss-120b
  42. 81.2 Qwen3.5-9B
  43. 80.6 Qwen3-32B
  44. 80.5 Cogito v1 Preview 70B
  45. 79.4 DeepSeek-V3
  46. 78.5 GPT-4.1 mini
  47. 78.4 Qwen3-30B-A3B
  48. 78.4 seed-oss-36b-instruct
  49. 78.0 Qwen2-72B
  50. 77.9 Qwen3-14B
  51. 76.6 Gemma 4 E4B
  52. 76.1 Qwen3.5-4B
  53. 75.7 gpt-oss-20b
  54. 74.6 Apple Foundation Model Server (2025)
  55. 74.4 Qwen3-8B
  56. 73.7 Granite 4.1 30B
  57. 69.9 Phi-3.5-MoE
  58. 69.8 Qwen3-4B
  59. 69.7 Granite 4.0 H Small
  60. 67.4 Gemma 4 E2B
  61. 66.9 GPT-4.1 nano
  62. 63.1 Qwen3.5-2B
  63. 59.1 Qwen3-1.7B
  64. 55.4 Phi-3.5-mini
  65. 55.3 LFM2-8B-A1B
  66. 49.3 Phi-4 Mini
  67. 47.3 phi-3-mini 3.8B
  68. 46.7 LFM2-1.2B
  69. 45.0 granite-4.0-1b
  70. 44.3 Qwen3.5-0.8B
  71. 35.6 Gemma 3n E4B
  1. 1 Claude Mythos Preview 92.7%
  2. 2 Gemini 3.1 Pro 92.6%
  3. 3 Gemini 3 Pro 91.8%
  4. 4 Gemini 3 Flash 91.8%
  5. 5 Claude Opus 4.7 91.5%
  6. 6 Claude Opus 4.6 91.1%
  7. 7 Claude Opus 4.5 90.8%
  8. 8 DeepSeek-V4-Pro 90.3%
  9. 9 Qwen3.7-Max 90.3%
  10. 10 Claude Opus 4.1 89.5%
  11. 11 Qwen 3.6 Plus 89.5%
  12. 12 Claude Sonnet 4.6 89.3%
  13. 13 Claude Sonnet 4.5 89.1%
  14. 14 Qwen3.7-Plus 89%
  15. 15 Gemini 3.1 Flash-Lite 88.9%
  16. 16 o3 88.8%
  17. 17 Claude Opus 4 88.8%
  18. 18 Qwen3.5 397B-A17B 88.5%
  19. 19 Gemma 4 31B IT 88.4%
  20. 20 Seed2.0 Pro 88.1%
  21. 21 Qwen3.5-122B-A10B 86.7%
  22. 22 K-EXAONE 2.0 86.6%
  23. 23 Claude Sonnet 4 86.5%
  24. 24 Gemma 4 26B A4B 86.3%
  25. 25 Cogito v2.1 86.2%
  26. 26 Claude 3.7 Sonnet 86.1%
  27. 27 Qwen3.5-27B 85.9%
  28. 28 Llama 4 Behemoth 85.8%
  29. 29 K-EXAONE 85.7%
  30. 30 EXAONE 4.0 32B 85.6%
  31. 31 Mistral Large 3 85.5%
  32. 32 EXAONE 4.5 33B 85.4%
  33. 33 o4-mini 85.2%
  34. 34 Qwen3.5-35B-A3B 85.2%
  35. 35 GPT-4.5 85.1%
  36. 36 Qwen3-235B-A22B 84.3%
  37. 37 Gemma 4 12B 83.4%
  38. 38 Claude Haiku 4.5 83%
  39. 39 Cogito v2 Preview 671B MoE 82.4%
  40. 40 DiffusionGemma 26B A4B 81.5%
  41. 41 gpt-oss-120b 81.3%
  42. 42 Qwen3.5-9B 81.2%
  43. 43 Qwen3-32B 80.6%
  44. 44 Cogito v1 Preview 70B 80.5%
  45. 45 DeepSeek-V3 79.4%
  46. 46 GPT-4.1 mini 78.5%
  47. 47 Qwen3-30B-A3B 78.4%
  48. 48 seed-oss-36b-instruct 78.4%
  49. 49 Qwen2-72B 78%
  50. 50 Qwen3-14B 77.9%
  51. 51 Gemma 4 E4B 76.6%
  52. 52 Qwen3.5-4B 76.1%
  53. 53 gpt-oss-20b 75.7%
  54. 54 Apple Foundation Model Server (2025) 74.6%
  55. 55 Qwen3-8B 74.4%
  56. 56 Granite 4.1 30B 73.7%
  57. 57 Phi-3.5-MoE 69.9%
  58. 58 Qwen3-4B 69.8%
  59. 59 Granite 4.0 H Small 69.7%
  60. 60 Gemma 4 E2B 67.4%
  61. 61 GPT-4.1 nano 66.9%
  62. 62 Qwen3.5-2B 63.1%
  63. 63 Qwen3-1.7B 59.1%
  64. 64 Phi-3.5-mini 55.4%
  65. 65 LFM2-8B-A1B 55.3%
  66. 66 Phi-4 Mini 49.3%
  67. 67 phi-3-mini 3.8B 47.3%
  68. 68 LFM2-1.2B 46.7%
  69. 69 granite-4.0-1b 45%
  70. 70 Qwen3.5-0.8B 44.3%
  71. 71 Gemma 3n E4B 35.6%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Claude Mythos Preview 92.7% Apr 2026 Lab model card
2 Gemini 3.1 Pro 92.6% Feb 2026 Lab model card
3 Gemini 3 Pro 91.8% Nov 2025 Lab model card
4 Gemini 3 Flash 91.8% Dec 2025 Lab model card
5 Claude Opus 4.7 91.5% Apr 2026 Lab model card
6 Claude Opus 4.6 91.1% Feb 2026 Lab model card
7 Claude Opus 4.5 90.8% Nov 2025 Lab model card
8 DeepSeek-V4-Pro 90.3% Apr 2026 Lab model card
9 Qwen3.7-Max 90.3% May 2026 Lab model card
10 Claude Opus 4.1 89.5% Aug 2025 Lab model card
11 Qwen 3.6 Plus 89.5% Apr 2026 Lab model card
12 Claude Sonnet 4.6 89.3% Feb 2026 Lab model card
13 Claude Sonnet 4.5 89.1% Sep 2025 Lab model card
14 Qwen3.7-Plus 89% Jun 2026 Lab model card
15 Gemini 3.1 Flash-Lite 88.9% Mar 2026 Lab model card
16 o3 88.8% Apr 2025 Lab model card
17 Claude Opus 4 88.8% May 2025 Lab model card
18 Qwen3.5 397B-A17B 88.5% Feb 2026 Lab model card
19 Gemma 4 31B IT 88.4% Apr 2026 Lab model card
20 Seed2.0 Pro 88.1% Feb 2026 Lab model card
21 Qwen3.5-122B-A10B 86.7% Feb 2026 Lab model card
22 K-EXAONE 2.0 86.6% Jul 2026 Lab model card
23 Claude Sonnet 4 86.5% May 2025 Lab model card
24 Gemma 4 26B A4B 86.3% Apr 2026 Lab model card
25 Cogito v2.1 86.2% Nov 2025 Lab model card
26 Claude 3.7 Sonnet 86.1% Feb 2025 Lab model card
27 Qwen3.5-27B 85.9% Feb 2026 Lab model card
28 Llama 4 Behemoth 85.8% Apr 2025 Lab model card
29 K-EXAONE 85.7% Dec 2025 Lab model card
30 EXAONE 4.0 32B 85.6% Jul 2025 Lab model card
31 Mistral Large 3 85.5% Dec 2025 Lab model card
32 EXAONE 4.5 33B 85.4% Apr 2026 Lab model card
33 o4-mini 85.2% Apr 2025 Lab model card
34 Qwen3.5-35B-A3B 85.2% Feb 2026 Lab model card
35 GPT-4.5 85.1% Feb 2025 Lab model card
36 Qwen3-235B-A22B 84.3% Apr 2025 Lab model card
37 Gemma 4 12B 83.4% Jun 2026 Lab model card
38 Claude Haiku 4.5 83% Oct 2025 Lab model card
39 Cogito v2 Preview 671B MoE 82.4% Jul 2025 Lab model card
40 DiffusionGemma 26B A4B 81.5% Jun 2026 Lab model card
41 gpt-oss-120b 81.3% Aug 2025 Lab model card
42 Qwen3.5-9B 81.2% Mar 2026 Lab model card
43 Qwen3-32B 80.6% May 2025 Lab model card
44 Cogito v1 Preview 70B 80.5% Apr 2025 Lab model card
45 DeepSeek-V3 79.4% Dec 2024 Lab model card
46 GPT-4.1 mini 78.5% Apr 2025 Lab model card
47 Qwen3-30B-A3B 78.4% May 2025 Lab model card
48 seed-oss-36b-instruct 78.4% Aug 2025 Lab model card
49 Qwen2-72B 78% Jun 2024 Lab model card
50 Qwen3-14B 77.9% May 2025 Lab model card
51 Gemma 4 E4B 76.6% Apr 2026 Lab model card
52 Qwen3.5-4B 76.1% Mar 2026 Lab model card
53 gpt-oss-20b 75.7% Aug 2025 Lab model card
54 Apple Foundation Model Server (2025) 74.6% Jul 2025 Lab model card
55 Qwen3-8B 74.4% May 2025 Lab model card
56 Granite 4.1 30B 73.7% Apr 2026 Lab model card
57 Phi-3.5-MoE 69.9% Aug 2024 Lab model card
58 Qwen3-4B 69.8% May 2025 Lab model card
59 Granite 4.0 H Small 69.7% Oct 2025 Lab model card
60 Gemma 4 E2B 67.4% Apr 2026 Lab model card
61 GPT-4.1 nano 66.9% Apr 2025 Lab model card
62 Qwen3.5-2B 63.1% Mar 2026 Lab model card
63 Qwen3-1.7B 59.1% May 2025 Lab model card
64 Phi-3.5-mini 55.4% Aug 2024 Lab model card
65 LFM2-8B-A1B 55.3% Oct 2025 Lab model card
66 Phi-4 Mini 49.3% Feb 2025 Lab model card
67 phi-3-mini 3.8B 47.3% Aug 2024 Lab model card
68 LFM2-1.2B 46.7% Jul 2025 Lab model card
69 granite-4.0-1b 45% Oct 2025 Lab model card
70 Qwen3.5-0.8B 44.3% Mar 2026 Lab model card
71 Gemma 3n E4B 35.6% Jun 2025 Lab model card

8 models have held number one in labs’ own numbers since its release in September 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen DeepSeek Anthropic OpenAI Google DeepMind
Lab claims Passed 85% in Feb 2025, by Claude 3.7 Sonnet
Claude Mythos Preview 92.7%

Source: Labs’ model cards

Anthropic’s own number for Claude 3.7 Sonnet passed 85% in February 2025

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Claude Mythos Preview Claude Opus 4.1 Claude 3.7 Sonnet DeepSeek-V3 Qwen2-72B 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Claude Mythos Preview Claude 3.7 Sonnet

Source: Labs’ model cards

3 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
Microsoft
DeepSeek
Anthropic
OpenAI
Meta
Deep Cogito
Google DeepMind
Liquid AI
LG AI Research
ByteDance Seed
IBM
Mistral AI
DateLaunch postScoreAs the post writes it
Jul 31, 2026 K-EXAONE 2.0 86.6 MMMLU (nine non-English supported languages)
Jun 10, 2026 DiffusionGemma 26B A4B 81.5 MMMLU
Jun 3, 2026 Gemma 4 12B 83.4 MMMLU
Jun 1, 2026 Qwen3.7-Plus 89 MMMLU
May 20, 2026 Qwen3.7-Max 90.3 MMMLU
Apr 29, 2026 Granite 4.1 30B 73.7 MMMLU (5-shot, 11 languages)
Apr 24, 2026 DeepSeek-V4-Pro 90.3 MMMLU (EM)
Apr 16, 2026 Claude Opus 4.7 91.5 Multilingual Q&A MMMLU
Apr 7, 2026 Claude Mythos Preview 92.7 MMMLU
Apr 2, 2026 Gemma 4 E4B 76.6 MMMLU
Apr 2, 2026 Gemma 4 E2B 67.4 MMMLU
Apr 2, 2026 Gemma 4 26B A4B 86.3 MMMLU
Apr 2, 2026 Qwen 3.6 Plus 89.5 MMMLU
Apr 2, 2026 Gemma 4 31B IT 88.4 MMMLU
Mar 3, 2026 Gemini 3.1 Flash-Lite 88.9 MMMLU
Mar 2, 2026 Qwen3.5-0.8B 44.3 MMMLU (Thinking)
Mar 2, 2026 Qwen3.5-2B 63.1 MMMLU (Thinking)
Mar 2, 2026 Qwen3.5-4B 76.1 MMMLU
Mar 2, 2026 Qwen3.5-9B 81.2 MMMLU
Feb 24, 2026 Qwen3.5-35B-A3B 85.2 MMMLU
Feb 24, 2026 Qwen3.5-27B 85.9 MMMLU
Feb 24, 2026 Qwen3.5-122B-A10B 86.7 MMMLU
Feb 19, 2026 Gemini 3.1 Pro 92.6 MMMLU
Feb 17, 2026 Claude Sonnet 4.6 89.3 Multilingual Q&A MMMLU
Feb 16, 2026 Qwen3.5 397B-A17B 88.5 MMMLU
Feb 14, 2026 Seed2.0 Pro 88.1 MMMLU
Feb 5, 2026 Claude Opus 4.6 91.1 Multilingual Q&A MMMLU
Dec 31, 2025 K-EXAONE 85.7 MMMLU
Dec 17, 2025 Gemini 3 Flash 91.8 MMMLU
Dec 2, 2025 Mistral Large 3 85.5 MMMLU (8-lang average) [base]
Nov 24, 2025 Claude Opus 4.5 90.8 Multilingual Q&A MMMLU
Nov 19, 2025 Cogito v2.1 86.2 Multilingual MMLU
Nov 18, 2025 Gemini 3 Pro 91.8 MMMLU
Oct 28, 2025 granite-4.0-1b 45 MMMLU (5-shot, 11 languages)
Oct 15, 2025 Claude Haiku 4.5 83 Multilingual Q&A MMMLU
Oct 7, 2025 LFM2-8B-A1B 55.3 MMMLU
Oct 2, 2025 Granite 4.0 H Small 69.7 MMMLU (5-shot, 11 languages)
Sep 29, 2025 Claude Sonnet 4.5 89.1 Multilingual Q&A MMMLU
Aug 20, 2025 seed-oss-36b-instruct 78.4 MMMLU
Aug 5, 2025 Claude Opus 4.1 89.5 Multilingual Q&A MMMLU
Jul 31, 2025 Cogito v2 Preview 671B MoE 82.4 MMMLU (thinking)
Jul 15, 2025 EXAONE 4.0 32B 85.6 MMMLU (ES), Spanish only (reasoning)
Jul 10, 2025 LFM2-1.2B 46.7 MMMLU
Jun 26, 2025 Gemma 3n E4B 35.6 OpenAI MMLU 0-shot (IT)
May 22, 2025 Claude Sonnet 4 86.5 Multilingual Q&A MMMLU
May 22, 2025 Claude Opus 4 88.8 Multilingual Q&A MMMLU
Apr 29, 2025 Qwen3-235B-A22B 84.3 MMMLU 14 languages
Apr 14, 2025 GPT-4.1 nano 66.9 Multilingual MMLU
Apr 14, 2025 GPT-4.1 mini 78.5 Multilingual MMLU
Apr 8, 2025 Cogito v1 Preview 70B 80.5 MMMLU (thinking)
Apr 5, 2025 Llama 4 Behemoth 85.8 Multilingual MMLU
Feb 27, 2025 GPT-4.5 85.1 MMMLU (multilingual)
Feb 26, 2025 Phi-4 Mini 49.3 Multilingual MMLU (5-shot)
Feb 24, 2025 Claude 3.7 Sonnet 86.1 Multilingual Q&A MMMLU (64K extended thinking)
Dec 26, 2024 DeepSeek-V3 79.4 MMMLU-non-English (Acc.)
Aug 20, 2024 Phi-3.5-MoE 69.9 Multilingual MMLU (5-shot)
Jun 7, 2024 Qwen2-72B 78 M-MMLU (5-shot)

42 system cards report it, from 9 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Aug 5, 2026 K-EXAONE 2.0 technical report 86.6 MMMLU, nine non-English languages (Table 6, Reasoning mode) Capabilities
Jul 31, 2026 DiffusionGemma 26B A4B technical report 81.5 MMMLU, TD, 76.3 no-think (Table 3) Capabilities
Jul 24, 2026 Gemma 4 E4B technical report 76.6 MMMLU, thinking (Table 5) Capabilities
Jul 24, 2026 Gemma 4 E2B technical report 67.4 MMMLU, thinking (Table 5) Capabilities
Jul 24, 2026 Gemma 4 31B IT technical report 88.4 MMMLU, thinking (Table 5) Capabilities
Jul 24, 2026 Gemma 4 26B A4B technical report 86.3 MMMLU, thinking (Table 5) Capabilities
Apr 16, 2026 Claude Opus 4.7 system card 91.5 MMMLU, non-English language pairings, adaptive thinking at max effort, average of 3 trials (Table 8.1.A, 8.5) Capabilities
Apr 9, 2026 EXAONE 4.5 33B technical report 85.4 MMMLU (ko, de, es, ja) (Table 3, Reasoning mode) Capabilities
Apr 7, 2026 Claude Mythos Preview system card 92.7 MMMLU, all non-English language pairings, adaptive thinking, max effort, average of 5 trials (Table 6.3.A; 92.67 in 6.7) Capabilities
Mar 3, 2026 Gemini 3.1 Flash-Lite model card 88.9 MMMLU (results table, p.4) Capabilities
Feb 19, 2026 Gemini 3.1 Pro model card 92.6 MMMLU (results table, p.4) Capabilities
Feb 17, 2026 Claude Sonnet 4.6 system card 89.3 MMMLU, non-English language pairings, adaptive thinking, max effort (Table 2.1.A, 2.11) Capabilities
Feb 14, 2026 Seed2.0 Pro model card 88.1 MMMLU (Table 3) Capabilities
Feb 5, 2026 Claude Opus 4.6 system card 91.1 MMMLU, all non-English language pairings, adaptive thinking, max effort, average of 5 trials (Table 2.3.A; 91.05 in 2.13) Capabilities
Jan 5, 2026 K-EXAONE technical report 85.7 MMMLU (ko, de, es, ja) (Table 3, Reasoning mode). Per language ko 85.6, de 85.1, es 86.6, ja 85.5 (Table 5). Non-reasoning 83.8 Capabilities
Dec 11, 2025 GPT-5.2 system card addendum MMLU Language (0-shot), gpt-5.2-thinking, 14 languages, no overall score (Table 8) Capabilities
Nov 28, 2025 LFM2-8B-A1B technical report 55.3 MMMLU, LFM2-8B-A1B (Table 7) Capabilities
Nov 28, 2025 LFM2-1.2B technical report 46.7 MMMLU, LFM2-1.2B (Table 6) Capabilities
Nov 24, 2025 Claude Opus 4.5 system card 90.8 MMMLU, 64k thinking, avg of 10 trials over 14 languages (Table 2.3.A; 90.77% in 2.19) Capabilities
Aug 7, 2025 GPT-5 system card MMLU Language (0-shot), 14 languages, no average (Table 11) Capabilities
Aug 5, 2025 gpt-oss-20b model card 75.7 MMMLU (Average) over 14 languages, high reasoning (Table 2, Table 3) Capabilities
Aug 5, 2025 gpt-oss-120b model card 81.3 MMMLU (Average) over 14 languages, high reasoning (Table 2, Table 3) Capabilities
Jul 17, 2025 Apple Foundation Model Server (2025) technical report 74.6 MMMLU, AFM-supported languages only, Simple-Evals (Table 2) Capabilities
Jul 15, 2025 EXAONE 4.0 32B technical report 85.6 MMMLU (ES), Spanish only (Table 3, Reasoning). Non-reasoning 80.6 (Table 4) Capabilities
May 14, 2025 Qwen3-8B technical report 74.4 MMMLU 14 languages, thinking mode (Table 17). Non-thinking 66.9 (Table 18) Capabilities
May 14, 2025 Qwen3-4B technical report 69.8 MMMLU 14 languages, thinking mode (Table 17). Non-thinking 61.7 (Table 18) Capabilities
May 14, 2025 Qwen3-32B technical report 80.6 MMMLU 14 languages, thinking mode (Table 13). Non-thinking 76.5 (Table 14) Capabilities
May 14, 2025 Qwen3-30B-A3B technical report 78.4 MMMLU 14 languages, thinking mode (Table 15). Non-thinking 73.8 (Table 16) Capabilities
May 14, 2025 Qwen3-235B-A22B technical report 84.3 MMMLU 14 languages, thinking mode (Table 11). Non-thinking 79.8 (Table 12) Capabilities
May 14, 2025 Qwen3-14B technical report 77.9 MMMLU 14 languages, thinking mode (Table 15). Non-thinking 72.6 (Table 16) Capabilities
May 14, 2025 Qwen3-1.7B technical report 59.1 MMMLU 14 languages, thinking mode (Table 19). Non-thinking 48.3 (Table 20) Capabilities
Apr 16, 2025 o4-mini system card 85.2 MMLU Language (0-shot), average of 14 languages, o4-mini-high (Table 16) Capabilities
Apr 16, 2025 o3 system card 88.8 MMLU Language (0-shot), average of 14 languages, o3-high (Table 16) Capabilities
Mar 3, 2025 Phi-4 Mini technical report 49.3 Multilingual-MMLU (5-Shot) (Table 7) Capabilities
Feb 27, 2025 GPT-4.5 system card MMLU Language (0-shot), 14 human-translated languages plus English, per-language scores with no average (Table 16) Capabilities
Jan 31, 2025 o3-mini system card MMLU Language (0-shot), 14 human-translated languages, per-language scores with no average (Table 14) Capabilities
Dec 5, 2024 o1 system card MMLU Language (0-shot), human translated MMLU in 14 languages plus English, no overall score (Table 17) Capabilities
Sep 12, 2024 o1-preview system card MMLU Language (0-shot), human translated MMLU in 14 languages plus English, no overall score (Table 12) Capabilities
Sep 12, 2024 o1-mini system card MMLU Language (0-shot), human translated MMLU in 14 languages plus English, no overall score (Table 12) Capabilities
Aug 30, 2024 phi-3-mini 3.8B technical report 47.3 MMLU-multilingual average (Section 4 text, Figure 4) Capabilities
Aug 30, 2024 Phi-3.5-MoE technical report 69.9 Ml MMLU (5-shot), multilingual MMLU (Table 3) Capabilities
Aug 30, 2024 Phi-3.5-mini technical report 55.4 Ml MMLU (5-shot), multilingual MMLU (Table 3) Capabilities

One logged change that moves scores

Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.

  1. Grader changed January 31st, 2025

    The maintainers merged improved multilingual answer parsing into the official evaluator. Evaluations using the earlier parser can classify the same responses differently. Source

More in Multilingual

All 120 in Multilingual
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Nemotron 3 Ultra 550B-A55B Base 90.1%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%
  • Alyah In use Models answer 1,173 four-option questions written by native Emirati speakers on greetings, idioms, figurative language, heritage and poetry in Emirati Arabic dialect. Gemma 3 27B 74.7%