MGSM
Cogito v2 Preview 671B MoE 93% leads since July 2025 250 GSM8K problems translated into 10 languages. Final-answer accuracy.
Saturated.
- Made by
- Google Research
- Released
- October 2022
- Kind
- Reasoning in many languages
- Level
- School
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- All public
- Strongest model tested
-
DeepSeek V4.1 Flash #34 on the capability index - Last in a launch post
- September 2026, dropped after it saturated
- Last model added
- October 2026
In the authors’ words
We evaluate the reasoning abilities of large language models in multilingual settings. We introduce the Multilingual Grade School Math (MGSM) benchmark, by manually translating 250 grade-school math problems from the GSM8K dataset (Cobbe et al., 2021) into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale, and that models have strikingly strong multilingual reasoning abilities, even in underrepresented languages such as Bengali and Swahili. Finally, we show that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment. The MGSM benchmark is publicly available at https://github.com/google-research/url-nlp.
57
Models scored in lab cards
33
Launch posts list it, from 10 labs
41
System cards report it, from 12 labs
Cogito v2 Preview 671B MoE leads with 93%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 93.0 Cogito v2 Preview 671B MoE
- 92.7 Cogito v1 Preview 70B
- 92.5 Claude 3.5 Sonnet (October 2024)
- 92.3
Llama 4 Maverick - 92.0 o3-mini
- 91.6 Claude 3.5 Sonnet
- 91.6 Llama 3.1-405B
- 91.1 Llama 3.3 70B Instruct
- 90.7 Claude 3 Opus
- 90.6 Llama 4 Scout
- 90.5 GPT-4o
- 90.1
Cohere Command A - 89.6 GPT-4 Turbo
- 89.3 o1
- 88.7
Gemini 1.5 Pro - 88.2
Qwen2.5-72B - 87.7
Nemotron 3 Ultra 550B-A55B Base - 87.5 Nemotron 3.5 Super 120B-A12B Base
- 87.5 Gemini 1.5 Pro
- 87.2 Qwen2.5-32B
- 87.1 Apple Foundation Model Server (2025)
- 87.0 PaLM 2-L
- 87.0 GPT-4o mini
- 86.9 Llama 3.1-70B
- 86.6 Qwen2-72B
- 85.6 Claude 3.5 Haiku
- 84.8 DiffusionGemma 26B A4B
- 84.4 DeepSeek-V4-Pro
- 83.5 Claude 3 Sonnet
- 83.0 Nemotron 3 Nano 30B-A3B Base
- 82.3 Qwen2.5-14B
- 82.3 Nemotron 3.5 Lightning 30B-A3B Base
- 80.6
Phi-4 - 80.2 DeepSeek V4.1 Flash
- 80.2 DeepSeek-V4.1-Flash Base
- 79.8 DeepSeek-V3
- 79.5 Aya Vision 32B
- 76.3
Falcon-H1-34B - 75.1 Claude 3 Haiku
- 74.3 Gemma 3 27B
- 74.0 DeepSeek LLM 67B
- 72.4 LFM2-8B-A1B
- 71.1 Granite 4.1 30B
- 68.9 Llama 3.1-8B
- 66.1 Qwen2.5-7B
- 64.3 Gemma 3 12B
- 64.3
Apertus 70B - 63.9 Phi-4 Mini
- 61.7 Qwen1.5-72B
- 60.7 Gemma 3n E4B
- 58.7 Phi-3.5-MoE
- 55.0 LFM2-1.2B
- 47.9 Phi-3.5-mini
- 38.7 Granite 4.0 H Small
- 37.8 granite-4.0-1b
- 34.7 Gemma 3 4B
- 2.04 Gemma 3 1B
- 1 Cogito v2 Preview 671B MoE 93%
- 2 Cogito v1 Preview 70B 92.7%
- 3 Claude 3.5 Sonnet (October 2024) 92.5%
- 4 Llama 4 Maverick 92.3%
- 5 o3-mini 92%
- 6 Claude 3.5 Sonnet 91.6%
- 7 Llama 3.1-405B 91.6%
- 8 Llama 3.3 70B Instruct 91.1%
- 9 Claude 3 Opus 90.7%
- 10 Llama 4 Scout 90.6%
- 11 GPT-4o May 2024 90.5%
- 12 Cohere Command A 90.1%
- 13 GPT-4 Turbo Apr 2024 89.6%
- 14 o1 89.3%
- 15 Gemini 1.5 Pro Feb 2024 88.7%
- 16 Qwen2.5-72B 88.2%
- 17 Nemotron 3 Ultra 550B-A55B Base 87.7%
- 18 Nemotron 3.5 Super 120B-A12B Base 87.5%
- 19 Gemini 1.5 Pro May 2024 87.5%
- 20 Qwen2.5-32B 87.2%
- 21 Apple Foundation Model Server (2025) 87.1%
- 22 PaLM 2-L 87%
- 23 GPT-4o mini 87%
- 24 Llama 3.1-70B 86.9%
- 25 Qwen2-72B 86.6%
- 26 Claude 3.5 Haiku 85.6%
- 27 DiffusionGemma 26B A4B 84.8%
- 28 DeepSeek-V4-Pro 84.4%
- 29 Claude 3 Sonnet 83.5%
- 30 Nemotron 3 Nano 30B-A3B Base 83%
- 31 Qwen2.5-14B 82.3%
- 32 Nemotron 3.5 Lightning 30B-A3B Base 82.3%
- 33 Phi-4 80.6%
- 34 DeepSeek V4.1 Flash 80.2%
- 35 DeepSeek-V4.1-Flash Base 80.2%
- 36 DeepSeek-V3 79.8%
- 37 Aya Vision 32B 79.5%
- 38 Falcon-H1-34B 76.3%
- 39 Claude 3 Haiku 75.1%
- 40 Gemma 3 27B 74.3%
- 41 DeepSeek LLM 67B 74%
- 42 LFM2-8B-A1B 72.4%
- 43 Granite 4.1 30B 71.1%
- 44 Llama 3.1-8B 68.9%
- 45 Qwen2.5-7B 66.1%
- 46 Gemma 3 12B 64.3%
- 47 Apertus 70B 64.3%
- 48 Phi-4 Mini 63.9%
- 49 Qwen1.5-72B 61.7%
- 50 Gemma 3n E4B 60.7%
- 51 Phi-3.5-MoE 58.7%
- 52 LFM2-1.2B 55%
- 53 Phi-3.5-mini 47.9%
- 54 Granite 4.0 H Small 38.7%
- 55 granite-4.0-1b 37.8%
- 56 Gemma 3 4B 34.7%
- 57 Gemma 3 1B 2.04%
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Cogito v2 Preview 671B MoE | 93% | Jul 2025 | Lab model card | |
| 2 | Cogito v1 Preview 70B | 92.7% | Apr 2025 | Lab model card | |
| 3 | Claude 3.5 Sonnet (October 2024) | 92.5% | Oct 2024 | Lab model card | |
| 4 | Llama 4 Maverick | 92.3% | Apr 2025 | Lab model card | |
| 5 | o3-mini | 92% | Jan 2025 | Lab model card | |
| 6 | Claude 3.5 Sonnet | 91.6% | Jun 2024 | Lab model card | |
| 7 | Llama 3.1-405B | 91.6% | Jul 2024 | Lab model card | |
| 8 | Llama 3.3 70B Instruct | 91.1% | Dec 2024 | Lab model card | |
| 9 | Claude 3 Opus | 90.7% | Mar 2024 | Lab model card | |
| 10 | Llama 4 Scout | 90.6% | Apr 2025 | Lab model card | |
| 11 | GPT-4o May 2024 | 90.5% | May 2024 | Lab model card | |
| 12 | Cohere Command A | 90.1% | Apr 2025 | Lab model card | |
| 13 | GPT-4 Turbo Apr 2024 | 89.6% | Apr 2024 | Lab model card | |
| 14 | o1 | 89.3% | Dec 2024 | Lab model card | |
| 15 | Gemini 1.5 Pro Feb 2024 | 88.7% | Feb 2024 | Lab model card | |
| 16 | Qwen2.5-72B | 88.2% | Sep 2024 | Lab model card | |
| 17 | Nemotron 3 Ultra 550B-A55B Base | 87.7% | Jun 2026 | Lab model card | |
| 18 | Nemotron 3.5 Super 120B-A12B Base | 87.5% | Oct 2026 | Lab model card | |
| 19 | Gemini 1.5 Pro May 2024 | 87.5% | May 2024 | Lab model card | |
| 20 | Qwen2.5-32B | 87.2% | Dec 2024 | Lab model card | |
| 21 | Apple Foundation Model Server (2025) | 87.1% | Jul 2025 | Lab model card | |
| 22 | PaLM 2-L | 87% | May 2023 | Lab model card | |
| 23 | GPT-4o mini | 87% | Jul 2024 | Lab model card | |
| 24 | Llama 3.1-70B | 86.9% | Jul 2024 | Lab model card | |
| 25 | Qwen2-72B | 86.6% | Jun 2024 | Lab model card | |
| 26 | Claude 3.5 Haiku | 85.6% | Oct 2024 | Lab model card | |
| 27 | DiffusionGemma 26B A4B | 84.8% | Jul 2026 | Lab model card | |
| 28 | DeepSeek-V4-Pro | 84.4% | Apr 2026 | Lab model card | |
| 29 | Claude 3 Sonnet | 83.5% | Mar 2024 | Lab model card | |
| 30 | Nemotron 3 Nano 30B-A3B Base | 83% | Dec 2025 | Lab model card | |
| 31 | Qwen2.5-14B | 82.3% | Dec 2024 | Lab model card | |
| 32 | Nemotron 3.5 Lightning 30B-A3B Base | 82.3% | Oct 2026 | Lab model card | |
| 33 | Phi-4 | 80.6% | Dec 2024 | Lab model card | |
| 34 | DeepSeek V4.1 Flash | 80.2% | Sep 2026 | Lab model card | |
| 35 | DeepSeek-V4.1-Flash Base | 80.2% | Sep 2026 | Lab model card | |
| 36 | DeepSeek-V3 | 79.8% | Dec 2024 | Lab model card | |
| 37 | Aya Vision 32B | 79.5% | May 2025 | Lab model card | |
| 38 | Falcon-H1-34B | 76.3% | Jul 2025 | Lab model card | |
| 39 | Claude 3 Haiku | 75.1% | Mar 2024 | Lab model card | |
| 40 | Gemma 3 27B | 74.3% | Mar 2025 | Lab model card | |
| 41 | DeepSeek LLM 67B | 74% | Jan 2024 | Lab model card | |
| 42 | LFM2-8B-A1B | 72.4% | Oct 2025 | Lab model card | |
| 43 | Granite 4.1 30B | 71.1% | Apr 2026 | Lab model card | |
| 44 | Llama 3.1-8B | 68.9% | Jul 2024 | Lab model card | |
| 45 | Qwen2.5-7B | 66.1% | Dec 2024 | Lab model card | |
| 46 | Gemma 3 12B | 64.3% | Mar 2025 | Lab model card | |
| 47 | Apertus 70B | 64.3% | Sep 2025 | Lab model card | |
| 48 | Phi-4 Mini | 63.9% | Feb 2025 | Lab model card | |
| 49 | Qwen1.5-72B | 61.7% | Feb 2024 | Lab model card | |
| 50 | Gemma 3n E4B | 60.7% | Jun 2025 | Lab model card | |
| 51 | Phi-3.5-MoE | 58.7% | Aug 2024 | Lab model card | |
| 52 | LFM2-1.2B | 55% | Jul 2025 | Lab model card | |
| 53 | Phi-3.5-mini | 47.9% | Aug 2024 | Lab model card | |
| 54 | Granite 4.0 H Small | 38.7% | Oct 2025 | Lab model card | |
| 55 | granite-4.0-1b | 37.8% | Oct 2025 | Lab model card | |
| 56 | Gemma 3 4B | 34.7% | Mar 2025 | Lab model card | |
| 57 | Gemma 3 1B | 2.04% | Mar 2025 | Lab model card |
7 models have held number one in labs’ own numbers since 2023
Lab claims count from the day the card came out, independent runs from the model’s release.
Other labs Google DeepMind Anthropic Deep Cogito
Source: Labs’ model cards
Its lab’s own number for PaLM 2-L passed 85% in May 2023
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Reported by the lab in its model card
Best so far
Source: Labs’ model cards
Labs headline it less often now, last in September 2026
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Sep 10, 2026 | DeepSeek V4.1 Flash | 80.2 | MGSM (EM) |
| Apr 29, 2026 | Granite 4.1 30B | 71.1 | MGSM (8-shot, 5 languages) |
| Apr 24, 2026 | DeepSeek-V4-Pro | 84.4 | MGSM (EM) |
| Oct 28, 2025 | granite-4.0-1b | 37.8 | MGSM (8-shot, 5 languages) |
| Oct 7, 2025 | LFM2-8B-A1B | 72.4 | MGSM |
| Oct 2, 2025 | Granite 4.0 H Small | 38.7 | MGSM (8-shot, 5 languages) |
| Jul 31, 2025 | Cogito v2 Preview 671B MoE | 93 | MGSM (thinking) |
| Jul 10, 2025 | LFM2-1.2B | 55 | MGSM |
| Jun 26, 2025 | Gemma 3n E4B | 60.7 | MGSM 0-shot (IT) |
| Apr 8, 2025 | Cogito v1 Preview 70B | 92.7 | MGSM (thinking) |
| Apr 5, 2025 | Llama 4 Scout | 90.6 | MGSM, average/em |
| Apr 5, 2025 | Llama 4 Maverick | 92.3 | MGSM, average/em |
| Mar 12, 2025 | Gemma 3 27B | 74.3 | MGSM (PT) |
| Feb 26, 2025 | Phi-4 Mini | 63.9 | MGSM (0-shot, CoT) |
| Jan 31, 2025 | o3-mini | 92 | MGSM (pass@1) |
| Dec 26, 2024 | DeepSeek-V3 | 79.8 | MGSM (EM) |
| Dec 17, 2024 | o1 | 89.3 | MGSM (pass @1) |
| Dec 12, 2024 | Phi-4 | 80.6 | MGSM (simple-evals) |
| Dec 6, 2024 | Llama 3.3 70B Instruct | 91.1 | MGSM, 0-shot, em |
| Oct 22, 2024 | Claude 3.5 Sonnet (October 2024) | 92.5 | MGSM (0-shot CoT) |
| Sep 19, 2024 | Qwen2.5-72B | 88.2 | MGSM8K (extended) |
| Aug 20, 2024 | Phi-3.5-MoE | 58.7 | MGSM (0-shot CoT) |
| Jul 23, 2024 | Llama 3.1-405B | 91.6 | MGSM (0-shot, CoT) |
| Jul 18, 2024 | GPT-4o mini | 87 | MGSM |
| Jun 20, 2024 | Claude 3.5 Sonnet | 91.6 | MGSM (0-shot CoT) |
| Jun 7, 2024 | Qwen2-72B | 86.6 | MGSM (0-shot, CoT) |
| May 14, 2024 | Gemini 1.5 Pro | 87.5 | MGSM, 8-shot |
| May 13, 2024 | GPT-4o | 90.5 | MGSM (0-shot CoT) |
| Apr 11, 2024 | GPT-4 Turbo | 89.6 | MGSM% |
| Mar 4, 2024 | Claude 3 Opus | 90.7 | MGSM (0-shot) |
| Feb 15, 2024 | Gemini 1.5 Pro | 88.7 | MGSM, 8-shot |
| Feb 5, 2024 | Qwen1.5-72B | 61.7 | Multilingual Math (MGSM, 8-shot) |
| May 10, 2023 | PaLM 2-L | 87 | MGSM, CoT + self-consistency (greedy CoT = 72.2) |
41 system cards report it, from 12 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Oct 15, 2026 | Nemotron 3.5 Super 120B-A12B Base technical report | 87.5 | MGSM, 8-shot native CoT, average across 6 languages, Table 6 | Capabilities |
| Oct 15, 2026 | Nemotron 3.5 Lightning 30B-A3B Base technical report | 82.3 | MGSM, 8-shot native CoT, average across 6 languages, Table 6 | Capabilities |
| Sep 10, 2026 | DeepSeek-V4.1-Flash Base model card | 80.2 | MGSM, EM, 8-shot | Capabilities |
| Jul 31, 2026 | DiffusionGemma 26B A4B technical report | 84.8 | MGSM, TD, 80.7 no-think (Table 3) | Capabilities |
| Jun 4, 2026 | Nemotron 3 Ultra 550B-A55B Base model card | 87.7 | MGSM, 8-shot native CoT avg | Capabilities |
| Dec 23, 2025 | Nemotron 3 Nano 30B-A3B Base technical report | 83 | MGSM, 8-shot average, base model Table 2 | Capabilities |
| Nov 28, 2025 | LFM2-8B-A1B technical report | 72.4 | MGSM, LFM2-8B-A1B (Table 7) | Capabilities |
| Nov 28, 2025 | LFM2-1.2B technical report | 55 | MGSM, LFM2-1.2B (Table 6) | Capabilities |
| Sep 17, 2025 | Apertus 70B technical report | 64.3 | MGSM, 0-shot CoT (Table 18) | Capabilities |
| Jul 30, 2025 | Falcon-H1-34B technical report | 76.3 | MGSM, flexible extract 8-shot native CoT, 6 languages (Tables 24 and 35) | Capabilities |
| Jul 17, 2025 | Apple Foundation Model Server (2025) technical report | 87.1 | MGSM, AFM-supported languages only, Simple-Evals (Table 2) | Capabilities |
| May 13, 2025 | Aya Vision 32B technical report | 79.5 | MGSM, Aya Vision languages (Table 5) | Capabilities |
| Apr 5, 2025 | Llama 4 Scout model card | 90.6 | MGSM, 0-shot, average/em, instruction tuned (Instruction tuned models table) | Capabilities |
| Apr 5, 2025 | Llama 4 Maverick model card | 92.3 | MGSM, 0-shot, average/em, instruction tuned (Instruction tuned models table) | Capabilities |
| Apr 1, 2025 | Cohere Command A technical report | 90.1 | MGSM average over 6 languages, simple-evals (Table 24) | Capabilities |
| Mar 25, 2025 | Gemma 3 4B technical report | 34.7 | MGSM, PT, 8-shot (Table 13) | Capabilities |
| Mar 25, 2025 | Gemma 3 27B technical report | 74.3 | MGSM, PT, 8-shot (Table 13) | Capabilities |
| Mar 25, 2025 | Gemma 3 1B technical report | 2.04 | MGSM, PT, 8-shot (Table 13) | Capabilities |
| Mar 25, 2025 | Gemma 3 12B technical report | 64.3 | MGSM, PT, 8-shot (Table 13) | Capabilities |
| Mar 3, 2025 | Phi-4 Mini technical report | 63.9 | MGSM (0-Shot, CoT) (Table 7) | Capabilities |
| Dec 19, 2024 | Qwen2.5-7B technical report | 66.1 | MGSM8K (extended), Qwen2.5-7B-Instruct (Table 14) | Capabilities |
| Dec 19, 2024 | Qwen2.5-72B technical report | 88.2 | MGSM8K (extended), Qwen2.5-72B-Instruct (Table 13) | Capabilities |
| Dec 19, 2024 | Qwen2.5-32B technical report | 87.2 | MGSM8K (extended), Qwen2.5-32B-Instruct (Table 13) | Capabilities |
| Dec 19, 2024 | Qwen2.5-14B technical report | 82.3 | MGSM8K (extended), Qwen2.5-14B-Instruct (Table 14) | Capabilities |
| Dec 12, 2024 | Phi-4 technical report | 80.6 | MGSM, simple-evals (Table 1) | Capabilities |
| Dec 6, 2024 | Llama 3.3 70B Instruct model card | 91.1 | MGSM, 0-shot, em (Instruction tuned models table) | Capabilities |
| Nov 23, 2024 | Llama 3.1-8B technical report | 68.9 | MGSM (0-shot, CoT), post-trained (Table 2, Table 20) | Capabilities |
| Nov 23, 2024 | Llama 3.1-70B technical report | 86.9 | MGSM (0-shot, CoT), post-trained (Table 2, Table 20) | Capabilities |
| Nov 23, 2024 | Llama 3.1-405B technical report | 91.6 | MGSM (0-shot, CoT), post-trained (Table 2, Table 20) | Capabilities |
| Oct 22, 2024 | Claude 3.5 Sonnet (October 2024) model card | 92.5 | MGSM, 0-shot CoT (Table 7) | Capabilities |
| Oct 22, 2024 | Claude 3.5 Haiku model card | 85.6 | MGSM, 0-shot CoT (Table 8) | Capabilities |
| Aug 30, 2024 | Phi-3.5-MoE technical report | 58.7 | MGSM (0-shot CoT) (Table 3) | Capabilities |
| Aug 30, 2024 | Phi-3.5-mini technical report | 47.9 | MGSM (0-shot CoT) (Table 3) | Capabilities |
| Jul 23, 2024 | Llama 3.1-8B model card | 68.9 | Multilingual MGSM (CoT), 0-shot, em, instruction tuned (Instruction tuned models table) | Capabilities |
| Jul 23, 2024 | Llama 3.1-70B model card | 86.9 | Multilingual MGSM (CoT), 0-shot, em, instruction tuned (Instruction tuned models table) | Capabilities |
| Jul 23, 2024 | Llama 3.1-405B model card | 91.6 | Multilingual MGSM (CoT), 0-shot, em, instruction tuned (Instruction tuned models table) | Capabilities |
| Jun 20, 2024 | Claude 3.5 Sonnet model card | 91.6 | MGSM, 0-shot CoT (Table 1) | Capabilities |
| Mar 4, 2024 | Claude 3 Sonnet model card | 83.5 | MGSM, 0-shot. 8-shot 83.7 (Table 1, Table 4) | Capabilities |
| Mar 4, 2024 | Claude 3 Opus model card | 90.7 | MGSM, 0-shot. 8-shot 90.5 (Table 1, Table 4) | Capabilities |
| Mar 4, 2024 | Claude 3 Haiku model card | 75.1 | MGSM, 0-shot. 8-shot 76.5 (Table 1, Table 4) | Capabilities |
| Jan 5, 2024 | DeepSeek LLM 67B technical report | 74 | MGSM-zh (Chinese subset only), chain of thought, DeepSeek LLM 67B Chat. 76.4 with tool-integrated reasoning (Table 16) | Capabilities |
Versions
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
K-EXAONE 90.5% - MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7% - Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Nemotron 3 Ultra 550B-A55B Base 90.1%
- MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%