- Best of MMLU [base pretrained] / Multilingual MMLU French [base] / Multilingual MMLU German [base] / Multilingual MMLU Spanish [base] / Multilingual MMLU Italian [base] 84
- MT Bench 8.63
- Wild Bench 56.3
- Arena Hard 73.2
- Best of Human Eval / Human Eval Plus 92
- Best of MBPP Base / MBPP Plus 80
- MultiPL-E Average (Python, C++, Bash, Java, TypeScript, PHP, C#) 76.9
- GSM8K (8-shot) 93
- Math Instruct (0-shot, CoT) [0-shot no CoT = 70%] 71.5
Mistral Large 2 Jul 2024
Mistral Large 2.
34
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Mistral AI Jul 2024 Mistral Large 2
Jul 2024 snapshot Open weights Flagship
126.3
#169 of 256
Epoch Capabilities Index
Self-Reported Index 126.9 #303 of 404
- GPQA Diamond 49% #159 /229
- KOR-Bench 53.1% #8 /33
- IHEval 29.4% #6 /10
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| EQ-Bench v2 | 85.0 | #5 of 85 | Official leaderboard | |
| WildBench | 55.8 | #5 of 47 | Official leaderboard | |
| MixEval | 86.1% | #5 of 34 | Official leaderboard | |
| MMLU-Pro | 65.9% | #97 of 212 | Official leaderboard | |
| MMLU | 84% | #60 of 131 | Lab model card | |
| SEAL Precise Instruction Following | 82.8 | #9 of 18 | Official leaderboard | |
| SEAL Multilingual | 1,032 | #10 of 19 | Official leaderboard | |
| MT-Bench | 8.63 | #14 of 23 | Lab model card | |
| Arena-Hard-Auto | 73.2% | #25 of 39 | Lab model card | |
| LMArena (Elo) | 1,314 | #215 of 335 | Official leaderboard | |
| LongBench v2 | 33.6% | #29 of 33 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ZebraLogic | 29% | #11 of 46 | Official leaderboard | |
| KOR-Bench | 53.1% | #8 of 33 | Official leaderboard | |
| GSM8K | 93% | #25 of 92 | Lab model card | |
| MATH | 71.5% | #42 of 99 | Lab model card | |
| GSM1k | 93.9% | #7 of 15 | Official leaderboard | |
| MATH Level 5 | 44.8% | #57 of 94 | Epoch AI run | |
| GPQA Diamond | 49% | #159 of 229 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 8.47% | #163 of 212 | Epoch AI run | |
| DTBench | 61.2% | #141 of 180 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 92% | #10 of 101 | Lab model card | |
| MBPP | 80% | #23 of 71 | Lab model card | |
| MHPP | 41.4% | #8 of 23 | Official leaderboard | |
| MultiPL-E | 76.9% | #7 of 18 | Lab model card | |
| BigCodeBench-Hard | 26% | #41 of 96 | Official leaderboard | |
| SEAL Coding | 1,029 | #13 of 26 | Official leaderboard | |
| SWT-bench | 16.3% | #4 of 7 | Official leaderboard | |
| Aider-Edit | 60.2% | #22 of 34 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ToolComp | 50.4% | #25 of 32 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 77.5% | #12 of 71 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| IHEval | 29.4% | #6 of 10 | Official leaderboard | |
| AIR-Bench 2024 | 35.3% | #77 of 82 | Official leaderboard | |
| AgentHarm | 69.1% | #8 of 8 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 74.4 | #19 of 71 | Official leaderboard |
Its lab's numbers put it at 126.9, in line with its ECI
The Self-Reported Index fits one score to the 7 capability results Mistral AI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 123.8 to 130.0. How we compute it.
Its lab's number next to a leaderboard
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Mistral AI's models over time
Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.