Mistral Large places in the top three on OR-Bench.
Mistral AI Released February 26th, 2024 Family Mistral Large The lab's model card Compare with others Mistral AI Feb 2024 Closed weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| EQ-Bench v2 | 85.2 | #4 of 85 | Official leaderboard | |
| WinoGrande | 86.7% | #5 of 47 | Lab model card | |
| HellaSwag | 89.2% | #9 of 53 | Lab model card | |
| ARC-Challenge | 94% | #10 of 49 | Lab model card | |
| MixEval | 84.2% | #8 of 34 | Official leaderboard | |
| AlpacaEval 2.0 | 32.6% | #14 of 47 | Official leaderboard | |
| TriviaQA | 82.7% | #15 of 40 | Lab model card | |
| WildBench | 42.3 | #23 of 47 | Official leaderboard | |
| MMLU | 81.2% | #70 of 131 | Lab model card | |
| CulturalBench | 43.9% | #9 of 15 | Official leaderboard | |
| LLM-AggreFact | 74.2% | #6 of 10 | Official leaderboard | |
| TruthfulQA | 50.5% | #20 of 30 | Lab model card | |
| LMArena (Elo) | 1,243 | #264 of 335 | Official leaderboard | |
| SEAL Precise Instruction Following | 79.9 | #16 of 18 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 91.2% | #38 of 92 | Lab model card | |
| MATH Level 5 | 24.5% | #71 of 94 | Epoch AI run | |
| MATH | 45% | #77 of 99 | Lab model card | |
| GPQA Diamond | 38.8% | #188 of 229 | Epoch AI run | |
| GSM1k | 87.5% | #13 of 15 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 1.94% | #197 of 212 | Epoch AI run | |
| ForecastBench | 0.17 | #78 of 82 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 62.2% | #25 of 46 | Official leaderboard | |
| MBPP | 73.1% | #39 of 71 | Lab model card | |
| MBPP+ | 59.5% | #22 of 35 | Official leaderboard | |
| BigCodeBench | 34.1% | #58 of 69 | Official leaderboard | |
| HumanEval | 45.1% | #87 of 101 | Lab model card | |
| BigCodeBench-Hard | 8.40% | #86 of 96 | Official leaderboard | |
| SEAL Coding | 811 | #24 of 26 | Official leaderboard | |
| DesignArena Agentic Overall | 908 | #72 of 74 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GTA | 17.1% | #5 of 15 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OR-Bench | 9.80% | #3 of 28 | Official leaderboard | |
| SEAL Adversarial Robustness | 37.0 | #7 of 8 | Official leaderboard | |
| DistressBench | 59.1% | #39 of 42 | Official leaderboard |
The Self-Reported Index fits one score to the 10 capability results Mistral AI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 111.1 to 120.5. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
The benchmarks the lab chose to headline, with the scores it gave.