Mixtral 8x22B
Mixtral 8x22B.
41
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Mistral AI Apr 2024 Mixtral 8x22B
Open weights
118.7
#188 of 256
Epoch Capabilities Index
- GPQA Diamond 34.1% #198 /229
- SimpleQA Verified 17.8% #43 /56
- MultiLoKo 19.3% #6 /7
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMLU-CF | 65.3% | #11 of 37 | Official leaderboard | |
| AlpacaEval 2.0 | 30.9% | #15 of 47 | Official leaderboard | |
| EQ-Bench v2 | 78.8 | #28 of 85 | Official leaderboard | |
| RepoQA | 67.8% | #11 of 22 | Official leaderboard | |
| CulturalBench | 44.3% | #8 of 15 | Official leaderboard | |
| MMLU-Pro | 56.3% | #122 of 212 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 4.70% | #86 of 148 | Official leaderboard | |
| SuperGPQA | 29.2% | #43 of 59 | Official leaderboard | |
| FACTS Grounding | 70.1% | #25 of 34 | Official leaderboard | |
| TurkishMMLU | 54.8% | #6 of 8 | Official leaderboard | |
| SimpleQA Verified | 17.8% | #43 of 56 | Official leaderboard | |
| ToRR | 41.1 | #11 of 14 | Official leaderboard | |
| ThaiExam | 37.7% | #19 of 24 | Official leaderboard | |
| LMArena (Elo) | 1,230 | #271 of 335 | Official leaderboard | |
| ECLeKTic | 8.52% | #29 of 35 | Official leaderboard | |
| MultiLoKo | 19.3% | #6 of 7 | Official leaderboard | |
| LLM-AggreFact | 71.5% | #9 of 10 | Official leaderboard | |
| IFEval | 72.4% | #58 of 60 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MATH Level 5 | 24.2% | #72 of 94 | Epoch AI run | |
| DTBench | 55.1% | #151 of 180 | Official leaderboard | |
| GPQA Diamond | 34.1% | #198 of 229 | Epoch AI run | |
| GPQA Main | 33.4% | #53 of 60 | Official leaderboard | |
| ForecastBench | 0.17 | #76 of 82 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 72% | #16 of 46 | Official leaderboard | |
| HumanEval | 76.2% | #19 of 46 | Official leaderboard | |
| BigCodeBench | 45.4% | #33 of 69 | Official leaderboard | |
| MBPP+ | 64.3% | #18 of 35 | Official leaderboard | |
| MBPP | 73.8% | #19 of 35 | Official leaderboard | |
| BigCodeBench-Hard | 19.9% | #62 of 96 | Official leaderboard | |
| MHPP | 21.9% | #15 of 23 | Official leaderboard | |
| SciCode | 0% | #16 of 18 | Official leaderboard | |
| SWT-bench | 0.70% | #7 of 7 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 70.4% | #23 of 71 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Cybench | 7.50% | #13 of 16 | Official leaderboard | |
| HELM Safety | 85.1 | #65 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 44% | #72 of 82 | Official leaderboard | |
| FORTRESS | 56.1% | #65 of 73 | Official leaderboard | |
| Anthropic Red Team | 96.2% | #67 of 75 | Official leaderboard | |
| SimpleSafetyTests | 91.5% | #71 of 75 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 70.5 | #22 of 71 | Official leaderboard | |
| HELM Capabilities | 47.9 | #55 of 60 | Official leaderboard |
Mistral AI's models over time
Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)