Benchmark Almanac

Mixtral 8x7B

Mixtral 8x7B.

Mistral AI Released December 11th, 2023 Family Mixtral The lab's model card Compare with others
58
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Mistral AI Dec 2023

Mixtral 8x7B

Open weights Open model

113.2 #198 of 256
Epoch Capabilities Index
Self-Reported Index 108.0 #381 of 404
  • GPQA Diamond 30.6% #208 /229
  • LiveSQLBench 2.33% #36 /36
  • Human Pathogen Capabilities Test 15.1% #85 /86

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
OpenBookQA 85.8% #5 of 41 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
AlpacaEval 94.8% #4 of 19 Official leaderboard
MuSR 44.1% #23 of 98 Official leaderboard
CriticEval 5.75 #4 of 12 Official leaderboard
WinoGrande 81.2% #18 of 47 Lab model card
PIQA 83.6% #15 of 37 Lab model card
HellaSwag 86.7% #22 of 53 Lab model card
ARC-Challenge 85.8% #22 of 49 Lab model card
RepoQA 68% #10 of 22 Official leaderboard
ANLI 55.2% #5 of 11 Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
ARC-Easy 83.1% #11 of 23 Lab model card
Passkey Retrieval 100% #1 of 2 Lab model card
AlpacaEval 2.0 23.7% #24 of 47 Official leaderboard
MMLU-CF 58.9% #19 of 37 Official leaderboard
EQ-Bench v2 72.4 #47 of 85 Official leaderboard
Natural Questions 30.6% #12 of 20 Lab model card
WildBench 35.0 #31 of 47 Official leaderboard
MMLU-Pro 43.3% #153 of 212 Official leaderboard
TriviaQA 71.5% #29 of 40 Lab model card
RewardBench 74.5% #35 of 48 Official leaderboard
MMLU 72.5% #96 of 131 Lab model card
LMArena (Elo) 1,197 #286 of 335 Official leaderboard
ToRR 34.7 #12 of 14 Official leaderboard
SuperGPQA 22.1% #51 of 59 Official leaderboard
MT-Bench 8.30 #20 of 23 Lab model card
ThaiExam 30% #22 of 24 Official leaderboard
Vectara Hallucination Leaderboard (HHEM) 20.1% #145 of 148 Official leaderboard
IFEval 57.5% #59 of 60 Official leaderboard

Math and science

BenchmarkScoreRankSource
ZebraLogic 8.70% #40 of 46 Official leaderboard
GSM8K 58.4% #81 of 92 Lab model card
MATH 28.4% #88 of 99 Lab model card
GPQA Diamond 30.6% #208 of 229 Epoch AI run
MATH Level 5 9.95% #89 of 94 Epoch AI run
GPQA Main 29.6% #58 of 60 Official leaderboard
ForecastBench 0.18 #80 of 82 Official leaderboard

Coding

BenchmarkScoreRankSource
CRUXEval 40.5% #6 of 11 Official leaderboard
HumanEval+ 39.6% #34 of 46 Official leaderboard
MBPP 60.7% #56 of 71 Lab model card
MBPP+ 49.7% #28 of 35 Official leaderboard
HumanEval 40.2% #91 of 101 Lab model card
MHPP 11% #21 of 23 Official leaderboard
LiveSQLBench 2.33% #36 of 36 Official leaderboard

Agents and tools

BenchmarkScoreRankSource
VisualWebArena 1.87% #4 of 7 Official leaderboard
GTA 9.77% #9 of 15 Official leaderboard
Spider2-V 0.80% #5 of 5 Official leaderboard

Professional work

BenchmarkScoreRankSource
MedQA 65.2% #40 of 71 Official leaderboard
LegalBench 55.8% #131 of 138 Official leaderboard

Safety and security

BenchmarkScoreRankSource
WMDP Cyber 52% #1 of 2 Official leaderboard
AIR-Bench 2024 39.1% #76 of 82 Official leaderboard
HELM Safety 81.4 #70 of 75 Official leaderboard
Anthropic Red Team 92.8% #72 of 75 Official leaderboard
SimpleSafetyTests 90.5% #72 of 75 Official leaderboard
BBQ (Bias Benchmark for QA) 56% #24 of 25 Lab model card
Human Pathogen Capabilities Test 15.1% #85 of 86 Official leaderboard
WMDP Biology 74.8% #2 of 2 Official leaderboard

Other

BenchmarkScoreRankSource
HELM Lite 51.0 #37 of 71 Official leaderboard
Open LLM Leaderboard v2 19.7 #56 of 99 Official leaderboard
HELM Capabilities 39.7 #59 of 60 Official leaderboard

Its lab's numbers put it at 108.0, 5.3 points below its ECI

The Self-Reported Index fits one score to the 12 capability results Mistral AI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 104.5 to 111.4. How we compute it.

Its lab's numbers next to 3 leaderboards

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap What differs
LMArena (Elo) 1,121 LMSys Arena Elo, Mixtral 8x7B Instruct v0.1, screenshot Dec 22, 2023 (Figure 6 caption) 1,197 Leaderboard −76 Mistral quotes Arena's December 2023 rating, while the board reports a later rating from its updated vote pool and scoring method.
HumanEval 40.2% HumanE [detailed table] 45.1% Leaderboard −4.9 pts
MBPP 60.7% MBPP (pass@1) 59.5% Leaderboard, EvalPlus subset, original tests +1.2 pts

Mistral AI's models over time

Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.

50 100 150 2023 2024 2025 2026 2027 Mistral Large 4 Mistral Medium 3.5 Mistral Medium 3 Mistral Large 2 Mistral NeMo Mixtral 8x7B Mistral 7B v0.1 50 100 150 2023 2024 2025 2026 2027 Mistral Large 4 Mistral Medium 3.5

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Mixtral of experts (launch post) Dec 11, 2023

Its system card

Every result in the longer documents the lab published with it. 7 of them aren't in the launch post.