Benchmark Almanac

Mistral Small 3

Mistral Small 3.

Mistral AI Released January 25th, 2025 Family Mistral Small The lab's model card Compare with others
35
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Mistral AI Jan 2025

Mistral Small 3

Open weights Small

124.0 #320 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • GPQA Diamond 47.3% #169 /229
  • BigCodeBench-Hard 20.6% #57 /96
  • Misguided Attention 27.9% #38 /45

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
Vectara Hallucination Leaderboard (2025 dataset) 5.10% #11 of 104 Official leaderboard
Vectara Hallucination Leaderboard (HHEM) 3.10% #49 of 148 Official leaderboard
ARC-Challenge 91.3% #17 of 49 Lab model card
Arena-Hard-Auto 87.3% #14 of 39 Lab model card
AGIEval 65.8% #17 of 40 Lab model card
StudyArena Text 1,236 #37 of 85 Official leaderboard
TriviaQA 80.3% #18 of 40 Lab model card
FaithJudge 17% #17 of 37 Official leaderboard
MMLU 80.7% #72 of 131 Lab model card
AraGen 39.8% #26 of 42 Official leaderboard
IFEval 82.9% #95 of 135 Lab model card
LMArena (Elo) 1,274 #251 of 335 Official leaderboard
WildBench 52.3 #3 of 4 Lab model card
MMLU-Pro 66.3% #134 of 167 Lab model card
MT-Bench 8.35 #19 of 23 Lab model card
LLM Confabulation Leaderboard 25.2% #48 of 58 Official leaderboard
EQ-Bench 3 862 #71 of 78 Official leaderboard
Creative Writing v3 42.4 #117 of 120 Official leaderboard

Math and science

BenchmarkScoreRankSource
MATH 70.6% #45 of 99 Lab model card
MATH Level 5 44.8% #58 of 94 Epoch AI run
GSM8K 80.7% #64 of 92 Lab model card
GPQA Diamond 47.3% #169 of 229 Epoch AI run
OTIS Mock AIME 2024-2025 6.67% #174 of 212 Epoch AI run
U-MATH 34.8% #25 of 30 Official leaderboard
Misguided Attention 27.9% #38 of 45 Official leaderboard

Coding

BenchmarkScoreRankSource
BigCodeBench 47.8% #29 of 69 Official leaderboard
HumanEval 84.8% #43 of 101 Lab model card
BigCodeBench-Hard 20.6% #57 of 96 Official leaderboard
MBPP 69.6% #46 of 71 Lab model card

Agents and tools

BenchmarkScoreRankSource
SnakeBench 25 #81 of 188 Official leaderboard
Elimination Game 3 #41 of 56 Official leaderboard

Safety and security

BenchmarkScoreRankSource
Anthropic Red Team 96.4% #66 of 75 Official leaderboard
HELM Safety 84.7 #66 of 75 Official leaderboard
SimpleSafetyTests 93.3% #66 of 75 Official leaderboard
AIR-Bench 2024 32.8% #78 of 82 Official leaderboard

Its lab's numbers put it at 124.0, number 320 of 404

The Self-Reported Index fits one score to the 12 capability results Mistral AI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 121.2 to 126.8. How we compute it.

Its lab's number next to a leaderboard

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark Lab says Board says Gap
GPQA Diamond 45.3% gpqa_main_cot_5shot_instruct 47.3% Epoch AI run −2.0 pts

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Mistral Small 3 (launch post) + HF model cards Mistral-Small-24B-Instruct-2501 / Base-2501 Jan 30, 2025