Mistral NeMo
Mistral NeMo.
28
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Mistral AI Jul 2024 Mistral NeMo
Open weights
113.9
#197 of 256
Epoch Capabilities Index
- GPQA Diamond 29.9% #209 /229
- BigCodeBench-Hard 16.6% #69 /96
- BALROG 17.6% #23 /33
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| PIQA | 83.5% | #11 of 58 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| EQ-Bench v2 | 77.1 | #30 of 85 | Official leaderboard | |
| WildBench | 46.9 | #17 of 47 | Official leaderboard | |
| BoolQ | 82.5% | #29 of 61 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| MLogiQA | 41.8% | #9 of 17 | Official leaderboard | |
| HELMET | 25% | #9 of 16 | Official leaderboard | |
| Slop Score | 54.4 | #10 of 17 | Official leaderboard | |
| MMLU-Pro | 44.8% | #146 of 212 | Official leaderboard | |
| Wer wird Millionär? LLM Benchmark | 224.0 | #59 of 77 | Official leaderboard | |
| CulturalBench | 37.7% | #13 of 15 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 11.2% | #132 of 148 | Official leaderboard | |
| Creative Writing v3 | 48.0 | #110 of 120 | Official leaderboard | |
| HUMAINE | 17 | #47 of 50 | Official leaderboard | |
| Longform Creative Writing (EQ-Bench) | 28.7 | #114 of 119 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 84.2% | #15 of 70 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| ZebraLogic | 11.8% | #33 of 46 | Official leaderboard | |
| GPQA Diamond | 29.9% | #209 of 229 | Epoch AI run | |
| MATH Level 5 | 10.8% | #88 of 94 | Epoch AI run | |
| DTBench | 48.6% | #170 of 180 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench-Hard | 16.6% | #69 of 96 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BFCL V4 | 27.6% | #45 of 65 | Official leaderboard | |
| BALROG | 17.6% | #23 of 33 | Official leaderboard | |
| Berkeley Function Calling Leaderboard (BFCL) | 51.5% | #48 of 67 | Official leaderboard | |
| SnakeBench | 15 | #173 of 188 | Official leaderboard | |
| LisanBench | 48.7 | #125 of 133 | Official leaderboard | |
| Galileo Agent Leaderboard | 0.66 | #28 of 29 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BrowseSafe++ | 59.7% | #14 of 15 | Official leaderboard | |
| nocot-bench (NCRI) | 42.5 | #176 of 185 | Official leaderboard |
Mistral AI's models over time
Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)