Mistral 7B v0.1.
Mistral AI Sep 2023 Open weights Open model
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenBookQA | 79.8% | #7 of 41 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| BIG-Bench Hard | 56.1% | #19 of 45 | Claim in Gemma: Open Models Based on Gemini Research and Technology, via Epoch AI | |
| MuSR | 41.4% | #46 of 98 | Official leaderboard | |
| PIQA | 83% | #18 of 37 | Lab model card | |
| BoolQ | 80.5% | #32 of 61 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| WinoGrande | 75.3% | #30 of 47 | Lab model card | |
| HellaSwag | 81.3% | #37 of 53 | Lab model card | |
| Natural Questions | 28.8% | #14 of 20 | Lab model card | |
| ANLI | 47.1% | #8 of 11 | Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI | |
| ARC-Easy | 80% | #17 of 23 | Lab model card | |
| TriviaQA | 69.9% | #31 of 40 | Lab model card | |
| ARC-Challenge | 55.5% | #40 of 49 | Lab model card | |
| MMLU-Pro | 30.9% | #180 of 212 | Official leaderboard | |
| MMLU | 60.1% | #117 of 131 | Lab model card | |
| MT-Bench | 6.84 | #22 of 23 | Lab model card | |
| LMArena (Elo) | 1,031 | #14 of 14 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CRUXEval | 34.3% | #8 of 11 | Official leaderboard | |
| MBPP | 47.5% | #61 of 71 | Lab model card | |
| MBPP+ | 42.1% | #31 of 35 | Official leaderboard | |
| HumanEval+ | 23.8% | #42 of 46 | Official leaderboard | |
| HumanEval | 30.5% | #95 of 101 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| T-Eval | 56% | #6 of 12 | Official leaderboard | |
| AgentBoard | 24.6% | #2 of 4 | Official leaderboard | |
| LLM-WikiRace | 1.00% | #12 of 21 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 52.5% | #57 of 71 | Official leaderboard | |
| LegalBench | 53.8% | #133 of 138 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MolecularIQ | 1.19% | #17 of 17 | Official leaderboard | |
| TrustLLM JailbreakTrigger | 59% | #4 of 4 | Official leaderboard | |
| Mistral 175 unsafe prompts | 100% | #1 of 1 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Open LLM Leaderboard v2 | 14.6 | #72 of 99 | Official leaderboard | |
| HELM Lite | 29.2 | #60 of 71 | Official leaderboard |
The Self-Reported Index fits one score to the 12 capability results Mistral AI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 96.8 to 101.9. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 6 of them aren't in the launch post.