Codestral 25.01 (Mistral news) Jan 13, 2025
- HumanEval (Python) 86.6
- MBPP 80.2
- CruxEval 55.5
- LiveCodeBench (version not stated) 37.9
- RepoBench 38
- Spider (SQL) 66.5
- CanItEdit 50.5
- HumanEval (average on several languages) 71.4
Codestral
Codestral.
14
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Mistral AI Jan 2025 Codestral
Open weights
No Epoch Capabilities Index score yet
- SimpleQA Verified 5.56% #54 /56
- BaxBench 15.8% #24 /34
- AutoBaxBench 9.52% #8 /11
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| FACTS Grounding | 59.7% | #32 of 34 | Official leaderboard | |
| ECLeKTic | 3.20% | #33 of 35 | Official leaderboard | |
| SimpleQA Verified | 5.56% | #54 of 56 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 80.2% | #20 of 71 | Lab model card | |
| HumanEval | 86.6% | #36 of 101 | Lab model card | |
| CanItEdit | 50.5% | #2 of 3 | Lab model card | |
| LiveCodeBench | 37.9% | #37 of 54 | Lab model card | |
| Spider | 66.5% | #3 of 4 | Lab model card | |
| CRUXEval | 55.5% | #7 of 9 | Lab model card | |
| HumanEval Multi | 71.4% | #4 of 5 | Lab model card | |
| Aider Polyglot | 11.1% | #42 of 46 | Official leaderboard | |
| RepoBench | 38% | #1 of 1 | Lab model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BaxBench | 15.8% | #24 of 34 | Official leaderboard | |
| AutoBaxBench | 9.52% | #8 of 11 | Official leaderboard |
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.