Code Llama 34B Instruct
Code Llama 34B Instruct.
19
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Meta Aug 2023 Code Llama 34B Instruct
Open weights Mid-size
No Epoch Capabilities Index score yet
- BigCodeBench-Hard 9.80% #84 /96
- RepoQA 41.6% #18 /22
- GSM1k 37.5% #15 /15
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| RepoQA | 41.6% | #18 of 22 | Official leaderboard | |
| LMArena (Elo) | 1,137 | #314 of 335 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM1k | 37.5% | #15 of 15 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MultiPL-E | 35.1% | #5 of 9 | Official leaderboard | |
| BigCodeBench | 32.3% | #60 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 9.80% | #84 of 96 | Official leaderboard | |
| SEAL Coding | 598 | #26 of 26 | Official leaderboard | |
| FullStackBench | 29.2% | #10 of 10 | Official leaderboard | |
| CodeEditorBench | 21.1% | #7 of 7 | Official leaderboard | |
| McEval | 29.1% | #4 of 4 | Official leaderboard | |
| DevBench Implementation | 0.60% | #3 of 3 | Official leaderboard | |
| DevBench Acceptance Testing | 4.50% | #2 of 2 | Official leaderboard | |
| DevBench Environment Setup | 16.7% | #2 of 2 | Official leaderboard | |
| DevBench Unit Testing | 21.1% | #2 of 2 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AgentBench | 0.96 | #5 of 5 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CyberSecEval MITRE | 58.8% | #2 of 2 | Official leaderboard | |
| CyberSecEval Autocomplete | 73.1% | #1 of 1 | Official leaderboard | |
| CyberSecEval Instruct | 62.5% | #1 of 1 | Official leaderboard | |
| CyberSecEval MITRE False Refusal Rate | 1.60% | #1 of 1 | Official leaderboard |