Llama 3.1-8B ranks first on AgentHarm.
Meta Jul 2024 Open weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AlGhafa | 70.9% | #11 of 39 | Official leaderboard | |
| HELMET | 37.1% | #6 of 16 | Official leaderboard | |
| ArabicMMLU | 53.6% | #15 of 39 | Official leaderboard | |
| ArbMMLU-HT | 47.9% | #15 of 39 | Official leaderboard | |
| BullshitBench | 34.5% | #43 of 107 | Official leaderboard | |
| Open Arabic LLM Leaderboard v2 | 55.4% | #17 of 39 | Official leaderboard | |
| MadinahQA | 43.1% | #21 of 39 | Official leaderboard | |
| ARC-Challenge | 83.4% | #27 of 49 | Lab model card | |
| Needle In A Haystack | 98.8% | #5 of 9 | Lab model card | |
| AlpacaEval 2.0 | 20.9% | #27 of 47 | Official leaderboard | |
| Open Arabic LLM Leaderboard v1 | 43% | #32 of 55 | Official leaderboard | |
| TriviaQA | 77.6% | #24 of 40 | Lab model card | |
| WinoGrande | 75.7% | #29 of 47 | Lab model card | |
| CommonSenseQA | 75% | #12 of 19 | Lab model card | |
| MLogiQA | 39.9% | #11 of 17 | Official leaderboard | |
| AP Calculus | 57.1% | #2 of 3 | Lab model card | |
| Vectara Hallucination Leaderboard (HHEM) | 5.40% | #99 of 148 | Official leaderboard | |
| MMLU-CF | 57.1% | #25 of 37 | Official leaderboard | |
| Phare | 62.5% | #42 of 62 | Official leaderboard | |
| AGIEval | 47.8% | #28 of 40 | Lab model card | |
| MMLU | 73% | #95 of 131 | Lab model card | |
| FaithJudge | 28.4% | #27 of 37 | Official leaderboard | |
| PIQA | 81% | #27 of 37 | Lab model card | |
| MuSR | 38.1% | #72 of 98 | Official leaderboard | |
| SQuAD v2 | 77% | #6 of 8 | Lab model card | |
| QIMMA | 45.3% | #19 of 25 | Official leaderboard | |
| BIG-Bench Hard | 64.2% | #56 of 73 | Lab model card | |
| IFEval | 80.4% | #104 of 135 | Lab model card | |
| MGSM | 68.9% | #44 of 57 | Lab model card | |
| BuzzBench | 32.9 | #16 of 20 | Official leaderboard | |
| CulturalBench | 38% | #12 of 15 | Official leaderboard | |
| SQuAD | 77% | #4 of 5 | Lab model card | |
| BullshitBench V2 | 14% | #104 of 129 | Official leaderboard | |
| Chinese SimpleQA | 17.7% | #22 of 27 | Official leaderboard | |
| AraGen | 24.9% | #35 of 42 | Official leaderboard | |
| SIQA | 49.5% | #20 of 24 | Lab model card | |
| KoBALT | 17% | #10 of 12 | Official leaderboard | |
| LMArena (Elo) | 1,211 | #280 of 335 | Official leaderboard | |
| Wer wird Millionär? LLM Benchmark | 118.0 | #67 of 77 | Official leaderboard | |
| ALRAGE | 49.9% | #34 of 39 | Official leaderboard | |
| BoolQ | 75% | #25 of 28 | Lab model card | |
| SuperGPQA | 20.5% | #53 of 59 | Official leaderboard | |
| BIG-Bench Extra Hard | 3.60% | #10 of 11 | Official leaderboard | |
| AgentIF | 53.6% | #11 of 12 | Official leaderboard | |
| SEA-HELM | 37.2 | #66 of 71 | Official leaderboard | |
| Creative Writing v3 | 47.2 | #112 of 120 | Official leaderboard | |
| MMLU-Pro | 48.3% | #156 of 167 | Lab model card | |
| WritingBench | 40.2 | #43 of 45 | Official leaderboard | |
| OpenBookQA | 45% | #23 of 24 | Lab model card | |
| ThaiExam | 24% | #23 of 24 | Official leaderboard | |
| MultiChallenge | 23.7% | #41 of 42 | Official leaderboard | |
| Judgemark v2.1 | 19.2 | #42 of 42 | Official leaderboard | |
| LongBench v2 | 30.4% | #33 of 33 | Official leaderboard | |
| ToRR | 28.9 | #14 of 14 | Official leaderboard | |
| BBEH micro average | 10.6% | #11 of 11 | Official leaderboard | |
| RACE | 54.3% | #8 of 8 | Lab model card | |
| QuALITY | 56% | #7 of 7 | Lab model card | |
| GRE Verbal | 149.0 | #6 of 6 | Lab model card | |
| WorldSense | 45.5% | #4 of 4 | Lab model card | |
| AP Physics | 57.1% | #3 of 3 | Lab model card | |
| GMAT Verbal | 65.7% | #3 of 3 | Lab model card | |
| InfiniteBench/En.MC | 65.1% | #3 of 3 | Lab model card | |
| InfiniteBench/En.QA | 27.1 | #3 of 3 | Lab model card | |
| Multilingual MMLU (Meta) | 58.6% | #3 of 3 | Lab model card | |
| SAT Reading | 57.4% | #3 of 3 | Lab model card | |
| ZeroSCROLLS/QuALITY | 81% | #3 of 3 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 84.5% | #57 of 92 | Lab model card | |
| OlympicArena | 19.4% | #16 of 25 | Official leaderboard | |
| ZebraLogic | 12.8% | #30 of 46 | Official leaderboard | |
| MATH | 51.9% | #66 of 99 | Lab model card | |
| KOR-Bench | 37.2% | #23 of 33 | Official leaderboard | |
| PHYSICS | 11.7% | #15 of 20 | Official leaderboard | |
| MATH Level 5 | 22.9% | #74 of 94 | Epoch AI run | |
| DTBench | 50.9% | #162 of 180 | Official leaderboard | |
| U-MATH | 29.6% | #27 of 30 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 1.67% | #200 of 212 | Epoch AI run | |
| GPQA Diamond | 27% | #217 of 229 | Epoch AI run | |
| IneqMath | 0% | #33 of 34 | Official leaderboard | |
| GPQA Main | 24.7% | #60 of 60 | Official leaderboard | |
| SciPredict | 14.7% | #12 of 12 | Official leaderboard | |
| GRE Quantitative | 152.0 | #6 of 6 | Lab model card | |
| GMAT Quantitative | 56% | #3 of 3 | Lab model card | |
| GRE Physics | 48% | #3 of 3 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 72.8% | #41 of 71 | Lab model card | |
| MBPP+ | 55.6% | #24 of 35 | Official leaderboard | |
| HumanEval | 72.6% | #70 of 101 | Lab model card | |
| BigCodeBench | 36.6% | #53 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 13.5% | #75 of 96 | Official leaderboard | |
| MHPP | 17.1% | #20 of 23 | Official leaderboard | |
| StructEval | 61.8% | #25 of 28 | Official leaderboard | |
| HumanEval+ | 67.1% | #26 of 27 | Lab model card | |
| BigO(Bench) | 21.8% | #18 of 18 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) | 76.1% | #6 of 41 | Lab model card | |
| BFCL V4 | 25.8% | #50 of 65 | Official leaderboard | |
| BALROG | 15.1% | #27 of 33 | Official leaderboard | |
| AceBench | 33.4% | #14 of 16 | Official leaderboard | |
| LLM Chess | -796 | #120 of 123 | Official leaderboard | |
| LisanBench | 12.7 | #131 of 133 | Official leaderboard | |
| ToolComp | 17.4% | #32 of 32 | Official leaderboard | |
| MCP-Bench | 0.43 | #15 of 15 | Official leaderboard | |
| ComplexFuncBench | 0.10% | #11 of 11 | Official leaderboard | |
| When2Call | 16.6% | #10 of 10 | Official leaderboard | |
| CRMArena | 3.10% | #9 of 9 | Official leaderboard | |
| AdaExploreBench | 13.8% | #8 of 8 | Official leaderboard | |
| API-Bank | 82.6% | #3 of 3 | Lab model card | |
| Gorilla APIBench | 8.20% | #3 of 3 | Lab model card | |
| Nexus | 38.5% | #3 of 3 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CorpFin v2 | 37.8% | #116 of 119 | Official leaderboard | |
| TaxEval v2 | 32.3% | #128 of 129 | Official leaderboard | |
| MedQA | 24.4% | #71 of 71 | Official leaderboard | |
| LEXam | 10% | #30 of 30 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SGP-Bench | 46.5% | #15 of 22 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AgentHarm | 3.10% | #1 of 8 | Official leaderboard | |
| CyberSecEval Prompt Injection | 19% | #1 of 3 | Lab model card | |
| AraTrust | 75.6% | #19 of 39 | Official leaderboard | |
| SimpleSafetyTests | 98.8% | #40 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 62.3% | #49 of 82 | Official leaderboard | |
| Anthropic Red Team | 96.8% | #61 of 75 | Official leaderboard | |
| HELM Safety | 86.2 | #61 of 75 | Official leaderboard | |
| AthenaBench | 38.8% | #5 of 6 | Official leaderboard | |
| nocot-bench (NCRI) | 40.6 | #177 of 185 | Official leaderboard | |
| BrowseSafe++ | 23.1% | #15 of 15 | Official leaderboard | |
| IHEval | 11.3% | #10 of 10 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Open LLM Leaderboard v2 | 14.4 | #73 of 99 | Official leaderboard | |
| HELM Lite | 30.3 | #58 of 71 | Official leaderboard | |
| ACCoRD | 33.7 | #138 of 152 | Official leaderboard | |
| Conceptual Reasoning Index | 13.6 | #140 of 151 | Official leaderboard | |
| LMCA | 5.37% | #142 of 147 | Official leaderboard | |
| HELM Capabilities | 44.4 | #58 of 60 | Official leaderboard |
The Self-Reported Index fits one score to the 24 capability results Meta reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 110.0 to 114.8. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap |
|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) | 76.1% BFCL, 0-shot, acc, instruction tuned (Instruction tuned models table) | 49.6% Leaderboard, prompted | +27 pts |
| IFEval | 80.4% IFEval, instruction tuned (Instruction tuned models table) | 74.3% Leaderboard, strict, share of instructions followed | +6.1 pts |
| GPQA Diamond | 32.8% GPQA (0-shot, CoT), subset not stated, post-trained (Table 2) | 27% Epoch AI run | +5.8 pts |
| MBPP | 72.8% MBPP ++ base version, 0-shot, pass@1, instruction tuned (Instruction tuned models table) | 68.3% Leaderboard, EvalPlus subset, original tests | +4.5 pts |
| HumanEval+ | 67.1% HumanEval+, pass@1, post-trained (Table 18) | 62.8% Leaderboard | +4.3 pts |
| MMLU-Pro | 48.3% MMLU-Pro (CoT), 5-shot, macro_avg/acc, instruction tuned (Instruction tuned models table) | 44.3% Leaderboard | +4.0 pts |
| HumanEval | 72.6% HumanEval, 0-shot, pass@1, instruction tuned (Instruction tuned models table) | 69.5% Leaderboard | +3.1 pts |
| GMAT Quantitative | 56% GMAT Quant., accuracy (Table 17) | 56% Leaderboard | Same |
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Every result in the longer documents the lab published with it. 25 of them aren't in the launch post.