Llama 3.1-405B ranks first on 6 of the 91 benchmarks with five or more models scored.
Meta Jul 2024 Open weights Open model
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TriviaQA | 91.8% | #1 of 40 | Lab model card | |
| HellaSwag | 89.2% | #2 of 63 | Claim in DeepSeek-V3 Technical Report, via Epoch AI | |
| ARC-Challenge | 96.9% | #2 of 49 | Lab model card | |
| Phare | 76.4% | #6 of 62 | Official leaderboard | |
| PIQA | 85.6% | #4 of 37 | Lab model card | |
| MGSM | 91.6% | #7 of 57 | Lab model card | |
| ThaiExam | 68.2% | #3 of 24 | Official leaderboard | |
| SQuAD v2 | 81.8% | #1 of 8 | Lab model card | |
| WinoGrande | 86.7% | #6 of 47 | Lab model card | |
| EQ-Bench v2 | 83.0 | #11 of 85 | Official leaderboard | |
| MultiLoKo | 31% | #1 of 7 | Official leaderboard | |
| AlpacaEval 2.0 | 39.3% | #7 of 47 | Official leaderboard | |
| CommonSenseQA | 85.8% | #3 of 19 | Lab model card | |
| MMLU | 88.6% | #23 of 131 | Lab model card | |
| SQuAD | 89.3% | #1 of 5 | Lab model card | |
| CulturalBench | 52.6% | #4 of 15 | Official leaderboard | |
| AGIEval | 71.6% | #11 of 40 | Lab model card | |
| QuALITY | 87.6% | #2 of 7 | Lab model card | |
| AP Calculus | 88.6% | #1 of 3 | Lab model card | |
| AP Physics | 92.9% | #1 of 3 | Lab model card | |
| GMAT Verbal | 86.6% | #1 of 3 | Lab model card | |
| InfiniteBench/En.MC | 83.4% | #1 of 3 | Lab model card | |
| Multilingual MMLU (Meta) | 83.2% | #1 of 3 | Lab model card | |
| SAT Reading | 74.8% | #1 of 3 | Lab model card | |
| ZeroSCROLLS/QuALITY | 95.2% | #1 of 3 | Lab model card | |
| BIG-Bench Hard | 85.9% | #25 of 73 | Lab model card | |
| ToRR | 46.1 | #5 of 14 | Official leaderboard | |
| IFEval | 88.6% | #49 of 135 | Lab model card | |
| SIQA | 53.7% | #9 of 24 | Lab model card | |
| SEAL Precise Instruction Following | 84.8 | #7 of 18 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 3.90% | #62 of 148 | Official leaderboard | |
| LLM-AggreFact | 74.4% | #5 of 10 | Official leaderboard | |
| GRE Verbal | 166.0 | #3 of 6 | Lab model card | |
| WorldSense | 63.7% | #2 of 4 | Lab model card | |
| BuzzBench | 45.9 | #11 of 20 | Official leaderboard | |
| LLM Confabulation Leaderboard | 17.6% | #32 of 58 | Official leaderboard | |
| SuperGPQA | 43.1% | #33 of 59 | Official leaderboard | |
| LMArena (Elo) | 1,335 | #195 of 335 | Official leaderboard | |
| BoolQ | 80% | #17 of 28 | Lab model card | |
| OpenBookQA | 49.2% | #16 of 24 | Lab model card | |
| Needle In A Haystack | 98.1% | #6 of 9 | Lab model card | |
| InfiniteBench/En.QA | 30.5 | #2 of 3 | Lab model card | |
| MMLU-Pro | 73.3% | #115 of 167 | Lab model card | |
| Creative Writing v3 | 51.5 | #104 of 120 | Official leaderboard | |
| RACE | 58.1% | #7 of 8 | Lab model card | |
| EQ-Bench 3 | 853 | #72 of 78 | Official leaderboard | |
| SEAL Multilingual | 915 | #19 of 19 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 96.8% | #1 of 92 | Lab model card | |
| KOR-Bench | 55.4% | #5 of 33 | Official leaderboard | |
| ZebraLogic | 32.6% | #9 of 46 | Official leaderboard | |
| GSM1k | 95.6% | #3 of 15 | Official leaderboard | |
| GRE Quantitative | 162.0 | #2 of 6 | Lab model card | |
| GMAT Quantitative | 96% | #1 of 3 | Lab model card | |
| GRE Physics | 80% | #1 of 3 | Lab model card | |
| MATH | 73.8% | #39 of 99 | Lab model card | |
| GPQA Main | 52.2% | #32 of 60 | Official leaderboard | |
| MATH Level 5 | 49.8% | #52 of 94 | Epoch AI run | |
| ForecastBench | 0.14 | #52 of 82 | Official leaderboard | |
| GPQA Diamond | 50.9% | #154 of 229 | Epoch AI run | |
| OTIS Mock AIME 2024-2025 | 9.72% | #162 of 212 | Epoch AI run | |
| DTBench | 61.4% | #138 of 180 | Official leaderboard | |
| CNMO 2024 | 6.80% | #6 of 7 | Official leaderboard | |
| SnorkelGraph | 20.5% | #21 of 24 | Official leaderboard | |
| SimpleBench | 23% | #94 of 106 | Official leaderboard | |
| TSP-Opt | 9.22% | #10 of 11 | Official leaderboard | |
| SnorkelSequences | 35.2% | #24 of 25 | Official leaderboard | |
| 3SAT-Search | 8.25% | #11 of 11 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 88.6% | #3 of 71 | Lab model card | |
| HumanEval | 89% | #16 of 101 | Lab model card | |
| BigO(Bench) | 38.3% | #5 of 18 | Official leaderboard | |
| BigCodeBench-Hard | 26.4% | #36 of 96 | Official leaderboard | |
| MHPP | 39.5% | #10 of 23 | Official leaderboard | |
| SEAL Coding | 1,022 | #14 of 26 | Official leaderboard | |
| HumanEval+ | 82.3% | #18 of 27 | Lab model card | |
| SciCode | 1.50% | #14 of 18 | Official leaderboard | |
| SWE-Bench Pro | 11.2% | #22 of 24 | Official leaderboard | |
| MLE-bench | 3.33% | #12 of 12 | Official leaderboard | |
| OIBench | 1.15% | #9 of 9 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) | 88.5% | #1 of 41 | Lab model card | |
| CRMArena | 51.3% | #3 of 9 | Official leaderboard | |
| API-Bank | 92.3% | #1 of 3 | Lab model card | |
| Gorilla APIBench | 35.3% | #1 of 3 | Lab model card | |
| Nexus | 58.7% | #1 of 3 | Lab model card | |
| LisanBench | 200.0 | #88 of 133 | Official leaderboard | |
| ToolComp | 50.4% | #24 of 32 | Official leaderboard | |
| ComplexFuncBench | 4.00% | #9 of 11 | Official leaderboard | |
| Elimination Game | 3 | #46 of 56 | Official leaderboard | |
| Step Race | 1 | #55 of 59 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 80.5% | #8 of 71 | Official leaderboard | |
| TheAgentCompany | 7.40% | #3 of 7 | Official leaderboard | |
| LEXam | 43.1% | #17 of 30 | Official leaderboard | |
| FinanceReasoning | 81.2% | #4 of 6 | Official leaderboard | |
| TaxEval v2 | 60.9% | #111 of 129 | Official leaderboard | |
| SnorkelUnderwrite | 20% | #26 of 27 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SGP-Bench | 58% | #6 of 22 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AgentHarm | 4.30% | #2 of 8 | Official leaderboard | |
| SEAL Adversarial Robustness | 10.0 | #2 of 8 | Official leaderboard | |
| FORTRESS | 20.6% | #33 of 73 | Official leaderboard | |
| CyberSecEval Spear Phishing | 14% | #1 of 2 | Lab model card | |
| SimpleSafetyTests | 98.8% | #39 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 58.6% | #54 of 82 | Official leaderboard | |
| HELM Safety | 89.7 | #50 of 75 | Official leaderboard | |
| CyberSecEval Prompt Injection | 21.7% | #2 of 3 | Lab model card | |
| Anthropic Red Team | 96.5% | #65 of 75 | Official leaderboard | |
| Cybench | 7.50% | #15 of 16 | Official leaderboard | |
| CyberSecEval Code Interpreter Abuse | 10.4% | #2 of 2 | Lab model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 85.4 | #8 of 71 | Official leaderboard | |
| HELM Capabilities | 61.8 | #38 of 60 | Official leaderboard |
The Self-Reported Index fits one score to the 24 capability results Meta reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 122.4 to 129.3. How we compute it.
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
| Benchmark | Lab says | Board says | Gap |
|---|---|---|---|
| IFEval | 88.6% IFEval | 81.1% Leaderboard, strict, share of instructions followed | +7.5 pts |
| GPQA Diamond | 51.1% GPQA (0-shot, CoT) | 50.9% Epoch AI run | +0.2 pts |
| GMAT Quantitative | 96% GMAT Quant., accuracy (Table 17) | 96% Leaderboard | Same |
| MMLU-Pro | 73.3% MMLU-Pro (5-shot, CoT) | 73.3% Leaderboard | Same |
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 13 of them aren't in the launch post.