Llama 2-34B.
Meta Jul 2023 Closed weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TriviaQA | 84.5% | #12 of 40 | Lab model card | |
| BoolQ | 83.7% | #9 of 28 | Lab model card | |
| Natural Questions | 32.8% | #9 of 20 | Lab model card | |
| OpenBookQA | 58.2% | #13 of 24 | Lab model card | |
| WinoGrande | 76.7% | #26 of 47 | Lab model card | |
| PIQA | 81.9% | #21 of 37 | Lab model card | |
| HellaSwag | 83.3% | #33 of 53 | Lab model card | |
| SIQA | 50.9% | #15 of 24 | Lab model card | |
| SQuAD v2 | 77.4% | #5 of 8 | Lab model card | |
| CommonSenseQA | 74.3% | #13 of 19 | Lab model card | |
| ARC-Easy | 79.4% | #18 of 23 | Lab model card | |
| AGIEval | 43.4% | #32 of 40 | Lab model card | |
| ARC-Challenge | 54.5% | #41 of 49 | Lab model card | |
| TruthfulQA | 43.5% | #26 of 30 | Lab model card | |
| MMLU | 62.6% | #115 of 131 | Lab model card | |
| BIG-Bench Hard | 44.1% | #66 of 73 | Lab model card |
The Self-Reported Index fits one score to the 19 capability results Meta reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 95.5 to 101.8. How we compute it.
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)
Every result in the longer documents the lab published with it. 25 of them aren't in the launch post.