Grok-2 Beta Release Aug 13, 2024
Grok-2 Aug 2024
Grok-2.
11
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
xAI Aug 2024
Grok-2
Aug 2024 snapshot Closed weights Flagship
128.0
#290 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
- MMLU-Pro 75.5% #108 /167
- MathVista 69% #48 /70
- MMLU 87.5% #32 /131
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMLU | 87.5% | #32 of 131 | Lab model card | |
| LMArena (Elo) | 1,336 | #192 of 335 | Official leaderboard | |
| MMLU-Pro | 75.5% | #108 of 167 | Lab model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MATH | 76.1% | #35 of 99 | Lab model card | |
| GPQA Diamond | 56% | #232 of 302 | Lab model card | |
| SimpleBench | 22.7% | #96 of 106 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 88.4% | #20 of 101 | Lab model card | |
| Aider-Edit | 58.6% | #24 of 34 | Official leaderboard |
Its lab's numbers put it at 128.0, number 290 of 404
The Self-Reported Index fits one score to the 8 capability results xAI reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 125.6 to 130.4. How we compute it.
Its lab's number next to a leaderboard
Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.