Benchmark Almanac

European LLM Leaderboard (OpenGPT-X)

Meta-Llama-3.1-70B-Instruct 70.6% tops its leaderboard Gemma 7B 56.7% leads the models we track
Not in model cards

Models answer exam, science, commonsense, truthfulness, and math questions across European languages. The leaderboard also includes multilingual reading comprehension.

The paper compares 40 models across 21 languages. The leaderboard README lists Belebele alongside the five translated benchmarks.

Made by
OpenGPT-X
Released
October 2024
Kind
Knowledge in many languages
Level
School
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Measures
Capability
Language
Many languages
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
Strongest model tested
Mixtral 8x7B #199 on the capability index
Last model added
July 2024
Scores refreshed
November 20th, 2026
52
Entries on its own leaderboard, 13 models we track

Meta-Llama-3.1-70B-Instruct tops the leaderboard with 70.6% Gemma 7B leads with 56.7% among models released by July 2024

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Gemma 7B, at 56.7%. This view shows only models we track, each at its best.

The newest model with a score here came out in July 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Meta-Llama-3.1-70B-Instruct Few-shot 70.6%
  2. 2 Gemma-2-27b-Instruct Few-shot 69.8%
  3. 3 Mistral-Nemo-Instruct-12.2B_2407 Few-shot 59.8%
  4. 4 Mixtral-8x7B-Instruct-v0.1 Few-shot 59.1%
  5. 5 Gemma-2-9b-Instruct Few-shot 58.1%
  6. 6 EuroLLM-9B-Instruct Few-shot 57.7%
  7. 7 EuroLLM-9B-4T Zero-shot 57.6%
  8. 8 Gemma 7B Zero-shot 56.7%
  9. 9 Mixtral 8x7B Zero-shot 56.2%
  10. 10 Mistral-Nemo-Base-12.2B_2407 Few-shot 56.2%
  11. 11 Meta-Llama-3.1-8B-Instruct Few-shot 56.2%
  12. 12 c4ai-command-r-35B-v01 Few-shot 55.7%
  13. 13 Meta-Llama-3-8B-Instruct Few-shot 54.7%
  14. 14 Salamandra-7b-instruct Zero-shot 54.4%
  15. 15 Qwen2-7B Few-shot 54.2%
  16. 16 Qwen2-7B-Instruct Few-shot 52.8%
  17. 17 Llama 3.1-8B Zero-shot 52.3%
  18. 18 Phi-3-medium-14B-4k-Instruct Few-shot 51.9%
  19. 19 Teuken-7B-instruct-v0.6 Few-shot 51.4%
  20. 20 Teuken-7B-sigma-v05 Few-shot 51.1%
  21. 21 Phi-3-medium-14B-128k-Instruct Few-shot 50.8%
  22. 22 Llama 3-8B Zero-shot 50.6%
  23. 23 Mistral-NeMo-Minitron-8B-Base Few-shot 50.2%
  24. 24 Salamandra-7b Zero-shot 49.4%
  25. 25 Mistral-7B-Instruct-v0.3 Zero-shot 49.1%
  26. 26 Teuken-7B-instruct-research-v0.4 Zero-shot 48.2%
  27. 27 Mistral-7B-Instruct-v0.2 Zero-shot 48.1%
  28. 28 Mistral 7B v0.1 Zero-shot 46.9%
  29. 29 Teuken-7B-base-v0.55 Zero-shot 46.8%
  30. 30 Teuken-7B-base-v0.50 Zero-shot 46.6%
  31. 31 Mistral 7B v0.3 Zero-shot 46.2%
  32. 32 Vicuna-33b-v1.3 Zero-shot 46%
  33. 33 Teuken-7B-base-v0.45 Zero-shot 45.9%
  34. 34 Teuken-7B-base-v0.4 Zero-shot 45.2%
  35. 35 Teuken-7B-instruct-commercial-v0.4 Few-shot 44.5%
  36. 36 Aya-23-8B Few-shot 44.1%
  37. 37 Occiglot-7b-eu5-Instruct Zero-shot 43%
  38. 38 Vicuna-13b-v1.5 Few-shot 42.5%
  39. 39 Phi-3-mini-3.8B-128k-Instruct Few-shot 42.4%
  40. 40 Occiglot-7b-eu5 Zero-shot 42.1%
  41. 41 Gemma 1.1 7B IT Zero-shot 42%
  42. 42 Mistral-7B-Instruct-v0.1 Zero-shot 41.9%
  43. 43 phi-3-mini 3.8B Few-shot 41.5%
  44. 44 Meta-Llama-2-13B-Chat Few-shot 40.7%
  45. 45 EuroLLM-1.7B-Instruct Zero-shot 38.8%
  46. 46 Llama 2-7B Zero-shot 38.5%
  47. 47 Meta-Llama-2-7B-Chat Zero-shot 38.4%
  48. 48 EuroLLM-1.7B Zero-shot 37.6%
  49. 49 Pharia-1-LLM-7B-control-aligned Zero-shot 37.3%
  50. 50 Pharia-1-LLM-7B-control Zero-shot 36.7%
  51. 51 Bloomz-7b1 Zero-shot 32.8%
  52. 52 Bloom-7b1 Zero-shot 29.9%

Source: Official European LLM Leaderboard (OpenGPT-X) leaderboard

#EntryScoreSettingDateLinkSource
1 Meta-Llama-3.1-70B-Instruct 70.6% Few-shot huggingface.co Official leaderboard
2 Gemma-2-27b-Instruct 69.8% Few-shot huggingface.co Official leaderboard
3 Mistral-Nemo-Instruct-12.2B_2407 59.8% Few-shot huggingface.co Official leaderboard
4 Mixtral-8x7B-Instruct-v0.1 59.1% Few-shot huggingface.co Official leaderboard
5 Gemma-2-9b-Instruct 58.1% Few-shot huggingface.co Official leaderboard
6 EuroLLM-9B-Instruct 57.7% Few-shot huggingface.co Official leaderboard
7 EuroLLM-9B-4T 57.6% Zero-shot huggingface.co Official leaderboard
8 Gemma 7B 56.7% Zero-shot Feb 2024 huggingface.co Official leaderboard
9 Mixtral 8x7B 56.2% Zero-shot Dec 2023 huggingface.co Official leaderboard
10 Mistral-Nemo-Base-12.2B_2407 56.2% Few-shot huggingface.co Official leaderboard
11 Meta-Llama-3.1-8B-Instruct 56.2% Few-shot huggingface.co Official leaderboard
12 c4ai-command-r-35B-v01 55.7% Few-shot huggingface.co Official leaderboard
13 Meta-Llama-3-8B-Instruct 54.7% Few-shot huggingface.co Official leaderboard
14 Salamandra-7b-instruct 54.4% Zero-shot huggingface.co Official leaderboard
15 Qwen2-7B 54.2% Few-shot Jun 2024 huggingface.co Official leaderboard
16 Qwen2-7B-Instruct 52.8% Few-shot Jun 2024 huggingface.co Official leaderboard
17 Llama 3.1-8B 52.3% Zero-shot Jul 2024 huggingface.co Official leaderboard
18 Phi-3-medium-14B-4k-Instruct 51.9% Few-shot huggingface.co Official leaderboard
19 Teuken-7B-instruct-v0.6 51.4% Few-shot huggingface.co Official leaderboard
20 Teuken-7B-sigma-v05 51.1% Few-shot huggingface.co Official leaderboard
21 Phi-3-medium-14B-128k-Instruct 50.8% Few-shot huggingface.co Official leaderboard
22 Llama 3-8B 50.6% Zero-shot Apr 2024 huggingface.co Official leaderboard
23 Mistral-NeMo-Minitron-8B-Base 50.2% Few-shot huggingface.co Official leaderboard
24 Salamandra-7b 49.4% Zero-shot huggingface.co Official leaderboard
25 Mistral-7B-Instruct-v0.3 49.1% Zero-shot huggingface.co Official leaderboard
26 Teuken-7B-instruct-research-v0.4 48.2% Zero-shot huggingface.co Official leaderboard
27 Mistral-7B-Instruct-v0.2 48.1% Zero-shot Dec 2023 huggingface.co Official leaderboard
28 Mistral 7B v0.1 46.9% Zero-shot Sep 2023 huggingface.co Official leaderboard
29 Teuken-7B-base-v0.55 46.8% Zero-shot huggingface.co Official leaderboard
30 Teuken-7B-base-v0.50 46.6% Zero-shot huggingface.co Official leaderboard
31 Mistral 7B v0.3 46.2% Zero-shot Sep 2023 huggingface.co Official leaderboard
32 Vicuna-33b-v1.3 46% Zero-shot huggingface.co Official leaderboard
33 Teuken-7B-base-v0.45 45.9% Zero-shot huggingface.co Official leaderboard
34 Teuken-7B-base-v0.4 45.2% Zero-shot huggingface.co Official leaderboard
35 Teuken-7B-instruct-commercial-v0.4 44.5% Few-shot huggingface.co Official leaderboard
36 Aya-23-8B 44.1% Few-shot huggingface.co Official leaderboard
37 Occiglot-7b-eu5-Instruct 43% Zero-shot huggingface.co Official leaderboard
38 Vicuna-13b-v1.5 42.5% Few-shot huggingface.co Official leaderboard
39 Phi-3-mini-3.8B-128k-Instruct 42.4% Few-shot huggingface.co Official leaderboard
40 Occiglot-7b-eu5 42.1% Zero-shot huggingface.co Official leaderboard
41 Gemma 1.1 7B IT 42% Zero-shot Apr 2024 huggingface.co Official leaderboard
42 Mistral-7B-Instruct-v0.1 41.9% Zero-shot Sep 2023 huggingface.co Official leaderboard
43 phi-3-mini 3.8B 41.5% Few-shot Apr 2024 huggingface.co Official leaderboard
44 Meta-Llama-2-13B-Chat 40.7% Few-shot huggingface.co Official leaderboard
45 EuroLLM-1.7B-Instruct 38.8% Zero-shot huggingface.co Official leaderboard
46 Llama 2-7B 38.5% Zero-shot Jul 2023 huggingface.co Official leaderboard
47 Meta-Llama-2-7B-Chat 38.4% Zero-shot huggingface.co Official leaderboard
48 EuroLLM-1.7B 37.6% Zero-shot huggingface.co Official leaderboard
49 Pharia-1-LLM-7B-control-aligned 37.3% Zero-shot huggingface.co Official leaderboard
50 Pharia-1-LLM-7B-control 36.7% Zero-shot huggingface.co Official leaderboard
51 Bloomz-7b1 32.8% Zero-shot huggingface.co Official leaderboard
52 Bloom-7b1 29.9% Zero-shot huggingface.co Official leaderboard
#ModelScoreSettingDateLinkSource
1 Gemma 7B 56.7% Zero-shot Feb 2024 huggingface.co Official leaderboard
2 Mixtral 8x7B 56.2% Zero-shot Dec 2023 huggingface.co Official leaderboard
3 Qwen2-7B 54.2% Few-shot Jun 2024 huggingface.co Official leaderboard
4 Qwen2-7B-Instruct 52.8% Few-shot Jun 2024 huggingface.co Official leaderboard
5 Llama 3.1-8B 52.3% Zero-shot Jul 2024 huggingface.co Official leaderboard
6 Llama 3-8B 50.6% Zero-shot Apr 2024 huggingface.co Official leaderboard
7 Mistral-7B-Instruct-v0.2 48.1% Zero-shot Dec 2023 huggingface.co Official leaderboard
8 Mistral 7B v0.1 46.9% Zero-shot Sep 2023 huggingface.co Official leaderboard
9 Mistral 7B v0.3 46.2% Zero-shot Sep 2023 huggingface.co Official leaderboard
10 Gemma 1.1 7B IT 42% Zero-shot Apr 2024 huggingface.co Official leaderboard
11 Mistral-7B-Instruct-v0.1 41.9% Zero-shot Sep 2023 huggingface.co Official leaderboard
12 phi-3-mini 3.8B 41.5% Few-shot Apr 2024 huggingface.co Official leaderboard
13 Llama 2-7B 38.5% Zero-shot Jul 2023 huggingface.co Official leaderboard

Rankings by part

We read 5 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as ARC, GSM8K and HellaSwag. Mixtral 8x7B leads 3 of them, and 2 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.

ARC

13 models we track have a score here.

  1. 1 Mixtral 8x7B ↑1 60.8%
  2. 2 Gemma 7B ↓1 60.7%
  3. 3 Llama 3.1-8B ↑2 55.4%
  4. 4 Llama 3-8B ↑2 55%
  5. 5 Mistral-7B-Instruct-v0.2 ↑2 52.4%
  6. 6 Mistral 7B v0.1 ↑2 52.1%
  7. 7 Mistral 7B v0.3 ↑2 51.3%
  8. 8 Qwen2-7B-Instruct ↓4 50.6%
  9. 9 Qwen2-7B ↓6 48.1%
  10. 10 Llama 2-7B ↑3 44.7%

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Gemma 7B
Few-shot Zero-shot
56.7%
Mixtral 8x7B
Few-shot
56.2%
Qwen2-7B
Zero-shot Few-shot
54.2%
Qwen2-7B-Instruct
Zero-shot Few-shot
52.8%
Llama 3.1-8B
Few-shot
52.3%
Llama 3-8B
Few-shot
50.6%
Mistral-7B-Instruct-v0.2
Few-shot
48.1%
Mistral 7B v0.1
Few-shot
46.9%
Mistral 7B v0.3
Few-shot
46.2%
Gemma 1.1 7B IT
Few-shot Zero-shot
42%
Mistral-7B-Instruct-v0.1
Few-shot Zero-shot
41.9%
phi-3-mini 3.8B
Zero-shot
41.5%

Source: Official European LLM Leaderboard (OpenGPT-X) leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Gemma 7B Zero-shot 56.7%
Mixtral 8x7B Zero-shot 56.2%
Mixtral 8x7B Few-shot 55.4%
Qwen2-7B Few-shot 54.2%
Gemma 7B Few-shot 53.5%
Qwen2-7B-Instruct Few-shot 52.8%
Llama 3.1-8B Zero-shot 52.3%
Llama 3.1-8B Few-shot 51.2%
Qwen2-7B-Instruct Zero-shot 50.7%
Llama 3-8B Zero-shot 50.6%
Llama 3-8B Few-shot 50.3%
Mistral-7B-Instruct-v0.2 Zero-shot 48.1%
Mistral-7B-Instruct-v0.2 Few-shot 47.5%
Mistral 7B v0.1 Zero-shot 46.9%
Qwen2-7B Zero-shot 46.3%
Mistral 7B v0.3 Zero-shot 46.2%
Mistral 7B v0.1 Few-shot 45.8%
Mistral 7B v0.3 Few-shot 45%
Gemma 1.1 7B IT Zero-shot 42%
Mistral-7B-Instruct-v0.1 Zero-shot 41.9%
phi-3-mini 3.8B Few-shot 41.5%
Mistral-7B-Instruct-v0.1 Few-shot 40%
phi-3-mini 3.8B Zero-shot 39.9%
Llama 2-7B Zero-shot 38.5%
Gemma 1.1 7B IT Few-shot 37.3%
Llama 2-7B Few-shot 36.2%

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Gemma 7B
Few-shot Zero-shot
56.7%
Mixtral 8x7B
Few-shot
56.2%
Qwen2-7B
Zero-shot Few-shot
54.2%
Qwen2-7B-Instruct
Zero-shot Few-shot
52.8%
Llama 3.1-8B
Few-shot
52.3%
Llama 3-8B
Few-shot
50.6%
Mistral-7B-Instruct-v0.2
Few-shot
48.1%
Mistral 7B v0.1
Few-shot
46.9%
Mistral 7B v0.3
Few-shot
46.2%
Gemma 1.1 7B IT
Few-shot Zero-shot
42%
Mistral-7B-Instruct-v0.1
Few-shot Zero-shot
41.9%
phi-3-mini 3.8B
Zero-shot
41.5%

Source: Official European LLM Leaderboard (OpenGPT-X) leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Gemma 7B Zero-shot 56.7%
Mixtral 8x7B Zero-shot 56.2%
Mixtral 8x7B Few-shot 55.4%
Qwen2-7B Few-shot 54.2%
Gemma 7B Few-shot 53.5%
Qwen2-7B-Instruct Few-shot 52.8%
Llama 3.1-8B Zero-shot 52.3%
Llama 3.1-8B Few-shot 51.2%
Qwen2-7B-Instruct Zero-shot 50.7%
Llama 3-8B Zero-shot 50.6%
Llama 3-8B Few-shot 50.3%
Mistral-7B-Instruct-v0.2 Zero-shot 48.1%
Mistral-7B-Instruct-v0.2 Few-shot 47.5%
Mistral 7B v0.1 Zero-shot 46.9%
Qwen2-7B Zero-shot 46.3%
Mistral 7B v0.3 Zero-shot 46.2%
Mistral 7B v0.1 Few-shot 45.8%
Mistral 7B v0.3 Few-shot 45%
Gemma 1.1 7B IT Zero-shot 42%
Mistral-7B-Instruct-v0.1 Zero-shot 41.9%
phi-3-mini 3.8B Few-shot 41.5%
Mistral-7B-Instruct-v0.1 Few-shot 40%
phi-3-mini 3.8B Zero-shot 39.9%
Llama 2-7B Zero-shot 38.5%
Gemma 1.1 7B IT Few-shot 37.3%
Llama 2-7B Few-shot 36.2%

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind
Own leaderboard Not tested since Oct 2024
Gemma 7B 56.7%

Source: Official European LLM Leaderboard (OpenGPT-X) leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) Gemma 7B Mistral 7B v0.1 Llama 2-7B 0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) Gemma 7B Mistral 7B v0.1

Source: Official European LLM Leaderboard (OpenGPT-X) leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Multilingual

All 545 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Claude Opus 5 95.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%

Data audit

9 of 10 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in July 2024, for Llama 3.1-8B. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 3 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (52 entries).
  • Pass Models scored 13 models we track, plus 39 other entries on its leaderboard.
  • Warning Newest score The newest came 2 years ago, in July 2024, for Llama 3.1-8B.
  • Note Frontier models None of the top 10 has a score. The strongest tested is Mixtral 8x7B (#199).
  • Pass Headroom The best is 56.7%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on huggingface.co: 26 rows for models we track and 39 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check