Benchmark Almanac

MAmmoTH2-8B-Plus

MAmmoTH2-8B-Plus places in the top three on Open Ko-LLM Leaderboard 2.

11
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
TIGER-Lab (University of Waterloo) May 2024

MAmmoTH2-8B-Plus

Open weights Small

No Epoch Capabilities Index score yet
  • Ko-IFEval (Upstage) 42.9% #6 /21
  • Open Ko-LLM Leaderboard 2 44.2% #2 /21
  • Open Ko-LLM Leaderboard 49% #4 /18

Where has it been number one?

MAmmoTH2-8B-Plus led 1 benchmark at some point, for a median of 97 days. It last lost Open Ko-LLM Leaderboard 2 to Qwen2.5-Coder-14B-Instruct on November 6th, 2024.

Source: Official leaderboards

How does it do against its rivals?

MAmmoTH2-8B-Plus beats Gemma 1.1 7B IT on 6 of the 9 benchmarks they share. Qwen2-7B does best against it, ahead on 6 of 9.

  1. Gemma 1.1 7B IT 6 ahead, 3 behind Compare
  2. Qwen1.5-14B-Chat 5 ahead, 4 behind Compare
  3. Yi-1.5-9B-Chat 5 ahead, 4 behind Compare
  4. Qwen1.5-14B 4 ahead, 5 behind Compare
  5. Qwen2-7B 3 ahead, 6 behind Compare

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
Open Ko-LLM Leaderboard 2 44.2% #2 of 21 Official leaderboard
Open Ko-LLM Leaderboard 49% #4 of 18 Official leaderboard
Ko-IFEval (Upstage) 42.9% #6 of 21 Official leaderboard
Open Chinese LLM Leaderboard 56.4 #18 of 35 Official leaderboard
HeQ 63.2% #51 of 88 Official leaderboard
Open Hebrew LLM Leaderboard 45.3 #52 of 88 Official leaderboard
MMLU-Pro 43.4% #152 of 212 Official leaderboard
Open Portuguese LLM Leaderboard 55.7 #128 of 175 Official leaderboard
BLUEX 30.3% #141 of 177 Official leaderboard

Professional work

BenchmarkScoreRankSource
MMLU Medical 63.9% #7 of 16 Official leaderboard

Other

BenchmarkScoreRankSource
Open LLM Leaderboard v1 67.5 #23 of 76 Official leaderboard

Who measured it?

Independent runs cover all 11 of its benchmarks.

  • An independent run only 11
Official leaderboards scored it on 11.

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards