Benchmark Almanac

SmolLM2 1.7B Instruct

SmolLM2 1.7B Instruct.

Hugging Face Released October 31st, 2024 Family SmolLM2 The lab's model card Compare with others
13
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Hugging Face Oct 2024

SmolLM2 1.7B Instruct

Open weights Small

No Epoch Capabilities Index score yet
  • Angry Tweets 15.7% #141 /149
  • FoQA 27.5% #69 /91
  • GermanQuAD 43.3% #64 /88

How does it do against its rivals?

SmolLM2 1.7B Instruct beats Gemma 3 1B on 7 of the 10 benchmarks they share. DeepSeek-R1-Distill-Qwen-14B does best against it, ahead on 5 of 8.

  1. Gemma 3 1B 7 ahead, 3 behind Compare
  2. Gemma 3 1B PT 6 ahead, 3 behind Compare
  3. DeepSeek-R1-Distill-Qwen-7B 5 ahead, 6 behind Compare
  4. Qwen3-8B 4 ahead, 5 behind Compare
  5. Qwen3-1.7B 4 ahead, 6 behind Compare
  6. DeepSeek-R1-Distill-Qwen-14B 3 ahead, 5 behind Compare

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
EuroEval Faroese Grammatical Error Correction 0.67% #39 of 71 Official leaderboard
EusTrivia 32.2% #15 of 24 Official leaderboard
EusReading 27% #16 of 24 Official leaderboard
EusExams 28.1% #17 of 24 Official leaderboard
GermanQuAD 43.3% #64 of 88 Official leaderboard
Open CoT Leaderboard 4.48% #24 of 32 Official leaderboard
FoQA 27.5% #69 of 91 Official leaderboard
Open Japanese LLM Leaderboard 0.31 #48 of 60 Official leaderboard
Biblically Informed Bot Learning Evaluation 30.4% #15 of 18 Official leaderboard
Dutch Book Reviews Dataset 74.1% #89 of 105 Official leaderboard
FoSent 5.78% #135 of 147 Official leaderboard
Angry Tweets 15.7% #141 of 149 Official leaderboard
LMArena (Elo) 1,115 #331 of 345 Official leaderboard

Who measured it?

Independent runs cover all 13 of its benchmarks.

  • An independent run only 13
Official leaderboards scored it on 13.

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards