Open Japanese LLM Leaderboard
Models perform Japanese language tasks drawn from more than 20 datasets.
LLM-jp and Hugging Face publish the board together.
- Made by
- LLM-jp and Hugging Face
- Released
- November 2024
- Kind
- Knowledge in many languages
- Level
- University
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- All public
- Measures
- Capability
- Subject
- Languages and literature
- Language
- Japanese
- Interaction
- Single turn
- Items from
- Another benchmark
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
-
QwQ-32B #137 on the capability index
- Last model added
- May 2025
- Scores refreshed
- November 20th, 2026
deep-analysis-research/Flux-Japanese-Qwen2.5-32B-Instruct-V1.0 tops the leaderboard with 0.74 QwQ-32B-Preview leads with 0.67 among models released by May 2025
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is QwQ-32B-Preview, at 0.67. This view shows only models we track, each at its best.
The newest model with a score here came out in May 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 0.74 deep-analysis-research/Flux-Japanese-Qwen2.5-32B-Instruct-V1.0
- 0.71 deep-analysis-research/D2IL-Japanese-Qwen2.5-32B-Instruct-v0.1
- 0.70 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V4-70B
- 0.70 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V5-70B
- 0.70 nitky/Llama-3.1-SuperSwallow-70B-Instruct-v0.1
- 0.70 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V4-70B
- 0.69 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V4-70B
- 0.69 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V2-70B
- 0.69 Saxo/Linkbricks-Horizon-AI-Japanese-Avdvanced-V11-70B
- 0.69 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V3-70B
- 0.68 Saxo/Linkbricks-Horizon-AI-Japanese-Avdvanced-V10-70B
- 0.68 allenai/Llama-3.1-Tulu-3-70B
- 0.68 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V5-70B
- 0.68 yellowtown/Japanese-Qwen2.5-14B-Instruct-V1
- 0.68 FINGU-AI/FINGU-2.5-instruct-32B-v1
- 0.68 karakuri-ai/karakuri-lm-32b-thinking-2501-exp
- 0.68 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-70B
- 0.68 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V2-70B
- 0.67 FINGU-AI/FINGU-2.5-instruct-70B
- 0.67 nitky/RoguePlanet-DeepSeek-R1-Qwen-32B-RP
- 0.67 Saxo/Linkbricks-Horizon-AI-Japanese-Avdvanced-V9-70B
- 0.67 JungZoona/T3Q-qwen2.5-32b-v1.2-e2
- 0.67
QwQ-32B-Preview
- 0.67 cyberagent/Llama-3.1-70B-Japanese-Instruct-2407
- 0.67 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V1-70B
- 0.67 huihui-ai/QwQ-32B-Coder-Fusion-9010
- 0.67 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V12-70B
- 0.67 maldv/Awqward2.5-32B-Instruct
- 0.67 Saxo/Linkbricks-Horizon-AI-Japanese-Base-32B
- 0.67 Triangle104/DeepSky-T100
- 0.67 NovaSky-AI/Sky-T1-32B-Flash
- 0.66 meta-llama/Llama-3.1-70B-Instruct
- 0.66 rinna/qwen2.5-bakeneko-32b-instruct-v2
- 0.66 Sao10K/32B-Qwen2.5-Kunou-v1
- 0.66
Llama-3.1-Nemotron-70B-Instruct
- 0.66 NovaSky-AI/Sky-T1-32B-Preview
- 0.66 spow12/ChatWaifu_32B_reasoning
- 0.66 simplescaling/s1-32B
- 0.66 maldv/Qwenstein2.5-32B-Instruct
- 0.66 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V7-70B
- 0.66 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V6-70B
- 0.66 deepcogito/cogito-v1-preview-qwen-32B
- 0.66 shisa-ai/shisa-v2-qwen2.5-32b
- 0.66 zetasepic/Qwen2.5-32B-Instruct-abliterated-v2
- 0.66 arcee-ai/Virtuoso-Medium-v2
- 0.66 Sakalti/ultiima-32B
- 0.66 rombodawg/Rombos-LLM-V2.5-Qwen-32b
- 0.66 prithivMLmods/Galactic-Qwen-14B-Exp2
- 0.66 deep-analysis-research/test-qwen2.5-32b
- 0.66 Qwen/Qwen2.5-32B-Instruct
- 0.66 TIGER-Lab/Qwen2.5-32B-Instruct-CFT
- 0.66 Sakalti/ultiima-52B
- 0.65 Sorawiz/Qwen2.5-14B-Instinct
- 0.65 hotmailuser/RombosBeagle-v2beta-MGS-32B
- 0.65 AXCXEPT/Llama-3.1-70B-EZO-1.1-it
- 0.65 mergekit-community/mergekit-model_stock-odyqbix
- 0.65 Saxo/Linkbricks-Horizon-AI-Korean-Advanced-70B
- 0.65 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V5-70B
- 0.65 llm-jp/llm-jp-3.1-13b-instruct4
- 0.65 Sorawiz/Qwen2.5-14B-Instinct-Talk
- 0.65 Sakalti/Saka-14B
- 0.65 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V3-70B
- 0.65 mergekit-community/mergekit-model_stock-lkrglxd
- 0.65 prithivMLmods/Sombrero-Opus-14B-Elite6
- 0.65 mergekit-community/mergekit-model_stock-qndyhny
- 0.65 Sorawiz/Qwen2.5-14B-Instinct-RP
- 0.65 mergekit-community/mergekit-ties-dvjxpbu
- 0.65 suayptalha/Lamarckvergence-14B
- 0.65 mergekit-community/mergekit-ties-knfxihf
- 0.65 mergekit-community/mergekit-della-efwskwi
- 0.65 prithivMLmods/Messier-Opus-14B-Elite7
- 0.65 mergekit-community/mergekit-model_stock-fuvlumz
- 0.65 rinna/llama-3-youko-70b-instruct
- 0.65 mergekit-community/mergekit-slerp-ryfxivm
- 0.65 mergekit-community/mergekit-model_stock-rmxmzvo
- 0.65 mergekit-community/mergekit-model_stock-jlodpmg
- 0.65 sometimesanotion/Lamarck-14B-v0.7
- 0.65 CultriX/Qwen2.5-14B-Hyperionv4
- 0.65 prithivMLmods/Equuleus-Opus-14B-Exp
- 0.65 sthenno/tempesthenno-nuslerp-0124
- 0.65 sometimesanotion/Qwenvergence-14B-v12-Prose-DS
- 0.65 sometimesanotion/Lamarck-14B-v0.7-Fusion
- 0.65 sthenno-com/miscii-14b-0218
- 0.65 mergekit-community/mergekit-model_stock-dmnvnmc
- 0.65 Sao10K/14B-Qwen2.5-Freya-x1
- 0.65 sometimesanotion/Qwenvergence-14B-v13-Prose-DS
- 0.65 mergekit-community/mergekit-model_stock-jwxwwmz
- 0.65 sthenno/tempesthenno-icy-0130
- 0.65 Sakalti/ultiima-14B-v0.4
- 0.65 qihoo360/Light-R1-32B
- 0.65 sthenno/tempesthenno-sft-0314-stage1-ckpt50
- 0.65 CultriX/Qwen2.5-14B-Ultimav2
- 0.65 JungZoona/T3Q-qwen2.5-14b-v1.2-e2
- 0.65 mergekit-community/mergekit-model_stock-batsnqf
- 0.65 Sakalti/ultiima-14B-v0.2
- 0.65 sometimesanotion/Lamarck-14B-v0.6
- 0.65 Cran-May/tempmotacilla-cinerea-0308
- 0.65 hotmailuser/QwenSlerp2-14B
- 0.65 sometimesanotion/Qwenvergence-14B-v11
- 0.65 mergekit-community/mergekit-model_stock-skvbpno
- 0.65 sthenno/tempesthenno-ppo-ckpt40
- 0.65 maldv/Qwentile2.5-32B-Instruct
- 0.65 sthenno-com/miscii-14b-1225
- 0.65
Qwen2.5-14B-Instruct-1M
- 0.64 deepcogito/cogito-v1-preview-qwen-14B
- 0.64 aixonlab/Zara-14b-v1.2
- 0.64 Nohobby/Q2.5-Qwetiapin-32B
- 0.64 mergekit-community/mergekit-model_stock-zelysxr
- 0.64 jpacifico/Chocolatine-2-14B-Instruct-v2.0b3
- 0.64 jpacifico/Chocolatine-2-14B-Instruct-v2.0.3
- 0.64 JungZoona/T3Q-qwen2.5-14b-v1.0-e3
- 0.64 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V3-70B
- 0.64 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V2-70B
- 0.64 sometimesanotion/Qwenvergence-14B-v10
- 0.64 fluently-lm/FluentlyLM-Prinum
- 0.64 prithivMLmods/Alpha-UMa-Qwen-14B
- 0.64 Sao10K/14B-Qwen2.5-Kunou-v1
- 0.64 sometimesanotion/Qwen2.5-14B-Vimarckoso-v3
- 0.64
Qwen2.5-14B
- 0.64 arcee-ai/Virtuoso-Small
- 0.64 soob3123/amoral-cogito-14b
- 0.64 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V4-70B
- 0.64 arcee-ai/Virtuoso-Small-v2
- 0.64 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V1-32B
- 0.64 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V1-70B
- 0.64 rubenroy/Zurich-14B-GCv2-5m
- 0.64 mergekit-community/mergekit-model_stock-grrwjhl
- 0.64 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V8-70B
- 0.64 Ttimofeyka/Tissint-14B-v1.2-128k-RP
- 0.63
Cogito v1 Preview 70B
- 0.63 Saxo/Linkbricks-Horizon-AI-Avengers-V6-32B
- 0.63 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V7-70B
- 0.63 nicoboss/DeepSeek-R1-Distill-Qwen-32B-Uncensored
- 0.63 Saxo/Linkbricks-Horizon-AI-Avengers-V3-32B
- 0.63 SicariusSicariiStuff/Impish_QWEN_14B-1M
- 0.63 Sakalti/oxyge1-33B
- 0.63 Sakalti/Oxyge1-33B
- 0.63 hotmailuser/Deeepseek-QwenSlerp4-32B
- 0.63 TeamDelta/Re-ultiima-14B
- 0.63 YOYO-AI/Qwen2.5-14B-YOYO-Average
- 0.63 Saxo/Linkbricks-Horizon-AI-Avengers-V1-32B
- 0.63 CultriX/Qwen2.5-14B-HyperSeekv5
- 0.63 Qwen/Qwen2.5-14B-Instruct
- 0.63 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V3-70B
- 0.63 Sakalti/ultiima-14B
- 0.63
Llama 3-70B
- 0.63 netease-youdao/Confucius-o1-14B
- 0.63 tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4
- 0.63 nitky/EZO-QwQ-32B-Preview
- 0.63 meta-llama/Meta-Llama-3-70B-Instruct
- 0.62 DataPilot/SKYDRIVE-32B-v0.1
- 0.62 tanliboy/lambda-qwen2.5-14b-dpo-test
- 0.62 deep-analysis-research/test-qwen2.5-14b-wo-system-v1
- 0.62 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V6-70B
- 0.62 AXCXEPT/EZO-Qwen2.5-32B-Instruct
- 0.62 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V1-70B
- 0.62 nitky/Llama-3.3-FakeSwallow-70B-Instruct-v0.1
- 0.62 Saxo/Linkbricks-Horizon-AI-Avengers-V2-32B
- 0.62 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.3
- 0.62 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.1
- 0.62 OpenBuddy/openbuddy-qwq-32b-v24.1-200k
- 0.62
Qwen2.5-Coder-32B-Instruct
- 0.62 DataPilot/Arrival-32B-Instruct-v0.5
- 0.62 nitky/Llama-3.3-SuperSwallow-70B-Instruct-v0.1
- 0.62 rombodawg/Rombos-Coder-V2.5-Qwen-32b
- 0.62 wanlige/li-14b-v0.4-slerp0.1
- 0.62 nbeerbower/DeepSeek-R1-Qwen-lorablated-32B
- 0.61 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V5-70B
- 0.61
Llama 3.3 70B Instruct
- 0.61 Qwen/Qwen2-57B-A14B
- 0.61 shisa-ai/shisa-v2-mistral-small-24b
- 0.61 nitky/RoguePlanet-DeepSeek-R1-Qwen-32B
- 0.61 Saxo/Linkbricks-Horizon-AI-Avengers-V4-32B
- 0.61 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V8-70B
- 0.61 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V2-70B
- 0.61 Saxo/Linkbricks-Horizon-AI-Llama-3.3-Japanese-70B-sft-dpo-base
- 0.61 Saxo/Linkbricks-Horizon-AI-Avengers-V5-32B
- 0.61 Steelskull/L3.3-MS-Nevoria-70b
- 0.61 RDson/WomboCombo-R1-Coder-14B-Preview
- 0.61 Saxo/Linkbricks-Horizon-AI-Japanese-Base-70B
- 0.61
Qwen2.5-32B
- 0.61 shisa-ai/shisa-v2-llama3.3-70b
- 0.60 wanlige/li-14b-v0.4
- 0.60 Ba2han/QwQenSeek-coder
- 0.60 TeetouchQQ/model_mergev2
- 0.60 Azure99/Blossom-V6-32B
- 0.60 abeja/ABEJA-Qwen2.5-32b-Japanese-v1.0
- 0.60 cyberagent/DeepSeek-R1-Distill-Qwen-32B-Japanese
- 0.60 abeja/ABEJA-Qwen2.5-32b-Japanese-v0.1
- 0.60 FINGU-AI/FINGU-2.5-instruct-32B
- 0.60 FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview
- 0.60 DreadPoor/Irix-12B-Model_Stock
- 0.59 RDson/CoderO1-DeepSeekR1-Coder-32B-Preview
- 0.59 spow12/ChatWaifu_v1.4
- 0.59
Llama 3.1-70B
- 0.59 Gen-Verse/ReasonFlux-F1
- 0.59
DeepSeek-R1-Distill-Qwen-32B
- 0.59 Saxo/Linkbricks-Horizon-AI-Llama-3.3-Japanese-70B-Advanced
- 0.59 NaniDAO/deepseek-r1-qwen-2.5-32B-ablated
- 0.59 llm-jp/llm-jp-3.1-13b
- 0.59 prithivMLmods/Qwen2.5-14B-DeepSeek-R1-1M
- 0.59 spow12/ChatWaifu_12B_v2.0
- 0.59
QwQ-32B
- 0.59 Saxo/Linkbricks-Horizon-AI-Korean-Superb-22B
- 0.58 Elizezen/Himeyuri-v0.1-12B
- 0.58
DeepSeek-R1-Distill-Llama-70B
- 0.58 mistralai/Mistral-Nemo-Instruct-2407
- 0.58 google/gemma-2-27b-it
- 0.58 anthracite-org/magnum-v2.5-12b-kto
- 0.58 Groq/Llama-3-Groq-70B-Tool-Use
- 0.58 karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- 0.58 llm-jp/llm-jp-3.1-1.8b-instruct4
- 0.58 Saxo/Linkbricks-Horizon-AI-Korean-Superb-27B
- 0.58 Azure99/Blossom-V6-14B
- 0.58 huihui-ai/DeepSeek-R1-Distill-Qwen-32B-abliterated
- 0.58 shisa-ai/shisa-v2-mistral-nemo-12b
- 0.58 spow12/ChatWaifu_v2.0_22B
- 0.57 CohereForAI/aya-23-35B
- 0.57 Zhihu-ai/Zhi-writing-dsr1-14b
- 0.57 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V1-22B
- 0.57 agentica-org/DeepCoder-14B-Preview
- 0.57 nvidia/AceReason-Nemotron-14B
- 0.57
DeepSeek-R1-Distill-Qwen-14B
- 0.57 qingy2024/NaturalLM
- 0.57 qihoo360/TinyR1-32B-Preview
- 0.57
Qwen1.5-32B
- 0.57 rubenroy/Geneva-12B-GCv2-5m
- 0.56 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.2
- 0.56 cyberagent/DeepSeek-R1-Distill-Qwen-14B-Japanese
- 0.56 Gen-Verse/ReasonFlux-F1-14B
- 0.56 cyberagent/calm3-22b-chat
- 0.56 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V1-27B
- 0.56 elyza/Llama-3-ELYZA-JP-8B
- 0.55 shisa-ai/shisa-v2-qwen2.5-7b
- 0.55 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3
- 0.55 shisa-ai/shisa-v1-llama3-70b
- 0.55 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.1
- 0.55 gmonsoon/SahabatAI-Lion-9B-TIES-v1
- 0.55 Qwen/Qwen2-57B-A14B-Instruct
- 0.55 tokyotech-llm/Llama-3.1-Swallow-8B-v0.1
- 0.55 Saxo/Linkbricks-Horizon-AI-Superb-27B
- 0.55 karakuri-ai/karakuri-lm-8x7b-instruct-v0.1
- 0.55 augmxnt/shisa-gamma-7b-v1
- 0.55 llm-jp/llm-jp-3-13b-instruct
- 0.55 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V7-70B
- 0.55 Reka Flash 3.1
- 0.54
Qwen1.5-14B
- 0.54 nbeerbower/Mistral-Nemo-Prism-12B-v2
- 0.54 rinna/llama-3-youko-8b-instruct
- 0.54 Kendamarron/Qwen2.5-7B-o1-ja-v0.1
- 0.54 lightblue/qwen2.5-7B-instruct-simpo
- 0.54 cyberagent/Mistral-Nemo-Japanese-Instruct-2408
- 0.54 meta-llama/Llama-2-70b-hf
- 0.54 Qwen/Qwen2.5-7B-Instruct-1M
- 0.54 AIDC-AI/Marco-o1
- 0.54
Qwen2.5-7B
- 0.54 tokyotech-llm/Llama-3.1-Swallow-8B-v0.2
- 0.54 Qwen2.5-Coder-14B-Instruct
- 0.54 rombodawg/Rombos-Coder-V2.5-Qwen-14b
- 0.54 Sakalti/deneb-v1-7b
- 0.54 Sakalti/Saka-7.6B
- 0.54 Saxo/Linkbricks-Horizon-AI-Korean-Advanced-56B
- 0.54 suayptalha/HomerCreativeAnvita-Mix-Qw7B
- 0.54 Qwen/Qwen1.5-32B-Chat
- 0.54 shisa-ai/shisa-v2-llama3.1-8b
- 0.54 DataPilot/Llama3.1-ArrowSE-v0.4
- 0.53 Azure99/Blossom-V6-7B
- 0.53 Nurture-intelligence/kEy-llama3.1-8b-v0.1
- 0.53 meta-llama/Llama-3.1-8B-Instruct
- 0.53 tokyo-electron-device-ai/llama3-tedllm-8b-v0-annealing
- 0.53 qihoo360/Light-R1-14B-DS
- 0.53 tokyotech-llm/Gemma-2-Llama-Swallow-27b-it-v0.1
- 0.53 Qwen/Qwen2.5-7B-Instruct
- 0.53 Qwen2-VL-7B-Instruct
- 0.53 AXCXEPT/EZO-Humanities-9B-gemma-2-it
- 0.53 fblgit/pancho-v1-qw25-3B-UNAMGS
- 0.53 AXCXEPT/EZO-Common-9B-gemma-2-it
- 0.53 deepcogito/cogito-v1-preview-llama-8B
- 0.53 Sakalti/sakalinear-1-1.0
- 0.52 AIDC-AI/Marco-LLM-GLO
- 0.52 rubenroy/Zurich-7B-GCv2-5m
- 0.52 Manual-Dataset-Creation-Project/Matsu-7B
- 0.52
Qwen2.5-3B
- 0.52
Qwen2-7B
- 0.52 google/gemma-2-9b-it
- 0.52 MaziyarPanahi/calme-3.2-instruct-3b
- 0.52 karakuri-ai/karakuri-lm-70b-chat-v0.1
- 0.52 Elizezen/Kudryavka-8B-alpha
- 0.52
Qwen2-7B-Instruct
- 0.52
Aya Expanse 8B
- 0.51 huihui-ai/DeepSeek-R1-Distill-Qwen-14B-abliterated-v2
- 0.51 Sakalti/Neptuno-Alpha
- 0.51 microsoft/Phi-3.5-mini-instruct
- 0.51
Aya Expanse 32B
- 0.51 Casual-Autopsy/Llama-3-VNTL-Yollisa-8B
- 0.51 huihui-ai/aya-expanse-32b-abliterated
- 0.51 Sakalti/Lunar-4B
- 0.50 ryota39/Tora-12B
- 0.50 SakanaAI/EvoLLM-JP-A-v1-7B
- 0.50
Qwen1.5-14B-Chat
- 0.50 meta-llama/Meta-Llama-3-8B-Instruct
- 0.49 llm-jp/llm-jp-3.1-1.8b
- 0.49 Granite-3.1-8B-Instruct
- 0.49
Llama 3.1-8B
- 0.49 Granite 3.2 8B Instruct
- 0.49
Llama 3-8B
- 0.49 llm-jp/llm-jp-3-13b
- 0.49 Qwen/Qwen2.5-Coder-7B-Instruct
- 0.48 CohereForAI/aya-23-8B
- 0.48 Qwen/Qwen2.5-3B-Instruct
- 0.48 meta-llama/Llama-2-70b-chat-hf
- 0.47
Qwen1.5-7B
- 0.47 tiiuae/Falcon3-10B-base
- 0.46 Rakuten/RakutenAI-7B-chat
- 0.46 deepcogito/cogito-v1-preview-llama-3B
- 0.46 llm-jp/llm-jp-3-3.7b-instruct
- 0.46 augmxnt/shisa-7b-v1
- 0.46 Granite-3.1-8B-Base
- 0.46 Elizezen/Phos-7B
- 0.45
Qwen2.5-1.5B
- 0.45
Qwen1.5-7B-Chat
- 0.45 Sakalti/Saba1.5-Pro
- 0.45 Sakalti/Saba2-Preview
- 0.45 Sakalti/tara-3.8B
- 0.45 Sakalti/Tara-3.8B-v1.1
- 0.45 jpacifico/Chocolatine-3B-Instruct-DPO-Revised
- 0.45 cognitivecomputations/Dolphin3.0-R1-Mistral-24B
- 0.45 Rakuten/RakutenAI-2.0-mini-instruct
- 0.45 rinna/gemma-2-baku-2b-it
- 0.45 Sakalti/Saka-1.5B
- 0.44 Qwen/Qwen2.5-1.5B-Instruct
- 0.44
Falcon3-10B-Instruct
- 0.44 Sakalti/SJT-1.5B-Alpha
- 0.44 Qwen/Qwen1.5-MoE-A2.7B
- 0.43 mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1
- 0.43 Granite-3.1-2B-Instruct
- 0.43 TIGER-Lab/Qwen2.5-Math-7B-CFT
- 0.43 llm-jp/llm-jp-3-3.7b-instruct3
- 0.43 llm-jp/llm-jp-3-3.7b-instruct2
- 0.43 Saxo/Linkbricks-Horizon-AI-Korean-LLAMA3blend-8x8b
- 0.43 meta-llama/Llama-2-13b-hf
- 0.42
Llama 3.2 3B
- 0.42 tiiuae/Falcon3-7B-base
- 0.41 deepseek-ai/DeepSeek-R1-Distill-Llama-8B
- 0.41 meta-llama/Llama-3.2-3B-Instruct
- 0.41 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V6-70B
- 0.41 Granite-3.1-2B-Base
- 0.41
Qwen2-1.5B-Instruct
- 0.41 meta-llama/Llama-2-13b-chat-hf
- 0.41
Gemma 2 2B IT
- 0.40 Qwen/Qwen1.5-MoE-A2.7B-Chat
- 0.40 Elizezen/SniffyOtter-7B
- 0.40 Elizezen/Antler-7B
- 0.40 rubenroy/Zurich-1.5B-GCv2-5m
- 0.40
Qwen1.5-4B-Chat
- 0.39
DeepSeek-R1-Distill-Qwen-7B
- 0.39 llm-jp/llm-jp-3-1.8b-instruct
- 0.39 Qwen/Qwen1.5-4B
- 0.39
Qwen2-1.5B
- 0.39 mobiuslabsgmbh/DeepSeek-R1-ReDistill-Qwen-7B-v1.1
- 0.38 Gen-Verse/ReasonFlux-F1-7B
- 0.38 meta-llama/Llama-2-7b-chat-hf
- 0.38 Granite-3.1-3B-A800M-Instruct
- 0.38 Sakalti/SJT-2.4B-Alpha
- 0.38 weblab-GENIAC/Tanuki-8B-dpo-v1.0
- 0.38 llm-jp/llm-jp-3-3.7b
- 0.38 Saxo/Linkbricks-Horizon-AI-Superb-70B
- 0.37 meta-llama/Llama-2-7b-hf
- 0.37 Granite-3.1-3B-A800M-Base
- 0.35
Falcon3-7B-Instruct
- 0.35 cyberagent/calm2-7b-chat
- 0.34 Elizezen/Hameln-japanese-mistral-7B
- 0.34 llm-jp/llm-jp-3-1.8b
- 0.33 prithivMLmods/Qwen2.5-32B-DeepSeek-R1-Instruct
- 0.33 llm-jp/llm-jp-3-980m-instruct2
- 0.33 llm-jp/llm-jp-3-980m-instruct3
- 0.32 Qwen/Qwen2.5-Math-7B-Instruct
- 0.31
Qwen2.5-0.5B
- 0.31
SmolLM2 1.7B Instruct
- 0.31 Qwen/Qwen2.5-0.5B-Instruct
- 0.31 Sakalti/CodeSaka-7.6B
- 0.31 meta-llama/Llama-3.2-1B-Instruct
- 0.30 tiiuae/Falcon3-3B-base
- 0.30 Qwen/Qwen1.5-1.8B
- 0.30 Qwen/Qwen1.5-1.8B-Chat
- 0.29
Qwen2-0.5B
- 0.29
Qwen2-0.5B-Instruct
- 0.29 Granite-3.1-1B-A400M-Instruct
- 0.29 llm-jp/llm-jp-3-980m
- 0.29 yamatazen/Twilight-SCE-12B-v2
- 0.29
Gemma 2B
- 0.29
Gemma 1.1 2B IT
- 0.29 sail/Sailor2-1B-Chat
- 0.28
Falcon3-3B-Instruct
- 0.28
Llama 3.2 1B
- 0.27 google/gemma-2b-it
- 0.27 Granite-3.1-1B-A400M-Base
- 0.26 agentica-org/DeepScaleR-1.5B-Preview
- 0.26 agentica-org/DeepCoder-1.5B-Preview
- 0.26
DeepSeek-R1-Distill-Qwen-1.5B
- 0.25 llm-jp/llm-jp-3-440m-instruct2
- 0.25 llm-jp/llm-jp-3-440m-instruct3
- 0.24
Falcon3-1B-Instruct
- 0.24 llm-jp/llm-jp-3-440m
- 0.24 rsh345/llama3.1-8b-swallow-openmath-dare_ties-d5w5_d5w5
- 0.23 nvidia/OpenMath2-Llama3.1-8B
- 0.23 Qwen/Qwen1.5-0.5B
- 0.21 tiiuae/Falcon3-1B-base
- 0.21 Sakalti/MathSaka-7.6B
- 0.21 CjangCjengh/GaLLM-multi-14B-v0.1
- 0.19 CjangCjengh/GaLLM-14B-v0.2
- 0.19 Qwen/Qwen1.5-0.5B-Chat
- 0.18 llm-jp/llm-jp-3-150m
- 0.16 HuggingFaceTB/SmolLM2-135M-Instruct
- 0.15 google/gemma-7b-it
- 0.15
Gemma 1.1 7B IT
- 0.04
Gemma 7B
- 0.03 Sakalti/Saka-7.2B
- 1 deep-analysis-research/Flux-Japanese-Qwen2.5-32B-Instruct-V1.0 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.74
- 2 deep-analysis-research/D2IL-Japanese-Qwen2.5-32B-Instruct-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.71
- 3 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V4-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.70
- 4 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V5-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.70
- 5 nitky/Llama-3.1-SuperSwallow-70B-Instruct-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.70
- 6 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V4-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.70
- 7 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V4-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.69
- 8 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V2-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.69
- 9 Saxo/Linkbricks-Horizon-AI-Japanese-Avdvanced-V11-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.69
- 10 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V3-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.69
- 11 Saxo/Linkbricks-Horizon-AI-Japanese-Avdvanced-V10-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.68
- 12 allenai/Llama-3.1-Tulu-3-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.68
- 13 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V5-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.68
- 14 yellowtown/Japanese-Qwen2.5-14B-Instruct-V1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.68
- 15 FINGU-AI/FINGU-2.5-instruct-32B-v1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.68
- 16 karakuri-ai/karakuri-lm-32b-thinking-2501-exp 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.68
- 17 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.68
- 18 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V2-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.68
- 19 FINGU-AI/FINGU-2.5-instruct-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 20 nitky/RoguePlanet-DeepSeek-R1-Qwen-32B-RP 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 21 Saxo/Linkbricks-Horizon-AI-Japanese-Avdvanced-V9-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 22 JungZoona/T3Q-qwen2.5-32b-v1.2-e2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 23
QwQ-32B-Preview 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 24 cyberagent/Llama-3.1-70B-Japanese-Instruct-2407 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 25 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V1-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 26 huihui-ai/QwQ-32B-Coder-Fusion-9010 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 27 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V12-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 28 maldv/Awqward2.5-32B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 29 Saxo/Linkbricks-Horizon-AI-Japanese-Base-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 30 Triangle104/DeepSky-T100 4-shot / float16 / main / vLLM v0.6.3.post1 0.67
- 31 NovaSky-AI/Sky-T1-32B-Flash 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 32 meta-llama/Llama-3.1-70B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 33 rinna/qwen2.5-bakeneko-32b-instruct-v2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 34 Sao10K/32B-Qwen2.5-Kunou-v1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 35
Llama-3.1-Nemotron-70B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 36 NovaSky-AI/Sky-T1-32B-Preview 4-shot / float16 / main / vLLM v0.6.3.post1 0.66
- 37 spow12/ChatWaifu_32B_reasoning 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 38 simplescaling/s1-32B 4-shot / float32 / main / vLLM v0.6.3.post1 0.66
- 39 maldv/Qwenstein2.5-32B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 40 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V7-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 41 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V6-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 42 deepcogito/cogito-v1-preview-qwen-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 43 shisa-ai/shisa-v2-qwen2.5-32b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 44 zetasepic/Qwen2.5-32B-Instruct-abliterated-v2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 45 arcee-ai/Virtuoso-Medium-v2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 46 Sakalti/ultiima-32B 4-shot / float16 / main / vLLM v0.6.3.post1 0.66
- 47 rombodawg/Rombos-LLM-V2.5-Qwen-32b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 48 prithivMLmods/Galactic-Qwen-14B-Exp2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 49 deep-analysis-research/test-qwen2.5-32b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 50 Qwen/Qwen2.5-32B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 51 TIGER-Lab/Qwen2.5-32B-Instruct-CFT 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 52 Sakalti/ultiima-52B 4-shot / float16 / main / vLLM v0.6.3.post1 0.66
- 53 Sorawiz/Qwen2.5-14B-Instinct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 54 hotmailuser/RombosBeagle-v2beta-MGS-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 55 AXCXEPT/Llama-3.1-70B-EZO-1.1-it 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 56 mergekit-community/mergekit-model_stock-odyqbix 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 57 Saxo/Linkbricks-Horizon-AI-Korean-Advanced-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 58 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V5-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 59 llm-jp/llm-jp-3.1-13b-instruct4 0-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 60 Sorawiz/Qwen2.5-14B-Instinct-Talk 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 61 Sakalti/Saka-14B 4-shot / float16 / main / vLLM v0.6.3.post1 0.65
- 62 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V3-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 63 mergekit-community/mergekit-model_stock-lkrglxd 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 64 prithivMLmods/Sombrero-Opus-14B-Elite6 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 65 mergekit-community/mergekit-model_stock-qndyhny 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 66 Sorawiz/Qwen2.5-14B-Instinct-RP 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 67 mergekit-community/mergekit-ties-dvjxpbu 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 68 suayptalha/Lamarckvergence-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 69 mergekit-community/mergekit-ties-knfxihf 4-shot / float16 / main / vLLM v0.6.3.post1 0.65
- 70 mergekit-community/mergekit-della-efwskwi 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 71 prithivMLmods/Messier-Opus-14B-Elite7 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 72 mergekit-community/mergekit-model_stock-fuvlumz 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 73 rinna/llama-3-youko-70b-instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 74 mergekit-community/mergekit-slerp-ryfxivm 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 75 mergekit-community/mergekit-model_stock-rmxmzvo 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 76 mergekit-community/mergekit-model_stock-jlodpmg 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 77 sometimesanotion/Lamarck-14B-v0.7 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 78 CultriX/Qwen2.5-14B-Hyperionv4 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 79 prithivMLmods/Equuleus-Opus-14B-Exp 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 80 sthenno/tempesthenno-nuslerp-0124 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 81 sometimesanotion/Qwenvergence-14B-v12-Prose-DS 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 82 sometimesanotion/Lamarck-14B-v0.7-Fusion 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 83 sthenno-com/miscii-14b-0218 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 84 mergekit-community/mergekit-model_stock-dmnvnmc 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 85 Sao10K/14B-Qwen2.5-Freya-x1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 86 sometimesanotion/Qwenvergence-14B-v13-Prose-DS 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 87 mergekit-community/mergekit-model_stock-jwxwwmz 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 88 sthenno/tempesthenno-icy-0130 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 89 Sakalti/ultiima-14B-v0.4 4-shot / float16 / main / vLLM v0.6.3.post1 0.65
- 90 qihoo360/Light-R1-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 91 sthenno/tempesthenno-sft-0314-stage1-ckpt50 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 92 CultriX/Qwen2.5-14B-Ultimav2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 93 JungZoona/T3Q-qwen2.5-14b-v1.2-e2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 94 mergekit-community/mergekit-model_stock-batsnqf 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 95 Sakalti/ultiima-14B-v0.2 4-shot / float16 / main / vLLM v0.6.3.post1 0.65
- 96 sometimesanotion/Lamarck-14B-v0.6 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 97 Cran-May/tempmotacilla-cinerea-0308 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 98 hotmailuser/QwenSlerp2-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 99 sometimesanotion/Qwenvergence-14B-v11 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 100 mergekit-community/mergekit-model_stock-skvbpno 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 101 sthenno/tempesthenno-ppo-ckpt40 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 102 maldv/Qwentile2.5-32B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 103 sthenno-com/miscii-14b-1225 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 104
Qwen2.5-14B-Instruct-1M 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 105 deepcogito/cogito-v1-preview-qwen-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 106 aixonlab/Zara-14b-v1.2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 107 Nohobby/Q2.5-Qwetiapin-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 108 mergekit-community/mergekit-model_stock-zelysxr 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 109 jpacifico/Chocolatine-2-14B-Instruct-v2.0b3 4-shot / float16 / main / vLLM v0.6.3.post1 0.64
- 110 jpacifico/Chocolatine-2-14B-Instruct-v2.0.3 4-shot / float16 / main / vLLM v0.6.3.post1 0.64
- 111 JungZoona/T3Q-qwen2.5-14b-v1.0-e3 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 112 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V3-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 113 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V2-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 114 sometimesanotion/Qwenvergence-14B-v10 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 115 fluently-lm/FluentlyLM-Prinum 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 116 prithivMLmods/Alpha-UMa-Qwen-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 117 Sao10K/14B-Qwen2.5-Kunou-v1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 118 sometimesanotion/Qwen2.5-14B-Vimarckoso-v3 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 119
Qwen2.5-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 120 arcee-ai/Virtuoso-Small 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 121 soob3123/amoral-cogito-14b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 122 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V4-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 123 arcee-ai/Virtuoso-Small-v2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 124 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V1-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 125 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V1-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 126 rubenroy/Zurich-14B-GCv2-5m 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 127 mergekit-community/mergekit-model_stock-grrwjhl 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 128 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V8-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 129 Ttimofeyka/Tissint-14B-v1.2-128k-RP 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 130
Cogito v1 Preview 70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 131 Saxo/Linkbricks-Horizon-AI-Avengers-V6-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 132 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V7-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 133 nicoboss/DeepSeek-R1-Distill-Qwen-32B-Uncensored 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 134 Saxo/Linkbricks-Horizon-AI-Avengers-V3-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 135 SicariusSicariiStuff/Impish_QWEN_14B-1M 4-shot / float16 / main / vLLM v0.6.3.post1 0.63
- 136 Sakalti/oxyge1-33B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 137 Sakalti/Oxyge1-33B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 138 hotmailuser/Deeepseek-QwenSlerp4-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 139 TeamDelta/Re-ultiima-14B 4-shot / float16 / main / vLLM v0.6.3.post1 0.63
- 140 YOYO-AI/Qwen2.5-14B-YOYO-Average 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 141 Saxo/Linkbricks-Horizon-AI-Avengers-V1-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 142 CultriX/Qwen2.5-14B-HyperSeekv5 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 143 Qwen/Qwen2.5-14B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 144 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V3-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 145 Sakalti/ultiima-14B 4-shot / float16 / main / vLLM v0.6.3.post1 0.63
- 146
Llama 3-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 147 netease-youdao/Confucius-o1-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 148 tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 149 nitky/EZO-QwQ-32B-Preview 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 150 meta-llama/Meta-Llama-3-70B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 151 DataPilot/SKYDRIVE-32B-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 152 tanliboy/lambda-qwen2.5-14b-dpo-test 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 153 deep-analysis-research/test-qwen2.5-14b-wo-system-v1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 154 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V6-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 155 AXCXEPT/EZO-Qwen2.5-32B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 156 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V1-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 157 nitky/Llama-3.3-FakeSwallow-70B-Instruct-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 158 Saxo/Linkbricks-Horizon-AI-Avengers-V2-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 159 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.3 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 160 tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 161 OpenBuddy/openbuddy-qwq-32b-v24.1-200k 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 162
Qwen2.5-Coder-32B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 163 DataPilot/Arrival-32B-Instruct-v0.5 4-shot / float16 / main / vLLM v0.6.3.post1 0.62
- 164 nitky/Llama-3.3-SuperSwallow-70B-Instruct-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 165 rombodawg/Rombos-Coder-V2.5-Qwen-32b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 166 wanlige/li-14b-v0.4-slerp0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 167 nbeerbower/DeepSeek-R1-Qwen-lorablated-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 168 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V5-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 169
Llama 3.3 70B Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 170 Qwen/Qwen2-57B-A14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 171 shisa-ai/shisa-v2-mistral-small-24b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 172 nitky/RoguePlanet-DeepSeek-R1-Qwen-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 173 Saxo/Linkbricks-Horizon-AI-Avengers-V4-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 174 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V8-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 175 Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V2-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 176 Saxo/Linkbricks-Horizon-AI-Llama-3.3-Japanese-70B-sft-dpo-base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 177 Saxo/Linkbricks-Horizon-AI-Avengers-V5-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 178 Steelskull/L3.3-MS-Nevoria-70b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 179 RDson/WomboCombo-R1-Coder-14B-Preview 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 180 Saxo/Linkbricks-Horizon-AI-Japanese-Base-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 181
Qwen2.5-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 182 shisa-ai/shisa-v2-llama3.3-70b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 183 wanlige/li-14b-v0.4 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.60
- 184 Ba2han/QwQenSeek-coder 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.60
- 185 TeetouchQQ/model_mergev2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.60
- 186 Azure99/Blossom-V6-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.60
- 187 abeja/ABEJA-Qwen2.5-32b-Japanese-v1.0 4-shot / float16 / main / vLLM v0.6.3.post1 0.60
- 188 cyberagent/DeepSeek-R1-Distill-Qwen-32B-Japanese 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.60
- 189 abeja/ABEJA-Qwen2.5-32b-Japanese-v0.1 4-shot / float16 / main / vLLM v0.6.3.post1 0.60
- 190 FINGU-AI/FINGU-2.5-instruct-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.60
- 191 FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.60
- 192 DreadPoor/Irix-12B-Model_Stock 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.60
- 193 RDson/CoderO1-DeepSeekR1-Coder-32B-Preview 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 194 spow12/ChatWaifu_v1.4 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 195
Llama 3.1-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 196 Gen-Verse/ReasonFlux-F1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 197
DeepSeek-R1-Distill-Qwen-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 198 Saxo/Linkbricks-Horizon-AI-Llama-3.3-Japanese-70B-Advanced 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 199 NaniDAO/deepseek-r1-qwen-2.5-32B-ablated 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 200 llm-jp/llm-jp-3.1-13b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 201 prithivMLmods/Qwen2.5-14B-DeepSeek-R1-1M 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 202 spow12/ChatWaifu_12B_v2.0 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 203
QwQ-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 204 Saxo/Linkbricks-Horizon-AI-Korean-Superb-22B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 205 Elizezen/Himeyuri-v0.1-12B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 206
DeepSeek-R1-Distill-Llama-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 207 mistralai/Mistral-Nemo-Instruct-2407 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 208 google/gemma-2-27b-it 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 209 anthracite-org/magnum-v2.5-12b-kto 4-shot / float16 / main / vLLM v0.6.3.post1 0.58
- 210 Groq/Llama-3-Groq-70B-Tool-Use 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 211 karakuri-ai/karakuri-lm-8x7b-chat-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 212 llm-jp/llm-jp-3.1-1.8b-instruct4 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 213 Saxo/Linkbricks-Horizon-AI-Korean-Superb-27B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 214 Azure99/Blossom-V6-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 215 huihui-ai/DeepSeek-R1-Distill-Qwen-32B-abliterated 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 216 shisa-ai/shisa-v2-mistral-nemo-12b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 217 spow12/ChatWaifu_v2.0_22B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 218 CohereForAI/aya-23-35B 4-shot / float16 / main / vLLM v0.6.3.post1 0.57
- 219 Zhihu-ai/Zhi-writing-dsr1-14b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 220 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V1-22B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 221 agentica-org/DeepCoder-14B-Preview 4-shot / float32 / main / vLLM v0.6.3.post1 0.57
- 222 nvidia/AceReason-Nemotron-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 223
DeepSeek-R1-Distill-Qwen-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 224 qingy2024/NaturalLM 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 225 qihoo360/TinyR1-32B-Preview 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 226
Qwen1.5-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 227 rubenroy/Geneva-12B-GCv2-5m 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 228 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.56
- 229 cyberagent/DeepSeek-R1-Distill-Qwen-14B-Japanese 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.56
- 230 Gen-Verse/ReasonFlux-F1-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.56
- 231 cyberagent/calm3-22b-chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.56
- 232 Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V1-27B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.56
- 233 elyza/Llama-3-ELYZA-JP-8B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.56
- 234 shisa-ai/shisa-v2-qwen2.5-7b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 235 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 236 shisa-ai/shisa-v1-llama3-70b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 237 tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 238 gmonsoon/SahabatAI-Lion-9B-TIES-v1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 239 Qwen/Qwen2-57B-A14B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 240 tokyotech-llm/Llama-3.1-Swallow-8B-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 241 Saxo/Linkbricks-Horizon-AI-Superb-27B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 242 karakuri-ai/karakuri-lm-8x7b-instruct-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 243 augmxnt/shisa-gamma-7b-v1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 244 llm-jp/llm-jp-3-13b-instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 245 Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V7-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 246 Reka Flash 3.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 247
Qwen1.5-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 248 nbeerbower/Mistral-Nemo-Prism-12B-v2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 249 rinna/llama-3-youko-8b-instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 250 Kendamarron/Qwen2.5-7B-o1-ja-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 251 lightblue/qwen2.5-7B-instruct-simpo 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 252 cyberagent/Mistral-Nemo-Japanese-Instruct-2408 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 253 meta-llama/Llama-2-70b-hf 4-shot / float16 / main / vLLM v0.6.3.post1 0.54
- 254 Qwen/Qwen2.5-7B-Instruct-1M 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 255 AIDC-AI/Marco-o1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 256
Qwen2.5-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 257 tokyotech-llm/Llama-3.1-Swallow-8B-v0.2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 258 Qwen2.5-Coder-14B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 259 rombodawg/Rombos-Coder-V2.5-Qwen-14b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 260 Sakalti/deneb-v1-7b 4-shot / float16 / main / vLLM v0.6.3.post1 0.54
- 261 Sakalti/Saka-7.6B 4-shot / float16 / main / vLLM v0.6.3.post1 0.54
- 262 Saxo/Linkbricks-Horizon-AI-Korean-Advanced-56B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 263 suayptalha/HomerCreativeAnvita-Mix-Qw7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 264 Qwen/Qwen1.5-32B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 265 shisa-ai/shisa-v2-llama3.1-8b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 266 DataPilot/Llama3.1-ArrowSE-v0.4 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 267 Azure99/Blossom-V6-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 268 Nurture-intelligence/kEy-llama3.1-8b-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 269 meta-llama/Llama-3.1-8B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 270 tokyo-electron-device-ai/llama3-tedllm-8b-v0-annealing 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 271 qihoo360/Light-R1-14B-DS 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 272 tokyotech-llm/Gemma-2-Llama-Swallow-27b-it-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 273 Qwen/Qwen2.5-7B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 274 Qwen2-VL-7B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 275 AXCXEPT/EZO-Humanities-9B-gemma-2-it 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 276 fblgit/pancho-v1-qw25-3B-UNAMGS 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 277 AXCXEPT/EZO-Common-9B-gemma-2-it 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 278 deepcogito/cogito-v1-preview-llama-8B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 279 Sakalti/sakalinear-1-1.0 4-shot / float16 / main / vLLM v0.6.3.post1 0.53
- 280 AIDC-AI/Marco-LLM-GLO 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 281 rubenroy/Zurich-7B-GCv2-5m 4-shot / float16 / main / vLLM v0.6.3.post1 0.52
- 282 Manual-Dataset-Creation-Project/Matsu-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 283
Qwen2.5-3B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 284
Qwen2-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 285 google/gemma-2-9b-it 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 286 MaziyarPanahi/calme-3.2-instruct-3b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 287 karakuri-ai/karakuri-lm-70b-chat-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 288 Elizezen/Kudryavka-8B-alpha 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 289
Qwen2-7B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 290
Aya Expanse 8B 4-shot / float16 / main / vLLM v0.6.3.post1 0.52
- 291 huihui-ai/DeepSeek-R1-Distill-Qwen-14B-abliterated-v2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.51
- 292 Sakalti/Neptuno-Alpha 4-shot / float16 / main / vLLM v0.6.3.post1 0.51
- 293 microsoft/Phi-3.5-mini-instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.51
- 294
Aya Expanse 32B 4-shot / float16 / main / vLLM v0.6.3.post1 0.51
- 295 Casual-Autopsy/Llama-3-VNTL-Yollisa-8B 4-shot / float32 / main / vLLM v0.6.3.post1 0.51
- 296 huihui-ai/aya-expanse-32b-abliterated 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.51
- 297 Sakalti/Lunar-4B 4-shot / float16 / main / vLLM v0.6.3.post1 0.51
- 298 ryota39/Tora-12B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.50
- 299 SakanaAI/EvoLLM-JP-A-v1-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.50
- 300
Qwen1.5-14B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.50
- 301 meta-llama/Meta-Llama-3-8B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.50
- 302 llm-jp/llm-jp-3.1-1.8b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 303 Granite-3.1-8B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 304
Llama 3.1-8B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 305 Granite 3.2 8B Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 306
Llama 3-8B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 307 llm-jp/llm-jp-3-13b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 308 Qwen/Qwen2.5-Coder-7B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 309 CohereForAI/aya-23-8B 4-shot / float16 / main / vLLM v0.6.3.post1 0.48
- 310 Qwen/Qwen2.5-3B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.48
- 311 meta-llama/Llama-2-70b-chat-hf 4-shot / float16 / main / vLLM v0.6.3.post1 0.48
- 312
Qwen1.5-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.47
- 313 tiiuae/Falcon3-10B-base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.47
- 314 Rakuten/RakutenAI-7B-chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.46
- 315 deepcogito/cogito-v1-preview-llama-3B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.46
- 316 llm-jp/llm-jp-3-3.7b-instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.46
- 317 augmxnt/shisa-7b-v1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.46
- 318 Granite-3.1-8B-Base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.46
- 319 Elizezen/Phos-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.46
- 320
Qwen2.5-1.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.45
- 321
Qwen1.5-7B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.45
- 322 Sakalti/Saba1.5-Pro 4-shot / float16 / main / vLLM v0.6.3.post1 0.45
- 323 Sakalti/Saba2-Preview 4-shot / float16 / main / vLLM v0.6.3.post1 0.45
- 324 Sakalti/tara-3.8B 4-shot / float16 / main / vLLM v0.6.3.post1 0.45
- 325 Sakalti/Tara-3.8B-v1.1 4-shot / float16 / main / vLLM v0.6.3.post1 0.45
- 326 jpacifico/Chocolatine-3B-Instruct-DPO-Revised 4-shot / float16 / main / vLLM v0.6.3.post1 0.45
- 327 cognitivecomputations/Dolphin3.0-R1-Mistral-24B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.45
- 328 Rakuten/RakutenAI-2.0-mini-instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.45
- 329 rinna/gemma-2-baku-2b-it 4-shot / float16 / main / vLLM v0.6.3.post1 0.45
- 330 Sakalti/Saka-1.5B 4-shot / float16 / main / vLLM v0.6.3.post1 0.45
- 331 Qwen/Qwen2.5-1.5B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.44
- 332
Falcon3-10B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.44
- 333 Sakalti/SJT-1.5B-Alpha 4-shot / float16 / main / vLLM v0.6.3.post1 0.44
- 334 Qwen/Qwen1.5-MoE-A2.7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.44
- 335 mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.43
- 336 Granite-3.1-2B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.43
- 337 TIGER-Lab/Qwen2.5-Math-7B-CFT 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.43
- 338 llm-jp/llm-jp-3-3.7b-instruct3 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.43
- 339 llm-jp/llm-jp-3-3.7b-instruct2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.43
- 340 Saxo/Linkbricks-Horizon-AI-Korean-LLAMA3blend-8x8b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.43
- 341 meta-llama/Llama-2-13b-hf 4-shot / float16 / main / vLLM v0.6.3.post1 0.43
- 342
Llama 3.2 3B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.42
- 343 tiiuae/Falcon3-7B-base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.42
- 344 deepseek-ai/DeepSeek-R1-Distill-Llama-8B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.41
- 345 meta-llama/Llama-3.2-3B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.41
- 346 Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V6-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.41
- 347 Granite-3.1-2B-Base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.41
- 348
Qwen2-1.5B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.41
- 349 meta-llama/Llama-2-13b-chat-hf 4-shot / float16 / main / vLLM v0.6.3.post1 0.41
- 350
Gemma 2 2B IT 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.41
- 351 Qwen/Qwen1.5-MoE-A2.7B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.40
- 352 Elizezen/SniffyOtter-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.40
- 353 Elizezen/Antler-7B 4-shot / float16 / main / vLLM v0.6.3.post1 0.40
- 354 rubenroy/Zurich-1.5B-GCv2-5m 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.40
- 355
Qwen1.5-4B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.40
- 356
DeepSeek-R1-Distill-Qwen-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.39
- 357 llm-jp/llm-jp-3-1.8b-instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.39
- 358 Qwen/Qwen1.5-4B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.39
- 359
Qwen2-1.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.39
- 360 mobiuslabsgmbh/DeepSeek-R1-ReDistill-Qwen-7B-v1.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.39
- 361 Gen-Verse/ReasonFlux-F1-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.38
- 362 meta-llama/Llama-2-7b-chat-hf 4-shot / float16 / main / vLLM v0.6.3.post1 0.38
- 363 Granite-3.1-3B-A800M-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.38
- 364 Sakalti/SJT-2.4B-Alpha 4-shot / float16 / main / vLLM v0.6.3.post1 0.38
- 365 weblab-GENIAC/Tanuki-8B-dpo-v1.0 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.38
- 366 llm-jp/llm-jp-3-3.7b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.38
- 367 Saxo/Linkbricks-Horizon-AI-Superb-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.38
- 368 meta-llama/Llama-2-7b-hf 4-shot / float16 / main / vLLM v0.6.3.post1 0.37
- 369 Granite-3.1-3B-A800M-Base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.37
- 370
Falcon3-7B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.35
- 371 cyberagent/calm2-7b-chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.35
- 372 Elizezen/Hameln-japanese-mistral-7B 4-shot / float16 / main / vLLM v0.6.3.post1 0.34
- 373 llm-jp/llm-jp-3-1.8b 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.34
- 374 prithivMLmods/Qwen2.5-32B-DeepSeek-R1-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.33
- 375 llm-jp/llm-jp-3-980m-instruct2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.33
- 376 llm-jp/llm-jp-3-980m-instruct3 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.33
- 377 Qwen/Qwen2.5-Math-7B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.32
- 378
Qwen2.5-0.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.31
- 379
SmolLM2 1.7B Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.31
- 380 Qwen/Qwen2.5-0.5B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.31
- 381 Sakalti/CodeSaka-7.6B 4-shot / float16 / main / vLLM v0.6.3.post1 0.31
- 382 meta-llama/Llama-3.2-1B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.31
- 383 tiiuae/Falcon3-3B-base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.30
- 384 Qwen/Qwen1.5-1.8B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.30
- 385 Qwen/Qwen1.5-1.8B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.30
- 386
Qwen2-0.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 387
Qwen2-0.5B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 388 Granite-3.1-1B-A400M-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 389 llm-jp/llm-jp-3-980m 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 390 yamatazen/Twilight-SCE-12B-v2 0-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 391
Gemma 2B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 392
Gemma 1.1 2B IT 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 393 sail/Sailor2-1B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 394
Falcon3-3B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.28
- 395
Llama 3.2 1B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.28
- 396 google/gemma-2b-it 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.27
- 397 Granite-3.1-1B-A400M-Base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.27
- 398 agentica-org/DeepScaleR-1.5B-Preview 4-shot / float32 / main / vLLM v0.6.3.post1 0.26
- 399 agentica-org/DeepCoder-1.5B-Preview 4-shot / float32 / main / vLLM v0.6.3.post1 0.26
- 400
DeepSeek-R1-Distill-Qwen-1.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.26
- 401 llm-jp/llm-jp-3-440m-instruct2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.25
- 402 llm-jp/llm-jp-3-440m-instruct3 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.25
- 403
Falcon3-1B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.24
- 404 llm-jp/llm-jp-3-440m 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.24
- 405 rsh345/llama3.1-8b-swallow-openmath-dare_ties-d5w5_d5w5 4-shot / float16 / main / vLLM v0.6.3.post1 0.24
- 406 nvidia/OpenMath2-Llama3.1-8B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.23
- 407 Qwen/Qwen1.5-0.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.23
- 408 tiiuae/Falcon3-1B-base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.21
- 409 Sakalti/MathSaka-7.6B 4-shot / float16 / main / vLLM v0.6.3.post1 0.21
- 410 CjangCjengh/GaLLM-multi-14B-v0.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.21
- 411 CjangCjengh/GaLLM-14B-v0.2 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.19
- 412 Qwen/Qwen1.5-0.5B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.19
- 413 llm-jp/llm-jp-3-150m 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.18
- 414 HuggingFaceTB/SmolLM2-135M-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.16
- 415 google/gemma-7b-it 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.15
- 416
Gemma 1.1 7B IT 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.15
- 417
Gemma 7B 0-shot / bfloat16 / main / vLLM v0.6.3.post1 0.04
- 418 Sakalti/Saka-7.2B 0-shot / float16 / main / vLLM v0.6.3.post1 0.03
Source: Official Open Japanese LLM Leaderboard leaderboard
- 0.67
QwQ-32B-Preview
- 0.66
Llama-3.1-Nemotron-70B-Instruct
- 0.65
Qwen2.5-14B-Instruct-1M
- 0.64
Qwen2.5-14B
- 0.63
Cogito v1 Preview 70B
- 0.63
Llama 3-70B
- 0.62
Qwen2.5-Coder-32B-Instruct
- 0.61
Llama 3.3 70B Instruct
- 0.61
Qwen2.5-32B
- 0.59
Llama 3.1-70B
- 0.59
DeepSeek-R1-Distill-Qwen-32B
- 0.59
QwQ-32B
- 0.58
DeepSeek-R1-Distill-Llama-70B
- 0.57
DeepSeek-R1-Distill-Qwen-14B
- 0.57
Qwen1.5-32B
- 0.55 Reka Flash 3.1
- 0.54
Qwen1.5-14B
- 0.54
Qwen2.5-7B
- 0.54 Qwen2.5-Coder-14B-Instruct
- 0.53 Qwen2-VL-7B-Instruct
- 0.52
Qwen2.5-3B
- 0.52
Qwen2-7B
- 0.52
Qwen2-7B-Instruct
- 0.52
Aya Expanse 8B
- 0.51
Aya Expanse 32B
- 0.50
Qwen1.5-14B-Chat
- 0.49 Granite-3.1-8B-Instruct
- 0.49
Llama 3.1-8B
- 0.49 Granite 3.2 8B Instruct
- 0.49
Llama 3-8B
- 0.47
Qwen1.5-7B
- 0.46 Granite-3.1-8B-Base
- 0.45
Qwen2.5-1.5B
- 0.45
Qwen1.5-7B-Chat
- 0.44
Falcon3-10B-Instruct
- 0.43 Granite-3.1-2B-Instruct
- 0.42
Llama 3.2 3B
- 0.41 Granite-3.1-2B-Base
- 0.41
Qwen2-1.5B-Instruct
- 0.41
Gemma 2 2B IT
- 0.40
Qwen1.5-4B-Chat
- 0.39
DeepSeek-R1-Distill-Qwen-7B
- 0.39
Qwen2-1.5B
- 0.38 Granite-3.1-3B-A800M-Instruct
- 0.37 Granite-3.1-3B-A800M-Base
- 0.35
Falcon3-7B-Instruct
- 0.31
Qwen2.5-0.5B
- 0.31
SmolLM2 1.7B Instruct
- 0.29
Qwen2-0.5B
- 0.29
Qwen2-0.5B-Instruct
- 0.29 Granite-3.1-1B-A400M-Instruct
- 0.29
Gemma 2B
- 0.29
Gemma 1.1 2B IT
- 0.28
Falcon3-3B-Instruct
- 0.28
Llama 3.2 1B
- 0.27 Granite-3.1-1B-A400M-Base
- 0.26
DeepSeek-R1-Distill-Qwen-1.5B
- 0.24
Falcon3-1B-Instruct
- 0.15
Gemma 1.1 7B IT
- 0.04
Gemma 7B
- 1
QwQ-32B-Preview 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.67
- 2
Llama-3.1-Nemotron-70B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.66
- 3
Qwen2.5-14B-Instruct-1M 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.65
- 4
Qwen2.5-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.64
- 5
Cogito v1 Preview 70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 6
Llama 3-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.63
- 7
Qwen2.5-Coder-32B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.62
- 8
Llama 3.3 70B Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 9
Qwen2.5-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.61
- 10
Llama 3.1-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 11
DeepSeek-R1-Distill-Qwen-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 12
QwQ-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.59
- 13
DeepSeek-R1-Distill-Llama-70B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.58
- 14
DeepSeek-R1-Distill-Qwen-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 15
Qwen1.5-32B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.57
- 16 Reka Flash 3.1 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.55
- 17
Qwen1.5-14B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 18
Qwen2.5-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 19 Qwen2.5-Coder-14B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.54
- 20 Qwen2-VL-7B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.53
- 21
Qwen2.5-3B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 22
Qwen2-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 23
Qwen2-7B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.52
- 24
Aya Expanse 8B 4-shot / float16 / main / vLLM v0.6.3.post1 0.52
- 25
Aya Expanse 32B 4-shot / float16 / main / vLLM v0.6.3.post1 0.51
- 26
Qwen1.5-14B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.50
- 27 Granite-3.1-8B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 28
Llama 3.1-8B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 29 Granite 3.2 8B Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 30
Llama 3-8B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.49
- 31
Qwen1.5-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.47
- 32 Granite-3.1-8B-Base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.46
- 33
Qwen2.5-1.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.45
- 34
Qwen1.5-7B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.45
- 35
Falcon3-10B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.44
- 36 Granite-3.1-2B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.43
- 37
Llama 3.2 3B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.42
- 38 Granite-3.1-2B-Base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.41
- 39
Qwen2-1.5B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.41
- 40
Gemma 2 2B IT 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.41
- 41
Qwen1.5-4B-Chat 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.40
- 42
DeepSeek-R1-Distill-Qwen-7B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.39
- 43
Qwen2-1.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.39
- 44 Granite-3.1-3B-A800M-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.38
- 45 Granite-3.1-3B-A800M-Base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.37
- 46
Falcon3-7B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.35
- 47
Qwen2.5-0.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.31
- 48
SmolLM2 1.7B Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.31
- 49
Qwen2-0.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 50
Qwen2-0.5B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 51 Granite-3.1-1B-A400M-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 52
Gemma 2B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 53
Gemma 1.1 2B IT 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.29
- 54
Falcon3-3B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.28
- 55
Llama 3.2 1B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.28
- 56 Granite-3.1-1B-A400M-Base 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.27
- 57
DeepSeek-R1-Distill-Qwen-1.5B 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.26
- 58
Falcon3-1B-Instruct 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.24
- 59
Gemma 1.1 7B IT 4-shot / bfloat16 / main / vLLM v0.6.3.post1 0.15
- 60
Gemma 7B 0-shot / bfloat16 / main / vLLM v0.6.3.post1 0.04
Source: Official Open Japanese LLM Leaderboard leaderboard
| # | Entry | Score | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | deep-analysis-research/Flux-Japanese-Qwen2.5-32B-Instruct-V1.0 32.8B | 0.74 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 2 | deep-analysis-research/D2IL-Japanese-Qwen2.5-32B-Instruct-v0.1 32.8B | 0.71 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 3 | Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V4-70B 70.6B | 0.70 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 4 | Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V5-70B 70.6B | 0.70 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 5 | nitky/Llama-3.1-SuperSwallow-70B-Instruct-v0.1 70.6B | 0.70 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 6 | Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V4-70B 70.6B | 0.70 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 7 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V4-70B 70.6B | 0.69 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 8 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V2-70B 70.6B | 0.69 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 9 | Saxo/Linkbricks-Horizon-AI-Japanese-Avdvanced-V11-70B 70.6B | 0.69 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 10 | Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V3-70B 70.6B | 0.69 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 11 | Saxo/Linkbricks-Horizon-AI-Japanese-Avdvanced-V10-70B 70.6B | 0.68 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 12 | allenai/Llama-3.1-Tulu-3-70B 70.6B | 0.68 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 13 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V5-70B 70.6B | 0.68 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 14 | yellowtown/Japanese-Qwen2.5-14B-Instruct-V1 14.8B | 0.68 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 15 | FINGU-AI/FINGU-2.5-instruct-32B-v1 32.8B | 0.68 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 16 | karakuri-ai/karakuri-lm-32b-thinking-2501-exp 32.8B | 0.68 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 17 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-70B 70.6B | 0.68 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 18 | Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V2-70B 70.6B | 0.68 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 19 | FINGU-AI/FINGU-2.5-instruct-70B 70.6B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 20 | nitky/RoguePlanet-DeepSeek-R1-Qwen-32B-RP 32.8B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 21 | Saxo/Linkbricks-Horizon-AI-Japanese-Avdvanced-V9-70B 70.6B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 22 | JungZoona/T3Q-qwen2.5-32b-v1.2-e2 32.8B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 23 | | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Nov 2024 | huggingface.co | Official leaderboard |
| 24 | cyberagent/Llama-3.1-70B-Japanese-Instruct-2407 70.6B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 25 | Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V1-70B 70.6B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 26 | huihui-ai/QwQ-32B-Coder-Fusion-9010 32.8B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 27 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V12-70B 70.6B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 28 | maldv/Awqward2.5-32B-Instruct 32.8B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 29 | Saxo/Linkbricks-Horizon-AI-Japanese-Base-32B 32.8B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 30 | Triangle104/DeepSky-T100 32.8B | 0.67 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 31 | NovaSky-AI/Sky-T1-32B-Flash 32.8B | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 32 | meta-llama/Llama-3.1-70B-Instruct 70.6B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 33 | rinna/qwen2.5-bakeneko-32b-instruct-v2 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 34 | Sao10K/32B-Qwen2.5-Kunou-v1 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 35 | | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Oct 2024 | huggingface.co | Official leaderboard |
| 36 | NovaSky-AI/Sky-T1-32B-Preview 32.8B | 0.66 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 37 | spow12/ChatWaifu_32B_reasoning 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 38 | simplescaling/s1-32B 32.8B | 0.66 | 4-shot / float32 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 39 | maldv/Qwenstein2.5-32B-Instruct 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 40 | Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V7-70B 70.6B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 41 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V6-70B 70.6B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 42 | deepcogito/cogito-v1-preview-qwen-32B 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 43 | shisa-ai/shisa-v2-qwen2.5-32b 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 44 | zetasepic/Qwen2.5-32B-Instruct-abliterated-v2 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 45 | arcee-ai/Virtuoso-Medium-v2 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 46 | Sakalti/ultiima-32B 32.8B | 0.66 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 47 | rombodawg/Rombos-LLM-V2.5-Qwen-32b 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 48 | prithivMLmods/Galactic-Qwen-14B-Exp2 14.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 49 | deep-analysis-research/test-qwen2.5-32b 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 50 | Qwen/Qwen2.5-32B-Instruct 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 51 | TIGER-Lab/Qwen2.5-32B-Instruct-CFT 32.8B | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 52 | Sakalti/ultiima-52B 52.3B | 0.66 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 53 | Sorawiz/Qwen2.5-14B-Instinct 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 54 | hotmailuser/RombosBeagle-v2beta-MGS-32B 32.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 55 | AXCXEPT/Llama-3.1-70B-EZO-1.1-it 70.6B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 56 | mergekit-community/mergekit-model_stock-odyqbix 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 57 | Saxo/Linkbricks-Horizon-AI-Korean-Advanced-70B 70.6B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 58 | Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V5-70B 70.6B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 59 | llm-jp/llm-jp-3.1-13b-instruct4 13.7B | 0.65 | 0-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 60 | Sorawiz/Qwen2.5-14B-Instinct-Talk 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 61 | Sakalti/Saka-14B 14.8B | 0.65 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 62 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V3-70B 70.6B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 63 | mergekit-community/mergekit-model_stock-lkrglxd 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 64 | prithivMLmods/Sombrero-Opus-14B-Elite6 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 65 | mergekit-community/mergekit-model_stock-qndyhny 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 66 | Sorawiz/Qwen2.5-14B-Instinct-RP 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 67 | mergekit-community/mergekit-ties-dvjxpbu 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 68 | suayptalha/Lamarckvergence-14B 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 69 | mergekit-community/mergekit-ties-knfxihf 14.8B | 0.65 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 70 | mergekit-community/mergekit-della-efwskwi 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 71 | prithivMLmods/Messier-Opus-14B-Elite7 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 72 | mergekit-community/mergekit-model_stock-fuvlumz 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 73 | rinna/llama-3-youko-70b-instruct 70.6B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 74 | mergekit-community/mergekit-slerp-ryfxivm 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 75 | mergekit-community/mergekit-model_stock-rmxmzvo 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 76 | mergekit-community/mergekit-model_stock-jlodpmg 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 77 | sometimesanotion/Lamarck-14B-v0.7 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 78 | CultriX/Qwen2.5-14B-Hyperionv4 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 79 | prithivMLmods/Equuleus-Opus-14B-Exp 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 80 | sthenno/tempesthenno-nuslerp-0124 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 81 | sometimesanotion/Qwenvergence-14B-v12-Prose-DS 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 82 | sometimesanotion/Lamarck-14B-v0.7-Fusion 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 83 | sthenno-com/miscii-14b-0218 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 84 | mergekit-community/mergekit-model_stock-dmnvnmc 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 85 | Sao10K/14B-Qwen2.5-Freya-x1 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 86 | sometimesanotion/Qwenvergence-14B-v13-Prose-DS 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 87 | mergekit-community/mergekit-model_stock-jwxwwmz 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 88 | sthenno/tempesthenno-icy-0130 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 89 | Sakalti/ultiima-14B-v0.4 14.8B | 0.65 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 90 | qihoo360/Light-R1-32B 32.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 91 | sthenno/tempesthenno-sft-0314-stage1-ckpt50 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 92 | CultriX/Qwen2.5-14B-Ultimav2 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 93 | JungZoona/T3Q-qwen2.5-14b-v1.2-e2 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 94 | mergekit-community/mergekit-model_stock-batsnqf 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 95 | Sakalti/ultiima-14B-v0.2 14.8B | 0.65 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 96 | sometimesanotion/Lamarck-14B-v0.6 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 97 | Cran-May/tempmotacilla-cinerea-0308 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 98 | hotmailuser/QwenSlerp2-14B 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 99 | sometimesanotion/Qwenvergence-14B-v11 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 100 | mergekit-community/mergekit-model_stock-skvbpno 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 101 | sthenno/tempesthenno-ppo-ckpt40 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 102 | maldv/Qwentile2.5-32B-Instruct 32.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 103 | sthenno-com/miscii-14b-1225 14.8B | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 104 | | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 105 | deepcogito/cogito-v1-preview-qwen-14B 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 106 | aixonlab/Zara-14b-v1.2 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 107 | Nohobby/Q2.5-Qwetiapin-32B 32.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 108 | mergekit-community/mergekit-model_stock-zelysxr 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 109 | jpacifico/Chocolatine-2-14B-Instruct-v2.0b3 14.8B | 0.64 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 110 | jpacifico/Chocolatine-2-14B-Instruct-v2.0.3 14.8B | 0.64 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 111 | JungZoona/T3Q-qwen2.5-14b-v1.0-e3 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 112 | Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V3-70B 70.6B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 113 | Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V2-70B 70.6B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 114 | sometimesanotion/Qwenvergence-14B-v10 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 115 | fluently-lm/FluentlyLM-Prinum 32.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 116 | prithivMLmods/Alpha-UMa-Qwen-14B 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 117 | Sao10K/14B-Qwen2.5-Kunou-v1 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 118 | sometimesanotion/Qwen2.5-14B-Vimarckoso-v3 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 119 | | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 120 | arcee-ai/Virtuoso-Small 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 121 | soob3123/amoral-cogito-14b 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 122 | Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V4-70B 70.6B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 123 | arcee-ai/Virtuoso-Small-v2 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 124 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V1-32B 32.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 125 | Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V1-70B 70.6B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 126 | rubenroy/Zurich-14B-GCv2-5m 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 127 | mergekit-community/mergekit-model_stock-grrwjhl 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 128 | Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V8-70B 70.6B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 129 | Ttimofeyka/Tissint-14B-v1.2-128k-RP 14.8B | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 130 | | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2025 | huggingface.co | Official leaderboard |
| 131 | Saxo/Linkbricks-Horizon-AI-Avengers-V6-32B 32.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 132 | Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V7-70B 70.6B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 133 | nicoboss/DeepSeek-R1-Distill-Qwen-32B-Uncensored 32.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 134 | Saxo/Linkbricks-Horizon-AI-Avengers-V3-32B 32.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 135 | SicariusSicariiStuff/Impish_QWEN_14B-1M 14.8B | 0.63 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 136 | Sakalti/oxyge1-33B 32.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 137 | Sakalti/Oxyge1-33B 32.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 138 | hotmailuser/Deeepseek-QwenSlerp4-32B 32.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 139 | TeamDelta/Re-ultiima-14B 14.8B | 0.63 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 140 | YOYO-AI/Qwen2.5-14B-YOYO-Average 14.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 141 | Saxo/Linkbricks-Horizon-AI-Avengers-V1-32B 32.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 142 | CultriX/Qwen2.5-14B-HyperSeekv5 14.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 143 | Qwen/Qwen2.5-14B-Instruct 14.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 144 | Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V3-70B 70.6B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 145 | Sakalti/ultiima-14B 14.8B | 0.63 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 146 | | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2024 | huggingface.co | Official leaderboard |
| 147 | netease-youdao/Confucius-o1-14B 14.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 148 | tokyotech-llm/Llama-3.3-Swallow-70B-Instruct-v0.4 70.6B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 149 | nitky/EZO-QwQ-32B-Preview 32.8B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 150 | meta-llama/Meta-Llama-3-70B-Instruct 70.6B | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 151 | DataPilot/SKYDRIVE-32B-v0.1 32.8B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 152 | tanliboy/lambda-qwen2.5-14b-dpo-test 14.8B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 153 | deep-analysis-research/test-qwen2.5-14b-wo-system-v1 14.8B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 154 | Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V6-70B 70.6B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 155 | AXCXEPT/EZO-Qwen2.5-32B-Instruct 32.8B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 156 | Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V1-70B 70.6B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 157 | nitky/Llama-3.3-FakeSwallow-70B-Instruct-v0.1 70.6B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 158 | Saxo/Linkbricks-Horizon-AI-Avengers-V2-32B 32.8B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 159 | tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.3 70.6B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 160 | tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.1 70.6B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 161 | OpenBuddy/openbuddy-qwq-32b-v24.1-200k 32.8B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 162 | | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Nov 2024 | huggingface.co | Official leaderboard |
| 163 | DataPilot/Arrival-32B-Instruct-v0.5 32.8B | 0.62 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 164 | nitky/Llama-3.3-SuperSwallow-70B-Instruct-v0.1 70.6B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 165 | rombodawg/Rombos-Coder-V2.5-Qwen-32b 32.8B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 166 | wanlige/li-14b-v0.4-slerp0.1 14.8B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 167 | nbeerbower/DeepSeek-R1-Qwen-lorablated-32B 32.8B | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 168 | Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V5-70B 70.6B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 169 | | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 170 | Qwen/Qwen2-57B-A14B 57.4B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 171 | shisa-ai/shisa-v2-mistral-small-24b 23.6B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 172 | nitky/RoguePlanet-DeepSeek-R1-Qwen-32B 32.8B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 173 | Saxo/Linkbricks-Horizon-AI-Avengers-V4-32B 32.8B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 174 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V8-70B 70.6B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 175 | Saxo/Linkbricks-Horizon-AI-Japanese-Avengers-V2-70B 70.6B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 176 | Saxo/Linkbricks-Horizon-AI-Llama-3.3-Japanese-70B-sft-dpo-base 70.6B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 177 | Saxo/Linkbricks-Horizon-AI-Avengers-V5-32B 32.8B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 178 | Steelskull/L3.3-MS-Nevoria-70b 70.6B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 179 | RDson/WomboCombo-R1-Coder-14B-Preview 14.8B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 180 | Saxo/Linkbricks-Horizon-AI-Japanese-Base-70B 70.6B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 181 | | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 182 | shisa-ai/shisa-v2-llama3.3-70b 70.6B | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 183 | wanlige/li-14b-v0.4 14.8B | 0.60 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 184 | Ba2han/QwQenSeek-coder 32.8B | 0.60 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 185 | TeetouchQQ/model_mergev2 32.8B | 0.60 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 186 | Azure99/Blossom-V6-32B 32.8B | 0.60 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 187 | abeja/ABEJA-Qwen2.5-32b-Japanese-v1.0 32.8B | 0.60 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 188 | cyberagent/DeepSeek-R1-Distill-Qwen-32B-Japanese 32.8B | 0.60 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 189 | abeja/ABEJA-Qwen2.5-32b-Japanese-v0.1 32.8B | 0.60 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 190 | FINGU-AI/FINGU-2.5-instruct-32B 32.8B | 0.60 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 191 | FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview 32.8B | 0.60 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 192 | DreadPoor/Irix-12B-Model_Stock 12.3B | 0.60 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 193 | RDson/CoderO1-DeepSeekR1-Coder-32B-Preview 32.8B | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 194 | spow12/ChatWaifu_v1.4 12.3B | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 195 | | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jul 2024 | huggingface.co | Official leaderboard |
| 196 | Gen-Verse/ReasonFlux-F1 32.8B | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 197 | | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 198 | Saxo/Linkbricks-Horizon-AI-Llama-3.3-Japanese-70B-Advanced 70.6B | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 199 | NaniDAO/deepseek-r1-qwen-2.5-32B-ablated 32.8B | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 200 | llm-jp/llm-jp-3.1-13b 13.7B | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 201 | prithivMLmods/Qwen2.5-14B-DeepSeek-R1-1M 14.8B | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 202 | spow12/ChatWaifu_12B_v2.0 12.3B | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 203 | | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Mar 2025 | huggingface.co | Official leaderboard |
| 204 | Saxo/Linkbricks-Horizon-AI-Korean-Superb-22B 22.3B | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 205 | Elizezen/Himeyuri-v0.1-12B 12.3B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 206 | | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 207 | mistralai/Mistral-Nemo-Instruct-2407 12.3B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 208 | google/gemma-2-27b-it 27.2B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 209 | anthracite-org/magnum-v2.5-12b-kto 12.3B | 0.58 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 210 | Groq/Llama-3-Groq-70B-Tool-Use 70.6B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 211 | karakuri-ai/karakuri-lm-8x7b-chat-v0.1 46.7B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 212 | llm-jp/llm-jp-3.1-1.8b-instruct4 1.9B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 213 | Saxo/Linkbricks-Horizon-AI-Korean-Superb-27B 27.2B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 214 | Azure99/Blossom-V6-14B 14.8B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 215 | huihui-ai/DeepSeek-R1-Distill-Qwen-32B-abliterated 32.8B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 216 | shisa-ai/shisa-v2-mistral-nemo-12b 12.3B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 217 | spow12/ChatWaifu_v2.0_22B 22.3B | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 218 | CohereForAI/aya-23-35B 35B | 0.57 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 219 | Zhihu-ai/Zhi-writing-dsr1-14b 14.8B | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 220 | Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V1-22B 22.3B | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 221 | agentica-org/DeepCoder-14B-Preview 14.8B | 0.57 | 4-shot / float32 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 222 | nvidia/AceReason-Nemotron-14B 14.8B | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 223 | | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 224 | qingy2024/NaturalLM 12.3B | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 225 | qihoo360/TinyR1-32B-Preview 32.8B | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 226 | | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 227 | rubenroy/Geneva-12B-GCv2-5m 12.3B | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 228 | tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.2 8B | 0.56 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 229 | cyberagent/DeepSeek-R1-Distill-Qwen-14B-Japanese 14.8B | 0.56 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 230 | Gen-Verse/ReasonFlux-F1-14B 14.8B | 0.56 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 231 | cyberagent/calm3-22b-chat 22.5B | 0.56 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 232 | Saxo/Linkbricks-Horizon-AI-Japanese-Superb-V1-27B 27.2B | 0.56 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 233 | elyza/Llama-3-ELYZA-JP-8B 8B | 0.56 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 234 | shisa-ai/shisa-v2-qwen2.5-7b 7.6B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 235 | tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3 8B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 236 | shisa-ai/shisa-v1-llama3-70b 70.6B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 237 | tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.1 8B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 238 | gmonsoon/SahabatAI-Lion-9B-TIES-v1 9.2B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 239 | Qwen/Qwen2-57B-A14B-Instruct 57.4B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 240 | tokyotech-llm/Llama-3.1-Swallow-8B-v0.1 8B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 241 | Saxo/Linkbricks-Horizon-AI-Superb-27B 27.2B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 242 | karakuri-ai/karakuri-lm-8x7b-instruct-v0.1 46.7B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 243 | augmxnt/shisa-gamma-7b-v1 7.2B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 244 | llm-jp/llm-jp-3-13b-instruct 13.7B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 245 | Saxo/Linkbricks-Horizon-AI-Japanese-Advanced-V7-70B 70.6B | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 246 | Reka Flash 3.1 | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | May 2025 | huggingface.co | Official leaderboard |
| 247 | | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 248 | nbeerbower/Mistral-Nemo-Prism-12B-v2 12.3B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 249 | rinna/llama-3-youko-8b-instruct 8B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 250 | Kendamarron/Qwen2.5-7B-o1-ja-v0.1 7.6B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 251 | lightblue/qwen2.5-7B-instruct-simpo 7.6B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 252 | cyberagent/Mistral-Nemo-Japanese-Instruct-2408 12.3B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 253 | meta-llama/Llama-2-70b-hf 69B | 0.54 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 254 | Qwen/Qwen2.5-7B-Instruct-1M 7.6B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 255 | AIDC-AI/Marco-o1 7.6B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 256 | | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 257 | tokyotech-llm/Llama-3.1-Swallow-8B-v0.2 8B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 258 | Qwen2.5-Coder-14B-Instruct | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Nov 2024 | huggingface.co | Official leaderboard |
| 259 | rombodawg/Rombos-Coder-V2.5-Qwen-14b 14.8B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 260 | Sakalti/deneb-v1-7b 7.6B | 0.54 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 261 | Sakalti/Saka-7.6B 7.6B | 0.54 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 262 | Saxo/Linkbricks-Horizon-AI-Korean-Advanced-56B 46.7B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 263 | suayptalha/HomerCreativeAnvita-Mix-Qw7B 7.6B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 264 | Qwen/Qwen1.5-32B-Chat 32.5B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 265 | shisa-ai/shisa-v2-llama3.1-8b 8B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 266 | DataPilot/Llama3.1-ArrowSE-v0.4 8B | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 267 | Azure99/Blossom-V6-7B 7.6B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 268 | Nurture-intelligence/kEy-llama3.1-8b-v0.1 8B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 269 | meta-llama/Llama-3.1-8B-Instruct 8B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 270 | tokyo-electron-device-ai/llama3-tedllm-8b-v0-annealing 8.1B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 271 | qihoo360/Light-R1-14B-DS 14.8B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 272 | tokyotech-llm/Gemma-2-Llama-Swallow-27b-it-v0.1 27.2B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 273 | Qwen/Qwen2.5-7B-Instruct 7.6B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 274 | Qwen2-VL-7B-Instruct | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Aug 2024 | huggingface.co | Official leaderboard |
| 275 | AXCXEPT/EZO-Humanities-9B-gemma-2-it 9.2B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 276 | fblgit/pancho-v1-qw25-3B-UNAMGS 3.4B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 277 | AXCXEPT/EZO-Common-9B-gemma-2-it 9.2B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 278 | deepcogito/cogito-v1-preview-llama-8B 8B | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 279 | Sakalti/sakalinear-1-1.0 7.6B | 0.53 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 280 | AIDC-AI/Marco-LLM-GLO 7.6B | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 281 | rubenroy/Zurich-7B-GCv2-5m | 0.52 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 282 | Manual-Dataset-Creation-Project/Matsu-7B 7.6B | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 283 | | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 284 | | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 285 | google/gemma-2-9b-it 9.2B | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 286 | MaziyarPanahi/calme-3.2-instruct-3b 3.1B | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 287 | karakuri-ai/karakuri-lm-70b-chat-v0.1 69.2B | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 288 | Elizezen/Kudryavka-8B-alpha 8B | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 289 | | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 290 | | 0.52 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Oct 2024 | huggingface.co | Official leaderboard |
| 291 | huihui-ai/DeepSeek-R1-Distill-Qwen-14B-abliterated-v2 14.8B | 0.51 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 292 | Sakalti/Neptuno-Alpha 3.4B | 0.51 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 293 | microsoft/Phi-3.5-mini-instruct 3.8B | 0.51 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 294 | | 0.51 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Oct 2024 | huggingface.co | Official leaderboard |
| 295 | Casual-Autopsy/Llama-3-VNTL-Yollisa-8B 8B | 0.51 | 4-shot / float32 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 296 | huihui-ai/aya-expanse-32b-abliterated 32.3B | 0.51 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 297 | Sakalti/Lunar-4B 3.8B | 0.51 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 298 | ryota39/Tora-12B 12.3B | 0.50 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 299 | SakanaAI/EvoLLM-JP-A-v1-7B 7.2B | 0.50 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 300 | | 0.50 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 301 | meta-llama/Meta-Llama-3-8B-Instruct 8B | 0.50 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 302 | llm-jp/llm-jp-3.1-1.8b 1.9B | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 303 | Granite-3.1-8B-Instruct | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 304 | | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jul 2024 | huggingface.co | Official leaderboard |
| 305 | Granite 3.2 8B Instruct | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2025 | huggingface.co | Official leaderboard |
| 306 | | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2024 | huggingface.co | Official leaderboard |
| 307 | llm-jp/llm-jp-3-13b 13.7B | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 308 | Qwen/Qwen2.5-Coder-7B-Instruct 7.6B | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 309 | CohereForAI/aya-23-8B 8B | 0.48 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 310 | Qwen/Qwen2.5-3B-Instruct 3.1B | 0.48 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 311 | meta-llama/Llama-2-70b-chat-hf 69B | 0.48 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 312 | | 0.47 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 313 | tiiuae/Falcon3-10B-base 10.3B | 0.47 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 314 | Rakuten/RakutenAI-7B-chat 7.4B | 0.46 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 315 | deepcogito/cogito-v1-preview-llama-3B 3.6B | 0.46 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 316 | llm-jp/llm-jp-3-3.7b-instruct 3.8B | 0.46 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 317 | augmxnt/shisa-7b-v1 8B | 0.46 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 318 | Granite-3.1-8B-Base | 0.46 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 319 | Elizezen/Phos-7B 7.2B | 0.46 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 320 | | 0.45 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 321 | | 0.45 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 322 | Sakalti/Saba1.5-Pro 1.5B | 0.45 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 323 | Sakalti/Saba2-Preview 1.5B | 0.45 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 324 | Sakalti/tara-3.8B 3.8B | 0.45 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 325 | Sakalti/Tara-3.8B-v1.1 3.8B | 0.45 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 326 | jpacifico/Chocolatine-3B-Instruct-DPO-Revised 3.8B | 0.45 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 327 | cognitivecomputations/Dolphin3.0-R1-Mistral-24B 23.6B | 0.45 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 328 | Rakuten/RakutenAI-2.0-mini-instruct 1.5B | 0.45 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 329 | rinna/gemma-2-baku-2b-it 2.6B | 0.45 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 330 | Sakalti/Saka-1.5B 1.8B | 0.45 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 331 | Qwen/Qwen2.5-1.5B-Instruct 1.5B | 0.44 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 332 | | 0.44 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 333 | Sakalti/SJT-1.5B-Alpha 1.8B | 0.44 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 334 | Qwen/Qwen1.5-MoE-A2.7B 14.3B | 0.44 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 335 | mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1 8B | 0.43 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 336 | Granite-3.1-2B-Instruct | 0.43 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 337 | TIGER-Lab/Qwen2.5-Math-7B-CFT 7.6B | 0.43 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 338 | llm-jp/llm-jp-3-3.7b-instruct3 3.8B | 0.43 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 339 | llm-jp/llm-jp-3-3.7b-instruct2 3.8B | 0.43 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 340 | Saxo/Linkbricks-Horizon-AI-Korean-LLAMA3blend-8x8b 47.5B | 0.43 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 341 | meta-llama/Llama-2-13b-hf 13B | 0.43 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 342 | | 0.42 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 343 | tiiuae/Falcon3-7B-base 7.5B | 0.42 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 344 | deepseek-ai/DeepSeek-R1-Distill-Llama-8B 8B | 0.41 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 345 | meta-llama/Llama-3.2-3B-Instruct 3.2B | 0.41 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 346 | Saxo/Linkbricks-Horizon-AI-Japanese-Pro-V6-70B 70.6B | 0.41 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 347 | Granite-3.1-2B-Base | 0.41 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 348 | | 0.41 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 349 | meta-llama/Llama-2-13b-chat-hf 13B | 0.41 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 350 | | 0.41 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jul 2024 | huggingface.co | Official leaderboard |
| 351 | Qwen/Qwen1.5-MoE-A2.7B-Chat 14.3B | 0.40 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 352 | Elizezen/SniffyOtter-7B 7.2B | 0.40 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 353 | Elizezen/Antler-7B 7.2B | 0.40 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 354 | rubenroy/Zurich-1.5B-GCv2-5m 1.5B | 0.40 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 355 | | 0.40 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 356 | | 0.39 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 357 | llm-jp/llm-jp-3-1.8b-instruct 1.9B | 0.39 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 358 | Qwen/Qwen1.5-4B 4B | 0.39 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 359 | | 0.39 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 360 | mobiuslabsgmbh/DeepSeek-R1-ReDistill-Qwen-7B-v1.1 7.6B | 0.39 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 361 | Gen-Verse/ReasonFlux-F1-7B 7.6B | 0.38 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 362 | meta-llama/Llama-2-7b-chat-hf 6.7B | 0.38 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 363 | Granite-3.1-3B-A800M-Instruct | 0.38 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 364 | Sakalti/SJT-2.4B-Alpha 2.2B | 0.38 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 365 | weblab-GENIAC/Tanuki-8B-dpo-v1.0 7.5B | 0.38 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 366 | llm-jp/llm-jp-3-3.7b 3.8B | 0.38 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 367 | Saxo/Linkbricks-Horizon-AI-Superb-70B 70.6B | 0.38 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 368 | meta-llama/Llama-2-7b-hf 6.7B | 0.37 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 369 | Granite-3.1-3B-A800M-Base | 0.37 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 370 | | 0.35 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 371 | cyberagent/calm2-7b-chat 7B | 0.35 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 372 | Elizezen/Hameln-japanese-mistral-7B 7.2B | 0.34 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 373 | llm-jp/llm-jp-3-1.8b 1.9B | 0.34 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 374 | prithivMLmods/Qwen2.5-32B-DeepSeek-R1-Instruct 32.8B | 0.33 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 375 | llm-jp/llm-jp-3-980m-instruct2 1B | 0.33 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 376 | llm-jp/llm-jp-3-980m-instruct3 1B | 0.33 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 377 | Qwen/Qwen2.5-Math-7B-Instruct 7.6B | 0.32 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 378 | | 0.31 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 379 | | 0.31 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Oct 2024 | huggingface.co | Official leaderboard |
| 380 | Qwen/Qwen2.5-0.5B-Instruct 0.5B | 0.31 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 381 | Sakalti/CodeSaka-7.6B 7.6B | 0.31 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 382 | meta-llama/Llama-3.2-1B-Instruct 1.2B | 0.31 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 383 | tiiuae/Falcon3-3B-base 3.2B | 0.30 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 384 | Qwen/Qwen1.5-1.8B 1.8B | 0.30 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 385 | Qwen/Qwen1.5-1.8B-Chat 1.8B | 0.30 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 386 | | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 387 | | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 388 | Granite-3.1-1B-A400M-Instruct | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 389 | llm-jp/llm-jp-3-980m 1B | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 390 | yamatazen/Twilight-SCE-12B-v2 12.3B | 0.29 | 0-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 391 | | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 392 | | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2024 | huggingface.co | Official leaderboard |
| 393 | sail/Sailor2-1B-Chat 1B | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 394 | | 0.28 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 395 | | 0.28 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 396 | google/gemma-2b-it 2.5B | 0.27 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 397 | Granite-3.1-1B-A400M-Base | 0.27 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 398 | agentica-org/DeepScaleR-1.5B-Preview 1.8B | 0.26 | 4-shot / float32 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 399 | agentica-org/DeepCoder-1.5B-Preview 1.8B | 0.26 | 4-shot / float32 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 400 | | 0.26 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 401 | llm-jp/llm-jp-3-440m-instruct2 0.4B | 0.25 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 402 | llm-jp/llm-jp-3-440m-instruct3 0.4B | 0.25 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 403 | | 0.24 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 404 | llm-jp/llm-jp-3-440m 0.4B | 0.24 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 405 | rsh345/llama3.1-8b-swallow-openmath-dare_ties-d5w5_d5w5 8B | 0.24 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 406 | nvidia/OpenMath2-Llama3.1-8B 8B | 0.23 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 407 | Qwen/Qwen1.5-0.5B 0.6B | 0.23 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 408 | tiiuae/Falcon3-1B-base 1.7B | 0.21 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 409 | Sakalti/MathSaka-7.6B 7.6B | 0.21 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 410 | CjangCjengh/GaLLM-multi-14B-v0.1 14.8B | 0.21 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 411 | CjangCjengh/GaLLM-14B-v0.2 14.8B | 0.19 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 412 | Qwen/Qwen1.5-0.5B-Chat 0.6B | 0.19 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 413 | llm-jp/llm-jp-3-150m 0.2B | 0.18 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 414 | HuggingFaceTB/SmolLM2-135M-Instruct 0.1B | 0.16 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 415 | google/gemma-7b-it 8.5B | 0.15 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard | |
| 416 | | 0.15 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2024 | huggingface.co | Official leaderboard |
| 417 | | 0.04 | 0-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 418 | Sakalti/Saka-7.2B 7.3B | 0.03 | 0-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | huggingface.co | Official leaderboard |
| # | Model | Score | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | | 0.67 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Nov 2024 | huggingface.co | Official leaderboard |
| 2 | | 0.66 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Oct 2024 | huggingface.co | Official leaderboard |
| 3 | | 0.65 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 4 | | 0.64 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 5 | | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2025 | huggingface.co | Official leaderboard |
| 6 | | 0.63 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2024 | huggingface.co | Official leaderboard |
| 7 | | 0.62 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Nov 2024 | huggingface.co | Official leaderboard |
| 8 | | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 9 | | 0.61 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 10 | | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jul 2024 | huggingface.co | Official leaderboard |
| 11 | | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 12 | | 0.59 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Mar 2025 | huggingface.co | Official leaderboard |
| 13 | | 0.58 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 14 | | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 15 | | 0.57 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 16 | Reka Flash 3.1 | 0.55 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | May 2025 | huggingface.co | Official leaderboard |
| 17 | | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 18 | | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 19 | Qwen2.5-Coder-14B-Instruct | 0.54 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Nov 2024 | huggingface.co | Official leaderboard |
| 20 | Qwen2-VL-7B-Instruct | 0.53 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Aug 2024 | huggingface.co | Official leaderboard |
| 21 | | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 22 | | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 23 | | 0.52 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 24 | | 0.52 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Oct 2024 | huggingface.co | Official leaderboard |
| 25 | | 0.51 | 4-shot / float16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Oct 2024 | huggingface.co | Official leaderboard |
| 26 | | 0.50 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 27 | Granite-3.1-8B-Instruct | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 28 | | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jul 2024 | huggingface.co | Official leaderboard |
| 29 | Granite 3.2 8B Instruct | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2025 | huggingface.co | Official leaderboard |
| 30 | | 0.49 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2024 | huggingface.co | Official leaderboard |
| 31 | | 0.47 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 32 | Granite-3.1-8B-Base | 0.46 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 33 | | 0.45 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 34 | | 0.45 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 35 | | 0.44 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 36 | Granite-3.1-2B-Instruct | 0.43 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 37 | | 0.42 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 38 | Granite-3.1-2B-Base | 0.41 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 39 | | 0.41 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 40 | | 0.41 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jul 2024 | huggingface.co | Official leaderboard |
| 41 | | 0.40 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 42 | | 0.39 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 43 | | 0.39 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 44 | Granite-3.1-3B-A800M-Instruct | 0.38 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 45 | Granite-3.1-3B-A800M-Base | 0.37 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 46 | | 0.35 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 47 | | 0.31 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 48 | | 0.31 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Oct 2024 | huggingface.co | Official leaderboard |
| 49 | | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 50 | | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jun 2024 | huggingface.co | Official leaderboard |
| 51 | Granite-3.1-1B-A400M-Instruct | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 52 | | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
| 53 | | 0.29 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2024 | huggingface.co | Official leaderboard |
| 54 | | 0.28 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 55 | | 0.28 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Sep 2024 | huggingface.co | Official leaderboard |
| 56 | Granite-3.1-1B-A400M-Base | 0.27 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 57 | | 0.26 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Jan 2025 | huggingface.co | Official leaderboard |
| 58 | | 0.24 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Dec 2024 | huggingface.co | Official leaderboard |
| 59 | | 0.15 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Apr 2024 | huggingface.co | Official leaderboard |
| 60 | | 0.04 | 0-shot / bfloat16 / main / vLLM v0.6.3.post1, llm-jp-eval v1.4.1 | Feb 2024 | huggingface.co | Official leaderboard |
Rankings by part
We read 54 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as AVG (NLI), AVG (QA) and AVG (RC). Aya Expanse 32B leads 17 of them, and 12 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.
AVG (NLI)
60 models we track have a score here.
- 1
Cogito v1 Preview 70B ↑4 0.83
- 2
Llama 3.3 70B Instruct ↑6 0.80
- 3
QwQ-32B-Preview ↓2 0.79
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.79
- 5
Qwen2.5-32B ↑4 0.78
- 6
Qwen2.5-14B-Instruct-1M ↓3 0.78
- 7
DeepSeek-R1-Distill-Qwen-32B ↑4 0.77
- 8
DeepSeek-R1-Distill-Qwen-14B ↑6 0.76
- 9
Qwen2.5-14B ↓5 0.76
- 10
DeepSeek-R1-Distill-Llama-70B ↑3 0.75
AVG (QA)
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.65
- 2
Llama 3.1-70B ↑8 0.65
- 3
Llama 3-70B ↑3 0.65
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.65
- 5
Llama 3.3 70B Instruct ↑3 0.64
- 6
Cogito v1 Preview 70B ↓1 0.62
- 7
QwQ-32B-Preview ↓6 0.58
- 8
DeepSeek-R1-Distill-Llama-70B ↑5 0.57
- 9
Qwen2.5-32B 0.57
- 10
Qwen2.5-14B ↓6 0.54
AVG (RC)
60 models we track have a score here.
- 1
Qwen2.5-32B ↑8 0.93
- 2
QwQ-32B-Preview ↓1 0.93
- 3
Llama 3.1-70B ↑7 0.92
- 4
Qwen2.5-14B 0.92
- 5
Llama 3-70B ↑1 0.92
- 6
Llama-3.1-Nemotron-70B-Instruct ↓4 0.92
- 7
DeepSeek-R1-Distill-Llama-70B ↑6 0.91
- 8
Aya Expanse 32B ↑17 0.91
- 9
Qwen1.5-32B ↑6 0.91
- 10 Qwen2.5-Coder-14B-Instruct ↑9 0.91
AVG (MC)
60 models we track have a score here.
- 1
QwQ-32B-Preview 0.90
- 2
Qwen2.5-32B ↑7 0.89
- 3
Llama 3.3 70B Instruct ↑5 0.89
- 4
QwQ-32B ↑8 0.89
- 5
Qwen2.5-14B ↓1 0.89
- 6
Llama-3.1-Nemotron-70B-Instruct ↓4 0.89
- 7
DeepSeek-R1-Distill-Qwen-32B ↑4 0.88
- 8
Cogito v1 Preview 70B ↓3 0.88
- 9
Qwen2.5-14B-Instruct-1M ↓6 0.88
- 10
Llama 3-70B ↓4 0.88
AVG (EL)
60 models we track have a score here.
- 1
Cogito v1 Preview 70B ↑4 0.59
- 2
DeepSeek-R1-Distill-Qwen-14B ↑12 0.58
- 3
QwQ-32B-Preview ↓2 0.58
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.57
- 5
Llama 3.3 70B Instruct ↑3 0.57
- 6
DeepSeek-R1-Distill-Qwen-32B ↑5 0.57
- 7
Qwen2.5-14B-Instruct-1M ↓4 0.56
- 8
QwQ-32B ↑4 0.56
- 9 Reka Flash 3.1 ↑7 0.56
- 10
Qwen2.5-32B ↓1 0.56
AVG (FA)
60 models we track have a score here.
- 1
Llama 3.1-70B ↑9 0.30
- 2
Llama 3-70B ↑4 0.29
- 3
Aya Expanse 32B ↑22 0.29
- 4
Cogito v1 Preview 70B ↑1 0.29
- 5 Reka Flash 3.1 ↑11 0.29
- 6
QwQ-32B-Preview ↓5 0.28
- 7
Qwen2.5-32B ↑2 0.28
- 8
Qwen2.5-14B-Instruct-1M ↓5 0.28
- 9
Qwen2.5-Coder-32B-Instruct ↓2 0.28
- 10
Llama-3.1-Nemotron-70B-Instruct ↓8 0.27
AVG (MR)
60 models we track have a score here.
- 1
Llama 3-70B ↑5 0.95
- 2
Llama 3.3 70B Instruct ↑6 0.94
- 3
QwQ-32B-Preview ↓2 0.94
- 4
Cogito v1 Preview 70B ↑1 0.94
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.93
- 6
DeepSeek-R1-Distill-Qwen-32B ↑5 0.92
- 7
Qwen2.5-32B ↑2 0.92
- 8
Qwen2.5-Coder-32B-Instruct ↓1 0.90
- 9
DeepSeek-R1-Distill-Llama-70B ↑4 0.89
- 10
Llama 3.1-70B 0.88
AVG (MT)
60 models we track have a score here.
- 1
Llama 3.1-70B ↑9 0.86
- 2
Aya Expanse 32B ↑23 0.86
- 3
Llama 3-70B ↑3 0.85
- 4
Llama 3.3 70B Instruct ↑4 0.85
- 5
Cogito v1 Preview 70B 0.85
- 6
Qwen2.5-32B ↑3 0.85
- 7
Llama-3.1-Nemotron-70B-Instruct ↓5 0.85
- 8
QwQ-32B-Preview ↓7 0.85
- 9
DeepSeek-R1-Distill-Llama-70B ↑4 0.85
- 10
Qwen2.5-14B ↓6 0.84
AVG (HE)
60 models we track have a score here.
- 1
Cogito v1 Preview 70B ↑4 0.83
- 2
QwQ-32B-Preview ↓1 0.78
- 3
Qwen2.5-32B ↑6 0.77
- 4
Llama 3.3 70B Instruct ↑4 0.77
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.77
- 6
DeepSeek-R1-Distill-Qwen-32B ↑5 0.75
- 7
Qwen2.5-14B-Instruct-1M ↓4 0.73
- 8
DeepSeek-R1-Distill-Llama-70B ↑5 0.72
- 9
Qwen2.5-14B ↓5 0.72
- 10
Llama 3-70B ↓4 0.72
AVG (CG)
60 models we track have a score here.
- 1
Qwen2.5-Coder-32B-Instruct ↑6 0.68
- 2
QwQ-32B-Preview ↓1 0.66
- 3
Qwen2.5-14B-Instruct-1M 0.64
- 4
Qwen2.5-14B 0.58
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.56
- 6
Qwen2.5-3B ↑15 0.46
- 7
Qwen1.5-14B ↑10 0.40
- 8
QwQ-32B ↑4 0.40
- 9
Llama 3-70B ↓3 0.30
- 10
Qwen2.5-1.5B ↑23 0.24
AVG (SUM)
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.14
- 2
Qwen1.5-32B ↑13 0.13
- 3
Qwen2.5-32B ↑6 0.12
- 4 Granite 3.2 8B Instruct ↑25 0.12
- 5
QwQ-32B-Preview ↓4 0.11
- 6
Qwen2.5-14B ↓2 0.11
- 7
Falcon3-10B-Instruct ↑28 0.11
- 8
Qwen2-7B ↑14 0.11
- 9
Aya Expanse 8B ↑15 0.11
- 10
DeepSeek-R1-Distill-Qwen-14B ↑4 0.11
ALT E to J BERT Score
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.88
- 2
Llama 3.1-70B ↑8 0.87
- 3
Cogito v1 Preview 70B ↑2 0.87
- 4
Llama 3-70B ↑2 0.87
- 5
Llama 3.3 70B Instruct ↑3 0.87
- 6
Llama-3.1-Nemotron-70B-Instruct ↓4 0.87
- 7
Qwen2.5-32B ↑2 0.87
- 8
Aya Expanse 8B ↑16 0.87
- 9
QwQ-32B-Preview ↓8 0.86
- 10
Qwen2.5-14B ↓6 0.86
ALT E to J BLEU
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.15
- 2
Llama 3-70B ↑4 0.14
- 3
Llama-3.1-Nemotron-70B-Instruct ↓1 0.14
- 4
Llama 3.1-70B ↑6 0.14
- 5
Cogito v1 Preview 70B 0.14
- 6
Llama 3.3 70B Instruct ↑2 0.14
- 7
Aya Expanse 8B ↑17 0.13
- 8
Qwen2.5-32B ↑1 0.13
- 9
QwQ-32B-Preview ↓8 0.13
- 10
Qwen2.5-14B-Instruct-1M ↓7 0.13
ALT E to J COMET WMT22 ⭐
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.92
- 2
Cogito v1 Preview 70B ↑3 0.91
- 3
Llama 3.1-70B ↑7 0.91
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.91
- 5
Qwen2.5-32B ↑4 0.91
- 6
Aya Expanse 8B ↑18 0.91
- 7
Llama 3.3 70B Instruct ↑1 0.91
- 8
QwQ-32B-Preview ↓7 0.91
- 9
Llama 3-70B ↓3 0.91
- 10
Qwen2.5-14B-Instruct-1M ↓7 0.91
ALT J to E BERT Score
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.96
- 2
Llama 3.1-70B ↑8 0.96
- 3
Cogito v1 Preview 70B ↑2 0.96
- 4
Llama 3-70B ↑2 0.96
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.96
- 6
Llama 3.3 70B Instruct ↑2 0.96
- 7
Qwen2.5-32B ↑2 0.96
- 8
DeepSeek-R1-Distill-Llama-70B ↑5 0.96
- 9
Qwen2.5-14B ↓5 0.96
- 10
QwQ-32B-Preview ↓9 0.96
ALT J to E BLEU
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.19
- 2
Llama 3-70B ↑4 0.19
- 3
Llama 3.1-70B ↑7 0.19
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.18
- 5
Llama 3.3 70B Instruct ↑3 0.18
- 6
Cogito v1 Preview 70B ↓1 0.18
- 7
Qwen2.5-32B ↑2 0.18
- 8
Qwen2.5-14B ↓4 0.18
- 9
DeepSeek-R1-Distill-Llama-70B ↑4 0.17
- 10
Qwen2.5-14B-Instruct-1M ↓7 0.17
ALT J to E COMET WMT22 ⭐
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.89
- 2
Cogito v1 Preview 70B ↑3 0.89
- 3
Llama 3.1-70B ↑7 0.89
- 4
Llama 3-70B ↑2 0.89
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.89
- 6
Llama 3.3 70B Instruct ↑2 0.89
- 7
Qwen2.5-32B ↑2 0.89
- 8
Qwen2.5-14B ↓4 0.89
- 9
QwQ-32B-Preview ↓8 0.89
- 10
DeepSeek-R1-Distill-Llama-70B ↑3 0.89
ChABSA ⭐
60 models we track have a score here.
- 1
Cogito v1 Preview 70B ↑4 0.59
- 2
DeepSeek-R1-Distill-Qwen-14B ↑12 0.58
- 3
QwQ-32B-Preview ↓2 0.58
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.57
- 5
Llama 3.3 70B Instruct ↑3 0.57
- 6
DeepSeek-R1-Distill-Qwen-32B ↑5 0.57
- 7
Qwen2.5-14B-Instruct-1M ↓4 0.56
- 8
QwQ-32B ↑4 0.56
- 9 Reka Flash 3.1 ↑7 0.56
- 10
Qwen2.5-32B ↓1 0.56
CommonSenseMoralJA ⭐
60 models we track have a score here.
- 1
QwQ-32B-Preview 0.91
- 2
Llama 3.3 70B Instruct ↑6 0.91
- 3
Llama 3.1-70B ↑7 0.91
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.91
- 5
Cogito v1 Preview 70B 0.91
- 6
Qwen2.5-14B ↓2 0.90
- 7
QwQ-32B ↑5 0.90
- 8
Llama 3-70B ↓2 0.90
- 9
DeepSeek-R1-Distill-Qwen-32B ↑2 0.90
- 10
Qwen2.5-32B ↓1 0.89
JAMP ⭐
60 models we track have a score here.
- 1
Cogito v1 Preview 70B ↑4 0.74
- 2
Llama 3.3 70B Instruct ↑6 0.67
- 3
Qwen2.5-32B ↑6 0.67
- 4
DeepSeek-R1-Distill-Qwen-32B ↑7 0.67
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.66
- 6
QwQ-32B-Preview ↓5 0.66
- 7
DeepSeek-R1-Distill-Llama-70B ↑6 0.66
- 8
Qwen2.5-Coder-32B-Instruct ↓1 0.65
- 9
QwQ-32B ↑3 0.65
- 10
Qwen2.5-14B-Instruct-1M ↓7 0.65
JANLI ⭐
60 models we track have a score here.
- 1
Cogito v1 Preview 70B ↑4 0.92
- 2
Llama 3.3 70B Instruct ↑6 0.91
- 3
Qwen2.5-Coder-32B-Instruct ↑4 0.87
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.87
- 5
Qwen2.5-14B-Instruct-1M ↓2 0.82
- 6
Qwen2.5-32B ↑3 0.82
- 7
QwQ-32B-Preview ↓6 0.81
- 8
QwQ-32B ↑4 0.79
- 9 Qwen2.5-Coder-14B-Instruct ↑10 0.79
- 10
Llama 3.1-70B 0.79
JCommonSenseQA ⭐
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.96
- 2
Qwen2.5-32B ↑7 0.96
- 3
Qwen2.5-14B ↑1 0.96
- 4
Qwen2.5-14B-Instruct-1M ↓1 0.96
- 5
DeepSeek-R1-Distill-Qwen-32B ↑6 0.95
- 6
QwQ-32B-Preview ↓5 0.95
- 7
Llama 3-70B ↓1 0.95
- 8
Llama-3.1-Nemotron-70B-Instruct ↓6 0.95
- 9
Llama 3.3 70B Instruct ↓1 0.95
- 10
QwQ-32B ↑2 0.94
JEMHopQA ⭐
60 models we track have a score here.
- 1
Llama 3.3 70B Instruct ↑7 0.65
- 2
Llama-3.1-Nemotron-70B-Instruct 0.62
- 3
QwQ-32B-Preview ↓2 0.62
- 4
Llama 3.1-70B ↑6 0.61
- 5
Llama 3-70B ↑1 0.61
- 6
Aya Expanse 32B ↑19 0.58
- 7
Qwen2.5-32B ↑2 0.57
- 8
Qwen2.5-14B ↓4 0.55
- 9
Cogito v1 Preview 70B ↓4 0.55
- 10
DeepSeek-R1-Distill-Llama-70B ↑3 0.54
JMMLU ⭐
60 models we track have a score here.
- 1
Cogito v1 Preview 70B ↑4 0.79
- 2
QwQ-32B-Preview ↓1 0.76
- 3
Qwen2.5-32B ↑6 0.75
- 4
Llama 3.3 70B Instruct ↑4 0.73
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.72
- 6
DeepSeek-R1-Distill-Qwen-32B ↑5 0.72
- 7
Qwen2.5-14B-Instruct-1M ↓4 0.70
- 8
Qwen2.5-14B ↓4 0.69
- 9
Llama 3-70B ↓3 0.69
- 10
QwQ-32B ↑2 0.68
JNLI ⭐
60 models we track have a score here.
- 1
QwQ-32B-Preview 0.90
- 2
Qwen2.5-32B ↑7 0.89
- 3
Cogito v1 Preview 70B ↑2 0.87
- 4
DeepSeek-R1-Distill-Qwen-32B ↑7 0.86
- 5
Qwen2.5-14B-Instruct-1M ↓2 0.84
- 6
Qwen2.5-7B ↑12 0.83
- 7
Aya Expanse 32B ↑18 0.83
- 8
Qwen1.5-32B ↑7 0.83
- 9
Qwen2.5-14B ↓5 0.83
- 10
DeepSeek-R1-Distill-Qwen-14B ↑4 0.82
JSEM ⭐
60 models we track have a score here.
- 1
QwQ-32B-Preview 0.80
- 2
QwQ-32B ↑10 0.80
- 3
Llama 3.1-70B ↑7 0.80
- 4
Llama 3-70B ↑2 0.79
- 5
Qwen2.5-32B ↑4 0.79
- 6
Llama 3.3 70B Instruct ↑2 0.79
- 7
DeepSeek-R1-Distill-Qwen-32B ↑4 0.79
- 8
Cogito v1 Preview 70B ↓3 0.79
- 9
Llama-3.1-Nemotron-70B-Instruct ↓7 0.79
- 10
Aya Expanse 32B ↑15 0.78
JSICK ⭐
60 models we track have a score here.
- 1
Llama-3.1-Nemotron-70B-Instruct ↑1 0.87
- 2
Qwen2.5-14B-Instruct-1M ↑1 0.85
- 3
DeepSeek-R1-Distill-Llama-70B ↑10 0.84
- 4
Cogito v1 Preview 70B ↑1 0.83
- 5
DeepSeek-R1-Distill-Qwen-14B ↑9 0.83
- 6
Llama 3.3 70B Instruct ↑2 0.82
- 7
Qwen2.5-7B ↑11 0.81
- 8
Qwen2-7B ↑14 0.79
- 9
Llama 3-70B ↓3 0.79
- 10
Qwen1.5-7B ↑21 0.78
JSquad ⭐
60 models we track have a score here.
- 1
Qwen2.5-32B ↑8 0.93
- 2
QwQ-32B-Preview ↓1 0.93
- 3
Llama 3.1-70B ↑7 0.92
- 4
Qwen2.5-14B 0.92
- 5
Llama 3-70B ↑1 0.92
- 6
Llama-3.1-Nemotron-70B-Instruct ↓4 0.92
- 7
DeepSeek-R1-Distill-Llama-70B ↑6 0.91
- 8
Aya Expanse 32B ↑17 0.91
- 9
Qwen1.5-32B ↑6 0.91
- 10 Qwen2.5-Coder-14B-Instruct ↑9 0.91
JSTS (Pearson)
60 models we track have a score here.
- 1
Qwen2.5-32B ↑8 0.91
- 2
Cogito v1 Preview 70B ↑3 0.91
- 3
Llama 3.1-70B ↑7 0.90
- 4
DeepSeek-R1-Distill-Qwen-32B ↑7 0.90
- 5
Qwen2.5-14B ↓1 0.90
- 6
QwQ-32B-Preview ↓5 0.90
- 7
Qwen2.5-14B-Instruct-1M ↓4 0.90
- 8
Llama-3.1-Nemotron-70B-Instruct ↓6 0.90
- 9 Reka Flash 3.1 ↑7 0.89
- 10
Llama 3-70B ↓4 0.89
JSTS (Spearman)
60 models we track have a score here.
- 1
Qwen2.5-32B ↑8 0.88
- 2
Qwen2.5-14B-Instruct-1M ↑1 0.88
- 3
DeepSeek-R1-Distill-Qwen-32B ↑8 0.88
- 4
Cogito v1 Preview 70B ↑1 0.88
- 5
QwQ-32B-Preview ↓4 0.87
- 6
Qwen2.5-14B ↓2 0.87
- 7
Llama 3.1-70B ↑3 0.87
- 8
Aya Expanse 32B ↑17 0.86
- 9
Llama-3.1-Nemotron-70B-Instruct ↓7 0.86
- 10
DeepSeek-R1-Distill-Llama-70B ↑3 0.86
KUCI ⭐
60 models we track have a score here.
- 1
Qwen2.5-32B ↑8 0.82
- 2
QwQ-32B-Preview ↓1 0.82
- 3
QwQ-32B ↑9 0.81
- 4
Llama 3.3 70B Instruct ↑4 0.80
- 5
Aya Expanse 32B ↑20 0.80
- 6
Llama-3.1-Nemotron-70B-Instruct ↓4 0.80
- 7
DeepSeek-R1-Distill-Qwen-32B ↑4 0.80
- 8
Cogito v1 Preview 70B ↓3 0.80
- 9
Qwen2.5-14B-Instruct-1M ↓6 0.79
- 10
Qwen2.5-14B ↓6 0.79
MAWPS ⭐
60 models we track have a score here.
- 1
Llama 3-70B ↑5 0.95
- 2
Llama 3.3 70B Instruct ↑6 0.94
- 3
QwQ-32B-Preview ↓2 0.94
- 4
Cogito v1 Preview 70B ↑1 0.94
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.93
- 6
DeepSeek-R1-Distill-Qwen-32B ↑5 0.92
- 7
Qwen2.5-32B ↑2 0.92
- 8
Qwen2.5-Coder-32B-Instruct ↓1 0.90
- 9
DeepSeek-R1-Distill-Llama-70B ↑4 0.89
- 10
Llama 3.1-70B 0.88
MBPP (exec) (0 shots only) ⭐
60 models we track have a score here.
- 1
Qwen2.5-Coder-32B-Instruct ↑6 0.68
- 2
QwQ-32B-Preview ↓1 0.66
- 3
Qwen2.5-14B-Instruct-1M 0.64
- 4
Qwen2.5-14B 0.58
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.56
- 6
Qwen2.5-3B ↑15 0.46
- 7
Qwen1.5-14B ↑10 0.40
- 8
QwQ-32B ↑4 0.40
- 9
Llama 3-70B ↓3 0.30
- 10
Qwen2.5-1.5B ↑23 0.24
MBPP (pylint) (0 shots only)
60 models we track have a score here.
- 1
Qwen2.5-14B-Instruct-1M ↑2 0.96
- 2
QwQ-32B-Preview ↓1 0.95
- 3
Qwen1.5-14B ↑14 0.91
- 4
Qwen2.5-3B ↑17 0.91
- 5
Qwen2.5-14B ↓1 0.89
- 6
Llama-3.1-Nemotron-70B-Instruct ↓4 0.89
- 7
Gemma 7B ↑53 0.84
- 8
Qwen2.5-Coder-32B-Instruct ↓1 0.76
- 9
QwQ-32B ↑3 0.68
- 10
Qwen2.5-1.5B ↑23 0.62
MMLU ⭐
60 models we track have a score here.
- 1
Cogito v1 Preview 70B ↑4 0.86
- 2
Llama 3.3 70B Instruct ↑6 0.82
- 3
Llama-3.1-Nemotron-70B-Instruct ↓1 0.81
- 4
QwQ-32B-Preview ↓3 0.80
- 5
Qwen2.5-32B ↑4 0.79
- 6
DeepSeek-R1-Distill-Qwen-32B ↑5 0.78
- 7
DeepSeek-R1-Distill-Llama-70B ↑6 0.77
- 8
Qwen2.5-14B-Instruct-1M ↓5 0.75
- 9
Qwen2.5-14B ↓5 0.75
- 10
Llama 3-70B ↓4 0.74
NIILC ⭐
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.62
- 2
Llama 3.1-70B ↑8 0.60
- 3
Llama 3-70B ↑3 0.60
- 4
Cogito v1 Preview 70B ↑1 0.59
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.59
- 6
Llama 3.3 70B Instruct ↑2 0.56
- 7
Qwen2.5-32B ↑2 0.55
- 8
Qwen2.5-14B ↓4 0.53
- 9
QwQ-32B-Preview ↓8 0.53
- 10
Qwen2.5-14B-Instruct-1M ↓7 0.52
JAQKET ⭐
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.75
- 2
Llama 3-70B ↑4 0.73
- 3
Llama 3.1-70B ↑7 0.73
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.73
- 5
Cogito v1 Preview 70B 0.72
- 6
Llama 3.3 70B Instruct ↑2 0.71
- 7
DeepSeek-R1-Distill-Llama-70B ↑6 0.66
- 8
QwQ-32B-Preview ↓7 0.59
- 9
Qwen2.5-32B 0.58
- 10
Qwen2.5-14B ↓6 0.55
Wiki Coreference ⭐
60 models we track have a score here.
- 1
Llama-3.1-Nemotron-70B-Instruct ↑1 0.10
- 2
Cogito v1 Preview 70B ↑3 0.08
- 3 Qwen2.5-Coder-14B-Instruct ↑16 0.06
- 4
Llama 3.1-70B ↑6 0.06
- 5
Qwen2.5-Coder-32B-Instruct ↑2 0.05
- 6
Qwen1.5-32B ↑9 0.05
- 7
Qwen2.5-14B-Instruct-1M ↓4 0.04
- 8
Llama 3.3 70B Instruct 0.04
- 9 Granite-3.1-8B-Instruct ↑18 0.04
- 10
Qwen2.5-14B ↓6 0.04
Wiki Dependency ⭐
60 models we track have a score here.
- 1 Reka Flash 3.1 ↑15 0.44
- 2
Qwen2.5-14B-Instruct-1M ↑1 0.41
- 3
Aya Expanse 32B ↑22 0.40
- 4
QwQ-32B-Preview ↓3 0.38
- 5
Llama 3-70B ↑1 0.38
- 6
QwQ-32B ↑6 0.37
- 7
Llama 3-8B ↑23 0.37
- 8
Qwen2.5-Coder-32B-Instruct ↓1 0.36
- 9
Llama 3.1-70B ↑1 0.36
- 10
DeepSeek-R1-Distill-Qwen-32B ↑1 0.36
Wiki NER ⭐
60 models we track have a score here.
- 1
QwQ-32B ↑11 0.18
- 2
Llama 3-70B ↑4 0.14
- 3
DeepSeek-R1-Distill-Qwen-32B ↑8 0.14
- 4
QwQ-32B-Preview ↓3 0.13
- 5
Llama 3.1-70B ↑5 0.12
- 6
Qwen2.5-32B ↑3 0.12
- 7
Qwen2.5-Coder-32B-Instruct 0.12
- 8
Cogito v1 Preview 70B ↓3 0.12
- 9
Gemma 2 2B IT ↑31 0.10
- 10
DeepSeek-R1-Distill-Qwen-14B ↑4 0.10
Wiki PAS ⭐
60 models we track have a score here.
- 1 Qwen2.5-Coder-14B-Instruct ↑18 0.11
- 2
Qwen2.5-14B-Instruct-1M ↑1 0.09
- 3
Aya Expanse 32B ↑22 0.09
- 4
Cogito v1 Preview 70B ↑1 0.09
- 5
Qwen2.5-32B ↑4 0.08
- 6
QwQ-32B-Preview ↓5 0.08
- 7 Reka Flash 3.1 ↑9 0.08
- 8
QwQ-32B ↑4 0.07
- 9
Qwen2.5-Coder-32B-Instruct ↓2 0.07
- 10
Qwen1.5-32B ↑5 0.07
Wiki Reading ⭐
60 models we track have a score here.
- 1
Llama 3.1-70B ↑9 0.87
- 2
Aya Expanse 32B ↑23 0.87
- 3
Llama 3-70B ↑3 0.87
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.86
- 5
Llama 3.3 70B Instruct ↑3 0.85
- 6
Cogito v1 Preview 70B ↓1 0.85
- 7 Reka Flash 3.1 ↑9 0.84
- 8
Qwen2.5-32B ↑1 0.82
- 9
QwQ-32B-Preview ↓8 0.79
- 10
Qwen2.5-14B-Instruct-1M ↓7 0.79
WikiCorpus E to J BERT Score
60 models we track have a score here.
- 1
Llama 3-70B ↑5 0.86
- 2
Llama 3.1-70B ↑8 0.86
- 3
Llama 3.3 70B Instruct ↑5 0.85
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.85
- 5
Aya Expanse 32B ↑20 0.85
- 6
Qwen2.5-32B ↑3 0.84
- 7
QwQ-32B-Preview ↓6 0.83
- 8
DeepSeek-R1-Distill-Llama-70B ↑5 0.83
- 9
Qwen2.5-14B ↓5 0.82
- 10
DeepSeek-R1-Distill-Qwen-32B ↑1 0.82
WikiCorpus E to J BLEU
60 models we track have a score here.
- 1
Llama-3.1-Nemotron-70B-Instruct ↑1 0.17
- 2
Llama 3-70B ↑4 0.16
- 3
Llama 3.3 70B Instruct ↑5 0.16
- 4
Llama 3.1-70B ↑6 0.15
- 5
Qwen2.5-32B ↑4 0.14
- 6
DeepSeek-R1-Distill-Llama-70B ↑7 0.13
- 7
Aya Expanse 32B ↑18 0.13
- 8
Cogito v1 Preview 70B ↓3 0.12
- 9
QwQ-32B-Preview ↓8 0.12
- 10
Llama 3.1-8B ↑18 0.11
WikiCorpus E to J COMET WMT22 ⭐
60 models we track have a score here.
- 1
Llama 3-70B ↑5 0.86
- 2
Llama 3.1-70B ↑8 0.85
- 3
Llama 3.3 70B Instruct ↑5 0.85
- 4
Aya Expanse 32B ↑21 0.85
- 5
Llama-3.1-Nemotron-70B-Instruct ↓3 0.84
- 6
Qwen2.5-32B ↑3 0.84
- 7
Cogito v1 Preview 70B ↓2 0.84
- 8
QwQ-32B-Preview ↓7 0.84
- 9
Qwen2.5-14B ↓5 0.84
- 10
Qwen2.5-14B-Instruct-1M ↓7 0.83
WikiCorpus J to E BERT Score
60 models we track have a score here.
- 1
Llama 3.1-70B ↑9 0.91
- 2
Llama 3-70B ↑4 0.91
- 3
Aya Expanse 32B ↑22 0.91
- 4
DeepSeek-R1-Distill-Llama-70B ↑9 0.91
- 5
Llama 3.3 70B Instruct ↑3 0.91
- 6
Qwen2.5-32B ↑3 0.91
- 7
Cogito v1 Preview 70B ↓2 0.91
- 8
Llama-3.1-Nemotron-70B-Instruct ↓6 0.91
- 9
QwQ-32B-Preview ↓8 0.91
- 10
DeepSeek-R1-Distill-Qwen-32B ↑1 0.90
WikiCorpus J to E BLEU
60 models we track have a score here.
- 1
Llama 3.1-70B ↑9 0.13
- 2
Llama 3-70B ↑4 0.13
- 3
Llama 3.3 70B Instruct ↑5 0.12
- 4
Llama-3.1-Nemotron-70B-Instruct ↓2 0.12
- 5
DeepSeek-R1-Distill-Llama-70B ↑8 0.12
- 6
Aya Expanse 32B ↑19 0.12
- 7
Qwen2.5-32B ↑2 0.12
- 8
QwQ-32B-Preview ↓7 0.11
- 9
Cogito v1 Preview 70B ↓4 0.11
- 10
Llama 3-8B ↑20 0.11
WikiCorpus J to E COMET WMT22 ⭐
60 models we track have a score here.
- 1
Llama 3.1-70B ↑9 0.77
- 2
Cogito v1 Preview 70B ↑3 0.76
- 3
DeepSeek-R1-Distill-Llama-70B ↑10 0.76
- 4
Aya Expanse 32B ↑21 0.76
- 5
QwQ-32B-Preview ↓4 0.76
- 6
Llama 3-70B 0.76
- 7
Llama 3.3 70B Instruct ↑1 0.76
- 8
Qwen2.5-32B ↑1 0.76
- 9
DeepSeek-R1-Distill-Qwen-32B ↑2 0.76
- 10
Llama-3.1-Nemotron-70B-Instruct ↓8 0.76
XL-Sum JA BERT Score (0 shots only)
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.72
- 2
Qwen1.5-32B ↑13 0.71
- 3 Granite 3.2 8B Instruct ↑26 0.71
- 4
Qwen2.5-32B ↑5 0.71
- 5
Qwen2.5-14B ↓1 0.70
- 6
Falcon3-10B-Instruct ↑29 0.70
- 7
QwQ-32B-Preview ↓6 0.70
- 8
Qwen2-7B ↑14 0.70
- 9 Granite-3.1-2B-Instruct ↑27 0.70
- 10 Granite-3.1-8B-Instruct ↑17 0.70
XL-Sum JA BLEU (0 shots only)
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.04
- 2
Llama-3.1-Nemotron-70B-Instruct 0.03
- 3 Granite-3.1-8B-Instruct ↑24 0.03
- 4
Llama 3.3 70B Instruct ↑4 0.03
- 5
QwQ-32B-Preview ↓4 0.03
- 6 Granite 3.2 8B Instruct ↑23 0.03
- 7
Qwen1.5-32B ↑8 0.03
- 8 Granite-3.1-2B-Base ↑30 0.03
- 9
Qwen2.5-14B-Instruct-1M ↓6 0.03
- 10
Qwen2.5-32B ↓1 0.03
XL-Sum ROUGE1 (0 shots only)
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.36
- 2
Qwen1.5-32B ↑13 0.36
- 3 Granite 3.2 8B Instruct ↑26 0.36
- 4
Falcon3-10B-Instruct ↑31 0.35
- 5
Qwen2.5-32B ↑4 0.32
- 6 Granite-3.1-2B-Instruct ↑30 0.32
- 7
Qwen2-7B ↑15 0.31
- 8
Aya Expanse 8B ↑16 0.31
- 9
DeepSeek-R1-Distill-Qwen-14B ↑5 0.30
- 10 Granite-3.1-8B-Instruct ↑17 0.30
XL-Sum ROUGE2 (0 shots only) ⭐
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.14
- 2
Qwen1.5-32B ↑13 0.13
- 3
Qwen2.5-32B ↑6 0.12
- 4 Granite 3.2 8B Instruct ↑25 0.12
- 5
Qwen2.5-14B ↓1 0.11
- 6
QwQ-32B-Preview ↓5 0.11
- 7
Falcon3-10B-Instruct ↑28 0.11
- 8
Qwen2-7B ↑14 0.11
- 9
Aya Expanse 8B ↑15 0.11
- 10
DeepSeek-R1-Distill-Qwen-14B ↑4 0.11
XL-Sum JA ROUGE2 Scaling
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.14
- 2
Qwen1.5-32B ↑13 0.13
- 3
Qwen2.5-32B ↑6 0.12
- 4 Granite 3.2 8B Instruct ↑25 0.12
- 5
QwQ-32B-Preview ↓4 0.11
- 6
Qwen2.5-14B ↓2 0.11
- 7
Falcon3-10B-Instruct ↑28 0.11
- 8
Qwen2-7B ↑14 0.11
- 9
Aya Expanse 8B ↑15 0.11
- 10
DeepSeek-R1-Distill-Qwen-14B ↑4 0.11
XL-Sum ROUGE-Lsum (0 shots only)
60 models we track have a score here.
- 1
Aya Expanse 32B ↑24 0.30
- 2
Qwen1.5-32B ↑13 0.30
- 3 Granite 3.2 8B Instruct ↑26 0.29
- 4
Qwen2.5-32B ↑5 0.27
- 5 Granite-3.1-2B-Instruct ↑31 0.26
- 6
Qwen2-7B ↑16 0.26
- 7
Qwen2.5-14B ↓3 0.26
- 8
Falcon3-10B-Instruct ↑27 0.25
- 9
QwQ-32B-Preview ↓8 0.25
- 10
QwQ-32B ↑2 0.25
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Source: Official Open Japanese LLM Leaderboard leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.67 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.66 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.65 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.64 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.63 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.63 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.62 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.61 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.61 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.59 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.59 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.59 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.58 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.58 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.57 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.57 | |||
| Reka Flash 3.1 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.55 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.54 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.54 | |||
| Qwen2.5-Coder-14B-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.54 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.53 | |||
| Qwen2-VL-7B-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.53 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.52 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.52 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.52 | |||
| 4-shot / float16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.52 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.51 | |||
| 4-shot / float16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.51 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.50 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.50 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.50 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.50 | |||
| Granite-3.1-8B-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.49 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.49 | |||
| Granite 3.2 8B Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.49 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.49 | |||
| 0-shot / float16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.48 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.47 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.47 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.46 | |||
| Granite-3.1-8B-Base | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.46 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.45 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.45 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.44 | |||
| Granite-3.1-2B-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.43 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.42 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.42 | |||
| Granite-3.1-2B-Base | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.41 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.41 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.41 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.41 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.40 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | |||
| Granite 3.2 8B Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | ||
| Granite-3.1-3B-A800M-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.38 | ||
| Reka Flash 3.1 | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.38 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.38 | |||
| Granite-3.1-8B-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.38 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.37 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.37 | |||
| 0-shot / float16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.37 | |||
| Granite-3.1-3B-A800M-Base | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.37 | ||
| Qwen2-VL-7B-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.36 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.35 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.34 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.34 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.33 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.33 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.33 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.31 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.31 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| Granite-3.1-1B-A400M-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| Granite-3.1-2B-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.27 | ||
| Granite-3.1-1B-A400M-Base | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.27 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.26 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.26 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.25 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.25 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.24 | |||
| Granite-3.1-3B-A800M-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.23 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.23 | |||
| Granite-3.1-8B-Base | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.22 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.22 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.22 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.21 | |||
| Qwen2.5-Coder-14B-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.20 | ||
| Granite-3.1-1B-A400M-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.18 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.18 | |||
| Granite-3.1-2B-Base | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.18 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.18 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.17 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.16 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.15 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.15 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.15 | |||
| Granite-3.1-3B-A800M-Base | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.15 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.14 | |||
| Granite-3.1-1B-A400M-Base | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.13 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.12 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.11 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.11 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.10 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.10 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.07 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.07 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.07 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.06 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.04 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.04 |
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Source: Official Open Japanese LLM Leaderboard leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.67 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.66 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.65 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.64 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.63 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.63 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.62 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.61 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.61 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.59 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.59 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.59 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.58 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.58 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.57 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.57 | |||
| Reka Flash 3.1 | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.55 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.54 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.54 | |||
| Qwen2.5-Coder-14B-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.54 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.53 | |||
| Qwen2-VL-7B-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.53 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.52 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.52 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.52 | |||
| 4-shot / float16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.52 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.51 | |||
| 4-shot / float16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.51 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.50 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.50 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.50 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.50 | |||
| Granite-3.1-8B-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.49 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.49 | |||
| Granite 3.2 8B Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.49 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.49 | |||
| 0-shot / float16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.48 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.47 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.47 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.46 | |||
| Granite-3.1-8B-Base | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.46 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.45 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.45 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.44 | |||
| Granite-3.1-2B-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.43 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.42 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.42 | |||
| Granite-3.1-2B-Base | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.41 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.41 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.41 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.41 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.40 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | |||
| Granite 3.2 8B Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.39 | ||
| Granite-3.1-3B-A800M-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.38 | ||
| Reka Flash 3.1 | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.38 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.38 | |||
| Granite-3.1-8B-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.38 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.37 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.37 | |||
| 0-shot / float16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.37 | |||
| Granite-3.1-3B-A800M-Base | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.37 | ||
| Qwen2-VL-7B-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.36 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.35 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.34 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.34 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.33 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.33 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.33 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.31 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.31 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| Granite-3.1-1B-A400M-Instruct | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | ||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.29 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.28 | |||
| Granite-3.1-2B-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.27 | ||
| Granite-3.1-1B-A400M-Base | 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.27 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.26 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.26 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.25 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.25 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.24 | |||
| Granite-3.1-3B-A800M-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.23 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.23 | |||
| Granite-3.1-8B-Base | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.22 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.22 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.22 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.21 | |||
| Qwen2.5-Coder-14B-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.20 | ||
| Granite-3.1-1B-A400M-Instruct | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.18 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.18 | |||
| Granite-3.1-2B-Base | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.18 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.18 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.17 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.16 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.15 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.15 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.15 | |||
| Granite-3.1-3B-A800M-Base | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.15 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.14 | |||
| Granite-3.1-1B-A400M-Base | 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.13 | ||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.12 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.11 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.11 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.10 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.10 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.07 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.07 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.07 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.06 | |||
| 0-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.04 | |||
| 4-shot / bfloat16 / main / vLLM v0.6.3.post1 | llm-jp-eval v1.4.1 | 0.04 |
2 models have held number one on the official leaderboard since its release in November 2024
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official Open Japanese LLM Leaderboard leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official Open Japanese LLM Leaderboard leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
K-EXAONE 90.5%
- MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language.
Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7%
- Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Claude Opus 5 95.7%
- MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%
Data audit
8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 18 months ago, in May 2025, for Reka Flash 3.1. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
- Pass Independent runs Scores from the official leaderboard (418 entries).
- Pass Models scored 60 models we track, plus 358 other entries on its leaderboard.
- Warning Newest score The newest came 18 months ago, in May 2025, for Reka Flash 3.1.
- Note Frontier models None of the top 10 has a score. The strongest tested is QwQ-32B (#137).
- Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
- Pass Importer We read the board on huggingface.co: 122 rows for models we track and 358 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.