Benchmark Almanac

LLM-Perf Leaderboard

Qwen1.5-110B 29.6 leads since April 2024
Not in model cards

Language models generate tokens on a specified hardware and software configuration. The board reports latency, throughput, memory and energy alongside quality scores.

The quality score comes from the Open LLM Leaderboard. Inference measurements depend on the hardware.

Made by
Hugging Face
Kind
Composite indices
Level
University
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Measures
Efficiency
Language
English
In and out
Text → Text
Interaction
Single turn
Items from
Another benchmark
Refresh
One fixed set
Run by
Anyone
Strongest model tested
phi-3-mini 3.8B #204 on the capability index
Last model added
June 2024
Scores refreshed
November 20th, 2026
46
Entries on its own leaderboard, 21 models we track

Qwen1.5-110B leads with 29.6 among models released by June 2024 Qwen1.5-110B leads with 29.6 among models released by June 2024

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in June 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 29.6
  2. 2 phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-eager 27.2
  3. 3 Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemm-fa2 26.7
  4. 4 meta-llama/Meta-Llama-3-8B-Instruct A10-24GB-150W / 8bit-bnb-eager 23.9
  5. 5 Yi-34B A10-24GB-150W / 4bit-awq-gemm-eager 22.3
  6. 6 Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemm-fa2 20.2
  7. 7 Mistral-7B-Instruct-v0.2 A10-24GB-150W / bfloat16-eager 18.4
  8. 8 Zephyr 7B Beta A10-24GB-150W / 8bit-bnb-eager 17.7
  9. 9 Gemma 7B A10-24GB-150W / 8bit-bnb-fa2 15.3
  10. 10 Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemm-fa2 15.2
  11. 11 Deci/DeciLM-7B A10-24GB-150W / 4bit-awq-gemm-eager 14.9
  12. 12 Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemm-fa2 14.5
  13. 13 TencentARC/Mistral_Pro_8B_v0.1 A10-24GB-150W / 4bit-awq-gemm-fa2 14.2
  14. 14 stabilityai/stablelm-2-12b A10-24GB-150W / 4bit-awq-gemm-eager 13.9
  15. 15 Yi 6B A10-24GB-150W / 4bit-awq-gemm-eager 13.6
  16. 16 google/recurrentgemma-9b A10-24GB-150W / 8bit-bnb-eager 13.5
  17. 17 Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-eager 13.4
  18. 18 Qwen/Qwen1.5-MoE-A2.7B A10-24GB-150W / 8bit-bnb-eager 12.4
  19. 19 Qwen/Qwen1.5-4B A10-24GB-150W / 4bit-awq-gemm-fa2 11.3
  20. 20 meta-llama/Llama-2-13b-hf A10-24GB-150W / 4bit-awq-gemm-eager 11.0
  21. 21 google/gemma-2-2b A10-24GB-150W / 8bit-bnb-fa2 10.3
  22. 22 LLaMA-13B A10-24GB-150W / 4bit-awq-gemm-eager 9.25
  23. 23 Qwen/Qwen1.5-1.8B A10-24GB-150W / 4bit-awq-gemm-fa2 9.12
  24. 24 meta-llama/Llama-2-7b-hf A10-24GB-150W / 4bit-awq-gemm-eager 8.72
  25. 25 Gemma 2B A10-24GB-150W / 8bit-bnb-eager 7.31
  26. 26 stabilityai/stablelm-3b-4e1t A10-24GB-150W / 4bit-awq-gemm-eager 7.26
  27. 27 Phi-1.5 A10-24GB-150W / 4bit-awq-gemm-eager 7.06
  28. 28 Qwen2-0.5B A10-24GB-150W / 8bit-bnb-fa2 7.06
  29. 29 google/recurrentgemma-2b A10-24GB-150W / 8bit-bnb-eager 6.94
  30. 30 GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-fa2 6.55
  31. 31 openai-community/gpt2 A10-24GB-150W / 8bit-bnb-eager 6.51
  32. 32 LLaMA-7B A10-24GB-150W / 4bit-awq-gemm-eager 6.39
  33. 33 GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-eager 6.34
  34. 34 facebook/opt-30b A10-24GB-150W / 4bit-awq-gemm-eager 6.20
  35. 35 GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-eager 5.99
  36. 36 EleutherAI/pythia-12b A10-24GB-150W / 4bit-awq-gemm-eager 5.93
  37. 37 EleutherAI/pythia-160m A10-24GB-150W / 4bit-awq-gemm-eager 5.62
  38. 38 openai-community/gpt2-large A10-24GB-150W / 8bit-bnb-eager 5.48
  39. 39 togethercomputer/RedPajama-INCITE-Base-3B-v1 A10-24GB-150W / 4bit-awq-gemm-eager 5.43
  40. 40 EleutherAI/gpt-neo-1.3B A10-24GB-150W / 4bit-awq-gemm-eager 5.33
  41. 41 stabilityai/stablelm-2-1_6b A10-24GB-150W / 4bit-awq-gemm-eager 5.22
  42. 42 Qwen/Qwen1.5-0.5B A10-24GB-150W / 8bit-bnb-fa2 5.14
  43. 43 EleutherAI/pythia-410m A10-24GB-150W / 4bit-awq-gemm-eager 5.11
  44. 44 Falcon-7B A10-24GB-150W / 8bit-bnb-eager 5.10
  45. 45 EleutherAI/gpt-neo-125m A10-24GB-150W / 8bit-bnb-eager 4.38
  46. 46 distilbert/distilgpt2 A10-24GB-150W / 8bit-bnb-eager 3.90

Source: Official LLM-Perf Leaderboard leaderboard

Source: Official LLM-Perf Leaderboard leaderboard

#EntryScoreSettingDateLinkSource
1 Qwen1.5-110B 29.6 A100-80GB-275W / 4bit-gptq-exllama-v2-eager Apr 2024 huggingface.co Official leaderboard
2 phi-3-mini 3.8B 27.2 A10-24GB-150W / 8bit-bnb-eager Apr 2024 huggingface.co Official leaderboard
3 Qwen1.5-32B 26.7 A10-24GB-150W / 4bit-awq-gemm-fa2 Feb 2024 huggingface.co Official leaderboard
4 meta-llama/Meta-Llama-3-8B-Instruct 23.9 A10-24GB-150W / 8bit-bnb-eager huggingface.co Official leaderboard
5 Yi-34B 22.3 A10-24GB-150W / 4bit-awq-gemm-eager Nov 2023 huggingface.co Official leaderboard
6 Qwen1.5-14B 20.2 A10-24GB-150W / 4bit-awq-gemm-fa2 Feb 2024 huggingface.co Official leaderboard
7 Mistral-7B-Instruct-v0.2 18.4 A10-24GB-150W / bfloat16-eager Dec 2023 huggingface.co Official leaderboard
8 Zephyr 7B Beta 17.7 A10-24GB-150W / 8bit-bnb-eager Oct 2023 huggingface.co Official leaderboard
9 Gemma 7B 15.3 A10-24GB-150W / 8bit-bnb-fa2 Feb 2024 huggingface.co Official leaderboard
10 Qwen1.5-7B 15.2 A10-24GB-150W / 4bit-awq-gemm-fa2 Feb 2024 huggingface.co Official leaderboard
11 Deci/DeciLM-7B 14.9 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
12 Mistral 7B v0.1 14.5 A10-24GB-150W / 4bit-awq-gemm-fa2 Sep 2023 huggingface.co Official leaderboard
13 TencentARC/Mistral_Pro_8B_v0.1 14.2 A10-24GB-150W / 4bit-awq-gemm-fa2 huggingface.co Official leaderboard
14 stabilityai/stablelm-2-12b 13.9 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
15 Yi 6B 13.6 A10-24GB-150W / 4bit-awq-gemm-eager Nov 2023 huggingface.co Official leaderboard
16 google/recurrentgemma-9b 13.5 A10-24GB-150W / 8bit-bnb-eager huggingface.co Official leaderboard
17 Llama 3-8B 13.4 A10-24GB-150W / 4bit-awq-gemm-eager Apr 2024 huggingface.co Official leaderboard
18 Qwen/Qwen1.5-MoE-A2.7B 12.4 A10-24GB-150W / 8bit-bnb-eager huggingface.co Official leaderboard
19 Qwen/Qwen1.5-4B 11.3 A10-24GB-150W / 4bit-awq-gemm-fa2 huggingface.co Official leaderboard
20 meta-llama/Llama-2-13b-hf 11.0 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
21 google/gemma-2-2b 10.3 A10-24GB-150W / 8bit-bnb-fa2 huggingface.co Official leaderboard
22 LLaMA-13B 9.25 A10-24GB-150W / 4bit-awq-gemm-eager Feb 2023 huggingface.co Official leaderboard
23 Qwen/Qwen1.5-1.8B 9.12 A10-24GB-150W / 4bit-awq-gemm-fa2 huggingface.co Official leaderboard
24 meta-llama/Llama-2-7b-hf 8.72 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
25 Gemma 2B 7.31 A10-24GB-150W / 8bit-bnb-eager Feb 2024 huggingface.co Official leaderboard
26 stabilityai/stablelm-3b-4e1t 7.26 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
27 Phi-1.5 7.06 A10-24GB-150W / 4bit-awq-gemm-eager Sep 2023 huggingface.co Official leaderboard
28 Qwen2-0.5B 7.06 A10-24GB-150W / 8bit-bnb-fa2 Jun 2024 huggingface.co Official leaderboard
29 google/recurrentgemma-2b 6.94 A10-24GB-150W / 8bit-bnb-eager huggingface.co Official leaderboard
30 GPT-J-6B 6.55 A10-24GB-150W / 4bit-awq-gemm-fa2 Aug 2021 huggingface.co Official leaderboard
31 openai-community/gpt2 6.51 A10-24GB-150W / 8bit-bnb-eager huggingface.co Official leaderboard
32 LLaMA-7B 6.39 A10-24GB-150W / 4bit-awq-gemm-eager Feb 2023 huggingface.co Official leaderboard
33 GPT-Neo-2.7B 6.34 A10-24GB-150W / 4bit-awq-gemm-eager Mar 2021 huggingface.co Official leaderboard
34 facebook/opt-30b 6.20 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
35 GPT-NeoX-20B 5.99 A10-24GB-150W / 4bit-awq-gemm-eager Apr 2022 huggingface.co Official leaderboard
36 EleutherAI/pythia-12b 5.93 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
37 EleutherAI/pythia-160m 5.62 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
38 openai-community/gpt2-large 5.48 A10-24GB-150W / 8bit-bnb-eager huggingface.co Official leaderboard
39 togethercomputer/RedPajama-INCITE-Base-3B-v1 5.43 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
40 EleutherAI/gpt-neo-1.3B 5.33 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
41 stabilityai/stablelm-2-1_6b 5.22 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
42 Qwen/Qwen1.5-0.5B 5.14 A10-24GB-150W / 8bit-bnb-fa2 huggingface.co Official leaderboard
43 EleutherAI/pythia-410m 5.11 A10-24GB-150W / 4bit-awq-gemm-eager huggingface.co Official leaderboard
44 Falcon-7B 5.10 A10-24GB-150W / 8bit-bnb-eager Apr 2023 huggingface.co Official leaderboard
45 EleutherAI/gpt-neo-125m 4.38 A10-24GB-150W / 8bit-bnb-eager huggingface.co Official leaderboard
46 distilbert/distilgpt2 3.90 A10-24GB-150W / 8bit-bnb-eager huggingface.co Official leaderboard
#ModelScoreSettingDateLinkSource
1 Qwen1.5-110B 29.6 A100-80GB-275W / 4bit-gptq-exllama-v2-eager Apr 2024 huggingface.co Official leaderboard
2 phi-3-mini 3.8B 27.2 A10-24GB-150W / 8bit-bnb-eager Apr 2024 huggingface.co Official leaderboard
3 Qwen1.5-32B 26.7 A10-24GB-150W / 4bit-awq-gemm-fa2 Feb 2024 huggingface.co Official leaderboard
4 Yi-34B 22.3 A10-24GB-150W / 4bit-awq-gemm-eager Nov 2023 huggingface.co Official leaderboard
5 Qwen1.5-14B 20.2 A10-24GB-150W / 4bit-awq-gemm-fa2 Feb 2024 huggingface.co Official leaderboard
6 Mistral-7B-Instruct-v0.2 18.4 A10-24GB-150W / bfloat16-eager Dec 2023 huggingface.co Official leaderboard
7 Zephyr 7B Beta 17.7 A10-24GB-150W / 8bit-bnb-eager Oct 2023 huggingface.co Official leaderboard
8 Gemma 7B 15.3 A10-24GB-150W / 8bit-bnb-fa2 Feb 2024 huggingface.co Official leaderboard
9 Qwen1.5-7B 15.2 A10-24GB-150W / 4bit-awq-gemm-fa2 Feb 2024 huggingface.co Official leaderboard
10 Mistral 7B v0.1 14.5 A10-24GB-150W / 4bit-awq-gemm-fa2 Sep 2023 huggingface.co Official leaderboard
11 Yi 6B 13.6 A10-24GB-150W / 4bit-awq-gemm-eager Nov 2023 huggingface.co Official leaderboard
12 Llama 3-8B 13.4 A10-24GB-150W / 4bit-awq-gemm-eager Apr 2024 huggingface.co Official leaderboard
13 LLaMA-13B 9.25 A10-24GB-150W / 4bit-awq-gemm-eager Feb 2023 huggingface.co Official leaderboard
14 Gemma 2B 7.31 A10-24GB-150W / 8bit-bnb-eager Feb 2024 huggingface.co Official leaderboard
15 Phi-1.5 7.06 A10-24GB-150W / 4bit-awq-gemm-eager Sep 2023 huggingface.co Official leaderboard
16 Qwen2-0.5B 7.06 A10-24GB-150W / 8bit-bnb-fa2 Jun 2024 huggingface.co Official leaderboard
17 GPT-J-6B 6.55 A10-24GB-150W / 4bit-awq-gemm-fa2 Aug 2021 huggingface.co Official leaderboard
18 LLaMA-7B 6.39 A10-24GB-150W / 4bit-awq-gemm-eager Feb 2023 huggingface.co Official leaderboard
19 GPT-Neo-2.7B 6.34 A10-24GB-150W / 4bit-awq-gemm-eager Mar 2021 huggingface.co Official leaderboard
20 GPT-NeoX-20B 5.99 A10-24GB-150W / 4bit-awq-gemm-eager Apr 2022 huggingface.co Official leaderboard
21 Falcon-7B 5.10 A10-24GB-150W / 8bit-bnb-eager Apr 2023 huggingface.co Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Qwen1.5-110B
A100-80GB-2…
29.6
phi-3-mini 3.8B
A10-24GB-15…
27.2
Qwen1.5-32B
A10-24GB-15…
26.7
Yi-34B
A10-24GB-15…
22.3
Qwen1.5-14B
A10-24GB-15…
20.2
Mistral-7B-Instruct-v0.2
A10-24GB-15…
18.4
Zephyr 7B Beta
A10-24GB-15…
17.7
Gemma 7B
A10-24GB-15…
15.3
Qwen1.5-7B
A10-24GB-15…
15.2
Mistral 7B v0.1
A10-24GB-15…
14.5
Yi 6B
A10-24GB-15…
13.6
Llama 3-8B
A10-24GB-15…
13.4

Source: Official LLM-Perf Leaderboard leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-bnb-fa2 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-bnb-eager 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-bnb-sdpa 29.6
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-bnb-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-bnb-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / float16-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / bfloat16-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemm-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemm-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / float32-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / float16-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemv-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / bfloat16-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemv-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-exllama-v1-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 8bit-bnb-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-bnb-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-gemm-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / float16-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / bfloat16-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-gemv-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-exllama-v1-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-eager-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-sdpa-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-sdpa-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-eager-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-eager-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-sdpa-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-eager 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-eager 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-eager 27.2
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemm-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemm-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemm-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemv-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemv-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v2-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v1-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemv-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / 8bit-bnb-eager 26.7
Qwen1.5-32B A100-80GB-275W / 8bit-bnb-fa2 26.7
Qwen1.5-32B A100-80GB-275W / 8bit-bnb-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-bnb-fa2 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-bnb-eager 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-bnb-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / bfloat16-sdpa 26.7
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-bnb-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-bnb-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / float16-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / bfloat16-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemm-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemm-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / float32-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / float16-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemv-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / bfloat16-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemv-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-exllama-v1-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 8bit-bnb-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-bnb-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-gemm-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / float16-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / bfloat16-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-gemv-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-exllama-v1-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-eager-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-sdpa-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-sdpa-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-eager-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-eager-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-sdpa-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-eager 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-eager 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-eager 26.0
Yi-34B A10-24GB-150W / 4bit-awq-gemm-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-eager 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-eager 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 22.3
Yi-34B A100-80GB-275W / 8bit-bnb-fa2 22.3
Yi-34B A100-80GB-275W / 8bit-bnb-sdpa 22.3
Yi-34B A100-80GB-275W / 8bit-bnb-eager 22.3
Yi-34B A100-80GB-275W / 4bit-bnb-eager 22.3
Yi-34B A100-80GB-275W / 4bit-bnb-fa2 22.3
Yi-34B A100-80GB-275W / 4bit-bnb-sdpa 22.3
Yi-34B A100-80GB-275W / bfloat16-eager 22.3
Yi-34B A100-80GB-275W / float16-eager 22.3
Yi-34B A100-80GB-275W / float16-sdpa 22.3
Yi-34B A100-80GB-275W / bfloat16-sdpa 22.3
Yi-34B A100-80GB-275W / bfloat16-fa2 22.3
Yi-34B A100-80GB-275W / float16-fa2 22.3
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemm-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemm-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemm-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-eager 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-eager 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemv-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemv-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v1-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v2-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemv-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B A100-80GB-275W / 8bit-bnb-eager 20.2
Qwen1.5-14B A100-80GB-275W / 8bit-bnb-fa2 20.2
Qwen1.5-14B A100-80GB-275W / 8bit-bnb-sdpa 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-bnb-eager 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-bnb-fa2 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-bnb-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-bnb-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-gemm-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-gemm-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-gemv-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-gemv-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-exllama-v1-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-exllama-v2-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / float32-eager-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / float16-eager-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 20.2
Mistral-7B-Instruct-v0.2 A10-24GB-150W / bfloat16-eager 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / float16-fa2 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / float16-eager 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / bfloat16-fa2 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / bfloat16-sdpa 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / float16-sdpa 18.4
Mistral-7B-Instruct-v0.2 T4-16GB-70W / 4bit-gptq-exllama-v2-eager 18.4
Mistral-7B-Instruct-v0.2 T4-16GB-70W / 4bit-gptq-exllama-v1-eager 18.4
Mistral-7B-Instruct-v0.2 T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 18.4
Mistral-7B-Instruct-v0.2 T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 18.4
Zephyr 7B Beta A10-24GB-150W / 8bit-bnb-eager 17.7
Zephyr 7B Beta A10-24GB-150W / 8bit-bnb-fa2 17.7
Zephyr 7B Beta A10-24GB-150W / 8bit-bnb-sdpa 17.7
Zephyr 7B Beta A10-24GB-150W / 4bit-bnb-fa2 17.7
Zephyr 7B Beta A10-24GB-150W / 4bit-bnb-eager 17.7
Zephyr 7B Beta A10-24GB-150W / float16-fa2 17.7
Zephyr 7B Beta A10-24GB-150W / bfloat16-fa2 17.7
Zephyr 7B Beta A10-24GB-150W / 4bit-bnb-sdpa 17.7
Zephyr 7B Beta A10-24GB-150W / bfloat16-eager 17.7
Zephyr 7B Beta A10-24GB-150W / float16-eager 17.7
Zephyr 7B Beta A10-24GB-150W / bfloat16-sdpa 17.7
Zephyr 7B Beta A10-24GB-150W / float16-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-bnb-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-bnb-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / 8bit-bnb-eager 17.7
Zephyr 7B Beta T4-16GB-70W / bfloat16-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / bfloat16-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 8bit-bnb-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / float16-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / float16-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-gptq-exllama-v1-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-gptq-exllama-v2-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / float32-eager 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / float32-sdpa 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / float16-eager 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / float16-sdpa 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / bfloat16-eager 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / bfloat16-sdpa 17.7
Gemma 7B A10-24GB-150W / 8bit-bnb-fa2 15.3
Gemma 7B A10-24GB-150W / 8bit-bnb-eager 15.3
Gemma 7B A10-24GB-150W / 8bit-bnb-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-bnb-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-bnb-fa2 15.3
Gemma 7B A10-24GB-150W / float16-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-bnb-sdpa 15.3
Gemma 7B A10-24GB-150W / bfloat16-fa2 15.3
Gemma 7B A10-24GB-150W / float16-fa2 15.3
Gemma 7B A10-24GB-150W / bfloat16-fa2 15.3
Gemma 7B A10-24GB-150W / bfloat16-eager 15.3
Gemma 7B A10-24GB-150W / bfloat16-eager 15.3
Gemma 7B A10-24GB-150W / float16-eager 15.3
Gemma 7B A10-24GB-150W / float16-eager 15.3
Gemma 7B A10-24GB-150W / bfloat16-sdpa 15.3
Gemma 7B A10-24GB-150W / bfloat16-sdpa 15.3
Gemma 7B A10-24GB-150W / float16-sdpa 15.3
Gemma 7B A10-24GB-150W / float16-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemm-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemm-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemm-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemv-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemv-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemv-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 15.3
Gemma 7B T4-16GB-70W / 8bit-bnb-eager 15.3
Gemma 7B T4-16GB-70W / 8bit-bnb-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-bnb-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-bnb-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-gemm-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-gemv-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-gemm-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-gemv-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 15.3
Gemma 7B Intel-Xeon-SPR-385W / float32-eager-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / float16-eager-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 15.3
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemm-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemm-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemm-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-eager 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-eager 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / float16-fa2 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-fa2 15.2
Qwen1.5-7B A10-24GB-150W / float16-fa2 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-fa2 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-eager 15.2
Qwen1.5-7B A10-24GB-150W / float16-eager 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / float16-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-eager 15.2
Qwen1.5-7B A10-24GB-150W / float16-eager 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / float16-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemv-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemv-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemv-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / 8bit-bnb-fa2 15.2
Qwen1.5-7B A100-80GB-275W / 8bit-bnb-eager 15.2
Qwen1.5-7B A100-80GB-275W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-bnb-fa2 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-bnb-eager 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / bfloat16-eager 15.2
Qwen1.5-7B A100-80GB-275W / float16-eager 15.2
Qwen1.5-7B A100-80GB-275W / float32-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / bfloat16-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / float16-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 8bit-bnb-eager 15.2
Qwen1.5-7B T4-16GB-70W / 8bit-bnb-eager 15.2
Qwen1.5-7B T4-16GB-70W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-bnb-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-bnb-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / bfloat16-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / bfloat16-eager 15.2
Qwen1.5-7B T4-16GB-70W / float16-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / float16-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemm-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemm-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemm-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemm-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemv-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemv-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemv-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemv-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / float32-eager-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / float16-eager-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 15.2
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemm-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemm-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemm-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v1-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v2-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 8bit-bnb-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 8bit-bnb-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 8bit-bnb-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-bnb-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-bnb-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-bnb-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / bfloat16-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / float16-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / bfloat16-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / float16-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemv-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / bfloat16-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / float16-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v2-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemv-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v1-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v1-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v2-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemv-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / 8bit-bnb-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 8bit-bnb-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-bnb-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-bnb-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / bfloat16-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / bfloat16-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / float16-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / float16-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-gptq-exllama-v1-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-gptq-exllama-v2-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / float32-eager-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / float16-eager-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / float32-sdpa-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / float16-sdpa-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 14.5
Yi 6B A10-24GB-150W / 4bit-awq-gemm-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-eager 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-sdpa 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-fa2 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-eager 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-fa2 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-eager 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-eager 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-fa2 13.6
Yi 6B A10-24GB-150W / float16-fa2 13.6
Yi 6B A10-24GB-150W / bfloat16-fa2 13.6
Yi 6B A10-24GB-150W / bfloat16-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-sdpa 13.6
Yi 6B A10-24GB-150W / float16-eager 13.6
Yi 6B A10-24GB-150W / bfloat16-sdpa 13.6
Yi 6B A10-24GB-150W / float16-sdpa 13.6
Yi 6B A10-24GB-150W / bfloat16-fa2 13.6
Yi 6B A10-24GB-150W / float16-fa2 13.6
Yi 6B A10-24GB-150W / float16-eager 13.6
Yi 6B A10-24GB-150W / bfloat16-eager 13.6
Yi 6B A10-24GB-150W / float16-sdpa 13.6
Yi 6B A10-24GB-150W / bfloat16-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-eager 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 13.6
Yi 6B A100-80GB-275W / 8bit-bnb-fa2 13.6
Yi 6B A100-80GB-275W / 8bit-bnb-sdpa 13.6
Yi 6B A100-80GB-275W / 8bit-bnb-eager 13.6
Yi 6B A100-80GB-275W / 4bit-bnb-eager 13.6
Yi 6B A100-80GB-275W / 4bit-bnb-fa2 13.6
Yi 6B A100-80GB-275W / 4bit-bnb-sdpa 13.6
Yi 6B A100-80GB-275W / float16-eager 13.6
Yi 6B A100-80GB-275W / bfloat16-eager 13.6
Yi 6B A100-80GB-275W / float16-sdpa 13.6
Yi 6B A100-80GB-275W / float32-eager 13.6
Yi 6B A100-80GB-275W / bfloat16-sdpa 13.6
Yi 6B A100-80GB-275W / float16-fa2 13.6
Yi 6B A100-80GB-275W / bfloat16-fa2 13.6
Yi 6B A100-80GB-275W / float32-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B T4-16GB-70W / 8bit-bnb-eager 13.6
Yi 6B T4-16GB-70W / 8bit-bnb-eager 13.6
Yi 6B T4-16GB-70W / 8bit-bnb-sdpa 13.6
Yi 6B T4-16GB-70W / 8bit-bnb-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-bnb-eager 13.6
Yi 6B T4-16GB-70W / 4bit-bnb-eager 13.6
Yi 6B T4-16GB-70W / 4bit-bnb-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-bnb-sdpa 13.6
Yi 6B T4-16GB-70W / bfloat16-sdpa 13.6
Yi 6B T4-16GB-70W / bfloat16-eager 13.6
Yi 6B T4-16GB-70W / bfloat16-eager 13.6
Yi 6B T4-16GB-70W / bfloat16-sdpa 13.6
Yi 6B T4-16GB-70W / float16-sdpa 13.6
Yi 6B T4-16GB-70W / float16-eager 13.6
Yi 6B T4-16GB-70W / float16-sdpa 13.6
Yi 6B T4-16GB-70W / float16-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemm-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemm-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemm-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemm-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v1-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemv-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemv-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v2-eager 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v1-eager 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v2-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemv-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemv-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-eager-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-eager-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 13.6
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-eager 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-eager 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-fa2 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-sdpa 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-sdpa 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-sdpa 13.4
Llama 3-8B A10-24GB-150W / float16-fa2 13.4
Llama 3-8B A10-24GB-150W / bfloat16-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-fa2 13.4
Llama 3-8B A10-24GB-150W / bfloat16-eager 13.4
Llama 3-8B A10-24GB-150W / float16-eager 13.4
Llama 3-8B A10-24GB-150W / bfloat16-sdpa 13.4
Llama 3-8B A10-24GB-150W / float16-sdpa 13.4
Llama 3-8B A10-24GB-150W / bfloat16-eager 13.4
Llama 3-8B A10-24GB-150W / float16-eager 13.4
Llama 3-8B A10-24GB-150W / bfloat16-fa2 13.4
Llama 3-8B A10-24GB-150W / float16-fa2 13.4
Llama 3-8B A10-24GB-150W / bfloat16-sdpa 13.4
Llama 3-8B A10-24GB-150W / float16-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 13.4
Llama 3-8B T4-16GB-70W / 8bit-bnb-eager 13.4
Llama 3-8B T4-16GB-70W / 8bit-bnb-sdpa 13.4
Llama 3-8B T4-16GB-70W / 8bit-bnb-eager 13.4
Llama 3-8B T4-16GB-70W / 8bit-bnb-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-bnb-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-bnb-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-bnb-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-bnb-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-gemm-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-gemm-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-gemv-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-exllama-v1-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-exllama-v2-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-gemv-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float16-eager-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float32-eager-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float32-eager 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float32-sdpa 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float16-eager 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float16-sdpa 13.4
Llama 3-8B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / bfloat16-eager 13.4
Llama 3-8B Intel-Xeon-SPR-385W / bfloat16-sdpa 13.4
LLaMA-13B A10-24GB-150W / 4bit-awq-gemm-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemm-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemm-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 9.25
LLaMA-13B A10-24GB-150W / 8bit-bnb-eager 9.25
LLaMA-13B A10-24GB-150W / 8bit-bnb-fa2 9.25
LLaMA-13B A10-24GB-150W / 8bit-bnb-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-bnb-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-bnb-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-bnb-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemv-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemv-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v2-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v1-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemv-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 9.25
LLaMA-13B T4-16GB-70W / 8bit-bnb-eager 9.25
LLaMA-13B T4-16GB-70W / 8bit-bnb-sdpa 9.25
LLaMA-13B T4-16GB-70W / 4bit-bnb-eager 9.25
LLaMA-13B T4-16GB-70W / 4bit-bnb-sdpa 9.25
LLaMA-13B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 9.25
LLaMA-13B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 9.25
LLaMA-13B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 9.25
LLaMA-13B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 9.25
LLaMA-13B Intel-Xeon-SPR-385W / float32-eager-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / float16-eager-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 9.25
Gemma 2B A10-24GB-150W / 8bit-bnb-eager 7.31
Gemma 2B A10-24GB-150W / 8bit-bnb-fa2 7.31
Gemma 2B A10-24GB-150W / 8bit-bnb-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-bnb-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-bnb-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-bnb-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemm-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemm-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemm-sdpa 7.31
Gemma 2B A10-24GB-150W / float32-sdpa 7.31
Gemma 2B A10-24GB-150W / float32-sdpa 7.31
Gemma 2B A10-24GB-150W / float32-eager 7.31
Gemma 2B A10-24GB-150W / float32-eager 7.31
Gemma 2B A10-24GB-150W / bfloat16-eager 7.31
Gemma 2B A10-24GB-150W / float16-eager 7.31
Gemma 2B A10-24GB-150W / bfloat16-eager 7.31
Gemma 2B A10-24GB-150W / float16-eager 7.31
Gemma 2B A10-24GB-150W / bfloat16-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemv-eager 7.31
Gemma 2B A10-24GB-150W / float16-fa2 7.31
Gemma 2B A10-24GB-150W / float16-fa2 7.31
Gemma 2B A10-24GB-150W / bfloat16-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v1-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemv-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v2-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 7.31
Gemma 2B A10-24GB-150W / float16-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 7.31
Gemma 2B A10-24GB-150W / float16-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 7.31
Gemma 2B A10-24GB-150W / bfloat16-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemv-sdpa 7.31
Gemma 2B A10-24GB-150W / bfloat16-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 7.31
Gemma 2B T4-16GB-70W / 8bit-bnb-eager 7.31
Gemma 2B T4-16GB-70W / 8bit-bnb-sdpa 7.31
Gemma 2B T4-16GB-70W / float32-sdpa 7.31
Gemma 2B T4-16GB-70W / float32-sdpa 7.31
Gemma 2B T4-16GB-70W / float32-eager 7.31
Gemma 2B T4-16GB-70W / float32-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-bnb-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-bnb-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-gemm-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-exllama-v1-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-gemm-sdpa 7.31
Gemma 2B T4-16GB-70W / float16-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-exllama-v2-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-gemv-eager 7.31
Gemma 2B T4-16GB-70W / float16-eager 7.31
Gemma 2B T4-16GB-70W / bfloat16-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 7.31
Gemma 2B T4-16GB-70W / bfloat16-eager 7.31
Gemma 2B T4-16GB-70W / bfloat16-sdpa 7.31
Gemma 2B T4-16GB-70W / float16-sdpa 7.31
Gemma 2B T4-16GB-70W / bfloat16-sdpa 7.31
Gemma 2B T4-16GB-70W / float16-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-gemv-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 7.31
Gemma 2B Intel-Xeon-SPR-385W / float16-eager-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / float32-eager-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 7.31
Phi-1.5 A10-24GB-150W / 4bit-awq-gemm-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemm-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemm-sdpa 7.06
Phi-1.5 A10-24GB-150W / 8bit-bnb-eager 7.06
Phi-1.5 A10-24GB-150W / 8bit-bnb-sdpa 7.06
Phi-1.5 A10-24GB-150W / 8bit-bnb-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v2-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v1-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-bnb-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-bnb-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-bnb-sdpa 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemv-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v2-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemv-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v1-eager 7.06
Phi-1.5 A10-24GB-150W / bfloat16-eager 7.06
Phi-1.5 A10-24GB-150W / float16-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v2-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v1-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemv-sdpa 7.06
Phi-1.5 A10-24GB-150W / bfloat16-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 7.06
Phi-1.5 A10-24GB-150W / float32-eager 7.06
Phi-1.5 A10-24GB-150W / float16-fa2 7.06
Phi-1.5 A10-24GB-150W / bfloat16-sdpa 7.06
Phi-1.5 A10-24GB-150W / float16-sdpa 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 7.06
Phi-1.5 A10-24GB-150W / float32-sdpa 7.06
Phi-1.5 A100-80GB-275W / float32-eager 7.06
Phi-1.5 A100-80GB-275W / bfloat16-eager 7.06
Phi-1.5 A100-80GB-275W / float16-eager 7.06
Phi-1.5 A100-80GB-275W / float16-fa2 7.06
Phi-1.5 A100-80GB-275W / bfloat16-fa2 7.06
Phi-1.5 A100-80GB-275W / float16-sdpa 7.06
Phi-1.5 A100-80GB-275W / bfloat16-sdpa 7.06
Phi-1.5 A100-80GB-275W / float32-sdpa 7.06
Phi-1.5 T4-16GB-70W / 8bit-bnb-eager 7.06
Phi-1.5 T4-16GB-70W / 8bit-bnb-sdpa 7.06
Phi-1.5 T4-16GB-70W / 4bit-bnb-eager 7.06
Phi-1.5 T4-16GB-70W / 4bit-bnb-sdpa 7.06
Phi-1.5 T4-16GB-70W / float32-sdpa 7.06
Phi-1.5 T4-16GB-70W / float32-sdpa 7.06
Phi-1.5 T4-16GB-70W / float32-eager 7.06
Phi-1.5 T4-16GB-70W / float32-eager 7.06
Phi-1.5 T4-16GB-70W / float16-eager 7.06
Phi-1.5 T4-16GB-70W / 4bit-gptq-exllama-v2-eager 7.06
Phi-1.5 T4-16GB-70W / float16-eager 7.06
Phi-1.5 T4-16GB-70W / bfloat16-eager 7.06
Phi-1.5 T4-16GB-70W / 4bit-gptq-exllama-v1-eager 7.06
Phi-1.5 T4-16GB-70W / bfloat16-eager 7.06
Phi-1.5 T4-16GB-70W / bfloat16-sdpa 7.06
Phi-1.5 T4-16GB-70W / bfloat16-sdpa 7.06
Phi-1.5 T4-16GB-70W / float16-sdpa 7.06
Phi-1.5 T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 7.06
Phi-1.5 T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 7.06
Phi-1.5 T4-16GB-70W / float16-sdpa 7.06
Phi-1.5 Intel-Xeon-SPR-385W / float32-eager-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / float32-sdpa-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / float16-eager-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / float16-sdpa-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 7.06
Qwen2-0.5B A10-24GB-150W / 8bit-bnb-fa2 7.06
Qwen2-0.5B A10-24GB-150W / 8bit-bnb-eager 7.06
Qwen2-0.5B A10-24GB-150W / 8bit-bnb-sdpa 7.06
Qwen2-0.5B A10-24GB-150W / 4bit-bnb-fa2 7.06
Qwen2-0.5B A10-24GB-150W / 4bit-bnb-eager 7.06
Qwen2-0.5B A10-24GB-150W / 4bit-bnb-sdpa 7.06
Qwen2-0.5B A10-24GB-150W / float16-fa2 7.06
Qwen2-0.5B A10-24GB-150W / float16-eager 7.06
Qwen2-0.5B A10-24GB-150W / bfloat16-eager 7.06
Qwen2-0.5B A10-24GB-150W / bfloat16-fa2 7.06
Qwen2-0.5B A10-24GB-150W / float16-sdpa 7.06
Qwen2-0.5B A10-24GB-150W / float32-eager 7.06
Qwen2-0.5B A10-24GB-150W / bfloat16-sdpa 7.06
Qwen2-0.5B A10-24GB-150W / float32-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / 8bit-bnb-eager 7.06
Qwen2-0.5B T4-16GB-70W / 8bit-bnb-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-bnb-eager 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-bnb-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / bfloat16-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / float16-eager 7.06
Qwen2-0.5B T4-16GB-70W / bfloat16-eager 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 7.06
Qwen2-0.5B T4-16GB-70W / float32-eager 7.06
Qwen2-0.5B T4-16GB-70W / float16-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / float32-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / float16-eager 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / float16-sdpa 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / float32-sdpa 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / bfloat16-eager 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / float32-eager 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / bfloat16-sdpa 7.06
GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B A10-24GB-150W / 8bit-bnb-fa2 6.55
GPT-J-6B A10-24GB-150W / 8bit-bnb-fa2 6.55
GPT-J-6B A10-24GB-150W / 8bit-bnb-eager 6.55
GPT-J-6B A10-24GB-150W / 8bit-bnb-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-bnb-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-bnb-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-bnb-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-bnb-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-fa2 6.55
GPT-J-6B A10-24GB-150W / float16-fa2 6.55
GPT-J-6B A10-24GB-150W / bfloat16-fa2 6.55
GPT-J-6B A10-24GB-150W / float16-fa2 6.55
GPT-J-6B A10-24GB-150W / bfloat16-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemv-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemv-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.55
GPT-J-6B A10-24GB-150W / bfloat16-eager 6.55
GPT-J-6B A10-24GB-150W / float16-eager 6.55
GPT-J-6B A10-24GB-150W / float16-eager 6.55
GPT-J-6B A10-24GB-150W / bfloat16-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemv-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemv-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 6.55
GPT-J-6B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 6.55
GPT-J-6B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B A100-80GB-275W / 8bit-bnb-fa2 6.55
GPT-J-6B A100-80GB-275W / 8bit-bnb-eager 6.55
GPT-J-6B A100-80GB-275W / 4bit-bnb-fa2 6.55
GPT-J-6B A100-80GB-275W / 4bit-bnb-eager 6.55
GPT-J-6B A100-80GB-275W / bfloat16-fa2 6.55
GPT-J-6B A100-80GB-275W / float16-fa2 6.55
GPT-J-6B A100-80GB-275W / float16-eager 6.55
GPT-J-6B A100-80GB-275W / bfloat16-eager 6.55
GPT-J-6B A100-80GB-275W / float32-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B T4-16GB-70W / 8bit-bnb-eager 6.55
GPT-J-6B T4-16GB-70W / 8bit-bnb-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-bnb-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-bnb-eager 6.55
GPT-J-6B T4-16GB-70W / bfloat16-eager 6.55
GPT-J-6B T4-16GB-70W / bfloat16-eager 6.55
GPT-J-6B T4-16GB-70W / float16-eager 6.55
GPT-J-6B T4-16GB-70W / float16-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-gemm-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-gemm-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-gemv-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-gemv-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-exllama-v2-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-exllama-v1-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-exllama-v2-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-exllama-v1-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float16-eager-pytorch 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float32-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float32-eager-pytorch 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float32-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float16-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float16-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 6.55
GPT-J-6B Intel-Xeon-SPR-385W / bfloat16-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / bfloat16-eager 6.55
LLaMA-7B A10-24GB-150W / 4bit-awq-gemm-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemm-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemm-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 6.39
LLaMA-7B A10-24GB-150W / 8bit-bnb-eager 6.39
LLaMA-7B A10-24GB-150W / 8bit-bnb-fa2 6.39
LLaMA-7B A10-24GB-150W / 8bit-bnb-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-bnb-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-bnb-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-bnb-sdpa 6.39
LLaMA-7B A10-24GB-150W / float16-fa2 6.39
LLaMA-7B A10-24GB-150W / bfloat16-fa2 6.39
LLaMA-7B A10-24GB-150W / bfloat16-eager 6.39
LLaMA-7B A10-24GB-150W / float16-eager 6.39
LLaMA-7B A10-24GB-150W / float16-fa2 6.39
LLaMA-7B A10-24GB-150W / bfloat16-fa2 6.39
LLaMA-7B A10-24GB-150W / bfloat16-sdpa 6.39
LLaMA-7B A10-24GB-150W / float16-sdpa 6.39
LLaMA-7B A10-24GB-150W / bfloat16-eager 6.39
LLaMA-7B A10-24GB-150W / float16-eager 6.39
LLaMA-7B A10-24GB-150W / bfloat16-sdpa 6.39
LLaMA-7B A10-24GB-150W / float16-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemv-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemv-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemv-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 6.39
LLaMA-7B T4-16GB-70W / 8bit-bnb-eager 6.39
LLaMA-7B T4-16GB-70W / 8bit-bnb-sdpa 6.39
LLaMA-7B T4-16GB-70W / 4bit-bnb-eager 6.39
LLaMA-7B T4-16GB-70W / 4bit-bnb-sdpa 6.39
LLaMA-7B T4-16GB-70W / bfloat16-sdpa 6.39
LLaMA-7B T4-16GB-70W / bfloat16-sdpa 6.39
LLaMA-7B T4-16GB-70W / bfloat16-eager 6.39
LLaMA-7B T4-16GB-70W / bfloat16-eager 6.39
LLaMA-7B T4-16GB-70W / float16-eager 6.39
LLaMA-7B T4-16GB-70W / float16-sdpa 6.39
LLaMA-7B T4-16GB-70W / float16-eager 6.39
LLaMA-7B T4-16GB-70W / float16-sdpa 6.39
LLaMA-7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.39
LLaMA-7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.39
LLaMA-7B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 6.39
LLaMA-7B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 6.39
LLaMA-7B Intel-Xeon-SPR-385W / float32-eager-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / float16-eager-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 6.39
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 8bit-bnb-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 8bit-bnb-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-bnb-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-bnb-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float32-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float32-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / bfloat16-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float16-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float16-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemv-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemv-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / bfloat16-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float16-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / bfloat16-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemv-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / bfloat16-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemv-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / float16-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / 8bit-bnb-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-bnb-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / bfloat16-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / float16-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / float32-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / float16-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / bfloat16-fa2 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / float32-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / float32-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-gemm-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-gemm-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / float16-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / float16-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / bfloat16-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / bfloat16-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-gemv-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-gemv-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float16-eager-pytorch 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float16-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float16-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float32-eager-pytorch 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float32-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float32-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / bfloat16-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / bfloat16-eager 6.34
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / 8bit-bnb-eager 5.99
GPT-NeoX-20B A100-80GB-275W / 8bit-bnb-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-bnb-eager 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-bnb-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / float32-eager 5.99
GPT-NeoX-20B A100-80GB-275W / float16-eager 5.99
GPT-NeoX-20B A100-80GB-275W / bfloat16-eager 5.99
GPT-NeoX-20B A100-80GB-275W / float16-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / bfloat16-fa2 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-bnb-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-bnb-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-bnb-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-bnb-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemm-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemm-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemv-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemv-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemm-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemv-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemm-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemv-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v1-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v1-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v2-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v2-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / float16-eager-pytorch 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / float16-sdpa 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / float16-eager 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / bfloat16-eager 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / bfloat16-sdpa 5.99
Falcon-7B A10-24GB-150W / 8bit-bnb-eager 5.10
Falcon-7B A10-24GB-150W / float16-eager 5.10
Falcon-7B A10-24GB-150W / bfloat16-eager 5.10
Falcon-7B A10-24GB-150W / 4bit-bnb-eager 5.10
Falcon-7B T4-16GB-70W / 4bit-bnb-eager 5.10
Falcon-7B T4-16GB-70W / 8bit-bnb-eager 5.10
Falcon-7B T4-16GB-70W / bfloat16-eager 5.10
Falcon-7B T4-16GB-70W / float16-eager 5.10

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Qwen1.5-110B
A100-80GB-2…
29.6
phi-3-mini 3.8B
A10-24GB-15…
27.2
Qwen1.5-32B
A10-24GB-15…
26.7
Yi-34B
A10-24GB-15…
22.3
Qwen1.5-14B
A10-24GB-15…
20.2
Mistral-7B-Instruct-v0.2
A10-24GB-15…
18.4
Zephyr 7B Beta
A10-24GB-15…
17.7
Gemma 7B
A10-24GB-15…
15.3
Qwen1.5-7B
A10-24GB-15…
15.2
Mistral 7B v0.1
A10-24GB-15…
14.5
Yi 6B
A10-24GB-15…
13.6
Llama 3-8B
A10-24GB-15…
13.4

Source: Official LLM-Perf Leaderboard leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-bnb-fa2 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-bnb-eager 29.6
Qwen1.5-110B A100-80GB-275W / 4bit-bnb-sdpa 29.6
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-bnb-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-bnb-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / float16-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / bfloat16-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemm-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemm-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / float32-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / float16-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemv-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / bfloat16-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemv-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-exllama-v1-eager 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 27.2
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 8bit-bnb-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-bnb-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-gemm-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / float16-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / bfloat16-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-gemv-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-exllama-v1-eager 27.2
phi-3-mini 3.8B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-eager-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-sdpa-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-sdpa-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-eager-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-eager-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-sdpa-onnxruntime 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-eager 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-eager 27.2
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-eager 27.2
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemm-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemm-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemm-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-bnb-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemv-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemv-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v2-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v1-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemv-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 26.7
Qwen1.5-32B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / 8bit-bnb-eager 26.7
Qwen1.5-32B A100-80GB-275W / 8bit-bnb-fa2 26.7
Qwen1.5-32B A100-80GB-275W / 8bit-bnb-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-bnb-fa2 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-bnb-eager 26.7
Qwen1.5-32B A100-80GB-275W / 4bit-bnb-sdpa 26.7
Qwen1.5-32B A100-80GB-275W / bfloat16-sdpa 26.7
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-bnb-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-bnb-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / float16-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / bfloat16-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemm-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemm-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / float32-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / float16-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemv-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / bfloat16-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-gemv-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-exllama-v1-eager 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 26.0
phi-3-mini 3.8B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 8bit-bnb-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-bnb-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-gemm-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / float16-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / bfloat16-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-gemv-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-awq-exllama-v1-eager 26.0
phi-3-mini 3.8B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-eager-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-sdpa-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-sdpa-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-eager-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-eager-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-sdpa-onnxruntime 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float32-eager 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / float16-eager 26.0
phi-3-mini 3.8B Intel-Xeon-SPR-385W / bfloat16-eager 26.0
Yi-34B A10-24GB-150W / 4bit-awq-gemm-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-eager 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-eager 22.3
Yi-34B A10-24GB-150W / 4bit-bnb-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemm-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-eager 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-gemv-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 22.3
Yi-34B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 22.3
Yi-34B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 22.3
Yi-34B A100-80GB-275W / 8bit-bnb-fa2 22.3
Yi-34B A100-80GB-275W / 8bit-bnb-sdpa 22.3
Yi-34B A100-80GB-275W / 8bit-bnb-eager 22.3
Yi-34B A100-80GB-275W / 4bit-bnb-eager 22.3
Yi-34B A100-80GB-275W / 4bit-bnb-fa2 22.3
Yi-34B A100-80GB-275W / 4bit-bnb-sdpa 22.3
Yi-34B A100-80GB-275W / bfloat16-eager 22.3
Yi-34B A100-80GB-275W / float16-eager 22.3
Yi-34B A100-80GB-275W / float16-sdpa 22.3
Yi-34B A100-80GB-275W / bfloat16-sdpa 22.3
Yi-34B A100-80GB-275W / bfloat16-fa2 22.3
Yi-34B A100-80GB-275W / float16-fa2 22.3
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemm-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemm-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemm-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-eager 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-eager 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 8bit-bnb-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemv-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemv-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v1-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v2-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemv-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B A100-80GB-275W / 8bit-bnb-eager 20.2
Qwen1.5-14B A100-80GB-275W / 8bit-bnb-fa2 20.2
Qwen1.5-14B A100-80GB-275W / 8bit-bnb-sdpa 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-bnb-eager 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-bnb-fa2 20.2
Qwen1.5-14B A100-80GB-275W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-bnb-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-bnb-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-bnb-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-gemm-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-gemm-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-gemv-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-gemv-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-exllama-v1-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-exllama-v2-eager 20.2
Qwen1.5-14B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / float32-eager-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / float16-eager-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 20.2
Qwen1.5-14B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 20.2
Mistral-7B-Instruct-v0.2 A10-24GB-150W / bfloat16-eager 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / float16-fa2 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / float16-eager 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / bfloat16-fa2 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / bfloat16-sdpa 18.4
Mistral-7B-Instruct-v0.2 A10-24GB-150W / float16-sdpa 18.4
Mistral-7B-Instruct-v0.2 T4-16GB-70W / 4bit-gptq-exllama-v2-eager 18.4
Mistral-7B-Instruct-v0.2 T4-16GB-70W / 4bit-gptq-exllama-v1-eager 18.4
Mistral-7B-Instruct-v0.2 T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 18.4
Mistral-7B-Instruct-v0.2 T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 18.4
Zephyr 7B Beta A10-24GB-150W / 8bit-bnb-eager 17.7
Zephyr 7B Beta A10-24GB-150W / 8bit-bnb-fa2 17.7
Zephyr 7B Beta A10-24GB-150W / 8bit-bnb-sdpa 17.7
Zephyr 7B Beta A10-24GB-150W / 4bit-bnb-fa2 17.7
Zephyr 7B Beta A10-24GB-150W / 4bit-bnb-eager 17.7
Zephyr 7B Beta A10-24GB-150W / float16-fa2 17.7
Zephyr 7B Beta A10-24GB-150W / bfloat16-fa2 17.7
Zephyr 7B Beta A10-24GB-150W / 4bit-bnb-sdpa 17.7
Zephyr 7B Beta A10-24GB-150W / bfloat16-eager 17.7
Zephyr 7B Beta A10-24GB-150W / float16-eager 17.7
Zephyr 7B Beta A10-24GB-150W / bfloat16-sdpa 17.7
Zephyr 7B Beta A10-24GB-150W / float16-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-bnb-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-bnb-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / 8bit-bnb-eager 17.7
Zephyr 7B Beta T4-16GB-70W / bfloat16-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / bfloat16-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 8bit-bnb-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / float16-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / float16-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-gptq-exllama-v1-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-gptq-exllama-v2-eager 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 17.7
Zephyr 7B Beta T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / float32-eager 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / float32-sdpa 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / float16-eager 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / float16-sdpa 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / bfloat16-eager 17.7
Zephyr 7B Beta Intel-Xeon-SPR-385W / bfloat16-sdpa 17.7
Gemma 7B A10-24GB-150W / 8bit-bnb-fa2 15.3
Gemma 7B A10-24GB-150W / 8bit-bnb-eager 15.3
Gemma 7B A10-24GB-150W / 8bit-bnb-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-bnb-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-bnb-fa2 15.3
Gemma 7B A10-24GB-150W / float16-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-bnb-sdpa 15.3
Gemma 7B A10-24GB-150W / bfloat16-fa2 15.3
Gemma 7B A10-24GB-150W / float16-fa2 15.3
Gemma 7B A10-24GB-150W / bfloat16-fa2 15.3
Gemma 7B A10-24GB-150W / bfloat16-eager 15.3
Gemma 7B A10-24GB-150W / bfloat16-eager 15.3
Gemma 7B A10-24GB-150W / float16-eager 15.3
Gemma 7B A10-24GB-150W / float16-eager 15.3
Gemma 7B A10-24GB-150W / bfloat16-sdpa 15.3
Gemma 7B A10-24GB-150W / bfloat16-sdpa 15.3
Gemma 7B A10-24GB-150W / float16-sdpa 15.3
Gemma 7B A10-24GB-150W / float16-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemm-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemm-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemm-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemv-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemv-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-gemv-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 15.3
Gemma 7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 15.3
Gemma 7B T4-16GB-70W / 8bit-bnb-eager 15.3
Gemma 7B T4-16GB-70W / 8bit-bnb-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-bnb-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-bnb-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-gemm-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-gemv-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-gemm-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-gemv-eager 15.3
Gemma 7B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 15.3
Gemma 7B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 15.3
Gemma 7B Intel-Xeon-SPR-385W / float32-eager-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / float16-eager-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 15.3
Gemma 7B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 15.3
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemm-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemm-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemm-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-eager 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-eager 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / float16-fa2 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-fa2 15.2
Qwen1.5-7B A10-24GB-150W / float16-fa2 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-fa2 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-eager 15.2
Qwen1.5-7B A10-24GB-150W / float16-eager 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / float16-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-eager 15.2
Qwen1.5-7B A10-24GB-150W / float16-eager 15.2
Qwen1.5-7B A10-24GB-150W / bfloat16-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / float16-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemv-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemv-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemv-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / 8bit-bnb-fa2 15.2
Qwen1.5-7B A100-80GB-275W / 8bit-bnb-eager 15.2
Qwen1.5-7B A100-80GB-275W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-bnb-fa2 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-bnb-eager 15.2
Qwen1.5-7B A100-80GB-275W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / bfloat16-eager 15.2
Qwen1.5-7B A100-80GB-275W / float16-eager 15.2
Qwen1.5-7B A100-80GB-275W / float32-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / bfloat16-sdpa 15.2
Qwen1.5-7B A100-80GB-275W / float16-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 8bit-bnb-eager 15.2
Qwen1.5-7B T4-16GB-70W / 8bit-bnb-eager 15.2
Qwen1.5-7B T4-16GB-70W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 8bit-bnb-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-bnb-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-bnb-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-bnb-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / bfloat16-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / bfloat16-eager 15.2
Qwen1.5-7B T4-16GB-70W / float16-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / float16-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemm-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemm-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemm-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemm-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemv-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemv-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemv-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-gemv-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 15.2
Qwen1.5-7B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / float32-eager-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / float16-eager-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 15.2
Qwen1.5-7B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 15.2
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemm-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemm-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemm-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v1-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v2-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 8bit-bnb-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 8bit-bnb-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 8bit-bnb-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-bnb-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-bnb-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-bnb-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / bfloat16-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / float16-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / bfloat16-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / float16-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemv-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / bfloat16-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / float16-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v2-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemv-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v1-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v1-fa2 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v2-eager 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemv-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 14.5
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / 8bit-bnb-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 8bit-bnb-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-bnb-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-bnb-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / bfloat16-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / bfloat16-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / float16-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / float16-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-gptq-exllama-v1-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-gptq-exllama-v2-eager 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 14.5
Mistral 7B v0.1 T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / float32-eager-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / float16-eager-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / float32-sdpa-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / float16-sdpa-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 14.5
Mistral 7B v0.1 Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 14.5
Yi 6B A10-24GB-150W / 4bit-awq-gemm-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-eager 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-sdpa 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-fa2 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-eager 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-fa2 13.6
Yi 6B A10-24GB-150W / 8bit-bnb-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-eager 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-eager 13.6
Yi 6B A10-24GB-150W / 4bit-bnb-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-fa2 13.6
Yi 6B A10-24GB-150W / float16-fa2 13.6
Yi 6B A10-24GB-150W / bfloat16-fa2 13.6
Yi 6B A10-24GB-150W / bfloat16-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemm-sdpa 13.6
Yi 6B A10-24GB-150W / float16-eager 13.6
Yi 6B A10-24GB-150W / bfloat16-sdpa 13.6
Yi 6B A10-24GB-150W / float16-sdpa 13.6
Yi 6B A10-24GB-150W / bfloat16-fa2 13.6
Yi 6B A10-24GB-150W / float16-fa2 13.6
Yi 6B A10-24GB-150W / float16-eager 13.6
Yi 6B A10-24GB-150W / bfloat16-eager 13.6
Yi 6B A10-24GB-150W / float16-sdpa 13.6
Yi 6B A10-24GB-150W / bfloat16-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-eager 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-gemv-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 13.6
Yi 6B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 13.6
Yi 6B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 13.6
Yi 6B A100-80GB-275W / 8bit-bnb-fa2 13.6
Yi 6B A100-80GB-275W / 8bit-bnb-sdpa 13.6
Yi 6B A100-80GB-275W / 8bit-bnb-eager 13.6
Yi 6B A100-80GB-275W / 4bit-bnb-eager 13.6
Yi 6B A100-80GB-275W / 4bit-bnb-fa2 13.6
Yi 6B A100-80GB-275W / 4bit-bnb-sdpa 13.6
Yi 6B A100-80GB-275W / float16-eager 13.6
Yi 6B A100-80GB-275W / bfloat16-eager 13.6
Yi 6B A100-80GB-275W / float16-sdpa 13.6
Yi 6B A100-80GB-275W / float32-eager 13.6
Yi 6B A100-80GB-275W / bfloat16-sdpa 13.6
Yi 6B A100-80GB-275W / float16-fa2 13.6
Yi 6B A100-80GB-275W / bfloat16-fa2 13.6
Yi 6B A100-80GB-275W / float32-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B T4-16GB-70W / 8bit-bnb-eager 13.6
Yi 6B T4-16GB-70W / 8bit-bnb-eager 13.6
Yi 6B T4-16GB-70W / 8bit-bnb-sdpa 13.6
Yi 6B T4-16GB-70W / 8bit-bnb-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-bnb-eager 13.6
Yi 6B T4-16GB-70W / 4bit-bnb-eager 13.6
Yi 6B T4-16GB-70W / 4bit-bnb-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-bnb-sdpa 13.6
Yi 6B T4-16GB-70W / bfloat16-sdpa 13.6
Yi 6B T4-16GB-70W / bfloat16-eager 13.6
Yi 6B T4-16GB-70W / bfloat16-eager 13.6
Yi 6B T4-16GB-70W / bfloat16-sdpa 13.6
Yi 6B T4-16GB-70W / float16-sdpa 13.6
Yi 6B T4-16GB-70W / float16-eager 13.6
Yi 6B T4-16GB-70W / float16-sdpa 13.6
Yi 6B T4-16GB-70W / float16-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemm-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemm-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemm-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemm-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v1-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemv-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemv-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v2-eager 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v1-eager 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v2-eager 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemv-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-gemv-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 13.6
Yi 6B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / float32-eager-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / float16-eager-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-eager 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-sdpa 13.6
Yi 6B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 13.6
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-eager 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-eager 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-fa2 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-sdpa 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-sdpa 13.4
Llama 3-8B A10-24GB-150W / 8bit-bnb-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-bnb-sdpa 13.4
Llama 3-8B A10-24GB-150W / float16-fa2 13.4
Llama 3-8B A10-24GB-150W / bfloat16-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-fa2 13.4
Llama 3-8B A10-24GB-150W / bfloat16-eager 13.4
Llama 3-8B A10-24GB-150W / float16-eager 13.4
Llama 3-8B A10-24GB-150W / bfloat16-sdpa 13.4
Llama 3-8B A10-24GB-150W / float16-sdpa 13.4
Llama 3-8B A10-24GB-150W / bfloat16-eager 13.4
Llama 3-8B A10-24GB-150W / float16-eager 13.4
Llama 3-8B A10-24GB-150W / bfloat16-fa2 13.4
Llama 3-8B A10-24GB-150W / float16-fa2 13.4
Llama 3-8B A10-24GB-150W / bfloat16-sdpa 13.4
Llama 3-8B A10-24GB-150W / float16-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-eager 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-gemv-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 13.4
Llama 3-8B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 13.4
Llama 3-8B T4-16GB-70W / 8bit-bnb-eager 13.4
Llama 3-8B T4-16GB-70W / 8bit-bnb-sdpa 13.4
Llama 3-8B T4-16GB-70W / 8bit-bnb-eager 13.4
Llama 3-8B T4-16GB-70W / 8bit-bnb-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-bnb-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-bnb-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-bnb-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-bnb-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-gemm-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-gemm-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-gemv-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-exllama-v1-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-exllama-v2-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-gemv-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 13.4
Llama 3-8B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float16-eager-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float32-eager-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float32-eager 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float32-sdpa 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float16-eager 13.4
Llama 3-8B Intel-Xeon-SPR-385W / float16-sdpa 13.4
Llama 3-8B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 13.4
Llama 3-8B Intel-Xeon-SPR-385W / bfloat16-eager 13.4
Llama 3-8B Intel-Xeon-SPR-385W / bfloat16-sdpa 13.4
LLaMA-13B A10-24GB-150W / 4bit-awq-gemm-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemm-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemm-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 9.25
LLaMA-13B A10-24GB-150W / 8bit-bnb-eager 9.25
LLaMA-13B A10-24GB-150W / 8bit-bnb-fa2 9.25
LLaMA-13B A10-24GB-150W / 8bit-bnb-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-bnb-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-bnb-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-bnb-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemv-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemv-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v2-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v1-eager 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-gemv-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 9.25
LLaMA-13B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 9.25
LLaMA-13B T4-16GB-70W / 8bit-bnb-eager 9.25
LLaMA-13B T4-16GB-70W / 8bit-bnb-sdpa 9.25
LLaMA-13B T4-16GB-70W / 4bit-bnb-eager 9.25
LLaMA-13B T4-16GB-70W / 4bit-bnb-sdpa 9.25
LLaMA-13B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 9.25
LLaMA-13B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 9.25
LLaMA-13B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 9.25
LLaMA-13B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 9.25
LLaMA-13B Intel-Xeon-SPR-385W / float32-eager-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / float16-eager-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 9.25
LLaMA-13B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 9.25
Gemma 2B A10-24GB-150W / 8bit-bnb-eager 7.31
Gemma 2B A10-24GB-150W / 8bit-bnb-fa2 7.31
Gemma 2B A10-24GB-150W / 8bit-bnb-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-bnb-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-bnb-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-bnb-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemm-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemm-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemm-sdpa 7.31
Gemma 2B A10-24GB-150W / float32-sdpa 7.31
Gemma 2B A10-24GB-150W / float32-sdpa 7.31
Gemma 2B A10-24GB-150W / float32-eager 7.31
Gemma 2B A10-24GB-150W / float32-eager 7.31
Gemma 2B A10-24GB-150W / bfloat16-eager 7.31
Gemma 2B A10-24GB-150W / float16-eager 7.31
Gemma 2B A10-24GB-150W / bfloat16-eager 7.31
Gemma 2B A10-24GB-150W / float16-eager 7.31
Gemma 2B A10-24GB-150W / bfloat16-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemv-eager 7.31
Gemma 2B A10-24GB-150W / float16-fa2 7.31
Gemma 2B A10-24GB-150W / float16-fa2 7.31
Gemma 2B A10-24GB-150W / bfloat16-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v1-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemv-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v2-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 7.31
Gemma 2B A10-24GB-150W / float16-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 7.31
Gemma 2B A10-24GB-150W / float16-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 7.31
Gemma 2B A10-24GB-150W / bfloat16-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-gemv-sdpa 7.31
Gemma 2B A10-24GB-150W / bfloat16-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 7.31
Gemma 2B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 7.31
Gemma 2B T4-16GB-70W / 8bit-bnb-eager 7.31
Gemma 2B T4-16GB-70W / 8bit-bnb-sdpa 7.31
Gemma 2B T4-16GB-70W / float32-sdpa 7.31
Gemma 2B T4-16GB-70W / float32-sdpa 7.31
Gemma 2B T4-16GB-70W / float32-eager 7.31
Gemma 2B T4-16GB-70W / float32-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-bnb-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-bnb-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-gemm-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-exllama-v1-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-gemm-sdpa 7.31
Gemma 2B T4-16GB-70W / float16-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-exllama-v2-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-gemv-eager 7.31
Gemma 2B T4-16GB-70W / float16-eager 7.31
Gemma 2B T4-16GB-70W / bfloat16-eager 7.31
Gemma 2B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 7.31
Gemma 2B T4-16GB-70W / bfloat16-eager 7.31
Gemma 2B T4-16GB-70W / bfloat16-sdpa 7.31
Gemma 2B T4-16GB-70W / float16-sdpa 7.31
Gemma 2B T4-16GB-70W / bfloat16-sdpa 7.31
Gemma 2B T4-16GB-70W / float16-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-gemv-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 7.31
Gemma 2B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 7.31
Gemma 2B Intel-Xeon-SPR-385W / float16-eager-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / float32-eager-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 7.31
Gemma 2B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 7.31
Phi-1.5 A10-24GB-150W / 4bit-awq-gemm-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemm-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemm-sdpa 7.06
Phi-1.5 A10-24GB-150W / 8bit-bnb-eager 7.06
Phi-1.5 A10-24GB-150W / 8bit-bnb-sdpa 7.06
Phi-1.5 A10-24GB-150W / 8bit-bnb-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v2-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v1-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 7.06
Phi-1.5 A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-bnb-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-bnb-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-bnb-sdpa 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemv-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v2-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemv-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v1-eager 7.06
Phi-1.5 A10-24GB-150W / bfloat16-eager 7.06
Phi-1.5 A10-24GB-150W / float16-eager 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v2-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v1-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-gemv-sdpa 7.06
Phi-1.5 A10-24GB-150W / bfloat16-fa2 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 7.06
Phi-1.5 A10-24GB-150W / float32-eager 7.06
Phi-1.5 A10-24GB-150W / float16-fa2 7.06
Phi-1.5 A10-24GB-150W / bfloat16-sdpa 7.06
Phi-1.5 A10-24GB-150W / float16-sdpa 7.06
Phi-1.5 A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 7.06
Phi-1.5 A10-24GB-150W / float32-sdpa 7.06
Phi-1.5 A100-80GB-275W / float32-eager 7.06
Phi-1.5 A100-80GB-275W / bfloat16-eager 7.06
Phi-1.5 A100-80GB-275W / float16-eager 7.06
Phi-1.5 A100-80GB-275W / float16-fa2 7.06
Phi-1.5 A100-80GB-275W / bfloat16-fa2 7.06
Phi-1.5 A100-80GB-275W / float16-sdpa 7.06
Phi-1.5 A100-80GB-275W / bfloat16-sdpa 7.06
Phi-1.5 A100-80GB-275W / float32-sdpa 7.06
Phi-1.5 T4-16GB-70W / 8bit-bnb-eager 7.06
Phi-1.5 T4-16GB-70W / 8bit-bnb-sdpa 7.06
Phi-1.5 T4-16GB-70W / 4bit-bnb-eager 7.06
Phi-1.5 T4-16GB-70W / 4bit-bnb-sdpa 7.06
Phi-1.5 T4-16GB-70W / float32-sdpa 7.06
Phi-1.5 T4-16GB-70W / float32-sdpa 7.06
Phi-1.5 T4-16GB-70W / float32-eager 7.06
Phi-1.5 T4-16GB-70W / float32-eager 7.06
Phi-1.5 T4-16GB-70W / float16-eager 7.06
Phi-1.5 T4-16GB-70W / 4bit-gptq-exllama-v2-eager 7.06
Phi-1.5 T4-16GB-70W / float16-eager 7.06
Phi-1.5 T4-16GB-70W / bfloat16-eager 7.06
Phi-1.5 T4-16GB-70W / 4bit-gptq-exllama-v1-eager 7.06
Phi-1.5 T4-16GB-70W / bfloat16-eager 7.06
Phi-1.5 T4-16GB-70W / bfloat16-sdpa 7.06
Phi-1.5 T4-16GB-70W / bfloat16-sdpa 7.06
Phi-1.5 T4-16GB-70W / float16-sdpa 7.06
Phi-1.5 T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 7.06
Phi-1.5 T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 7.06
Phi-1.5 T4-16GB-70W / float16-sdpa 7.06
Phi-1.5 Intel-Xeon-SPR-385W / float32-eager-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / float32-sdpa-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / float16-eager-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / float16-sdpa-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 7.06
Phi-1.5 Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 7.06
Qwen2-0.5B A10-24GB-150W / 8bit-bnb-fa2 7.06
Qwen2-0.5B A10-24GB-150W / 8bit-bnb-eager 7.06
Qwen2-0.5B A10-24GB-150W / 8bit-bnb-sdpa 7.06
Qwen2-0.5B A10-24GB-150W / 4bit-bnb-fa2 7.06
Qwen2-0.5B A10-24GB-150W / 4bit-bnb-eager 7.06
Qwen2-0.5B A10-24GB-150W / 4bit-bnb-sdpa 7.06
Qwen2-0.5B A10-24GB-150W / float16-fa2 7.06
Qwen2-0.5B A10-24GB-150W / float16-eager 7.06
Qwen2-0.5B A10-24GB-150W / bfloat16-eager 7.06
Qwen2-0.5B A10-24GB-150W / bfloat16-fa2 7.06
Qwen2-0.5B A10-24GB-150W / float16-sdpa 7.06
Qwen2-0.5B A10-24GB-150W / float32-eager 7.06
Qwen2-0.5B A10-24GB-150W / bfloat16-sdpa 7.06
Qwen2-0.5B A10-24GB-150W / float32-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / 8bit-bnb-eager 7.06
Qwen2-0.5B T4-16GB-70W / 8bit-bnb-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-bnb-eager 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-bnb-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / bfloat16-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / float16-eager 7.06
Qwen2-0.5B T4-16GB-70W / bfloat16-eager 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 7.06
Qwen2-0.5B T4-16GB-70W / float32-eager 7.06
Qwen2-0.5B T4-16GB-70W / float16-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / float32-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 7.06
Qwen2-0.5B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / float16-eager 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / float16-sdpa 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / float32-sdpa 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / bfloat16-eager 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / float32-eager 7.06
Qwen2-0.5B Intel-Xeon-SPR-385W / bfloat16-sdpa 7.06
GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B A10-24GB-150W / 8bit-bnb-fa2 6.55
GPT-J-6B A10-24GB-150W / 8bit-bnb-fa2 6.55
GPT-J-6B A10-24GB-150W / 8bit-bnb-eager 6.55
GPT-J-6B A10-24GB-150W / 8bit-bnb-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-bnb-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-bnb-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-bnb-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-bnb-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-fa2 6.55
GPT-J-6B A10-24GB-150W / float16-fa2 6.55
GPT-J-6B A10-24GB-150W / bfloat16-fa2 6.55
GPT-J-6B A10-24GB-150W / float16-fa2 6.55
GPT-J-6B A10-24GB-150W / bfloat16-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemv-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemv-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.55
GPT-J-6B A10-24GB-150W / bfloat16-eager 6.55
GPT-J-6B A10-24GB-150W / float16-eager 6.55
GPT-J-6B A10-24GB-150W / float16-eager 6.55
GPT-J-6B A10-24GB-150W / bfloat16-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemv-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-gemv-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 6.55
GPT-J-6B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 6.55
GPT-J-6B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B A100-80GB-275W / 8bit-bnb-fa2 6.55
GPT-J-6B A100-80GB-275W / 8bit-bnb-eager 6.55
GPT-J-6B A100-80GB-275W / 4bit-bnb-fa2 6.55
GPT-J-6B A100-80GB-275W / 4bit-bnb-eager 6.55
GPT-J-6B A100-80GB-275W / bfloat16-fa2 6.55
GPT-J-6B A100-80GB-275W / float16-fa2 6.55
GPT-J-6B A100-80GB-275W / float16-eager 6.55
GPT-J-6B A100-80GB-275W / bfloat16-eager 6.55
GPT-J-6B A100-80GB-275W / float32-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B T4-16GB-70W / 8bit-bnb-eager 6.55
GPT-J-6B T4-16GB-70W / 8bit-bnb-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-bnb-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-bnb-eager 6.55
GPT-J-6B T4-16GB-70W / bfloat16-eager 6.55
GPT-J-6B T4-16GB-70W / bfloat16-eager 6.55
GPT-J-6B T4-16GB-70W / float16-eager 6.55
GPT-J-6B T4-16GB-70W / float16-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-gemm-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-gemm-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-gemv-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-gemv-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-exllama-v2-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-exllama-v1-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-exllama-v2-eager 6.55
GPT-J-6B T4-16GB-70W / 4bit-awq-exllama-v1-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float16-eager-pytorch 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float32-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float32-eager-pytorch 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float32-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float16-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / float16-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 6.55
GPT-J-6B Intel-Xeon-SPR-385W / bfloat16-eager 6.55
GPT-J-6B Intel-Xeon-SPR-385W / bfloat16-eager 6.55
LLaMA-7B A10-24GB-150W / 4bit-awq-gemm-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemm-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemm-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 6.39
LLaMA-7B A10-24GB-150W / 8bit-bnb-eager 6.39
LLaMA-7B A10-24GB-150W / 8bit-bnb-fa2 6.39
LLaMA-7B A10-24GB-150W / 8bit-bnb-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-bnb-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-bnb-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-bnb-sdpa 6.39
LLaMA-7B A10-24GB-150W / float16-fa2 6.39
LLaMA-7B A10-24GB-150W / bfloat16-fa2 6.39
LLaMA-7B A10-24GB-150W / bfloat16-eager 6.39
LLaMA-7B A10-24GB-150W / float16-eager 6.39
LLaMA-7B A10-24GB-150W / float16-fa2 6.39
LLaMA-7B A10-24GB-150W / bfloat16-fa2 6.39
LLaMA-7B A10-24GB-150W / bfloat16-sdpa 6.39
LLaMA-7B A10-24GB-150W / float16-sdpa 6.39
LLaMA-7B A10-24GB-150W / bfloat16-eager 6.39
LLaMA-7B A10-24GB-150W / float16-eager 6.39
LLaMA-7B A10-24GB-150W / bfloat16-sdpa 6.39
LLaMA-7B A10-24GB-150W / float16-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemv-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemv-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-gemv-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 6.39
LLaMA-7B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 6.39
LLaMA-7B T4-16GB-70W / 8bit-bnb-eager 6.39
LLaMA-7B T4-16GB-70W / 8bit-bnb-sdpa 6.39
LLaMA-7B T4-16GB-70W / 4bit-bnb-eager 6.39
LLaMA-7B T4-16GB-70W / 4bit-bnb-sdpa 6.39
LLaMA-7B T4-16GB-70W / bfloat16-sdpa 6.39
LLaMA-7B T4-16GB-70W / bfloat16-sdpa 6.39
LLaMA-7B T4-16GB-70W / bfloat16-eager 6.39
LLaMA-7B T4-16GB-70W / bfloat16-eager 6.39
LLaMA-7B T4-16GB-70W / float16-eager 6.39
LLaMA-7B T4-16GB-70W / float16-sdpa 6.39
LLaMA-7B T4-16GB-70W / float16-eager 6.39
LLaMA-7B T4-16GB-70W / float16-sdpa 6.39
LLaMA-7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.39
LLaMA-7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.39
LLaMA-7B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 6.39
LLaMA-7B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 6.39
LLaMA-7B Intel-Xeon-SPR-385W / float32-eager-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / float32-sdpa-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / float16-eager-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 6.39
LLaMA-7B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 6.39
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 8bit-bnb-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 8bit-bnb-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-bnb-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-bnb-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float32-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float32-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / bfloat16-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float16-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float16-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemv-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemv-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / bfloat16-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v2-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v1-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B A10-24GB-150W / float16-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / bfloat16-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemv-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / bfloat16-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemv-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / float16-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / 8bit-bnb-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / 4bit-bnb-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / bfloat16-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / float16-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / float32-eager 6.34
GPT-Neo-2.7B A100-80GB-275W / float16-fa2 6.34
GPT-Neo-2.7B A100-80GB-275W / bfloat16-fa2 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 8bit-bnb-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / float32-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / float32-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-bnb-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-gemm-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-gemm-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / float16-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / float16-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / bfloat16-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / bfloat16-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-gemv-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-gemv-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-exllama-v1-eager 6.34
GPT-Neo-2.7B T4-16GB-70W / 4bit-awq-exllama-v2-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float16-eager-pytorch 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float16-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float16-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float32-eager-pytorch 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float32-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / float32-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / bfloat16-eager 6.34
GPT-Neo-2.7B Intel-Xeon-SPR-385W / bfloat16-eager 6.34
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 8bit-bnb-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-bnb-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v1-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemv-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 5.99
GPT-NeoX-20B A10-24GB-150W / 4bit-awq-exllama-v2-sdpa 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / 8bit-bnb-eager 5.99
GPT-NeoX-20B A100-80GB-275W / 8bit-bnb-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-bnb-eager 5.99
GPT-NeoX-20B A100-80GB-275W / 4bit-bnb-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / float32-eager 5.99
GPT-NeoX-20B A100-80GB-275W / float16-eager 5.99
GPT-NeoX-20B A100-80GB-275W / bfloat16-eager 5.99
GPT-NeoX-20B A100-80GB-275W / float16-fa2 5.99
GPT-NeoX-20B A100-80GB-275W / bfloat16-fa2 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-bnb-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-bnb-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-bnb-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-bnb-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemm-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemm-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemv-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemv-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemm-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemv-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemm-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-gemv-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v1-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v1-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v1-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v2-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v1-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v2-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v2-eager 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 5.99
GPT-NeoX-20B T4-16GB-70W / 4bit-awq-exllama-v2-sdpa 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / float16-sdpa-pytorch 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / float16-eager-pytorch 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / float16-sdpa 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / float16-eager 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / bfloat16-eager-pytorch 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / bfloat16-eager 5.99
GPT-NeoX-20B Intel-Xeon-SPR-385W / bfloat16-sdpa 5.99
Falcon-7B A10-24GB-150W / 8bit-bnb-eager 5.10
Falcon-7B A10-24GB-150W / float16-eager 5.10
Falcon-7B A10-24GB-150W / bfloat16-eager 5.10
Falcon-7B A10-24GB-150W / 4bit-bnb-eager 5.10
Falcon-7B T4-16GB-70W / 4bit-bnb-eager 5.10
Falcon-7B T4-16GB-70W / 8bit-bnb-eager 5.10
Falcon-7B T4-16GB-70W / bfloat16-eager 5.10
Falcon-7B T4-16GB-70W / float16-eager 5.10

8 models have held number one on the official leaderboard since 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

EleutherAI Meta Mistral AI Hugging Face 01.AI Qwen Microsoft
Own leaderboard Not tested since Jun 2024

Source: Official LLM-Perf Leaderboard leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard
5.00 10.0 15.0 20.0 25.0 30.0 2021 2022 2023 2024 5.00 10.0 15.0 20.0 25.0 30.0 2021 2022 2023 2024

Source: Official LLM-Perf Leaderboard leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Other

All 69 in Other
  • Artificial Analysis Intelligence Index v5 In use Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. Claude Fable 5.2 64.6
  • Personal Superintelligence Index (Meta) In use Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO.
  • Vals Science Index In use Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. GPT-6.2 Astra 68.4%
  • Decision Index Public v0.3 In use Decision models return probabilities over labeled options in public tasks covering knowledge, language, retrieval, tools, and arts. A chance-corrected weighted aggregate combines the task scores. Granite Switch 4.1 30B 51.3
  • SAGE v1.1 (Vals) In use Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading. Claude Opus 5.5 91.3%
  • MTEB Fading Massive Text Embedding Benchmark: retrieval, clustering, classification across languages and modalities. Qwen3-Embedding-8B 72.2%

Data audit

8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in June 2024, for Qwen2-0.5B. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
  • Pass Independent runs Scores from the official leaderboard (46 entries).
  • Pass Models scored 21 models we track, plus 25 other entries on its leaderboard.
  • Warning Newest score The newest came 2 years ago, in June 2024, for Qwen2-0.5B.
  • Note Frontier models None of the top 10 has a score. The strongest tested is phi-3-mini 3.8B (#204).
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on huggingface.co: 1,299 rows for models we track and 25 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check