LLM-Perf Leaderboard
Language models generate tokens on a specified hardware and software configuration. The board reports latency, throughput, memory and energy alongside quality scores.
The quality score comes from the Open LLM Leaderboard. Inference measurements depend on the hardware.
- Made by
- Hugging Face
- Kind
- Composite indices
- Level
- University
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- All public
- Measures
- Efficiency
- Language
- English
- Interaction
- Single turn
- Items from
- Another benchmark
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
-
phi-3-mini 3.8B #204 on the capability index
- Last model added
- June 2024
- Scores refreshed
- November 20th, 2026
Qwen1.5-110B leads with 29.6 among models released by June 2024 Qwen1.5-110B leads with 29.6 among models released by June 2024
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
The newest model with a score here came out in June 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 29.6
Qwen1.5-110B
- 27.2
phi-3-mini 3.8B
- 26.7
Qwen1.5-32B
- 23.9 meta-llama/Meta-Llama-3-8B-Instruct
- 22.3
Yi-34B
- 20.2
Qwen1.5-14B
- 18.4
Mistral-7B-Instruct-v0.2
- 17.7
Zephyr 7B Beta
- 15.3
Gemma 7B
- 15.2
Qwen1.5-7B
- 14.9 Deci/DeciLM-7B
- 14.5
Mistral 7B v0.1
- 14.2 TencentARC/Mistral_Pro_8B_v0.1
- 13.9 stabilityai/stablelm-2-12b
- 13.6
Yi 6B
- 13.5 google/recurrentgemma-9b
- 13.4
Llama 3-8B
- 12.4 Qwen/Qwen1.5-MoE-A2.7B
- 11.3 Qwen/Qwen1.5-4B
- 11.0 meta-llama/Llama-2-13b-hf
- 10.3 google/gemma-2-2b
- 9.25
LLaMA-13B
- 9.12 Qwen/Qwen1.5-1.8B
- 8.72 meta-llama/Llama-2-7b-hf
- 7.31
Gemma 2B
- 7.26 stabilityai/stablelm-3b-4e1t
- 7.06
Phi-1.5
- 7.06
Qwen2-0.5B
- 6.94 google/recurrentgemma-2b
- 6.55 GPT-J-6B
- 6.51 openai-community/gpt2
- 6.39
LLaMA-7B
- 6.34 GPT-Neo-2.7B
- 6.20 facebook/opt-30b
- 5.99 GPT-NeoX-20B
- 5.93 EleutherAI/pythia-12b
- 5.62 EleutherAI/pythia-160m
- 5.48 openai-community/gpt2-large
- 5.43 togethercomputer/RedPajama-INCITE-Base-3B-v1
- 5.33 EleutherAI/gpt-neo-1.3B
- 5.22 stabilityai/stablelm-2-1_6b
- 5.14 Qwen/Qwen1.5-0.5B
- 5.11 EleutherAI/pythia-410m
- 5.10
Falcon-7B
- 4.38 EleutherAI/gpt-neo-125m
- 3.90 distilbert/distilgpt2
- 1
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 29.6
- 2
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-eager 27.2
- 3
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemm-fa2 26.7
- 4 meta-llama/Meta-Llama-3-8B-Instruct A10-24GB-150W / 8bit-bnb-eager 23.9
- 5
Yi-34B A10-24GB-150W / 4bit-awq-gemm-eager 22.3
- 6
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemm-fa2 20.2
- 7
Mistral-7B-Instruct-v0.2 A10-24GB-150W / bfloat16-eager 18.4
- 8
Zephyr 7B Beta A10-24GB-150W / 8bit-bnb-eager 17.7
- 9
Gemma 7B A10-24GB-150W / 8bit-bnb-fa2 15.3
- 10
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemm-fa2 15.2
- 11 Deci/DeciLM-7B A10-24GB-150W / 4bit-awq-gemm-eager 14.9
- 12
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemm-fa2 14.5
- 13 TencentARC/Mistral_Pro_8B_v0.1 A10-24GB-150W / 4bit-awq-gemm-fa2 14.2
- 14 stabilityai/stablelm-2-12b A10-24GB-150W / 4bit-awq-gemm-eager 13.9
- 15
Yi 6B A10-24GB-150W / 4bit-awq-gemm-eager 13.6
- 16 google/recurrentgemma-9b A10-24GB-150W / 8bit-bnb-eager 13.5
- 17
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-eager 13.4
- 18 Qwen/Qwen1.5-MoE-A2.7B A10-24GB-150W / 8bit-bnb-eager 12.4
- 19 Qwen/Qwen1.5-4B A10-24GB-150W / 4bit-awq-gemm-fa2 11.3
- 20 meta-llama/Llama-2-13b-hf A10-24GB-150W / 4bit-awq-gemm-eager 11.0
- 21 google/gemma-2-2b A10-24GB-150W / 8bit-bnb-fa2 10.3
- 22
LLaMA-13B A10-24GB-150W / 4bit-awq-gemm-eager 9.25
- 23 Qwen/Qwen1.5-1.8B A10-24GB-150W / 4bit-awq-gemm-fa2 9.12
- 24 meta-llama/Llama-2-7b-hf A10-24GB-150W / 4bit-awq-gemm-eager 8.72
- 25
Gemma 2B A10-24GB-150W / 8bit-bnb-eager 7.31
- 26 stabilityai/stablelm-3b-4e1t A10-24GB-150W / 4bit-awq-gemm-eager 7.26
- 27
Phi-1.5 A10-24GB-150W / 4bit-awq-gemm-eager 7.06
- 28
Qwen2-0.5B A10-24GB-150W / 8bit-bnb-fa2 7.06
- 29 google/recurrentgemma-2b A10-24GB-150W / 8bit-bnb-eager 6.94
- 30 GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-fa2 6.55
- 31 openai-community/gpt2 A10-24GB-150W / 8bit-bnb-eager 6.51
- 32
LLaMA-7B A10-24GB-150W / 4bit-awq-gemm-eager 6.39
- 33 GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-eager 6.34
- 34 facebook/opt-30b A10-24GB-150W / 4bit-awq-gemm-eager 6.20
- 35 GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-eager 5.99
- 36 EleutherAI/pythia-12b A10-24GB-150W / 4bit-awq-gemm-eager 5.93
- 37 EleutherAI/pythia-160m A10-24GB-150W / 4bit-awq-gemm-eager 5.62
- 38 openai-community/gpt2-large A10-24GB-150W / 8bit-bnb-eager 5.48
- 39 togethercomputer/RedPajama-INCITE-Base-3B-v1 A10-24GB-150W / 4bit-awq-gemm-eager 5.43
- 40 EleutherAI/gpt-neo-1.3B A10-24GB-150W / 4bit-awq-gemm-eager 5.33
- 41 stabilityai/stablelm-2-1_6b A10-24GB-150W / 4bit-awq-gemm-eager 5.22
- 42 Qwen/Qwen1.5-0.5B A10-24GB-150W / 8bit-bnb-fa2 5.14
- 43 EleutherAI/pythia-410m A10-24GB-150W / 4bit-awq-gemm-eager 5.11
- 44
Falcon-7B A10-24GB-150W / 8bit-bnb-eager 5.10
- 45 EleutherAI/gpt-neo-125m A10-24GB-150W / 8bit-bnb-eager 4.38
- 46 distilbert/distilgpt2 A10-24GB-150W / 8bit-bnb-eager 3.90
Source: Official LLM-Perf Leaderboard leaderboard
- 29.6
Qwen1.5-110B
- 27.2
phi-3-mini 3.8B
- 26.7
Qwen1.5-32B
- 22.3
Yi-34B
- 20.2
Qwen1.5-14B
- 18.4
Mistral-7B-Instruct-v0.2
- 17.7
Zephyr 7B Beta
- 15.3
Gemma 7B
- 15.2
Qwen1.5-7B
- 14.5
Mistral 7B v0.1
- 13.6
Yi 6B
- 13.4
Llama 3-8B
- 9.25
LLaMA-13B
- 7.31
Gemma 2B
- 7.06
Phi-1.5
- 7.06
Qwen2-0.5B
- 6.55 GPT-J-6B
- 6.39
LLaMA-7B
- 6.34 GPT-Neo-2.7B
- 5.99 GPT-NeoX-20B
- 5.10
Falcon-7B
- 1
Qwen1.5-110B A100-80GB-275W / 4bit-gptq-exllama-v2-eager 29.6
- 2
phi-3-mini 3.8B A10-24GB-150W / 8bit-bnb-eager 27.2
- 3
Qwen1.5-32B A10-24GB-150W / 4bit-awq-gemm-fa2 26.7
- 4
Yi-34B A10-24GB-150W / 4bit-awq-gemm-eager 22.3
- 5
Qwen1.5-14B A10-24GB-150W / 4bit-awq-gemm-fa2 20.2
- 6
Mistral-7B-Instruct-v0.2 A10-24GB-150W / bfloat16-eager 18.4
- 7
Zephyr 7B Beta A10-24GB-150W / 8bit-bnb-eager 17.7
- 8
Gemma 7B A10-24GB-150W / 8bit-bnb-fa2 15.3
- 9
Qwen1.5-7B A10-24GB-150W / 4bit-awq-gemm-fa2 15.2
- 10
Mistral 7B v0.1 A10-24GB-150W / 4bit-awq-gemm-fa2 14.5
- 11
Yi 6B A10-24GB-150W / 4bit-awq-gemm-eager 13.6
- 12
Llama 3-8B A10-24GB-150W / 4bit-awq-gemm-eager 13.4
- 13
LLaMA-13B A10-24GB-150W / 4bit-awq-gemm-eager 9.25
- 14
Gemma 2B A10-24GB-150W / 8bit-bnb-eager 7.31
- 15
Phi-1.5 A10-24GB-150W / 4bit-awq-gemm-eager 7.06
- 16
Qwen2-0.5B A10-24GB-150W / 8bit-bnb-fa2 7.06
- 17 GPT-J-6B A10-24GB-150W / 4bit-awq-gemm-fa2 6.55
- 18
LLaMA-7B A10-24GB-150W / 4bit-awq-gemm-eager 6.39
- 19 GPT-Neo-2.7B A10-24GB-150W / 4bit-awq-gemm-eager 6.34
- 20 GPT-NeoX-20B A10-24GB-150W / 4bit-awq-gemm-eager 5.99
- 21
Falcon-7B A10-24GB-150W / 8bit-bnb-eager 5.10
Source: Official LLM-Perf Leaderboard leaderboard
| # | Entry | Score | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | | 29.6 | A100-80GB-275W / 4bit-gptq-exllama-v2-eager | Apr 2024 | huggingface.co | Official leaderboard |
| 2 | | 27.2 | A10-24GB-150W / 8bit-bnb-eager | Apr 2024 | huggingface.co | Official leaderboard |
| 3 | | 26.7 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Feb 2024 | huggingface.co | Official leaderboard |
| 4 | meta-llama/Meta-Llama-3-8B-Instruct | 23.9 | A10-24GB-150W / 8bit-bnb-eager | huggingface.co | Official leaderboard | |
| 5 | | 22.3 | A10-24GB-150W / 4bit-awq-gemm-eager | Nov 2023 | huggingface.co | Official leaderboard |
| 6 | | 20.2 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Feb 2024 | huggingface.co | Official leaderboard |
| 7 | | 18.4 | A10-24GB-150W / bfloat16-eager | Dec 2023 | huggingface.co | Official leaderboard |
| 8 | | 17.7 | A10-24GB-150W / 8bit-bnb-eager | Oct 2023 | huggingface.co | Official leaderboard |
| 9 | | 15.3 | A10-24GB-150W / 8bit-bnb-fa2 | Feb 2024 | huggingface.co | Official leaderboard |
| 10 | | 15.2 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Feb 2024 | huggingface.co | Official leaderboard |
| 11 | Deci/DeciLM-7B | 14.9 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 12 | | 14.5 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Sep 2023 | huggingface.co | Official leaderboard |
| 13 | TencentARC/Mistral_Pro_8B_v0.1 | 14.2 | A10-24GB-150W / 4bit-awq-gemm-fa2 | huggingface.co | Official leaderboard | |
| 14 | stabilityai/stablelm-2-12b | 13.9 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 15 | | 13.6 | A10-24GB-150W / 4bit-awq-gemm-eager | Nov 2023 | huggingface.co | Official leaderboard |
| 16 | google/recurrentgemma-9b | 13.5 | A10-24GB-150W / 8bit-bnb-eager | huggingface.co | Official leaderboard | |
| 17 | | 13.4 | A10-24GB-150W / 4bit-awq-gemm-eager | Apr 2024 | huggingface.co | Official leaderboard |
| 18 | Qwen/Qwen1.5-MoE-A2.7B | 12.4 | A10-24GB-150W / 8bit-bnb-eager | huggingface.co | Official leaderboard | |
| 19 | Qwen/Qwen1.5-4B | 11.3 | A10-24GB-150W / 4bit-awq-gemm-fa2 | huggingface.co | Official leaderboard | |
| 20 | meta-llama/Llama-2-13b-hf | 11.0 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 21 | google/gemma-2-2b | 10.3 | A10-24GB-150W / 8bit-bnb-fa2 | huggingface.co | Official leaderboard | |
| 22 | | 9.25 | A10-24GB-150W / 4bit-awq-gemm-eager | Feb 2023 | huggingface.co | Official leaderboard |
| 23 | Qwen/Qwen1.5-1.8B | 9.12 | A10-24GB-150W / 4bit-awq-gemm-fa2 | huggingface.co | Official leaderboard | |
| 24 | meta-llama/Llama-2-7b-hf | 8.72 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 25 | | 7.31 | A10-24GB-150W / 8bit-bnb-eager | Feb 2024 | huggingface.co | Official leaderboard |
| 26 | stabilityai/stablelm-3b-4e1t | 7.26 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 27 | | 7.06 | A10-24GB-150W / 4bit-awq-gemm-eager | Sep 2023 | huggingface.co | Official leaderboard |
| 28 | | 7.06 | A10-24GB-150W / 8bit-bnb-fa2 | Jun 2024 | huggingface.co | Official leaderboard |
| 29 | google/recurrentgemma-2b | 6.94 | A10-24GB-150W / 8bit-bnb-eager | huggingface.co | Official leaderboard | |
| 30 | GPT-J-6B | 6.55 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Aug 2021 | huggingface.co | Official leaderboard |
| 31 | openai-community/gpt2 | 6.51 | A10-24GB-150W / 8bit-bnb-eager | huggingface.co | Official leaderboard | |
| 32 | | 6.39 | A10-24GB-150W / 4bit-awq-gemm-eager | Feb 2023 | huggingface.co | Official leaderboard |
| 33 | GPT-Neo-2.7B | 6.34 | A10-24GB-150W / 4bit-awq-gemm-eager | Mar 2021 | huggingface.co | Official leaderboard |
| 34 | facebook/opt-30b | 6.20 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 35 | GPT-NeoX-20B | 5.99 | A10-24GB-150W / 4bit-awq-gemm-eager | Apr 2022 | huggingface.co | Official leaderboard |
| 36 | EleutherAI/pythia-12b | 5.93 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 37 | EleutherAI/pythia-160m | 5.62 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 38 | openai-community/gpt2-large | 5.48 | A10-24GB-150W / 8bit-bnb-eager | huggingface.co | Official leaderboard | |
| 39 | togethercomputer/RedPajama-INCITE-Base-3B-v1 | 5.43 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 40 | EleutherAI/gpt-neo-1.3B | 5.33 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 41 | stabilityai/stablelm-2-1_6b | 5.22 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 42 | Qwen/Qwen1.5-0.5B | 5.14 | A10-24GB-150W / 8bit-bnb-fa2 | huggingface.co | Official leaderboard | |
| 43 | EleutherAI/pythia-410m | 5.11 | A10-24GB-150W / 4bit-awq-gemm-eager | huggingface.co | Official leaderboard | |
| 44 | | 5.10 | A10-24GB-150W / 8bit-bnb-eager | Apr 2023 | huggingface.co | Official leaderboard |
| 45 | EleutherAI/gpt-neo-125m | 4.38 | A10-24GB-150W / 8bit-bnb-eager | huggingface.co | Official leaderboard | |
| 46 | distilbert/distilgpt2 | 3.90 | A10-24GB-150W / 8bit-bnb-eager | huggingface.co | Official leaderboard |
| # | Model | Score | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | | 29.6 | A100-80GB-275W / 4bit-gptq-exllama-v2-eager | Apr 2024 | huggingface.co | Official leaderboard |
| 2 | | 27.2 | A10-24GB-150W / 8bit-bnb-eager | Apr 2024 | huggingface.co | Official leaderboard |
| 3 | | 26.7 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Feb 2024 | huggingface.co | Official leaderboard |
| 4 | | 22.3 | A10-24GB-150W / 4bit-awq-gemm-eager | Nov 2023 | huggingface.co | Official leaderboard |
| 5 | | 20.2 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Feb 2024 | huggingface.co | Official leaderboard |
| 6 | | 18.4 | A10-24GB-150W / bfloat16-eager | Dec 2023 | huggingface.co | Official leaderboard |
| 7 | | 17.7 | A10-24GB-150W / 8bit-bnb-eager | Oct 2023 | huggingface.co | Official leaderboard |
| 8 | | 15.3 | A10-24GB-150W / 8bit-bnb-fa2 | Feb 2024 | huggingface.co | Official leaderboard |
| 9 | | 15.2 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Feb 2024 | huggingface.co | Official leaderboard |
| 10 | | 14.5 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Sep 2023 | huggingface.co | Official leaderboard |
| 11 | | 13.6 | A10-24GB-150W / 4bit-awq-gemm-eager | Nov 2023 | huggingface.co | Official leaderboard |
| 12 | | 13.4 | A10-24GB-150W / 4bit-awq-gemm-eager | Apr 2024 | huggingface.co | Official leaderboard |
| 13 | | 9.25 | A10-24GB-150W / 4bit-awq-gemm-eager | Feb 2023 | huggingface.co | Official leaderboard |
| 14 | | 7.31 | A10-24GB-150W / 8bit-bnb-eager | Feb 2024 | huggingface.co | Official leaderboard |
| 15 | | 7.06 | A10-24GB-150W / 4bit-awq-gemm-eager | Sep 2023 | huggingface.co | Official leaderboard |
| 16 | | 7.06 | A10-24GB-150W / 8bit-bnb-fa2 | Jun 2024 | huggingface.co | Official leaderboard |
| 17 | GPT-J-6B | 6.55 | A10-24GB-150W / 4bit-awq-gemm-fa2 | Aug 2021 | huggingface.co | Official leaderboard |
| 18 | | 6.39 | A10-24GB-150W / 4bit-awq-gemm-eager | Feb 2023 | huggingface.co | Official leaderboard |
| 19 | GPT-Neo-2.7B | 6.34 | A10-24GB-150W / 4bit-awq-gemm-eager | Mar 2021 | huggingface.co | Official leaderboard |
| 20 | GPT-NeoX-20B | 5.99 | A10-24GB-150W / 4bit-awq-gemm-eager | Apr 2022 | huggingface.co | Official leaderboard |
| 21 | | 5.10 | A10-24GB-150W / 8bit-bnb-eager | Apr 2023 | huggingface.co | Official leaderboard |
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Source: Official LLM-Perf Leaderboard leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 29.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 29.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 29.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 29.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 29.6 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 29.6 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 29.6 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 29.6 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 27.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 27.2 | ||||
| A10-24GB-150W / float16-fa2 | 27.2 | ||||
| A10-24GB-150W / bfloat16-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 27.2 | ||||
| A10-24GB-150W / float32-eager | 27.2 | ||||
| A10-24GB-150W / float16-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 27.2 | ||||
| A10-24GB-150W / bfloat16-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 27.2 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 27.2 | ||||
| T4-16GB-70W / float16-eager | 27.2 | ||||
| T4-16GB-70W / bfloat16-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 27.2 | ||||
| Intel-Xeon-SPR-385W / float16-eager-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / float32-eager-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 27.2 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 27.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 26.7 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 26.7 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 26.7 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 26.7 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 26.7 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 26.7 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 26.7 | ||||
| A100-80GB-275W / bfloat16-sdpa | 26.7 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 26.0 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 26.0 | ||||
| A10-24GB-150W / float16-fa2 | 26.0 | ||||
| A10-24GB-150W / bfloat16-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 26.0 | ||||
| A10-24GB-150W / float32-eager | 26.0 | ||||
| A10-24GB-150W / float16-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 26.0 | ||||
| A10-24GB-150W / bfloat16-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 26.0 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 26.0 | ||||
| T4-16GB-70W / float16-eager | 26.0 | ||||
| T4-16GB-70W / bfloat16-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 26.0 | ||||
| Intel-Xeon-SPR-385W / float16-eager-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / float32-eager-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 26.0 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 26.0 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 22.3 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 22.3 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 22.3 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 22.3 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 22.3 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 22.3 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 22.3 | ||||
| A100-80GB-275W / bfloat16-eager | 22.3 | ||||
| A100-80GB-275W / float16-eager | 22.3 | ||||
| A100-80GB-275W / float16-sdpa | 22.3 | ||||
| A100-80GB-275W / bfloat16-sdpa | 22.3 | ||||
| A100-80GB-275W / bfloat16-fa2 | 22.3 | ||||
| A100-80GB-275W / float16-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 20.2 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 20.2 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 20.2 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 20.2 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 20.2 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 20.2 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 20.2 | ||||
| A10-24GB-150W / bfloat16-eager | 18.4 | ||||
| A10-24GB-150W / float16-fa2 | 18.4 | ||||
| A10-24GB-150W / float16-eager | 18.4 | ||||
| A10-24GB-150W / bfloat16-fa2 | 18.4 | ||||
| A10-24GB-150W / bfloat16-sdpa | 18.4 | ||||
| A10-24GB-150W / float16-sdpa | 18.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 18.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 18.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 18.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 18.4 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 17.7 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 17.7 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 17.7 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 17.7 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 17.7 | ||||
| A10-24GB-150W / float16-fa2 | 17.7 | ||||
| A10-24GB-150W / bfloat16-fa2 | 17.7 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 17.7 | ||||
| A10-24GB-150W / bfloat16-eager | 17.7 | ||||
| A10-24GB-150W / float16-eager | 17.7 | ||||
| A10-24GB-150W / bfloat16-sdpa | 17.7 | ||||
| A10-24GB-150W / float16-sdpa | 17.7 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 17.7 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 17.7 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 17.7 | ||||
| T4-16GB-70W / bfloat16-sdpa | 17.7 | ||||
| T4-16GB-70W / bfloat16-eager | 17.7 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 17.7 | ||||
| T4-16GB-70W / float16-sdpa | 17.7 | ||||
| T4-16GB-70W / float16-eager | 17.7 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 17.7 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 17.7 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 17.7 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 17.7 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 17.7 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 17.7 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 17.7 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 17.7 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 17.7 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 17.7 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 15.3 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 15.3 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 15.3 | ||||
| A10-24GB-150W / float16-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 15.3 | ||||
| A10-24GB-150W / bfloat16-fa2 | 15.3 | ||||
| A10-24GB-150W / float16-fa2 | 15.3 | ||||
| A10-24GB-150W / bfloat16-fa2 | 15.3 | ||||
| A10-24GB-150W / bfloat16-eager | 15.3 | ||||
| A10-24GB-150W / bfloat16-eager | 15.3 | ||||
| A10-24GB-150W / float16-eager | 15.3 | ||||
| A10-24GB-150W / float16-eager | 15.3 | ||||
| A10-24GB-150W / bfloat16-sdpa | 15.3 | ||||
| A10-24GB-150W / bfloat16-sdpa | 15.3 | ||||
| A10-24GB-150W / float16-sdpa | 15.3 | ||||
| A10-24GB-150W / float16-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 15.3 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 15.3 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 15.3 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 15.2 | ||||
| A10-24GB-150W / float16-fa2 | 15.2 | ||||
| A10-24GB-150W / bfloat16-fa2 | 15.2 | ||||
| A10-24GB-150W / float16-fa2 | 15.2 | ||||
| A10-24GB-150W / bfloat16-fa2 | 15.2 | ||||
| A10-24GB-150W / bfloat16-eager | 15.2 | ||||
| A10-24GB-150W / float16-eager | 15.2 | ||||
| A10-24GB-150W / bfloat16-sdpa | 15.2 | ||||
| A10-24GB-150W / float16-sdpa | 15.2 | ||||
| A10-24GB-150W / bfloat16-eager | 15.2 | ||||
| A10-24GB-150W / float16-eager | 15.2 | ||||
| A10-24GB-150W / bfloat16-sdpa | 15.2 | ||||
| A10-24GB-150W / float16-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 15.2 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 15.2 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 15.2 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 15.2 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 15.2 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 15.2 | ||||
| A100-80GB-275W / bfloat16-eager | 15.2 | ||||
| A100-80GB-275W / float16-eager | 15.2 | ||||
| A100-80GB-275W / float32-sdpa | 15.2 | ||||
| A100-80GB-275W / bfloat16-sdpa | 15.2 | ||||
| A100-80GB-275W / float16-sdpa | 15.2 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 15.2 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 15.2 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 15.2 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 15.2 | ||||
| T4-16GB-70W / bfloat16-sdpa | 15.2 | ||||
| T4-16GB-70W / bfloat16-eager | 15.2 | ||||
| T4-16GB-70W / float16-sdpa | 15.2 | ||||
| T4-16GB-70W / float16-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 15.2 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 14.5 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 14.5 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 14.5 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 14.5 | ||||
| A10-24GB-150W / bfloat16-fa2 | 14.5 | ||||
| A10-24GB-150W / float16-fa2 | 14.5 | ||||
| A10-24GB-150W / bfloat16-eager | 14.5 | ||||
| A10-24GB-150W / float16-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 14.5 | ||||
| A10-24GB-150W / bfloat16-sdpa | 14.5 | ||||
| A10-24GB-150W / float16-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 14.5 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 14.5 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 14.5 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 14.5 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 14.5 | ||||
| T4-16GB-70W / bfloat16-sdpa | 14.5 | ||||
| T4-16GB-70W / bfloat16-eager | 14.5 | ||||
| T4-16GB-70W / float16-sdpa | 14.5 | ||||
| T4-16GB-70W / float16-eager | 14.5 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 14.5 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 14.5 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 14.5 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 14.5 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 13.6 | ||||
| A10-24GB-150W / float16-fa2 | 13.6 | ||||
| A10-24GB-150W / bfloat16-fa2 | 13.6 | ||||
| A10-24GB-150W / bfloat16-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 13.6 | ||||
| A10-24GB-150W / float16-eager | 13.6 | ||||
| A10-24GB-150W / bfloat16-sdpa | 13.6 | ||||
| A10-24GB-150W / float16-sdpa | 13.6 | ||||
| A10-24GB-150W / bfloat16-fa2 | 13.6 | ||||
| A10-24GB-150W / float16-fa2 | 13.6 | ||||
| A10-24GB-150W / float16-eager | 13.6 | ||||
| A10-24GB-150W / bfloat16-eager | 13.6 | ||||
| A10-24GB-150W / float16-sdpa | 13.6 | ||||
| A10-24GB-150W / bfloat16-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 13.6 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 13.6 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 13.6 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 13.6 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 13.6 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 13.6 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 13.6 | ||||
| A100-80GB-275W / float16-eager | 13.6 | ||||
| A100-80GB-275W / bfloat16-eager | 13.6 | ||||
| A100-80GB-275W / float16-sdpa | 13.6 | ||||
| A100-80GB-275W / float32-eager | 13.6 | ||||
| A100-80GB-275W / bfloat16-sdpa | 13.6 | ||||
| A100-80GB-275W / float16-fa2 | 13.6 | ||||
| A100-80GB-275W / bfloat16-fa2 | 13.6 | ||||
| A100-80GB-275W / float32-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 13.6 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 13.6 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 13.6 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 13.6 | ||||
| T4-16GB-70W / bfloat16-sdpa | 13.6 | ||||
| T4-16GB-70W / bfloat16-eager | 13.6 | ||||
| T4-16GB-70W / bfloat16-eager | 13.6 | ||||
| T4-16GB-70W / bfloat16-sdpa | 13.6 | ||||
| T4-16GB-70W / float16-sdpa | 13.6 | ||||
| T4-16GB-70W / float16-eager | 13.6 | ||||
| T4-16GB-70W / float16-sdpa | 13.6 | ||||
| T4-16GB-70W / float16-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 13.4 | ||||
| A10-24GB-150W / float16-fa2 | 13.4 | ||||
| A10-24GB-150W / bfloat16-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 13.4 | ||||
| A10-24GB-150W / bfloat16-eager | 13.4 | ||||
| A10-24GB-150W / float16-eager | 13.4 | ||||
| A10-24GB-150W / bfloat16-sdpa | 13.4 | ||||
| A10-24GB-150W / float16-sdpa | 13.4 | ||||
| A10-24GB-150W / bfloat16-eager | 13.4 | ||||
| A10-24GB-150W / float16-eager | 13.4 | ||||
| A10-24GB-150W / bfloat16-fa2 | 13.4 | ||||
| A10-24GB-150W / float16-fa2 | 13.4 | ||||
| A10-24GB-150W / bfloat16-sdpa | 13.4 | ||||
| A10-24GB-150W / float16-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 13.4 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 13.4 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 13.4 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 13.4 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 13.4 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 13.4 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 13.4 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 13.4 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 13.4 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 13.4 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 9.25 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 9.25 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 9.25 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 9.25 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 9.25 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 9.25 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 9.25 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 9.25 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 9.25 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 9.25 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 9.25 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 9.25 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 9.25 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 7.31 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 7.31 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 7.31 | ||||
| A10-24GB-150W / float32-sdpa | 7.31 | ||||
| A10-24GB-150W / float32-sdpa | 7.31 | ||||
| A10-24GB-150W / float32-eager | 7.31 | ||||
| A10-24GB-150W / float32-eager | 7.31 | ||||
| A10-24GB-150W / bfloat16-eager | 7.31 | ||||
| A10-24GB-150W / float16-eager | 7.31 | ||||
| A10-24GB-150W / bfloat16-eager | 7.31 | ||||
| A10-24GB-150W / float16-eager | 7.31 | ||||
| A10-24GB-150W / bfloat16-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 7.31 | ||||
| A10-24GB-150W / float16-fa2 | 7.31 | ||||
| A10-24GB-150W / float16-fa2 | 7.31 | ||||
| A10-24GB-150W / bfloat16-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 7.31 | ||||
| A10-24GB-150W / float16-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 7.31 | ||||
| A10-24GB-150W / float16-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 7.31 | ||||
| A10-24GB-150W / bfloat16-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 7.31 | ||||
| A10-24GB-150W / bfloat16-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 7.31 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 7.31 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 7.31 | ||||
| T4-16GB-70W / float32-sdpa | 7.31 | ||||
| T4-16GB-70W / float32-sdpa | 7.31 | ||||
| T4-16GB-70W / float32-eager | 7.31 | ||||
| T4-16GB-70W / float32-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 7.31 | ||||
| T4-16GB-70W / float16-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 7.31 | ||||
| T4-16GB-70W / float16-eager | 7.31 | ||||
| T4-16GB-70W / bfloat16-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 7.31 | ||||
| T4-16GB-70W / bfloat16-eager | 7.31 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.31 | ||||
| T4-16GB-70W / float16-sdpa | 7.31 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.31 | ||||
| T4-16GB-70W / float16-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 7.31 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 7.06 | ||||
| A10-24GB-150W / bfloat16-eager | 7.06 | ||||
| A10-24GB-150W / float16-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 7.06 | ||||
| A10-24GB-150W / bfloat16-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 7.06 | ||||
| A10-24GB-150W / float32-eager | 7.06 | ||||
| A10-24GB-150W / float16-fa2 | 7.06 | ||||
| A10-24GB-150W / bfloat16-sdpa | 7.06 | ||||
| A10-24GB-150W / float16-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 7.06 | ||||
| A10-24GB-150W / float32-sdpa | 7.06 | ||||
| A100-80GB-275W / float32-eager | 7.06 | ||||
| A100-80GB-275W / bfloat16-eager | 7.06 | ||||
| A100-80GB-275W / float16-eager | 7.06 | ||||
| A100-80GB-275W / float16-fa2 | 7.06 | ||||
| A100-80GB-275W / bfloat16-fa2 | 7.06 | ||||
| A100-80GB-275W / float16-sdpa | 7.06 | ||||
| A100-80GB-275W / bfloat16-sdpa | 7.06 | ||||
| A100-80GB-275W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 7.06 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 7.06 | ||||
| T4-16GB-70W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / float32-eager | 7.06 | ||||
| T4-16GB-70W / float32-eager | 7.06 | ||||
| T4-16GB-70W / float16-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 7.06 | ||||
| T4-16GB-70W / float16-eager | 7.06 | ||||
| T4-16GB-70W / bfloat16-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 7.06 | ||||
| T4-16GB-70W / bfloat16-eager | 7.06 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.06 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.06 | ||||
| T4-16GB-70W / float16-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 7.06 | ||||
| T4-16GB-70W / float16-sdpa | 7.06 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 7.06 | ||||
| A10-24GB-150W / float16-fa2 | 7.06 | ||||
| A10-24GB-150W / float16-eager | 7.06 | ||||
| A10-24GB-150W / bfloat16-eager | 7.06 | ||||
| A10-24GB-150W / bfloat16-fa2 | 7.06 | ||||
| A10-24GB-150W / float16-sdpa | 7.06 | ||||
| A10-24GB-150W / float32-eager | 7.06 | ||||
| A10-24GB-150W / bfloat16-sdpa | 7.06 | ||||
| A10-24GB-150W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 7.06 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 7.06 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.06 | ||||
| T4-16GB-70W / float16-eager | 7.06 | ||||
| T4-16GB-70W / bfloat16-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 7.06 | ||||
| T4-16GB-70W / float32-eager | 7.06 | ||||
| T4-16GB-70W / float16-sdpa | 7.06 | ||||
| T4-16GB-70W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 7.06 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 7.06 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 7.06 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 7.06 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 7.06 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 7.06 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 7.06 | ||||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemm-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 8bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 8bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / float16-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / bfloat16-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / float16-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / bfloat16-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemm-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / float16-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / float16-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemv-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemv-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 8bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / bfloat16-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / float16-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / float16-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / float32-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / float16-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / float16-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-gemm-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-gemm-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-gemv-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-gemv-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float16-eager-pytorch | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float32-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float32-eager-pytorch | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float32-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float16-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float16-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / bfloat16-eager | 6.55 | |||
| A10-24GB-150W / 4bit-awq-gemm-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 6.39 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 6.39 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 6.39 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 6.39 | ||||
| A10-24GB-150W / float16-fa2 | 6.39 | ||||
| A10-24GB-150W / bfloat16-fa2 | 6.39 | ||||
| A10-24GB-150W / bfloat16-eager | 6.39 | ||||
| A10-24GB-150W / float16-eager | 6.39 | ||||
| A10-24GB-150W / float16-fa2 | 6.39 | ||||
| A10-24GB-150W / bfloat16-fa2 | 6.39 | ||||
| A10-24GB-150W / bfloat16-sdpa | 6.39 | ||||
| A10-24GB-150W / float16-sdpa | 6.39 | ||||
| A10-24GB-150W / bfloat16-eager | 6.39 | ||||
| A10-24GB-150W / float16-eager | 6.39 | ||||
| A10-24GB-150W / bfloat16-sdpa | 6.39 | ||||
| A10-24GB-150W / float16-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 6.39 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 6.39 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 6.39 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 6.39 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 6.39 | ||||
| T4-16GB-70W / bfloat16-sdpa | 6.39 | ||||
| T4-16GB-70W / bfloat16-sdpa | 6.39 | ||||
| T4-16GB-70W / bfloat16-eager | 6.39 | ||||
| T4-16GB-70W / bfloat16-eager | 6.39 | ||||
| T4-16GB-70W / float16-eager | 6.39 | ||||
| T4-16GB-70W / float16-sdpa | 6.39 | ||||
| T4-16GB-70W / float16-eager | 6.39 | ||||
| T4-16GB-70W / float16-sdpa | 6.39 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.39 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.39 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 6.39 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 6.39 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 6.39 | ||||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemm-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 8bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 8bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemm-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemv-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemv-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / bfloat16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / bfloat16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 8bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / float16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / bfloat16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-gemm-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-gemm-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-gemv-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-gemv-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float16-eager-pytorch | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float32-eager-pytorch | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / bfloat16-eager | 6.34 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 8bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 8bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / float32-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / float16-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / bfloat16-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / float16-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / bfloat16-fa2 | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemm-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemm-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemv-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemv-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemm-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemv-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemm-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemv-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / float16-eager-pytorch | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / float16-sdpa | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / float16-eager | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / bfloat16-eager | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / bfloat16-sdpa | 5.99 | |||
| A10-24GB-150W / 8bit-bnb-eager | 5.10 | ||||
| A10-24GB-150W / float16-eager | 5.10 | ||||
| A10-24GB-150W / bfloat16-eager | 5.10 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 5.10 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 5.10 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 5.10 | ||||
| T4-16GB-70W / bfloat16-eager | 5.10 | ||||
| T4-16GB-70W / float16-eager | 5.10 |
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Source: Official LLM-Perf Leaderboard leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 29.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 29.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 29.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 29.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 29.6 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 29.6 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 29.6 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 29.6 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 27.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 27.2 | ||||
| A10-24GB-150W / float16-fa2 | 27.2 | ||||
| A10-24GB-150W / bfloat16-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 27.2 | ||||
| A10-24GB-150W / float32-eager | 27.2 | ||||
| A10-24GB-150W / float16-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 27.2 | ||||
| A10-24GB-150W / bfloat16-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 27.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 27.2 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 27.2 | ||||
| T4-16GB-70W / float16-eager | 27.2 | ||||
| T4-16GB-70W / bfloat16-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 27.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 27.2 | ||||
| Intel-Xeon-SPR-385W / float16-eager-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / float32-eager-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-onnxruntime | 27.2 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 27.2 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 27.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 27.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 26.7 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 26.7 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 26.7 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 26.7 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 26.7 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 26.7 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 26.7 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 26.7 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 26.7 | ||||
| A100-80GB-275W / bfloat16-sdpa | 26.7 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 26.0 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 26.0 | ||||
| A10-24GB-150W / float16-fa2 | 26.0 | ||||
| A10-24GB-150W / bfloat16-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 26.0 | ||||
| A10-24GB-150W / float32-eager | 26.0 | ||||
| A10-24GB-150W / float16-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 26.0 | ||||
| A10-24GB-150W / bfloat16-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 26.0 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 26.0 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 26.0 | ||||
| T4-16GB-70W / float16-eager | 26.0 | ||||
| T4-16GB-70W / bfloat16-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 26.0 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 26.0 | ||||
| Intel-Xeon-SPR-385W / float16-eager-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / float32-eager-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-onnxruntime | 26.0 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 26.0 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 26.0 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 26.0 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 22.3 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 22.3 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 22.3 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 22.3 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 22.3 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 22.3 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 22.3 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 22.3 | ||||
| A100-80GB-275W / bfloat16-eager | 22.3 | ||||
| A100-80GB-275W / float16-eager | 22.3 | ||||
| A100-80GB-275W / float16-sdpa | 22.3 | ||||
| A100-80GB-275W / bfloat16-sdpa | 22.3 | ||||
| A100-80GB-275W / bfloat16-fa2 | 22.3 | ||||
| A100-80GB-275W / float16-fa2 | 22.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 20.2 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 20.2 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 20.2 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 20.2 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 20.2 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 20.2 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 20.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 20.2 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 20.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 20.2 | ||||
| A10-24GB-150W / bfloat16-eager | 18.4 | ||||
| A10-24GB-150W / float16-fa2 | 18.4 | ||||
| A10-24GB-150W / float16-eager | 18.4 | ||||
| A10-24GB-150W / bfloat16-fa2 | 18.4 | ||||
| A10-24GB-150W / bfloat16-sdpa | 18.4 | ||||
| A10-24GB-150W / float16-sdpa | 18.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 18.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 18.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 18.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 18.4 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 17.7 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 17.7 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 17.7 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 17.7 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 17.7 | ||||
| A10-24GB-150W / float16-fa2 | 17.7 | ||||
| A10-24GB-150W / bfloat16-fa2 | 17.7 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 17.7 | ||||
| A10-24GB-150W / bfloat16-eager | 17.7 | ||||
| A10-24GB-150W / float16-eager | 17.7 | ||||
| A10-24GB-150W / bfloat16-sdpa | 17.7 | ||||
| A10-24GB-150W / float16-sdpa | 17.7 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 17.7 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 17.7 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 17.7 | ||||
| T4-16GB-70W / bfloat16-sdpa | 17.7 | ||||
| T4-16GB-70W / bfloat16-eager | 17.7 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 17.7 | ||||
| T4-16GB-70W / float16-sdpa | 17.7 | ||||
| T4-16GB-70W / float16-eager | 17.7 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 17.7 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 17.7 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 17.7 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 17.7 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 17.7 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 17.7 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 17.7 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 17.7 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 17.7 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 17.7 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 15.3 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 15.3 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 15.3 | ||||
| A10-24GB-150W / float16-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 15.3 | ||||
| A10-24GB-150W / bfloat16-fa2 | 15.3 | ||||
| A10-24GB-150W / float16-fa2 | 15.3 | ||||
| A10-24GB-150W / bfloat16-fa2 | 15.3 | ||||
| A10-24GB-150W / bfloat16-eager | 15.3 | ||||
| A10-24GB-150W / bfloat16-eager | 15.3 | ||||
| A10-24GB-150W / float16-eager | 15.3 | ||||
| A10-24GB-150W / float16-eager | 15.3 | ||||
| A10-24GB-150W / bfloat16-sdpa | 15.3 | ||||
| A10-24GB-150W / bfloat16-sdpa | 15.3 | ||||
| A10-24GB-150W / float16-sdpa | 15.3 | ||||
| A10-24GB-150W / float16-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 15.3 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 15.3 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 15.3 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 15.3 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 15.3 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 15.3 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 15.3 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 15.3 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 15.2 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 15.2 | ||||
| A10-24GB-150W / float16-fa2 | 15.2 | ||||
| A10-24GB-150W / bfloat16-fa2 | 15.2 | ||||
| A10-24GB-150W / float16-fa2 | 15.2 | ||||
| A10-24GB-150W / bfloat16-fa2 | 15.2 | ||||
| A10-24GB-150W / bfloat16-eager | 15.2 | ||||
| A10-24GB-150W / float16-eager | 15.2 | ||||
| A10-24GB-150W / bfloat16-sdpa | 15.2 | ||||
| A10-24GB-150W / float16-sdpa | 15.2 | ||||
| A10-24GB-150W / bfloat16-eager | 15.2 | ||||
| A10-24GB-150W / float16-eager | 15.2 | ||||
| A10-24GB-150W / bfloat16-sdpa | 15.2 | ||||
| A10-24GB-150W / float16-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 15.2 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 15.2 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 15.2 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 15.2 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 15.2 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 15.2 | ||||
| A100-80GB-275W / bfloat16-eager | 15.2 | ||||
| A100-80GB-275W / float16-eager | 15.2 | ||||
| A100-80GB-275W / float32-sdpa | 15.2 | ||||
| A100-80GB-275W / bfloat16-sdpa | 15.2 | ||||
| A100-80GB-275W / float16-sdpa | 15.2 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 15.2 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 15.2 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 15.2 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 15.2 | ||||
| T4-16GB-70W / bfloat16-sdpa | 15.2 | ||||
| T4-16GB-70W / bfloat16-eager | 15.2 | ||||
| T4-16GB-70W / float16-sdpa | 15.2 | ||||
| T4-16GB-70W / float16-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 15.2 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 15.2 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 15.2 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 15.2 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 14.5 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 14.5 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 14.5 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 14.5 | ||||
| A10-24GB-150W / bfloat16-fa2 | 14.5 | ||||
| A10-24GB-150W / float16-fa2 | 14.5 | ||||
| A10-24GB-150W / bfloat16-eager | 14.5 | ||||
| A10-24GB-150W / float16-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 14.5 | ||||
| A10-24GB-150W / bfloat16-sdpa | 14.5 | ||||
| A10-24GB-150W / float16-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 14.5 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 14.5 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 14.5 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 14.5 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 14.5 | ||||
| T4-16GB-70W / bfloat16-sdpa | 14.5 | ||||
| T4-16GB-70W / bfloat16-eager | 14.5 | ||||
| T4-16GB-70W / float16-sdpa | 14.5 | ||||
| T4-16GB-70W / float16-eager | 14.5 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 14.5 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 14.5 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 14.5 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 14.5 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 14.5 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 14.5 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 13.6 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 13.6 | ||||
| A10-24GB-150W / float16-fa2 | 13.6 | ||||
| A10-24GB-150W / bfloat16-fa2 | 13.6 | ||||
| A10-24GB-150W / bfloat16-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 13.6 | ||||
| A10-24GB-150W / float16-eager | 13.6 | ||||
| A10-24GB-150W / bfloat16-sdpa | 13.6 | ||||
| A10-24GB-150W / float16-sdpa | 13.6 | ||||
| A10-24GB-150W / bfloat16-fa2 | 13.6 | ||||
| A10-24GB-150W / float16-fa2 | 13.6 | ||||
| A10-24GB-150W / float16-eager | 13.6 | ||||
| A10-24GB-150W / bfloat16-eager | 13.6 | ||||
| A10-24GB-150W / float16-sdpa | 13.6 | ||||
| A10-24GB-150W / bfloat16-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 13.6 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 13.6 | ||||
| A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 13.6 | ||||
| A100-80GB-275W / 8bit-bnb-fa2 | 13.6 | ||||
| A100-80GB-275W / 8bit-bnb-sdpa | 13.6 | ||||
| A100-80GB-275W / 8bit-bnb-eager | 13.6 | ||||
| A100-80GB-275W / 4bit-bnb-eager | 13.6 | ||||
| A100-80GB-275W / 4bit-bnb-fa2 | 13.6 | ||||
| A100-80GB-275W / 4bit-bnb-sdpa | 13.6 | ||||
| A100-80GB-275W / float16-eager | 13.6 | ||||
| A100-80GB-275W / bfloat16-eager | 13.6 | ||||
| A100-80GB-275W / float16-sdpa | 13.6 | ||||
| A100-80GB-275W / float32-eager | 13.6 | ||||
| A100-80GB-275W / bfloat16-sdpa | 13.6 | ||||
| A100-80GB-275W / float16-fa2 | 13.6 | ||||
| A100-80GB-275W / bfloat16-fa2 | 13.6 | ||||
| A100-80GB-275W / float32-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 13.6 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 13.6 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 13.6 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 13.6 | ||||
| T4-16GB-70W / bfloat16-sdpa | 13.6 | ||||
| T4-16GB-70W / bfloat16-eager | 13.6 | ||||
| T4-16GB-70W / bfloat16-eager | 13.6 | ||||
| T4-16GB-70W / bfloat16-sdpa | 13.6 | ||||
| T4-16GB-70W / float16-sdpa | 13.6 | ||||
| T4-16GB-70W / float16-eager | 13.6 | ||||
| T4-16GB-70W / float16-sdpa | 13.6 | ||||
| T4-16GB-70W / float16-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 13.6 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 13.6 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 13.6 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 13.4 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 13.4 | ||||
| A10-24GB-150W / float16-fa2 | 13.4 | ||||
| A10-24GB-150W / bfloat16-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 13.4 | ||||
| A10-24GB-150W / bfloat16-eager | 13.4 | ||||
| A10-24GB-150W / float16-eager | 13.4 | ||||
| A10-24GB-150W / bfloat16-sdpa | 13.4 | ||||
| A10-24GB-150W / float16-sdpa | 13.4 | ||||
| A10-24GB-150W / bfloat16-eager | 13.4 | ||||
| A10-24GB-150W / float16-eager | 13.4 | ||||
| A10-24GB-150W / bfloat16-fa2 | 13.4 | ||||
| A10-24GB-150W / float16-fa2 | 13.4 | ||||
| A10-24GB-150W / bfloat16-sdpa | 13.4 | ||||
| A10-24GB-150W / float16-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 13.4 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 13.4 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 13.4 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 13.4 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 13.4 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 13.4 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 13.4 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 13.4 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 13.4 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 13.4 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 13.4 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 13.4 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 13.4 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 9.25 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 9.25 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 9.25 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 9.25 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 9.25 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 9.25 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 9.25 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 9.25 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 9.25 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 9.25 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 9.25 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 9.25 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 9.25 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 9.25 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 9.25 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 7.31 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 7.31 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 7.31 | ||||
| A10-24GB-150W / float32-sdpa | 7.31 | ||||
| A10-24GB-150W / float32-sdpa | 7.31 | ||||
| A10-24GB-150W / float32-eager | 7.31 | ||||
| A10-24GB-150W / float32-eager | 7.31 | ||||
| A10-24GB-150W / bfloat16-eager | 7.31 | ||||
| A10-24GB-150W / float16-eager | 7.31 | ||||
| A10-24GB-150W / bfloat16-eager | 7.31 | ||||
| A10-24GB-150W / float16-eager | 7.31 | ||||
| A10-24GB-150W / bfloat16-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 7.31 | ||||
| A10-24GB-150W / float16-fa2 | 7.31 | ||||
| A10-24GB-150W / float16-fa2 | 7.31 | ||||
| A10-24GB-150W / bfloat16-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 7.31 | ||||
| A10-24GB-150W / float16-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 7.31 | ||||
| A10-24GB-150W / float16-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 7.31 | ||||
| A10-24GB-150W / bfloat16-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 7.31 | ||||
| A10-24GB-150W / bfloat16-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 7.31 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 7.31 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 7.31 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 7.31 | ||||
| T4-16GB-70W / float32-sdpa | 7.31 | ||||
| T4-16GB-70W / float32-sdpa | 7.31 | ||||
| T4-16GB-70W / float32-eager | 7.31 | ||||
| T4-16GB-70W / float32-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-gemm-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-gemm-sdpa | 7.31 | ||||
| T4-16GB-70W / float16-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-gemv-eager | 7.31 | ||||
| T4-16GB-70W / float16-eager | 7.31 | ||||
| T4-16GB-70W / bfloat16-eager | 7.31 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 7.31 | ||||
| T4-16GB-70W / bfloat16-eager | 7.31 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.31 | ||||
| T4-16GB-70W / float16-sdpa | 7.31 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.31 | ||||
| T4-16GB-70W / float16-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-gemv-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 7.31 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 7.31 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 7.31 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 7.31 | ||||
| A10-24GB-150W / 4bit-awq-gemm-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 7.06 | ||||
| A10-24GB-150W / bfloat16-eager | 7.06 | ||||
| A10-24GB-150W / float16-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 7.06 | ||||
| A10-24GB-150W / bfloat16-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 7.06 | ||||
| A10-24GB-150W / float32-eager | 7.06 | ||||
| A10-24GB-150W / float16-fa2 | 7.06 | ||||
| A10-24GB-150W / bfloat16-sdpa | 7.06 | ||||
| A10-24GB-150W / float16-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 7.06 | ||||
| A10-24GB-150W / float32-sdpa | 7.06 | ||||
| A100-80GB-275W / float32-eager | 7.06 | ||||
| A100-80GB-275W / bfloat16-eager | 7.06 | ||||
| A100-80GB-275W / float16-eager | 7.06 | ||||
| A100-80GB-275W / float16-fa2 | 7.06 | ||||
| A100-80GB-275W / bfloat16-fa2 | 7.06 | ||||
| A100-80GB-275W / float16-sdpa | 7.06 | ||||
| A100-80GB-275W / bfloat16-sdpa | 7.06 | ||||
| A100-80GB-275W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 7.06 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 7.06 | ||||
| T4-16GB-70W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / float32-eager | 7.06 | ||||
| T4-16GB-70W / float32-eager | 7.06 | ||||
| T4-16GB-70W / float16-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 7.06 | ||||
| T4-16GB-70W / float16-eager | 7.06 | ||||
| T4-16GB-70W / bfloat16-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 7.06 | ||||
| T4-16GB-70W / bfloat16-eager | 7.06 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.06 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.06 | ||||
| T4-16GB-70W / float16-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 7.06 | ||||
| T4-16GB-70W / float16-sdpa | 7.06 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 7.06 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 7.06 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 7.06 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 7.06 | ||||
| A10-24GB-150W / float16-fa2 | 7.06 | ||||
| A10-24GB-150W / float16-eager | 7.06 | ||||
| A10-24GB-150W / bfloat16-eager | 7.06 | ||||
| A10-24GB-150W / bfloat16-fa2 | 7.06 | ||||
| A10-24GB-150W / float16-sdpa | 7.06 | ||||
| A10-24GB-150W / float32-eager | 7.06 | ||||
| A10-24GB-150W / bfloat16-sdpa | 7.06 | ||||
| A10-24GB-150W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 7.06 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 7.06 | ||||
| T4-16GB-70W / bfloat16-sdpa | 7.06 | ||||
| T4-16GB-70W / float16-eager | 7.06 | ||||
| T4-16GB-70W / bfloat16-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 7.06 | ||||
| T4-16GB-70W / float32-eager | 7.06 | ||||
| T4-16GB-70W / float16-sdpa | 7.06 | ||||
| T4-16GB-70W / float32-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 7.06 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 7.06 | ||||
| Intel-Xeon-SPR-385W / float16-eager | 7.06 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa | 7.06 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa | 7.06 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager | 7.06 | ||||
| Intel-Xeon-SPR-385W / float32-eager | 7.06 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa | 7.06 | ||||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemm-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 8bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 8bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / float16-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / bfloat16-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / float16-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / bfloat16-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemm-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / float16-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / float16-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemv-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-gemv-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 8bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-bnb-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / bfloat16-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / float16-fa2 | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / float16-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | A100-80GB-275W / float32-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 8bit-bnb-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-bnb-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / float16-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / float16-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-gemm-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-gemm-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-gemv-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-gemv-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 6.55 | |||
| GPT-J-6B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float16-eager-pytorch | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float32-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float32-eager-pytorch | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float32-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float16-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / float16-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / bfloat16-eager | 6.55 | |||
| GPT-J-6B | Intel-Xeon-SPR-385W / bfloat16-eager | 6.55 | |||
| A10-24GB-150W / 4bit-awq-gemm-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemm-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 6.39 | ||||
| A10-24GB-150W / 8bit-bnb-eager | 6.39 | ||||
| A10-24GB-150W / 8bit-bnb-fa2 | 6.39 | ||||
| A10-24GB-150W / 8bit-bnb-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-bnb-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-bnb-sdpa | 6.39 | ||||
| A10-24GB-150W / float16-fa2 | 6.39 | ||||
| A10-24GB-150W / bfloat16-fa2 | 6.39 | ||||
| A10-24GB-150W / bfloat16-eager | 6.39 | ||||
| A10-24GB-150W / float16-eager | 6.39 | ||||
| A10-24GB-150W / float16-fa2 | 6.39 | ||||
| A10-24GB-150W / bfloat16-fa2 | 6.39 | ||||
| A10-24GB-150W / bfloat16-sdpa | 6.39 | ||||
| A10-24GB-150W / float16-sdpa | 6.39 | ||||
| A10-24GB-150W / bfloat16-eager | 6.39 | ||||
| A10-24GB-150W / float16-eager | 6.39 | ||||
| A10-24GB-150W / bfloat16-sdpa | 6.39 | ||||
| A10-24GB-150W / float16-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemv-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-gemv-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 6.39 | ||||
| A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 6.39 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 6.39 | ||||
| T4-16GB-70W / 8bit-bnb-sdpa | 6.39 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 6.39 | ||||
| T4-16GB-70W / 4bit-bnb-sdpa | 6.39 | ||||
| T4-16GB-70W / bfloat16-sdpa | 6.39 | ||||
| T4-16GB-70W / bfloat16-sdpa | 6.39 | ||||
| T4-16GB-70W / bfloat16-eager | 6.39 | ||||
| T4-16GB-70W / bfloat16-eager | 6.39 | ||||
| T4-16GB-70W / float16-eager | 6.39 | ||||
| T4-16GB-70W / float16-sdpa | 6.39 | ||||
| T4-16GB-70W / float16-eager | 6.39 | ||||
| T4-16GB-70W / float16-sdpa | 6.39 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.39 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.39 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 6.39 | ||||
| T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 6.39 | ||||
| Intel-Xeon-SPR-385W / float32-eager-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / float32-sdpa-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / float16-eager-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 6.39 | ||||
| Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 6.39 | ||||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemm-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 8bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 8bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemm-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemv-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemv-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / bfloat16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / bfloat16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / float16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 8bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / 4bit-bnb-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / float16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | A100-80GB-275W / bfloat16-fa2 | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 8bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-bnb-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-gemm-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-gemm-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-gemv-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-gemv-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 6.34 | |||
| GPT-Neo-2.7B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float16-eager-pytorch | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float16-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float32-eager-pytorch | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / float32-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / bfloat16-eager | 6.34 | |||
| GPT-Neo-2.7B | Intel-Xeon-SPR-385W / bfloat16-eager | 6.34 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 8bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemm-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-gemv-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-gptq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | A10-24GB-150W / 4bit-awq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-gptq-exllama-v1-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-gptq-exllama-v2-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 8bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 8bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / 4bit-bnb-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / float32-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / float16-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / bfloat16-eager | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / float16-fa2 | 5.99 | |||
| GPT-NeoX-20B | A100-80GB-275W / bfloat16-fa2 | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-bnb-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-bnb-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemm-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemm-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemv-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemv-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemm-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemv-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemm-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-gemv-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v1-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v1-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-gptq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v2-eager | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | T4-16GB-70W / 4bit-awq-exllama-v2-sdpa | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / float16-sdpa-pytorch | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / float16-eager-pytorch | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / float16-sdpa | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / float16-eager | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / bfloat16-eager-pytorch | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / bfloat16-sdpa-pytorch | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / bfloat16-eager | 5.99 | |||
| GPT-NeoX-20B | Intel-Xeon-SPR-385W / bfloat16-sdpa | 5.99 | |||
| A10-24GB-150W / 8bit-bnb-eager | 5.10 | ||||
| A10-24GB-150W / float16-eager | 5.10 | ||||
| A10-24GB-150W / bfloat16-eager | 5.10 | ||||
| A10-24GB-150W / 4bit-bnb-eager | 5.10 | ||||
| T4-16GB-70W / 4bit-bnb-eager | 5.10 | ||||
| T4-16GB-70W / 8bit-bnb-eager | 5.10 | ||||
| T4-16GB-70W / bfloat16-eager | 5.10 | ||||
| T4-16GB-70W / float16-eager | 5.10 |
8 models have held number one on the official leaderboard since 2023
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official LLM-Perf Leaderboard leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official LLM-Perf Leaderboard leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Other
- Artificial Analysis Intelligence Index v5 In use Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. Claude Fable 5.2 64.6
- Personal Superintelligence Index (Meta) In use Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO.
- Vals Science Index In use Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. GPT-6.2 Astra 68.4%
- Decision Index Public v0.3 In use Decision models return probabilities over labeled options in public tasks covering knowledge, language, retrieval, tools, and arts. A chance-corrected weighted aggregate combines the task scores. Granite Switch 4.1 30B 51.3
- SAGE v1.1 (Vals) In use Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading. Claude Opus 5.5 91.3%
- MTEB Fading Massive Text Embedding Benchmark: retrieval, clustering, classification across languages and modalities.
Qwen3-Embedding-8B 72.2%
Data audit
8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in June 2024, for Qwen2-0.5B. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
- Pass Independent runs Scores from the official leaderboard (46 entries).
- Pass Models scored 21 models we track, plus 25 other entries on its leaderboard.
- Warning Newest score The newest came 2 years ago, in June 2024, for Qwen2-0.5B.
- Note Frontier models None of the top 10 has a score. The strongest tested is phi-3-mini 3.8B (#204).
- Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
- Pass Importer We read the board on huggingface.co: 1,299 rows for models we track and 25 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.