Results
| Model Hugging Face ID | Hardware GPU and quantity | Workload tokens in → tokens out | Capacity workload throughput | Responsiveness p95 TTFT | Stream speed p95 floor | Concurrency test load |
|---|---|---|---|---|---|---|
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4NVFP4Speculative · NVFP4 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 10.64 req/s | 517.1 ms | 72.8 tok/s | 16 |
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4NVFP4 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 9.62 req/s | 531.8 ms | 155.2 tok/s | 16 |
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4NVFP4DFlash · NVFP4 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 9.49 req/s | 523.7 ms | 59.3 tok/s | 16 |
nvidia/Gemma-4-26B-A4B-NVFP4NVFP4 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 8.82 req/s | 786.1 ms | 77.1 tok/s | 32 |
nvidia/Gemma-4-26B-A4B-NVFP4NVFP4Speculative | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 8.05 req/s | 440.7 ms | 79.2 tok/s | 16 |
nvidia/Qwen3.6-35B-A3B-NVFP4NVFP4 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 7.92 req/s | 968 ms | 65.4 tok/s | 32 |
gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090NVFP4DFlash2 · NVFP4 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 6.17 req/s | 541.1 ms | 210.6 tok/s | 8 |
gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090NVFP4DFlash2 · BF16 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 4.4 req/s | 225.2 ms | 285.5 tok/s | 4 |
google/gemma-4-12B-itFP8 per tensor | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 3.87 req/s | 931 ms | 65.6 tok/s | 16 |
RedHatAI/gemma-4-31B-it-NVFP4NVFP4Speculative | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 2.29 req/s | 232.3 ms | 48.9 tok/s | 8 |
gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090NVFP4 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 2.22 req/s | 500.7 ms | 73.5 tok/s | 8 |
unsloth/Qwen3.8-27B-NVFP4NVFP4 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 1.68 req/s | 702.4 ms | 58.4 tok/s | 8 |
meta-models/Muse-Glimmer-30B-GGUFQ4_K_MSpeculative · Q4_K_M | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 1.59 req/s | 284 ms | 725.2 tok/s | 1 |
meta-models/Muse-Glimmer-30B-GGUFQ4_K_XLSpeculative · Q4_K_M | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 1.47 req/s | 309.6 ms | 651.1 tok/s | 1 |
google/gemma-4-12B-itBF16 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 1.41 req/s | 878.3 ms | 48.7 tok/s | 8 |
RedHatAI/gemma-4-31B-it-NVFP4NVFP4 | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 1.34 req/s | 561.6 ms | 43.3 tok/s | 8 |
meta-models/Muse-Glimmer-30B-GGUFkquant 17gb Q4 k m | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 0.3 req/s | 226.7 ms | 82.5 tok/s | 1 |
meta-models/Muse-Glimmer-30B-GGUFkquant dynamic Q4 k xl | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 0.27 req/s | 236.9 ms | 72 tok/s | 1 |
meta-models/Muse-Glimmer-30B-GGUFdynamic Q4 k xl | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 0.25 req/s | 501.9 ms | 71 tok/s | 1 |
malaiwah/Qwen3.8-27B-EXL3-K5K6-hydratedexl3 mixed k5 k6 hydrated | 1× NVIDIA GeForce RTX 5090 | 1024 → 256 | 0.11 req/s | 473.8 ms | 29.7 tok/s | 1 |
- nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
NVFP4Speculative · NVFP4 draft · 1× NVIDIA GeForce RTX 5090 · concurrency 1610.64 req/sCapacity517.1 msp95 TTFT72.8 tok/sp95 speed - nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
NVFP41× NVIDIA GeForce RTX 5090 · concurrency 169.62 req/sCapacity531.8 msp95 TTFT155.2 tok/sp95 speed - nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
NVFP4DFlash · NVFP4 draft · 1× NVIDIA GeForce RTX 5090 · concurrency 169.49 req/sCapacity523.7 msp95 TTFT59.3 tok/sp95 speed - nvidia/Gemma-4-26B-A4B-NVFP4
NVFP41× NVIDIA GeForce RTX 5090 · concurrency 328.82 req/sCapacity786.1 msp95 TTFT77.1 tok/sp95 speed - nvidia/Gemma-4-26B-A4B-NVFP4
NVFP4Speculative decoding · 1× NVIDIA GeForce RTX 5090 · concurrency 168.05 req/sCapacity440.7 msp95 TTFT79.2 tok/sp95 speed - nvidia/Qwen3.6-35B-A3B-NVFP4
NVFP41× NVIDIA GeForce RTX 5090 · concurrency 327.92 req/sCapacity968 msp95 TTFT65.4 tok/sp95 speed - gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
NVFP4DFlash2 · calibrated NVFP4 draft · 1× NVIDIA GeForce RTX 5090 · concurrency 86.17 req/sCapacity541.1 msp95 TTFT210.6 tok/sp95 speed - gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
NVFP4DFlash2 · BF16 draft · 1× NVIDIA GeForce RTX 5090 · concurrency 44.4 req/sCapacity225.2 msp95 TTFT285.5 tok/sp95 speed - google/gemma-4-12B-it
FP8 per tensor1× NVIDIA GeForce RTX 5090 · concurrency 163.87 req/sCapacity931 msp95 TTFT65.6 tok/sp95 speed - RedHatAI/gemma-4-31B-it-NVFP4
NVFP4Speculative decoding · 1× NVIDIA GeForce RTX 5090 · concurrency 82.29 req/sCapacity232.3 msp95 TTFT48.9 tok/sp95 speed - gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
NVFP41× NVIDIA GeForce RTX 5090 · concurrency 82.22 req/sCapacity500.7 msp95 TTFT73.5 tok/sp95 speed - unsloth/Qwen3.8-27B-NVFP4
NVFP41× NVIDIA GeForce RTX 5090 · concurrency 81.68 req/sCapacity702.4 msp95 TTFT58.4 tok/sp95 speed - meta-models/Muse-Glimmer-30B-GGUF
Q4_K_MSpeculative · Q4_K_M draft · 1× NVIDIA GeForce RTX 5090 · concurrency 11.59 req/sCapacity284 msp95 TTFT725.2 tok/sp95 speed - meta-models/Muse-Glimmer-30B-GGUF
Q4_K_XLSpeculative · Q4_K_M draft · 1× NVIDIA GeForce RTX 5090 · concurrency 11.47 req/sCapacity309.6 msp95 TTFT651.1 tok/sp95 speed - google/gemma-4-12B-it
BF161× NVIDIA GeForce RTX 5090 · concurrency 81.41 req/sCapacity878.3 msp95 TTFT48.7 tok/sp95 speed - RedHatAI/gemma-4-31B-it-NVFP4
NVFP41× NVIDIA GeForce RTX 5090 · concurrency 81.34 req/sCapacity561.6 msp95 TTFT43.3 tok/sp95 speed - meta-models/Muse-Glimmer-30B-GGUF
kquant 17gb Q4 k m1× NVIDIA GeForce RTX 5090 · concurrency 10.3 req/sCapacity226.7 msp95 TTFT82.5 tok/sp95 speed - meta-models/Muse-Glimmer-30B-GGUF
kquant dynamic Q4 k xl1× NVIDIA GeForce RTX 5090 · concurrency 10.27 req/sCapacity236.9 msp95 TTFT72 tok/sp95 speed - meta-models/Muse-Glimmer-30B-GGUF
dynamic Q4 k xl1× NVIDIA GeForce RTX 5090 · concurrency 10.25 req/sCapacity501.9 msp95 TTFT71 tok/sp95 speed - malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated
exl3 mixed k5 k6 hydrated1× NVIDIA GeForce RTX 5090 · concurrency 10.11 req/sCapacity473.8 msp95 TTFT29.7 tok/sp95 speed