nvidia/Gemma-4-26B-A4B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
1024 → 256
Runs
29 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)07030.5012run_0HUfv_uWk5lwg0Sf: 3.77 req/s, p95 TTFT 205.1 msrun_ftYKQ0ifof0ywoQK: 2.04 req/s, p95 TTFT 111.3 msrun_NOfq_eOE4LS2eviE: 0.71 req/s, p95 TTFT 37.1 msrun_wTG22EngDCsTeJnL: 6.04 req/s, p95 TTFT 407.4 msrun_ckJR2ADVhxPmDuSb: 5.88 req/s, p95 TTFT 416.2 msrun_a1sHSnxByBc5ptLf: 5.99 req/s, p95 TTFT 402 msrun_rT8CVflR0Kd8APoq: 7.29 req/s, p95 TTFT 1,064.4 msrun_Uvao52W66Efh1ZHX: 7.71 req/s, p95 TTFT 1,033.5 msrun_2jvsd98TgETbZEsH: 7.73 req/s, p95 TTFT 996.1 msrun_V83TA8wr3_OaEaX9: 7.71 req/s, p95 TTFT 1,034.2 msrun_qsvZrwESjm5uzv3C: 7.54 req/s, p95 TTFT 1,826.8 msrun_SoAp9OP_2rmsyrwT: 5.67 req/s, p95 TTFT 1,960.4 msrun_b2vivTgE3a5FuG1M: 3.2 req/s, p95 TTFT 271.7 msrun_Z8f07104ef2SG7dS: 10.51 req/s, p95 TTFT 2,054.9 msrun_7Em55k5Y9sAXC3z9: 10.53 req/s, p95 TTFT 1,939.5 msrun_a2Z0OVHnBcxgGkdK: 1.78 req/s, p95 TTFT 148.4 msrun_QKC9Ar5WoFHGl0tW: 7.89 req/s, p95 TTFT 1,056.6 msrun_YtkxDSzawuJmnnyo: 7.72 req/s, p95 TTFT 1,032.7 msrun_Yz14XIGl1C_Mhdtb: 7.73 req/s, p95 TTFT 925 msrun_XZjhEjD_Jrh4MfBc: 0.59 req/s, p95 TTFT 43.6 msrun_jLFwtr_PCHrbfGrp: 5.18 req/s, p95 TTFT 546.2 msrun_kNWGNT6KdEkKGM0R: 12.19 req/s, p95 TTFT 1,527.5 msrun_78yDkL5Nwiksfkny: 6.03 req/s, p95 TTFT 411.4 msrun_qq_m6ocYRPHq-MwX: 2.11 req/s, p95 TTFT 115.6 msrun_FfmlMvD9VPPI2Crh: 1.19 req/s, p95 TTFT 70.4 msrun_uX4ej3GSleJuun4i: 12.36 req/s, p95 TTFT 7,030.5 msrun_R05DohvT71HmzPHY: 8.82 req/s, p95 TTFT 786.1 msrun_sGelkB_gvuMWm1Us: 3.84 req/s, p95 TTFT 214 msrun_feGAJd6Zcv58f_0c: 0.71 req/s, p95 TTFT 40.6 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=8964.9205.1 ms8.14 ms2,148.7 ms0run_0HUfv_uWk5lwg0Sf
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4521.8111.3 ms7.6 ms2,000.4 ms0run_ftYKQ0ifof0ywoQK
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1181.337.1 ms5.4 ms1,413.7 ms0run_NOfq_eOE4LS2eviE
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=161,547.3407.4 ms10.1 ms2,719.1 ms0run_wTG22EngDCsTeJnL
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=161,504.1416.2 ms10.35 ms2,805.6 ms0run_ckJR2ADVhxPmDuSb
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=161,534.1402 ms10.29 ms2,765.8 ms0run_a1sHSnxByBc5ptLf
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=321,866.11,064.4 ms16.68 ms4,487.2 ms0run_rT8CVflR0Kd8APoq
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=321,973.61,033.5 ms15.38 ms4,240.9 ms0run_Uvao52W66Efh1ZHX
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=321,980.1996.1 ms14.67 ms4,266.8 ms0run_2jvsd98TgETbZEsH
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=321,974.11,034.2 ms15.38 ms4,241.9 ms0run_V83TA8wr3_OaEaX9
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=321,931.11,826.8 ms14.69 ms4,892.4 ms0run_qsvZrwESjm5uzv3C
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=321,450.71,960.4 ms19.57 ms6,421.9 ms0run_SoAp9OP_2rmsyrwT
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8820.4271.7 ms9.62 ms2,566.5 ms0run_b2vivTgE3a5FuG1M
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=642,6912,054.9 ms22.14 ms6,501 ms0run_Z8f07104ef2SG7dS
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=642,695.41,939.5 ms22.86 ms6,600.1 ms0run_7Em55k5Y9sAXC3z9
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4454.5148.4 ms8.64 ms2,298.1 ms0run_a2Z0OVHnBcxgGkdK
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=322,020.31,056.6 ms15.35 ms4,108.2 ms0run_QKC9Ar5WoFHGl0tW
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=321,977.31,032.7 ms15.45 ms4,205.6 ms0run_YtkxDSzawuJmnnyo
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=321,978925 ms15.73 ms4,226.7 ms0run_Yz14XIGl1C_Mhdtb
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1149.843.6 ms6.54 ms1,710.5 ms0run_XZjhEjD_Jrh4MfBc
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,324.8546.2 ms11.85 ms3,156.4 ms0run_jLFwtr_PCHrbfGrp
vllm 0.27.1dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=643,1201,527.5 ms19.96 ms5,446.9 ms0run_kNWGNT6KdEkKGM0R
vllm 0.27.1dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=161,544.8411.4 ms9.89 ms2,653.1 ms0run_78yDkL5Nwiksfkny
vllm 0.27.1dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=4540.9115.6 ms7.32 ms1,934.6 ms0run_qq_m6ocYRPHq-MwX
vllm 0.27.1dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=2304.570.4 ms6.49 ms1,701.1 ms0run_FfmlMvD9VPPI2Crh
vllm 0.27.1dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=1283,163.17,030.5 ms21.86 ms12,159.7 ms0run_uX4ej3GSleJuun4i
vllm 0.27.1dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=322,258.2786.1 ms12.97 ms3,472.3 ms0run_R05DohvT71HmzPHY
vllm 0.27.1dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=8983.2214 ms8.01 ms2,128.6 ms0run_sGelkB_gvuMWm1Us
vllm 0.27.1dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=1181.340.6 ms5.39 ms1,413.2 ms0run_feGAJd6Zcv58f_0c
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
964.9
p95 TTFT / TPOT
205.1 ms / 8.14 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
521.8
p95 TTFT / TPOT
111.3 ms / 7.6 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
181.3
p95 TTFT / TPOT
37.1 ms / 5.4 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,547.3
p95 TTFT / TPOT
407.4 ms / 10.1 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,504.1
p95 TTFT / TPOT
416.2 ms / 10.35 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,534.1
p95 TTFT / TPOT
402 ms / 10.29 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,866.1
p95 TTFT / TPOT
1,064.4 ms / 16.68 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,973.6
p95 TTFT / TPOT
1,033.5 ms / 15.38 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,980.1
p95 TTFT / TPOT
996.1 ms / 14.67 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,974.1
p95 TTFT / TPOT
1,034.2 ms / 15.38 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,931.1
p95 TTFT / TPOT
1,826.8 ms / 14.69 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=true max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,450.7
p95 TTFT / TPOT
1,960.4 ms / 19.57 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
820.4
p95 TTFT / TPOT
271.7 ms / 9.62 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
2,691
p95 TTFT / TPOT
2,054.9 ms / 22.14 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
2,695.4
p95 TTFT / TPOT
1,939.5 ms / 22.86 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
454.5
p95 TTFT / TPOT
148.4 ms / 8.64 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2,020.3
p95 TTFT / TPOT
1,056.6 ms / 15.35 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,977.3
p95 TTFT / TPOT
1,032.7 ms / 15.45 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,978
p95 TTFT / TPOT
925 ms / 15.73 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
149.8
p95 TTFT / TPOT
43.6 ms / 6.54 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,324.8
p95 TTFT / TPOT
546.2 ms / 11.85 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
3,120
p95 TTFT / TPOT
1,527.5 ms / 19.96 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,544.8
p95 TTFT / TPOT
411.4 ms / 9.89 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
540.9
p95 TTFT / TPOT
115.6 ms / 7.32 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
304.5
p95 TTFT / TPOT
70.4 ms / 6.49 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=128
Req/s
Output tok/s
3,163.1
p95 TTFT / TPOT
7,030.5 ms / 21.86 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2,258.2
p95 TTFT / TPOT
786.1 ms / 12.97 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
983.2
p95 TTFT / TPOT
214 ms / 8.01 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=128 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
181.3
p95 TTFT / TPOT
40.6 ms / 5.39 ms