nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
1024 → 256
Runs
40 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)016045.3011run_12_tjWo50qiwYR1d: 5.09 req/s, p95 TTFT 255.7 msrun_1ZbsgDg-I6obrHKX: 3.37 req/s, p95 TTFT 143.1 msrun_7OraySujvrrZs-BU: 1.38 req/s, p95 TTFT 45.6 msrun_xoUDvjYMWzyIWthM: 7.2 req/s, p95 TTFT 536.6 msrun_zxoiwUHi9gS31kt6: 7.21 req/s, p95 TTFT 552.8 msrun_fB_Pc7BWItDUcfEn: 7.21 req/s, p95 TTFT 509.1 msrun_K8uStGcRdcvzahD0: 6.77 req/s, p95 TTFT 255.3 msrun_XDBJi4jfue_kNy_u: 4.69 req/s, p95 TTFT 88.7 msrun_UFGlXBUP9NAQD_sQ: 1.89 req/s, p95 TTFT 48.2 msrun_E_EyiiK-84uLIiuI: 9.62 req/s, p95 TTFT 531.8 msrun_RGHlaSQzZ43srC1g: 9.2 req/s, p95 TTFT 575.2 msrun_AuCf_DdIzXOLg9Mi: 9.27 req/s, p95 TTFT 532.4 msrun_ynKk6gHhnnQ6PpYI: 6.48 req/s, p95 TTFT 509.7 msrun_edMbg06JmRwKljUi: 6.61 req/s, p95 TTFT 540 msrun_49NUV3wogr7nqifk: 6.59 req/s, p95 TTFT 536.5 msrun_EbSvHpdEliAjuYVw: 6.97 req/s, p95 TTFT 532.1 msrun_sJzPr80T1LNAm4lF: 7.03 req/s, p95 TTFT 551.4 msrun_dfYS5j3MhCMXxJb2: 7.05 req/s, p95 TTFT 551.2 msrun_RA9rTqCtTtbv4niz: 7.2 req/s, p95 TTFT 554.8 msrun_lMwFwBtgx9yB2RKY: 6.44 req/s, p95 TTFT 540.5 msrun_GTnVSctrP0Zqm_yL: 6.96 req/s, p95 TTFT 508.8 msrun_HkCNlQ5ZUmN5C6hi: 6.84 req/s, p95 TTFT 264.1 msrun_xDtMfdL70YO89KpN: 7.04 req/s, p95 TTFT 547.3 msrun_4wgaCEdacYp11s20: 4.85 req/s, p95 TTFT 267 msrun_GpGQPBB2j-pe7Kmv: 10.54 req/s, p95 TTFT 4,864.7 msrun_Xqf1BDcpPJhDbiWl: 3.42 req/s, p95 TTFT 137 msrun_PpX5UPldjUg2TFDG: 9.65 req/s, p95 TTFT 1,051.3 msrun_OaV6IpykgvSeETwm: 9.76 req/s, p95 TTFT 1,049.5 msrun_iiU3dK-Z2HJ9FY7E: 1.36 req/s, p95 TTFT 40.7 msrun_WkleiNzbUySlY_p8: 6.95 req/s, p95 TTFT 528.9 msrun_gPjsZ999nG5FaRxY: 6.97 req/s, p95 TTFT 530.5 msrun_TMh1PCshUYBnqr-L: 7.09 req/s, p95 TTFT 530.2 msrun_HJDaSqdyhNft8pVW: 7.74 req/s, p95 TTFT 8,007.1 msrun_Vdb2xjv84i9Abmi5: 7.14 req/s, p95 TTFT 577.7 msrun_W2tnclvkJAT98nrh: 3.42 req/s, p95 TTFT 151.2 msrun_oc6dFQgjZlPB1Ip2: 2.16 req/s, p95 TTFT 96 msrun_CJZN3utcuw8ac2Yq: 7.68 req/s, p95 TTFT 16,045.3 msrun_Gf4yXnX4Cvip9JPH: 7.66 req/s, p95 TTFT 3,216.6 msrun_9QHXZMzYBddmjC6T: 5.03 req/s, p95 TTFT 273 msrun_8JunaIIoNugjxgjQ: 1.47 req/s, p95 TTFT 51.2 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=81,302.8255.7 ms6.02 ms1,646.8 ms0run_12_tjWo50qiwYR1d
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=4864143.1 ms4.55 ms1,223.2 ms0run_1ZbsgDg-I6obrHKX
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1353.145.6 ms2.68 ms728.9 ms0run_7OraySujvrrZs-BU
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=161,844536.6 ms8.47 ms2,276.8 ms0run_xoUDvjYMWzyIWthM
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=161,846.9552.8 ms8.35 ms2,300.4 ms0run_zxoiwUHi9gS31kt6
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=161,844.5509.1 ms8.35 ms2,286.3 ms0run_fB_Pc7BWItDUcfEn
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=81,733.7255.3 ms4.6 ms1,276.5 ms0run_K8uStGcRdcvzahD0
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=41,199.788.7 ms3.39 ms917.3 ms0run_XDBJi4jfue_kNy_u
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=1482.748.2 ms2.08 ms575.4 ms0run_UFGlXBUP9NAQD_sQ
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,461.8531.8 ms6.44 ms1,783.2 ms0run_E_EyiiK-84uLIiuI
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,355.6575.2 ms6.61 ms1,849.5 ms0run_RGHlaSQzZ43srC1g
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384c=162,374.1532.4 ms6.69 ms1,839.2 ms0run_AuCf_DdIzXOLg9Mi
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,660509.7 ms9.37 ms2,500.7 ms0run_ynKk6gHhnnQ6PpYI
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,691.1540 ms9.12 ms2,510.8 ms0run_edMbg06JmRwKljUi
vllm 0.28.0dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,686536.5 ms9.19 ms2,500 ms0run_49NUV3wogr7nqifk
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,783.1532.1 ms8.73 ms2,369.4 ms0run_EbSvHpdEliAjuYVw
vllm 0.28.0dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,800.8551.4 ms8.56 ms2,372.5 ms0run_sJzPr80T1LNAm4lF
vllm 0.28.0dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,805551.2 ms8.55 ms2,340.1 ms0run_dfYS5j3MhCMXxJb2
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,842.9554.8 ms8.33 ms2,331.6 ms0run_RA9rTqCtTtbv4niz
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,649.1540.5 ms9.37 ms2,565.8 ms0run_lMwFwBtgx9yB2RKY
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192c=161,783508.8 ms8.58 ms2,377.8 ms0run_GTnVSctrP0Zqm_yL
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048c=161,750264.1 ms8.92 ms2,411.4 ms0run_HkCNlQ5ZUmN5C6hi
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,801.3547.3 ms8.56 ms2,349.3 ms0run_xDtMfdL70YO89KpN
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=81,241.6267 ms6.31 ms1,697 ms0run_4wgaCEdacYp11s20
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=642,697.44,864.7 ms15.15 ms8,728.4 ms0run_GpGQPBB2j-pe7Kmv
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4876137 ms4.49 ms1,201.4 ms0run_Xqf1BDcpPJhDbiWl
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=322,4711,051.3 ms12.55 ms3,384.4 ms0run_PpX5UPldjUg2TFDG
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=322,497.41,049.5 ms12.45 ms3,385.6 ms0run_OaV6IpykgvSeETwm
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1349.240.7 ms2.73 ms734.8 ms0run_iiU3dK-Z2HJ9FY7E
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,778528.9 ms8.72 ms2,385.7 ms0run_WkleiNzbUySlY_p8
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,785.5530.5 ms8.69 ms2,337 ms0run_gPjsZ999nG5FaRxY
vllm 0.28.0dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,815.5530.2 ms8.5 ms2,322.2 ms0run_TMh1PCshUYBnqr-L
vllm 0.27.1dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=641,9818,007.1 ms8.88 ms10,170.7 ms0run_HJDaSqdyhNft8pVW
vllm 0.27.1dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=161,826.8577.7 ms8.01 ms2,321.5 ms0run_Vdb2xjv84i9Abmi5
vllm 0.27.1dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=4876.4151.2 ms4.36 ms1,203.1 ms0run_W2tnclvkJAT98nrh
vllm 0.27.1dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=255296 ms3.38 ms944.8 ms0run_oc6dFQgjZlPB1Ip2
vllm 0.27.1dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=1281,967.216,045.3 ms9.87 ms18,092.9 ms0run_CJZN3utcuw8ac2Yq
vllm 0.27.1dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=321,961.53,216.6 ms8.95 ms5,359.4 ms0run_Gf4yXnX4Cvip9JPH
vllm 0.27.1dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=81,288.8273 ms5.97 ms1,703.4 ms0run_9QHXZMzYBddmjC6T
vllm 0.27.1dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384c=1376.651.2 ms2.47 ms681.7 ms0run_8JunaIIoNugjxgjQ
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,302.8
p95 TTFT / TPOT
255.7 ms / 6.02 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
864
p95 TTFT / TPOT
143.1 ms / 4.55 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
353.1
p95 TTFT / TPOT
45.6 ms / 2.68 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,844
p95 TTFT / TPOT
536.6 ms / 8.47 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,846.9
p95 TTFT / TPOT
552.8 ms / 8.35 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,844.5
p95 TTFT / TPOT
509.1 ms / 8.35 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,733.7
p95 TTFT / TPOT
255.3 ms / 4.6 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
1,199.7
p95 TTFT / TPOT
88.7 ms / 3.39 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
482.7
p95 TTFT / TPOT
48.2 ms / 2.08 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,461.8
p95 TTFT / TPOT
531.8 ms / 6.44 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,355.6
p95 TTFT / TPOT
575.2 ms / 6.61 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.88 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
2,374.1
p95 TTFT / TPOT
532.4 ms / 6.69 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=false gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,660
p95 TTFT / TPOT
509.7 ms / 9.37 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,691.1
p95 TTFT / TPOT
540 ms / 9.12 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,686
p95 TTFT / TPOT
536.5 ms / 9.19 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,783.1
p95 TTFT / TPOT
532.1 ms / 8.73 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=32 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,800.8
p95 TTFT / TPOT
551.4 ms / 8.56 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=128 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,805
p95 TTFT / TPOT
551.2 ms / 8.55 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,842.9
p95 TTFT / TPOT
554.8 ms / 8.33 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,649.1
p95 TTFT / TPOT
540.5 ms / 9.37 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=8192
Test load
c=16
Req/s
Output tok/s
1,783
p95 TTFT / TPOT
508.8 ms / 8.58 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=2048
Test load
c=16
Req/s
Output tok/s
1,750
p95 TTFT / TPOT
264.1 ms / 8.92 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=auto prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,801.3
p95 TTFT / TPOT
547.3 ms / 8.56 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,241.6
p95 TTFT / TPOT
267 ms / 6.31 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
2,697.4
p95 TTFT / TPOT
4,864.7 ms / 15.15 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
876
p95 TTFT / TPOT
137 ms / 4.49 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2,471
p95 TTFT / TPOT
1,051.3 ms / 12.55 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2,497.4
p95 TTFT / TPOT
1,049.5 ms / 12.45 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
349.2
p95 TTFT / TPOT
40.7 ms / 2.73 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,778
p95 TTFT / TPOT
528.9 ms / 8.72 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,785.5
p95 TTFT / TPOT
530.5 ms / 8.69 ms
Runtime
vllm 0.28.0
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,815.5
p95 TTFT / TPOT
530.2 ms / 8.5 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
1,981
p95 TTFT / TPOT
8,007.1 ms / 8.88 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,826.8
p95 TTFT / TPOT
577.7 ms / 8.01 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
876.4
p95 TTFT / TPOT
151.2 ms / 4.36 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=2
Req/s
Output tok/s
552
p95 TTFT / TPOT
96 ms / 3.38 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=128
Req/s
Output tok/s
1,967.2
p95 TTFT / TPOT
16,045.3 ms / 9.87 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
1,961.5
p95 TTFT / TPOT
3,216.6 ms / 8.95 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
1,288.8
p95 TTFT / TPOT
273 ms / 5.97 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=95 max_model_len=16384 kv_cache_dtype=fp8 prefix_caching=false gpu_memory_utilization=0.90 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
376.6
p95 TTFT / TPOT
51.2 ms / 2.47 ms