nvidia/Qwen3.6-35B-A3B-NVFP4

Hardware
1× NVIDIA GeForce RTX 5090
Workload
1024 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)01710.4010run_5IRiVnlGaUSU8uaR: 5.49 req/s, p95 TTFT 490.9 msrun_2xdHQyYtmRfrbwyy: 10.48 req/s, p95 TTFT 1,710.4 msrun_M5yVjJeMh_MUJgiX: 2.12 req/s, p95 TTFT 145.6 msrun_lFLGR_wUI1_FpZWD: 7.92 req/s, p95 TTFT 968 msrun_yH65DQ1c3UQ_ELrG: 0.67 req/s, p95 TTFT 53.3 msrun_eX1Y4pM699fDJG5o: 3.45 req/s, p95 TTFT 248.8 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=161,404.9490.9 ms11.01 ms2,964.7 ms0run_5IRiVnlGaUSU8uaR
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=642,683.41,710.4 ms22.29 ms6,523.2 ms0run_2xdHQyYtmRfrbwyy
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=4543.4145.6 ms7.22 ms1,899.3 ms0run_M5yVjJeMh_MUJgiX
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=322,027.4968 ms15.3 ms4,101.3 ms0run_lFLGR_wUI1_FpZWD
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=1170.553.3 ms5.69 ms1,504 ms0run_yH65DQ1c3UQ_ELrG
vllm 0.27.1dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384c=8882.3248.8 ms8.92 ms2,360.8 ms0run_eX1Y4pM699fDJG5o
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=16
Req/s
Output tok/s
1,404.9
p95 TTFT / TPOT
490.9 ms / 11.01 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=64
Req/s
Output tok/s
2,683.4
p95 TTFT / TPOT
1,710.4 ms / 22.29 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=4
Req/s
Output tok/s
543.4
p95 TTFT / TPOT
145.6 ms / 7.22 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=32
Req/s
Output tok/s
2,027.4
p95 TTFT / TPOT
968 ms / 15.3 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
170.5
p95 TTFT / TPOT
53.3 ms / 5.69 ms
Runtime
vllm 0.27.1
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 server_variant=baseline enable_prefix_caching=false enable_chunked_prefill=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384
Test load
c=8
Req/s
Output tok/s
882.3
p95 TTFT / TPOT
248.8 ms / 8.92 ms