google/gemma-4-12B-it

Hardware
1× NVIDIA GeForce RTX 5090
Workload
1024 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)03687.908run_-AnWupcaqeX6vvZT: 2.32 req/s, p95 TTFT 482.6 msrun_Iew7zCX9D_S5J9iI: 7.57 req/s, p95 TTFT 3,687.9 msrun_LRkzeKNM2Eew0zZy: 1.29 req/s, p95 TTFT 254.1 msrun_Ra6nwx9OE8nBeaye: 5.72 req/s, p95 TTFT 1,667.3 msrun_49U9yOF1_hJ7QVzU: 0.35 req/s, p95 TTFT 73.8 msrun_v2NtG4twUBQd45oz: 3.87 req/s, p95 TTFT 931 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=8593.8482.6 ms13.17 ms3,466.7 ms0run_-AnWupcaqeX6vvZT
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=641,938.93,687.9 ms28.93 ms9,502.6 ms0run_Iew7zCX9D_S5J9iI
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=4330.8254.1 ms11.8 ms3,104 ms0run_LRkzeKNM2Eew0zZy
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=321,463.41,667.3 ms21.11 ms5,615 ms0run_Ra6nwx9OE8nBeaye
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=190.873.8 ms10.79 ms2,825.1 ms0run_49U9yOF1_hJ7QVzU
vllm 0.23.0+cu129dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensorc=16990.1931 ms15.24 ms4,146.1 ms0run_v2NtG4twUBQd45oz
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=8
Req/s
Output tok/s
593.8
p95 TTFT / TPOT
482.6 ms / 13.17 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=64
Req/s
Output tok/s
1,938.9
p95 TTFT / TPOT
3,687.9 ms / 28.93 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=4
Req/s
Output tok/s
330.8
p95 TTFT / TPOT
254.1 ms / 11.8 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=32
Req/s
Output tok/s
1,463.4
p95 TTFT / TPOT
1,667.3 ms / 21.11 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=1
Req/s
Output tok/s
90.8
p95 TTFT / TPOT
73.8 ms / 10.79 ms
Runtime
vllm 0.23.0+cu129
Material parameters
dtype=bfloat16 max_num_seqs=64 enforce_eager=false max_model_len=9216 kv_cache_dtype=fp8 prefix_caching=false chunked_prefill=true expert_parallel=false async_scheduling=true gpu_memory_utilization=0.92 max_num_batched_tokens=16384 runtime_weight_quantization=fp8_per_tensor
Test load
c=16
Req/s
Output tok/s
990.1
p95 TTFT / TPOT
931 ms / 15.24 ms