meta-models/Muse-Glimmer-30B-GGUF

Hardware
1× NVIDIA GeForce RTX 5090
Workload
1024 → 256
Runs
6 eligible
Efficient frontierOther eligible run
p95 TTFT (ms)Request throughput (req/s)0227.700run_6JaGDduA6V6UUZAO: 0.3 req/s, p95 TTFT 227.7 msrun_rnFdIe1VGz_CdGHB: 0.3 req/s, p95 TTFT 226.7 msrun_hZrPPIaCESBXNmfi: 0.3 req/s, p95 TTFT 227.2 msrun_NP7Cgj167353GT8X: 0.3 req/s, p95 TTFT 221.5 msrun_vG1F0-GzWHpPsQuI: 0.3 req/s, p95 TTFT 222 msrun_R6cE0ka8-q28MGRv: 0.3 req/s, p95 TTFT 222 ms

Runs

All published runs with the same model build, accelerator setup, and request shape. Benchmark methodology remains attached to each run.

RuntimeMaterial parametersTest loadReq/sOutput tok/sp95 TTFTp95 TPOTp95 E2EFailuresRun
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=177.2227.7 ms12.14 ms3,322.7 ms0run_6JaGDduA6V6UUZAO
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=177.2226.7 ms12.12 ms3,317.5 ms0run_rnFdIe1VGz_CdGHB
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=177.1227.2 ms12.15 ms3,325.3 ms0run_hZrPPIaCESBXNmfi
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=177.1221.5 ms12.17 ms3,324.9 ms0run_NP7Cgj167353GT8X
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=177.1222 ms12.16 ms3,321.7 ms0run_vG1F0-GzWHpPsQuI
llama.cpp b10689ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384c=177222 ms12.18 ms3,328.1 ms0run_R6cE0ka8-q28MGRv
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
77.2
p95 TTFT / TPOT
227.7 ms / 12.14 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
77.2
p95 TTFT / TPOT
226.7 ms / 12.12 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
77.1
p95 TTFT / TPOT
227.2 ms / 12.15 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
77.1
p95 TTFT / TPOT
221.5 ms / 12.17 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
77.1
p95 TTFT / TPOT
222 ms / 12.16 ms
Runtime
llama.cpp b10689
Material parameters
ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Test load
c=1
Req/s
Output tok/s
77
p95 TTFT / TPOT
222 ms / 12.18 ms