meta-models/Muse-Glimmer-30B-GGUF

completed
Request throughput
1.58 req/s
Output throughput
405.5 tok/s
p95 TTFT
281.4 ms
p95 TPOT
1.39 ms

Manifest warnings

Review these published caveats before interpreting or comparing the metrics.

  • Review imported facts against the local model, hardware, launch command, and benchmark command before publication.

Server launch command

CUDA_VISIBLE_DEVICES=2 <local-path> serve \
  --model meta-models/Muse-Glimmer-30B-GGUF \
  --alias meta-models/Muse-Glimmer-30B-GGUF \
  --host 127.0.0.1 \
  --port 9002 \
  --ctx-size 147456 \
  --parallel 16 \
  --batch-size 16384 \
  --ubatch-size 512 \
  --n-gpu-layers all \
  --flash-attn on \
  --jinja \
  --metrics \
  --no-webui \
  --no-cache-prompt \
  --cache-ram 0 \
  --no-cache-idle-slots \
  --ctx-checkpoints 0 \
  --model-draft meta-models/Muse-Glimmer-30B-GGUF \
  --spec-type draft-dflash \
  --spec-draft-n-max 15 \
  --n-gpu-layers-draft all

Run details

Workload
1024 → 256
Methodology
closed loop · streaming
Test load
1 concurrent requests
Endpoint
openai-compatible-completions
Requests
8 unmeasured warmups per invocation after one readiness request.
Dataset
sha256=b4d953b7093a94ac24ca491ac83b7a1919ec0023baff4a27d838e2cbbeef119a generator=runpile-sweeps-tokenizer-verified-jsonl synthetic=true generator_revision=a3d215d3624741ac0031b7b6ec24bc4600b7dadd2dcdf9ba58f18e56ea0e3b64 shared_prefix_tokens=0
Sampling
mode=sampling top_k=64 top_p=0.95 temperature=1
Stopping
ignore_eos=true max_output_tokens=256
Prefix and cache
cache_state=warm-host cache_enabled=false shared_prefix_tokens=0
Client
name=vllm bench serve backend=vllm version=0.28.0 placement=same-host
Hardware
1× NVIDIA GeForce RTX 5090
Runtime
llama.cpp b10689
Decoding method
Speculative · Q4_K_M draft
Draft model
meta-models/Muse-Glimmer-30B-GGUF · Q4_K_Mrevision 70bf1b61ac09f91b24d39038091b41c582bc5d7a
Runtime configuration
speculative.type=draft-dflash speculative.n_max=15 ubatch_size=512 context_size=147456 max_num_seqs=16 flash_attention=on weight_artifact=Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf max_model_len_per_slot=9216 max_num_batched_tokens=16384
Measurement
vllm bench serve 0.28.0
Timing
client-observed streaming request latency · client overhead included
Repetitions
1
Failure handling
Failed requests counted; no client retries.
Quantization
format=KQuant 17GB Q4_K_M target plus Q4_K_M DFlash
Model revision
70bf1b61ac09f91b24d39038091b41c582bc5d7a
Evidence
Raw benchmark evidence attached · eligible and ranked
Published
by anonymous-EVjB2S

Benchmark command

vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:9002 \
  --endpoint /v1/completions \
  --model meta-models/Muse-Glimmer-30B-GGUF \
  --tokenizer meta-models/Muse-Glimmer-30B \
  --dataset-name custom \
  --dataset-path <EXACT_SYNTHETIC_DATASET> \
  --custom-output-len 256 \
  --skip-chat-template \
  --disable-shuffle \
  --num-prompts 128 \
  --max-concurrency 1 \
  --request-rate inf \
  --seed 0 \
  --ignore-eos \
  --temperature 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir <local-path> \
  --result-filename glimmer30b-q4km-dflash15-gpu2__baseline__serve-fixed-1k-256-v1__c1__r2__attempt1.json \
  --num-warmups 8 \
  --metadata campaign=runpile-rtx5090-glimmer-gguf-dflash15-20260831-v3 config_id=glimmer30b-q4km-dflash15-gpu2 server_variant=baseline workload_key=serve-fixed-1k-256-v1 repetition=2 aggregate_only=true