meta-models/Muse-Glimmer-30B-GGUF

completed

Request throughput: 0.63 req/s
Output throughput: 160.6 tok/s
p95 TTFT: 1968.5 ms
p95 TPOT: 17.44 ms

Server launch command

CUDA_VISIBLE_DEVICES=0 llama-server -m Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf -a meta-models/Muse-Glimmer-30B-GGUF -ngl 99 -c 262144 -np 128 \
  --flash-attn on -ctk q8_0 -ctv q8_0 \
  --host 127.0.0.1 \
  --port 9001 \
  --jinja \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 -t 16 -tb 16
RUNPILE_BACKENDS=<replica-urls> RUNPILE_PROXY_PORT=9000 python llama_compat_proxy.py

Run details

Workload: 1024 → 256
Methodology: closed loop · streaming
Test load: 4 concurrent requests
Endpoint: openai_completions
Requests: 16 explicit requests before the curve; the harness also issues one unmeasured endpoint test immediately before each timed point.
Dataset: name=vllm-random range_ratio=0 random_prefix_tokens=0
Sampling: mode=greedy temperature=0
Stopping: ignore_eos=true max_tokens=256
Prefix and cache: shared_prefix_tokens=0 server_prefix_caching=false
Client: name=vllm bench serve backend=openai version=0.27.1
Hardware: 1× NVIDIA GeForce RTX 5090
Runtime: llama.cpp b10353
Runtime configuration: dflash=false kv_cache=Q8_0 slots_per_replica=128 context_tokens_per_slot=2048 context_tokens_per_replica=262144
Measurement: vllm bench serve 0.27.1
Timing: Client-observed, streaming online serving interval after the harness initial test request. · client overhead included
Repetitions: 1
Failure handling: No client retries; failed requests are counted as failures.
Quantization: format=Dynamic Q4_K_XL checkpoint=GGUF
Model revision: 70bf1b61ac09f91b24d39038091b41c582bc5d7a
Evidence: Raw benchmark evidence attached · eligible and ranked
Published: 8/23/2026, 11:58:50 PM by anonymous-TWTejS

Benchmark command

vllm bench serve \
  --backend openai \
  --host localhost \
  --port 9000 \
  --endpoint /v1/completions \
  --model meta-models/Muse-Glimmer-30B-GGUF \
  --tokenizer meta-models/Muse-Glimmer-30B \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --random-range-ratio 0 \
  --num-prompts 200 \
  --request-rate inf \
  --max-concurrency 4 \
  --ignore-eos \
  --temperature 0 \
  --seed 20260823 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm \
  --save-result \
  --result-dir results \
  --result-filename muse-glimmer-30b-dynamic-q4-k-xl__r1__i1024__o256__c4.json