Command Templates

Metrum Insights uses command templates so framework startup and benchmark tool execution are reproducible. Templates are versioned and parameterized; run settings store the resolved values used by each run.

Template Types

Template	Purpose
Framework command template	Starts model serving, for example vLLM, SGLang, TGI, Dynamo, or a custom server.
Tool command template	Runs the benchmark client, for example metrumbench-llm or GenAI-Perf.
Engine args set	Named framework preset such as baseline, FP8, KV offload, or high-throughput.
Run settings	Explicit values resolved for a run/job/replica.

Reproducibility Contract

Before a paid or long-running benchmark, verify the resolved command:

no literal placeholders remain;
model id, port, tensor parallelism, precision, max model length, and cache settings match the plan;
optional flags are omitted when unset;
boolean flags render as flags only when enabled;
replica-specific values such as port and GPU assignment are correct.

vLLM Baseline Example

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 1 \
  --dtype auto \
  --max-model-len 4096 \
  --kv-cache-dtype auto \
  --cpu-offload-gb 0

Use this shape for a small baseline before enabling aggressive flags.

Create this as a workload with an engine args set such as vllm-default. When comparing other serving modes, keep the project and scenarios constant and change the workload or engine args set.

vLLM Engine Args Examples

Engine args sets are the normal way to compare serving behavior. Do not edit a command line by hand for one run and then report it as the same workload.

Long Context And Scheduler Headroom

Use this when the scenario matrix includes long ISL/OSL cells or high concurrency. max_model_len must cover input plus output tokens; scheduler limits should leave enough headroom for active sequences.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 65536 \
  --gpu-memory-utilization 0.92

Example engine args set:

Arg key	Value
`dtype`	`bfloat16`
`max_model_len`	`32768`
`max_num_seqs`	`512`
`max_num_batched_tokens`	`65536`
`gpu_memory_utilization`	`0.92`

KV Cache FP8

Use a separate workload or engine args set for KV cache dtype changes. This is not just a memory optimization; it can change quality, compatibility, and latency.

vllm serve Qwen/Qwen3-8B \
  --port 8000 \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90

Example engine args set: vllm-kv-fp8.

Arg key	Value
`dtype`	`bfloat16`
`max_model_len`	`8192`
`kv_cache_dtype`	`fp8`
`enable_prefix_caching`	`true`
`gpu_memory_utilization`	`0.90`

Record whether the framework uses per-tensor, per-channel, or dynamic scale calculation for FP8 KV cache when that detail is exposed by the runtime.

CPU Offload

CPU offload is useful for fitting a model that is close to the GPU memory limit. It should be tested separately from the baseline because it may reduce throughput and increase latency.

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
  --port 8000 \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --max-model-len 16384 \
  --cpu-offload-gb 16 \
  --gpu-memory-utilization 0.88

Capture these SUT details with the result:

CPU model, socket count, NUMA layout, RAM size, and memory bandwidth;
pinned-memory or hugepage settings if used;
observed PCIe/NVLink transfer pressure from telemetry or logs.

Different KV Block Sizes

Use block-size sweeps to understand fragmentation and scheduler behavior.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --block-size 16

Run separate engine args sets:

Set code	`block_size`	Intended comparison
`vllm-block16`	`16`	Smaller blocks, less waste, more metadata pressure.
`vllm-block32`	`32`	Common middle ground.
`vllm-block256`	`256`	Large-block experimental or specialized runtime behavior.

DeepSeek V4 / MoE Style Configuration

Large MoE models often need a much richer template than the 7B baseline. Represent every special runtime flag as a template variable so the resolved command is auditable.

vllm serve deepseek-ai/DeepSeek-V4-Pro \
  --port 8000 \
  --tensor-parallel-size 1 \
  --data-parallel-size 8 \
  --max-model-len 4096 \
  --kv-cache-dtype fp8 \
  --reasoning-parser deepseek_v4 \
  --block-size 256 \
  --max-num-seqs 1024 \
  --max-num-batched-tokens 65536 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --moe-backend deep_gemm

Example engine args set names:

Set code	Target
`vllm-deepseek-v4-pro-b300-dp8`	8-GPU B300 system, DP8 plus MoE backend tuning.
`vllm-deepseek-v4-pro-b200-dp8`	8-GPU B200 system with the same model/runtime recipe.

Do not mix these rows with a baseline vllm-default workload. They represent a different runtime recipe.

SGLang Examples

SGLang command templates should capture the server mode, tensor parallelism, memory fraction, context length, and any attention backend or JIT/runtime settings.

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --host 0.0.0.0 \
  --port 8000 \
  --tp-size 1 \
  --mem-fraction-static 0.88 \
  --context-length 8192 \
  --dtype bfloat16

Example engine args set:

Arg key	Value
`tp_size`	`1`
`mem_fraction_static`	`0.88`
`context_length`	`8192`
`dtype`	`bfloat16`
`attention_backend`	runtime-specific value, if selected explicitly.

For apples-to-apples comparisons against vLLM, create parallel workloads:

Workload	Framework	Engine args set
`qwen3-vllm-bf16`	`vllm`	`vllm-default`
`qwen3-vllm-kvfp8`	`vllm`	`vllm-kv-fp8`
`qwen3-sglang-bf16`	`sglang`	`sglang-default`

Keep scenario codes identical across the workloads.

Dynamo Example Shape

Dynamo deployments usually involve multiple processes or services: frontend, prefill, decode, and routing components. A Metrum Insights template should make each component explicit and connect them through named settings.

Example dimensions to model:

model id and tokenizer;
frontend OpenAI-compatible port;
prefill/worker process count;
tensor/data/pipeline parallelism;
routing policy;
KV cache transfer configuration;
container image and runtime variables.

The exact Dynamo command depends on the deployment package. Record the final resolved command and service topology in SUT evidence.

Example topology:

# Frontend
dynamo-frontend \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
  --listen 0.0.0.0:8000 \
  --router-policy prefix-cache-aware

# Prefill worker
dynamo-worker \
  --role prefill \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
  --tp-size 1 \
  --kv-transfer-backend nixl \
  --kv-cache-capacity-gb 48

# Decode worker
dynamo-worker \
  --role decode \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
  --tp-size 1 \
  --kv-transfer-backend nixl \
  --kv-cache-capacity-gb 48

Template variables should include at least:

Arg key	Example value
`router_policy`	`prefix-cache-aware`
`prefill_replicas`	`1`
`decode_replicas`	`1`
`kv_transfer_backend`	`nixl`
`kv_cache_capacity_gb`	`48`
`frontend_port`	`8000`

KV Cache Offload And Advanced Serving

Advanced serving features need their own engine args set rather than ad hoc command edits:

Feature	Settings to capture
CPU offload	offload GB, pinned memory, NUMA assumptions.
KV cache offload	cache backend, block size, transfer engine, capacity, eviction policy.
FP8 KV cache	KV dtype, calibration or scale source, model support.
Expert parallelism	expert-parallel flag, parallel sizes, all-to-all backend.
Long context	max model length, block size, prefix cache, chunked prefill.

Create separate workloads or engine args sets for these features so reports do not mix baseline and experimental behavior.

Custom And Non-NVIDIA Hardware

Custom accelerators should be modeled explicitly. Do not force every accelerator into an NVIDIA/vLLM mental model. The server config should capture the real vendor, runtime, driver stack, memory topology, and container image.

Common cases:

Hardware family	Runtime examples	Template concerns
AMD Instinct	vLLM ROCm, SGLang ROCm	`HIP_VISIBLE_DEVICES`, ROCm version, image tag, supported dtypes, attention backend.
Google TPU	SGLang-JAX, custom JAX server	topology, TPU count, zone, XLA flags, model sharding, serving port.
Intel Gaudi	Optimum Habana, TGI/Habana, custom OpenAI-compatible server	HPU graph flags, lazy/eager mode, SynapseAI version, tensor parallel support.
AWS Trainium/Inferentia	NeuronX runtime, vLLM Neuron variants, custom endpoint	Neuron core count, compiled artifact path, batch constraints, model cache.
Private ASIC	OpenAI-compatible gateway or BYOE endpoint	endpoint health, runtime version, hardware counters, vendor-specific batch knobs.

AMD ROCm vLLM Example

HIP_VISIBLE_DEVICES=0 \
vllm serve Qwen/Qwen3-8B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

Capture in SUT evidence:

ROCm version and kernel driver;
GPU model and memory, for example MI300X OAM;
container image, for example a ROCm PyTorch or ROCm vLLM image;
known unsupported vLLM flags or dtype restrictions.

TPU / JAX Serving Example

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --host 0.0.0.0 \
  --port 8000 \
  --tp-size 4 \
  --context-length 8192 \
  --dtype bfloat16

Pair this with server config fields such as TPU accelerator, topology, zone, node machine type, and runtime package. For example, a v5e-16 system should not be reported as just "TPU".

BYOE / Custom OpenAI-Compatible Endpoint

Some systems do not expose a local framework process that Metrum Insights starts. In that case, represent the candidate as a bring-your-own-endpoint workload and capture the endpoint metadata rather than a process command.

curl -fsS "$MODEL_ENDPOINT_URL/v1/models" \
  -H "Authorization: Bearer <endpoint-token>"

Record:

endpoint URL and provider kind;
model id returned by the endpoint;
claimed framework/runtime and version;
accelerator vendor and instance id;
rate limits, max context, supported modalities, and streaming support.

Template Variable Checklist

Use stable normalized keys. The exact flag can differ by framework, but the meaning should remain queryable.

Concept	vLLM key examples	SGLang/custom key examples
Port	`port`	`port`
Context length	`max_model_len`	`context_length`
Tensor parallelism	`tensor_parallel_size`	`tp_size`
Data parallelism	`data_parallel_size`	`dp_size`
Batch/sequences	`max_num_seqs`, `max_num_batched_tokens`	`schedule_policy`, `max_running_requests`
KV dtype	`kv_cache_dtype`	runtime-specific cache dtype key
KV layout	`block_size`, `enable_prefix_caching`	radix/prefix cache flags
Offload	`cpu_offload_gb`, cache backend keys	runtime-specific offload keys
Precision	`dtype`, `quantization_code`	`dtype`, runtime quantization key
Runtime tuning	`compilation_config`, `moe_backend`	backend/JIT/vendor flags

Failure Mode: Unresolved Placeholders

If a resolved command contains text like {data_parallel_size} or {block_size}, stop before launching the benchmark. This indicates a template or seed-data regression. Fix the command variable mapping first, then rerun the benchmark from a clean run.

Template Types​

Reproducibility Contract​

vLLM Baseline Example​

vLLM Engine Args Examples​

Long Context And Scheduler Headroom​

KV Cache FP8​

CPU Offload​

Different KV Block Sizes​

DeepSeek V4 / MoE Style Configuration​

SGLang Examples​

Dynamo Example Shape​

KV Cache Offload And Advanced Serving​

Custom And Non-NVIDIA Hardware​

AMD ROCm vLLM Example​

TPU / JAX Serving Example​

BYOE / Custom OpenAI-Compatible Endpoint​

Template Variable Checklist​

Failure Mode: Unresolved Placeholders​