Choosing engine arguments
This is the section that saves you from failed runs. Engine arguments are the startup flags handed to the serving framework (vLLM, SGLang, or TensorRT-LLM). They are grouped into a named engine args set that a workload points at.
Why you must not guess
Every model plus GPU combination has a known-good set of flags. Choosing flags at random has real consequences:
- The server refuses to start. A
max_model_lenlarger than the model supports, or a tensor-parallel size that does not divide the number of attention heads, fails at launch. The job then fails before a single request is sent. - The GPU runs out of memory. Set
gpu_memory_utilizationtoo high, ormax_num_batched_tokenstoo large for the card, and the model out-of-memories (OOM) partway through the run. - The numbers are quietly wrong. A context length that cannot hold your input plus output silently truncates requests, so throughput looks good but the model was never doing the work you measured.
The fix is simple: start from a recipe that was verified for your model and your hardware, then change one thing at a time.
Start from the official recipes
Two upstream sources publish tuned launch settings for popular models on common GPUs. Read the page for your model before you build an engine args set.
- vLLM recipes: https://recipes.vllm.ai/ - per-model pages with the exact
vllm serveflags for each GPU class (H100, H200, B200, GB200, and others), including tensor parallelism, KV cache dtype, and context length. - SGLang cookbook: https://lmsysorg.mintlify.app/cookbook/intro - the equivalent for SGLang, with per-model launch commands and hardware notes, including AMD MI300X-class parts.
Find your exact model on the matching page, read the recommended command for the GPU you are running, and copy the flags into an engine args set. This is the difference between a recipe and a guess.
The platform already ships verified recipes
You often do not need to build a set by hand. Metrum AI Bench seeds verified recipe argsets: named presets whose values were copied directly from the vLLM recipes and SGLang cookbook pages, keyed to a specific model, GPU, framework version, and tensor-parallel size. The set code encodes exactly what it is for. For example:
| Set code | Model | Hardware | Parallelism |
|---|---|---|---|
vllm-llama31-8b-instruct-h100-h200-trillium-xeon6-tp1 | Llama 3.1 8B Instruct | H100, H200, Trillium, Xeon6 | TP1 |
vllm-llama33-70b-instruct-h100-h200-tp8 | Llama 3.3 70B Instruct | H100, H200 | TP8 |
vllm-llama33-70b-instruct-b200-gb200-tp1-fused | Llama 3.3 70B Instruct | B200, GB200 | TP1, fused compilation |
Verified argsets are seeded today for the Llama, DeepSeek, Qwen 3.5 and 3.6, Gemma, GLM, GPT-OSS, Kimi, Mistral, and Nemotron families. When a preset exists for your model and GPU, pick it. Its values already match the upstream recipe, so you avoid the whole guessing problem.
How to choose, step by step
- Identify your model and GPU. For example, Llama 3.3 70B Instruct on eight H100s.
- Look for a verified argset that matches. Filter the engine args sets for your model and hardware. If one exists (for example
vllm-llama33-70b-instruct-h100-h200-tp8), use it and skip to step 6. - If none matches, open the recipe. Go to recipes.vllm.ai (or the SGLang cookbook for SGLang) and open your model's page.
- Read the flags for your GPU. Recipes list settings per GPU class. Take the tensor-parallel size, KV cache dtype, context length, and any model-specific flags from the row that matches your card.
- Build an engine args set from those values. Create a new set (via the API today) with those flags. Give it a descriptive code, such as
vllm-<model>-<gpu>-tp<n>. - Sanity-check three things before you launch:
- Tensor parallelism divides the GPU count.
tensor_parallel_sizemust equal the number of GPUs you are giving the model (or evenly divide it). TP8 needs 8 GPUs. - Context length covers your traffic.
max_model_lenmust be at least your largest ISL plus OSL. If you benchmarkisl2048-osl1024,max_model_lenmust be 3072 or more. - Memory headroom exists. Leave
gpu_memory_utilizationaround 0.90. Lower it if the server is shared or if you see OOM.
- Tensor parallelism divides the GPU count.
- Run one smoke scenario first. Use
concurrency=1, a tiny ISL and OSL, andnum_requests=2. If the server starts and the job completes, your engine args are valid. Then run the full matrix.
The arguments you are most likely to set
| Argument | vLLM key | SGLang key | What it controls | Rule of thumb |
|---|---|---|---|---|
| Tensor parallelism | tensor_parallel_size | tp_size | How many GPUs the model is split across. | Set to the number of GPUs the model needs to fit. Large models need more. |
| Context length | max_model_len | context_length | The longest input-plus-output the server will accept. | At least the largest ISL plus OSL in your scenario matrix. |
| KV cache dtype | kv_cache_dtype | runtime cache dtype | The numeric format of the attention cache. fp8 roughly halves cache memory. | Use the value the recipe specifies. Test it as a separate set, since it can affect quality. |
| GPU memory fraction | gpu_memory_utilization | mem_fraction_static | The share of GPU memory the server may claim. | Around 0.90. Lower on a shared box. |
| Batch sizing | max_num_seqs, max_num_batched_tokens | scheduler flags | How aggressively requests are batched together. | Take from the recipe. Raising max_num_batched_tokens can lift throughput if it plateaus early. |
| Prefix caching | enable_prefix_caching | radix cache flags | Reuse of shared prompt prefixes across requests. | On for repeated-prefix traffic, such as a fixed system prompt. |
Model-server replicas
Tensor parallelism sets how many GPUs one model server uses (tensor-parallel size times data-parallel size). When that footprint is smaller than the GPUs on the host, Metrum fills the rest by running several model-server replicas on the same machine, so a benchmark uses the whole box instead of leaving GPUs idle.
- Replica count is the total GPUs divided by the GPUs per replica, where GPUs per replica is TP times DP. Eight GPUs with a TP2 recipe (DP left at its default of 1) run four replicas; the same eight GPUs with a TP8 recipe run one. If you also set DP2 alongside TP2, each replica now needs 4 GPUs (TP2 × DP2), so those same eight GPUs run two replicas instead of four.
- Each replica is isolated. It is pinned to its own GPU set and listens on its own port (the base port plus the replica index), so replicas never share GPUs or collide on a port.
- Load is spread across replicas. The benchmark tool sends the scenario's traffic across every replica endpoint, so the reported throughput reflects the full host, not a single replica.
Quantization
Quantization is the numeric format the model weights are served in. It is set by the Quantization field on the workload card, but treat it as part of the engine configuration: the recipe for your model and GPU already specifies the format it was tuned for, so match it rather than picking one on its own.
| Quantization | Quality impact | Throughput / memory impact | Typical use |
|---|---|---|---|
fp16 | Baseline | Baseline | Reference baseline |
bf16 | About identical to fp16 | Same | When the hardware prefers bf16 |
fp8 | Small | About 1.7x throughput, about 0.5x memory | Production default on H100 and newer |
int8 | Noticeable | About 2x throughput, about 0.5x memory | Cost-optimized deployments |
awq / gptq | Small to moderate | About 3x throughput, about 0.25x memory | Memory-constrained scenarios |
Weight quantization and KV cache dtype are separate knobs. fp8 weights and an fp8 kv_cache_dtype are set independently, and each can affect quality. Change one at a time.
If quality matters as much as throughput, pair any quantization change with a KYAI evaluation on the same endpoints, so a throughput gain does not hide a quality regression.
Compare recipes cleanly
When you want to compare two settings, make each one its own engine args set and its own workload, and keep the scenarios identical. Never hand-edit a command line for a single run and then report it as the same workload. See Compare vLLM Versions And Settings and Command Templates for worked examples, including FP8 KV cache, long context, CPU offload, and MoE-style configurations.
A tuning change is a win only if it improves the primary metric without raising the failure rate, memory pressure, startup failures, or quality regressions. Pair an engine-args change with a KYAI evaluation when quality matters, so a throughput gain does not hide a quality loss.