Skip to main content

Choosing engine arguments

This is the section that saves you from failed runs. Engine arguments are the startup flags handed to the serving framework (vLLM, SGLang, or TensorRT-LLM). They are grouped into a named engine args set that a workload points at.

Why you must not guess​

Every model plus GPU combination has a known-good set of flags. Choosing flags at random has real consequences:

  • The server refuses to start. A max_model_len larger than the model supports, or a tensor-parallel size that does not divide the number of attention heads, fails at launch. The job then fails before a single request is sent.
  • The GPU runs out of memory. Set gpu_memory_utilization too high, or max_num_batched_tokens too large for the card, and the model out-of-memories (OOM) partway through the run.
  • The numbers are quietly wrong. A context length that cannot hold your input plus output silently truncates requests, so throughput looks good but the model was never doing the work you measured.

The fix is simple: start from a recipe that was verified for your model and your hardware, then change one thing at a time.

Start from the official recipes​

Two upstream sources publish tuned launch settings for popular models on common GPUs. Read the page for your model before you build an engine args set.

  • vLLM recipes: https://recipes.vllm.ai/ - per-model pages with the exact vllm serve flags for each GPU class (H100, H200, B200, GB200, and others), including tensor parallelism, KV cache dtype, and context length.
  • SGLang cookbook: https://lmsysorg.mintlify.app/cookbook/intro - the equivalent for SGLang, with per-model launch commands and hardware notes, including AMD MI300X-class parts.

Find your exact model on the matching page, read the recommended command for the GPU you are running, and copy the flags into an engine args set. This is the difference between a recipe and a guess.

The platform already ships verified recipes​

You often do not need to build a set by hand. Metrum AI Bench seeds verified recipe argsets: named presets whose values were copied directly from the vLLM recipes and SGLang cookbook pages, keyed to a specific model, GPU, framework version, and tensor-parallel size. The set code encodes exactly what it is for. For example:

Set codeModelHardwareParallelism
vllm-llama31-8b-instruct-h100-h200-trillium-xeon6-tp1Llama 3.1 8B InstructH100, H200, Trillium, Xeon6TP1
vllm-llama33-70b-instruct-h100-h200-tp8Llama 3.3 70B InstructH100, H200TP8
vllm-llama33-70b-instruct-b200-gb200-tp1-fusedLlama 3.3 70B InstructB200, GB200TP1, fused compilation

Verified argsets are seeded today for the Llama, DeepSeek, Qwen 3.5 and 3.6, Gemma, GLM, GPT-OSS, Kimi, Mistral, and Nemotron families. When a preset exists for your model and GPU, pick it. Its values already match the upstream recipe, so you avoid the whole guessing problem.

How to choose, step by step​

  1. Identify your model and GPU. For example, Llama 3.3 70B Instruct on eight H100s.
  2. Look for a verified argset that matches. Filter the engine args sets for your model and hardware. If one exists (for example vllm-llama33-70b-instruct-h100-h200-tp8), use it and skip to step 6.
  3. If none matches, open the recipe. Go to recipes.vllm.ai (or the SGLang cookbook for SGLang) and open your model's page.
  4. Read the flags for your GPU. Recipes list settings per GPU class. Take the tensor-parallel size, KV cache dtype, context length, and any model-specific flags from the row that matches your card.
  5. Build an engine args set from those values. Create a new set (via the API today) with those flags. Give it a descriptive code, such as vllm-<model>-<gpu>-tp<n>.
  6. Sanity-check three things before you launch:
    • Tensor parallelism divides the GPU count. tensor_parallel_size must equal the number of GPUs you are giving the model (or evenly divide it). TP8 needs 8 GPUs.
    • Context length covers your traffic. max_model_len must be at least your largest ISL plus OSL. If you benchmark isl2048-osl1024, max_model_len must be 3072 or more.
    • Memory headroom exists. Leave gpu_memory_utilization around 0.90. Lower it if the server is shared or if you see OOM.
  7. Run one smoke scenario first. Use concurrency=1, a tiny ISL and OSL, and num_requests=2. If the server starts and the job completes, your engine args are valid. Then run the full matrix.

The arguments you are most likely to set​

ArgumentvLLM keySGLang keyWhat it controlsRule of thumb
Tensor parallelismtensor_parallel_sizetp_sizeHow many GPUs the model is split across.Set to the number of GPUs the model needs to fit. Large models need more.
Context lengthmax_model_lencontext_lengthThe longest input-plus-output the server will accept.At least the largest ISL plus OSL in your scenario matrix.
KV cache dtypekv_cache_dtyperuntime cache dtypeThe numeric format of the attention cache. fp8 roughly halves cache memory.Use the value the recipe specifies. Test it as a separate set, since it can affect quality.
GPU memory fractiongpu_memory_utilizationmem_fraction_staticThe share of GPU memory the server may claim.Around 0.90. Lower on a shared box.
Batch sizingmax_num_seqs, max_num_batched_tokensscheduler flagsHow aggressively requests are batched together.Take from the recipe. Raising max_num_batched_tokens can lift throughput if it plateaus early.
Prefix cachingenable_prefix_cachingradix cache flagsReuse of shared prompt prefixes across requests.On for repeated-prefix traffic, such as a fixed system prompt.

Model-server replicas​

Tensor parallelism sets how many GPUs one model server uses (tensor-parallel size times data-parallel size). When that footprint is smaller than the GPUs on the host, Metrum fills the rest by running several model-server replicas on the same machine, so a benchmark uses the whole box instead of leaving GPUs idle.

  • Replica count is the total GPUs divided by the GPUs per replica, where GPUs per replica is TP times DP. Eight GPUs with a TP2 recipe (DP left at its default of 1) run four replicas; the same eight GPUs with a TP8 recipe run one. If you also set DP2 alongside TP2, each replica now needs 4 GPUs (TP2 × DP2), so those same eight GPUs run two replicas instead of four.
  • Each replica is isolated. It is pinned to its own GPU set and listens on its own port (the base port plus the replica index), so replicas never share GPUs or collide on a port.
  • Load is spread across replicas. The benchmark tool sends the scenario's traffic across every replica endpoint, so the reported throughput reflects the full host, not a single replica.

Quantization​

Quantization is the numeric format the model weights are served in. It is set by the Quantization field on the workload card, but treat it as part of the engine configuration: the recipe for your model and GPU already specifies the format it was tuned for, so match it rather than picking one on its own.

QuantizationQuality impactThroughput / memory impactTypical use
fp16BaselineBaselineReference baseline
bf16About identical to fp16SameWhen the hardware prefers bf16
fp8SmallAbout 1.7x throughput, about 0.5x memoryProduction default on H100 and newer
int8NoticeableAbout 2x throughput, about 0.5x memoryCost-optimized deployments
awq / gptqSmall to moderateAbout 3x throughput, about 0.25x memoryMemory-constrained scenarios
tip

Weight quantization and KV cache dtype are separate knobs. fp8 weights and an fp8 kv_cache_dtype are set independently, and each can affect quality. Change one at a time.

warning

If quality matters as much as throughput, pair any quantization change with a KYAI evaluation on the same endpoints, so a throughput gain does not hide a quality regression.

Compare recipes cleanly​

When you want to compare two settings, make each one its own engine args set and its own workload, and keep the scenarios identical. Never hand-edit a command line for a single run and then report it as the same workload. See Compare vLLM Versions And Settings and Command Templates for worked examples, including FP8 KV cache, long context, CPU offload, and MoE-style configurations.

A tuning change is a win only if it improves the primary metric without raising the failure rate, memory pressure, startup failures, or quality regressions. Pair an engine-args change with a KYAI evaluation when quality matters, so a throughput gain does not hide a quality loss.