Skip to main content

Every argument explained

This section defines every value you can set, in plain language, so nothing on the workload card or in an API call is a mystery. The three workloads share most arguments. Anything specific to one workload is called out again inside that workload's guide.

Arguments live at two levels. Workload arguments describe what is being served. Scenario arguments describe the traffic sent to it.

Workload arguments​

ArgumentUI labelWhat it meansHow to choose
ModelModelThe AI model to benchmark, picked from the registered catalog.Pick the exact model you plan to deploy.
FrameworkFrameworkThe serving engine that runs the model: vLLM, SGLang, or TensorRT-LLM.Start with vLLM. See Picking a framework.
VersionVersionThe framework version, which also selects the container image.Use a stable version unless you are specifically testing an upgrade.
QuantizationQuantizationThe numeric format the model weights are served in (fp16, bf16, fp8, int8, awq, gptq). Smaller formats use less memory and run faster, sometimes at a quality cost.Match the recipe for your model and GPU. See Quantization under engine arguments.
Engine args setEngine argsA named preset of framework startup flags, for example tensor parallelism, KV cache dtype, and maximum context length.Do not guess these. See Choosing engine arguments.
Tool(set automatically)The benchmark client: metrumbench-llm, metrumbench-asr, or metrumbench-vlm.Selected automatically from the model type.

Scenario arguments (shared by all three workloads)​

ArgumentAPI fieldWhat it meansTypical values
Concurrencyp_concurrencyHow many requests are in flight at the same time. This is the single most important knob for throughput and latency.1, 8, 32, 64
Input sequence length (ISL)p_input_sequence_lengthThe target size of each input, in tokens. For LLM and VLM this is the prompt length. It models how large your real inputs are.128 to 4096
Output sequence length (OSL)p_output_sequence_lengthThe target number of tokens the model should generate per request.128 to 1024
Requests per scenariop_num_requests (API and CLI only)How many total requests to send in the scenario. The wizard has no field for it. When you leave it out, the platform derives it from concurrency, as How many requests a scenario sends shows.Leave unset in the wizard. Set it from the API or CLI for an exact count
Max tokensp_max_tokensA hard cap on generated tokens per request. When left unset, the tool uses the OSL value.Usually leave equal to OSL
Streamingp_streaming and the Streaming toggleWhether the response is consumed token by token (server-sent events) or waited for as one final body. Streaming is required to measure Time To First Token.On for latency studies

How many requests a scenario sends​

A scenario sends the number of requests you set in p_num_requests. When you leave it out, which is always the case in the wizard, the platform derives it from the scenario's concurrency. The rule depends on the tool.

ToolRequests when p_num_requests is not set
MetrumBench LLM, VLM and ASR, GenAI-PerfThe greater of 200 and 10 times the concurrency
MetrumBench ImageGenThe greater of 4 and 4 times the concurrency
InferenceXThe greater of 16 and 8 times the concurrency
vLLM benchp_num_prompts when set, else 1

A single token is roughly three-quarters of an English word. So an ISL of 512 tokens is about 380 words of prompt, and an OSL of 256 is about 190 words of answer.

Tool-level arguments (advanced, usually left at defaults)​

The platform sets these to safe defaults. You rarely change them, but here is what each one does.

ArgumentApplies toWhat it means
modeLLMchat wraps the prompt in a messages array with a system message (the normal case). completion sends the raw prompt to the legacy completions endpoint.
temperatureLLM, VLMSampling randomness. Benchmarks keep this low and fixed so runs are repeatable.
ramp_up_secondsLLM, VLMA warm-up window during which concurrency climbs from 1 to the target. Metrics from this window are discarded, so the reported numbers reflect steady state, not cold start.
request_timeoutallMaximum seconds to wait for a single request before it is counted as failed.
connect_timeoutallMaximum seconds allowed to open the TCP connection.
job_timeout_secondsallHard wall-clock limit for the whole job.
num_images_batchVLMNumber of images attached to each request.
image_detailVLMlow sends the image at reduced fidelity (fewer image tokens, faster) or high sends full fidelity (more accurate, slower). Only these two values are valid.
inputASRPath or URL to the JSONL manifest listing the audio files to transcribe.
ground_truthASRPath or URL to reference transcripts. When present, the tool computes accuracy (Word Error Rate and Character Error Rate).
response_formatASRThe transcription response shape: verbose_json, json, text, srt, or vtt.