LLM benchmarking (metrumbench-llm)
This is the most common workload. It measures how fast a text language model serves generation requests under load, using the metrumbench-llm tool. If you followed the Quickstart, you have already run one. This section is the complete reference.
What it measures and when to use it
metrumbench-llm drives an OpenAI-compatible chat or completion endpoint with real prompts, at a concurrency you choose, and records the timing of every request. Use it to answer questions like:
- How many tokens per second can this model serve on this GPU?
- What is the latency my users would feel at 8, 32, and 64 concurrent requests?
- How much faster is fp8 than bf16 on the same hardware, and does quality hold?
- Which serving framework, vLLM or SGLang, is faster for my model?
The headline metrics are throughput (tokens per second), TTFT (time to first token), and TPOT (time per output token). They are defined in Understanding key metrics.
For an NVIDIA-reference baseline or exact synthetic ISL/OSL, see GenAI-Perf benchmarking.
Step 1: create the workload (Web UI)
- Go to Projects, then New Project (or open a project and select Edit Project, or Copy Project when the project has run).
- Give the workload a clear name, for example
llama-70b-vllm-bf16. - Model: pick the language model from the dropdown. The list is the registered model catalog.
- Framework: choose
vLLMto start. See Picking a framework for when to use SGLang or TensorRT-LLM. - Version: pick a stable framework version.
- Quantization (optional): leave at the model default, or pick fp8 to test a faster, smaller format. See Quantization.
- Engine args: attach the verified recipe set for your model and GPU. This step matters; read Choosing engine arguments first.
Step 2: define the scenario matrix
In the workload card, the Scenario Matrix uses chip buttons. Toggle the values you want on each axis, and the platform multiplies them out.
- Concurrency chips:
1, 4, 8, 16, 32, 64, 128. - Input Sequence Length (ISL) chips:
128, 256, 512, 1024, 2048. - Output Sequence Length (OSL) chips:
64, 128, 256, 512. - Requests per scenario: the wizard has no field for it. The platform sets it from the concurrency of each scenario.
- Streaming: leave on to measure TTFT.
Selecting concurrency 1, 8, 32 with one ISL and one OSL produces three scenarios. Each becomes one job named c<concurrency>-isl<isl>-osl<osl>, for example c8-isl512-osl256. Click Show details to preview the rolled-out scenarios. Start small: a 24-scenario matrix on a 70B model can take an hour.
Match the ISL and OSL to your real traffic. If your production prompts are 4k tokens and you benchmark at 512, your numbers will be optimistic.
Step 3: run it
Select Save & Run. The editor checks the project, saves it, starts its runs, and opens the Monitoring page. Each scenario runs as one job, which moves through pending, queued, running, and completed. To run the project later, select Save instead. See What Save & Run does.
Step 4: read the results
Once a job completes, open the Reporting page, filter to your run, and read throughput, TTFT P50 and P99, and TPOT. Open the Concurrency chart tab to find the saturation point: throughput rises then flattens, and P99 latency stays flat then climbs. The concurrency at the elbow is the sweet spot for this model on this hardware.
Running across multiple servers
To spread a metrumbench-llm workload across more than one server (tensor
and pipeline parallelism across a cluster), see
Multi-Node vLLM Deployment. Every
server you plan to group must have had its IP address set when it was
onboarded; a server onboarded without one cannot join a multi-node cluster.
The API path
The same workload and scenarios can be created over the API. See metrumbench-llm Workload And Scenario and Project Lifecycle for the full API reference.
LLM-specific arguments
Beyond the shared scenario arguments, metrumbench-llm exposes a few tool-level settings. They are defaulted for you; change them only with a reason.
| Argument | What it means | Default behavior |
|---|---|---|
mode | chat sends prompts through the chat completions endpoint with a system message. completion uses the legacy completions endpoint. | chat |
streaming | Consume the response as a server-sent event stream so TTFT can be measured chunk by chunk. | On |
ramp_up_seconds | Warm-up window where concurrency climbs to the target; warm-up metrics are discarded to report steady state. | Set by the platform |
temperature | Sampling randomness, kept low and fixed for repeatable runs. | Low fixed value |
max_tokens | Hard cap on generated tokens per request. When unset, OSL is used. | Equals OSL |
Token counts in the results come from the model server's own usage response, not from local tokenization, so they reflect exactly what the server did.
Common pitfalls
- Concurrency too high for the model. Concurrency
128on a 70B model with 80 GB of memory will OOM. Start at8, find saturation, then push higher. - Context too short for the scenario. If your engine args set a
max_model_lenbelow ISL plus OSL, requests are truncated. See Choosing engine arguments. - Too few requests. A scenario created from the API or CLI with a small
p_num_requestsis a smoke test, not a measurement. Leave it unset in the wizard, or set hundreds to thousands for numbers you will report.