GenAI-Perf Methodology
GenAI-Perf is NVIDIA’s generative-AI load generator from the Triton Perf Analyzer ecosystem. In Metrum Insights it is a third-party benchmark tool used to measure serving performance of OpenAI-compatible chat endpoints under controlled concurrency and synthetic sequence lengths.
Use GenAI-Perf when you need:
- An NVIDIA-reference baseline for throughput, latency, TTFT, and inter-token latency
- Exact synthetic ISL/OSL (mean with stddev 0) for fair comparisons
- Cross-checks against metrumbench-llm on the same model-server recipe
Do not use GenAI-Perf for answer quality. It measures serving performance only.
How Insights Uses GenAI-Perf
- Resolves a framework recipe (vLLM / SGLang / TensorRT-LLM) and starts or attaches to one OpenAI-compatible chat server.
- Targets that server with a single base URL (
--url). - Runs the tool wrapper
python3 -m insights_cli.tools.genai_perf_runner, which invokesgenai-perf profileon the host or inside the Triton SDK container. - Generates synthetic / random token sequences (not a prompt library) with fixed means and zero stddev.
- Ingests vendor JSON/CSV artifacts and exposes them through
v_genai_perf_outputs.
Principles
- Synthetic and deterministic — every request targets the same ISL/OSL.
- Load-shaped, not content-shaped — concurrency and token pressure only.
- Streaming is user-controlled — on or off from the scenario setting. TTFT and inter-token latency need streaming on; non-streaming still yields end-to-end latency and throughput.
Tool Contract
| Item | Value |
|---|---|
| Tool code | genai-perf |
| Tool type | third_party |
| Modality | llm only |
| Pinned version | genai-perf==0.0.16 |
| Dataset | Synthetic / random token sequences (not a prompt library) |
| Endpoint type | chat |
| Random seed | 0 |
| Input / output stddev | 0 (exact sequence lengths) |
| Default request count | max(200, concurrency × 10) |
| Streaming | On or off from the user’s scenario setting |
| Serving frameworks | vLLM, SGLang, TensorRT-LLM |
| Load balancing | None — single endpoint URL |
| Replicas | 1 |
| Tensor parallelism | TP = GPU count on the selected hardware |
Dataset
GenAI-Perf does not use a prompt library, JSONL file, or real text content.
The dataset is synthetic: random token ID sequences produced by a seeded
generator. This is the NVIDIA synthetic-input path (--synthetic-input-tokens-*),
not file-backed prompts.
--synthetic-input-tokens-mean {input_sequence_length}
--synthetic-input-tokens-stddev 0
--output-tokens-mean {output_sequence_length}
--output-tokens-stddev 0
--num-dataset-entries {num_requests}
--request-count {num_requests}
--random-seed {random_seed}
| Control | Meaning |
|---|---|
| Input sequence length (ISL) | Target prompt length in tokens |
| Output sequence length (OSL) | Target generation length in tokens |
| Request count | Total requests sent; also sets how many synthetic entries are generated |
| Random seed | Seed for the synthetic generator (0 by default) |
With stddev 0 and seed 0, every request in a scenario targets the same ISL
and OSL, and runs with the same seed are comparable. The tokenizer is the
workload model (--tokenizer {model}).
For real prompt libraries or production-like content, use metrumbench-llm.
Serving Topology
GenAI-Perf is not load-balanced. It talks to one server process.
| Choice | Behavior |
|---|---|
| Endpoint | Single --url base URL |
| Replicas | 1 |
| Tensor parallelism | TP = number of GPUs on the selected hardware |
On a 1× GPU host: TP = 1, replicas = 1. On an 8× GPU host: TP = 8, replicas = 1 — the framework shards the model across all GPUs, but GenAI-Perf still hits one URL. Results are single-endpoint capacity, not multi-replica aggregate capacity.
Supported Serving Frameworks
GenAI-Perf targets OpenAI-compatible chat endpoints. Framework choice sets the engine arguments (model load, memory, scheduling). Those come from the workload Engine Args Set and are independent of GenAI-Perf — the tool only drives the client side of the measurement.
vLLM
Default OpenAI-compatible server. Key engine args: tensor-parallel-size (set
to hardware GPU count), max-model-len, gpu-memory-utilization, dtype,
enable-chunked-prefill. Runs as vllm serve.
See vLLM recipes for per-model launch flags.
SGLang
OpenAI-compatible server with RadixAttention. Key engine args: tp (tensor
parallelism), mem-fraction-static, enable-prefix-caching,
reasoning-parser. Runs as python3 -m sglang.launch_server.
See the SGLang cookbook
for per-model launch configurations.
TensorRT-LLM
NVIDIA-optimised serving via trtllm-serve. Key engine args:
tensor-parallel-size, max-num-tokens, kv-cache-free-gpu-mem-fraction,
backend. Often needs an engine build before serve.
See the TensorRT-LLM documentation
for engine building and serving guides.
Metrics
GenAI-Perf writes *_genai_perf.json and *_genai_perf.csv. Insights normalizes
vendor keys into seeded metric templates (v_genai_perf_outputs).
| Metric | Unit | Definition |
|---|---|---|
| TTFT (mean, p50, p95, p99) | ms | Time to first token |
| Inter-token latency (mean, p95, p99) | ms | Gap between consecutive output tokens |
| Request latency (mean, p95, p99) | ms | End-to-end request latency |
| Output token throughput | tokens/sec | Aggregate output tokens per second |
| Request throughput | requests/sec | Completed requests per second |
| Input / output sequence length | tokens | Reported lengths |
| Concurrency / request count | — | Configured load shape |
| Successful / failed request count | requests | Completion counts |
Extended percentiles are retained when present.
System telemetry: During the run, the Metrum agent samples GPU power,
utilization, memory, and CPU metrics from the worker host and stores them
alongside the GenAI-Perf results. Use the Benchmarks page Hardware chart
tab or telemetry views (v_telemetry_job_summary) to see GPU power draw,
memory pressure, and utilization — these are not from GenAI-Perf's optional
DCGM scrape but from the platform's own agent telemetry path.
Reading results: throughput for capacity; TTFT for interactive UX; ITL for streaming smoothness; p95/p99 for tail risk; failed requests near zero before treating a cell as valid. Use streaming on when you need TTFT/ITL.
Scenario → CLI Mapping
| Scenario field | GenAI-Perf flag |
|---|---|
| Concurrency | --concurrency |
| Input Sequence Length | --synthetic-input-tokens-mean |
| Output Sequence Length | --output-tokens-mean |
| Number of Requests | --request-count, --num-dataset-entries |
| Streaming | --streaming when the user sets streaming on |
| Model / tokenizer | -m, --tokenizer |
| Endpoint type | --endpoint-type chat |
| Random seed | --random-seed 0 |
python3 -m insights_cli.tools.genai_perf_runner \
-m {model} \
{endpoint_args} \
--tokenizer {model} \
--endpoint-type {endpoint_type} \
{streaming} \
--synthetic-input-tokens-mean {input_sequence_length} \
--synthetic-input-tokens-stddev 0 \
--output-tokens-mean {output_sequence_length} \
--output-tokens-stddev 0 \
--request-count {num_requests} \
--num-dataset-entries {num_requests} \
--concurrency {concurrency} \
--random-seed {random_seed} \
--profile-export-file {profile_export_file} \
--artifact-dir {artifact_dir}
{endpoint_args} is the server base URL (not /v1/chat/completions) plus
headers:
--url http://<host>:<port> \
-H "Authorization: Bearer <api_key>" \
-H "Accept: text/event-stream"
The wrapper uses host GenAI-Perf when available; otherwise the Triton SDK
container (nvcr.io/nvidia/tritonserver:26.04-py3-sdk). High concurrency
(≥ 512) needs ulimit -n 65536 on the worker.
GenAI-Perf vs metrumbench-llm
| Dimension | GenAI-Perf | metrumbench-llm |
|---|---|---|
| Ownership | Third-party (NVIDIA) | First-party (Metrum) |
| Dataset | Synthetic / random tokens | Prompt library / JSONL |
| ISL/OSL | Exact mean, stddev 0 | Driven by prompt content |
| Endpoint | Base URL, single endpoint | Full chat path; multi-endpoint supported |
| Topology | Replicas = 1, TP = hardware GPUs | Same managed recipes; multi-endpoint client available |
| Default request count | max(200, concurrency × 10) | Same shared default |
For a fair comparison, hold model, framework, engine args, hardware, replicas, TP, ISL, OSL, concurrency, request count, and streaming fixed. Vary only the tool.
Summary
GenAI-Perf is the NVIDIA-reference LLM load tool in Insights: synthetic /
random dataset (not a prompt library), random_seed = 0, OpenAI chat,
single endpoint with replicas = 1 and TP = hardware GPU count, against
vLLM, SGLang, or TensorRT-LLM.
It answers “How fast can this single endpoint serve under this load shape?”
Operator steps: User Guide → GenAI-Perf benchmarking.