GenAI-Perf benchmarking
GenAI-Perf is NVIDIA's third-party LLM load tool from the Triton Perf Analyzer ecosystem. It is still an LLM workload: same modality, same OpenAI-compatible chat endpoint, same scenario axes (concurrency, ISL, OSL, streaming). The differences from metrumbench-llm are the load source (synthetic token lengths only, no prompt library) and the serving topology (single endpoint, replicas = 1, TP = hardware GPU count).
Use GenAI-Perf when you want:
- An NVIDIA-reference baseline for throughput, latency, TTFT, and inter-token latency
- Exact synthetic ISL/OSL (mean with stddev 0) for controlled grids
- A second opinion beside metrumbench-llm on the same model-server recipe
Use metrumbench-llm when you need real prompt content, prompt libraries, multi-replica load-balanced endpoints, or richer first-party metrics. Full metric definitions, defaults, and known issues live in GenAI-Perf Methodology.
Step 1: create the project and pick GenAI-Perf
- Go to Projects → New Project.
- Enter a Project Name and set Visibility (Private or Team).
- Under Workloads, open Text Generation and click GenAI-Perf to add a workload card.
Step 2: configure the workload
Work through the card in order:
- Target Hardware: select a server chip and set GPU Count if needed.
- Models: type or select HuggingFace model IDs and press Enter. Add multiple models for a multi-model run: every model shares the same engine args, scenario matrix, and server. A heads-up banner reminds you of that shared setup.
- HuggingFace Token: if a model is gated, click + Add Token and paste your token (account-level).
- Framework: pick vLLM, SGLang, or TensorRT-LLM. If only one framework is compatible, it auto-selects and locks.
- Version: pick the framework release. The latest compatible version auto-selects.
- Engine Args Set: pick a pre-configured template, or leave unset for the framework default. Custom org sets show "(custom)".
- Build Profile: pick the hardware build/launch variant (for example Default (CUDA) or ROCm) when profiles are available.
- GenAI-Perf Args: expand to edit tool args. Exclude with X, Add custom arg, or Save arg set team-wide after edits.
- Framework Args (for example vLLM Args): same edit/add/remove/save mechanics as tool args.
- Dataset: GenAI-Perf is synthetic-only, so this field is hidden or shows a synthetic source.
Step 3: configure the scenario matrix
| Field | What it controls |
|---|---|
| Streaming | Yes / No. Required for TTFT and inter-token latency. |
| Concurrency | Parallel requests. Presets include 1 through 8192. |
| Input Sequence Length | Synthetic prompt length in tokens (exact, stddev 0). |
| Output Sequence Length | Synthetic generation length in tokens (exact, stddev 0). |
Every combination becomes one scenario. The preview panel shows codes such as c1-isl256-osl128.
Request count is not exposed in the UI; it defaults to max(200, concurrency × 10) per scenario (same shared default as metrumbench-llm).
Endpoint type is fixed to chat and the random seed is fixed to 0 for reproducibility.
High concurrency (≥ 512, especially 1024) needs ulimit -n 65536 on the worker.
The benchmark targets a single endpoint URL with replicas = 1 and TP = GPU count on the selected hardware. If you need multi-replica load-balanced benchmarks, use metrumbench-llm's multi-endpoint YAML instead.
Step 4: read results
Filter the Reporting page to the GenAI-Perf workload. Review per scenario:
- Output token throughput and request throughput
- TTFT mean / p50 / p95 / p99 (plus extended percentiles on the detail view)
- Inter-token latency mean / p95 / p99
- Request latency mean / p95 / p99
- Successful vs failed request counts
- Hardware tab: GPU power, utilization, memory, and CPU metrics captured
by the agent during the run (telemetry views:
v_telemetry_job_summary,v_telemetry_per_gpu_summary)
Streaming is required for useful TTFT and inter-token latency. Non-streaming still produces end-to-end latency and throughput.
When comparing GenAI-Perf to metrumbench-llm, hold model, framework, engine args, hardware, replicas, TP, ISL, OSL, concurrency, and streaming fixed. Vary only the tool. Treat absolute number parity as directional: GenAI-Perf uses synthetic tokens; metrumbench-llm uses prompt content.
The API path
GenAI-Perf workloads and scenarios can be created over the API through the
same RPCs used by metrumbench-llm (only the p_tool_code changes to
genai-perf). See
metrumbench-llm Workload And Scenario
and Project Lifecycle for the API reference.