Skip to main content

GenAI-Perf benchmarking

GenAI-Perf is NVIDIA's third-party LLM load tool from the Triton Perf Analyzer ecosystem. It is still an LLM workload: same modality, same OpenAI-compatible chat endpoint, same scenario axes (concurrency, ISL, OSL, streaming). The differences from metrumbench-llm are the load source (synthetic token lengths only, no prompt library) and the serving topology (single endpoint, replicas = 1, TP = hardware GPU count).

Use GenAI-Perf when you want:

  • An NVIDIA-reference baseline for throughput, latency, TTFT, and inter-token latency
  • Exact synthetic ISL/OSL (mean with stddev 0) for controlled grids
  • A second opinion beside metrumbench-llm on the same model-server recipe

Use metrumbench-llm when you need real prompt content, prompt libraries, multi-replica load-balanced endpoints, or richer first-party metrics. Full metric definitions, defaults, and known issues live in GenAI-Perf Methodology.

Step 1: create the project and pick GenAI-Perf​

  1. Go to Projects → New Project.
  2. Enter a Project Name and set Visibility (Private or Team).
  3. Under Workloads, open Text Generation and click GenAI-Perf to add a workload card.

Step 2: configure the workload​

Work through the card in order:

  1. Target Hardware: select a server chip and set GPU Count if needed.
  2. Models: type or select HuggingFace model IDs and press Enter. Add multiple models for a multi-model run: every model shares the same engine args, scenario matrix, and server. A heads-up banner reminds you of that shared setup.
  3. HuggingFace Token: if a model is gated, click + Add Token and paste your token (account-level).
  4. Framework: pick vLLM, SGLang, or TensorRT-LLM. If only one framework is compatible, it auto-selects and locks.
  5. Version: pick the framework release. The latest compatible version auto-selects.
  6. Engine Args Set: pick a pre-configured template, or leave unset for the framework default. Custom org sets show "(custom)".
  7. Build Profile: pick the hardware build/launch variant (for example Default (CUDA) or ROCm) when profiles are available.
  8. GenAI-Perf Args: expand to edit tool args. Exclude with X, Add custom arg, or Save arg set team-wide after edits.
  9. Framework Args (for example vLLM Args): same edit/add/remove/save mechanics as tool args.
  10. Dataset: GenAI-Perf is synthetic-only, so this field is hidden or shows a synthetic source.

Step 3: configure the scenario matrix​

FieldWhat it controls
StreamingYes / No. Required for TTFT and inter-token latency.
ConcurrencyParallel requests. Presets include 1 through 8192.
Input Sequence LengthSynthetic prompt length in tokens (exact, stddev 0).
Output Sequence LengthSynthetic generation length in tokens (exact, stddev 0).

Every combination becomes one scenario. The preview panel shows codes such as c1-isl256-osl128.

Request count is not exposed in the UI; it defaults to max(200, concurrency × 10) per scenario (same shared default as metrumbench-llm).

Endpoint type is fixed to chat and the random seed is fixed to 0 for reproducibility.

High concurrency (≥ 512, especially 1024) needs ulimit -n 65536 on the worker.

The benchmark targets a single endpoint URL with replicas = 1 and TP = GPU count on the selected hardware. If you need multi-replica load-balanced benchmarks, use metrumbench-llm's multi-endpoint YAML instead.

Step 4: read results​

Filter the Reporting page to the GenAI-Perf workload. Review per scenario:

  • Output token throughput and request throughput
  • TTFT mean / p50 / p95 / p99 (plus extended percentiles on the detail view)
  • Inter-token latency mean / p95 / p99
  • Request latency mean / p95 / p99
  • Successful vs failed request counts
  • Hardware tab: GPU power, utilization, memory, and CPU metrics captured by the agent during the run (telemetry views: v_telemetry_job_summary, v_telemetry_per_gpu_summary)

Streaming is required for useful TTFT and inter-token latency. Non-streaming still produces end-to-end latency and throughput.

When comparing GenAI-Perf to metrumbench-llm, hold model, framework, engine args, hardware, replicas, TP, ISL, OSL, concurrency, and streaming fixed. Vary only the tool. Treat absolute number parity as directional: GenAI-Perf uses synthetic tokens; metrumbench-llm uses prompt content.

The API path​

GenAI-Perf workloads and scenarios can be created over the API through the same RPCs used by metrumbench-llm (only the p_tool_code changes to genai-perf). See metrumbench-llm Workload And Scenario and Project Lifecycle for the API reference.