Skip to main content

Understanding key metrics

Four metrics carry most of the signal; the rest are breakdowns or telemetry. For the full list and exact formulas, see the Metric Definitions Reference.

MetricWhat it measuresWhat moves itWhat to watch
Throughput (tokens/sec)Total output tokens generated per second across all concurrent requests. The headline number.Rises with concurrency until the GPU saturates, then plateaus. Rises with smaller models and aggressive quantization. Falls with longer OSL.The plateau is the server's ceiling on this hardware.
TTFT (Time To First Token)Wall-clock from request arrival to the first output token. Captures prefill and queueing.Grows with concurrency (queueing) and ISL (prefill work).Track P99, not P50, for latency SLAs.
TPOT (Time Per Output Token)Average time between output tokens after the first. Captures steady-state generation speed.Grows mildly with concurrency until saturation. Roughly model-size-bound.User-perceived tokens/sec per request is about 1 / TPOT.
Latency percentilesP50, P90, and P99 for both TTFT and TPOT.The tail widens under queueing or jitter.P99 far above P50 means queueing; raise capacity or lower concurrency.
GPU utilizationFrom telemetry. Fraction of GPU compute cycles in use.Rises with concurrency on a healthy run.Below 50% at high concurrency means the workload is CPU-, network-, or memory-bandwidth-bound. Check the Hardware chart tab.
note

Streaming must be on to measure TTFT. Without it, the tool only sees the final response body.