Understanding key metrics
Four metrics carry most of the signal; the rest are breakdowns or telemetry. For the full list and exact formulas, see the Metric Definitions Reference.
| Metric | What it measures | What moves it | What to watch |
|---|---|---|---|
| Throughput (tokens/sec) | Total output tokens generated per second across all concurrent requests. The headline number. | Rises with concurrency until the GPU saturates, then plateaus. Rises with smaller models and aggressive quantization. Falls with longer OSL. | The plateau is the server's ceiling on this hardware. |
| TTFT (Time To First Token) | Wall-clock from request arrival to the first output token. Captures prefill and queueing. | Grows with concurrency (queueing) and ISL (prefill work). | Track P99, not P50, for latency SLAs. |
| TPOT (Time Per Output Token) | Average time between output tokens after the first. Captures steady-state generation speed. | Grows mildly with concurrency until saturation. Roughly model-size-bound. | User-perceived tokens/sec per request is about 1 / TPOT. |
| Latency percentiles | P50, P90, and P99 for both TTFT and TPOT. | The tail widens under queueing or jitter. | P99 far above P50 means queueing; raise capacity or lower concurrency. |
| GPU utilization | From telemetry. Fraction of GPU compute cycles in use. | Rises with concurrency on a healthy run. | Below 50% at high concurrency means the workload is CPU-, network-, or memory-bandwidth-bound. Check the Hardware chart tab. |
note
Streaming must be on to measure TTFT. Without it, the tool only sees the final response body.