Performance Methodology
This page defines how Metrum AI Bench CLI measures client-observed performance.
All intervals use std::time::Instant. ISO timestamps are metadata only.
If a value is undefined, it serializes as null or absent - never as a measured zero.
Release performance validation
The versioned performance-validation harness
compares a previous release and a candidate on the same reference GPU and
serving stack. spec.yaml pins the serving image, model revisions, workload,
request parameters, and W1/W2 at concurrency 1, 8, and 32. The runner
interleaves the two binaries at each point (30 s warmup and 180 s measured
issuance). The GPU run is a pre-release gate, not a normal CI test.
From a checkout, prepare the datasets and binaries as documented in the harness README; first inspect the plan without GPU traffic:
bash scripts/validation/perf/runner.sh --plan q38-27b /path/to/baseline-llm /path/to/candidate-llm /path/to/results 1
# After preparing W1.jsonl and W2.jsonl and the pinned serving stack:
export DATASETS_DIR=/path/to/datasets
bash scripts/validation/perf/runner.sh q38-27b /path/to/baseline-llm /path/to/candidate-llm /path/to/results 1
python3 scripts/validation/perf/compare.py --baseline-root /path/to/results/bench-rel --candidate-root /path/to/results/bench-src --model q38-27b --pass 1
Repeat with q38-27b-nvfp4 for the quantized model. The default gate requires
system output tokens/s and TPOT p50 within 5% at each point, or the recorded
per-point noise floor if wider. Missing points fail; TTFT is informational,
not a gate. Use the same dataset bytes, hardware, serving configuration, and
measurement method for each pair. A comparison is not a quality assessment or
an MLPerf submission; TTFT differences require cohort and issuance caveats.
See the harness README for dataset freezing, setup, and limitations.
Latency
| Metric | Definition |
|---|---|
| E2E latency | Response body completion minus actual send. Successful measure-phase requests only. Field: latency_s. |
| Coordinated-omission latency | E2E latency plus delay between scheduled arrival and actual send. Headline open-loop latency. |
| TTFT | First nonempty reasoning or visible output delta minus send. Role, usage and finish frames do not count. A reasoning-only length finish is reasoning_truncated; other completed no-answer streams are no_output_token. Includes connection setup, TLS, and queueing. Non-streaming TTFT is absent unless HTTP first-byte approximation is explicitly requested; it is never fabricated from E2E. Network RTT affects client-observed TTFT. |
| First byte | Response headers received minus send (first_byte_s). Separates gateway/header delay from body wait. |
| Connect | TCP/TLS connector time minus pool reuse (connect_s). 0 means the request reused a pooled connection. Prefill proxy: prefill_s approximates TTFT minus connect when both are present. TTFT still includes connect/TLS/queue by design. |
| First reasoning | First non-empty reasoning_content / reasoning delta minus send; reported separately from TTFT, including completed no-answer streams. |
| ITL | Every successive visible-output chunk timestamp delta, pooled across measured successes. |
| TPOT | (e2e - ttft) / (completion_tokens - 1), defined only for at least two completion tokens. |
Streaming chat/VLM report visible-answer timing separately. The deprecated
--ttft-fallback first-token is redundant; --allow-missing-ttft relaxes the
audit. Explicit --infer-ttft-from-first-byte uses HTTP headers, not a token.
Reasoning truncation is non-failing by default but is not an answer; strict
mode and --ci-safety reject it. Choice index 0 is measured. Current
request/summary output is v5; historical v4 TTFT has a different meaning.
Workload
Measurement definitions are meaningless without a defined workload. This section specifies what is sent; the sections that follow specify how the response is measured.
Prompt source
| Term | Definition |
|---|---|
| Prompt set | A JSONL file, one request per line, produced by metrum-ai-bench-cli-prompts from the Hugging Face metrum-ai/prompt-library dataset. Prompts are real text, not synthetic filler. |
| Revision | Default Hub ref is main (latest); the mix report stores the resolved commit SHA. For publishable runs, pass that SHA (optionally with --require-pinned-revision) so the workload cannot drift. |
| Selection seed | --seed, stamped in the mix report and run summary. The same revision, seed, count, and targets must reproduce the same prompt set. |
| Config | sample or full. sample is a smoke-test subset and must not be used for published results. full is the publishable corpus. |
A single literal prompt repeated across every request is not a workload. It holds input sequence length constant, which holds prefill cost and per-request KV growth constant, and produces a uniform batch the scheduler never has to reconcile. Results obtained this way describe the harness, not the system.
Sequence length
| Term | Definition |
|---|---|
| ISL | Input sequence length in tokens, targeted by --isl-target with --isl-stat (mean or median) and --isl-tolerance. The achieved distribution is reported, not assumed. |
| OSL | Output sequence length in tokens, targeted by --osl-target with --osl-stat and --osl-tolerance. |
| Achieved ISL/OSL | Measured input_tokens / output_tokens across measured requests (and optional mix-report targets). Runtime gates: --isl-target / --osl-target / --prompt-mix-report, mismatch counts on summary, --fail-on-osl-mismatch. |
| max_tokens | Required whenever OSL is targeted on strategic (--max-tokens with --prompts). metrum-ai-bench-cli-prompts emits recommended_max_tokens in its mix report for this purpose. |
Named profiles
Published results should state a profile. Ad-hoc ISL/OSL pairs are permitted but are not comparable to profile-named results unless ISL/OSL and SUT also match.
| Profile | ISL target | OSL target | Represents |
|---|---|---|---|
chat-short | 256 | 64 | Short conversational turns |
chat-medium | 512 | 128 | Longer chat turns |
code-medium | 1024 | 512 | Code completion and edit loops |
rag-medium | 2048 | 256 | Retrieval-augmented generation |
summarize-long | 4096 | 512 | Document summarization |
Profiles are selected with metrum-ai-bench-cli-prompts --profile … and stamped
in the mix report (profile.name / profile.version).
Warmup
| Term | Definition |
|---|---|
| Warmup requests | Issued and fully completed before measurement (--warmup-requests) in llm, vlm, asr, imagegen and every strategic stage. Strategic then resets the measurement epoch; open-loop modality runs shift the measured schedule by the barrier wait and keep the seeded gaps. Excluded from every measured distribution and from the throughput window. On strategic, measured prompt indexing restarts at zero so warmup does not shift the mix. |
| Real endpoints | Use enough warmup to cover model load, CUDA graph capture, and JIT kernel compilation. Prefer a warmup count at least as large as stage concurrency. |
| Dummy only | --warmup-requests 0 is valid for determinism checks against the mock/dummy server, where no weights are loaded. Do not use zero warmup against a real GPU endpoint for publishable numbers. |
| Exact output length | For fixed-length throughput studies, modality and strategic chat paths support --ignore-eos and optional --min-tokens with --max-tokens (engine extensions). Pair with --fail-on-osl-mismatch for publishable gates. Not a default for natural-EOS or reasoning workloads. |
Repetition and significance
Cross-run aggregation and the seeded 10,000-resample percentile-bootstrap 95% confidence interval are defined under Distributions. Prefer multiple independent runs before claiming a difference between systems; a point-estimate delta without overlapping-interval analysis is not a result.
Load generator
The harness is part of the system under test until proven otherwise. A measurement is valid only when the client is not the limiting factor.
| Term | Definition |
|---|---|
| Client headroom | Establish by running a high-concurrency configuration against the mock/dummy server (near-zero service time) and confirming client throughput exceeds the real target by a clear margin. |
| On-host loadgen | For publishable TTFT and concurrency claims, run the load generator on the serving host (loopback). Laptop/WAN loadgen adds RTT into TTFT. |
Throughput and goodput
| Metric | Definition |
|---|---|
| Request throughput | Completed requests divided by the explicit window: first measured send to last measured terminal completion of any outcome, from monotonic send_offset_s plus latency_s. Warmup excluded; drain for requests issued during measurement included. |
| Throughput bins | Fixed-width bins over send offsets (open-loop: scheduled_offset_s; closed-loop: send_offset_s), measured from the first measured request's offset. Each bin is divided by its actual width so a trailing partial bin is not under-normalized. |
| Token throughput | Server-reported completion tokens, including reasoning, divided by the same window. Incomplete usage makes the headline unavailable; local tokenizer estimates do not fill the gap. |
| Error rate | Measured failures divided by measured attempts. |
| Goodput | Measured successes satisfying all configured SLOs, excluding explicitly unanswered rows, divided by the window. Unmarked legacy/ASR/imagegen rows are treated as answered for goodput. |
Effective max concurrency is stamped on summary.v5.config as
effective_max_concurrency - --max-concurrency when set, otherwise the
closed-loop --concurrency value that caps outstanding work. Observed
concurrency (in-flight mean/p50/max, cap-engagement) is reported separately on
modality and strategic summaries.
Distributions
Distributions report n, min, max, arithmetic mean, sample standard deviation,
median absolute deviation, and Hyndman-Fan type 7 p50/p90/p95/p99.
P99 is marked unreliable when fewer than 100 samples exist. Small-n p95/p90 also carry unreliability flags.
Multi-endpoint aggregate distributions are labeled pooled_mixture; the same
full distributions are emitted independently per endpoint.
Cross-run aggregation uses sample dispersion and a seeded 10,000-resample percentile-bootstrap 95% confidence interval.
ASR quality and RTFx
| Metric | Definition |
|---|---|
| WER / CER | Edit distance after normalization, divided by the normalized reference word/character count. --normalizer selects whisper-english (default), whisper-basic, or none. |
| RTFx (client) | Audio duration divided by client request duration (modality_metrics.rtfx_client). Sole RTFx definition. |
Imagegen and VLM payload notes
- Imagegen latency: time until response body bytes are fully read. Decode, hash, and artifact writes happen after the timer stops.
- VLM image payload: source bytes are sent unchanged unless resize /
re-encode flags apply (
--max-image-dimension,--reencode-jpeg).modality_metrics.image_bytesalways matches what the server received.
Console vs JSONL
Printed end-of-run statistics come from the same RunSummary / DistSummary
values written as summary.v5. There is no separate nearest-rank console
estimator.