Performance Methodology

Metrum Insights uses four first-party benchmarking tools to measure AI model serving performance across different modalities. Each tool targets a specific workload type, sends controlled traffic to an OpenAI-compatible endpoint, and produces structured output with timing, throughput, and accuracy metrics.

Tool	Modality	Protocol	Primary Use
metrumbench-llm	LLM text generation	JSON POST to /v1/chat/completions	Load testing language models with configurable concurrency, streaming, and prompt control
metrumbench-vlm	VLM image + text	JSON POST to /v1/chat/completions with image payloads	Benchmarking vision-language models with image inputs
metrumbench-asr	ASR audio transcription	Multipart POST to /v1/audio/transcriptions	Benchmarking speech-to-text models with accuracy measurement
metrumbench-imagegen	Image generation	JSON POST to /v1/images/generations	Benchmarking image generation models with artifact collection and integrity verification

All four tools share a common foundation: the same Rust crate, the same HTTP client library, and the same concurrency control approach. metrumbench-llm, metrumbench-vlm, and metrumbench-asr also share the same NTP time-sync preflight and endpoint resolution system. This means measurement methodology is consistent across modalities.

---

Platform Architecture

The benchmark tools do not run in isolation. They are the measurement layer of a larger system that decides what to run, provisions where to run it, executes the work, captures the results, and makes them queryable. Understanding this split explains where every number in a report comes from.

The two halves: control plane and workers

Metrum Insights separates the control plane from the workers.

• The control plane is the system of record. It holds every project, workload, scenario, run, job, result, telemetry sample, and hardware record, and it serves the product UI and the public API.
• A worker is a GPU host that serves models and generates load. An agent runs on each worker: it receives jobs, starts the model server with the resolved framework command, runs the benchmark tool against it, samples hardware telemetry during the run, and uploads results back to the control plane.

A worker never writes to the control plane's database directly. It sends every result through the public API, which enforces authentication and access control on every request. That keeps the control plane the only writer of record and keeps each tenant's data isolated.

The natural-key contract

Every external interface uses natural keys, never internal serial IDs. A workload is addressed by its workload_code, a model by its model_code, a framework by its framework_code and version string, a server by its config code and hostname. Internal numeric IDs exist in the database but never appear in an API contract, a test, or a benchmark report. This is what makes a benchmark reproducible: the exact same codes reproduce the exact same run on any deployment.

The traceability chain

Results are only trustworthy if you can prove what produced them. Every result links back through an unbroken chain of records:

project -> workload -> scenario -> run -> job -> tool_invocation -> output metrics
                                          |
                                          +-> telemetry samples (per GPU, per CPU, system)
                                          +-> logs and SUT (system-under-test) evidence

• The workload pins the model, framework, version, quantization, and engine args set.
• The scenario pins the traffic shape (concurrency, ISL, OSL, request count).
• The job pins the concrete server instance it ran on.
• The tool invocation records the exact resolved command line the benchmark tool executed.
• The output rows hold the typed metrics.
• Telemetry samples captured during the job hold GPU power, utilization, memory, and CPU and system metrics.

Because each layer references the one above it, any number on the Reporting page can be traced to the command, the hardware, and the configuration that produced it.

End-to-end flow of a single benchmark

1. Define. A user creates a project, a workload, and one or more scenarios, either in the Web UI or over the API. Nothing runs yet.
2. Select or provision hardware. The workload targets a registered on-prem server or a cloud SKU. For cloud, the platform provisions the instance on demand.
3. Check readiness before the first job. The platform checks that the model fits the GPU memory at the chosen precision, that the framework version supports the hardware, that required tokens are present, and that the server is online. A failure here stops the run with a specific, actionable error rather than a late crash.
4. Resolve commands. For each job, the control plane resolves the framework command template (how to start the model server) and the tool command template (how to run the benchmark client), substituting run settings and engine args into named placeholders. An unresolved placeholder is treated as a defect, not a warning.
5. Serve and load. The agent starts the model server, waits for readiness, then runs the benchmark tool against it at the scenario's concurrency.
6. Sample telemetry. Throughout the job, the agent samples hardware metrics and tags each sample with a UTC timestamp inside the job's time window.
7. Ingest results. The tool writes a JSON summary line. The agent reads it, maps it to typed columns, and inserts it through a dedicated ingest RPC.
8. Check the results after the last job. The platform flags suspect data (all-zero latencies, implausible token counts, missing telemetry) without discarding it.
9. Report. SQL views join outputs with job, workload, scenario, hardware, and telemetry context, so the UI and API can present and compare results.

The full operational version of this flow, with the decision framework that should precede it, is in Benchmark Lifecycle.

---

Shared Foundations

Clock Synchronization

Before any benchmark run, each tool contacts public NTP servers (pool.ntp.org, time.google.com, time.cloudflare.com, time.apple.com) and verifies the system clock is within one second of NTP time. This prevents skewed timestamps in results. The check can be skipped for air-gapped environments by setting METRUM_SKIP_NTP_CHECK=true.

HTTP Client

All tools use the same HTTP client configured with:

• Request timeout: maximum wall-clock time for a single request (default 120 seconds for metrumbench-llm/metrumbench-vlm, 120 seconds for metrumbench-asr CLI, 300 seconds when run through Metrum jobs).
• Connect timeout: time allowed to establish the TCP connection (default 30 seconds).
• Connection pool: idle connections are kept open and reused, sized to match the concurrency level. Idle timeout and TCP keepalive are configurable.

Endpoint Resolution

Each tool supports two endpoint modes:

• Single endpoint: one URL and one API key.
• Multi-endpoint: a YAML file listing multiple endpoints with optional weights. Requests are distributed in weighted round-robin order. This is useful for testing load balancers or comparing endpoints side by side.

Concurrency Control

All tools use an async semaphore to limit in-flight requests. The semaphore is sized to the --concurrency value. Each request acquires a permit before sending and releases it on completion. This ensures the target concurrency is never exceeded, regardless of response time variation.

Output Format

Each tool writes one JSON summary line to a JSONL file per benchmark run. This summary contains the full configuration, all computed metrics, error details, and per-endpoint breakdowns. The summary is always written, even when errors occur, so partial results are never lost.

Every summary includes:

• A UUID and a human-readable ID for easy reference.
• An ISO 8601 UTC timestamp.
• The tool version and Rust compiler version used to build the binary.
• The full configuration echoed back for reproducibility.

---

Model-Server Replicas and Tensor Parallelism

When a model server's tensor-parallel footprint is smaller than the GPUs on the host, Metrum runs multiple replicas so the whole machine gets used instead of leaving GPUs idle, and spreads the scenario's traffic across every replica endpoint. See Model-server replicas in the User Guide for how replica count is derived.

---

metrumbench-llm: LLM Performance

metrumbench-llm measures how fast a language model endpoint processes text generation requests under controlled load.

How Requests Work

metrumbench-llm sends JSON POST requests to an OpenAI-compatible chat completions or legacy completions endpoint. Each request includes a prompt from a JSONL prompt file, the target model name, a maximum token limit, and a sampling temperature.

In chat mode, prompts are wrapped in a messages array with a system message ("You are a helpful assistant.") and the user prompt. In completion mode, the prompt is sent directly.

Streaming mode uses Server-Sent Events (SSE). The tool processes the byte stream chunk by chunk, extracting content tokens as they arrive. When streaming is enabled, the tool also requests token usage statistics in the stream.

Non-streaming mode waits for the complete response body before parsing.

Prompt Handling

Prompts are loaded from a JSONL file (local path or HTTP URL). Each line must contain a JSON object with a prompt field. Before the run, all prompts are shuffled randomly for even distribution, then cycled deterministically through the request sequence. If there are fewer prompts than requests, prompts are reused in order.

When running through the Metrum platform, prompts are sampled from a prompt library based on target input and output token lengths. The platform can also run a calibration step to determine the token-to-word ratio for the specific model being tested.

What Gets Measured

Per request:

Metric	What it is	How it is captured
Response Time	Total wall-clock time from request start to completion	Instant::now() before the HTTP call, measured at stream end or body receipt
Time to First Token (TTFT)	Time from request start to the first output content chunk	Streaming: recorded when the first non-empty content chunk arrives. Non-streaming: equals response time
Time Per Output Token (TPOT)	Average time to generate each output token after the first	(Response Time - TTFT) / completion_tokens
Prompt Tokens	Number of input tokens	Read from the server's usage response
Completion Tokens	Number of output tokens generated	Read from the server's usage response
Prompt Words	Word count in the input prompt	Whitespace split count
Completion Words	Word count in the generated text	Whitespace split count

Key point: metrumbench-llm does not tokenize locally. Token counts come entirely from the model server's usage response. Word counts are computed locally as a tokenizer-independent fallback metric.

Aggregate statistics computed across all successful requests:

• Response times: min, max, average, p50, p90, p95, p99
• TTFT: min, max, average, p50, p90, p95, p99
• TPOT: min, max, average, p50, p90, p95, p99
• Token counts: total, min, max, average, p50, p95, tokens per second
• Word counts: total, min, max, average, words per second
• Throughput: requests per second, prompt tokens per second, completion tokens per second, total tokens per second
• Errors: count, error rate percentage, error type breakdown, OOM detection

Ramp-up Mode

metrumbench-llm supports a ramp-up period where concurrency increases linearly from one to the target value over a configurable number of seconds. During this period:

• Requests are still executed (load is applied to the server).
• Metrics from ramp-up requests are discarded.
• Only requests completing after the ramp-up period contribute to the final statistics.

This produces steady-state metrics that are not distorted by cold-start behavior or initial connection establishment.

Error Classification

metrumbench-llm classifies errors by HTTP status code:

Status	Classification
429	Rate limit
401, 403	Authentication
400	Bad request
500-599	Server error

Stream errors are classified by failure type: timeout, connection failure, body processing error, decoding error, or redirect error. The tool also detects out-of-memory (OOM) conditions from error messages.

Multi-endpoint Support

When using multiple endpoints, metrumbench-llm tracks per-endpoint metrics separately: response times, TTFT, TPOT, token counts, errors, and request rates. The JSONL output includes both aggregate and per-endpoint breakdowns.

---

metrumbench-vlm: VLM Performance

metrumbench-vlm measures vision-language model performance by sending prompts with one or more images to an OpenAI-compatible chat completions endpoint.

How Requests Work

metrumbench-vlm sends the same JSON POST requests as metrumbench-llm, but with image content in the messages array. Images can be included in two ways:

• Base64 encoding (default): images are loaded, optionally resized, and encoded directly into the request payload.
• Server-side download: image URLs are included in the request, and the model server downloads them.

Each prompt line in the input JSONL must have a prompt field and an image_urls array containing one or more image references (HTTP URLs, local file paths, or file:// URIs).

Image Processing

metrumbench-vlm includes built-in image handling:

• Resizing: large images can be automatically resized to a maximum dimension while preserving aspect ratio (--max-image-dimension).
• Caching: an LRU cache stores processed images to avoid redundant loading and encoding.
• Detail levels: images can be sent at low or high detail level, which affects token usage and processing time.
• Multiple images per prompt: a single request can include multiple images for multi-image reasoning tasks.

What Gets Measured

metrumbench-vlm collects the same core metrics as metrumbench-llm:

• Response times with percentile distributions
• TTFT and TPOT for streaming requests
• Token counts (prompt, completion, total) from the server
• Word counts for prompts and completions
• Throughput rates (tokens per second, requests per second)
• Error counts and classifications
• Per-endpoint breakdowns for multi-endpoint runs

Additional metrumbench-vlm-specific data in the output:

• Image processing times
• Image sizes and dimensions
• Per-request image metadata

Shared Methodology

metrumbench-vlm shares the same concurrency model, timing approach, percentile calculations, ramp-up support, error handling, and output format as metrumbench-llm. The only difference is the payload shape (images in the messages array) and the input JSONL format.

---

metrumbench-asr: ASR Performance

metrumbench-asr measures audio transcription model performance by uploading audio files to an OpenAI-compatible transcription endpoint and evaluating both speed and accuracy.

ASR is scoped to a single model family and a single framework: the Whisper family (for example openai/whisper-large-v3) served on vLLM. vLLM exposes the Whisper transcription endpoint that metrumbench-asr uploads audio to. SGLang and TensorRT-LLM are not productized for ASR, so vLLM is the only supported framework.

How Requests Work

metrumbench-asr sends multipart form POST requests to an audio transcription endpoint. Each request includes:

• The audio file binary content
• The model name
• The requested response format (verbose_json, json, text, srt, or vtt)
• An optional language code (default: English)
• Timestamp granularity request (word level for JSON formats)

Audio files can be provided as local paths or URLs. Remote files are downloaded and cached locally before the benchmark starts.

Supported Audio Formats

Format	MIME Type
MP3	audio/mpeg
WAV	audio/wav
WebM	audio/webm
OGG	audio/ogg
M4A / MP4	audio/mp4
FLAC	audio/flac

What Gets Measured

Per request:

Metric	What it is	How it is captured
Response Time	Total wall-clock time including file upload and download	Measured from before file read to after response receipt
Inference Time	Server-side processing time	From the inference_time field in JSON responses, or wall-clock time as fallback
RTF (Real-Time Factor)	Ratio of inference time to audio duration	inference_time / audio_duration. Values below 1.0 mean faster than real-time
Words Per Second	Transcription throughput by word count	word_count / inference_time
Characters Per Second	Transcription throughput by character count	char_count / inference_time
Bytes Sent	Size of audio file uploaded	File size in bytes
Bytes Received	Size of transcription response	Response body size in bytes

Aggregate statistics:

• Response times: min, max, average, p50, p90, p95, p99
• Inference times: min, max, average, p50, p90, p95, p99
• RTF: min, max, average, p50, p90, p95, p99
• Throughput: audio seconds processed per wall-clock second, characters per second, words per second, requests per second, data transfer rate
• Data transfer: total bytes sent, received, and transferred
• Audio format breakdown: count of files processed by format
• Errors: count, error rate percentage

Accuracy Metrics

metrumbench-asr computes accuracy metrics when a ground truth transcript file is provided. Ground truth is a JSONL file where each line has an id field matching an audio sample and a transcript field with the reference text.

Word Error Rate (WER):

WER measures how many word-level edits are needed to transform the transcription into the reference text. It uses the Levenshtein edit distance algorithm at the word level with case-insensitive comparison.

• A WER of 0% means perfect transcription.
• A WER of 100% means every word is wrong.
• WER can exceed 100% if the transcription has more words than the reference (many insertions).

The formula: WER = (substitutions + insertions + deletions) / reference word count

Character Error Rate (CER):

CER applies the same Levenshtein algorithm at the character level. It is more granular than WER and useful for evaluating partial-word accuracy and languages where word boundaries are less clear.

Aggregate accuracy statistics: min, max, average, p50, p90, p95, p99 for both WER and CER across all samples with ground truth.

Input Format

The audio sample manifest is a JSONL file where each line describes one audio file:

{"id": "sample-001", "path": "/path/to/audio.flac", "format": "flac", "duration": 12.5}
{"id": "sample-002", "url": "https://storage.example.com/audio.mp3", "format": "mp3", "duration": 8.3}

• id is required and must be unique.
• Either path (local file) or url (remote download) must be present.
• format determines the MIME type for upload.
• duration is needed for RTF calculation. Without it, RTF is skipped.

Audio samples are cycled round-robin when the request count exceeds the sample count.

---

metrumbench-imagegen: Image Generation Performance

metrumbench-imagegen measures image generation model performance by sending prompts to an OpenAI-compatible image generation endpoint, collecting generated image artifacts, and tracking generation latency and throughput.

Supported frameworks

Image generation is served by:

Framework	Role	Latest CUDA 13.0 (default)	Latest AMD / ROCm
vLLM-Omni (vllm-omni)	Omni image-generation path (vllm serve … --omni). Separate from plain text vLLM.	0.24.0	0.22.0 (amd-rocm)
SGLang (sglang)	Diffusion image-generation path (sglang serve / serve-diffusion).	0.5.15.post1	0.5.15.post1 (amd-mi300 / amd-mi355)

ImageGen engine args are tool-scoped and do not reuse LLM engine-args sets.

Default ImageGen engine-args sets (single-GPU-safe):

Framework	Default set	Defaults
vLLM-Omni	vllm-omni-metrumbench-imagegen	tensor_parallel_size=1, usp=1, ring=1
SGLang	sglang-metrumbench-imagegen	tp_size=1, ulysses_degree=1, ring_degree=1

• TP (tensor_parallel_size / tp_size) — shards model weights across GPUs. Default 1.
• USP / Ulysses (usp / ulysses_degree) — sequence parallelism for diffusion. Default 1 (off).
• Ring (ring / ring_degree) — ring sequence parallelism for diffusion. Default 1 (off).

Raise TP / USP / Ring only for multi-GPU or high-resolution workloads.

How Requests Work

metrumbench-imagegen sends JSON POST requests to an /images/generations endpoint. Each request includes:

• The text prompt describing the desired image
• The model name
• The number of images to generate per request (n)
• The target image size (for example 1024x1024)
• The response format (b64_json for base64-encoded images, or url for download links)
• Optional parameters: negative prompt, inference steps, guidance scale, and custom body fields

Load shape (same idea as metrumbench-llm / VLM / ASR):

Control	Meaning	Default when unset
Concurrency	Parallel in-flight image requests	1
Number of requests	Total requests in the job (platform-derived; not a user scenario field)	max(4, concurrency × 4)
Images per request (n)	Images asked for in each API call	1
Image size	Target resolution	1024x1024

Request count is derived by the platform from concurrency when the scenario does not set it: max(4, concurrency × 4). At concurrency 1 that is 4 requests; at concurrency 8 that is 32 requests (about 4 rounds per concurrent slot). Users configure concurrency (and n / size); they do not configure request count in the ImageGen scenario UI.

Images per request (n) is set by the user (scenario num_images_batch):

• vLLM-Omni: n > 1 is supported.
• SGLang: depends on the model pipeline. When n > 1, each diffusion pipeline must broadcast its text conditioning to the expanded sample batch. SGLang's shared denoising stage does not do this automatically:
  • Pipelines that broadcast correctly (for example SD 3.5 Large Turbo, FLUX.1-dev) work with n > 1.
  • Pipelines that do not (for example Qwen-Image, Z-Image-Turbo) fail at n > 1 with a conditioning / tensor-shape mismatch.
  • Prefer n = 1 on SGLang unless the specific model is known to support multi-image requests.

Prompts come from the ImageGen prompt snapshot (resolved to JSONL). Each line can include per-prompt overrides for size and negative prompt.

Endpoint and Load Balancing

metrumbench-imagegen supports multiple endpoint modes:

• Single endpoint: one --url with an --api-key.
• Repeated endpoints: multiple --endpoint flags for simple multi-endpoint runs.
• Endpoints file: a JSON or JSONL file listing endpoints with optional names and weights.

Four load balancing strategies are available:

• Round-robin (default): requests cycle through endpoints in order.
• Weighted round-robin: endpoints with higher weights receive proportionally more requests.
• Least in-flight: each request goes to the endpoint with the fewest active requests at that moment.
• Random: each request goes to a randomly chosen endpoint.

metrumbench-imagegen also supports optional endpoint health checks before the benchmark starts, and configurable retry attempts with backoff for failed requests.

Artifact Collection

When using the b64_json response format, metrumbench-imagegen decodes and saves each generated image to a local artifact directory. For every saved image, it records:

• File path and index within the request
• SHA-256 hash of the image bytes for integrity verification
• File size in bytes
• Image dimensions (width and height)
• MIME type

Raw API responses can also be saved as JSON files for debugging or analysis.

What Gets Measured

Per request:

Metric	What it is	How it is captured
Latency	Total wall-clock time from request start to completion	Measured as the difference between UTC start and completion timestamps
Images Requested	Number of images asked for (n parameter)	From the request configuration
Images Returned	Number of images in the API response	Count of items in the response data array
Response Bytes	Size of the full API response	Byte length of the response body
Image Artifacts	Decoded image details	SHA-256, dimensions, file size per saved image

Aggregate statistics:

• Latency: min, mean, p50, p90, p95, p99, max (in milliseconds)
• Image byte sizes: min, mean, p50, p90, p95, p99, max
• Throughput: images per second, requests per second
• Megapixels per second: resolution-normalised throughput (sum of each image's actual decoded width × height / 1e6, divided by run duration)
• Success rate: fraction of requests that completed without error
• Image counts: total images requested vs total images generated
• Per-endpoint breakdown: request count, success/failure counts, images generated, images per second, requests per second, and latency percentiles for each endpoint

Error Handling

metrumbench-imagegen classifies errors into these types:

Error Type	Meaning
timeout	Request exceeded the configured timeout
connection_error	TCP connection or response body read failure
http_error	Server returned a non-success HTTP status
schema_error	Response body could not be parsed or was missing expected fields
decode_error	Base64 image data could not be decoded
artifact_error	Image file could not be saved to disk
request_error	Other request-level failure

The summary groups errors by type and HTTP status code so you can quickly see whether failures are server-side, network-related, or format mismatches.

Output Format

metrumbench-imagegen produces two output files:

• Data log (JSONL): one line per request with full request details, latency, status, image artifacts, attempt history, and error information.
• Summary (JSON): a single file with aggregate metrics, per-endpoint breakdowns, error summaries, and artifact paths.

Each per-request record in the data log follows the metrumbench-imagegen.request.v1 schema. The summary follows the metrumbench-imagegen.summary.v1 schema.

---

Datasets And Prompt Sources

A benchmark is only as meaningful as the data it sends. Metrum Insights models every input source as a prompt source that a workload references, so the exact data used by a run is recorded and reproducible. The three workload types draw on different kinds of source.

LLM prompt libraries

metrumbench-llm draws prompts from a prompt library: a named collection of text prompts. When a scenario runs, the platform samples prompts whose length matches the scenario's target input and output token lengths, so a isl512 run actually sends roughly 512-token prompts. The platform can run a calibration step to learn the token-to-word ratio for the specific model being tested, which makes the length targeting accurate across tokenizers.

Prompt libraries are discoverable through v_prompt_libraries_with_prompt_count. Choose the library by the question you are asking:

• random token-like prompts for raw capacity sweeps;
• a fixed prompt library for run-to-run reproducibility;
• domain prompts (medical, legal, code) for application-like latency;
• long-context prompts to stress memory and the scheduler.

To bring your own, upload a JSONL with one prompt per line and reference it from the workload by its library code. An optional expected_output_tokens column overrides the OSL per prompt.

VLM image snapshots

metrumbench-vlm draws from an image snapshot. The default is metrumbench-vlm-4k-v1, a set of photographs paired with description prompts. Each prompt row carries a prompt and an image reference. The image bytes themselves are stored centrally (Restic) and loaded onto the worker separately from the metadata rows, so the database stays lean while the worker still gets the real pixels.

Each request can carry one or more images (num_images_batch) at low or high detail (image_detail). To bring your own, supply a JSONL where each line has a prompt and an image_urls array of HTTP URLs, local paths, or file:// URIs.

ASR audio snapshots

metrumbench-asr draws from an audio snapshot. Two are seeded by default:

Snapshot code	Source	Character
librispeech-test-clean	OpenSLR LibriSpeech test-clean	Clear, read English speech. The standard baseline.
librispeech-test-other	OpenSLR LibriSpeech test-other	Noisier, harder speech for stress-testing accuracy.

The audio files are stored in Restic and restored to the worker during agent bootstrap. The snapshot rows in the database are metadata only, which lets a workload auto-derive its --input manifest path from the workload's prompt source. Both LibriSpeech sets ship with reference transcripts, so Word Error Rate and Character Error Rate are available without extra setup. To bring your own audio, supply an input manifest (one audio file per JSONL line) and, for accuracy, a ground-truth transcript file.

ImageGen prompt snapshots

metrumbench-imagegen draws from an image-generation prompt snapshot (prompt_source type metrumbench_imagegen_snapshot), not an LLM prompt library. Each row is a text prompt (and optional per-prompt fields such as negative prompt). The platform resolves the snapshot into a JSONL prompts file for the tool.

Snapshot	Prompts	Role
text2image-multiprompt-1024	1024	Dataset used for ImageGen (manifest-pinned product set)

Why prompt sources are recorded, not assumed

Because the workload references a specific prompt source by code, a report can prove exactly which data produced a result. Two runs that cite the same prompt library or snapshot code sent comparable inputs. This is the data-side counterpart to the natural-key contract described in the architecture section.

---

Engine Arguments And Verified Recipes

The single largest cause of a failed or misleading serving benchmark is a bad set of framework startup flags. Metrum Insights treats these engine arguments as first-class, versioned, reproducible data rather than something typed on a command line for one run.

The data model

Engine arguments are stored in three layers, all keyed to a specific framework version:

• framework_engine_args_templates defines which flags exist for a framework version, their key, type, and default (for example tensor_parallel_size, max_model_len, kv_cache_dtype).
• framework_engine_args_sets is a named preset (a set_code) that a workload points at.
• framework_engine_args_values holds the concrete value each set assigns to each flag.

A workload references one set by code. When a job runs, those values are substituted into the framework command template to produce the exact resolved command. There is no hidden state and no manual editing, so the same set code reproduces the same launch command every time.

Verified recipes, sourced from upstream

Metrum Insights seeds verified recipe argsets whose values are copied directly from the two authoritative upstream sources for serving configuration:

• vLLM recipes: https://recipes.vllm.ai/
• SGLang cookbook: https://lmsysorg.mintlify.app/cookbook/intro

Each seeded set is keyed to a specific model, GPU class, framework version, and parallelism, and the set code says so. For example vllm-llama33-70b-instruct-h100-h200-tp8 is the Llama 3.3 70B Instruct recipe for H100 and H200 at tensor-parallel size 8, and vllm-llama33-70b-instruct-b200-gb200-tp1-fused is the same model on B200 and GB200 with fused compilation. Verified argsets are seeded for the Llama, DeepSeek, Qwen, Gemma, GLM, GPT-OSS, Kimi, Mistral, and Nemotron families, across vLLM and SGLang, and are reconciled against the upstream pages when versions change.

Why this matters for measurement

Sourcing engine arguments from verified recipes removes three failure modes that otherwise corrupt results:

• a launch that fails outright because a flag is incompatible with the model or the GPU count;
• an out-of-memory partway through a run because memory or batch limits were set too aggressively;
• a silent correctness problem, such as a context length that cannot hold the input plus output, so requests are truncated and the throughput number describes work the model never did.

The user-facing procedure for selecting arguments, including how to read a recipe page and how to sanity-check a set before launch, is in the User Guide section Choosing engine arguments. Worked command examples for FP8 KV cache, long context, CPU offload, block-size sweeps, and MoE configurations are in Command Templates.

---

Percentile Calculations

All four tools compute percentile statistics using the same approach:

1. Sort all values in ascending order.
2. Calculate the index as a function of the desired percentile and array length.
3. Return the value at that index.

Standard percentiles reported: p50 (median), p90, p95, and p99. These provide a clear picture of typical performance (p50), tail latency (p95/p99), and overall distribution shape.

---

How Results Reach the Database

Benchmark results flow through a consistent pipeline regardless of which tool produced them:

1. Tool writes JSONL: the benchmark tool appends one summary JSON line to its data log file.
2. Agent reads summary: the Metrum agent daemon reads the last line of the JSONL file after the tool exits.
3. Agent uploads results: the agent maps the summary fields to the tool-specific output records and inserts them through the platform API.
4. Views make results queryable: SQL views join the raw output tables with job, workload, scenario, and hardware information for reporting.

Each tool has dedicated database tables:

Tool	Invocation Table	Output Table	Primary View
metrumbench-llm	metrumbench_llm_invocations	metrumbench_llm_outputs	v_metrumbench_llm_outputs
metrumbench-vlm	metrumbench_vlm_invocations	metrumbench_vlm_outputs	v_metrumbench_vlm_outputs
metrumbench-asr	metrumbench_asr_invocations	metrumbench_asr_outputs	v_metrumbench_asr_outputs
metrumbench-imagegen	tool_invocations	metrumbench_imagegen_outputs	v_metrumbench_imagegen_outputs

The invocation table stores the CLI configuration used for the run. The output table stores all computed metrics in typed columns for efficient querying. ImageGen also stores per-endpoint rows, image artifact metadata, and error counts in dedicated tables joined through the same invocation.

---

System Metrics (Hardware Telemetry)

In addition to benchmark tool metrics, Metrum Insights collects hardware telemetry during benchmark runs. These system-level metrics show what the hardware was doing while the benchmark tool was generating load, providing the full picture of serving performance and resource utilization.

What Gets Collected

Telemetry is sampled as a time series throughout the benchmark run. Each sample captures:

GPU Metrics (per GPU):

Metric	Unit	Description
GPU Power	Watts	Power consumption of each GPU during the run
GPU Memory Used	Bytes	GPU memory actively in use at sample time
GPU Memory Total	Bytes	Total GPU memory capacity
GPU Utilization	0.0 to 1.0	Fraction of GPU compute cycles actively used

On multi-GPU servers, each GPU is tracked independently using a GPU index (0, 1, 2, ...). This lets you see whether load is evenly distributed across GPUs or if one GPU is a bottleneck.

CPU Metrics (per CPU socket):

Metric	Unit	Description
CPU Utilization	0.0 to 1.0	Fraction of CPU cycles actively used

On multi-socket systems, each CPU socket is tracked independently.

System-wide Metrics:

Metric	Unit	Description
System RAM Used	Bytes	Host memory currently in use
System RAM Total	Bytes	Total host memory capacity
System Power	Watts	Total system power draw (when available)

TPU Metrics (for Google TPU workloads):

Metric	Unit	Description
Duty Cycle (avg)	Percentage	Average TPU duty cycle across the sample window
Duty Cycle (peak)	Percentage	Peak TPU duty cycle observed during the run
Tensor Core Utilization (avg)	Percentage	Average tensor core usage across the sample window
Tensor Core Utilization (peak)	Percentage	Peak tensor core usage observed during the run
HBM Capacity Used	Bytes	High-bandwidth memory in use at time of sample
HBM Capacity Total	Bytes	Total HBM available on the TPU chip

How Telemetry is Collected

The Metrum agent on the benchmark worker samples hardware metrics at regular intervals during the run. Collection is tied to the job lifecycle:

• Samples are only accepted within the run's time window (between run start and run completion).
• Each sample carries a UTC timestamp for time series analysis.
• Samples are rejected if they fall outside the run period, ensuring data integrity.

The agent uses NVML (NVIDIA Management Library) for GPU metrics on NVIDIA hardware, and standard OS interfaces for CPU and system metrics. For TPU workloads, telemetry comes from LibTPU (the TPU runtime library), which exposes per-chip utilization, HBM capacity, and tensor core activity directly from the TPU hardware counters. The agent queries the LibTPU metrics endpoint on each TPU worker node and forwards the samples to the control plane.

Aggregated Views

Raw telemetry samples are aggregated into summary views for easy consumption:

v_telemetry_job_summary - one row per job with:

• Sample count and collection duration
• GPU count detected
• Average and max GPU power (watts)
• Average GPU utilization
• Average and max GPU memory percentage
• Average CPU utilization
• Average system RAM percentage

v_telemetry_per_gpu_summary - one row per GPU per job for multi-GPU analysis:

• Average, max, and min GPU power per GPU
• Average and max GPU utilization per GPU
• Average and max GPU memory usage per GPU

v_job_with_telemetry - benchmark jobs joined with their telemetry summaries, giving a single view of model, framework, status, and resource utilization.

Why System Metrics Matter

System metrics answer questions that benchmark tool metrics cannot:

Question	Tool Metrics Alone	With System Metrics
Is the GPU fully utilized?	No visibility	GPU utilization shows headroom or saturation
Why did throughput drop?	Error rate may increase	GPU memory pressure or thermal throttling visible in power/memory trends
Is my model too large for this GPU?	OOM errors in tool output	GPU memory usage as percentage shows how close you are to the limit
Is the CPU a bottleneck?	Not visible	CPU utilization spikes during tokenization or data loading
How much power does serving cost?	Not visible	GPU and system power draw in watts, useful for cost-per-token analysis
Is load balanced across GPUs?	Not visible	Per-GPU utilization and power show imbalances

Availability

System metrics are available for managed workloads where the Metrum agent runs on the same host as the model server. When the model server runs on external infrastructure that the agent is not on, GPU and system metrics are typically not available because the agent does not have access to the endpoint's hardware. In those cases, attach external monitoring data or note the gap in your benchmark report.

---

Metric Definitions Reference

Latency Metrics

Metric	Applies To	Definition
Response Time	All tools	Total wall-clock time from request start to response completion
TTFT (Time to First Token)	metrumbench-llm, metrumbench-vlm	Time from request start to first output content in the SSE stream. Equals response time for non-streaming requests
TPOT (Time Per Output Token)	metrumbench-llm, metrumbench-vlm	Average generation time per token after the first: (Response Time - TTFT) / completion_tokens
Inference Time	metrumbench-asr	Server-reported processing time. Falls back to response time when not reported
Latency	metrumbench-imagegen	Wall-clock time from request start to image generation completion

Throughput Metrics

Metric	Applies To	Definition
Tokens Per Second	metrumbench-llm, metrumbench-vlm	Total tokens processed divided by elapsed time (prompt, completion, and total reported separately)
Words Per Second	metrumbench-llm, metrumbench-vlm, metrumbench-asr	Total words divided by elapsed time
Requests Per Second	All tools	Total completed requests divided by elapsed time
Images Per Second	metrumbench-imagegen	Total images generated divided by elapsed time
Megapixels Per Second	metrumbench-imagegen	Total decoded megapixels (width × height / 1e6) divided by elapsed time. Fairer than images/sec when comparing different resolutions
Audio Per Second	metrumbench-asr	Total audio duration processed divided by wall-clock time. Values above 1.0 mean processing is faster than real-time
Characters Per Second	metrumbench-asr	Total transcribed characters divided by elapsed time
Data Transfer Rate	metrumbench-asr	Total bytes (sent + received) divided by elapsed time

Accuracy and Quality Metrics

Metric	Applies To	Definition
WER (Word Error Rate)	metrumbench-asr	Levenshtein word-level edit distance divided by reference word count. 0% is perfect
CER (Character Error Rate)	metrumbench-asr	Levenshtein character-level edit distance divided by reference character count. 0% is perfect
RTF (Real-Time Factor)	metrumbench-asr	Inference time divided by audio duration. Below 1.0 means faster than real-time
Image SHA-256	metrumbench-imagegen	Per-image cryptographic hash for artifact integrity and reproducibility verification
Success Rate	metrumbench-imagegen	Fraction of requests that completed without error

Error Metrics

Metric	Applies To	Definition
Error Count	All tools	Number of failed requests
Error Rate	All tools	Failed requests as a percentage of total requests
OOM Detected	metrumbench-llm	Whether any error message indicates an out-of-memory condition
Timeout Count	metrumbench-imagegen	Number of requests that exceeded the configured timeout

---

When to Use Each Tool

Question	Tool	Why
How fast does my LLM serve text under load?	metrumbench-llm	Measures token throughput, TTFT, TPOT, and latency percentiles with streaming support
How does my VLM handle image inputs?	metrumbench-vlm	Sends images with prompts and measures the same LLM metrics plus image processing overhead
How fast and accurate is my speech-to-text model?	metrumbench-asr	Measures transcription latency, real-time factor, and accuracy against reference transcripts
How fast does my image generation model produce images?	metrumbench-imagegen	Measures generation latency, images per second, and collects artifacts with integrity hashes
Do different benchmark tools agree on performance?	All four (plus third-party tools)	Run the same scenario with multiple tools and compare
What is the steady-state performance after warm-up?	metrumbench-llm	Use --ramp-up-seconds to discard warm-up metrics
How does my endpoint handle increasing load?	metrumbench-llm	Use ramp-up with increasing concurrency to find saturation points
Is my transcription accurate enough for production?	metrumbench-asr	Compare WER/CER against ground truth transcripts
Are my generated images reproducible across runs?	metrumbench-imagegen	Compare SHA-256 hashes of artifacts across runs under the same prompt snapshot
How does my image model perform across multiple GPU endpoints?	metrumbench-imagegen	Use multi-endpoint mode with least-in-flight or weighted round-robin load balancing