Performance Methodology

Metrum Insights uses four first-party benchmarking tools to measure AI model serving performance across different modalities. Each tool targets a specific workload type, sends controlled traffic to an OpenAI-compatible endpoint, and produces structured output with timing, throughput, and accuracy metrics.

Tool	Modality	Protocol	Primary Use
metrumbench-llm	LLM text generation	JSON POST to /v1/chat/completions	Load testing language models with configurable concurrency, streaming, and prompt control
metrumbench-vlm	VLM image + text	JSON POST to /v1/chat/completions with image payloads	Benchmarking vision-language models with image inputs
metrumbench-asr	ASR audio transcription	Multipart POST to /v1/audio/transcriptions	Benchmarking speech-to-text models with accuracy measurement
metrumbench-imagegen	Image generation	JSON POST to /v1/images/generations	Benchmarking image generation models with artifact collection and integrity verification

All four tools share a common foundation: the same Rust crate, the same HTTP client library, and the same concurrency control approach. metrumbench-llm, metrumbench-vlm, and metrumbench-asr also share the same NTP time-sync preflight and endpoint resolution system. This means measurement methodology is consistent across modalities.

---

Shared Foundations

Clock Synchronization

Before any benchmark run, each tool contacts public NTP servers (pool.ntp.org, time.google.com, time.cloudflare.com, time.apple.com) and verifies the system clock is within one second of NTP time. This prevents skewed timestamps in results. The check can be skipped for air-gapped environments by setting METRUM_SKIP_NTP_CHECK=true.

HTTP Client

All tools use the same HTTP client configured with:

• Request timeout: maximum wall-clock time for a single request (default 120 seconds for metrumbench-llm/metrumbench-vlm, 120 seconds for metrumbench-asr CLI, 300 seconds when run through Metrum jobs).
• Connect timeout: time allowed to establish the TCP connection (default 30 seconds).
• Connection pool: idle connections are kept open and reused, sized to match the concurrency level. Idle timeout and TCP keepalive are configurable.

Endpoint Resolution

Each tool supports two endpoint modes:

• Single endpoint: one URL and one API key.
• Multi-endpoint: a YAML file listing multiple endpoints with optional weights. Requests are distributed in weighted round-robin order. This is useful for testing load balancers or comparing endpoints side by side.

Concurrency Control

All tools use an async semaphore to limit in-flight requests. The semaphore is sized to the --concurrency value. Each request acquires a permit before sending and releases it on completion. This ensures the target concurrency is never exceeded, regardless of response time variation.

Output Format

Each tool writes one JSON summary line to a JSONL file per benchmark run. This summary contains the full configuration, all computed metrics, error details, and per-endpoint breakdowns. The summary is always written, even when errors occur, so partial results are never lost.

Every summary includes:

• A UUID and a human-readable ID for easy reference.
• An ISO 8601 UTC timestamp.
• The tool version and Rust compiler version used to build the binary.
• The full configuration echoed back for reproducibility.

---

metrumbench-llm: LLM Performance

metrumbench-llm measures how fast a language model endpoint processes text generation requests under controlled load.

How Requests Work

metrumbench-llm sends JSON POST requests to an OpenAI-compatible chat completions or legacy completions endpoint. Each request includes a prompt from a JSONL prompt file, the target model name, a maximum token limit, and a sampling temperature.

In chat mode, prompts are wrapped in a messages array with a system message ("You are a helpful assistant.") and the user prompt. In completion mode, the prompt is sent directly.

Streaming mode uses Server-Sent Events (SSE). The tool processes the byte stream chunk by chunk, extracting content tokens as they arrive. When streaming is enabled, the tool also requests token usage statistics in the stream.

Non-streaming mode waits for the complete response body before parsing.

Prompt Handling

Prompts are loaded from a JSONL file (local path or HTTP URL). Each line must contain a JSON object with a prompt field. Before the run, all prompts are shuffled randomly for even distribution, then cycled deterministically through the request sequence. If there are fewer prompts than requests, prompts are reused in order.

When running through the Metrum platform, prompts are sampled from a prompt library based on target input and output token lengths. The platform can also run a calibration step to determine the token-to-word ratio for the specific model being tested.

What Gets Measured

Per request:

Metric	What it is	How it is captured
Response Time	Total wall-clock time from request start to completion	Instant::now() before the HTTP call, measured at stream end or body receipt
Time to First Token (TTFT)	Time from request start to the first output content chunk	Streaming: recorded when the first non-empty content chunk arrives. Non-streaming: equals response time
Time Per Output Token (TPOT)	Average time to generate each output token after the first	(Response Time - TTFT) / completion_tokens
Prompt Tokens	Number of input tokens	Read from the server's usage response
Completion Tokens	Number of output tokens generated	Read from the server's usage response
Prompt Words	Word count in the input prompt	Whitespace split count
Completion Words	Word count in the generated text	Whitespace split count

Key point: metrumbench-llm does not tokenize locally. Token counts come entirely from the model server's usage response. Word counts are computed locally as a tokenizer-independent fallback metric.

Aggregate statistics computed across all successful requests:

• Response times: min, max, average, p50, p90, p95, p99
• TTFT: min, max, average, p50, p90, p95, p99
• TPOT: min, max, average, p50, p90, p95, p99
• Token counts: total, min, max, average, p50, p95, tokens per second
• Word counts: total, min, max, average, words per second
• Throughput: requests per second, prompt tokens per second, completion tokens per second, total tokens per second
• Errors: count, error rate percentage, error type breakdown, OOM detection

Ramp-up Mode

metrumbench-llm supports a ramp-up period where concurrency increases linearly from one to the target value over a configurable number of seconds. During this period:

• Requests are still executed (load is applied to the server).
• Metrics from ramp-up requests are discarded.
• Only requests completing after the ramp-up period contribute to the final statistics.

This produces steady-state metrics that are not distorted by cold-start behavior or initial connection establishment.

Error Classification

metrumbench-llm classifies errors by HTTP status code:

Status	Classification
429	Rate limit
401, 403	Authentication
400	Bad request
500-599	Server error

Stream errors are classified by failure type: timeout, connection failure, body processing error, decoding error, or redirect error. The tool also detects out-of-memory (OOM) conditions from error messages.

Multi-endpoint Support

When using multiple endpoints, metrumbench-llm tracks per-endpoint metrics separately: response times, TTFT, TPOT, token counts, errors, and request rates. The JSONL output includes both aggregate and per-endpoint breakdowns.

---

metrumbench-vlm: VLM Performance

metrumbench-vlm measures vision-language model performance by sending prompts with one or more images to an OpenAI-compatible chat completions endpoint.

How Requests Work

metrumbench-vlm sends the same JSON POST requests as metrumbench-llm, but with image content in the messages array. Images can be included in two ways:

• Base64 encoding (default): images are loaded, optionally resized, and encoded directly into the request payload.
• Server-side download: image URLs are included in the request, and the model server downloads them.

Each prompt line in the input JSONL must have a prompt field and an image_urls array containing one or more image references (HTTP URLs, local file paths, or file:// URIs).

Image Processing

metrumbench-vlm includes built-in image handling:

• Resizing: large images can be automatically resized to a maximum dimension while preserving aspect ratio (--max-image-dimension).
• Caching: an LRU cache stores processed images to avoid redundant loading and encoding.
• Detail levels: images can be sent at low or high detail level, which affects token usage and processing time.
• Multiple images per prompt: a single request can include multiple images for multi-image reasoning tasks.

What Gets Measured

metrumbench-vlm collects the same core metrics as metrumbench-llm:

• Response times with percentile distributions
• TTFT and TPOT for streaming requests
• Token counts (prompt, completion, total) from the server
• Word counts for prompts and completions
• Throughput rates (tokens per second, requests per second)
• Error counts and classifications
• Per-endpoint breakdowns for multi-endpoint runs

Additional metrumbench-vlm-specific data in the output:

• Image processing times
• Image sizes and dimensions
• Per-request image metadata

Shared Methodology

metrumbench-vlm shares the same concurrency model, timing approach, percentile calculations, ramp-up support, error handling, and output format as metrumbench-llm. The only difference is the payload shape (images in the messages array) and the input JSONL format.

---

metrumbench-asr: ASR Performance

metrumbench-asr measures audio transcription model performance by uploading audio files to an OpenAI-compatible transcription endpoint and evaluating both speed and accuracy.

How Requests Work

metrumbench-asr sends multipart form POST requests to an audio transcription endpoint. Each request includes:

• The audio file binary content
• The model name
• The requested response format (verbose_json, json, text, srt, or vtt)
• An optional language code (default: English)
• Timestamp granularity request (word level for JSON formats)

Audio files can be provided as local paths or URLs. Remote files are downloaded and cached locally before the benchmark starts.

Supported Audio Formats

Format	MIME Type
MP3	audio/mpeg
WAV	audio/wav
WebM	audio/webm
OGG	audio/ogg
M4A / MP4	audio/mp4
FLAC	audio/flac

What Gets Measured

Per request:

Metric	What it is	How it is captured
Response Time	Total wall-clock time including file upload and download	Measured from before file read to after response receipt
Inference Time	Server-side processing time	From the inference_time field in JSON responses, or wall-clock time as fallback
RTF (Real-Time Factor)	Ratio of inference time to audio duration	inference_time / audio_duration. Values below 1.0 mean faster than real-time
Words Per Second	Transcription throughput by word count	word_count / inference_time
Characters Per Second	Transcription throughput by character count	char_count / inference_time
Bytes Sent	Size of audio file uploaded	File size in bytes
Bytes Received	Size of transcription response	Response body size in bytes

Aggregate statistics:

• Response times: min, max, average, p50, p90, p95, p99
• Inference times: min, max, average, p50, p90, p95, p99
• RTF: min, max, average, p50, p90, p95, p99
• Throughput: audio seconds processed per wall-clock second, characters per second, words per second, requests per second, data transfer rate
• Data transfer: total bytes sent, received, and transferred
• Audio format breakdown: count of files processed by format
• Errors: count, error rate percentage

Accuracy Metrics

metrumbench-asr computes accuracy metrics when a ground truth transcript file is provided. Ground truth is a JSONL file where each line has an id field matching an audio sample and a transcript field with the reference text.

Word Error Rate (WER):

WER measures how many word-level edits are needed to transform the transcription into the reference text. It uses the Levenshtein edit distance algorithm at the word level with case-insensitive comparison.

• A WER of 0% means perfect transcription.
• A WER of 100% means every word is wrong.
• WER can exceed 100% if the transcription has more words than the reference (many insertions).

The formula: WER = (substitutions + insertions + deletions) / reference word count

Character Error Rate (CER):

CER applies the same Levenshtein algorithm at the character level. It is more granular than WER and useful for evaluating partial-word accuracy and languages where word boundaries are less clear.

Aggregate accuracy statistics: min, max, average, p50, p90, p95, p99 for both WER and CER across all samples with ground truth.

Input Format

The audio sample manifest is a JSONL file where each line describes one audio file:

{"id": "sample-001", "path": "/path/to/audio.flac", "format": "flac", "duration": 12.5}
{"id": "sample-002", "url": "https://storage.example.com/audio.mp3", "format": "mp3", "duration": 8.3}

• id is required and must be unique.
• Either path (local file) or url (remote download) must be present.
• format determines the MIME type for upload.
• duration is needed for RTF calculation. Without it, RTF is skipped.

Audio samples are cycled round-robin when the request count exceeds the sample count.

---

metrumbench-imagegen: Image Generation Performance

metrumbench-imagegen measures image generation model performance by sending prompts to an OpenAI-compatible image generation endpoint, collecting generated image artifacts, and tracking generation latency and throughput.

How Requests Work

metrumbench-imagegen sends JSON POST requests to an /images/generations endpoint. Each request includes:

• The text prompt describing the desired image
• The model name
• The number of images to generate per request (n)
• The target image size (for example 1024x1024)
• The response format (b64_json for base64-encoded images, or url for download links)
• Optional parameters: seed, negative prompt, inference steps, guidance scale, and custom body fields

Prompts can be provided as a single --prompt string or loaded from a JSONL file with --prompts. Each JSONL line can include per-prompt overrides for seed, size, and negative prompt.

Seed Control

metrumbench-imagegen supports three seed modes for reproducibility:

• Fixed: every request uses the same seed value.
• Increment: the seed increases by one for each request, producing deterministic variation across the run.
• Prompt: seeds are read from the prompt JSONL file, allowing full control over which seed each prompt uses.

Endpoint and Load Balancing

metrumbench-imagegen supports multiple endpoint modes:

• Single endpoint: one --url with an --api-key.
• Repeated endpoints: multiple --endpoint flags for simple multi-endpoint runs.
• Endpoints file: a JSON or JSONL file listing endpoints with optional names and weights.

Four load balancing strategies are available:

• Round-robin (default): requests cycle through endpoints in order.
• Weighted round-robin: endpoints with higher weights receive proportionally more requests.
• Least in-flight: each request goes to the endpoint with the fewest active requests at that moment.
• Random: each request goes to a randomly chosen endpoint.

metrumbench-imagegen also supports optional endpoint health checks before the benchmark starts, and configurable retry attempts with backoff for failed requests.

Artifact Collection

When using the b64_json response format, metrumbench-imagegen decodes and saves each generated image to a local artifact directory. For every saved image, it records:

• File path and index within the request
• SHA-256 hash of the image bytes for integrity verification
• File size in bytes
• Image dimensions (width and height)
• MIME type

Raw API responses can also be saved as JSON files for debugging or analysis.

What Gets Measured

Per request:

Metric	What it is	How it is captured
Latency	Total wall-clock time from request start to completion	Measured as the difference between UTC start and completion timestamps
Images Requested	Number of images asked for (n parameter)	From the request configuration
Images Returned	Number of images in the API response	Count of items in the response data array
Response Bytes	Size of the full API response	Byte length of the response body
Image Artifacts	Decoded image details	SHA-256, dimensions, file size per saved image

Aggregate statistics:

• Latency: min, mean, p50, p90, p95, p99, max (in milliseconds)
• Image byte sizes: min, mean, p50, p90, p95, p99, max
• Throughput: images per second, requests per second
• Success rate: fraction of requests that completed without error
• Image counts: total images requested vs total images generated
• Per-endpoint breakdown: request count, success/failure counts, images generated, images per second, requests per second, and latency percentiles for each endpoint

Error Handling

metrumbench-imagegen classifies errors into these types:

Error Type	Meaning
timeout	Request exceeded the configured timeout
connection_error	TCP connection or response body read failure
http_error	Server returned a non-success HTTP status
schema_error	Response body could not be parsed or was missing expected fields
decode_error	Base64 image data could not be decoded
artifact_error	Image file could not be saved to disk
request_error	Other request-level failure

The summary groups errors by type and HTTP status code so you can quickly see whether failures are server-side, network-related, or format mismatches.

Output Format

metrumbench-imagegen produces two output files:

• Data log (JSONL): one line per request with full request details, latency, status, image artifacts, attempt history, and error information.
• Summary (JSON): a single file with aggregate metrics, per-endpoint breakdowns, error summaries, and artifact paths.

Each per-request record in the data log follows the metrumbench-imagegen.request.v1 schema. The summary follows the metrumbench-imagegen.summary.v1 schema.

---

Percentile Calculations

All four tools compute percentile statistics using the same approach:

1. Sort all values in ascending order.
2. Calculate the index as a function of the desired percentile and array length.
3. Return the value at that index.

Standard percentiles reported: p50 (median), p90, p95, and p99. These provide a clear picture of typical performance (p50), tail latency (p95/p99), and overall distribution shape.

---

How Results Reach the Database

Benchmark results flow through a consistent pipeline regardless of which tool produced them:

1. Tool writes JSONL: the benchmark tool appends one summary JSON line to its data log file.
2. Agent reads summary: the Metrum agent daemon reads the last line of the JSONL file after the tool exits.
3. Agent uploads to PostgREST: the agent maps the summary fields to the tool-specific database tables and inserts them via the API.
4. Views make results queryable: SQL views join the raw output tables with job, workload, scenario, and hardware information for reporting.

Each tool has dedicated database tables:

Tool	Invocation Table	Output Table	Primary View
metrumbench-llm	metrumbench_llm_invocations	metrumbench_llm_outputs	v_metrumbench_llm_outputs
metrumbench-vlm	metrumbench_vlm_invocations	metrumbench_vlm_outputs	v_metrumbench_vlm_outputs
metrumbench-asr	metrumbench_asr_invocations	metrumbench_asr_outputs	v_metrumbench_asr_outputs
metrumbench-imagegen	metrumbench_imagegen_invocations	metrumbench_imagegen_outputs	v_metrumbench_imagegen_outputs

The invocation table stores the CLI configuration used for the run. The output table stores all computed metrics in typed columns for efficient querying.

---

System Metrics (Hardware Telemetry)

In addition to benchmark tool metrics, Metrum Insights collects hardware telemetry during benchmark runs. These system-level metrics show what the hardware was doing while the benchmark tool was generating load, providing the full picture of serving performance and resource utilization.

What Gets Collected

Telemetry is sampled as a time series throughout the benchmark run. Each sample captures:

GPU Metrics (per GPU):

Metric	Unit	Description
GPU Power	Watts	Power consumption of each GPU during the run
GPU Memory Used	Bytes	GPU memory actively in use at sample time
GPU Memory Total	Bytes	Total GPU memory capacity
GPU Utilization	0.0 to 1.0	Fraction of GPU compute cycles actively used

On multi-GPU servers, each GPU is tracked independently using a GPU index (0, 1, 2, ...). This lets you see whether load is evenly distributed across GPUs or if one GPU is a bottleneck.

CPU Metrics (per CPU socket):

Metric	Unit	Description
CPU Utilization	0.0 to 1.0	Fraction of CPU cycles actively used

On multi-socket systems, each CPU socket is tracked independently.

System-wide Metrics:

Metric	Unit	Description
System RAM Used	Bytes	Host memory currently in use
System RAM Total	Bytes	Total host memory capacity
System Power	Watts	Total system power draw (when available)

TPU Metrics (for Google TPU workloads):

Metric	Unit	Description
Duty Cycle	Percentage	Average and peak TPU duty cycle
Tensor Core Utilization	Percentage	Average and peak tensor core usage
HBM Capacity Used	Bytes	High-bandwidth memory in use
HBM Capacity Total	Bytes	Total HBM available

How Telemetry is Collected

The Metrum agent on the benchmark worker samples hardware metrics at regular intervals during the run. Collection is tied to the job lifecycle:

• Samples are only accepted within the run's time window (between run start and run completion).
• Each sample carries a UTC timestamp for time series analysis.
• Samples are rejected if they fall outside the run period, ensuring data integrity.

The agent uses NVML (NVIDIA Management Library) for GPU metrics on NVIDIA hardware, and standard OS interfaces for CPU and system metrics. For TPU workloads, telemetry comes from LibTPU metrics.

Aggregated Views

Raw telemetry samples are aggregated into summary views for easy consumption:

v_telemetry_job_summary - one row per job with:

• Sample count and collection duration
• GPU count detected
• Average and max GPU power (watts)
• Average GPU utilization
• Average and max GPU memory percentage
• Average CPU utilization
• Average system RAM percentage

v_telemetry_per_gpu_summary - one row per GPU per job for multi-GPU analysis:

• Average, max, and min GPU power per GPU
• Average and max GPU utilization per GPU
• Average and max GPU memory usage per GPU

v_job_with_telemetry - benchmark jobs joined with their telemetry summaries, giving a single view of model, framework, status, and resource utilization.

Why System Metrics Matter

System metrics answer questions that benchmark tool metrics cannot:

Question	Tool Metrics Alone	With System Metrics
Is the GPU fully utilized?	No visibility	GPU utilization shows headroom or saturation
Why did throughput drop?	Error rate may increase	GPU memory pressure or thermal throttling visible in power/memory trends
Is my model too large for this GPU?	OOM errors in tool output	GPU memory usage as percentage shows how close you are to the limit
Is the CPU a bottleneck?	Not visible	CPU utilization spikes during tokenization or data loading
How much power does serving cost?	Not visible	GPU and system power draw in watts, useful for cost-per-token analysis
Is load balanced across GPUs?	Not visible	Per-GPU utilization and power show imbalances

Availability

System metrics are available for managed workloads where the Metrum agent runs on the same host as the model server. For BYOE workloads (where the model server runs on external infrastructure), GPU and system metrics are typically not available because the agent does not have access to the endpoint's hardware. In BYOE cases, attach external monitoring data or note the gap in your benchmark report.

---

Metric Definitions Reference

Latency Metrics

Metric	Applies To	Definition
Response Time	All tools	Total wall-clock time from request start to response completion
TTFT (Time to First Token)	metrumbench-llm, metrumbench-vlm	Time from request start to first output content in the SSE stream. Equals response time for non-streaming requests
TPOT (Time Per Output Token)	metrumbench-llm, metrumbench-vlm	Average generation time per token after the first: (Response Time - TTFT) / completion_tokens
Inference Time	metrumbench-asr	Server-reported processing time. Falls back to response time when not reported
Latency	metrumbench-imagegen	Wall-clock time from request start to image generation completion

Throughput Metrics

Metric	Applies To	Definition
Tokens Per Second	metrumbench-llm, metrumbench-vlm	Total tokens processed divided by elapsed time (prompt, completion, and total reported separately)
Words Per Second	metrumbench-llm, metrumbench-vlm, metrumbench-asr	Total words divided by elapsed time
Requests Per Second	All tools	Total completed requests divided by elapsed time
Images Per Second	metrumbench-imagegen	Total images generated divided by elapsed time
Audio Per Second	metrumbench-asr	Total audio duration processed divided by wall-clock time. Values above 1.0 mean processing is faster than real-time
Characters Per Second	metrumbench-asr	Total transcribed characters divided by elapsed time
Data Transfer Rate	metrumbench-asr	Total bytes (sent + received) divided by elapsed time

Accuracy and Quality Metrics

Metric	Applies To	Definition
WER (Word Error Rate)	metrumbench-asr	Levenshtein word-level edit distance divided by reference word count. 0% is perfect
CER (Character Error Rate)	metrumbench-asr	Levenshtein character-level edit distance divided by reference character count. 0% is perfect
RTF (Real-Time Factor)	metrumbench-asr	Inference time divided by audio duration. Below 1.0 means faster than real-time
Image SHA-256	metrumbench-imagegen	Per-image cryptographic hash for artifact integrity and reproducibility verification
Success Rate	metrumbench-imagegen	Fraction of requests that completed without error

Error Metrics

Metric	Applies To	Definition
Error Count	All tools	Number of failed requests
Error Rate	All tools	Failed requests as a percentage of total requests
OOM Detected	metrumbench-llm	Whether any error message indicates an out-of-memory condition
Timeout Count	metrumbench-imagegen	Number of requests that exceeded the configured timeout

---

When to Use Each Tool

Question	Tool	Why
How fast does my LLM serve text under load?	metrumbench-llm	Measures token throughput, TTFT, TPOT, and latency percentiles with streaming support
How does my VLM handle image inputs?	metrumbench-vlm	Sends images with prompts and measures the same LLM metrics plus image processing overhead
How fast and accurate is my speech-to-text model?	metrumbench-asr	Measures transcription latency, real-time factor, and accuracy against reference transcripts
How fast does my image generation model produce images?	metrumbench-imagegen	Measures generation latency, images per second, and collects artifacts with integrity hashes
Do different benchmark tools agree on performance?	All four (plus third-party tools)	Run the same scenario with multiple tools and compare
What is the steady-state performance after warm-up?	metrumbench-llm	Use --ramp-up-seconds to discard warm-up metrics
How does my endpoint handle increasing load?	metrumbench-llm	Use ramp-up with increasing concurrency to find saturation points
Is my transcription accurate enough for production?	metrumbench-asr	Compare WER/CER against ground truth transcripts
Are my generated images reproducible across runs?	metrumbench-imagegen	Use fixed or increment seed modes and compare SHA-256 hashes across runs
How does my image model perform across multiple GPU endpoints?	metrumbench-imagegen	Use multi-endpoint mode with least-in-flight or weighted round-robin load balancing