InferenceX Benchmark

InferenceX is a third-party LLM serving benchmark tool integrated into Metrum Insights. It follows a SemiAnalysis-style synthetic serving methodology that focuses on GPU-normalized throughput and advanced serving architecture metrics like disaggregated prefill/decode, expert parallelism, and multi-node configurations.

Unlike metrumbench-llm, which uses real prompts from a prompt library and measures TTFT/TPOT at the HTTP client level, InferenceX generates synthetic workloads with fixed input/output token distributions and reports metrics normalized per GPU. This makes it useful for hardware-to-hardware comparisons and for evaluating advanced parallelism strategies.

---

When to Use InferenceX

Use InferenceX when you need to:

• Compare raw GPU serving throughput across different hardware configurations
• Evaluate advanced serving architectures: disaggregated prefill/decode, expert parallelism, tensor parallelism, data-parallel attention
• Benchmark multi-node inference setups
• Produce results in a format that aligns with SemiAnalysis-style industry benchmarks
• Measure per-GPU efficiency rather than aggregate endpoint throughput

Use metrumbench-llm instead when you need real-prompt workloads, streaming TTFT/TPOT at the HTTP client level, prompt library integration, or ramp-up steady-state analysis.

---

How It Works

Workload Generation

InferenceX generates synthetic chat workloads with configurable input and output token length distributions. Rather than sending real text prompts, it uses controlled synthetic token sequences that produce predictable, repeatable load patterns. This removes prompt variability from the measurement, making it easier to isolate hardware and framework performance differences.

Key parameters that define the synthetic workload:

Parameter	What it controls
ISL (Input Sequence Length)	Number of input tokens per request
OSL (Output Sequence Length)	Number of output tokens generated per request
Concurrency	Number of simultaneous in-flight requests

Execution

InferenceX is daemon-managed. The agent daemon:

1. Clones the pinned upstream InferenceX checkout under {work_dir}/inferencex/{version} and creates a shared venv.
2. Launches upstream utils/bench_serving/benchmark_serving.py as a subprocess with cwd=checkout_root and the venv on PATH / VIRTUAL_ENV / PYTHONPATH.
3. Rewrites daemon-owned result_dir / result_filename job params from /workspace/... to the agent logs directory (same pattern as metrumbench-llm data_log).
4. After the job finishes, runs upstream process_result.py and POSTs the aggregate metrics to ingest_inferencex_results.

Users configure only the synthetic workload shape: model, concurrency, input_sequence_length, output_sequence_length, num_requests, random_range_ratio. Everything else (tool version, server config, paths, env vars) is derived from the control plane or set by the daemon.

The tool targets OpenAI-compatible chat completion endpoints, the same endpoints used by metrumbench-llm and metrumbench-vlm.

Results Ingestion

Results are ingested into the database through the ingest_inferencex_results RPC function. This is a single atomic call that:

1. Validates that the job exists and belongs to the InferenceX tool.
2. Checks that the caller has access to the project.
3. Creates a tool invocation record with the benchmark dimensions.
4. Stores the output metrics in typed columns.
5. Marks the job as completed.

---

What Gets Measured

Throughput Metrics (Per GPU)

InferenceX normalizes all throughput metrics by GPU count, which makes cross-hardware comparisons straightforward.

Metric	Definition
Tokens Per GPU	Total tokens (input + output) processed per second per GPU
Output Tokens Per GPU	Output tokens generated per second per GPU
Input Tokens Per GPU	Input tokens processed per second per GPU

Latency Metrics

InferenceX reports four latency metrics. Each one is captured as a fan-out of percentiles so the head and the tail are both visible.

Metric Group	What it measures	Reported percentiles
TTFT (Time to First Token)	Wall-clock from request arrival to the first output token. Captures prefill and queueing.	Mean, median, P90, P99, P99.9
TPOT (Time Per Output Token)	Average time between output tokens after the first. Captures steady-state generation speed.	Mean, median, P90, P99, P99.9
E2EL (End-to-End Latency)	Total request latency from arrival to final response.	Mean, median, P90, P99, P99.9
Interactivity (ITL)	User-perceived tokens per second: 1000 / TPOT. Complements TPOT for interactivity studies.	Mean, median, plus values at TPOT P90, P99, and P99.9 thresholds

The full column list in v_inferencex_outputs is listed in How Results Appear.

Benchmark Dimensions

Each InferenceX run records the full configuration so that results are reproducible and comparable:

Dimension	What it records
Hardware	GPU/accelerator type used
Framework	Serving framework (vLLM, SGLang, TensorRT-LLM, etc.)
Inference Precision	Data type used for inference (FP16, BF16, FP8, INT8)
Speculative Decoding	Whether speculative decoding was enabled and what kind
ISL / OSL / Concurrency	The traffic shape used for the benchmark
Tensor Parallelism (TP)	Number of GPUs used for tensor parallelism
Expert Parallelism (EP)	Number of GPUs used for expert parallelism (MoE models)
DP Attention	Data-parallel attention configuration
Disaggregated	Whether prefill and decode run on separate GPU groups
Multi-node	Whether the benchmark spans multiple physical nodes
Container Image	The exact container image used for the model server
Model ID	The model identifier being served

Disaggregated Prefill/Decode Dimensions

For advanced serving architectures that separate prefill and decode onto different GPU groups, InferenceX records additional dimensions:

Dimension	What it records
Prefill GPU Count	Number of GPUs dedicated to prefill
Decode GPU Count	Number of GPUs dedicated to decode
Prefill TP / EP	Tensor and expert parallelism for the prefill group
Decode TP / EP	Tensor and expert parallelism for the decode group
Prefill Workers	Number of prefill worker processes
Decode Workers	Number of decode worker processes
Prefill DP Attention	Data-parallel attention config for prefill
Decode DP Attention	Data-parallel attention config for decode

---

System Metrics (Hardware Telemetry)

InferenceX runs collect the same hardware telemetry as all other Metrum Insights benchmark workloads. The platform captures system-level metrics from the host during the run so benchmark tool numbers can be interpreted in the context of what the hardware was doing.

What Gets Collected

Telemetry is sampled as a time series throughout the benchmark run.

GPU Metrics (per GPU):

Metric	Unit	Description
GPU Power	Watts	Power consumption of each GPU during the run
GPU Memory Used	Bytes	GPU memory actively in use at sample time
GPU Memory Total	Bytes	Total GPU memory capacity
GPU Utilization	0.0 to 1.0	Fraction of GPU compute cycles actively used

On multi-GPU servers, each GPU is tracked independently using a GPU index (0, 1, 2, ...). This identifies load imbalances across GPUs.

CPU Metrics (per CPU socket):

Metric	Unit	Description
CPU Utilization	0.0 to 1.0	Fraction of CPU cycles actively used

System-wide Metrics:

Metric	Unit	Description
System RAM Used	Bytes	Host memory currently in use
System RAM Total	Bytes	Total host memory capacity
System Power	Watts	Total system power draw (when available)

Aggregated Views

• v_telemetry_job_summary - one row per job with sample count, GPU count, average and max GPU power, average GPU utilization, average and max GPU memory percentage, average CPU utilization, and average system RAM percentage.
• v_telemetry_per_gpu_summary - one row per GPU per job for multi-GPU analysis: average, max, and min GPU power; average and max GPU utilization; average and max GPU memory usage per GPU.
• v_job_with_telemetry - benchmark jobs joined with their telemetry summaries for a single view of model, framework, status, and resource utilization.

Why System Metrics Matter for InferenceX

System metrics are especially important for interpreting InferenceX results because the tool generates synthetic fixed-length load that pushes hardware to a steady state. The telemetry reveals whether that steady state is limited by compute, memory, or power:

Question	How to answer
Is the GPU saturated at this concurrency?	GPU utilization near 100% with the target ISL/OSL means compute-bound. Below 50% at high concurrency suggests a CPU, network, or memory bottleneck.
Is the model close to the GPU memory limit?	GPU memory percentage shows how much headroom remains before OOM.
Is power delivery the bottleneck?	GPU power flat at TDP indicates thermal or power capping.
Is load balanced across GPUs?	Per-GPU utilization and power trends from v_telemetry_per_gpu_summary.
Did the GPU throttle during a benchmark?	Power oscillation or a sustained drop in utilization with no change in concurrency suggests throttling.

Availability

System metrics are available when the Metrum agent runs on the same host as the model server, which is the normal case for InferenceX. The agent uses NVML for NVIDIA GPU metrics and standard OS interfaces for CPU and system metrics. Full collection details are in the Performance Methodology.

---

How Results Appear

InferenceX results are available through v_inferencex_outputs and the unified v_benchmark_results view.

v_inferencex_outputs

A SQL view that joins output metrics with tool invocation and job IDs. Every column is prefixed with ix_ so InferenceX metrics stay distinct when joined with other tool results:

Throughput:

Column	Unit	Definition
ix_tokens_per_second	tokens/s	Aggregate total tokens processed per second (input + output, all GPUs). Restored from per-GPU storage by multiplying back the tensor-parallel size.
ix_output_tput_per_gpu	tokens/s/GPU	Output tokens generated per second per GPU
ix_input_tput_per_gpu	tokens/s/GPU	Input tokens processed per second per GPU

TTFT (Time to First Token):

Column	Unit	Definition
ix_time_to_first_token_ms	ms	Mean TTFT across all requests
ix_ttft_median_ms	ms	Median TTFT
ix_ttft_p90_ms	ms	P90 TTFT
ix_ttft_p99_ms	ms	P99 TTFT
ix_ttft_p99_9_ms	ms	P99.9 TTFT
ix_p99_ttft_s	s	P99 TTFT in seconds (same value as ix_ttft_p99_ms, different unit)

TPOT (Time Per Output Token):

Column	Unit	Definition
ix_mean_tpot_ms	ms	Mean TPOT across all requests
ix_tpot_median_ms	ms	Median TPOT
ix_tpot_p90_ms	ms	P90 TPOT
ix_tpot_p99_ms	ms	P99 TPOT
ix_tpot_p99_9_ms	ms	P99.9 TPOT

E2EL (End-to-End Latency):

Column	Unit	Definition
ix_mean_e2el_s	s	Mean end-to-end request latency
ix_e2el_median_s	s	Median end-to-end request latency
ix_e2el_p90_s	s	P90 end-to-end request latency
ix_e2el_p99_s	s	P99 end-to-end request latency
ix_e2el_p99_9_s	s	P99.9 end-to-end request latency

Interactivity (ITL: Interactive Tokens per Second):

Column	Unit	Definition
ix_mean_interactivity_tps	tok/s/user	Mean interactivity = 1000 / mean_tpot_ms
ix_intvty_median_tps	tok/s/user	Median interactivity = 1000 / median_tpot_ms
ix_intvty_at_p90_tpot_tps	tok/s/user	Interactivity computed at the TPOT P90 threshold
ix_intvty_at_p99_tpot_tps	tok/s/user	Interactivity computed at the TPOT P99 threshold
ix_intvty_at_p99_9_tpot_tps	tok/s/user	Interactivity computed at the TPOT P99.9 threshold

v_benchmark_results

The unified project dashboard view includes all InferenceX columns alongside metrumbench-llm and metrumbench-vlm results. All tools can be compared in one query or export.

---

Comparison With Other Tools

Aspect	metrumbench-llm	InferenceX
Workload	Real prompts from a prompt library	Synthetic fixed-length token sequences
Throughput unit	Tokens per second (aggregate)	Tokens per second per GPU
TTFT measurement	Client-side SSE stream timing	Reported from tool output
Prompt control	Full prompt library with ISL/OSL targeting	Synthetic token distributions
Ramp-up	Supported (gradual concurrency increase)	Not applicable
Streaming	Supported (SSE with chunk-level timing)	Depends on tool configuration
Disaggregated serving	Not recorded	Full prefill/decode GPU breakdown
Multi-node	Not explicitly recorded	Recorded as a dimension
Word metrics	Prompt and completion word counts	Not applicable
Best for	Real-world serving performance with prompt diversity	Hardware efficiency, GPU-normalized cross-platform comparison