InferenceX Benchmark

InferenceX is a third-party LLM serving benchmark tool integrated into Metrum Insights. It follows a SemiAnalysis-style synthetic serving methodology that focuses on GPU-normalized throughput and advanced serving architecture metrics like disaggregated prefill/decode, expert parallelism, and multi-node configurations.

Unlike metrumbench-llm, which uses real prompts from a prompt library and measures TTFT/TPOT at the HTTP client level, InferenceX generates synthetic workloads with fixed input/output token distributions and reports metrics normalized per GPU. This makes it useful for hardware-to-hardware comparisons and for evaluating advanced parallelism strategies.

---

When to Use InferenceX

Use InferenceX when you need to:

• Compare raw GPU serving throughput across different hardware configurations
• Evaluate advanced serving architectures: disaggregated prefill/decode, expert parallelism, tensor parallelism, data-parallel attention
• Benchmark multi-node inference setups
• Produce results in a format that aligns with SemiAnalysis-style industry benchmarks
• Measure per-GPU efficiency rather than aggregate endpoint throughput

Use metrumbench-llm instead when you need real-prompt workloads, streaming TTFT/TPOT at the HTTP client level, prompt library integration, or ramp-up steady-state analysis.

---

How It Works

Workload Generation

InferenceX generates synthetic chat workloads with configurable input and output token length distributions. Rather than sending real text prompts, it uses controlled synthetic token sequences that produce predictable, repeatable load patterns. This removes prompt variability from the measurement, making it easier to isolate hardware and framework performance differences.

Key parameters that define the synthetic workload:

Parameter	What it controls
ISL (Input Sequence Length)	Number of input tokens per request
OSL (Output Sequence Length)	Number of output tokens generated per request
Concurrency	Number of simultaneous in-flight requests

Execution

InferenceX runs through a Python wrapper (insights_cli.tools.inferencex_runner) that handles:

• Launching the benchmark against the target model server
• Collecting the aggregate metrics from the tool output
• Mapping results to the Metrum Insights database format

The tool targets OpenAI-compatible chat completion endpoints, the same endpoints used by metrumbench-llm and metrumbench-vlm.

Results Ingestion

Results are ingested into the database through the ingest_inferencex_results RPC function. This is a single atomic call that:

1. Validates that the job exists and belongs to the InferenceX tool.
2. Checks that the caller has access to the project.
3. Creates a tool invocation record with the benchmark dimensions.
4. Stores the output metrics in typed columns.
5. Marks the job as completed.

---

What Gets Measured

Throughput Metrics (Per GPU)

InferenceX normalizes all throughput metrics by GPU count, which makes cross-hardware comparisons straightforward.

Metric	Definition
Tokens Per GPU	Total tokens (input + output) processed per second per GPU
Output Tokens Per GPU	Output tokens generated per second per GPU
Input Tokens Per GPU	Input tokens processed per second per GPU

Latency Metrics

Metric	Definition
Mean TTFT	Average time to first token across all requests (seconds)
p99 TTFT	99th percentile time to first token (seconds)
Mean E2E Latency	Average end-to-end request latency (seconds)
Mean TPOT	Average time per output token (seconds)
Mean Interactivity TPS	Average interactive tokens per second (a user-perceived throughput metric)

Benchmark Dimensions

Each InferenceX run records the full configuration so that results are reproducible and comparable:

Dimension	What it records
Hardware	GPU/accelerator type used
Framework	Serving framework (vLLM, SGLang, TensorRT-LLM, etc.)
Inference Precision	Data type used for inference (FP16, BF16, FP8, INT8)
Speculative Decoding	Whether speculative decoding was enabled and what kind
ISL / OSL / Concurrency	The traffic shape used for the benchmark
Tensor Parallelism (TP)	Number of GPUs used for tensor parallelism
Expert Parallelism (EP)	Number of GPUs used for expert parallelism (MoE models)
DP Attention	Data-parallel attention configuration
Disaggregated	Whether prefill and decode run on separate GPU groups
Multi-node	Whether the benchmark spans multiple physical nodes
Container Image	The exact container image used for the model server
Model ID	The model identifier being served

Disaggregated Prefill/Decode Dimensions

For advanced serving architectures that separate prefill and decode onto different GPU groups, InferenceX records additional dimensions:

Dimension	What it records
Prefill GPU Count	Number of GPUs dedicated to prefill
Decode GPU Count	Number of GPUs dedicated to decode
Prefill TP / EP	Tensor and expert parallelism for the prefill group
Decode TP / EP	Tensor and expert parallelism for the decode group
Prefill Workers	Number of prefill worker processes
Decode Workers	Number of decode worker processes
Prefill DP Attention	Data-parallel attention config for prefill
Decode DP Attention	Data-parallel attention config for decode

---

How Results Appear

InferenceX results are available through:

• v_inferencex_outputs: a SQL view that joins output metrics with job IDs. Columns are prefixed with ix_ for clarity when joined with other tool results: ix_tokens_per_second, ix_time_to_first_token_ms, ix_mean_e2el_s, ix_p99_ttft_s, ix_output_tput_per_gpu, ix_input_tput_per_gpu, ix_mean_interactivity_tps.

• v_benchmark_results: the unified project dashboard view includes InferenceX metrics alongside metrumbench-llm and metrumbench-vlm results, so all tools can be compared in one query or export.

• Web UI: InferenceX job results appear on the project results page alongside other tool results.

---

Comparison With Other Tools

Aspect	metrumbench-llm	InferenceX
Workload	Real prompts from a prompt library	Synthetic fixed-length token sequences
Throughput unit	Tokens per second (aggregate)	Tokens per second per GPU
TTFT measurement	Client-side SSE stream timing	Reported from tool output
Prompt control	Full prompt library with ISL/OSL targeting	Synthetic token distributions
Ramp-up	Supported (gradual concurrency increase)	Not applicable
Streaming	Supported (SSE with chunk-level timing)	Depends on tool configuration
Disaggregated serving	Not recorded	Full prefill/decode GPU breakdown
Multi-node	Not explicitly recorded	Recorded as a dimension
Word metrics	Prompt and completion word counts	Not applicable
Best for	Real-world serving performance with prompt diversity	Hardware efficiency, GPU-normalized cross-platform comparison

---

Creating InferenceX Jobs

Through the API

curl -fsS -X POST "$METRUM_API_URL/rpc/create_inferencex_job" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_job_name\":\"ix-h100-llama70b-isl128-osl128-c64\",
    \"p_framework_code\":\"vllm\",
    \"p_version\":\"0.20.0\",
    \"p_model_code\":\"meta-llama/Llama-3.1-70B-Instruct\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_config_code\":\"shadeform-h100-1x\",
    \"p_server_hostname\":\"$SERVER_HOSTNAME\"
  }" | jq

Ingesting Results

After the benchmark completes, ingest the aggregate metrics:

curl -fsS -X POST "$METRUM_API_URL/rpc/ingest_inferencex_results" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_job_id\":$JOB_ID,
    \"p_run_id\":$RUN_ID,
    \"p_tool_version\":\"1.0.0\",
    \"p_hw\":\"H100\",
    \"p_framework\":\"vllm\",
    \"p_precision\":\"bf16\",
    \"p_isl\":128,
    \"p_osl\":128,
    \"p_conc\":64,
    \"p_tp\":1,
    \"p_tput_per_gpu\":1250.5,
    \"p_output_tput_per_gpu\":625.3,
    \"p_input_tput_per_gpu\":625.2,
    \"p_mean_ttft_s\":0.045,
    \"p_p99_ttft_s\":0.120,
    \"p_mean_e2el_s\":0.890,
    \"p_mean_tpot_s\":0.0065
  }" | jq

---

Querying InferenceX Results

curl -fsS "$METRUM_API_URL/v_inferencex_outputs?job_id=eq.$JOB_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

For cross-tool comparison within a project:

curl -fsS "$METRUM_API_URL/v_benchmark_results?project_id=eq.$PROJECT_ID&select=job_id,tool_code,tokens_per_second,ix_tokens_per_second,ttft_ms,ix_time_to_first_token_ms" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq