Skip to main content

InferenceX benchmarking

InferenceX is a third-party LLM serving benchmark integrated into Metrum AI Bench. It follows a SemiAnalysis-style synthetic serving methodology: fixed input and output token lengths, throughput normalized per GPU, and optional dimensions for advanced serving architectures (disaggregated prefill/decode, expert parallelism, multi-node).

It is still an LLM workload: same modality, same OpenAI-compatible chat endpoint, same traffic axes (concurrency, ISL, OSL). The differences from metrumbench-llm are the load source (synthetic token sequences, no prompt library) and the metric focus (tokens per second per GPU, plus architecture dimensions such as TP, EP, and disaggregation).

Results appear on the Reporting page after a job completes.

What it measures and when to use it​

InferenceX drives a running model server with synthetic chat traffic at a concurrency you choose, then reports GPU-normalized throughput and latency percentiles. Use it to answer:

  • How many tokens per second per GPU can this model serve on this hardware?
  • How does serving efficiency compare across H100, H200, and B200 for the same model and recipe?
  • What is the latency profile (TTFT, TPOT, end-to-end) under a fixed synthetic ISL/OSL grid?
  • How do advanced layouts (tensor parallelism, expert parallelism, disaggregated prefill/decode, multi-node) change efficiency?

Use InferenceX when you need:

  • GPU-normalized throughput for hardware-to-hardware comparison
  • Results in a format aligned with SemiAnalysis-style industry benchmarks
  • Dimensions for disaggregated prefill/decode, expert parallelism, or multi-node setups
  • Synthetic fixed-length ISL/OSL with no prompt variability

Use metrumbench-llm when you need real prompt content, prompt libraries, multi-replica load-balanced endpoints, ramp-up steady-state analysis, or richer first-party client-side streaming metrics. Full metric definitions and architecture notes live in InferenceX Methodology.

Step 1: create the job​

Create an InferenceX job pinned to a project, model, framework version, and target server. Name jobs so the traffic shape is obvious, for example ix-<gpu>-<model>-isl<isl>-osl<osl>-c<concurrency>.

Step 2: set the traffic shape​

InferenceX uses synthetic random token sequences. The shape is defined by job parameters (not a prompt library):

ArgumentWhat it meansHow to choose
concurrencyMaximum simultaneous in-flight requests.Start at 8 or 64 depending on model size; raise to find the per-GPU plateau.
input_sequence_lengthSynthetic input length in tokens (random dataset).Match the ISL you want to compare across hardware. Common grids: 128, 512, 2048.
output_sequence_lengthSynthetic output length in tokens.Match the OSL you want to compare. Common grids: 128, 256, 512.
num_requestsTotal prompts to send.Hundreds for stable numbers. Too few requests make percentiles noisy.
random_range_ratioWidth of the random length distribution around ISL/OSL. 1.0 is fixed length (matches the upstream default for controlled grids).Leave at 1.0 unless you intentionally want length variance.

Daemon-managed paths (result_dir, result_filename) and the tool checkout/venv are set by the agent. Do not configure them yourself.

The benchmark targets a single endpoint with replicas = 1 and tensor parallelism equal to the GPU count on the selected hardware. If you need multi-replica load-balanced benchmarks, use metrumbench-llm instead.

Step 3: run and ingest​

Once the job is queued on a server with a healthy agent, the daemon:

  1. Bootstraps the pinned InferenceX checkout and shared venv.
  2. Starts the model server from the workload's framework and version.
  3. Runs the upstream serving benchmark against the OpenAI-compatible endpoint.
  4. Processes the result file and POSTs aggregate metrics to ingest_inferencex_results.

You do not call ingest by hand in normal operation. The agent owns that step. If ingestion fails, the agent retries from a pending record on the next cycle.

Step 4: read the results​

Filter the Reporting page to the InferenceX job. Headline metrics:

MetricWhat it meansWhat to watch
Tokens per GPUTotal tokens (input + output) per second per GPU.Primary efficiency number for hardware comparison.
Output tokens per GPUGenerated tokens per second per GPU.Decode-side serving efficiency.
Input tokens per GPUPrefill tokens per second per GPU.Prefill-side efficiency; rises with ISL.
Mean / p99 TTFTTime to first token, average and tail (seconds).Track p99 for latency SLAs under load.
Mean TPOTAverage time per output token (seconds).User-perceived tokens/sec per request is about 1 / TPOT.
Mean E2E latencyEnd-to-end request latency (seconds).Full request cost including queueing and generation.
Mean interactivity TPSInteractive tokens per second (user-perceived throughput).Complements TPOT for interactivity studies.

Architecture dimensions recorded with each run (when applicable): hardware, framework, inference precision, speculative decoding, ISL/OSL/concurrency, tensor parallelism (TP), expert parallelism (EP), data-parallel attention, disaggregated prefill/decode GPU counts and workers, multi-node flag, container image, and model id. Full column definitions are in InferenceX Methodology.

InferenceX vs metrumbench-llm​

Aspectmetrumbench-llmInferenceX
WorkloadReal prompts from a prompt librarySynthetic fixed-length token sequences
Throughput unitTokens per second (aggregate)Tokens per second per GPU
TTFT measurementClient-side SSE stream timingReported from tool output
Ramp-upSupported (gradual concurrency increase)Not applicable
Disaggregated servingNot recorded as dimensionsPrefill/decode GPU and worker breakdown
Multi-nodeNot explicitly recordedRecorded as a dimension
Best forReal-world serving with prompt diversityHardware efficiency and GPU-normalized comparison

When comparing the two tools, hold model, framework, engine args, hardware, ISL, OSL, concurrency, and streaming fixed. Vary only the tool. Treat absolute number parity as directional: synthetic tokens and per-GPU normalization are not the same measurement as real-prompt aggregate throughput.

Common pitfalls​

  • Comparing aggregate tokens/sec to tokens/sec per GPU. Multiply or divide by GPU count before ranking hardware, or always report the InferenceX per-GPU columns.
  • Too few requests. Percentiles (p90, p99, p999) need enough samples. Use hundreds of requests for numbers you will publish.
  • Leaving length variance on when you want a fixed grid. Set random_range_ratio to 1.0 for exact synthetic ISL/OSL.
  • Expecting prompt-library behavior. InferenceX does not use real prompts. For production traffic shape or domain-specific prompts, use metrumbench-llm.
  • Skipping engine args validation. Bad max_model_len or tensor-parallel size fails before any request is sent. Start from a verified recipe; see Choosing engine arguments.