InferenceX benchmarking
InferenceX is a third-party LLM serving benchmark integrated into Metrum AI Bench. It follows a SemiAnalysis-style synthetic serving methodology: fixed input and output token lengths, throughput normalized per GPU, and optional dimensions for advanced serving architectures (disaggregated prefill/decode, expert parallelism, multi-node).
It is still an LLM workload: same modality, same OpenAI-compatible chat endpoint, same traffic axes (concurrency, ISL, OSL). The differences from metrumbench-llm are the load source (synthetic token sequences, no prompt library) and the metric focus (tokens per second per GPU, plus architecture dimensions such as TP, EP, and disaggregation).
Results appear on the Reporting page after a job completes.
What it measures and when to use it
InferenceX drives a running model server with synthetic chat traffic at a concurrency you choose, then reports GPU-normalized throughput and latency percentiles. Use it to answer:
- How many tokens per second per GPU can this model serve on this hardware?
- How does serving efficiency compare across H100, H200, and B200 for the same model and recipe?
- What is the latency profile (TTFT, TPOT, end-to-end) under a fixed synthetic ISL/OSL grid?
- How do advanced layouts (tensor parallelism, expert parallelism, disaggregated prefill/decode, multi-node) change efficiency?
Use InferenceX when you need:
- GPU-normalized throughput for hardware-to-hardware comparison
- Results in a format aligned with SemiAnalysis-style industry benchmarks
- Dimensions for disaggregated prefill/decode, expert parallelism, or multi-node setups
- Synthetic fixed-length ISL/OSL with no prompt variability
Use metrumbench-llm when you need real prompt content, prompt libraries, multi-replica load-balanced endpoints, ramp-up steady-state analysis, or richer first-party client-side streaming metrics. Full metric definitions and architecture notes live in InferenceX Methodology.
Step 1: create the job
Create an InferenceX job pinned to a project, model, framework version, and target server. Name jobs so the traffic shape is obvious, for example ix-<gpu>-<model>-isl<isl>-osl<osl>-c<concurrency>.
Step 2: set the traffic shape
InferenceX uses synthetic random token sequences. The shape is defined by job parameters (not a prompt library):
| Argument | What it means | How to choose |
|---|---|---|
concurrency | Maximum simultaneous in-flight requests. | Start at 8 or 64 depending on model size; raise to find the per-GPU plateau. |
input_sequence_length | Synthetic input length in tokens (random dataset). | Match the ISL you want to compare across hardware. Common grids: 128, 512, 2048. |
output_sequence_length | Synthetic output length in tokens. | Match the OSL you want to compare. Common grids: 128, 256, 512. |
num_requests | Total prompts to send. | Hundreds for stable numbers. Too few requests make percentiles noisy. |
random_range_ratio | Width of the random length distribution around ISL/OSL. 1.0 is fixed length (matches the upstream default for controlled grids). | Leave at 1.0 unless you intentionally want length variance. |
Daemon-managed paths (result_dir, result_filename) and the tool checkout/venv are set by the agent. Do not configure them yourself.
The benchmark targets a single endpoint with replicas = 1 and tensor parallelism equal to the GPU count on the selected hardware. If you need multi-replica load-balanced benchmarks, use metrumbench-llm instead.
Step 3: run and ingest
Once the job is queued on a server with a healthy agent, the daemon:
- Bootstraps the pinned InferenceX checkout and shared venv.
- Starts the model server from the workload's framework and version.
- Runs the upstream serving benchmark against the OpenAI-compatible endpoint.
- Processes the result file and POSTs aggregate metrics to
ingest_inferencex_results.
You do not call ingest by hand in normal operation. The agent owns that step. If ingestion fails, the agent retries from a pending record on the next cycle.
Step 4: read the results
Filter the Reporting page to the InferenceX job. Headline metrics:
| Metric | What it means | What to watch |
|---|---|---|
| Tokens per GPU | Total tokens (input + output) per second per GPU. | Primary efficiency number for hardware comparison. |
| Output tokens per GPU | Generated tokens per second per GPU. | Decode-side serving efficiency. |
| Input tokens per GPU | Prefill tokens per second per GPU. | Prefill-side efficiency; rises with ISL. |
| Mean / p99 TTFT | Time to first token, average and tail (seconds). | Track p99 for latency SLAs under load. |
| Mean TPOT | Average time per output token (seconds). | User-perceived tokens/sec per request is about 1 / TPOT. |
| Mean E2E latency | End-to-end request latency (seconds). | Full request cost including queueing and generation. |
| Mean interactivity TPS | Interactive tokens per second (user-perceived throughput). | Complements TPOT for interactivity studies. |
Architecture dimensions recorded with each run (when applicable): hardware, framework, inference precision, speculative decoding, ISL/OSL/concurrency, tensor parallelism (TP), expert parallelism (EP), data-parallel attention, disaggregated prefill/decode GPU counts and workers, multi-node flag, container image, and model id. Full column definitions are in InferenceX Methodology.
InferenceX vs metrumbench-llm
| Aspect | metrumbench-llm | InferenceX |
|---|---|---|
| Workload | Real prompts from a prompt library | Synthetic fixed-length token sequences |
| Throughput unit | Tokens per second (aggregate) | Tokens per second per GPU |
| TTFT measurement | Client-side SSE stream timing | Reported from tool output |
| Ramp-up | Supported (gradual concurrency increase) | Not applicable |
| Disaggregated serving | Not recorded as dimensions | Prefill/decode GPU and worker breakdown |
| Multi-node | Not explicitly recorded | Recorded as a dimension |
| Best for | Real-world serving with prompt diversity | Hardware efficiency and GPU-normalized comparison |
When comparing the two tools, hold model, framework, engine args, hardware, ISL, OSL, concurrency, and streaming fixed. Vary only the tool. Treat absolute number parity as directional: synthetic tokens and per-GPU normalization are not the same measurement as real-prompt aggregate throughput.
Common pitfalls
- Comparing aggregate tokens/sec to tokens/sec per GPU. Multiply or divide by GPU count before ranking hardware, or always report the InferenceX per-GPU columns.
- Too few requests. Percentiles (p90, p99, p999) need enough samples. Use hundreds of requests for numbers you will publish.
- Leaving length variance on when you want a fixed grid. Set
random_range_ratioto1.0for exact synthetic ISL/OSL. - Expecting prompt-library behavior. InferenceX does not use real prompts. For production traffic shape or domain-specific prompts, use
metrumbench-llm. - Skipping engine args validation. Bad
max_model_lenor tensor-parallel size fails before any request is sent. Start from a verified recipe; see Choosing engine arguments.