User Guide

This guide walks through the workflows that make up day-to-day work in Metrum Insights. Each section is task-oriented (e.g. "how do I run a benchmark", "how do I evaluate quality with KYAI") and links to the Feature Reference for parameter-level detail.

If you haven't run anything yet, do the Quickstart first. It gets you to a completed benchmark in 15 minutes. This guide assumes you've made it that far.

Sections:

• Benchmark building blocks
• Every argument explained
• Running benchmarks
• Choosing engine arguments
• LLM benchmarking (metrumbench-llm)
• ASR benchmarking (metrumbench-asr)
• VLM benchmarking (metrumbench-vlm)
• Image generation benchmarking (metrumbench-imagegen)
• KYAI - qualitative evaluation
• GenAI-Perf benchmarking
• InferenceX benchmarking
• KV cache offload benchmarking
• Reading results on the Reporting page
• Understanding key metrics
• Server management
• When a run fails
• Email notifications
• Telemetry
• Leaderboard

The sections LLM benchmarking, ASR benchmarking, VLM benchmarking, Image generation benchmarking, KYAI, GenAI-Perf benchmarking, and InferenceX benchmarking are self-contained, step-by-step guides for each workload type.

---

Benchmark building blocks

Before the workload guides, here is the vocabulary. Every benchmark in Metrum Insights is built from five nested objects. Once these five click, every screen and every API call reads the same way.

Object	Plain-language meaning	Example
Project	A named folder for one benchmarking question. It holds everything below it.	llama-70b-h100-throughput
Workload	One thing you want to measure. It is a single model, served by a single framework version, driven by one benchmark tool. Change the model, framework, version, quantization, or tool, and you have a new workload.	Llama 3.1 70B on vLLM 0.20.0, driven by metrumbench-llm
Scenario	One traffic shape applied to a workload. It sets how many requests run at once, how long the inputs and outputs are, and how many requests to send.	c8-isl512-osl256
Run	One execution of a workload across its scenarios, on a server you choose.	Run 1 of the workload above
Job	One scenario executing inside a run. A workload with 6 scenarios produces 6 jobs per run.	The c8-isl512-osl256 job

Read it as a sentence: a project groups workloads; each workload is measured under one or more scenarios; executing a workload creates a run; each scenario in that run becomes a job.

The tool that generates load is chosen per workload, based on the kind of model. This guide covers the three first-party tools plus InferenceX, a third-party LLM serving benchmark:

Tool	Workload type	What it drives
metrumbench-llm	Text language models (LLM)	An OpenAI-compatible chat or completion endpoint
metrumbench-asr	Speech-to-text models (ASR)	An OpenAI-compatible audio transcription endpoint
metrumbench-vlm	Vision-language models (VLM, image plus text)	An OpenAI-compatible chat endpoint with image payloads
inferencex	Text language models (LLM), synthetic load	An OpenAI-compatible chat endpoint, with GPU-normalized metrics

The first-party tools send real traffic to a running model server, measure every request, and write the results back to the platform. InferenceX uses synthetic fixed-length token sequences and reports throughput per GPU. The Performance Methodology and InferenceX Methodology explain exactly how each metric is captured.

---

Every argument explained

This section defines every value you can set, in plain language, so nothing on the workload card or in an API call is a mystery. The three workloads share most arguments. Anything specific to one workload is called out again inside that workload's guide.

Arguments live at two levels. Workload arguments describe what is being served. Scenario arguments describe the traffic sent to it.

Workload arguments

Argument	UI label	What it means	How to choose
Model	Model	The AI model to benchmark, picked from the registered catalog.	Pick the exact model you plan to deploy.
Framework	Framework	The serving engine that runs the model: vLLM, SGLang, or TensorRT-LLM.	Start with vLLM. See Picking a framework.
Version	Version	The framework version, which also selects the container image.	Use a stable version unless you are specifically testing an upgrade.
Quantization	Quantization	The numeric format the model weights are served in (fp16, bf16, fp8, int8, awq, gptq). Smaller formats use less memory and run faster, sometimes at a quality cost.	Match the recipe for your model and GPU. See Quantization under engine arguments.
Engine args set	Engine args	A named preset of framework startup flags, for example tensor parallelism, KV cache dtype, and maximum context length.	Do not guess these. See Choosing engine arguments.
Tool	(set automatically)	The benchmark client: metrumbench-llm, metrumbench-asr, or metrumbench-vlm.	Selected automatically from the model type.

Scenario arguments (shared by all three workloads)

Argument	API field	What it means	Typical values
Concurrency	p_concurrency	How many requests are in flight at the same time. This is the single most important knob for throughput and latency.	1, 8, 32, 64
Input sequence length (ISL)	p_input_sequence_length	The target size of each input, in tokens. For LLM and VLM this is the prompt length. It models how large your real inputs are.	128 to 4096
Output sequence length (OSL)	p_output_sequence_length	The target number of tokens the model should generate per request.	128 to 1024
Requests per scenario	p_num_requests	How many total requests to send in the scenario. More requests give more stable averages.	10 for a smoke test, hundreds to thousands for real numbers
Max tokens	p_max_tokens	A hard cap on generated tokens per request. When left unset, the tool uses the OSL value.	Usually leave equal to OSL
Streaming	p_streaming and the Streaming toggle	Whether the response is consumed token by token (server-sent events) or waited for as one final body. Streaming is required to measure Time To First Token.	On for latency studies

A single token is roughly three-quarters of an English word. So an ISL of 512 tokens is about 380 words of prompt, and an OSL of 256 is about 190 words of answer.

Tool-level arguments (advanced, usually left at defaults)

The platform sets these to safe defaults. You rarely change them, but here is what each one does.

Argument	Applies to	What it means
mode	LLM	chat wraps the prompt in a messages array with a system message (the normal case). completion sends the raw prompt to the legacy completions endpoint.
temperature	LLM, VLM	Sampling randomness. Benchmarks keep this low and fixed so runs are repeatable.
ramp_up_seconds	LLM, VLM	A warm-up window during which concurrency climbs from 1 to the target. Metrics from this window are discarded, so the reported numbers reflect steady state, not cold start.
request_timeout	all	Maximum seconds to wait for a single request before it is counted as failed.
connect_timeout	all	Maximum seconds allowed to open the TCP connection.
job_timeout_seconds	all	Hard wall-clock limit for the whole job.
num_images_batch	VLM	Number of images attached to each request.
image_detail	VLM	low sends the image at reduced fidelity (fewer image tokens, faster) or high sends full fidelity (more accurate, slower). Only these two values are valid.
input	ASR	Path or URL to the JSONL manifest listing the audio files to transcribe.
ground_truth	ASR	Path or URL to reference transcripts. When present, the tool computes accuracy (Word Error Rate and Character Error Rate).
response_format	ASR	The transcription response shape: verbose_json, json, text, srt, or vtt.

---

Running benchmarks

The core unit of work is a project containing one or more workloads. A workload describes what to benchmark (model + framework + scenario matrix); the project organizes them and produces runs when executed.

Single-workload runs (the Quickstart path)

The simplest path is a project with one workload, executed against one server. The Quickstart covers this end-to-end. The shorthand:

1. Projects -> New Project.
2. Fill in project name + visibility (the form default in the live UI is Organization).
3. Configure one workload card (Model, Framework, Version, Concurrency chips, ISL chips, OSL chips, Requests per scenario, Streaming).
4. Pick a server.
5. Save and Execute.

Save and Execute is only available when there's exactly one workload and a server is selected. Multi-workload projects save as drafts via Save Project and execute via a separate flow ~ see below.

Multi-workload projects

Use multi-workload projects when you want to compare several related configurations as a single deliverable (e.g. "Llama 70B on vLLM, SGLang, and TRT-LLM, at three concurrencies each").

1. Projects -> New Project.
2. Add the first workload card.
3. Click Add Workload to add more cards. Each card has its own model, framework, scenario matrix, and (optionally) server. Each card also offers Add Benchmark and Add KYAI Run buttons for adding additional benchmark or quality-evaluation steps.
4. Click Save Project (not "Save and Execute"). The project is saved as a draft.
5. From the project detail page, trigger runs per workload.

Multi-workload projects are useful for keeping related work together ~ every run from the project lands under the same Runs tab, and the Reporting page can filter to "this project" for cross-workload comparison.

Scenario matrices

Within a workload, concurrency, ISL, and OSL chips multiply out:

Concurrency	ISL	OSL	Scenarios
1	512	256	1
1, 8, 32	512	256	3
1, 8, 32	512, 2048	256	6
1, 8, 32	512, 2048	256, 512	12

Each scenario becomes one job in the run, named c<concurrency>-isl<isl>-osl<osl>. A 12-scenario run produces 12 rows in the Jobs table, run sequentially on the selected server. Start small ~ running a 24-scenario run on a 70B model can take an hour.

Available chip values in the workload card today are: concurrency 1, 4, 8, 16, 32, 64, 128; ISL 128, 256, 512, 1024, 2048; OSL 64, 128, 256, 512.

Picking a framework

Framework	Best for
vllm	General-purpose; widest model coverage; most portable
sglang	High-throughput serving with structured output
trt-llm	NVIDIA-only; highest peak throughput on H100/H200/B200

Defaults: start with vLLM. Add SGLang and TRT-LLM when comparing peak performance. Other framework codes (tgi, llama-cpp, ollama, dynamo, openai) are also registered in the catalog ~ see Feature Reference - Other registered frameworks for the support level of each.

Engine arguments (advanced)

Engine arguments are the flags passed to the serving framework when it starts the model. They decide how much GPU memory the model uses, how long a context it can handle, how many GPUs it spreads across, and how requests are batched. Getting them wrong is the most common cause of a failed or misleading run.

Do not pick these by guessing. The platform ships verified presets sourced from the official vLLM and SGLang recipes, matched to specific models and hardware. The full method for choosing them is its own section: Choosing engine arguments. Read it before you launch anything larger than a smoke test.

---

Choosing engine arguments

This is the section that saves you from failed runs. Engine arguments are the startup flags handed to the serving framework (vLLM, SGLang, or TensorRT-LLM). They are grouped into a named engine args set that a workload points at.

Why you must not guess

Every model plus GPU combination has a known-good set of flags. Choosing flags at random has real consequences:

• The server refuses to start. A max_model_len larger than the model supports, or a tensor-parallel size that does not divide the number of attention heads, fails at launch. The job then fails before a single request is sent.
• The GPU runs out of memory. Set gpu_memory_utilization too high, or max_num_batched_tokens too large for the card, and the model out-of-memories (OOM) partway through the run.
• The numbers are quietly wrong. A context length that cannot hold your input plus output silently truncates requests, so throughput looks good but the model was never doing the work you measured.

The fix is simple: start from a recipe that was verified for your model and your hardware, then change one thing at a time.

Start from the official recipes

Two upstream sources publish tuned launch settings for popular models on common GPUs. Read the page for your model before you build an engine args set.

• vLLM recipes: https://recipes.vllm.ai/ - per-model pages with the exact vllm serve flags for each GPU class (H100, H200, B200, GB200, and others), including tensor parallelism, KV cache dtype, and context length.
• SGLang cookbook: https://lmsysorg.mintlify.app/cookbook/intro - the equivalent for SGLang, with per-model launch commands and hardware notes, including AMD MI300X-class parts.

Find your exact model on the matching page, read the recommended command for the GPU you are running, and copy the flags into an engine args set. This is the difference between a recipe and a guess.

The platform already ships verified recipes

You often do not need to build a set by hand. Metrum Insights seeds verified recipe argsets: named presets whose values were copied directly from the vLLM recipes and SGLang cookbook pages, keyed to a specific model, GPU, framework version, and tensor-parallel size. The set code encodes exactly what it is for. For example:

Set code	Model	Hardware	Parallelism
vllm-llama31-8b-instruct-h100-h200-trillium-xeon6-tp1	Llama 3.1 8B Instruct	H100, H200, Trillium, Xeon6	TP1
vllm-llama33-70b-instruct-h100-h200-tp8	Llama 3.3 70B Instruct	H100, H200	TP8
vllm-llama33-70b-instruct-b200-gb200-tp1-fused	Llama 3.3 70B Instruct	B200, GB200	TP1, fused compilation

Verified argsets are seeded today for the Llama, DeepSeek, Qwen 3.5 and 3.6, Gemma, GLM, GPT-OSS, Kimi, Mistral, and Nemotron families. When a preset exists for your model and GPU, pick it. Its values already match the upstream recipe, so you avoid the whole guessing problem.

How to choose, step by step

1. Identify your model and GPU. For example, Llama 3.3 70B Instruct on eight H100s.
2. Look for a verified argset that matches. Filter the engine args sets for your model and hardware. If one exists (for example vllm-llama33-70b-instruct-h100-h200-tp8), use it and skip to step 6.
3. If none matches, open the recipe. Go to recipes.vllm.ai (or the SGLang cookbook for SGLang) and open your model's page.
4. Read the flags for your GPU. Recipes list settings per GPU class. Take the tensor-parallel size, KV cache dtype, context length, and any model-specific flags from the row that matches your card.
5. Build an engine args set from those values. Create a new set (via the API today) with those flags. Give it a descriptive code, such as vllm-<model>-<gpu>-tp<n>.
6. Sanity-check three things before you launch:
   • Tensor parallelism divides the GPU count. tensor_parallel_size must equal the number of GPUs you are giving the model (or evenly divide it). TP8 needs 8 GPUs.
   • Context length covers your traffic. max_model_len must be at least your largest ISL plus OSL. If you benchmark isl2048-osl1024, max_model_len must be 3072 or more.
   • Memory headroom exists. Leave gpu_memory_utilization around 0.90. Lower it if the server is shared or if you see OOM.
7. Run one smoke scenario first. Use concurrency=1, a tiny ISL and OSL, and num_requests=2. If the server starts and the job completes, your engine args are valid. Then run the full matrix.

The arguments you are most likely to set

Argument	vLLM key	SGLang key	What it controls	Rule of thumb
Tensor parallelism	tensor_parallel_size	tp_size	How many GPUs the model is split across.	Set to the number of GPUs the model needs to fit. Large models need more.
Context length	max_model_len	context_length	The longest input-plus-output the server will accept.	At least the largest ISL plus OSL in your scenario matrix.
KV cache dtype	kv_cache_dtype	runtime cache dtype	The numeric format of the attention cache. fp8 roughly halves cache memory.	Use the value the recipe specifies. Test it as a separate set, since it can affect quality.
GPU memory fraction	gpu_memory_utilization	mem_fraction_static	The share of GPU memory the server may claim.	Around 0.90. Lower on a shared box.
Batch sizing	max_num_seqs, max_num_batched_tokens	scheduler flags	How aggressively requests are batched together.	Take from the recipe. Raising max_num_batched_tokens can lift throughput if it plateaus early.
Prefix caching	enable_prefix_caching	radix cache flags	Reuse of shared prompt prefixes across requests.	On for repeated-prefix traffic, such as a fixed system prompt.

Model-server replicas

Tensor parallelism sets how many GPUs one model server uses (tensor-parallel size times data-parallel size). When that footprint is smaller than the GPUs on the host, Metrum fills the rest by running several model-server replicas on the same machine, so a benchmark uses the whole box instead of leaving GPUs idle.

• Replica count is the total GPUs divided by the GPUs per replica, where GPUs per replica is TP times DP. Eight GPUs with a TP2 recipe (DP left at its default of 1) run four replicas; the same eight GPUs with a TP8 recipe run one. If you also set DP2 alongside TP2, each replica now needs 4 GPUs (TP2 × DP2), so those same eight GPUs run two replicas instead of four.
• Each replica is isolated. It is pinned to its own GPU set and listens on its own port (the base port plus the replica index), so replicas never share GPUs or collide on a port.
• Load is spread across replicas. The benchmark tool sends the scenario's traffic across every replica endpoint, so the reported throughput reflects the full host, not a single replica.

Quantization

Quantization is the numeric format the model weights are served in. It is set by the Quantization field on the workload card, but treat it as part of the engine configuration: the recipe for your model and GPU already specifies the format it was tuned for, so match it rather than picking one on its own.

Quantization	Quality impact	Throughput / memory impact	Typical use
fp16	Baseline	Baseline	Reference baseline
bf16	About identical to fp16	Same	When the hardware prefers bf16
fp8	Small	About 1.7x throughput, about 0.5x memory	Production default on H100 and newer
int8	Noticeable	About 2x throughput, about 0.5x memory	Cost-optimized deployments
awq / gptq	Small to moderate	About 3x throughput, about 0.25x memory	Memory-constrained scenarios

tip

Weight quantization and KV cache dtype are separate knobs. fp8 weights and an fp8 kv_cache_dtype are set independently, and each can affect quality. Change one at a time.

warning

If quality matters as much as throughput, pair any quantization change with a KYAI evaluation on the same endpoints, so a throughput gain does not hide a quality regression.

Compare recipes cleanly

When you want to compare two settings, make each one its own engine args set and its own workload, and keep the scenarios identical. Never hand-edit a command line for a single run and then report it as the same workload. See Compare vLLM Versions And Settings and Command Templates for worked examples, including FP8 KV cache, long context, CPU offload, and MoE-style configurations.

A tuning change is a win only if it improves the primary metric without raising the failure rate, memory pressure, startup failures, or quality regressions. Pair an engine-args change with a KYAI evaluation when quality matters, so a throughput gain does not hide a quality loss.

---

LLM benchmarking (metrumbench-llm)

This is the most common workload. It measures how fast a text language model serves generation requests under load, using the metrumbench-llm tool. If you followed the Quickstart, you have already run one. This section is the complete reference.

What it measures and when to use it

metrumbench-llm drives an OpenAI-compatible chat or completion endpoint with real prompts, at a concurrency you choose, and records the timing of every request. Use it to answer questions like:

• How many tokens per second can this model serve on this GPU?
• What is the latency my users would feel at 8, 32, and 64 concurrent requests?
• How much faster is fp8 than bf16 on the same hardware, and does quality hold?
• Which serving framework, vLLM or SGLang, is faster for my model?

The headline metrics are throughput (tokens per second), TTFT (time to first token), and TPOT (time per output token). They are defined in Understanding key metrics.

For an NVIDIA-reference baseline or exact synthetic ISL/OSL, see GenAI-Perf benchmarking.

Step 1: create the workload (Web UI)

1. Go to Projects, then New Project (or open an existing project and use Add Workload).
2. Give the workload a clear name, for example llama-70b-vllm-bf16.
3. Model: pick the language model from the dropdown. The list is the registered model catalog.
4. Framework: choose vLLM to start. See Picking a framework for when to use SGLang or TensorRT-LLM.
5. Version: pick a stable framework version.
6. Quantization (optional): leave at the model default, or pick fp8 to test a faster, smaller format. See Quantization.
7. Engine args: attach the verified recipe set for your model and GPU. This step matters; read Choosing engine arguments first.

Step 2: define the scenario matrix

In the workload card, the Scenario Matrix uses chip buttons. Toggle the values you want on each axis, and the platform multiplies them out.

• Concurrency chips: 1, 4, 8, 16, 32, 64, 128.
• Input Sequence Length (ISL) chips: 128, 256, 512, 1024, 2048.
• Output Sequence Length (OSL) chips: 64, 128, 256, 512.
• Requests per scenario: total requests fired per scenario (default 10; raise it for stable numbers).
• Streaming: leave on to measure TTFT.

Selecting concurrency 1, 8, 32 with one ISL and one OSL produces three scenarios. Each becomes one job named c<concurrency>-isl<isl>-osl<osl>, for example c8-isl512-osl256. Click Show details to preview the rolled-out scenarios. Start small: a 24-scenario matrix on a 70B model can take an hour.

Match the ISL and OSL to your real traffic. If your production prompts are 4k tokens and you benchmark at 512, your numbers will be optimistic.

Step 3: run it

For a single-workload project with a server selected, click Save and Execute. The run starts immediately and you land on the Run Detail page, which shows one job per scenario moving through pending, queued, running, completed. For multi-workload projects, click Save Project, then launch runs from the project detail page. See Running benchmarks for the difference.

Step 4: read the results

Once a job completes, open the Reporting page, filter to your run, and read throughput, TTFT P50 and P99, and TPOT. Open the Concurrency chart tab to find the saturation point: throughput rises then flattens, and P99 latency stays flat then climbs. The concurrency at the elbow is the sweet spot for this model on this hardware.

The API path

The same workload and scenarios can be created over the API. See metrumbench-llm Workload And Scenario and Project Lifecycle for the full API reference.

LLM-specific arguments

Beyond the shared scenario arguments, metrumbench-llm exposes a few tool-level settings. They are defaulted for you; change them only with a reason.

Argument	What it means	Default behavior
mode	chat sends prompts through the chat completions endpoint with a system message. completion uses the legacy completions endpoint.	chat
streaming	Consume the response as a server-sent event stream so TTFT can be measured chunk by chunk.	On
ramp_up_seconds	Warm-up window where concurrency climbs to the target; warm-up metrics are discarded to report steady state.	Set by the platform
temperature	Sampling randomness, kept low and fixed for repeatable runs.	Low fixed value
max_tokens	Hard cap on generated tokens per request. When unset, OSL is used.	Equals OSL

Token counts in the results come from the model server's own usage response, not from local tokenization, so they reflect exactly what the server did.

Common pitfalls

• Concurrency too high for the model. Concurrency 128 on a 70B model with 80 GB of memory will OOM. Start at 8, find saturation, then push higher.
• Context too short for the scenario. If your engine args set a max_model_len below ISL plus OSL, requests are truncated. See Choosing engine arguments.
• Too few requests. Ten requests per scenario is a smoke test, not a measurement. Use hundreds to thousands for numbers you will report.

---

ASR benchmarking (metrumbench-asr)

ASR stands for Automatic Speech Recognition: models that turn audio into text, such as Whisper. The metrumbench-asr tool measures both how fast a transcription endpoint runs and, when you provide reference transcripts, how accurate it is.

UI status in v4.0: ASR execution is wired through the API. The dedicated ASR launch surface in the workload card is being staged and is not yet exposed in the live UI. Submit ASR runs via the API today.

Supported models and frameworks

ASR benchmarking is intentionally scoped to one model family and one serving framework:

Dimension	Supported	Notes
Model family	Whisper only (OpenAI Whisper family, e.g. openai/whisper-large-v3)	This is the only ASR model family seeded on the platform. Other speech-to-text models are not supported.
Framework	vLLM only (framework_code: vllm)	vLLM serves the Whisper transcription endpoint that metrumbench-asr targets. SGLang and TensorRT-LLM are not productized for ASR.

If you pass any other model family or framework for an ASR workload, the run will not resolve a valid command template. Keep p_model_code on a Whisper model and p_framework_code set to vllm.

What it measures and when to use it

metrumbench-asr uploads audio files to an OpenAI-compatible transcription endpoint (/v1/audio/transcriptions) and records timing and accuracy for each file. Use it to answer:

• Can this model transcribe faster than real time on this hardware?
• How accurate is it, in Word Error Rate, on my kind of audio?
• How does accuracy trade off against speed across quantizations or frameworks?

The headline metrics are RTF (Real-Time Factor), WER (Word Error Rate), and CER (Character Error Rate). They are defined in ASR-specific metrics below.

Step 1: pick or prepare a dataset

An ASR benchmark needs audio to transcribe. Two built-in datasets are seeded:

Dataset code	What it is
librispeech-test-clean	The LibriSpeech "test-clean" set: clear, read English speech. The standard baseline.
librispeech-test-other	The LibriSpeech "test-other" set: noisier, harder speech. Use it to stress accuracy.

The audio files are stored centrally and restored to the benchmark worker during agent bootstrap, so you do not upload them yourself when using a built-in dataset.

To use your own audio, supply an input manifest: a JSONL file where each line describes one audio file.

{"id": "sample-001", "path": "/data/audio/sample-001.flac", "format": "flac", "duration": 12.5}
{"id": "sample-002", "url": "https://storage.example.com/sample-002.mp3", "format": "mp3", "duration": 8.3}

• id is required and must be unique.
• Provide either path (a local file on the worker) or url (downloaded and cached before the run).
• format sets the upload MIME type. Supported: MP3, WAV, WebM, OGG, M4A/MP4, FLAC.
• duration (seconds) is needed to compute RTF. Without it, RTF is skipped for that file.

Step 2: prepare ground truth (optional, for accuracy)

Accuracy metrics (WER and CER) are computed only when you provide a ground truth file: a JSONL where each line pairs an audio id with its reference transcript.

{"id": "sample-001", "transcript": "the quick brown fox jumps over the lazy dog"}
{"id": "sample-002", "transcript": "a second reference transcription"}

The built-in LibriSpeech datasets ship with reference transcripts, so WER and CER are available out of the box. Without ground truth, you still get full speed metrics, just no accuracy.

Step 3: create the workload and scenario (API)

The same ASR workload and scenarios can be created over the API. See metrumbench-asr Workload And Scenario and Project Lifecycle for the API reference.

ASR-specific arguments

Argument	What it means	How to choose
concurrency	Audio files transcribed at the same time.	Start at 8, raise to find saturation.
num_requests	Total transcriptions to send. Audio samples cycle round-robin if requests exceed the sample count.	Hundreds for stable numbers.
input	The audio manifest (built-in dataset resolves this for you).	Use a built-in dataset unless testing your own audio.
ground_truth	Reference transcripts for accuracy.	Provide it to get WER and CER.
response_format	The transcription response shape: verbose_json, json, text, srt, or vtt. verbose_json returns the richest data.	Leave at the default unless you need a specific format.

There is no ISL, OSL, or streaming for ASR. The traffic shape is defined by the audio itself and by concurrency.

ASR-specific metrics

Metric	What it means	Good value
RTF (Real-Time Factor)	Inference time divided by audio duration.	Below 1.0 means faster than real time. 0.1 means ten times faster than real time.
WER (Word Error Rate)	The fraction of words that must be inserted, deleted, or substituted to match the reference.	Lower is better. 0% is perfect. It can exceed 100% if the model produces far more words than the reference.
CER (Character Error Rate)	The same idea at the character level. More granular than WER.	Lower is better.
Audio per second	Seconds of audio processed per wall-clock second across all concurrency.	Higher is better.

WER and CER appear only when ground truth is provided. RTF appears only when the manifest includes duration.

Common pitfalls

• No duration in the manifest. RTF is skipped silently. Add duration to every line.
• Expecting WER without ground truth. Accuracy needs reference transcripts. Use a built-in dataset or supply your own ground truth file.
• Reading accuracy on the wrong audio. Test-clean flatters a model; test-other is closer to real-world noise. Report which set you used.

---

VLM benchmarking (metrumbench-vlm)

VLM stands for Vision-Language Model: models that take an image plus text and produce text, such as describing a photo or reading a chart. The metrumbench-vlm tool measures how fast a VLM serves these image-plus-text requests.

UI status in v4.0: VLM execution is wired through the API. The dedicated VLM launch surface in the workload card is being staged and is not yet exposed in the live UI. Submit VLM runs via the API today.

What it measures and when to use it

metrumbench-vlm sends the same kind of chat request as metrumbench-llm, but each request carries one or more images alongside the text prompt. Everything else, concurrency, timing, throughput, TTFT, and TPOT, works the same way. Use it to answer:

• How fast does my VLM serve image-plus-text requests under load?
• How much does image preprocessing add to latency?
• Is the server GPU-bound or CPU-bound when handling images?

Step 1: understand how images reach the model

Each prompt line in a VLM dataset has a text prompt and an image_urls array with one or more image references. There are two ways the image gets to the model:

• Base64 encoding (default): the tool loads the image, optionally resizes it, and embeds it directly in the request. This is the realistic path for most deployments.
• Server-side download: the request carries an image URL and the model server fetches it.

The built-in dataset metrumbench-vlm-4k-v1 provides a set of photographs with description prompts, ready to use. To bring your own images, supply a JSONL where each line has a prompt and an image_urls array (HTTP URLs, local paths, or file:// URIs).

Step 2: create the workload and scenario (API)

The same VLM workload and scenarios can be created over the API. See metrumbench-vlm Workload And Scenario and Project Lifecycle for the API reference.

VLM-specific arguments

Argument	API field	What it means	How to choose
Images per request	p_num_images_batch	How many images are attached to each request.	Usually 1. Raise it only to test multi-image reasoning.
Image detail	p_image_detail	low sends the image at reduced fidelity (fewer image tokens, faster). high sends full fidelity (more accurate, slower). These are the only two valid values.	Use high for accuracy-sensitive tests, low for throughput tests.
Input sequence length	p_input_sequence_length	The text portion of the prompt, in tokens. The image adds its own tokens on top.	Match your real prompts.
Output sequence length	p_output_sequence_length	Tokens generated per request.	Match your real answers.

Note on image detail: only low and high are valid. Earlier documentation mentioned a medium level; it does not exist. The tool can also resize oversized images to a maximum dimension before sending, which keeps image token counts predictable.

VLM-specific metrics

VLM reports the same core metrics as LLM (throughput, TTFT, TPOT, latency percentiles, token counts), plus image handling data:

• Image preprocessing time, included in TTFT. Loading, resizing, and encoding an image happens on the CPU on most paths.
• Image sizes and dimensions per request.

Because preprocessing is CPU work, watch the Hardware chart tab on the Reporting page. If CPU utilization is high while GPU utilization is low, you are preprocessing-bound, not inference-bound. Move to a server with more CPU cores, use low image detail, or resize images ahead of time.

Common pitfalls

• Blaming the GPU for a CPU bottleneck. High image throughput can be capped by CPU preprocessing. Check the CPU view before concluding the GPU is slow.
• Comparing low and high in one run. They produce different token counts and latencies. Make each its own scenario so the comparison is clean.
• Forgetting the image adds tokens. A "256 token" VLM prompt is 256 text tokens plus the image tokens. Budget context length accordingly in your engine args.

---

Image generation benchmarking (metrumbench-imagegen)

Image generation measures how fast a text-to-image model produces images under load, using the metrumbench-imagegen tool. It targets an OpenAI-compatible /v1/images/generations endpoint, collects image artifacts, and reports latency and throughput.

Metric definitions and serving details live in Performance Methodology → metrumbench-imagegen.

Supported frameworks

Framework	Role	Latest CUDA 13.0 (default)	Latest AMD / ROCm
vLLM-Omni (vllm-omni)	Omni image-generation path (vllm serve … --omni)	0.24.0	0.22.0 (amd-rocm)
SGLang (sglang)	Diffusion path (sglang serve / serve-diffusion)	0.5.15.post1	0.5.15.post1 (amd-mi300 / amd-mi355)

Default engine-args sets are single-GPU-safe:

Framework	Default set	Defaults
vLLM-Omni	vllm-omni-metrumbench-imagegen	tensor_parallel_size=1, usp=1, ring=1
SGLang	sglang-metrumbench-imagegen	tp_size=1, ulysses_degree=1, ring_degree=1

• TP — shards model weights across GPUs. Default 1.
• USP / Ulysses — sequence parallelism for diffusion. Default 1 (off).
• Ring — ring sequence parallelism. Default 1 (off).

You can alter, add, or remove engine args to match your hardware and model (for example changing TP from 1 to 2 on a multi-GPU node).

What it measures and when to use it

Use ImageGen when you need:

• Images per second and megapixels per second on a given GPU
• Generation latency (p50 / p95 / p99) under concurrency
• A fair comparison of vLLM-Omni vs SGLang diffusion on the same model family

Headline metrics: images/sec, megapixels/sec, latency, success rate.

Step 1: create the workload

1. Go to Projects → New Project (or open a project and Add Workload).
2. Under Workloads, open Image Generation and select MetrumBench ImageGen.
3. Target Hardware — select the server chip / GPU count.
4. Models — pick an image-generation model (for example Tongyi-MAI/Z-Image-Turbo).
5. Framework — pick vLLM-Omni or SGLang (diffusion).
6. Version — use the latest CUDA 13.0 default above unless you need ROCm.
7. Engine Args Set — use the ImageGen default set for that framework, or a verified set matched to your model and hardware. You can add, edit, or remove engine args (for example changing tensor_parallel_size or tp_size to match the GPU count).

The dataset (text2image-multiprompt-1024, 1024 prompts) is preselected automatically.

Step 2: configure the scenario matrix

Field	What it controls	Default when unset
Concurrency	Parallel image-generation requests	1
Images per request (n)	How many images each request asks for	1
Image size	Target resolution (for example 1024x1024)	1024x1024

Request count is set by the platform from concurrency (not a user field). See Performance Methodology → metrumbench-imagegen.

Images per request (n):

• vLLM-Omni: n > 1 is supported.
• SGLang: prefer n = 1. Multi-image requests rely on each diffusion pipeline broadcasting text conditioning to the expanded sample batch, which SGLang's shared infrastructure does not do automatically. Some models work (SD 3.5, FLUX); others fail (Qwen-Image, Z-Image-Turbo). This is a known SGLang upstream issue — see Performance Methodology.

ImageGen does not use ISL/OSL or streaming the way LLM workloads do.

Step 3: run and read results

1. Click Save or Save & Execute.
2. On the Reporting page, filter to the ImageGen workload.
3. Review per scenario:
   • Images per second and megapixels per second
   • Latency mean / p50 / p90 / p95 / p99
   • Success rate and failed / timeout counts

When comparing frameworks, hold model, hardware, dataset, concurrency, image size, and n fixed. Vary only the framework. For SGLang comparisons, keep n = 1 unless you know that model supports multi-image requests.

Common pitfalls

• Setting n > 1 on SGLang. Prefer n = 1 on SGLang; multi-image support is model-pipeline dependent (known upstream limitation).
• Comparing different image sizes on images/sec alone. Prefer megapixels/sec when resolutions differ.
• Reusing LLM engine-args sets. ImageGen sets are separate (vllm-omni-metrumbench-imagegen, sglang-metrumbench-imagegen).

The API path

The same ImageGen workload and scenarios can be created over the API. See metrumbench-imagegen Workload And Scenario and Project Lifecycle for the API reference.

---

KYAI

KYAI ("Know Your AI") evaluates inference quality — how accurate a model's outputs are, not how fast it serves them. It sends a fixed set of questions to the model, then scores each response against a ground truth using a platform-managed judge.

The headline metric is the KYAI Score (average correctness, 0.0 to 1.0). Generation latency is also captured and used as a tiebreaker on the leaderboard when scores are equal.

Full methodology: KYAI Methodology.

Step 1: create the project and pick KYAI

1. Go to Projects → New Project.
2. Enter a Project Name and set Visibility (Private or Team).
3. Under Workloads, open Quality Evaluation and click KYAI to add a workload card.

Step 2: configure the candidate model

Work through the card in order:

1. Target Hardware — select a server chip and GPU Count.
2. Models — pick the model to evaluate.
3. Framework — choose vLLM, SGLang, or TensorRT-LLM. If only one framework is compatible, it auto-selects and locks.
4. Version — pick the framework release. The latest compatible version auto-selects.
5. Build Profile — pick the hardware build/launch variant (for example Default (CUDA) or ROCm) when profiles are available.
6. Engine Args Set — pick a pre-configured template, or leave unset for the framework default. Custom org sets show "(custom)".

When editing engine args, ensure:

• Tensor parallelism (TP / tp) is set to the same value as the hardware GPU count. KYAI runs with replicas = 1 (no replica fan-out), so all GPUs belong to one server process.
• For default thinking / reasoning models (for example DeepSeek-R1 or any model that emits chain-of-thought blocks), add a reasoning parser to the engine args so the framework separates chain-of-thought from the final answer. Without it, thinking tokens mix into the response the judge scores.

Step 3: configure the dataset

KYAI evaluates against fixed question snapshots or custom uploads.

Built-in snapshots (100 questions each, preselected):

Dataset	Domain	Answer format
MMLU-Pro	Academic reasoning (14 disciplines)	Answer: X (A–J)
GPQA Diamond	Graduate-level expert science MCQ	Answer: X (A–D)
MATH-500	Competition mathematics	Final answer: <answer>
Python Codes 25K	Practical Python generation	Valid Python, no fences
HumanEval	Python function completion	Indented function body only
Quantum Mechanics	Physics chain-of-thought	Step-by-step + final answer
IFEval	Instruction-following precision	Satisfies all stated constraints

Custom dataset (user-uploaded): Upload a JSONL or CSV file with two mandatory fields — instruction (prompt sent to the candidate) and ground_truth_response (reference answer). You also provide two prompts: a generation system prompt and a judge prompt. There is no built-in prompt picker for custom uploads.

Step 4: run and read results

Click Save & Execute (or Add KYAI Run). The run progresses through two phases:

1. Generation — the candidate receives each prompt and produces a response. All outputs are stored verbatim.
2. Judging — the Metrum AI Judge scores each response against the ground truth. The judge does not know which model produced the response.

Filter the Reporting page to the KYAI workload. Review per evaluation:

• KYAI Score — mean correctness across the fixed question set (0.0 to 1.0). Errors count as zero.
• Per-prompt scores and reasoning — inspect low scores for failure modes.
• Leaderboard — candidates on the same dataset + same prompts ranked by mean score (latency as tiebreaker).

See KYAI Methodology for scoring rules, fail-closed behavior, and comparison keys.

The API path

KYAI workloads and runs can be created over the API. See KYAI Workload And Run and KYAI Evaluation for the full API reference.

GenAI-Perf benchmarking

GenAI-Perf is NVIDIA's third-party LLM load tool from the Triton Perf Analyzer ecosystem. It is still an LLM workload: same modality, same OpenAI-compatible chat endpoint, same scenario axes (concurrency, ISL, OSL, streaming). The differences from metrumbench-llm are the load source (synthetic token lengths only, no prompt library) and the serving topology (single endpoint, replicas = 1, TP = hardware GPU count).

Use GenAI-Perf when you want:

• An NVIDIA-reference baseline for throughput, latency, TTFT, and inter-token latency
• Exact synthetic ISL/OSL (mean with stddev 0) for controlled grids
• A second opinion beside metrumbench-llm on the same model-server recipe

Use metrumbench-llm when you need real prompt content, prompt libraries, multi-replica load-balanced endpoints, or richer first-party metrics. Full metric definitions, defaults, and known issues live in GenAI-Perf Methodology.

Step 1: create the project and pick GenAI-Perf

1. Go to Projects → New Project.
2. Enter a Project Name and set Visibility (Private or Team).
3. Under Workloads, open Text Generation and click GenAI-Perf to add a workload card.

Step 2: configure the workload

Work through the card in order:

1. Target Hardware — select a server chip and set GPU Count if needed.
2. Models — type or select HuggingFace model IDs and press Enter. Add multiple models for a multi-model run: every model shares the same engine args, scenario matrix, and server. A heads-up banner reminds you of that shared setup.
3. HuggingFace Token — if a model is gated, click + Add Token and paste your token (account-level).
4. Framework — pick vLLM, SGLang, or TensorRT-LLM. If only one framework is compatible, it auto-selects and locks.
5. Version — pick the framework release. The latest compatible version auto-selects.
6. Engine Args Set — pick a pre-configured template, or leave unset for the framework default. Custom org sets show "(custom)".
7. Build Profile — pick the hardware build/launch variant (for example Default (CUDA) or ROCm) when profiles are available.
8. GenAI-Perf Args — expand to edit tool args. Exclude with X, Add custom arg, or Save arg set team-wide after edits.
9. Framework Args (for example vLLM Args) — same edit/add/remove/save mechanics as tool args.
10. Dataset — GenAI-Perf is synthetic-only, so this field is hidden or shows a synthetic source.

Step 3: configure the scenario matrix

Field	What it controls
Streaming	Yes / No. Required for TTFT and inter-token latency.
Concurrency	Parallel requests. Presets include 1 through 8192.
Input Sequence Length	Synthetic prompt length in tokens (exact, stddev 0).
Output Sequence Length	Synthetic generation length in tokens (exact, stddev 0).

Every combination becomes one scenario. The preview panel shows codes such as c1-isl256-osl128.

Request count is not exposed in the UI; it defaults to max(200, concurrency × 10) per scenario (same shared default as metrumbench-llm).

Endpoint type is fixed to chat and the random seed is fixed to 0 for reproducibility.

High concurrency (≥ 512, especially 1024) needs ulimit -n 65536 on the worker.

The benchmark targets a single endpoint URL with replicas = 1 and TP = GPU count on the selected hardware. If you need multi-replica load-balanced benchmarks, use metrumbench-llm's multi-endpoint YAML instead.

Step 4: read results

Filter the Reporting page to the GenAI-Perf workload. Review per scenario:

• Output token throughput and request throughput
• TTFT mean / p50 / p95 / p99 (plus extended percentiles on the detail view)
• Inter-token latency mean / p95 / p99
• Request latency mean / p95 / p99
• Successful vs failed request counts
• Hardware tab: GPU power, utilization, memory, and CPU metrics captured by the agent during the run (telemetry views: v_telemetry_job_summary, v_telemetry_per_gpu_summary)

Streaming is required for useful TTFT and inter-token latency. Non-streaming still produces end-to-end latency and throughput.

When comparing GenAI-Perf to metrumbench-llm, hold model, framework, engine args, hardware, replicas, TP, ISL, OSL, concurrency, and streaming fixed. Vary only the tool. Treat absolute number parity as directional: GenAI-Perf uses synthetic tokens; metrumbench-llm uses prompt content.

The API path

GenAI-Perf workloads and scenarios can be created over the API through the same RPCs used by metrumbench-llm (only the p_tool_code changes to genai-perf). See metrumbench-llm Workload And Scenario and Project Lifecycle for the API reference.

---

InferenceX benchmarking

InferenceX is a third-party LLM serving benchmark integrated into Metrum Insights. It follows a SemiAnalysis-style synthetic serving methodology: fixed input and output token lengths, throughput normalized per GPU, and optional dimensions for advanced serving architectures (disaggregated prefill/decode, expert parallelism, multi-node).

It is still an LLM workload: same modality, same OpenAI-compatible chat endpoint, same traffic axes (concurrency, ISL, OSL). The differences from metrumbench-llm are the load source (synthetic token sequences, no prompt library) and the metric focus (tokens per second per GPU, plus architecture dimensions such as TP, EP, and disaggregation).

UI status in v4.0: InferenceX execution is wired through the control plane. The dedicated InferenceX launch surface in the workload card is being staged and is not yet exposed in the live UI. Results appear on the Reporting page once a job completes.

What it measures and when to use it

InferenceX drives a running model server with synthetic chat traffic at a concurrency you choose, then reports GPU-normalized throughput and latency percentiles. Use it to answer:

• How many tokens per second per GPU can this model serve on this hardware?
• How does serving efficiency compare across H100, H200, and B200 for the same model and recipe?
• What is the latency profile (TTFT, TPOT, end-to-end) under a fixed synthetic ISL/OSL grid?
• How do advanced layouts (tensor parallelism, expert parallelism, disaggregated prefill/decode, multi-node) change efficiency?

Use InferenceX when you need:

• GPU-normalized throughput for hardware-to-hardware comparison
• Results in a format aligned with SemiAnalysis-style industry benchmarks
• Dimensions for disaggregated prefill/decode, expert parallelism, or multi-node setups
• Synthetic fixed-length ISL/OSL with no prompt variability

Use metrumbench-llm when you need real prompt content, prompt libraries, multi-replica load-balanced endpoints, ramp-up steady-state analysis, or richer first-party client-side streaming metrics. Full metric definitions and architecture notes live in InferenceX Methodology.

Step 1: create the job

Create an InferenceX job pinned to a project, model, framework version, and target server. Name jobs so the traffic shape is obvious, for example ix-<gpu>-<model>-isl<isl>-osl<osl>-c<concurrency>.

Step 2: set the traffic shape

InferenceX uses synthetic random token sequences. The shape is defined by job parameters (not a prompt library):

Argument	What it means	How to choose
concurrency	Maximum simultaneous in-flight requests.	Start at 8 or 64 depending on model size; raise to find the per-GPU plateau.
input_sequence_length	Synthetic input length in tokens (random dataset).	Match the ISL you want to compare across hardware. Common grids: 128, 512, 2048.
output_sequence_length	Synthetic output length in tokens.	Match the OSL you want to compare. Common grids: 128, 256, 512.
num_requests	Total prompts to send.	Hundreds for stable numbers. Too few requests make percentiles noisy.
random_range_ratio	Width of the random length distribution around ISL/OSL. 1.0 is fixed length (matches the upstream default for controlled grids).	Leave at 1.0 unless you intentionally want length variance.

Daemon-managed paths (result_dir, result_filename) and the tool checkout/venv are set by the agent. Do not configure them yourself.

The benchmark targets a single endpoint with replicas = 1 and tensor parallelism equal to the GPU count on the selected hardware. If you need multi-replica load-balanced benchmarks, use metrumbench-llm instead.

Step 3: run and ingest

Once the job is queued on a server with a healthy agent, the daemon:

1. Bootstraps the pinned InferenceX checkout and shared venv.
2. Starts the model server from the workload's framework and version.
3. Runs the upstream serving benchmark against the OpenAI-compatible endpoint.
4. Processes the result file and POSTs aggregate metrics to ingest_inferencex_results.

You do not call ingest by hand in normal operation. The agent owns that step. If ingestion fails, the agent retries from a pending record on the next cycle.

Step 4: read the results

Filter the Reporting page to the InferenceX job. Headline metrics:

Metric	What it means	What to watch
Tokens per GPU	Total tokens (input + output) per second per GPU.	Primary efficiency number for hardware comparison.
Output tokens per GPU	Generated tokens per second per GPU.	Decode-side serving efficiency.
Input tokens per GPU	Prefill tokens per second per GPU.	Prefill-side efficiency; rises with ISL.
Mean / p99 TTFT	Time to first token, average and tail (seconds).	Track p99 for latency SLAs under load.
Mean TPOT	Average time per output token (seconds).	User-perceived tokens/sec per request is about 1 / TPOT.
Mean E2E latency	End-to-end request latency (seconds).	Full request cost including queueing and generation.
Mean interactivity TPS	Interactive tokens per second (user-perceived throughput).	Complements TPOT for interactivity studies.

Architecture dimensions recorded with each run (when applicable): hardware, framework, inference precision, speculative decoding, ISL/OSL/concurrency, tensor parallelism (TP), expert parallelism (EP), data-parallel attention, disaggregated prefill/decode GPU counts and workers, multi-node flag, container image, and model id. Full column definitions are in InferenceX Methodology.

InferenceX vs metrumbench-llm

Aspect	metrumbench-llm	InferenceX
Workload	Real prompts from a prompt library	Synthetic fixed-length token sequences
Throughput unit	Tokens per second (aggregate)	Tokens per second per GPU
TTFT measurement	Client-side SSE stream timing	Reported from tool output
Ramp-up	Supported (gradual concurrency increase)	Not applicable
Disaggregated serving	Not recorded as dimensions	Prefill/decode GPU and worker breakdown
Multi-node	Not explicitly recorded	Recorded as a dimension
Best for	Real-world serving with prompt diversity	Hardware efficiency and GPU-normalized comparison

When comparing the two tools, hold model, framework, engine args, hardware, ISL, OSL, concurrency, and streaming fixed. Vary only the tool. Treat absolute number parity as directional: synthetic tokens and per-GPU normalization are not the same measurement as real-prompt aggregate throughput.

Common pitfalls

• Comparing aggregate tokens/sec to tokens/sec per GPU. Multiply or divide by GPU count before ranking hardware, or always report the InferenceX per-GPU columns.
• Too few requests. Percentiles (p90, p99, p999) need enough samples. Use hundreds of requests for numbers you will publish.
• Leaving length variance on when you want a fixed grid. Set random_range_ratio to 1.0 for exact synthetic ISL/OSL.
• Expecting prompt-library behavior. InferenceX does not use real prompts. For production traffic shape or domain-specific prompts, use metrumbench-llm.
• Skipping engine args validation. Bad max_model_len or tensor-parallel size fails before any request is sent. Start from a verified recipe; see Choosing engine arguments.

---

KV cache offload benchmarking

KV cache offload measures how well a model server holds onto conversation history across multiple back-and-forth turns, by moving KV cache blocks off the GPU when it runs out of room and pulling them back when a conversation continues. It replays real multi-turn conversation datasets rather than single-shot prompts.

Use it to answer:

• Does offloading KV cache to CPU RAM or NVMe disk help or hurt time-to-first-token on a long multi-turn conversation?
• How much conversation history can this server retain before it has to evict and recompute?
• Which storage tier (GPU-only, CPU RAM, or NVMe SSD) gives the best latency/capacity trade-off for my hardware?

The engine underneath CPU and NVMe offload is LMCache, running as a standalone lmcache-server sidecar process alongside vLLM. The GPU tier does not use LMCache at all; it is plain vLLM serving off its own in-VRAM prefix cache.

Full methodology, including the sampling approach behind each metric and the dataset session-count ceiling that limits concurrency: KV Cache Offload Methodology.

Step 1: create the workload and pick a model

1. Go to Projects → New Project (or open a project and Add Workload).
2. Under Workloads, add a KV Cache Offload workload card.
3. Model — pick the language model to benchmark from the registered catalog.
4. HuggingFace Token — if the model is gated, click + Add Token and paste your token (account-level).

Step 2: pick the dataset

KV cache offload replays a real multi-turn conversation dataset, not a synthetic prompt library. Pick one from the Dataset dropdown:

Dataset code	What it is
sammshen	sammshen/lmcache-agentic-traces — real agentic multi-turn conversation traces.
tau2-bench-v1	Tau2-Bench (v1) — a multi-turn conversational benchmark dataset.

Each session in the dataset is one simulated multi-turn conversation; num_rounds (set in the scenario matrix, see Step 5) controls how many of its turns are actually replayed per session.

Both datasets hold a fixed, finite number of real sessions, and each session is sliced into fixed-length chunks of num_rounds turns — a session shorter than num_rounds produces no chunk at all. That means the number of independent conversations a dataset can supply at a given num_rounds is fixed and shrinks as num_rounds grows. Requesting a concurrency higher than that number does not scale further: the run silently replays whatever the dataset can actually supply, not the number you asked for. Check the session counts in your job's results (complete/incomplete/total sessions) before trusting a comparison at high concurrency — if the achieved count is lower than requested, every scenario in that comparison needs to hit the same ceiling to stay comparable.

Step 3: pick the KV Cache Storage Tier

This is the central choice for this workload. It sets where offloaded KV cache blocks live once GPU memory fills up:

Tier	Where cache lives	Needs a cache server?
GPU	Stays in vLLM's own in-VRAM prefix cache. No offload, no second process.	No
CPU RAM	Offloaded to a pinned system-RAM pool, managed by a standalone lmcache-server sidecar.	Yes
NVMe SSD	Offloaded to a small CPU staging pool plus a larger disk-backed pool on NVMe.	Yes

Picking CPU RAM or NVMe SSD automatically selects the matching vLLM engine args (below) and adds an LMCache Server Args panel for the cache-server sidecar. GPU needs neither.

Step 4: pick the build (NVIDIA default, or AMD)

For the CPU RAM and NVMe SSD tiers only, pick the hardware build:

• NVIDIA (default) — selected automatically. Use this unless you are running on AMD GPUs.
• AMD — switch to this explicitly if your target server has AMD ROCm GPUs. It selects the AMD-built lmcache-server image and the matching AMD-pinned vLLM profile instead of the NVIDIA ones.

The GPU tier has no build choice — it is plain vLLM and does not depend on this setting.

Framework and version: vLLM is currently the only framework supported for KV cache offload, and the latest version is used by default (0.28.0 as of this writing). Newer vLLM versions are added to the catalog as they are validated for this recipe; use the latest one shown unless you have a specific reason to pin an older version.

Step 5: configure vLLM args (all tiers)

These are the vLLM engine args editable for every tier:

Argument	What it controls	Default
gpu_memory_utilization	Share of GPU memory vLLM may claim.	0.80
max_model_len	Context length the server accepts. auto infers it from the model config.	auto
tensor_parallel_size	How many GPUs the model is split across.	1
reasoning_parser	Separates chain-of-thought from the final answer for reasoning models. Leave blank for non-reasoning models.	blank
trust_remote_code	Allows the model's own custom code (needed by some HuggingFace model repos).	true

On the CPU RAM and NVMe SSD tiers, two additional vLLM args wire the connection to the cache server sidecar: kv_connector (LMCacheMPConnector) and kv_role (kv_both). These are set automatically by the tier picker — leave them as-is. Changing kv_connector to point at a different connector implementation is possible but is an advanced, unsupported customization: the rest of this tool's wiring (the cache-server sidecar, its args panel, its command template) assumes LMCache specifically.

Step 6: configure LMCache Server args (CPU RAM and NVMe SSD tiers)

This panel only appears for the CPU RAM and NVMe SSD tiers — it configures the standalone lmcache-server sidecar process, separate from the vLLM args above.

CPU RAM tier defaults:

Argument	What it controls	Default
l1_size_gb	Size of the pinned RAM pool holding offloaded cache.	128
l1_init_size_gb	How much of the pool to allocate upfront (the rest grows lazily).	20
l1_use_lazy	Grow the RAM pool gradually instead of allocating it all at once.	true
eviction_policy	How cache is evicted when the pool fills.	LRU
eviction_ratio / eviction_trigger_watermark	How much to evict, and at what fill level eviction kicks in.	0.2 / 0.8
chunk_size	Granularity, in tokens, that cache is stored and evicted in.	256

NVMe SSD tier defaults (a small CPU staging pool plus a larger disk-backed pool):

Argument	What it controls	Default
l1_size_gb	Size of the small CPU staging pool in front of the disk. Deliberately much smaller than the CPU-tier's, since disk provides the real capacity.	32
l1_init_size_gb	Upfront allocation for that staging pool.	8
l2_path	Disk directory the overflow pool is written to.	/data/lmcache/l2
max_capacity_gb	Cap on the disk pool's size. 0 means unlimited (bounded only by the disk itself).	0
num_workers	Parallel I/O workers writing to and reading from disk.	4
use_odirect	Bypass the OS page cache for disk I/O (O_DIRECT). Off by default; only enable it if your storage and filesystem are confirmed to support it, since raw device or unusual filesystem setups can behave unpredictably with it on.	false
l2_store_policy / l2_prefetch_policy	How data is written to and read back from the disk pool.	default / default
l2_prefetch_max_in_flight	How many prefetch reads from disk can be outstanding at once.	8

Every value in both tables is editable — these are starting points, not fixed limits. Change one at a time and re-run a small smoke scenario before committing to a full matrix.

Step 7: configure the scenario matrix

The scenario matrix uses chip buttons, the same way concurrency, ISL, and OSL do elsewhere in the platform, with one addition:

Field	What it controls	Default when unset
Concurrency	Simultaneous conversation sessions in flight.	1
Input sequence length (ISL)	Target size of each turn's input, in tokens.	2048
Output sequence length (OSL)	Target number of tokens generated per turn.	256
Num rounds	How many back-and-forth turns to replay per conversation session.	25

Match num_rounds to how deep the conversations you care about actually go. Do not go below 5: for tau2-bench-v1 and any other non-sammshen dataset, the runner drops any conversation chunk shorter than 5 rounds outright, so a lower value produces zero virtual users instead of a valid first-turn-latency measurement. A value close to 5 stresses first-turn latency; a higher value stresses how well the offloaded tier retains and reuses earlier turns as the conversation grows.

Step 8: run the workload

Click Save and Execute (single workload) or Save Project then launch from the project detail page (multi-workload).

Metrics collected

Every tier collects the same core serving metrics. CPU RAM and NVMe SSD additionally collect LMCache cache metrics, since only those two tiers run the cache server. NVMe SSD collects a further set of disk metrics on top of that, since it is the only tier writing cache data to physical storage.

Core serving metrics (all tiers: GPU, CPU RAM, NVMe SSD):

Metric	What it means
Mean / p50 / p90 / p95 / p99 TTFT	Time to first token, average and tail percentiles.
Mean TPOT	Average time per output token after the first.
Output throughput (tokens/sec)	Generated tokens per second.
Input throughput (tokens/sec)	Prompt tokens processed per second.
Request throughput (requests/sec)	Completed requests per second.
TTFT degradation (median / p95)	How much TTFT worsens on later turns of a conversation compared to the first, as percentage change.
Session counts	Total, complete, incomplete, and error session counts, plus failed-request count, per job.
GPU power / utilization / memory, CPU utilization	Hardware telemetry captured during the run.

LMCache cache metrics (CPU RAM and NVMe SSD tiers only):

Metric	What it means
Cache hit rate (local and external)	Share of KV cache lookups served from vLLM's own cache versus the offloaded LMCache tier.
KV cache usage (%)	How full the active KV cache is.
Cache evictions	Count of cache entries evicted to make room.
Cache usage (bytes)	Total offloaded cache currently held.
L1 usage ratio, L1 eviction loop ticks / triggered	How full the fast (L1) pool is, and whether it is actively cycling through eviction.
L0-L1 and L2 load/store throughput (GB/s)	Data movement rate between vLLM's own cache, the offload pool, and (for NVMe) the disk-backed pool.
Chunks loaded, lookup hit tokens, lookup requested tokens	Counts describing how much cache content was found and reused versus requested.
Shared-prefix / per-user history reuse opportunity (%)	How much of the traffic could theoretically reuse a cached prefix, whether or not it actually did.

Disk metrics (NVMe SSD tier only):

Metric	What it means
Disk read / write throughput (GB/s)	Data rate to and from the NVMe-backed cache pool.
Disk read / write IOPS	I/O operations per second against the disk pool.
Disk read / write latency (ms)	Per-operation latency against the disk pool.
Disk utilization (%) and queue depth	How busy the disk is and how many I/O operations are queued.
Total disk read / write (GB)	Cumulative data moved over the job.

Common pitfalls

• Comparing tiers without holding everything else fixed. Change only the tier (and its build, if switching NVIDIA/AMD); keep model, concurrency, ISL, OSL, and num_rounds identical across the comparison.
• Requesting concurrency higher than the dataset can supply. Each dataset has a fixed number of real sessions, sliced into num_rounds-sized chunks — that ceiling shrinks as num_rounds grows. A concurrency above it does not add more load; the run replays fewer sessions than requested instead. Always check the achieved session count in the results, not just the requested concurrency, and make sure every scenario in a comparison hits the same ceiling.
• Setting num_rounds below 5. For every dataset except sammshen, chunks shorter than 5 rounds are dropped entirely, so the run replays zero virtual users. Even at 5 rounds and above, a short conversation rarely fills GPU memory enough for offloading to matter — use enough rounds that the conversation genuinely outgrows GPU capacity.
• Editing kv_connector casually. It is pre-wired to LMCache for a reason; changing it without changing the rest of the recipe will fail to launch.
• Picking AMD build on NVIDIA hardware (or vice versa). The build selector must match the actual GPU vendor on the target server, or the wrong container image is launched.

---

Reading results on the Reporting page

The Reporting page (/dashboard/reporting) is the unified results explorer, accessible from the Dashboard subnav. The page subtitle reads "Full results across all benchmark runs. Filter, compare, and analyse performance." (The marketing name "Pulse" appears in some legacy docs; the live UI surface is named Reporting.)

The filter bar

Above the results table. The live UI exposes a Filters dropdown panel and a My runs toggle pill. The panel surfaces the standard slicing dimensions:

• Status: completed ~ exclude in-progress and failed jobs.
• My runs ~ toggle on when you just want yours.
• Model + Framework ~ for cross-comparison studies.
• Date range ~ recent only, or a specific period.

Filter state is preserved as you navigate; deep-link to a filtered view via the URL.

The results table

Paginated, sortable. Click any row to expand its detailed metrics.

Columns you'll use most:

• Job name ~ your scenario identifier (e.g. c8-isl512-osl256).
• Model + Framework + Quant ~ the workload config.
• Throughput ~ tokens/sec.
• TTFT P50 / P99 ~ Time To First Token, median and tail.
• TPOT ~ Time Per Output Token, steady-state generation speed.
• Status ~ completed, failed, etc.

The chart panel

Below the table. The live chart sub-tabs are Model, Framework, Quantization, Concurrency, Cost, and Hardware:

• Model ~ same framework, vary model. Useful for picking a model for a use case.
• Framework ~ same model, vary framework. Useful for picking a backend.
• Quantization ~ same model + framework, vary quantization. Useful for quality-vs-throughput tradeoffs.
• Concurrency ~ throughput and latency as concurrency rises. Finds the saturation point.
• Cost ~ tokens per dollar and tokens per hour, when cost data is available.
• Hardware ~ GPU power, util, memory, CPU, RAM. Diagnoses underutilized hardware.

Exporting

CSV export of the filtered view is planned ~ the explicit Export -> CSV button is not yet exposed in v4.0.

Pagination notes

The Reporting page uses range-based pagination ~ page jumps stay fast even on large result sets. The page indicator shows "X-Y of Z" where Z is the total filtered count.

---

Understanding key metrics

Four metrics carry most of the signal; the rest are breakdowns or telemetry. For the full list and exact formulas, see the Metric Definitions Reference.

Metric	What it measures	What moves it	What to watch
Throughput (tokens/sec)	Total output tokens generated per second across all concurrent requests. The headline number.	Rises with concurrency until the GPU saturates, then plateaus. Rises with smaller models and aggressive quantization. Falls with longer OSL.	The plateau is the server's ceiling on this hardware.
TTFT (Time To First Token)	Wall-clock from request arrival to the first output token. Captures prefill and queueing.	Grows with concurrency (queueing) and ISL (prefill work).	Track P99, not P50, for latency SLAs.
TPOT (Time Per Output Token)	Average time between output tokens after the first. Captures steady-state generation speed.	Grows mildly with concurrency until saturation. Roughly model-size-bound.	User-perceived tokens/sec per request is about 1 / TPOT.
Latency percentiles	P50, P90, and P99 for both TTFT and TPOT.	The tail widens under queueing or jitter.	P99 far above P50 means queueing; raise capacity or lower concurrency.
GPU utilization	From telemetry. Fraction of GPU compute cycles in use.	Rises with concurrency on a healthy run.	Below 50% at high concurrency means the workload is CPU-, network-, or memory-bandwidth-bound. Check the Hardware chart tab.

note

Streaming must be on to measure TTFT. Without it, the tool only sees the final response body.

---

Server management

Adding a server

If you have hardware access:

1. Hardware -> Add Server.
2. In the modal, pick a Server Configuration preset (e.g. local-dummy), then fill in Hostname, optional IP Address, optional Bootstrap Script Filename (default agent-bootstrap.sh), and a Bootstrap Comment.
3. Click Generate Bootstrap Script.
4. Run the displayed bootstrap script on the GPU host as a sudoer.

The server appears in the Servers table on the Infrastructure page within ~30 seconds.

Hosted v4.0 note: some hosted Metrum deployments return 404 PGRST202 from POST https://insights.metrum.ai/api/rpc/create_server_config_instance when Generate Bootstrap Script runs. If your deployment shows this error, register servers via the API or via the bootstrap helper described in the Admin Guide.

For the full operational walkthrough (including troubleshooting agent connectivity) see Admin Guide - Registering a server.

Server states

State	Meaning
online	Agent has reported a heartbeat within the last 2 minutes
stale	No heartbeat in 2-10 minutes
offline	No heartbeat in 10+ minutes
requires_reonboarding	Registered under v3.x; needs the v4 bootstrap run

Jobs queued for a stale or offline server remain queued until it returns. Use the planned Monitoring -> Cancel Stale Jobs action (currently API only) if you need to clear them.

Cloud-provisioned servers

If your workspace has cloud credentials configured (see Admin Guide - Cloud provisioning), you can target cloud SKUs directly in a workload ~ Metrum provisions an instance on demand, runs the job, and tears the instance down after the idle timeout.

---

When a run fails

A run can stop before results appear, or complete with numbers that look wrong. Most causes are configuration, not platform faults, and the logs tell you which.

Common causes

Symptom	Likely cause	Fix
Out of memory (OOM)	The model does not fit the GPU at the chosen precision, or gpu_memory_utilization / max_num_batched_tokens is set too high.	Use a smaller quantization, add GPUs by raising tensor parallelism, or pick a server with more GPU memory.
Model server never becomes ready	Bad engine args: a max_model_len the model does not support, or a tensor-parallel size that does not divide the GPU count.	Start from a verified recipe. See Choosing engine arguments.
Framework version not supported on the hardware	The selected version does not match the GPU or driver.	Pick a different version from the dropdown.
Gated model will not load	The Hugging Face token is missing.	Add it in the Profile HF-tokens panel (API only in v4.0).
Server offline	The host missed its heartbeat.	Pick another server or wait for the heartbeat to resume.
All-zero latencies or implausible token counts	The framework did not actually serve the requests.	Open the logs below and read the Framework stream.

Read the logs

Every job records its logs, and the fastest way to see why a run failed is to read them.

Open the Logs drawer from Monitoring: on an execution row or a workload row, click View logs. The drawer lists each job and gives three streams per job:

Stream	What it holds
Framework	The model server's own output (vLLM, SGLang, TensorRT-LLM). OOM errors, rejected flags, and startup failures appear here.
Tool	The benchmark client's output: request pacing, per-request errors, and timeouts.
Agent	The worker agent's lifecycle log: model-server launch, readiness polling, and teardown.

tip

For a failed run, start with the Framework stream. An OOM or an invalid engine argument shows up there first.

If a run completes but a results may be unreliable badge appears, the data is still on the Reporting page. Treat it with extra skepticism, and read the logs before trusting the numbers.

---

Email notifications

Per-user notification preferences are planned via a Profile -> Notifications panel (not yet exposed in the live UI in v4.0). The intended defaults are:

Notification	Default	Notes
Run completed	On	One email per run, regardless of job count
Run failed	On	Includes a link to the Run Detail page
Weekly summary	Off	Aggregate of your activity over the past week
Quota approaching	Admin only	Sent to workspace admins at 80% and 95% of quota

Notifications are user-scoped ~ even if a teammate triggers a run, only people who opted in get the email.

---

Telemetry

Hardware telemetry - what was happening on the GPU server while a job ran.

Accessing

Telemetry is accessible from the job detail page under Dashboard -> Monitoring (/dashboard/monitoring). The left panel is a job selector; the main panel renders charts for the selected job.

What's captured

Metric	Source
GPU power (W)	NVIDIA NVML / ROCm SMI / TPU runtime
GPU utilization (%)	Same
GPU memory (%)	Same
CPU utilization (%)	OS-level
System RAM (%)	OS-level
Per-GPU breakdown	When multiple GPUs per server

TPU jobs use a parallel TPU-specific telemetry path; surfaced via v_tpu_telemetry_job_summary. The agent collects these metrics from LibTPU, Google's TPU runtime library, which exposes hardware counters for per-chip duty cycle, tensor core utilization, and HBM capacity directly through its metrics endpoint. LibTPU provides these readings at sub-second granularity, allowing the agent to capture the same telemetry fidelity as NVML on GPU workloads. For the full metric list, units, and how each is collected, see System Metrics in the methodology.

What to look for

• GPU power flat at TDP ~ fully utilizing the hardware. Good for benchmark runs.
• GPU power oscillating ~ likely batching dynamics; usually fine.
• GPU utilization < 50% at high concurrency ~ workload is bottlenecked elsewhere (CPU preprocessing, network, memory bandwidth).
• High CPU during VLM jobs ~ image preprocessing path. Try a higher-core-count server.

Telemetry summary card

For a quick at-a-glance, the right side of the panel summarizes mean / peak / min for each metric over the job's duration. Useful when you want the headline number without scrubbing a chart.

---

Leaderboard

Ranked LLM throughput across all visible benchmark runs.

Accessing

Leaderboard in the sidebar.

What it shows

The live /leaderboard view exposes a tab strip for Throughput, Cost Efficiency, Quality, and Sustainability, plus a hero card for the current top performer and a paginated ranked list. Use the filter controls to slice by model, framework, hardware, and quantization.

Visibility

Leaderboard rows are subject to the same visibility rules as the underlying runs:

• Public runs appear platform-wide.
• Organization runs appear within your workspace's view.
• Private runs appear only to the owner.

KYAI leaderboard

The KYAI leaderboard surfaces are planned (KYAI -> Leaderboard and the Quality tab on /leaderboard); they rank by mean quality score rather than throughput. Underlying data is available today via v_kyai_leaderboard through the API.

---

See also

• Quickstart ~ get your first benchmark running in under 15 minutes.
• Feature Reference ~ per-feature parameter detail.
• Hardware Compatibility ~ supported accelerators and frameworks per workload type.
• InferenceX Methodology ~ GPU-normalized metrics, architecture dimensions, and ingest details.
• Performance Methodology ~ how first-party tools measure throughput, latency, and accuracy.