Skip to main content

User Guide

This guide walks through the workflows that make up day-to-day work in Metrum Insights. Each section is task-oriented (e.g. "how do I run a benchmark", "how do I evaluate quality with KYAI") and links to the Feature Reference for parameter-level detail.

If you haven't run anything yet, do the Quickstart first. It gets you to a completed benchmark in 15 minutes. This guide assumes you've made it that far.

Sections:

Running benchmarks​

The core unit of work is a project containing one or more workloads. A workload describes what to benchmark (model + framework + scenario matrix); the project organizes them and produces runs when executed.

Single-workload runs (the Quickstart path)​

The simplest path is a project with one workload, executed against one server. The Quickstart covers this end-to-end. The shorthand:

  1. Projects -> New Project.
  2. Fill in project name + visibility (the form default in the live UI is Organization).
  3. Configure one workload card (Model, Framework, Version, Concurrency chips, ISL chips, OSL chips, Requests per scenario, Streaming).
  4. Pick a server.
  5. Save and Execute.

Save and Execute is only available when there's exactly one workload and a server is selected. Multi-workload projects save as drafts via Save Project and execute via a separate flow ~ see below.

Multi-workload projects​

Use multi-workload projects when you want to compare several related configurations as a single deliverable (e.g. "Llama 70B on vLLM, SGLang, and TRT-LLM, at three concurrencies each").

  1. Projects -> New Project.
  2. Add the first workload card.
  3. Click Add Workload to add more cards. Each card has its own model, framework, scenario matrix, and (optionally) server. Each card also offers Add Benchmark and Add KYAI Run buttons for adding additional benchmark or quality-evaluation steps.
  4. Click Save Project (not "Save and Execute"). The project is saved as a draft.
  5. From the project detail page, trigger runs per workload.

Multi-workload projects are useful for keeping related work together ~ every run from the project lands under the same Runs tab, and the Benchmarks page can filter to "this project" for cross-workload comparison.

Scenario matrices​

Within a workload, concurrency, ISL, and OSL chips multiply out:

ConcurrencyISLOSLScenarios
15122561
1, 8, 325122563
1, 8, 32512, 20482566
1, 8, 32512, 2048256, 51212

Each scenario becomes one job in the run, named c<concurrency>-isl<isl>-osl<osl>. A 12-scenario run produces 12 rows in the Jobs table, run sequentially on the selected server. Start small ~ running a 24-scenario run on a 70B model can take an hour.

Available chip values in the workload card today are: concurrency 1, 4, 8, 16, 32, 64, 128; ISL 128, 256, 512, 1024, 2048; OSL 64, 128, 256, 512.

Picking a framework​

FrameworkBest for
vllmGeneral-purpose; widest model coverage; most portable
sglangHigh-throughput serving with structured output
trt-llmNVIDIA-only; highest peak throughput on H100/H200/B200

Defaults: start with vLLM. Add SGLang and TRT-LLM when comparing peak performance. Other framework codes (tgi, llama-cpp, ollama, dynamo, openai) are also registered in the catalog ~ see Feature Reference - Other registered frameworks for the support level of each.

Picking a quantization​

QuantizationQuality impactThroughput / memory impactTypical use
fp16BaselineBaselineReference baseline
bf16~identical to fp16SameWhen hardware prefers bf16
fp8Small~1.7x throughput, ~0.5x memoryProduction default on H100+
int8Noticeable~2x throughput, ~0.5x memoryCost-optimized deployments
awq/gptqSmall to moderate~3x throughput, ~0.25x memoryMemory-constrained scenarios

If quality matters as much as throughput, pair quantization comparison with a KYAI evaluation on the same endpoints to catch quality regressions.

Engine arguments (advanced)​

Most users don't touch these ~ defaults are sane. When you do need to, the most common changes:

  • max_num_batched_tokens ~ if you see throughput plateau early on vLLM, raising this from the default usually helps. See Feature Reference -> vLLM for full parameters.
  • gpu_memory_utilization ~ lower it (e.g. 0.85) if the server is shared with other workloads.
  • enforce_eager ~ set true only for debugging; it disables CUDA graphs and tanks throughput.

Reusable engine-args templates are managed via the planned Settings -> Engine args templates Admin panel (not yet exposed in the live UI; managed via API today).

Reading results on the Benchmarks page​

The Benchmarks page (/benchmarks) is the unified results explorer. The page subtitle reads "Full results across all benchmark runs. Filter, compare, and analyse performance." (The marketing name "Pulse" appears in some legacy docs; the live UI surface is named Benchmarks.)

The filter bar​

Above the results table. The live UI exposes a Filters dropdown panel and a My runs toggle pill. The panel surfaces the standard slicing dimensions:

  • Status: completed ~ exclude in-progress and failed jobs.
  • My runs ~ toggle on when you just want yours.
  • Model + Framework ~ for cross-comparison studies.
  • Date range ~ recent only, or a specific period.

Filter state is preserved as you navigate; deep-link to a filtered view via the URL.

The results table​

Paginated, sortable. Click any row to expand its detailed metrics.

Columns you'll use most:

  • Job name ~ your scenario identifier (e.g. c8-isl512-osl256).
  • Model + Framework + Quant ~ the workload config.
  • Throughput ~ tokens/sec.
  • TTFT P50 / P99 ~ Time To First Token, median and tail.
  • TPOT ~ Time Per Output Token, steady-state generation speed.
  • Status ~ completed, failed, etc.

The chart panel​

Below the table. The live chart sub-tabs are Model, Framework, Quantization, Concurrency, Cost, and Hardware:

  • Model ~ same framework, vary model. Useful for picking a model for a use case.
  • Framework ~ same model, vary framework. Useful for picking a backend.
  • Quantization ~ same model + framework, vary quantization. Useful for quality-vs-throughput tradeoffs.
  • Concurrency ~ throughput and latency as concurrency rises. Finds the saturation point.
  • Cost ~ tokens per dollar and tokens per hour, when cost data is available.
  • Hardware ~ GPU power, util, memory, CPU, RAM. Diagnoses underutilized hardware.

Exporting​

CSV export of the filtered view is planned ~ the explicit Export -> CSV button is not yet exposed in v4.0. Use the API to pull the filtered set today (see the API Reference).

Pagination notes​

The Benchmarks page uses range-based pagination ~ page jumps stay fast even on large result sets. The page indicator shows "X-Y of Z" where Z is the total filtered count.

Understanding key metrics​

Four metrics carry most of the signal. The rest are usually breakdowns or telemetry.

Throughput (tokens/sec)​

Total output tokens generated per second across all concurrent requests. Headline number for benchmarking.

  • Scales up with concurrency until the GPU saturates, then plateaus.
  • Scales up with smaller models or aggressive quantization.
  • Scales down with longer OSL (more steady-state generation), and slightly down with longer ISL (more prefill work).

TTFT (Time To First Token)​

Wall-clock from request arrival to the first output token returned. Captures prefill + queueing latency.

  • TTFT P50 ~ typical user experience.
  • TTFT P99 ~ tail latency. Often more important than P50 for SLAs.

Grows with concurrency (queueing) and ISL (prefill work).

TPOT (Time Per Output Token)​

Average time between consecutive output tokens, after the first one. Captures steady-state generation speed.

  • Grows mildly with concurrency until saturation.
  • Roughly model-size-bound ~ a 70B model has higher TPOT than a 7B model on the same hardware.

A user-perceived "tokens per second per request" is roughly 1 / TPOT (in seconds).

Latency percentiles​

The Benchmarks page reports P50, P90, and P99 for both TTFT and TPOT. The shape across percentiles tells you about tail behavior:

  • P50 ~ P99 ~ healthy, well-batched.
  • P99 >> P50 ~ queueing or jitter; raise capacity or lower concurrency.

GPU utilization​

From telemetry. Healthy benchmarks push utilization above 80% at the higher concurrency levels. Persistent <50% utilization at high concurrency usually means the workload is CPU-bound, network-bound, or memory-bandwidth-bound ~ the Benchmarks page Hardware chart tab will tell you which.

KYAI​

KYAI ("Know Your AI") evaluates inference quality, distinct from performance which is what the rest of the platform measures.

UI status in v4.0: KYAI execution is wired through the Add KYAI Run action on each project workload card and the underlying RPCs. The dedicated KYAI -> ... navigation surface (Endpoints & Judges, Prompt Sets, Results, Leaderboard) referenced below is being staged and is not yet exposed in the live sidebar ~ those panels are managed via the API today and will land in subsequent releases.

Concepts​

Three objects:

  • Candidate endpoint ~ what you're evaluating. Either a deployed inference server registered in Metrum, or a BYOE endpoint.
  • Judge config ~ the LLM that scores candidate outputs, plus the scoring method.
  • Prompt set ~ instructions + (optional) ground truths. Built-in or user-uploaded.

A KYAI run pairs one candidate x one judge x one prompt set and produces a row per (instruction, response, score, reasoning).

Running an evaluation​

  1. From a project workload card, click Add KYAI Run to attach a quality-evaluation step to the workload. (The standalone KYAI -> Overview -> New Evaluation flow is planned but not yet exposed in the live UI.)
  2. Provide the candidate endpoint, judge config, and prompt set ~ either pick from existing API-managed records or create new ones via the API. Field reference is in Feature Reference - KYAI.
  3. Submit. The run progresses through generation (Phase 1) and judging (Phase 2).

Reading results​

The KYAI Results Explorer (planned KYAI -> Results, API only in v4.0) supports the following filters:

  • Job IDs ~ narrow to one or a few evaluations.
  • Score range ~ find the worst (e.g. score <= 3) or best (e.g. score >= 9) outputs.
  • Candidate endpoint ~ across all evaluations of one endpoint.
  • Judge config ~ to compare how different judges rate the same outputs.

Each row shows the instruction, the candidate's actual response, the ground truth (if any), the score, the judge's reasoning, and the configuration that produced it. Until the dedicated UI lands, query v_kyai_outputs directly via the API.

What to look for​

  • Mean score ~ overall quality. Compare across candidate endpoints.
  • Score distribution ~ a candidate with mean 7 but bimodal at 4 and 10 is unreliable.
  • Low-score reasoning ~ read the judge's explanations on the worst outputs. Patterns often emerge (refuses to answer, ignores instructions, hallucinates facts).
  • Disagreement across judges ~ running the same candidate through two judges with different scoring methods checks judge robustness.

Cost note​

Judges are LLMs ~ they consume tokens. A KYAI run on a 1,000-prompt set with a 70B judge can run hundreds of thousands of judge tokens. Metering tracks this; quotas can cap it. Start with smaller prompt sets when calibrating.

Leaderboard​

v_kyai_leaderboard ranks runs by mean score, filtered by visibility. Public runs appear platform-wide; Organization runs in your workspace; Private to you only. The dedicated KYAI leaderboard surface is planned and not yet exposed in the live UI.

ASR benchmarking​

Speech-to-text models. Same scenario-matrix model as LLM, with the ISL axis replaced by audio duration buckets.

UI status in v4.0: ASR execution is wired through the API. The dedicated ASR launch surface in the workload card (audio dataset picker, audio-duration bucket controls, sample-length controls) is being staged and not yet exposed in the live UI; submit ASR runs via the API today.

Configuring a workload​

  1. Projects -> New Project -> Add Workload.
  2. Pick an ASR model (e.g. whisper-large-v3).
  3. Pick a framework that supports it (vLLM or SGLang where available ~ see Hardware Compatibility - ASR).
  4. Set audio duration buckets ~ xs, s, m, l, xl. Bucket -> typical duration range in the Feature Reference.
  5. Pick an ASR dataset snapshot. Built-in: LibriSpeech, AMI, etc. Custom: see BYOW.
  6. Run as normal.

ASR-specific metrics​

  • RTF (Real-Time Factor) ~ wall-clock seconds per audio second. RTF < 1 means faster than real-time.
  • WER (Word Error Rate) ~ quality, vs. ground truth if the dataset has transcripts.
  • TTFT ~ time to first transcribed word.

WER is reported when the dataset includes ground truth; absent otherwise.

VLM benchmarking​

Image-to-text (vision-language) models. Same scenario-matrix model as LLM, with image-related parameters added.

UI status in v4.0: VLM execution is wired through the API. The dedicated VLM launch surface in the workload card (image dataset picker, image_resolution selector, images_per_request control) is being staged and not yet exposed in the live UI; submit VLM runs via the API today.

Configuring a workload​

  1. Projects -> New Project -> Add Workload.
  2. Pick a VLM (e.g. llava-1.6-mistral-7b, qwen2-vl-7b).
  3. Set ISL (text portion of prompt), OSL, concurrency as normal.
  4. Set image resolution (low, medium, high) and images per request (typically 1).
  5. Run as normal.

VLM-specific metrics​

  • Image preprocessing time ~ included in TTFT. CPU-bound on most paths.

The Benchmarks page Hardware chart tab (CPU view) breaks out the preprocessing contribution. If CPU utilization is high while GPU is low, you're preprocessing-bound, not inference-bound ~ try a server with a higher core count or pipeline preprocessing.

Bring Your Own​

Two related capabilities.

BYOE - Bring Your Own Endpoint​

Point Metrum at any OpenAI-compatible inference endpoint ~ your own deployment, a third-party API, a model served from anywhere.

  1. Register the endpoint via the API today (the dedicated KYAI -> Endpoints & Judges -> Add Candidate Endpoint panel is planned and not yet exposed in the live UI), or Projects -> New Project and set the workload target to BYOE.
  2. Provide endpoint URL, model name, optional auth, max concurrent requests, and retry settings.
  3. Submit benchmark or KYAI runs against it.

Caveats:

  • No telemetry. GPU power, utilization, and memory will be empty ~ Metrum can't see your endpoint's hardware.
  • TTFT timeout. Default 60 seconds; configurable per endpoint up to 600s for cold-start testing.
  • Network is in the path. TTFT and TPOT include network round-trip time. This is realistic for production, but worth noting if you're comparing BYOE to a local benchmark.

BYOW - Bring Your Own Workload​

Supply your own prompt dataset.

For benchmarking: upload a CSV or JSONL file with one prompt per row via the planned Settings -> Prompt libraries panel (API only in v4.0). Optional expected_output_tokens column overrides the workload's OSL per-prompt. Reference the library from a workload via its prompt_library_code.

For KYAI: upload via the planned KYAI -> Prompt Sets -> Upload panel (API only in v4.0). Same format, plus an optional ground_truth column used by the judge.

When to use BYOW:

  • Production traffic shape matters and the built-in datasets don't match it.
  • You're benchmarking a domain-specific model on its domain (medical, legal, code).
  • You're validating that a deployment handles your exact prompts before going live.

Server management​

Adding a server​

If you have hardware access:

  1. Hardware -> Add Server.
  2. In the modal, pick a Server Configuration preset (e.g. local-dummy), then fill in Hostname, optional IP Address, optional Bootstrap Script Filename (default agent-bootstrap.sh), and a Bootstrap Comment.
  3. Click Generate Bootstrap Script.
  4. Run the displayed bootstrap script on the GPU host as a sudoer.

The server appears in the Hardware table within ~30 seconds.

Hosted v4.0 note: some hosted Metrum deployments return 404 PGRST202 from POST https://<your-control-plane>/api/rpc/create_server_config_instance when Generate Bootstrap Script runs. If your deployment shows this error, register servers via the API or via the bootstrap helper described in the Admin Guide.

For the full operational walkthrough (including troubleshooting agent connectivity) see Admin Guide - Registering a server.

Server states​

StateMeaning
onlineAgent has reported a heartbeat within the last 2 minutes
staleNo heartbeat in 2-10 minutes
offlineNo heartbeat in 10+ minutes
requires_reonboardingRegistered under v3.x; needs the v4 bootstrap run

Jobs queued for a stale or offline server remain queued until it returns. Use the planned Monitoring -> Cancel Stale Jobs action (currently API only) if you need to clear them.

Cloud-provisioned servers​

If your workspace has cloud credentials configured (see Admin Guide - Cloud provisioning), you can target cloud SKUs directly in a workload ~ Metrum provisions an instance on demand, runs the job, and tears the instance down after the idle timeout.

Pre- and post-run validation​

Every run passes through a validator before the first job starts (pre-run) and after the last completes (post-run).

Pre-run failures​

If pre-run validation fails, the run never starts and you see a structured error on the Run Detail page. The validator gives you a specific suggestion ~ not just "validation failed" but, for example, "model requires 140 GB at fp16; selected server has 80 GB; try quantization_code=fp8 or a server with >= 160 GB GPU memory."

Common pre-run failures:

  • Insufficient GPU memory ~ use a smaller quantization or a different server.
  • Framework version not supported on selected hardware ~ pick a different version from the dropdown.
  • HF token missing ~ for gated models. Add via the planned Profile -> HF tokens panel (API only in v4.0).
  • Server offline ~ pick a different server or wait for heartbeat to resume.

Post-run warnings​

Post-run validation flags suspect data but doesn't block the run:

  • All-zero latencies ~ usually means the framework didn't actually serve requests.
  • Implausible output token counts ~ way under or over the configured OSL.
  • Missing telemetry ~ the agent didn't capture metrics for one or more jobs.

A "results may be unreliable" badge appears on the Run Detail page if any post-run check fails. The data is still on the Benchmarks page ~ just treat it with extra skepticism.

Strict vs. warn mode​

Admins can configure validation severity per-workspace. Default is strict (fail-fast on pre-run issues). See Feature Reference - Pre/Post-run validation.

Email notifications​

Per-user notification preferences are planned via a Profile -> Notifications panel (not yet exposed in the live UI in v4.0). The intended defaults are:

NotificationDefaultNotes
Run completedOnOne email per run, regardless of job count
Run failedOnIncludes a link to the Run Detail page
Weekly summaryOffAggregate of your activity over the past week
Quota approachingAdmin onlySent to workspace admins at 80% and 95% of quota

Notifications are user-scoped ~ even if a teammate triggers a run, only people who opted in get the email.

Telemetry​

Hardware telemetry - what was happening on the GPU server while a job ran.

Accessing​

Telemetry in the sidebar. The left panel is a job selector; the main panel renders charts for the selected job.

What's captured​

MetricSource
GPU power (W)NVIDIA NVML / ROCm SMI / TPU runtime
GPU utilization (%)Same
GPU memory (%)Same
CPU utilization (%)OS-level
System RAM (%)OS-level
Per-GPU breakdownWhen multiple GPUs per server

TPU jobs use a parallel TPU-specific telemetry path; surfaced via v_tpu_telemetry_job_summary.

What to look for​

  • GPU power flat at TDP ~ fully utilizing the hardware. Good for benchmark runs.
  • GPU power oscillating ~ likely batching dynamics; usually fine.
  • GPU utilization < 50% at high concurrency ~ workload is bottlenecked elsewhere (CPU preprocessing, network, memory bandwidth).
  • High CPU during VLM jobs ~ image preprocessing path. Try a higher-core-count server.

Telemetry summary card​

For a quick at-a-glance, the right side of the panel summarizes mean / peak / min for each metric over the job's duration. Useful when you want the headline number without scrubbing a chart.

Leaderboard​

Ranked LLM throughput across all visible benchmark runs.

Accessing​

Leaderboard in the sidebar.

What it shows​

The live /leaderboard view exposes a tab strip for Throughput, Cost Efficiency, Quality, and Sustainability, plus a hero card for the current top performer and a paginated ranked list. Use the filter controls to slice by model, framework, hardware, and quantization.

Visibility​

Leaderboard rows are subject to the same visibility rules as the underlying runs:

  • Public runs appear platform-wide.
  • Organization runs appear within your workspace's view.
  • Private runs appear only to the owner.

KYAI leaderboard​

The KYAI leaderboard surfaces are planned (KYAI -> Leaderboard and the Quality tab on /leaderboard); they rank by mean quality score rather than throughput. Underlying data is available today via v_kyai_leaderboard through the API.

See also​