Feature Reference

This page is a configuration reference for every feature shipping in Metrum Insights v4.0. Each section covers the feature's purpose, the parameters that configure it, and the surface (UI + API) for working with it.

If you're new to a feature, the User Guide covers the conceptual workflow first ~ this page is the lookup, not the tutorial.

Features documented here:

• Inference backends: vLLM, Multi-node vLLM, SGLang, TensorRT-LLM, Other registered frameworks
• KYAI v2 + Leaderboard
• ASR benchmarking
• VLM benchmarking
• Bring Your Own Workload (BYOW)
• Reporting (formerly Pulse)
• Pre/Post-run validation
• Metering and Stripe billing
• RBAC
• Cloud provisioning (Shadeform)
• Google TPU
• GDPR and compliance
• Backup and recovery

Note on UI status: a number of admin surfaces referenced below (Settings -> Engine args templates, Settings -> Engine presets, Settings -> Validation, Settings -> Cloud credentials, Billing (/billing), Settings -> Compliance, Settings -> Backup, Settings -> Prompt libraries, KYAI -> *) are not yet exposed in the live sidebar in v4.0. Where they are referenced, the underlying database objects and RPCs are present and the corresponding actions are available via the API; the dedicated UI panels are being staged and will land in subsequent releases.

---

Inference backends

Three primary inference backends ship with v4.0 (vLLM, SGLang, TensorRT-LLM). They're surfaced as framework_code values throughout the platform. The framework catalog also registers additional codes for adjacent runners ~ see Other registered frameworks below.

vLLM

Single-node vLLM, configured per workload.

Parameter	Where set	Notes
framework_code	Workload card → Framework	Use vllm
version	Workload card → Version	e.g. 0.6.3. For AMD ROCm, use a -rocm-tagged build
quantization_code	Workload card → Quantization	fp16, bf16, fp8, int8, awq, gptq. Default: fp16
tensor_parallel_size	Engine args (advanced)	Defaults to GPU count on the selected server. When TP is smaller than the GPU count, multiple replicas are created. See Model-server replicas.
max_num_seqs	Engine args (advanced)	Defaults to max(256, concurrency × 2)
max_num_batched_tokens	Engine args (advanced)	Defaults: chunked-prefill-friendly value based on ISL/OSL
block_size	Engine args (advanced)	Default 16. Raise for very long contexts
gpu_memory_utilization	Engine args (advanced)	Default 0.90
enforce_eager	Engine args (advanced)	Default false. Set true to disable CUDA graphs for debugging
enable_prefix_caching	Engine args (advanced)	Default true. Powers KV-cache reuse measurements

For a model that does not fit one server, or a concurrency target that needs tensor and pipeline parallelism spread across hosts, see Multi-node vLLM below.

Engine args templates live under the planned Settings -> Engine args templates Admin panel (not yet in the live UI); reference them via engine_args_template_code on a workload to avoid setting individual parameters. The underlying templates are managed via API today.

Multi-node vLLM (Kubernetes / llm-d)

vLLM served across multiple servers as a Kubernetes deployment, for models too large for one server's GPU memory or concurrency targets that need tensor and pipeline parallelism spread across hosts. Metrum Insights forms a Kubernetes cluster (K3s) across the servers you group together, then deploys the model through llm-d (Gateway API, the Gateway API Inference Extension, and a LeaderWorkerSet) in place of a single-process vLLM server. SGLang and TensorRT-LLM do not have a multi-node path; use single-node vLLM above for those backends. Verified live on NVIDIA GPUs only ~ AMD device plugin and cross-node RCCL support exist in the bootstrap code but have not been confirmed end-to-end.

Parameter	Where set	Notes
Cluster group	Cloud provisioning / hardware onboarding	A named, reusable group of onboarded servers. Every server in a group must share a GPU vendor and have an IP address on file; see Hardware Onboarding.
node_group_selections	Workload (API only in v4.0)	Attaches a cluster group to a workload. Only one selection is supported per workload today.
tensor_parallel_size / pipeline_parallel_size	Engine args (advanced)	Same engine args set as single-node vLLM; read from there for the deployment.
Inference pool / gateway namespace / chart version	Workload (API only in v4.0)	Deployment-identity overrides. Defaults are auto-provisioned on first run when left unset.

There is no dedicated UI for grouping hardware or attaching a cluster group to a workload in v4.0; this feature is API-only today. Cluster formation (K3s join across every group member, then Gateway API/GAIE/LeaderWorkerSet controller bootstrap) happens automatically at run-launch time and, on first use of a cluster group, can take several minutes; a cluster group that has already formed a cluster for a prior run reuses it.

See Multi-Node vLLM Deployment for the full curl walkthrough: grouping hardware, creating a multi-node workload, launching a run, and monitoring it.

SGLang

Parameter	Where set	Notes
framework_code	Workload card → Framework	Use sglang
version	Workload card → Version	e.g. 0.3.5
quantization_code	Workload card → Quantization	fp16, bf16, fp8. awq/gptq supported on some models
tp_size	Engine args	Defaults to GPU count
mem_fraction_static	Engine args	Default 0.88
schedule_policy	Engine args	lpm (default), fcfs, dfs-weight
chunked_prefill_size	Engine args	Default 8192

TensorRT-LLM

NVIDIA-only. Requires pre-built engine files for the target (model, quantization, GPU) tuple.

Parameter	Where set	Notes
framework_code	Workload card → Framework	Use trt-llm
version	Workload card → Version	e.g. 0.13.0. Must match the engine file's TRT-LLM version
engine_preset_code	Workload card → Engine preset	References a pre-built engine bundle
kv_cache_free_gpu_mem_fraction	Engine args	Default 0.9
max_batch_size	Engine args	Default 64
max_num_tokens	Engine args	Default depends on engine preset

Engine presets are managed under the planned Settings -> Engine presets Admin panel (not yet in the live UI). For Llama 70B TP4 specifically, the platform ships pre-built presets ~ see schema file 37_llama70b_tp4_engine_presets.sql.

Other registered frameworks

The framework catalog (v_frameworks_versions) also exposes framework_code values you'll see on the project workload card today, but with varying levels of end-to-end support:

framework_code	Status
vllm	Primary backend, fully supported.
sglang	Primary backend, fully supported.
trt-llm	Primary backend, fully supported (NVIDIA only).
tgi	HuggingFace TGI ~ wired in the catalog, intended for parity benchmarks.
llama-cpp	llama.cpp ~ wired in the catalog, useful for CPU and Apple-silicon comparisons.
ollama	Wired in the catalog; full E2E support deferred to v4.1.
dynamo	NVIDIA Dynamo ~ wired in the catalog, end-to-end run path deferred to v4.1.
openai	Target for any OpenAI-compatible endpoint.

Use one of the three primary backends for production benchmarking unless you are explicitly comparing against an alternative runner.

---

KYAI v2

KYAI evaluates inference quality through a two-phase pipeline. KYAI execution is wired through the Add KYAI Run action on each project workload card and the underlying RPCs; the dedicated KYAI -> ... navigation surface (Endpoints & Judges, Prompt Sets, Results) referenced below is not yet exposed in the live sidebar in v4.0 ~ those panels are managed via the API today and will land in subsequent releases.

Architecture

• Phase 1 ~ Generation. Runs a candidate endpoint over a prompt set, captures responses.
• Phase 2 ~ Judging. Runs a configured judge over each (instruction, candidate_response, ground_truth) triple and produces a score plus reasoning.

Three first-class objects:

• Candidate endpoint ~ what you're evaluating (a deployed inference server or a remote OpenAI-compatible endpoint).
• Judge config ~ the LLM that scores outputs and the scoring method.
• Prompt set ~ instructions + optional ground truths. Either built-in or user-uploaded.

Candidate endpoint parameters

Configured via the planned KYAI -> Endpoints & Judges -> Candidate Endpoints panel (API only in v4.0):

Parameter	Required	Notes
Name	yes	Display name
API base URL	yes	OpenAI-compatible endpoint
Model name	yes	Model identifier sent in the request
Max concurrent requests	yes	In-flight cap; default 8
Retry attempts	no	Default 3
Retry backoff	no	Default exponential, base 500ms
Authentication	no	Optional bearer token or API key
Workspace	yes	Which workspace owns this endpoint

Judge config parameters

Configured via the planned KYAI -> Endpoints & Judges -> Judge Configs panel (API only in v4.0):

Parameter	Required	Notes
Name	yes	Display name
Scoring method	yes	numeric_1_to_10, pairwise_preference, pass_fail, rubric
API base URL	yes	OpenAI-compatible endpoint hosting the judge model
Model name	yes	Judge model (e.g. a 70B+ instruct model)
Max concurrent requests	yes	Default 4. Judges are token-heavy ~ be conservative
Retry attempts	no	Default 3
Judge prompt template	no	Override the default; placeholders {{instruction}}, {{response}}, {{ground_truth}}

Scoring methods:

• numeric_1_to_10 ~ judge returns a 1-10 integer plus reasoning.
• pairwise_preference ~ compare two candidates; judge picks one or "tie".
• pass_fail ~ binary, with reasoning.
• rubric ~ multi-dimensional scoring against a user-supplied rubric.

Prompt sets

Either:

• Built-in. Listed under v_kyai_builtin_sets_with_prompt_count. Curated for common evaluations (general instruction-following, code, math, reasoning).
• User-uploaded. CSV or JSONL with instruction and optional ground_truth columns. Uploaded via the planned KYAI -> Prompt Sets -> Upload panel (API only in v4.0).

Results

Phase-1 outputs in v_kyai_generation_outputs; phase-2 scored outputs in v_kyai_outputs. The planned Results Explorer (KYAI -> Results, not yet exposed in the live UI) supports:

• Job ID filtering
• Score range (min/max)
• Candidate endpoint filter
• Judge config filter

Each row shows: instruction, candidate response, ground truth (if present), score, judge reasoning, candidate endpoint, judge config, prompt-set snapshot name, created timestamp.

Leaderboard

v_kyai_leaderboard ranks scored runs. Visibility follows project visibility ~ Public KYAI runs appear in the cross-workspace leaderboard; Organization-scoped runs in your workspace's view only; Private to the owner only.

---

ASR benchmarking

ASR (Automatic Speech Recognition) benchmarks use the metrumbench-asr tool.

ASR support is intentionally narrow: the Whisper model family only (for example whisper-large-v3) and the vLLM framework only. SGLang and TensorRT-LLM are not productized for ASR.

Workload parameters

Parameter	Required	Notes
model_code	yes	Whisper-family ASR model only (e.g. whisper-large-v3)
framework_code	yes	vllm only. SGLang and TensorRT-LLM are not supported for ASR
version	yes	Framework version
concurrency	yes	In-flight transcription requests
audio_duration_bucket	yes	Bucket key ~ see below
dataset_snapshot_code	yes	ASR snapshot reference (v_metrumbench_asr_outputs)

Audio duration buckets

ISL is replaced by audio duration buckets in ASR. Standard buckets:

Bucket	Duration range	Typical use case
xs	< 5 seconds	Short commands
s	5-30 seconds	Voice messages
m	30-120 seconds	Meeting clips
l	2-10 minutes	Podcasts, lectures
xl	10+ minutes	Long-form transcription

Metrics

• RTF (Real-Time Factor) ~ wall-clock seconds per second of audio. RTF < 1 means faster than real-time.
• WER (Word Error Rate) ~ output quality vs ground truth (if dataset has transcripts).
• TTFT (audio) ~ time to first transcribed word.

Dataset snapshots

ASR datasets are versioned via 36_asr_snapshots.sql. The platform ships a set of standard snapshots (LibriSpeech, AMI, etc.); custom snapshots are uploaded via the BYOW flow (see Bring Your Own Workload).

---

VLM benchmarking

Vision-language model benchmarking uses the metrumbench-vlm tool.

Workload parameters

Same shape as LLM benchmarking, plus image-related fields:

Parameter	Required	Notes
model_code	yes	VLM (e.g. llava-1.6-mistral-7b, qwen2-vl-7b)
framework_code	yes	vllm, sglang, trt-llm where supported
input_sequence_length	yes	Text portion of the prompt
image_resolution	yes	low, medium, high ~ maps to model-specific patch counts
images_per_request	yes	Typically 1; multi-image supported on some models
output_sequence_length	yes	Standard OSL
concurrency	yes	In-flight requests

Metrics

Same as LLM, with one addition:

• Image preprocessing time ~ included in TTFT. CPU-bound. The Reporting page breaks it out separately under the Hardware chart tab (CPU view).

---

Bring Your Own Workload (BYOW)

Supply your own prompt dataset for benchmarking or KYAI evaluation.

For benchmarking: upload a CSV/JSONL prompt library via the planned Settings -> Prompt libraries panel (API only in v4.0). Reference it from a workload via prompt_library_code.

For KYAI: see Prompt sets above.

CSV format for benchmarking workloads:

prompt,expected_output_tokens
"Translate to French: Hello world",10
"Summarize the following article: ...",256

Expected output tokens is optional; if absent, the workload's OSL is used.

---

Reporting (formerly Pulse)

The Reporting page (/dashboard/reporting) is the results explorer, accessible from the Dashboard subnav. The page subtitle reads "Full results across all benchmark runs. Filter, compare, and analyse performance." The marketing name "Pulse" still appears in some legacy material; the live UI surface is named Reporting.

Filters

The Reporting page exposes a Filters dropdown panel and a My runs toggle pill above the results table. The filterable dimensions are:

Filter	Backed by
Status	v_benchmark_results.status_code
Tool	v_benchmark_results.tool_code
Model	v_benchmark_results.model_code
Framework	v_benchmark_results.framework_code
Server config	v_benchmark_results.config_code
Date range	v_benchmark_results.created_at
My runs only	v_benchmark_results.owner_account_id

Charts

The chart panel below the results table has the following sub-tabs (live UI labels):

Tab	View	What it shows
Model	v_model_comparison	Throughput, latency, GPU utilization across models
Framework	v_framework_comparison	Same metrics grouped by framework
Quantization	v_quantization_comparison	Impact of quantization on throughput and quality
Concurrency	v_concurrency_scaling	Throughput and latency vs concurrency
Cost	v_cost_efficiency	tokens/$ and tokens/hour
Hardware	v_hardware_utilization	GPU power, util, memory, CPU, RAM

Pagination

Range-based via Range/Range-Unit: items headers. See API Reference - Pagination.

Exporting

CSV export of the filtered results view is planned; the explicit Export -> CSV button on the results table is not yet exposed in v4.0. Use the API to pull the filtered set today.

---

Pre/Post-run validation

Pre-run

Runs before the first job starts. Checks:

• Model is accessible to the target server (via HuggingFace if applicable).
• Framework + version + accelerator combination is supported.
• GPU memory is sufficient for the largest scenario in the run.
• HF token (if required) is present and valid.
• Server is online with recent heartbeat.

Failures surface as a structured error on the Run Detail page with a specific suggestion (e.g. "use quantization_code=fp8" or "select a server with ≥160 GB GPU memory").

Post-run

Runs after the last job completes. Checks:

• Output token counts in plausible ranges.
• No all-zero latency rows.
• Telemetry actually attached to each job.

Surfaces as a "results may be unreliable" badge on the Run Detail page if any check fails. Doesn't block the run from completing ~ it just flags suspect data.

Configuration

Validation severity is configurable per-workspace via the planned Settings -> Validation Admin panel (not yet exposed in the live UI; managed via API today):

• Strict (default) ~ fail-fast on any pre-run issue.
• Warn ~ log warnings but allow the run to proceed.
• Off ~ disable (not recommended; useful only for debugging the validator itself).

---

Metering and Stripe billing

See Admin Guide → Billing for the operational view.

Metered events

Event	Recorded in
GPU-hour (cloud)	v_cloud_resource_lifecycle
Cloud spend pass-through	v_run_cost_summary
KYAI judge tokens	v_kyai_outputs (aggregated)
API request volume	Internal; will surface under the planned Billing -> Usage view

Quotas

Quotas are configured via the API in v4.0; the dedicated Billing -> Quotas UI panel is planned for a subsequent release.

• GPU-hours (monthly)
• Judge tokens (monthly)
• Cloud spend (monthly, in USD)

Hitting a quota returns HTTP 402 Payment Required at job submission.

---

RBAC

Four roles: Owner, Admin, Engineer, Viewer. See Admin Guide → RBAC for the full responsibility matrix.

Enforcement points

Surface	Where enforced
RPC calls	19_rbac.sql ~ checked per-RPC
View reads	Row-level security policies on each view
UI	Frontend hides forbidden actions; not source of truth

Custom roles

Custom roles are not supported in v4.0 ~ the four built-in roles are fixed. Per-project visibility (Private / Organization / Public) provides per-resource access control alongside the role system.

---

Cloud provisioning (Shadeform)

Unified provisioning across RunPod, Shadeform, Lambda, and Google Cloud.

Configuration

Configured via the planned Settings -> Cloud credentials Admin/Owner panel (not yet exposed in the live UI; managed via API today). Per-provider keys are stored encrypted-at-rest.

Per-user overrides via the planned Profile -> Cloud credentials panel ~ if set, used in place of workspace credentials for jobs submitted by that user.

Provider matrix

See Admin Guide - Cloud provisioning for the provider x GPU matrix.

Lifecycle parameters

Parameter	Default	Notes
Idle teardown timeout	5 min	Configurable per cloud provider
Provisioning timeout	15 min	If exceeded, request fails with provisioning_timeout
Auto-retry on failure	1	One automatic retry on transient provider errors
Max instance lifetime	24 hr	Hard cap; instances are torn down regardless of activity

Cost controls

• Per-workspace monthly cap (Owner).
• Per-user daily cap (Admin or Owner).

Both surface under the Billing page (/billing) in the sidebar.

---

Google TPU

Google Cloud-managed support across Google TPU generations and shapes - not limited to a single TPU type.

Prerequisites

• Google Cloud cluster with a supported TPU node pool.
• Workload Identity enabled.
• Service account JSON registered via the planned Settings -> Cloud credentials -> Google Cloud panel (API only in v4.0).

Workload configuration

Parameter	Notes
framework_code	vllm (TPU backend) ~ SGLang and TRT-LLM not supported on TPU
gpu_code	TPU family code for the selected generation (e.g. tpu-v5e, tpu-v5p)
tpu_topology	e.g. 2x2, 4x4 ~ defines the slice

Telemetry

TPU telemetry uses the dedicated v_tpu_telemetry_job_summary view. Per-chip utilization, HBM usage, and tensorcore activity are surfaced.

Limitations

• VLM and ASR support on TPU are experimental ~ see Hardware Compatibility.

---

GDPR and compliance

See Admin Guide → GDPR and compliance for the operational view.

Configurable retention

Per category, via the planned Settings -> Compliance -> Data retention Admin panel (not yet exposed in the live UI):

• Audit logs: 90 days to 7 years
• Telemetry: 90 days to 365 days
• Benchmark results, KYAI outputs: indefinite by default, per-workspace TTL configurable

Data deletion

POST /rpc/request_account_deletion initiates a 14-day pending deletion. After the window, the account is hard-deleted; audit and results data is anonymized (account ID hashed).

---

Backup and recovery

See Admin Guide → Backup and recovery.

Schedule

• Nightly automatic snapshot, 30-day retention.
• Manual snapshots via the planned Settings -> Backup -> Take snapshot Admin panel (not yet exposed in the live UI).

Restore

Support-assisted in v4.0. Self-service restore is planned for a later release.

---

See also

• User Guide ~ workflows that combine these features.
• API Reference ~ programmatic access to every feature.
• Admin Guide ~ operational settings and access control.
• Hardware Compatibility ~ supported accelerators per feature.