Feature Reference

This page is a configuration reference for every feature shipping in Metrum Insights v4.0. Each section covers the feature's purpose, the parameters that configure it, and the surface (UI + API) for working with it.

If you're new to a feature, the User Guide covers the conceptual workflow first ~ this page is the lookup, not the tutorial.

Features documented here:

• Inference backends: vLLM, SGLang, TensorRT-LLM, Other registered frameworks
• KYAI v2 + Leaderboard
• ASR benchmarking
• VLM benchmarking
• BYOE / BYOW
• Sizer
• InferenceX
• Benchmarks (formerly Pulse)
• Pre/Post-run validation
• Metering and Stripe billing
• RBAC
• Cloud provisioning (Shadeform)
• Google TPU
• GDPR and compliance
• Backup and recovery

Note on UI status: a number of admin surfaces referenced below (Settings -> Engine args templates, Settings -> Engine presets, Settings -> Validation, Settings -> Cloud credentials, Settings -> Billing, Settings -> Compliance, Settings -> Backup, Settings -> Prompt libraries, KYAI -> *) are not yet exposed in the live sidebar in v4.0. Where they are referenced, the underlying database objects and RPCs are present and the corresponding actions are available via the API; the dedicated UI panels are being staged and will land in subsequent releases.

---

Inference backends

Three primary inference backends ship with v4.0 (vLLM, SGLang, TensorRT-LLM). They're surfaced as framework_code values throughout the platform. The framework catalog also registers additional codes for adjacent runners and BYOE-style targets ~ see Other registered frameworks below.

vLLM

Single-node vLLM, configured per workload.

Parameter	Where set	Notes
framework_code	Workload card → Framework	Use vllm
version	Workload card → Version	e.g. 0.6.3. For AMD ROCm, use a -rocm-tagged build
quantization_code	Workload card → Quantization	fp16, bf16, fp8, int8, awq, gptq. Default: fp16
tensor_parallel_size	Engine args (advanced)	Defaults to GPU count on the selected server
max_num_seqs	Engine args (advanced)	Defaults to max(256, concurrency × 2)
max_num_batched_tokens	Engine args (advanced)	Defaults: chunked-prefill-friendly value based on ISL/OSL
block_size	Engine args (advanced)	Default 16. Raise for very long contexts
gpu_memory_utilization	Engine args (advanced)	Default 0.90
enforce_eager	Engine args (advanced)	Default false. Set true to disable CUDA graphs for debugging
enable_prefix_caching	Engine args (advanced)	Default true. Powers KV-cache reuse measurements

Multi-node vLLM is not supported in v4.0 ~ see Release Notes.

Engine args templates live under the planned Settings -> Engine args templates Admin panel (not yet in the live UI); reference them via engine_args_template_code on a workload to avoid setting individual parameters. The underlying templates are managed via API today.

SGLang

Parameter	Where set	Notes
framework_code	Workload card → Framework	Use sglang
version	Workload card → Version	e.g. 0.3.5
quantization_code	Workload card → Quantization	fp16, bf16, fp8. awq/gptq supported on some models
tp_size	Engine args	Defaults to GPU count
mem_fraction_static	Engine args	Default 0.88
schedule_policy	Engine args	lpm (default), fcfs, dfs-weight
chunked_prefill_size	Engine args	Default 8192

TensorRT-LLM

NVIDIA-only. Requires pre-built engine files for the target (model, quantization, GPU) tuple.

Parameter	Where set	Notes
framework_code	Workload card → Framework	Use trt-llm
version	Workload card → Version	e.g. 0.13.0. Must match the engine file's TRT-LLM version
engine_preset_code	Workload card → Engine preset	References a pre-built engine bundle
kv_cache_free_gpu_mem_fraction	Engine args	Default 0.9
max_batch_size	Engine args	Default 64
max_num_tokens	Engine args	Default depends on engine preset

Engine presets are managed under the planned Settings -> Engine presets Admin panel (not yet in the live UI). For Llama 70B TP4 specifically, the platform ships pre-built presets ~ see schema file 37_llama70b_tp4_engine_presets.sql.

Other registered frameworks

The framework catalog (v_frameworks_versions) also exposes framework_code values you'll see in the Sizer dropdowns and project workload card today, but with varying levels of end-to-end support:

framework_code	Status
vllm	Primary backend, fully supported.
sglang	Primary backend, fully supported.
trt-llm	Primary backend, fully supported (NVIDIA only).
tgi	HuggingFace TGI ~ wired in the catalog, intended for parity benchmarks.
llama-cpp	llama.cpp ~ wired in the catalog, useful for CPU and Apple-silicon comparisons.
ollama	Wired in the catalog; full E2E support deferred to v4.1.
dynamo	NVIDIA Dynamo ~ exposed in Sizer today, end-to-end run path deferred to v4.1.
openai	BYOE-style target for any OpenAI-compatible endpoint.

Use one of the three primary backends for production benchmarking unless you are explicitly comparing against an alternative runner.

---

KYAI v2

KYAI evaluates inference quality through a two-phase pipeline. KYAI execution is wired through the Add KYAI Run action on each project workload card and the underlying RPCs; the dedicated KYAI -> ... navigation surface (Endpoints & Judges, Prompt Sets, Results) referenced below is not yet exposed in the live sidebar in v4.0 ~ those panels are managed via the API today and will land in subsequent releases.

Architecture

• Phase 1 ~ Generation. Runs a candidate endpoint over a prompt set, captures responses.
• Phase 2 ~ Judging. Runs a configured judge over each (instruction, candidate_response, ground_truth) triple and produces a score plus reasoning.

Three first-class objects:

• Candidate endpoint ~ what you're evaluating (a deployed inference server or a BYOE endpoint).
• Judge config ~ the LLM that scores outputs and the scoring method.
• Prompt set ~ instructions + optional ground truths. Either built-in or user-uploaded.

Candidate endpoint parameters

Configured via the planned KYAI -> Endpoints & Judges -> Candidate Endpoints panel (API only in v4.0):

Parameter	Required	Notes
Name	yes	Display name
API base URL	yes	OpenAI-compatible endpoint
Model name	yes	Model identifier sent in the request
Max concurrent requests	yes	In-flight cap; default 8
Retry attempts	no	Default 3
Retry backoff	no	Default exponential, base 500ms
Authentication	no	Optional bearer token or API key
Workspace	yes	Which workspace owns this endpoint

Judge config parameters

Configured via the planned KYAI -> Endpoints & Judges -> Judge Configs panel (API only in v4.0):

Parameter	Required	Notes
Name	yes	Display name
Scoring method	yes	numeric_1_to_10, pairwise_preference, pass_fail, rubric
API base URL	yes	OpenAI-compatible endpoint hosting the judge model
Model name	yes	Judge model (e.g. a 70B+ instruct model)
Max concurrent requests	yes	Default 4. Judges are token-heavy ~ be conservative
Retry attempts	no	Default 3
Judge prompt template	no	Override the default; placeholders {{instruction}}, {{response}}, {{ground_truth}}

Scoring methods:

• numeric_1_to_10 ~ judge returns a 1-10 integer plus reasoning.
• pairwise_preference ~ compare two candidates; judge picks one or "tie".
• pass_fail ~ binary, with reasoning.
• rubric ~ multi-dimensional scoring against a user-supplied rubric.

Prompt sets

Either:

• Built-in. Listed under v_kyai_builtin_sets_with_prompt_count. Curated for common evaluations (general instruction-following, code, math, reasoning).
• User-uploaded. CSV or JSONL with instruction and optional ground_truth columns. Uploaded via the planned KYAI -> Prompt Sets -> Upload panel (API only in v4.0).

Results

Phase-1 outputs in v_kyai_generation_outputs; phase-2 scored outputs in v_kyai_outputs. The planned Results Explorer (KYAI -> Results, not yet exposed in the live UI) supports:

• Job ID filtering
• Score range (min/max)
• Candidate endpoint filter
• Judge config filter

Each row shows: instruction, candidate response, ground truth (if present), score, judge reasoning, candidate endpoint, judge config, prompt-set snapshot name, created timestamp.

Leaderboard

v_kyai_leaderboard ranks scored runs. Visibility follows project visibility ~ Public KYAI runs appear in the cross-workspace leaderboard; Organization-scoped runs in your workspace's view only; Private to the owner only.

---

ASR benchmarking

ASR (Automatic Speech Recognition) benchmarks use the metrumbench-asr tool.

Workload parameters

Parameter	Required	Notes
model_code	yes	ASR model (e.g. whisper-large-v3)
framework_code	yes	vllm or sglang where supported
version	yes	Framework version
concurrency	yes	In-flight transcription requests
audio_duration_bucket	yes	Bucket key ~ see below
dataset_snapshot_code	yes	ASR snapshot reference (v_metrumbench_asr_outputs)

Audio duration buckets

ISL is replaced by audio duration buckets in ASR. Standard buckets:

Bucket	Duration range	Typical use case
xs	< 5 seconds	Short commands
s	5-30 seconds	Voice messages
m	30-120 seconds	Meeting clips
l	2-10 minutes	Podcasts, lectures
xl	10+ minutes	Long-form transcription

Metrics

• RTF (Real-Time Factor) ~ wall-clock seconds per second of audio. RTF < 1 means faster than real-time.
• WER (Word Error Rate) ~ output quality vs ground truth (if dataset has transcripts).
• TTFT (audio) ~ time to first transcribed word.

Dataset snapshots

ASR datasets are versioned via 36_asr_snapshots.sql. The platform ships a set of standard snapshots (LibriSpeech, AMI, etc.); custom snapshots are uploaded via the BYOW flow (see BYOE / BYOW).

---

VLM benchmarking

Vision-language model benchmarking uses the metrumbench-vlm tool.

Workload parameters

Same shape as LLM benchmarking, plus image-related fields:

Parameter	Required	Notes
model_code	yes	VLM (e.g. llava-1.6-mistral-7b, qwen2-vl-7b)
framework_code	yes	vllm, sglang, trt-llm where supported
input_sequence_length	yes	Text portion of the prompt
image_resolution	yes	low, medium, high ~ maps to model-specific patch counts
images_per_request	yes	Typically 1; multi-image supported on some models
output_sequence_length	yes	Standard OSL
concurrency	yes	In-flight requests

Metrics

Same as LLM, with one addition:

• Image preprocessing time ~ included in TTFT. CPU-bound. The Benchmarks page breaks it out separately under the Hardware chart tab (CPU view).

---

BYOE / BYOW

BYOE - Bring Your Own Endpoint

Point Metrum at any OpenAI-compatible inference endpoint.

Parameter	Notes
Endpoint URL	OpenAI-compatible base URL
Model name	What to request
Authentication	Optional bearer/API key
Request timeout	Default 60s; raise for cold-start scenarios
Max concurrent requests	Default 8; honor your endpoint's limits

BYOE endpoints register via the planned KYAI -> Endpoints & Judges -> Candidate Endpoints panel (for evaluation, API only in v4.0) or as a workload-level target (for benchmarking).

Limitations:

• The benchmark runner kills BYOE jobs whose TTFT exceeds 60 seconds (configurable per endpoint up to 600s).
• Telemetry is unavailable ~ Metrum doesn't have access to your endpoint's hardware. GPU power, utilization, and memory will be empty for BYOE jobs.

BYOW - Bring Your Own Workload

Supply your own prompt dataset for benchmarking or KYAI evaluation.

For benchmarking: upload a CSV/JSONL prompt library via the planned Settings -> Prompt libraries panel (API only in v4.0). Reference it from a workload via prompt_library_code.

For KYAI: see Prompt sets above.

CSV format for benchmarking workloads:

prompt,expected_output_tokens
"Translate to French: Hello world",10
"Summarize the following article: ...",256

Expected output tokens is optional; if absent, the workload's OSL is used.

---

Sizer

Hardware recommendation tool. Full walkthrough in the Sizer Usage Guide.

Inputs

Parameter	Required	Notes
model_code or model class	yes	Specific model preferred
Use case	yes	Chat, RAG, code completion, batch, other
Concurrent users OR tokens/sec	yes	One of these must be set
TTFT P95	no	If set, Sizer enforces as constraint
TPOT P95	no	If set, Sizer enforces as constraint
ISL	yes	Input sequence length, tokens
OSL	yes	Output sequence length, tokens
Quantization preference	no	Defaults to "best available"
Hardware constraint	no	Restrict to a vendor or SKU family

Output

Ranked list. Each entry: hardware + quantity, framework + quantization, predicted TTFT/TPOT/throughput, confidence, cost per million tokens, notes.

Backed by reference benchmark data in v_benchmark_results and the 45_sizer_reference_views.sql views.

---

InferenceX

Replaces the v3 InferenceMax engine. Computes capacity estimates with confidence intervals.

What it produces

For a given (model, framework, quantization, GPU, scenario shape) tuple, InferenceX estimates:

• Maximum sustainable throughput (tokens/sec).
• Maximum concurrency before latency SLA violation.
• Cost per million tokens.
• Confidence interval (lower, upper) on each estimate.

Available via v_inferencex_outputs. The Sizer tool layers on top of these estimates.

Ingest

The InferenceX pipeline ingests fresh benchmark results into the estimation model on each completed run, via the 34_inferencex_ingest_rpc.sql RPC. There's no manual ingest step.

---

Benchmarks (formerly Pulse)

The Benchmarks page (/benchmarks) is the results explorer. The page subtitle reads "Full results across all benchmark runs. Filter, compare, and analyse performance." The marketing name "Pulse" still appears in some legacy material; the live UI surface is named Benchmarks.

Filters

The Benchmarks page exposes a Filters dropdown panel and a My runs toggle pill above the results table. The filterable dimensions are:

Filter	Backed by
Status	v_benchmark_results.status_code
Tool	v_benchmark_results.tool_code
Model	v_benchmark_results.model_code
Framework	v_benchmark_results.framework_code
Server config	v_benchmark_results.config_code
Date range	v_benchmark_results.created_at
My runs only	v_benchmark_results.owner_account_id

Charts

The chart panel below the results table has the following sub-tabs (live UI labels):

Tab	View	What it shows
Model	v_model_comparison	Throughput, latency, GPU utilization across models
Framework	v_framework_comparison	Same metrics grouped by framework
Quantization	v_quantization_comparison	Impact of quantization on throughput and quality
Concurrency	v_concurrency_scaling	Throughput and latency vs concurrency
Cost	v_cost_efficiency	tokens/$ and tokens/hour
Hardware	v_hardware_utilization	GPU power, util, memory, CPU, RAM

Pagination

Range-based via Range/Range-Unit: items headers. See API Reference - Pagination.

Exporting

CSV export of the filtered results view is planned; the explicit Export -> CSV button on the results table is not yet exposed in v4.0. Use the API to pull the filtered set today.

---

Pre/Post-run validation

Pre-run

Runs before the first job starts. Checks:

• Model is accessible to the target server (via HuggingFace if applicable).
• Framework + version + accelerator combination is supported.
• GPU memory is sufficient for the largest scenario in the run.
• HF token (if required) is present and valid.
• Server is online with recent heartbeat.

Failures surface as a structured error on the Run Detail page with a specific suggestion (e.g. "use quantization_code=fp8" or "select a server with ≥160 GB GPU memory").

Post-run

Runs after the last job completes. Checks:

• Output token counts in plausible ranges.
• No all-zero latency rows.
• Telemetry actually attached to each job.

Surfaces as a "results may be unreliable" badge on the Run Detail page if any check fails. Doesn't block the run from completing ~ it just flags suspect data.

Configuration

Validation severity is configurable per-workspace via the planned Settings -> Validation Admin panel (not yet exposed in the live UI; managed via API today):

• Strict (default) ~ fail-fast on any pre-run issue.
• Warn ~ log warnings but allow the run to proceed.
• Off ~ disable (not recommended; useful only for debugging the validator itself).

---

Metering and Stripe billing

See Admin Guide → Billing for the operational view.

Metered events

Event	Recorded in
GPU-hour (cloud)	v_cloud_resource_lifecycle
Cloud spend pass-through	v_run_cost_summary
KYAI judge tokens	v_kyai_outputs (aggregated)
API request volume	Internal; will surface under the planned Billing -> Usage view

Quotas

Configured via the planned Settings -> Billing -> Quotas Admin panel (not yet exposed in the live UI):

• GPU-hours (monthly)
• Judge tokens (monthly)
• Cloud spend (monthly, in USD)

Hitting a quota returns HTTP 402 Payment Required at job submission.

---

RBAC

Four roles: Owner, Admin, Engineer, Viewer. See Admin Guide → RBAC for the full responsibility matrix.

Enforcement points

Surface	Where enforced
RPC calls	19_rbac.sql ~ checked per-RPC
View reads	Row-level security policies on each view
UI	Frontend hides forbidden actions; not source of truth

Custom roles

Custom roles are not supported in v4.0 ~ the four built-in roles are fixed. Per-project visibility (Private / Organization / Public) provides per-resource access control alongside the role system.

---

Cloud provisioning (Shadeform)

Unified provisioning across RunPod, Shadeform, Lambda, and GKE.

Configuration

Configured via the planned Settings -> Cloud credentials Admin/Owner panel (not yet exposed in the live UI; managed via API today). Per-provider keys are stored encrypted-at-rest.

Per-user overrides via the planned Profile -> Cloud credentials panel ~ if set, used in place of workspace credentials for jobs submitted by that user.

Provider matrix

See Admin Guide - Cloud provisioning for the provider x GPU matrix.

Lifecycle parameters

Parameter	Default	Notes
Idle teardown timeout	5 min	Configurable per cloud provider
Provisioning timeout	15 min	If exceeded, request fails with provisioning_timeout
Auto-retry on failure	1	One automatic retry on transient provider errors
Max instance lifetime	24 hr	Hard cap; instances are torn down regardless of activity

Cost controls

• Per-workspace monthly cap (Owner).
• Per-user daily cap (Admin or Owner).

Both surface under the planned Settings -> Billing Admin panel (not yet exposed in the live UI).

---

Google TPU

GKE-managed TPU v5e and v5p support.

Prerequisites

• GKE cluster with TPU v5e or v5p node pools.
• Workload Identity enabled.
• Service account JSON registered via the planned Settings -> Cloud credentials -> GKE panel (API only in v4.0).

Workload configuration

Parameter	Notes
framework_code	vllm (TPU backend) ~ SGLang and TRT-LLM not supported on TPU
gpu_code	tpu-v5e or tpu-v5p
tpu_topology	e.g. 2x2, 4x4 ~ defines the slice

Telemetry

TPU telemetry uses the dedicated v_tpu_telemetry_job_summary view. Per-chip utilization, HBM usage, and tensorcore activity are surfaced.

Limitations

• VLM and ASR support on TPU are experimental ~ see Hardware Compatibility.
• Sizer recommendations are NVIDIA/AMD only in v4.0.

---

GDPR and compliance

See Admin Guide → GDPR and compliance for the operational view.

Configurable retention

Per category, via the planned Settings -> Compliance -> Data retention Admin panel (not yet exposed in the live UI):

• Audit logs: 90 days to 7 years
• Telemetry: 90 days to 365 days
• Benchmark results, KYAI outputs: indefinite by default, per-workspace TTL configurable

Data deletion

POST /rpc/request_account_deletion initiates a 14-day pending deletion. After the window, the account is hard-deleted; audit and results data is anonymized (account ID hashed).

---

Backup and recovery

See Admin Guide → Backup and recovery.

Schedule

• Nightly automatic snapshot, 30-day retention.
• Manual snapshots via the planned Settings -> Backup -> Take snapshot Admin panel (not yet exposed in the live UI).

Restore

Support-assisted in v4.0. Self-service restore is planned for a later release.

---

See also

• User Guide ~ workflows that combine these features.
• API Reference ~ programmatic access to every feature.
• Admin Guide ~ operational settings and access control.
• Hardware Compatibility ~ supported accelerators per feature.