Benchmark Lifecycle

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

A useful benchmark starts with a decision record, not with a job launch. Metrum Insights keeps the decision, execution, telemetry, and reports traceable through project, workload, scenario, server, run, and job ids.

1. Decide The Benchmark Question

Capture the question in plain language. A good benchmark question names the thing being varied, the things being held constant, and the metric that decides the answer.

Common decision questions:

Question	Hold Constant	Vary	Primary Metrics
How does hardware config A perform vs hardware config B for the same model?	Model, framework, version, quantization, command template, traffic matrix	Server config and concrete server instance	Output tokens/sec, tokens/sec/accelerator, TTFT, TPOT, p95/p99 latency, utilization, cost/run
How does hardware config A compare across a list of models?	Framework version, traffic matrix, quantization policy	Model and server config	Tokens/sec, tokens/sec/accelerator, memory headroom, error rate, model load time
How does model A perform vs model B on the same hardware?	Server instance, framework, quantization policy, traffic matrix	Model	Throughput, TTFT, TPOT, latency percentiles, quality score, cost per million tokens
How does accuracy change with quantization level?	Model family, dataset/snapshot, prompts, judge/eval method, hardware if possible	BF16/FP16/FP8/INT8/AWQ/GPTQ/vendor format	KYAI score, task accuracy, error rate, refusal/regression notes, throughput gain
How do concurrency, ISL, and OSL affect serving behavior?	Model, hardware, framework, command template	concurrency x input_sequence_length x output_sequence_length	Tokens/sec, TTFT, TPOT, p50/p95/p99 latency, failures/timeouts
What is the throughput per accelerator across hardware options?	Model, framework, quantization, traffic shape	GPU/accelerator type and count	Tokens/sec, tokens/sec/GPU or accelerator, tokens/sec/watt, tokens/sec/dollar
Which runtime recipe is better for the same system?	Model, hardware, traffic matrix	Engine args set, command template, KV cache settings, offload, scheduler limits	Throughput, latency, memory, stability, quality impact
Does a workload fit reliably on this exact installed host?	Model and runtime recipe	Concrete server instance	Readiness, model load success, memory headroom, telemetry, SUT evidence

Use the same scenarios when comparing across workloads or hardware. For example, a hardware comparison should run identical isl128-osl128-c32, isl128-osl1024-c256, isl1024-osl1024-c512, and isl1024-osl128-c1024 scenarios on each server config. A model comparison should reuse the same hardware instance and scenario codes for every model.

Before creating the project, answer these setup questions:

• Which model or model family are you evaluating?
• Are you comparing one model, a model list, or one model across runtime recipes?
• Which modality is in scope: LLM text, VLM image+text, ASR/audio, metrumbench-imagegen image generation, or KYAI evaluation?
• Which serving framework and version are in scope?
• Which precision and quantization should be tested: BF16, FP16, FP8, INT8, AWQ, GPTQ, or provider-specific formats?
• Which traffic shapes matter: input sequence length, output sequence length, image count, audio duration, concurrency, request count, streaming mode, and batch behavior?
• Which hardware target is being evaluated: a generic class, a specific system configuration, or one concrete installed host?
• What normalized metrics do you need: tokens/sec, tokens/sec/GPU, tokens/sec/accelerator, tokens/sec/watt, tokens/sec/dollar, or cost per million tokens?
• What quality metric is required: KYAI score, task accuracy, judge score, benchmark-specific exact match, or human review?
• What does success mean: throughput, TTFT, TPOT, latency percentiles, utilization, memory headroom, error rate, cost, or quality score?

The answer determines project structure:

• hardware comparison: same workloads and scenarios, multiple server configs or instances;
• model comparison: multiple workloads, same scenarios and hardware;
• quantization study: one workload per quantization/runtime recipe, same dataset and quality-evaluation method;
• traffic sweep: one scenario per ISL x OSL x concurrency point;
• accelerator-normalized report: collect accelerator count, power, cost, and SUT evidence for every run.

2. Create The Project

A project groups one benchmark campaign. Use separate projects when the answer will be reported separately, for example:

• deepseek-qwen15b-rtxpro6000-smoke
• vlm-llava-h200-serving-comparison
• kyai-quantum-dataset-candidate-eval

The project owns workloads, scenarios, runs, jobs, artifacts, reports, and access control.

API details: Project Lifecycle.

For a complete copy-pasteable API flow, start with End To End Benchmark API Example.

3. Create Workloads

A workload is the model/tool/framework definition. Create separate workloads when any of these change:

• modality or benchmark tool, such as metrumbench-llm, GenAI-Perf, KYAI, metrumbench-vlm, or metrumbench-imagegen;
• model code or model endpoint;
• serving framework and version;
• engine argument preset;
• precision or quantization mode;
• command template family.

Example workload split:

Workload	Purpose
deepseek-qwen15b-vllm-bf16	Baseline vLLM BF16 serving.
deepseek-qwen15b-vllm-fp8	FP8 serving comparison.
deepseek-qwen15b-dynamo-kv-offload	Dynamo deployment with KV offload.
kyai-candidate-openrouter	Candidate endpoint for KYAI scoring.

API details: metrumbench-llm Workload And Scenario.

4. Create Scenarios

A scenario is the traffic shape. Keep scenario codes readable and stable:

• isl128-osl128-c32
• isl128-osl1024-c256
• isl1024-osl1024-c512
• isl1024-osl128-c1024

For sweeps, create one scenario per point. This keeps jobs and results easy to compare. Include num_requests large enough to stabilize metrics at the chosen concurrency.

5. Model And Select Hardware

Metrum Insights distinguishes hardware classes from concrete systems:

• hardware catalog: vendors, GPU models, CPU models, chassis/platforms;
• server config: an exact benchmarkable configuration, such as 1x RTX PRO 6000 Blackwell, CPU model, RAM, disk, NIC, driver assumptions, and framework support;
• server instance: one concrete host or cloud instance with hostname, provider metadata, heartbeat state, and onboarding token.

Do not benchmark against a vague label such as "RTX PRO 6000" when the report needs to compare real systems. Record the exact instance and make the SUT export prove what ran.

Guide: Hardware Modeling.

6. Onboard Or Provision Capacity

For existing on-prem systems, register the server, generate onboarding, run the worker install command, and wait for readiness.

For disposable cloud systems, use managed onboarding where available. The managed path records provider instance ids, server ids, bootstrap state, and teardown evidence.

API details:

• Hardware Onboarding
• Cloud Benchmark Execution

7. Execute Runs And Jobs

Launch scenarios only after the target server is ready. A healthy run should produce:

• run id and job id;
• job status transitions;
• framework and tool command resolution;
• benchmark outputs;
• telemetry samples;
• logs and SUT artifacts.

Stop and diagnose when command templates have unresolved placeholders, workers stop heartbeating, or jobs stay pending after the bounded wait.

8. Collect Evidence From The Database

Use database-backed evidence rather than console screenshots:

• v_benchmark_results for throughput, latency, model, framework, and hardware;
• v_metrumbench_llm_outputs for per-job metrumbench-llm metrics;
• telemetry summary RPCs for utilization, memory, power, and duration;
• job logs and SUT CSV downloads for reproducibility;
• cloud cost summaries for price/performance analysis.

API details: Results And Reports.

9. Package And Compare Results

A final benchmark package should include:

• project, workload, scenario, run, job, server, and provider ids;
• model, framework, precision, command template, and engine args;
• hardware instance and SUT summary;
• throughput, TTFT, TPOT, latency percentiles, error rate, and concurrency;
• telemetry and cost summaries;
• report URLs and downloadable artifacts;
• caveats, failures, and rerun instructions.