Workflow Examples Overview

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Metrum Insights benchmarks are easiest to reason about when each project has a single question, each workload represents one runtime recipe, and each scenario represents one traffic or evaluation point.

Start with the End To End Benchmark Runbook for a complete Shadeform + metrumbench-llm + KYAI example. Use the patterns below when the benchmark question changes.

Detailed workflow pages:

Common Setup

For API examples, use these variables:

export METRUM_WEBUI_URL="https://<control-plane-host-or-dns>"
export METRUM_API_URL="https://api.<control-plane-domain>"
export METRUM_JWT_TOKEN="<postgrest-jwt>"
export METRUM_ACCOUNT_ID="<account-uuid>"
export METRUM_ORG_ID="<org-uuid>"
export PROJECT_NAME="<project-name>"
export SERVER_HOSTNAME="<registered-worker-hostname>"

Use METRUM_WEBUI_URL for browser workflows. Use the Caddy API METRUM_API_URL for shell examples; it reverse-proxies to PostgREST behind the edge. The new Web UI /api/postgrest path is an authenticated browser proxy, not the recommended endpoint for standalone curl commands.

Web UI users perform the same lifecycle from the project page:

Create or open the project.
Add one workload for every model, framework, version, tool, precision, or engine-args recipe being compared.
Add scenarios for each traffic shape or quality dataset slice.
Run a smoke scenario first.
Run the full matrix.
Review results, telemetry, logs, SUT evidence, and exports.

Compare metrumbench-llm, genai-perf, And inferenceX

Use this when you want to know whether different benchmark tools report consistent performance for the same model and hardware.

Hold constant:

model, precision, and model-server command;
hardware config and concrete server instance;
framework and framework version;
ISL, OSL, concurrency, request count, and streaming mode;
endpoint and port if the tools call an already running server.

Vary:

benchmark tool: metrumbench-llm, genai-perf, and inferencex.

Project structure:

Object	Example
Project	`qwen15b-tool-comparison-h100`
Workload 1	`qwen15b-vllm020-metrumbench-llm`
Workload 2	`qwen15b-vllm020-genaiperf`
Workload 3	`qwen15b-vllm020-inferencex`
Scenarios	`isl128-osl128-c32`, `isl128-osl1024-c64`, `isl1024-osl128-c512`

API pattern:

for tool in metrumbench-llm genai-perf inferencex; do
  workload_code="qwen15b-vllm020-${tool}"

  curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
    -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
      \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
      \"p_project_name\":\"$PROJECT_NAME\",
      \"p_workload_name\":\"Qwen 1.5B vLLM 0.20 ${tool}\",
      \"p_workload_code\":\"$workload_code\",
      \"p_tool_code\":\"$tool\",
      \"p_model_code\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
      \"p_framework_code\":\"vllm\",
      \"p_version\":\"0.20.0\",
      \"p_quantization_code\":\"bf16\",
      \"p_engine_args_set_code\":\"vllm-default\"
    }" | jq
done

Create identical scenario codes under each workload. Execute each tool workload on the same SERVER_HOSTNAME, then compare by workload_code, scenario_code, and job_id.

curl -fsS "$METRUM_API_URL/v_benchmark_results?org_id=eq.$METRUM_ORG_ID&project_name=eq.$PROJECT_NAME&select=workload_code,scenario_code,server_hostname,output_tokens_per_second,ttft_ms,tpot_ms,p95_latency_ms,error_count" \
  -H "X-Org-Id: $METRUM_ORG_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Interpretation:

metrumbench-llm is the recommended temporal serving baseline for ISL/OSL sweeps.
genai-perf is useful when you need compatibility with NVIDIA-oriented performance reporting.
inferenceX is useful when comparing against its client/runtime conventions.
If tools disagree, inspect prompt generation, streaming mode, tokenizer behavior, warmup, request timeout, and whether all tools measured the same model-server process.

Compare Hardware A And Hardware B

Use this when the model and runtime are fixed and the hardware is the variable.

Hold constant:

model, framework, version, quantization, engine args set;
scenario matrix;
benchmark tool;
dataset or prompt source;
model-server replica count unless that is the planned variable.

Vary:

server_config_code and concrete server_config_instance;
optionally cloud provider, CPU, RAM, disk, network, GPU count, and cost.

Recommended project layout:

Hardware	Server Config	Server Instance
H100	`shadeform-h100-1x`	`shadeform-h100-001`
H200	`shadeform-h200-1x`	`shadeform-h200-001`
B200	`shadeform-b200-1x`	`shadeform-b200-001`

Create one workload and run the same scenarios on every server:

for server in shadeform-h100-001 shadeform-h200-001 shadeform-b200-001; do
  for scenario in isl128-osl128-c32 isl128-osl1024-c64 isl1024-osl128-c512; do
    curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
      -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
        \"p_project_name\":\"$PROJECT_NAME\",
        \"p_workload_code\":\"qwen15b-vllm020-metrumbench-llm\",
        \"p_scenario_code\":\"$scenario\",
        \"p_server_hostname\":\"$server\",
        \"p_manage_model_server\":true,
        \"p_job_name\":\"$scenario-on-$server\"
      }" | jq
  done
done

Report both raw throughput and normalized throughput:

output tokens/sec;
output tokens/sec/GPU or tokens/sec/accelerator;
output tokens/sec/watt when power telemetry exists;
output tokens/sec/dollar when cloud cost is known;
TTFT, TPOT, p95, p99, and failure rate.

Compare Model A And Model B

Use this when the hardware and runtime are fixed and the model is the variable.

Hold constant:

server instance;
framework and version;
engine args set where compatible;
precision/quantization policy;
traffic matrix and request count.

Vary:

p_model_code;
workload code and name.

Create one workload per model:

for model in \
  "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B" \
  "Qwen/Qwen3-8B"; do
  slug="$(printf '%s' "$model" | tr '/[:upper:]' '-[:lower:]' | tr -cs 'a-z0-9-' '-')"

  curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
    -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
      \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
      \"p_project_name\":\"$PROJECT_NAME\",
      \"p_workload_name\":\"$model vLLM 0.20\",
      \"p_workload_code\":\"${slug}vllm020\",
      \"p_tool_code\":\"metrumbench-llm\",
      \"p_model_code\":\"$model\",
      \"p_framework_code\":\"vllm\",
      \"p_version\":\"0.20.0\",
      \"p_engine_args_set_code\":\"vllm-default\"
    }" | jq
done

Use the same scenario codes for every workload. Compare throughput and latency, then add KYAI if the decision also depends on response quality.

Measure Model Output Quality With KYAI

KYAI evaluates a candidate model endpoint against a snapshot or built-in prompt set. Use it in the same project as the performance workload when you need a single report that covers both speed and quality.

Hold constant:

prompt snapshot or built-in set;
judge configuration;
generation system prompt;
candidate endpoint;
number of samples.

Vary:

model, quantization, engine args, or hardware, depending on the question.

Typical API sequence:

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_candidate_endpoint" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_endpoint_code\":\"qwen15b-local-vllm\",
    \"p_endpoint_name\":\"Qwen 1.5B local vLLM\",
    \"p_api_base_url\":\"http://127.0.0.1:8000/v1\",
    \"p_model_name\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
    \"p_framework_code\":\"vllm\",
    \"p_framework_version\":\"0.20.0\",
    \"p_config_code\":\"shadeform-vllm-starter\"
  }" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"qwen15b-kyai\",
    \"p_workload_name\":\"Qwen 1.5B KYAI quality\",
    \"p_candidate_endpoint_code\":\"qwen15b-local-vllm\"
  }" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"qwen15b-kyai\",
    \"p_scenario_code\":\"quantum-30\",
    \"p_scenario_name\":\"Quantum 30 prompt quality\",
    \"p_snapshot_code\":\"quantum-mechanics\",
    \"p_num_samples\":30,
    \"p_batch_size\":4,
    \"p_max_concurrent_requests\":8
  }" | jq

Run generation first, then run judging with the returned generation job id. Review v_kyai_results for per-prompt scores, reasoning, and failures.

Add Your Own KYAI Dataset

Use a KYAI snapshot when the question is spatial quality: correctness on a fixed set of instructions, images, domains, or task types.

Dataset rows should include:

Column	Purpose
`instruction`	Prompt sent to the candidate model.
`ground_truth_response`	Reference answer used by the judge or scorer.
`external_row_id`	Stable id from your source dataset.
`row_index`	Reproducible ordering for sampling.

API flow:

Create a snapshot with create_kyai_snapshot.
Insert rows into kyai_snapshot_prompt_staging with one shared session_id.
Call import_kyai_snapshot_prompts with the snapshot code and session id.
Use that snapshot_code in create_kyai_scenario.

Default datasets can be discovered with:

curl -fsS "$METRUM_API_URL/kyai_snapshots?select=snapshot_code,snapshot_name,source_type&order=snapshot_code.asc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/v_kyai_builtin_sets_with_prompt_count?select=set_code,set_name,prompt_count&order=set_code.asc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Use temporal datasets for metrumbench-llm, genai-perf, and inferenceX when the question is serving behavior under traffic shape. Use KYAI datasets when the question is model output quality.

Compare vLLM, SGLang, And TensorRT-LLM

Use this when the model and hardware stay fixed but the serving engine changes.

Hold constant:

model;
hardware instance;
precision and quantization intent;
scenarios;
benchmark tool;
target endpoint behavior.

Vary:

p_framework_code;
p_version;
p_engine_args_set_code;
command template and container image.

Recommended workload split:

Workload	Framework	Engine Args Set
`qwen15b-vllm020`	`vllm`	`vllm-default`
`qwen15b-sglang`	`sglang`	SGLang args set selected by admin
`qwen15b-trtllm`	`trtllm`	TensorRT-LLM args set selected by admin

Run the same scenario matrix for every workload. If one engine requires a different max sequence length or batch limit, record that as a recipe change in the workload name and report caveats.

Compare vLLM Versions

Use this when you want to test whether a vLLM upgrade improves throughput, latency, or stability.

Hold constant:

model;
hardware;
quantization;
traffic matrix;
command template family;
engine args values.

Vary:

vLLM version, for example 0.19.0, 0.20.0, 0.20.1;
container image or CUDA image tag when version implies a different image.

Create one workload per version:

for version in 0.19.0 0.20.0 0.20.1; do
  code="qwen15b-vllm-${version//./}"

  curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
    -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
      \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
      \"p_project_name\":\"$PROJECT_NAME\",
      \"p_workload_name\":\"Qwen 1.5B vLLM $version\",
      \"p_workload_code\":\"$code\",
      \"p_tool_code\":\"metrumbench-llm\",
      \"p_model_code\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
      \"p_framework_code\":\"vllm\",
      \"p_version\":\"$version\",
      \"p_engine_args_set_code\":\"vllm-default\"
    }" | jq
done

Interpretation:

A version win should improve at least one primary metric without increasing failures, memory pressure, or output-quality regressions.
If a newer image changes CUDA, tokenizer behavior, or default scheduling, say that in the report.
Keep the old version in the same project so side-by-side comparisons are traceable.

Test Engine Args Impact

Use this when you want to tune vLLM or another engine. Each recipe should be a separate workload or engine args set so results remain reproducible.

Common variables:

Variable	Examples	Why It Matters
Attention backend	flash attention, paged attention, vendor backend	Changes memory use and latency.
KV cache dtype	fp16, bf16, fp8	Trades quality/risk against memory and throughput.
KV cache size	GPU memory utilization, cache block size	Controls max concurrency and OOM risk.
Max model length	4096, 8192, 32768	Determines supported ISL/OSL and memory footprint.
Prefix caching	on/off	Helps repeated-prefix workloads.
Scheduler settings	max batched tokens, max sequences	Controls batching and TPOT under load.
Parallelism	tensor parallel, data parallel, replicas	Changes scaling, ports, and GPU assignment.
Speculative decoding	draft model, token count	Can improve decode speed with added complexity.
Quantization	bf16, fp8, awq, gptq	Changes memory, speed, and quality.

Recommended structure:

Workload	Engine Args Set	Purpose
`qwen15b-vllm-default`	`vllm-default`	Baseline.
`qwen15b-vllm-fp8kv`	`vllm-kv-cache-fp8`	KV cache dtype comparison.
`qwen15b-vllm-longctx`	`vllm-max-len-32768`	Long-context memory and latency.
`qwen15b-vllm-prefix-cache`	`vllm-prefix-cache`	Repeated-prefix traffic.

Admin-created command templates and engine args sets should encode the actual startup arguments. Operators should not edit commands by hand during a run.

Test Image Generation With metrumbench-imagegen

Use metrumbench-imagegen when the output is generated images rather than text tokens.

Hold constant:

image model;
prompt dataset or snapshot;
image size, count, quality, and scheduler settings;
hardware and command template.

Vary:

model;
hardware;
image resolution;
batch size;
serving engine or container image.

Project structure:

Object	Example
Project	`sdxl-metrumbench-imagegen-h100-sweep`
Workload	`sdxl-metrumbench-imagegen-h100`
Scenarios	`img1024-b1`, `img1024-b4`, `img512-b8`

Collect:

images/sec or seconds/image;
p50/p95 generation time;
GPU utilization and memory;
failed image count;
output artifact paths;
qualitative review notes when needed.

If the Web UI does not expose metrumbench-imagegen workload creation in your build, create the workload through the API and inspect jobs/results from the project page.

Test Google TPU

Use TPU workflows when the hardware target is a TPU slice rather than a GPU server.

Document the exact TPU target:

TPU generation and slice shape;
host CPU and memory;
framework and version;
model;
compiler/runtime options;
dataset and traffic matrix.

Keep TPU comparisons separate from GPU comparisons unless the report explicitly normalizes by accelerator count, cost, and power. TPU telemetry can differ from GPU telemetry, so the report should state which utilization and memory fields are available.

Test SSH Hardware With BYOH

Use BYOH when the hardware is available through SSH and Metrum Insights should still set up the worker, start the model server, run readiness checks, execute the benchmark, and ingest results through the normal job lifecycle.

BYOH is the right path for manually provisioned Vast.ai containers, manually created cloud VMs, on-prem lab servers, and customer-owned accelerator hosts. It is not just a direct endpoint smoke test: the benchmark still runs through project, workload, scenario, execute_scenario, worker, job, and result views.

Operator flow:

Confirm provider or hardware owner, accelerator, model, expected cost, and teardown expectation.
Expose PostgREST and WebUI/packages through public DNS or Cloudflared URLs.
Register the server config and concrete server instance.
Install target-side prerequisites, including pyenv, uv, nvm, and shell bootstrap tools.
Bootstrap the Metrum benchmark worker with Restic credentials and the explicit package tag.
Create workload and scenario records, dispatch execute_scenario, and verify the resolved model-server command before the benchmark measures it.
Monitor worker logs, model-server logs, job status, v_metrumbench_llm_outputs, and teardown.

See Bring Your Own Hardware for the concrete Vast.ai RTX PRO 6000 example.

Test Custom Model Servers With BYOE

BYOE means Metrum Insights benchmarks an endpoint you operate. Metrum Insights does not start or stop the model server for that workload.

Use BYOE when:

the hardware is not managed by Metrum Insights;
the model server is custom;
the accelerator is non-NVIDIA or has a vendor runtime;
the endpoint is already deployed in Kubernetes, on-prem, or another cloud.

Operator flow:

Register the hardware or endpoint owner in the project notes.
Create a BYOE workload with the endpoint URL, transport, model name, and authentication reference.
Create the same scenarios used for managed workloads.
Run scenarios against the endpoint.
Attach external telemetry or SUT evidence when the worker cannot collect it directly.

Report caveats should say which responsibilities are outside Metrum Insights: model-server startup, autoscaling, load balancer behavior, endpoint auth, external telemetry, and teardown. If the starting point is SSH hardware rather than an already-running endpoint, use BYOH instead.

Download Data For One Project Or Many Projects

Use API exports for reproducible analysis. Export CSV for spreadsheets or JSON for automated analysis.

Single project:

curl -fsS "$METRUM_API_URL/v_benchmark_results?org_id=eq.$METRUM_ORG_ID&project_name=eq.$PROJECT_NAME&select=*" \
  -H "X-Org-Id: $METRUM_ORG_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Accept: text/csv" \
  -o "benchmark-results-$PROJECT_NAME.csv"

curl -fsS "$METRUM_API_URL/v_benchmark_results?org_id=eq.$METRUM_ORG_ID&project_name=eq.$PROJECT_NAME&select=job_id,tool_code" \
  -H "X-Org-Id: $METRUM_ORG_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -o /tmp/metrum-benchmark-jobs.json

export METRUMBENCH_LLM_JOB_IDS="$(
  jq -r '[.[] | select(.tool_code=="metrumbench-llm") | .job_id] | join(",")' \
    /tmp/metrum-benchmark-jobs.json
)"

curl -fsS "$METRUM_API_URL/v_metrumbench_llm_outputs?job_id=in.($METRUMBENCH_LLM_JOB_IDS)&select=*" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Accept: text/csv" \
  -o "metrumbench-llm-outputs-$PROJECT_NAME.csv"

curl -fsS "$METRUM_API_URL/v_kyai_results?select=*" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Accept: text/csv" \
  -o "kyai-results.csv"

Multiple projects:

curl -fsS "$METRUM_API_URL/v_benchmark_results?org_id=eq.$METRUM_ORG_ID&project_name=in.(project-a,project-b,project-c)&select=*" \
  -H "X-Org-Id: $METRUM_ORG_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Accept: text/csv" \
  -o "benchmark-results-project-set.csv"

For final reports, include:

project, run, job, workload, scenario, server, and provider ids;
model, framework, version, quantization, command template, and engine args;
raw and normalized performance metrics;
quality metrics when KYAI is used;
telemetry and SUT evidence;
logs and failed-job notes;
cost and teardown evidence for cloud runs.

Admin: Add Command Templates Without Code Changes

Admins add new runtime recipes by registering framework versions, command templates, and engine args sets. Operators then select those recipes as workloads without changing Metrum Insights code.

Template design checklist:

name the framework, version, image, and supported hardware;
define required placeholders such as model id, port, tensor parallel size, and cache settings;
provide defaults for optional arguments;
include a health check or readiness condition;
state whether the template supports managed lifecycle or BYOE only;
validate that rendered commands have no unresolved placeholders;
test with a smoke scenario before publishing to operators.

Example recipe families:

Recipe	Example Arguments
Default vLLM	`--model`, `--served-model-name`, `--port`, `--tensor-parallel-size`
vLLM FP8 KV cache	`--kv-cache-dtype fp8`, cache scaling flags, memory utilization
Long context	`--max-model-len 32768`, adjusted batched-token limits
SGLang	scheduler and memory fraction arguments
TensorRT-LLM	engine path, runtime options, batch limits
Vendor accelerator	vendor runtime launcher, device selection, telemetry adapter

Changing a command template changes the runtime recipe. Create a new engine args set or versioned template when you need results to remain comparable over time.

Private Stack Exposure​

Common Setup​

Compare metrumbench-llm, genai-perf, And inferenceX​

Compare Hardware A And Hardware B​

Compare Model A And Model B​

Measure Model Output Quality With KYAI​

Add Your Own KYAI Dataset​

Compare vLLM, SGLang, And TensorRT-LLM​

Compare vLLM Versions​

Test Engine Args Impact​

Test Image Generation With metrumbench-imagegen​

Test Google TPU​

Test SSH Hardware With BYOH​

Test Custom Model Servers With BYOE​

Download Data For One Project Or Many Projects​

Admin: Add Command Templates Without Code Changes​

Private Stack Exposure

Common Setup

Compare metrumbench-llm, genai-perf, And inferenceX

Compare Hardware A And Hardware B

Compare Model A And Model B

Measure Model Output Quality With KYAI

Add Your Own KYAI Dataset

Compare vLLM, SGLang, And TensorRT-LLM

Compare vLLM Versions

Test Engine Args Impact

Test Image Generation With metrumbench-imagegen

Test Google TPU

Test SSH Hardware With BYOH

Test Custom Model Servers With BYOE

Download Data For One Project Or Many Projects

Admin: Add Command Templates Without Code Changes