Model And Hardware Examples

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Use this page as the public operator map for model + hardware benchmark targets. Each example names the benchmark contract first, then shows the provider path used to make that hardware available. Keep the model, hardware, framework, package tag, and scenario matrix fixed during a run; changing the provider should not silently change the benchmark being measured.

Common Operator Path

Every model + hardware run follows the same Metrum lifecycle:

Select or create an account-owned or org-scoped project.
Create a workload for the exact model, framework, framework version, quantization, engine-args set, and benchmark tool.
Create one scenario per traffic shape or dataset slice.
Make hardware available through a managed provider or BYOH SSH host.
Verify the resolved model-server runtime contract before readiness.
Wait for worker readiness, then run the smoke scenario.
Run the remaining scenario matrix only after the smoke succeeds.
Query result views, logs, and artifacts.
Tear down provider hardware or Metrum-managed runtime processes.

For local control planes, expose PostgREST and package downloads through public Cloudflared URLs before provisioning remote hardware:

cloudflared tunnel --url http://localhost:23040
cloudflared tunnel --url http://localhost:25041

Mint 12-hour PostgREST tokens through the database RPC:

curl -fsS -X POST "$METRUM_API_URL/rpc/mint_postgrest_write_jwt" \
  -H "Content-Type: application/json" \
  -d "$(jq -n \
    --arg p_jwt_secret "$PGRST_JWT_SECRET" \
    --arg p_account_id "$METRUM_ACCOUNT_ID" \
    --arg p_email "$METRUM_ACCOUNT_EMAIL" \
    '{
      p_jwt_secret:$p_jwt_secret,
      p_account_id:$p_account_id,
      p_email:$p_email,
      p_expires_in_seconds:43200
    }')" > /tmp/metrum.jwt.json

export METRUM_JWT_TOKEN="$(jq -er 'if type=="string" then . else .token end' /tmp/metrum.jwt.json)"

Verify the tagged package set before launch. The tag must include worker-startup.sh, restic-restore.sh, metrumbench-llm, the insights-cli wheel, all wheel dependencies, and any runbook-specific artifacts.

: "${INSIGHTS_PACKAGE_TAG:?set the exact package tag}"
: "${RESTIC_REPOSITORY:?set Restic repository without printing it}"
: "${RESTIC_PASSWORD:?set Restic password without printing it}"

restic snapshots --tag "$INSIGHTS_PACKAGE_TAG" --json >/tmp/metrum-package-snapshots.json

curl -fsS "$PACKAGES_URL/api/scripts/worker-startup.sh" >/tmp/worker-startup.sh.check
curl -fsS "$PACKAGES_URL/api/scripts/restic-restore.sh" >/tmp/restic-restore.sh.check
curl -fsS "$PACKAGES_URL/api/packages/?tag=$INSIGHTS_PACKAGE_TAG" >/tmp/packages-tag.check.json
curl -fsS "$PACKAGES_URL/downloads/metrumbench-llm" -o /tmp/metrumbench-llm.check

Quick Selection

Target	Model	Hardware	Tooling	Provider path
H100 small LLM baseline	`Qwen/Qwen3-8B`	1x H100	metrumbench-llm + vLLM	RunPod managed execution
H100 DeepSeek distill baseline	`deepseek-ai/DeepSeek-R1-Distill-Llama-8B`	1x H100	metrumbench-llm + vLLM	Shadeform or BYOH SSH host
H200 DeepSeek Flash validation	`deepseek-ai/DeepSeek-V4-Flash`	1x H200	metrumbench-llm + vLLM	RunPod managed execution
Large DeepSeek Flash throughput	`deepseek-ai/DeepSeek-V4-Flash`	8x B200	metrumbench-llm + vLLM	Vast.ai/manual BYOH
Large DeepSeek Pro matrix	`deepseek-ai/DeepSeek-V4-Pro`	8x B300	metrumbench-llm, GenAI-Perf, KYAI	Shadeform/manual host
Large Qwen FP8 benchmark	`Qwen/Qwen3.5-122B-A10B-FP8`	1x B300	metrumbench-llm + vLLM	Shadeform managed execution
Budget LLM smoke	`deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B`	1x RTX PRO 6000	metrumbench-llm + vLLM	Shadeform or Vast.ai
Image generation benchmark	`Tongyi-MAI/Z-Image-Turbo`	1x RTX PRO 6000	metrumbench-imagegen + vLLM-Omni	Shadeform
Speech transcription benchmark	`openai/whisper-large-v3`	4x L40S or A6000 fallback	metrumbench-asr ASR	Shadeform
VLM workflow benchmark	`Qwen/Qwen3.5-2B`	1x A6000 or fallback	metrumbench-vlm VLM	Shadeform
H100 versus TPU comparison	`google/gemma-4-31B-it`	8x H100 and TPU v6e-8	metrumbench-llm and KYAI	Manual H100 and GKE TPU

Qwen 3 8B On 1x H100

Use this target for a small LLM latency and throughput baseline on H100.

Field	Value
Model	`Qwen/Qwen3-8B`
Hardware	1x H100
Framework	vLLM `0.20.1`
Benchmark	metrumbench-llm
Scenario	`c1-in128-out128` smoke, then the selected ISL/OSL matrix
Provider path	RunPod managed cloud execution

Resolve the runtime contract before launching:

curl -fsS -X POST "$METRUM_API_URL/rpc/resolve_model_server_runtime_contract" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "p_framework_code":"vllm",
    "p_framework_version":"0.20.1",
    "p_model_identifier":"Qwen/Qwen3-8B",
    "p_engine_args_set_code":"vllm-default",
    "p_bind_host":"0.0.0.0",
    "p_port":8000,
    "p_launch_adapter":"direct_process"
  }' | tee /tmp/qwen3-h100-runtime.json

The command should serve Qwen/Qwen3-8B on port 8000 with readiness at /v1/models. Then submit the managed cloud execution:

curl -fsS -X POST "$METRUM_API_URL/rpc/create_cloud_execution_request" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "p_owner_account_id":"<ACCOUNT_ID>",
    "p_org_id":null,
    "p_project_name":"Reference Benchmark Comparison",
    "p_workload_code":"qwen3-h100-nostream",
    "p_config_code":"runpod-h100",
    "p_scenario_code":"c1-in128-out128",
    "p_auto_terminate":true
  }' | tee /tmp/qwen3-h100-execution.json

Monitor the request, instance, provider pod, job, and metrumbench-llm output:

export REQUEST_ID="$(jq -er '.request_id' /tmp/qwen3-h100-execution.json)"

curl -fsS "$METRUM_API_URL/cloud_provisioning_requests?request_id=eq.$REQUEST_ID&select=request_id,status_code,error_message,updated_at" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN"

curl -fsS "$METRUM_API_URL/cloud_instances?provisioning_request_id=eq.$REQUEST_ID&select=instance_id,provider_instance_id,status_code,server_hostname" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN"

curl -fsS "$METRUM_API_URL/jobs?select=job_id,status_code,error_message,created_at,updated_at&order=created_at.desc&limit=10" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN"

Use RunPod GraphQL through curl for provider status and termination checks. Do not treat a RunPod pod as benchmark-ready until Metrum worker readiness and the runtime contract have both been verified.

DeepSeek R1 Distill Llama 8B On 1x H100

Use this target when H100 capacity is available as an SSH host and Metrum should prepare the runtime before benchmarking.

Field	Value
Model	`deepseek-ai/DeepSeek-R1-Distill-Llama-8B`
Hardware	1x H100
Framework	vLLM
Benchmark	metrumbench-llm
Provider path	Shadeform or BYOH SSH host

The HOW TO path is BYOH:

Provision or select an SSH-accessible H100 host.
Confirm target prerequisites on that host: pyenv, uv, nvm, shell bootstrap tools, NVIDIA driver, and enough disk for weights and artifacts.
Create or reuse a server_config for the exact H100 shape.
Create a concrete server_config_instance and onboarding token.
Copy a private worker env file to the host.
Restore packages from Restic, start the worker, and let Metrum start vLLM.
Dispatch metrumbench-llm only after the worker reports ready.

The BYOH control-plane shape looks like:

curl -fsS -X POST "$METRUM_API_URL/rpc/create_server_config_instance" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n \
    --arg org "$METRUM_ORG_ID" \
    --arg code "byoh-h100-deepseek-distill" \
    --arg host "$WORKER_HOSTNAME" \
    --arg ip "$WORKER_PUBLIC_IP" \
    '{
      p_org_id:$org,
      p_config_code:$code,
      p_hostname:$host,
      p_ip_address:$ip,
      p_availability_code:"pending_registration"
    }')" | tee /tmp/byoh-h100-server.json

After the worker joins, execute the smoke scenario against the registered hostname:

curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "p_owner_account_id":"<ACCOUNT_ID>",
    "p_project_name":"DeepSeek R1 Distill Llama 8B H100",
    "p_workload_code":"deepseek-r1-distill-llama8b-h100-vllm",
    "p_scenario_code":"c1-in128-out128",
    "p_server_hostname":"<WORKER_HOSTNAME>",
    "p_manage_model_server":true,
    "p_job_name":"deepseek-llama8b-h100-smoke"
  }'

Record server_hostname, job ids, v_metrumbench_llm_outputs rows, model-server logs, and provider teardown evidence.

DeepSeek V4 Flash On H200 Or 8x B200

Use H200 for single-accelerator validation and B200 when the question is large-model throughput on a multi-GPU host.

Shape	Provider path	HOW TO
1x H200	RunPod managed execution	Resolve the DeepSeek V4 Flash vLLM contract, submit `create_cloud_execution_request` for `runpod-h200`, watch worker readiness, run the smoke scenario, then auto-terminate.
8x B200	Vast.ai/manual BYOH	Provision an SSH host with `uvx --from vastai vastai`, verify 8 B200 GPUs, install/restore the runtime, run two vLLM replicas or the validated recipe, then dispatch metrumbench-llm through the registered BYOH worker.

For Vast.ai, use uvx for the provider CLI:

uvx --from vastai vastai --api-key "$VAST_AI_API_KEY" --raw search offers \
  'rentable=true verified=true num_gpus=8 gpu_ram>=180 direct_port_count>=2' \
  --limit 20 -o 'dph' > /tmp/vast-b200-offers.json

After a host is created, verify the actual GPU inventory before onboarding:

ssh -p "$WORKER_SSH_PORT" "root@$WORKER_PUBLIC_IP" \
  'nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader'

Keep the same model, precision, prompt source, and scenario matrix when comparing H200 and B200. Report output tokens/sec, TTFT, TPOT, p95, p99, error count, GPU memory, provider cost, and teardown time.

DeepSeek V4 Pro On 8x B300

Use this target for the high-end DeepSeek matrix.

Field	Value
Model	`deepseek-ai/DeepSeek-V4-Pro`
Hardware	8x B300
Tooling	metrumbench-llm, GenAI-Perf, KYAI
Provider path	Shadeform or manual BYOH host

The HOW TO path is:

Query provider capacity for 8x B300 and capture the provider offer or instance type used.
Create the project with one workload per tool: metrumbench-llm, GenAI-Perf, and KYAI.
Create the scenario matrix before provisioning so job ids are traceable.
Verify the vLLM/MoE runtime contract, including tensor parallelism, expert parallelism, FP8/KV-cache settings, and required runtime environment.
Start one reusable worker and keep it warm only for the validated matrix.
Run metrumbench-llm ISL/OSL sweeps first, then GenAI-Perf cells, then KYAI snapshots.
Export results and delete the provider host.

Use the same result query shape across tools:

curl -fsS "$METRUM_API_URL/jobs?project_id=eq.<PROJECT_ID>&select=job_id,workload_id,scenario_id,status_code,error_message,updated_at&order=created_at.desc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN"

curl -fsS "$METRUM_API_URL/v_metrumbench_llm_outputs?job_id=eq.<JOB_ID>&limit=100" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN"

Qwen 3.5 122B A10B FP8 On 1x B300

Use this target when validating a large Qwen FP8 serving recipe on B300.

Field	Value
Model	`Qwen/Qwen3.5-122B-A10B-FP8`
Hardware	1x B300
Framework	vLLM `0.20.1`
Benchmark	metrumbench-llm
Provider path	Shadeform managed execution

The resolved command must include the Qwen FP8 recipe and the B300 runtime environment:

curl -fsS -X POST "$METRUM_API_URL/rpc/resolve_model_server_runtime_contract" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "p_framework_code":"vllm",
    "p_framework_version":"0.20.1",
    "p_model_identifier":"Qwen/Qwen3.5-122B-A10B-FP8",
    "p_engine_args_set_code":"vllm-qwen35-122b-a10b-b300-fp8",
    "p_bind_host":"0.0.0.0",
    "p_port":8000,
    "p_launch_adapter":"direct_process"
  }'

Expected runtime environment values:

VLLM_ATTENTION_BACKEND=FLASHINFER
VLLM_USE_DEEP_GEMM=1
VLLM_MULTI_STREAM_GEMM_TOKEN_THRESHOLD=256
VLLM_TORCH_COMPILE_LEVEL=O2

Run the first cloud execution with auto_terminate=false if you intend to reuse the loaded model for the full matrix. After readiness, dispatch additional scenarios to the same server_hostname. For API-oriented details, see Qwen3.5 122B A10B On Shadeform B300.

RTX PRO 6000 Budget Paths

Use RTX PRO 6000 when the goal is lower-cost smoke coverage, benchmark-client comparison, or a model that does not require H100/H200/B-series memory.

Model	Tooling	Provider path	HOW TO
`deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B`	metrumbench-llm + vLLM	Shadeform or Vast.ai BYOH	Provision SSH hardware, register a BYOH server instance, resolve vLLM, run smoke and matrix scenarios.
`deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B`	metrumbench-llm + GenAI-Perf	Shadeform	Start one endpoint, run both benchmark clients with identical ISL/OSL/concurrency settings.
`Tongyi-MAI/Z-Image-Turbo`	metrumbench-imagegen + vLLM-Omni	Shadeform	Restore metrumbench-imagegen artifacts from the package tag, start vLLM-Omni, run image prompts, collect image artifact URIs.

For Vast.ai direct hardware acquisition:

uvx --from vastai vastai --api-key "$VAST_AI_API_KEY" --raw search offers \
  'rentable=true verified=true gpu_ram>=90 num_gpus=1 direct_port_count>=2' \
  --limit 20 -o 'dph' > /tmp/vast-rtxpro6000-offers.json

Use BYOH, not BYOE, when Metrum installs the runtime and starts the endpoint on that SSH host.

Speech And VLM Workloads

These examples use non-chat benchmark tools but still follow the same Metrum project, workload, scenario, worker, result, and teardown lifecycle.

Workload	Model	Hardware	HOW TO
Speech transcription	`openai/whisper-large-v3`	4x L40S or A6000 fallback	Restore metrumbench-asr artifacts, create ASR scenarios from audio inputs, run the worker, and query ASR result artifacts.
Vision-language workflow	`Qwen/Qwen3.5-2B`	1x A6000 or fallback	Restore metrumbench-vlm artifacts, create VLM scenarios with image inputs, run the worker, and query VLM outputs.

Keep runbook-specific binaries and assets in the tagged package set before launch. ASR, VLM, and image paths typically need artifacts beyond metrumbench-llm.

BYOH Versus BYOE

Use BYOH when you have hardware access over SSH and want Metrum to prepare the runtime endpoint before benchmarking. Use BYOE only when the GenAI API endpoint already exists outside Metrum and Metrum should benchmark that endpoint as-is.

Path	You provide	Metrum does
BYOH	SSH hardware, GPU shape, provider lifecycle access	Installs prerequisites, restores packages, starts model server, checks readiness, runs benchmarks
BYOE	Existing OpenAI-compatible endpoint	Creates endpoint records, runs benchmarks/evaluations, collects results

For the full BYOH procedure, see Bring Your Own Hardware. For endpoint-only testing, see Bring Your Own Endpoint.

Private Stack Exposure​

Common Operator Path​

Quick Selection​

Qwen 3 8B On 1x H100​

DeepSeek R1 Distill Llama 8B On 1x H100​

DeepSeek V4 Flash On H200 Or 8x B200​

DeepSeek V4 Pro On 8x B300​

Qwen 3.5 122B A10B FP8 On 1x B300​

RTX PRO 6000 Budget Paths​

Speech And VLM Workloads​

BYOH Versus BYOE​