Model And Hardware Runbooks

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Metrum runbooks are organized by the benchmark target first: model, tool, scenario shape, and accelerator. Provider mechanics are subsections inside that target runbook. This keeps the benchmark contract stable when capacity moves between Shadeform, RunPod, Vast.ai, or a manual host.

Provider availability changes frequently. Do not swap hardware or model during a paid validation run unless the operator explicitly confirms the new target after seeing the availability result.

Operator Rules

• Mint PostgREST tokens with POST /rpc/mint_postgrest_write_jwt and p_expires_in_seconds: 43200; do not use CLI JWT minting.
• Use curl for Metrum/PostgREST and provider HTTP operations.
• Use uv or uvx for project-local Python and provider CLIs such as Vast.ai; do not install into system Python.
• Resolve Restic repository/auth before paid launch. A package URL is not a fallback for missing Restic credentials.
• Require an explicit INSIGHTS_PACKAGE_TAG and verify the complete tagged worker package set before launch: worker-startup.sh, restic-restore.sh, metrumbench-llm, the insights-cli wheel, all wheel dependencies, and runbook-specific artifacts.
• Verify the resolved runtime contract, image, command, and readiness endpoint before dispatching benchmark scenarios.
• Record provider instance ids, execution/request ids, job ids, result locations, useful redacted log excerpts, and teardown status.

Maintained Targets

Model / workload	Hardware target	Provider sections
Qwen/Qwen3-8B metrumbench-llm	1x H100	RunPod
deepseek-ai/DeepSeek-R1-Distill-Llama-8B metrumbench-llm	1x H100	Shadeform
deepseek-ai/DeepSeek-V4-Flash metrumbench-llm	1x H200	RunPod
deepseek-ai/DeepSeek-V4-Flash metrumbench-llm	8x B200	Vast.ai / manual host
deepseek-ai/DeepSeek-V4-Pro metrumbench-llm / GenAI-Perf / KYAI	8x B300	Shadeform
Qwen/Qwen3.5-122B-A10B-FP8 metrumbench-llm	1x B300	Shadeform
deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B metrumbench-llm	1x RTX PRO 6000	Shadeform, Vast.ai
Tongyi-MAI/Z-Image-Turbo metrumbench-imagegen	1x RTX PRO 6000	Shadeform
openai/whisper-large-v3 metrumbench-asr ASR	4x L40S or A6000 fallback	Shadeform
Qwen/Qwen3.5-2B metrumbench-vlm VLM	1x A6000 or fallback	Shadeform

Use the repository runbook catalog for exact file paths and validation evidence: docs/runbooks/README.md.

Provider Sections

A provider section must include the exact capacity query, create/status calls, runtime launch inputs, readiness checks, result checks, and teardown commands that were validated for that target. If a provider path only validates a direct model endpoint and not the full Metrum worker lifecycle, label it that way.

BYOH Provider Sections

Use BYOH when provider capacity is made available as SSH hardware but Metrum Insights should still perform worker setup, model-server startup, readiness, benchmark execution, result ingestion, and managed runtime teardown. A BYOH section belongs under the model+hardware target, alongside managed provider sections such as Shadeform or RunPod.

BYOH sections must document:

• the provider capacity query and create/status/teardown commands;
• the public PostgREST and WebUI/package URL requirement;
• target-side pyenv, uv, nvm, and shell bootstrap prerequisites;
• Restic credential resolution and explicit package tag verification;
• server config, GPU metadata, concrete server instance, and onboarding token;
• worker bootstrap method matched to the artifact layout;
• resolved model-server runtime contract before readiness and benchmark;
• job ids, result views, log locations, and teardown status.

Do not call a BYOH run BYOE. BYOE starts from an already-running GenAI API endpoint. BYOH starts from hardware access and Metrum prepares the benchmark endpoint. See Bring Your Own Hardware.

For Vast.ai direct endpoint validation, use uvx --from vastai vastai and a non-secret metrumbench-llm placeholder such as --api-key local-vllm when the local vLLM endpoint does not enforce auth.

For RunPod, direct provider status and termination use RunPod GraphQL through curl. The benchmark launch itself should come from the model/hardware runbook and, for managed runs, create_cloud_execution_request.