Bring Your Own Hardware

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

BYOH means you provide an SSH-accessible machine and Metrum Insights does the benchmark setup on that machine: worker install, agent join, model-server startup, readiness checks, benchmark execution, result ingestion, and teardown of Metrum-managed runtime processes.

Use BYOH when the hardware is available outside a Metrum-managed cloud provisioner but the benchmark should still use the normal Metrum lifecycle. Examples include a Vast.ai SSH container, a manually-created Shadeform VM, an on-prem lab server, or a customer-owned accelerator host.

BYOH is not BYOE. BYOE starts from an already-running GenAI API endpoint and Metrum does not own model-server startup. BYOH starts from hardware access and Metrum prepares the endpoint before measuring it.

Operator Contract

Before a paid BYOH run, confirm:

• provider or hardware owner;
• exact accelerator model and count;
• model and framework version;
• expected hourly cost or fixed reservation cost;
• teardown expectation after success or unrecoverable failure.

Use external Cloudflared or DNS URLs for bootstrap. Remote hosts must not be given localhost PostgREST or WebUI URLs.

Use 12-hour PostgREST JWTs minted through:

curl -fsS -X POST "$METRUM_API_URL/rpc/mint_postgrest_write_jwt" \
  -H "Content-Type: application/json" \
  -d "$(jq -n \
    --arg p_jwt_secret "$JWT_SECRET" \
    --arg p_account_id "$METRUM_ACCOUNT_ID" \
    --arg p_email "$METRUM_ACCOUNT_EMAIL" \
    '{
      p_jwt_secret:$p_jwt_secret,
      p_account_id:$p_account_id,
      p_email:$p_email,
      p_expires_in_seconds:43200
    }')" | jq -er 'if type=="string" then . else .token end'

Never print JWTs, provider keys, Restic credentials, encrypted credentials, or Hugging Face tokens. Store temporary env files with mode 600.

Prerequisites

The target host must have:

• SSH access for the operator;
• NVIDIA driver and visible GPU devices for GPU benchmarks;
• shell bootstrap tools: bash, curl, git, ca-certificates, and common build packages;
• pyenv, uv, and nvm;
• enough local disk for model weights, benchmark artifacts, and package restore;
• network egress to the public PostgREST URL, WebUI/package URL, model registry, and Restic repository.

Use uv or uvx for Python/project commands. Do not install custom Python packages into system Python.

Restic repository/auth must be present before a paid launch. A package URL is not a fallback for missing Restic credentials. For golden-image BYOH bootstraps, create or select an audited golden-image Restic snapshot and pass its exact snapshot id to the worker bootstrap.

Control-Plane Setup

Create or reuse a server config that describes the hardware being benchmarked, then create a concrete server instance for the SSH host.

export CONFIG_CODE="byoh-rtxpro6000-vast"
export WORKER_HOSTNAME="vast-rtxpro6000-$VAST_INSTANCE_ID"

curl -fsS -X POST "$METRUM_API_URL/rpc/create_server_config" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n \
    --arg org "$METRUM_ORG_ID" \
    --arg code "$CONFIG_CODE" \
    '{
      p_org_id:$org,
      p_config_name:"BYOH Vast RTX PRO 6000",
      p_config_code:$code,
      p_config_type_code:"on-prem"
    }')" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/create_server_config_gpu" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n \
    --arg org "$METRUM_ORG_ID" \
    --arg code "$CONFIG_CODE" \
    '{
      p_org_id:$org,
      p_config_code:$code,
      p_vendor_code:"NV",
      p_model_code:"rtx-pro-6000-blackwell-server-edition",
      p_gpu_count:1
    }')" | jq

SERVER_JSON="$(curl -fsS -X POST "$METRUM_API_URL/rpc/create_server_config_instance" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n \
    --arg org "$METRUM_ORG_ID" \
    --arg code "$CONFIG_CODE" \
    --arg host "$WORKER_HOSTNAME" \
    --arg ip "$WORKER_PUBLIC_IP" \
    '{
      p_org_id:$org,
      p_config_code:$code,
      p_hostname:$host,
      p_ip_address:$ip,
      p_availability_code:"pending_registration"
    }')")"

SERVER_ID="$(printf '%s' "$SERVER_JSON" | jq -er '.server_id')"
ONBOARDING_TOKEN="$(printf '%s' "$SERVER_JSON" | jq -er '.onboarding_token')"

The project itself must be account-owned or org-scoped. Do not describe an account-owned project as ownerless simply because org_id is null.

Vast.ai Example

Vast.ai is useful when managed-provider capacity is scarce. Use uvx for the provider CLI:

uvx --from vastai vastai --api-key "$VAST_AI_API_KEY" --raw show instances-v1 \
  | jq '{total_instances, instances:[.instances[]? | {id,label,actual_status,gpu_name,num_gpus,dph_total}]}'

uvx --from vastai vastai --api-key "$VAST_AI_API_KEY" --raw search offers \
  'rentable=true verified=true gpu_ram>=90 num_gpus=1 direct_port_count>=2' \
  --limit 20 -o 'dph' \
  > /tmp/vast-rtxpro6000-offers.json

jq '[.[] | select((.gpu_name // "") | test("RTX PRO 6000"; "i")) |
  {id,gpu_name,num_gpus,gpu_ram,dph_total,reliability,geolocation,direct_port_count,disk_space}]' \
  /tmp/vast-rtxpro6000-offers.json

Create a host that stays idle for BYOH bootstrap. Use a versioned provider container image when the runtime is meant to come from the provider image.

export VAST_OFFER_ID="<confirmed-offer-id>"
export VAST_LABEL="metrum-byoh-rtxpro6000-qwen15b-$(date -u +%Y%m%d%H%M%S)"

uvx --from vastai vastai --api-key "$VAST_AI_API_KEY" --raw create instance "$VAST_OFFER_ID" \
  --image vllm/vllm-openai:v0.20.0 \
  --disk 120 \
  --ssh --direct \
  --env '-p 8000:8000' \
  --label "$VAST_LABEL" \
  --onstart-cmd 'mkdir -p /workspace/logs; sleep infinity' \
  --cancel-unavail \
  > /tmp/vast-byoh-create.json

export VAST_INSTANCE_ID="$(jq -er '.new_contract // .id' /tmp/vast-byoh-create.json)"

Poll until actual_status is running, then verify SSH and GPU identity:

uvx --from vastai vastai --api-key "$VAST_AI_API_KEY" --raw show instance "$VAST_INSTANCE_ID" \
  > /tmp/vast-byoh-instance.json

export WORKER_PUBLIC_IP="$(jq -er '.public_ipaddr' /tmp/vast-byoh-instance.json)"
export WORKER_SSH_PORT="$(jq -er '.ports["22/tcp"][0].HostPort' /tmp/vast-byoh-instance.json)"

ssh -o StrictHostKeyChecking=accept-new -p "$WORKER_SSH_PORT" "root@$WORKER_PUBLIC_IP" \
  'nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader'

Verify the provider image runtime before bootstrap:

ssh -o StrictHostKeyChecking=accept-new -p "$WORKER_SSH_PORT" "root@$WORKER_PUBLIC_IP" \
  'command -v vllm && python3 - <<'"'"'PY'"'"'
import vllm
print(vllm.__version__)
PY'

Bootstrap The Worker

Write a private env file locally, copy it to the host, and never print it:

cat > /tmp/metrum-byoh-worker.env <<EOF
export METRUM_API_URL='$METRUM_API_URL'
export METRUM_WEBUI_URL='$METRUM_WEBUI_URL'
export METRUM_JWT_TOKEN='$METRUM_JWT_TOKEN'
export METRUM_HOSTNAME='$WORKER_HOSTNAME'
export METRUM_CONFIG_CODE='$CONFIG_CODE'
export METRUM_ONBOARDING_TOKEN='$ONBOARDING_TOKEN'
export METRUM_WORKER_MODE='benchmark'
export METRUM_BOOTSTRAP_MODE='full'
export RESTIC_REPOSITORY='$RESTIC_REPOSITORY'
export RESTIC_PASSWORD='$RESTIC_PASSWORD'
export RESTIC_RESTORE_SNAPSHOT_ID='$RESTIC_RESTORE_SNAPSHOT_ID'
export INSIGHTS_LAUNCH_ADAPTER='provider_container'
export INSIGHTS_FRAMEWORK_CODE='vllm'
export INSIGHTS_FRAMEWORK_VERSION='0.20.0'
export INSIGHTS_ENGINE_ARGS_SET_CODE='vllm-default'
export INSIGHTS_MODEL_ID='deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B'
export METRUM_SERVER_CONTAINER_IMAGE='vllm/vllm-openai:v0.20.0'
export METRUM_SERVER_HOST='0.0.0.0'
export METRUM_SERVER_PORT='8000'
export METRUM_SERVER_READY_PATH='/v1/models'
export METRUM_SERVER_COMMAND='vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --port 8000 --gpu-memory-utilization 0.90 --tensor-parallel-size 1 --dtype auto --max-model-len 4096 --kv-cache-dtype auto --cpu-offload-gb 0 --host 0.0.0.0'
EOF
chmod 600 /tmp/metrum-byoh-worker.env

scp -P "$WORKER_SSH_PORT" /tmp/metrum-byoh-worker.env \
  "root@$WORKER_PUBLIC_IP:/root/metrum-byoh-worker.env"

Install target-side shell prerequisites if the image does not already include them:

ssh -p "$WORKER_SSH_PORT" "root@$WORKER_PUBLIC_IP" 'bash -lc "
  set -euo pipefail
  apt-get update -qq
  apt-get install -y -qq curl git ca-certificates build-essential
  test -d ~/.pyenv || git clone --depth 1 https://github.com/pyenv/pyenv.git ~/.pyenv
  export PYENV_ROOT=~/.pyenv
  export PATH=\"\$PYENV_ROOT/bin:\$HOME/.local/bin:\$PATH\"
  pyenv --version
  command -v uv >/dev/null || curl -LsSf https://astral.sh/uv/install.sh | sh
  uv --version
  test -d ~/.nvm || PROFILE=~/.bashrc bash -c \"curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash\"
  . ~/.nvm/nvm.sh
  nvm --version
"'

Create and audit the golden worker snapshot before a paid run:

export RESTIC_SNAPSHOT_CHANNEL=golden-image
export RESTIC_TAG_BRANCH="$(git rev-parse --abbrev-ref HEAD)"
export RESTIC_TAG_RUN_ID="byoh-$(date -u +%Y%m%d%H%M%S)"
./scripts/cloud/create-golden-snapshot

export RESTIC_RESTORE_TAGS="golden-image,branch-${RESTIC_TAG_BRANCH}"
./scripts/cloud/audit-golden-snapshot

Set RESTIC_RESTORE_SNAPSHOT_ID to the audited golden-image snapshot id. Then run the public worker-startup.sh from the WebUI/package URL:

ssh -p "$WORKER_SSH_PORT" "root@$WORKER_PUBLIC_IP" 'bash -lc "
  set -euo pipefail
  source /root/metrum-byoh-worker.env
  curl -fsSL \"\$METRUM_WEBUI_URL/api/scripts/worker-startup.sh\" -o /root/worker-startup.sh
  chmod +x /root/worker-startup.sh
  nohup /root/worker-startup.sh > /workspace/logs/worker-bootstrap.log 2>&1 &
"'

Execute The Benchmark

Create project, workload, and scenario with the normal APIs. For provider containers that serve the Hugging Face id as the OpenAI model name, make the Metrum model code match that served model name or configure vLLM with --served-model-name to match the Metrum model code. Then dispatch against the already-ready BYOH endpoint:

EXECUTION_JSON="$(curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n \
    --arg account "$METRUM_ACCOUNT_ID" \
    --arg project "$PROJECT_NAME" \
    --arg workload "$WORKLOAD_CODE" \
    --arg scenario "$SCENARIO_CODE" \
    --arg host "$WORKER_HOSTNAME" \
    '{
      p_owner_account_id:$account,
      p_project_name:$project,
      p_workload_code:$workload,
      p_scenario_code:$scenario,
      p_server_hostname:$host,
      p_job_name:($scenario + "-on-" + $host),
      p_manage_model_server:false
    }')")"

export JOB_ID="$(printf '%s' "$EXECUTION_JSON" | jq -er '.job_id')"
export RUN_ID="$(printf '%s' "$EXECUTION_JSON" | jq -er '.run_id')"

Before the worker claims the job, verify the resolved setup contract for the run. The expected RTX PRO 6000 Qwen 1.5B command shape is:

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 1 \
  --dtype auto \
  --max-model-len 4096 \
  --kv-cache-dtype auto \
  --cpu-offload-gb 0 \
  --host 0.0.0.0

Poll job status and results:

curl -fsS -X POST "$METRUM_API_URL/rpc/get_job_status" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --argjson job "$JOB_ID" '{p_job_id:$job}')" | jq

curl -fsS -G "$METRUM_API_URL/v_metrumbench_llm_outputs" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  --data-urlencode "job_id=eq.$JOB_ID" | jq

Monitor the remote logs:

ssh -p "$WORKER_SSH_PORT" "root@$WORKER_PUBLIC_IP" \
  'tail -f /workspace/agent.log ~/.metrum/agent/logs/managed-model-servers/*.log 2>/dev/null'

Teardown

Teardown is mandatory for paid BYOH providers:

printf 'y\n' | uvx --from vastai vastai --api-key "$VAST_AI_API_KEY" \
  destroy instance "$VAST_INSTANCE_ID"

uvx --from vastai vastai --api-key "$VAST_AI_API_KEY" --raw show instances-v1 \
  | jq '{total_instances, instances:[.instances[]? | {id,label,actual_status,gpu_name,num_gpus,dph_total}]}'

For on-prem hardware, stop the agent, stop managed model-server processes, and record the final server status and result locations in the benchmark report.