Cloud Benchmark Execution

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Use this flow for paid cloud validation runs. Confirm provider, hardware, model, expected cost/risk, and teardown expectations before launch.

Choose the model+hardware runbook first. Provider-specific launch/status commands live as subsections inside that target runbook so the benchmark contract stays stable when capacity moves between providers.

Set these environment variables for the examples:

export METRUM_API_URL="https://api.<control-plane-domain>"
export METRUM_WEBUI_URL="https://<control-plane-host-or-dns>"
export METRUM_JWT_TOKEN="<12-hour-db-minted-postgrest-jwt>"
export METRUM_ACCOUNT_ID="<account-uuid>"
export METRUM_ORG_ID="<org-uuid>"

Mint benchmark-run tokens through POST /rpc/mint_postgrest_write_jwt with p_expires_in_seconds: 43200. Do not use CLI JWT minting in runbooks.

Managed Shadeform Onboarding

For disposable Shadeform workers, use the WebUI-managed onboarding endpoint instead of hand-assembling provider startup scripts.

curl -fsS -X POST "$METRUM_WEBUI_URL/api/cloud/onboard" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"org_id\":\"$METRUM_ORG_ID\",
    \"hostname\":\"shadeform-b200-1x-001\",
    \"cloud_provider\":\"shadeform\",
    \"provider_params\":{
      \"shade_instance_type\":\"RTX PRO 6000\",
      \"gpu_type\":\"RTX PRO 6000\",
      \"gpu_count\":1,
      \"num_gpus\":1,
      \"cloud\":\"example-cloud\",
      \"region\":\"example-region\",
      \"os\":\"ubuntu22.04_cuda12.6_shade_os\",
      \"disk_gb\":250,
      \"hourly_cost_usd\":0
    }
  }" | jq

Important response fields:

Field	Description
server_id	Metrum Insights server instance id.
provider_instance_id	Real provider instance id for teardown and reconciliation.
hostname	Hostname to use for readiness and benchmark execution.
config_code	Generated or selected server config code.
onboarding_token	Worker bootstrap token. Do not log or share it.

Execute Scenario

Starts a benchmark job for a created scenario on a ready server.

curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"deepseek-qwen15b-rtxpro6000-smoke\",
    \"p_workload_code\":\"deepseek-qwen15b-vllm-latest\",
    \"p_scenario_code\":\"isl128-osl1024-c256\",
    \"p_server_hostname\":\"shadeform-b200-1x-001\"
  }" | jq

Parameters:

Name	Required	Description
p_owner_account_id	Yes	Account UUID that owns the project.
p_project_name	Yes	Project containing the workload and scenario.
p_workload_code	Yes	Workload code.
p_scenario_code	Yes	Scenario code.
p_server_hostname	Yes	Ready server hostname.
p_scheduled_for	No	Optional scheduled time.

Get Job Status

Checks benchmark job status.

curl -fsS -X POST "$METRUM_API_URL/rpc/get_job_status" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"p_job_id":123}' | jq

Parameters:

Name	Required	Description
p_job_id	Yes	Benchmark job id returned by execution.

List Jobs

Lists jobs for inspection when the runbook needs to recover ids.

curl -fsS -X POST "$METRUM_API_URL/rpc/list_jobs" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"deepseek-qwen15b-rtxpro6000-smoke\",
    \"p_tool_code\":\"metrumbench-llm\",
    \"p_dummy\":1
  }" | jq

Parameters:

Name	Required	Description
p_account_id	Yes	Account UUID for access filtering.
p_project_name	No	Project name filter.
p_status_code	No	Job status filter.
p_tool_code	No	Tool code filter.
p_dummy	Yes	Compatibility parameter required by this RPC. Use 1.

If a job remains pending and the worker heartbeat stops updating, tear down the provider instance and report a stale-heartbeat or pending-job failure instead of waiting indefinitely.