Automate Custom Recipes With Command Templates

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Admins add new runtime recipes by registering framework versions, command templates, and engine args sets. Operators then select those recipes as workloads.

Template Checklist

• Name the framework, version, image, and supported hardware.
• Define required placeholders such as model id, port, tensor parallel size, and cache settings.
• Provide defaults for optional arguments.
• Include a health check or readiness condition.
• State whether the template supports managed lifecycle or BYOE only.
• Validate that rendered commands have no unresolved placeholders.
• Test with a smoke scenario before publishing.

Recipe Families

Recipe	Example Arguments
Default vLLM	--model, --served-model-name, --port, --tensor-parallel-size
vLLM FP8 KV cache	--kv-cache-dtype fp8, cache scaling flags, memory utilization
Long context	--max-model-len 32768, adjusted batched-token limits
SGLang	scheduler and memory fraction arguments
TensorRT-LLM	engine path, runtime options, batch limits
Vendor accelerator	vendor runtime launcher, device selection, telemetry adapter

Operator Impact

Changing a command template changes the runtime recipe. Create a new engine args set or versioned template when results need to remain comparable over time.

API

List framework versions and existing engine args sets before adding a custom recipe:

curl -fsS -X POST "$METRUM_API_URL/rpc/list_framework_versions_catalog" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_framework_code\":\"vllm\"}" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/list_engine_args_for_framework_version" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_framework_code\":\"vllm\",
    \"p_version\":\"0.20.0\",
    \"p_org_id\":\"$METRUM_ORG_ID\"
  }" | jq

Create a team-scoped engine args set. Unknown argument keys are ignored, so compare the response from list_engine_args_for_framework_version before and after saving.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_custom_engine_args_set" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_framework_code\":\"vllm\",
    \"p_version\":\"0.20.0\",
    \"p_set_name\":\"vLLM FP8 KV cache\",
    \"p_set_code\":\"vllm-kv-cache-fp8\",
    \"p_arg_keys\":[
      \"kv_cache_dtype\",
      \"gpu_memory_utilization\",
      \"max_model_len\",
      \"enable_prefix_caching\"
    ],
    \"p_arg_values\":[\"fp8\", \"0.90\", \"4096\", \"true\"]
  }" | jq

Use the resulting p_engine_args_set_code when creating workloads. Keep the old set in place for historical comparisons instead of mutating it after a report is published.