Developer Guide

This guide shows how to build an integration that creates and runs a complete benchmark project through the Metrum Insights HTTP APIs. Use these examples as the starting point for scripts, CI jobs, internal portals, and agent workflows.

For the full benchmark operator walkthrough, see End To End Benchmark API Example.

0. Environment

Use the Web UI origin as the browser control-plane URL. Use the Caddy API endpoint for automation; in hosted deployments this is the api. hostname that reverse-proxies to PostgREST.

export METRUM_WEBUI_URL="https://<control-plane-host-or-dns>"
export METRUM_API_URL="https://api.<control-plane-domain>"
export METRUM_JWT_TOKEN="<12-hour-db-minted-postgrest-jwt>"
export METRUM_ACCOUNT_ID="<account-uuid>"
export METRUM_ORG_ID="<org-uuid>"

export PROJECT_NAME="api-dev-smoke-qwen15b"
export WORKLOAD_CODE="qwen15b-vllm-bf16"
export SCENARIO_CODE="smoke-isl32-osl16-c1"
export SERVER_HOSTNAME="gpu-worker-001"

Mint METRUM_JWT_TOKEN with POST /rpc/mint_postgrest_write_jwt and p_expires_in_seconds: 43200 for benchmark automation. Do not use CLI JWT minting in runbooks or integrations.

Every request below uses:

-H "Authorization: Bearer $METRUM_JWT_TOKEN"
-H "Content-Type: application/json"

Never bake bearer tokens or provider keys into source code. Inject them through your deployment secret manager.

1. Verify Caller Context

Start every integration by checking identity and organization membership.

curl -fsS -X POST "$METRUM_API_URL/rpc/get_current_user_context" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\"}" | jq

Expected use:

• confirm the account is valid;
• confirm the user or service account can access the target org;
• store org/account IDs in your job log.

2. Discover Catalog Codes

Do not hardcode model and framework codes until your integration has verified they exist in the target environment.

curl -fsS "$METRUM_API_URL/ai_models?model_code=ilike.qwen%25&select=model_id,model_code,model_name&limit=20" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/frameworks?framework_code=eq.vllm&select=framework_id,framework_code,framework_name" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/framework_versions?select=version_id,framework_id,version&limit=100" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Pick stable catalog values for the rest of the flow:

export MODEL_CODE="deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"
export FRAMEWORK_CODE="vllm"
export FRAMEWORK_VERSION="0.20.0"
export ENGINE_ARGS_SET_CODE="vllm-default"

3. Create A Project

Use one project per reportable benchmark campaign.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_project" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_visibility_code\":\"private\",
    \"p_created_by_account_id\":\"$METRUM_ACCOUNT_ID\"
  }" | tee /tmp/metrum-project.json | jq

List projects when recovering from an interrupted script:

curl -fsS -X POST "$METRUM_API_URL/rpc/list_projects" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_include_shared\":true}" | jq

4. Create A Workload

A workload is the model, benchmark tool, framework, version, quantization, and engine-args recipe.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"$WORKLOAD_CODE\",
    \"p_workload_name\":\"DeepSeek Qwen 1.5B vLLM BF16\",
    \"p_tool_code\":\"metrumbench-llm\",
    \"p_model_code\":\"$MODEL_CODE\",
    \"p_framework_code\":\"$FRAMEWORK_CODE\",
    \"p_version\":\"$FRAMEWORK_VERSION\",
    \"p_quantization_code\":\"bf16\",
    \"p_engine_args_set_code\":\"$ENGINE_ARGS_SET_CODE\"
  }" | tee /tmp/metrum-workload.json | jq

Create separate workloads for changes that affect reproducibility:

Change	New workload?
model	Yes
framework or version	Yes
quantization or dtype	Yes
engine args set, such as KV FP8 or CPU offload	Yes
only ISL/OSL/concurrency	No, create scenarios instead

5. Create Scenarios

Create one scenario per traffic point. Start with a smoke scenario before a full sweep.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"$WORKLOAD_CODE\",
    \"p_scenario_code\":\"$SCENARIO_CODE\",
    \"p_scenario_name\":\"$SCENARIO_CODE\",
    \"p_concurrency\":1,
    \"p_input_sequence_length\":32,
    \"p_output_sequence_length\":16,
    \"p_num_requests\":2,
    \"p_job_params\":{
      \"dataset_name\":\"random\",
      \"server_side_download\":true,
      \"streaming\":true
    }
  }" | tee /tmp/metrum-scenario.json | jq

Example matrix loop:

for pair in 128:128 128:1024 1024:1024 1024:128; do
  isl="${pair%%:*}"
  osl="${pair##*:}"

  for concurrency in 32 256 512 1024; do
    code="isl${isl}-osl${osl}-c${concurrency}"

    curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
      -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
        \"p_project_name\":\"$PROJECT_NAME\",
        \"p_workload_code\":\"$WORKLOAD_CODE\",
        \"p_scenario_code\":\"$code\",
        \"p_scenario_name\":\"$code\",
        \"p_concurrency\":$concurrency,
        \"p_input_sequence_length\":$isl,
        \"p_output_sequence_length\":$osl,
        \"p_num_requests\":1024
      }" | jq
  done
done

6. Check Server Readiness

Your integration should not execute jobs until the selected server is ready.

curl -fsS -X POST "$METRUM_API_URL/rpc/validate_server_readiness" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_server_hostname\":\"$SERVER_HOSTNAME\"}" | jq

Bound the wait:

for attempt in $(seq 1 20); do
  curl -fsS -X POST "$METRUM_API_URL/rpc/validate_server_readiness" \
    -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"p_server_hostname\":\"$SERVER_HOSTNAME\"}" | tee /tmp/readiness.json | jq

  jq -e '.ready == true' /tmp/readiness.json >/dev/null && break
  sleep 15
done

7. Execute A Scenario

Execute the smoke scenario first.

curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"$WORKLOAD_CODE\",
    \"p_scenario_code\":\"$SCENARIO_CODE\",
    \"p_server_hostname\":\"$SERVER_HOSTNAME\",
    \"p_manage_model_server\":true,
    \"p_job_name\":\"$SCENARIO_CODE-on-$SERVER_HOSTNAME\"
  }" | tee /tmp/metrum-execute.json | jq

Capture the returned job ID:

export JOB_ID="$(jq -r '.job_id // .benchmark_job_id // empty' /tmp/metrum-execute.json)"
test -n "$JOB_ID"

If your deployment returns a nested payload, inspect the response once and map the correct job field in your integration.

8. Poll Job Status

Poll by job ID and stop on terminal states.

for attempt in $(seq 1 120); do
  curl -fsS -X POST "$METRUM_API_URL/rpc/get_job_status" \
    -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"p_job_id\":$JOB_ID}" | tee /tmp/job-status.json | jq

  jq -e '.status_code | IN("completed","failed","cancelled")' /tmp/job-status.json >/dev/null && break
  sleep 30
done

Recover job IDs by project when needed:

curl -fsS -X POST "$METRUM_API_URL/rpc/list_jobs" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_tool_code\":\"metrumbench-llm\",
    \"p_limit\":100,
    \"p_offset\":0,
    \"p_dummy\":1
  }" | jq

9. Read Results

Use result views for integration output instead of scraping logs.

curl -fsS "$METRUM_API_URL/v_benchmark_results?org_id=eq.$METRUM_ORG_ID&project_name=eq.$PROJECT_NAME&order=scenario_code.asc&limit=200" \
  -H "X-Org-Id: $METRUM_ORG_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/v_metrumbench_llm_outputs?job_id=eq.$JOB_ID&limit=100" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Telemetry:

curl -fsS -X POST "$METRUM_API_URL/rpc/get_telemetry_summary_for_job" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_job_id\":$JOB_ID}" | jq

Downloads:

curl -fsS -OJ "$METRUM_WEBUI_URL/api/jobs/$JOB_ID/sut.csv" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN"

curl -fsS -OJ "$METRUM_WEBUI_URL/api/run-report/<run-id>" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN"

10. Integration Checklist

Before shipping a workflow integration:

• verify identity with get_current_user_context;
• verify catalog codes before creating workloads;
• create one workload per reproducibility recipe;
• create one scenario per traffic point;
• bound every readiness and job-status poll;
• persist project, workload, scenario, server, run, and job IDs;
• query results from views and telemetry RPCs;
• download SUT CSV for every reported benchmark;
• fail fast when commands contain unresolved placeholders;
• never log secrets or bootstrap tokens.