metrumbench-llm Workload And Scenario

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Use this flow to create the model/framework workload and the input/output token scenario matrix before benchmark execution.

Set these environment variables for the examples:

export METRUM_API_URL="https://api.<control-plane-domain>"
export METRUM_JWT_TOKEN="<12-hour-db-minted-postgrest-jwt>"
export METRUM_ACCOUNT_ID="<account-uuid>"

List Model Catalog

Find model catalog entries.

curl -fsS "$METRUM_API_URL/ai_models?model_code=ilike.deepseek%25&select=model_id,model_code,model_name,parameter_count_display,is_chat_model,family_id&limit=20" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Query parameters:

Name	Required	Description
model_code	No	PostgREST filter. Example: ilike.deepseek%.
select	No	Projection for catalog fields.
limit	No	Maximum model rows to return.

List Framework Catalog

Find framework versions and framework codes.

curl -fsS "$METRUM_API_URL/frameworks?framework_code=eq.vllm&select=framework_id,framework_code,framework_name&limit=20" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/framework_versions?select=version_id,framework_id,version&limit=60" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Query parameters:

Name	Required	Description
framework_code	No	PostgREST filter. Example: eq.vllm.
select	No	Projection for framework or version fields.
limit	No	Maximum rows to return.

Create Workload

Creates a benchmark workload for a project, tool, model, framework, version, and engine args set.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"deepseek-qwen15b-rtxpro6000-smoke\",
    \"p_workload_code\":\"deepseek-qwen15b-vllm-latest\",
    \"p_workload_name\":\"deepseek-qwen15b-vllm-latest\",
    \"p_tool_code\":\"metrumbench-llm\",
    \"p_model_code\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
    \"p_framework_code\":\"vllm\",
    \"p_version\":\"0.20.0\",
    \"p_engine_args_set_code\":\"vllm-default\"
  }" | jq

Parameters:

Name	Required	Description
p_owner_account_id	Yes	Account UUID that owns the project.
p_project_name	Yes	Existing project name.
p_workload_code	Yes	Stable workload code. Use lowercase words separated by hyphens.
p_workload_name	Yes	Display name. Often the same as p_workload_code.
p_tool_code	Yes	Benchmark tool code, usually metrumbench-llm.
p_model_code	Yes	Model catalog code.
p_framework_code	Yes	Serving framework code, for example vllm.
p_version	Yes	Framework version.
p_quantization_code	No	Quantization code, when applicable.
p_engine_args_set_code	No	Engine argument preset, for example vllm-default.

Create Scenario

Creates one benchmark scenario for a workload.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"deepseek-qwen15b-rtxpro6000-smoke\",
    \"p_workload_code\":\"deepseek-qwen15b-vllm-latest\",
    \"p_scenario_code\":\"isl128-osl1024-c256\",
    \"p_scenario_name\":\"isl128-osl1024-c256\",
    \"p_concurrency\":256,
    \"p_input_sequence_length\":128,
    \"p_output_sequence_length\":1024,
    \"p_num_requests\":1024
  }" | jq

Parameters:

Name	Required	Description
p_owner_account_id	Yes	Account UUID that owns the project.
p_project_name	Yes	Existing project name.
p_workload_code	Yes	Workload code created earlier.
p_scenario_code	Yes	Stable scenario code. Include ISL, OSL, and concurrency.
p_scenario_name	Yes	Display name. Often the same as p_scenario_code.
p_concurrency	No	Concurrent requests.
p_input_sequence_length	No	Requested prompt/input token count.
p_output_sequence_length	No	Requested output token count.
p_num_images_batch	No	Images per request for multimodal scenarios.
p_num_requests	No	Total requests for the scenario.

For a small smoke run before scaling, use p_input_sequence_length=32, p_output_sequence_length=16, p_concurrency=1, and p_num_requests=2.

List Workloads And Scenarios

Inspect created objects.

curl -fsS -X POST "$METRUM_API_URL/rpc/list_workloads" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"p_project_id":1}' | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/list_scenarios" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"p_workload_id":1}' | jq

Parameters:

RPC	Parameter	Required	Description
list_workloads	p_project_id	Yes	Project numeric id.
list_scenarios	p_workload_id	Yes	Workload numeric id.