metrumbench-vlm Workload And Scenario

Use this flow to create a VLM workload and the image-plus-text scenario matrix before benchmark execution.

Set these environment variables for the examples:

export METRUM_API_URL="https://api.<control-plane-domain>"
export METRUM_JWT_TOKEN="<12-hour-db-minted-postgrest-jwt>"
export METRUM_ACCOUNT_ID="<account-uuid>"

Create Workload

Creates a benchmark workload for a project, tool, model, framework, version, and engine args set.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"qwen-vl-h100\",
    \"p_workload_code\":\"qwen2-vl-7b-vllm\",
    \"p_workload_name\":\"qwen2-vl-7b-vllm\",
    \"p_tool_code\":\"metrumbench-vlm\",
    \"p_model_code\":\"Qwen/Qwen2-VL-7B-Instruct\",
    \"p_framework_code\":\"vllm\",
    \"p_version\":\"0.20.0\"
  }" | jq

Parameters:

Name	Required	Description
`p_owner_account_id`	Yes	Account UUID that owns the project.
`p_project_name`	Yes	Existing project name.
`p_workload_code`	Yes	Stable workload code. Use lowercase words separated by hyphens.
`p_workload_name`	Yes	Display name. Often the same as `p_workload_code`.
`p_tool_code`	Yes	Benchmark tool code, usually `metrumbench-vlm`.
`p_model_code`	Yes	Model catalog code.
`p_framework_code`	Yes	Serving framework code, for example `vllm`.
`p_version`	Yes	Framework version.
`p_quantization_code`	No	Quantization code, when applicable.
`p_engine_args_set_code`	No	Engine argument preset, for example `vllm-default`.

Create Scenario

Creates one benchmark scenario for a workload. For VLM, the workload pulls images from the prompt source referenced on the workload; the scenario adds concurrency, ISL/OSL targets, and per-request image parameters.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"qwen-vl-h100\",
    \"p_workload_code\":\"qwen2-vl-7b-vllm\",
    \"p_scenario_code\":\"c8-isl256-osl128-img1-high\",
    \"p_scenario_name\":\"c8-isl256-osl128-img1-high\",
    \"p_concurrency\":8,
    \"p_input_sequence_length\":256,
    \"p_output_sequence_length\":128,
    \"p_num_images_batch\":1,
    \"p_image_detail\":\"high\",
    \"p_num_requests\":200
  }" | jq

Parameters:

Name	Required	Description
`p_owner_account_id`	Yes	Account UUID that owns the project.
`p_project_name`	Yes	Existing project name.
`p_workload_code`	Yes	Workload code created earlier.
`p_scenario_code`	Yes	Stable scenario code. Include ISL, OSL, concurrency.
`p_scenario_name`	Yes	Display name. Often the same as `p_scenario_code`.
`p_concurrency`	No	Concurrent requests.
`p_input_sequence_length`	No	Requested prompt/input token count.
`p_output_sequence_length`	No	Requested output token count.
`p_num_images_batch`	No	Images per request.
`p_image_detail`	No	`low` or `high`. Affects token cost.
`p_num_requests`	No	Total requests for the scenario.

Create Workload​

Create Scenario​

See also​

Create Workload

Create Scenario

See also