metrumbench-asr Workload And Scenario
Use this flow to create an ASR workload and its audio scenario matrix. ASR is scoped to the Whisper model family on vLLM only.
Set these environment variables for the examples:
export METRUM_API_URL="https://api.<control-plane-domain>"
export METRUM_JWT_TOKEN="<12-hour-db-minted-postgrest-jwt>"
export METRUM_ACCOUNT_ID="<account-uuid>"
Create Workload
Creates a benchmark workload for a project, tool, model, framework, version, and engine args set.
curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_project_name\":\"whisper-asr-h100\",
\"p_workload_code\":\"whisper-largev3-vllm\",
\"p_workload_name\":\"whisper-largev3-vllm\",
\"p_tool_code\":\"metrumbench-asr\",
\"p_model_code\":\"openai/whisper-large-v3\",
\"p_framework_code\":\"vllm\",
\"p_version\":\"0.20.0\"
}" | jq
Parameters:
| Name | Required | Description |
|---|---|---|
p_owner_account_id | Yes | Account UUID that owns the project. |
p_project_name | Yes | Existing project name. |
p_workload_code | Yes | Stable workload code. Use lowercase words separated by hyphens. |
p_workload_name | Yes | Display name. Often the same as p_workload_code. |
p_tool_code | Yes | Benchmark tool code, usually metrumbench-asr. |
p_model_code | Yes | Whisper model code, for example openai/whisper-large-v3. |
p_framework_code | Yes | vllm only. SGLang and TensorRT-LLM are not productized for ASR. |
p_version | Yes | vLLM framework version. |
p_quantization_code | No | Quantization code, when applicable. |
p_engine_args_set_code | No | Engine argument preset, for example vllm-default. |
Create Scenario
Creates one benchmark scenario for an ASR workload. For ASR, the main knobs are
p_concurrency and p_num_requests; audio is pulled from the prompt source
referenced on the workload.
curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_project_name\":\"whisper-asr-h100\",
\"p_workload_code\":\"whisper-largev3-vllm\",
\"p_scenario_code\":\"c8-librispeech-clean\",
\"p_scenario_name\":\"c8-librispeech-clean\",
\"p_concurrency\":8,
\"p_num_requests\":200
}" | jq
Parameters:
| Name | Required | Description |
|---|---|---|
p_owner_account_id | Yes | Account UUID that owns the project. |
p_project_name | Yes | Existing project name. |
p_workload_code | Yes | Workload code created earlier. |
p_scenario_code | Yes | Stable scenario code. Include concurrency. |
p_scenario_name | Yes | Display name. Often the same as p_scenario_code. |
p_concurrency | No | Concurrent transcription requests. |
p_num_requests | No | Total transcriptions to send. |
p_response_format | No | verbose_json, json, text, srt, or vtt. Default verbose_json. |