Multi-Node vLLM Deployment

Use multi-node deployment when a model does not fit one server's GPU memory, or the concurrency target needs tensor and pipeline parallelism spread across several hosts. Metrum Insights forms a Kubernetes cluster across the servers you group together, then deploys the model through llm-d (Gateway API, the Gateway API Inference Extension, and a LeaderWorkerSet) instead of launching a single-process model server on one host.

This is a deployment topology for vLLM specifically, not a separate benchmark tool: it builds on the workload and scenario created in metrumbench-llm Workload And Scenario (same tool, same create_workload/create_scenario shapes for anything not covered below) with framework_code: "vllm". SGLang and TensorRT-LLM do not have a multi-node path today.

Set these environment variables for the examples, alongside the ones from metrumbench-llm Workload And Scenario:

export METRUM_ORG_ID="<org-uuid>"

Prerequisites

• Every server you plan to group must already be onboarded (see Hardware Onboarding) with p_ip_address set. K3s cluster formation joins workers to the head over that address; a server onboarded without one cannot join a multi-node cluster.
• Every server in a group must have the same GPU vendor. The cluster's GPU device plugin is selected once, from the head server's detected vendor, and applies to every member; a mixed-vendor group leaves the other members' GPUs unusable to the deployment.
• Verified live on NVIDIA GPUs only. The bootstrap installs an AMD GPU device plugin and configures RCCL cross-node environment variables as well, but that path has not been confirmed end-to-end the way NVIDIA has. Treat AMD multi-node as unverified until it has been.
• A cluster group needs at least one server, but multi-node deployment is only useful with two or more.

Group Hardware Into A Cluster

Creates a reusable, org-level group of onboarded servers. A workload later references this group by id; the group itself is not tied to any one project or workload.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_cluster_group" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_group_name\":\"h200-pair\",
    \"p_server_ids\":[101,102]
  }" | jq

Parameters:

Name	Required	Description
p_account_id	Yes	Account UUID; must match the authenticated caller.
p_org_id	Yes	Organization UUID that owns the group.
p_group_name	Yes	Display name. A URL-safe group code is derived from it.
p_server_ids	Yes	Server numeric ids (server_config_instances.server_id), from Register Server or List Servers in Hardware Onboarding. Position 0 becomes the cluster's head node.

The response's group_id is the numeric id used everywhere below.

List Cluster Groups

Inspect existing groups, their members, and any workload already using them.

curl -fsS -X POST "$METRUM_API_URL/rpc/list_cluster_groups" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_org_id\":\"$METRUM_ORG_ID\"}" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/list_cluster_group_members" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_group_id\":1}" | jq

Parameters:

RPC	Parameter	Required	Description
list_cluster_groups	p_account_id	Yes	Account UUID.
list_cluster_groups	p_org_id	Yes	Organization UUID.
list_cluster_group_members	p_account_id	Yes	Account UUID.
list_cluster_group_members	p_group_id	Yes	Cluster group numeric id.

Choosing Tensor and Pipeline Parallelism

Set tensor_parallel_size to the number of GPUs on one machine, and pipeline_parallel_size to the number of machines in the cluster group. vLLM launches every group member as one rank (--nnodes equal to the group's member count, one pod per member), so tensor parallelism stays inside a single node's fast NVLink/PCIe interconnect, and pipeline parallelism is what crosses the slower network between nodes. Do not reverse them: splitting tensor-parallel ranks across nodes puts the all-reduce traffic vLLM runs on every forward pass onto the network link instead of NVLink/PCIe, which is far slower and typically dominates latency.

For a 2-node group where each node has 8 GPUs, set tensor_parallel_size: 8 and pipeline_parallel_size: 2 (16 GPUs total). Both are engine args on the engine_args_set_code referenced below; see Command Templates for the vLLM engine args reference.

No seeded preset carries a pipeline_parallel_size above 1 today (the catalog's TP8 presets, vllm-default included, target a single node), so a genuine multi-node deployment needs its own engine args set with both values explicit. Creating one requires a metrum_admin or service-level caller, not a regular org member's own JWT:

curl -fsS -X POST "$METRUM_API_URL/rpc/create_framework_engine_args_set" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "p_framework_code":"vllm",
    "p_version":"0.20.0",
    "p_set_code":"vllm-deepseek-r1-tp8-pp2-multinode",
    "p_set_name":"DeepSeek-R1 TP8/PP2 multi-node",
    "p_args_json":"{\"tensor_parallel_size\":\"8\",\"pipeline_parallel_size\":\"2\"}"
  }' | jq

Parameters:

Name	Required	Description
p_framework_code / p_version	Yes	Must match an onboarded framework version, e.g. vllm / 0.20.0.
p_set_code	Yes	Stable code referenced as engine_args_set_code below.
p_set_name	No	Display name.
p_args_json	Yes	JSON object of arg_key: arg_value pairs, validated against that framework version's registered engine arg keys. Values are strings even for numeric/flag args.

An existing p_set_code for the same framework version is replaced, not duplicated, so re-running this call to adjust tensor_parallel_size or pipeline_parallel_size is safe.

Create A Multi-Node Project And Workload

A multi-node workload is created through create_project_with_workloads, not the plain Create Workload call in metrumbench-llm Workload And Scenario: only this RPC stamps the workload's deployment mode and attaches its cluster group selection in the same transaction. workloads[].node_group_selections is what makes a workload multi-node; a workload created here without it behaves like any other single-node metrumbench-llm workload.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_project_with_workloads" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_payload\":{
      \"owner_account_id\":\"$METRUM_ACCOUNT_ID\",
      \"project_name\":\"deepseek-r1-h200pair-multinode\",
      \"visibility_code\":\"private\",
      \"workloads\":[{
        \"type\":\"benchmark\",
        \"workload_code\":\"deepseek-r1-vllm-multinode\",
        \"workload_name\":\"deepseek-r1-vllm-multinode\",
        \"tool_code\":\"metrumbench-llm\",
        \"model_codes\":[\"deepseek-ai/DeepSeek-R1\"],
        \"framework_code\":\"vllm\",
        \"version\":\"0.20.0\",
        \"engine_args_set_code\":\"vllm-deepseek-r1-tp8-pp2-multinode\",
        \"node_group_selections\":[{
          \"cluster_group_id\":1,
          \"selection_name\":\"h200-pair\"
        }],
        \"scenarios\":[{
          \"scenario_code\":\"isl128-osl1024-c256\",
          \"scenario_name\":\"isl128-osl1024-c256\",
          \"concurrency\":256,
          \"input_sequence_length\":128,
          \"output_sequence_length\":1024,
          \"num_requests\":1024
        }]
      }]
    }
  }" | jq

Parameters:

Name	Required	Description
p_org_id	Yes	Organization UUID. Gates access via rpc_permission_map.
p_payload.owner_account_id	Yes	Account UUID that owns the project.
p_payload.project_name	Yes	New project name.
p_payload.workloads[].tool_code	No	Defaults to metrumbench-llm.
p_payload.workloads[].model_codes	Yes	Array of model catalog codes (plural, unlike Create Workload's single p_model_code).
p_payload.workloads[].framework_code / version	Yes	Serving framework and version. Use vllm; SGLang and TensorRT-LLM have no multi-node path.
p_payload.workloads[].engine_args_set_code	No	Engine argument preset. Tensor and pipeline parallelism (tensor_parallel_size, pipeline_parallel_size) for the deployment are read from here; use the set created in Choosing Tensor and Pipeline Parallelism, not a single-node preset like vllm-default.
p_payload.workloads[].node_group_selections[].cluster_group_id	Yes	group_id from Group Hardware Into A Cluster. Exactly one selection is supported per workload today.
p_payload.workloads[].node_group_selections[].selection_name	Yes	Display name for this workload's use of the group.
p_payload.workloads[].scenarios[]	No	Same fields as Create Scenario in metrumbench-llm Workload And Scenario.

Configure The llm-d Deployment (Optional)

Sets deployment-identity fields for the workload's Helm release: the Inference Pool name Gateway API routes into, the target namespace, and the Helm chart version. Skip this call to accept the defaults execute_project_run provisions automatically on first run (a pool name derived from the workload code, default namespace). Chart version has no pinned default: leaving p_chart_version unset means helm upgrade --install runs with no --version flag, so Helm installs whatever the chart repository's latest release happens to be at deploy time. Set p_chart_version explicitly for a reproducible deployment.

curl -fsS -X POST "$METRUM_API_URL/rpc/upsert_llm_d_workload_config" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_workload_id\":1,
    \"p_inference_pool_name\":\"deepseek-r1-vllm-multinode\",
    \"p_gateway_namespace\":\"default\"
  }" | jq

Parameters:

Name	Required	Description
p_account_id	Yes	Account UUID; must match the authenticated caller.
p_workload_id	Yes	Workload numeric id. Must already be k8s_multinode_llm_d deployment mode.
p_inference_pool_name	Yes	Name of the Gateway API Inference Extension InferencePool this workload's deployments register into.
p_gateway_namespace	No	Kubernetes namespace for the Gateway/HTTPRoute. Defaults to default.
p_chart_version	No	Pins the Helm chart version. Leaving it unset installs the chart repository's current latest release, not a validated default.
p_readiness_timeout_seconds	No	helm upgrade --install --wait --timeout value. Defaults to 900.
p_cleanup_policy	No	uninstall (default, tears the release down) or retain (leaves it installed for debugging).

Run A Multi-Node Project

Launch with execute_project_run, not execute_scenario: pass no p_server_id/p_server_config_instance_id at all. Metrum resolves the target automatically to the cluster group's head server, forms the Kubernetes cluster across every group member if one is not already formed, and deploys the llm-d chart before any benchmark job starts.

curl -fsS -X POST "$METRUM_API_URL/rpc/execute_project_run" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"deepseek-r1-h200pair-multinode\",
    \"p_run_name\":\"deepseek-r1-h200pair-multinode-run1\",
    \"p_workload_code\":\"deepseek-r1-vllm-multinode\"
  }" | jq

Parameters:

Name	Required	Description
p_owner_account_id	Yes	Account UUID that owns the project.
p_project_name	Yes	Project created above.
p_run_name	Yes	Display name for this run.
p_workload_code	No but recommended	Restricts the run to this workload when the project holds others.

Cluster formation (K3s join across every group member, then Gateway API/GAIE/ Agentgateway/LeaderWorkerSet controller bootstrap) is asynchronous and, on first use of a cluster group, can take several minutes. A cluster group that has already formed a cluster for a prior run reuses it.

Monitor A Multi-Node Run

Poll job status the same way as any other workload, via Get Job Status and List Jobs in Cloud Benchmark Execution. get_job_status reports the benchmark job's own status only; it has no visibility into cluster formation or the llm-d deployment, which run as a separate setup stage before the agent starts the job. Expect the job to report as not yet started for the full duration of that setup stage (several minutes on a cluster group's first use), then move to running once the deployment is ready and the benchmark actually begins.