Skip to main content

Distributed KV Cache

A distributed KV cache workload shares or moves KV cache between the machines of a cluster group. In the API, it is a KV Cache Offload workload (tool code multiroundqa) that carries distributed KV settings. The run starts one pool of engines on the cluster group and then runs the workload's scenarios.

The Distributed KV Cache User Guide explains each setting and lists the stacks. This page shows the API calls.

The MCP server has no distributed KV tool. Use the RPCs on this page.

Set these environment variables for the examples:

export METRUM_API_URL="https://<control-plane-host>/api/postgrest"
export METRUM_JWT_TOKEN="<personal-access-token>"
export METRUM_ACCOUNT_ID="<account-uuid>"
export METRUM_ORG_ID="<org-uuid>"

Workflow​

StepCallResult
1list_server_config_instancesThe machines of your workspace and their server_id values.
2list_cluster_groups or create_cluster_groupA cluster group and its group_code.
3GET /kv_stacks and list_kv_cache_tiers_for_recipeA stack code and a KV cache tier for your engine.
4preview_distributed_kv_topology (optional)The per-machine plan, or the reason the settings cannot run.
5save_project_configA project with the distributed KV workload.
6preflight_project_runPlan, hardware, and busy cluster checks.
7execute_project_runA run, started now or queued for the cluster group.
8get_distributed_kv_run_status, get_run_cluster_queue_statePool and machine state, and the place in the queue.
9get_distributed_kv_run_report, list_distributed_kv_workload_resultsResults per job.

Permissions use two Casbin resources. cluster_group:read belongs to the owner, admin, member, and viewer roles. cluster_group:write belongs to the owner, admin, and member roles. An RPC without a p_org_id parameter is checked against the workspace in your token's org_id claim.

Step 1: List Machines​

curl -fsS -X POST "$METRUM_API_URL/rpc/list_server_config_instances" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_org_id\":\"$METRUM_ORG_ID\",\"p_account_id\":\"$METRUM_ACCOUNT_ID\"}" | jq
NameTypeRequiredDescription
p_org_iduuidYesWorkspace.
p_status_codevarcharNoFilter by machine status.
p_account_iduuidNoThe caller. If you omit it, the RPC uses the account in your token. A value that is not your account is refused.

Returns a JSON array of machines, ordered by hostname. Each element has server_id, hostname, availability_code, config_code, config_gpu_model, config_gpu_count, hardware_label, last_seen_at, and other hardware fields. When you do not belong to the workspace, it returns an error object: Access denied: account is not a member of the specified org. Through the hosted proxy, that error object arrives as HTTP 400 with code RPC_ERROR.

Step 2: Cluster Groups​

list_cluster_groups​

Lists the cluster groups of one workspace, with member and GPU counts. A group that holds a decommissioned machine is left out.

curl -fsS -X POST "$METRUM_API_URL/rpc/list_cluster_groups" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_org_id\":\"$METRUM_ORG_ID\"}" | jq
NameTypeRequiredDescription
p_account_iduuidYesThe caller. It must be the account in your token.
p_org_iduuidYesWorkspace.
ColumnTypeDescription
group_idintegerInternal id.
group_namevarcharName.
group_codevarcharCode. The distributed KV settings name the group by this code.
member_countintegerMachines in the group.
total_gpu_countintegerGPUs across the machines.

The result is empty, not an error, when you do not own or belong to the workspace. Permission: the RPC checks workspace membership itself.

create_cluster_group​

Creates a named group of machines. The first machine in p_server_ids is the head. The head coordinates a distributed KV run and starts the shared tier.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_cluster_group" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_org_id\":\"$METRUM_ORG_ID\",
\"p_group_name\":\"L40S pair\",
\"p_server_ids\":[<head-server-id>, <second-server-id>]
}" | jq
NameTypeRequiredDescription
p_account_iduuidYesThe caller. It must be the account in your token.
p_org_iduuidYesWorkspace. Every machine must belong to it.
p_group_namevarcharYesName. The code is derived from it, for example l40s-pair.
p_server_idsinteger[]YesMachines, head first.

Returns one row with group_id. Call list_cluster_groups to read the group_code.

Errors: p_account_id must match the authenticated caller, p_group_name is required, p_server_ids must contain at least one server, server_id ... does not belong to org ..., and account ... is not authorized to create a cluster group for org .... Each one is an HTTP 400 response with code P0001.

Permission: cluster_group:write.

Step 3: Stacks And Tiers​

A stack is one engine, KV mode, cache layer, and transfer path that the runtime can launch. Read the active stacks:

curl -fsS "$METRUM_API_URL/kv_stacks?is_active=eq.true&order=sort_order&select=code,display_name,min_engine_version,max_engine_version,supports_nvidia,supports_amd,frameworks(framework_code)" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

A stack runs on one engine. The workload's KV cache tier must match the stack: a stack whose cache layer needs an LMCache server takes the cpu or nvme tier, and every other stack takes the gpu tier. List the tiers for an engine:

curl -fsS -X POST "$METRUM_API_URL/rpc/list_kv_cache_tiers_for_recipe" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{"p_tool_code":"multiroundqa","p_framework_code":"vllm"}' | jq
NameTypeRequiredDescription
p_tool_codevarcharYesmultiroundqa.
p_framework_codevarcharYesEngine, for example vllm.

Each row has tier_code, display_name, description, requires_cache_server, model_server_engine_args_set_code, and cache_server_engine_args_set_code, plus the AMD variants of both sets. Use model_server_engine_args_set_code as the workload's engine_args_set_code. For the cpu and nvme tiers, use cache_server_engine_args_set_code as lmcache_engine_args_set_code.

Permission: catalog:read.

Step 4: Preview The Plan​

Shows the plan a distributed KV workload would run with, before you save it. It raises the same message that saving or running the workload would raise.

curl -fsS -X POST "$METRUM_API_URL/rpc/preview_distributed_kv_topology" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_org_id\":\"$METRUM_ORG_ID\",
\"p_cluster_group_code\":\"l40s-pair\",
\"p_stack_code\":\"vllm-native-offload\",
\"p_framework_code\":\"vllm\",
\"p_framework_version\":\"<engine-version>\",
\"p_engine_args_set_code\":\"<model-server-engine-args-set-code>\",
\"p_model_code\":\"<model-code>\"
}" | jq
NameTypeRequiredDescription
p_account_iduuidYesThe caller.
p_org_iduuidYesWorkspace of the cluster group.
p_cluster_group_codevarcharYesCluster group.
p_stack_codevarcharYesStack.
p_framework_codevarcharYesEngine.
p_framework_versionvarcharYesEngine version.
p_engine_args_set_codevarcharYesEngine arguments set. Tensor and data parallel sizes come from it.
p_orchestrator_codevarcharNoDefault bare_metal.
p_max_gpus_per_nodeintegerNoGPUs to use per machine.
p_prefill_node_countintegerNoPrefill machines of a prefill and decode stack.
p_prefill_tensor_parallel_sizeintegerNoTensor parallel size of the prefill engines.
p_prefill_gpu_countintegerNoPrefill GPUs on a one-machine prefill and decode group.
p_model_codevarcharNoModel. Required when the tensor parallel size is auto.

Returns one row per machine: server_id, node_index, node_role, hostname, gpu_vendor, gpu_model, gpu_family_code, gpu_family_name, gpu_count, gpus_used, tensor_parallel_size, replica_count, is_head, stack_verdict, and prefill_replica_count.

Permission: cluster_group:read.

Step 5: Save The Project​

Send a benchmark workload with tool_code multiroundqa and a distributed_kv object to save_project_config. A distributed KV workload runs on its cluster group, so it has no target_selections. The Project Lifecycle page documents the RPC, its return row, and its errors.

curl -fsS -X POST "$METRUM_API_URL/rpc/save_project_config" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_org_id\":\"$METRUM_ORG_ID\",
\"p_project_uuid\":null,
\"p_save_as_new\":false,
\"p_expected_revision\":null,
\"p_payload\":{
\"project_name\":\"dkv-native-offload-l40s\",
\"visibility_code\":\"org\",
\"workloads\":[{
\"type\":\"benchmark\",
\"workload_code\":\"dkv-native-offload\",
\"tool_code\":\"multiroundqa\",
\"model_codes\":[\"<model-code>\"],
\"framework_code\":\"vllm\",
\"version\":\"<engine-version>\",
\"engine_args_set_code\":\"<model-server-engine-args-set-code>\",
\"kv_cache_tier_code\":\"gpu\",
\"target_selections\":[],
\"scenarios\":[{
\"scenario_code\":\"c32-r3\",
\"concurrency\":32,
\"job_params\":{\"num_rounds\":\"3\"}
}],
\"distributed_kv\":{
\"cluster_group_code\":\"l40s-pair\",
\"stack_code\":\"vllm-native-offload\",
\"host_tier_gb\":64
}
}]
}
}" | jq

The distributed_kv object​

cluster_group_code and stack_code are required. Every other key is optional. An omitted code takes the catalog default for the stack. A code resolves inside the workspace of the workload's project.

KeyTypeDescription
cluster_group_codetextCluster group.
stack_codetextStack.
transfer_backend_codetextTransfer backend. Default: the stack's default backend.
transport_codetextNetwork protocol. Default auto: every machine's InfiniBand or RoCE ports when all machines have them, otherwise the fastest TCP interface they share. The run records the path it chose and why.
routing_policy_codetextRouter policy.
orchestrator_codetextDefault bare_metal.
gpus_per_nodeintegerGPUs to use per machine.
shared_tier_gbnumberShared store size, for a stack that starts a shared store.
host_tier_gbnumberHost memory tier per machine, for a stack that offloads to host memory.
disk_tier_gb, disk_tier_pathnumber, textDisk tier size and absolute path.
prefill_node_count, prefill_tensor_parallel_size, prefill_gpu_countintegerPrefill sizing of a prefill and decode stack.
rdma_device_nametextRDMA ports to use, comma-separated, for example bnxt_re0,bnxt_re1 or mlx5_0:1. Valid for the rdma and auto transports. Leave it out to use every active port on every machine.
tcp_interface_nametextNetwork interface for KV over TCP, for example eno8303. Valid for the tcp and auto transports. Every machine must have it. Leave it out to use the fastest interface.
decode_prefix_cachingbooleanfalse turns prompt caching off on decode engines only, so each request moves its whole prompt KV from prefill to decode. Default true. Prefill and decode stacks only.
node_layoutarrayPrefill and decode GPUs per machine, in cluster group order, as [{"prefill_gpus": 2, "decode_gpus": 6}, {"prefill_gpus": 4, "decode_gpus": 4}]. Prefill and decode stacks only. Needs a numeric tensor parallel size for each role, and cannot be combined with gpus_per_node, prefill_node_count or prefill_gpu_count.
expected_fabric_codetextNetwork fabric the run must use.
declared_pool_state_code, prepare_action_codetextCache state at the start of a job, and how the run prepares it.
load_failure_policy_codetextWhat happens when one machine fails to load.
working_set_rationumberWorking set as a multiple of the measured GPU KV capacity, from 0.05 to 100.
repeat_countintegerRepeats per scenario, from 1 to 20. Fewer than 3 marks results preliminary.
load_secondsintegerHold each job's load for this long, from 120 to 14400 seconds. Without it, the run replays one session per simulated request stream.
warmup_secondsintegerPart of the held load the figures leave out, from 0 to 3600 seconds and less than half of load_seconds.
fault_injection_code, fault_paramstext, objectFault to inject, and its settings as an object of names to numbers.
workloadobjectThe workload shape. null returns it to the default shape. See the next table.

The workload object takes shape_code, arrival_pattern_code, arrival_rate_qps, burstiness_cv, max_requests, max_duration_s, overlength_policy_code, prompt_mode_code, client_workers, seed, trace_format_code, trace_source, time_scale, min_session_turns, prefix_tokens, prefix_group_count, users_per_prefix, turns_per_user, turn_input_tokens, output_tokens, context_growth_tokens, subagent_fanout, subagent_input_tokens, think_time_distribution_code, think_time_mean_s, shared_system_prompt_tokens, user_history_prompt_tokens, and rate_steps. rate_steps is an array of {"rate_qps": <number>, "duration_s": <integer>, "request_count": <integer>}. Only the agentic data set replay shape reads a scenario's dataset_name. Any other shape with a dataset_name is refused.

The save refuses settings that cannot run, with the first reason. Examples: Cluster group ... not found in this organization, ... needs an LMCache server on every node. Choose the CPU RAM or NVMe SSD tier for this workload., and ... offloads to host memory on each node. Set the host tier size in GB.

Change the settings of a saved workload​

upsert_workload_distributed_kv_config sets or replaces the distributed KV settings of one saved workload. It takes the keys of the distributed_kv object as parameters with a p_ prefix, except these four:

distributed_kv keyRPC parameter
gpus_per_nodep_max_gpus_per_node
fault_paramsp_fault_param_keys and p_fault_param_values (text arrays in matching order)
workloadNot taken. Use upsert_workload_distributed_kv_workload.
(none)p_account_id (uuid, required) and p_workload_id (integer, required)

p_cluster_group_code and p_stack_code are required. It returns one row with workload_id, node_count, and stack_verdict. Read the current settings with get_workload_distributed_kv_config(p_account_id, p_workload_id), and the shape with get_workload_distributed_kv_workload(p_account_id, p_workload_id). list_project_edit_workloads returns the workload_id.

Permission: cluster_group:write for the upsert, cluster_group:read for the reads.

A project that has run cannot be changed. A direct call to upsert_workload_distributed_kv_config, remove_workload_distributed_kv_config, upsert_workload_distributed_kv_workload, remove_workload_distributed_kv_workload, create_workload_node_group_selection, or remove_workload_node_group_selection on a workload of such a project is refused with project_executed. To change the distributed KV settings of a project that has run, copy it with copy_project and change the copy.

Step 6: Check Before You Run​

preflight_project_run takes the same payload. For a distributed KV workload it returns no_hardware when the cluster group is not in the workspace, not_in_plan when the workspace plan does not include distributed KV cache, and the note cluster_busy when another run uses the group. See preflight_project_run.

Step 7: Run​

Call execute_project_run without a machine. The run goes to the head of the workload's cluster group.

curl -fsS -X POST "$METRUM_API_URL/rpc/execute_project_run" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_project_name\":\"dkv-native-offload-l40s\",
\"p_project_uuid\":\"<project-uuid>\",
\"p_run_name\":\"dkv-native-offload-run-1\"
}" | jq

How the call behaves:

ProjectResult
One distributed KV workloadOne run on the cluster group.
Several workloads, no p_workload_codeOne run per workload under one execution_group_id. Single-machine workloads start first on the machine you pass. Distributed KV workloads join the execution's queue, because one cluster group runs one pool at a time. The response has run_ids, queued_workloads, and split_by_workload set to true.
A queued workloadThe response has queued set to true, queue_position, queue_size, and status waiting.
A cluster group that another run usesThe run is created and holds every machine of the group from its first job. While another run still uses a machine of the group, the run waits and its job stays pending. get_run_cluster_queue_state returns waiting with the runs ahead. The run starts when the machines are free. If it cannot start, the job failure shows the database's reason, which names the machine and the run.

Errors raised for distributed KV, each an HTTP 400 response with code P0001:

messageCause
A distributed KV workload runs on its own cluster group. Run it by itself by naming its workload.p_workload_code matches more than one workload.
A distributed KV workload runs on its cluster group, coordinated by the group's first machine. Start it without choosing a server.You passed a machine that is not the head.

Permission: project:execute.

Step 8: Monitor​

get_distributed_kv_run_status​

Returns one row per machine of every pool of a run, newest pool first. It never returns the store password or the engine API key.

curl -fsS -X POST "$METRUM_API_URL/rpc/get_distributed_kv_run_status" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_run_id\":<run-id>}" | jq
NameTypeRequiredDescription
p_account_iduuidYesThe caller.
p_run_idintegerYesRun id from execute_project_run.

Key columns (the RPC returns 50; the OpenAPI document lists them all):

ColumnTypeDescription
pool_run_id, epochintegerThe pool and its attempt number.
phasevarcharPool phase.
failure_reasontextWhy the pool failed.
failed_server_idintegerThe machine at fault.
stack_code, stack_verdictvarcharStack and its verdict on this engine version.
node_count, total_replicasintegerMachines and engine replicas in the pool.
server_id, hostname, node_role, is_headmixedThe machine this row describes.
member_statevarcharState of this machine in the pool.
gpus_used, replica_countintegerGPU use on this machine.
engine_version, kv_cache_dtypevarcharResolved engine settings.
gpu_kv_capacity_tokensbigintMeasured GPU KV capacity.
heartbeat_age_secondsintegerSeconds since the machine's last heartbeat.
member_errortextThe machine's error.

Errors, each an HTTP 400 response with code P0001: p_account_id is required, run ... not found, and insufficient_privilege when you cannot access the project.

Permission: cluster_group:read.

get_run_cluster_queue_state​

Returns where the newest run of a workload stands in line for its cluster group.

curl -fsS -X POST "$METRUM_API_URL/rpc/get_run_cluster_queue_state" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_org_id\":\"$METRUM_ORG_ID\",\"p_workload_id\":<workload-id>}" | jq
NameTypeRequiredDescription
p_account_iduuidYesThe caller.
p_org_iduuidYesWorkspace.
p_workload_idintegerYesWorkload.
p_execution_group_iduuidNoLimit to one execution.
ColumnTypeDescription
r_statevarcharwaiting when the run has not started, none when it is not queued.
r_cluster_group_namevarcharCluster group.
r_holder_run_idintegerRun that holds the group now. Hidden when you cannot open it.
r_runs_aheadintegerRuns that go first, including the holder and the runs that still use a machine of the group.
r_queue_lengthintegerRuns in line.
r_waiting_sincetimestamptzWhen this run started waiting.

Errors: workload ... was not found (P0002, HTTP 500), Caller may only act as their own account and insufficient_privilege (42501, HTTP 403).

Permission: project:read.

stop_project_execution_queue​

Stops the queue of an execution. Every workload still waiting is marked not run. The workload that is running finishes.

curl -fsS -X POST "$METRUM_API_URL/rpc/stop_project_execution_queue" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_org_id\":\"$METRUM_ORG_ID\",\"p_execution_group_id\":\"<execution-group-id>\"}" | jq
NameTypeRequiredDescription
p_account_iduuidYesThe caller.
p_org_iduuidYesWorkspace.
p_execution_group_iduuidYesexecution_group_id from execute_project_run.

Returns one row with execution_group_id, dequeued_workloads, and running_workloads.

Errors: This execution has no queue of workloads to stop. and insufficient_privilege. A metrum admin, the workspace owner, a workspace admin, or the account that ran the project can stop the queue.

Permission: project:execute.

Step 9: Read The Results​

get_distributed_kv_run_report​

Returns one row per distributed KV job of a run, with the figures the report leads with.

curl -fsS -X POST "$METRUM_API_URL/rpc/get_distributed_kv_run_report" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_run_id\":<run-id>}" | jq

Parameters are the same as get_distributed_kv_run_status.

Key columns (the RPC returns 48; the OpenAPI document lists them all):

ColumnTypeDescription
job_id, scenario_name, concurrency, repeat_indexmixedThe job.
job_statusvarcharJob status.
share_from_other_machinesnumericShare of prompt tokens served from another machine (shared pool plus peer).
share_from_own_gpunumericShare served from the machine's own GPU cache.
share_computednumericShare computed again.
attribution_completebooleanWhether every token is attributed. A share is null when attribution is incomplete.
tokens_unattributednumericTokens left unattributed.
kv_transfer_gbpsnumericKV transfer bandwidth.
kv_usage_peaknumericPeak GPU KV usage.
mean_ttft_s, p50_ttft_s, p90_ttft_s, p99_ttft_snumericTime to first token, in seconds.
output_throughput_tps, output_tps_per_gpunumericOutput tokens per second, and per GPU.
is_validbooleanWhether the job result is valid.
invalid_reasonstextEvery reason when it is not.

Errors and permission: the same as get_distributed_kv_run_status.

list_distributed_kv_workload_results​

Returns one row per distributed KV job of a run, with its data set profile and cache hit rates.

curl -fsS -X POST "$METRUM_API_URL/rpc/list_distributed_kv_workload_results" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_run_id\":<run-id>}" | jq

Key columns (the RPC returns 31): job_id, shape_code, requests_scheduled, requests_dropped, requests_trimmed, oracle_basis_code, oracle_hit_rate, achieved_hit_rate, achieved_hit_source, achieved_over_oracle, step_count, and calibration_failed_steps.

Permission: cluster_group:read.