Distributed KV Cache
A distributed KV cache workload shares or moves KV cache between the machines
of a cluster group. In the API, it is a KV Cache Offload workload (tool code
multiroundqa) that carries distributed KV settings. The run starts one pool
of engines on the cluster group and then runs the workload's scenarios.
The Distributed KV Cache User Guide explains each setting and lists the stacks. This page shows the API calls.
The MCP server has no distributed KV tool. Use the RPCs on this page.
Set these environment variables for the examples:
export METRUM_API_URL="https://<control-plane-host>/api/postgrest"
export METRUM_JWT_TOKEN="<personal-access-token>"
export METRUM_ACCOUNT_ID="<account-uuid>"
export METRUM_ORG_ID="<org-uuid>"
Workflow
| Step | Call | Result |
|---|---|---|
| 1 | list_server_config_instances | The machines of your workspace and their server_id values. |
| 2 | list_cluster_groups or create_cluster_group | A cluster group and its group_code. |
| 3 | GET /kv_stacks and list_kv_cache_tiers_for_recipe | A stack code and a KV cache tier for your engine. |
| 4 | preview_distributed_kv_topology (optional) | The per-machine plan, or the reason the settings cannot run. |
| 5 | save_project_config | A project with the distributed KV workload. |
| 6 | preflight_project_run | Plan, hardware, and busy cluster checks. |
| 7 | execute_project_run | A run, started now or queued for the cluster group. |
| 8 | get_distributed_kv_run_status, get_run_cluster_queue_state | Pool and machine state, and the place in the queue. |
| 9 | get_distributed_kv_run_report, list_distributed_kv_workload_results | Results per job. |
Permissions use two Casbin resources. cluster_group:read belongs to the
owner, admin, member, and viewer roles. cluster_group:write belongs to the
owner, admin, and member roles. An RPC without a p_org_id parameter is
checked against the workspace in your token's org_id claim.
Step 1: List Machines
curl -fsS -X POST "$METRUM_API_URL/rpc/list_server_config_instances" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_org_id\":\"$METRUM_ORG_ID\",\"p_account_id\":\"$METRUM_ACCOUNT_ID\"}" | jq
| Name | Type | Required | Description |
|---|---|---|---|
p_org_id | uuid | Yes | Workspace. |
p_status_code | varchar | No | Filter by machine status. |
p_account_id | uuid | No | The caller. If you omit it, the RPC uses the account in your token. A value that is not your account is refused. |
Returns a JSON array of machines, ordered by hostname. Each element has
server_id, hostname, availability_code, config_code,
config_gpu_model, config_gpu_count, hardware_label, last_seen_at, and
other hardware fields. When you do not belong to the workspace, it returns an
error object: Access denied: account is not a member of the specified org.
Through the hosted proxy, that error object arrives as HTTP 400 with code
RPC_ERROR.
Step 2: Cluster Groups
list_cluster_groups
Lists the cluster groups of one workspace, with member and GPU counts. A group that holds a decommissioned machine is left out.
curl -fsS -X POST "$METRUM_API_URL/rpc/list_cluster_groups" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_org_id\":\"$METRUM_ORG_ID\"}" | jq
| Name | Type | Required | Description |
|---|---|---|---|
p_account_id | uuid | Yes | The caller. It must be the account in your token. |
p_org_id | uuid | Yes | Workspace. |
| Column | Type | Description |
|---|---|---|
group_id | integer | Internal id. |
group_name | varchar | Name. |
group_code | varchar | Code. The distributed KV settings name the group by this code. |
member_count | integer | Machines in the group. |
total_gpu_count | integer | GPUs across the machines. |
The result is empty, not an error, when you do not own or belong to the workspace. Permission: the RPC checks workspace membership itself.
create_cluster_group
Creates a named group of machines. The first machine in p_server_ids is the
head. The head coordinates a distributed KV run and starts the shared tier.
curl -fsS -X POST "$METRUM_API_URL/rpc/create_cluster_group" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_org_id\":\"$METRUM_ORG_ID\",
\"p_group_name\":\"L40S pair\",
\"p_server_ids\":[<head-server-id>, <second-server-id>]
}" | jq
| Name | Type | Required | Description |
|---|---|---|---|
p_account_id | uuid | Yes | The caller. It must be the account in your token. |
p_org_id | uuid | Yes | Workspace. Every machine must belong to it. |
p_group_name | varchar | Yes | Name. The code is derived from it, for example l40s-pair. |
p_server_ids | integer[] | Yes | Machines, head first. |
Returns one row with group_id. Call list_cluster_groups to read the
group_code.
Errors: p_account_id must match the authenticated caller,
p_group_name is required, p_server_ids must contain at least one server,
server_id ... does not belong to org ..., and
account ... is not authorized to create a cluster group for org .... Each
one is an HTTP 400 response with code P0001.
Permission: cluster_group:write.
Step 3: Stacks And Tiers
A stack is one engine, KV mode, cache layer, and transfer path that the runtime can launch. Read the active stacks:
curl -fsS "$METRUM_API_URL/kv_stacks?is_active=eq.true&order=sort_order&select=code,display_name,min_engine_version,max_engine_version,supports_nvidia,supports_amd,frameworks(framework_code)" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq
A stack runs on one engine. The workload's KV cache tier must match the stack:
a stack whose cache layer needs an LMCache server takes the cpu or nvme
tier, and every other stack takes the gpu tier. List the tiers for an engine:
curl -fsS -X POST "$METRUM_API_URL/rpc/list_kv_cache_tiers_for_recipe" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{"p_tool_code":"multiroundqa","p_framework_code":"vllm"}' | jq
| Name | Type | Required | Description |
|---|---|---|---|
p_tool_code | varchar | Yes | multiroundqa. |
p_framework_code | varchar | Yes | Engine, for example vllm. |
Each row has tier_code, display_name, description,
requires_cache_server, model_server_engine_args_set_code, and
cache_server_engine_args_set_code, plus the AMD variants of both sets.
Use model_server_engine_args_set_code as the workload's
engine_args_set_code. For the cpu and nvme tiers, use
cache_server_engine_args_set_code as lmcache_engine_args_set_code.
Permission: catalog:read.
Step 4: Preview The Plan
Shows the plan a distributed KV workload would run with, before you save it. It raises the same message that saving or running the workload would raise.
curl -fsS -X POST "$METRUM_API_URL/rpc/preview_distributed_kv_topology" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_org_id\":\"$METRUM_ORG_ID\",
\"p_cluster_group_code\":\"l40s-pair\",
\"p_stack_code\":\"vllm-native-offload\",
\"p_framework_code\":\"vllm\",
\"p_framework_version\":\"<engine-version>\",
\"p_engine_args_set_code\":\"<model-server-engine-args-set-code>\",
\"p_model_code\":\"<model-code>\"
}" | jq
| Name | Type | Required | Description |
|---|---|---|---|
p_account_id | uuid | Yes | The caller. |
p_org_id | uuid | Yes | Workspace of the cluster group. |
p_cluster_group_code | varchar | Yes | Cluster group. |
p_stack_code | varchar | Yes | Stack. |
p_framework_code | varchar | Yes | Engine. |
p_framework_version | varchar | Yes | Engine version. |
p_engine_args_set_code | varchar | Yes | Engine arguments set. Tensor and data parallel sizes come from it. |
p_orchestrator_code | varchar | No | Default bare_metal. |
p_max_gpus_per_node | integer | No | GPUs to use per machine. |
p_prefill_node_count | integer | No | Prefill machines of a prefill and decode stack. |
p_prefill_tensor_parallel_size | integer | No | Tensor parallel size of the prefill engines. |
p_prefill_gpu_count | integer | No | Prefill GPUs on a one-machine prefill and decode group. |
p_model_code | varchar | No | Model. Required when the tensor parallel size is auto. |
Returns one row per machine: server_id, node_index, node_role,
hostname, gpu_vendor, gpu_model, gpu_family_code, gpu_family_name,
gpu_count, gpus_used, tensor_parallel_size, replica_count, is_head,
stack_verdict, and prefill_replica_count.
Permission: cluster_group:read.
Step 5: Save The Project
Send a benchmark workload with tool_code multiroundqa and a
distributed_kv object to save_project_config. A distributed KV workload
runs on its cluster group, so it has no target_selections. The
Project Lifecycle page documents
the RPC, its return row, and its errors.
curl -fsS -X POST "$METRUM_API_URL/rpc/save_project_config" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_org_id\":\"$METRUM_ORG_ID\",
\"p_project_uuid\":null,
\"p_save_as_new\":false,
\"p_expected_revision\":null,
\"p_payload\":{
\"project_name\":\"dkv-native-offload-l40s\",
\"visibility_code\":\"org\",
\"workloads\":[{
\"type\":\"benchmark\",
\"workload_code\":\"dkv-native-offload\",
\"tool_code\":\"multiroundqa\",
\"model_codes\":[\"<model-code>\"],
\"framework_code\":\"vllm\",
\"version\":\"<engine-version>\",
\"engine_args_set_code\":\"<model-server-engine-args-set-code>\",
\"kv_cache_tier_code\":\"gpu\",
\"target_selections\":[],
\"scenarios\":[{
\"scenario_code\":\"c32-r3\",
\"concurrency\":32,
\"job_params\":{\"num_rounds\":\"3\"}
}],
\"distributed_kv\":{
\"cluster_group_code\":\"l40s-pair\",
\"stack_code\":\"vllm-native-offload\",
\"host_tier_gb\":64
}
}]
}
}" | jq
The distributed_kv object
cluster_group_code and stack_code are required. Every other key is
optional. An omitted code takes the catalog default for the stack. A code
resolves inside the workspace of the workload's project.
| Key | Type | Description |
|---|---|---|
cluster_group_code | text | Cluster group. |
stack_code | text | Stack. |
transfer_backend_code | text | Transfer backend. Default: the stack's default backend. |
transport_code | text | Network protocol. Default auto: every machine's InfiniBand or RoCE ports when all machines have them, otherwise the fastest TCP interface they share. The run records the path it chose and why. |
routing_policy_code | text | Router policy. |
orchestrator_code | text | Default bare_metal. |
gpus_per_node | integer | GPUs to use per machine. |
shared_tier_gb | number | Shared store size, for a stack that starts a shared store. |
host_tier_gb | number | Host memory tier per machine, for a stack that offloads to host memory. |
disk_tier_gb, disk_tier_path | number, text | Disk tier size and absolute path. |
prefill_node_count, prefill_tensor_parallel_size, prefill_gpu_count | integer | Prefill sizing of a prefill and decode stack. |
rdma_device_name | text | RDMA ports to use, comma-separated, for example bnxt_re0,bnxt_re1 or mlx5_0:1. Valid for the rdma and auto transports. Leave it out to use every active port on every machine. |
tcp_interface_name | text | Network interface for KV over TCP, for example eno8303. Valid for the tcp and auto transports. Every machine must have it. Leave it out to use the fastest interface. |
decode_prefix_caching | boolean | false turns prompt caching off on decode engines only, so each request moves its whole prompt KV from prefill to decode. Default true. Prefill and decode stacks only. |
node_layout | array | Prefill and decode GPUs per machine, in cluster group order, as [{"prefill_gpus": 2, "decode_gpus": 6}, {"prefill_gpus": 4, "decode_gpus": 4}]. Prefill and decode stacks only. Needs a numeric tensor parallel size for each role, and cannot be combined with gpus_per_node, prefill_node_count or prefill_gpu_count. |
expected_fabric_code | text | Network fabric the run must use. |
declared_pool_state_code, prepare_action_code | text | Cache state at the start of a job, and how the run prepares it. |
load_failure_policy_code | text | What happens when one machine fails to load. |
working_set_ratio | number | Working set as a multiple of the measured GPU KV capacity, from 0.05 to 100. |
repeat_count | integer | Repeats per scenario, from 1 to 20. Fewer than 3 marks results preliminary. |
load_seconds | integer | Hold each job's load for this long, from 120 to 14400 seconds. Without it, the run replays one session per simulated request stream. |
warmup_seconds | integer | Part of the held load the figures leave out, from 0 to 3600 seconds and less than half of load_seconds. |
fault_injection_code, fault_params | text, object | Fault to inject, and its settings as an object of names to numbers. |
workload | object | The workload shape. null returns it to the default shape. See the next table. |
The workload object takes shape_code, arrival_pattern_code,
arrival_rate_qps, burstiness_cv, max_requests, max_duration_s,
overlength_policy_code, prompt_mode_code, client_workers, seed,
trace_format_code, trace_source, time_scale, min_session_turns,
prefix_tokens, prefix_group_count, users_per_prefix, turns_per_user,
turn_input_tokens, output_tokens, context_growth_tokens,
subagent_fanout, subagent_input_tokens, think_time_distribution_code,
think_time_mean_s, shared_system_prompt_tokens,
user_history_prompt_tokens, and rate_steps. rate_steps is an array of
{"rate_qps": <number>, "duration_s": <integer>, "request_count": <integer>}.
Only the agentic data set replay shape reads a scenario's dataset_name. Any
other shape with a dataset_name is refused.
The save refuses settings that cannot run, with the first reason. Examples:
Cluster group ... not found in this organization,
... needs an LMCache server on every node. Choose the CPU RAM or NVMe SSD tier for this workload.,
and ... offloads to host memory on each node. Set the host tier size in GB.
Change the settings of a saved workload
upsert_workload_distributed_kv_config sets or replaces the distributed KV
settings of one saved workload. It takes the keys of the distributed_kv
object as parameters with a p_ prefix, except these four:
distributed_kv key | RPC parameter |
|---|---|
gpus_per_node | p_max_gpus_per_node |
fault_params | p_fault_param_keys and p_fault_param_values (text arrays in matching order) |
workload | Not taken. Use upsert_workload_distributed_kv_workload. |
| (none) | p_account_id (uuid, required) and p_workload_id (integer, required) |
p_cluster_group_code and p_stack_code are required. It returns one row
with workload_id, node_count, and stack_verdict. Read the current
settings with get_workload_distributed_kv_config(p_account_id, p_workload_id), and the shape with get_workload_distributed_kv_workload(p_account_id, p_workload_id). list_project_edit_workloads returns the workload_id.
Permission: cluster_group:write for the upsert, cluster_group:read for the
reads.
A project that has run cannot be changed. A direct call to
upsert_workload_distributed_kv_config, remove_workload_distributed_kv_config,
upsert_workload_distributed_kv_workload, remove_workload_distributed_kv_workload,
create_workload_node_group_selection, or remove_workload_node_group_selection
on a workload of such a project is refused with project_executed. To change
the distributed KV settings of a project that has run, copy it with
copy_project and change the copy.
Step 6: Check Before You Run
preflight_project_run takes the same payload. For a distributed KV workload
it returns no_hardware when the cluster group is not in the workspace,
not_in_plan when the workspace plan does not include distributed KV cache,
and the note cluster_busy when another run uses the group. See
preflight_project_run.
Step 7: Run
Call execute_project_run without a machine. The run goes to the head of the
workload's cluster group.
curl -fsS -X POST "$METRUM_API_URL/rpc/execute_project_run" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_project_name\":\"dkv-native-offload-l40s\",
\"p_project_uuid\":\"<project-uuid>\",
\"p_run_name\":\"dkv-native-offload-run-1\"
}" | jq
How the call behaves:
| Project | Result |
|---|---|
| One distributed KV workload | One run on the cluster group. |
Several workloads, no p_workload_code | One run per workload under one execution_group_id. Single-machine workloads start first on the machine you pass. Distributed KV workloads join the execution's queue, because one cluster group runs one pool at a time. The response has run_ids, queued_workloads, and split_by_workload set to true. |
| A queued workload | The response has queued set to true, queue_position, queue_size, and status waiting. |
| A cluster group that another run uses | The run is created and holds every machine of the group from its first job. While another run still uses a machine of the group, the run waits and its job stays pending. get_run_cluster_queue_state returns waiting with the runs ahead. The run starts when the machines are free. If it cannot start, the job failure shows the database's reason, which names the machine and the run. |
Errors raised for distributed KV, each an HTTP 400 response with code P0001:
message | Cause |
|---|---|
A distributed KV workload runs on its own cluster group. Run it by itself by naming its workload. | p_workload_code matches more than one workload. |
A distributed KV workload runs on its cluster group, coordinated by the group's first machine. Start it without choosing a server. | You passed a machine that is not the head. |
Permission: project:execute.
Step 8: Monitor
get_distributed_kv_run_status
Returns one row per machine of every pool of a run, newest pool first. It never returns the store password or the engine API key.
curl -fsS -X POST "$METRUM_API_URL/rpc/get_distributed_kv_run_status" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_run_id\":<run-id>}" | jq
| Name | Type | Required | Description |
|---|---|---|---|
p_account_id | uuid | Yes | The caller. |
p_run_id | integer | Yes | Run id from execute_project_run. |
Key columns (the RPC returns 50; the OpenAPI document lists them all):
| Column | Type | Description |
|---|---|---|
pool_run_id, epoch | integer | The pool and its attempt number. |
phase | varchar | Pool phase. |
failure_reason | text | Why the pool failed. |
failed_server_id | integer | The machine at fault. |
stack_code, stack_verdict | varchar | Stack and its verdict on this engine version. |
node_count, total_replicas | integer | Machines and engine replicas in the pool. |
server_id, hostname, node_role, is_head | mixed | The machine this row describes. |
member_state | varchar | State of this machine in the pool. |
gpus_used, replica_count | integer | GPU use on this machine. |
engine_version, kv_cache_dtype | varchar | Resolved engine settings. |
gpu_kv_capacity_tokens | bigint | Measured GPU KV capacity. |
heartbeat_age_seconds | integer | Seconds since the machine's last heartbeat. |
member_error | text | The machine's error. |
Errors, each an HTTP 400 response with code P0001: p_account_id is required,
run ... not found, and insufficient_privilege when you cannot access the
project.
Permission: cluster_group:read.
get_run_cluster_queue_state
Returns where the newest run of a workload stands in line for its cluster group.
curl -fsS -X POST "$METRUM_API_URL/rpc/get_run_cluster_queue_state" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_org_id\":\"$METRUM_ORG_ID\",\"p_workload_id\":<workload-id>}" | jq
| Name | Type | Required | Description |
|---|---|---|---|
p_account_id | uuid | Yes | The caller. |
p_org_id | uuid | Yes | Workspace. |
p_workload_id | integer | Yes | Workload. |
p_execution_group_id | uuid | No | Limit to one execution. |
| Column | Type | Description |
|---|---|---|
r_state | varchar | waiting when the run has not started, none when it is not queued. |
r_cluster_group_name | varchar | Cluster group. |
r_holder_run_id | integer | Run that holds the group now. Hidden when you cannot open it. |
r_runs_ahead | integer | Runs that go first, including the holder and the runs that still use a machine of the group. |
r_queue_length | integer | Runs in line. |
r_waiting_since | timestamptz | When this run started waiting. |
Errors: workload ... was not found (P0002, HTTP 500),
Caller may only act as their own account and insufficient_privilege
(42501, HTTP 403).
Permission: project:read.
stop_project_execution_queue
Stops the queue of an execution. Every workload still waiting is marked not run. The workload that is running finishes.
curl -fsS -X POST "$METRUM_API_URL/rpc/stop_project_execution_queue" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_org_id\":\"$METRUM_ORG_ID\",\"p_execution_group_id\":\"<execution-group-id>\"}" | jq
| Name | Type | Required | Description |
|---|---|---|---|
p_account_id | uuid | Yes | The caller. |
p_org_id | uuid | Yes | Workspace. |
p_execution_group_id | uuid | Yes | execution_group_id from execute_project_run. |
Returns one row with execution_group_id, dequeued_workloads, and
running_workloads.
Errors: This execution has no queue of workloads to stop. and
insufficient_privilege. A metrum admin, the workspace owner, a workspace
admin, or the account that ran the project can stop the queue.
Permission: project:execute.
Step 9: Read The Results
get_distributed_kv_run_report
Returns one row per distributed KV job of a run, with the figures the report leads with.
curl -fsS -X POST "$METRUM_API_URL/rpc/get_distributed_kv_run_report" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_run_id\":<run-id>}" | jq
Parameters are the same as get_distributed_kv_run_status.
Key columns (the RPC returns 48; the OpenAPI document lists them all):
| Column | Type | Description |
|---|---|---|
job_id, scenario_name, concurrency, repeat_index | mixed | The job. |
job_status | varchar | Job status. |
share_from_other_machines | numeric | Share of prompt tokens served from another machine (shared pool plus peer). |
share_from_own_gpu | numeric | Share served from the machine's own GPU cache. |
share_computed | numeric | Share computed again. |
attribution_complete | boolean | Whether every token is attributed. A share is null when attribution is incomplete. |
tokens_unattributed | numeric | Tokens left unattributed. |
kv_transfer_gbps | numeric | KV transfer bandwidth. |
kv_usage_peak | numeric | Peak GPU KV usage. |
mean_ttft_s, p50_ttft_s, p90_ttft_s, p99_ttft_s | numeric | Time to first token, in seconds. |
output_throughput_tps, output_tps_per_gpu | numeric | Output tokens per second, and per GPU. |
is_valid | boolean | Whether the job result is valid. |
invalid_reasons | text | Every reason when it is not. |
Errors and permission: the same as get_distributed_kv_run_status.
list_distributed_kv_workload_results
Returns one row per distributed KV job of a run, with its data set profile and cache hit rates.
curl -fsS -X POST "$METRUM_API_URL/rpc/list_distributed_kv_workload_results" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_run_id\":<run-id>}" | jq
Key columns (the RPC returns 31): job_id, shape_code,
requests_scheduled, requests_dropped, requests_trimmed,
oracle_basis_code, oracle_hit_rate, achieved_hit_rate,
achieved_hit_source, achieved_over_oracle, step_count, and
calibration_failed_steps.
Permission: cluster_group:read.