Skip to main content

KV Cache Offload (Single Node)

A KV Cache Offload workload measures how a vLLM server performs when its KV cache lives in GPU memory, in CPU RAM, or on an NVMe disk. In the API it is a workload with tool code multiroundqa and a KV cache tier. Each run starts vLLM on one machine and, for the cpu and nvme tiers, an LMCache server next to it. The benchmark then replays multi-round conversations from a dataset.

The API, the CLI, the MCP server, and the web app call the same RPCs and store the same values. A workload saved through the API runs exactly like one saved in the New Project wizard. The KV Cache Offload methodology explains the tiers and the metrics. For more than one machine, see Distributed KV Cache.

Set these environment variables for the examples:

export METRUM_API_URL="https://<control-plane-host>/api/postgrest"
export METRUM_JWT_TOKEN="<personal-access-token>"
export METRUM_ACCOUNT_ID="<account-uuid>"
export METRUM_ORG_ID="<org-uuid>"

Workflow​

StepCallResult
1list_kv_cache_tiers_for_recipeThe tiers and the settings each one uses.
2create_projectAn empty project in your workspace.
3create_workload with p_kv_cache_tier_codeA workload with its tier and the tier's settings stored.
4create_scenario with p_job_paramsA scenario with users, token lengths, rounds, and dataset stored.
5execute_scenarioA job on your machine.
6get_job_statuspending, running, then completed, partial, or failed.
7list_multiroundqa_outputs_for_jobThe results of the job.

A project that has run is locked. To change it, copy it first (copy_project) and change the copy. To run the next tier, create a workload for it in a project that has not run yet.

Step 1: List the tiers​

curl -fsS -X POST "$METRUM_API_URL/rpc/list_kv_cache_tiers_for_recipe" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d '{"p_tool_code":"multiroundqa","p_framework_code":"vllm"}' | jq
TierWhere the KV cache livesvLLM settingsCache server settings
gpuGPU memory only. No cache server.kv-cache-offload-gpuNone
cpuGPU memory, then an LMCache server's CPU RAM pool.kv-cache-offload-cpulmcache-server-cpu-default
nvmeGPU memory, a small CPU RAM pool, then a disk folder.kv-cache-offload-cpulmcache-server-nvme-default

The nvme tier writes to /data/lmcache/l2 by default. The agent creates that folder with sudo -n when it does not exist, and changes the owner of that new folder only. It never changes a folder that already exists, never uses sudo inside a system folder such as /etc, /usr or /var, and refuses a relative l2_path. A folder the agent's user can already write works wherever it is. When the folder is on the same disk as the operating system, the setup log says so: the run works, but it does not measure a separate SSD. Mount an SSD at that path, or set l2_path in your own cache server settings, to test a real disk tier.

Step 2: Create a project​

curl -fsS -X POST "$METRUM_API_URL/rpc/create_project" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_name\":\"kv-cpu-qwen3-8b\",
\"p_visibility_code\":\"private\",\"p_created_by_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_org_id\":\"$METRUM_ORG_ID\"}" | jq

Step 3: Create the workload​

curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_name\":\"kv-cpu-qwen3-8b\",
\"p_workload_name\":\"cpu tier\",\"p_workload_code\":\"cpu-tier\",
\"p_tool_code\":\"multiroundqa\",\"p_model_code\":\"qwen-3-8b\",
\"p_framework_code\":\"vllm\",\"p_version\":\"0.32.0\",
\"p_kv_cache_tier_code\":\"cpu\"}" | jq
NameTypeRequiredDescription
p_tool_codetextYesmultiroundqa.
p_model_code, p_framework_code, p_versiontextYesModel, vllm, and a vLLM version that has the KV Cache Offload recipe.
p_kv_cache_tier_codetextNogpu, cpu, or nvme. Default gpu. An unknown tier is refused.
p_engine_args_set_codetextNoYour own vLLM settings. Default: the tier's vLLM settings.
p_lmcache_engine_args_set_codetextNoYour own cache server settings for the cpu or nvme tier. Default: the tier's cache server settings.

Leave the two settings codes out to get the same values the web app stores.

Step 4: Create a scenario​

curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_name\":\"kv-cpu-qwen3-8b\",
\"p_workload_code\":\"cpu-tier\",\"p_scenario_name\":\"c8-r5\",\"p_scenario_code\":\"c8-r5\",
\"p_concurrency\":8,\"p_job_params\":{\"num_rounds\":\"5\"}}" | jq

The response shows the values the run will use:

{"scenario_code": "c8-r5", "tool_code": "multiroundqa", "concurrency": 8,
"input_sequence_length": 8192, "output_sequence_length": 512,
"num_rounds": 5, "dataset_name": "sammshen"}
ValueParameterDefaultRule
Usersp_concurrency1Conversations that run at the same time.
Input tokensp_input_sequence_length8192Longest prompt sent. Older turns are dropped to fit.
Output tokensp_output_sequence_length512Longest answer per request.
Roundsp_job_params.num_rounds25A whole number, 5 or more. The benchmark drops conversations shorter than 5 rounds, so fewer is refused.
Datasetp_dataset_name or p_job_params.dataset_namesammshenA registered dataset, such as sammshen or tau2-bench-v1.

A workload has one dataset. A second scenario that names a different dataset is refused with workload <code> already uses dataset <name>. A scenario that names no dataset uses the workload's dataset. An unknown dataset is refused, and the message lists the valid ones.

Step 5: Run the scenario​

curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_name\":\"kv-cpu-qwen3-8b\",
\"p_workload_code\":\"cpu-tier\",\"p_scenario_code\":\"c8-r5\",
\"p_server_hostname\":\"<machine-hostname>\",\"p_job_name\":\"kv-cpu-c8-r5\",
\"p_gpu_count\":1}" | jq
{"job_id": 760, "run_id": 699, "job_name": "kv-cpu-c8-r5", "tool_code": "multiroundqa",
"tier_code": "cpu", "status": "pending", "cache_server_replica_id": 3405}

p_gpu_count limits the run to that many of the machine's GPUs. Leave it out to use every GPU. The job stores tier_code and num_rounds, and its command carries --num-rounds. Run one KV Cache Offload job at a time on a machine: the LMCache server uses a fixed port.

Step 6: Poll the job​

curl -fsS -X POST "$METRUM_API_URL/rpc/get_job_status" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d '{"p_job_id":760}' | jq '.status_code'
StatusMeaning
completedEvery conversation finished.
partialSome conversations or requests failed. The results cover the rest.
failedNo request succeeded, or the benchmark saved no results. The job notes say why.

The first run on a machine downloads the model and the engine image. The agent waits up to 4 hours for the server to answer and stops early if the container exits.

Step 7: Read the results​

curl -fsS -X POST "$METRUM_API_URL/rpc/list_multiroundqa_outputs_for_job" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_job_id\":760,\"p_org_id\":\"$METRUM_ORG_ID\"}" | jq '.[0] |
{tier_code, num_users_actual, num_rounds, request_count, failed_request_count,
output_throughput_tps, mean_ttft_s, p95_ttft_s, vllm_local_cache_hit_rate,
cache_hit_rate, is_valid_run}'
{"tier_code": "cpu", "num_users_actual": 8, "num_rounds": 5, "request_count": 40,
"failed_request_count": 0, "output_throughput_tps": 249.77, "mean_ttft_s": 0.413,
"p95_ttft_s": 1.184, "vllm_local_cache_hit_rate": 0.9028, "cache_hit_rate": 0.7658,
"is_valid_run": true}
FieldMeaning
output_throughput_tps, input_throughput_tpsTokens per second generated and sent.
mean_ttft_s, p50_ttft_s, p95_ttft_s, p99_ttft_sTime to first token, in seconds.
mean_tpot_sTime per output token, in seconds.
vllm_local_cache_hit_rateShare of prompt tokens served from GPU memory.
vllm_external_cache_hit_rateShare served from the cpu or nvme tier. 0 when everything fit in GPU memory.
cache_hit_rate, cache_usage_bytesLMCache lookups that found the data, and bytes it holds (cpu and nvme).
cache_l2_store_throughput_gbps, ssd_write_gb_totalDisk writes of the nvme tier.
is_valid_run, invalid_run_reasonWhether the numbers can be compared.

Only members of the workspace can read the results. An anonymous call returns nothing.

Use your own settings​

The tier's settings are defaults. To change a value, save your own set for your team and name it when you create the workload. A saved set holds exactly the values you send, so send the default's full list with your changes. Read the default's values from GET /v_framework_engine_args_sets?set_code=eq.<code> (args_json).

curl -fsS -X POST "$METRUM_API_URL/rpc/create_custom_engine_args_set" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_org_id\":\"$METRUM_ORG_ID\",\"p_framework_code\":\"lmcache-server\",\"p_version\":\"0.5.5\",
\"p_set_name\":\"CPU pool 20 GB\",\"p_set_code\":\"my-lmcache-cpu\",\"p_tool_code\":\"multiroundqa\",
\"p_arg_keys\":[\"chunk_size\",\"eviction_policy\",\"eviction_ratio\",\"eviction_trigger_watermark\",\"http_port\",
\"l1_init_size_gb\",\"l1_size_gb\",\"l1_use_lazy\",\"prometheus_port\",\"supported_transfer_mode\"],
\"p_arg_values\":[\"256\",\"LRU\",\"0.2\",\"0.8\",\"8099\",\"10\",\"20\",\"true\",\"9094\",\"auto\"]}" | jq

Then pass the codes to create_workload:

ParameterSet it replaces
p_engine_args_set_codeThe tier's vLLM settings, for example kv-cache-offload-cpu.
p_lmcache_engine_args_set_codeThe tier's cache server settings, for example lmcache-server-cpu-default.
FrameworkValues to keep from the default
vllm (cpu and nvme tiers)kv_connector LMCacheMPConnector and kv_role kv_both. With them the set gets the kv-cache-offload-mp build profile, which starts vLLM with the cache connector.
lmcache-server (nvme tier)l2_path and the other disk keys. With l2_path the set gets the disk build profile.

create_custom_engine_args_set needs engine_args:write on the team. Saving the same code again replaces its values. Give a cpu tier a cache server set made for the cpu tier: a set with the nvme disk keys is not refused yet, and its run fails.

Change a scenario​

Change a scenario before its project runs. The same rules apply as when you create one: fewer than 5 rounds is refused and nothing is saved. An empty string removes the value, so the run uses the default of 25.

RPCFinds the scenario byTool settings parameter
update_scenariop_scenario_idp_job_params
update_scenario_by_codep_project_id, p_workload_code, p_scenario_codep_job_params
update_project_config, save_project_configThe whole project, as the web editor sends itjob_params of each scenario
curl -fsS -X POST "$METRUM_API_URL/rpc/update_scenario_by_code" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_id\":<project-id>,\"p_workload_code\":\"cpu-tier\",
\"p_scenario_code\":\"c8-r5\",\"p_job_params\":{\"num_rounds\":\"12\"}}" | jq

<project-id> is the project_id that create_project and get_project_by_name return.

Cancel or delete a run​

RPCEffect
cancel_benchmark_job with p_account_id, p_org_id, p_job_id and an optional p_reasonCancels a job that has not finished. The job must belong to a project you can access, and p_org_id must be that project's organization.
delete_execution_groupDeletes every job and run of one execution, as the Monitoring page does.

Errors​

MessageCauseFix
num_rounds must be a whole number, 5 or more, for KV cache offload, got 4Fewer than 5 rounds, or a value that is not a whole number.Send 5 or more, or leave it out.
KV cache tier not found for this tool/framework recipeUnknown tier code.Use a code from step 1.
workload <code> already uses dataset <name>A second dataset in one workload.Create another workload for the other dataset.
dataset_name <name> is not a registered KV cache offload datasetUnknown dataset.Use one of the codes the message lists.
project_executedThe project has run.Copy the project and change the copy.
The <dataset> dataset supports at most <n> users at <r> rounds per user, got <c>. Use fewer users or fewer rounds per user.More users than the dataset gives at that rounds-per-user value, through concurrency or p_job_params.num_users, on a single-machine or a distributed workload. For sammshen the limit is 598 at 25 rounds per user.Use <n> or fewer users, use fewer rounds per user, or use another dataset.
Job <id> not found, or you cannot change it.The job does not exist, or it belongs to a project you cannot access.Check the job ID and the selected organization.

Permissions​

RPCPermission
create_project, create_workload, create_scenario, update_scenario, update_scenario_by_codeproject:write
execute_scenariojob:execute
list_multiroundqa_outputs_for_jobjob:read
list_kv_cache_tiers_for_recipecatalog:read

MCP tools​

The MCP server offers the same flow: list_kv_cache_tiers, create_workload (kv_cache_tier_code), create_scenario (num_rounds, dataset_name), execute_scenario (gpu_count), get_job_status, query_kv_cache_offload_results, cancel_job, and delete_job_execution.