KV Cache Offload (Single Node)
A KV Cache Offload workload measures how a vLLM server performs when its KV cache
lives in GPU memory, in CPU RAM, or on an NVMe disk. In the API it is a workload
with tool code multiroundqa and a KV cache tier. Each run starts vLLM on one
machine and, for the cpu and nvme tiers, an LMCache server next to it. The
benchmark then replays multi-round conversations from a dataset.
The API, the CLI, the MCP server, and the web app call the same RPCs and store the same values. A workload saved through the API runs exactly like one saved in the New Project wizard. The KV Cache Offload methodology explains the tiers and the metrics. For more than one machine, see Distributed KV Cache.
Set these environment variables for the examples:
export METRUM_API_URL="https://<control-plane-host>/api/postgrest"
export METRUM_JWT_TOKEN="<personal-access-token>"
export METRUM_ACCOUNT_ID="<account-uuid>"
export METRUM_ORG_ID="<org-uuid>"
Workflow
| Step | Call | Result |
|---|---|---|
| 1 | list_kv_cache_tiers_for_recipe | The tiers and the settings each one uses. |
| 2 | create_project | An empty project in your workspace. |
| 3 | create_workload with p_kv_cache_tier_code | A workload with its tier and the tier's settings stored. |
| 4 | create_scenario with p_job_params | A scenario with users, token lengths, rounds, and dataset stored. |
| 5 | execute_scenario | A job on your machine. |
| 6 | get_job_status | pending, running, then completed, partial, or failed. |
| 7 | list_multiroundqa_outputs_for_job | The results of the job. |
A project that has run is locked. To change it, copy it first (copy_project) and
change the copy. To run the next tier, create a workload for it in a project that
has not run yet.
Step 1: List the tiers
curl -fsS -X POST "$METRUM_API_URL/rpc/list_kv_cache_tiers_for_recipe" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d '{"p_tool_code":"multiroundqa","p_framework_code":"vllm"}' | jq
| Tier | Where the KV cache lives | vLLM settings | Cache server settings |
|---|---|---|---|
gpu | GPU memory only. No cache server. | kv-cache-offload-gpu | None |
cpu | GPU memory, then an LMCache server's CPU RAM pool. | kv-cache-offload-cpu | lmcache-server-cpu-default |
nvme | GPU memory, a small CPU RAM pool, then a disk folder. | kv-cache-offload-cpu | lmcache-server-nvme-default |
The nvme tier writes to /data/lmcache/l2 by default. The agent creates that
folder with sudo -n when it does not exist, and changes the owner of that new
folder only. It never changes a folder that already exists, never uses sudo inside a
system folder such as /etc, /usr or /var, and refuses a relative l2_path. A folder
the agent's user can already write works wherever it is. When the folder is on the same disk
as the operating system, the setup log says so: the run works, but it does not
measure a separate SSD. Mount an SSD at that path, or set l2_path in your own
cache server settings, to test a real disk tier.
Step 2: Create a project
curl -fsS -X POST "$METRUM_API_URL/rpc/create_project" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_name\":\"kv-cpu-qwen3-8b\",
\"p_visibility_code\":\"private\",\"p_created_by_account_id\":\"$METRUM_ACCOUNT_ID\",
\"p_org_id\":\"$METRUM_ORG_ID\"}" | jq
Step 3: Create the workload
curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_name\":\"kv-cpu-qwen3-8b\",
\"p_workload_name\":\"cpu tier\",\"p_workload_code\":\"cpu-tier\",
\"p_tool_code\":\"multiroundqa\",\"p_model_code\":\"qwen-3-8b\",
\"p_framework_code\":\"vllm\",\"p_version\":\"0.32.0\",
\"p_kv_cache_tier_code\":\"cpu\"}" | jq
| Name | Type | Required | Description |
|---|---|---|---|
p_tool_code | text | Yes | multiroundqa. |
p_model_code, p_framework_code, p_version | text | Yes | Model, vllm, and a vLLM version that has the KV Cache Offload recipe. |
p_kv_cache_tier_code | text | No | gpu, cpu, or nvme. Default gpu. An unknown tier is refused. |
p_engine_args_set_code | text | No | Your own vLLM settings. Default: the tier's vLLM settings. |
p_lmcache_engine_args_set_code | text | No | Your own cache server settings for the cpu or nvme tier. Default: the tier's cache server settings. |
Leave the two settings codes out to get the same values the web app stores.
Step 4: Create a scenario
curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_name\":\"kv-cpu-qwen3-8b\",
\"p_workload_code\":\"cpu-tier\",\"p_scenario_name\":\"c8-r5\",\"p_scenario_code\":\"c8-r5\",
\"p_concurrency\":8,\"p_job_params\":{\"num_rounds\":\"5\"}}" | jq
The response shows the values the run will use:
{"scenario_code": "c8-r5", "tool_code": "multiroundqa", "concurrency": 8,
"input_sequence_length": 8192, "output_sequence_length": 512,
"num_rounds": 5, "dataset_name": "sammshen"}
| Value | Parameter | Default | Rule |
|---|---|---|---|
| Users | p_concurrency | 1 | Conversations that run at the same time. |
| Input tokens | p_input_sequence_length | 8192 | Longest prompt sent. Older turns are dropped to fit. |
| Output tokens | p_output_sequence_length | 512 | Longest answer per request. |
| Rounds | p_job_params.num_rounds | 25 | A whole number, 5 or more. The benchmark drops conversations shorter than 5 rounds, so fewer is refused. |
| Dataset | p_dataset_name or p_job_params.dataset_name | sammshen | A registered dataset, such as sammshen or tau2-bench-v1. |
A workload has one dataset. A second scenario that names a different dataset is
refused with workload <code> already uses dataset <name>. A scenario that names
no dataset uses the workload's dataset. An unknown dataset is refused, and the
message lists the valid ones.
Step 5: Run the scenario
curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_name\":\"kv-cpu-qwen3-8b\",
\"p_workload_code\":\"cpu-tier\",\"p_scenario_code\":\"c8-r5\",
\"p_server_hostname\":\"<machine-hostname>\",\"p_job_name\":\"kv-cpu-c8-r5\",
\"p_gpu_count\":1}" | jq
{"job_id": 760, "run_id": 699, "job_name": "kv-cpu-c8-r5", "tool_code": "multiroundqa",
"tier_code": "cpu", "status": "pending", "cache_server_replica_id": 3405}
p_gpu_count limits the run to that many of the machine's GPUs. Leave it out to
use every GPU. The job stores tier_code and num_rounds, and its command carries
--num-rounds. Run one KV Cache Offload job at a time on a machine: the LMCache
server uses a fixed port.
Step 6: Poll the job
curl -fsS -X POST "$METRUM_API_URL/rpc/get_job_status" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d '{"p_job_id":760}' | jq '.status_code'
| Status | Meaning |
|---|---|
completed | Every conversation finished. |
partial | Some conversations or requests failed. The results cover the rest. |
failed | No request succeeded, or the benchmark saved no results. The job notes say why. |
The first run on a machine downloads the model and the engine image. The agent waits up to 4 hours for the server to answer and stops early if the container exits.
Step 7: Read the results
curl -fsS -X POST "$METRUM_API_URL/rpc/list_multiroundqa_outputs_for_job" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_job_id\":760,\"p_org_id\":\"$METRUM_ORG_ID\"}" | jq '.[0] |
{tier_code, num_users_actual, num_rounds, request_count, failed_request_count,
output_throughput_tps, mean_ttft_s, p95_ttft_s, vllm_local_cache_hit_rate,
cache_hit_rate, is_valid_run}'
{"tier_code": "cpu", "num_users_actual": 8, "num_rounds": 5, "request_count": 40,
"failed_request_count": 0, "output_throughput_tps": 249.77, "mean_ttft_s": 0.413,
"p95_ttft_s": 1.184, "vllm_local_cache_hit_rate": 0.9028, "cache_hit_rate": 0.7658,
"is_valid_run": true}
| Field | Meaning |
|---|---|
output_throughput_tps, input_throughput_tps | Tokens per second generated and sent. |
mean_ttft_s, p50_ttft_s, p95_ttft_s, p99_ttft_s | Time to first token, in seconds. |
mean_tpot_s | Time per output token, in seconds. |
vllm_local_cache_hit_rate | Share of prompt tokens served from GPU memory. |
vllm_external_cache_hit_rate | Share served from the cpu or nvme tier. 0 when everything fit in GPU memory. |
cache_hit_rate, cache_usage_bytes | LMCache lookups that found the data, and bytes it holds (cpu and nvme). |
cache_l2_store_throughput_gbps, ssd_write_gb_total | Disk writes of the nvme tier. |
is_valid_run, invalid_run_reason | Whether the numbers can be compared. |
Only members of the workspace can read the results. An anonymous call returns nothing.
Use your own settings
The tier's settings are defaults. To change a value, save your own set for your
team and name it when you create the workload. A saved set holds exactly the
values you send, so send the default's full list with your changes. Read the
default's values from GET /v_framework_engine_args_sets?set_code=eq.<code>
(args_json).
curl -fsS -X POST "$METRUM_API_URL/rpc/create_custom_engine_args_set" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_org_id\":\"$METRUM_ORG_ID\",\"p_framework_code\":\"lmcache-server\",\"p_version\":\"0.5.5\",
\"p_set_name\":\"CPU pool 20 GB\",\"p_set_code\":\"my-lmcache-cpu\",\"p_tool_code\":\"multiroundqa\",
\"p_arg_keys\":[\"chunk_size\",\"eviction_policy\",\"eviction_ratio\",\"eviction_trigger_watermark\",\"http_port\",
\"l1_init_size_gb\",\"l1_size_gb\",\"l1_use_lazy\",\"prometheus_port\",\"supported_transfer_mode\"],
\"p_arg_values\":[\"256\",\"LRU\",\"0.2\",\"0.8\",\"8099\",\"10\",\"20\",\"true\",\"9094\",\"auto\"]}" | jq
Then pass the codes to create_workload:
| Parameter | Set it replaces |
|---|---|
p_engine_args_set_code | The tier's vLLM settings, for example kv-cache-offload-cpu. |
p_lmcache_engine_args_set_code | The tier's cache server settings, for example lmcache-server-cpu-default. |
| Framework | Values to keep from the default |
|---|---|
vllm (cpu and nvme tiers) | kv_connector LMCacheMPConnector and kv_role kv_both. With them the set gets the kv-cache-offload-mp build profile, which starts vLLM with the cache connector. |
lmcache-server (nvme tier) | l2_path and the other disk keys. With l2_path the set gets the disk build profile. |
create_custom_engine_args_set needs engine_args:write on the team. Saving the
same code again replaces its values. Give a cpu tier a cache server set made for
the cpu tier: a set with the nvme disk keys is not refused yet, and its run fails.
Change a scenario
Change a scenario before its project runs. The same rules apply as when you create one: fewer than 5 rounds is refused and nothing is saved. An empty string removes the value, so the run uses the default of 25.
| RPC | Finds the scenario by | Tool settings parameter |
|---|---|---|
update_scenario | p_scenario_id | p_job_params |
update_scenario_by_code | p_project_id, p_workload_code, p_scenario_code | p_job_params |
update_project_config, save_project_config | The whole project, as the web editor sends it | job_params of each scenario |
curl -fsS -X POST "$METRUM_API_URL/rpc/update_scenario_by_code" \
-H "Authorization: Bearer $METRUM_JWT_TOKEN" -H "Content-Type: application/json" \
-d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_project_id\":<project-id>,\"p_workload_code\":\"cpu-tier\",
\"p_scenario_code\":\"c8-r5\",\"p_job_params\":{\"num_rounds\":\"12\"}}" | jq
<project-id> is the project_id that create_project and get_project_by_name return.
Cancel or delete a run
| RPC | Effect |
|---|---|
cancel_benchmark_job with p_account_id, p_org_id, p_job_id and an optional p_reason | Cancels a job that has not finished. The job must belong to a project you can access, and p_org_id must be that project's organization. |
delete_execution_group | Deletes every job and run of one execution, as the Monitoring page does. |
Errors
| Message | Cause | Fix |
|---|---|---|
num_rounds must be a whole number, 5 or more, for KV cache offload, got 4 | Fewer than 5 rounds, or a value that is not a whole number. | Send 5 or more, or leave it out. |
KV cache tier not found for this tool/framework recipe | Unknown tier code. | Use a code from step 1. |
workload <code> already uses dataset <name> | A second dataset in one workload. | Create another workload for the other dataset. |
dataset_name <name> is not a registered KV cache offload dataset | Unknown dataset. | Use one of the codes the message lists. |
project_executed | The project has run. | Copy the project and change the copy. |
The <dataset> dataset supports at most <n> users at <r> rounds per user, got <c>. Use fewer users or fewer rounds per user. | More users than the dataset gives at that rounds-per-user value, through concurrency or p_job_params.num_users, on a single-machine or a distributed workload. For sammshen the limit is 598 at 25 rounds per user. | Use <n> or fewer users, use fewer rounds per user, or use another dataset. |
Job <id> not found, or you cannot change it. | The job does not exist, or it belongs to a project you cannot access. | Check the job ID and the selected organization. |
Permissions
| RPC | Permission |
|---|---|
create_project, create_workload, create_scenario, update_scenario, update_scenario_by_code | project:write |
execute_scenario | job:execute |
list_multiroundqa_outputs_for_job | job:read |
list_kv_cache_tiers_for_recipe | catalog:read |
MCP tools
The MCP server offers the same flow: list_kv_cache_tiers, create_workload
(kv_cache_tier_code), create_scenario (num_rounds, dataset_name),
execute_scenario (gpu_count), get_job_status, query_kv_cache_offload_results,
cancel_job, and delete_job_execution.