Skip to main content

KV cache offload benchmarking

KV cache offload measures how well a model server holds onto conversation history across multiple back-and-forth turns, by moving KV cache blocks off the GPU when it runs out of room and pulling them back when a conversation continues. It replays real multi-turn conversation datasets rather than single-shot prompts.

Use it to answer:

  • Does offloading KV cache to CPU RAM or NVMe disk help or hurt time-to-first-token on a long multi-turn conversation?
  • How much conversation history can this server retain before it has to evict and recompute?
  • Which storage tier (GPU-only, CPU RAM, or NVMe SSD) gives the best latency/capacity trade-off for my hardware?

The engine underneath CPU and NVMe offload is LMCache, running as a standalone lmcache-server sidecar process alongside vLLM. The GPU tier does not use LMCache at all; it is plain vLLM serving off its own in-VRAM prefix cache.

Full methodology, including the sampling approach behind each metric and the dataset session-count ceiling that limits concurrency: KV Cache Offload Methodology.

To spread the KV cache across two or more machines, with shared pools, split prefill and decode, and RDMA transfer on NVIDIA or AMD GPUs, follow the Distributed KV Cache User Guide. The Distributed KV Cache Methodology explains what it measures.

Before you start​

You need:

  • A GPU machine that is already onboarded to your organization.
  • The model to test. If it is gated or private, also a HuggingFace access token.
  • For the NVMe SSD tier only: an SSD mounted on that machine. See Mounting the SSD and setting l2_path.

The workload card has four stages: Model, Serving engine, Hardware and Load. The steps below follow them in order.

Step 1: Create the project and add the workload​

  1. Go to Projects, then select New Project.
  2. Enter a Project name.
  3. Under Who can see it, choose Team or Private.
  4. Under Add a workload, select KV Cache Offload. The workload card opens.

A new workload already has values filled in: cache storage NVMe SSD, concurrency 1, input length 8192, output length 512 and 25 rounds. Change only what you need. To move to the next stage, select Next.

Step 2: Model stage​

  1. Models (required). Pick a model from the catalog, or paste a HuggingFace ID such as Qwen/Qwen3-235B-A22B-FP8 and press Enter.

  2. Datasets. Pick the conversations the benchmark replays. Leave it blank to use the default set.

    OptionConversations
    sammshen/lmcache-agentic-traces767
    Tau2-Bench (v1)10832
  3. HuggingFace token (only for gated or private models). Select Add a token, paste the token, then select Save. The token is stored encrypted and reused. Without it, the download of a gated model fails when the run starts.

A dataset holds a fixed number of conversations. A concurrency higher than that number adds no load. See Common pitfalls.

Step 3: Serving engine stage: framework and version​

  1. Serving framework. vLLM is already set. It is the only framework for this benchmark.
  2. Version. The newest validated version is filled in. Keep it unless you need an older one.

Step 4: Choose Cache storage​

Under Cache storage, choose one:

OptionWhere the cache livesCache server
GPUIn vLLM's own GPU memory. No offload.No
CPU RAMIn a pinned RAM pool, managed by an lmcache-server sidecar.Yes
NVMe SSDIn a small RAM pool plus a larger pool on the SSD.Yes

Choose the tier before you edit any arguments, because changing the tier resets them to the new tier's defaults. To compare tiers, add one workload per tier to the same project and keep everything else identical. See Comparing Tiers Correctly.

Step 5: Cache server build and version (CPU RAM and NVMe SSD only)​

  1. Cache server build. Keep NVIDIA (vllm/vllm-openai). Choose AMD ROCm (rocm/vllm-dev) only if your machine has AMD GPUs.
  2. Cache server version. Keep the version that is filled in. You need one selected to save LMCache Server Args.

For the GPU tier, neither field appears. A Build variant field under Advanced settings has its default already set.

Step 6: Arguments under Advanced settings​

Open Advanced settings. It holds the vLLM Args editor and, for CPU RAM and NVMe SSD, the LMCache Server Args editor. The tier fills in working defaults, so for a first run you can leave them as they are.

To change a value:

  1. Type the new value in its row. To add an argument that is not listed, select Add custom arg.
  2. Select Save arg set. An Arg set name box appears first, so edit the name if you like.
  3. A message confirms: Arg set "<name>" saved and selected. The set is saved for your whole team and used by this workload.

An edit that you do not save is ignored. The workload keeps using the selected arg set. Select Reset arg set to drop your edits.

vLLM Args (all tiers)​

Check vLLM recipes for your model first, because a model-specific recipe can call for different values.

These are the vLLM engine args editable for every tier:

ArgumentWhat it controlsDefault
gpu_memory_utilizationShare of GPU memory vLLM may claim.0.80
max_model_lenContext length the server accepts. auto infers it from the model config.auto
tensor_parallel_sizeHow many GPUs the model is split across. Recommended: set it equal to the number of GPUs on the server.1
reasoning_parserSeparates chain-of-thought from the final answer. Set it if the model supports reasoning. Leave blank for non-reasoning models.blank
trust_remote_codeAllows the model's own custom code (needed by some HuggingFace model repos).true

OpenAI's gpt-oss models use the Harmony chat format, so the benchmark sends tool definitions with every request for them, and vLLM needs a tool-call parser to accept these. Set these two args if your model supports tool calling, using the values below for gpt-oss.

ArgumentValue
enable_auto_tool_choicetrue
tool_call_parseropenai

For CPU RAM and NVMe SSD, the tier adds kv_connector (LMCacheMPConnector) and kv_role (kv_both). Leave both as they are.

LMCache Server Args (CPU RAM and NVMe SSD only)​

These configure the lmcache-server sidecar, separate from vLLM. For a hybrid (Mamba or GDN style) model, check the LMCache hybrid model recipes first.

CPU RAM tier defaults:

ArgumentWhat it controlsDefault
l1_size_gbSize of the pinned RAM pool holding offloaded cache.auto (sized to the machine's RAM)
l1_init_size_gbHow much of the pool to allocate upfront (the rest grows lazily).20
l1_use_lazyGrow the RAM pool gradually instead of allocating it all at once.true
eviction_policyHow cache is evicted when the pool fills.LRU
eviction_ratio / eviction_trigger_watermarkHow much to evict, and at what fill level eviction kicks in.0.2 / 0.8
chunk_sizeGranularity, in tokens, that cache is stored and evicted in.256

NVMe SSD tier defaults (a small CPU staging pool plus a larger disk-backed pool):

ArgumentWhat it controlsDefault
l1_size_gbSize of the small CPU staging pool in front of the disk. Deliberately much smaller than the CPU-tier's, since disk provides the real capacity.32
l1_init_size_gbUpfront allocation for that staging pool.8
l2_pathThe disk folder the overflow pool writes to. Mount your SSD first, then set this to a folder on it. The agent creates the folder, unless the machine needs a password for sudo. See Mounting the SSD and setting l2_path./data/lmcache/l2
max_capacity_gbCap on the disk pool's size. 0 means unlimited (bounded only by the disk itself).0
num_workersParallel I/O workers writing to and reading from disk.4
use_odirectBypass the OS page cache for disk I/O (O_DIRECT). Off by default; only enable it if your storage and filesystem are confirmed to support it, since raw device or unusual filesystem setups can behave unpredictably with it on.false
l2_store_policy / l2_prefetch_policyHow data is written to and read back from the disk pool.default / default
l2_prefetch_max_in_flightHow many prefetch reads from disk can be outstanding at once.8

Every value in both tables is editable. These are starting points, not fixed limits. Change one at a time and re-run a small smoke scenario before committing to a full matrix.

Mounting the SSD and setting l2_path​

Mount your SSD on the GPU machine first, then point l2_path at a folder on it. These steps mount the SSD at /data and use the catalog default, /data/lmcache/l2. If you mount your SSD somewhere else, use that path in place of /data, and set l2_path to a folder under it. Run the commands on the GPU machine over SSH.

Is the SSD mounted?​

lsblk

No sudo needed, this only reads device info. Look for your SSD in the list.

  • Already has a value under MOUNTPOINT? It's mounted. Set l2_path to a folder on that mount.
  • Nothing under MOUNTPOINT? It's not mounted yet, and any path you set will land on the regular boot disk instead of the SSD. Mount it first, using the partition name lsblk just showed you (for example sdb1 or nvme1n1):
sudo mkdir -p /data
sudo mount /dev/nvme1n1 /data

(Replace /dev/nvme1n1 with your own partition name.) This mount does not survive a reboot; add it to /etc/fstab if you need it to.

The folder and its permissions​

After you mount the SSD, there is usually nothing more to do. Before the cache server starts, the agent creates the l2_path folder when it does not exist and gives its own user ownership of that new folder. It never changes the owner of a folder that already exists, never uses sudo inside a system folder such as /etc, /usr or /var, and refuses a relative l2_path. A folder the agent's user can already write works wherever it is. If the folder already exists and the agent's user cannot write to it, give that user ownership yourself.

The agent needs sudo for this only when its own user cannot write there, for example under a mount owned by root. It uses sudo -n, which never asks for a password. Check that it will work by running this on the GPU machine, as the user that runs the agent:

sudo -n true && echo "passwordless sudo works"
  • It prints passwordless sudo works. You do nothing. The agent creates the folder.

  • It prints a password is required. The agent cannot create the folder, and the run stops before anything starts. Do this once after you mount the SSD, using your own l2_path in place of the example. First find the agent's user:

    ps -eo user,args | grep '[i]nsights-cli agent'

    The first column of the line it prints is the user. Then create the folder and hand it over:

    sudo mkdir -p /data/lmcache/l2
    sudo chown <agent-user>:<agent-group> /data/lmcache/l2

    The error message in the run shows these same commands, already filled in with your path and user. If the agent runs as root, skip this.

The agent cannot tell whether a path is on your SSD. If the SSD is not mounted, the folder is created on the boot disk, the run still starts, and the agent writes a warning to its log. Mount the SSD before you run.

Step 7: Hardware stage​

  1. Deployment. Keep One machine. Several machines is Distributed KV, which has its own guide.
  2. Machines (required). Choose the onboarded machine. For the NVMe SSD tier, choose one with the SSD mounted.
  3. GPUs to use. Leave it blank to use all GPUs, or enter a number. It must match tensor_parallel_size in the vLLM Args.

Step 8: Load stage​

Each value you enter becomes its own scenario. Type a value and press Enter after each one.

FieldWhat it setsStarting value
ConcurrencyConversations running at the same time.1
Input Sequence LengthTokens in each turn's input. Type 8k for 8000.8192
Output Sequence LengthTokens generated per turn.512
Rounds per ConversationTurns replayed per conversation. Use 5 or more.25

Rounds below 5 are refused. For most datasets the benchmark drops any chunk shorter than 5 rounds, so a lower value runs with no users, and one rule applies to every dataset. A value near 5 stresses first-turn latency. A higher value stresses how well the cache keeps earlier turns.

Select Review scenarios to see every scenario the workload will run.

Step 9: Save and run​

  1. Select Save to run later, or Save & Run to start now. Save & Run opens Monitoring, where each scenario runs as one job. See What Save & Run does.
  2. If a required field is missing, the save stops and lists the workload. Select Finish to complete it.

Worked examples: one model, three tiers​

Build these three workloads in one project and compare them. They ran on one machine with 4 NVIDIA H200 GPUs. Everything is the same except the storage tier and its cache server settings. Leave every field that is not listed at its default.

Model stage (all three workloads)​

FieldValue
ModelsQwen/Qwen3-235B-A22B-FP8
Datasetssammshen/lmcache-agentic-traces

Serving engine stage​

FieldWorkload 1Workload 2Workload 3
Cache storageGPUCPU RAMNVMe SSD

In Advanced settings, change these and select Save arg set for each editor:

EditorArgumentWorkload 1Workload 2Workload 3
vLLM Argstensor_parallel_size444
LMCache Server Argsl1_size_gbNone10241024
LMCache Server Argsl1_init_size_gbNone2020
LMCache Server Argsl2_pathNoneNone/mnt/kvcache/l2
LMCache Server Argsmax_capacity_gbNoneNone13000
LMCache Server Argsuse_odirectNoneNonetrue

Workload 1 has no LMCache Server Args. For Workload 2, l1_init_size_gb is already 20. Choosing CPU RAM or NVMe SSD adds kv_connector and kv_role for you.

Hardware stage (all three workloads)​

FieldValue
MachinesYour machine with 4 H200 GPUs
GPUs to use4

Load stage (all three workloads)​

FieldValue
Concurrency16, 32, 64, 128, 256, 512
Input Sequence Length39680
Output Sequence Length512
Rounds per Conversation25

That makes six scenarios per workload. An input of 39680 plus an output of 512 is 40192 tokens, which fits this model's 40960-token window. For a model with a smaller window, lower the input length.

Before you select Save & Run​

  • Choose l1_size_gb well below your machine's total RAM. The 1024 here fits this machine only.
  • For Workload 3, mount the SSD and set l2_path to a folder on it. See Mounting the SSD and setting l2_path.
  • Keep max_capacity_gb below the free space on the SSD. 0 means no limit.
  • Turn use_odirect on only if your disk and filesystem support it.
  • Use LMCache Server Args saved for the same tier. A CPU RAM workload cannot use arguments saved for NVMe SSD.

Metrics collected​

Open the Reporting page and filter to your run. KV Cache Offload charts plot each metric against concurrency, with one line per cache storage tier. Not every metric exists for every tier, because only CPU RAM and NVMe SSD run a cache server, and only NVMe SSD writes to a disk.

Charted on the Reporting page:

MetricWhat it tells youGPUCPU RAMNVMe SSD
Output token throughputTokens generated per second, across all users.YesYesYes
Decode throughputToken generation speed while a request is generating.YesYesYes
Input (prefill) throughputPrompt tokens processed per second.YesYesYes
Request throughputCompleted requests per second.YesYesYes
TTFT (mean, P50, P90, P95, P99)Time to first token: the average and the slower tail.YesYesYes
TPOT (mean)Time per output token after the first.YesYesYes
Change in TTFT, first to last roundHow much time to first token changes from a conversation's first round to its last. Median and P95.YesYesYes
vLLM prefill timeMean time vLLM spends on the prompt. Shown only when your vLLM build reports it.YesYesYes
GPU KV cache usageHow full vLLM's own GPU cache gets.YesYesYes
vLLM local (in-GPU) cache hit rateShare of lookups served from vLLM's own GPU cache.YesYesYes
vLLM external (LMCache) cache hit rateShare of lookups served from the offloaded cache. It is zero for GPU, which has no offload.ZeroYesYes
LMCache server hit rateShare of lookups served by the cache server.NoYesYes
LMCache GPU to CPU load and store throughputData moved between GPU memory and the RAM pool, in GB per second.NoYesYes
LMCache L2 load and store throughputData moved between the RAM pool and the disk pool, in GB per second.NoNoYes
GPU power, GPU utilization, CPU utilizationHardware telemetry during the run.YesYesYes
SSD read and write throughput, IOPS, read and write latency, utilization, queue depth, data transfer summaryHow hard the disk works.NoNoYes

Stored with every job, but not charted. Read them with the CLI command analytics kv-cache-results, or through the API function list_multiroundqa_outputs_for_job.

MetricWhat it means
Session countsTotal, complete, incomplete and error sessions, and the number of failed requests.
Run validityWhether the run is valid. When it is not, the reason is stored with it. Do not use the numbers of an invalid run.
Cache evictions, cache usage (bytes)Entries the cache server evicted, and how much it holds at the end (CPU RAM and NVMe SSD).
L1 usage and eviction activityHow full the RAM pool is, and whether it was evicting (CPU RAM and NVMe SSD).
Chunks loaded, lookup hit tokens, lookup requested tokensHow much cache content was found and reused versus requested (CPU RAM and NVMe SSD).
Shared-prefix and per-user history reuse opportunity (%)An estimate of how much traffic could reuse a cached prefix, not a measured hit rate. The shared-prefix estimate is not available for every dataset.

Common pitfalls​

  • Comparing tiers without holding everything else fixed. Change only the tier. Keep the model, dataset, concurrency, input length, output length and rounds identical.
  • Asking for more concurrency than the dataset can supply. A dataset holds a fixed number of conversations, and each one is cut into chunks of Rounds per Conversation turns. More rounds means fewer chunks. A concurrency above that number adds no load, and the run replays fewer conversations than you asked for. Compare the session counts in the results, not the requested concurrency.
  • Rounds per Conversation below 5. It is refused. For most datasets the benchmark drops chunks shorter than 5 rounds, so the run would have no users, and one rule applies to every dataset. Even at 5 or more, a short conversation may never fill GPU memory, so offloading shows no effect. Use enough rounds that the conversation outgrows the GPU.
  • Editing an argument and not saving it. Only a saved arg set is used. Select Save arg set after every change.
  • Changing the tier after editing arguments. The arguments reset to the new tier's defaults.
  • Running the NVMe SSD tier without a mounted SSD. The folder is created on the boot disk and the run still starts, so you measure the boot disk. Mount the SSD first.
  • Using LMCache Server Args saved for another tier. An NVMe SSD set includes disk settings that a CPU RAM workload cannot use. Use a set saved for the same tier.
  • Editing kv_connector. It is wired to LMCache. Changing it without changing the rest of the setup stops the server from launching.
  • Choosing the AMD build on NVIDIA GPUs, or the reverse. The build must match the GPU vendor of the machine, or the wrong container image starts.