KV cache offload benchmarking
KV cache offload measures how well a model server holds onto conversation history across multiple back-and-forth turns, by moving KV cache blocks off the GPU when it runs out of room and pulling them back when a conversation continues. It replays real multi-turn conversation datasets rather than single-shot prompts.
Use it to answer:
- Does offloading KV cache to CPU RAM or NVMe disk help or hurt time-to-first-token on a long multi-turn conversation?
- How much conversation history can this server retain before it has to evict and recompute?
- Which storage tier (GPU-only, CPU RAM, or NVMe SSD) gives the best latency/capacity trade-off for my hardware?
The engine underneath CPU and NVMe offload is LMCache, running as a standalone lmcache-server sidecar process alongside vLLM. The GPU tier does not use LMCache at all; it is plain vLLM serving off its own in-VRAM prefix cache.
Full methodology, including the sampling approach behind each metric and the dataset session-count ceiling that limits concurrency: KV Cache Offload Methodology.
To spread the KV cache across two or more machines, with shared pools, split prefill and decode, and RDMA transfer on NVIDIA or AMD GPUs, follow the Distributed KV Cache User Guide. The Distributed KV Cache Methodology explains what it measures.
Before you start
You need:
- A GPU machine that is already onboarded to your organization.
- The model to test. If it is gated or private, also a HuggingFace access token.
- For the NVMe SSD tier only: an SSD mounted on that machine. See Mounting the SSD and setting
l2_path.
The workload card has four stages: Model, Serving engine, Hardware and Load. The steps below follow them in order.
Step 1: Create the project and add the workload
- Go to Projects, then select New Project.
- Enter a Project name.
- Under Who can see it, choose Team or Private.
- Under Add a workload, select KV Cache Offload. The workload card opens.
A new workload already has values filled in: cache storage NVMe SSD, concurrency 1, input length 8192, output length 512 and 25 rounds. Change only what you need. To move to the next stage, select Next.
Step 2: Model stage
-
Models (required). Pick a model from the catalog, or paste a HuggingFace ID such as
Qwen/Qwen3-235B-A22B-FP8and press Enter. -
Datasets. Pick the conversations the benchmark replays. Leave it blank to use the default set.
Option Conversations sammshen/lmcache-agentic-traces767 Tau2-Bench (v1) 10832 -
HuggingFace token (only for gated or private models). Select Add a token, paste the token, then select Save. The token is stored encrypted and reused. Without it, the download of a gated model fails when the run starts.
A dataset holds a fixed number of conversations. A concurrency higher than that number adds no load. See Common pitfalls.
Step 3: Serving engine stage: framework and version
- Serving framework. vLLM is already set. It is the only framework for this benchmark.
- Version. The newest validated version is filled in. Keep it unless you need an older one.
Step 4: Choose Cache storage
Under Cache storage, choose one:
| Option | Where the cache lives | Cache server |
|---|---|---|
| GPU | In vLLM's own GPU memory. No offload. | No |
| CPU RAM | In a pinned RAM pool, managed by an lmcache-server sidecar. | Yes |
| NVMe SSD | In a small RAM pool plus a larger pool on the SSD. | Yes |
Choose the tier before you edit any arguments, because changing the tier resets them to the new tier's defaults. To compare tiers, add one workload per tier to the same project and keep everything else identical. See Comparing Tiers Correctly.
Step 5: Cache server build and version (CPU RAM and NVMe SSD only)
- Cache server build. Keep NVIDIA (vllm/vllm-openai). Choose AMD ROCm (rocm/vllm-dev) only if your machine has AMD GPUs.
- Cache server version. Keep the version that is filled in. You need one selected to save LMCache Server Args.
For the GPU tier, neither field appears. A Build variant field under Advanced settings has its default already set.
Step 6: Arguments under Advanced settings
Open Advanced settings. It holds the vLLM Args editor and, for CPU RAM and NVMe SSD, the LMCache Server Args editor. The tier fills in working defaults, so for a first run you can leave them as they are.
To change a value:
- Type the new value in its row. To add an argument that is not listed, select Add custom arg.
- Select Save arg set. An Arg set name box appears first, so edit the name if you like.
- A message confirms:
Arg set "<name>" saved and selected.The set is saved for your whole team and used by this workload.
An edit that you do not save is ignored. The workload keeps using the selected arg set. Select Reset arg set to drop your edits.
vLLM Args (all tiers)
Check vLLM recipes for your model first, because a model-specific recipe can call for different values.
These are the vLLM engine args editable for every tier:
| Argument | What it controls | Default |
|---|---|---|
gpu_memory_utilization | Share of GPU memory vLLM may claim. | 0.80 |
max_model_len | Context length the server accepts. auto infers it from the model config. | auto |
tensor_parallel_size | How many GPUs the model is split across. Recommended: set it equal to the number of GPUs on the server. | 1 |
reasoning_parser | Separates chain-of-thought from the final answer. Set it if the model supports reasoning. Leave blank for non-reasoning models. | blank |
trust_remote_code | Allows the model's own custom code (needed by some HuggingFace model repos). | true |
OpenAI's gpt-oss models use the Harmony chat format, so the benchmark sends tool definitions with every request for them, and vLLM needs a tool-call parser to accept these. Set these two args if your model supports tool calling, using the values below for gpt-oss.
| Argument | Value |
|---|---|
enable_auto_tool_choice | true |
tool_call_parser | openai |
For CPU RAM and NVMe SSD, the tier adds kv_connector (LMCacheMPConnector) and kv_role (kv_both). Leave both as they are.
LMCache Server Args (CPU RAM and NVMe SSD only)
These configure the lmcache-server sidecar, separate from vLLM. For a hybrid (Mamba or GDN style) model, check the LMCache hybrid model recipes first.
CPU RAM tier defaults:
| Argument | What it controls | Default |
|---|---|---|
l1_size_gb | Size of the pinned RAM pool holding offloaded cache. | auto (sized to the machine's RAM) |
l1_init_size_gb | How much of the pool to allocate upfront (the rest grows lazily). | 20 |
l1_use_lazy | Grow the RAM pool gradually instead of allocating it all at once. | true |
eviction_policy | How cache is evicted when the pool fills. | LRU |
eviction_ratio / eviction_trigger_watermark | How much to evict, and at what fill level eviction kicks in. | 0.2 / 0.8 |
chunk_size | Granularity, in tokens, that cache is stored and evicted in. | 256 |
NVMe SSD tier defaults (a small CPU staging pool plus a larger disk-backed pool):
| Argument | What it controls | Default |
|---|---|---|
l1_size_gb | Size of the small CPU staging pool in front of the disk. Deliberately much smaller than the CPU-tier's, since disk provides the real capacity. | 32 |
l1_init_size_gb | Upfront allocation for that staging pool. | 8 |
l2_path | The disk folder the overflow pool writes to. Mount your SSD first, then set this to a folder on it. The agent creates the folder, unless the machine needs a password for sudo. See Mounting the SSD and setting l2_path. | /data/lmcache/l2 |
max_capacity_gb | Cap on the disk pool's size. 0 means unlimited (bounded only by the disk itself). | 0 |
num_workers | Parallel I/O workers writing to and reading from disk. | 4 |
use_odirect | Bypass the OS page cache for disk I/O (O_DIRECT). Off by default; only enable it if your storage and filesystem are confirmed to support it, since raw device or unusual filesystem setups can behave unpredictably with it on. | false |
l2_store_policy / l2_prefetch_policy | How data is written to and read back from the disk pool. | default / default |
l2_prefetch_max_in_flight | How many prefetch reads from disk can be outstanding at once. | 8 |
Every value in both tables is editable. These are starting points, not fixed limits. Change one at a time and re-run a small smoke scenario before committing to a full matrix.
Mounting the SSD and setting l2_path
Mount your SSD on the GPU machine first, then point l2_path at a folder on it. These steps mount the SSD at /data and use the catalog default, /data/lmcache/l2. If you mount your SSD somewhere else, use that path in place of /data, and set l2_path to a folder under it. Run the commands on the GPU machine over SSH.
Is the SSD mounted?
lsblk
No sudo needed, this only reads device info. Look for your SSD in the list.
- Already has a value under
MOUNTPOINT? It's mounted. Setl2_pathto a folder on that mount. - Nothing under
MOUNTPOINT? It's not mounted yet, and any path you set will land on the regular boot disk instead of the SSD. Mount it first, using the partition namelsblkjust showed you (for examplesdb1ornvme1n1):
sudo mkdir -p /data
sudo mount /dev/nvme1n1 /data
(Replace /dev/nvme1n1 with your own partition name.) This mount does not survive a reboot; add it to /etc/fstab if you need it to.
The folder and its permissions
After you mount the SSD, there is usually nothing more to do. Before the cache server starts, the agent creates the l2_path folder when it does not exist and gives its own user ownership of that new folder. It never changes the owner of a folder that already exists, never uses sudo inside a system folder such as /etc, /usr or /var, and refuses a relative l2_path. A folder the agent's user can already write works wherever it is. If the folder already exists and the agent's user cannot write to it, give that user ownership yourself.
The agent needs sudo for this only when its own user cannot write there, for example under a mount owned by root. It uses sudo -n, which never asks for a password. Check that it will work by running this on the GPU machine, as the user that runs the agent:
sudo -n true && echo "passwordless sudo works"
-
It prints
passwordless sudo works. You do nothing. The agent creates the folder. -
It prints
a password is required. The agent cannot create the folder, and the run stops before anything starts. Do this once after you mount the SSD, using your ownl2_pathin place of the example. First find the agent's user:ps -eo user,args | grep '[i]nsights-cli agent'The first column of the line it prints is the user. Then create the folder and hand it over:
sudo mkdir -p /data/lmcache/l2sudo chown <agent-user>:<agent-group> /data/lmcache/l2The error message in the run shows these same commands, already filled in with your path and user. If the agent runs as root, skip this.
The agent cannot tell whether a path is on your SSD. If the SSD is not mounted, the folder is created on the boot disk, the run still starts, and the agent writes a warning to its log. Mount the SSD before you run.
Step 7: Hardware stage
- Deployment. Keep One machine. Several machines is Distributed KV, which has its own guide.
- Machines (required). Choose the onboarded machine. For the NVMe SSD tier, choose one with the SSD mounted.
- GPUs to use. Leave it blank to use all GPUs, or enter a number. It must match
tensor_parallel_sizein the vLLM Args.
Step 8: Load stage
Each value you enter becomes its own scenario. Type a value and press Enter after each one.
| Field | What it sets | Starting value |
|---|---|---|
| Concurrency | Conversations running at the same time. | 1 |
| Input Sequence Length | Tokens in each turn's input. Type 8k for 8000. | 8192 |
| Output Sequence Length | Tokens generated per turn. | 512 |
| Rounds per Conversation | Turns replayed per conversation. Use 5 or more. | 25 |
Rounds below 5 are refused. For most datasets the benchmark drops any chunk shorter than 5 rounds, so a lower value runs with no users, and one rule applies to every dataset. A value near 5 stresses first-turn latency. A higher value stresses how well the cache keeps earlier turns.
Select Review scenarios to see every scenario the workload will run.
Step 9: Save and run
- Select Save to run later, or Save & Run to start now. Save & Run opens Monitoring, where each scenario runs as one job. See What Save & Run does.
- If a required field is missing, the save stops and lists the workload. Select Finish to complete it.
Worked examples: one model, three tiers
Build these three workloads in one project and compare them. They ran on one machine with 4 NVIDIA H200 GPUs. Everything is the same except the storage tier and its cache server settings. Leave every field that is not listed at its default.
Model stage (all three workloads)
| Field | Value |
|---|---|
| Models | Qwen/Qwen3-235B-A22B-FP8 |
| Datasets | sammshen/lmcache-agentic-traces |
Serving engine stage
| Field | Workload 1 | Workload 2 | Workload 3 |
|---|---|---|---|
| Cache storage | GPU | CPU RAM | NVMe SSD |
In Advanced settings, change these and select Save arg set for each editor:
| Editor | Argument | Workload 1 | Workload 2 | Workload 3 |
|---|---|---|---|---|
| vLLM Args | tensor_parallel_size | 4 | 4 | 4 |
| LMCache Server Args | l1_size_gb | None | 1024 | 1024 |
| LMCache Server Args | l1_init_size_gb | None | 20 | 20 |
| LMCache Server Args | l2_path | None | None | /mnt/kvcache/l2 |
| LMCache Server Args | max_capacity_gb | None | None | 13000 |
| LMCache Server Args | use_odirect | None | None | true |
Workload 1 has no LMCache Server Args. For Workload 2, l1_init_size_gb is already 20. Choosing CPU RAM or NVMe SSD adds kv_connector and kv_role for you.
Hardware stage (all three workloads)
| Field | Value |
|---|---|
| Machines | Your machine with 4 H200 GPUs |
| GPUs to use | 4 |
Load stage (all three workloads)
| Field | Value |
|---|---|
| Concurrency | 16, 32, 64, 128, 256, 512 |
| Input Sequence Length | 39680 |
| Output Sequence Length | 512 |
| Rounds per Conversation | 25 |
That makes six scenarios per workload. An input of 39680 plus an output of 512 is 40192 tokens, which fits this model's 40960-token window. For a model with a smaller window, lower the input length.
Before you select Save & Run
- Choose
l1_size_gbwell below your machine's total RAM. The 1024 here fits this machine only. - For Workload 3, mount the SSD and set
l2_pathto a folder on it. See Mounting the SSD and settingl2_path. - Keep
max_capacity_gbbelow the free space on the SSD.0means no limit. - Turn
use_odirecton only if your disk and filesystem support it. - Use LMCache Server Args saved for the same tier. A CPU RAM workload cannot use arguments saved for NVMe SSD.
Metrics collected
Open the Reporting page and filter to your run. KV Cache Offload charts plot each metric against concurrency, with one line per cache storage tier. Not every metric exists for every tier, because only CPU RAM and NVMe SSD run a cache server, and only NVMe SSD writes to a disk.
Charted on the Reporting page:
| Metric | What it tells you | GPU | CPU RAM | NVMe SSD |
|---|---|---|---|---|
| Output token throughput | Tokens generated per second, across all users. | Yes | Yes | Yes |
| Decode throughput | Token generation speed while a request is generating. | Yes | Yes | Yes |
| Input (prefill) throughput | Prompt tokens processed per second. | Yes | Yes | Yes |
| Request throughput | Completed requests per second. | Yes | Yes | Yes |
| TTFT (mean, P50, P90, P95, P99) | Time to first token: the average and the slower tail. | Yes | Yes | Yes |
| TPOT (mean) | Time per output token after the first. | Yes | Yes | Yes |
| Change in TTFT, first to last round | How much time to first token changes from a conversation's first round to its last. Median and P95. | Yes | Yes | Yes |
| vLLM prefill time | Mean time vLLM spends on the prompt. Shown only when your vLLM build reports it. | Yes | Yes | Yes |
| GPU KV cache usage | How full vLLM's own GPU cache gets. | Yes | Yes | Yes |
| vLLM local (in-GPU) cache hit rate | Share of lookups served from vLLM's own GPU cache. | Yes | Yes | Yes |
| vLLM external (LMCache) cache hit rate | Share of lookups served from the offloaded cache. It is zero for GPU, which has no offload. | Zero | Yes | Yes |
| LMCache server hit rate | Share of lookups served by the cache server. | No | Yes | Yes |
| LMCache GPU to CPU load and store throughput | Data moved between GPU memory and the RAM pool, in GB per second. | No | Yes | Yes |
| LMCache L2 load and store throughput | Data moved between the RAM pool and the disk pool, in GB per second. | No | No | Yes |
| GPU power, GPU utilization, CPU utilization | Hardware telemetry during the run. | Yes | Yes | Yes |
| SSD read and write throughput, IOPS, read and write latency, utilization, queue depth, data transfer summary | How hard the disk works. | No | No | Yes |
Stored with every job, but not charted. Read them with the CLI command analytics kv-cache-results, or through the API function list_multiroundqa_outputs_for_job.
| Metric | What it means |
|---|---|
| Session counts | Total, complete, incomplete and error sessions, and the number of failed requests. |
| Run validity | Whether the run is valid. When it is not, the reason is stored with it. Do not use the numbers of an invalid run. |
| Cache evictions, cache usage (bytes) | Entries the cache server evicted, and how much it holds at the end (CPU RAM and NVMe SSD). |
| L1 usage and eviction activity | How full the RAM pool is, and whether it was evicting (CPU RAM and NVMe SSD). |
| Chunks loaded, lookup hit tokens, lookup requested tokens | How much cache content was found and reused versus requested (CPU RAM and NVMe SSD). |
| Shared-prefix and per-user history reuse opportunity (%) | An estimate of how much traffic could reuse a cached prefix, not a measured hit rate. The shared-prefix estimate is not available for every dataset. |
Common pitfalls
- Comparing tiers without holding everything else fixed. Change only the tier. Keep the model, dataset, concurrency, input length, output length and rounds identical.
- Asking for more concurrency than the dataset can supply. A dataset holds a fixed number of conversations, and each one is cut into chunks of Rounds per Conversation turns. More rounds means fewer chunks. A concurrency above that number adds no load, and the run replays fewer conversations than you asked for. Compare the session counts in the results, not the requested concurrency.
- Rounds per Conversation below 5. It is refused. For most datasets the benchmark drops chunks shorter than 5 rounds, so the run would have no users, and one rule applies to every dataset. Even at 5 or more, a short conversation may never fill GPU memory, so offloading shows no effect. Use enough rounds that the conversation outgrows the GPU.
- Editing an argument and not saving it. Only a saved arg set is used. Select Save arg set after every change.
- Changing the tier after editing arguments. The arguments reset to the new tier's defaults.
- Running the NVMe SSD tier without a mounted SSD. The folder is created on the boot disk and the run still starts, so you measure the boot disk. Mount the SSD first.
- Using LMCache Server Args saved for another tier. An NVMe SSD set includes disk settings that a CPU RAM workload cannot use. Use a set saved for the same tier.
- Editing
kv_connector. It is wired to LMCache. Changing it without changing the rest of the setup stops the server from launching. - Choosing the AMD build on NVIDIA GPUs, or the reverse. The build must match the GPU vendor of the machine, or the wrong container image starts.