KV Cache Offload Methodology

KV cache offload measures how well a model server retains conversation history across multiple back-and-forth turns by moving KV cache blocks off the GPU when it runs out of room, and pulling them back when a conversation continues. Unlike metrumbench-llm or GenAI-Perf, it replays real, recorded multi-turn conversations, not single-shot or synthetic prompts — the point of the benchmark is to see what happens when a real conversation genuinely outgrows GPU memory.

The engine underneath CPU and NVMe offload is LMCache, running as a standalone lmcache-server sidecar process alongside vLLM. The GPU tier does not use LMCache at all — it is plain vLLM serving off its own in-VRAM prefix cache, and exists as the no-offload baseline every comparison is measured against.

Use it to answer:

• Does offloading KV cache to CPU RAM or NVMe disk help or hurt time-to-first-token on a long multi-turn conversation?
• How much conversation history can this server retain before it has to evict and recompute?
• Which storage tier (GPU-only, CPU RAM, or NVMe SSD) gives the best latency/capacity trade-off for a given hardware config?

---

How It Works

Storage Tiers

Tier	Where cache lives	Cache server?
GPU	vLLM's own in-VRAM prefix cache. No offload.	No
CPU RAM	A pinned system-RAM pool, managed by a standalone lmcache-server sidecar.	Yes
NVMe SSD	A small CPU staging pool in front of a larger disk-backed pool on NVMe.	Yes

CPU RAM and NVMe SSD both go through LMCacheMPConnector — LMCache's own MP (multi-process) mode: the cache lives in a separate lmcache-server process, not embedded inside vLLM. vLLM is wired to it with kv_connector=LMCacheMPConnector, kv_role=kv_both, and a kv_connector_extra_config that points at the server's fixed ZMQ port (lmcache.mp.port: 5555, LMCache's own upstream default) with a 300-second connect timeout. Exactly one lmcache-server replica runs per benchmark run — never one per vLLM replica — and every vLLM replica in that run talks to it as a shared peer, whether there is one vLLM replica or several. GPU never connects to a cache server at all; there is nothing to offload to.

Execution

1. The workload's tier selection resolves a vLLM engine-args profile (and, for CPU RAM/NVMe SSD, a matching lmcache-server profile) — NVIDIA or AMD build, selected explicitly.
2. For CPU RAM and NVMe SSD, lmcache-server starts first as the sidecar; vLLM starts next, wired to it via kv_connector=LMCacheMPConnector / kv_role=kv_both.
3. The runner (insights_cli.tools.multiroundqa_runner, wrapping a vendored fork of LMCache's own multi-round-qa.py) replays the selected dataset against the running server(s) at the requested concurrency, ISL/OSL, and num_rounds.
4. Results are parsed from the runner's own JSONL output and ingested via ingest_multiroundqa_results, alongside a Prometheus scrape of vLLM's (and, for CPU RAM/NVMe SSD, LMCache's) own metrics.

Multi-Replica Serving: Caddy Load Balancing

multi-round-qa.py takes exactly one --base-url — it has no client-side round-robin across endpoints. When a scenario launches 2 or more vLLM replicas, the agent starts a local Caddy reverse proxy in front of them so every replica actually receives traffic; --base-url points at Caddy, not at any one replica. lmcache-server is never behind Caddy — it stays a single shared peer every vLLM replica connects to directly.

The load-balancing policy is chosen per tier, not fixed:

Tier	Policy	Why
CPU RAM / NVMe SSD	least_conn (routes to whichever replica has the fewest in-flight requests)	A shared lmcache-server means any vLLM replica can retrieve a prefix another replica stored, so there's no cost to a conversation's rounds landing on different replicas — load-adaptive routing is strictly better here.
GPU	header X-User-Id (sticky — the same virtual user always routes to the same replica)	GPU has no shared cache server. Without stickiness, successive rounds of one conversation could land on different vLLM replicas and miss that replica's own local KV cache entirely, silently degrading the exact cache-hit/TTFT comparison this benchmark exists to measure.

The sticky routing needs no client-side change: multi-round-qa.py already sends an x-user-id header on every request (for its own virtual-user tracking), and Caddy's header policy hashes on it. Caddy also runs active health checks (/health, 10s interval) and passive failure detection (stops routing to a replica after 3 failures in 30s), so a replica that crashes mid-run stops receiving traffic instead of failing every request sent to it.

Dataset: Real Conversations, Not Synthetic Load

Dataset code	What it is
sammshen	sammshen/lmcache-agentic-traces — real agentic multi-turn conversation traces.
tau2-bench-v1	Tau2-Bench (v1) — a multi-turn conversational benchmark dataset.

Each dataset is a fixed, finite pool of real recorded sessions. Every session is chronologically ordered and sliced into fixed-length chunks of num_rounds turns — each chunk becomes one independent "virtual user" the runner replays as its own conversation. A session shorter than num_rounds contributes no chunk at all.

This has a direct consequence for concurrency. The number of virtual users a dataset can produce at a given num_rounds is fixed and shrinks as num_rounds grows — a dataset that yields thousands of virtual users at num_rounds=1 yields far fewer at num_rounds=25. Requesting a concurrency higher than that number does not create more load: the runner replays every virtual user it has and stops there, so the achieved session count silently falls short of what was requested. This is not a bug to fix — it's a property of replaying real, finite conversation data instead of synthetic traffic. It does mean:

• A concurrency sweep that goes past the dataset's ceiling for the chosen num_rounds stops adding real load partway through the sweep. Numbers past that point reflect the same fixed session count, not the requested concurrency.
• Every scenario in a comparison must land at the same achieved session count to stay comparable — check the job's session counts (total / complete / incomplete), not just the requested concurrency, before drawing conclusions from a high-concurrency result.
• tau2-bench-v1 and every dataset except sammshen go through a generic loader with its own minimum chunk length (5 rounds) — a num_rounds below that floor produces zero virtual users, not a smaller but valid run.

multi-round-qa.py Fixed Defaults

Beyond the scenario matrix (concurrency, ISL/OSL, num_rounds) and the dataset choice, every KV cache offload run passes a set of fixed flags the UI does not expose, because they are internal safety knobs rather than values worth sweeping:

Flag	Value	Why it's fixed
--agentic-traces	always on	The mode gate for closed-loop, fixed-N-concurrent-user replay — this is what the whole tool does, not a variable setting.
--min-chunk-rounds	5	Matches the tool's own argparse default exactly. See the dataset-ceiling section above for what this does to tau2-bench-v1 and similar datasets.
--warmup	always on	Runs 4 short throwaway requests before timing starts, so the first real request of every scenario cell doesn't pay a one-time cold-start cost (first connection, tokenizer setup) inside the measured average.
--request-timeout	64800 seconds (18 hours)	Generous enough that no realistic scenario times out mid-run; the platform's own per-job timeout is what actually bounds run length.
--tokenizer	the model under test	The agentic-traces depth filter tokenizes against the real model, not a stand-in.
Agentic tool-call flags (--enable-agentic-tools, --ask-mode, --sanitize-harmony-tokens, --agentic-max-attempts 3)	on for gpt-oss models only	gpt-oss's harmony decoder is strict about tool-call formatting; the sammshen dataset's inline tool-call text has no tools schema on the request, which gpt-oss specifically mishandles without these flags. Resolves to nothing for every other model — their resolved command is byte-identical to a run without this row.

---

What Gets Measured

Core Serving Metrics (all tiers)

Metric	What it means
Mean / p50 / p90 / p95 / p99 TTFT	Time to first token, average and tail percentiles.
Mean TPOT	Average time per output token after the first.
Output / input throughput (tokens/sec)	Generated and prompt tokens processed per second.
Request throughput (requests/sec)	Completed requests per second.
TTFT degradation (median / p95)	How much TTFT worsens on later turns of a conversation versus the first, as a percentage change — the signal that a tier is failing to retain history cheaply.
Session counts	Total, complete, incomplete, and error session counts, plus failed-request count, per job.
GPU power / utilization / memory, CPU utilization	Hardware telemetry captured during the run.

LMCache Cache Metrics (CPU RAM and NVMe SSD tiers only)

Metric	What it means
Cache hit rate (local and external)	Share of KV cache lookups served from vLLM's own in-VRAM cache versus the offloaded LMCache tier.
KV cache usage (%)	How full the active KV cache is.
Cache evictions	Count of cache entries evicted to make room.
Cache usage (bytes)	Total offloaded cache currently held.
L1 usage ratio, L1 eviction loop ticks / triggered	How full the fast (L1) pool is, and whether it is actively cycling through eviction.
L0-L1 and L2 load/store throughput (GB/s)	Data movement rate between vLLM's own cache, the offload pool, and (for NVMe) the disk-backed pool.
Chunks loaded, lookup hit tokens, lookup requested tokens	Counts describing how much cache content was found and reused versus requested.
Shared-prefix / per-user history reuse opportunity (%)	How much of the traffic could theoretically reuse a cached prefix, whether or not it actually did.

Disk Metrics (NVMe SSD tier only)

Metric	What it means
Disk read / write throughput (GB/s)	Data rate to and from the NVMe-backed cache pool.
Disk read / write IOPS	I/O operations per second against the disk pool.
Disk read / write latency (ms)	Per-operation latency against the disk pool.
Disk utilization (%) and queue depth	How busy the disk is and how many I/O operations are queued.
Total disk read / write (GB)	Cumulative data moved over the job.

---

Sampling Methodology: Continuous vs. Before/After

Two vLLM cache metrics are collected as a continuous time series during the run rather than a single measurement:

• Local cache hit rate (vLLM's own in-VRAM radix cache — active on every tier, including GPU).
• External cache hit rate (the KV-connector-backed cache — CPU RAM and NVMe SSD only, since GPU has no connector).

Both are sampled off vLLM's /metrics endpoint every 15 seconds throughout the job. Rather than averaging raw ratios, the platform derives a rolling rate() over a 300-second window — matching the reference HPE benchmark methodology's own stated approach ("Prometheus rate window: 300s, time-series step: 15s") — and reports the mean of the final quarter of that windowed series as a steady-state approximation. This deliberately isolates the plateau after cache warm-up from the cold-start ramp-up period, which a naive whole-run average would blend together and understate.

Two other metrics — mean prefill time and mean queue time — remain a single before/after measurement: one scrape immediately before the benchmark's traffic starts, one immediately after it ends, diffed. These come from vLLM's own request-timing histograms, which are not independently confirmed to exist on every pinned vLLM build, so they stay null rather than a false zero when the underlying histogram isn't present.

KV cache usage (%) is a continuously-sampled average, not a rate: the same 15-second scrape reads the instantaneous gauge and the platform reports the mean across the run. A single end-of-run snapshot was tried first and found to read a flat 0% for offload-tier jobs regardless of concurrency, because by the time that scrape ran the benchmark's own traffic had already drained and freed the blocks it briefly held — the continuous average avoids that blind spot.

If a job finishes before a single 15-second sample is taken, every one of these falls back to the before/after diff automatically, so short runs still get a value rather than null.

---

Comparing Tiers Correctly

• Change only the tier under test (and its build, if switching NVIDIA/AMD). Keep model, concurrency, ISL, OSL, and num_rounds identical across the comparison — a tier comparison is only meaningful when everything else is held fixed.
• Confirm every scenario in the comparison hit the same achieved session count (see the dataset ceiling above) before comparing throughput or latency across them.
• num_rounds controls how deep the replayed conversations go, and by extension how much offload actually gets exercised. A conversation short enough to fit entirely in GPU memory never triggers offload on any tier, making the comparison a no-op — num_rounds needs to be high enough that the conversation genuinely outgrows GPU capacity before a tier comparison says anything about offload behavior specifically.