A 10.5M-token session rebuilt through prefill on 8xH100 takes 1 hour 42 minutes. Restored from a tiered KV cache, the same session is serving in seconds, with bit-identical output.
That ratio is the headline, and it widens with context: published joint results with FarmGPU and ScaleFlux put time-to-first-token between 100x and 280x across models from 131K to 1M tokens, and beyond 1000x at 10M. This post is about the mechanisms underneath it, why the naive version of this idea does not work, and which parts of the problem we have not solved.
What is the KV cache? When a model reads your prompt, every token it attends to produces a key and a value vector at every layer. Those vectors are the model’s working memory of the conversation so far: they are what lets token 900,000 attend to token 12. Together they are the KV cache. It is not the model weights, which are fixed and shared across every request; it is per-session state. It grows linearly with context length, and at long context it dwarfs the weights. Throw it away and the only way to get it back is to run the entire prompt through the model again. That re-run is called prefill.
Why the Obvious Approach Fails
“Put the KV cache on NVMe and load it back” is an idea everyone has within about ten minutes of seeing the problem. It does not work as stated, and the reason is instructive.
Attention is not a bulk read. Generating one token requires attending over the full history, layer by layer, and the kernel wants each block at a specific moment in a tight schedule. Fetch a block when the kernel asks for it and you have inserted a network round trip into the middle of an operation measured in microseconds. Do that per block, per layer, per token, and the “optimized” path is slower than recomputing, which is why cache offload has a reputation for being a good idea that does not pay off.
So the problem is not bandwidth. RDMA has plenty of bandwidth. The problem is latency hiding: the block has to already be in flight before attention asks for it. Everything below follows from that constraint.
Mechanism 1: Virtual Paging
The first move is to stop requiring that a sequence be fully resident in HBM to be served.
Classic serving treats the KV cache for a sequence as one allocation: either it is in GPU memory or the sequence cannot run. Virtual paging breaks that binding. A sequence has a logical extent, all 10.5M tokens of it, and a much smaller physical residency. The pages that attention needs now are resident; the rest live on the tier below, addressable and fetchable.
This is the same abstraction an operating system applies to process memory, and it buys the same thing: a working set smaller than the address space. The difference is that a page fault here costs microseconds against an attention kernel’s schedule, not milliseconds against a syscall, which is why the prefetch has to be good rather than merely present.
The important consequence is one that gets lost in performance framing:
Paging is not a speedup on the warm path. It is the reason the warm path exists at all.
At 10.5M tokens on 8xH100, the cache does not fit in HBM. The baseline is not “slower”; it cannot run. Comparing seconds against 1 hour 42 minutes is really comparing a session that runs against a session you would have to refuse or re-architect around. The ratio at that length runs beyond 1000x, but the categorical difference matters more than the multiple.
Mechanism 2: streaming windowed attention with double buffering
Given a sequence whose cache is mostly not in HBM, attention must be restructured to consume it as a stream rather than an array.
The windowed form processes history in blocks: stage a window of KV into a GPU-side buffer, run attention against the query for that window, and combine the result into a running accumulator. Because attention’s softmax normalization is associative under the right bookkeeping (carry the running maximum and the sum of exponentials alongside the partial output, the standard online-softmax construction), the streamed result is numerically identical to the monolithic one. That is what makes “bit-identical output” a claim rather than an aspiration: this is not an approximation with a quality/latency knob; it is the same computation with a different memory schedule.
Double buffering is what keeps the GPU busy. Two staging buffers: attention computes against buffer A while the transfer engine fills buffer B, then they swap. If the fill of B completes before the compute on A does, the transfer is entirely hidden and the GPU never stalls. If it does not, you see the difference as a bubble.
That inequality, transfer time per window versus compute time per window, is the single most important relationship in the design. It sets the window size, it sets how much staging memory you need, and it determines whether the whole approach pays off on a given fabric. A window too small and per-window fixed overheads dominate. Too large and the staging buffer stops fitting.
Mechanism 3: Prefetch That Anticipates
Double buffering hides the transfer for the next window. It does nothing for the first one, and nothing for a window whose location was not predictable.
This is where prefetch comes in, and where the interesting problem lives. Sequential prefetch, “fetch block n+1 while working on block n”, handles the easy case, and for a straight walk through history it is most of what you need. It is not sufficient for two reasons: the start of a turn has no n to extrapolate from, and attention with sparsity or top-k block selection does not walk history sequentially.
Predicting which blocks will be wanted, from content similarity and access history rather than from position, is a large enough topic to be its own post. SLAP, a subsequent blog post, covers the two signals it combines and why either one alone fails on agentic traffic.
The Ladder, and How to Read It
| Context | Cost to rebuild through prefill | With the cache restored |
|---|---|---|
| 100K | 7.4 seconds | sub-second |
| 1M | 2 minutes | seconds |
| 10.5M | 1 hour 42 minutes | seconds |
Time-to-first-token improves by two to three orders of magnitude on returning long-context sessions, and the multiple rises with context length.
Three things to notice.
The multiple grows with context, because prefill is quadratic in sequence length while restore is closer to linear in bytes moved. The gap widens as contexts grow, which is the opposite of the usual pattern where optimizations wash out at scale.
Output is bit-identical. Verified, not asserted. The online-softmax construction above is what makes it possible, and it is the property that lets you treat this as an infrastructure change rather than a model change. No eval regression, no quality conversation.
It survives engine and fabric restarts. The cache outlives the process that created it. Restart vLLM, restart the fabric, and sessions stay warm. This is a durability property rather than a performance one, and it is the difference between a cache and a tier.
These are prefill numbers, and at these lengths prefill is the turn. Whole-turn improvement is for prefill share p, which converges on the prefill improvement as p approaches 1. On multi-million-token sessions, prefill is almost the entire turn, so the two figures land in the same order of magnitude. On short-context traffic, they do not, because decode dominates and decode is untouched. Model at the context lengths you serve.
What We Have Not Solved
The honest list, because a performance post without one is marketing.
Fabric contention under many-node prefetch. The measurements above are not from a fleet where hundreds of nodes are aggressively prefetching across a shared fabric simultaneously. Prefetch is speculative by nature (it issues reads for blocks that may not be needed) and speculation that is free at one node can be expensive in aggregate. We do not currently model the contention, and we would rather say so than extrapolate.
NVMe read amplification. Block sizes on the storage tier do not align perfectly with the granularity of attention requests, and the resulting amplification depends on the access pattern. We have not characterized it across the range of workloads people actually run.
Miss cost. The ladder above is the hit path. A miss, a block that was not prefetched in time, is not free: it is a stall in the attention schedule. The distribution of misses under realistic traffic, and the tail latency they produce, is the weakest-modelled part of the whole system. The mean is good. We have less to say about p99 than we would like.
These are named on the methodology page as the weakest assumptions, and they are the first things to interrogate in a proof of value on your own traffic.
Reproducing It
The ladder configuration is available as a reproducible recipe (model, quantization, hardware, fabric, and the traffic generator) so you can check the numbers rather than take them on faith. If you want to run it against your own workload rather than the reference one, the PoC configuration is the place to start, and it is worth doing precisely because the three unsolved items above are workload-dependent.
Methodology note. The calculator at open.lightbits.ai is a simulation parameterized by these measurements, not a measurement of your fleet. Validate with a proof of value before committing capital.