Part of an ongoing series. See Why Inference Needs a Rethink for the broader argument, How to Measure Effective GPU Utilization and its Effects on Tokenomics for the unit economics, and The Two Interfaces the KV Path Is Missing for the vendor collaboration this points toward.
In July 2025, I published Lessons Learned Scaling LLM Training and Inference with Direct Memory Access: Part 2, which described extending GPU memory into what I called an AI Token Warehouse: treating fast NVMe as a seamless extension of HBM, and moving KV cache pages over RDMA so a second turn could skip the prefill the first turn had already paid for.
The measured result then, on Llama-70B with an FP16 KV cache:
| Prompt length | Turn-2, paging off | Turn-2, paging on |
|---|---|---|
| 8,000 tokens | 1,199 ms | 38.8 ms |
| 32,000 tokens | 4,935 ms | 88.7 ms |
| 128,000 tokens | ~23 s | ~300 ms |
And, reported at the time, a +10% penalty at 50 tokens, the fixed cost of the caching layer when there is nothing worth paging.
That post closed with a section called Toward 10 Million-Token Contexts, which I described as “both a thought experiment and a blueprint.” This post is the follow-up: what building it actually taught us, where it stands, and what we are doing next.
What the blueprint got right
The KV cache is a storage problem wearing a GPU costume. Everything since has reinforced it. The cache is large, deterministic, written once and read many times, and highly repetitive across sessions: every property of a storage workload, and none that would justify treating it as a scratch buffer.
Reuse is the whole mechanism. Part 2 was explicit that if you never reuse the cache, paging does not help and, in fact, costs you a little. That is still exactly true, and still the first thing to establish about a workload before modeling any benefit.
Multi-turn is the real shape of the work. This aged well. The rise of agentic traffic, where each step resends the accumulated trajectory plus one new observation, made multi-turn the dominant pattern rather than a special case. Step 20 of an agent loop contains all of steps 1 through 19, which is close to the ideal case for reuse.
Four Things We Learned by Building It
1. Prefetch volume should track sparsity, not context length
Part 2 treated prefetch as a scheduling problem: know which pages the next operation touches, start early, hide the latency behind compute. For a dense pass over contiguous history, that works.
Building it at scale surfaced the underlying scaling argument. Prefetching every block a dense attention window could touch is linear in context length: fine at 128K, untenable at 10M, where you would stream the whole context to find the handful of blocks that matter. And long-context attention is not dense: top-k selection, block-sparse routing and retrieval-style attention all touch a small, content-dependent subset.
That gave us the design constraint we now build to:
Prefetch volume should be proportional to the sparsity of the attention pattern, not to the length of the context.
It is a harder target than scheduling, because it means predicting which blocks matter rather than fetching known ones earlier. It also makes 10M tractable rather than merely possible.
2. “KV page fault” is the framing that carries the right intuitions
The most useful takeaway from the work was a name. When an attention kernel touches an offloaded block, the system must fetch it and wait; the GPU is idle for that request until the block lands. That is a KV page fault, and it sits squarely in the critical path.
The name does real work, because page faults come with decades of intuition attached. You do not fix them by making the disk faster; you fix them by not taking them. The metric that matters is fault rate, not fetch bandwidth, and the tail matters more than the mean, because under multi-tenant load a small fault rate produces a large p99.
It also clarifies where effort pays. Faster storage reduces the cost of a fault. Better prediction reduces how often they happen. The second is worth considerably more, and that realization redirected most of our engineering.
3. Fail-closed is a design principle, not a configuration choice
Part 2 contained a passage about GPUDirect Storage falling back to POSIX when misconfigured, noting almost in passing that this creates a DMA-to-POSIX downgrade path for an adversary wanting to bypass protected execution. It then explained why the fallback existed: GDS is genuinely hard to configure, node configuration drifts across reboots in autoscaling fleets, and a fatal error would mean outages.
Both halves of that were true, and holding them together turned out to be the interesting part. The operability problem is real. The answer we settled on is to solve it properly, detecting topology automatically rather than depending on hand-maintained configuration files, so the security control does not have to pay for it.
That generalised into a principle we now apply everywhere: a control with a degraded mode runs degraded. It gets bypassed under load, then permanently, and the failure it existed to catch goes undetected alongside it. Every gate in the current design fails closed, and the operability work happens on the other side of that line.
4. A cache index is a convention; an interface is a contract
Part 2’s cache was globally addressable by convention: a metadata key written alongside each block on a shared filesystem, so any GPU could find any page. It worked. Every engine that has tried this has built something similar, and none of them interoperate.
Working through what it would take to make that portable produced the more valuable insight. The cost of hand-rolled copy loops is not only duplicated effort and zero portability. It is that there is no seam: nowhere to attach encryption, integrity, tenancy, or attestation, because there is no boundary, only an internal implementation detail.
You cannot secure an abstraction that does not exist, and you cannot schedule it, measure it, or buy it from a second vendor either. That is what turned the work toward defining an open contract rather than optimizing a private one.
The Question We Build to Now
Part 2 asked: how do we move KV pages fast enough? The answer was DMA and RDMA, and it was a good answer; the 2025 measurements above are real and still hold.
The question that governs long-context economics turns out to be one layer up:
How do we know which blocks to move, early enough that the move is invisible?
Three consequences follow, and they shape the current roadmap:
- Latency hiding is the requirement, not bandwidth. The block must already be in flight when attention asks for it. A fetch issued at the moment of use is too late regardless of fabric speed.
- Prediction quality dominates transport speed once transport is adequate. Halving fetch latency helps; halving the fault rate helps far more.
- Prediction should run close to its inputs. Today’s signals are the ones userspace can see: request context, block metadata, engine timing, observed reuse. That is enough to produce the large long-context wins the current numbers are built on. The state that most directly determines which blocks matter is produced inside the attention computation itself, and getting closer to it is the next significant step.
Where it stands
The 10M-token case that Part 2 framed as a thought experiment now runs. On 8xH100, a 10.5M-token session that takes 1 hour 42 minutes to rebuild through prefill is restored in seconds, with bit-identical output, surviving engine and fabric restarts. Time-to-first-token on returning long-context sessions improves by two to three orders of magnitude, and the multiple rises with context length.
Bit-identical is not a rounding claim. The streamed windowed form uses an online softmax accumulation, carrying the running maximum and the sum of exponentials alongside the partial output, which is numerically identical to the monolithic computation. Same computation, different memory schedule. That makes this an infrastructure change rather than a model change: no eval regression, no quality conversation.
Why the multiple grows with the context
The improvement isn’t a fixed number; it depends on sequence length.
Prefill computes attention between every pair of tokens in the prompt, so its cost carries a term that grows with the square of the context. Restoring a cache doesn’t work that way: a KV cache is linear in tokens, so twice the context means twice the bytes to read back. Recompute is superlinear, restore is linear, and the gap between them widens with every additional token of context. That is why results improve as sessions get longer, the opposite of the usual pattern where an optimization washes out at scale.
Published joint results with FarmGPU and ScaleFlux put time-to-first-token improvements between 100x and 280x across models from 131K to 1M tokens, and beyond 1000x at 10M tokens. Those are prefill-dominated turns, which is the regime this work is built for.
At those lengths, prefill is the turn. A multi-million-token session spends almost all its time building the cache and very little generating the answer, so the improvement a user waits through and the improvement in GPU-seconds consumed land in the same order of magnitude. The relationship is exact: for a prefill share p of turn time and a prefill improvement S, the turn improves by 1 / ((1 - p) + p/S), which converges on S itself as p approaches 1.
| What the turn looks like | Prefill share of turn time | Whole-turn improvement |
|---|---|---|
| Short prompt, long answer | 20% | 1.25x |
| Mixed traffic | 50% | about 2x |
| Long context, hundreds of thousands of tokens | 90% | about 9x |
| Multi-million-token sessions | 99% | about 50x |
Where it does not help, stated plainly. Decode is untouched: the same tokens through the same weights at the same rate. A short prompt with a long answer is mostly decode, so there is little for a cache tier to recover, and the honest answer is that this is not the technology for that workload. The returns track context length, which is the same statement as above, read from the other end.
And where no multiple applies at all. At 10.5M tokens on 8xH100, the baseline is not slow; it doesn’t run: the cache doesn’t fit, and a session requiring over an hour and a half of prefill isn’t a product. That comparison is between an offering and a refusal, which is a categorical difference rather than a ratio, and it is the one that matters most commercially.
What is still being characterized
Three areas are still being characterized, and we would rather name them than let people discover them: fabric contention when many nodes prefetch speculatively across a shared fabric; NVMe read amplification when block granularity does not match request granularity; and the distribution of miss cost under realistic traffic. The means are good. The p99 is where the most work remains.
Where we are going
An open contract, with a reference implementation. The most durable output of this work is not a speedup; it is a boundary: deterministic block identity, typed metadata, an asynchronous data plane, and mandatory tenant and session scoping, defined so that engines, storage backends, and security controls can all attach to the same seam. That is the Open KV Cache API, with Inferra as its reference implementation. A boundary that exists in one vendor’s product is a feature; one with independent implementations and conformance testing is infrastructure.
Prediction closer to the signal. The next architectural step is narrowing the distance between where attention state is produced and where prefetch decisions are made. That needs interfaces that don’t exist yet on both the storage side and the device side, and it is work we would rather do with accelerator and memory-fabric vendors than around them.
Security designed in rather than retrofitted. The KV cache is a lossy but invertible encoding of the prompt: published work recovers exact tokens from a cache dump. Designing for that from the start (ciphertext everywhere except inside the GPU, keys sealed to a measured platform state, retention verifiable per turn) is far cheaper than adding it later, and it is what makes the result deployable by enterprise and public-sector buyers rather than only by hyperscalers.
Workloads that were previously infeasible. This is the part worth being excited about. A 10M-token context is not a faster version of a 100K one; it enables whole-codebase reasoning, multi-day agent sessions that never forget, and document analysis that doesn’t need chunking or a retrieval pipeline to work around a memory limit. Those were architectural compromises made because we couldn’t keep the cache. They do not have to be.