The inference industry has spent the last two years optimizing around the KV cache without optimizing the KV cache itself. Paged attention fixed fragmentation. Prefix caching fixed redundant prefill for shared prompts. Quantization shrank the footprint. Disaggregated prefill/decode separated two workloads with different resource profiles. Each of these was a genuine, necessary advance — and each treats the KV cache as a side effect of attention that needs to be managed, rather than as the central resource the entire serving stack should be designed around. The result is a serving stack of well-engineered patches built on an unexamined assumption: that the KV cache lives or dies with whatever GPU memory happens to be attached to it. That assumption is now the actual ceiling on inference economics. The obvious-seeming fix — add more HBM — is not a solution; it is the industry paying an ever-larger capital and power bill to defer a problem that more memory doesn’t structurally address. This article argues that closing the gap requires a first-principles rethink of what KV cache is to the serving stack: not a cache that happens to sit in GPU memory, but a first-class, addressable, tiered data object with its own lifecycle, spanning from HBM through host memory to persistent tiers, moved and placed by a system designed around it. We describe what that rethink looks like in practice, and how Inferra‘s current work—rearchitecting the inference stack’s relationship to KV cache, not reinventing storage hardware—fits as an early, right step toward it.
1. Problem of “Problems outgrowing fixes over time”
Look at the last two years of inference serving research and engineering, and a pattern emerges: nearly every major advance is a response to the KV cache outgrowing its container.
Paged attention and block-based KV management solved a real problem — naive KV allocation fragmented GPU memory so badly that usable capacity fell far below physical capacity. Treating KV storage like virtual memory pages fixed that, and it’s now table stakes in every serious serving engine.
Prefix and session caching solved a different problem — the same system prompt, few-shot block, or RAG context being recomputed identically across thousands of concurrent requests. Caching and sharing that prefix’s KV state avoids redundant prefill computation.
KV quantization and compression (INT8/INT4 KV, low-rank approaches, architectural reductions like multi-query and grouped-query attention) attack the footprint directly, trading some precision or architecture flexibility for a smaller per-token memory cost.
Eviction heuristics (sliding windows, attention-sink-aware eviction, importance-scored retention) decide which KV entries to drop when everything can’t be kept, betting that most historical context contributes little to the next token.
Disaggregated prefill/decode serving separates the compute-bound prefill phase from the memory-bandwidth-bound decode phase onto independently scaled pools, so neither starves the other of the resource it actually needs.
Tiered offload pushes cold KV out of HBM to CPU DRAM, then to pooled or NVMe storage, when it doesn’t fit or isn’t immediately needed.
Every one of these is a legitimate, valuable piece of engineering. And every one of them was invented as a workaround for the same underlying fact: KV cache is now large enough, and grows unpredictably enough, that it doesn’t reliably fit where the system was designed to keep it. These techniques treat that fact as a constraint to route around. None of them ask the more basic question: what if the serving stack were designed around KV cache as its central resource from the start, instead of retrofitting workarounds onto a design that assumed KV cache would stay small?
2. Why “Just Add More HBM” Isn’t the Answer
The instinctive response to a memory-capacity problem is to buy more memory. For KV cache, this instinct is worth resisting, for reasons that are structural rather than incidental:
The economics point the wrong way. HBM is, per gigabyte, among the most expensive and power-hungry memory in the industry, constrained further by packaging and supply. Provisioning HBM capacity for peak KV cache footprint—the worst case of long context and high concurrency—means paying premium prices for capacity that sits mostly idle outside peak load. That’s capital sunk into headroom, not into serving more tokens.
Most KV cache, most of the time, is cold. In real traffic, the vast majority of active sessions are idle at any given instant — waiting on a user’s next message, or holding context that won’t be touched again for seconds or minutes. Provisioning expensive, power-dense HBM to hold cache that is statistically cold most of the time is solving a hot-path problem with a hot-path-priced resource applied indiscriminately to cold data.
It treats a data-locality and movement problem as a capacity problem. The real failure mode—GPUs stalling while waiting for KV blocks, or GPUs redundantly recomputing cache that already exists somewhere—needs a solution beyond more HBM, since that only buys time before demand outgrows the “more capacity.” Now, if the system doesn’t know where the right KV blocks are, can’t fetch them fast enough, or can’t tell a stale cache from a reusable one, adding capacity just gives it a larger space in which to have the same problem.
3. Extracting the Value: KV Cache as the center of the system architecture
If point fixes and brute-force capacity both fall short, the alternative is to stop treating KV cache as an artifact that happens to live in GPU memory, and start treating it as what it actually is: a data object with a lifecycle that should be managed, placed, and moved by a system designed around that lifecycle from the ground up.
That reframing has concrete implications:
Temperature-aware placement. KV cache should live in a tier appropriate to how likely it is to be touched next — hot in HBM, warm in host memory, cold in a fast persistent tier — with the system actively managing that placement rather than treating eviction as a last resort.
Locality and addressability across the pipeline. A KV block should be findable and fetchable fast regardless of which tier it’s currently in, and regardless of which node or GPU is asking — not tied to the accident of which GPU generated it.
Reuse as the default, not the exception. The system should know when cache already exists for a given prefix or session before deciding whether to recompute, rather than recompute being the default fallback because checking is inconvenient.
Utilization of all available signals across the stack. Making the judgment of what to keep, where to offload, what to evict and what would be needed next requires reading signals from across the stack, upstream and downstream: request patterns arriving at the router, session and prefix history, scheduler state, and the current shape of memory pressure across tiers. Taken together, those signals are what let a system run without interruption
A data path built for this access pattern, not repurposed from one that wasn’t. Most of the storage and memory-movement infrastructure inference stacks rely on today was designed for very different workloads—bulk sequential reads, checkpointing, general-purpose caching. KV cache access is limited, bursty, latency-sensitive, and constantly shifting in what’s hot. The path data takes between tiers needs to be engineered for that pattern specifically, or every technique layered on top of it — paging, prefix caching, offload — will keep hitting the same ceiling when it needs data back quickly.
This is a genuinely different design center than “attach more memory” or “cache more cleverly at the attention layer.” It’s a claim that the serving stack, the scheduler, the network, and the data path all need to be designed with the KV cache’s lifecycle as the organizing principle — not as a side effect that each layer discovers it has to accommodate independently.
4. Where Inferra™ Fits
Inferra’s current work is a first step in this direction, and it’s worth being precise about what that does and doesn’t mean. Inferra is not rethinking storage hardware — the goal isn’t a better disk or a novel memory technology. Inferra is rethinking the inference stack’s relationship to KV cache: treating it as a managed, tiered, addressable resource with its own lifecycle, and building the data path that moves it — between HBM, host memory, and persistent tiers — to match the access pattern KV cache actually has, rather than the access pattern general-purpose infrastructure assumes.
That distinction matters because it changes where the engineering effort goes. It’s not about buying denser or faster storage components — plenty of vendors can sell you those. It’s about the system that decides what stays hot, what moves cold, what gets reused instead of recomputed, and how fast any of that can get back in front of a waiting GPU. That’s a systems and software problem sitting above the hardware, and it’s the layer that’s been comparatively under-invested relative to the attention-algorithm and hardware-capacity work the rest of the industry has focused on.
Part of how Inferra approaches that problem is by treating placement as a prediction problem, not just a reaction to memory pressure. Rather than waiting for a tier to fill up and then deciding what to evict, the system draws on historical access patterns and live signals from upstream in the stack — request arrival patterns, session and prefix behavior, scheduler and routing state — to anticipate which KV data will be needed next, and stage it accordingly. The goal is for data to already be in place, or already in motion, by the time compute asks for it, rather than triggering a fetch only after a GPU is already stalled waiting. This turns “keep the most expensive resource in the system fed” from a best-effort property of the stack into something the system actively and continuously manages.
This is just a step toward treating KV cache as the central design constraint it has already become in practice — not the fourth or fifth thing a serving stack was patched to deal with.
Want to learn more about Inferra Optimized AI Inference? Download the tech paper or talk to one of our tech experts.