LLM inference at long context is limited by GPU memory, not GPU compute. A joint study by Solidigm and Lightbits Labs found that reloading the KV cache from NVMe storage is 4.6x to 21.6x faster than recomputing it on the GPU, and lets the same GPUs start up to 8.4x more conversations per minute.
Why LLM Inference is Hitting the GPU Memory Wall
Long-context LLM inference has outgrown the GPU. Agentic workflows, retrieval-augmented generation (RAG), and long-document reasoning are pushing context windows from tens of thousands of tokens toward ten million. The key-value (KV) cache that holds that context grows linearly with sequence length and again with every concurrent session. HBM capacity cannot keep pace, even on the newest accelerators.
The math is unforgiving. One 100K-token conversation on gpt-oss-120B consumes 7.08 GB of KV cache. An eight-GPU NVIDIA B200 node carries 1.46 TB of HBM, a hard ceiling of roughly 206 such conversations before a single model weight is loaded.
The Hidden Cost of Recompute in LLM Inference
When context no longer fits in HBM, teams face three bad options: recompute it, truncate it and lose accuracy, or offload it to storage that stalls the pipeline. Most default to recompute, and the cost is larger than it looks.
Prefill is not just slow. A prefill stream monopolizes the GPU, so decode work for every other user waits behind it, including sessions whose context never left memory. Every redundant prefill burns power and GPU-seconds to reproduce work the system already did once.
This is why cost per token is an incomplete LLM inference metric for infrastructure and platform leaders. It prices each unit of work but ignores how often a GPU rebuilds the same context. The better measure is cost per useful outcome, and its sharpest lever is prefill GPU-seconds avoided per request. Where the KV cache lives determines whether you pay for context once or on every request.
A Fourth Option for LLM Inference: Disaggregated KV Cache Storage
The Solidigm AI Central Lab and Lightbits Labs set out to answer a narrow question. When context exceeds HBM, is reloading the KV cache from NVMe faster than recomputing it on the GPU, and by how much?
The testbed paired Inferra™ by Lightbits, a KV cache orchestration engine, with a five-node Solidigm D7-PS1010 PCIe 5.0 NVMe cluster (607 TB usable) and an eight-GPU NVIDIA B200 node over four 200G RoCEv2 rails.
Earlier attempts to offload KV cache to general-purpose network storage stalled on queuing. This study shows that three ingredients, working together, clear the bar: NVMe with predictable latency under sustained load, an RDMA data path configured for losslessness, and orchestration that stages cache segments ahead of the request.
Results: KV Cache Reload vs. GPU Recompute for LLM Inference
Reload beat recompute on every axis tested, across two models (Llama-4-Scout and gpt-oss-120B), and the gap widened as workloads got harder.
- Long context: At 4.8M tokens, time to first token fell from 332.77 seconds to 15.39 seconds, 21.6x faster. Users wait 15 seconds instead of five and a half minutes. Across a 6.2x increase in context length, recompute time rose 27.5x while reload time rose only 5.8x.
- Concurrency: At 100K tokens on two GPUs, reload was 5.5x faster at a single request and 9.2x faster at 16 concurrent requests.
- Throughput: On identical hardware, the node started up to 8.4x more conversations per minute (337.7 vs. 40.3), with gains of 3.6x to 8.4x across four configurations.
- Capacity: The NVMe tier held 69,244 live 100K-token conversations, 336x the HBM ceiling, and contexts sampled mid-fill still reloaded in under 2 seconds.
- Correctness: Output from reloaded cache was byte-identical to recomputed output, with zero fetch errors.
What This Means for Your LLM Inference Platform
The data makes an economic case as much as a technical one. GPUs are the scarcest, most expensive layer of the inference stack. A disaggregated KV cache lets context length and concurrency scale in a tier that costs a fraction as much per unit of capacity. At 4.8M tokens, reload freed roughly 2,540 GPU-seconds on an eight-GPU node per request.
For Neoclouds and serverless or managed inference providers, that means more sessions and more revenue per GPU without buying more GPUs. For financial services teams running long-document analysis and agentic workflows, it means full context without minutes-long waits. For e-commerce platforms, it means multi-turn assistants that remember the whole session and still respond in real time.
Inferra by Lightbits is built for exactly this problem. It orchestrates KV cache on high-performance NVMe and stages context ahead of the request, so GPUs spend their cycles on decode instead of rebuilding prefill.
Keep your GPUs pinned. Power them with Solidigm and Inferra by Lightbits. Download the full Solidigm and Lightbits technical brief.
Frequently Asked Questions About LLM Inference and KV Cache
What is the KV cache in LLM inference?
The KV cache stores the key and value tensors for every prior token, so the model does not recompute attention at each generation step. It grows linearly with context length and with concurrent sessions, and by default it lives in GPU HBM.
Why is long-context LLM inference so expensive?
When the KV cache exceeds HBM, the engine evicts it and recomputes prefill. Recompute occupies the GPU and blocks decode for other users. At 4.8M tokens, recompute took 332.77 seconds on an eight-GPU NVIDIA B200 node.
Is reloading KV cache from storage faster than recomputing it?
Yes, with the right architecture. In Solidigm and Lightbits testing, reloading KV cache from PCIe 5.0 NVMe over RoCEv2 was 4.6x to 21.6x faster than GPU recompute, with byte-identical output.
How does Inferra improve LLM inference efficiency?
Inferra by Lightbits orchestrates KV cache on NVMe storage and stages context ahead of each request, freeing GPUs for decode. In testing, the same node started up to 8.4x more conversations per minute.
