As Large Language Model (LLM) context windows push past a million tokens, the KV cache driving fast inference outgrows high-bandwidth GPU memory (HBM). Recomputing this context on the GPU creates severe bottlenecks, driving up time-to-first-token (TTFT) and capping session concurrency.
This technical report details a joint proof of value by Lightbits Labs and Solidigm, showing how offloading and orchestrating KV cache reload over a disaggregated NVMe cluster drastically outperforms traditional GPU recomputation.