Large-context AI inference on Solidigm D7-PS1010 and Inferra™ by Lightbits
Long contexts are outgrowing GPU memory. When the KV cache no longer fits, teams either recompute it at high GPU cost or truncate it and lose accuracy. Solidigm and Lightbits tested a third option: reloading the KV cache from NVMe. The results show it’s faster at every context length tested, and the advantage grows as context grows.
What you’ll learn:
- 15 seconds, not 5½ minutes. Time to first token at 4.8 million tokens, 21.6x faster than recomputing on the GPU.
- Up to 8.4x more conversations per minute on the same GPUs.
- 69,244 live conversations held, 336x what the node’s GPU memory can hold.
See the full test results, system configuration and what they mean for your inference costs. Get the technical brief.