The structural problem with being a NeoCloud is that you are selling the same GPUs as everyone else, sourced from the same vendor, at a price the market sets. You can compete on how fast you can rack them and how cheaply you can power them, and both are races the largest player wins by default.
Differentiation has to come from what the GPUs deliver, not from what they are. This post is about three things you can put in a contract once the KV cache is tiered rather than discarded, and just as importantly, what each one actually costs you to honor.
What is the KV cache? When a model reads your prompt, every token it attends to produces a key and a value vector at every layer. Those vectors are the model’s working memory of the conversation so far: they are what lets token 900,000 attend to token 12. Together they are the KV cache. It is not the model weights, which are fixed and shared across every request; it is per-session state. It grows linearly with context length, and at long context it dwarfs the weights. Throw it away and the only way to get it back is to run the entire prompt through the model again. That re-run is called prefill.
Tier 1: A latency SLA on returning sessions
What you sell: a p95 time-to-first-token guarantee on returning sessions, under load.
Every provider quotes TTFT. Almost nobody guarantees it at p95 on a returning session, because on a cache-regeneration architecture that number isn’t a property of your fleet; it is a property of how long the customer’s conversation happens to be. A returning user with a 200K-token history gets a TTFT determined by prefill on 200K tokens, and prefill is quadratic. You cannot commit to a number when the customer controls the input to it.
When the cache is restored instead of recomputed, TTFT on a returning session stops scaling with history length and starts looking like a fetch. That is a number you can put in a contract, because it is bounded by your infrastructure rather than by your customer’s behavior.
What it costs you to honor. The guarantee holds while the session is warm, that is, while its cache is still resident somewhere on your tier ladder. It degrades to baseline when the session has aged out. So the real commitment is not “fast TTFT”; it is “fast TTFT within a retention window”, and the retention window is a capacity decision you are making on the customer’s behalf. Size the NVMe tier for the window you sell, not the window you hope for.
How to write it. Guarantee p95 TTFT for sessions that have been active within the retention window; state the window explicitly; state the cold-start behaviour outside it. A guarantee with a clearly stated boundary is worth more than a vague one, and it is the only kind you can actually operate.
Tier 2: A context SLA
What you sell: 1M-token and 10M-token sessions, served with usable latency on returning turns.
This tier is genuinely hard to copy because it is not a performance difference; it is a feasibility difference. From the published benchmark ladder, rebuilding a 10.5M-token session through prefill takes 1 hour 42 minutes. That is not a slow product. It is not a product. No competitor on a discard-the-cache architecture can list it on comparable hardware, regardless of how they price it.
Restored rather than recomputed, the same session comes back in seconds, with bit-identical output.
The commercial point is that you are not 40% cheaper in a crowded category. You are the only listing in an empty one. Long-context document review, multi-day agent sessions, whole-repository code assistants: all of these are workloads that currently get architected around the limitation, by chunking and summarizing and losing fidelity. Being able to serve them intact is a different conversation with the customer than a discount.
What it costs you to honor. Long-context sessions are large on the storage tier and they are the ones most likely to be resumed after a long gap. Your NVMe sizing and your retention policy are doing the real work here. Price accordingly; see burst pricing below.
Tier 3: A privacy SLA
What you sell: per-tenant encryption keys, mandatory tenant scoping on every cache block, and a stated roadmap toward attested zero-data-retention.
The enterprise buyer’s actual fear is not that you are malicious. It is that they cannot tell. Their conversation data passes through infrastructure they do not control, is cached in tiers they cannot inspect, and their only recourse is a clause in a PDF and an annual audit.
What you can commit to today is concrete: cache blocks encrypted at rest and on the wire, keys scoped per tenant, tenant and session identity mandatory on every block so cross-tenant reuse is structurally impossible rather than policy-prohibited.
Be honest about what is on the roadmap. Verifiable per-turn attestation, in which the tenant checks cryptographically, before each request, that the encryption policy and platform state hold, is a proposal and a reference implementation, not a shipping feature. Post 8 in this series covers it in detail and says so plainly. Sell the encryption and the scoping as facts; sell the attestation as a direction you are committed to and can co-design with them. Enterprises respond well to a credible roadmap and badly to discovering a shipped-sounding feature was aspirational.
Pricing Mechanics that Map to Real Costs
The tiers above are only sellable if the pricing recovers what they cost. Four mechanisms, each tied to a cost you actually bear:
Burst pricing over context thresholds. Price bands at 128K, 256K, and 1M+ context. This is not arbitrary segmentation: cache size scales linearly with context, and a 1M-token session occupies roughly eight times the storage tier of a 128K one for the same retention window. The band structure recovers a real, proportional cost.
Cache-miss surcharge. When a session has aged out and must be rebuilt through prefill, you incur the full GPU cost of recomputation. Charging differently for a cold turn than a warm one is honest pricing, and it has a useful side effect: it makes the customer’s own usage pattern visible to them, which tends to move traffic toward the warm path that is cheaper for both of you.
Cached-input pricing. Charge less per token for input that was served from cache than input that had to be prefilled. This is the cleanest expression of the underlying economics and it has become a common market convention. It also prices the thing the customer can influence.
Retention window as a product. How long a session stays warm, hours rather than seconds, is a capacity commitment with a direct storage cost. Sell it as a tier. A customer who needs 24-hour warm resumption should pay for the NVMe that makes it possible; a customer whose sessions are always within the hour should not subsidize them.
The discipline that makes all four work: each is priced against a cost you can point to. That survives a procurement conversation. Segmentation invented to capture willingness-to-pay does not, and it poisons the relationship when the customer works it out.
Sizing the Promise Before You Make It
The failure mode is selling a retention window your storage tier cannot hold under peak concurrency, then quietly evicting sessions and missing the latency SLA you sold alongside it.
The arithmetic you need before signing anything:
- Cache bytes per session at your target context lengths, for your model and quantization.
- Concurrent warm sessions at peak, not average. SLAs are breached at peak.
- Retention window you intend to sell.
- Multiply, add headroom, and compare against your actual NVMe tier.
The cache capacity panel in the calculator at open.lightbits.ai does this arithmetic, and the Marketplace mode will take your list price and rank it against live listings so you can see what the premium tiers are worth against the current floor.
Methodology note. The calculator is a simulation parameterized by measured benchmarks, not a measurement of your fleet. Treat its output as a hypothesis worth testing, and validate with a proof of value before committing capital, particularly before committing to an SLA, where being wrong has contractual consequences.
Sequencing the Rollout
- Measure your reuse share and your context-length distribution. Everything else depends on these and most providers have never looked.
- Size the storage tier for the retention window you intend to sell, at peak concurrency.
- Launch the latency SLA first; it is the easiest to honor and the easiest for a customer to verify, which builds the credibility for the rest.
- Launch the context SLA next, as a named product, to customers currently architecting around the limitation.
- Commit to the privacy roadmap only at a pace you can actually ship, and be explicit about what is shipping versus proposed.
To learn more about improving your tokenomics and optimizing your KV cache, read these blogs: