Ask an inference provider what their infrastructure costs and you will almost always get a number denominated in GPUs: dollars per GPU per hour, or a count of H100s. Ask what they sell and the answer is denominated in tokens. The gap between those two units is where inference businesses quietly lose their margin, and closing it changes which investments look sensible.
What is the KV cache? When a model reads your prompt, every token it attends to produces a key and a value vector at every layer. Those vectors are the model’s working memory of the conversation so far: they are what lets token 900,000 attend to token 12. Together they are the KV cache. It is not the model weights, which are fixed and shared across every request; it is per-session state, it grows linearly with context length, and at long context it dwarfs the weights. Throw it away and the only way to get it back is to run the entire prompt through the model again. That re-run is called prefill, and it is the single largest avoidable cost in inference today.
The unit that actually matters
$/GPU/hr is a procurement unit. It tells you what you pay a vendor. It tells you almost nothing about whether you have a business, because it is silent on the only question that matters: how many sellable tokens does an hour of that GPU produce?
The unit to run your business on is fully loaded cost per million tokens. Fully loaded means:
- Amortized capex: the card, the chassis, the network, the storage, over its real depreciation life, not its warranty period.
- Power: at your actual contracted rate, not a national average.
- PUE: the facility multiplier on that power. A PUE of 1.5 means half again on every watt the GPU draws.
- Facilities and staff: rack space, cooling capex, the people who replace the cards.
Two providers with identical $/GPU/hr can have cost-per-million-tokens numbers that differ by more than a factor of two, because the denominator is a function of what the fleet does with its hours, not what the hours cost.
And here is the uncomfortable part: on long-context and returning traffic, a large share of those hours produces zero sellable tokens. Prefill on a session the fleet has already seen generates no output the customer pays for. It is pure denominator destruction. You bought an hour of GPU, you spent forty minutes of it recomputing state you previously had, and you sold the tokens from the remaining twenty.
What the Multiple Depends On
The improvement is not a single number. It is a function of context length, and knowing which regime your traffic sits in matters more than any figure we could quote.
Prefill computes attention between every pair of tokens in the prompt, so its cost carries a term that grows with the square of the context. Restoring a cache does not work that way: a KV cache is linear in tokens, so twice the context is twice the bytes to read back. Recompute is superlinear, restore is linear, and the gap between them widens with every additional token of context. Published joint results with FarmGPU and ScaleFlux put time-to-first-token improvements between 100x and 280x across models from 131K to 1M tokens, and beyond 1000x at 10M tokens.
The same length that drives the ratio also decides how much of it reaches the whole turn. If prefill is a fraction p of turn time and it becomes S times faster, the turn improves by:
1 / ((1 - p) + p/S) |
As p approaches 1, that converges on S itself. That is exactly what happens at long context: prefill is almost the entire turn, so the whole-turn improvement lands in the same order as the prefill improvement. At short context, p is small, decode dominates, and the same restore moves the total very little.
| What a turn looks like | Prefill share of turn time | Whole-turn improvement |
|---|---|---|
| Short prompt, long answer | 20% | 1.25x |
| Mixed traffic | 50% | about 2x |
| Long context, hundreds of thousands of tokens | 90% | about 9x |
| Multi-million-token sessions | 99% | about 50x |
Decode is untouched throughout: the same tokens through the same weights at the same rate. Everything here moves the prefill share and nothing else, which is why your context-length distribution is the single most important input to the model.
Even the modest end of that table is worth having, because it applies to every GPU-hour in the fleet continuously and lands entirely on the denominator of the unit you sell in. At the long-context end, it stops being an efficiency argument at all and becomes a feasibility one: the session either runs or it does not.
Reuse share is the second lever and it is workload-dependent. Agentic traffic, which replays a growing trajectory on every step, has very high reuse and accumulating context, so it sits high on both. One-shot classification of short inputs sits low on both. Know which you are before you model anything.
Where the Margin Actually Comes From
Three distinct mechanisms, worth separating because they compound differently.
1. Recovered GPU-time becomes sellable tokens. This is the direct one. Prefill hours that were producing nothing become decode hours that produce billable output. Fleet size is unchanged; sellable output goes up. On a fleet that was spending 30% of its cycles on recomputation, this is roughly a 30% capacity increase with no capex.
2. Higher tenant density. Freeing HBM from cache duty and moving it down the tier ladder means more concurrent sessions per node. FarmGPU measured 3x more requests served on the same GPUs. Density is the mechanism that turns a latency improvement into a margin improvement, because it changes how many customers a node can carry at once.
3. Products your competitors cannot list. This is the most valuable and the least modelled. A 1M-token or 10M-token session is not a faster version of a product your competitors offer. On cache-regeneration architectures, it is a product they physically cannot serve on the same hardware, because an hour and forty-two minutes of prefill per session is not a service; it is an outage. Being the only listing in a category is worth more than being 40% cheaper in a crowded one.
Market Floor Dynamics
If you sell on an open marketplace, your price is not set by your costs. The cheapest credible listing sets it, and your costs only determine whether you can survive there.
This produces two regimes, and it is worth knowing which one you are in:
At or near the floor. The market sets the price; your cost advantage converts directly into margin. A 40% cost reduction at a market-set price is a 40-point improvement in contribution margin on that traffic. You are not competing on price; you are simply keeping more of it.
Above the floor. You are charging a premium for something (latency guarantees, context length, privacy posture) and demand is elastic. Here the cost advantage buys you the option to defend the premium or to undercut, and the right move depends on whether your differentiation is defensible. Cost advantage alone is not differentiation; it is the ability to fund differentiation.
The trap in marketplace selling is optimizing a model you can serve profitably, but nobody routes traffic to. Demand comes first. A perfect margin on zero volume is zero.
Re-Deriving Your Own Unit Cost
Rerun your own numbers in the right unit. Specifically:
- Compute fully loaded cost per million tokens today: capex amortization, power at your rate, PUE, facilities.
- Estimate what fraction of your GPU-hours goes to prefill on traffic you have already seen. This is usually the number nobody has.
- Work the whole-turn effect from your own context-length distribution and reuse share, not ours.
- Ask what that cost reduction is worth in your regime: margin if you are at the floor, funded differentiation if you are above it.
The Revenue, margin & demand panel in the calculator at open.lightbits.ai does steps 1, 3, and 4 with your inputs, and ranks your list price against live marketplace listings so you can see which regime you are actually in.
Methodology note. The calculator is a simulation parameterized by measured benchmarks, not a measurement of your fleet. Treat its output as a hypothesis worth testing, and validate with a proof of value before committing capital.
Interested in learning more about this topic, read the blog: The memory wall is now a memory bill