Serious inference operators tend to be power-constrained before they are capital-constrained. You can buy the GPUs. Getting the megawatts, the cooling, and the interconnect approved on a timeline that matters is the hard part. Which makes a question worth asking precisely: of the power your fleet draws today, how much of it is spent computing something it already computed?
What is the KV cache? When a model reads your prompt, every token it attends to produces a key and a value vector at every layer. Those vectors are the model’s working memory of the conversation so far: they are what lets token 900,000 attend to token 12. Together they are the KV cache. It is not the model weights, which are fixed and shared across every request; it is per-session state, it grows linearly with context length, and at long context it dwarfs the weights. Throw it away and the only way to get it back is to run the entire prompt through the model again. That re-run is called prefill, and it is the single largest avoidable cost in inference today.
The Energy Footprint is GPU-Hours, and a Large Share of Them are Repeats
Inference energy is overwhelmingly dominated by GPU draw. Everything else in the node (CPU, networking, storage, fans) is a rounding error against eight accelerators under load. So the energy question reduces to a utilization question: what are the GPU-hours doing?
On reuse-heavy traffic, a substantial fraction of them are recomputing KV cache the fleet previously held and discarded. That work is thermodynamically real. It draws full power, produces heat you must remove, consumes facility capacity you had to get approved, and produces no token the customer pays for and no information the fleet did not already have.
This is the cleanest efficiency argument in the stack, because it does not require anything to get faster. It requires something to stop happening twice.
Density is the Efficiency Mechanism
The instinct in efficiency work is to chase performance per watt on the chip. That is a hardware vendor’s lever; it moves on silicon cycles, and you do not control it.
The lever you do control is how many tenants a node carries at once. FarmGPU measured 3x more requests served on the same GPUs with the cache tiered rather than discarded. Same cards, same power envelope, same facility footprint, three times the work.
Stated as an efficiency figure, that is a 3x improvement in requests per kilowatt-hour, achieved with no change to the silicon. There is a reasonable argument that the greenest GPU available to you is the one you already own, running at three times the tenant density.
The mechanism is straightforward. Freeing HBM from cache duty leaves more of it available for concurrent sessions. More concurrent sessions per node means fewer nodes for the same offered load. Fewer nodes mean less power, less cooling, and less facility.
The Metric: Tokens Per Kilowatt-Hour
If you are power-constrained, your binding constraint is not dollars and not rack units. It is watts. The metric that should drive your architecture decisions is therefore tokens per kWh: how much sellable output you extract from each unit of energy you get approved.
This is a better yardstick than performance per watt for two reasons. It is denominated in the thing you sell rather than the thing you compute, so it captures wasted work rather than hiding it. It’s measured at the facility boundary, so it includes PUE, meaning it prices the cooling consequences of a decision, not just its compute consequences.
The calculator at open.lightbits.ai reports tokens/kWh directly. Give it your facility power draw and your PUE and it will tell you what your energy actually buys, in the unit you sell.
The Power-Cap Optimizer
For power-constrained sites, the useful question is inverted. Not “what does this cluster cost in watts?” but “given that I have N megawatts and cannot get more this year, what is the best cluster I can build inside it?”
That is a constrained optimization, and the calculator will run it. Give it a kilowatt budget, and it designs the cluster within the envelope, selecting model, quantization, and hardware mix to maximize sellable tokens under the cap rather than under a dollar budget.
This tends to produce different answers than a cost-first design. Under a power cap, tiering the cache is unusually attractive, because it raises the output of an envelope you are not permitted to expand.
Be Honest About the Net: Storage Draws Power Too
An NVMe tier is not free in watts, and any efficiency claim that ignores this is not worth trusting.
A populated NVMe tier and the fabric to reach it draw real power continuously: tens of watts per drive, plus the NICs and switching. On a node that already draws several kilowatts of GPU under load, this is a small percentage, but it is not zero and it runs whether or not the cache is being read.
So the honest framing of the trade is:
- You add a persistent storage and fabric power draw, roughly constant, modest against GPU load.
- You remove a large, bursty GPU draw that produced no sellable output: the recomputation.
The net is strongly favorable on reuse-heavy traffic and can be unfavorable on traffic with no reuse at all, where you have added storage power and removed nothing. A fleet serving purely one-shot requests with no returning sessions should not expect an efficiency gain here, and we would rather say that plainly than let you discover it after the capex.
The determining variable, again, is reuse share. Know yours before modelling.
There is also a second-order effect worth noting: recomputation is bursty, and bursty draw is harder to cool and harder to provision for than steady draw. Replacing a spiky GPU load with a flat storage load tends to improve the shape of your power profile, not just its integral. Whether that matters depends on how close you run to your headroom.
Establishing Your Own Numbers
- Establish your reuse share. This determines everything downstream and most operators have never measured it.
- Compute your current tokens/kWh at the facility boundary, PUE included.
- Run the power-cap optimizer with your real kW ceiling and see what the best cluster inside it looks like.
- Include the storage tier’s power in the comparison. If a vendor’s efficiency number does not, ask why.
Methodology note. The calculator is a simulation parameterized by measured benchmarks, not a measurement of your fleet. Treat its output as a hypothesis worth testing, and validate with a proof of value before committing capital.
To learn more about improving the performance and efficiency of your GPU fleet, read: Why token economics matter more than GPU count