Right-Sizing Private Inference for the Enterprise

Arthur Rasmusson Director of AI Infrastructure at Lightbits Labs
Arthur Rasmusson
Director of AI Architecture
November 16, 2026

If you are a bank, a hospital system, a government department, or anyone else running inference on your own infrastructure, you are solving a different problem than a token seller, and most tooling in this space is built for the token seller.

They are maximizing revenue per GPU. You are not selling tokens. Your objective is the smallest cluster that serves peak load with acceptable headroom, because every GPU beyond that is capital you didn’t need to spend and power you didn’t need to contract.

What is the KV cache? When a model reads your prompt, every token it attends to produces a key and a value vector at every layer. Those vectors are the model’s working memory of the conversation so far: they are what lets token 900,000 attend to token 12. Together they are the KV cache. It is not the model weights, which are fixed and shared across every request; it is per-session state. It grows linearly with context length, and at long context it dwarfs the weights. Throw it away and the only way to get it back is to run the entire prompt through the model again. That re-run is called prefill.

A Different Shape of Question

Switch the calculator at open.lightbits.ai into Private inference mode and it changes shape. Revenue, margin, price rank, and marketplace capture all disappear, because none of them are decisions you make. What leads instead is total cost of ownership against a served-load requirement.

This matters because optimizing for the wrong objective produces genuinely wrong answers, not just irrelevant ones. A configuration that maximizes tokens per dollar will happily run at 95% utilization, which is excellent for a seller and unacceptable for an internal platform where a latency regression at 9am on Monday is an incident rather than a pricing signal.

Headroom is a Policy Setting, Not a Guess

The single most consequential number in a private deployment is the one nobody writes down: how much spare capacity you hold at peak.

Most organisations arrive at it by instinct: someone says “let’s add 30%” and it becomes permanent. It is worth making explicit, because it is a policy decision with a price tag, and the right value genuinely differs across services.

Set it from the questions that actually determine it:

  • What happens at saturation? If the failure mode is a slow response, you can run tight. If it is a queue that grows without bound and takes the service down, you cannot.
  • How fast can you add capacity? If a new node takes six months to procure, your headroom has to cover six months of growth. If you can burst to cloud in an afternoon, it does not.
  • How spiky is your load? Internal traffic is often much spikier than consumer traffic: everyone arrives at 9 am, and month-end is not like mid-month.
  • What is the cost of the incident? A developer tool degrading is an annoyance. A clinical decision support tool degrading is not.

Treat headroom as a number in the design document with a justification next to it. In Private mode, it is an explicit input rather than an assumption baked into someone’s spreadsheet.

The canonical case: coding agents

A diagram depicting KV cache concurrency caps
The load curve and the eviction curve peak together. That coupling, not raw capacity, is what makes the fleet larger than its compute requirement implies.

The workload that drives most enterprise private-inference sizing right now is internal coding assistants, and it is worth walking through because it is close to the worst case for cache handling and therefore the best case for fixing it.

Assume a mid-sized engineering organization:

  • 800 developers, of whom perhaps 300 are active in any given hour at peak.
  • Long sessions. An agent working in a repository accumulates context: files it has read, diffs it has produced, test output. Sessions run for hours and contexts reach hundreds of thousands of tokens.
  • Heavy reuse, by construction. Every step of an agent loop resends the accumulated trajectory. Step 20 contains all of steps 1–19.
  • Bursty. Concentrated in working hours, in one or two time zones.

Now the sizing question. On a discard-the-cache architecture, each of those 300 concurrent sessions holds its KV cache in HBM or loses it. HBM per node is finite and modest, so concurrency per node is capped by cache residency rather than by compute. You end up buying GPUs for their memory, and their compute sits underused, which is a very expensive way to buy memory.

Worse, sessions evicted under pressure come back through prefill, and prefill on a 200K-token agent trajectory is not free. At peak, when you can least afford it, the fleet starts spending its cycles rebuilding state it just discarded. The load curve and the eviction curve peak together.

Tiering the cache breaks that coupling. Concurrency stops being bounded by HBM residency, so a node carries more sessions, and the sessions that do fall out of HBM are restored rather than recomputed. The fleet stops buying GPUs to hold memory and starts buying them to do arithmetic.

A worked example, with assumptions stated. Take the 300 concurrent sessions above at an average of 250K tokens of context. Run the cache capacity arithmetic for your model and quantization to get bytes per session, multiply by 300, and compare that against HBM available for cache after weights and activations. In most realistic configurations, the required cache footprint is several times the available HBM, which is precisely why concurrency is capped and why the fleet is larger than its compute requirement implies. The same footprint against a populated NVMe tier is unremarkable.

The calculator’s agent workload table has the per-session figures for common model and quantization combinations; use those rather than the round numbers here, which are illustrative.

Sovereignty is the Other Half of the Argument

For a regulated enterprise, the reason to run privately is usually not cost. It is that the data cannot leave.

This is where the cache deserves attention it usually doesn’t get, because it expands the set of places customer content resides, and most data-handling documents haven’t been updated to reflect that. The KV cache is a lossy but invertible encoding of the prompt: published work recovers exact tokens from a cache dump. Wherever the cache is at rest in plaintext, the conversation is at rest in plaintext.

Run it yourself and the answers are yours to give: the cache is on your NVMe, in your rack, in your facility, under keys you hold. No third party to ask, no shared storage tier to reason about, and no cross-tenant surface because there is one tenant.

That is a materially easier story to take to a regulator than any assurance a provider can offer about infrastructure you cannot inspect, and it is worth writing down explicitly in the design document, because it is one of the few places where the private deployment is straightforwardly better rather than merely equivalent.

The Sizing Sequence

  1. Measure the load you actually have. Peak concurrent sessions, context-length distribution, and reuse share. Not averages; the averages will size you for a fleet that falls over at 9 am.
  2. Set headroom as a policy, with the justification written next to the number.
  3. Compute the cache footprint at peak and compare it against HBM. If it exceeds HBM, your concurrency is memory-bound and the fleet is larger than your compute needs.
  4. Size the NVMe tier for peak concurrent sessions times your retention window.
  5. Model both configurations in Private mode and compare TCO over the real depreciation life, including power at your contracted rate and your facility PUE.
  6. Validate with a proof of value before committing. This is capital.

Methodology note. The calculator is a simulation parameterized by measured benchmarks, not a measurement of your fleet. Treat its output as a hypothesis worth testing, and validate with a proof of value before committing capital.

Private mode will export the whole thing as a PDF, which is generally the artifact a steering committee actually wants: the assumptions, the sizing, and the TCO comparison in one document they can take into a capital approval rather than a screenshot of a web tool.

To learn more on this topic, read my post, The KV cache is the conversation, which covers the cache-as-content argument in full, and Deploying Inferra on Kubernetes, for the build-out

About the writer
Arthur Rasmusson Director of AI Infrastructure at Lightbits Labs
Arthur Rasmusson
Director of AI Architecture