Deploying Inferra on Kubernetes: from Prerequisites to First Warm Hit

This post is the operational path: what has to be true of your cluster before you start, how to size the storage tier with actual arithmetic, and how to verify you are getting warm hits rather than assuming it.

Exact commands, chart versions, and values-file keys move between releases, so the authoritative reference is the Deployment and Usage Guide on docs.lightbits.ai. This post gives you the shape of the work and the decisions that are expensive to get wrong: the parts that don’t change between releases.

Prerequisites

Node labelling. GPU nodes, storage nodes, and the control plane get distinct labels; scheduling depends on them. Decide early whether storage is colocated with GPU nodes or on dedicated nodes. Colocated is simpler and keeps a fetch on-node; dedicated scales the tiers independently and is usually right above a handful of nodes. Changing your mind later means re-labeling and re-scheduling.

RDMA fabric. This prerequisite most often blocks a deployment, and it is worth validating before installing anything else. The requirement is not just “we have InfiniBand or RoCE”; it is that RDMA works from inside a pod, which means the device plugin is installed, the network device is exposed into the container, and the pod has the capabilities it needs. Prove this with a point-to-point RDMA test between two pods on two nodes before you deploy anything else. A fabric problem discovered after install shows up as mysterious performance, not a clear error.

NVMe. Sized per the arithmetic below, and, this matters, sized for throughput as well as capacity. The tier is read-heavy and latency-sensitive on the load path. A large, slow tier will meet your capacity number and miss your latency target.

Kubernetes version and GPU operator. Both are pinned to what the release notes specify. This is not a place to be adventurous.

Sizing the NVMe Tier

Do this arithmetic carefully, because it determines whether you can honor a retention window, and it is expensive to revise.

Step 1: bytes per token. For a full-attention model:

bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

The leading 2 are keys and values. bytes_per_element is 2 for FP16/BF16, 1 for FP8. On a hybrid model, count only the full-attention layers: state-space layers carry recurrent state, not a per-token KV history, and are not stored here.

Compute this for your actual model and quantization. It is the input everything else multiplies.

Step 2: bytes per session. bytes_per_token × context_length. Use the context length you intend to support, not your average, because the sessions that stress the tier are the long ones.

Step 3: capacity from your retention target. Two ways to get there, and it is worth computing both and taking the larger:

From concurrency:

capacity = bytes_per_session × peak_concurrent_warm_sessions

From arrival rate:

capacity = bytes_per_token × token_arrival_rate × retention_window

The first is right when sessions are long-lived and bounded in number. The second is right when sessions churn and the window is what holds them. Real traffic is a mix, which is why you take the larger and then add headroom.

Step 4: the MetaDB floor. Block metadata is small per block, but it isn’t zero, and it has a floor that doesn’t scale down. Take the figure from the sizing guide for your block size and add it. Under-provisioning metadata produces a failure that looks nothing like “storage is full”, which is what makes it worth planning rather than discovering.

Step 5: headroom. Explicit, justified, written down. Eviction at the watermark is normal; running at the watermark continuously means you are silently shortening the retention window you think you are offering.

The cache capacity panel in the calculator at open.lightbits.ai runs this for common model and quantisation combinations if you would rather not do it by hand.

Install and Verify

The install is a Helm chart with a values file. The values that matter are the ones above: node selectors, the storage tier’s size and class, the RDMA device configuration, and the model/block-size parameters that must match your vLLM configuration. Take the current chart and the current key names from the guide.

What matters more than the install is verifying it in the right order, because each layer’s failure looks like the next layer’s problem:

1. Pods healthy. All components running and passing readiness. A component stuck in readiness is nearly always a fabric or device-plugin problem, not a configuration one.
2. Fabric reachable from pods. Re-run the RDMA check from inside the deployed pods, not just from the host. Container networking is where this breaks.
3. Storage tier writable and sized as intended. Confirm the cluster’s actual capacity matches what you sized. Storage-class defaults can silently substitute something smaller.
4. vLLM sees the connector. The engine should log the connector initializing. If it does not, nothing downstream will work and everything will still appear to run, correctly and with no reuse.
5. Turn 1 stores. Send a request and confirm blocks were written. A common early failure is a configuration where loads are attempted and stores never happen, which produces a permanently cold cache that looks like a prediction problem.
6. Turn 2 hits. Send the same prompt again and measure.

The Measurement that Actually Tells You It Works

Step 6 is the only test that matters, and it has to be done carefully to mean anything.

Send an identical long prompt twice and compare time-to-first-token. Turn 1 is cold: full prefill. Turn 2 should be dramatically faster.

Three things to control for, or you will measure something else:

vLLM’s own prefix cache. If turn 2 hits the engine’s in-GPU prefix cache, you have measured vLLM, not the storage tier. Either evict between turns, or use a prompt large enough that it cannot be resident, or run enough intervening traffic to displace it. This is the most common way a first measurement is accidentally meaningless.

Prompt length. At short context, the saving is real but small, and easily lost in noise. Use a context long enough that prefill dominates the turn: hundreds of thousands of tokens if you can.

Cold-start effects. The first request after deployment pays one-time costs. Discard it.

Then check output equivalence. The streamed path is numerically identical by construction, so a greedy-decoded turn 2 should match turn 1 token for token. If it does not, that is a real finding and worth raising rather than tuning around.

Day 2

Upgrades. Release notes first, every time, specifically the section on what changed in the on-disk or on-wire format, which determines whether an upgrade is rolling or requires draining the cache.

Errata. Check it before debugging anything unusual. A known interaction with a specific vLLM version is a much more likely explanation than a novel bug, and the errata will usually have the workaround.

Watch eviction pressure, not just capacity. Utilization tells you how full the tier is. Eviction rate tells you whether you are holding the retention window you promised. A tier at 70% with a high eviction rate is under-provisioned for its workload regardless of what the capacity number says.

Watch the miss path. Hit rate is the headline, but tail latency on misses is what your users feel. Post 9 is candid that p99 under realistic traffic is the least-modelled part of the system; on your own fleet, it is measurable and worth measuring.