Every inference provider quotes the same headline numbers — GPU count, peak FLOPs, HBM capacity — as if silicon alone determined economics. It doesn’t. The number that actually sets the price of a token is effective utilization: the fraction of a GPU’s theoretical capability that is converted into useful, billable work over time. In production LLM serving, that fraction is routinely 20–40%, and the gap is rarely where people look.
This article walks through how to measure effective GPU utilization properly—beyond the misleading nvidia-smi “GPU-Util” number—across compute, memory bandwidth, and interconnect dimensions, and shows how each maps directly to tokenomics: dollars per million tokens, latency-adjusted throughput, and margin per query. We survey the classic bottlenecks (PCIe and NVLink saturation, network fabric contention, storage I/O for model and KV data), then narrow in on the bottleneck that dominates modern autoregressive serving at scale: KV cache growth colliding with the memory wall.
We examine the two symptoms this produces — GPUs stalling because data cannot reach compute in time, and GPUs burning cycles on redundant recomputation because it’s cheaper than waiting or storing — and survey the architectural responses the industry has converged on: paged attention, prefix/session caching, disaggregated prefill-decode serving, and tiered KV cache offload. We close with how Inferra’s approach to data placement and GPU pinning is built to close this gap, keeping compute fed rather than idle or duplicated.
1. The Metric Everyone Uses, and Why It Is Misleading
Ask most infrastructure teams how utilized their GPU fleet is, and they’ll cite SM occupancy or “GPU-Util%” reported by nvidia-smi. That number answers one narrow question: was any kernel running on the GPU during the sampling window? It says nothing about whether that kernel was doing useful work relative to what the hardware can actually do.
The metric that matters is Model FLOPs Utilization (MFU): the ratio of FLOPs actually delivered toward the model’s forward pass to the GPU’s theoretical peak FLOPs, over the same time window. A GPU can show 99% “GPU-Util” while running at 15% MFU — fully busy, mostly wasting cycles on small, memory-starved matrix multiplies during autoregressive decode. Alongside MFU, three companion metrics complete the picture:
- Memory Bandwidth Utilization (MBU) — actual HBM bytes moved versus peak bandwidth. Decode-heavy inference is almost always memory-bound, not compute-bound, so MBU is frequently the more honest number.
- Tokens per GPU-second, at a fixed latency SLA — throughput without a latency constraint is meaningless; a batch size that maximizes tokens/sec but blows past your p99 latency target isn’t usable capacity.
- Effective vs. nominal cost per million tokens — nominal cost assumes 100% utilization; effective cost divides your actual cloud/hardware spend by tokens actually served, which is the number that shows up in a P&L.
Effective utilization, properly measured, is the product of all of these — and it is the direct input to tokenomics. A model serving at 25% MFU doesn’t just run slow; it means your cost per token is roughly 3–4x what the hardware could support, which flows straight into either compressed margins or an uncompetitive price per token.
2. The Bottleneck Menu: It’s Rarely Just the GPU
Before narrowing to the dominant failure mode, it’s worth being honest about the full list of places tokens die on the way to being generated:
- PCIe and NVLink saturation — moving activations, KV blocks, or model shards between GPUs or between host and device competes for finite interconnect bandwidth, especially in tensor-parallel and pipeline-parallel configurations.
- Network fabric contention — in disaggregated or multi-node serving, RDMA/NVLink-fabric congestion between prefill and decode nodes, or between routing/scheduling layers, adds latency that shows up as downstream GPU idle time.
- Storage I/O for model weights and context — cold model loads, checkpoint restores, and increasingly, KV cache and prompt-cache reads/writes to external memory tiers —compete with the compute path.
- Scheduling and batching inefficiency — poor continuous-batching implementations leave GPUs waiting on stragglers, fragment memory, or force smaller batches than the hardware could otherwise support.
- Power and thermal throttling — real but comparatively minor at data-center scale with modern designs; worth ruling out but rarely the dominant term.
Each of these deserves engineering attention, and a mature observability stack should instrument all of them. But in production LLM inference at any meaningful scale, one failure mode has grown to dwarf the others as context windows, concurrent sessions, and multi-turn conversation lengths have exploded: the KV cache.
3. The Real Culprit: KV Cache Growth and the Memory Wall
Every token generated autoregressively requires attention over all prior tokens’ key and value projections — the KV cache. Unlike model weights, which are fixed, the KV cache grows linearly with context length and the number of concurrent sequences a server handles. A single long-context session, or a modest number of concurrent multi-turn conversations, can require KV cache footprints many times larger than the model weights themselves.
This creates the modern memory wall for inference: GPU compute has grown far faster than memory bandwidth and capacity, so the constraint on how many tokens a GPU can serve per second is no longer “how fast can it multiply matrices” — it’s “how fast can it get the right KV blocks in front of the compute units, and how much can it hold in HBM before something has to move, evict, or recompute.”
This bottleneck manifests as two distinct and expensive failure modes:
- Stalls — compute waiting on data. When the KV cache for an active sequence doesn’t fit in HBM, or when a batch scheduler has to fetch cache blocks from host memory, another node, or slower storage, the GPU’s compute units sit idle mid-generation. This is pure waste: silicon drawing power, occupying capacity, producing nothing, while the data path catches up.
- Redundant recompute — compute wasted on work already done. The more insidious failure happens when systems choose not to store or fetch cache and instead recompute it. This shows up in several forms: re-running the prefill pass for a multi-turn conversation because the prior turn’s KV cache wasn’t retained; recomputing shared system-prompt or RAG context prefixes across thousands of concurrent requests that all begin with the same tokens; and repeating speculative-decode draft passes because verification and draft paths aren’t sharing cached state efficiently. Each of these is, in effect, paying full compute price for an answer the system already computed once — a direct, measurable, and entirely avoidable tax on tokenomics. At scale, this recompute tax is often the single largest source of “phantom” GPU demand: fleets are sized not for the unique work being done, but for the unique work plus everything being redundantly redone.
Both failure modes point to the same root cause: data is not arriving at compute fast enough, cheaply enough, or from a persistent enough location, so the system either waits or recomputes.
4. How the Industry Is Responding
A cluster of architectural patterns has emerged specifically to attack this problem:
- Paged attention / block-based KV management — treats the KV cache like virtual memory, allocating it in fixed-size pages so it can be shared, evicted, and reused without fragmentation, dramatically improving effective HBM utilization.
- Prefix and session caching — recognizes that many requests share a common prefix (system prompts, few-shot examples, RAG context) and caches the KV state for that prefix once, serving it to every subsequent request that shares it rather than recomputing.
- Disaggregated prefill/decode serving — separates the compute-bound prefill phase from the memory-bandwidth-bound decode phase onto different GPU pools, sized and scheduled independently, so neither phase starves the other of the resource it actually needs.
- Tiered KV cache offload — pushes cold or lower-priority KV cache out of HBM to CPU DRAM, and further to fast NVMe or pooled network storage, with the goal of making that tier fast enough that eviction doesn’t become the next stall.
Each of these narrows the gap between nominal and effective utilization. But they share a common dependency: they only work as well as the data path beneath them. Paging, offload, and prefix reuse all assume that when a cache block needs to come back, it comes back fast enough not to stall the GPU that’s waiting on it — and that assumption is exactly where most implementations quietly fail, because they’re built on storage and networking layers that weren’t designed for this access pattern.
5. Where Inferra Fits
Inferra is built around a simple observation: closing the utilization gap described above is fundamentally a data-placement problem, not just a scheduling or attention-algorithm problem. Inferra, with its intelligent predictive algorithms, proactively fetches the required KV blocks to the HBM from different memory tiers, creating the illusion of an infinite HBM memory pool. In practice, that means fewer stalls waiting on cache fetches, and less need to fall back to recompute as the cheaper-seeming option, because fetching the right data is no longer the slow path. This delivers tremendous improvements in the key metrics the AI industry cares about — low-latency to output tokens, high token throughput, improved GPU effective utilization, and more simultaneous active sessions with better SLAs.
The result isn’t a new attention algorithm or a new scheduler — it’s the foundation those techniques need to actually deliver the utilization gains they promise on paper. For neoclouds and hyperscalers whose margins are set at the token level, that foundation is where tokenomics is actually won or lost.
Measure the efficiency of your own data infrastructure with our Pod Efficiency Analyzer; try it now.