SLAP: Predicting Which KV Blocks the Model Will Want Next

Arthur Rasmusson Director of AI Infrastructure at Lightbits Labs
Arthur Rasmusson
Director of AI Architecture
November 21, 2026

RDMA is fast. That is not the hard part.

The hard part is that “fast” has to mean already in flight. Attention runs on a schedule measured in microseconds, and a block fetched when the kernel asks for it arrives far too late, no matter how good the fabric is. A tiered KV cache is only viable if you can answer a harder question than “can I read this quickly”: which blocks will be needed before anything asks for them.

That is a prediction problem, and SLAP is the predictor.

What is the KV cache? When a model reads your prompt, every token it attends to produces a key and a value vector at every layer. Those vectors are the model’s working memory of the conversation so far: they are what lets token 900,000 attend to token 12. Together they are the KV cache. It is not the model weights, which are fixed and shared across every request; it is per-session state. It grows linearly with context length, and at long context it dwarfs the weights.

Why Sequential Prefetch is Not Enough

The cheap predictor is positional: attention is walking history in order, so while working on block n, fetch n+1. For a dense pass over a contiguous history, this is most of what you need, and it costs nothing to implement.

It fails in two places that matter.

At the start of a turn. There is no n to extrapolate from. The first blocks of a resumed session are exactly the ones where a stall is most visible to the user, because they land in time-to-first-token, and positional prefetch has nothing to say about them.

When attention is not sequential. Sparse and top-k block selection are standard at long context: attention picks the blocks that matter rather than reading everything. The selected set is content-dependent and scattered. A positional predictor is not merely less accurate here; it is predicting the wrong variable entirely.

Both failure modes are most severe at exactly the context lengths where tiering matters most. So the predictor has to reason about what a block contains and how it has been used, not where it sits.

Two Signals, Because One is Not Enough

SLAP combines a similarity signal and a temporal signal. The design is worth understanding by seeing what each one gets wrong on its own.

The similarity signal: what does this block contain?

The first signal asks: which stored blocks resemble what is being processed now?

Blocks are summarized into compact fingerprints with one essential property: similar content produces similar fingerprints, so proximity in fingerprint space approximates proximity in content space. Candidate lookup then becomes a cheap neighborhood query over small fingerprints rather than an expensive comparison against block contents.

This is the signal that handles the cold start. A resumed session arrives with a prefix, and content similarity can identify relevant blocks even with no access history for that session. Positional prefetch cannot do this because there is no position to extrapolate from, but similarity doesn’t need one.

Where similarity alone fails. It is blind to recency and to how a session actually behaves. In a long-running session with heavy internal repetition, such as an agent that has read the same file a dozen times, many blocks are similar to the current context and similarity alone cannot rank among them. It answers “what is related?” and not “what is about to be used?”, and prefetching everything related is indistinguishable from not predicting at all.

The temporal signal: what has this session been touching?

The second signal asks: what has this session actually been touching, and how recently?

A decaying recency score tracks how recently each block was touched, weighting recent accesses more heavily and aging older ones out, without storing per-block access history.

Access frequency is tracked in sublinear space using a probabilistic counting structure. This matters more than it might appear: a session at 10M tokens has an enormous number of blocks, and per-block exact counters would be a significant memory cost inside the system whose job is to save memory. Trading a bounded, one-sided error for a large constant-factor space saving is the right call when the consumer is a prefetch heuristic rather than an accounting ledger.

Where temporal alone fails. It cannot predict a block that has never been accessed in this session, which is precisely the cold-start case. It predicts continuation well and resumption poorly.

Why the combination is the point

A diagram depicting two KV cache fetch signals
Agentic traffic alternates between these two regimes constantly, which is why a predictor carrying only one signal degrades on half the trace.

The two signals fail in complementary places, which is the entire argument for carrying both:

SimilarityTemporal
Cold start/resumptionStrongBlind
Long session continuationWeak, too many candidatesStrong
Sparse / top-k selectionStrong, content-drivenModerate
Cost to maintainA fingerprint per blockCompact recency + frequency state

Agentic traffic exercises both regimes constantly, which is why this workload motivated the design. An agent loop resumes (cold, needs similarity), then runs many steps over an accumulating trajectory (warm, needs temporal), then reads a file it has not touched in twenty steps (cold again, within a warm session). A predictor carrying only one signal degrades on half of that trace.

Eviction is the Other Half

Prefetch decides what comes in. Something has to decide what goes out, and a good predictor paired with a bad eviction policy performs like a bad predictor: you will spend your bandwidth re-fetching blocks you just discarded.

Two design choices are worth naming.

Capacity watermarks rather than a fixed cap. Eviction is driven by high and low watermarks: pressure above the high mark triggers reclamation down to the low mark. This is the standard approach in memory management for a reason: it batches the work and avoids thrashing at a single threshold, where every admission triggers exactly one eviction and the system spends its time in the reclaim path.

Residency scored by access history, not by a fixed window. The naive policy is a sliding window: keep the last N tokens. It is easy to implement and it is wrong for the traffic that matters, because a system prompt or a foundational document read at step 1 and attended to at every subsequent step is old and hot at once. A window evicts it on age. History-based scoring keeps it, and evicts the genuinely cold middle of the conversation instead.

The same temporal machinery that drives prefetch, the recency and frequency state, drives the eviction score. That symmetry is deliberate: admission and eviction reasoning over the same signal cannot disagree, which is the failure mode where a block is evicted and immediately predicted.

What Determines Whether It Pays Off

Prefetch is speculative, and speculation costs. Three quantities decide whether it is worth it on a given deployment:

Hit rate under prefetch. How often the block is already resident when attention asks. This is the headline quality metric.

Wasted bandwidth. Blocks fetched and never used. At single-node scale this is close to free; the bandwidth was idle. Across a fleet sharing a fabric, it is not, and this is one of the known-unmodelled areas flagged in this post, Attention Prefetch over RDMA: 10M tokens in seconds.

Tail behavior. The mean is comfortable; a miss is a stall in the attention schedule, and the p99 under realistic traffic is the number we have least to say about. It is the first thing to measure in a proof of value on your own workload.

Scope Note

This post describes the shape of the predictor: two complementary signals, where each fails, and how eviction shares the temporal machinery. The specific constructions, parameterizations and scoring function are deliberately out of scope, and nothing here should be read as a description of how any of them are implemented.

If you are evaluating this for a workload with an unusual access pattern, very high sparsity, or sessions that resume after long gaps, that is a conversation worth having directly; connect with an Inferra expert, because those are the regimes where a general-purpose predictor is least likely to be well-tuned out of the box.

About the writer
Arthur Rasmusson Director of AI Infrastructure at Lightbits Labs
Arthur Rasmusson
Director of AI Architecture