This post is adapted from “All You Need Is MMU,” Part 3 of the Tokenium series originally published on vgpu.io by Arthur Rasmusson, Director of AI Architecture at Lightbits Labs. Read the full original article.
In a multi-tenant inference platform, the wall between one customer’s conversation and another’s is a software ledger, not a hardware boundary. That is the uncomfortable starting point of Arthur Rasmusson‘s latest Tokenium essay, and it matters to anyone running shared GPU fleets: neoclouds, managed inference providers, and enterprises consolidating AI workloads.
The asset at stake is the KV cache, the keys and values a model stores for every token it has processed. As Lightbits has argued before, the KV cache is the conversation: it is a reversible encoding of the prompt, not a lossy summary. Whoever can read it can reconstruct what the user said. Rasmusson’s thesis is that the GPU’s own page tables can protect this memory, and that the first step requires no new silicon.
Every Safe Way to Share a Machine Has Been a New Layer of Translation
Computing history offers a consistent pattern. The CPU’s memory management unit let processes share a processor. Nested paging let virtual machines (VMs) share a host. The IOMMU let devices share both. Each time, the mechanism was the same: give every tenant its own address table, and have the hardware refuse to translate anything outside it.
Rasmusson calls this isolation by absence. Another tenant’s page isn’t locked; from your perspective, it simply doesn’t exist. That approach is older and cheaper than any runtime permission check.
An inference server already contains three such units: the CPU MMU, the GPU’s own MMU (the GMMU), and the platform IOMMU (the SMMU on Arm). Yet none of them protects the KV cache. Serving engines like vLLM manage KV memory in fixed-size blocks, with a per-request block table that maps logical blocks to physical ones. That table is effectively a page table, but only software enforces it. To the hardware, another request’s blocks are just memory in the same context.
What the GPU’s MMU Already Gives, and What It Doesn’t
Every engine on a GPU, from CUDA kernels to copy engines, reaches memory through the GMMU. Drawing on the Open-IOV project’s documentation of the Turing generation, Rasmusson separates what is proven from what is assumed:
- Proven: each GPU context gets its own page directory. One context cannot name another’s mappings because they are not in its tree. This per-context boundary is today’s only demonstrated hardware isolation primitive.
- Proven absent: page-table entries carry no tenant, access-control, or process-identifier field.
- Unverified: whether the GPU actually enforces the read-only bit. That bit is the key to safely sharing a common prompt prefix across tenants, a major memory saving.
- Open: full invalidation behavior, and the entry formats on Ampere, Hopper, and Blackwell.
Two practical conclusions follow. The wall available now is coarse but real: one tenant per context. And sharing is the hard part. Until the read-only bit is tested, any shared page must be treated as writable by everyone who can see it.
There is also an operational subtlety. The GPU caches page-table entries in lines of sixteen, so changing a mapping is a protocol, not a single write: update the entry, then invalidate every cache that may hold the old one.
Lessons from the Arm SMMU
When a GPU reaches system memory, it passes through two independent translation stages: the GMMU, programmed by the GPU driver, then the SMMU, programmed by the hypervisor. The SMMU already solved the problem the GMMU now faces: many requesters, one memory, each owed its own view. Rasmusson draws four lessons from it:
- Name the requester in hardware. The SMMU identifies who is asking before deciding what they may reach, using an ID the hardware supplies rather than one software claims. On the GPU, the channel and its instance block can play that role.
- Keep protection domains in memory, not registers. SMMUv2’s register-based contexts cap out in the hundreds or low thousands. SMMUv3 moved that state into in-memory tables and supports roughly a million address spaces per device. With AI agents numbering in the thousands, scale must be designed in.
- Compose the stages. A platform-owned outer table plus a tenant-owned inner table, where every access must pass both.
- Refuse without killing. SMMUv3 can park a faulting transaction and let software resume or terminate it. That is the difference between blocking a stray access and taking down the tenant who made it.
A Software-First Roadmap, and the One Metric That Matters
The proposal scales across three granularities: a page directory per tenant (deployable now), channels per agent (so one agent’s faults are charged only to that agent), and a protection tag on every KV block (research). The long-term hardware wish list includes pages sized to match KV blocks, a protection domain on every page, verified read-only aliases for shared prefixes, and a single invalidation that spans CPU, SMMU, and GMMU caches. Each has precedent in existing CPUs or SMMUs.
Much of this can be built and measured before any chip changes. The first design needs no guest driver modification: the serving engine reserves an address range per agent, and the GPU mediation layer checks at map time and at submit time that each agent’s work touches only its own range. A stray access finds no entry in the offending agent’s tree, faults, and is charged to that agent alone. The roadmap orders the work carefully: key all state by owner, prove two guests apart, finish invalidation, build per-agent ranges, then probe the read-only bit under supervision before enabling any sharing.
Across all of it, one number is non-negotiable: reads across a tenant boundary must be zero, in every run, under attack, and as performance improves. Throughput, memory savings, and latency are all tradeable. Isolation is not.
Where Lightbits Fits
The KV cache rarely stays in GPU memory. As context windows grow and agentic workloads multiply, providers increasingly offload KV cache to a shared tier so it can be reused rather than recomputed. That makes the isolation question bigger, not smaller: the conversation now lives on the GPU, across the fabric, and in storage.
Lightbits builds for exactly that tier. Inferra™ by Lightbits is a KV-cache acceleration engine for LLM inference, built on Lightbits’ software-defined, multi-tenant NVMe storage foundation. For infrastructure and platform leaders, the principle Rasmusson applies to page tables should apply to every layer the KV cache touches: each tenant’s cache addressable only by that tenant, with isolation designed into the data path rather than bolted on in a scheduler’s bookkeeping.
The economic and security cases point the same way. Reusing KV cache cuts GPU cost per token; doing it safely across tenants is what lets a neocloud or managed inference provider actually capture that saving.
Key takeaways
- The KV cache is sensitive data. It encodes the conversation itself, and today it is separated between tenants mostly by software bookkeeping.
- Isolation by absence is the proven model. Every prior generation of safe sharing, from processes to VMs to devices, relied on per-tenant translation tables enforced in hardware.
- A real wall exists today. One tenant per GPU context, one page directory per tenant, is deployable now without new silicon.
- Sharing is the open risk. Safe cross-tenant prefix reuse depends on a read-only bit whose enforcement hasn’t been verified.
- Start in software, measure everything. Per-agent address ranges, checked at map and submit time, can be built and tested this year.
- Zero cross-tenant reads is the only acceptable result. Every other metric is negotiable.
Read the complete article, including GMMU page-walk details, SMMUv2/v3 internals, and the full Tokenium series, at vgpu.io.