Recently, customers informed us that they are moving from their existing hypervisor environment to vCluster. The reasons are the usual mix of technology challenges: two separate stacks for VMs and containers, network challenges, tenants asking for more independence, and license renewals that arrive with the warmth of a tax audit.
“How easy is it to run vCluster with KubeVirt on Lightbits software-defined storage, with live migration on block storage, and fully multi-tenant?”
That’s four buzzwords in one sentence, which in our line of work is either a sales pitch or a very good lab weekend. We chose the lab. This is what happened.
The Wish List
The customer’s requirements fit on a sticky note, which is always suspicious:
- VMs on Kubernetes. Their existing VMs have to keep running, just not on the old hypervisor.
- Multi-tenant, for real. Every tenant gets its own cluster, not “a namespace and a stern email about quotas”.
- Live migration on block storage. Patching a host must not mean rebooting someone’s database at 2 a.m.
- One storage platform. No local disks scattered across compute nodes, no separate SAN just for VMs, and no NFS.
The ingredients we picked: vCluster with private nodes for the tenant clusters, KubeVirt to run VMs as Kubernetes workloads, and Lightbits NVMe/TCP for storage. The whole thing ran on three nodes, imaginatively named vcluster-1, vcluster-2 and vcluster-3 for the vCluster and three nodes from Lightbits.
The Magic Trick: Moving a Running VM
Here’s the part that makes infrastructure people a little emotional. A Lightbits block volume in ReadWriteMany mode can be attached to two nodes at the same time. So when a VM live-migrates, the disk doesn’t go anywhere. Only memory and CPU state travel across the network.
In the lab, our test VM, an Ubuntu 24.04 cloud image with a 10 GiB Lightbits disk, sat happily on vcluster-3. One command later:
| Shell virtctl migrate test-vm kubectl get vmi test-vm -o wide # NODE: vcluster-1 |
The VM was on vcluster-1, still running, with no reboot. It’s the infrastructure equivalent of changing seats on a train without spilling your coffee. The best part is that a node drain now does this automatically for every VM on the node, so maintenance stops being an event and becomes a Tuesday.
Plot Twists (every good lab has them)
It wouldn’t be an honest blog without the bits that didn’t work the first time.
Plot twist 1: “Permission denied.” Our first disk import crashed on repeat, with the log politely stating blockdev: cannot open /dev/cdi-block-volume: Permission denied. The importer runs as a non-root user, and containerd doesn’t hand raw block devices to non-root pods by default. One line fixes it in the file /etc/containerd/config.toml.
| None device_ownership_from_security_context = true |
Plot twist 2: the node we forgot. We fixed containerd, restarted it and deleted the importer pod. The import failed again, which is when you start questioning your career choices. The Kubernetes scheduler, with impeccable comic timing, had put the new importer on vcluster-3, the one node we hadn’t fixed yet. Lesson learned: “every node” means every node.
Plot twist 3: “What’s the password?” Once the VM booted, the question came up that every VM story eventually reaches. The answer for Ubuntu cloud images is user ubuntu, and whatever you put in cloud-init. For anything beyond a lab, the answer should be “there is none; use SSH keys”.
None of these were Lightbits problems or vCluster problems, just the normal friction of putting several good pieces of software together. And each is now a single line in the deployment checklist.
Multi-Tenancy: Good Fences Make Good Neighbors
Multi-tenancy is one of those words that means “separate floors in one building” to some people and “a shared kitchen with labeled yogurt” to others. The customer wanted separate floors.
So every tenant gets a matching pair:
| Layer | Tenant A | Tenant B |
|---|---|---|
| Kubernetes | Its own vCluster, with its own API server | Its own vCluster, with its own API server |
| Compute | Its own private nodes | Its own private nodes |
| Storage | Lightbits project tenant-a | Lightbits project tenant-b |
| Credentials | A JWT that only works for tenant-a | A JWT that only works for tenant-b |
On the Lightbits side, a project is a tenant. Each tenant cluster stores its own token as a Kubernetes Secret, and its StorageClass points to its own project. Tenant A can create, attach and live-migrate as many volumes as it likes, but it cannot even see Tenant B’s volumes, let alone attach one.
Tenants share a Lightbits storage cluster, but not a single credential. Optional per-tenant QoS policies make sure one enthusiastic tenant running a benchmark doesn’t turn everyone else’s Monday into a support ticket.
Conclusion: So, how easy is it?
The honest answer to our customer’s question is: easier than they expected, and the hard parts come down to three settings you only have to learn once.
Why it’s a good fit technically:
- vCluster gives each tenant a real cluster, with its own API server, nodes and operators, without the cost of a physical cluster per tenant.
- KubeVirt puts VMs and containers on one platform, managed with one set of tools.
- Lightbits SDS provides what live migration can’t do without: a fast block volume that two nodes can attach at the same time over plain Ethernet.
What it means for the business:
- One platform instead of two. One stack to run, secure, staff and budget for, instead of a hypervisor plus a Kubernetes platform.
- Maintenance without maintenance windows. Patching and hardware work move VMs instead of stopping them, so you no longer have to negotiate downtime with every tenant.
- Tenants as a product. Isolated clusters with their own storage scope make multi-tenant hosting something you can offer and bill, not just something you tolerate.
- Storage that scales separately. Lightbits SDS grows storage independently of compute, so you buy capacity where you need it instead of disks in every server.
- A clear exit from legacy virtualization. Existing VMs move over and keep their live migration, while new workloads land as containers on the same platform.
Planning a similar move? Download the tech paper with a full reference architecture, including multi-tenant Lightbits projects and the complete deployment steps. Reach out to talk with an expert, and we’ll walk you through it.
