What I learned wiring VMs to an NVMe/TCP cluster — so you don’t have to

The goal was simple: run an AlmaLinux VM on Harvester, with its disk living on a Lightbits NVMe/TCP cluster. Then pile on everything I’d actually want in production: snapshots, cloning, multiple tenants, live migration, QoS, compression, replication. The full works.

It did not go simply at first. But here’s the good news: every wall I hit had a clean, repeatable fix — and now the whole thing just works. What follows is the friendly tour of what broke and why, followed by a white paper you can download and follow step by step.

The payoff, up front

Before the war stories, here’s where it landed. Three-node Harvester, three-node Lightbits, VMs booting from Lightbits over NVMe/TCP. Snapshots and clones both working straight from the UI. Per-tenant storage isolation, QoS, and compression all in place. And the crowd-pleaser: live migration moving running VMs between nodes with zero downtime.

All of it reproducible from the white paper. Now, how we got there. 🙂

The greatest hits (and their one-line fixes)

None of these were hard once I understood them. Each one is a story of an error message that pointed one way while the real problem sat a layer beneath it.

1. “Unschedulable — insufficient memory” on an empty node

A units slip, nothing more.

Memory had been set to a giant MiB value. Switch to sane units like 2Gi and it schedules immediately. Move on.

2. A scary certificate error, once a minute

Looks like the culprit. Isn’t.

It’s cosmetic log noise in KubeVirt 1.7 — the VMs boot fine despite it. The lesson: don’t chase the loudest error, chase the real one.

3. VMs stuck starting

The image was the problem.

The AlmaLinux image had been uploaded, and after a reboot Longhorn couldn’t rebuild it. The fix is a one-line habit: import images by URL instead of uploading them. URL-based images heal themselves after reboots, which quietly removes a whole class of pain.

4. “VM Paused due to IO error” the moment I wrote to disk

A block-size mismatch.

Lightbits volumes are 4K-native, but the guest was seeing 512-byte blocks — so writes didn’t line up. Present the disk at 4K with blockSize.matchVolume, set before you install. Bake it into a template and forget about it.

5. “No bootable device” on every reboot

Wrong firmware.

The VM was installed as BIOS; it needed UEFI. Enable EFI first, reinstall with the Lightbits disk as the only target, drop the ISO — and it boots every time. Pick your firmware before you install, not after.

6. Snapshots worked in the CLI but not in the UI

Harvester just needed to be told.

The backend was fine all along. Add a VolumeSnapshotClass for Lightbits, register the driver in csi-driver-config, annotate the StorageClass — and the snapshot button appears.

7. A second tenant’s volume stayed Pending

Three things have to agree.

The JWT’s project, the StorageClass project-name, and the actual Lightbits project (with its qos-policy-name). Line all three up and each tenant gets its own isolated storage.

8. Live migration — and this one just worked 🎉

The reward for doing the plumbing right.

With RWX Lightbits disks and every node connected over NVMe/TCP, a running VM hops between hosts with zero downtime. Set evictionStrategy: LiveMigrateIfPossible and node maintenance becomes a non-event.

The one habit that saved me

Notice the pattern? Every “the VM won’t start” turned out to be storage or firmware plumbing a layer or two beneath the error I was staring at.

The trick that worked over and over: read the innermost status. The attachment pod’s events, the VMI conditions, the provisioner log, not the top-level message. The real cause is almost always down there, stated plainly, waiting for you to scroll to it.

Grab the white paper

I turned all of this into a proper step-by-step white paper: every command, the exact StorageClass and VolumeSnapshotClass manifests, the 4K and UEFI procedures, live migration, multi-tenancy, and a quick verification checklist at the end.

➤ Download: Harvester-Lightbits-whitepaper

Adapt the names to your environment, keep it next to your cluster, and if something breaks that isn’t covered, read the innermost status. It’ll tell you. Happy building! 🚀

Watch the High-Performance Storage for Harvester: Lightbits NVMe/TCP Setup Video