There’s a quiet controversy brewing in the world of enterprise storage — one that’s getting serious attention in Linux kernel communities and among storage architects. Two technical proposals from the NVMe consortium, TP8028 and TP4129, are being pushed into the Linux kernel to fix a serious data corruption risk in NVMe over TCP (NVMe/TCP) deployments. If you’ve heard about this and wondered whether your NVMe/TCP storage is safe, here’s what’s actually going on — and why Lightbits users don’t need to lose any sleep over it.
Before you read another word, here is the most important thing to understand: this problem does not affect Lightbits. Not because Lightbits has patched around it. Not because Lightbits is waiting for a fix. But because Lightbits’ architecture makes the underlying hazard physically impossible in the first place. The vulnerability exists in storage systems in which the host determines how to handle path failures. Lightbits removed that decision from the host entirely — and did so from day one. Everything that follows explains why that matters, and why other vendors are scrambling to catch up.
The pizza delivery analogy
Imagine you order a pizza (let’s call it Write #1). The delivery driver gets stuck in traffic and goes completely silent — no updates, no ETA. You wait, assume it’s lost, and place a second order (Write #2). The second driver arrives, delivers your pizza, and everything seems fine.
Twenty minutes later, there’s another knock at the door. It’s the first driver — still carrying the original order. And somehow, against all logic, he walks in and replaces the pizza you’re already eating with his older one. Your receipt shows two successful deliveries. The kitchen has no idea that anything went wrong. The damage is silent and undetected.
That’s not just a bad night for pizza. In certain NVMe/TCP storage architectures, the exact equivalent of this scenario is a genuine and serious risk — known as a write-after-write hazard.
Here’s the technical version: your server (the “host”) sends a write command over a network path to a storage array. The network link goes down. The host can’t tell whether the write has completed or is still in transit. So it does the sensible thing — it picks a different network path and resends the command (Write #2). Write #2 lands successfully and the data is updated. But then the original Write #1 finally limps through the failed path and lands after Write #2. Older data silently overwrites newer data. Your storage thinks everything is fine. Your application has no idea.
How Lightbits handles it: the second driver never knocks
Now imagine a smarter pizzeria. Same scenario — driver A gets stuck, so driver B is dispatched. But the moment driver B knocks on your door and confirms delivery, the pizzeria’s system instantly radios driver A: turn around, order cancelled, do not deliver. Driver A never reaches your door. There is no second knock. No duplicate. No confusion.
The old order is killed the instant the new one succeeds — guaranteed.
That is exactly how Lightbits works. The moment a new path is opened and confirmed, the Lightbits cluster synchronously fences all in-flight writes on the old path. The “first driver” — the stale write on the failed path — receives an immediate cancellation signal and is rejected by the cluster before it can ever land. There is no window in which both the old and new writes can simultaneously be accepted.
The industry’s fix — and why it’s painful
For storage vendors that rely on the host to make path decisions, the NVMe consortium has introduced two safety standards to address this problem:
TP4129 is the cautious approach: force the host to wait out long, mathematically safe timeouts before retrying on another path. It works, but during a failover, your applications can be left waiting for seconds — an eternity in high-performance storage.
TP8028 is the more aggressive approach: have the host actively signal the storage target to shut down the broken path before retrying. Faster, but it adds significant complexity to the host driver — and if that signal fails, you fall back to TP4129’s long timeouts anyway.
Both proposals are being championed by legacy storage vendors — vendors whose architectures rely on the host to make path decisions. Red Hat and the Linux kernel community are pushing these changes precisely to protect users of those architectures.
Going back to our analogy: TP4129 is the pizzeria telling you to wait 30 minutes before reordering, just to be safe. TP8028 is the pizzeria trying to radio the first driver — but if the radio doesn’t connect, you’re back to waiting 30 minutes anyway. Neither is elegant. Both are workarounds for a deeper architectural problem.
Why Lightbits is different
Here’s the key insight: the vulnerability isn’t in the NVMe/TCP protocol itself. It’s in who gets to make path decisions.
In legacy architectures, the host is the traffic cop. When a road closes, the host decides where to reroute traffic — and that’s where the race condition happens.
Lightbits flips this model entirely. The Lightbits cluster is the traffic cop, not the host. The host doesn’t get to decide which path to use. It doesn’t get to decide when to retry. The cluster tells it what to do, and the host follows.
When a path fails, here’s what actually happens in Lightbits:
- The cluster detects the failure and synchronously fences any in-flight writes on the failed path — cancelling them before they can ever complete.
- Only once the cluster has mathematically guaranteed that no stale writes can arrive does it signal the host to start using a new path.
- The host simply follows instructions. There is no race condition because there is no moment where both the old and new paths are simultaneously accepting writes.
The second driver never knocks. The cluster made sure of it before the first driver even left the building.
What this means for you
If you’re running Legacy storage that was never designed to use NVMe/TCP and now tries to circumvent this severe problem, your Linux administrators will need to update their kernel drivers to incorporate TP4129 and/or TP8028. This, of course, will include 100s or 1000s of hosts (clients) reboots, a challenging task. You’ll also need to test these changes and accept real trade-offs — either longer failover times or added driver complexity.
If you’re running Lightbits, none of that applies to you. Lightbits’ architecture inherently prevents the write-after-write hazard. There’s no timeout to configure, no cross-controller reset handshake to worry about, and no latency penalty during failover.
The broader message is that NVMe/TCP is not the problem. The protocol itself is sound. What matters is the architectural decision of who controls path management — and Lightbits made the right call from the start by putting that control in the cluster, where it belongs.
For storage architects who want to go deeper: the relevant kernel discussions center on the nvme-tcp driver changes for CCR (Cross-Controller Reset) in TP8028, and the KATO/CQT timeout mechanics in TP4129. Lightbits’ target-driven state transitions make both irrelevant at the architectural level.
