HN Reader

I tried longhorn on my homelab cluster. I'll admit it's possible that I did something wrong, but I managed to somehow get it into a state where it seemed my volumes got permanently corrupted. At the very least I couldn't figure out how to get my volumes working again.

When restoring from backup I went with Rook (which is a wrapper on ceph) instead and it's been much more stable, even able to recover (albeit with some manual intervention needed) from a total node hardware failure.

Kubernetes CSI drivers are surprisingly easy to write. You basically just have to implement a number of gRPC procedures that manipulate your system's storage as the Kubernetes control plane calls them. I wrote one that uses file-level syncing between hosts using Syncthing to "fake" network volumes.

https://kubernetes-csi.github.io/docs/developing.html

There are 4 gRPCs listed in the overview, that literally is all you need.

Longhorn was the codename for Windows Vista... so not a great choice of a name (IMO).

(Copied from[0] when this was posted to lobste.rs) Longhorn was nothing but trouble for me. Issues with mount paths, uneven allocation of volumes, orphaned undeletable data taking up space. It’s entirely possible that this was a skill issue, but still - never touching it again. Democratic-csi[1] has been a breath of fresh air by comparison.

[0] https://lobste.rs/s/vmardk/longhorn_kubernetes_native_filesy... [1] https://github.com/democratic-csi/democratic-csi

Go with Ceph… a little more of a learning curve but overall better.

i am using nfs and i think its pretty simple and just works

As an Enterprise user of Rancher, we had long discussions with Suse about Longhorn. And we are not using it.

You need a separate storage lan, a seriously beafy one at to use Longhorn. But even 25Gbit was not enough to keep volumes from being corrupted.

When rebuilds take too long, longhorn fails, crashes, hangs, etc, etc.

We will never make the mistake of using Longhorn again.

Anyone knows what's the story with NVMEoF/SPDK support these days? A couple years ago Mayastor/OpenEBS was running laps around Longhorn on every performance metrics big time, not sure if anything changed there...

Be aware of its security flaws -- https://github.com/longhorn/longhorn/issues/1983

Allowing anyone to delete all your data is not great. When I found this I gave up on Longhorn and installed Ceph.

Longhorn is a poorly implemented distributed storage layer. You are better off with Ceph.

For homelab uses, I've been enjoying Linstor/Piraeus a lot more than longhorn lately. Less issues overall so far and simpler.

Where I work, we primarily use Ceph for the a K8s Native Filesystem. Though we still use OpenEBS for block store and are actively watching OpenEBS mayastor