[s3rg4fts]

Wasn't aware of HomelabOS, looks pretty interesting!

How is your Longhorn performing? I tried setting it up on my nodes but with Gigabit networking and possibly the Pis pretty average CPUs I would get pretty awful performance both in distributed volumes (with replicas etc) but also on strict-local ones (for reasons I haven't yet figured out).

I am now considering using something like https://github.com/rancher/local-path-provisioner, since I mainly intend to use Longhorn for DBs that handle fault tolerance / backups etc on their own.

[psini]

My experience with longhorn, or rook/ceph, or any other StorageClass solution that enables read write many is that eventually, despite all the "self-healing" and "highly available" claims, it all explodes at some point.

Volumes become unmountable, pools stay yellow forever, desynchronization happens... Perhaps (most probably even) I just don't know how to configure them properly, but maybe that is telling of the difficulty to operate/maintain these solutions.

What I took away from this is to try my most to never deploy anything that requires shared persistent volumes. If I need something stateful, it needs to speak to a database or S3 backend, or something that handles redundancy at some other level than filesystem. If I really really need to have a local volume, I'll use local-path-provisoner like you said, which means pinning the pod to a single node, but really that is a concession I am willing to make to not deal with ceph/rook.

Great writeup, it's like I'm reading about my own journey managing kubernetes clusters!

Best of luck

[NickBusey]

I haven't gotten to performance testing just yet, that should happen pretty soon. Thanks for the local-path-provisioner recommendation, I'll definitely give it a go!

[s3rg4fts]

Good luck! This is pretty useful for benchmarks https://github.com/longhorn/kbench?tab=readme-ov-file