[rtempaccount1]

I think what type of k8s environment you use very much depends on what you're looking to get out of it.

If it's experience deploying applications into containerized environments, then micro-k8s and k3s seem like reasonable choices, you don't really care about the setup of the underlying components, just that they present the k8s API.

If you're looking for experience of managing k8s clusters, then either the distribution you're looking to run in prod. or something like kubeadm are perhaps a better option. kubeadm is very "vanilla" in terms of how it's deployed so it's quite representative of production (on-prem) deployments, perhaps unlike k3s which makes changes to how k8s works.

If you're looking to quickly test things in k8s, I'd recommend kind as the easiest way to stand up and remove clusters quickly.

And if you're looking for something to run your home services long term, I would recommend not using Kubernetes :) (unless you have a really complex home network which might justify adding k8s to the mix)

[ClumsyPilot]

I think you are right on the money - the effort of hosting it at home is only worthwhile from educational standpoint I do try to keep things as simple as possible so that I could re-deploy the cluster quickly. Also learning to debug issues in kubernetes is a skill in it's own right.

Regarding Kubeadm - some storage solutions don't even work on K3S, or at least i don't see how you can make them work. For example EdgeFS Rook integrated CSI driver requires you to deal with feature gates.

[charlieegan3]

> if you're looking for something to run your home services long term, I would recommend not using Kubernetes

If we disregard one’s experience with Kubernetes as a factor, are there any other reasons you see to not use k8s at home?

[erinaceousjones]

I suppose it depends on what your intentions are:

* "I want to run my stuff from home indefinitely, in the background": Don't use kubernetes. You'll spend ages setting up your cluster, you'll be fighting to keep it up. Most things you'll want to run probably have an OS package and run happily as systemd services. It's unnecessary overhead, both on the hardware, and mentally. Internal certificates [used to, at least] expire after a year on kubeadm created clusters, so if you don't look at it occasionally your control plane goes down. It doesn't handle the case where all your machines are nearly at full capacity (i.e. >85% disk, or nearly all memory used) and the defaults are to kick things off nodes with "pressure" which is a huge PITA when you don't have a dynamically scalable cluster - nothing makes my blood boil than seeing 200 lines of "Evicted" in kubectl . It's designed for huge workloads in datacentres, after all. You really don't need it for hosting a single user blog and a NAS.

* "I want to set up a homelab and learn k8s": Definitely use kubernetes. You'll learn how painful and time-consuming it is to manage onprem installs, but you'll also learn a lot. A lot of packaged kubernetes solutions follow the same patterns (ingress controllers like nginx / ambassador / envoy, istio service mesh, flannel etc for VLANs, prometheus for monitoring, helm / argocd deployments, ...) so it's super useful if you need to use kubernetes at work. You'll come to realize just how much awful, awful stupid rolling-release "this was deprecated on Tuesday and all our config schemas have changed" bullshit you're protected from when using someone else's (i.e. BigCorp's cloud solution) managed kubernetes cloud thing, and screaming "NO" at anyone who starts to utter "on-prem" will become ingrained in you.

:-)

[vetinari]

Do you really have that many computers at home, so you need container orchestration for that? A single machine is capable of handling most home workloads.

So outside of playing with Kubernetes for experience, why would you do that?

[rtempaccount1]

complexity and overhead. Maintaining Kubernetes is non-trivial (IMO ofc). It has a 9 month support lifecycle, so you need to redeploy everything regularly. There are also API deprecations to deal with periodically, which if you're not keeping up with k8s developments, could be disruptive.

From a complexity perspective, you're adding more places for things to break. Instead of (say) running your apps in containers with Docker, you add pods, services and ingress as layers, which is more places for things to go wrong.