[fideloper]

"Low maintenance", welp.

I suppose that's true in one sense - in that I'm using EKS heavily, and don't maintain cluster health myself (other than all the creative ways I find to fuck up a node). And perhaps in another sense: It'll try its hardest to run some containers so matter how many times I make it OOMkill itself.

Buttttttttt Kubernetes is almost pure maintenance in reality. Don't get me wrong, it's amazing to just submit some yaml and get my software out into the world. But the trade off is pure maintenance.

The workflows to setup a cluster, decide which chicken-egg trade-off you want to get ArgoCD running, register other clusters if you're doing a hub-and-spoke model ... is just, like, one single act in the circus.

Then there's installing all the operators of choice from https://landscape.cncf.io/. I mean that page is a meme, but how many of us run k8s clusters without at least 30 pods running "ancillary" tooling? (Is "ancillary" the right word? It's stuff we need, but it's not our primary workloads).

A repeat circus is spending hours figuring out just the right values.yaml (or, more likely, hours templating it, since we're ArgoCD'ing it all, right?)

> As an side, I once spent HORUS figuring out to (incorrectly) pass boolean values around from a Secrets Manager Secret, to a k8s secret - via External Secrets, another operator! - to an ArgoCD ApplicationSet definition, to another values.yaml file.

And then you have to operationalize updating your clusters - and all the operators you installed/painstakingly configured. Given the pace of releases, this is literally, pure maintenance that is always present.

Finally, if you're autoscaling (Karpenter in our case), there's a whole other act in the circus (wait, am I still using that analogy?) of replacing your nodes "often" without downtime, which gets fun in a myriad of interesting ways (running apps with state is fun in kubernetes!)

So anyway, there's my rant. Low fucking maintenance!

[aljgz]

"Low Maintenance" is relative to alternatives. In my experience, any time I was dealing with K8s I needed much lower maintenance to get the same quality of service (everything from [auto]scaling, to faileover, deployment, rollback, disaster recovery, DevOps, ease of spinning up a completely independent cluster) compared to not using it. YMMV.

[SOLAR_FIELDS]

It's yet another argument, like many arguments against Kubernetes, that essentially boils down to "Kubernetes is complicated!"

No, deployment of a distributed system itself is complicated, regardless of the platform you deploy it on. Kubernetes is only "complicated" because it can do all of the things you need to do to deploy software, in a standard way. You can simplify by not using Kube, but then you have to hand roll all of the capabilities that Kube just gives you for free. If you don't need a good hunk of those capabilities, you probably don't need Kube.

[ljm]

I’ve been running k3s on hetzner for over 2 years now with 100% uptime.

In fact, it was so low maintenance that I lost my SSH key for the master node and I had to reprovision the entire cluster. Took about 90 mins including the time spent updating my docs. If it was critical I could have got that down to 15 mins tops.

20€/mo for a k8s cluster using k3s, exclusively on ARM, 3 nodes 1 master, some storage, and a load balancer with automatic dns on cloudflare.

[busterarm]

People talking about the maintenance pain of running Kubernetes have actual customers and are not running their whole infrastructure on 20€/mo.

Anecdotes like these are not helpful.

I have thousands of services I'm running on ~10^5 hosts and all kinds of compliance and contractual requirements to how I maintain my systems. Maintenance pain is a very real table-stakes conversation for people like us.

Your comment is like pure noise in our space.

[verst]

How often do you perform version upgrades? Patching of the operation system of the nodes or control plane etc? Things quickly get complex if application uptime / availability is critical.

[Bombthecat]

Yeah, as soon as you got your helm charts and node installers.

Installing is super fast.

We don't do backup of the cluster for example for that reason ( except databases etc) we just reprovision the whole cluster.

[zzyzxd]

But you are not talking about maintaining Kubernetes, you are talking about maintaining a CI/CD system, a secret management system, some automation to operate databases, and so on.

Instead of editing some YAML files, in the "old" days these software vendors would've asked you to maintain a cronjob, ansible playbooks, systemd unit, bash scripts...

[eddythompson80]

Yeah, they are basically DIY-ing their own "cloud" in a way, which is what kuberetes was designed for.

It's indeed a lot of maintenance to run things thins way. You're no longer operationalizing your own code, you're also operating (as you mentioned) a CI/CD, secret management, logging, analytics, storage, databases, cron tasks, message brokers, etc. You're doing everything.

On one (if you're not doing anything super esoteric or super cloud specific) migrating kubernetes based deployments between clouds has always been super easy for me. I'm currently managing a k3s cluster that's running a nodepool on AWS and a nodepool on Azure.

[turtlebits]

Sounds self inflicted. Stop installing so much shit. Everything you add is just tech debt and has a cost associated, even if the product is free.

If autoscaling doesn't save more $$ than the tech debt/maintenance burden, turn it off.

[ozim]

I agree with your take.

But I think a lot of people are in state where they need to run stuff the way it is because “just turn it off” won’t work.

Like system after years on k8s coupled to its quirks. People not knowing how to setup and run stuff without k8s.

[globular-toast]

It all depends what you are doing. If you just want to run simple webservers then it's certainly lower maintenance than having a fleet of named/Simpson servers to run them.

The trouble is you then start doing more. You start going way beyond what you were doing before. Like you ditch RDS and just run your DBs in cluster. You stop checking your pipelines manually because you implement auto-scaling etc.

It's not free, nobody ever said it was, but could you do all the stuff you mentioned on another system with a lower maintenance burden? I doubt it.

What it boils down to is running services has maintenance still, but it's hopefully lower than before and much of the burden is amortized across many services.

But you definitely need to keep an eye on things. Don't implement auto-scaling unless you're spending a lot of your time manually scaling. Otherwise you've now got something new to maintain without any payoff.

[pnathan]

vis-a-vis running a roughly equivalent set of services cobbled together, its wildly low maintenance to the point of fire and forget.

you do have to know what you're doing and not fall prey to the "install the cool widget" trap.

[Spivak]

The fundamental tension is that there is real complexity to running services in production that you can't avoid/hide with a better abstraction. So your options are to deal with the complexity (the k8s case) or pay someone else to do it (one of the many severless "just upload your code" platforms).

You can hide the ops person but it but you can't remove them from the equation which is what people seem to want.