[itsnowandnever]

etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. much of the "secret sauce" of kubernetes is the "watch etcd" logic that "watches" desired state and does the cybernetic loop to bring the observed state adhere to the desired state.

[trenchpilgrim]

The API and controller loops are the point of k8s. etcd is an implementation detail and lots of clusters swap it out for something else like sqlite. I'm pretty sure that GCP and Azure are using Spanner or Cosmos instead of etcd for their managed offerings.

[alphabettsy]

Yep. K3s can use SQLite or Postgres.

[itsnowandnever]

not exactly a fair assessment since neither of those were out and/or available to the kubernetes team at the time. sure, some things at many times from now into eternity may be or become better suited for the kubernetes data plane but at the time if etcd wasn't used there would be no kubernetes today

[jauntywundrkind]

The API server is the thing. It so happens that the API server can mostly be a thin shell over etcd. But etcd itself while so common is not sacrosanct.

https://github.com/k3s-io/kine is a reasonably adequate substitute for etcd. sqlite, MySQL, PostgreSQL can also be substituted in. Etcd is from the ground up built to be more scale-out reliable, and that rocks to have baked in. But given how easy it is to substitute etcd out, I feel like we are at least a little off if we're trying to say "etcd is also the entire point of k8s" (the APIserver is)

[dmlittle]

It's been a while since I've checked this but a few years ago we tried to limit test kine on a large-ish cluster and it performed pretty poorly. It's fine for small clusters but the way they have to implement the watch semantics makes it perform poorly (at least this was the case a few years ago).

[itsnowandnever]

that's fair but that 99% of all apiserver deployments in the world have the same standard boilerplate footprint is a large part of why it became so ubiquitous. that people running it locally don't have to make any decisions about how to deploy which database or why to use this one over that one... and that's also the same situation in production so people doing stuff in dev aren't punched in the face by an exponentially more complex system in production is huge.

[geoctl]

Is it? I honestly kinda believe that etcd is probably the weakest point in vanilla k8s. It is simply unsuitable for heavy write environments and causes lots of consistency problems under heavy write loads, it's generally slow, it has value size constraints, it offers very primitive querying, etc... Why not replace etcd altogether with something like Postgres + Redis/NATS?

[itsnowandnever]

that touches on what I consider the dichotomy of k8s: it's a really scalable system that makes it easy to spin up a cluster locally on your laptop and interact with the full API locally just like in prod. so it's a super scalable system with a dense array of features. but paradoxically most shops won't need the vast majority of k8s features ever and by the time they scale to where they do need a ton of distributed init features they're extremely close to the point where they'd be better served by a bespoke system conceived from scratch in house that solves problems very specific to the business in question. if you have many thousands of k8s nodes, you're probably in the gray area of if using k8s is worth it because the loop of k8s will never be as fast as a centralized push control plane vs the k8s pull/watch control plane. and naturally at scale that problem will only compound

[varispeed]

> Why not replace etcd altogether with something like Postgres + Redis/NATS?

Holy Raft protocol is the blockchain of cloud.

[cyberax]

> etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd.

Sorry, this is just BS. etcd is a fifth wheel in most k8s installations. Even the largest clusters are better off with something like a large-ish instance running a regular DB for the control plane state storage.

Yes, etcd theoretically protects against any kind of node failures and network partitions. But in practice, well, nobody really cares about the control plane being resilient against meteorite strikes and Cthulhu rising from the deeps.

[itsnowandnever]

that's not my point - my point is it would not have gotten the adoption it has without etcd and the fact that it was resilient and scalable out of the box