[jsmthrowaway]

Indeed. The universe where you aren’t trying to replace Borg, and instead have other motives for conceiving of Kubernetes in the first place. That’s all I’m trying to say.

Hadoop is nowhere close to the original thinking behind MapReduce, and Kubernetes is nowhere close to the original thinking behind Borg. The difference is Yahoo! wrote one, and the other leads us to speculate about why they’d want to offer such a good concept to the world, then ship something that so spectacularly misses that mark, particularly scalewise, but get away with it because it’s Google doing it.

There are only a couple explanations, especially when you factor in that corporations always act in their interest. That’s not a moral dimension, that’s just how it works. What upside is there for Google having everyone do cluster orchestration the way they do? Really, I challenge you to think of one. When I was there, you couldn’t even say the word Borg externally (despite the original article not being the first security-based leak of it). Operations and engineering velocity was, and remains, a crucial competitive advantage for Google. Since I’ve left, there’s a clear shift toward teaching the industry how to do operations, from the Borg paper on through to the SRE book. Why? Why suddenly cede that competitive advantage, the ability to ship a whole production service in like, a weekend?

Because you do it poorly, drive people to Google Cloud because administering Kubernetes at any scale is like self-flagellating with a rake (don’t get me started and be satisfied that I’m speaking from ulcers here), and keep ahead of the competition who are forced to respond to GKE now that a whole community is clamoring for Docker in prod. It’s the only explanation that fits. Maybe it wasn’t even intentional, but it’s what has played out.

Another possible explanation is that Google wanted to end run Mesos, which was threatening to dominate until Kubernetes got a bunch of mindshare and venture capital. Mesos, with care and careful choice of schedulers, can be on the same field as Borg. Twitter is very happy with it at scales that would decimate Kubernetes, as well as many others I cannot name. Google’s VC endeavors have been dumping into Kubernetes startups, and now you don’t hear about Mesos any more. Just like Docker before it, we just can’t get enough bad operational systems as a profession, apparently.

My whole career I’ve been obsessed with high scale orchestration because it is a uniquely difficult problem. I saw Borg and went Jesus, we are all behind the curve as an industry. Then the first few releases of Kubernetes came out and I killed it dead with a few dozen nodes and moderate load. It’s gotten better, yes, but Borg has literally centuries of engineering in it, and a steering committee is going to push Kubernetes to OpenStack or Java status (I’d argue it’s already there, given that people are now shipping distributions of it). And now operations as a discipline suffers for it. I can’t get a SRE gig any more without a dev team who doesn’t need Kubernetes, at all, making it a showstopper to ship.

Kinda funny, by the way, that this thread has been off the front page for hours and is still active with new faces. I must have ruffled eng-misc. :)