Kubernetes V1 Released

[StevePerkins]

Can anyone recommend a good "Consultant's", "Solutions Architect's", or "Top-Right Quadrant on a Silly Gardner Chart" overview of how Kubernetes competes or cooperates with Mesos? It feels like there's quite a bit of dense conceptual reading you have to plow through before you can even start to talk about what these things do.

[bkeroack]

To add to the others, in a nutshell:

- Mesos is a generalized, low level framework for distributing workloads over multiple nodes. It provides mechanism not policy. Therefore it requires quite a bit of up-front work to build something usable for any given application.

- Kubernetes is an opinionated cluster execution tool. It provides tools and a curated workflow for running distributed containerized applications. It's generally pretty quick and easy to get running.

- Mesos has a rich, resource-aware task scheduler. You can specify that your application requires X CPU units and Y RAM units and it will find the optimum node to run the task on.

- By contrast, the Kubernetes scheduler currently is rather dumb[1]. There's no way to specify the expected resource utilization for pods, and the scheduler simply tries to spread out replicas as much as possible throughout the available nodes.

People are (rightly) excited about things like Mesosphere which could allow the best of both worlds: the ease and API of Kubernetes with a powerful Mesos resource scheduler, not to mention nice-to-haves like a Web UI with pretty visualizations.

You can now cut me a check for 50% of the consulting revenue you get from this information. :)

1. The scheduler is intentionally simple and pluggable, to allow improvements easily in the future. My statements only apply to the current state of Kubernetes as deployed today.

[ghshephard]

Add to that, how does this relate to Docker-Swarm & Core-OS Fleet.

StackOverflow has at least one quick briefer, but wow, this field is growing quickly:

http://stackoverflow.com/questions/27640633/docker-swarm-kub...

[mml]

Kubernetes can run on Mesos as a Mesos service. Thereby giving you a more opinionated layout for your compute cluster, while running other peer Mesos services, such as Hadoop, Chronos And Marathon.

https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...

[WestCoastJustin]

At their most basic level, Kubernetes [1] and Mesos [2] both use a client/server type architecture, where you install a client on many compute nodes, then the server part farms out jobs (in the form of containers) to these client compute nodes. Mesos does not do scheduling, and is pretty low-level in terms of interfaces, so you would typically run some type of software which talks to it via API, like Marathon [3], Aurora [4], or Hadoop [5]. Then Marathon/Aurora/Hadoop/etc tells Mesos to farm out compute jobs to these end client nodes (aka schedules). Complexity can quickly go up depending on your hosting environment, scale, and HA requirements. There is actually a really good high-level overview diagram of what a mesos/hadoop/zookeeper setup looks like here [6].

The stack looks something like this for Kubernetes/Mesos (top down):

  - Kubernetes and Mesos (client/server packages depending on node type)
  - Docker (container engine)
  - OS (Ubuntu/RHEL/etc)

What are some use-cases?

  - you have more work than can fit into one server
  - need to distribute load across N+ nodes
  - Google heavily uses containers (not k8s, but that inspired these patterns)
    - Gmail/Search/etc all run in containers [7]
  - Apple, Twitter, and Airbnb are running Mesos today [8, 9]

There are a bunch of revolving services, like:

  - distributed key/values stores (etcd/zookeeper)
  - load balancers
  - image registries
  - user interfaces
  - cli tools
  - logging/monitoring/alerting
  - etc

But, to answer your question, the main difference between Kubernetes and Mesos, is that Kubernetes offers an opinionated workflow, built-in scheduler, and patterns for how containers are deployed into this cluster of compute notes. The pattern is baked in from the start via Pods, Labels, Services, and Replication Controllers. It also helps to know that Kubernetes comes from Google, where they have been running containers in-house, so much of this workflow (pods, services, replication controllers, etc), comes from their internal use-cases. That's the 10,000 foot view.

[1] https://github.com/GoogleCloudPlatform/kubernetes

[2] http://mesos.apache.org/

[3] https://mesosphere.github.io/marathon/

[4] http://aurora.apache.org/

[5] https://hadoop.apache.org/

[6] http://mesos.apache.org/assets/img/documentation/architectur...

[7] http://www.wired.com/2013/03/google-borg-twitter-mesos/

[8] http://www.infoq.com/news/2015/05/mesos-powers-apple-siri

[9] https://www.youtube.com/watch?v=E4lxX6epM_U

[MehdiEG]

To add to discussion, I've found this interview with Malte Schwarzkopf to be quite enlighning to understand how Omega, Kubernetes and Mesos fit together: https://blog.kismatic.com/qa-with-malte-schwarzkopf-on-distr...

[jcastro]

If you're using Ubuntu we've got a bundle of Juju charms for those who want to get up and running:

- http://insights.ubuntu.com/2015/07/21/introducing-kubernetes...

- https://jujucharms.com/u/kubernetes/kubernetes-cluster

PRs and comments welcome!

[nickbauman]

I've found that Kubernetes is a big hammer. If your problem can be backed by a web app, you should start with AppEngine. If you need specialty library support or wider computational space per instance, you can move your AppEngine app to ManagedVMs (which uses Kubernetes under the covers). If you need special, "grid-like" services where you need exquisite control over the entire stack, only then does it make sense to use raw Kubernetes and Docker. And you will spend a lot of time getting it right.

[puja108]

ManagedVMs don't use k8s under the covers afaik. However there's GKE (Google Container Engine), which goes on top of (at least 3) MVMs and that one does use k8s. While it is true that going with a PaaS(-like) solution like AppEngine or Heroku is easy in the beginning it can get pretty expensive pretty fast and it limits you in the choice of languages, frameworks, and data stores you can use. This can in some instances bring technical debt with it that will pose a hurdle when growing. Actually, using Docker combined with an orchestration layer like k8s is supposed to give the the ease-of-use of PaaS with the flexibility of IaaS (or MVMs), however, managing sth like k8s by yourself is not that easy and you will need quite a bunch of other tools on top, i.e. for monitoring and stuff, which paves the way for Container-as-a-Service solutions, like RancherOS, tutum, or Giant Swarm (disclaimer, I'm part of the latter company)

[geku]

Getting started with Kubernetes is pretty easy and I wrote a guide to quickly run it on your local machine: https://www.cloudgear.net/blog/2015/5-minutes-kubernetes-set...

It's not up to date with Kubernetes 1.0.0 but I'll update the images as soon as the final version 1 is tagged.

[TheIronYuppie]

I definitely don't want this to come off as a sales pitch, but you can get started in one click using Google Container Engine (and $300 in free credit) as well.

Full disclosure: I work at Google on Kubernetes

[rjusher]

I might come across as ignorant but what is the relationship between Kubernetes and Docker, because when I was reading the article I tought of it as a Docker competitor, but further down in the comments, there is one that says they do different jobs.

And that confused me.

[TheIronYuppie]

Docker and Kubernetes works hand in hand. That is to say, if you choose Docker as your container format, Kubernetes runs Docker on every node to run your container. Kubernetes focuses on _what_ Docker should run, and how to move those container workloads around.

Docker also has Docker Swarm, which can be thought of as a competitor in some ways. But Google will be a heavy supporter of their container format for a long time to come.

Full Disclosure: I work at Google on Kubernetes

[pat2man]

Kubernetes is a scheduler for Docker containers. So lets say you want the following:

1 x MySQL Master

2 x MySQL Slave

5 x PHP Server

1 x Monitoring Script

Each of those would be a docker container. Kubernetes would figure out which host to place them on and verify that they are running, rebooting them on another host if one of your hosts goes down.

[ecnahc515]

Docker as an organization, is a competitor to Google and Kubernetes, but Docker as a tool is complemented by Kubernetes.

This is because Docker is building it's own scheduling/orchestration tools, which is what Kubernetes is for. However, as a container runtime, Kubernetes works great with Docker.

[Oletros]

And related to that, CoreOS has launcehd a preview of its Kubernetes comercial platform [0]

[0] http://techcrunch.com/2015/07/21/coreos-launches-preview-of-...

[kordless]

Wercker has a post on continous integration for containers to Kubernetes which Micha, their CEO, did back in June: http://blog.wercker.com/2015/06/23/Deploying-minimal-contain..., which uses Google's Container Registry for pushing images.

If anyone is interested, I just wrapped up a similar post for deploying containers to Giant Swarm from Wercker, no Docker required: I just got done doing a continuos integration post for containers using Wercker and Giant Swarm: https://github.com/giantswarm/swarm-wercker.

[arianvanp]

I'm hoping for ACI support soon so that I can use rkt instead of docker. :-)

[TheIronYuppie]

I have good news for you :)

https://blog.kismatic.com/running-rkt-on-kubernetes/

Full Disclosure: I work at Google on Kubernetes

[smegel]

> containers scheduled < 5s on average

That seems awfully slow for a fancy chroot. I use KVM to bring up WinXP snapshot VMs in around 2s to a running state...maybe they mean 5ms?

[cjcullen]

5s is the right number, but it leaves out that it is 99%ile, on a 100-node, 3000-pod cluster. See https://github.com/GoogleCloudPlatform/kubernetes/issues/395....

[zwischenzug]

It's the cadence of the etcd read cycle chosen for kubernetes. They could do it much faster, but I guess there are engineering tradeoffs based on the size of the cluster and the speed of the network.

[jacques_chester]

Interesting to see IBM placing a bet each way by joining the new foundation, seeing as they're already in the Cloud Foundry Foundation.

Edit: and I see the Cloud Foundry Foundation logo on the Cloud Native Foundation homepage. It's Foundations all the way down.

(Disclaimer: I work for another CFF member, Pivotal).

[justincormack]

Its pretty orthogonal to cloud foundry, which is a composite infrastructure, while these are lower level products that can be used to build PaaS.

[alanh]

Is Kubernetes pronounced "koo bur NET ease" or am I way off?

[a-robinson]

Yes, that's how it's pronounced.

[wereHamster]

What's with the recent adoption of the .io TLD @ google? cncf.io, gcr.io, ...

[dguaraglia]

I guess it's driven by the same motivation everyone else has: it's impossible to get a decent .com domain.

[barkingcat]

how do you pronounce that word?

[packetslave]

coo-bur-net-eees