Some google-managed base images would be helpful. The last time I checked, some of the major public docker images were still shellshock-vulnerable. Pre-installed GCE tools would be helpful. Perhaps automated environment variables about region, etc.
Given that this is the alpha launch of Container Engine, and the Beta launch of Autoscaling and Managed VMs, what is the anticipated timeframe for this to be ready for production workloads?
I'm assuming that there'll be a beta of Container Engine in the near future, and then a stable 1.0 launch?
We use AWS Elastic Beanstalk for Docker deployment. Can you comment on how this is different? I see it supports multiple containers per VM - are there other significant distinguishing factors?
>>Because this new service is officially in alpha, you’ll need to apply for access and be whitelisted to use the service. It also means the service isn’t feature complete and the whole infrastructure could melt down at any minute.
Not sure what the take away here is other than, this may one day be a feature. No information about pricing, support, or sla. My attitude is wait and see. Should it be otherwise?
Edit: Can someone tell me why this isn't a valid question? Why the down votes?
My biggest question on Docker / k8s: How does one control where persistent data lives and the required performance characteristics? Example: I need SSD + a certain level of availability? Is there a good strategy for managing this?
Is it possible to run a KVM VM inside one of the containers? I have an application for launching and managing VMs and letting people spin up a Google container to try it out would be neat.
Do you see this being useful for analytical workloads (ie i need a bunch of cores for a short time to run a parallel job) or more focused on scaling web apps and services?
Does this mean that Google feels confident in running untrusted code inside containers? Or is each container actually running in an isolated VM?
It is my understanding that Docker containers are "generally" secure (https://docs.docker.com/articles/security/). But that statement isn't enough for me to use them to power a multi-user production hosting environment.
Can someone explain the difference between Apache Mesos, Apache Spark, and Kubernetes. As someone reading just announcements, and having never experimented with any of these tools, they sound like they promise the same capabilities.
Is it just that they are different open source projects aiming for the same goal ... or are their goals different?
Mesos and Kubernetes share lot of commonalities: both are useful for managing (virtual) compute resources for a given application. Primarily, as an app developer, you are interesting in specifying the required resources at a high level: 10 front end instances, 20 middle tier, and 3 back end database instances. You may not care how the resources are provisioned as long as some underlying SLA is satisfied. This is useful for boosting service uptime since in any long running service the nodes may go offline anytime and the framework will re-provision the resources automatically--often within few mins.
Mesos is useful to share resources across multiple clusters (i.e., multiple departments using Hadoop within a same organization may want to share the resources), where as Kubernetes seems to focus on container applications using docker. In addition, Mesos implements dominant-resource-fairness (DRF) scheduling, which has some nice properties [0].
Lastly, Spark is just an application running on the top of Mesos or Kubernetes. Using my earlier example, Spark and Hadoop can run in a single datacenter on the top of Mesos without statically partitioning the clusters a priori each having an illusion that they own the entire datacenter.
I'll give it a shot, with the disclaimer that this explanation is based on conference talks I've seen and documentation I've read.
Spark is a successor to Hadoop, aiming to perform big distributed data crunching jobs more quickly by not limiting itself to a map-reduce paradigm and by holding more data in memory.
Apache Mesos is a resource scheduler for a cluster of machines. It is architected to be agnostic about the kind of application running on it, but I get the impression it's primarily used to make things like Hadoop and Spark run on the same cluster without stepping on each other's toes (and not used so much for long-running services like a web app). An application says to Mesos "hey I'd like to run job X", outlining the resources that it needs, and then Mesos looks at the cluster to see the best place to run the job. Mesos has some Docker integration but it's weird. Instead of launching your job and creating cgroups directly as it usually does, Mesos has to run a proxy process that talks to Docker and asks it to do it. (This is because Docker no longer has standalone mode. Which is my biggest gripe about Docker.)
Kubernetes is solving a similar problem to Mesos, how to allocate resources in a cluster to all the jobs that you might want to run on it. It's explicitly focused on "containers", though not necessarily just Docker containers. (The README at https://github.com/GoogleCloudPlatform/kubernetes makes no mention of Docker.) I get the impression it's focused more on long-running processes like web servers and less on ephemeral Hadoop-style jobs.
Apache Mesos is a distributed system that is kind of a "bottom layer" for computations and storage (whether in-memory or on disk).
Apache Spark is a distributed application that runs on top of Mesos and does computations that takes advantage of cluster computing. It can do classic MapReduce or other algorithms that you write using its API.
Kubernetes is a distributed system that runs Docker in a cluster. Docker is a way to run sandboxed applications. Kubernetes can run on Mesos.
Apache Mesos is based on Twitter's expertise in deploying their cluster, Kubernetes on Google's. They do things in different ways. Spark seems to be in the same area as Hadoop, so not relevant to the conversation.
Strongly agree with this. I've been interviewing for DevOps jobs and it seems like everyone thinks they need to be using Docker for some reason or another. Studying up on it has improved my interview feedback substantially.
Hi Joe, I guess you folks are not running containers from customers side-by-side in the same host. So if I scale up my cluster size while I have containers deployed, do you folks rebalance the load on the host machines on the fly (by stopping/relocating some containers) or do you rebalance the new containers?
I hope this have the ripple effect of allowing enterprises to feel comfortable using things like Firebase and Docker. I floated the idea of using Docker but the idea was turned down because we have no idea how it affects performance in a production environment.
[+] [-] jbeda|11 years ago|reply
I'm on the Kubernetes/GKE team and happy to answer any questions you all might have.
We also all hang out on IRC at #google-containers on freenode.
[+] [-] philip1209|11 years ago|reply
[+] [-] 23david|11 years ago|reply
I'm assuming that there'll be a beta of Container Engine in the near future, and then a stable 1.0 launch?
[+] [-] jread|11 years ago|reply
[+] [-] marktangotango|11 years ago|reply
Not sure what the take away here is other than, this may one day be a feature. No information about pricing, support, or sla. My attitude is wait and see. Should it be otherwise?
Edit: Can someone tell me why this isn't a valid question? Why the down votes?
[+] [-] wstrange|11 years ago|reply
[+] [-] jff|11 years ago|reply
[+] [-] micro_cam|11 years ago|reply
[+] [-] lbotos|11 years ago|reply
pick my vm size/ pick my docker image/ and go?
AKA does this bring me one step closer to docker-oku-aaS?
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] espeed|11 years ago|reply
[deleted]
[+] [-] roberthbailey|11 years ago|reply
[+] [-] philip1209|11 years ago|reply
[+] [-] deweller|11 years ago|reply
It is my understanding that Docker containers are "generally" secure (https://docs.docker.com/articles/security/). But that statement isn't enough for me to use them to power a multi-user production hosting environment.
[+] [-] jbeda|11 years ago|reply
My view is that the surface area for cgroups/kernel namespaces is just too large and isn't appropriate for hostile untrusted workloads right now.
More nuanced statement on this here: http://googlecloudplatform.blogspot.com/2014/08/containers-v...
[+] [-] TheMagicHorsey|11 years ago|reply
Is it just that they are different open source projects aiming for the same goal ... or are their goals different?
[+] [-] wowmsi|11 years ago|reply
Mesos is useful to share resources across multiple clusters (i.e., multiple departments using Hadoop within a same organization may want to share the resources), where as Kubernetes seems to focus on container applications using docker. In addition, Mesos implements dominant-resource-fairness (DRF) scheduling, which has some nice properties [0].
Lastly, Spark is just an application running on the top of Mesos or Kubernetes. Using my earlier example, Spark and Hadoop can run in a single datacenter on the top of Mesos without statically partitioning the clusters a priori each having an illusion that they own the entire datacenter.
[0] https://www.usenix.org/event/nsdi11/tech/full_papers/Hindman...
[+] [-] bjt|11 years ago|reply
Spark is a successor to Hadoop, aiming to perform big distributed data crunching jobs more quickly by not limiting itself to a map-reduce paradigm and by holding more data in memory.
Apache Mesos is a resource scheduler for a cluster of machines. It is architected to be agnostic about the kind of application running on it, but I get the impression it's primarily used to make things like Hadoop and Spark run on the same cluster without stepping on each other's toes (and not used so much for long-running services like a web app). An application says to Mesos "hey I'd like to run job X", outlining the resources that it needs, and then Mesos looks at the cluster to see the best place to run the job. Mesos has some Docker integration but it's weird. Instead of launching your job and creating cgroups directly as it usually does, Mesos has to run a proxy process that talks to Docker and asks it to do it. (This is because Docker no longer has standalone mode. Which is my biggest gripe about Docker.)
Kubernetes is solving a similar problem to Mesos, how to allocate resources in a cluster to all the jobs that you might want to run on it. It's explicitly focused on "containers", though not necessarily just Docker containers. (The README at https://github.com/GoogleCloudPlatform/kubernetes makes no mention of Docker.) I get the impression it's focused more on long-running processes like web servers and less on ephemeral Hadoop-style jobs.
[+] [-] shepardrtc|11 years ago|reply
Apache Mesos is a distributed system that is kind of a "bottom layer" for computations and storage (whether in-memory or on disk).
Apache Spark is a distributed application that runs on top of Mesos and does computations that takes advantage of cluster computing. It can do classic MapReduce or other algorithms that you write using its API.
Kubernetes is a distributed system that runs Docker in a cluster. Docker is a way to run sandboxed applications. Kubernetes can run on Mesos.
[+] [-] vertex-four|11 years ago|reply
[+] [-] krschultz|11 years ago|reply
[+] [-] RenegadeofFunk|11 years ago|reply
[+] [-] gfodor|11 years ago|reply
[+] [-] jread|11 years ago|reply
[+] [-] lxcp|11 years ago|reply
[+] [-] jbeda|11 years ago|reply
We don't do rebalancing/rescheduling/repacking yet. Those are the types of things that we will be working on moving forward.
[+] [-] yim|11 years ago|reply
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] omouse|11 years ago|reply
[+] [-] tomcart|11 years ago|reply
[+] [-] chx|11 years ago|reply
[+] [-] anon1385|11 years ago|reply
Submissions from bullshit SV news sites is the price you pay for using a site that primarily exists as a marketing exercise.
Note that 'lower quality' sites about other sections of the tech industry have been banned for years (appleinsider, winsupersite etc).
[+] [-] dang|11 years ago|reply
(We changed the URL from http://techcrunch.com/2014/11/04/google-launches-managed-ser...)
[+] [-] ossreality|11 years ago|reply
[deleted]
[+] [-] unknown|11 years ago|reply
[deleted]