Google has plenty of experience with containers already, since they heavily use cgroups and the concept of containers in their production environment for isolation and resource control, never mind the fact that two of their engineers had written much of the initial cgroups code. I talked about this in my "Introduction to Containers on Linux using LXC" screencast [1]. Briefly, Google is in the process of open sourcing their internal container code [2], there was a Wired article that talked about their container orchestration system [3], and finally, there was John Wilkes (Google Cluster Management), who talks about the container management system [4].
Docker adds very interesting filesystem ideas and software for the management of container images. Personally, I think we are on the cusp of a transition from VPS (xen/hvm) to VPS (containers). I also hope that Google throws some of their concepts at the Docker project. Interesting times for this space.
"Personally, I think we are on the cusp of a transition from VPS (xen/hvm) to VPS (containers)."
A transition back to, I think ... the very first VPS provider (JohnCompanies, 2001)[1] was based entirely on FreeBSD jail and was entirely container based, even though those containers were presented as standalone FreeBSD systems.
[1] Yes, verio did have that odd VPS-like enterprise service earlier than that, but JC did the first VPS as we came to know it.
> Personally, I think we are on the cusp of a transition from VPS (xen/hvm) to VPS (containers).
I'm not so sure of that. I think a lot of the use-cases for VMs are based on isolation between users and making sure everybody gets a fair slice. Something like docker would work well with a single tenant but for multi-tenant usage docker would give you all the headaches of a shared host and very little of the benefits of a VM. For those use cases you're probably going to see multiple docker instances for a single tenant riding on top of a VM.
The likes of Heroku, AWS, Google etc will likely use docker or something very much like it as a basic unit to talk to their customers, but underneath it they'll (hopefully) be walling off tenants with VMs first. VMs don't have to play friendly with each other, docker containers likely will have to behave nicely if they're not to monopolize the underlying machine.
I'm always surprised that OpenVZ[1] doesn't come up more in discussions about containers. I used to use it extensively in for small lightweight VPS's (usually just an apache or mysql instance) and always found it to be pretty impressive. I've used Docker to make a debian-packaged app available on a bunch of CentOS machines and it saved me a huge headache (the dependency tree for the app was huge) so I'm a fan - but still a little puzzled at OpenVZ's seeming invisibility.
There's also the recent article about automatic and machine-learning based power and core management.[1], [2]
If anyone here specializes in similar things, I would be curious to know if this Pegasus system runs on top of or underneath Borg/Omega (or perhaps replaced it?), or is a separate system altogether.
> Personally, I think we are on the cusp of a transition from VPS (xen/hvm) to VPS (containers).
There may be some of that, but I think more common will be continuing to have tradition IaaS bring-your-OS-image services, with a new tier in between IaaS and PaaS (call it CaaS -- Container host as a service), plus a common use of IaaS being to deploy an OS that works largely as a container host (something like CoreOS).
For at least the last 15 years I have almost always done 'devops' myself (before it was called that). Managing deployment goes with designing and building systems.
My problem is that I am sort of stuck in the past. Whether I am using VPSs, AWS, or rented physical servers, I have only a partially automated way to set up servers. This scales to small numbers of servers just fine, and that is mostly the world I live in, but I need to improve my workflow. This really hit home yesterday when I had to upgrade a Haskell GHC/platform because I tweaked a Haskell app making incompatible with an old GHC 7.4.* setup on an older server, and ended up wasting some time before fixing things.
Working as a contractor at Google last year was an eye opener. I really loved their infrastructure. Eye opening experience.
You have hit the nail on the head. The thing people want in the real world is improved workflow.
In the real world, everyone wants infrastructure to have the same sexy qualities: automated deployment (CD/CI), automated scaling, automated failover/high availability, automated service discovery (read: functional service topology dependency resolution), security, resource and capacity planning support, real time least-cost-overhead provider selection for third party infrastructure providers meeting some disparate set of performance requirements, etc. Unfortunately, it's not an easy problem area to deliver a one size fits all solution to.
Docker doesn't really have most of that stuff in scope yet, even vaguely. Actually, it seems to have a really weird scope: it wants to wrap different LXC implementations and other container-style unix environments (potentially supporting non-Linux platforms) but doesn't want to deal with managing the host systems themselves, having - kind of, for practical reasons (though not entirely!) - outsourced this to CoreOS (ie. some particularly specific configuration of a Linux host system).
Whether all of this recent Redhat/Google docker bandwagon jumping will amount to any real solution remains to be seen .. Google AFAIK effectively runs its services on fat clusters made of commodity hardware, organized in to segments ('cells'), running highly customised Linux distributions, and so does Redhat where HA is required. I'm pretty familiar with these configurations as I do this myself. So will we ever see meaningful support for other OSs? Other distros? Physical systems via PXE to support these clusters? Hypervisor guests managed with the same developer and operations workflow?
My wager is not soon, at least in a manner that everyone agrees on... Google will keep doing its thing (using its unlimited supply of internal, world-class nerds to deliver and manage services on their custom OS in a custom way because saving 1/2c a month per machine pays ten world class nerd salaries at their scale), Redhat will keep doing its thing (selling prebuilt systems at expensive prices that still comfortably undercut the likes of IBM, pretending they are manageable, but actually rejigging the whole infrastructure every system release leaving little in the way of realistic upgrade paths without expensive consulting) and you and I will be left wondering where that magical docker solution went that everyone was talking about in early 2014.
I agree.. I had a very similar 'eye opening' experience as you did with Google's infrastructure. I am now working on a internal projects as a contractor / partner. Solid.
Is Docker one of those things that works better for big projects than small ones? I tried using it for 3 little projects, just to see what all the fuss was about. But every time it ended up being easier to use bare LXC.
Yes. It is like configuration management [e.g. Chef, Salt]. It doesn't make sense to do for one off projects [unless you already have it as part of your workflow].
Like, I have a generic Flask container repository that I clone as a starting base...change the domain name(s) on the config file, app name, etc. Then just copy things into the src/ directory and spin it up.
For me, this is really easy to spin up a new flask project. I do the same with golang [sort of, it really just downloads a binary+config file into a container that has all of my logging, etc setup].
The problem with doing LXC is where you have inheritance chains like this:
[Base Container of Ubuntu + Logging + Monitoring] -> [Base Flask Container] -> [Flask Container for Website X]
[Base Container of Ubuntu + Logging + Monitoring] -> [Base Flask Container] -> [Flask Container for API for Website X]
[Base Container of Ubuntu + Logging + Monitoring] -> [Container for Mail Service]
[Base Container of Ubuntu + Logging + Monitoring] -> [Container for Kibana]
[Base Container of Ubuntu + Logging + Monitoring] -> [Container for Redis-Cache]
etc.
Tbh, I think that is what docker really fixes. The ability to easily inherit containers so you only have to make changes in one spot to change all of them.
My bet? Docker increasingly is something that will be used even if the user don't know it.
Builds, as an example. This was my first introduction to Docker - scaling Jenkins worker nodes was allocating yet another VM. As you scale to tens or hundreds of build slaves, you realize utilization across the cluster is down. Scaling builds with Docker was simple and efficient to implement and it allowed me to drive utilization up dramatically.
Now there's 15 different companies, integrations, and tooling surrounding this space to make that process repeatable to the masses. Amazing.
We're seeing the same thing in the orchestration, paas, sdn, iaas, YOUR ACRONYM OF CHOICE, and one common theme is that ecosystem is standardizing around Docker. That's pretty powerful.
1. When I need to move application between different systems running Ubuntu, Debian etc, now I use Virtual box. Can I use Docker now on?
2. A quick reading about docker tells me that instead of running a guest OS as in Virtual box docker only holds application related things. Then how could it handle deployment between Debian Squeeze and Ubuntu 14.04. I mean old and new Linux version
3. Compared to virtual box how easy it is to install and use Docker
4. Can you please tell some places where you people use docker
5. How many of you have migrated to Docker from virtual box and related things?
Does the complexity of using containers bother anybody? It just seems like this adds so many more attack points by removing the virtual machine which was a good way to organize services.
> Does the complexity of using containers bother anybody?
It doesn't seem all that complex; sure, its (in the typical cloud use case) another level of organization, but done right it should actually simplify organization and deployments.
> It just seems like this adds so many more attack points by removing the virtual machine which was a good way to organize services.
Containers are different than VMs, but using them doesn't mean "removing the virtual machine". Particularly in the use cases that Google is embracing (e.g., on a cloud platform where the containers are for use on VMs.) How, specifically, does it add "attack points"?
Increasing parallelization has been done in evolutionary steps for the last 10 years. This is a great step, but I'm sure your amazons and rackspaces of the world will soon be building docker++ (and docker really is just ec2/AMI++)
It's really amusing to me that FreeBSD Jails have existed for quite some time already. I think the novelty though is "shared configurations" and shareable environments that are not VMs.
Right now "cloud" stuff is looking a lot like Subversion back in the day - I'll love it when the "distributed cloud" arrives (mesh networks are kind of the first iteration of that but they won't catch on until more people have fiber internet and there comes a commercial use for it like what Kickstarter did for crowdfunding).
[EDIT] The downvote trolls have been out recently - this guy has a legit comment; it's not detracting and it provided an interesting point of discussion that I myself took up. There's no real foul in his comment.
[+] [-] WestCoastJustin|11 years ago|reply
Docker adds very interesting filesystem ideas and software for the management of container images. Personally, I think we are on the cusp of a transition from VPS (xen/hvm) to VPS (containers). I also hope that Google throws some of their concepts at the Docker project. Interesting times for this space.
[1] http://sysadmincasts.com/episodes/24-introduction-to-contain...
[2] https://github.com/google/lmctfy
[3] http://www.wired.com/2013/03/google-borg-twitter-mesos/all/
[4] http://www.youtube.com/watch?v=0ZFMlO98Jkc
[+] [-] rsync|11 years ago|reply
A transition back to, I think ... the very first VPS provider (JohnCompanies, 2001)[1] was based entirely on FreeBSD jail and was entirely container based, even though those containers were presented as standalone FreeBSD systems.
[1] Yes, verio did have that odd VPS-like enterprise service earlier than that, but JC did the first VPS as we came to know it.
[+] [-] jacquesm|11 years ago|reply
I'm not so sure of that. I think a lot of the use-cases for VMs are based on isolation between users and making sure everybody gets a fair slice. Something like docker would work well with a single tenant but for multi-tenant usage docker would give you all the headaches of a shared host and very little of the benefits of a VM. For those use cases you're probably going to see multiple docker instances for a single tenant riding on top of a VM.
The likes of Heroku, AWS, Google etc will likely use docker or something very much like it as a basic unit to talk to their customers, but underneath it they'll (hopefully) be walling off tenants with VMs first. VMs don't have to play friendly with each other, docker containers likely will have to behave nicely if they're not to monopolize the underlying machine.
[+] [-] tzm|11 years ago|reply
Also check out Joe Beda's deck from GlueCon: http://slides.eightypercent.net/GlueCon%202014%20-%20Contain...
Docker is a natural fit for GCE.
[+] [-] billybofh|11 years ago|reply
[1] http://openvz.org/Main_Page
[+] [-] mikecb|11 years ago|reply
If anyone here specializes in similar things, I would be curious to know if this Pegasus system runs on top of or underneath Borg/Omega (or perhaps replaced it?), or is a separate system altogether.
[1] http://www.theregister.co.uk/2014/06/05/google_pegasus_syste...
Edit: [2] http://gigaom.com/2014/05/28/google-is-harnessing-machine-le...
[+] [-] dragonwriter|11 years ago|reply
There may be some of that, but I think more common will be continuing to have tradition IaaS bring-your-OS-image services, with a new tier in between IaaS and PaaS (call it CaaS -- Container host as a service), plus a common use of IaaS being to deploy an OS that works largely as a container host (something like CoreOS).
[+] [-] mark_l_watson|11 years ago|reply
My problem is that I am sort of stuck in the past. Whether I am using VPSs, AWS, or rented physical servers, I have only a partially automated way to set up servers. This scales to small numbers of servers just fine, and that is mostly the world I live in, but I need to improve my workflow. This really hit home yesterday when I had to upgrade a Haskell GHC/platform because I tweaked a Haskell app making incompatible with an old GHC 7.4.* setup on an older server, and ended up wasting some time before fixing things.
Working as a contractor at Google last year was an eye opener. I really loved their infrastructure. Eye opening experience.
Docker seems like my best path forward.
[+] [-] contingencies|11 years ago|reply
In the real world, everyone wants infrastructure to have the same sexy qualities: automated deployment (CD/CI), automated scaling, automated failover/high availability, automated service discovery (read: functional service topology dependency resolution), security, resource and capacity planning support, real time least-cost-overhead provider selection for third party infrastructure providers meeting some disparate set of performance requirements, etc. Unfortunately, it's not an easy problem area to deliver a one size fits all solution to.
Docker doesn't really have most of that stuff in scope yet, even vaguely. Actually, it seems to have a really weird scope: it wants to wrap different LXC implementations and other container-style unix environments (potentially supporting non-Linux platforms) but doesn't want to deal with managing the host systems themselves, having - kind of, for practical reasons (though not entirely!) - outsourced this to CoreOS (ie. some particularly specific configuration of a Linux host system).
Whether all of this recent Redhat/Google docker bandwagon jumping will amount to any real solution remains to be seen .. Google AFAIK effectively runs its services on fat clusters made of commodity hardware, organized in to segments ('cells'), running highly customised Linux distributions, and so does Redhat where HA is required. I'm pretty familiar with these configurations as I do this myself. So will we ever see meaningful support for other OSs? Other distros? Physical systems via PXE to support these clusters? Hypervisor guests managed with the same developer and operations workflow?
My wager is not soon, at least in a manner that everyone agrees on... Google will keep doing its thing (using its unlimited supply of internal, world-class nerds to deliver and manage services on their custom OS in a custom way because saving 1/2c a month per machine pays ten world class nerd salaries at their scale), Redhat will keep doing its thing (selling prebuilt systems at expensive prices that still comfortably undercut the likes of IBM, pretending they are manageable, but actually rejigging the whole infrastructure every system release leaving little in the way of realistic upgrade paths without expensive consulting) and you and I will be left wondering where that magical docker solution went that everyone was talking about in early 2014.
[+] [-] olalonde|11 years ago|reply
Makes me feel old... I used to "deploy" with cPanel or DirectAdmin.
[+] [-] tzm|11 years ago|reply
[+] [-] sp332|11 years ago|reply
[+] [-] opendais|11 years ago|reply
Like, I have a generic Flask container repository that I clone as a starting base...change the domain name(s) on the config file, app name, etc. Then just copy things into the src/ directory and spin it up.
For me, this is really easy to spin up a new flask project. I do the same with golang [sort of, it really just downloads a binary+config file into a container that has all of my logging, etc setup].
The problem with doing LXC is where you have inheritance chains like this:
[Base Container of Ubuntu + Logging + Monitoring] -> [Base Flask Container] -> [Flask Container for Website X]
[Base Container of Ubuntu + Logging + Monitoring] -> [Base Flask Container] -> [Flask Container for API for Website X]
[Base Container of Ubuntu + Logging + Monitoring] -> [Container for Mail Service]
[Base Container of Ubuntu + Logging + Monitoring] -> [Container for Kibana]
[Base Container of Ubuntu + Logging + Monitoring] -> [Container for Redis-Cache]
etc.
Tbh, I think that is what docker really fixes. The ability to easily inherit containers so you only have to make changes in one spot to change all of them.
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] nickstinemates|11 years ago|reply
Builds, as an example. This was my first introduction to Docker - scaling Jenkins worker nodes was allocating yet another VM. As you scale to tens or hundreds of build slaves, you realize utilization across the cluster is down. Scaling builds with Docker was simple and efficient to implement and it allowed me to drive utilization up dramatically.
Now there's 15 different companies, integrations, and tooling surrounding this space to make that process repeatable to the masses. Amazing.
We're seeing the same thing in the orchestration, paas, sdn, iaas, YOUR ACRONYM OF CHOICE, and one common theme is that ecosystem is standardizing around Docker. That's pretty powerful.
[+] [-] dominotw|11 years ago|reply
[+] [-] general_failure|11 years ago|reply
[+] [-] jestinjoy1|11 years ago|reply
1. When I need to move application between different systems running Ubuntu, Debian etc, now I use Virtual box. Can I use Docker now on?
2. A quick reading about docker tells me that instead of running a guest OS as in Virtual box docker only holds application related things. Then how could it handle deployment between Debian Squeeze and Ubuntu 14.04. I mean old and new Linux version
3. Compared to virtual box how easy it is to install and use Docker
4. Can you please tell some places where you people use docker
5. How many of you have migrated to Docker from virtual box and related things?
Disclaimer: Noob detected :)
[+] [-] ladzoppelin|11 years ago|reply
[+] [-] dragonwriter|11 years ago|reply
It doesn't seem all that complex; sure, its (in the typical cloud use case) another level of organization, but done right it should actually simplify organization and deployments.
> It just seems like this adds so many more attack points by removing the virtual machine which was a good way to organize services.
Containers are different than VMs, but using them doesn't mean "removing the virtual machine". Particularly in the use cases that Google is embracing (e.g., on a cloud platform where the containers are for use on VMs.) How, specifically, does it add "attack points"?
[+] [-] blazespin|11 years ago|reply
[+] [-] nickstinemates|11 years ago|reply
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] ixmatus|11 years ago|reply
Right now "cloud" stuff is looking a lot like Subversion back in the day - I'll love it when the "distributed cloud" arrives (mesh networks are kind of the first iteration of that but they won't catch on until more people have fiber internet and there comes a commercial use for it like what Kickstarter did for crowdfunding).
[EDIT] The downvote trolls have been out recently - this guy has a legit comment; it's not detracting and it provided an interesting point of discussion that I myself took up. There's no real foul in his comment.