I've since moved on to Kubernetes and SaltStack, but I got into production Docker containers with Deis[1] a year ago. With full Heroku buildpack support, Kubernetes backed, and great container tools. I feel like it's sad that this project doesn't get more love. It's a 20 minute deploy to GKE or AWS (I recommend GKE).
If you run OSX, TheNewNormal's solo and cluster Kube setup is a breeze and has the Deis Workflow as a one command install, then you have full local parity with deployment.
When I started with straight Docker figuring out the transition to the real world was be really daunting. Getting into a real production environment that I could compost quickly as I learned learn the paradigms was invaluable. Articles like this seem to buzzword their way into making the path muddier than it actually is.
Another big help for me was any time I needed to set something up, first thing, I'd look at how the bleeding edge was being done with Alpine. Practically you might not always run that light, but that community has all the answers.
When I first heard about Deis it was heavily compared to Flynn [1]. It's a little hard to tell from the outside if the two are still competitors or if Deis is focusing on a lower level of abstraction.
Title really shouldn't include "Mesosphere." It should be "Marathon" or "DC/OS". Or, if you keep Mesosphere, it should be "Google, Mesosphere, and Docker."
Aside from that, it's not a bad overview. I particularly like this quote:
“Mesos and Kubernetes are largely aimed at solving similar problems of running clustered applications; they have different histories and different approaches to solving the problem. Mesos focuses its energy on very generic scheduling and plugging in multiple different schedulers.” In contrast, says Burns, “Kubernetes was designed from the ground up to be an environment for building distributed applications from containers. It includes primitives for replication and service discovery as core primitives, whereas such things are added via frameworks in Mesos. … Swarm is an effort by Docker to extend the existing Docker API to make a cluster of machines look like a single Docker API.”
Mesos is useful when you have requirements that don't fit in k8s' rather opinionated world view. If you can use k8s, it's great, but it's also changing rapidly, so be aware you're on the bleeding edge. DockerSwarm is Docker finding a business model. (I'm biased here)
Better bleeding edge than end-of-life. We're using Marathon, and suddenly the entire UI is end-of-life, and your options are "use Marathon back-end only without a UI" or "migrate to DC/OS" which is even more opinionated than K8s. I think the real tell is that Mesosphere don't have the cash/people to maintain a non-DC/OS UI for marathon, and there isn't a community able/willing to do it for them.
> Swarm is an effort by Docker to extend the existing Docker API to make a cluster of machines look like a single Docker API
This isn't really true any more; it's a reference to the original version of Swarm. The version of Swarm that came with Docker 1.12 is completely rethought and not constrained in the same way.
We've been running Kubernetes in production for over a year. It has a steep learning curve and it takes a lot of time to fine-tune the underlying infrastructure (we use CoreOs) to make it production-ready. There still seems to be a lot of shortcomings of k8s that seem like should have been addressed by now:
- it is impossible to trigger a rescheduling / rebalancing. When a new node comes in (via AutoScaling policy or whatever), kubernetes doesn't do anything. Thus, the new nodes can be sitting there doing nothing.
- once a pod has been scheduled onto a node, it never reschedules it anywhere else. The node may be experiencing problems, thus the pod is affected. k8s doesn't do anything to heal that -- it could simply delete the pod so it is rescheduled somewhere else.
- docker itself constantly ships with a lot of bugs. To this day (we are on docker 1.12.6), we constantly have problems with the docker daemon hanging or becoming unresponsive. I'm not sure if k8s can do much about this, but I feel like it should since we don't directly control docker.
- doesn't integrate more tightly with the OS / cloud provider. For example, it could perform health checks and decide if the node should be restarted, terminated, or idle.
All of our services are stateless, so it would be nice to have the option for all the above, especially k8s started as being the solution for stateless apps.
This is right on point with our experience. Kubernetes is great tech but it could be smarter with respect to scheduling decisions.
Docker is always packed full of bugs; we had to increase redundancy everywhere in our stack. It's consistently been the source for me getting paged at night, and now it's the first thing I look at when there is a new issue. Some components I feel could fail more easily: our etcd and Consul clusters have been chugging along fore more than a year, even though they solve a problem much more complex than Docker does. Docker has been "production-ready" for years now, but I would not recommend it. The developers always fix that critical, production-affecting bug in the next release, but it's been disappointing for some time. I look forward to rkt + Kubernetes getting more mature.
> - once a pod has been scheduled onto a node, it never reschedules it anywhere else. The node may be experiencing problems, thus the pod is affected. k8s doesn't do anything to heal that -- it could simply delete the pod so it is rescheduled somewhere else.
IIRC, if a node goes from NodeReady to NodeNotReady, pods are drained from it.
We've been using Docker as well and I can attest to the shear number of bugs and constant regressions. I don't think a release has gone by without a bug that prevents us from updating. From slow pulls, to slow extractions, to issues with SELinux, to namespace incompatibilities, to the daemon just hanging repeatedly. It's frustrating because the technologies around Docker (Mesos, Kubernetes, Marathon, etc) seem to be getting better and more stable while Docker just continues to have issues.
> - docker itself constantly ships with a lot of bugs. To this day (we are on docker 1.12.6), we constantly have problems with the docker daemon hanging or becoming unresponsive. I'm not sure if k8s can do much about this, but I feel like it should since we don't directly control docker.
To be fair, sometimes these problems are due to the kernel. Specifically, the infamous unregister_netdevice ref count issue (https://github.com/docker/docker/issues/5618) has been around for years. One of the comments from a kubernetes dev says they're bypassing the cause and don't see it in GKE production.
> - once a pod has been scheduled onto a node, it never reschedules it anywhere else. The node may be experiencing problems, thus the pod is affected. k8s doesn't do anything to heal that -- it could simply delete the pod so it is rescheduled somewhere else.
Can you please elaborate this? When you are using replication controllers or deployments, don’t they drive the state to the desired/goal state, which is N replicas of a pod? So when the node is shut down, I guess it should be rescheduling those dead pods somewhere else to satisfy the goal state?
With all of this talk about docker, Kubernetes and the rest I feel peer pressured into ditching my monolithic heroku rails app and switching to the distributed services heaven that docker seems too advertise.
Can anybody that has made the switch give me a convincing argument about why I should switch to (or not)?
My feeling that docker is great if you are VP of Engineering at Netflix, but is probably not the best thing if you are starting a startup and just need to get things done.
Disclaimer: I'm not religious about this and I'm totally open to being convinced that I'm wrong.
> With all of this talk about docker, Kubernetes and the rest I feel peer pressured into ditching my monolithic heroku rails app and switching to the distributed services heaven that docker seems too advertise.
I successfully run lots of Docker microservices in production, and I strongly advise you to keep your app on Heroku as long as you can. :-)
Microservices make sense in two circumstances:
1. You have multiple teams of developers, and you want them to have different release cycles and loosely-coupled APIs. In this case, you can let each team have a microservice.
2. There's a module in your app which is self-contained and naturally isolated, with a stable API. You could always just make this a separate Heroku app with its own REST API.
But in general, microservices add complexity and make it harder to do certain kinds of refactorings. You can make them work, if you know what you're doing. For example, ECS +RDS+ALBs is halfway civilized, especially if you manage the configuration with Terraform, and set up a CI server to build Docker images and run tests. But it's still a lot more complex than a single, well-refactored app on Heroku.
I need to write my journey through docker article. Now I get it I'm pretty much 100% convinced it's the best way to go, simply because everyone who needs to work on something gets a clean dev environment that just works. The bit's about deployment/test/ci/production all just about being the same is wonderful.
The next thing I realise is that I'm very happy dealing with infrastructure now in a way I wasn't before; I've looked through docker files and know what they install and why and if anything goes wrong it provides me with an immediate goto which is let's add more servers or lets rebuild the environment and switch over (should there be more users or more load than expected). Docker will buy you time here.
Docker removes the temptation to start editing stuff on servers in the event of issues.
In terms of doing a startup I think other people here are better to advise; if you are MVP no but anything bigger than that I think it'll pay off.
Managing secrets is still an absolute pain though...
It sounds like your current setup is working just fine, so I don't see a compelling reason to switch.
When you start to deploy many applications, and they need to talk to each other, and you need service discovery, automatic restart, rolling upgrades, horizontal scaling, etc - then Kubernetes brings a lot of value.
If you're a startup, then I'd look at the growth you're expecting. Containers scale well, and when you're big, maintaining multiple heroku apps eats into developer time that can be better spent somewhere else.
Of course, if you've just started, and are getting an MVP out the door, don't worry about docker just yet. And also don't listen to the microservices people. It'll be like putting the cart before the horse.
You can switch to Pivotal Web Services[0] (disclosure: I work for Pivotal on Cloud Foundry) and get the best of both worlds.
PWS is based on Cloud Foundry, which allows routing by path. So as an intermediate step towards decomposing your app into services, you can deploy different copies and have them respond to particular routes.
Cloud Foundry uses Heroku's buildpack code, with very minor changes, with additional testing. Your app will stage and run identically.
If you decide to switch to docker images, Cloud Foundry can run those too.
Cloud Foundry is a complete platform, rather than a collection of components. Test-driven, pair programmed, all that jazz. More mature and production-tested than any alternative that I'm aware of.
I think it's awesome, but I'm biased. Feel free to email me.
There are plenty of upsides to the distributed approach. But there are downsides to distributed too which don't get discussed as much. Things like communication between nodes, fault tolerance, monitoring and handling failure. Same case with having many microservices. Also this stuff becomes time consuming if you are a solo dev / small dev team.
IMO one approach isn't better than the other for all cases. Maybe I'm a bit of a laggard here, but I still like Heroku and believe in just doing enough infrastructure to support where your app is / is going in the near future.
The ecosystem around containerization is still emerging and is in a rapid state of flux. I really wouldn't recommend getting production anywhere near it for the next 2 years minimum, and realistically, you probably want to wait more like 5.
Both Docker and k8s change quickly and both lack functionality that most people would consider pretty basic. Google may have transcended into a plane where persistent storage and individually-addressable servers are a thing of the past, but the rest of us haven't. Many things that an admin takes for granted on a normal setup are missing, difficult, convoluted, or flat out impossible on k8s/Docker.
We're converting our 100+ "traditional" cloud servers into a Docker/k8s cluster now, and it's a nightmare. There's really no reason for it. The biggest benefit is a consistent image, but you can get that with much less ridiculous tooling, like Ansible.
My opinion on the long-term: containers will have a permanent role, but I don't think it will be nearly as big as many think. Kubernetes will be refined and become the de-facto "cluster definition language" for deployments and will take a much larger role than containers. It will learn to address all types of networked units (already underway) and cloud interfaces/APIs will likely just be layers on top of it.
The hugely embarrassing bugs and missing features in both k8s and Docker will get fleshed out and fixed over the next 2-3 years, and in 5 years, just as the sheen wears off of this containerland architecture and people start looking for the next premature fad to waste millions of dollars blindly pursuing, it will probably start to be reasonable to run some stable, production-level services in Docker/k8s. ;) It will never be appropriate for everything, despite protests to the contrary.
I think the long-term future for k8s is much brighter than the future for Docker. If Docker can survive under the weight of the VC investments they've taken, they'll probably become a repository management company (and a possible acquisition target for a megacorp that wants to control that) and the docker engine will fall out of use, mostly replaced by a combination of container runtimes: rkt, lxc, probably a forthcoming containerization implementation from Microsoft, and a smattering of smaller ones.
The important thing to remember about Docker and containers is that they're not really new. Containers used to be called jails, zones, etc. They didn't revolutionize infrastructure then and I don't think they will now. The hype is mostly because Docker has hundreds of millions of VC money to burn on looking cool.
If Docker has a killer feature, it's the image registry that makes it easy to "docker pull upstream/image", but the Dockerfile spec itself is too sloppy to really provide the simplicity that people think they're getting, the security practices are abysmal and there will be large-scale pwnage due to it sometime in the not-too-distant future, and the engine's many quirks, bugs, and stupid behaviors do no favors to either the runtime or the company.
If Docker can nurse the momentum from the registry, they may have a future, but the user base for Docker is pretty hard to lock in, so I dunno.
tl;dr Don't use either. Learn k8s slowly over the next 2 years as they work out the kinks, since it will play a larger role in the future. In 5 years, you may want to use some of this on an important project, but right now, it's all a joke, and, in general, the companies that are switching now are making a very bad decision.
Isn't the goal of software development to reduce complexity, both in real life and the software itself? I would argue is if your work is on Heroku and you know what your doing, why chase after intermingled microservice hell?
I haven't worked anywhere that's doing containerization since the rise of the container managers being discussed here, but I can comment on some of the benefits we saw from deploying Docker at a previous job in what was, at the time, a fairly standard Consul/Consul-Template/HAProxy configuration with Terraform to provision all the host instances.
1. We could run sanity tests, in production, prior to a release going live. Our deploy scripts would bring up the Docker container and, prior to registering as live with Consul, make a few different requests to ensure that the app had started up cleanly and was communicating with the database. Only after the new container was up and handling live traffic would the old Docker container be removed from Consul and stopped. This only caught one bad release, but that's short bit of unpleasantness that we avoided for our customers.
2. Deployments were immutable, which made rollbacks a breeze. Just update Consul to indicate that a previous image ID is the current ID and re-deployment would be triggered automatically. We wrote a script to handle querying our private Docker registry based on a believed-good date and updating Consul with the proper ID. Thankfully, we only had to run the script a couple of times.
3. Deployments were faster and easier to debug. CI was responsible for running all tests, building the Docker image and updating Consul. That's it. Each running instance had an agent that was responsible for monitoring Consul for changes and the deploy process was a single tens-of-megabytes download and an almost-instantaneous Docker start. Our integration testing environment could also pull changes and deploy in parallel.
4. Setting up a separate testing environment was trivial. We'd just re-run our Terraform scripts to point to a different instance of Consul, and it would provision everything just as it would in production, albeit sized (number/size of instances) according to parameterized Terraform values.
Docker also made a lot of things easier on the development side too. We made a development version of our database into a Docker image, so the instructions for setting up a completely isolated, offline capable dev environment were literally install Docker/Fig (this was before Docker had equivalent functionality), clone the repo and tell Fig to start (then wait for Docker to pull several gigs of images, but that was a one-time cost.)
As I see it, the main thing that Kubernetes and the rest of the container managers will buy you is better utilization of your hardware/instances. We had to provision instances that were dedicated to a specific function (i.e. web tier) or make the decision to re-use instances for two different purposes. But the mapping between docker container and instance/auto-scaling group was static. Container managers can dynamically shift workloads to ensure that as much of your compute capacity is used as possible. It was something we considered, but decided our AWS spend wasn't large enough to justify the dev time to replace our existing setup.
Having not used Heroku, I can't say how much of this their tooling gives you, but I think it comes down to running your own math around your size and priorities to say whether Docker and/or any of the higher-level abstractions are worth it for your individual situation. Containers are established enough to make a pretty good estimate for how long it will take you to come up to speed on the technologies and design/implement your solution. For reference, it took 1 developer about a week for us to work up our Docker/Consul/Terraform solution. If you look at the problems you're solving, you should be able to make a pretty good swag at how much those problems are costing you (kinda the way that we did when we decided that Kubernetes wouldn't save us enough AWS spend to justify the dev time to modify our setup). Then compare that to the value of items on your roadmap and do whatever has the highest value. There's no universally correct answer.
>" I’ve spoken with several DevOps who argue that compared with running a Kubernetes or Mesos cluster, Docker Swarm is a snap."
I'm always leary of statements like that. My experience with cluster managers(Yarn, Mesos) and distributed systems in general is that they are almost never "a snap" to run once you move past trivial workloads and requirements.
Not many startups move past trivial workloads. And if you do, it's a good problem to have. If swarm can get your mvp up quickly, it might be a good choice.
Yep, that's my exact experience with Swarm (1.13). I want the higher-level abstractions like K8s Pods and Deployments, but I absolutely cannot wait around for Swarm to stabilize those kind of things.
The author is (mis)quoting me there. I meant the initial process of installing and getting Swarm running is much easier than k8s or Mesos, not that on-going maintenance was easier.
Also, since I wrote that, k8s has done a lot of work to simplify installation with kubeadm and other improvements.
Quoting from the disadvantages section: "Second, Kubernetes excels at automatically fixing problems. But it’s so good at it that containers can crash and be restarted so fast you don’t notice your containers are crashing. "
A key part of Kubernetes is to the bring the state of the system to that of the spec, through a series of watches - and in some places polling (config maps watching in kubectl as of 1.4.x) - this may look like magic, but it's not fixing problems per se its that your desired state of the system (the spec) is different to what is observed (the status). This is not a disadvantage. Kubernetes if not for someone sitting at the terminal expecting a bell or stop the world when problems happens, although I guess you could configure it that way and fight against the system.
Unfortunately, Mesosphere is no longer focusing on Marathon as an Apache Mesos framework and is instead integrating it into DC/OS, their commercial offering. Open-source Mesos+Marathon no longer appears to be an option.
I built Jollyturns (https://jollyturns.com) which has a fairly large server-side component. Before starting the work on Jollyturns I worked at Google (I left in 2010) on a variety of infrastructure projects, where I got to use first hand Borg and all the other services running on it.
When I started more than 5 years ago few of the options mentioned in the article were available, so I just used Xen running on bare metal. I just wanted to get things done as opposed to forever experimenting with infrastructure. Not to mention the load was inexistent, so everything was easy to manage.
After I launched the mobile app, I decided to spend some time on the infrastructure. 2 years ago I experimented with Marathon, which was originally developed at Twitter, probably by a bunch of former Google employees. The reason is Marathon felt very much like Borg: you could see the jobs you launched in a pretty nice Web interface, including their log files, resource utilization and so on. Deploying it on bare metal machines however was an exercise in frustration since Marathon relied heavily on Apache Mesos. For a former Googler, Mesos had some weird terminology and was truly difficult to understand what the heck was going on. Running Marathon on top of it had major challenges: when long-running services failed you could not figure out why things were not working. So after about 2 weeks I gave up on it, and went back to manually managed Xen VMs.
Around the same time I spent few days with Docker Swarm. If you're familiar with Borg/Kubernetes, Swarm is very different from it since - at least when it started, you had to allocate services to machines by hand. I wrote it off quickly since allocating dockerized services on physical machines was not my idea of cluster management.
Last year I switched to Kubernetes (version 1.2) since it's the closest to what I expect from a cluster management system. The version I've been using in production has a lot of issues: high availability (HA) for its components is almost non-existent. I had to setup Kube in such a way that its components have some resemblance of HA. In the default configuration, Kubernetes installs its control components on a single machine. If that machine fails or is rebooted your entire cluster disappears.
Even with all these issues, Kubernetes solves a lot of problems for you. The flannel networking infrastructure greatly simplifies the deployment of docker containers, since you don't need to worry about routing traffic between your containers.
Don't be fooled by the title of the bug report, the same component used in kubectl to implement HA could be used inside Kube' servers for the same reason. I guess these days the Google engineers working on Kubernetes have no real experience deploying large services on Borg inside Google. Such a pity!
FYI, Kubernetes can do HA: the issues you pointed to are both about client connectivity to multiple API servers, which is typically solved today either using a load-balancer, or by using DNS with multiple A records (which actually works surprisingly well with go clients). The issues you pointed to are about a potential third way, where you would configure a client with multiple server names/addresses, and the client would failover between them.
[+] [-] stuntkite|9 years ago|reply
If you run OSX, TheNewNormal's solo and cluster Kube setup is a breeze and has the Deis Workflow as a one command install, then you have full local parity with deployment.
When I started with straight Docker figuring out the transition to the real world was be really daunting. Getting into a real production environment that I could compost quickly as I learned learn the paradigms was invaluable. Articles like this seem to buzzword their way into making the path muddier than it actually is.
Another big help for me was any time I needed to set something up, first thing, I'd look at how the bleeding edge was being done with Alpine. Practically you might not always run that light, but that community has all the answers.
[1]https://deis.com/ [2]https://github.com/TheNewNormal/kube-solo-osx [3]https://github.com/TheNewNormal/kube-cluster-osx
[+] [-] tedmiston|9 years ago|reply
https://flynn.io/
[+] [-] bogomipz|9 years ago|reply
I curious how those relate to each other. Are you using SaltStack to bootstrap Kubernetes?
[+] [-] jaaron|9 years ago|reply
Aside from that, it's not a bad overview. I particularly like this quote:
“Mesos and Kubernetes are largely aimed at solving similar problems of running clustered applications; they have different histories and different approaches to solving the problem. Mesos focuses its energy on very generic scheduling and plugging in multiple different schedulers.” In contrast, says Burns, “Kubernetes was designed from the ground up to be an environment for building distributed applications from containers. It includes primitives for replication and service discovery as core primitives, whereas such things are added via frameworks in Mesos. … Swarm is an effort by Docker to extend the existing Docker API to make a cluster of machines look like a single Docker API.”
Mesos is useful when you have requirements that don't fit in k8s' rather opinionated world view. If you can use k8s, it's great, but it's also changing rapidly, so be aware you're on the bleeding edge. DockerSwarm is Docker finding a business model. (I'm biased here)
[+] [-] lowbloodsugar|9 years ago|reply
[+] [-] amouat|9 years ago|reply
This isn't really true any more; it's a reference to the original version of Swarm. The version of Swarm that came with Docker 1.12 is completely rethought and not constrained in the same way.
[+] [-] ldar15|9 years ago|reply
[deleted]
[+] [-] webo|9 years ago|reply
- it is impossible to trigger a rescheduling / rebalancing. When a new node comes in (via AutoScaling policy or whatever), kubernetes doesn't do anything. Thus, the new nodes can be sitting there doing nothing.
- once a pod has been scheduled onto a node, it never reschedules it anywhere else. The node may be experiencing problems, thus the pod is affected. k8s doesn't do anything to heal that -- it could simply delete the pod so it is rescheduled somewhere else.
- docker itself constantly ships with a lot of bugs. To this day (we are on docker 1.12.6), we constantly have problems with the docker daemon hanging or becoming unresponsive. I'm not sure if k8s can do much about this, but I feel like it should since we don't directly control docker.
- doesn't integrate more tightly with the OS / cloud provider. For example, it could perform health checks and decide if the node should be restarted, terminated, or idle.
All of our services are stateless, so it would be nice to have the option for all the above, especially k8s started as being the solution for stateless apps.
[+] [-] Artemis2|9 years ago|reply
Docker is always packed full of bugs; we had to increase redundancy everywhere in our stack. It's consistently been the source for me getting paged at night, and now it's the first thing I look at when there is a new issue. Some components I feel could fail more easily: our etcd and Consul clusters have been chugging along fore more than a year, even though they solve a problem much more complex than Docker does. Docker has been "production-ready" for years now, but I would not recommend it. The developers always fix that critical, production-affecting bug in the next release, but it's been disappointing for some time. I look forward to rkt + Kubernetes getting more mature.
> - once a pod has been scheduled onto a node, it never reschedules it anywhere else. The node may be experiencing problems, thus the pod is affected. k8s doesn't do anything to heal that -- it could simply delete the pod so it is rescheduled somewhere else.
IIRC, if a node goes from NodeReady to NodeNotReady, pods are drained from it.
[+] [-] cddotdotslash|9 years ago|reply
[+] [-] alxvio|9 years ago|reply
To be fair, sometimes these problems are due to the kernel. Specifically, the infamous unregister_netdevice ref count issue (https://github.com/docker/docker/issues/5618) has been around for years. One of the comments from a kubernetes dev says they're bypassing the cause and don't see it in GKE production.
[+] [-] alpb|9 years ago|reply
Can you please elaborate this? When you are using replication controllers or deployments, don’t they drive the state to the desired/goal state, which is N replicas of a pod? So when the node is shut down, I guess it should be rescheduling those dead pods somewhere else to satisfy the goal state?
[+] [-] jaxxstorm|9 years ago|reply
Are you saying this doesn't work?
[+] [-] tom_pulo|9 years ago|reply
Can anybody that has made the switch give me a convincing argument about why I should switch to (or not)? My feeling that docker is great if you are VP of Engineering at Netflix, but is probably not the best thing if you are starting a startup and just need to get things done.
Disclaimer: I'm not religious about this and I'm totally open to being convinced that I'm wrong.
[+] [-] ekidd|9 years ago|reply
I successfully run lots of Docker microservices in production, and I strongly advise you to keep your app on Heroku as long as you can. :-)
Microservices make sense in two circumstances:
1. You have multiple teams of developers, and you want them to have different release cycles and loosely-coupled APIs. In this case, you can let each team have a microservice.
2. There's a module in your app which is self-contained and naturally isolated, with a stable API. You could always just make this a separate Heroku app with its own REST API.
But in general, microservices add complexity and make it harder to do certain kinds of refactorings. You can make them work, if you know what you're doing. For example, ECS +RDS+ALBs is halfway civilized, especially if you manage the configuration with Terraform, and set up a CI server to build Docker images and run tests. But it's still a lot more complex than a single, well-refactored app on Heroku.
[+] [-] andy_ppp|9 years ago|reply
The next thing I realise is that I'm very happy dealing with infrastructure now in a way I wasn't before; I've looked through docker files and know what they install and why and if anything goes wrong it provides me with an immediate goto which is let's add more servers or lets rebuild the environment and switch over (should there be more users or more load than expected). Docker will buy you time here.
Docker removes the temptation to start editing stuff on servers in the event of issues.
In terms of doing a startup I think other people here are better to advise; if you are MVP no but anything bigger than that I think it'll pay off.
Managing secrets is still an absolute pain though...
[+] [-] wstrange|9 years ago|reply
When you start to deploy many applications, and they need to talk to each other, and you need service discovery, automatic restart, rolling upgrades, horizontal scaling, etc - then Kubernetes brings a lot of value.
[+] [-] ktta|9 years ago|reply
Of course, if you've just started, and are getting an MVP out the door, don't worry about docker just yet. And also don't listen to the microservices people. It'll be like putting the cart before the horse.
[+] [-] jacques_chester|9 years ago|reply
PWS is based on Cloud Foundry, which allows routing by path. So as an intermediate step towards decomposing your app into services, you can deploy different copies and have them respond to particular routes.
Cloud Foundry uses Heroku's buildpack code, with very minor changes, with additional testing. Your app will stage and run identically.
If you decide to switch to docker images, Cloud Foundry can run those too.
Cloud Foundry is a complete platform, rather than a collection of components. Test-driven, pair programmed, all that jazz. More mature and production-tested than any alternative that I'm aware of.
I think it's awesome, but I'm biased. Feel free to email me.
[0] https://run.pivotal.io/
[+] [-] tedmiston|9 years ago|reply
"The grass is always greener on the other side"
There are plenty of upsides to the distributed approach. But there are downsides to distributed too which don't get discussed as much. Things like communication between nodes, fault tolerance, monitoring and handling failure. Same case with having many microservices. Also this stuff becomes time consuming if you are a solo dev / small dev team.
IMO one approach isn't better than the other for all cases. Maybe I'm a bit of a laggard here, but I still like Heroku and believe in just doing enough infrastructure to support where your app is / is going in the near future.
[+] [-] cookiecaper|9 years ago|reply
Both Docker and k8s change quickly and both lack functionality that most people would consider pretty basic. Google may have transcended into a plane where persistent storage and individually-addressable servers are a thing of the past, but the rest of us haven't. Many things that an admin takes for granted on a normal setup are missing, difficult, convoluted, or flat out impossible on k8s/Docker.
We're converting our 100+ "traditional" cloud servers into a Docker/k8s cluster now, and it's a nightmare. There's really no reason for it. The biggest benefit is a consistent image, but you can get that with much less ridiculous tooling, like Ansible.
My opinion on the long-term: containers will have a permanent role, but I don't think it will be nearly as big as many think. Kubernetes will be refined and become the de-facto "cluster definition language" for deployments and will take a much larger role than containers. It will learn to address all types of networked units (already underway) and cloud interfaces/APIs will likely just be layers on top of it.
The hugely embarrassing bugs and missing features in both k8s and Docker will get fleshed out and fixed over the next 2-3 years, and in 5 years, just as the sheen wears off of this containerland architecture and people start looking for the next premature fad to waste millions of dollars blindly pursuing, it will probably start to be reasonable to run some stable, production-level services in Docker/k8s. ;) It will never be appropriate for everything, despite protests to the contrary.
I think the long-term future for k8s is much brighter than the future for Docker. If Docker can survive under the weight of the VC investments they've taken, they'll probably become a repository management company (and a possible acquisition target for a megacorp that wants to control that) and the docker engine will fall out of use, mostly replaced by a combination of container runtimes: rkt, lxc, probably a forthcoming containerization implementation from Microsoft, and a smattering of smaller ones.
The important thing to remember about Docker and containers is that they're not really new. Containers used to be called jails, zones, etc. They didn't revolutionize infrastructure then and I don't think they will now. The hype is mostly because Docker has hundreds of millions of VC money to burn on looking cool.
If Docker has a killer feature, it's the image registry that makes it easy to "docker pull upstream/image", but the Dockerfile spec itself is too sloppy to really provide the simplicity that people think they're getting, the security practices are abysmal and there will be large-scale pwnage due to it sometime in the not-too-distant future, and the engine's many quirks, bugs, and stupid behaviors do no favors to either the runtime or the company.
If Docker can nurse the momentum from the registry, they may have a future, but the user base for Docker is pretty hard to lock in, so I dunno.
tl;dr Don't use either. Learn k8s slowly over the next 2 years as they work out the kinks, since it will play a larger role in the future. In 5 years, you may want to use some of this on an important project, but right now, it's all a joke, and, in general, the companies that are switching now are making a very bad decision.
[+] [-] ben_jones|9 years ago|reply
[+] [-] luisehk|9 years ago|reply
[+] [-] curun1r|9 years ago|reply
1. We could run sanity tests, in production, prior to a release going live. Our deploy scripts would bring up the Docker container and, prior to registering as live with Consul, make a few different requests to ensure that the app had started up cleanly and was communicating with the database. Only after the new container was up and handling live traffic would the old Docker container be removed from Consul and stopped. This only caught one bad release, but that's short bit of unpleasantness that we avoided for our customers.
2. Deployments were immutable, which made rollbacks a breeze. Just update Consul to indicate that a previous image ID is the current ID and re-deployment would be triggered automatically. We wrote a script to handle querying our private Docker registry based on a believed-good date and updating Consul with the proper ID. Thankfully, we only had to run the script a couple of times.
3. Deployments were faster and easier to debug. CI was responsible for running all tests, building the Docker image and updating Consul. That's it. Each running instance had an agent that was responsible for monitoring Consul for changes and the deploy process was a single tens-of-megabytes download and an almost-instantaneous Docker start. Our integration testing environment could also pull changes and deploy in parallel.
4. Setting up a separate testing environment was trivial. We'd just re-run our Terraform scripts to point to a different instance of Consul, and it would provision everything just as it would in production, albeit sized (number/size of instances) according to parameterized Terraform values.
Docker also made a lot of things easier on the development side too. We made a development version of our database into a Docker image, so the instructions for setting up a completely isolated, offline capable dev environment were literally install Docker/Fig (this was before Docker had equivalent functionality), clone the repo and tell Fig to start (then wait for Docker to pull several gigs of images, but that was a one-time cost.)
As I see it, the main thing that Kubernetes and the rest of the container managers will buy you is better utilization of your hardware/instances. We had to provision instances that were dedicated to a specific function (i.e. web tier) or make the decision to re-use instances for two different purposes. But the mapping between docker container and instance/auto-scaling group was static. Container managers can dynamically shift workloads to ensure that as much of your compute capacity is used as possible. It was something we considered, but decided our AWS spend wasn't large enough to justify the dev time to replace our existing setup.
Having not used Heroku, I can't say how much of this their tooling gives you, but I think it comes down to running your own math around your size and priorities to say whether Docker and/or any of the higher-level abstractions are worth it for your individual situation. Containers are established enough to make a pretty good estimate for how long it will take you to come up to speed on the technologies and design/implement your solution. For reference, it took 1 developer about a week for us to work up our Docker/Consul/Terraform solution. If you look at the problems you're solving, you should be able to make a pretty good swag at how much those problems are costing you (kinda the way that we did when we decided that Kubernetes wouldn't save us enough AWS spend to justify the dev time to modify our setup). Then compare that to the value of items on your roadmap and do whatever has the highest value. There's no universally correct answer.
[+] [-] bogomipz|9 years ago|reply
I'm always leary of statements like that. My experience with cluster managers(Yarn, Mesos) and distributed systems in general is that they are almost never "a snap" to run once you move past trivial workloads and requirements.
[+] [-] atomi|9 years ago|reply
[+] [-] btmiller|9 years ago|reply
[+] [-] Axsuul|9 years ago|reply
[+] [-] amouat|9 years ago|reply
Also, since I wrote that, k8s has done a lot of work to simplify installation with kubeadm and other improvements.
[+] [-] sitepodmatt|9 years ago|reply
A key part of Kubernetes is to the bring the state of the system to that of the spec, through a series of watches - and in some places polling (config maps watching in kubectl as of 1.4.x) - this may look like magic, but it's not fixing problems per se its that your desired state of the system (the spec) is different to what is observed (the status). This is not a disadvantage. Kubernetes if not for someone sitting at the terminal expecting a bell or stop the world when problems happens, although I guess you could configure it that way and fight against the system.
[+] [-] technologyvault|9 years ago|reply
Mesosphere looks like it works similar to Nanobox (http://nanobox.io)
[+] [-] lowbloodsugar|9 years ago|reply
[+] [-] nemothekid|9 years ago|reply
[+] [-] bogomipz|9 years ago|reply
https://mesosphere.com/blog/2016/04/19/open-source-dcos/
[+] [-] poooogles|9 years ago|reply
[+] [-] ovidiup|9 years ago|reply
When I started more than 5 years ago few of the options mentioned in the article were available, so I just used Xen running on bare metal. I just wanted to get things done as opposed to forever experimenting with infrastructure. Not to mention the load was inexistent, so everything was easy to manage.
After I launched the mobile app, I decided to spend some time on the infrastructure. 2 years ago I experimented with Marathon, which was originally developed at Twitter, probably by a bunch of former Google employees. The reason is Marathon felt very much like Borg: you could see the jobs you launched in a pretty nice Web interface, including their log files, resource utilization and so on. Deploying it on bare metal machines however was an exercise in frustration since Marathon relied heavily on Apache Mesos. For a former Googler, Mesos had some weird terminology and was truly difficult to understand what the heck was going on. Running Marathon on top of it had major challenges: when long-running services failed you could not figure out why things were not working. So after about 2 weeks I gave up on it, and went back to manually managed Xen VMs.
Around the same time I spent few days with Docker Swarm. If you're familiar with Borg/Kubernetes, Swarm is very different from it since - at least when it started, you had to allocate services to machines by hand. I wrote it off quickly since allocating dockerized services on physical machines was not my idea of cluster management.
Last year I switched to Kubernetes (version 1.2) since it's the closest to what I expect from a cluster management system. The version I've been using in production has a lot of issues: high availability (HA) for its components is almost non-existent. I had to setup Kube in such a way that its components have some resemblance of HA. In the default configuration, Kubernetes installs its control components on a single machine. If that machine fails or is rebooted your entire cluster disappears.
Even with all these issues, Kubernetes solves a lot of problems for you. The flannel networking infrastructure greatly simplifies the deployment of docker containers, since you don't need to worry about routing traffic between your containers.
Even now Kubernetes doesn't do HA:
https://github.com/kubernetes/kubernetes/issues/26852
https://github.com/kubernetes/kubernetes/issues/18174
Don't be fooled by the title of the bug report, the same component used in kubectl to implement HA could be used inside Kube' servers for the same reason. I guess these days the Google engineers working on Kubernetes have no real experience deploying large services on Borg inside Google. Such a pity!
[+] [-] justinsb|9 years ago|reply