Etcd and Zookeeper provide essentially the same functionality. They are both a strongly consistent key/value stores that support notifications to clients of changes. These two projects are limited to service discovery, there are a number of projects that depend on zookeeper (that theoretically could be ported to etcd) for various distributed state management such as Kafka, Storm, and Hadoop.
Skydock (Skydns) and Consul provide automagic service discovery primarily through dns. They use a strongly consistent backend (think Zookeeper or Etcd), to keep track of what servers are active and what applications they run. They then expose a DNS that routes through said servers. So lets say you had a client application that would talk to a node application that could be on any number of servers. What you could do is hard code that list into your application and randomly select one, in order to "fake" load balancing. However every time a machine went up or down you would have to update that list. What Consul provides is you just tell your app to connect to "mynodeapp.consul" and then consul will give you the proper address of one of your node apps.
Consul and Skydock are both applications that build on top of a tool like Zookeeper and Etcd. [1]
Finally all these tools, you can essentially consider "ops" tools to manage the lifetime of application, services and servers "in the cloud". What a developer ideally wants to do is just push code and not have to worry about what servers are running what, and worry about failover and the like. What Flynn provides (if I get it), is a diy Heroku like platform that makes use of tools like Etcd, Skydns, Docker(?) and others. With Flynn, I believe the goal is to radically simplify ops. Another project that I believe may be similar to Flynn is Apache Mesos.
[1] Technically Consul and Skydock, IIRC don't actually use Etcd or Zookeeper, but both implement an underlying protocol to achieve the same thing - Raft, but they could use Etcd or Zookeeper instead of implementing the protocol themselves.
What does Flynn do? The word "ops" is so central to the pitch and so ambiguous to me that I have absolutely no idea what the function of this software (it is software, right?) is.
In short yes, it's like self-hosted heroku. They use docker and the heroku buildpack interface and many different components to coordinate container management, logging, etc to provide a whole system.
Can anyone from the Flynn project comment on this versus Deis? I've tried Dokku and thought that was pretty slick, but ultimately decided to move on to a Deis/Chef devops setup, because dokku was too lightweight for my needs and Flynn wasn't ready.
Running it on your own server: We're currently only providing step by step instructions for vagrant, but you should be able to use the script in the vagrant file on your own host with some modifications on your own hardware. Happy to help in IRC (#flynn on freenode)
Flynn Dashboard: At least some part of this will end up as open source in the not distant future, we're trying to figure out the best strategy for additional tools like the dashboard and what parts should be packaged.
On that note, it would be great to know what AWS `User Policy` Flynn needs / expects so I could grant it a unique IAM key with limited access for creating servers, etc.
Very nit-picky comment, but: Can you explain in one or two sentences what the value proposition is?
I might be dense, but it took me quite a bit of clicking around, reading HN comments, and poking at the Omega paper to understand what you want to do their. (Well, I think I understand. The "private Heroku" line points in the right direction, at least)
Flynn is the product ops should provide to developers.
Flynn is a single platform that runs all your services from databases to applications to individual Linux processes. With Flynn ops teams can stop being consultants and start providing a single product to their internal "customers". Basically you can deploy and scale whatever you want all in one place on any infrastructure without having to think about individual hosts.
Are these sorts of services good enough? I want somewhere to deploy my Django apps, ideally I should be able to get a beefy server and deploy every app as a single thing on that server, completely compartmentalized.
The problem I keep hitting is that these things have a tradeoff between ease of use and power. I want to use Docker, but it has no easy way to say "take this file that contains instructions and make everything". You can write Dockerfiles, but you can only use one part of the stack in them, otherwise you run into trouble.
I'm hoping Flynn will allow me to use a database, Redis, uwsgi, nginx, etc with a simple deployment command. Does anyone know of anything else like this?
Right now Flynn Layer 0 is very immature compared to Mesos, but yes they're trying to solve similar problems. After Flynn reaches production stability and builds out more features, we expect Layer 0 to be a valid (and much lighter weight) alternative to Mesos that we hope will be a superior solution for a broad class of users.
I feel like the projects have very different prospective user bases and communities (Mesos is an Apache project, hundreds of thousands of lines of C++; Flynn Layer 0 is run by a startup and only a few thousand lines of Go) and will likely develop in very different directions to service those communities.
That being said, we've explored creating a version of Flynn layer 1 components that run on Mesos instead of Flynn Layer 0 for users who are already deeply invested in the Mesos ecosystem.
1. Mesos is very mature software, we take reliability, quality, and backwards compatible upgrades very seriously as there are companies currently relying on these properties.
2. At a high level, Mesos aims to provide abstraction for building distributed applications. This means "frameworks" are either built on top, like Aurora, Marathon, Chronos, etc. Or frameworks are existing distributed applications that are made to run on top, like Spark, Hadoop, Jenkins, distcc, etc. The goal being to run these distributed applications together in the same cluster in order to simplify operational complexity and gain efficiency. In this sense, Mesos is trying to build and grow the common lower level abstraction, akin to a "kernel" for the datacenter.
3. Flynn is aiming to solve a much broader set of problems, by providing a PaaS, (Mesos is more like an IaaS, PaaS should be built / run on top). Flynn is aiming to provide something that is immediately useful on its own, that means things in the layer 1 listed on the website are included. Flynn is aiming to provide some of these "schedulers" out of the box. That is my understanding from reading their website.
4. I'm not sure the authors of Flynn fully comprehend the subtlety that exists between Omega and Mesos. Unfortunately, there are some primitives in Mesos that have been discussed for quite some time and have yet to be implemented that aim to alleviate the issues brought up the Omega paper. I think the Omega model makes sense at Google, where they have complete control over the schedulers. However, in the open source world, I think the Mesos model is more appropriate (this claim really warrants it's own post). With additional primitives, like "optimistic offers", "revocable offers", and "over-subscription", many of the issues discussed in the Omega paper should be remediated.
It seems to partially duplicate the functionality of Mesos, as they are writing their own task scheduling framework [0] based on Google's Omega [1].
The Omega authors claim that Mesos' system of application specific schedulers accepting resource offers from the Mesos master is well suited toward short-lived jobs (think ephemeral Map/Reduce or MPI type workloads) but is not well suited for long lived 'service' jobs (like a Rails app or DB server). As this seems to be an important use-case for Flynn, it would seem like a valid architecture decision to not use Mesos.
Does anyone know if Flynn components can be run on CoreOS?
Also, does Fylnn offer the ability to use sub-domains for accessing apps, with either a random string or custom, eg. //dev.example.com? Thx
Flynn is very interesting! I've been following its development since the project was made public and the team behind it made great progress during the last months.
My first impression was that it has to do something with TRON's Kevin Flynn. Quoting him:
"I tried to picture clusters of information as they moved through the computer. What did they look like? Ships? Motorcycles? Were the circuits like freeways?"
I too am interested in the comparison with such projects and the differing requirements they cater for. I'm also working on some notes on https://wiki.thingsandstuff.org/Stack if anyone is interested in bettering the order/outline/etc.
[+] [-] dominotw|12 years ago|reply
[1]. etcd [2] skydock [3] consul [4] zookeper ( ok this is not new) [5] flynn
They all seem to be doing the exact same to me.
1. https://github.com/coreos/etcd 2. https://github.com/crosbymichael/skydock 3. https://github.com/hashicorp/consul/ 5.https://flynn.io/
Edit: flynn uses discoverd (https://github.com/flynn/discoverd)
[+] [-] nemothekid|12 years ago|reply
Skydock (Skydns) and Consul provide automagic service discovery primarily through dns. They use a strongly consistent backend (think Zookeeper or Etcd), to keep track of what servers are active and what applications they run. They then expose a DNS that routes through said servers. So lets say you had a client application that would talk to a node application that could be on any number of servers. What you could do is hard code that list into your application and randomly select one, in order to "fake" load balancing. However every time a machine went up or down you would have to update that list. What Consul provides is you just tell your app to connect to "mynodeapp.consul" and then consul will give you the proper address of one of your node apps.
Consul and Skydock are both applications that build on top of a tool like Zookeeper and Etcd. [1]
Finally all these tools, you can essentially consider "ops" tools to manage the lifetime of application, services and servers "in the cloud". What a developer ideally wants to do is just push code and not have to worry about what servers are running what, and worry about failover and the like. What Flynn provides (if I get it), is a diy Heroku like platform that makes use of tools like Etcd, Skydns, Docker(?) and others. With Flynn, I believe the goal is to radically simplify ops. Another project that I believe may be similar to Flynn is Apache Mesos.
[1] Technically Consul and Skydock, IIRC don't actually use Etcd or Zookeeper, but both implement an underlying protocol to achieve the same thing - Raft, but they could use Etcd or Zookeeper instead of implementing the protocol themselves.
[+] [-] andrewmunsell|12 years ago|reply
[+] [-] SEJeff|12 years ago|reply
http://www.consul.io/intro/vs/index.html
Also, discoverd is pluggable, but currently relies on etcd.
[+] [-] sirsar|12 years ago|reply
What does Flynn do? The word "ops" is so central to the pitch and so ambiguous to me that I have absolutely no idea what the function of this software (it is software, right?) is.
[+] [-] benatkin|12 years ago|reply
Its goals are more ambitious than Dokku but it's too early to tell if it will be any good.
[+] [-] Jemaclus|12 years ago|reply
[+] [-] jmspring|12 years ago|reply
[+] [-] erkkie|12 years ago|reply
[+] [-] cellis|12 years ago|reply
[+] [-] danielsiders|12 years ago|reply
I haven't looked closely at Deis in several months, but Flynn doesn't use Chef (or anything similar) anywhere and don't plan to.
[+] [-] akhatri_aus|12 years ago|reply
Is there a way to install it manually on a server with stuff already on it?
Also is the source of the aws flynn.cupcake.io tool on github, it would be nice to see whats being installed onto an aws cluster.
[+] [-] danielsiders|12 years ago|reply
Flynn Dashboard: At least some part of this will end up as open source in the not distant future, we're trying to figure out the best strategy for additional tools like the dashboard and what parts should be packaged.
[+] [-] ghayes|12 years ago|reply
[+] [-] groby_b|12 years ago|reply
I might be dense, but it took me quite a bit of clicking around, reading HN comments, and poking at the Omega paper to understand what you want to do their. (Well, I think I understand. The "private Heroku" line points in the right direction, at least)
[+] [-] danielsiders|12 years ago|reply
Flynn is a single platform that runs all your services from databases to applications to individual Linux processes. With Flynn ops teams can stop being consultants and start providing a single product to their internal "customers". Basically you can deploy and scale whatever you want all in one place on any infrastructure without having to think about individual hosts.
[+] [-] zakelfassi|12 years ago|reply
[+] [-] StavrosK|12 years ago|reply
The problem I keep hitting is that these things have a tradeoff between ease of use and power. I want to use Docker, but it has no easy way to say "take this file that contains instructions and make everything". You can write Dockerfiles, but you can only use one part of the stack in them, otherwise you run into trouble.
I'm hoping Flynn will allow me to use a database, Redis, uwsgi, nginx, etc with a simple deployment command. Does anyone know of anything else like this?
[+] [-] danielsiders|12 years ago|reply
[+] [-] cheez|12 years ago|reply
[+] [-] danielsiders|12 years ago|reply
[1] https://news.ycombinator.com/item?id=6058662
[+] [-] necubi|12 years ago|reply
[+] [-] danielsiders|12 years ago|reply
I feel like the projects have very different prospective user bases and communities (Mesos is an Apache project, hundreds of thousands of lines of C++; Flynn Layer 0 is run by a startup and only a few thousand lines of Go) and will likely develop in very different directions to service those communities.
That being said, we've explored creating a version of Flynn layer 1 components that run on Mesos instead of Flynn Layer 0 for users who are already deeply invested in the Mesos ecosystem.
[+] [-] mahler|12 years ago|reply
2. At a high level, Mesos aims to provide abstraction for building distributed applications. This means "frameworks" are either built on top, like Aurora, Marathon, Chronos, etc. Or frameworks are existing distributed applications that are made to run on top, like Spark, Hadoop, Jenkins, distcc, etc. The goal being to run these distributed applications together in the same cluster in order to simplify operational complexity and gain efficiency. In this sense, Mesos is trying to build and grow the common lower level abstraction, akin to a "kernel" for the datacenter.
3. Flynn is aiming to solve a much broader set of problems, by providing a PaaS, (Mesos is more like an IaaS, PaaS should be built / run on top). Flynn is aiming to provide something that is immediately useful on its own, that means things in the layer 1 listed on the website are included. Flynn is aiming to provide some of these "schedulers" out of the box. That is my understanding from reading their website.
4. I'm not sure the authors of Flynn fully comprehend the subtlety that exists between Omega and Mesos. Unfortunately, there are some primitives in Mesos that have been discussed for quite some time and have yet to be implemented that aim to alleviate the issues brought up the Omega paper. I think the Omega model makes sense at Google, where they have complete control over the schedulers. However, in the open source world, I think the Mesos model is more appropriate (this claim really warrants it's own post). With additional primitives, like "optimistic offers", "revocable offers", and "over-subscription", many of the issues discussed in the Omega paper should be remediated.
Dislaimer: I am a Mesos PMC member. :)
[+] [-] hendzen|12 years ago|reply
The Omega authors claim that Mesos' system of application specific schedulers accepting resource offers from the Mesos master is well suited toward short-lived jobs (think ephemeral Map/Reduce or MPI type workloads) but is not well suited for long lived 'service' jobs (like a Rails app or DB server). As this seems to be an important use-case for Flynn, it would seem like a valid architecture decision to not use Mesos.
[0] - https://github.com/flynn/flynn-host/tree/master/sampi
[1] - http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/S...
[+] [-] gales|12 years ago|reply
[+] [-] danielsiders|12 years ago|reply
You can add any route to any app in Flynn, so different subdomains, domains, and TCP ports are pretty easy.
[+] [-] Artemis2|12 years ago|reply
[+] [-] danielsiders|12 years ago|reply
[+] [-] andrewmunsell|12 years ago|reply
[+] [-] Titanous|12 years ago|reply
[+] [-] auvi|12 years ago|reply
[+] [-] ochoseis|12 years ago|reply
[Edit] I would also ask the same about Apache Mesos since that seems like it's coming up a lot in this discussion as well.
[+] [-] milkmiruku|12 years ago|reply
[+] [-] sdegutis|12 years ago|reply
[+] [-] RivieraKid|12 years ago|reply
[+] [-] toisanji|12 years ago|reply
[+] [-] deltron|12 years ago|reply
[+] [-] Goranek|12 years ago|reply
[+] [-] unknown|12 years ago|reply
[deleted]