[cromantin]

In my anecdotal experience with 10 machines and 40 services, half of machines dedicated and half in amazon - i've tried:

k8s - great but for >>10 machines and you need a devops to watch and configure it + every configuration example on site is broken in one way or another. skydns is working for dns. has it's own overlay network.

nomad - great, but doesn't manage dns or network + i didn't get how to save it to file, only upload to nomad. What if it fails? Where cluster config is stored? May be it should be used with Terraform, but i did't like it. Thou for it's purposes it working good. Has concept of long running tasks and can run VM's

helios - i don't remember exactly why but i didn't like it. Something with configuration management. No network or dns.

maestro-ng - this is as raw as it gets. Basically docker compose. I was using it for a while, but it hit it's limits pretty fast. No network or dns.

rancher - after seeing it in action i fell in love. Beautiful UI (you can even view logs and start shell from web ui, and it works great!). Has DNS, manages your overlay network. Store configuration in mysql (i use amazon's rds with MZ). It has it's own perfectly capable scheduler - Cattle. And it can run k8s and mesos if you want it to. I highly recommend OpenVPN package that will connect you to internal overlay network and you will be able to work with services walled from outside world.

[Veratyr]

Another thing on the k8s configuration guides is that few (none?) of them include SSL authentication. All the ones I tried were entirely insecure, allowing anyone to point the command line client at the cluster and do whatever they pleased. There was also no documentation for fixing this or any warning that the example configurations were entirely lacking in security.

If you want to use k8s, I strongly suggest that you entirely ignore their own documentation and instead use the excellent documentation from CoreOS.

[lobster_johnson]

Helios is probably closer to Docker Swarm more than any of the other tools mentioned. It starts containers and isn't very sophisticated beyond that.

We evaluated it for a while, but discarded for various reasons, particularly the fact that it doesn't support private container registries [1].

I'd say Helios suffers from being an internal Spotify project. They're not under pressure to support any use cases except their own. For example, Helios is tightly coupled with Zookeeper, and they have rejected the possibility of supporting anything else (e.g., etcd). Their prerogative, of course.

[1] https://github.com/spotify/helios/issues/462

[icebraining]

Last time I tried RancherOS is seemed quite heavy, with a 2GB RAM footprint for the initial run (without any nodes attached). What's your experience?

[daveguy]

I found the same. It is an all-in-ram distribution, so every docker layer is fully loaded into ram (I think there are 3). The core images must be built on something like Ubuntu. Maybe they have trimmed it down more recently. With something like an Alpine base you could probably get to a few hundred MB ram footprint without any loss in functionality. It would be great to see the bare minimum -- something like busybox+runit/dumb-init. Something with < 100MB footprint would be great for those cheap 512MB vps systems.

[cromantin]

RancherOS is separate entity from Rancher. It's their attempt at smallest possible OS for running docker. You may use rancher and never use rancher os.

Said that i personally use RancherOS on t2.micro instances without any problems.