top | item 6801286

Docker 0.7 runs on all Linux distributions

435 points| zrail | 12 years ago |blog.docker.io | reply

128 comments

order
[+] shykes|12 years ago|reply
A few details on the "standard linux support" part.

To remove the hard dependency on the AUFS patches, we moved it to an optional storage driver, and shipped a second driver which uses thin LVM snapshots (via libdevmapper) for copy-on-write. The big advantage of devicemapper/lvm, of course, is that it's part of the mainline kernel.

If your system supports AUFS, Docker will continue to use the AUFS driver. Otherwise it will pick lvm. Either way, the image format is preserved and all images on the docker index (http://index.docker.io) or any instance of the open-source registry will continue to work on all drivers.

It's pretty easy to develop new drivers, and there is a btrfs one on the way: https://github.com/shykes/docker/pull/65

If you want to hack your own driver, there are basically 4 methods you need to implement: Create, Get, Remove and Cleanup. Take a look at the graphdriver/ package: https://github.com/dotcloud/docker/tree/master/graphdriver

As usual don't hesitate to come ask questions on IRC! #docker/freenode for users, #docker-dev/freenode for aspiring contributors.

[+] dgregd|12 years ago|reply
When it will be "production ready"? It seems that the schedule in "GETTING TO DOCKER 1.0" post is outdated.
[+] mateuszf|12 years ago|reply
Does the LVM part mean that the partition/disk used to store/run Docker images has to use LVM?
[+] knitatoms|12 years ago|reply
Any drawbacks to using lvm comapared to AUFS? I'd like to switch to Debian from Ubuntu - will I notice any performance differences or other restrictions?
[+] DannoHung|12 years ago|reply
Is there a document on the relative costs and benefits of the two drivers?
[+] andyl|12 years ago|reply
does this mean that docker will run on 32 bit systems ?
[+] Legion|12 years ago|reply
Could someone explain the logistics of Docker in a distributed app development scenario? I feel like I am on the outskirts of understanding.

My goal is having a team of developers use Docker to have their local development environments match the production environment. The production environment should use the same Docker magic to define its environment.

Is the idea that developers define their Docker environment in the Dockerfile, and then on app deployment, the production environment builds its world from the same Dockerfile? How does docker push/pull of images factor into that, if at all?

Or is the idea that developers push a container, which contains the app code, up to production?

What happens when a developer makes changes to his/her environment from the shell rather than scripted in the Dockerfile?

What about dealing with differences in configuration between production and dev? (Eg. developers need a PostgreSQL server to develop, but on production, the Postgres host is separate from the app server - ideally running PG in a Docker container, but the point being multiple apps share a PG server rather than each running their own individual PG instance). Is the idea that in local dev, the app server and PG are in two separate Docker containers, and then in deployment, that separation allows for the segmentation of app server and PG instance?

I see the puzzle pieces but I am not quite fitting them together into a cohesive understanding. Or possibly I am misunderstanding entirely.

[+] peterwwillis|12 years ago|reply
> Could someone explain the logistics of Docker in a distributed app development scenario?

Docker lets you build an environment (read: put together a bunch of files) for you to run an app in. It also has other features, like reducing space if lots of your apps [on the same host] use the same docker images, and networking stuff.

> My goal is having a team of developers use Docker to have their local development environments match the production environment

You run a container. You use images to distribute files. A Dockerfile is a loose set of instructions to build the images.

The basic idea is that, any single program that you want to be able to run anywhere, you make into a container. You can run it here, you can run it there, you can run it anywhere. The whole "running it anywhere" concept comes from the idea that all of your containers are created based on the same images, so no matter what kind of crazy mix of machines you have, your containers will just work - because you're literally shipping them a micro linux distribution in which to run your application. And since all the applications are running in isolated little identical containers, you can run as many of them as you want, independent of each other, in whatever configuration you want.

You'll have a PSQL container and an App container, and you'll manage them separately, even if they share images - the changes they make get saved off to a temp folder so they don't impact each other. Your environment stays the same only as long as the containers and images you're using are the same.

There will always be differences between development and production. You have to focus on managing those differences so you have confidence that what you're shipping to production actually works. The only thing Docker really does there is make sure the files are basically the same.

[+] cschmidt|12 years ago|reply
Since no one has answered you, I'll take a stab. I haven't actually used Docker yet, but I've been lurking extensively. Hopefully someone will correct me if I get this wrong.

A developer will use a Dockerfile to build the main dependencies that you need, and then save an image. Then to deploy code (or other frequently changing data), you pull that base image, add or update whatever else is needed, and save it again. (You can add layers to an image as often as desired.)

Then this saved image is pushed into a production environment. It seems like configuration data is often saved in images, so they require zero configuration. However, you can pass as many command line parameters as desired to run the image in production, so you can keep your configuration separated (like http://12factor.net/config) if you want.

[+] Sprint|12 years ago|reply
I looked at it several times but never really got it. Can I use Docker to isolate different servers (think http, xmpp, another http on another port) on a server so that if one of them was exploited, the attacker would be constrained to inside the container? Or is it "just" of a convenient way to put applications into self-contained packages?
[+] shykes|12 years ago|reply
It's both. Everyone using docker benefits from the "software distribution" feature. Some people using docker also benefit from the security and isolation features - it depends on your needs and the security profile of your application. Because the underlying namespacing features of the kernel are still young, it's recommended to avoid running untrusted code as root inside a container on a shared machine. If you drop privileges inside the container, use an additional layer of security like SELinux, grsec, apparmor etc. then it is absolutely possible and viable in production.

It's only a matter of time before linux namespaces get more scrutiny and people grow more comfortable with them as a first-class security mechanism. It took a while for OpenVZ to get there, but not there are hosting providers using it for VPS offerings.

On top of namespacing, docker manages other aspects of the isolation including firewalling between containers and the outside world (currently optional but soon to be mandatory), whitelisting of traffic between certain containers, etc.

[+] zrail|12 years ago|reply
> Can I use Docker to isolate different servers

That's the idea. Docker is based on LXC which provides configurable isolation between process groups. So, you could run your http in one container, your database in another, and your application server in a 3rd and have reasonable confidence that they can only talk to each other the way you want them to.

[+] yebyen|12 years ago|reply
I have heard two opinions. One of them is paranoid: "if you give up root on the container you give up root on the host system." I am not sure I believe this, I can't show you how to do the compromise and escalation, but I know there is a -privileged mode you can use on your containers in which Root user can do things like create device nodes, reset the system clock, and presumably other things you should not allow if you are concerned about this kind of attack.

The other opinion (given that you don't enable -privileged mode) is that lxc namespaces _do_ protect your host and other containers from being exploited due to compromise of a contained process, and cgroups _do_ protect your precious i/o and cpu cycles from being fully consumed and deadlocking your whole system when a rogue container decides to start infinite looping.

The competent answer of course is "try it", you should know that there are ways your processes can be compromised, and you should check if the exploits you know about result in the same or different privilege escalations when you apply them to a service hosted within a docker instance. That being said, you can't really prove a negative...

[+] neals|12 years ago|reply
I see docker come around every now and then here. I'm a smalltime developer shop, small team, small webspps. What can docker do for me?

Can this reduce the time it takes me to put up and Ubuntu installation on Digital Ocean?

Is this more for larger companies ?

[+] shykes|12 years ago|reply
Yes, Docker should be able to help you. It is definitely not just for large companies.

Docker can help you streamline your development and deployment process across all the machines, from the development laptop to the production server. The result should be that your development environment is easier to setup and replicate, more similar to production, and easier to customize as you evaluate new tools, hosting providers and software components.

I recommend checking out the "Getting started" page, it has a great online tutorial: http://www.docker.io/gettingstarted/

You should also browse through this list of real-world use cases. http://www.docker.io/community/#What-people-have-already-bui...

I hope this helps!

[+] mateuszf|12 years ago|reply
You will be able to have multiple Ubuntu servers running on one Ubuntu VPS on Digital Ocean. Each of them may have some initial state (installed some software, configuration) that you will be able to spawn as separate virtual machine. The "inside" machine will consume mnimal amount of memory (for running applications) and not for whole OS as in case of "real virtual machine". Also, you will be able to spawn and destroy new VMS very fast (under 1s).
[+] Nux|12 years ago|reply
EL6 users (RHEL, CentOS, SL), I've just learned Docker is now in EPEL (testing for now, but will hit release soon):

yum --enablerepo=epel-testing install docker-io

PS: make sure you have "cgconfig" service running

[+] klaruz|12 years ago|reply
Very nice, I can stop updating my fork of the other rpm spec that's on github.

Any info on how it sets up networking? I see -b none in the startup... Is there some other tool that does it on EL6?

Do you know where devel talks for this rpm are happening? I did not see any mention of it in the EPEL list archives.

[+] jackielii|12 years ago|reply
Hmm. Thanks for this, but I couldn't find package "docker-io" in epel-testing.

Could you post a detailed instruction?

Thanks a lot!

[+] SEJeff|12 years ago|reply
Thanks for paying it forward :)
[+] speeq|12 years ago|reply
Does anyone know if it is possible to set a disk quota on a container?
[+] geku|12 years ago|reply
Interested in that too.
[+] larsmak|12 years ago|reply
I'd be interested in this as well. The workarounds mentioned seems to require a bit to much hacking for my liking. Are there any actual plans for implementing simple disk-usage (and perhaps network) limiting?
[+] apphrase|12 years ago|reply
Can anyone please tell about the overhead of Docker, compared to no-container scenario (not against a fat vm scenario)? I am a "dev" not "ops", but we might make use of Docker in our rapidly growing service oriented backend... Thanks
[+] shykes|12 years ago|reply
For process isolation, the overhead is approximately zero. It adds a few milliseconds to initial execution time, then CPU and memory consumption of your process should be undistinguishable from a no-container scenario.

For disk IO, as long as you use data volumes [1] for performance-critical directories (typically the database files), then overhead is also zero. For other directories, you incur the overhead of the underlying copy-on-write driver. That overhead varies from negligible (aufs) to very small (devicemapper). Either way that overhead ends up not mattering in production because those files are typically read/written very unfrequently (otherwise they would be in volumes).

[1] http://docs.docker.io/en/master/use/working_with_volumes/

[+] Xelom|12 years ago|reply
Will it be possible to run Docker containers on Android? I may be asking this incorrectly. So correct me if I have a mistake. My question might be "Will it be possible to run Docker containers on Dalvik VM?" or "Can I run an Android in Docker container?"
[+] justincormack|12 years ago|reply
You cant "run docker on Dalvik VM" as it runs on Linux not on a JVM. You also can't run Android on a Docker container as Android needs some system calls not in a standard Linux kernel, although there may be some ways around that.

You might be able to run docker on Android though, but you may need to compile a kernel with containers (namespace) support as I don't know that Android ships with them (I forget).

[+] T-zex|12 years ago|reply
Is it possible to have multiple instances of the same app running in Docker containers and having readonly access to a "global" memory linked file? What I'm trying to achieve is having sand-boxed consumers having access to some shared resource.
[+] sown|12 years ago|reply
Hey,

docker newb here. Can I easily put my own software in it? I've got this c++ program that has a few dependencies in ubuntu.

[+] jeffheard|12 years ago|reply
This is crazy talk of course, but I wonder if there'd be some way to use rsync or git to support distributed development of images the way git does with code?

I mean, it'd be neat to be able to do a "pull" of diffs from one image into another related image. Merge branches and so on. I don't know, possibly this would be just too unreliable, but I would have previously thought that what docker is doing right now would be too unreliable for production use, and lo and behold we have it and it's awesome.

[+] shykes|12 years ago|reply
It wouldn't be the craziest thing people do with docker:

http://blog.bittorrent.com/2013/10/22/sync-hacks-deploy-bitt...

http://blog.docker.io/2013/07/docker-desktop-your-desktop-ov...

There has been discussion of taking the git analogy further. We actually experimented with a lot of that early on (https://github.com/dotcloud/cloudlets) and I can tell you it's definitely possible to take the scm analogy too far :)

I do think we can still borrow a few interesting things from git. Including, potentially, their packfile format and cryptographic signatures of each diff. We'll see!

Here's a relevant discussion thread: https://groups.google.com/forum/#!msg/docker-user/CWc5HB6kAN...

[+] jfchevrette|12 years ago|reply
Unfortunately it looks like the documentation has not been updated yet...

So much for feature #7. Documentation should be part of the development/release process

[+] julien421|12 years ago|reply
The doc has been updated, but there was a problem while pushing it this morning. We are working on it.
[+] shykes|12 years ago|reply
Documentation is definitely part of the process :) Apparently the documentation service triggered an incorrect build overnight. Until we fix that you can browse the latest version of the docs from the master branch: http://docs.docker.io/en/master
[+] dmunoz|12 years ago|reply
Nice to see Docker 0.7 hit with some very useful changes.

I see lots of people are getting some generic Docker questions answered in here, and want to ask one I have been wondering about.

What is the easiest way to use dockers like I would virtual machines? I want to boot an instance, make some changes e.g. apt-get install or edit config files, shutdown the instance, and have the changes available next time I boot that instance. Unless I misunderstand something, Docker requires me to take snapshots of the running instance before I shut it down, which takes an additional terminal window if I started into the instance with something like docker run -i -t ubuntu /bin/bash. I know there are volumes that I can attach/detach to instances, but this doesn't help for editing something like /etc/ssh/sshd_config.

[+] MoosePlissken|12 years ago|reply
There's no need to manually take snapshots, docker does this automatically every time you run a process inside a container. In your example of running /bin/bash, after you exit bash and return to the host machine docker will give you the id for the container which has your changes. You can restart the container or run a new command inside it and your changes will still be there. If you want to access it more easily later, you can run 'docker commit' which will create an image from the container with a name you can reference. You can also use that new image as a base for other containers.

This is great for development or playing around with something new, but the best practice for creating a reusable image with your custom changes would be to write a Dockerfile which describes the steps necessary to build the image: http://docs.docker.io/en/latest/use/builder/

[+] yebyen|12 years ago|reply
You don't have to take the snapshot before you shut it down. The container persists after its main process is terminated. It's safe to take the snapshot any time after.

Your question is very near to me, I published a teeny-tiny script called 'urbinit'[1] for doing exactly what you are asking. It's meant to use with urbit, where you start by creating a 'pier' and use it to begin 'ships'. The ships must not be discarded (unless you don't mind having no real identity, and many parts of urbit require you to maintain a persistent identity). If you re-do the key exchange dance again from a fresh pier to identify yourself as the same 'destroyer' (having discarded the previous incarnation), you wind up as a perfectly viable ship with packet sequence numbers that are out of order and can't really receive any messages or acknowledge any packets that are waiting to be delivered in the correct order from your neighbor ships.

Anyway, urbinit is a manifestation of the fact that Docker doesn't really have anything to help you deal with this (very common and not at all unique to Urbit) problem. Docker offers that you can use a volume... make the state you need to be persistent part of a volume. Unfortunately this means it won't be present in your images; or you can do roughly what urbinit does, which is a very simple and classic problem all Computer Scientists past year 1 should already know.

Launch the container in detached mode (-d) and save the ID. Create a lock file. Urbinit won't do step 1 if there is already a lock file created that was never cleared. This is to prevent you from launching the same ship twice at one time with two addresses, which is very confusing to the network.

Attach if you need to interact through terminal, or however you want, affect the container by sending packets to it (eg. login via ssh and do stuff). Terminate your process or let it end when it's ready to end naturally, or the power goes out. If you're using urbinit (and the power did not go out), this is the point where the commit process comes along and commits your container ID back to the named image that it was created from, simultaneously clearing the lock created in step 2.

If you're doing this with several different images on one host that are meant to run simultaneously, a nice trick is to name your lock file after the actual docker image that's used in the commit step.

[1]: http://nerdland.info/urbit-docker

[+] gexla|12 years ago|reply
So, I assume that if you aren't using AUFS then you don't have to deal with potentially bumping up against the 42 layer limit? Or does this update also address the issue with AUFS?
[+] shykes|12 years ago|reply
The 42 layer limit is still there... But not for long! There is a fix underway to remove the limitation from all drivers, including aufs.

Instead of lifting the limit for some drivers first (which would mean some images on the index could only be used by certain drivers - something we really want to avoid), we're artificially enforcing the limit on all drivers until it can be lifted altogether.

If you want to follow the progress of the fix: https://github.com/shykes/docker/pull/66

(This pull request is on my personal fork because that's where the storage driver feature branch lived until it was merged. Most of the action is usually on the main repo).

[+] shimon_e|12 years ago|reply
The links feature will make deploying sites a million times easier.
[+] neumino|12 years ago|reply
You guys are awesome, just awesome!

I was pretty sure that the requirement for AUFS would stick for a long time -- I was resigned to use a special kernel. But again, you folks surprise me!

You guys just rock!

[+] oskarhane|12 years ago|reply
Hmm, not sure I'm understanding #1 correct. Can I install it on, let's say, Debian without Vagrant/Virtualbox now?

I can't find the info in the docs.

[+] chc|12 years ago|reply
They don't have a package repository for Debian yet, but I think you should just be able to make/install it.
[+] chr15|12 years ago|reply
For local development, I use Vagrant + Chef cookbooks to setup my environment. The same Chef cookbooks are used to provision the production servers.

It's not clear to me how I can benefit from Docker given my setup above. Any comments?

[+] kro0ub|12 years ago|reply
Can someone please explain what docker does and brings to the table, what all the fuss seems to be about? I've looked into it several times and really can't tell from any of what I've found.
[+] saboot|12 years ago|reply
I have heard / read about docker for quite some time, yet it is unclear still how this is useful.

Let me ask a direct need I have, would docker allow me to use newer c++ compilers on redhat so I can code in c++11?