That's a valid way to look at it, but there are other ways. Containers are also a simple, practical way to bundle applications and their dependencies in a relatively standardized way, so they can be run on different compute fabrics.
That sense of the term isn't loaded with any specific notion of how attack surfaces should work. I think modern "Docker"'s security properties are underrated†. But you still can't run multitenant workloads from arbitrary untrusted tenants on shared-kernel isolation. It turns out to be pretty damned useful to be able to ship application components as containers, but have them run as VMs.
Your first paragraph pretty much sums up what docker is, it’s a convenient way to design and build a system, but it is not a security mechanism.
If you’re building a system that’s handling classified information, there is probably not an accreditation authority in the world that would let you use containers or even hypervisors as a way to separate different information classes.
> I think modern "Docker"'s security properties are underrated†
100% agree.
The docker/CRI-de-jour (by default) strips off many "dangerous" system capabilities. By default a pid on linux gets something like over one hundred system capabilities, and most container runtimes strip that down to around 50. Those number are not exact.
Stripping down the system level capabilities of your workload is assuredly a security improvement over running that workload "bare metal" on the system.
Technically what you're describing is an image. Might sound pedantic but interchanging container and image does often cause confusion in my experience.
> Containers are also a simple, practical way to bundle applications and their dependencies in a relatively standardized way, so they can be run on different compute fabrics.
What I find interesting, is that many uses of containers are just reinventing statically linked binaries in a more complicated form.
Should be noted that a portion of this (valid) criticism applies specifically to the most prominent "container" implementation; Docker. Not containers as a whole.
For example resources isolation with the Solaris / Illumos container implementation (zones) works just as well as full blown virtualization. You are just as well equipped to handle noisy neighbors with zones as you are with hardware VM's.
> Much as you’d likely choose to live in a two-bedroom townhouse over a tent, if what you need is a lightweight operating system, containers aren’t your best option.
So I think this is true for Docker but doesn't really do justice to other container implementations such as FreeBSD jails and Solaris / Illumos zones. Because those containers are really just lightweight operating systems.
In the end Docker started out and was designed to be a deployment tool. Not necessarily an isolation tool in all aspects. And yeah, it shows.
I can not agree more. It is the saddest thing the appalling implementation of Docker, and the whole lack of security around the ecosystem, made people think Containers equal to Docker. Docker is what happens when you put your security
implementation in the hands of your Developer team and not
in the hands of your DevSecOps people.
Enjoyed the article but having watched containerization and kubernetes maturing over the last 5 years (especially at an enterprise level), I'd say a huge part of the value proposition is (and this applies more to K8) it really catalyses prototyping/experimenting and (depending on the org I suppose) promotes a lot of autonomy for app teams who'd historically have to log calls to infrastructure to get compute, network/lb/dns, databases et al. built up before kicking the tyres on something. I've seen those types of things take months in large orgs. And then there's the inevitable drift between tiered environments that happens over time in richer operating environments (I've seen VMs so laden with superfluous monitoring and agentware they fall over all the time, while simultaneously being on completely different OS and patch versions from dev to prod). Containers provide immutability at the service layer, so I have confidence in at least having that level of parity between dev and prod (albeit hardly ever at a data or network layer).
I believe, that success of containers is not because of lightweightness or other isolation properties of them.
Containers won dev mindshare because of ease packaging and distribution of the artifacts. Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up.
> Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up.
IMO the important, catalyzing difference is that Docker containers have a standard interface for logging, monitoring, process management, etc which allow us to just think in terms of “the app” rather than the app plus the SSH daemon, log exfiltration, host metrics daemon, etc. In other words, Docker got the abstraction right: I only care about the app, not all of the ceremony required to run my app in a VM. These common interfaces allow orchestration tools to provide more value: they aren’t just scheduling VMs, they’re also managing your log exfiltration, your process management, your SSH connection, your metrics, etc, and all of those things are configurable in the same declarative format rather than configuring them with some fragile Ansible playbook that requires you to understand each of the daemons it is configuring, possibly including their unique configuration file/filesystem conventions and syntaxes.
> Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up.
Packaged VM's existed for a while already with thing like Vagrant on top, there was also already LXC which leaned more into the VM concept. Where Docker made the difference imho is with Dockerfiles and the layered/cached build steps.
IMO comparing containers to an apartment is more accurate than a tent.
Because with an apartment each tenant gets to share certain infrastructure like heating and plumbing from the apartment building, just like containers get to the share things from the Linux host they run on. In the end both houses and apartments protect you from outside guests, just in their own way.
I went into this analogy in my Dive into Docker course. Here's a video link to this exact point: https://youtu.be/TvnZTi_gaNc?t=427, that video was recorded back in 2017 but it still applies today.
This is oversimplifying containers and VM by using the house vs tent analogy. Just talking about Docker weakens this too, because Docker is not the only way to setup containers.
> Tents, after all, aren’t a particularly secure place to store your valuables. Your valuables in a tent in your living room, however? Pretty secure.
Containers do provide strong security features, and sometimes the compromises you have to make hosting something on a VM vs. a container will make the container more secure.
> While cgroups are pretty neat as an isolation mechanism, they’re not hardware-level guarantees against noisy neighbors. Because cgroups were a later addition to the kernel, it’s not always possible to ensure they’re taken into account when making system-wide resource management decisions.
Cgroups are more than a neat resource isolation mechanism, they work. That's really all there is to it.
Paranoia around trusting the Linux kernel is unnecessary if at the end of the day you end up running Linux in production. If anything breaks, security patches will come quick and the general security attitude of the Linux community is improving everyday. If you are really paranoid, perhaps run BSD, use grsec, or the best choice is to use SELinux IMO.
If anything, you will be pwned because you have a service open to the world, not because cgroups or containers let you down.
Modern containers do provide lots of security features with namespaces, seccomp, cgroups (to some extent)
The author seems to largely ignore this. I would consider that a bit stronger than a "tent wall". Comparing it to a tent seems more akin to a plain chroot.
If I have my tent right next to someone else, I can trivially "IPC" just speaking out loud which would be prevented by an IPC namespace (which is Docker's current default container setup)
Also worth mentioning, turning a container into a VM (for enhanced security) is generally easier than trying to do the opposite. AWS Lambda basically does that as do a lot of the minimal "cloud" Linux distributions that just run Docker with a stripped down userland (like Container Linux and whatever its successors are)
I’m a big proponent of containers, but in fairness to TFA, I don’t know how to configure namespaces, second, or cgroups and I don’t know what settings my orchestrator uses by default. If containers can be secure but we don’t enable those security features properly, then it’s a bit of a moot point. That said, I think (but am not sure) most of us understand enough not to trust containers for isolation between untrusted processes, so I don’t regard containers as lightweight VMs, but rather collocated processes with their own namespaces. When I run untrusted code, like jupyterhub, I make sure those untrusted containers get scheduled onto their own dedicated mode pool with single tenancy (at which point the container is more of a tooling/orchestration convenience than a resource optimization tool).
>Finally, there’s the whole business of resource isolation. While cgroups are pretty neat as an isolation mechanism, they’re not hardware-level guarantees against noisy neighbors. Because cgroups were a later addition to the kernel, it’s not always possible to ensure they’re taken into account when making system-wide resource management decisions.
I don't think virtualization really offers hardware-level guarantees against noisy neighbours either.
VMs provide stronger guarantees for maximum CPU, network, and disk usage, as well as memory size consumption. This because the abstraction boundaries are fairly clear (e.g. threads and virtual devices).
Starts off saying VMs are like brick and mortar houses and containers are like tents.
I agree somewhat but there has been significant progress to sandbox containers with the same security we'd expect from a VM. It isn't a ridiculous idea that VMs will one day be antiquated, but probably won't happen for a few more years.
Do you have any links to secure container runtimes that don’t either virtualize or replace all the system calls of the container such that it might as well be virtual?
You can't make the Linux kernel isolation of processes as secure as Xen or Firecracker or SEL4 can. Yes, processes can be restricted to subset of syscalls and system resources but Linux is just too big and its attack surface is too big to put it on the same level of confidence as above hypervisors.
i don't think that is necessarily the case but instead i believe in the near future the differences between container sandboxes and virtual machines might be less clear.
I imagine CPU and memory namespaces coming implemented on hardware isolation features like VT-d io-mmus and alike thus making virtual machines integrated into some sandboxing feature.
Am I the only one getting tired of people stating confidently that containers don't improve safety _at all_ because they run on the same kernel? It's just not true.
I place all my tents in a house (Docker VMs inside unprivileged LXC containers on Proxmox - yes, unprivileged = not a brick house, more like wood).
The only reason I use Docker is that I can access the system design knowledge that is available with docker-compose.yml's. Last example: Gitlab. Could not get it running on unrivileged LXC using the official installation instructions, with Docker it was simply editing the `.env` and then `docker-compose up -d`. All of this in a local, non-public (ipsec-distributed) network. I often find myself creating a single, separate unprivileged LXC container->Docker nesting for each new container, because I do not need to follow the complicated and error prone installation instructions for native installs.
Container tech can be used for small scale "pet" deployment, but my understanding is that the true benefit of containers come with seeing them as "cattle".
You should never login to the shell of a container for config. All application state lives elsewhere, and any new commit to your app triggers a new build.
If that's not for you, then while containers like proxmox/LXC can still be handy, you're just doing VM at a different layer.
The article was a bit hand wavey about how "they" complain about containers, and then uses the analogy more than explaining the problems and solutions.
>You should never login to the shell of a container for config
I absolutely do this and think it works great. Fedora has built a tool called "toolbox" which is basically a wrapper on podman which can create and enter containers where you can install development tools without touching your actual OS.
I basically do all of my development inside a container where the source code is volume mounted in but git/ruby/etc only exist in the container.
This has the benefit of letting me very quickly create a fresh env to test something out. Recently I wanted to try doing a git pull using project access tokens on gitlab and containers let me have a copy of git which does not have my ssh keys in it.
This is somewhat of an edge case though, for a server deployment, yes you shouldn't rely on anything changed inside the container and should use volume mounts or modify the image.
Only time it should be utilized as a small scale "pet" is when you externalize the storage mounts and it is an on-demand non-persistent virtual environment not directly connected to a data sensitive environment. That's mainly my take on it. I will often use docker locally to test out some kubernetes service connectivity, but never bring my Frankenstein stuff into an environment any higher than local.
I’ve found systemd-nspawn useful. Use debootstrap to install a minimal Debian system inside your system, then boot it with this command. It isolates the filesystem while sharing the network interface, and is convenient for most things that I guess people use Docker for. I wonder why it’s not mentioned more often.
I explain that if the an Amazon Virtual Private Cloud (VPC) is a datacenter "cloud", then a container implementation is a "puff".
Virtualizing the kernel like the Amazon Machine Image (AMI) virtualizes a chip core sounds great. But now, in the "puff", all of those networking details that AWS keeps below the hypervizor line confront us. Storage, load balancing, name services, firewalls. . .
Containers can solve packaging issues, but wind up only relocating a vast swath of other problems.
If VM is like a nuclear war bunker, containers are like brick and mortar houses. They are not air tight and have glass windows which can be easily broken, but that's where most people live. They're cheaper to build, comfortable enough, and can last a human lifetime most of the time.
I've seen the problems of treating containers as houses, primarily during development: Multiple different processes inside a single container, with a wrapper around them (inside the container) that makes it even more difficult to debug.
So, assuming I understood correctly, treating them like tents is infinitely the better choice.
I was totally expecting this to go in the direction about tech debt with a homeless analogy, but it was about destructability. Yes we know this already and if you catch people treating it as a persistent host, slap their hands and say no.
I'd be curious to see services designed to run as PID 1 inside containers, and contain or run nothing else other than the required binaries. Maybe someone is doing this.
It's pretty common for languages that compile to binaries. Golang, for example. Just inherit your container "FROM scratch". I have a number of service containers that are around 8 - 12MB, which contain compiled Haskell binaries.
Pretty sure this is what tini[1] is for, and there's supervisord[2] for more complex use cases, e.g. running multiple processes. Neither are a full replacement for traditional systemd, which I heard you can technically run in a container these days, but I've never seen anyone try.
containers are cattle, VMs were pets. If one does not get the operational differences nor understands that these are completely two different usescases then probably should not work in IT industry
VMs can be cattle. Physical machines can be cattle. This is not tied to the runtime technology, but to how you design and manage your machines and applications.
tptacek|4 years ago
That sense of the term isn't loaded with any specific notion of how attack surfaces should work. I think modern "Docker"'s security properties are underrated†. But you still can't run multitenant workloads from arbitrary untrusted tenants on shared-kernel isolation. It turns out to be pretty damned useful to be able to ship application components as containers, but have them run as VMs.
† https://fly.io/blog/sandboxing-and-workload-isolation/
Saint_Genet|4 years ago
If you’re building a system that’s handling classified information, there is probably not an accreditation authority in the world that would let you use containers or even hypervisors as a way to separate different information classes.
djmetzle|4 years ago
100% agree.
The docker/CRI-de-jour (by default) strips off many "dangerous" system capabilities. By default a pid on linux gets something like over one hundred system capabilities, and most container runtimes strip that down to around 50. Those number are not exact.
Stripping down the system level capabilities of your workload is assuredly a security improvement over running that workload "bare metal" on the system.
Ref: https://www.redhat.com/en/blog/secure-your-containers-one-we...
globular-toast|4 years ago
nijave|4 years ago
All those bargain basement OpenVZ "VPS" providers beg to differ :)
There's also gVisor
RcouF1uZ4gsC|4 years ago
What I find interesting, is that many uses of containers are just reinventing statically linked binaries in a more complicated form.
jsiepkes|4 years ago
For example resources isolation with the Solaris / Illumos container implementation (zones) works just as well as full blown virtualization. You are just as well equipped to handle noisy neighbors with zones as you are with hardware VM's.
> Much as you’d likely choose to live in a two-bedroom townhouse over a tent, if what you need is a lightweight operating system, containers aren’t your best option.
So I think this is true for Docker but doesn't really do justice to other container implementations such as FreeBSD jails and Solaris / Illumos zones. Because those containers are really just lightweight operating systems.
In the end Docker started out and was designed to be a deployment tool. Not necessarily an isolation tool in all aspects. And yeah, it shows.
belter|4 years ago
nyx__|4 years ago
One could argue that zones are distinct from containers (a Linux implementation), with both being OS specific versions of jails.
jb_gericke|4 years ago
rossmohax|4 years ago
Containers won dev mindshare because of ease packaging and distribution of the artifacts. Somehow it is Docker, not VM vendors came up with a standard for packaging, distributing and indexing for glorified tarballs and it quickly picked up.
throwaway894345|4 years ago
IMO the important, catalyzing difference is that Docker containers have a standard interface for logging, monitoring, process management, etc which allow us to just think in terms of “the app” rather than the app plus the SSH daemon, log exfiltration, host metrics daemon, etc. In other words, Docker got the abstraction right: I only care about the app, not all of the ceremony required to run my app in a VM. These common interfaces allow orchestration tools to provide more value: they aren’t just scheduling VMs, they’re also managing your log exfiltration, your process management, your SSH connection, your metrics, etc, and all of those things are configurable in the same declarative format rather than configuring them with some fragile Ansible playbook that requires you to understand each of the daemons it is configuring, possibly including their unique configuration file/filesystem conventions and syntaxes.
aequitas|4 years ago
Packaged VM's existed for a while already with thing like Vagrant on top, there was also already LXC which leaned more into the VM concept. Where Docker made the difference imho is with Dockerfiles and the layered/cached build steps.
enw|4 years ago
Calling container images glorified tarballs is like calling cars glorified lawnmowers.
nickjj|4 years ago
Because with an apartment each tenant gets to share certain infrastructure like heating and plumbing from the apartment building, just like containers get to the share things from the Linux host they run on. In the end both houses and apartments protect you from outside guests, just in their own way.
I went into this analogy in my Dive into Docker course. Here's a video link to this exact point: https://youtu.be/TvnZTi_gaNc?t=427, that video was recorded back in 2017 but it still applies today.
fsociety|4 years ago
> Tents, after all, aren’t a particularly secure place to store your valuables. Your valuables in a tent in your living room, however? Pretty secure.
Containers do provide strong security features, and sometimes the compromises you have to make hosting something on a VM vs. a container will make the container more secure.
> While cgroups are pretty neat as an isolation mechanism, they’re not hardware-level guarantees against noisy neighbors. Because cgroups were a later addition to the kernel, it’s not always possible to ensure they’re taken into account when making system-wide resource management decisions.
Cgroups are more than a neat resource isolation mechanism, they work. That's really all there is to it.
Paranoia around trusting the Linux kernel is unnecessary if at the end of the day you end up running Linux in production. If anything breaks, security patches will come quick and the general security attitude of the Linux community is improving everyday. If you are really paranoid, perhaps run BSD, use grsec, or the best choice is to use SELinux IMO.
If anything, you will be pwned because you have a service open to the world, not because cgroups or containers let you down.
unknown|4 years ago
[deleted]
nijave|4 years ago
The author seems to largely ignore this. I would consider that a bit stronger than a "tent wall". Comparing it to a tent seems more akin to a plain chroot.
If I have my tent right next to someone else, I can trivially "IPC" just speaking out loud which would be prevented by an IPC namespace (which is Docker's current default container setup)
Also worth mentioning, turning a container into a VM (for enhanced security) is generally easier than trying to do the opposite. AWS Lambda basically does that as do a lot of the minimal "cloud" Linux distributions that just run Docker with a stripped down userland (like Container Linux and whatever its successors are)
throwaway894345|4 years ago
eVeechu7|4 years ago
I don't think virtualization really offers hardware-level guarantees against noisy neighbours either.
amarshall|4 years ago
habeebtc|4 years ago
encryptluks2|4 years ago
I agree somewhat but there has been significant progress to sandbox containers with the same security we'd expect from a VM. It isn't a ridiculous idea that VMs will one day be antiquated, but probably won't happen for a few more years.
kasey_junk|4 years ago
effie|4 years ago
mercora|4 years ago
I imagine CPU and memory namespaces coming implemented on hardware isolation features like VT-d io-mmus and alike thus making virtual machines integrated into some sandboxing feature.
znpy|4 years ago
Marketing. Because of Marketing.
0xcmoney|4 years ago
Helmut10001|4 years ago
The only reason I use Docker is that I can access the system design knowledge that is available with docker-compose.yml's. Last example: Gitlab. Could not get it running on unrivileged LXC using the official installation instructions, with Docker it was simply editing the `.env` and then `docker-compose up -d`. All of this in a local, non-public (ipsec-distributed) network. I often find myself creating a single, separate unprivileged LXC container->Docker nesting for each new container, because I do not need to follow the complicated and error prone installation instructions for native installs.
unethical_ban|4 years ago
You should never login to the shell of a container for config. All application state lives elsewhere, and any new commit to your app triggers a new build.
If that's not for you, then while containers like proxmox/LXC can still be handy, you're just doing VM at a different layer.
The article was a bit hand wavey about how "they" complain about containers, and then uses the analogy more than explaining the problems and solutions.
foobar33333|4 years ago
I absolutely do this and think it works great. Fedora has built a tool called "toolbox" which is basically a wrapper on podman which can create and enter containers where you can install development tools without touching your actual OS.
I basically do all of my development inside a container where the source code is volume mounted in but git/ruby/etc only exist in the container.
This has the benefit of letting me very quickly create a fresh env to test something out. Recently I wanted to try doing a git pull using project access tokens on gitlab and containers let me have a copy of git which does not have my ssh keys in it.
This is somewhat of an edge case though, for a server deployment, yes you shouldn't rely on anything changed inside the container and should use volume mounts or modify the image.
unknown|4 years ago
[deleted]
Sparkyte|4 years ago
leephillips|4 years ago
dijit|4 years ago
I'd be interested in trying it out but I don't want to spend some hours reading documentation trying to get it working.
lamontcg|4 years ago
smitty1e|4 years ago
Virtualizing the kernel like the Amazon Machine Image (AMI) virtualizes a chip core sounds great. But now, in the "puff", all of those networking details that AWS keeps below the hypervizor line confront us. Storage, load balancing, name services, firewalls. . .
Containers can solve packaging issues, but wind up only relocating a vast swath of other problems.
zaphirplane|4 years ago
wmf|4 years ago
unknown|4 years ago
[deleted]
srg0|4 years ago
An analogy can go a long way. Both ways.
luord|4 years ago
So, assuming I understood correctly, treating them like tents is infinitely the better choice.
unknown|4 years ago
[deleted]
Sparkyte|4 years ago
johbjo|4 years ago
antonvs|4 years ago
onei|4 years ago
[1] https://github.com/krallin/tini
[2] https://github.com/Supervisor/supervisor
nijave|4 years ago
1MachineElf|4 years ago
eyeyeyerg|4 years ago
detaro|4 years ago