The main reason why containers are (a lot) better than the former status quo is shockingly simple: Dockerfiles. They list the steps that you need to build a working image. And yep, there are some caveats, but most of the time they do not cause problems (example: I have containers where :latest has been fine for over 5 years.)
I'll go as far as to say that if you want reproducible images that don't unexpectedly break without some kind of ability to trace it back to a change, always use sha256 digests in your `FROM` clauses, never fetch files directly from URLs (and/or check sha256 digests of things that you do fetch,) be thoughtful about the way your container is designed, and favor multi-stage builds and other OCI image builders to writing silly shell scripts that wrangle the build command in unusual ways.
But generally? It's still a massive improvement just having the steps written out somewhere. When I first started, it seemed the best you could get is a Vagrantfile that basically worked after you stepped on a bunch of rakes to figure out exactly what quirks you had to work around. With containers, things break a lot more predictably in my experience.
I really don't understand what Dockerfiles offer in terms of reproducible builds that a minimal install of a distro + a config manager didn't.
I feel like we took Ansible roles (for example, it could be Puppet, CFEngine, whatever) and spread them in uncountable and not reusable Dockefiles and bash scripts (oh the irony). But we still have the config managers too, because who could have imagined, you still have to have bare metal underneath.
Docker (like every other tool before it) started nice, clean and simple, because it didn't cover all the real needs. As those where added on we ended up with tens of tools on top of k8s and now here we are in yaml hell with version compatibility still not solved. And a new generation will come along and repeat it with another set of tools becase "this time it will be different". You can not get rid of complexity, you can only move it from here to there. And if there == YAML then may God have mercy on you.
Even if I wasn't using containers for production, the ability to make a repeatable build allows you to make software with complex dependencies extremely easy to develop in. Making it possible for a new developer in a new environment to get a build working on their own laptop, in the first day on the job, didn't use to be simple.
And being able to make hermetic environments for integration tests used to be almost impossible, and today, depending on your stack, it is trivial with libraries like testcontainers.
On one hand, layered docker builds mean that with some care you can only care about the top layer and treat base layers as immutable. As long as they are not repulled.
On the other hand, to have actual reproducibility you need to self build and self host everything down from base userland. However, once you achieve that, reproducible machines are one `apt install devenv` away.
What docker/containers do and excel at, compared to traditional workstation provisioning, is reduction of dependency trees via isolation. With one single user process running dependency trees are shaken and chances of accidental dependency collision drop. Does this count as solving dependency problem? Personally, I say no, but understand the other side of the argument.
Yes I remember the problems with Vagrant. I'm unsure about what's making Docker more predictable across machines than Vagrant. Possible reasons
- it's usually headless
- it comes with a layer that mounts the host file system, instead of installing extensions
- better testing of that layer on all platforms, especially the ones that need to add a Linux kernel? (Windows and Mac)
- it's harder to ssh into a container, manually fix things and persist the changes without updating the Dockerfile. We can do that with a Vagrant machine.
What you suggested about the listed steps is a bad suggestion. Docker should crash if you don't use a sha256 in a Dockerfile, or at least make some sort of lock file. But it instead allows you to make mistakes.
I recently contributed to the Linux kernel and they often get irate over the contributor not following manual steps. They could automate a lot of it with basic features like CI, and your suggestion that it is easy to make things reproducible if you just follow a list of instructions is part of this problem. "Just follow a list of instructions" will never be a solution to a bad workflow, and it is no replacement for a good methodology.
If you do not force best practices in the tool, it permits mistakes. Something you probably don't want to allow in a tool that builds software and manages the software supply chain. Docker doesn't provide a language for describing things correctly, you can make all the mistakes you want to make. Nix, for example, is a domain specific language which won't permit you to commit crimes like not pinning your dependencies, at least in 2023 with pure evaluation mode (on by default in flakes).
> They list the steps that you need to build a working image.
No they don't. They typically list the steps that you need to build a working image yesterday, not today, or in 5 years, which is a very important thing to be aware of, otherwise you might assume the instructions in the Dockerfile were crafted with any intention of working tomorrow. There's no reason to believe this is true, unless you know the author really did follow your list of suggestions.
Nix crashes when you make mistakes in your .nix expression. `docker build` won't crash when you make mistakes in your build, it is unaware and doesn't enforce a reproducibility methodology like Nix outlines in the thesis, an obvious example being the unconditional internet access given by the Docker "sandbox".
Docker does not make distinctions between fetching source code and operating/building that source code. In Nix these happen in two separate steps, and you can't accidentally implement your build instructions in the same step as fetching your program, which would otherwise lead you to execute the build on untrusted/unexpected input. This is just one part of the methodology outlined in the Nix thesis.
TL;DR Nix doesn't suggest you follow a list of instructions to make things reproducible, it just doesn't permit a lot of common mistakes that lead to unreproducibility such as not pinning or hashing inputs.
Dev would get the thing working on their machine configured for a customer. We'd take their machine and put it in the server room, and use it as the server for that customer. Dev would get a new machine.
Yes, I know it's stupid. But if it's stupid and it works, it isn't stupid.
DLL Hell was real. Spending days trying to get the exact combination of runtimes and DLL's that made the thing spring into life wasn't fun, especially with a customer waiting and management breathing down our necks. This became the easiest option. We started speccing dev machines with half an eye on "this might end up in the server room".
> "I don't know whats wrong, it works on my machine"
I had one of these years ago where QA had an issue that I couldn't reproduce.
I walked over to his desk, watched the repro and realized that he was someone who clicked to open a dropdown and then clicked again to select, while I would hold the mouse button down and then let up to select.
I have never been able to realize the alleged ergonomic gains of containers. Ever. It always adds more friction to actually getting something initially stood up, prototyped, and deployed.
I'm guessing it may be one of these things where it only starts to make sense after something has matured enough to warrant being replicated en-masse in a data-center environment.
Then again, I tend to live in a world where I'm testing the ever-loving crap out of everything; and all that instrumentation has to go somewhere!
That's basically what Smalltalk was back in the 20th Century.
The OS, the development environment and the application (both code and live objects) where one and the same thing. To ship an "app" you would export the image and the user would load it into their Smalltalk VM.
Owner hired an extremely “senior” developer. Was told to let him do his thing.
After he spent 3 months building a web app, I asked him how he wanted to deploy it.
Perfectly straight face he said we would take his developer machine to a local data center and plug it in. We could then buy him a new developer machine.
It went downhill from there.
I ended up writing the application from scratch and deploying it that same evening.
And dependencies are about the way that we went from a blank slate to a working system.
If you can't retrace that path, you can't debug. If you don't have tools to track the path, you will make mistakes. At best, you can exactly replicate the system that you need to fix -- and fixing is changing.
Especially for linux hosts this misses that containers will still run on the same kernel (version) as your host OS, inheriting a lot of settings and limitations from there as well (for example net.core.somaxconn). The most obvious ones are sysctl settings, many of which must be set system-wide. Common ones which can drastically change characteristics of database software are vm.nr_hugepages (postgres - why are we OOM or latency is spotty?) and vm.overcommit_memory (redis - why does background saving not work?).
I also looked at this topic, see [1]. Some points are similar to the article posted by OP. My findings were:
- Docker Desktop and Docker engine (CE) behave differently, e.g. bind mounts, or file system ownerships.
- CPU/Platform differences (ARM vs. AMD64): many devs don't realize they use ARM on their mac, thus ARM images are used by default, and tools you run in it (or want to install) may be have differently, or may be missing entirely
- Incompatible Linux kernel APIs (when containerized binaries make syscalls not supported by the the host's kernel, for whatever reason)
- Using the same version tags, expecting the same result (--> insanity, as you know it :D)
- Different engines (e.g. Docker Desktop vs. colima) change the execution behavior (RUNNING containers)
- Different build engines (e.g. kaniko vs. BuildKit vs. buildah) change the BUILD behavior
For anyone who is interested: more details in [1].
I think a lot of this comes down to a broader difference between Mac/Windows Docker Desktop and "plain" Docker on Linux. The former is actually backed by a VM, so a lot of the painless simplicity comes from having a true virtual machine involved, rather than just a layer of namespacing.
A lot of people are in here complaining about how Docker is not reproducible enough. But reproducibility of image builds is a matter of diminishing returns, and there are other problems to worry about, like the ones you are pointing out.
Speaking of which, it's probably good to get in the habit of installing some Linux OS in a VM and trying to run your container images inside that (with "plain" Docker, no inner VM), before pushing it to your cloud host and waiting for it to fail there.
I feel like most of the problems raised in this blog post can be solved with a proper reproducible build system - for example NixOs (or guix if you will) derivations.
It's true that Dockerfiles are not reproducible, but at least they're human friendly and easy to deploy. If you need something more deterministic, I really encourage you to try NixOs. It's (almost) 100% reproducible and works for any real-world use-case that I've ever had. Dockerfiles have a different use case - they are a formal version of a installation instuction that you would give to a new hire in the older times.
There are a number of incorrect statements in this post.
1) One should neither be using the "latest" nor just the "version" tag as the version can still vary depending on when it is pulled.
Instead, one should use a combination of version + hash, say alpine:3.18.2@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 for reproducibility reasons. This provides for human readable versions as well as the specific hash.
2) Next, afaik, Compose has removed the need for version tags. All of the compose.yml files that I now use do not specify versions.
"version + hash" is ugly though. I trust the publisher of my base image to keep compatibility even if they update their image and trust my test suites to detect any issues, so I just use version without the hash nowadays.
The article seems to miss the point that we are able talk about these differences in terms of containers as they are abstracted into a purely software system that are reproducible. Versus a customized hardware+software system with no means to reproduce it. Containers were a huge step forward because they raised so much more of the software stack into a simple, repeatable, defined systems than were previously much harder to obtain.
Controversial opinion but I still deeply believe that any building tools offering a “latest” option in its build configuration file and not as an option to update said file is doing something deeply wrong and just poorly designed. Everything pooled from outside should be using a checksum.
People want and need the ability to pin their environment. If you want to avoid silly surprises when going to production or through your CI system, you need everything to be identical: same versions, same configuration. Tooling should help you do that not introduce new way to shoot you in the foot.
I got -3 points on a comment on a different post where I stated what I didn't like about docker. I was bullied by docker fan boys. Glad to see many here agree with me. It sucks to be surrounded by a mob.
took a peek at your comment, and i do agree with you. docker can add unnecessary bloat to projects.
docker shouldn't be used for everything. if you provide a docker version of something, it's a smart idea to also publish (for example) an appimage or deb file for people who can't or don't want to use docker.
like for example, at my work we don't want to use docker because we will have to get approval from corporate for every little script we want to run in a container because corporate identifies a container as a separate application so it must go through the approval process (which takes 4-8 weeks).
In short, make sure to pin the exact version of dependencies, set the user and keep in mind OS-specific restrictions.
Containers are still a great improvement over running on dev machines directly: besides cleanly separating environment per project, an unsung benefit is being able to get rid of the environment when you're done working on a project instead of accumulating bloat over time.
While the comments here talk lot about pinning and locking everything down, I'll offer alternative viewpoint: test your application against wider range of environments and versions. Docker is great for that too, you can easily spin up your application in Debian oldstable or latest Fedora and see how it behaves. Your software will become less fragile as a result and better in the long term.
Somehow the old adage of portable software being good software seems to have been lost to the ages now that we as developers have attained such precise control over the environment our software runs in.
I once worked with a scientist who could only replicate their results on a single computer which had been extensively modified over time. Like, hot patching the C library and stuff. They were absolutely stuck on this machine, and never once considered the possibility that their results were due to the local modifications.
In retrospect this is not completely surprising given the incentive system in science.
How do you specify the need for reproducible software installations in an incentive system?
Also, what kind of scientist was this? I'm a physicist. I'm deeply concerned about the reproducibility of my results. I periodically try to rebuild my software systems on a clean computer to make sure it's both possible and well documented.
One thing I've learned when deploying: Pin absolutely everything. From the exact Docker base image version, to the package installer (e.g. Poetry) version, to all dependencies.
For everyone that went straight into the comments, this article is meant to guide people through _preventing_ situations like "it works in my container."
Although this was always a problem until the mac M1 chips it didn’t matter much. Now it happens almost every week. I would prefer to have at least the same architecture between my local and prod environments.
There are definitely footguns with docker. I still feel like if I build the image I have a lot more control over it than when I'm trying to document build scripts, create the right package files or lock files, etc. I don't buy this argument and stopped reading when it was obvious that these issues could be solved by using a clearly specific build tag for the base image, etc.
Imo, the strength of containers is portability, not reproducible builds. Even portability isn't perfect, since images are sensitive to CPU architectures and containers can still actually rely on host system configurations (mounting part of fs, privileged mode).
[+] [-] jchw|2 years ago|reply
I'll go as far as to say that if you want reproducible images that don't unexpectedly break without some kind of ability to trace it back to a change, always use sha256 digests in your `FROM` clauses, never fetch files directly from URLs (and/or check sha256 digests of things that you do fetch,) be thoughtful about the way your container is designed, and favor multi-stage builds and other OCI image builders to writing silly shell scripts that wrangle the build command in unusual ways.
But generally? It's still a massive improvement just having the steps written out somewhere. When I first started, it seemed the best you could get is a Vagrantfile that basically worked after you stepped on a bunch of rakes to figure out exactly what quirks you had to work around. With containers, things break a lot more predictably in my experience.
[+] [-] trabant00|2 years ago|reply
I feel like we took Ansible roles (for example, it could be Puppet, CFEngine, whatever) and spread them in uncountable and not reusable Dockefiles and bash scripts (oh the irony). But we still have the config managers too, because who could have imagined, you still have to have bare metal underneath.
Docker (like every other tool before it) started nice, clean and simple, because it didn't cover all the real needs. As those where added on we ended up with tens of tools on top of k8s and now here we are in yaml hell with version compatibility still not solved. And a new generation will come along and repeat it with another set of tools becase "this time it will be different". You can not get rid of complexity, you can only move it from here to there. And if there == YAML then may God have mercy on you.
[+] [-] greiskul|2 years ago|reply
And being able to make hermetic environments for integration tests used to be almost impossible, and today, depending on your stack, it is trivial with libraries like testcontainers.
[+] [-] friendzis|2 years ago|reply
On the other hand, to have actual reproducibility you need to self build and self host everything down from base userland. However, once you achieve that, reproducible machines are one `apt install devenv` away.
What docker/containers do and excel at, compared to traditional workstation provisioning, is reduction of dependency trees via isolation. With one single user process running dependency trees are shaken and chances of accidental dependency collision drop. Does this count as solving dependency problem? Personally, I say no, but understand the other side of the argument.
[+] [-] Corsome|2 years ago|reply
[+] [-] pmontra|2 years ago|reply
- it's usually headless
- it comes with a layer that mounts the host file system, instead of installing extensions
- better testing of that layer on all platforms, especially the ones that need to add a Linux kernel? (Windows and Mac)
- it's harder to ssh into a container, manually fix things and persist the changes without updating the Dockerfile. We can do that with a Vagrant machine.
Anything else?
[+] [-] matthewcroughan|2 years ago|reply
I recently contributed to the Linux kernel and they often get irate over the contributor not following manual steps. They could automate a lot of it with basic features like CI, and your suggestion that it is easy to make things reproducible if you just follow a list of instructions is part of this problem. "Just follow a list of instructions" will never be a solution to a bad workflow, and it is no replacement for a good methodology.
If you do not force best practices in the tool, it permits mistakes. Something you probably don't want to allow in a tool that builds software and manages the software supply chain. Docker doesn't provide a language for describing things correctly, you can make all the mistakes you want to make. Nix, for example, is a domain specific language which won't permit you to commit crimes like not pinning your dependencies, at least in 2023 with pure evaluation mode (on by default in flakes).
> They list the steps that you need to build a working image.
No they don't. They typically list the steps that you need to build a working image yesterday, not today, or in 5 years, which is a very important thing to be aware of, otherwise you might assume the instructions in the Dockerfile were crafted with any intention of working tomorrow. There's no reason to believe this is true, unless you know the author really did follow your list of suggestions.
Nix crashes when you make mistakes in your .nix expression. `docker build` won't crash when you make mistakes in your build, it is unaware and doesn't enforce a reproducibility methodology like Nix outlines in the thesis, an obvious example being the unconditional internet access given by the Docker "sandbox".
Docker does not make distinctions between fetching source code and operating/building that source code. In Nix these happen in two separate steps, and you can't accidentally implement your build instructions in the same step as fetching your program, which would otherwise lead you to execute the build on untrusted/unexpected input. This is just one part of the methodology outlined in the Nix thesis.
TL;DR Nix doesn't suggest you follow a list of instructions to make things reproducible, it just doesn't permit a lot of common mistakes that lead to unreproducibility such as not pinning or hashing inputs.
[+] [-] somat|2 years ago|reply
Manager: "Fine, then we will ship your machine"
And thus docker was born.
[+] [-] marcus_holmes|2 years ago|reply
Dev would get the thing working on their machine configured for a customer. We'd take their machine and put it in the server room, and use it as the server for that customer. Dev would get a new machine.
Yes, I know it's stupid. But if it's stupid and it works, it isn't stupid.
DLL Hell was real. Spending days trying to get the exact combination of runtimes and DLL's that made the thing spring into life wasn't fun, especially with a customer waiting and management breathing down our necks. This became the easiest option. We started speccing dev machines with half an eye on "this might end up in the server room".
[+] [-] dunham|2 years ago|reply
I had one of these years ago where QA had an issue that I couldn't reproduce.
I walked over to his desk, watched the repro and realized that he was someone who clicked to open a dropdown and then clicked again to select, while I would hold the mouse button down and then let up to select.
[+] [-] andrewedstrom|2 years ago|reply
[+] [-] salawat|2 years ago|reply
I'm guessing it may be one of these things where it only starts to make sense after something has matured enough to warrant being replicated en-masse in a data-center environment.
Then again, I tend to live in a world where I'm testing the ever-loving crap out of everything; and all that instrumentation has to go somewhere!
[+] [-] rahoulb|2 years ago|reply
The OS, the development environment and the application (both code and live objects) where one and the same thing. To ship an "app" you would export the image and the user would load it into their Smalltalk VM.
[+] [-] bandrami|2 years ago|reply
[+] [-] RF_Savage|2 years ago|reply
Turns out they did ship a 1:1 image of his machine.
[+] [-] treeman79|2 years ago|reply
After he spent 3 months building a web app, I asked him how he wanted to deploy it.
Perfectly straight face he said we would take his developer machine to a local data center and plug it in. We could then buy him a new developer machine. It went downhill from there.
I ended up writing the application from scratch and deploying it that same evening.
Owner hired a lot Of strange people.
[+] [-] dsr_|2 years ago|reply
And dependencies are about the way that we went from a blank slate to a working system.
If you can't retrace that path, you can't debug. If you don't have tools to track the path, you will make mistakes. At best, you can exactly replicate the system that you need to fix -- and fixing is changing.
[+] [-] ttymck|2 years ago|reply
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] chmod775|2 years ago|reply
[+] [-] mshekow|2 years ago|reply
- Docker Desktop and Docker engine (CE) behave differently, e.g. bind mounts, or file system ownerships.
- CPU/Platform differences (ARM vs. AMD64): many devs don't realize they use ARM on their mac, thus ARM images are used by default, and tools you run in it (or want to install) may be have differently, or may be missing entirely
- Incompatible Linux kernel APIs (when containerized binaries make syscalls not supported by the the host's kernel, for whatever reason)
- Using the same version tags, expecting the same result (--> insanity, as you know it :D)
- Different engines (e.g. Docker Desktop vs. colima) change the execution behavior (RUNNING containers)
- Different build engines (e.g. kaniko vs. BuildKit vs. buildah) change the BUILD behavior
For anyone who is interested: more details in [1].
[1] https://www.augmentedmind.de/2023/04/02/docker-portability-i...
[+] [-] nerdponx|2 years ago|reply
A lot of people are in here complaining about how Docker is not reproducible enough. But reproducibility of image builds is a matter of diminishing returns, and there are other problems to worry about, like the ones you are pointing out.
Speaking of which, it's probably good to get in the habit of installing some Linux OS in a VM and trying to run your container images inside that (with "plain" Docker, no inner VM), before pushing it to your cloud host and waiting for it to fail there.
[+] [-] msm_|2 years ago|reply
It's true that Dockerfiles are not reproducible, but at least they're human friendly and easy to deploy. If you need something more deterministic, I really encourage you to try NixOs. It's (almost) 100% reproducible and works for any real-world use-case that I've ever had. Dockerfiles have a different use case - they are a formal version of a installation instuction that you would give to a new hire in the older times.
[+] [-] newman314|2 years ago|reply
1) One should neither be using the "latest" nor just the "version" tag as the version can still vary depending on when it is pulled.
Instead, one should use a combination of version + hash, say alpine:3.18.2@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 for reproducibility reasons. This provides for human readable versions as well as the specific hash.
2) Next, afaik, Compose has removed the need for version tags. All of the compose.yml files that I now use do not specify versions.
See https://github.com/compose-spec/compose-spec/blob/master/04-...
[+] [-] dikei|2 years ago|reply
[+] [-] eikenberry|2 years ago|reply
[+] [-] paulddraper|2 years ago|reply
"It works on my machine"
"Okay well here is the exact image digest and configuration from docker inspect"
"Thanks I can reproduce the problem now"
[+] [-] brmgb|2 years ago|reply
People want and need the ability to pin their environment. If you want to avoid silly surprises when going to production or through your CI system, you need everything to be identical: same versions, same configuration. Tooling should help you do that not introduce new way to shoot you in the foot.
[+] [-] _9za9|2 years ago|reply
[+] [-] kaaaate|2 years ago|reply
docker shouldn't be used for everything. if you provide a docker version of something, it's a smart idea to also publish (for example) an appimage or deb file for people who can't or don't want to use docker.
like for example, at my work we don't want to use docker because we will have to get approval from corporate for every little script we want to run in a container because corporate identifies a container as a separate application so it must go through the approval process (which takes 4-8 weeks).
[+] [-] wildpeaks|2 years ago|reply
Containers are still a great improvement over running on dev machines directly: besides cleanly separating environment per project, an unsung benefit is being able to get rid of the environment when you're done working on a project instead of accumulating bloat over time.
[+] [-] crooked-v|2 years ago|reply
[+] [-] KingMob|2 years ago|reply
[+] [-] zokier|2 years ago|reply
Somehow the old adage of portable software being good software seems to have been lost to the ages now that we as developers have attained such precise control over the environment our software runs in.
[+] [-] ilyt|2 years ago|reply
Only feeble component is CA bundle for SSL-related stuff as that by nature is changeable.
[+] [-] chomp5977|2 years ago|reply
[+] [-] zX41ZdbW|2 years ago|reply
[+] [-] lmm|2 years ago|reply
[+] [-] dekhn|2 years ago|reply
In retrospect this is not completely surprising given the incentive system in science.
[+] [-] analog31|2 years ago|reply
Also, what kind of scientist was this? I'm a physicist. I'm deeply concerned about the reproducibility of my results. I periodically try to rebuild my software systems on a clean computer to make sure it's both possible and well documented.
[+] [-] stavros|2 years ago|reply
[+] [-] remram|2 years ago|reply
Otherwise distro package installs are not reproducible even if you lock the base image (and apt-get with a specific version will most likely fail).
[+] [-] greatpostman|2 years ago|reply
[+] [-] nunez|2 years ago|reply
[+] [-] bogota|2 years ago|reply
[+] [-] xrd|2 years ago|reply
[+] [-] yeck|2 years ago|reply