As much as I admire the people involved with Nix, taking the initiative to solve what they believe can be improved and the other developers like her, toying with new things and successfully managing to figure all this stuff out... I'm actually disappointed.
The thing is, Nix is an absurdly complex piece of software, perhaps on a similar order of magnitude as maybe Kubernetes or other gizmos, requiring a very large time commitment to understand and to use properly.
Just for fun, have any of you tried to keep up during the Docker bigbang, with all these technologies popping up left and right, switching names, dying abruptly, getting super-seeded or hacked with vulnerabilities? That was one of my most mentally draining year for me and I've been in the field for a while now.
See... along the way, I've learned that Computer Science is always about tradeoffs. Now when I'm being asked to trade away my sanity learning this nonsense for in return a couple of megabyte of disk space that could've been wasted, I just don't see the value.
Seriously, are we optimizing for the right stuff here? I know Nix is doing package isolation with a programmable environment defined by text files, it's immutable and all. I get it. But we have alternatives for these problems. They do a pretty good job overall and we understand them.
Ironically, even those solutions I'm comfortable with are still struggling to get good adoption! How the heck is our field going to avoid fragmentation when it keep growing exponentially like this.
Perhaps it's just me aging and getting groggy; though there must be an explanation for this phenomenon. Please, it definitely gnaws at me.
If you like Nix than probably try GNU Guix [1] and if possible use Guix System on server side. This is one of the most modern development in operating systems.
In the modern security conscious world where most companies run their workloads on virtual machines controlled by other companies, it's imperative that applications are deployed on such predictable, secure and reproducible operating systems.
Guix supports transactional upgrades and roll-backs, unprivileged package management, and more. When used as a standalone distribution, Guix supports declarative system configuration for transparent and reproducible operating systems.
I still think Amazon, Google and Azure will take a decade or two to build and offer something like this to it's customers. Indeed Google's Fuschia is trying to do the same, but I feel it's at least a decade away.
It has one of the best wonderful documentation [2] to get started and explore with all the details.
As a Nix user, Guix interests me but I have three concerns, namely my pain points with Nix:
1. Small package base.
My workflow includes running Julia on CUDA, editing from VS Code. Remarkably, Guix has both Julia and (on a third-party PKG repo) CUDA. What it's missing is VS Code. I understand that VS Code is non-free (like Chrome, it attaches trademarks on its branding), but there's no VS Codium I could find either. I also can't find Atom. Am I missing something?
2. Package index mirroring.
Nix has an issue where Python, Node and other "third-party" packages have to be manually added to nixpkgs. If you want celery or requests, you're in luck because those are popular. If you want something more obscure, like pywikibot, you have to add it yourself (or run pip/npm yourself, and forego the benefits of Nix/Guix.) Does Guix address this?
3. Ease of scripting.
This might be a plus for Guix, since Nix design patterns are arcane (the language is easy, things like overrides and needing to replace broken URLs is hard), but how easy is it to debug failing builds? Is there something like a debugger, or at least a way to shell-in and call things yourself?
Bonus #4: broken URLs. is there a cache? what can be done?
I'd really love to use GuixSD. I'd much prefer Lisp to Nix. But the strict licensing is just too much of a handicap. You've really got to build your system around the underlying licenses. I've tried installing it on 3 or 4 systems, and was always tripped up by some critical missing driver.
I admire the ideological purity, but it kinda renders it irrelevant to me.
If I could trouble you for the common discourse here, would you mind summarizing why one may prefer to use Guix in the place of Nix? They seem to be based on the very same ideas and Guix even admits to being inspired by Nix.
I actually tried guix several times, and for my trouble I got my tmux sessions (and random processes on my box) killed on a box with 32 GB of ram. #guix on freenode was confused. I intend to wait until guix is a bit more stable on Ubuntu.
Guix is technically interesting, but as a fsf endorsed distro it makes running common hardware a challenge. Are there linux-antilibre packages out there for guix?
I've been using NixOS on my laptop for a couple of years now and I am still embarrasingly undereducated on how to do some of even basic things. I feel I still haven't fully grokked "the nix way". You will need to understand it pretty well to do even basic things like running specific versions of ruby or python with specific native libraries. I have spent a couple of hours here and there on nix pills but I think one really needs to set aside some concerted time to get into it properly. I consider myself fairly proficient with Linux in general and never had this kind of steep learning curve with other distros.
(This is a not me criticizing NixOS, just noting that they do things very differently and you really need to invest some time and effort to start being able to solve your own problems rather than relying on what's provided by others)
I used NixOS for 6 months and dropped it for the same reasons. I absolutely love the idea of my entire system being configured through text files, and being easily reproducible, but the amount of work you have to put in to get anything working in that distribution is insane.
I bounced off it two or three years ago after trying to get it running a basic GUI desktop environment by following a guide (IIRC an official one of some sort? It seemed legit, certainly) and after an hour or so of installing and faffing about was nowhere near having it working or even feeling like I might, after another hour, get there. Nothing seemed to work right or do what it was supposed to, troubleshooting was hard, and learning some new (and, at least initially, quite unpleasant) language just to configure this one package manager... ugh, no.
For reference I ran Gentoo on an IBM Thinkpad as my main computer for a few years in the oughts, among a bunch of other Linux work, so I, by necessity, at least kind of know what I'm doing with desktop Linux installation and config, from a fairly low level on up. At least that knowledge has proven highly transferable and, over and over again, relevant and useful. I didn't get the impression I was learning anything from NixOS other than how to make NixOS work, and I wasn't even making much headway at that.
You frighten me a bit here. I looked at the article and have the ability to build a web site in a minimum locked env is great. Just try to taste it and it seemed not as he said. Wonder now.
It's been years since I last tried NixOS. One of the things that I really didn't like was that changing something that had a lot of dependencies meant that everything that depended on it, needed to be rebuilt. That's so even if I knew the change couldn't affect how the dependencies would be built.
I know why NixOS requires that. I think that's really cool from a safety side, but it was also really annoying to have to wait so, so long to use the system after an insignificant change. I wish there were a "I know what I'm doing; let me risk screwing up" mechanism. I don't know if there's one now, but I kind of doubt it.
Happy to see more people interest in Nix. At the end of the post there are some links on learning resources, that list is 100% the best way to get started.
If they built their app differently, such as using a temporary directory rather than building it in tree in the release path, they could've used staged builds to cache their steps in docker. What they do get from Nix package management is freedom from dependency hell.
Actually, I prefer habitat (hab) for portable Linux packaging because it supports a crazy amount of caching, it does GC on unneeded packages, it has an internet-visible package repo to publish to and has a simple DSL that I can teach to anyone. I don't use its other features that much, and I don't use it on FreeBSD but I use hab studio on macOS now and then.
Nix is pretty awesome because it almost marries package management and configuration management... if you think about it, the definition of a conventional OS package is an arbitrary one that limits configurability and customization. Gentoo, Arch and others try to work around it, but you really can't unless you can drive and hook intelligently into specification language that built the package and can accept modifications at different points without a lot of kludgy hacks or forks. Furthermore, the conventional package paradigm assumes one version and one configuration for everything, unless you want to get messy with variant packages. Nix and hab solve these issues.
The problem is I can't suggest it in a lot of usage contexts because it would be like if I recommended Rust, Elixir or Haskell to a PHP shop. ;) I really want to but I know it's probably not a good idea. Nix is awesome wherever the badassometer says "above average." Give it a try, but use what works best.
>If they built their app differently, such as using a temporary directory rather than building it in tree in the release path, they could've used staged builds to cache their steps in docker
Can you explain this just a bit more, please?
I was just reading the two articles you linked today, and your comment seems there is some piece of insight that's missing.
If you search the multi-stage build article for "cache" the only hits are related to not using cache.
What should I do if I want to leverage the cache as much as possible in a multi stage build?
I've tried to use Nix in the past, but I can't get past the layers of abstraction. Normally I go and consult the relevant docs for what I'm trying to configure (postgres, systemd, etc), but with Nix I also need to consult the Nix docs and figure out the discrepancies/map values between them. I love the idea of systems configuration like this, but I'd much rather see less abstraction ala cloud-init with nixos-rebuild.
For those who use Nix, how do you deal with this kind of abstraction? Additionally, how soon do you see security updates land?
I have to do a bit of DevOps at work, and I absolutely hate every second of it, hence I tried Nix, and I think it is absolutely awesome. As a software developer, I find the ability to take a cryptographically consistent closure of a program's environment and deploy it in a reproducible way across your dev, loadtest and prod machines _the only reasonable way_ to do things. Not to mention that Nix community is very knowledgeable and open to patches and suggestions.
However after a brief investigation period I abandoned this idea altogether, here's why.
In my opinion, Nix is only worth climbing its steep learning curve when you really utilize its "reproducible environment" concept. As a counterexample, take your home PC environment, which is 1) unique 2) quickly changing. You don't need to reproduce it very often (how often do you upgrade your laptop?). You don't really want a hash-perfect reproducibility there, and a simple Ansible script will work just fine. So, the target of Nix should be "serious enterprise", where you deploy hundreds of machines, right?
Well, "serious enterprise" where I am at (banking) will _never_ adopt Nix. There are two main reasons why:
1) NixOS doesn't follow standard Linux filesystem hierarchy. They get around this by patching everything, both manually and automatically. nixpkgs repo contains thousands of patches. To put it in the perspective, OS is one of your trust anchors. I don't want to sound as "no one got fired for choosing IBM" guy, but legal people will scream bloody murder if they see extensive level of patching going on in the distro, for a variety of reasons, not necessarily tied to security. Some of their patching goes very deeply, BTW, and straight up changes behavior of the software.
2) Secrets management: it doesn't exist in Nix. nix store itself is world-readable. There are fundamental problems with some cryptographic algorithms that require entropy, and therefore go against "reproducible environment" concept.
YMMV, but for me Nix didn't do the job, although I studied it deeply, and contributed some patches to nixpkgs. It looks like a futuristic research project. On the outside it is brilliant, but in brutal reality you need to run the ugliest scripts to patch shebangs and what not.
I find it pretty stunning that Nix has decided to drop support for OS X over ideological reasons like this. There are workarounds, but the reality is that they're upset at apple's decision so they just dropped support. We're not close to a year in to this these issue threads. Any software which would be so anti-user does not deserve to be used. Nix might be magical but that sucks.
I think it's a lot more complicated than that. Yes there are plenty of people that use Docker as a package manager, but I think that's a failing of traditional package managers rather than a failing of Docker. You could, for exactly the same reasons, say that Nix is Nix Containers[1] Done Right but if that were true then Nix Containers wouldn't need to exist in the first place.
I use Nix for every project once it grows enough for me to have to pin a directory. And sometimes sooner.
I don't know it that well but I've picked up on design patterns, use the repl to figure stuff out, write small abstractions when necessary..and it works pretty great. And the gains grow as my dependencies become polyglot (even C + another language like Haskell..but when you add compile-to-JS, static asset generation into the mix it's amazing)
Yeah it's hard to learn. No denying it. But every other "alternative" I've found doesn't actually approach it. So I've kind of gotten sick of reading about "alternatives" because it always feels like they don't actually solve the problem Nix solves. I always end up using the alternative+Nix. Classic example of this is the Haskell stack build tool.
I could've bemoaned complexity and various capitalist criticisms when I first ran into it at a job. But I'm so much better off for not.
> The image age might look weird at first, but it’s part of the reproducibility Nix offers. The date an image was built is something that can change with time and is actually a part of the resulting file. This means that an image built one second after another has a different cryptographic hash. It helpfully pins all images to Unix timestamp 0, which just happens to be about 50 years ago.
This doesn’t mean builds done by Nix are bit-for-bit reproducible by default, does it?
There are a lot of other ways to introduce non-determinism in builds, like rand() or network requests (which I think could only be eliminated in a generic way by literally emulating the CPU of the machine doing the build and not allowing external communication?)
It provides the hash of files as a method of verifying the input identity. As long as the build file is pure function of build tools (gcc, make, etc), and the build tools themselves are deterministic, the build outputs should be reproducible. Though I've never seen anything like `-frandom-seed=<input-file-name>` in build files. I think bit-by-bit reproduciblity is one problem tackled by guix.
The nix by default makes builds in a sandbox. It is purposefully making certain operations unavailable (local home directory, local configuration, network access etc). I don't think it prevents Rand(), but that's not as common.
I still haven't figured out the nix language, how to find which package installs a particular binary if it isn't the name of the package, or how to find config options for packages I do have.
Those three problems are what's holding it back for me. I still enjoy it so far, though.
Most of their complaints about Docker, specifically:
* The package manager is included in the image
* The package manager’s database is included in the image
* An entire copy of the C library is included in the image (even though the binary was statically linked to specifically avoid this)
Can be solved with Docker multi-stage builds[0]. This is essentially a separate build stage inside your single Dockerfile that sets up, builds, and creates artifacts. Those artifacts are then copied over into a later build stage. The resulting docker image is lean and yet the artifacts were built at docker build time, and its all contained in a single file.
Looks like that author prefixes the month with "M". Dunno why.
Sometimes DD-MM-YYYY and MM-DD-YYYY can be confused, such that if one of those formats is used, it could be helpful to clarify. However, YYYY-MM-DD is typically pretty unambiguous, so it seems odd here.
I guess it's probably meant in analogy to, e.g., "2020-Q1" (which would be the first Quarter of 2020) or "2020-H1" (which would be the first Half of 2020).
[+] [-] nitrix|6 years ago|reply
The thing is, Nix is an absurdly complex piece of software, perhaps on a similar order of magnitude as maybe Kubernetes or other gizmos, requiring a very large time commitment to understand and to use properly.
Just for fun, have any of you tried to keep up during the Docker bigbang, with all these technologies popping up left and right, switching names, dying abruptly, getting super-seeded or hacked with vulnerabilities? That was one of my most mentally draining year for me and I've been in the field for a while now.
See... along the way, I've learned that Computer Science is always about tradeoffs. Now when I'm being asked to trade away my sanity learning this nonsense for in return a couple of megabyte of disk space that could've been wasted, I just don't see the value.
Seriously, are we optimizing for the right stuff here? I know Nix is doing package isolation with a programmable environment defined by text files, it's immutable and all. I get it. But we have alternatives for these problems. They do a pretty good job overall and we understand them.
Ironically, even those solutions I'm comfortable with are still struggling to get good adoption! How the heck is our field going to avoid fragmentation when it keep growing exponentially like this.
Perhaps it's just me aging and getting groggy; though there must be an explanation for this phenomenon. Please, it definitely gnaws at me.
[+] [-] dragonsh|6 years ago|reply
In the modern security conscious world where most companies run their workloads on virtual machines controlled by other companies, it's imperative that applications are deployed on such predictable, secure and reproducible operating systems.
Guix supports transactional upgrades and roll-backs, unprivileged package management, and more. When used as a standalone distribution, Guix supports declarative system configuration for transparent and reproducible operating systems.
I still think Amazon, Google and Azure will take a decade or two to build and offer something like this to it's customers. Indeed Google's Fuschia is trying to do the same, but I feel it's at least a decade away.
It has one of the best wonderful documentation [2] to get started and explore with all the details.
[1] https://guix.gnu.org/
[2] https://guix.gnu.org/manual/en/html_node/
[+] [-] sterlind|6 years ago|reply
1. Small package base.
My workflow includes running Julia on CUDA, editing from VS Code. Remarkably, Guix has both Julia and (on a third-party PKG repo) CUDA. What it's missing is VS Code. I understand that VS Code is non-free (like Chrome, it attaches trademarks on its branding), but there's no VS Codium I could find either. I also can't find Atom. Am I missing something?
2. Package index mirroring.
Nix has an issue where Python, Node and other "third-party" packages have to be manually added to nixpkgs. If you want celery or requests, you're in luck because those are popular. If you want something more obscure, like pywikibot, you have to add it yourself (or run pip/npm yourself, and forego the benefits of Nix/Guix.) Does Guix address this?
3. Ease of scripting.
This might be a plus for Guix, since Nix design patterns are arcane (the language is easy, things like overrides and needing to replace broken URLs is hard), but how easy is it to debug failing builds? Is there something like a debugger, or at least a way to shell-in and call things yourself?
Bonus #4: broken URLs. is there a cache? what can be done?
[+] [-] ertian|6 years ago|reply
I admire the ideological purity, but it kinda renders it irrelevant to me.
[+] [-] ozataman|6 years ago|reply
[+] [-] xena|6 years ago|reply
[+] [-] kalium_xyz|6 years ago|reply
[+] [-] CameronNemo|6 years ago|reply
[+] [-] Legogris|6 years ago|reply
(This is a not me criticizing NixOS, just noting that they do things very differently and you really need to invest some time and effort to start being able to solve your own problems rather than relying on what's provided by others)
[+] [-] Kaze404|6 years ago|reply
Maybe we're just not there yet.
[+] [-] SlowRobotAhead|6 years ago|reply
I’ve never had to search so hard to change the link negotiate settings on a nic before.
Maybe it comes more natural over time, but I’d be concerned about not remembering “the voodoo” to make things work when I came back from vacation.
[+] [-] karatestomp|6 years ago|reply
For reference I ran Gentoo on an IBM Thinkpad as my main computer for a few years in the oughts, among a bunch of other Linux work, so I, by necessity, at least kind of know what I'm doing with desktop Linux installation and config, from a fairly low level on up. At least that knowledge has proven highly transferable and, over and over again, relevant and useful. I didn't get the impression I was learning anything from NixOS other than how to make NixOS work, and I wasn't even making much headway at that.
[+] [-] ngcc_hk|6 years ago|reply
[+] [-] newnewpdro|6 years ago|reply
[+] [-] jolmg|6 years ago|reply
I know why NixOS requires that. I think that's really cool from a safety side, but it was also really annoying to have to wait so, so long to use the system after an insignificant change. I wish there were a "I know what I'm doing; let me risk screwing up" mechanism. I don't know if there's one now, but I kind of doubt it.
I imagine Guix would have the same problem.
[+] [-] Ericson2314|6 years ago|reply
[+] [-] corysama|6 years ago|reply
That works wonderfully until it bites your face.
[+] [-] jolmg|6 years ago|reply
[+] [-] oldwinebottles|6 years ago|reply
[+] [-] thundergolfer|6 years ago|reply
It really clearly explores the problem space Nix dives into. I learnt a lot about why Nix needs to exist.
[+] [-] anonsivalley652|6 years ago|reply
Actually, I prefer habitat (hab) for portable Linux packaging because it supports a crazy amount of caching, it does GC on unneeded packages, it has an internet-visible package repo to publish to and has a simple DSL that I can teach to anyone. I don't use its other features that much, and I don't use it on FreeBSD but I use hab studio on macOS now and then.
Nix is pretty awesome because it almost marries package management and configuration management... if you think about it, the definition of a conventional OS package is an arbitrary one that limits configurability and customization. Gentoo, Arch and others try to work around it, but you really can't unless you can drive and hook intelligently into specification language that built the package and can accept modifications at different points without a lot of kludgy hacks or forks. Furthermore, the conventional package paradigm assumes one version and one configuration for everything, unless you want to get messy with variant packages. Nix and hab solve these issues.
The problem is I can't suggest it in a lot of usage contexts because it would be like if I recommended Rust, Elixir or Haskell to a PHP shop. ;) I really want to but I know it's probably not a good idea. Nix is awesome wherever the badassometer says "above average." Give it a try, but use what works best.
Best practices > Leverage build cache:
https://docs.docker.com/develop/develop-images/dockerfile_be...
Multi-stage build:
https://docs.docker.com/develop/develop-images/multistage-bu...
[+] [-] ses1984|6 years ago|reply
Can you explain this just a bit more, please?
I was just reading the two articles you linked today, and your comment seems there is some piece of insight that's missing.
If you search the multi-stage build article for "cache" the only hits are related to not using cache.
What should I do if I want to leverage the cache as much as possible in a multi stage build?
[+] [-] candiddevmike|6 years ago|reply
For those who use Nix, how do you deal with this kind of abstraction? Additionally, how soon do you see security updates land?
[+] [-] zegl|6 years ago|reply
[+] [-] h91wka|6 years ago|reply
However after a brief investigation period I abandoned this idea altogether, here's why.
In my opinion, Nix is only worth climbing its steep learning curve when you really utilize its "reproducible environment" concept. As a counterexample, take your home PC environment, which is 1) unique 2) quickly changing. You don't need to reproduce it very often (how often do you upgrade your laptop?). You don't really want a hash-perfect reproducibility there, and a simple Ansible script will work just fine. So, the target of Nix should be "serious enterprise", where you deploy hundreds of machines, right? Well, "serious enterprise" where I am at (banking) will _never_ adopt Nix. There are two main reasons why:
1) NixOS doesn't follow standard Linux filesystem hierarchy. They get around this by patching everything, both manually and automatically. nixpkgs repo contains thousands of patches. To put it in the perspective, OS is one of your trust anchors. I don't want to sound as "no one got fired for choosing IBM" guy, but legal people will scream bloody murder if they see extensive level of patching going on in the distro, for a variety of reasons, not necessarily tied to security. Some of their patching goes very deeply, BTW, and straight up changes behavior of the software.
2) Secrets management: it doesn't exist in Nix. nix store itself is world-readable. There are fundamental problems with some cryptographic algorithms that require entropy, and therefore go against "reproducible environment" concept.
YMMV, but for me Nix didn't do the job, although I studied it deeply, and contributed some patches to nixpkgs. It looks like a futuristic research project. On the outside it is brilliant, but in brutal reality you need to run the ugliest scripts to patch shebangs and what not.
[+] [-] fctorial|6 years ago|reply
Can you give some examples?
[+] [-] srik|6 years ago|reply
[+] [-] echelon|6 years ago|reply
This reminds me of GOROOT/GOPATH and other complexities that arise when systems care too much about paths.
[+] [-] kohtatsu|6 years ago|reply
[+] [-] anarchogeek|6 years ago|reply
[+] [-] kragen|6 years ago|reply
[+] [-] Sean1708|6 years ago|reply
[1]: https://nixos.org/nixos/manual/#ch-containers
[+] [-] whateveracct|6 years ago|reply
I don't know it that well but I've picked up on design patterns, use the repl to figure stuff out, write small abstractions when necessary..and it works pretty great. And the gains grow as my dependencies become polyglot (even C + another language like Haskell..but when you add compile-to-JS, static asset generation into the mix it's amazing)
Yeah it's hard to learn. No denying it. But every other "alternative" I've found doesn't actually approach it. So I've kind of gotten sick of reading about "alternatives" because it always feels like they don't actually solve the problem Nix solves. I always end up using the alternative+Nix. Classic example of this is the Haskell stack build tool.
I could've bemoaned complexity and various capitalist criticisms when I first ran into it at a job. But I'm so much better off for not.
[+] [-] tlrobinson|6 years ago|reply
This doesn’t mean builds done by Nix are bit-for-bit reproducible by default, does it?
There are a lot of other ways to introduce non-determinism in builds, like rand() or network requests (which I think could only be eliminated in a generic way by literally emulating the CPU of the machine doing the build and not allowing external communication?)
[+] [-] fctorial|6 years ago|reply
[+] [-] takeda|6 years ago|reply
[+] [-] eximius|6 years ago|reply
Those three problems are what's holding it back for me. I still enjoy it so far, though.
[+] [-] SitiSchu|6 years ago|reply
Another way would be reading the nix expression on the nixpkgs GitHub repository.
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] sagebird|6 years ago|reply
I know that include os stuff is young.
But I wonder, would it be easier to build the website with go, with an os included, and then deploy that?
Both the docker and nix approach seem like alot of random tasks or knowledge is required. Are there bad things about including an OS I am forgetting?
[+] [-] kalium_xyz|6 years ago|reply
[+] [-] daenz|6 years ago|reply
0. https://docs.docker.com/develop/develop-images/multistage-bu...
[+] [-] takeda|6 years ago|reply
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] dfc|6 years ago|reply
What is "2020 M2 10"?
[+] [-] _Nat_|6 years ago|reply
Sometimes DD-MM-YYYY and MM-DD-YYYY can be confused, such that if one of those formats is used, it could be helpful to clarify. However, YYYY-MM-DD is typically pretty unambiguous, so it seems odd here.
I guess it's probably meant in analogy to, e.g., "2020-Q1" (which would be the first Quarter of 2020) or "2020-H1" (which would be the first Half of 2020).
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] bb010g|6 years ago|reply
[+] [-] xena|6 years ago|reply
[+] [-] pmarreck|6 years ago|reply