[cloudripper]

I gave LFS a go earlier this year. I learned a lot through the process - but I definitely went outside the guardrails. I use NixOS as my daily driver and found myself curious of whether I could complete LFS using a "Nix" approach. I was only a basic Nix user at that time and that choice made a difficult process much more difficult. However, the declarative nature of Nix meant that I had clear notes of every step of my process - and if something didn't work, I could backtrack and troubleshoot to find the root of the cause. The end result is here [0].

My understanding of Linux, of bootstrapping, cross-compilation, and Nix has grown tremendously as a result of the time I took on this project - and I still go back and reference the work from time to time. When I get some time to revisit the Nix-based LFS project, there are quite a few things I would like to clean-up, including setting kernel configs and handling post-build permissions.

Nix-complexities aside, I highly recommend LFS if you like to understand how things work and don't mind a little suffering along the way.

[0]: https://github.com/cloudripper/NixLFS

[hi-v-rocknroll]

While Nix/NixOS solved RPM dependency hell and side-by-side multiple concurrent version installation issues, it created its own problems.

The problem with Nix/NixOS is everything is too isolated to be configurable or usable, and so almost everything is broken because it doesn't work like any other OS. Chasing down all of the oddities and gotcha problems becomes a full-time IT job that slows everything else down, and then with a fragile special snowflake of patches and workarounds, it becomes very expensive and slow to do anything or to count on it being reliable. Furthermore, the syntax and nature of the packaging DSL is also too clever special snowflake that could've made do with declarative data or imperative procedural with something widespread like Python, Bourne shell, or JS to reduce the friction of customization and contribution.

Like Qubes, an over-optimization for particular goal(s) ends up becoming Achille's footguns because of a lack of compatibility and usability with everything else that came before.

[ertian]

You're basically just saying it's unfamiliar, and not universal yet.

In my experience, using & configuring NixOS is much simpler and easier than most distros...until it's not, at which point, yes, the difficulty curve is very steep.

Even in those cases, though, it's got the nice feature that once it's solved, it's solved. All the config is in one place, and you can apply the same config across all your hosts. You don't get those situations where you get something working after tweaking configs, env vars and systems jobs, and can't remember how to reproduce it.

And though some of the conventions around NixOS are pretty arcane and convoluted (though the language itself is pretty simple), it has the advantage that you only need to learn it once and you can use it all over, instead of switching between bash, python, yaml, toml, systemd units and even more obscure formats (nginx, apache, postfix, etc).

IMHO, the tradeoff is completely worth it.

[VTimofeenko]

Chasing down oddities on predictably isolated environments is much easier though. And since the environment is declarative, one can actually roll back to the previous working state without breaking other things. For that alone the learning curve of the DSL is worth it IMHO, but I am not saying that the language and ecosystem are perfect just the way they are.

[therein]

If the DSL was easier to work with, more familiar; and if there was some good IDE support with autocomplete and documentation, I think it would be amazing.

[tomberek]

This is excellent. Have you considered making a presentation or a write-up of the experience?

[cloudripper]

Thanks for the input. A couple of folks suggested that recently as well. Once I can clear up some bandwidth, I do intend to follow through on that. It gave me a huge appreciation for Nix as a build system, and I would love to share that if it were helpful to others.

[ayakang31415]

The first thing that pops out on Google search of NixOS is "Declarative builds and deployments." What exactly is NixOS different from other distros, such as Ubuntu?

[hollerith]

NixOS separates packages. If the package foo contains a file /usr/bin/foo, NixOS installs it in /nix/store/67c25d7ad7b2b64c67c25d7ad7b2b64c-foo/usr/bin/foo. In order to make this separation work, Nix must sometimes rewrite binaries so that all references in the binary to /usr/bin/foo becomes references to /nix/store/67c25d7ad7b2b64c67c25d7ad7b2b64c-foo/usr/bin/foo.

The advantage of this approach is that it gives more control to the distro maintainers and the admins of the computer, taking that control away from the "upstream" maintainers of the software being packaged. For example the software being packaged cannot just call the library bar because bar is not at /usr/lib/bar.so like it is in most Linux distros -- it is at /nix/store/17813e8b97b84e0317813e8b97b84e03-bar/usr/lib/bar.so, but of course the software does not know that unless the person creating the Nix package (the packager) arranges for the software to know it (again sometimes by doing a search-and-replace on binaries).

If the upstream maintainer of foo thinks foo should link to version 6 of library bar, but you think it should link to version 5, NixOS makes it easier for you to arrange for foo to link to version 5 than most distros do (even if version 6 of bar is needed by other packages you have installed which you need to use at the same times as your using foo).

Note that if this separation of packages imposed by NixOS has any beneficial security properties, it is merely security through obscurity because there is nothing preventing a binary from covertly searching through the directory listing of /nix/store/ for the name of the library it wants to call. Nevertheless it turns out the be useful to seize some control away from upstream in this way even if technically upstream could seize the control back if it were willing to complicate the software to do so.

People, including the creator of Nix and NixOS (Dolstra), will tell you that NixOS's main advantage is "declarativeness" (which in the past Dolstra called "purity") or the fact that the compilation / building process is deterministic. I believe both positions (NixOS's advantage is declarativeness and the advantage is deterministic builds) are wrong. Specifically, I believe that although deterministic builds are useful, the separation of packages I described is much more useful to most users and prospective users of NixOS.

Another way to summarize it is that NixOS package maintainers routinely modify the software they are packaging to use less "ambient authority".

[MuffinFlavored]

Why did you need to chmod every ~2,600 files? man pages, docs, includes. Why not chmod -R or something?

https://github.com/cloudripper/NixLFS/blob/64e333f60cca4bec5...

[cloudripper]

Fair questions. If the question is "why did I need to chmod those files", the answer is that permissions modifications in the mkderivation build environment are not propagated to the build output. The quick and dirty solution I came up with was to make a chmod wrapper that logged the calls so that I could apply them after the entire build completed.

For why I did not chmod the directories - that is something I should probably do when I get time again (cleaning up permissions handling more broadly). In my case at the time, I had the logs from the wrappers and did a quick filter before directly adding them to the script as a last hurdle to having a bootable machine after a month-long grind.

The kernel config handling needs a good amount of attention too. I have a cleaner approach to that in mind that I haven't had the opportunity to work on.

It will likely be December before I am able to invest much more time into it, but definitely open to any input generally.

[danlitt]

oh, fun, I was planning to do the same thing!

[ocean_moist]

> This results in a significant storage footprint, often exceeding 12GB.

12GB being considered significant makes me feel good, a return to simplicity. The other day I couldn't provision a VM with less than 512GB of storage...

I can't even play my favorite games without giving up >150GBs...