Building a Linux distro from scratch has been one of the most tasking projects I've attempted. There are two phases: the frustration phase, and the tedious phase. Bootstrapping it is an incredibly frustrating process - I restarted from scratch 5 times before any attempt ever got to the tedious phase. The tedious phase never ends. You have to make hundreds of packages to get to a usable desktop system. I've made 407 packages and I still don't have a desktop to show for it (I'm hoping to get sway working tomorrow, I think I have about 30 packages left).
Still, I've finally gotten to a somewhat practical self-hosting system. My laptop runs it on the metal and all of the infrastructure (website, bugzilla, git hosting, mirrors) is on a server running my distro too. It's taken almost a year of frustration, but it's very rewarding.
I'm just curious: In what way is it rewarding? Just for educational purpose? Or anything else? Or is it just a hobby? I mean, you are probably investing a lot of time into this. This time you could spend on other things instead.
I see the educational purpose. Although you probably get similar experience/understanding/education if you use something like Gentoo and you would have to invest much less time because all the nasty corner cases are already taken care of.
Good for you for sticking with it. I've started in that direction a few times, but it really is amazing just how necessary all those hacky distro-maintained build scripts and patches are. It's tempting to think that they're just tweaking things to fit their conventions and handling edge cases, but they're often just getting it to not explode by default.
I really wondered what's so special about this project that aims to achieve what probably more projects aimed to achieved than there are lines of code in the kernel.
It's not the goal, it's not the OS. But the documentation is a sight to behold! Very clear, detailed, interesting writing style, and it puts together quite a few frustrating topics in a simple, structured matter. Wow and kudos! Keep on writing docs, please!
Your kind words are much appreciated. Many people seem to indeed be missing the point. I wasn't trying to create a new distro, I'm just trying to teach people in a way so that it would just take a few hours of your time.
Both this and LFS reminds me that Linux makes sense until you get the DEs involved. At that point shit just sprawls all over the place as there are no longer any notion of layers.
Back in 2002 I ran LFS for a semester and a half. Then I got tired of saving all my ./configure flags and switch to Redhat and gained a respect for package management. :-P Eventually I started using Gentoo.
Still, I'm really glad I built and ran an LFS for nearly a year. It helped me build a really in-dept knowledge of Linux that's helped me as I later moved into embedded development.
Today I still use Gentoo (which feels like LFS with package management) and Void and would recommend either for Linux devs.
I'm running i3, polybar, connman and rofi as my Linux desktop environment and its very productive, configurable, lightweight and reliable. My only worry is that one day I'll be forced to move to something akin to what you describe. I blame object-oriented programmers.
The problem is described perfectly with the name: desktop environment.
The very goal of a desktop environment is to create an entire environment, rather than using the one that is already there.
Desktop environments like GNOME and MADE have so many worthless parts that are tightly coupled dependencies, they end up becoming the same bloated mess that Windows and OS X are.
If a desktop environment focused on creating separate packages that played well with the underlying system, they would not have this problem.
A similar process is building a custom kernel and rootfs for an ARM device such as the beaglebone. Olimex actually has a good tutorial for their device: https://www.olimex.com/wiki/AM335x
This is only slightly more complicated due to the need to cross-build but I found it fairly easy with qemu-static-arm and prebuilt cross toolchain packages for Ubuntu/ Debian.
The benefit is that you can develop for a target device that is not your PC, so no worry about messing up the bootloader and leaving your PC in a state where you need a recovery CD to fix it and boot. Just get a USB-serial cable :)
You can also try buildroot or yocto, although I had no interest in building every package manually versus relying on Debian's repos.
Actually LFS teaches you about the whole GNU Userland, getting a kernel up and running with busybox is much easier. This project does just that - teaching you the bare minimum.
It's supposed to be a lot shorter and simpler. Which means many important things were left out for the sake of brevity. I also think LFS doesn't always explain why we are doing certain things a certain way or how things work. Sure you get a working system but it's often assumed you know why it's working. I'm not sure if I did a better job though.
I see this project as the next logical step beyond LFS. LFS will help you get one up and running from start to finish, but once you do that, you might want to try it a bit more free-form. That's where this project seems to fit in.
I have a question related to this article, though not directly. If building completely from scratch turns out too cumbersome and time-consuming, what would be the easiest way of building a minimal, fast starting distro with graphical user interface and networking whose only purpose is to run one application on x86 hardware in kiosk mode?
NixOS is awesome for stuff like this. Very easy to define your own "base image" and layer whatever you want on top of it. Binary packages by default, but you can override the compilation flags for any any package, Gentoo-style. Configuration files are also packages, which frees you from the cruft that systems like Chef or Puppet tend to accumulate.
Oh, and NixOps is awesome for managing multiple machines. And it can automagically set up VMs corresponding to your production network.
SUSE Studio has a nice GUI and is easy to get started with, but AFAIK it doesn't have a nice story for configuration management, or managing the systems after they've been deployed.
Sure there is LFS. I suppose arch could give you a fairly minimal installation. Really when you think of it harddisc usage is hardly the problem here so even though it might seem silly having a 10GB installation, it will cost you nothing and save you time.
But if you want it quick install debian or arch or whatever and simply start only X11 and whatever application you want to run. If there is no window manager or an extremely limited mode, you got kiosk mode.
Following this guide will get you something very close to a base Alpine Linuxwith busybox. Alpine is fairly minimal out of the box and even eschews grub for syslinux.
The upside with Alpine is if you need features and packages they are an install away. But if the purpose is to learn about compiling the kernel and how the system initalizes this is a decent start.
Huge fang on Linux From Scratch myself, after reading this I wonder if someone has tried the same with FreeBSD! Or the Darwin sources released for OSX (.. not talking about dormant PureDarwin project)
This is awesome. I've been building almost the exact same project, along almost the same timeline (based on the commit history). Mostly an excuse to learn more advanced Bash, and Linux Internals/Features I've never had a good excuse to explore. Gonna release next week. Hope I get as warm a reception. Kudos on an awesome project!
I am following this guide to build a kernel. But it seems like that instead of getting the headers from the kernel source they are using a github repository which only contains the headers to save downloading time. All fine and dandy if the latest commit to this repo wasn't from 3 years ago!!
This is an awesome write-up. did not know losetup can do what kpartx does now with the option -P, I did similar things in the past but this is a good update for me.
LFS is infinitely more involved and am not sure for good reasons. I don't like to say it because it's someone else project but I feel important parts are missing, format is wrong, and without BLFS the goal is only half realized.
[+] [-] ddevault|9 years ago|reply
Still, I've finally gotten to a somewhat practical self-hosting system. My laptop runs it on the metal and all of the infrastructure (website, bugzilla, git hosting, mirrors) is on a server running my distro too. It's taken almost a year of frustration, but it's very rewarding.
[+] [-] albertzeyer|9 years ago|reply
I see the educational purpose. Although you probably get similar experience/understanding/education if you use something like Gentoo and you would have to invest much less time because all the nasty corner cases are already taken care of.
[+] [-] 0xcde4c3db|9 years ago|reply
[+] [-] erikb|9 years ago|reply
It's not the goal, it's not the OS. But the documentation is a sight to behold! Very clear, detailed, interesting writing style, and it puts together quite a few frustrating topics in a simple, structured matter. Wow and kudos! Keep on writing docs, please!
[+] [-] michielderhaeg|9 years ago|reply
[+] [-] digi_owl|9 years ago|reply
[+] [-] djsumdog|9 years ago|reply
Still, I'm really glad I built and ran an LFS for nearly a year. It helped me build a really in-dept knowledge of Linux that's helped me as I later moved into embedded development.
Today I still use Gentoo (which feels like LFS with package management) and Void and would recommend either for Linux devs.
[+] [-] willtim|9 years ago|reply
[+] [-] thomastjeffery|8 years ago|reply
The very goal of a desktop environment is to create an entire environment, rather than using the one that is already there.
Desktop environments like GNOME and MADE have so many worthless parts that are tightly coupled dependencies, they end up becoming the same bloated mess that Windows and OS X are.
If a desktop environment focused on creating separate packages that played well with the underlying system, they would not have this problem.
[+] [-] thom_nic|9 years ago|reply
This is only slightly more complicated due to the need to cross-build but I found it fairly easy with qemu-static-arm and prebuilt cross toolchain packages for Ubuntu/ Debian.
The benefit is that you can develop for a target device that is not your PC, so no worry about messing up the bootloader and leaving your PC in a state where you need a recovery CD to fix it and boot. Just get a USB-serial cable :)
You can also try buildroot or yocto, although I had no interest in building every package manually versus relying on Debian's repos.
[+] [-] asciimo|9 years ago|reply
[+] [-] duncan_bayne|9 years ago|reply
[+] [-] philbo|9 years ago|reply
[+] [-] fizixer|9 years ago|reply
[1] http://www.linuxfromscratch.org/
[+] [-] accurrent|9 years ago|reply
[+] [-] partycoder|9 years ago|reply
[+] [-] michielderhaeg|9 years ago|reply
[+] [-] exDM69|9 years ago|reply
[+] [-] jokr004|9 years ago|reply
[+] [-] lanna|9 years ago|reply
[+] [-] btgeekboy|9 years ago|reply
[+] [-] jonathanstrange|9 years ago|reply
Is there a distro builder for dummies?
[+] [-] slezyr|9 years ago|reply
You can pick packages and build a distro from browser.
NixOS - You can make one config file to specify how your system should look like. It's easy to make GUI for it.
[+] [-] Nullabillity|9 years ago|reply
Oh, and NixOps is awesome for managing multiple machines. And it can automagically set up VMs corresponding to your production network.
SUSE Studio has a nice GUI and is easy to get started with, but AFAIK it doesn't have a nice story for configuration management, or managing the systems after they've been deployed.
[+] [-] rijoja|9 years ago|reply
But if you want it quick install debian or arch or whatever and simply start only X11 and whatever application you want to run. If there is no window manager or an extremely limited mode, you got kiosk mode.
[+] [-] ChickeNES|9 years ago|reply
[+] [-] kasabali|8 years ago|reply
[+] [-] throw2016|9 years ago|reply
The upside with Alpine is if you need features and packages they are an install away. But if the purpose is to learn about compiling the kernel and how the system initalizes this is a decent start.
[+] [-] thu|9 years ago|reply
[+] [-] marenkay|9 years ago|reply
[+] [-] blanket_the_cat|9 years ago|reply
[+] [-] peterwwillis|9 years ago|reply
[+] [-] rijoja|9 years ago|reply
[+] [-] agumonkey|9 years ago|reply
https://wiki.gentoo.org/wiki/Comparison_of_init_systems
[+] [-] colemickens|9 years ago|reply
[+] [-] felixsanz|9 years ago|reply
[+] [-] ausjke|9 years ago|reply
[+] [-] apeacox|9 years ago|reply
[+] [-] agumonkey|9 years ago|reply
[+] [-] Jaruzel|9 years ago|reply