OK, so here's a question. I've always wanted to spend some time reading through the Linux kernel - I really enjoyed operating systems when I studied it (and reading through MINIX src) but I don't really know where/how to start with Linux.
I mean, I guess I can just dive in, and maybe that's the best approach, but is there a strategy anyone would recommend for reading the Linux source in terms of it making sense as a combined unit of code (as opposed to a collection of algorithms, if that makes sense)?
I recently did exactly what you describe. My first contribution to the Linux kernel actually shipped with this release. If I recall correctly it took me about 80 hours to go from no knowledge to first patch. Assuming no knowledge, it is a two step process.
(1) Learn basic OS concepts through xv6.
xv6 is a reimplementation of an early version of Unix, designed to be as simple as possible and accompanied by a whole book of commentary. Get the book and the source cost listing printed and bound: http://pdos.csail.mit.edu/6.828/2012/xv6.html. Work through the book and exercises. Use the lecture videos from 6.828 from 2011 if you need extra material in order to understand: http://pdos.csail.mit.edu/6.828/2011/schedule.html.
(2) Pick a part of OSes that you are interested in. Contribute to that part in Linux.
Figure out where that part is in the Linux kernel. Find a bug in the bug tracker and submit the patch. I found filesystems interesting, so I fixed a small bug in one of the filesystems. Use a cross reference, it will save you a lot of time: http://lxr.free-electrons.com/source/include/linux/cpu.h. Also feel free to subscribe to the Linux kernel subreddit: http://www.reddit.com/r/kernel. I've set up the sidebar with a lot of useful links.
The Linux kernel is large and complex. You need to equipe yourself with a mental model of an OS through xv6 and then pick one small, specific part to attack in Linux. Be tactical! Otherwise you will be overwhelmed.
--
As an aside, I'm actually currently working on a tool that parses the Linux source code to find symbol definitions and then works its way back through the Git history to find the commit message for when the symbol was first defined. These commit messages usually contain really useful information about the original intent of the symbol and implementation details. Currently fighting with a few bugs in my C grammar, but should be able to work through those soon. Please feel free to email me at [email protected] if you want to be pinged when the tool is released.
This is going to be unsatisfying, but the only thing that works for motivating me to understand a piece of kernel code has been wanting to change it and having to learn exactly how it works to achieve that.
If you just want to learn more about how the kernel fits together, reading http://lwn.net/Kernel/LDD3/ (Linux Device Drivers, freely downloadable) is a fine start.
One thing I found a little different was that the OS has its own libraries for everything (string.h etc..) which makes sense if you think about it.
If you want to browse the source code on-line with this software website thing called lxr (Linux Cross Reference). Its got a good search tool and linked headers. Clicking on a function name shows you where that function is used.
You can install it yourself and I think its much faster..
http://lxr.linux.no/linux+v3.6/http://rhkernel.org/
There is linux weekly news too, which is a decent site when I was still in the Linux porting world. Like many Linux sites, seems to lack in style, but makes up in content.
http://lwn.net/
start off with reading 1) linux kernel development 2) linux device drivers 3) linux kernel module programming guide
have - understanding the linux kernel as your reference manual.
by now, you should be comfortable to read/understand the kernel source; download linux kernel source and start browsing through the code.
simply reading books wont get you anywhere - you need to play around with kernel source inorder to understand the linux kernel behavior and different problems you may come across. write simple kernel modules to get a hang of how you can interact/modify with the kernel.
join some opensource project and start fixing bugs you're comfortable with or just play around with your local linux kernel source - make changes; build and deploy and observe what happens.
EDIT:
if you have no prior knowledge of OS Theory and Fundamentals; then you should start here first - read either of the following books
1) Operating System Concepts by Galvin, Silberschatz OR 2)Modern Operating Systems by Tanenbaum
For programming related - system calls and stuff
read 1) Advanced Programming in the UNIX Environment - by Richard Stevens
You may want to start with the earliest Linux kernels as they will be considerably smaller and simpler. You'll also be able to fit the whole kernel in your head.
From that point you can look at yearly diffs to find subsystems that have changed of interest. (Are 4 rewrites of USB interesting?)
I'd really like to see btrfs start to stabilize and become more mainstream, but everything still seems to say that it's experimental.
The main reasons that I want it on my desktop are:
* easy backups
* checksums
* maybe it will become integrated with package manager (I'm on ubuntu) so that I can roll back package updates if they don't work properly
It is integrated with the package manager on Ubuntu already. I just upgraded my laptop to Quantal and part of the upgrade process spotted I was using btrfs and created a snapshot. (It was a complete waste of time as I have automatically created hourly, daily, weekly and monthly snapshots.) Doing regular package updates is very slow with btrfs. This is due to fsync() being called several times per file.
You'll also want Quantal for an updated btrfs-tools package. For example it lets you do scrubbing. You also need it (and kernel 3.3+) to change raid levels after filesystem creation. The Ubuntu 12.04 installer does create and install to btrfs but didn't let you configure things like raid level so you were stuck with whatever it did.
I've been running all my systems on btrfs for several months now on Ubuntu 12.04 (kernel 3.2) on both SSD and HDD including using RAID 0, RAID 1, dmcrypt/LUKS, inplace conversion from ext4 and who knows what else. (Across my server, workstation, laptop and HTPC.)
The only problem I have had is when filesystems fill up. I've never lost data but it can be quite frustrating trying to find files to delete (also need to be removed from snapshots), rebalancing etc to get things running smoothly again. The various ways of doing df are mostly a work of fiction.
My data is duplicated to offline backups, Dropbox, multiple systems and git/hg repositories so the failure of any system would be annoying but I'd never lose anything. You should make sure you are in that position first, independent of what filesystems are being used.
The fsck for btrfs is still not quite there. If you are using multiple disks with btrfs's mirroring/checksums, hopefully you won't get hit by HDD hiccups leading to metadata corruptions which btrfs's fsck can't yet handle. I certainly wouldn't recommend running btrfs on a single HDD/SSD unless you are doing frequent backups and don't mind losing state between backups if you get unlucky.
Also, please note that btrfs snapshots != backups. They will not save you in case of device failure. Checksums also won't help you if there are kernel bugs.
I have been running Ubuntu 12.04 with my / as btrfs for a couple of months now, with no trouble (and not even thinking about it truth be told). That said, I left my /home ext4 because I didn't know how much to trust it.
And I still can't sleep my Powerbook /w Radeon because post KMS radeon doesn't support it yet (no one ported the pre-KMS code from the X driver portion of radeon into the post-KMS radeon kernel driver).
OTOH, I can just leave KMS off and have no 2D or 3D acceleration.
The code is already out there, all 3.7 integration would represent is a stamp of approval. Consider that there is no hardware for the architecture yet, I don't know why approval would be very interesting:
From the desktop perspective, I really like the "Suspend to disk and memory at the same time" part. It's great that you don't have to think anymore if you're going to use your laptop in the next 2 days or not. If it runs out of power, it does - nothing gets lost!
Adding bufferbloat pieces into the main release is also good news (although that probably won't hit the servers that really need it for many more months...)
[+] [-] alexholehouse|13 years ago|reply
I mean, I guess I can just dive in, and maybe that's the best approach, but is there a strategy anyone would recommend for reading the Linux source in terms of it making sense as a combined unit of code (as opposed to a collection of algorithms, if that makes sense)?
[+] [-] timsally|13 years ago|reply
(1) Learn basic OS concepts through xv6.
xv6 is a reimplementation of an early version of Unix, designed to be as simple as possible and accompanied by a whole book of commentary. Get the book and the source cost listing printed and bound: http://pdos.csail.mit.edu/6.828/2012/xv6.html. Work through the book and exercises. Use the lecture videos from 6.828 from 2011 if you need extra material in order to understand: http://pdos.csail.mit.edu/6.828/2011/schedule.html.
(2) Pick a part of OSes that you are interested in. Contribute to that part in Linux.
Figure out where that part is in the Linux kernel. Find a bug in the bug tracker and submit the patch. I found filesystems interesting, so I fixed a small bug in one of the filesystems. Use a cross reference, it will save you a lot of time: http://lxr.free-electrons.com/source/include/linux/cpu.h. Also feel free to subscribe to the Linux kernel subreddit: http://www.reddit.com/r/kernel. I've set up the sidebar with a lot of useful links.
The Linux kernel is large and complex. You need to equipe yourself with a mental model of an OS through xv6 and then pick one small, specific part to attack in Linux. Be tactical! Otherwise you will be overwhelmed.
--
As an aside, I'm actually currently working on a tool that parses the Linux source code to find symbol definitions and then works its way back through the Git history to find the commit message for when the symbol was first defined. These commit messages usually contain really useful information about the original intent of the symbol and implementation details. Currently fighting with a few bugs in my C grammar, but should be able to work through those soon. Please feel free to email me at [email protected] if you want to be pinged when the tool is released.
[+] [-] seiji|13 years ago|reply
A good "gentle introduction" book is the Love book (440 pages): http://www.amazon.com/Linux-Kernel-Development-Robert-Love/d...
[+] [-] cjbprime|13 years ago|reply
If you just want to learn more about how the kernel fits together, reading http://lwn.net/Kernel/LDD3/ (Linux Device Drivers, freely downloadable) is a fine start.
[+] [-] twoodfin|13 years ago|reply
http://www.amazon.com/Linux-Kernel-Development-Robert-Love/d...
[+] [-] acomjean|13 years ago|reply
The instructor was excellent. https://www.redhat.com/training/courses/rhd361/course-exam-o...
Well thats kinda expensive....
One thing I found a little different was that the OS has its own libraries for everything (string.h etc..) which makes sense if you think about it.
If you want to browse the source code on-line with this software website thing called lxr (Linux Cross Reference). Its got a good search tool and linked headers. Clicking on a function name shows you where that function is used. You can install it yourself and I think its much faster.. http://lxr.linux.no/linux+v3.6/ http://rhkernel.org/
There is linux weekly news too, which is a decent site when I was still in the Linux porting world. Like many Linux sites, seems to lack in style, but makes up in content. http://lwn.net/
[+] [-] Jarihd|13 years ago|reply
have - understanding the linux kernel as your reference manual.
by now, you should be comfortable to read/understand the kernel source; download linux kernel source and start browsing through the code.
simply reading books wont get you anywhere - you need to play around with kernel source inorder to understand the linux kernel behavior and different problems you may come across. write simple kernel modules to get a hang of how you can interact/modify with the kernel.
join some opensource project and start fixing bugs you're comfortable with or just play around with your local linux kernel source - make changes; build and deploy and observe what happens.
EDIT:
if you have no prior knowledge of OS Theory and Fundamentals; then you should start here first - read either of the following books 1) Operating System Concepts by Galvin, Silberschatz OR 2)Modern Operating Systems by Tanenbaum
For programming related - system calls and stuff read 1) Advanced Programming in the UNIX Environment - by Richard Stevens
[+] [-] rogerbinns|13 years ago|reply
From that point you can look at yearly diffs to find subsystems that have changed of interest. (Are 4 rewrites of USB interesting?)
[+] [-] jey|13 years ago|reply
[+] [-] jeltz|13 years ago|reply
http://kernelnewbies.org/Linux_3.6
[+] [-] jeffdavis|13 years ago|reply
The main reasons that I want it on my desktop are:
[+] [-] rogerbinns|13 years ago|reply
It is integrated with the package manager on Ubuntu already. I just upgraded my laptop to Quantal and part of the upgrade process spotted I was using btrfs and created a snapshot. (It was a complete waste of time as I have automatically created hourly, daily, weekly and monthly snapshots.) Doing regular package updates is very slow with btrfs. This is due to fsync() being called several times per file.
You'll also want Quantal for an updated btrfs-tools package. For example it lets you do scrubbing. You also need it (and kernel 3.3+) to change raid levels after filesystem creation. The Ubuntu 12.04 installer does create and install to btrfs but didn't let you configure things like raid level so you were stuck with whatever it did.
I've been running all my systems on btrfs for several months now on Ubuntu 12.04 (kernel 3.2) on both SSD and HDD including using RAID 0, RAID 1, dmcrypt/LUKS, inplace conversion from ext4 and who knows what else. (Across my server, workstation, laptop and HTPC.)
The only problem I have had is when filesystems fill up. I've never lost data but it can be quite frustrating trying to find files to delete (also need to be removed from snapshots), rebalancing etc to get things running smoothly again. The various ways of doing df are mostly a work of fiction.
My data is duplicated to offline backups, Dropbox, multiple systems and git/hg repositories so the failure of any system would be annoying but I'd never lose anything. You should make sure you are in that position first, independent of what filesystems are being used.
[+] [-] tytso|13 years ago|reply
Also, please note that btrfs snapshots != backups. They will not save you in case of device failure. Checksums also won't help you if there are kernel bugs.
[+] [-] triplepoint217|13 years ago|reply
[+] [-] DASD|13 years ago|reply
[+] [-] nine_k|13 years ago|reply
[+] [-] DiabloD3|13 years ago|reply
OTOH, I can just leave KMS off and have no 2D or 3D acceleration.
/rant
[+] [-] mtgx|13 years ago|reply
http://www.phoronix.com/scan.php?page=news_item&px=MTE5N...
[+] [-] ajross|13 years ago|reply
http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux-aar...
[+] [-] Nux|13 years ago|reply
[+] [-] viraptor|13 years ago|reply
Adding bufferbloat pieces into the main release is also good news (although that probably won't hit the servers that really need it for many more months...)