So many thank to all of you for your support!
This project has represented a lot of efforts for me and it means a lot!
Right now the website seems to be pretty slow/down. There is a lot of traffic, which was not expected. I also suspect there might be a DoS attack going on.
I will try to make it work better when I get home! (I am currently at work so I cannot give much attention to it right now)
Sorry for the inconvenience, but glad you appreciate the project!
This is a cool project; getting bash to run on it seems like a real milestone. I dabbled with my own unix-like OS right out of college in C (no rust in 2010) and it was great fun.
I'm jealous you were able to make time to get this far!
The navbar takes like 33% screen state and can't be removed.
I never understand why people want to make them sticky and steal valuable reading screen space. You can, if you want, always scroll to the top in like 300 ms.
What a cool little project. It's astonishing how far this can boot with less than a third of the syscalls of Linux implemented.
However, my guess is that the ones that are missing are the more complicated ones. The TTY layer, for example, looks rather basic at the moment. Getting this right will probably be a lot of work.
So don't hold your breath for Maestro running your Linux applications in the next 3 years or so (even without taking into account all the thousands of drivers that Linux has)
>It's astonishing how far this can boot with less than a third of the syscalls of Linux implemented.
It's a great project, but I don't find this ratio surprising at all. Any mature platform builds up logic to enable scenarios such that most things don't need most of the system. As the saying goes, no one uses more than 10% of Excel, but it's a different 10% for everyone.
You could implement 30% of Excel functions and probably have an engine which opens 99% of spreadsheets out there.....though if you wanted full doc compatibility you would still have a long journey ahead of you.
> What a cool little project. It's astonishing how far this can boot with less than a third of the syscalls of Linux implemented.
My hobby OS is more or less a FreeBSD compatible kernel for one specific language VM[1]; it looks like I support 61 syscalls out of 424, and it's been a while since I ran across one I missed (sometimes syscalls are only called in some code paths, or when I target a newer kernel, there may be newer syscalls)
There are a lot of syscalls, and some of them are pretty esoteric; eventually a fully openended replacement will get to most of them, but a third is a good start.
[1] I wanted VM on metal and/or boot to VM, and it became apparent that this is the least effort way to get there, other than probably just having init=/path/to/the/vm; but that doesn't get me what I really want (hardware drivers and tcp stack in the VM language).
Is there maybe a subset of Linux applications that it could run soon? A proxy, nfs, some database server, http server, firewall?
I think it doesn't need to run Steam, libreoffice and Firefox to be useful. Many parts in a common server or microservices architecture are relatively simple in what they do and would probably benefit a lot from a safe, simple kernel.
Sounds like a fun project. Curious though: most of the drawbacks to using C and difficulties with developing an OS are around debugging.
I assume that the switch to Rust eliminated a certain class of memory error but is debugging still a pain? Or is there less of it than before the switch making debugging more tolerable?
A lot of memory and concurrency issues have been eliminated.
It is still a pain to debug, but a lot less than it was before though.
As an example, there is not a lot of chances you forget to use a mutex since the compiler would remind it to you by an error.
This is not a silver bullet though, things such as deadlocks are still present. Especially with interruptions.
To give an example, if you decide to lock a mutex, then an interruption happens, the code that locks the mutex will stop running until the interruption is over. If the interruption itself tries to lock the same mutex, then you have a deadlock, and the typing system cannot help you with this kind of problem.
The solution is to disable interruptions handling while the mutex is locked, but the compiler cannot enforce it.
Tangent, but I love this Gource thing that the author made the contribution video with. I'd never seen it before but had an idea to try making something like it a couple of years back - no original ideas it seems!
I really like the idea of building a kernel, especiallly for learning purposes. Curious about the resources you used to understand the whole kernel/OS thing
I’m surprised to see no one mentioned Theseus OS. It’s a WIP OS built from scratch with Rust, using Rust compiler’s guarantees around memory safety for automatically providing many things which require careful plumbing in conventional OSes. As an example it’s SPL, SAS (single privilege level, single address space) and yet is secure. All parts of the OS are hot pluggable too. The drawback is that all native code needs to be Rust, though a WASM runtime is implemented last year. Related thread from 3 years ago:
https://news.ycombinator.com/item?id=25741729
Wow, i didn't think someone would be crazy enough to take on the ft_penguin project (is it still its name?). Really good job to be honest, when i was still at school (7 years ago now) i thought this was probably one of the project with the worse reward compared to the time needed.
Did you start the project with a friend at first (before the rust rewrite)? Did you work on other projects at the same time?
Looking up at your code, remembering how fun it was, i now kinda want to stop working in devops and start doing embedded or any low-level work like i intended to at first.
"Since the OS started as a school project, I had to cut corners in order to finish it on time. But now is the time to pay back the technical debt I accumulated."
This is obviously impressive. Did you think from the beginning monolithic/module-based like linux was the way to go or did you consider making it a hybrid/micro kernel.
Compatible means "syscall compatible" (I get that from the article). I wonder if it also means kernel module compatible (I dont think so, as the API touch point surface is much larger), but if it strives to be that'd be great (use all hardware that works on Linux).
A memory safe linux kernel would be a fairly incredible thing. If you could snap your fingers and have it, the wins would be huge.
Consider that right now a docker container can't be relied upon to contain arbitrary malware, exactly because the Linux kernel has so many security issues and they're exposed to containers. The reason why a VM like Firecracker is so much safer is that it removes the kernel as the primary security boundary.
Imagine if containers were actually vm-level safe? The performance and operational simplicity of a container with the security of a VM.
I'm not saying this is practical, at this point the C version of Linux is here to stay for quite a while and I think, if anything, Fuschia is the most likely successor (and is unlikely to give us the memory safety that a Rust kernel would). But damn, if Linux had been built with safety in mind security would be a lot simpler. Being able to trust the kernel would be so nice.
edit: OK OK. Yeesh. I meant this to be a hypothetical, I got annoyed at so many of the replies, and this has spiraled. I'm signing off.
I apologize if I was rude! Not a fun start to the morning.
Memory safety isn’t why containers are considered insufficient as a security boundary. It’s exposing essentially the entire Linux feature surface, and the ability to easily interact with the host/other containers that makes them unsafe by themselves. What you’re saying about VMs vs containers makes no sense to me. VMs are used to sandbox containers. You still need to sandbox containers if your kernel is written in rust
Even just considering Linux security itself: there are so, so many ways OS security can break besides a slight (you’re going to have to use unsafe a whole lot) increase in memory safety
If you find this amazing, perhaps you should take a look at seL4, which has formal proofs of correctness, going all the way down to the generated assembly code still satisfying the requirements.
It also has a much better overall architecture, the best currently available: A third generation microkernel multiserver system.
It provides a protected (with proof of isolation) RTOS with hard realtime, proof of worst case timing as well as mixed criticality support. No other system can currently make such claims.
Container vulnerabilities are rarely related to memory bugs. Most vulnerabilities in container deployments are due to logical bugs, misconfiguration, etc. C-level memory stuff is absolutely NOT the reason why virtualization is safer, and not something Rust would greatly
improve. On the opposite end of the spectrum, you have hardware vulnerabilities that Rust also wouldn't help you with.
Rust is a good language and I like using it, but there's a lot of magical thinking around the word "safe". Rust's definition of what "safe" means is fairly narrow, and while the things it fixes are big wins, the majority of CVEs I've seen in my career are not things that Rust would have prevented.
> if Linux had been built with safety in mind security would be a lot simpler
I'm replying simply because you're getting defensive with your edits, but you're missing a few important points, IMO.
First of all, the comment I quoted falls straight into the category of if only we knew back then what we know now.
What does it even mean "built with safety in mind" for a project like Linux?
No one could predict that Linux (which was born as a kernel) would run on billions of devices that people keep in their pockets and constantly use for everything, from booking a table at the restaurant to checking the weather, from chatting with other people to accessing their bank accounts. And that said banks would use it too.
Literally no one.
Computers were barely connected back then, internet wasn't even a thing outside of research centers and universities.
So, what kind of safety should he have planned for?
And to safeguard what from what and who from who?
Secondly, Linux was born as a collaborative effort to write something already old: a monolithic Unix like kernel, nothing fancy, nothing new, nothing experimental, just plain old established stuff for Linus to learn how that kernel thing worked.
The most important thing about it was to be a collaborative effort so he used a language that he and many others already knew.
Did Linus use something more suited for stronger safety guarantees, such as Ada (someone else already mentioned it), Linux wouldn't be the huge success it is now and we would not be having this conversation.
Lastly, the strongest Linux safety guarantee is IMO the GPL license, that conveniently all these Rust rewrites are turning into more permissive licenses. Which steers away from what Linux was, and still largely is, a community effort based on the work of thousands of volunteers.
> Imagine if containers were actually vm-level safe? The performance and operational simplicity of a container with the security of a VM.
As far as I know, the order of magnitudes of container security flaws from memory safety is the same as security flaws coming from namespace logic issues, and you'll have to top that with hardware issues. I'm sorry but rust or not, there will never be a world where you can 100% trust running a malware.
> Fuschia [...] is unlikely to give us the memory safety that a Rust kernel would
Well being micro kernel make it easier to migrate bits by bits, and not care about ABI
Rust is not a magic bullet, it just reduces the attack surface by isolating the unsafe parts. Another way to reduce the attack surface would be to use a microkernel architecture, it has a cost though.
> Consider that right now a docker container can't be relied upon to contain arbitrary malware, exactly because the Linux kernel has so many security issues and they're exposed to containers
If you don't run docker as root, it's fairly ok for normal software. Kernel memory safety is not the main issue with container escapes. Even with memory safety, you can have logical bugs that result in privilege escalation scenarios. Is docker itself in Rust?
Memory safety is not a magic bullet, the Linux kernel isn't exactly trivial to exploit either these days, although still not as hardened as windows (if you don't consider stuff like win32k.sys font parsing kernel space since NT is hybrid after all) in my humble opinion.
> Linux had been built with safety in mind security would be a lot simpler
I think it was, given the resources available in 1993. But if Trovalds caved in and allowed a mini-kernel or NT like hybrid design instead if hard-core monolithic unix, it would have been a game changer. In 1995, Ada was well accepted mainstream, it was memory safe and even Rust devs learned a lot from it. It just wasn't fun to use for the devs (on purpose, so devs were forced to do tedious stuff to prevent even non-memory bugs). But since it is developed by volunteers, they used what attracts the most volunteers.
The main benefit of Rust is not it's safety but its popularity. Ada has been running on missiles, missile defense, subways, aircraft, etc... for a long time and it even has a formally verified subset (SPARK).
In my opinion, even today Ada is a better suit technically for the kernel than Rust because it is time tested and version stable and it would open up the possibility easily formal-verifying parts of the kernel.
Given how widely used Linux is, it would require a massive backing fund to pay devs to write something not so fun like Ada though.
Containers became popular because it doesn't make much sense to be running full blown virtual machines just to run simple single process services.
You can lock down the allowed kernel syscalls with seccomp and go further with confining the processes with apparmor. Docker has good enough defaults for these 2 security approaches.
Full fat VMs are not immune to malware infection (the impact still applies to the permitted attack surface). Might not be able to easily escape to host but the risk is still there.
> Consider that right now a docker container can't be relied upon to contain arbitrary malware, exactly because the Linux kernel has so many security issues and they're exposed to containers.
No, Docker container was never meant for that. Never use containers with untrustable binary. There is Vagrant and others for that.
"gVisor is an application kernel for containers. It limits the host kernel surface accessible to the application while still giving the application access to all the features it expects. Unlike most kernels, gVisor does not assume or require a fixed set of physical resources; instead, it leverages existing host kernel functionality and runs as a normal process. In other words, gVisor implements Linux by way of Linux."
I like Rust and work in it fulltime, and like its memory-safety aspects but I think it's a bit of a stretch to be able to claim memory safety guarantees of any kind when we're talking about low-level code like a kernel.
Because in reality, the kernel will have to do all sorts of "unsafe" things even just to provide for basic memory management services for itself and applications, or for interacting with hardware.
You can confine these bits to verified and well-tested parts of the code, but they're still there. And because we're human beings, they will inevitably have bugs that get exploited.
TLDR being written in Rust is an improvement but no guarantee of lack of memory safety issues. It's all how you hold the tool.
I didn't know Firecracker existed, that's really awesome. Looks to be in Rust as well. I'll have to look at how this differs from the approach that Docker uses, my understanding is that Docker uses cgroups and some other built-in Linux features.
[+] [-] llenotre|2 years ago|reply
Right now the website seems to be pretty slow/down. There is a lot of traffic, which was not expected. I also suspect there might be a DoS attack going on.
I will try to make it work better when I get home! (I am currently at work so I cannot give much attention to it right now)
Sorry for the inconvenience, but glad you appreciate the project!
[+] [-] dash2|2 years ago|reply
[+] [-] joshmarlow|2 years ago|reply
I'm jealous you were able to make time to get this far!
[+] [-] TheHiddenSun|2 years ago|reply
The navbar takes like 33% screen state and can't be removed.
I never understand why people want to make them sticky and steal valuable reading screen space. You can, if you want, always scroll to the top in like 300 ms.
[+] [-] dark-star|2 years ago|reply
However, my guess is that the ones that are missing are the more complicated ones. The TTY layer, for example, looks rather basic at the moment. Getting this right will probably be a lot of work.
So don't hold your breath for Maestro running your Linux applications in the next 3 years or so (even without taking into account all the thousands of drivers that Linux has)
[+] [-] Arainach|2 years ago|reply
It's a great project, but I don't find this ratio surprising at all. Any mature platform builds up logic to enable scenarios such that most things don't need most of the system. As the saying goes, no one uses more than 10% of Excel, but it's a different 10% for everyone.
You could implement 30% of Excel functions and probably have an engine which opens 99% of spreadsheets out there.....though if you wanted full doc compatibility you would still have a long journey ahead of you.
[+] [-] toast0|2 years ago|reply
My hobby OS is more or less a FreeBSD compatible kernel for one specific language VM[1]; it looks like I support 61 syscalls out of 424, and it's been a while since I ran across one I missed (sometimes syscalls are only called in some code paths, or when I target a newer kernel, there may be newer syscalls)
There are a lot of syscalls, and some of them are pretty esoteric; eventually a fully openended replacement will get to most of them, but a third is a good start.
[1] I wanted VM on metal and/or boot to VM, and it became apparent that this is the least effort way to get there, other than probably just having init=/path/to/the/vm; but that doesn't get me what I really want (hardware drivers and tcp stack in the VM language).
[+] [-] berkes|2 years ago|reply
I think it doesn't need to run Steam, libreoffice and Firefox to be useful. Many parts in a common server or microservices architecture are relatively simple in what they do and would probably benefit a lot from a safe, simple kernel.
[+] [-] gardaani|2 years ago|reply
[1] https://news.ycombinator.com/item?id=28986229
[+] [-] MuffinFlavored|2 years ago|reply
[+] [-] pizza234|2 years ago|reply
[+] [-] agentultra|2 years ago|reply
I assume that the switch to Rust eliminated a certain class of memory error but is debugging still a pain? Or is there less of it than before the switch making debugging more tolerable?
[+] [-] llenotre|2 years ago|reply
As an example, there is not a lot of chances you forget to use a mutex since the compiler would remind it to you by an error.
This is not a silver bullet though, things such as deadlocks are still present. Especially with interruptions.
To give an example, if you decide to lock a mutex, then an interruption happens, the code that locks the mutex will stop running until the interruption is over. If the interruption itself tries to lock the same mutex, then you have a deadlock, and the typing system cannot help you with this kind of problem.
The solution is to disable interruptions handling while the mutex is locked, but the compiler cannot enforce it.
[+] [-] lucasyvas|2 years ago|reply
[+] [-] llenotre|2 years ago|reply
[+] [-] willangelo|2 years ago|reply
[+] [-] llenotre|2 years ago|reply
[+] [-] Aerbil313|2 years ago|reply
[+] [-] orwin|2 years ago|reply
Did you start the project with a friend at first (before the rust rewrite)? Did you work on other projects at the same time?
Looking up at your code, remembering how fun it was, i now kinda want to stop working in devops and start doing embedded or any low-level work like i intended to at first.
[+] [-] internet101010|2 years ago|reply
Please try to keep this mindset.
[+] [-] dottedmag|2 years ago|reply
[+] [-] _flux|2 years ago|reply
For running on bare iron.. I suppose there's no short-term solution for that.
[+] [-] potato24|2 years ago|reply
[+] [-] omeid2|2 years ago|reply
But motor-os is literally the repository name. Sometimes?
[+] [-] Zambyte|2 years ago|reply
This does not bode well for computers.
[+] [-] cies|2 years ago|reply
[+] [-] mgoetzke|2 years ago|reply
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] insanitybit|2 years ago|reply
Consider that right now a docker container can't be relied upon to contain arbitrary malware, exactly because the Linux kernel has so many security issues and they're exposed to containers. The reason why a VM like Firecracker is so much safer is that it removes the kernel as the primary security boundary.
Imagine if containers were actually vm-level safe? The performance and operational simplicity of a container with the security of a VM.
I'm not saying this is practical, at this point the C version of Linux is here to stay for quite a while and I think, if anything, Fuschia is the most likely successor (and is unlikely to give us the memory safety that a Rust kernel would). But damn, if Linux had been built with safety in mind security would be a lot simpler. Being able to trust the kernel would be so nice.
edit: OK OK. Yeesh. I meant this to be a hypothetical, I got annoyed at so many of the replies, and this has spiraled. I'm signing off.
I apologize if I was rude! Not a fun start to the morning.
[+] [-] opportune|2 years ago|reply
Even just considering Linux security itself: there are so, so many ways OS security can break besides a slight (you’re going to have to use unsafe a whole lot) increase in memory safety
[+] [-] snvzz|2 years ago|reply
It also has a much better overall architecture, the best currently available: A third generation microkernel multiserver system.
It provides a protected (with proof of isolation) RTOS with hard realtime, proof of worst case timing as well as mixed criticality support. No other system can currently make such claims.
[+] [-] t8sr|2 years ago|reply
Rust is a good language and I like using it, but there's a lot of magical thinking around the word "safe". Rust's definition of what "safe" means is fairly narrow, and while the things it fixes are big wins, the majority of CVEs I've seen in my career are not things that Rust would have prevented.
[+] [-] peoplefromibiza|2 years ago|reply
I'm replying simply because you're getting defensive with your edits, but you're missing a few important points, IMO.
First of all, the comment I quoted falls straight into the category of if only we knew back then what we know now.
What does it even mean "built with safety in mind" for a project like Linux?
No one could predict that Linux (which was born as a kernel) would run on billions of devices that people keep in their pockets and constantly use for everything, from booking a table at the restaurant to checking the weather, from chatting with other people to accessing their bank accounts. And that said banks would use it too.
Literally no one.
Computers were barely connected back then, internet wasn't even a thing outside of research centers and universities.
So, what kind of safety should he have planned for?
And to safeguard what from what and who from who?
Secondly, Linux was born as a collaborative effort to write something already old: a monolithic Unix like kernel, nothing fancy, nothing new, nothing experimental, just plain old established stuff for Linus to learn how that kernel thing worked.
The most important thing about it was to be a collaborative effort so he used a language that he and many others already knew.
Did Linus use something more suited for stronger safety guarantees, such as Ada (someone else already mentioned it), Linux wouldn't be the huge success it is now and we would not be having this conversation.
Lastly, the strongest Linux safety guarantee is IMO the GPL license, that conveniently all these Rust rewrites are turning into more permissive licenses. Which steers away from what Linux was, and still largely is, a community effort based on the work of thousands of volunteers.
[+] [-] K0nserv|2 years ago|reply
> But damn, if Linux had been built with safety in mind security would be a lot simpler. Being able to trust the kernel would be so nice.
For its time, it was built with safety in mind, we can't hold it to a standard that wasn't prevalent until ~20 years later
[+] [-] phh|2 years ago|reply
As far as I know, the order of magnitudes of container security flaws from memory safety is the same as security flaws coming from namespace logic issues, and you'll have to top that with hardware issues. I'm sorry but rust or not, there will never be a world where you can 100% trust running a malware.
> Fuschia [...] is unlikely to give us the memory safety that a Rust kernel would
Well being micro kernel make it easier to migrate bits by bits, and not care about ABI
[+] [-] GuB-42|2 years ago|reply
Rust is not a magic bullet, it just reduces the attack surface by isolating the unsafe parts. Another way to reduce the attack surface would be to use a microkernel architecture, it has a cost though.
[+] [-] badrabbit|2 years ago|reply
If you don't run docker as root, it's fairly ok for normal software. Kernel memory safety is not the main issue with container escapes. Even with memory safety, you can have logical bugs that result in privilege escalation scenarios. Is docker itself in Rust?
Memory safety is not a magic bullet, the Linux kernel isn't exactly trivial to exploit either these days, although still not as hardened as windows (if you don't consider stuff like win32k.sys font parsing kernel space since NT is hybrid after all) in my humble opinion.
> Linux had been built with safety in mind security would be a lot simpler
I think it was, given the resources available in 1993. But if Trovalds caved in and allowed a mini-kernel or NT like hybrid design instead if hard-core monolithic unix, it would have been a game changer. In 1995, Ada was well accepted mainstream, it was memory safe and even Rust devs learned a lot from it. It just wasn't fun to use for the devs (on purpose, so devs were forced to do tedious stuff to prevent even non-memory bugs). But since it is developed by volunteers, they used what attracts the most volunteers.
The main benefit of Rust is not it's safety but its popularity. Ada has been running on missiles, missile defense, subways, aircraft, etc... for a long time and it even has a formally verified subset (SPARK).
In my opinion, even today Ada is a better suit technically for the kernel than Rust because it is time tested and version stable and it would open up the possibility easily formal-verifying parts of the kernel.
Given how widely used Linux is, it would require a massive backing fund to pay devs to write something not so fun like Ada though.
[+] [-] Timber-6539|2 years ago|reply
You can lock down the allowed kernel syscalls with seccomp and go further with confining the processes with apparmor. Docker has good enough defaults for these 2 security approaches.
Full fat VMs are not immune to malware infection (the impact still applies to the permitted attack surface). Might not be able to easily escape to host but the risk is still there.
[+] [-] Alifatisk|2 years ago|reply
No, Docker container was never meant for that. Never use containers with untrustable binary. There is Vagrant and others for that.
[+] [-] mikepurvis|2 years ago|reply
"gVisor is an application kernel for containers. It limits the host kernel surface accessible to the application while still giving the application access to all the features it expects. Unlike most kernels, gVisor does not assume or require a fixed set of physical resources; instead, it leverages existing host kernel functionality and runs as a normal process. In other words, gVisor implements Linux by way of Linux."
https://github.com/google/gvisor
[+] [-] cmrdporcupine|2 years ago|reply
Because in reality, the kernel will have to do all sorts of "unsafe" things even just to provide for basic memory management services for itself and applications, or for interacting with hardware.
You can confine these bits to verified and well-tested parts of the code, but they're still there. And because we're human beings, they will inevitably have bugs that get exploited.
TLDR being written in Rust is an improvement but no guarantee of lack of memory safety issues. It's all how you hold the tool.
[+] [-] jmakov|2 years ago|reply
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] giancarlostoro|2 years ago|reply
[+] [-] kossTKR|2 years ago|reply
I'm honestly interested to know, because it sounds like a huge deal here, but in my laymans ears very cool and sci fi!
[+] [-] maayank|2 years ago|reply
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] scoot|2 years ago|reply
"to not contain"?
Edit to contain (ahem!) the downvotes: I was genuinely confused by the ambiguous use of "contain", but comments below cleared that up.
[+] [-] unknown|2 years ago|reply
[deleted]