<<There are a class of "ideal attractors" in engineering, concepts like "everything is an object," "homoiconicity," "purely functional," "pure capability system," etc. Engineers fall into orbit around these ideas quite easily. Systems that follow these principles often get useful properties out of the deal.
However, going too far in any of these directions is also a great way to find a deep reservoir of unsolved problems, which is part of why these are popular directions in academia.
In the interest of shipping, we are consciously steering around unsolved problems, even when it means we lose some attractive features.>>
I wonder if the attributes of Hubris and similar systems -- real-time, lack of dynamism -- will become "ideal attractors" for developers not working in problem domains where these things are absolutely required, especially as the backlash against the complexity at higher layers of the software stack continues to grow. In other words, I wonder if a sizable number of developers will convince themselves that they need an embedded RTOS like Hubris for a project where Linux running on an off-the-shelf SBC (or even PC) would work well enough.
I have an embedded real-time control project that is currently written in Rust, but runs with RTIC (https://rtic.rs/), a framework which is conceptually similar (no dynamic allocation of tasks or resources) but also has some differences. RTIC is more of a framework for locks and critical sections in an interrupt based program than a full fledged RTOS. Looking through the docs, here's the main differences (for my purposes) I see:
1. In Hubris, all interrupt handlers dispatch to a software task. In RTIC, you can dispatch to a software task, but you can also run the code directly in the interrupt handler. RTIC is reliant on Cortex-M's NVIC for preemption, whereas Hubris can preempt in software (assuming it is implemented). This does increase the minimum effective interrupt latency in Hubris, and if not very carefully implemented, the jitter also.
2. Hubris compiles each task separately and then pastes the binaries together, presumably with a fancy linker script. RTIC can have everything in one source file and builds everything into one LTO'd blob. I see the Hubris method as mostly a downside (unless you want to integrate binary blobs, for example), but it might have been needed for:
3. Hubris supports Cortex-M memory protection regions. This is pretty neat and something that is mostly out of scope for RTIC (being built around primitives that allow shared memory, trying to map into the very limited number of MPU regions would be difficult at best). Of course, it's Rust, so in theory you wouldn't need the MPU protections, but if you have to run any sort of untrusted code this is definitely the winner.
Hubris does support shared memory via leases, but I'm not sure how it manages to map them into the very limited 8 Cortex-M MPU regions. I'm quite interested to look at the implementation when the source code is released.
Edit: I forgot to mention the biggest difference, which is that because tasks have separate stacks in Hubris, you can do blocking waits. RTIC may support async in the future but for now you must manually construct state machines.
> Hubris does support shared memory via leases, but I'm not sure how it manages to map them into the very limited 8 Cortex-M MPU regions.
What I did in a similar kernel was dynamically map them from a larger table on faults, sort of like you would with a soft fill TLB. When you turn off the MPU in supervisor mode you get a sane 'map everything' mapping, leaving all 8 entries to user code.
The way LDM/STM restart after faults is amenable to this model on the M series cores.
I don't think they link the binaries. It's more like, put them each on executable flash in separate places and the kernel just calls them.
The intent here seems to be that each binary has no need (and no ability) to get all up in another binary's business. Nothing shared, except access to the RTOS.
> instead of having an operating system that knows how to dynamically create tasks at run-time (itself a hallmark of multiprogrammed, general purpose systems), Cliff had designed Hubris to fully specify the tasks for a particular application at build time, with the build system then combining the kernel with the selected tasks to yield a single (attestable!) image.
I worked briefly at John Deere, and their home-grown operating system (called "JDOS", written in C) also baked every application into the system at compile time. This was my only embedded experience, but I assumed this was somewhat common for embedded operating systems?
I'd like to hear more about Oxide's development process. Was this designed on an index card, and then implemented? Or was it done with piles and piles of diagrams and documents before the first code was committed? Was it treated as a cool, out-there idea that's worth exploring, and then it gradually looked better and better?
It's hard to get software organizations to do ambitious things like this, and it's impressive that this was done on a relatively short timescale. I think the industry could learn a lot from how this was managed.
So, the Hubris repo itself will show a bunch of that history, but in particular, Cliff used the "sketch" nomenclature for the earliest ideas. I think in those first days, our thinking was that we were hitting a bunch of headwind on other approaches -- and had an increasingly concrete idea of what our own alternative might look like. I think whenever doing something new and bold, you want to give yourself some amount of time to at least get to an indicator that the new path is promising. For Hubris, this period was remarkably brief (small number of weeks), which is a tribute to the tremendous focus that Cliff had, but also how long some of the ideas had been germinating. Cliff also made the decision to do the earliest work on the STM32F407 Discovery; we knew that this wouldn't be the ultimate hardware that we would use for anything, but it is (or was!) readily attainable and just about everything about that board is known.
To summarize, it didn't take long to get to the point where Hubris was clearly the direction -- and a working artifact was really essential for that.
Intersting choices of names, Hubris and Humility. Combined with the style of the page, it gives to me a solemn and heavy feeling. Especially compared to most projects presented that tend to be very "positive energy and emojis". Their website is also beautiful https://oxide.computer/. Though I wonder who's the target for this. Is this for cloud provider themselves, for people that self host, for hosters? For everyone?
The target market is users that want to build their own cloud infrastructure, but don't have the scale required to go directly to ODM's to have their own custom designs manufactured.
When Cantrill and his team works on something, HN listens, and for good reason. Startups like Oxide show that there's room for a lot of innovation still on a smaller scale, even within fields like HW.
Yeah also noticed and got a little bit upset about this.. I mean publishing a website with broken links does not seems very smart nor makes very much sense to me..
Has Oxide released any information on the price range of one of their machines? I assume if they're targeting mid-size enterprises it would be outside what I would consider buying for hobby use, but it would be sweet in the future if there was a mini-Oxide suitable for home labs.
> no C code in the system. This removes, by construction, a lot of the attack surface normally present in similar systems.
Not to be too pedantic here, but it's important to note that the absence of C code, while arguably a benefit overall, doesn't by itself guarantee anything with regards to safety/security...I suppose there's going to necessarily be at least some "unsafe" Rust and/or raw assembly instructions sprinkled throughout, but I can't yet see that myself (as of the time of writing this comment, the GitHub links are responding with 404). Nonetheless, it's always refreshing to see some good documentation and source code being provided for these kinds of things. Many companies in this space, even these days, sadly continue to live by some outdated values of hiding behind "security through obscurity", which is somehow championed (though using different words) as a benefit even to their own customers, so it's refreshing that others (Oxide among them) are really starting to take a different approach and making their software/firmware publicly available for inspection by anyone inclined to do so.
To be clear, that sentence refers to the sum total of the things in the previous sentence, not just the bit about C. And it's "a lot of the attack surface" and not "guarantee" for a reason. We don't believe that simply being written in Rust makes things automatically safe or secure.
There is some unsafe Rust and some inline assembly, yes. I imagine a lot less than folks may think.
This needs citation, but what does the "that" even refer to? I'm genuinely curious because there's little on it; did you use it? Did it survive? And what particular aspect of Hubris and Humility reminds you of this system?
As someone who's only worked with a prepared hardware kit (a dsPIC33F on an Explorer 16 that came with cables and the debugging puck), if I want to pick up the board they recommend in the blog post, do I need to make sure I get any other peripherals?
This all seems very cool, and I badly want to poke at embedded stuff again, but I have whatever the opposite of a green thumb is for hardware. Advice would be appreciated ^_^
For anyone still reading this a week on, I got the recommended board in, and I needed a micro-USB cable -- preferably two, since it has two sockets (one for power and one for, I assume, programming). Luckily, I already have a few of these hanging around for charging my wireless headphones.
It is completely beyond me, however, why one of the sockets is mounted upside down relative to the other one.
There's an open PR with the details, I set it to deploy every push to that PR for now so we could make quick fixes. It just runs asciidoctor, nothing fancy.
[+] [-] JulianMorrison|4 years ago|reply
<<There are a class of "ideal attractors" in engineering, concepts like "everything is an object," "homoiconicity," "purely functional," "pure capability system," etc. Engineers fall into orbit around these ideas quite easily. Systems that follow these principles often get useful properties out of the deal.
However, going too far in any of these directions is also a great way to find a deep reservoir of unsolved problems, which is part of why these are popular directions in academia.
In the interest of shipping, we are consciously steering around unsolved problems, even when it means we lose some attractive features.>>
[+] [-] mwcampbell|4 years ago|reply
[+] [-] SideburnsOfDoom|4 years ago|reply
Huh, this is more like pragmatism than "Hubris".
[+] [-] panick21_|4 years ago|reply
- Hubris for deep embedded
- Redox OS for Desktop/Server (https://www.redox-os.org/)
- Tock for embedded (https://www.tockos.org/)
- Xous for trusted devices (https://xobs.io/announcing-xous-the-betrusted-operating-syst...)
I assume there are more.
[+] [-] TD-Linux|4 years ago|reply
1. In Hubris, all interrupt handlers dispatch to a software task. In RTIC, you can dispatch to a software task, but you can also run the code directly in the interrupt handler. RTIC is reliant on Cortex-M's NVIC for preemption, whereas Hubris can preempt in software (assuming it is implemented). This does increase the minimum effective interrupt latency in Hubris, and if not very carefully implemented, the jitter also.
2. Hubris compiles each task separately and then pastes the binaries together, presumably with a fancy linker script. RTIC can have everything in one source file and builds everything into one LTO'd blob. I see the Hubris method as mostly a downside (unless you want to integrate binary blobs, for example), but it might have been needed for:
3. Hubris supports Cortex-M memory protection regions. This is pretty neat and something that is mostly out of scope for RTIC (being built around primitives that allow shared memory, trying to map into the very limited number of MPU regions would be difficult at best). Of course, it's Rust, so in theory you wouldn't need the MPU protections, but if you have to run any sort of untrusted code this is definitely the winner.
Hubris does support shared memory via leases, but I'm not sure how it manages to map them into the very limited 8 Cortex-M MPU regions. I'm quite interested to look at the implementation when the source code is released.
Edit: I forgot to mention the biggest difference, which is that because tasks have separate stacks in Hubris, you can do blocking waits. RTIC may support async in the future but for now you must manually construct state machines.
[+] [-] monocasa|4 years ago|reply
What I did in a similar kernel was dynamically map them from a larger table on faults, sort of like you would with a soft fill TLB. When you turn off the MPU in supervisor mode you get a sane 'map everything' mapping, leaving all 8 entries to user code.
The way LDM/STM restart after faults is amenable to this model on the M series cores.
[+] [-] JulianMorrison|4 years ago|reply
The intent here seems to be that each binary has no need (and no ability) to get all up in another binary's business. Nothing shared, except access to the RTOS.
[+] [-] wyldfire|4 years ago|reply
[1] https://rtic.rs/
[+] [-] panick21_|4 years ago|reply
[+] [-] throwaway894345|4 years ago|reply
I worked briefly at John Deere, and their home-grown operating system (called "JDOS", written in C) also baked every application into the system at compile time. This was my only embedded experience, but I assumed this was somewhat common for embedded operating systems?
[+] [-] stusmall|4 years ago|reply
[+] [-] 7thaccount|4 years ago|reply
[+] [-] tambourine_man|4 years ago|reply
That is some great naming
[+] [-] solmag|4 years ago|reply
https://www.youtube.com/watch?v=XbBzSSvT_P0
https://www.youtube.com/watch?v=cuvp-e4ztC0
[+] [-] cpach|4 years ago|reply
[+] [-] spenczar5|4 years ago|reply
It's hard to get software organizations to do ambitious things like this, and it's impressive that this was done on a relatively short timescale. I think the industry could learn a lot from how this was managed.
[+] [-] bcantrill|4 years ago|reply
To summarize, it didn't take long to get to the point where Hubris was clearly the direction -- and a working artifact was really essential for that.
[+] [-] mbag|4 years ago|reply
But someone from Oxide would need to tell you exactly how many RFDs took to desing and implement Hubris.
[1] https://oxide.computer/blog/rfd-1-requests-for-discussion
[+] [-] Zababa|4 years ago|reply
[+] [-] thesuperbigfrog|4 years ago|reply
The OS is named Hubris. Building a new Operating System does take a lot of confidence.
The debugger is named Humility. It can be humbling to know your program is not working correctly and use a tool to discover how it is broken.
Impatience would be a great name for the task scheduler. (Because you want your task to run NOW!)
Laziness would be a great name for a hardware-based watchdog timer. (Because you keep on putting it off / resetting it until later.)
Compare: https://www.threevirtues.com/
[+] [-] pjmlp|4 years ago|reply
[+] [-] barrenko|4 years ago|reply
[+] [-] noisy_boy|4 years ago|reply
[+] [-] jabl|4 years ago|reply
[+] [-] omnicognate|4 years ago|reply
But horribly broken for me (mobile firefox), with text cut off at borders and overlaid by images.
[+] [-] sgt|4 years ago|reply
[+] [-] emptyparadise|4 years ago|reply
[+] [-] Tuna-Fish|4 years ago|reply
[+] [-] detaro|4 years ago|reply
[+] [-] HeavenSmile|4 years ago|reply
[+] [-] pjmlp|4 years ago|reply
[+] [-] scudd|4 years ago|reply
[+] [-] floatboth|4 years ago|reply
Since they aim to open source everything, there probably will be a way to use their management plane and stuff with a homelab eventually :)
[+] [-] sbarre|4 years ago|reply
As much as most of this is way over my head, I'm always fascinated to read about new ground-up work like this.
[+] [-] ls65536|4 years ago|reply
Not to be too pedantic here, but it's important to note that the absence of C code, while arguably a benefit overall, doesn't by itself guarantee anything with regards to safety/security...I suppose there's going to necessarily be at least some "unsafe" Rust and/or raw assembly instructions sprinkled throughout, but I can't yet see that myself (as of the time of writing this comment, the GitHub links are responding with 404). Nonetheless, it's always refreshing to see some good documentation and source code being provided for these kinds of things. Many companies in this space, even these days, sadly continue to live by some outdated values of hiding behind "security through obscurity", which is somehow championed (though using different words) as a benefit even to their own customers, so it's refreshing that others (Oxide among them) are really starting to take a different approach and making their software/firmware publicly available for inspection by anyone inclined to do so.
[+] [-] steveklabnik|4 years ago|reply
There is some unsafe Rust and some inline assembly, yes. I imagine a lot less than folks may think.
[+] [-] pjmlp|4 years ago|reply
[+] [-] Animats|4 years ago|reply
[+] [-] bcantrill|4 years ago|reply
[+] [-] Twisol|4 years ago|reply
This all seems very cool, and I badly want to poke at embedded stuff again, but I have whatever the opposite of a green thumb is for hardware. Advice would be appreciated ^_^
[+] [-] jeffrallen|4 years ago|reply
1: https://www.st.com/en/evaluation-tools/nucleo-h753zi.html
[+] [-] Twisol|4 years ago|reply
It is completely beyond me, however, why one of the sockets is mounted upside down relative to the other one.
[+] [-] jbott|4 years ago|reply
[+] [-] steveklabnik|4 years ago|reply
https://github.com/oxidecomputer/hubris/pull/272
(specifically https://github.com/oxidecomputer/hubris/pull/272/files#diff-... )
[+] [-] ABS|4 years ago|reply
<meta name="generator" content="Asciidoctor 2.0.16">
so I guess https://asciidoctor.org/
[+] [-] bo0tzz|4 years ago|reply
[+] [-] rossmohax|4 years ago|reply
[+] [-] steveklabnik|4 years ago|reply
[+] [-] cute_boi|4 years ago|reply
https://hubris.oxide.computer/reference
Looks amazing imo. Waiting for github code :D
[+] [-] moondev|4 years ago|reply