This looks interesting (although I'm not in the target market, too small)...
But if I were looking at this, judging from the quality of people they've amassed in their engineering team, is there any chance they won't be acquired in 6 months?
To anyone looking to take a bet on this, what is the answer to "what's your plan for when your stellar team gets acquired?" And what answer will satisfy that buyer?
Update: Adding another question, does this "environment" (where any really great product with great talent in it can be acquired very quickly) have a chilling effect on purchases for products like this?
Hi! So, at every step -- from conception to funding to building the team and now building the product -- we have done so to build a big, successful public company. Not only do we (the founders) share that conviction, but it is shared by our investors and employees as well. For better or for ill, we are -- as we memorably declared to one investor -- ride or die.
Also, if it's of any solace, I really don't think any of the existing players would be terribly interested in buying a company that has so thoroughly and unequivocally rejected so many of their accrued decisions! ;) I'm pretty sure their due diligence would reveal that we have taken a first principles approach here that is anathema to the iterative one they have taken for decades -- and indeed, these companies have shown time and time again that they don't want to risk their existing product lines to a fresh approach, no matter how badly customers want it.
Going by that logic, you should never take a chance on a bad company because they are bad, and a good company because they are too good and might get acquired. So should you just never rely on a small company for anything?
Private companies can't just get bought out. They have to agree to be acquired. There is not some roaming force of Big Corp M&A people who forcefully acquihire companies.
But second, I'd love to understand the compute vs storage tradeoff chosen here. Looking at the (pretty!) picture [1], I was shocked to see "Wow, it's mostly storage?". Is that from going all flash?
Given how much of the rack is storage, I'm not sure which Milan was chosen (and so whether that's 2048 threads or 4196 [edit: real cores, 4196 threads]), but it seems like visually 4U is compute? [edit: nope] Is that a mistake on my part, because dual-socket Milan at 128 threads per socket is 256 threads per server, so you need at least 8 servers to hit 2048 "somethings", or do the storage nodes also
have Milans [would make sense] and their compute is included [also fine!] -- and so similarly that's how you get a funky 30 TiB of memory?
[Top-level edit from below: the green stuff are the nodes, including the compute. The 4U near the middle is the fiber]
P.S.: the "NETWORK SPEED 100 GB/S" in all caps / CSS loses the presumably 100 Gbps (though the value in the HTML is 100 gb/s which is also unclear).
Power footprint also confirms that the compute density is pretty low.
We built a few racks of Supermicro AMD servers (4 X computes in 2U), and we load tested it to 23kva peak usage (about 1/2 full with nthat type of nodes only, our DC would let us go further)
Were also over 1 PB of disks (unclear how much of this is redundancy), also in NVMe (15.36 TB x 24 in 2U is a lot of storage...)
Other then that not a bad concept, not sure of a premium they will charge or what will be comparable on price.
They basically reinvented mainframes.
Seems it has a lot in common with Z series.
Scalable locked in hardware, virtualization,
reliability, engineered for hardware swaps,
upgrades.
A proprietary operating system (?) from what someone said.
(Offshoot of Solaris +++ (???) By that I mean that most of it, or all of it might be open sourced forks, but it will be an OS only meant to run on their systems.
(It would be fun to get it working at home, on a couple of PCs or a bunch of PIs)
They lack specialized processors to offload some
workloads to.
Perhaps in modern terms shelfs of GPUs or a shelf fast FPGA
, DSP processors. The possibilities are huge.
I didn't find any mention of from what I read.
They also lack the gigantic legacy effort to be
compatible, which is a good thing.
Their approach to reliability isn't quite on par with mainframes, AIUI. At least, not yet. And the programming model is also quite different - a mainframe can seamlessly scale from lots of tiny VM workloads (what Oxide seems to be going for) to large vertically-scaled shared-everything SSI, and anything in between.
It’s all fun stories from people doing amazing things with computer hardware and low level software. Like Ring-Sub-Zero and DRAM driver level software.
> Our firmware is open source. We will be transparent about bug fixes. No longer will you be gaslit by vendors about bugs being fixed but not see results or proof.
There are lots of reasons to be enthusiastic about Oxide but for me, this one takes the cake. I hope they are successful, and I hope this attitude spreads far and wide.
- vendor-locked at the rack - if you have hardware from someone else, it can't live in the same cabinet
I guess if you just want a pretty data center in a box and look like what they consider a 'normal' enterprise to be, it might appeal. But I'm not sure how many people asked for Apple-style hardware in the DC.
Why is it important what kind of virtualization? It works and since it is built for this hardware it will likely be more reliable then anything you're putting together yourself.
The specs are damn good. When it is all top-of-the-line, inflexibility is kind of a mute point. Where else are you going to go?
> But I'm not sure how many people asked for Apple-style hardware in the DC.
Well integrated, performant and reliable hardware that runs VMs where you can put anything on it is pretty much all everyone running their own hardware is looking for.
Honestly I am surprised how many here completely misunderstand what their value proposition is.
As I understand it, Oxide is going to have deep software integration into their hardware. So the expectation isn't that the servers in this rack will be running Windows or a generic Linux distribution. In case anyone from Oxide is here, is my understanding correct? And if so, will there be a way to run a smaller version of an Oxide system, say for testing or development, without purchasing an entire rack at a time?
Anyway, glad to finally get a glimpse of what Oxide has to offer. Looking forward to seeing a lot more.
My understanding is you will use a API to provision virtual machines on top of the Oxide hypervisor/software stack, which is bhyve running on Illumos. So you can still just run your favorite Linux distro or windows or a BSD if you want[1].
Agreed, I would love to hear more about the management plane. I'm glad it's API-driven, but I still have some questions about things like which hypervisor they are using.
If it's a custom software stack, might be nice to get a miniature dev-kit!
In reality I would never want this type of hardware... It reminds me of the old boat anchor bladecenter rigs we used to use. They were great, up until you had to replace one of the blades after the support was up. It's not always practical to replace hardware every 3 years like we're supposed to, so this type of stuff sticks around and gets some barnacles.
What would be fantastic would be if the entire industry committed to an open spec for large chassis like this with a standardized networking and storage overlay... But that would never happen because vendor lock-in is the big money maker in 'enterprise'.
> What would be fantastic would be if the entire industry committed to an open spec for large chassis like this with a standardized networking and storage overlay
Isn't the Open Compute Project supposed to be working on that kind of stuff?
It seems like a lot of Oxide information is currently hiding out in podcasts and other media - does anyone know how the AuthN, AuthZ, ACL system is going to work?
One of the most powerful elements of the trust root system is in audit ability and access control for both service-to-service and human-to-system aspects and I'm really interested in seeing how this plays out.
For example, a service mesh where hosts can be identified securely and authorized in a specific role unlocks a lot of low-friction service-to-service security. I'm curious what Oxide plan to provide in this space API and SDK wise.
I see some Zanzibar related projects on their GitHub, so it can be assumed the ACL system will be based on the principles there - but that's more a framework than an implementation.
The storage is always the difficult part in these architectures. Are you distributing across all nodes? It appears that each sled is an individual compute unit with 10 drives. Are the drives on a proverbial island and only accessible to that local node, or is there some distributed storage going on that you can talk about?
On paper with RDMA and NVMe-OF you could access any drive from any compute unit... but that's easier said than done :)
Soo... we’re switching back to blade servers again?
The problem with this model is its no longer commodity hardware. You are kinda locked into their exosystem of specialized network and server equipment.
And it of course introduces some unique failure modes to mitigate too.
Not to say its not a cool idea, it’s just interesting to see how hardware trends oscillate between commodity and highly specialized proprietary designs.
Congrats Oxide team! More competition in this space is always a good thing.
I'm curious about management. Can the rack operate completely standalone? I assume when you have multiple there will be some management abstraction above the rack layer?
The closest direct equivalent that I can think of to this is AWS outposts. Are there any others that I'm forgetting?
Also thinking if the Website is not finished? All the "Read More" actually hide very little information, if so why hide it? And doesn't seems to explain the company very well. Seems like we need to listen to their PodCast to find out what is going on. ( Edit: Found a Youtube Video about it https://www.youtube.com/watch?v=vvZA9n3e5pc )
>Get the most efficient power rating and network speeds of 100GBps without the pain of cable
100GBps would be impressive, 100Gbps would be ... not much?
A interesting thing is all the terminal like graphics are actually HTML/CSS and not images.
It's interesting that the RAM/CPU ratio is about double the default shapes from AWS/GCP. In practice I have generally seen those shapes run on the low side of CPU utilization for most workloads, so I think the choice makes sense.
I'm curious if ARC will be running with primarycache=metadata to rely on low latency storage and in-VM cache, otherwise I could see ARC using a fair bit of that RAM overhead in the hosts.
Intel is losing on the client and server. everyone is jumping ship to either ARM or AMD for client/server. hopefully Intel new engineer CEO can turn it around like AMD engineer CEO (Lisa Su)
This is a solid private-cloud play aimed at those corporations (probably mainly financials, but other sectors too I'm sure...) who don't want to outsource to the likes of AWS / GCP.
Not just “don’t want to”, I’d hope: They should be able to win on the economics, too, assuming customers that care more about TCO for a fixed or steady-growing workload rather than elasticity.
[+] [-] atonse|4 years ago|reply
But if I were looking at this, judging from the quality of people they've amassed in their engineering team, is there any chance they won't be acquired in 6 months?
To anyone looking to take a bet on this, what is the answer to "what's your plan for when your stellar team gets acquired?" And what answer will satisfy that buyer?
Update: Adding another question, does this "environment" (where any really great product with great talent in it can be acquired very quickly) have a chilling effect on purchases for products like this?
Hopefully some Oxide people can answer :-)
[+] [-] bcantrill|4 years ago|reply
Also, if it's of any solace, I really don't think any of the existing players would be terribly interested in buying a company that has so thoroughly and unequivocally rejected so many of their accrued decisions! ;) I'm pretty sure their due diligence would reveal that we have taken a first principles approach here that is anathema to the iterative one they have taken for decades -- and indeed, these companies have shown time and time again that they don't want to risk their existing product lines to a fresh approach, no matter how badly customers want it.
[+] [-] emmelaich|4 years ago|reply
Not sure that 'with the software baked in' is a good phrase to use. Sounds inflexible. Perhaps a different phrasing would help?
[+] [-] paxys|4 years ago|reply
[+] [-] dadrian|4 years ago|reply
I don't understand this concern at all.
[+] [-] boulos|4 years ago|reply
But second, I'd love to understand the compute vs storage tradeoff chosen here. Looking at the (pretty!) picture [1], I was shocked to see "Wow, it's mostly storage?". Is that from going all flash?
Heading to https://oxide.computer/product for more details, lists:
- 2048 cores
- 30 TB of memory
- 1024 TB of flash (1 PiB)
Given how much of the rack is storage, I'm not sure which Milan was chosen (and so whether that's 2048 threads or 4196 [edit: real cores, 4196 threads]), but it seems like visually 4U is compute? [edit: nope] Is that a mistake on my part, because dual-socket Milan at 128 threads per socket is 256 threads per server, so you need at least 8 servers to hit 2048 "somethings", or do the storage nodes also have Milans [would make sense] and their compute is included [also fine!] -- and so similarly that's how you get a funky 30 TiB of memory?
[Top-level edit from below: the green stuff are the nodes, including the compute. The 4U near the middle is the fiber]
P.S.: the "NETWORK SPEED 100 GB/S" in all caps / CSS loses the presumably 100 Gbps (though the value in the HTML is 100 gb/s which is also unclear).
[1] https://oxide.computer/_next/image?url=%2Fimages%2Frenders%2...
[+] [-] ptomato|4 years ago|reply
eta: also suspect 30TB total just means they're leaving 64GB ram for the hypervisor OS on each node.
[+] [-] richardwhiuk|4 years ago|reply
Guessing they aren't counting threads (they say "cores"), so 64 cores per socket, 128 cores per server, 16 servers => 2048 cores.
[+] [-] neurotixz|4 years ago|reply
We built a few racks of Supermicro AMD servers (4 X computes in 2U), and we load tested it to 23kva peak usage (about 1/2 full with nthat type of nodes only, our DC would let us go further)
Were also over 1 PB of disks (unclear how much of this is redundancy), also in NVMe (15.36 TB x 24 in 2U is a lot of storage...)
Other then that not a bad concept, not sure of a premium they will charge or what will be comparable on price.
[+] [-] jsolson|4 years ago|reply
- There's a bunch of RJ45 up top that I don't quite understand :)
- A bunch of storage sleds
- A compute sled, 100G QSFP switch, compute sled sandwich
- Power distribution (rectifiers, I'd think, unless it's AC to the trays?)
- Another CSC sandwich
- More storage.
I assume in reality we'd have many more cables making things less pretty, given the number of front-facing QSFPs on those ToRs.
[+] [-] ThinkBeat|4 years ago|reply
They basically reinvented mainframes. Seems it has a lot in common with Z series.
Scalable locked in hardware, virtualization, reliability, engineered for hardware swaps, upgrades.
A proprietary operating system (?) from what someone said. (Offshoot of Solaris +++ (???) By that I mean that most of it, or all of it might be open sourced forks, but it will be an OS only meant to run on their systems.
(It would be fun to get it working at home, on a couple of PCs or a bunch of PIs)
They lack specialized processors to offload some workloads to.
Perhaps in modern terms shelfs of GPUs or a shelf fast FPGA , DSP processors. The possibilities are huge.
I didn't find any mention of from what I read.
They also lack the gigantic legacy effort to be compatible, which is a good thing.
[+] [-] zozbot234|4 years ago|reply
[+] [-] discardable_dan|4 years ago|reply
[+] [-] corysama|4 years ago|reply
It’s all fun stories from people doing amazing things with computer hardware and low level software. Like Ring-Sub-Zero and DRAM driver level software.
[+] [-] ohazi|4 years ago|reply
There are lots of reasons to be enthusiastic about Oxide but for me, this one takes the cake. I hope they are successful, and I hope this attitude spreads far and wide.
[+] [-] _jal|4 years ago|reply
- dedicated to virtualization, done their way
- rather inflexible in hardware specs
- vendor-locked at the rack - if you have hardware from someone else, it can't live in the same cabinet
I guess if you just want a pretty data center in a box and look like what they consider a 'normal' enterprise to be, it might appeal. But I'm not sure how many people asked for Apple-style hardware in the DC.
[+] [-] rapsey|4 years ago|reply
The specs are damn good. When it is all top-of-the-line, inflexibility is kind of a mute point. Where else are you going to go?
> But I'm not sure how many people asked for Apple-style hardware in the DC.
Well integrated, performant and reliable hardware that runs VMs where you can put anything on it is pretty much all everyone running their own hardware is looking for.
Honestly I am surprised how many here completely misunderstand what their value proposition is.
[+] [-] loudmax|4 years ago|reply
Anyway, glad to finally get a glimpse of what Oxide has to offer. Looking forward to seeing a lot more.
[+] [-] 2trill2spill|4 years ago|reply
[1]: https://soundcloud.com/user-760920229/why-your-servers-suck-...
[+] [-] jhickok|4 years ago|reply
If it's a custom software stack, might be nice to get a miniature dev-kit!
[+] [-] yabones|4 years ago|reply
In reality I would never want this type of hardware... It reminds me of the old boat anchor bladecenter rigs we used to use. They were great, up until you had to replace one of the blades after the support was up. It's not always practical to replace hardware every 3 years like we're supposed to, so this type of stuff sticks around and gets some barnacles.
What would be fantastic would be if the entire industry committed to an open spec for large chassis like this with a standardized networking and storage overlay... But that would never happen because vendor lock-in is the big money maker in 'enterprise'.
But wow, absolutely gorgeous machines.
[+] [-] zozbot234|4 years ago|reply
Isn't the Open Compute Project supposed to be working on that kind of stuff?
[+] [-] bri3d|4 years ago|reply
One of the most powerful elements of the trust root system is in audit ability and access control for both service-to-service and human-to-system aspects and I'm really interested in seeing how this plays out.
For example, a service mesh where hosts can be identified securely and authorized in a specific role unlocks a lot of low-friction service-to-service security. I'm curious what Oxide plan to provide in this space API and SDK wise.
I see some Zanzibar related projects on their GitHub, so it can be assumed the ACL system will be based on the principles there - but that's more a framework than an implementation.
[+] [-] tw04|4 years ago|reply
On paper with RDMA and NVMe-OF you could access any drive from any compute unit... but that's easier said than done :)
[+] [-] deeblering4|4 years ago|reply
The problem with this model is its no longer commodity hardware. You are kinda locked into their exosystem of specialized network and server equipment.
And it of course introduces some unique failure modes to mitigate too.
Not to say its not a cool idea, it’s just interesting to see how hardware trends oscillate between commodity and highly specialized proprietary designs.
[+] [-] psanford|4 years ago|reply
I'm curious about management. Can the rack operate completely standalone? I assume when you have multiple there will be some management abstraction above the rack layer?
The closest direct equivalent that I can think of to this is AWS outposts. Are there any others that I'm forgetting?
[+] [-] ksec|4 years ago|reply
(Edit: Previous Discussions https://news.ycombinator.com/item?id=21682360 )
Also thinking if the Website is not finished? All the "Read More" actually hide very little information, if so why hide it? And doesn't seems to explain the company very well. Seems like we need to listen to their PodCast to find out what is going on. ( Edit: Found a Youtube Video about it https://www.youtube.com/watch?v=vvZA9n3e5pc )
>Get the most efficient power rating and network speeds of 100GBps without the pain of cable
100GBps would be impressive, 100Gbps would be ... not much?
A interesting thing is all the terminal like graphics are actually HTML/CSS and not images.
[+] [-] tom_mellior|4 years ago|reply
[+] [-] mindfulplay|4 years ago|reply
Excited to see tech startups do actual tech instead of chasing VC funded growth hacking.
I wonder what sort of enterprise customers this targets.. (definitely not for individual devs)
[+] [-] benlivengood|4 years ago|reply
I'm curious if ARC will be running with primarycache=metadata to rely on low latency storage and in-VM cache, otherwise I could see ARC using a fair bit of that RAM overhead in the hosts.
[+] [-] MangoCoffee|4 years ago|reply
Intel is losing on the client and server. everyone is jumping ship to either ARM or AMD for client/server. hopefully Intel new engineer CEO can turn it around like AMD engineer CEO (Lisa Su)
[+] [-] tgtweak|4 years ago|reply
Second thoughts: Oh it's former SUN/joyent guys, nvm.
Will the console have fancy ncurses reporting and analytics like then product page?
[+] [-] gjvc|4 years ago|reply
This is a solid private-cloud play aimed at those corporations (probably mainly financials, but other sectors too I'm sure...) who don't want to outsource to the likes of AWS / GCP.
[+] [-] twoodfin|4 years ago|reply
[+] [-] numbsafari|4 years ago|reply
Government, too (especially non-US).