This is one of those headlines that initially reads like a joke until you think of the economics of it. "cost is about 5-10% of the cost of an equivalent system built with off-the-shelf computer parts."
I wonder if this is due to the fact that the Playstation hardware is (was?) competitively priced to encourage revenue generation through games? Or was Sony simply very good at mass-producing these units?
Sony subsidizes the cost of the hardware, in order to sell games. They were not happy with the Airforce doing this and it probably played a role in Sony pushing out an update that disabled Linux on all PS3s.
The PS3 probably was the cheapest way to play with it in practice. There is a reason why modern GPUs share an architecture with video gamers.
It's not about technical advancement, as much as economics. There are two groups of people who want TFlops of SIMD compute. Supercomputer groups, and video gamers.
Cell / PS3 was one attempt at making one device work with both groups, sharing research and economic investment.
NVidia over the next 15 years would execute these economics better however.
I thought PlayStation and XBoxes are subsidized by the Sony and Microsoft respectively as they make most of the money off the games and the network subscription. Nintendo is the only one that does not do that.
>I wonder if this is due to the fact that the Playstation hardware is (was?) competitively priced to encourage revenue generation through games? Or was Sony simply very good at mass-producing these units?
> The PS3s are the older, larger variety, since the newer slim models don't allow for the installation of Linux.
Big caveat there. Sony later removed the ability to install "OtherOS" (a.k.a. Linux or FreeBSD) on PS3s from those models. [0] I wonder if that firmware update basically neutered the supercomputer or if they were able to keep them from updating automatically.
1. No one updates supercomputers automatically. Instead you firewall the compute nodes and have a team dedicated to maintenance and support to do periodic upgrades.
2. I'm sure they had a support contract, which may have included running an entirely different firmware. I have no specific knowledge, but based on other supercomputers I've worked with, these are often very custom machines, despite the "commodity" hardware. The odds that they were running the consumer version of the firmware are low to nil.
UGH! All this infrastructure and all this work into a software platform and it's something that will be obsoleted in a few years by off the shelf GPUs built on an standard development environment backed by industry
That's the problem of building something super-cutting-edge on a grand scale - you run the risk of making a super evolved version of an evolutionary dead end.
Sure it sounds like a cool idea at the time, the Cell optimized SETI At Home demo around the PS3 launch ripped through work units far more quickly than any Intel system.
I wonder what became of the cluster once it was decommissioned? Military surplus PS3s anyone?
The challenge was not rigging this up - but making things usable & efficient. The Cell Broadband Engine was notoriously hard in architecture & needed lots of optimization.
Part of the reason why PS4 & Xbox later pivoted towards x86-64 ISA
If you were building a supercomputer the CBE’s peculiarities (specifically the SPEs) were what you wanted.
And the 360 didn’t use SPEs, it has a pretty standard 3-core CPU built out of the PPE. The weird-ass structure of the CBE was not why it switched (although PPC might be).
I remember in college we had a lab full of PS3s because they were cheap and powerful and there was a course you could take to learn to develop on them. The lab later expanded after a big donation from nvidia of high end gaming machines to teach students CUDA.
The courses sucked though, because everyone would take them just to get access to the labs.
In the PS2 era I considering buying it and using the "linux kit" instead of buying a desktop. I was prepared for the differences in performance but then I knew that sony didn't release all the drivers or specs and graphic acceleration was not available. Just gave up the idea.
The PS3 could be used as a computer, this allowed sony to pay less taxes in the EU. Since it is such a closed platform the "install other OS" feature could be disable remotely by the vendor automatically and, I think, without any user intervention. When it became economically "better" for sony, they disable the feature.
These are good examples of the problems with such closed systems.
I worked on a DARPA project a few years before this - where we were using CBE as the core for a polymorphic processor (one with an FPGA attached to every IO). We were also gutting PS3s to make mission computers for early unmanned systems - running Ubuntu on top. USAF wasn't the only one - not only were there commercial supply challenges with the PS3, various components were being horded by various nation states. We were pretty sure they didn't even no what to do with the parts, but was a basic attempt to prevent projects like this from getting off the ground.
The Cell's SPEs were such an inspired design - if you look at CPU architecture, 99% of the complexity comes from pretending there's a flat memory space - data hazards, cache coherency, latency hiding, prediction etc.
If you make a beefy processor that works like a microcontroller - reading and writing everything from SRAM, and making all main memory DMA explicit - you get all the speed a fraction of the logic budget.
If you look at a modern CPU pipeline, it's 20ish stages, with most of them about handling the aforementioned complexities, with fetch-decode-execute taking up at most like 6 of them. In MCUs (and SPEs) it's basically all there is.
Unfortunately they never did figure out how to write programs for such a curiosity in an accessible manner.
I'm curious if it's worth taking another crack at this architecture, this time with better tooling.
I remember reading about the air force buying a lot of ps3's back in the 2000s, and that just cemented to me that information is power and any government will do all they can to own as much of it as possible. IBM just published some quantum breakthrough[1] and my first thought was that it will be used by the military or intelligence agencies. Hopefully by the time this technology trickles down to more corrupt agencies the people will have caught up.
You might as well be talking about magic when it comes to quantum computing and me, I have no idea if that new openssh standard really does protect against a quantum computer trying to break it.
Wouldn't seeing the Air Force clamoring for commodity gaming hardware usually destined for a kid's bedroom be an inspiring example of equal access to technology?
Is there any evidence of the resulting “supercomputer” being used? It’s one thing to buy two thousand PS3s. Another to deploy them into a data center. And yet another to build processes and software that uses it. The challenges involved seem discouraging.
The most surprising aspect of this story to me is that the Air Force was allowed to exercise this level of resourcefulness. I just assumed that the procurement process would stipulate 10 year support contracts or something similar.
If some officer can cobble together a proof of concept from COTS parts and show that it is just as reliable and effective as the boutique stuff from Raytheon, et al., at a way cheaper price, you bet that it will be met with considerable attention if not approval. Most ground-based military drones these days are controlled with Xbox controllers because those controllers are cheap and ubiquitous. In the 90s, Doom (with mods) was pressed into service as a simulator to help teach Marines fireteam tactics.
About the most stiff-necked branch of the U.S. military is the Navy. I'd be more surprised if the Navy approved a PS3 supercomputer, especially if it were to be run aboard ship.
I was there, but not on that project. It worked well. The reasons it was attractive were the cost savings, but more importantly the unique aspects of the Cell processor in the PS3s.
What's the best lengthy write-up of the Air Force's PS3 cluster (if any exist)? From the headlines, I had always thought it to be some marketing gimmick.
Why is that? What was it about the Cell processor at the time that was desirable?
Afaik PS3s were sold at a loss for Sony, so it seems likely that they were very beefy computationally wise, but I am use USAF could have gotten an incredible deal with Intel, IBM or AMD, so what is it?
Sony actually officially supported it at the time... and then realized that since PS3 was sold at a massive loss, selling thousands of units to bulk customers who would never buy any games was costing them millions.
Subsequent consoles dropped the "sell the hardware at a loss, make it up on games" model, they only lose money during a brief window at launch and then economies of scale take over and most of the console life it's profitable. It's likely that other kinds of hardware also share this model but we just don't hear about it - steam deck is running tight margins and day-1 sales are more expensive than the "average" unit sold mid-gen or late-gen, so similarly they are probably selling day-1 units at a loss even if they make a profit on the expected cost.
Of course... even though sony is selling the hardware at a profit, they still maintain the lock-in, just like apple/etc. The linux feature never came back even though the reason for the removal went away.
> I believe they had to jailbreak these to get Linux installed on them.
OtherOS was a fully supported feature on the PS3, that enabled you to install Linux on a secondary partition on the built in HDD. At least until George Hotz used it to get hypervisor access and exploit the PS3.
This lead to the feature being removed and him being sued by Sony.
Nope. PS3s were initially released with a built-in feature to install other OSes. This Air Force project got a ton of publicity. It directly led to Sony deciding to lock down the hardware and firmware on future releases.
[+] [-] chrisma0|3 years ago|reply
I wonder if this is due to the fact that the Playstation hardware is (was?) competitively priced to encourage revenue generation through games? Or was Sony simply very good at mass-producing these units?
[+] [-] riskneutral|3 years ago|reply
[+] [-] dragontamer|3 years ago|reply
The PS3 probably was the cheapest way to play with it in practice. There is a reason why modern GPUs share an architecture with video gamers.
It's not about technical advancement, as much as economics. There are two groups of people who want TFlops of SIMD compute. Supercomputer groups, and video gamers.
Cell / PS3 was one attempt at making one device work with both groups, sharing research and economic investment.
NVidia over the next 15 years would execute these economics better however.
[+] [-] manishsharan|3 years ago|reply
[+] [-] throwaway0a5e|3 years ago|reply
By 2010? Likely both.
[+] [-] masklinn|3 years ago|reply
[+] [-] zappo2938|3 years ago|reply
[+] [-] AdmiralAsshat|3 years ago|reply
Big caveat there. Sony later removed the ability to install "OtherOS" (a.k.a. Linux or FreeBSD) on PS3s from those models. [0] I wonder if that firmware update basically neutered the supercomputer or if they were able to keep them from updating automatically.
[0] https://en.wikipedia.org/wiki/OtherOS
[+] [-] eslaught|3 years ago|reply
2. I'm sure they had a support contract, which may have included running an entirely different firmware. I have no specific knowledge, but based on other supercomputers I've worked with, these are often very custom machines, despite the "commodity" hardware. The odds that they were running the consumer version of the firmware are low to nil.
[+] [-] Scuds|3 years ago|reply
That's the problem of building something super-cutting-edge on a grand scale - you run the risk of making a super evolved version of an evolutionary dead end.
Sure it sounds like a cool idea at the time, the Cell optimized SETI At Home demo around the PS3 launch ripped through work units far more quickly than any Intel system.
I wonder what became of the cluster once it was decommissioned? Military surplus PS3s anyone?
[+] [-] avian|3 years ago|reply
If it touched at any time any kind of confidential data or algorithms it was probably shredded into tiny pieces.
[+] [-] HPsquared|3 years ago|reply
[+] [-] srvmshr|3 years ago|reply
Part of the reason why PS4 & Xbox later pivoted towards x86-64 ISA
[+] [-] masklinn|3 years ago|reply
And the 360 didn’t use SPEs, it has a pretty standard 3-core CPU built out of the PPE. The weird-ass structure of the CBE was not why it switched (although PPC might be).
[+] [-] bilekas|3 years ago|reply
[+] [-] mywittyname|3 years ago|reply
I remember in college we had a lab full of PS3s because they were cheap and powerful and there was a course you could take to learn to develop on them. The lab later expanded after a big donation from nvidia of high end gaming machines to teach students CUDA.
The courses sucked though, because everyone would take them just to get access to the labs.
[+] [-] trimbo|3 years ago|reply
https://en.wikipedia.org/wiki/Roadrunner_(supercomputer)
[+] [-] schaefer|3 years ago|reply
[+] [-] marcodiego|3 years ago|reply
The PS3 could be used as a computer, this allowed sony to pay less taxes in the EU. Since it is such a closed platform the "install other OS" feature could be disable remotely by the vendor automatically and, I think, without any user intervention. When it became economically "better" for sony, they disable the feature.
These are good examples of the problems with such closed systems.
[+] [-] kitplummer|3 years ago|reply
[+] [-] kitplummer|3 years ago|reply
[+] [-] omershapira|3 years ago|reply
To be fair, he could have been more upfront about it. Some marketing managers would have shown that at E3.
[1] https://www.reddit.com/r/AskHistorians/comments/l3hp2i/did_s...
[+] [-] jlink|3 years ago|reply
[+] [-] torginus|3 years ago|reply
If you make a beefy processor that works like a microcontroller - reading and writing everything from SRAM, and making all main memory DMA explicit - you get all the speed a fraction of the logic budget.
If you look at a modern CPU pipeline, it's 20ish stages, with most of them about handling the aforementioned complexities, with fetch-decode-execute taking up at most like 6 of them. In MCUs (and SPEs) it's basically all there is.
Unfortunately they never did figure out how to write programs for such a curiosity in an accessible manner.
I'm curious if it's worth taking another crack at this architecture, this time with better tooling.
[+] [-] pyinstallwoes|3 years ago|reply
[+] [-] INTPenis|3 years ago|reply
You might as well be talking about magic when it comes to quantum computing and me, I have no idea if that new openssh standard really does protect against a quantum computer trying to break it.
1. https://www.techradar.com/news/ibm-claims-to-have-mapped-out...
[+] [-] zaptrem|3 years ago|reply
[+] [-] blueboo|3 years ago|reply
[+] [-] thetinguy|3 years ago|reply
https://en.wikipedia.org/wiki/PlayStation_3_cluster
[+] [-] dc-programmer|3 years ago|reply
[+] [-] smallmouth|3 years ago|reply
[+] [-] bitwize|3 years ago|reply
About the most stiff-necked branch of the U.S. military is the Navy. I'd be more surprised if the Navy approved a PS3 supercomputer, especially if it were to be run aboard ship.
[+] [-] tomrod|3 years ago|reply
[+] [-] fdr|3 years ago|reply
[+] [-] Communitivity|3 years ago|reply
[+] [-] danso|3 years ago|reply
[+] [-] PartiallyTyped|3 years ago|reply
Afaik PS3s were sold at a loss for Sony, so it seems likely that they were very beefy computationally wise, but I am use USAF could have gotten an incredible deal with Intel, IBM or AMD, so what is it?
[+] [-] hypeatei|3 years ago|reply
Just imagining an Air Force employee (?) browsing community modding forums and such for older firmware and tools, kinda funny.
[+] [-] paulmd|3 years ago|reply
Subsequent consoles dropped the "sell the hardware at a loss, make it up on games" model, they only lose money during a brief window at launch and then economies of scale take over and most of the console life it's profitable. It's likely that other kinds of hardware also share this model but we just don't hear about it - steam deck is running tight margins and day-1 sales are more expensive than the "average" unit sold mid-gen or late-gen, so similarly they are probably selling day-1 units at a loss even if they make a profit on the expected cost.
https://www.extremetech.com/gaming/325504-sony-finally-turns...
Of course... even though sony is selling the hardware at a profit, they still maintain the lock-in, just like apple/etc. The linux feature never came back even though the reason for the removal went away.
[+] [-] flutas|3 years ago|reply
OtherOS was a fully supported feature on the PS3, that enabled you to install Linux on a secondary partition on the built in HDD. At least until George Hotz used it to get hypervisor access and exploit the PS3.
This lead to the feature being removed and him being sued by Sony.
https://en.wikipedia.org/wiki/George_Hotz#Sony_lawsuit
[+] [-] humanistbot|3 years ago|reply
[+] [-] smm11|3 years ago|reply
[+] [-] latchkey|3 years ago|reply
https://www.techspot.com/news/93980-14800-asrock-mining-rig-...
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] Barrin92|3 years ago|reply