My concern about this is that with the failure of Qualcomm Centriq, there is no industry standard/affordable/easy to buy ARM based server platform ordinary persons and small/medium sized businesses can acquire.
It's great that Amazon has ARM based stuff, but it's something proprietary they're purchasing in large quantities from a manufacturer they have a very close relationship with. Undoubtedly the physical hypervisor platform and motherboard these things are running on is something totally bespoke and designed to Amazon's unique requirements.
I can't pull out my visa card and go buy a (atx, microatx, mini-itx) format motherboard for an ARM CPU, the CPU itself, RAM, etc, and build a system to run debian, centos, RHEL, ubuntu whatever on.
This means that, sure, you can get an EC2 ARM based server, but it's something you can't physically own and you'll be paying cloud based service rates forever if you want to keep running it. There are some categories of business and government entities where not having things on-premises, or fully owning and controlling the hypervisor all the way down to the bare metal, is a non starter.
If the ARM platform Amazon is buying becomes truly price/performance competitive with a single/dual socket xeon or threadripper/epyc, it also gives a possible competitive advantage to Amazon over any medium-sized cloud based VM provider out there currently selling (xen, kvm) based VMs on x86-64 hypervisors.
Based on what's available on the market right now I see no signs of there being a viable hardware-purchasing alternative to Intel or AMD based motherboards and CPUs.
> I can't pull out my visa card and go buy a (atx, microatx, mini-itx) format motherboard for an ARM CPU, the CPU itself, RAM, etc, and build a system to run debian, centos, RHEL, ubuntu whatever on.
The difference between ARM and x86 is that there's no standardized ARM interface between the motherboard and the CPU—no ARM equivalent to ACPI. Every integration of an ARM CPU into a board is bespoke. So you have to buy "a board with an ARM CPU on it", not just an ARM CPU and board separately.
But, if you relax that restriction, it's not like it's hard to acquire "a board with an ARM CPU on it." 80% of single-board computers (e.g. the Raspberry Pi) are "a board with an ARM CPU on it." You can wipe pretty much any Android device (not just phones, but also HDMI "streaming boxes", which are a convenient form-factor for basing a workstation on) and install Linux on them. There are also some higher-end development/SDK boards for ARM embedded systems, like the Nvidia Jetson. What more do you need?
As a normal person, you're mostly stuck with SolidRun's offerings if you want something that's affordable but serious (i.e. can run PCIe devices).
I run a MACCHIATObin as a desktop with FreeBSD, it's fine, great open source UEFI-ACPI firmware (upstream EDK2), generic ECAM PCIe works (with a quirk but still), but it's not a powerful system. Four A72 cores, single DDR4 channel. It's basically a rather-old-ultrabook level of performance.
Then there's their new thing with the NXP LX2160A 16-core dual-channel chip — there was a loooot of doubt about whether they could achieve good ACPI support on that SoC. But recently NXP have pushed a commit that introduces a generic description of the PCIe controller to the ACPI tables..
I agree, as someone that's had some "off mainstream" hardware over the years, I'd kind of like to have a 16+ core ARM server to play with; just because that's the kind of dork that I am.
Really though, I think this is other things, if amazon can become independent from Intel and AMD? that gives them huge leverage in terms of negotiations with them. Really, it sets AWS up quite nicely for the time when they are broken up and split out.
> I can't pull out my visa card and go buy a (atx, microatx, mini-itx) format motherboard for an ARM CPU, the CPU itself, RAM, etc, and build a system to run debian, centos, RHEL, ubuntu whatever on.
You can buy a SBC though, so you don't necessarily need to. There's SBC's with more power than the Raspberry Pi (at least up to 3). They just cost more. I'm starting to get into the habit of using more Raspberry Pi's at home for network stuff, and I'm debating setting one up for a web development target platform, something that can be code-pen like but on my Pi. Beats running a bunch of VMs and cheaper than buying a whole rig just for VMs.
An important update has been made to the microbenchmarks to this page. I will quote for you my editorial comment:
"Editor’s Note: The microbenchmarks in this article have been updated to reflect the fact that running a single instance of stress-ng would skew the results in favor of the x86 platforms, since in SMT architectures a single thread may not be enough to use all resources available in the physical core. Thanks to our readers for bringing this to our attention."
You'll note that the newly-used stress-ng command is: "stress-ng --metrics-brief --cache 16 --icache 16 --matrix 16 --cpu 16 --memcpy 16 --qsort 16 --dentry 16 --timer 16 -t 1m"
The count on the flags under the old numbers was 1. This update shows even better numbers for Arm than we originally produced. Thanks to our assiduous readers for pointing this out.
I think one of the big issues may be with high performance multi-threaded code. x86(I am including x64 in this designation) is a lot stronger memory model than ARM. This has two implications. First, x86 is a lot more tolerant of data races, and missing explicit memory fences. When you port server applications that have been running well on x86 to ARM, you may be in for some surprises as data races and missing fences now manifest as data corruption. The other implication is that on x86, the gap between a sequentially consistent memory order and a relaxed memory order is not that great. Thus, many programmers may use atomics with sequentially consistent memory order to reduce the complexity. On x86, this will generally yield decent performance. On ARM, that gap is much bigger and you are liable to have severe performance regressions.
If I'm reading this correctly, you're basically implying that every serious QA department in a a large company might benefit from buying some ARM hardware for running their test suites in order to reveal multithreading bugs and inadvertent data races in their code?
Do you happen to know of any studies or benchmarks that show how much of a difference in performance there is between strong and relaxed memory consistency models for real-world workloads? It's something that I've been curious about for a while.
One upside of code increasingly being written in languages that are predominantly single threaded like Python/JS is that these issues do not matter as much.
"AWS, the biggest of the existing cloud providers released an Arm-based offering in 2018 and now in 2019 catapults that offering to a world-class spot. With results comparable to x86-based instances and AWS’s sure ability to offer a lower price due to well known attributes of the Arm-based servers like power efficiency, we consider the new M6g instances to be a game changer in a red-hot market ripe for change."
I'm not sure how that conclusion follows from the numbers presented? Yes, the new ARM processor has become much faster than the older one, but clearly looses against x86 in cpu-heavy benchmarks.
Might be a good option for I/O limited workloads, as the NVMe storage is newer and therefore faster.
ARM has recently made some real progress into HPC. HPE is delivering ARM clusters in Europe. Fujitsu has developed a "optimised" ARM CPU with SVE and it will power Japans exascale HPC system[1].
I tried ARM servers in Scaleway and honestly, unless your profile is sort of a sysadmin or you're motivated, it's just dealing with some issues and less power overall.
Also, AFAIR they were around the same price of X86 instances.
But then again, I have almost no sysadmin skills, so maybe it was my lack of knowledge.
My experience with ARM (locally anyway, raspberry pi's and pinebooks) is that everything works great, if it works.
ARM binaries available? Great. You're all set. (hopefully its for the right generation of ARM though, I can't get SteamLink software to run on my pinebook because it does raspberry pi hardware version checks that obviously fail).
Not available? You're gonna run into one of two scenarios:
Closed source app? Sorry. Just never going to get it (unless you want to run it through qemu at a huge performance hit and YMMV).
Open source app? Compile it yourself! Which takes noticeably longer than x86 for me. Maybe in server this works out. For desktop, compiling st myself was actually the path of least resistance to getting a terminal I liked. I wouldn't have wanted to compile firefox on ARM though without some serious server horsepower. When I used to build docker images they would sometimes take hours, for things that were less than a minute on x86.
Disclosure: I currently work at Scaleway but wasn't there at the time when we rolled out the 1st ever ARM baremetal offering.
It was massively cheaper than their x86 counterparts at Scaleway or competitors, thanks in no small parts to very clever electrical engineering by my colleagues.
I think we never publicly disclosed power consumption levels but.... let's say our ARM boards are very power efficient :)
ARMv7 as a physical server. No virtualization. Very low performance, very cheap. Good for running your personal web server or similar, but not for anything seeing load. Looks they have not maintained the kernel for more than a year. They let it die, would be my guess.
ARMv8. A virtualized platform. I have not evaluated it myself yet.
I don't know much about the cloud hosting for ARM stuff (since I don't work in that space), but I have been extremely happy with my ARM home-server setup in my basement. Docker swarm has been extremely nice on my ODroids, and I recently upgraded to the Nvidia Jetson Nano, which has perfectly fine Kubernetes support.
I'll admit that maybe I'm not doing the most elaborate stress tests, but I mostly use them for my video transcoding and my (very) recent interest in machine learning, and I haven't had much issue. The thing that's given me the biggest headache is older versions of Ubuntu's mediocre support of ZFS, which has largely been fixed.
I've been thinking about moving my home server to an ARM-based setup to reduce power consumption/fan noise. My situation is closer to 'glorified NAS' that runs a few additional oddball things, though. Are you just using USB 3.0 to SATA in the cases where such is needed?
I bought an AArch64 desktop board based on one of the newer NXP manycore CPUs, because it seems to be the first of its kind.
Every SoC vendor should sell an mATX or Mini-ITX board compatible with PC components, if they want server adoption.
That goes especially for any vendor facing the even harder uphill battle of bringing RISC-V to servers: the server was dominated by PC clones for a reason.
I don't think quite yet. As others have noted, porting x86 apps to ARM can be fraught with issues concerning memory models and concurrency. Especially apps written using unmanaged languages. Newer apps written in things like .NET Core (especially if you can keep any native dependencies out of the equation) are probably going to be a lot easier to port when the time is right.
I think we'd have to be at a point where the ARM server is <50% the cost of the x86 server while offering equivalent real-world performance to make the jump worth it for the average shop. You'd also have to have a very accessible ecosystem of reliable ARM machines that developers could purchase and hack on. There are many businesses that will happily incinerate millions of dollars to keep x86 around just because changing things is frowned upon or otherwise scary.
For some applications ARM is today and it's an excellent approach. But, for most it's still somewhere on the horizon.
Can you meaningfully benchmark stuff in the cloud? It seems to make any claims on price/performance you'd want to use two local servers dedicated to the benchmark, with as near identical hardware as possible (apart from the CPUs of course), and you'd want to know the full cost of both servers.
You can benchmark using bare metal instances to eliminate noisy neighbours, and then you can meaningfully benchmark against other types of instance from the same cloud provider to get a relative cost benefit. I'd agree that obtaining the cost benefit in terms of on prem hardware is a much more involved calculation.
I'd like to see cheap ARM server hardware for small servers available for purchase.
Something relatively low power (also power efficient) that can replace an entry-level Intel Atom server like those offered by Kimsufi (OVH) and Online.net.
The Raspberry Pi 4 with 4GB RAM is getting close in terms of performance but it lacks some things I'd like to see in a server, i.e. at least two SATA or NVME ports and two LAN ports.
If you want to acquire large numbers of ARM boards for servers, our neighbours in Zhongshan are Firefly. They make a cluster server capable of 11 x their own RK3399 6 core 64-bit 'core boards', so 66 cores in total. It's cheap. http://shop.t-firefly.com/goods.php?id=111
At Kubecon this year, there were three different vendors doing ARM based storage. Two were capable with NVMe and one only with SATA/SAS. I am sure the answer to the article's title question is a no for right now, but in terms of disaggregated storage, I think the answer is yes!
The fact that ARM is written as Arm really threw me for a loop here. At first, I thought they were talking about a new programming language or something.
That comparison was never done. All CPU tests are on EBS backed instances, and in the end the I/O subsystem is compared in isolation for NVMe-backed instances in both cases.
[+] [-] walrus01|6 years ago|reply
It's great that Amazon has ARM based stuff, but it's something proprietary they're purchasing in large quantities from a manufacturer they have a very close relationship with. Undoubtedly the physical hypervisor platform and motherboard these things are running on is something totally bespoke and designed to Amazon's unique requirements.
I can't pull out my visa card and go buy a (atx, microatx, mini-itx) format motherboard for an ARM CPU, the CPU itself, RAM, etc, and build a system to run debian, centos, RHEL, ubuntu whatever on.
This means that, sure, you can get an EC2 ARM based server, but it's something you can't physically own and you'll be paying cloud based service rates forever if you want to keep running it. There are some categories of business and government entities where not having things on-premises, or fully owning and controlling the hypervisor all the way down to the bare metal, is a non starter.
If the ARM platform Amazon is buying becomes truly price/performance competitive with a single/dual socket xeon or threadripper/epyc, it also gives a possible competitive advantage to Amazon over any medium-sized cloud based VM provider out there currently selling (xen, kvm) based VMs on x86-64 hypervisors.
Based on what's available on the market right now I see no signs of there being a viable hardware-purchasing alternative to Intel or AMD based motherboards and CPUs.
[+] [-] derefr|6 years ago|reply
The difference between ARM and x86 is that there's no standardized ARM interface between the motherboard and the CPU—no ARM equivalent to ACPI. Every integration of an ARM CPU into a board is bespoke. So you have to buy "a board with an ARM CPU on it", not just an ARM CPU and board separately.
But, if you relax that restriction, it's not like it's hard to acquire "a board with an ARM CPU on it." 80% of single-board computers (e.g. the Raspberry Pi) are "a board with an ARM CPU on it." You can wipe pretty much any Android device (not just phones, but also HDMI "streaming boxes", which are a convenient form-factor for basing a workstation on) and install Linux on them. There are also some higher-end development/SDK boards for ARM embedded systems, like the Nvidia Jetson. What more do you need?
[+] [-] floatboth|6 years ago|reply
As a normal person, you're mostly stuck with SolidRun's offerings if you want something that's affordable but serious (i.e. can run PCIe devices).
I run a MACCHIATObin as a desktop with FreeBSD, it's fine, great open source UEFI-ACPI firmware (upstream EDK2), generic ECAM PCIe works (with a quirk but still), but it's not a powerful system. Four A72 cores, single DDR4 channel. It's basically a rather-old-ultrabook level of performance.
Then there's their new thing with the NXP LX2160A 16-core dual-channel chip — there was a loooot of doubt about whether they could achieve good ACPI support on that SoC. But recently NXP have pushed a commit that introduces a generic description of the PCIe controller to the ACPI tables..
[+] [-] moyix|6 years ago|reply
[+] [-] colechristensen|6 years ago|reply
https://www.gigabyte.com/ARM-Server
https://www.officedepot.com/a/products/6889935/Gigabyte-R120...
[+] [-] rjsw|6 years ago|reply
[1] https://www.solid-run.com/nxp-lx2160a-family/honeycomb-works...
[+] [-] TheCondor|6 years ago|reply
Really though, I think this is other things, if amazon can become independent from Intel and AMD? that gives them huge leverage in terms of negotiations with them. Really, it sets AWS up quite nicely for the time when they are broken up and split out.
[+] [-] giancarlostoro|6 years ago|reply
You can buy a SBC though, so you don't necessarily need to. There's SBC's with more power than the Raspberry Pi (at least up to 3). They just cost more. I'm starting to get into the habit of using more Raspberry Pi's at home for network stuff, and I'm debating setting one up for a web development target platform, something that can be code-pen like but on my Pi. Beats running a bunch of VMs and cheaper than buying a whole rig just for VMs.
[+] [-] WanderPanda|6 years ago|reply
[+] [-] NicoJuicy|6 years ago|reply
I paid 1000€, 140€ shipping
Planning to do something similar for a 8 node rpi 4 cluster, depending the results ( 600€ because of PoE)
I'm pretty sure I will come out insanely more cheaper to run my microservice experiments with docker then hosting it in the cloud.
I don't care/talk about ARM currently. Where I work they will go for performance / € in the cloud.
Not many people care about ARM in the cloud. I don't know why the topic of coming from.
Or what I'm missing?
[+] [-] PeterCorless|6 years ago|reply
"Editor’s Note: The microbenchmarks in this article have been updated to reflect the fact that running a single instance of stress-ng would skew the results in favor of the x86 platforms, since in SMT architectures a single thread may not be enough to use all resources available in the physical core. Thanks to our readers for bringing this to our attention."
You'll note that the newly-used stress-ng command is: "stress-ng --metrics-brief --cache 16 --icache 16 --matrix 16 --cpu 16 --memcpy 16 --qsort 16 --dentry 16 --timer 16 -t 1m"
The count on the flags under the old numbers was 1. This update shows even better numbers for Arm than we originally produced. Thanks to our assiduous readers for pointing this out.
https://www.scylladb.com/2019/12/05/is-arm-ready-for-server-...
[+] [-] RcouF1uZ4gsC|6 years ago|reply
[+] [-] PeterisP|6 years ago|reply
[+] [-] morning_gelato|6 years ago|reply
[+] [-] dman|6 years ago|reply
[+] [-] MrBuddyCasino|6 years ago|reply
I'm not sure how that conclusion follows from the numbers presented? Yes, the new ARM processor has become much faster than the older one, but clearly looses against x86 in cpu-heavy benchmarks.
Might be a good option for I/O limited workloads, as the NVMe storage is newer and therefore faster.
[+] [-] timthorn|6 years ago|reply
[+] [-] aloknnikhil|6 years ago|reply
Perhaps, the power saved is translated to cheaper instances. But I don't think it's worth the performance penalty.
[+] [-] magriz|6 years ago|reply
[1]https://en.wikipedia.org/wiki/Fugaku_(supercomputer)
[+] [-] iagovar|6 years ago|reply
Also, AFAIR they were around the same price of X86 instances.
But then again, I have almost no sysadmin skills, so maybe it was my lack of knowledge.
[+] [-] bisby|6 years ago|reply
ARM binaries available? Great. You're all set. (hopefully its for the right generation of ARM though, I can't get SteamLink software to run on my pinebook because it does raspberry pi hardware version checks that obviously fail).
Not available? You're gonna run into one of two scenarios:
Closed source app? Sorry. Just never going to get it (unless you want to run it through qemu at a huge performance hit and YMMV).
Open source app? Compile it yourself! Which takes noticeably longer than x86 for me. Maybe in server this works out. For desktop, compiling st myself was actually the path of least resistance to getting a terminal I liked. I wouldn't have wanted to compile firefox on ARM though without some serious server horsepower. When I used to build docker images they would sometimes take hours, for things that were less than a minute on x86.
[+] [-] karambahh|6 years ago|reply
It was massively cheaper than their x86 counterparts at Scaleway or competitors, thanks in no small parts to very clever electrical engineering by my colleagues.
I think we never publicly disclosed power consumption levels but.... let's say our ARM boards are very power efficient :)
[+] [-] usr1106|6 years ago|reply
ARMv7 as a physical server. No virtualization. Very low performance, very cheap. Good for running your personal web server or similar, but not for anything seeing load. Looks they have not maintained the kernel for more than a year. They let it die, would be my guess.
ARMv8. A virtualized platform. I have not evaluated it myself yet.
[+] [-] sanxiyn|6 years ago|reply
[+] [-] tombert|6 years ago|reply
I'll admit that maybe I'm not doing the most elaborate stress tests, but I mostly use them for my video transcoding and my (very) recent interest in machine learning, and I haven't had much issue. The thing that's given me the biggest headache is older versions of Ubuntu's mediocre support of ZFS, which has largely been fixed.
[+] [-] syntheticnature|6 years ago|reply
[+] [-] microcolonel|6 years ago|reply
Every SoC vendor should sell an mATX or Mini-ITX board compatible with PC components, if they want server adoption.
That goes especially for any vendor facing the even harder uphill battle of bringing RISC-V to servers: the server was dominated by PC clones for a reason.
[+] [-] bob1029|6 years ago|reply
I think we'd have to be at a point where the ARM server is <50% the cost of the x86 server while offering equivalent real-world performance to make the jump worth it for the average shop. You'd also have to have a very accessible ecosystem of reliable ARM machines that developers could purchase and hack on. There are many businesses that will happily incinerate millions of dollars to keep x86 around just because changing things is frowned upon or otherwise scary.
For some applications ARM is today and it's an excellent approach. But, for most it's still somewhere on the horizon.
[+] [-] rwmj|6 years ago|reply
[+] [-] timthorn|6 years ago|reply
[+] [-] jafingi|6 years ago|reply
[+] [-] emmanueloga_|6 years ago|reply
The table of support for ARM of sbcl [1] and ccl [2] shows only Linux is supported on ARM (versus all sorts of OSs for x86/AMD64).
I imagine the Intel targets of these compilers are a lot more widely used and hence had had more opportunities for bug ridding.
[1]: http://www.sbcl.org/platform-table.html
[2]: https://ccl.clozure.com/
[+] [-] aidenn0|6 years ago|reply
Obviously there's no ARM64 windows or solaris port in the works.
[+] [-] Tepix|6 years ago|reply
The Raspberry Pi 4 with 4GB RAM is getting close in terms of performance but it lacks some things I'd like to see in a server, i.e. at least two SATA or NVME ports and two LAN ports.
[+] [-] 8fingerlouie|6 years ago|reply
Only 2GB RAM, but 4 SATA ports.
Otherwise, Hardkernel.com might have something in their Odroid lineup you might like.
[+] [-] contingencies|6 years ago|reply
[+] [-] birdyrooster|6 years ago|reply
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] luord|6 years ago|reply
[+] [-] pnako|6 years ago|reply
[+] [-] ryuukk_|6 years ago|reply
[deleted]
[+] [-] techie128|6 years ago|reply
[+] [-] glommer|6 years ago|reply