These benchmarks were done by someone who seems to have no idea about the ins-and-outs of EC2. For example, there's a lot of IO bound tests... but they're on instance storage. What about Instance storage vs EBS? What about network performance over time?
All I can get from these is that Amazon isn't lying that their instance sizes are different in terms of compute units and memory.
It looks strange to me that High-CPU Instances didn't come as clear winners in compilation tests. I thought of choosing only between c1.medium and c1.xlarge for my CPU-intensive tasks. Looks like I should look closely into all types of instances instead, except for micro and small.
According to the specs Amazon gives us, "High-Memory" instances have 3.25 Compute Units per virtual core, while "High-CPU" only 2.5. It would be interesting to know what is going on: are their compilation tests unable to properly utilize many cores, or the nodes assigned to "High-Memory" instances tend to have more powerful hardware, or what?
I was hoping for more extensive and definitive. Bandwidth? Latency? Startup time? A meaningful sample with variation from day to day and an idea of variation across instances running at the same time?
For all the dollars being thrown at EC2, I'm surprised someone hasn't gone nuts with benchmarking this for the authority, notoriety and seo link juice.
I've done some long-running benchmarks of EC2, mostly for kicks and personal edification. For about 8 months now, I've had a job running on an EC2 "small" instance, that performs a suite of microbencharks every 10 seconds and records the results. Lots of raw data can be found here:
This covers a variety of AWS operations, ranging from simple EC2 CPU benchmarks to SimpleDB and RDS operations. I also include some measurements of Google App Engine. I've put a lot of focus on measuring variation over time, and measuring outliers (i.e. 99.9th percentile, not just mean). I'd like to measure variation across instances, but haven't gotten around to it.
[+] [-] shizcakes|15 years ago|reply
All I can get from these is that Amazon isn't lying that their instance sizes are different in terms of compute units and memory.
[+] [-] listic|15 years ago|reply
According to the specs Amazon gives us, "High-Memory" instances have 3.25 Compute Units per virtual core, while "High-CPU" only 2.5. It would be interesting to know what is going on: are their compilation tests unable to properly utilize many cores, or the nodes assigned to "High-Memory" instances tend to have more powerful hardware, or what?
[+] [-] jcborro|15 years ago|reply
[+] [-] ericb|15 years ago|reply
For all the dollars being thrown at EC2, I'm surprised someone hasn't gone nuts with benchmarking this for the authority, notoriety and seo link juice.
[+] [-] snewman|15 years ago|reply
http://amistrongeryet.com/dashboard.jsp
This covers a variety of AWS operations, ranging from simple EC2 CPU benchmarks to SimpleDB and RDS operations. I also include some measurements of Google App Engine. I've put a lot of focus on measuring variation over time, and measuring outliers (i.e. 99.9th percentile, not just mean). I'd like to measure variation across instances, but haven't gotten around to it.
The web site linked above has no documentation at the moment, and is a bit broken in other ways, but the data may be of interest. I have a little blog (http://amistrongeryet.blogspot.com/) where I've discussed some of my findings, e.g. http://amistrongeryet.blogspot.com/2010/04/early-results-on-..., http://amistrongeryet.blogspot.com/2010/04/beginnings-of-rob..., and http://amistrongeryet.blogspot.com/2010/04/cloud-data-storag....
If there's interest, I'll put a little effort into cleaning up and documenting the data reporting site.
[+] [-] DrJosiah|15 years ago|reply
[+] [-] samuel1604|15 years ago|reply
[+] [-] eclark|15 years ago|reply
[+] [-] snissn|15 years ago|reply