Summary: if you want performance, use Java, Scala, Go, Clojure, Lua, or C++.
Honestly, now with all the great Scala frameworks, Clojure, and the ability to run Rails, plus Cassandra, Storm, etc, I'm a little creeped out that I'm actually strongly considering building my current new project completely on the JVM.
I'm a fan of Go and an even bigger fan of Lua, and I'm not sure I agree with your summary. Go and OpenResty can push 300,000 "Hello World" plaintext responses a second, and some Java stuff can do x2 that, so JVM frameworks are strictly better? And something called cpoll_cppsp trumps a sane language choice? Don't be cute. :)
It is perfectly fine to not squeeze every drop from your hardware and pick Go or Lua. In the same vain, Flask may be x20 slower at a meaningless benchmark but you'd note that the ones that involve actual work, you know, the kind your complex app will actually perform, the more involved it is the more the difference shrinks. All the way down to x4, in other words not an order of magnitutde.
The real takeaway summary should probably be: a well made framework like flask doesn't have so much overhead that the productivity gains it offers are not worth trading a bit of performance for.
A terribly made framework (I won't name names) will bite you in the ass. Choose carefully.
I was surprised not to see Revel included in the round 7 results. You can filter results by it but nothing is shown.
This is significant because, on round 6, Revel was performing better than raw Go in at least some of the test scenarios, which is not something you can say about many frameworks.
Was there ever an explanation why Go without any frameworks was performing worse than Revel? Simply because not much effort had been put into optimizing the Go code?
I have the dubious honor of the slowest framework on the test on page #1! For some reason the i7 tests perform really badly for Phreeze, whereas the EC2 tests it performs near the top of all the PHP frameworks.
I've been too busy with work to look at the tests but my goal is to make a proper showing in round #8!
Benchmarks aside, I'm really impressed with the whole One ASP.NET thing. Seeing the whole MVC with web api and SignalR all now coming together now is making me finally feel excited about developing on the MS stack again.
I’d like to see ASP.net MVC natively on Windows, as well as ServiceStack. I think they’d compete. Mono is just not a thing for those of us in the .Net world, frankly.
If you notice the specific filters I've set, PHP actually outperforms Go for many cases. That's actually nice to see (PHP gets better with each release version). Well, if there was something like Coffeescript, but for PHP, then there's no better time than now to make it.
I really love the simplicity of Ruby, the performance of Golang/JVM and the massive popularity of PHP and it's compatibility with budget hardware (Shared hosting, etc). If there was a "Write your code in Ruby and we'll compile it to PHP" kind of generator/converter/service, that would be so awesome :D
(If anyone knows something like that that already exists, then I would be extremely thankful).
This is interesting, but why is there such a huge gap between raw php, and most of the php frameworks?
The one PHP framework that seems to do well is yaf. However, Yaf is actually written in C and is a php extension. I would expect it to do better than all the php frameworks, but that doesn't account for the huge gap between raw php and the other frameworks.
> For each request, an object mapping the key message to Hello, World! must be instantiated.
Why mandate how the benchmark is implemented? What if some frameworks could be faster by not instantiating an object? For maximum speed, you'd want to stream-process the input into output while materializing as few intermediate objects as possible.
Benchmarks should be defined in terms of their observable inputs/outputs. Otherwise it's like defining a car's 0-60 measurement with a rule that says "the car must then take gas from its tank and inject it into the engine." And then a Tesla could only compete by adding a gas engine that it doesn't need or want.
In an era when VMs are bought by the RAM size, this benchmark still does not show RAM usage, or performance at a specific RAM limit. In such a benchmark JVM is allowed to take advantage of its weakness. :)
We do have a fairly long-standing issue to add collection of resource statistics while tests are running [1]. It has not yet been implemented, however. I do want to get that implemented sometime soon because I too would like to see CPU utilization (some frameworks do not successfully saturate all of the cores!), memory utilization, and IO utilization.
I'm rather unimpressed by the implementations of the Python benchmarks, and surprised there isn't a single mention of gevent - that alone would have speeded up, say, Bottle by nearly an order of magnitude.
We use uWSGI + bottle + gevent for REST APIs and a relatively humble box can reach flood a gig Ethernet link with _useful_ replies. And, of course, any decent implementation will cache computed results, etc. (but toss Varnish in - which we haven't, yet, since there's no need - and you'll outperform anything else for cacheable replies).
With all the nice alternative frameworks available for the JVM these days, including some pretty micro/bare-bones alternatives, I wonder if Java will find new favour with startups or if they'll stick with Rails. The old "the Java ecosystem is too complex/over-engineered" excuse falls down with stuff like Play or Grizzly.
I fully realise the language itself is annoying and clunky and that Scala is not necessarily the answer. I'm just saying that there are some serious upsides to consider beyond language aesthetics.
Which is more expensive, servers or developers? Which is more expensive, server or opportunity cost?
In a lot of cases, performance of the server isn't the determining variable in the overall cost calculation.
In addition, the relationships between those variables likely changes over time.
I'm guessing that for most startups, HR expenses and opportunity costs are pretty important relative to actual server performance.
Obviously, if the point of your startup is some kind of high-performance computing thing, the equation is tipped more towards server performance. But, you probably aren't using off-the-shelf frameworks in that case, either.
This less forgiving stance with respect to glitches comes from our intent to publish a round of tests every month from this point forward (assuming we have the manpower to do so). In order to pull that off, we need to be less forgiving with problems that we don't have the know-how to fix immediately. We investigated Rebar for a bit but ultimately conceded defeat, skipped the tests, and posted the issue linked above. We'll revisit it again for Round 8.
Go's numbers are quite impressive considering how young the language is. That combined with the concurrency primitives and ease of deployment makes Go very appealing.
Although this is Round 7, we have no delusions that there is not a lot of room for improvement (particularly in the toolset, although that's a separate matter). If there are tuning tweaks, we'd love to receive a pull request.
I'm curious as to why Django was paired with gUnicorn and only gUnicorn for its tests. I would have loved to see it benchmarked with uWSGI, gEvent or something with a little more gusto.
Probably because the testers aren't experts in all frameworks and languages and deployment options, and don't have infinite time and resources to try all combinations of options to test stuff.
Maybe suggest some updates to the people doing the benchmarking. Or, god forbid, do the testing yourself and post some results for people.
Are there enough HN readers interested in contributing new benchmark scenarios for their favourite language?
Ideas I had were:
- Generating a nested JSON structure weighing in at 30kb (from a database)
- Other more-real-word scenarios which I haven't thought up yet ;)
For Round 8, it would interesting to see Cognitect's (http://cognitect.com/) new Pedestal framework (http://pedestal.io/) for Clojure. Pedestal is Ring compatible but it doesn't use Ring.
Instead, Pedestal's new interceptors abstraction decouples http requests from threads. This provides better concurrency support because it enables processing a single request across multiple threads (http://pedestal.io/documentation/service-interceptors/).
Honestly, that's what I began looking at also. Django and Rails are compared frequently - even if those comparisons aren't hostile.
This is the first time I've seen this benchmark, and it was really really interesting to see how Djangos performance degraded as more queries were executed. They mention that the lack of connection pooling is likely a big factor. I never realised how much that could affect an application.
84 frameworks and no GWT? The original cross-compiling web framework, being used for Adwords, Adsense, Google Play store, Amazon AWS, Google Groups, and lots of other huge deployments?
JRebel's web framework comparison this summer had GWT as the front runner, pitted against Spring MVC, Play, Grails, etc.:
[+] [-] rallison|12 years ago|reply
Round 6: https://news.ycombinator.com/item?id=5979766
Round 5: https://news.ycombinator.com/item?id=5727012
Round 4: https://news.ycombinator.com/item?id=5644880
Round 3: https://news.ycombinator.com/item?id=5573532
Round 2: https://news.ycombinator.com/item?id=5498869
Round 1: https://news.ycombinator.com/item?id=5454775
Lots of solid discussion.
[+] [-] integraton|12 years ago|reply
Honestly, now with all the great Scala frameworks, Clojure, and the ability to run Rails, plus Cassandra, Storm, etc, I'm a little creeped out that I'm actually strongly considering building my current new project completely on the JVM.
[+] [-] recuter|12 years ago|reply
It is perfectly fine to not squeeze every drop from your hardware and pick Go or Lua. In the same vain, Flask may be x20 slower at a meaningless benchmark but you'd note that the ones that involve actual work, you know, the kind your complex app will actually perform, the more involved it is the more the difference shrinks. All the way down to x4, in other words not an order of magnitutde.
The real takeaway summary should probably be: a well made framework like flask doesn't have so much overhead that the productivity gains it offers are not worth trading a bit of performance for.
A terribly made framework (I won't name names) will bite you in the ass. Choose carefully.
[+] [-] hbbio|12 years ago|reply
And Java/Scala are disqualified pretty quickly.
[+] [-] TylerE|12 years ago|reply
[+] [-] thezilch|12 years ago|reply
[+] [-] stesch|12 years ago|reply
Sorry, round 7 isn't helping much. Wait for round 8.
[+] [-] arocks|12 years ago|reply
[+] [-] unknown|12 years ago|reply
[deleted]
[+] [-] raveli|12 years ago|reply
This is significant because, on round 6, Revel was performing better than raw Go in at least some of the test scenarios, which is not something you can say about many frameworks.
Was there ever an explanation why Go without any frameworks was performing worse than Revel? Simply because not much effort had been put into optimizing the Go code?
[+] [-] iends|12 years ago|reply
[+] [-] wheaties|12 years ago|reply
[+] [-] jakejake|12 years ago|reply
I've been too busy with work to look at the tests but my goal is to make a proper showing in round #8!
[+] [-] bhauer|12 years ago|reply
Yes, if you have time, please do help us resolve the issue (and I am guessing it is some configuration issue) for Round 8.
[+] [-] sker|12 years ago|reply
Slightly sad to see the Mono performance is still abysmal.
Edit: link to the blog. It was submitted earlier, but apparently got killed:
http://www.techempower.com/blog/2013/10/31/framework-benchma...
[+] [-] bigdubs|12 years ago|reply
In my own testing there was another ~60% performance to gain with better tweaks.
[+] [-] cmircea|12 years ago|reply
I'm impressed by the Java options. Definitely something to consider if absolute performance is the requirement.
[+] [-] redact207|12 years ago|reply
[+] [-] omphalos|12 years ago|reply
[+] [-] mwsherman|12 years ago|reply
[+] [-] neya|12 years ago|reply
If you notice the specific filters I've set, PHP actually outperforms Go for many cases. That's actually nice to see (PHP gets better with each release version). Well, if there was something like Coffeescript, but for PHP, then there's no better time than now to make it.
I really love the simplicity of Ruby, the performance of Golang/JVM and the massive popularity of PHP and it's compatibility with budget hardware (Shared hosting, etc). If there was a "Write your code in Ruby and we'll compile it to PHP" kind of generator/converter/service, that would be so awesome :D
(If anyone knows something like that that already exists, then I would be extremely thankful).
[+] [-] iends|12 years ago|reply
The one PHP framework that seems to do well is yaf. However, Yaf is actually written in C and is a php extension. I would expect it to do better than all the php frameworks, but that doesn't account for the huge gap between raw php and the other frameworks.
[+] [-] martinml|12 years ago|reply
[+] [-] haberman|12 years ago|reply
> For each request, an object mapping the key message to Hello, World! must be instantiated.
Why mandate how the benchmark is implemented? What if some frameworks could be faster by not instantiating an object? For maximum speed, you'd want to stream-process the input into output while materializing as few intermediate objects as possible.
Benchmarks should be defined in terms of their observable inputs/outputs. Otherwise it's like defining a car's 0-60 measurement with a rule that says "the car must then take gas from its tank and inject it into the engine." And then a Tesla could only compete by adding a gas engine that it doesn't need or want.
[+] [-] cies|12 years ago|reply
[+] [-] bhauer|12 years ago|reply
[1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/10...
[+] [-] cies|12 years ago|reply
[+] [-] rcarmo|12 years ago|reply
We use uWSGI + bottle + gevent for REST APIs and a relatively humble box can reach flood a gig Ethernet link with _useful_ replies. And, of course, any decent implementation will cache computed results, etc. (but toss Varnish in - which we haven't, yet, since there's no need - and you'll outperform anything else for cacheable replies).
[+] [-] cgh|12 years ago|reply
I fully realise the language itself is annoying and clunky and that Scala is not necessarily the answer. I'm just saying that there are some serious upsides to consider beyond language aesthetics.
[+] [-] numbsafari|12 years ago|reply
In a lot of cases, performance of the server isn't the determining variable in the overall cost calculation.
In addition, the relationships between those variables likely changes over time.
I'm guessing that for most startups, HR expenses and opportunity costs are pretty important relative to actual server performance.
Obviously, if the point of your startup is some kind of high-performance computing thing, the equation is tipped more towards server performance. But, you probably aren't using off-the-shelf frameworks in that case, either.
[+] [-] ollysb|12 years ago|reply
https://grizzly.java.net/quickstart.html
[+] [-] tracker1|12 years ago|reply
[+] [-] nivertech|12 years ago|reply
[+] [-] bhauer|12 years ago|reply
See https://github.com/TechEmpower/FrameworkBenchmarks/issues/49...
This less forgiving stance with respect to glitches comes from our intent to publish a round of tests every month from this point forward (assuming we have the manpower to do so). In order to pull that off, we need to be less forgiving with problems that we don't have the know-how to fix immediately. We investigated Rebar for a bit but ultimately conceded defeat, skipped the tests, and posted the issue linked above. We'll revisit it again for Round 8.
[+] [-] strmpnk|12 years ago|reply
[+] [-] mayhew|12 years ago|reply
[+] [-] koblas|12 years ago|reply
wsgi:
tornado: What you have is the slower way to contruct a dictionary and then passing to the Python native JSON vs. a C optimized JSON for output.[+] [-] bhauer|12 years ago|reply
[+] [-] e12e|12 years ago|reply
[+] [-] shijie|12 years ago|reply
[+] [-] meepmorp|12 years ago|reply
Maybe suggest some updates to the people doing the benchmarking. Or, god forbid, do the testing yourself and post some results for people.
[+] [-] porker|12 years ago|reply
Ideas I had were: - Generating a nested JSON structure weighing in at 30kb (from a database) - Other more-real-word scenarios which I haven't thought up yet ;)
[+] [-] espeed|12 years ago|reply
Instead, Pedestal's new interceptors abstraction decouples http requests from threads. This provides better concurrency support because it enables processing a single request across multiple threads (http://pedestal.io/documentation/service-interceptors/).
[+] [-] mcv|12 years ago|reply
[+] [-] hiddentaco|12 years ago|reply
[+] [-] jsmeaton|12 years ago|reply
This is the first time I've seen this benchmark, and it was really really interesting to see how Djangos performance degraded as more queries were executed. They mention that the lack of connection pooling is likely a big factor. I never realised how much that could affect an application.
These results are really interesting!
[+] [-] steveklabnik|12 years ago|reply
[+] [-] jbeja|12 years ago|reply
[+] [-] sfjailbird|12 years ago|reply
JRebel's web framework comparison this summer had GWT as the front runner, pitted against Spring MVC, Play, Grails, etc.:
http://zeroturnaround.com/rebellabs/the-curious-coders-java-...
Glaring omission for a comprehensive comparison, IMHO.
[+] [-] virtualwhys|12 years ago|reply
Surprised it came in so high (6), no wonder Spray was acquired by Typesafe...and why Spray will take place of Netty in Play stack.