cgbystrom's comments

Doesn't Linux schedule interrupts on CPU0 by default?

I tried enabling RPS/RFS, which to my understanding, did this; load balance the interrupt handling among multiple cores. With this enabled, I saw little to no difference in connection rate. But then again I'm guru, I might as well double check this.

Updated my little "action plan" in the original Serverfault question with this info.

Thanks, noted. Updated the "action plan" in my question at Serverfault.

(I wrote the Serverfault question)

Thanks for the suggestions, here's some clarifications:

* I clarified what hardware I've tested in a comment (see below).

* I've run tests with my server ranging from 10 sec up to 10 minutes. They're consistently bad unfortunately.

* Interesting what you say about dom0/domU. I'm no Xen guru, but the culprit is probably something like that. I've been using a backlog of 1024 for the server tests (set both in Java land and sysctl.conf). The netperf are all defaults, both in terms of run time and backlog. Was actually trying to monitor the backlog somehow, but I'm not sure that's even possible in Linux?

* The server isn't doing anything disk IO-bound so this shouldn't be the case.

* syncookies seems like a good idea, I will definitely try that along with a lot bigger backlog and see if it makes any difference. I'll also try and see if netperf can be tweaked as well to provide a better, isolated test case.

Writing this off as Xen overhead would be such a shame, virtualization should not cause this much overhead. I'll continue investigating!

I've tried it on variety of different servers/hardware using both my server+apachebench and netperf as benchmarking tools.

* On EC2 I've tried the c1.xlarge and the giant cc1.4xlarge. With cc1.4xlarge, I saw maybe a ~10% increase accept() rate. * Two separate, virtualized servers at the office. * A private, virtualized server on Rackspace was briefly tested as well.

A compilation of netperf results are available at https://gist.github.com/985475

Open source at EA: http://gpl.ea.com/

Makes me think of why Backblaze built their own "storage pod". http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-h...

According to them, their storage costs are ~20 times less than those of S3. A bit different use-case for sure. But as wave says, once you become a storage company you will inevitably need to do it in-house to be cost-effective.

(Disclaimer: I'm the project lead on Beaconpush)

Beaconpush is a good alternative for doing this. We support user presence and you can use the API to find out what users are connected. You can even log out users from the API.

We have a contributed Ruby library (called Dimma) made by one of our users. Don't think it's EventMachine based though. Read more at https://github.com/Burgestrand/Dimma

node.js with socket.io is also a nice alternative. Played with it somewhat myself and I like it! But hosting it yourself will make you deal with scaling, deploying, uptime and hosting in general. Not exactly stuff you want to focus on when developing things on a tight schedule.

So if you're asking me, I'd naturally say Beaconpush (http://beaconpush.com) :)

Same for us, their gateway turned us down.

Are you sure it was CheddarGetter that turned you down? I've also been denied but it was one of their supported payment gateways that denied me (even though I applied through CG).

Recently been thinking of changing to Chargify now that they support a payment gateway in Scandinavia. We (http://beaconpush.com, push service for Web Sockets) currently have our custom-built system for recurring payments. It works, but I really don't want to spend time hacking on it. I rather be focusing on our core business of developing our service.

Agree with you on most points.

However on some points I'm not as convinced. Take the DOM abstraction, it works and is implemented by several SPA frameworks. You can design good looking, "desktop class" applications pretty fast. But problem arises quickly when your graphic designers send you those PSD mockups, full of great looking artwork waiting to come to life.

With a normal DOM approach you've always been able to solve this. With hacking some HTML, tuning CSS and a lot of swearing you pull through. But with SPA frameworks that favor "components" over low-level, raw DOM elements things usually aren't as straightforward. Very often you need to start picking apart the provided "ready-made components" to get any work done. Ends up being very contra-productive and usually takes way longer to do.

It has happened to me numerous times before with both GWT and Adobe Flex. Nice and shiny, given you don't try changing the layouts too much. Surely, I'd be one happy camper if this wasn't the case, web development need to move forward. And I hope the goals proclaimed by both Cappuccino and Sproutcore will work in practice some day.

Regarding your other point about routing, departing from the route paradigm will be only be true if what your designing is not a document-centric application. In my world, an SPA can be either document-centric or desktop-like (containing a lot of UI state, as you mention). Think the answer to that is the boring "it depends".

Worth mentioning, many of these issues are stuff we've been trying to resolve with the Planet Framework (http://www.planetframework.com), essentially bridging the gap between client and server.

Netty is not the problem. Java NIO has that limitation. And performance isn't in any way bad, just that there's more to extract (I believe).

It turns out that Java synchronizes on accept() capping the number of socket accepts that can made each second on a single port.

But even if that would be solved or if you used something more low-level like, say nginx, you'd still be limited of how fast you can accept().

Try using ApacheBench and hit one server with keep-alive on and off respectively. Even nginx, which I would say is a performant web server, show quite some discrepancies between having to reconnect each time and having a permanent connection.

I file this under the penalty of having to set up and tear down a socket each time. Something that only can be avoided by doing less reconnecting.

I could of course be wrong but these are the findings from doing performance testing and discussing it on the Netty mailing list. But I'd gladly accept suggestions on how to improve this situation for our service.

An ordinary Comet server with long-poll can actually end up eating quite some CPU.

I run beaconpush.com which is a cloud-based service for browser push messages. We're based on Netty and Java, which works out well for us. We've looked at Erlang and wrote some initial prototypes in it but Java became our language of choice. Mainly because we knew it and had some experiences with Netty, a NIO library for Java. Netty did also outperform Erlang and mochiweb (both vanilla configured).

Anyway, the problem tend to be with long-poll that users re-establish their connection back to the servers each time they receive a message or navigate to another page (or reload the page). This end up taking a big toll on the system. We're to some extent limited to how fast we can perform accept() on our sockets.

If you use long-polling with multi-part you can get away with sending more messages on a single established connection (it becomes request/response/response/response or similar). That can reduce the system load and the use of WebSockets can eliminate the use of reconnections altogether (disregarding any page reloads).

Facebook's use of AJAX navigation (i.e not reloading the entire page when user clicks on links etc) also reduces this load. This due to not having to re-establish a connection each time a user reloads a page.

So yes, we're actually CPU bound by the accept() behavior (at least to how the JVM does accept()). But would our connection be more permanent in nature then no, we would of course be more I/O bound if not completely.

You host a lot of open-source projects and GitHub itself is even built on it.

What would you say is the best way to monetize your open-source product?

I find Celerity with its completely headless browser much more interesting for doing acceptance/functional tests.

Sports both CSS and JavaScript without requiring rendering.

Thanks for your feedback!

1. Right now we offer up to one million messages for free. After that, we charge per each initiated million messages delivered. Meaning, if you send 900 000 you will be charged for 1 MM. Much how Amazon bills hours on EC2.

2. Well, sure. Our endpoint server (where all browsers connect) is written in Java. We're using the Netty framework, which we are incredibly happy with. Highly recommend it for doing high-performance networking in Java. Also, using the JVM instead of, say node.js, makes it very easy to utilize all our CPUs.

The user-facing website is implemented in our own framework called Planet. Developed in-house using Python with focus on real-time web sites. Both Beacon and Planet share the same underlying push technology and we have some major customers using it. You can read more about it at http://www.esn.me

The REST API is also implemented in Python and coordinates the message sending among our end-point servers. It's quite simple actually, just 200 lines of code.

3. Sounds like a good idea. It's just a few lines of code that's needed to integrate it. But yeah, a simple chat app hosted on GitHub would be helpful for newcomers.

-- Carl

For Java reloading, I'd recommend you check out http://www.zeroturnaround.com/jrebel/

I think Alex confuses "scaling in the small" with performance. Rewriting Starling in Scala most probably improved Twitter but it did it make it more scalable?

To me as an outsider, it sounds more as Starling (or Kestrel) got its performance upgraded, not scalability.

May sound pedantic, but it's important to differentiate performance from scalability.

This the exact reason why I dumped Twisted in favor for gevent (greenlet/coroutine-based NIO library)

Paul Tyma discusses some other myths regarding evented vs threaded servers: http://www.mailinator.com/tymaPaulMultithreaded.pdf