top | item 31990089

Ejabberd scalability: single node with 2M concurrent users (2016)

53 points| richardboegli | 3 years ago |process-one.net | reply

15 comments

order
[+] flakiness|3 years ago|reply
How is ejabberd or XMPP in general doing these days?

I remember Nintendo adopted it for their push platform [1] but it was old news. Are there any interesting development and/or adoption stories in recent years?

[1] https://www.process-one.net/blog/ejabberd-nintendo-switch-np...

[+] nivertech|3 years ago|reply
Modern versions of ejabberd and MongooseIM are much better than earlier versions in regards to scaling. But XMPP is still a mess. If you need to develop a custom messaging app, it's much better to start from a clean slate, rather than trying to fit your data and interaction models into an XMPP framework. As a transport you can use Websockets, MQTT, and/or GraphQL.

The main problems with the earlier versions of ejabberd were:

1. Using Erlang strings (linked lists of UTF32 codepoints) instead of UTF8 binaries, which led to 16x more RAM use.

2. Over-reliance on Mnesia for session management which is a maintenance nightmare when used in large clusters. Now it can be replaced by Redis or your own DB.

3. Non-scalable default implementations for mod_muc (Multi-User Chat) and mod_pubsub.

4. Absence of built-in mobile push support.

That was before WhatsApp, which later committed upstream many of their scalability improvement to the BEAM VM.

Source: I scaled an old ejabberd cluster to 500K concurrent users, and wrote my own Erlang-based Long-Polling/Websocket Pub/Sub servers and MQTT brokers scaling to 1M concurrent connections per node in a cluster. Also Elixir-based specialized messaging systems.

[+] aidenn0|3 years ago|reply
As of 2 years ago, there were good XMPP clients for every platform except the iPhone. On the server side, both ejabberd and Prosody are in great shape. There is zero excitement for administering them.
[+] akho|3 years ago|reply
With ejabberd, your scalability problem is that you don’t have enough users to load a single node. So you have a popularity problem, but not a technical one.

To get back to familiar technical ground, try Synapse. A node can support, like, 3 concurrent users.

[+] nisa|3 years ago|reply
> A node can support, like, 3 concurrent users.

I wish you would joking. But it doesn't depend so much on the users but more on the size of the room / amount of events and the amount of other servers in the room.

However Matrix solves a different problem https://github.com/jryans/awesome-matrix#research so it's a bit unfair to compare both.

I hope someday someone pulls off an memory/cpu efficient matrix protocol implementation. https://conduit.rs/ looks good and there is still work on construct the c++ server: https://github.com/jevolk/charybdis - but both only implement a subset of the rather huge and complicated spec at the moment.

[+] alberth|3 years ago|reply
There’s some active HN users who developed WhatsApp.

I’m sure they have a wealth of knowledge related to scaling since WhatsApp was initially based on ejabberd.

[+] nivertech|3 years ago|reply
There are several talks by WhatsApp people who describe how they're scaled their systems. They run on FreeBSD and used Dtrace to quickly find bottlenecks in the BEAM VM. BEAM has a very good builtin instrumentation, but if you're only looking on them, you're missing a lot. Also they committed upstream many of their improvements.

Not only WhatsApp, but also the initial implementation of APNs (Apple Push Notification System) was based on ejabberd. They even used the same port numbers. I did an ejabberd scaling training in SF in 2011, and there were several Apple guys. It doesn't mean that they're used ejabberd for sure, but it's a hint.

[+] imoverclocked|3 years ago|reply
I like the federation abilities of xmpp and ran ejabberd for a previous job. It’s perfect for an organization … in theory. Client support for the protocol was always hit or miss though.

Have file transfers over xmpp gotten any better in the last decade?

[+] leetnewb|3 years ago|reply
Haven't tested much. But I think HTTP upload is reliable and has been ok for me on Snikket server (Prosody) between Snikket clients (Conversations) and between Snikket and Gajim.