Riot Games Messaging Service

[matt_oriordan]

That's am impressive bit of technology, and nice to see it's making extensive use of Erlang (we're using Elixir).

I'd be interested to know if the service works across multiple regions though. Some of the biggest challenges we've faced when engineering our realtime platform has been in having no single point of congestion and effectively peer-to-peer routing within the cluster. This is not that important if all your servers can be in a single region, however if you want your users in Australia for example to have a similar latency profile to those in USA, then clients in Oz need to connect to servers in Oz and all routing for customers in Oz does not need to traverse continents for other customers in Oz, but does for customers in US. I'd be interested to know if that was tackled in the design. Michal you following this thread?

Matt Ably realtime - https://www.ably.io

[cyunker]

It does work across multiple Riot regions, however it's meant for applications which can tolerate a little (~1 sec) latency. So we didn't really need to address that concern.

(I work on same team with Michal)

[grey]

Riot accounts are tied to different geographical regions, so I would expect the service doesn't have to span across them, http://leagueoflegends.wikia.com/wiki/Servers

[sanqui]

I realize the article isn't actually about a chat service, but I continue to be confused because there exists Riot[1], the messaging client of the Matrix network.

[1] https://riot.im/

[trqx]

Off-topic: do you guys at matrix/riot have a bot that automatically creates a ticket when it finds keywords such as {IRC|messaging|chat} in any hn thread without any mention of {riot.im|matrix.org} URL being also in the comments? Honest question.

Isn't there a publicly hosted database of hn comments so someone could make stats on this topic? I'm sure you are close to 100% match.

[hd4]

I thought the same thing.

[shuntress]

Edit: Upon more careful reading, it appears that a response is sent from the server to the client via WebSocket which contains the resource and method that the client can request using HTTP if it wishes to update.

Original question: Can someone explain to me why the JSON message sent to the REST interface seems to contain URL and Type as data?

Is it because they are transmitted using web sockets but the concepts of "resource" (ie: clubs/v1/clubs/665632A9-EF44-41CB-BF03-01F2BA533FE7) and "method" (ie: GET) still just make sense and happens to be very similar to HTTP?

[pjmlp]

Nice to see yet another game studio making use of Erlang.

[shoover]

I thought the post might shed more light on their use of Erlang for this service, but it focuses more on high level topology, AWS infrastructure, and load balancing. It's interesting to get a sense of the level of high level deployment tooling they used, because in a way Erlang marketing gives the impression that the language and VM are all you need.

[javitury]

This article reminds me of a previous HN post. The program discussed was called pushpin or pinpush.

These solutions are the glue between stateless and stateful services. I guess they will become increasingly important as some services specialize towards responsiveness. The use websockets or http2 without giving up the simplification provided by restful architecture.

[biot]

Pushpin: https://news.ycombinator.com/item?id=9005724

[je42]

wondering why they have the edge nodes and not just a load balancer. looking at their responsibilities it looks like a lot of the overlap with the responsibilities of a load balancer.

[dorfsmay]

Looks like a load issue, a single server can only take care of so many conn, auth, etc... So having several solves the problem and can scale horizontally easily, but each edge maintains conn information, so a given user needs to be routed to that same server every time. On the other hand, all LBs share a single address and all should be replaceable by each other, they exist purely to route the user to the right edge server (existing conn or least busy).

Question to OP: do you run your edge servers in pairs or some kind of cluster?

[Zekio]

Their JSON have an key called payload which saves a String rather than an Object, is there any benefits to doing this? since they also escape the " in the String

[marksomnian]

Probably that services aren't required to use JSON (since they use many different service languages, it's not unlikely that they use many different serialisation methods - JSON, protobuf, etc.)

[tcard]

Separation of concerns: the code handling the envelope doesn't need to know about the format of the payload. The JSON in that payload will only be encoded (on the sender) and decoded (on the recipient) once.

[nodivbyzero]

I'd like to see more info on their customized Erlang/OTP mnesia . michalptaszek mentioned it in the comment: http://disq.us/p/1gltwfs

[michalptaszek]

Sure, few words on customizations we've built into mnesia: - we integrated it with our eureka-based service discovery mechanism, so that it can automatically cluster with other servers that are spun up in a process of cluster bootstrap/resizing. Also relaxed constraints when merging 2 identical schemas of separate clusters (when table cookies don't match, but everything else matches we still want to merge and take a union of already existing data) - we've added a bunch of auto-merge code (heavily inspired by Ulf's wonderful https://github.com/uwiger/unsplit library) in case of network partitions - we've also added support for maintaining pools of processes for each table for dirty updates (as opposed of going through mnesia_tm for every single operation, including transactions as well as dirty_asyncs)

I'm 100% aware that these changes are RMS/Riot specific and won't work in many other situations (e.g. they violate certain transaction isolation properties).

[shoover]

Do people use zeromq for this kind of work? It seems the various socket types are tailored to building such load balancing and routing architectures, but I'm not aware if zeromq is in use at this scale.

[jkarneges]

Yup, we use ZeroMQ for internal routing for this kind of thing. Each instance of our edge server (Pushpin) binds on a SUB socket to receive messages destined for external clients. This makes it possible for internal publishers (using PUB) to route among edge servers using a brokerless mesh.