top | item 31019234

Why Discord is switching from Go to Rust (2020)

77 points| hiena03 | 3 years ago |discord.com

59 comments

order

verdagon|3 years ago

I suspect that GC'd languages could mitigate this problem by introducing regions; separate areas of memory that cannot point at each other. Pony actors [0] have them, and Cone [1] and Vale [2] are trying new things with them.

If golang had this, then it might not ever need to run its GC because it could just fire up a new region for every request. The request will likely end and blast away its memory before it needs to collect, or it could choose to collect only when that particular goroutine/region is blocked.

Extra benefit: if there's an error in one region, we can blast it away and the rest of the program continues!

[0] https://tutorial.ponylang.io/types/actors.html#concurrent

[1] https://cone.jondgoodwin.com/fast.html

[2] https://verdagon.dev/blog/seamless-fearless-structured-concu...

jatone|3 years ago

or you know; just pace the GC mark and sweep algorithm. which is what go is doing now.

erikbye|3 years ago

Not discrediting Rust, but I've noticed you rarely hear "we improved performance" by rewriting our implementation using the same language... Although this, too, can yield similar performance improvements.

stonewareslord|3 years ago

In the article they describe their attempts at tuning and weren't happy with the results

abeltensor|3 years ago

The main issue with Go in this regard is that it generally doesn't have multiple ways of doing the same thing at least idiomatically. I mean sure, there are architectural choices you could make in a rewrite but the actual structure of the code itself is going to be very similar.

ceeplusplus|3 years ago

It does look like with some GC tuning (e.g. manually triggering GC's at a smaller interval than the Go automatic GC threshold) they might've mitigated the spikes, although I don't think they would have gotten the level of perf improvement they did. Golang assembly code IME is not very optimized compared to Rust/C++.

edit: reading comprehension skills are lacking, please see comment below for why I'm wrong

pfraze|3 years ago

I’m told the Go GC has gotten better in recent years. Has anybody run a similar program in Go lately that can confirm that?

phendrenad2|3 years ago

(2020)

(Anyone know if they're still using Rust?)

jhgg|3 years ago

Yes we are using rust in a big way. We have multiple teams now full time working on Rust. It is being used on both the client and server, as native modules, web assembly, and also native rust services and NIFs that embed themselves in our elixir services.

It has been an incredible success. I plan to blog more about it in the coming months. Our usage of Rust is continuing to grow, and if you check out our jobs page, you might notice all backend / infra jobs list Rust in them now :)

I think probably 40% of requests are handled directly by rust services now, with the rest involving one or more rust service called from our Python API layer.

eatonphil|3 years ago

Their blog doesn't list all articles on a single page (so you could ctrl-f) and doesn't have its own search and doesn't have it's own domain (so googleing `site:discord.com rust` returns a mix of Discord communities and blog posts).

Makes it pretty hard to find stuff!

mc4ndr3|3 years ago

How illuminating. From CloudFlare posts, I had been under the impression that Go's gc was incredibly unintrusive, near-real time performance for applications operating in increments of a few hundred milliseconds. For example, CloudFlare uses Go to analyze network traffic.

Yes, Rust provides a more predictable, faster memory management model than Go. At the expense of unpredictable, expensive memory leaks triggering application termination.

Curious how much time and effort was dedicated to improving gc, which is a useful endeavor in its own right.

bcrl|3 years ago

Personally, I prefer the choice of simply using a language that doesn't have GC. No GC, no GC tuning problem. Seems ideal!

That said, I've worked on several embedded systems, and the never allocate memory rule that most of them had for runtime was critical to maintaining real-time-like performance. One was written in C++, which meant that we basically couldn't make use of most of the stl and boost. We had to roll our own implementations of plenty of data structures used on the performance critical threads as a result. I couldn't imagine using a language with GC baked in for such a system. But the results spoke for themselves: microsecond level latencies and performance that scaled well with increased CPU core counts.

whoisburbansky|3 years ago

Where do you get that sense that Rust results in memory leaks? Is that just an assumption you’re making about languages without garbage collection, or are there examples you’re aware of Rust applications having to deal with runaway memory consumption?

steveklabnik|3 years ago

Cloudflare also uses a significant amount of Rust, in many business-critical places.

abeltensor|3 years ago

Hmm, well Go itself has gone through a ton of revisions where the GC saw large improvements. And other languages coming up are experimenting with various types of GCs. Its really up to the use case though.

With discord, I imagine a big reason why Rust was considered as an alternative to Go is because they already have a substantial Elixir codebase. Rust and Elixir have a very easy time communicating with one another via Erlang NIFs (native function interfaces). You can embed languages like C/C++/Rust into elixir without much overhead. While I've never personally tried do to such a thing with go, I can't imagine its a smooth experience. You'd probably need to use Ports or CNodes for Go simply for this reason.

I love go myself, but one of the biggest turn offs for the language is its FFI support for C and other C connected languages. CGo is a relatively expensive investment when compared to many other comparable alternatives and it should be avoided if possible.

kaba0|3 years ago

> At the expense of unpredictable, expensive memory leaks triggering application termination.

Rust’s borrow checker does defend against it as well for safe rust, unless you are doing something very stupid, this is just false.

alberth|3 years ago

Isn’t this a function of them being such heavy Erlang users and are writing NIFs (via Rustler) in Rust.

midrus|3 years ago

Or they just got bored and wanted to try some shinier toy. I've seen this happen dozens of time, all the bullshit for justifying it is just that, bullshit.

Not saying this is the case here but highly likely.

butterisgood|3 years ago

Well... is this still true? Go's had a lot of perf improvements in the last two years.

mohanmcgeek|3 years ago

It wasn't true even when they wrote the blog. Realistically this should read 2018 because apparently they waited two years before writing this blogpost.

Andys|3 years ago

There was a major performance boost in Go GC just after this happened

boxingrock|3 years ago

isn't the tldr on this that Go let them scale up for years before it became the bottleneck? a natural progression for any successful project...