top | item 34002526

Lichess gets a big upgrade. It doesn't go as planned

478 points| _fizz_buzz_ | 3 years ago |lichess.org

213 comments

order
[+] jakub_g|3 years ago|reply
For those short on time: "It doesn't go as planned" is explained in the middle of the article:

> When everything compiled, I shipped it. And to everyone's surprise, apart from a few bugs I had created while rewriting thousands of lines of code... it worked. It just did. No explosions, no obscure bugs, no memory leak, no performance degradation. That was rather unexpected.

...followed by

> Then we saw the JVM CPU usage rise to alarming heights, with unusual patterns. And no obvious culprit in the thread dumps...

> it was just the JVM that needed some tuning.

Details: https://lichess.org/@/thibault/blog/lichess-on-scala3-help-n...

[+] lamontcg|3 years ago|reply
> "It doesn't go as planned"

bit of an editorialized clickbait title really.

[+] bravetraveler|3 years ago|reply
'if it builds it ships' in action, love to see it

edit: Since I got sniped pretty hard for not divulging further, allow me to 'kill the joke' by explaining. This will read more harsh than I intended

Shipping it isn't the problem. In fact, that's the expectation. Just... don't ship it to a place that matters first.

I feel baited by an otherwise cavalier operator, footguns be footguns.

as planned -- I'm not convinced there was a plan

[+] AuthorizedCust|3 years ago|reply
JVM tuning is frustrating. The need for that is generally not an issue on Microsoft .NET.
[+] ummonk|3 years ago|reply
I feel like “it didn’t go as anticipated” would have created the same suspense without being a lie.
[+] nicky0|3 years ago|reply
This whole subthread reads like an HN parody.
[+] warent|3 years ago|reply

[deleted]

[+] TulliusCicero|3 years ago|reply
> Improved type inference

> We want types, not boilerplate. Sometimes it's best to let the compiler figure out what things are by itself.

At first I thought type inference was cool too, but more and more I don't like it. I prefer being able to see a variable's type easily, even when skimming over code. Type inference makes code less readable to me.

[+] matsemann|3 years ago|reply
Type inference locally is great. Doing `val something = MyClass()` or `val something = someFunc()` . No ambiguity, just less typing and clutter. At bounds however (function input/output for instance) I prefer explicit types, less chance for mistakes and the signature acts as documentation to what the function does.
[+] vkou|3 years ago|reply
Type inference everywhere can make code less readable, but type inference when it's clearly the code author's intent that the type of the variable should not be of much interest to anyone improves readability.

A common example of this is a variable that's initialized in one line, fed into another function in the very next line, and then never used again. [1] If the function call that produces it, and the function call that consumes it is named well, the variable's type is often, but not always superfluous.

It's like naming for loop variables. Sometimes, you want a specific, detailed name for them. Other times, you just call it 'i', because what it is... Is not very important. It's a counter, counter goes up until we hit the end of a list, don't think about it too much.

[1] The two function calls could have been mashed together, but that would make the code less readable. The full variable type could have also been included, but its not very relevant to the intent of the code - which is simply[2] feeding the output of one function as an input into the other.

[2] Obviously, when contextual information (like real-world side effects of creating a particular variable - say, a thread, or a record...) that complicates the "Create -> Use" case is relevant, or if the APIs in question have a lot of function overloading, or poorly-named interfaces, or if non-trivial casting is going to take place, it may still be beneficial to explicitly provide the variable type.

[+] 63|3 years ago|reply
One of the things that's made me the most frustrated learning rust is all the type coercion and inferencing. Sometimes code will just work when I have the wrong mental model because types are coerced but then it ends up biting me later when suddenly everything breaks because it can't coerce anymore and I find out nothing was the type I thought it was. I wish there was a way to turn it off while I'm learning just so I can understand what's really happening.
[+] rubyist5eva|3 years ago|reply
My IDE can show me inferred types as an overlay in my editor (even in languages with dynamic types like Ruby, jetbrains inference is pretty good when the code is simple), perhaps look for an extension for your preferred editor?
[+] treis|3 years ago|reply
The problem is when you get things like:

var someLongThing <application.someHierarchy.someLongAssPackage.someLongThing> = new application.someHierarchy.someLongAssPackage.someLongThing()

[+] dkarl|3 years ago|reply
Explicit types are useful in some places. Type inference doesn't mean you can't use them, just that you don't have to spell them out everywhere even when they aren't useful.
[+] blt|3 years ago|reply
yeah, to be fair I only know it through the C++ `auto` keyword, but I dislike its overuse. It's great for weird templated iterator types but I prefer to see the concrete type if it's a reasonable length.
[+] Barrin92|3 years ago|reply
that's more an issue of editor integration. F# in Vscode for example overlays what types it has inferred automatically.
[+] spullara|3 years ago|reply
I try to avoid using it where the type isn't obvious from the context. But when it is, I like it a lot.
[+] nextaccountic|3 years ago|reply
> I prefer being able to see a variable's type easily, even when skimming over code. Type inference makes code less readable to me.

An IDE can still show the inferred types!

[+] jandrese|3 years ago|reply
Or the question you should always ask: What if the compiler gets it wrong? Is there a way for you to tell?
[+] winrid|3 years ago|reply
There's a lot of ignorance in this thread around the JVM, which is forgivable I suppose since it's not the hot new thing atm :)

The parameters they are tuning are available in any good VM with a JIT. You won't see these tuning knobs with Go because there's no JIT. These are trade offs. Scala is also known to create more garbage than Java.

The reason for the CPU spikes are code is getting compiled via JIT which takes CPU. If isn't enough space for JIT compiled code then you'll be recompiling code often, burning CPU.

This is similar to database cache tuning. Too little cache and you burn CPU and IOPS. Too much cache and you don't leave enough memory for other operations.

These parameters have good defaults but large apps can surpass them. It's important it's not automatic as this could have dire consequences (oom etc).

[+] kevincox|3 years ago|reply
> If isn't enough space for JIT compiled code then you'll be recompiling code often, burning CPU.

This sounds like a bug to me. If you are out of space for compiled code stop compiling. Or at least start slowly raising the threshold at which you recompile something so that the compilation cost is amortized over time.

If your recompiling is using more CPU than was being used before it started something is wrong. You should only be compiling code when it is expected to save CPU long-term.

[+] d2049|3 years ago|reply
> To be able to tell if Scala 3 itself is faster, we would have to rollback to Scala 2 and try it with the proper JVM tuning. I'm not willing to do that, sorry! Once you've tried Scala 3, there's no going back.

Pardon my ignorance. Why can't you set up a performance test in both environments and measure the speed?

[+] agilob|3 years ago|reply
Setting up reliable and reproducible performance environments is as much work as running good quality prod systems. It requires monitoring thing rarely available to monitor, extending prod metrics, creating snapshots, running repeatable tests on different scales. This isn't work for a single person. I'm in a team of 7 working to acquire next customer who will be on average 5x bigger than peaks of our current biggest customer. We've been designing, configuring and creating metrics on EKS since September, alongside training devs and developing performance engineering framework in k6. Lichess doesn't need that much effort and accuracy, but it's still lots of work. Did you know Fargate will schedule you Intel Skylake or Broadwell and there's even 30% difference in performance between them on certain tasks?
[+] toast0|3 years ago|reply
> Pardon my ignorance. Why can't you set up a performance test in both environments and measure the speed?

Everyone has a performance test environment, not everyone has one that's separate from production. In this case, generating realistic test patterns is probably very difficult, so you get one environment for everything.

[+] ARandumGuy|3 years ago|reply
I think it's a matter of effort. Many performance bottlenecks only really show up when you scale up to massive levels. While it's possible to replicate that on a test environment, it takes time and effort to set that up. While doing that is worth it for some applications, I can see why the Lichess developers didn't want to bother.
[+] typingmonkey|3 years ago|reply
That would only confirm that the new lichess version is faster, but not that scala3 is faster then scala2.
[+] cpleppert|3 years ago|reply
The JVM only runs out of codecache (usually) after a period of time running with a production load.
[+] zulban|3 years ago|reply
Of course they can. They've proven themselves very capable. The question is: where does this sit on their list of literally thousands of tasks to do?
[+] syastrov|3 years ago|reply
> That's where things got a bit hairy. Lila is built on Play Framework which is not yet ported to Scala 3.

> So I forked it and butchered it to remove everything we don't need - which is actually most of the framework.

I guess there is hope that Play framework itself will be migrated to Scala 3 and that the dependency on the fork can be removed, but this is taking on a risk - what if there are security updates to the upstream in the mean time?

[+] tasuki|3 years ago|reply
I think the plan might be to get rid of the play framework altogether and rather use a couple small, independent libraries to achieve the same.

The likelihood of getting security fixes in the future is about the same as getting new security holes created by whatever updates. Butchering away everything they didn't need certainly didn't harm security.

[+] 12345hn6789|3 years ago|reply
Isn't light bend more or less falling apart? I thought they announced they would no longer publish changes to play
[+] eppp|3 years ago|reply
But what was the JVM tuning? Thats the most interesting part!
[+] stareatgoats|3 years ago|reply
Got sucked into reading the article because of the "didn't go as planned" but it left my thirst for failure stories unquenched. Should have known.

That aside, that thing with opaque types in Scala sounds interesting - but how would the compiler know that the parameter was a UserId and not just a random string?

[+] jbm|3 years ago|reply
I truly wish I could get Opaque Types (Newtypes?) in Typescript. Great writeup!
[+] tareqak|3 years ago|reply
Is there an automatic refactoring that changes type-inferred variable declarations and changes then into typed declarations?

I can see the value when writing code in the early stages of a project to both have flexibility and save keystrokes with type inference to help prototype and explore the problem space. Later on, I can see the benefit of switching to typed variable declarations for readability as time goes on and that part of the code base slips away from my working memory.

[+] ajnin|3 years ago|reply
Interestingly the source code base does not seem to have any tests. I guess it speaks of the strength of Scala's type system or the rigorousness of the developer.

I tried Scala a long time ago but I didn't like it, it felt every student of Martin Odersky's CS course at the EPFL got to design their own language feature, and they often picked the same one but came up with different implementations.

[+] shmageggy|3 years ago|reply
Yeah I was kinda surprised how YOLO some of this was. For example

> At some point I had to rewrite the Glicko2 rating system from Java to Scala 3 [...] No-one noticed broken ratings, so I suppose it worked.

Sheesh. Seems like just the kind of thing for which a couple unit tests would be relatively easy to write and would add a lot of confidence.

[+] Kukumber|3 years ago|reply
The power of open source!

Congrats to lichess and everyone involved who helped!

[+] aiwv|3 years ago|reply
> All we need now is for some brave soul to improve Scala 3 support for treesitter,

Brave soul is an understatement.

[+] WhiteBlueSkies|3 years ago|reply
I wonder how node.js would have performed at that scale.
[+] philipwhiuk|3 years ago|reply
> Significant indentation and optional braces

And this is why it's annoying to use Scala. They just decided to rewrite the syntax of the language

[+] exabrial|3 years ago|reply
-1 for Type inference and `var`. It literally save you no time, increases cognitive load, and is an exposure point for future bugs.

Just use Explicit types. They aren't hard, time consuming, or bad. They are your friends.

[+] jdhdjdbdjdbd|3 years ago|reply
lol no? type inference is great u just suck at programming.