Erlang/OTP 24 highlights

[travisgriggs]

> there are still 260k lines of code added and 320k lines removed

So a net reduction of 60K lines of code? And yet functionality was added to the system as well. That's praiseworthy IMO.

Imagine if "we did more with less" infected much of software development today.

[nullgeo]

Recently I read somewhere that writing long prose is easy but writing something succinct takes way longer. I think that translates to writing code as well.

[deeviant]

Imagine if we had a metric of "how easy is the code to understand and work with" rather than "can we do it with less lines of code".

[CRConrad]

> So a net reduction of 60K lines of code? And yet functionality was added to the system as well. That's praiseworthy IMO.

Yup.

> Imagine if "we did more with less" infected much of software development today.

Taken to its logical conclusion this should mean that the ultimate piece of software, that does everything and does it perfectly, is 0 LoC.

Hmm, fells like I need to check my logic here...

[rubyn00bie]

OMG, YES! The JIT is here... and holy crap, if you haven't tried it yet... it's awesome (everything feels snappier). I'm particularly stoked for the receive optimizations and process aliases.

BEAM just gets better and better. It's a good time to be an Erlanger/Elixirist...

[sivakon]

I just ran programs from computer language benchmarks with OTP24. Every program ran considerably faster almost 30% speed up.

[latch]

I was hoping our `mix test` would be faster, but it doesn't appear to be. It would be nice if this got some attention fro the Elixir team.

[the_duke]

It's noteworthy that the JIT doesn't (yet?) do runtime optimization or specialization, so gains should be moderate. The very low end of double digits.

Not comparable to going from a Javascript interpreter to v8.

But it's a great starting point.

[macintux]

Erlang has long, long needed better error messages. These changes look very welcome.

Overall this looks like quite a nice set of improvements. Kudos to the team.

[niek_pas]

The column number for errors and warnings is a welcome, long overdue addition.

[sgrytoyr]

Working with Elixir on a daily basis is mostly a pleasure, but those "ArgumentError" errors have been super annoying.

Looks like Elixir 1.12 will take full advantage of EEP 54:

https://github.com/elixir-lang/elixir/releases/tag/v1.12.0-r...

[losvedir]

Awesome, thanks for linking that! I was wondering how these improvements would flow through to Elixir.

Looks like we get the JIT performance improvements for free.

And unrelated to the Erlang/OTP changes, Elixir 1.12 looks awesome. Totally small, but such an unexpected little quality of life improvement, Kernel.then/2 for pipelines, looks great. I love the core team's focus on the UX of the language.

[semilattice]

Seems like there is less and less reason for not using Erlang for distributed coordinated compute broker tasks.

I have a stateless java backend servicing REST APIs. These backends are load-balanced by nginx.

Now the time is approaching where I need to introduce some form of global state (that involves global caching, message passing, registering/discovery of known workers, periodic (cron-like tasks), etc).

I would much prefer a single tool to add to my backend stack (currently Java + postgres) to cover most of the above needs.

With the performance improvement + persistent_term [1] -- in current Erlang, I basically get:

- a light weight distributed K/V cache

- a ZeroMQ like on-wire messaging system (built into erlang)

- discovery/coordination

- a distributed compute grid

- a way to write my own routines within that compute grid, and have them exposed via Java interface to my existing backend.

I do not need 'fastest' possible performance or least memory consumption. I just need them to be 'reasonable', 'known' and 'controllable' (not to exceed some baseline).

Erlang is just looking better and better (and I prefer its syntax to Elixir, for some reason.).

[1] https://erlang.org/doc/man/persistent_term.html

[pdimitar]

Do consider Elixir for its metaprogramming (macros) and the goodies that come with them, namely fantastic libraries and frameworks like Ecto and Phoenix.

[namelosw]

Nice update on the error message!

Speaking of error message. I love that Erlang gives the exact parameter values in the call stack. (I guess this is generally possible because immutable data structures are enforced in the whole language?) It saves a lot of time than just speculating with call stack only.

[exciteabletom]

I love the new error messages. Rust seems to have started that trend, Python also added better errors recently.

[steveklabnik]

Okay, lots of people arguing about this, so I'm gonna reply to you, the top-most person in this sub-thread :) I think the reason that it's easy to argue is that you can be talking about similar but slightly different things.

When Rust was created doesn't really matter, exactly. For a very long time, errors looked something like this:

  hello.rs:2:4: 2:16 error: unresolved name: print_with_unicorns
  hello.rs:2     print_with_unicorns("hello?");
                 ^~~~~~~~~~~~~~~~~~~

At some point, "improving the error messages" became a project goal, and Jonathan Turner decided to take this on. This is described in https://blog.rust-lang.org/2016/08/10/Shape-of-errors-to-com... . The post explicitly cites Elm as inspiration because Rust was directly inspired by Elm in this regard.

Yes, other languages may have started this trend earlier. Yes, maybe they influenced Elm, which influenced Rust. Yes, Rust may be "older" than Elm. But if you ask the people who began this effort, they will name Elm as the inspiration.

(And yeah, then you can try to argue about who has influenced the broader public, which is effectively impossible to prove, IMHO.)

Today that error looks like

  error[E0425]: cannot find function `print_with_unicorns` in this scope
   --> src/main.rs:2:5
    |
  2 |     print_with_unicorns("hello?");
    |     ^^^^^^^^^^^^^^^^^^^ not found in this scope

incidentally, and there's also JSON output, and if you're not on a terminal, you get the line numbers like before... lots of things are improved. This particular error doesn't show off some of the nicer things. Esteban Küber has taken up where Jonathan left off, and has been doing amazing work.

[bla3]

expressive diagnostics was one of clang's early selling points: https://clang.llvm.org/diagnostics.html

Clang was released 2007 and was usable 2009/2010-ish. Rust dev started 2010.

I'm not saying the trend of having good diagnostics was started by clang, but it's a more believable than the claim that it was started by Rust.

--

Rust-the-language is nice, but the Rust community feeling the need to mention Rust on every unrelated thread is a bit of a turn-off for me.

[lelanthran]

> Rust seems to have started that trend,

No it didn't. Lets not try to rewrite history here, clang started the trend of meaningful error messages, gcc quickly caught up.

[Skinney]

Rust was inspired by Elm, if I’m not much mistaken.

[niho]

Improved error messages is great. But one area specifically that is in desperate need of some love is the type errors from Dialyzer. They are rarely helpful to identify what is actually wrong in the code. Typically the error message will point to something several layers up or down in the call stack and prints out a huge blob of unreadable and unhelpful type signatures. Most of the time the best you can do is to simply compile and run the system to figure out what is wrong with the types. The obscurity of the type errors remind me of C++ template errors.

[di4na]

Part of the problem here is that dialyzer is a quite ... spaghetti codebase, that noone really want to fund that work and that noone really knows how this stuff work that is interested in doing that work.

There is a lot of research work to do and noone to pay for it. That said, if you know of a company interested to fund this work, i am interested and would love to hear more about it.

[AlchemistCamp]

I've been looking forward to the JIT for months. This is fantastic!

[olafura]

Years here

[mrslave]

Slightly off-topic from someone who hasn't touched Erlang in almost 20 years: if my app is distributed with Erlang/Elixir, I kind of feel like my choice of database should be something with excellent horizontal scaling too. Can someone offer some insight into the trends in 3rd-party tools like databases for Erlang applications?

[danpetrov]

Erlang already comes with a distributed DBMS called Mnesia https://erlang.org/doc/man/mnesia.html , which under the hood uses ETS/DETS depending on the configuration. In distributed Erlang projects you'll find that or abstractions over it. Mne sia makes the most sense in mu opinion when you only have 1-2 relations or just need some kind of distributed cache.

In Elixir apps you'll frequently find the aforementioned Ecto "ORM", which has adapters to different DBMSs like MySQL, PostgreSQL, and even Mnesia.

[derethanhausen]

Postgres? ;) In all seriousness though, I wouldn't really expect database needs for Erlang & Friends to be that different from other languages. My current employer has a vertically scaled AWS RDS Postgres as companion for an Elixir app and it's worked great. (Ecto in particular is an excellent ORM.) If your app truly needs zero downtime or ultra-low latency then there are other options.

[lawik]

I've had interesting exchanges with a guy who does work on CouchDB. That is built with Erlang and should scale quite well in what sounds like a resilient way. Haven't dug in.

Horizontal scaling of the DB seems like a problem in many architectures. For SQL I'd be looking at CockroachDB as it is Postgres-compatible and is built with scaling out in mind. Haven't tried it though. Most of my work hasn't needed that for the DB recently.

[spamizbad]

Probably CockroachDB (https://www.cockroachlabs.com/)? It's wire-compatible with postgres so any library with a postgres driver will work.

[yewenjie]

Elixir has the excellent Ecto package which talks to many SQL dbs, Postgres being the default.

[tiffanyh]

BeamASM vs HiPE?

Since BeamASM doesn't support HiPE - has anyone seen benchmarks of BeamASM (JIT) vs HiPE. I've searched and searched and can't find such analysis.

(Super excited the JIT work is seeing light after 10+ years)

[igouy]

fwiw elapsed seconds

    Erlang/OTP 23 [erts-11.1] [hipe]
    Erlang/OTP 24 [erts-12.0] [jit]

    binarytrees,hipe,1,7.531
    binarytrees,erlang,1,11.768

    binarytrees,hipe,2,4.172
    binarytrees,erlang,2,5.149

    fannkuchredux,hipe,1,59.079
    fannkuchredux,erlang,1,73.151

    fasta,hipe,1,57.006
    fasta,erlang,1,50.843

    fasta,erlang,2,20.209

    knucleotide,hipe,1,92.662
    knucleotide,erlang,1,80.360

    knucleotide,hipe,3,86.793
    knucleotide,erlang,3,70.949

    mandelbrot,hipe,1,118.623
    mandelbrot,erlang,1,48.756

    mandelbrot,hipe,2,101.211
    mandelbrot,erlang,2,46.154

    mandelbrot,hipe,3,87.793
    mandelbrot,erlang,3,44.633

    mandelbrot,hipe,4,84.878
    mandelbrot,erlang,4,44.976

    nbody,hipe,3,140.025
    nbody,erlang,3,100.630

    pidigits,hipe,1,8.791
    pidigits,erlang,1,8.008

    pidigits,hipe,2,8.515
    pidigits,erlang,2,8.673

    pidigits,hipe,3,7.935
    pidigits,erlang,3,7.748

    regexredux,hipe,6,42.757
    regexredux,erlang,40.402

    revcomp,hipe,1,25.622
    revcomp,erlang,1,23.070

    revcomp,hipe,3,188.990
    revcomp,erlang,3,155.024

    revcomp,hipe,4,122.507
    revcomp,erlang,4,106.456

    spectralnorm,hipe,1,92.347
    spectralnorm,erlang,1,62.519

    spectralnorm,hipe,2,11.176
    spectralnorm,erlang,2,11.460
    

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

[ergl]

HiPE support is getting wonky, I don't know if many people use it, and I'm not sure it supports the latest OTP versions.

The Erlang folks are looking for maintainer volunteers to continue working on HiPE support, they don't have the manpower right now to maintain it themselves.

[toast0]

I'm not sure if there's a point in the history where you can run with (this) JIT or with HiPE on the same commit. Which makes an apples to apples comparison difficult. If you compare HiPE on OTP 23 with JIT on OTP 24, you're also getting the large amount of other changes as well.

Both HiPE and this JIT have drastically different improvements depending on the specific code that's running, which makes it challenging to have a real world benchmark as well.

[ramchip]

There's a comment here: https://blog.erlang.org/the-road-to-the-jit/#maturing-the-ne...

I feel like I saw graphs in one of the presentations, but I don't recall which, or if it was about the final iteration of the JIT.

I suspect HiPE would beat the JIT on tight loops, but the JIT wins in general because of the lack of switching cost between native and interpreted code.

[ch4s3]

I believe HiPE is going away in the future.

[fredrikholm]

Always puts a smile on my face when I read these, the team does an amazing job. Cheers! :)

[btbuildem]

Sometimes I'll sit there, look at the error trying to decipher it, and I forget who I am and what I was doing.

Great work on the improvements!

[lpgauth]

I'm going to miss those `badarg` errors :D

[erinan]

I know you're being sarcastic but I will definitely not miss those when working with Elixir......

[ch4s3]

Seriously. I'm forever forgetting that Erlang functions called from Elixir often accept charlists as args.

[devoutsalsa]

Found the person who likes writing an in-house ETS wrapper on every new job!

[dang]

One past thread:

Erlang/OTP 24 Release Candidate 1 - https://news.ycombinator.com/item?id=26260127 - Feb 2021 (15 comments)

[truth_seeker]

Adoption of JIT on robust platform like BEAM is excellent news. Although i would wait for one or more release major releases till I see the significant improvement in computational aspect of the code.

[gideon13]

bring on the ARM support