We developed Online Charging System in Erlang that served couple million subscribers for close to three years. I found the whole experience fairly terrible.
Erlang is nice enough language and I don't mind the syntax but it's also kinda cumbersome and verbose at times. For example, adding `true -> ok` to every if statement gets old fast. Similarly, Erlang/OPT is a nice platform but some parts are fairly bad. Looking at you `inets/httpc`.
But the really big problem was the whole ecosystem. Build process was needlessly complicated. Surprisingly enough multicore support was not great. Performance was not that great. It seemed like a lot of libraries were abandoned after 2013. I could go on.
We ended up rewriting the whole thing in Java and, so far, it worked out great. And after Java 8, Java the language is not so bad. I still have somewhat found memories of Erlang but I don't miss it.
Not to refute your experience, but "adding true -> ok to every if statement" sounds like an anti-pattern. A couple of notes to elaborate:
1. If-statementes should rarely be used in Erlang because case is preferred
2. If you're returning ok everywhere, you could also just let it crash by doing something akin to true = function(). Always go for let it crash first, unless you really know you can handle the error sensibly (and then use a case-statement)
3. httpc is kind of known to not be the most formidable HTTP client. Hackney is a more modern and better choice. The good thing with httpc is that it is included with OTP, otherwise there are better choices (scalability- and feature-wise)
4. When someone says "multicore support" and "performance" support is not great it's usually one of two things: (a) they're developing a use case that is not fit for Erlang (e.g. compute heavy) or (b) they don't know how to use Erlang properly (e.g. too complex process setups)
Now, many of these things are not obvious to people new to Erlang and takes some experience to know about. This I would say is the bigger problem with the Erlang eco-system.
I can't refute your personal experience, but 10 years of Erlang work left me longing for more. It sounds as if perhaps Erlang was not used optimally, or, alternatively, best practices may not have been followed.
10 years ago the ecosystem was horrendous, and is much better now. Granted, the build experience could be better but I didn't find it that bad.
I find that people's experiences are relative to what they are accustomed to. Early years for me included compiling and linking C in Windows (segmented) and that was truly horrible!
Interesting, do you think Elixir would be better in your case? Also what was the amount of requests per second? Looks like you didn't really need Erlang's scalability.
If you use Java I'm assuming you are not using process. If so, I believe Phoenix is no worse than Java applications in terms of ease of use and scalability etc.
And it in terms of performance it indeed require some design experience on Erlang process. But it shouldn't be bad. After all WhatsApp and Discord have run on top of it with very few engineers to support massive customers before. I doubt online charging system could be more chatty.
Although not in Erlang, Discord uses Elixir which runs on the BEAM VM to much success as well for our real time distributed system as well as VOIP signaling components. Looking back years later - it’s safe to say we wouldn’t have chosen anything else.
Having worked extensively with both Erlang and Elixir in the last 7 years, the main advantage for me has always been the smaller code base that those projects tend to produce when compared to other languages. Especially for projects that have some long-running components to them the Erlang built-in features allow writing really short yet fault resistant systems.
The Supervisor system, the per process garbage collection (no stop the world), 'start_link', "Let it crash", and pattern matching are for me the superpowers of beam. Elixir is contributing a great package ecosystem, 'mix format', and better features for the new live web of websockets
At least some of those examples are highly misleading:
> There is no better example of Erlang’s reliability than the English National Health Service (NHS). [...] Using Riak (written in Erlang), the NHS has managed 99.999% availability for over five years.
From what I could find out, NHS's Spine2 is mostly Python, with "a bit" of Erlang and Javascript. They do use Riak, but they also use Redis, RabbitMQ, Python/Tornado, Ubuntu and lots of other components.
This sentence could have said "Using redis (written in C)", or "Using Tornado (written in Python)", and still be fully correct.
I have never programmed Erlang but it feels like it is the currently only language that is some kind of secret weapon. It has similar aura than Lisp had before that with Erlang you can do stuff beyond "normal" languages.
Awesome write-up, Francesco! To add to the success stories, our team's previous startup (Bugsense) was a huge user of Erlang, which allowed to scale to hundreds of thousands of concurrent devices with just a handful of servers [1]. It was a no brainer that we would now use Elixir for our current startup (AgentRisk [2,3]) and we wouldn't have been happier with our choice, especially paired with Phoenix, which is hands-down the best web framework we've ever worked with.
Fairly blind question: does Erlang|Elixir and BEAM make any sense in a pseudo-embedded, edge compute or IoT environment with constrained resources (CPU and memory), or even ARM architecture? Aside from the developer-side things, if AMQP were to the centralized communications hub (e.g. RabbitMQ) does using Nerves on devices make any performance benefit - meaning BEAM and app(s) there?
The intent would be pre-process data and stream inward, or simply be a command-and-control interface -> process and provide response.
They make the most sense if you want lots and lots of simultaneous clients. Plus it gives you a little bit of safety if you write crappy code.
Typically not suited for embedded.
(Assuming you mean embedded as in severely limited in resouces, and not "modern embedded" where you have lots and lots of unused system resources. If you mean the latter then anything goes.)
worked with Erlang lately in a side project, I think the actor model is really cool (and the FP style), but the error messages are not helpful at all (at least for a beginner), I wonder how other people manage.
I have been building grouper.ai to integrate voice ai systems with 2600hz's kazoo apis, and have enjoyed doing it as a Greenfield side project. Nerves for hardware on embedded Linux, c/c++ for the voice apis, ports in erlang/elixir to talk the low level calls, using elixir Phoenix in the cloud on the beam. Have used erlang also in day job for custom industrial M2M protocol translation (serial / Ethernet) very successfully also.
I have seen that Erlang is used in RabbitMQ, if I recall right their argument was that Erlang is particularily well suited for that kind of asynchonous message queue stuff. I donlt know all too much about Erlang, so I cannot judge if that claim makes sense.
I'm an Erlang novice, so take this with a grain of salt: Erlang's use in RMQ seems to have helped them immensely through the first few years of the system's life. They got an "all in one box" performant message broker off the ground, and relatively reliable, quite quickly.
However, RMQ did hit numerous issues that they had to work around on the Erlang platform. They had to engineer what was effectively their own scheduler on top of parts of the native Erlang one in order to prevent some starvation cases [1], and I have also heard that issues with Mnesia are at the root of some of the pathological behaviors (data loss) of some versions of RabbitMQ when restarting from a crash (this is admittedly anecdotal, so take it with a grain of salt).
My only exposure to Erlang has been through RabbitMQ and while I can't comment much on the language itself, managing the packages is a dependency nightmare. Like some twisted, mutant, supervillain version of Python, a given version of RabbitMQ will only work with a certain version of Erlang (with horrible, hard-to-understand error messages on incorrect versions), and on RedHat/CentOS, the Erlang packages all misadvertise what exact versions they supply, necessitating awkward forcing of the dependencies or else Yum screws everything up.
We're happily plugging along on KAZOO [0] since 2010 at 2600Hz. Erlang is a force-multiplier for us. Approaching 300K LOC and ~10 Erlang engineers (depends if Karl is writing code or CTO-ing that day) today.
Something I find interesting is that the tech stack is often chosen by non-engineers/founders. So languages and frameworks are marketed to business people. Freelance work is often weak on the requirements, but they have strong opinions on what language and framework to use.
If you really want to find yourself digging into a lot of different features in the Elixir ecosystem, you can’t go wrong with collaborative tools. If you really want to play with the latest hotness in Elixir, and dig into a challenge or two, I’d suggest building a light multi-user blogging system that leverages Phoenix LiveView for real-time Markdown->HTML previews, while allowing simultaneous editing/previewing by two users. Put a DB behind it so you can also play with Ecto. Outside of the 3 lines of JS you need to wire up LiveView, limit yourself to writing 0 additional lines of JS. Do everything in Elixir and eex/leex templates only. Since you’re focused on learning, skip dealing with user auth stuff. There will be some challenge to deal with handling multiple edits at the same time. May as well play with doing DB persistence in your LiveView, without any refresh. :)
Online gaming can be a good option—if building games is your thing. Personally, I find game-based demos/exercises with certain tools to be a bit too far removed from my daily work to help certain concepts really take root.
[+] [-] ihuk|6 years ago|reply
Erlang is nice enough language and I don't mind the syntax but it's also kinda cumbersome and verbose at times. For example, adding `true -> ok` to every if statement gets old fast. Similarly, Erlang/OPT is a nice platform but some parts are fairly bad. Looking at you `inets/httpc`.
But the really big problem was the whole ecosystem. Build process was needlessly complicated. Surprisingly enough multicore support was not great. Performance was not that great. It seemed like a lot of libraries were abandoned after 2013. I could go on.
We ended up rewriting the whole thing in Java and, so far, it worked out great. And after Java 8, Java the language is not so bad. I still have somewhat found memories of Erlang but I don't miss it.
[+] [-] eproxus|6 years ago|reply
1. If-statementes should rarely be used in Erlang because case is preferred
2. If you're returning ok everywhere, you could also just let it crash by doing something akin to true = function(). Always go for let it crash first, unless you really know you can handle the error sensibly (and then use a case-statement)
3. httpc is kind of known to not be the most formidable HTTP client. Hackney is a more modern and better choice. The good thing with httpc is that it is included with OTP, otherwise there are better choices (scalability- and feature-wise)
4. When someone says "multicore support" and "performance" support is not great it's usually one of two things: (a) they're developing a use case that is not fit for Erlang (e.g. compute heavy) or (b) they don't know how to use Erlang properly (e.g. too complex process setups)
Now, many of these things are not obvious to people new to Erlang and takes some experience to know about. This I would say is the bigger problem with the Erlang eco-system.
[+] [-] defined|6 years ago|reply
10 years ago the ecosystem was horrendous, and is much better now. Granted, the build experience could be better but I didn't find it that bad.
I find that people's experiences are relative to what they are accustomed to. Early years for me included compiling and linking C in Windows (segmented) and that was truly horrible!
[+] [-] sandGorgon|6 years ago|reply
it regularly tops performance benchmarks - https://www.techempower.com/benchmarks/#section=data-r18&hw=... and super simple heroku deployments - https://github.com/vert-x3/vertx-examples/tree/master/heroku...
Also Kotlin !! https://vertx.io/docs/vertx-core/kotlin/ (kotlin vertx on heroku: https://github.com/vert-x3/vertx-examples/tree/master/kotlin...)
[+] [-] mapcars|6 years ago|reply
[+] [-] cryptos|6 years ago|reply
[+] [-] namelosw|6 years ago|reply
And it in terms of performance it indeed require some design experience on Erlang process. But it shouldn't be bad. After all WhatsApp and Discord have run on top of it with very few engineers to support massive customers before. I doubt online charging system could be more chatty.
[+] [-] lelf|6 years ago|reply
Considering that it’s state of the art, you must have been doing it terribly wrong.
> adding `true -> ok` to every if statement
That confirms it.
[+] [-] jhgg|6 years ago|reply
[+] [-] faitswulff|6 years ago|reply
[+] [-] erk__|6 years ago|reply
[+] [-] uasm|6 years ago|reply
[deleted]
[+] [-] dominicl|6 years ago|reply
The Supervisor system, the per process garbage collection (no stop the world), 'start_link', "Let it crash", and pattern matching are for me the superpowers of beam. Elixir is contributing a great package ecosystem, 'mix format', and better features for the new live web of websockets
[+] [-] verttii|6 years ago|reply
[+] [-] theamk|6 years ago|reply
> There is no better example of Erlang’s reliability than the English National Health Service (NHS). [...] Using Riak (written in Erlang), the NHS has managed 99.999% availability for over five years.
From what I could find out, NHS's Spine2 is mostly Python, with "a bit" of Erlang and Javascript. They do use Riak, but they also use Redis, RabbitMQ, Python/Tornado, Ubuntu and lots of other components.
This sentence could have said "Using redis (written in C)", or "Using Tornado (written in Python)", and still be fully correct.
[+] [-] i_feel_great|6 years ago|reply
[+] [-] melicerte|6 years ago|reply
[+] [-] dmitriid|6 years ago|reply
- telecoms, as it was specifically designed for telecoms.
- "glue" in networked applications: to orchestrate, pass messages around and the like. See GitHub [1] or CoD [2] etc.
In a sense, chats are a mix of both. Erlang is rarely a good fit in pretty much anything else.
[1] https://www.infoq.com/interviews/erlang-and-github/
[2] https://www.erlang-factory.com/upload/presentations/395/Erla...
[+] [-] ailideex|6 years ago|reply
Really? I am not sure I trust this claim.
[+] [-] atupis|6 years ago|reply
[+] [-] MetalMASK|6 years ago|reply
[+] [-] aloukissas|6 years ago|reply
[1] http://www.erlang-factory.com/static/upload/media/1394234889...
[2] https://agentrisk.com
[3] https://blog.agentrisk.com
[+] [-] kitplummer|6 years ago|reply
The intent would be pre-process data and stream inward, or simply be a command-and-control interface -> process and provide response.
[+] [-] Globz|6 years ago|reply
[+] [-] derpherpsson|6 years ago|reply
Typically not suited for embedded.
(Assuming you mean embedded as in severely limited in resouces, and not "modern embedded" where you have lots and lots of unused system resources. If you mean the latter then anything goes.)
[+] [-] vasilakisfil|6 years ago|reply
[+] [-] tickbw|6 years ago|reply
[+] [-] atoav|6 years ago|reply
[+] [-] zbentley|6 years ago|reply
However, RMQ did hit numerous issues that they had to work around on the Erlang platform. They had to engineer what was effectively their own scheduler on top of parts of the native Erlang one in order to prevent some starvation cases [1], and I have also heard that issues with Mnesia are at the root of some of the pathological behaviors (data loss) of some versions of RabbitMQ when restarting from a crash (this is admittedly anecdotal, so take it with a grain of salt).
[1] http://alvaro-videla.com/2013/09/rabbitmq-internals-credit-f...
[+] [-] HeXetic|6 years ago|reply
[+] [-] not_a_cop75|6 years ago|reply
[+] [-] liveoneggs|6 years ago|reply
https://www.erlang-solutions.com/resources/download.html
Linux distros have always been terrible places to find reliable packages.
[+] [-] miki123211|6 years ago|reply
Just wondering if such a system used in Poland, actually called Elixir[1], has a line of Erlang/Elixir in its codebase.
[1] https://pl.wikipedia.org/wiki/Elixir_(system_rozliczeń_międz...
[+] [-] mc_|6 years ago|reply
[0] https://github.com/2600hz/kazoo/
[+] [-] latentpot|6 years ago|reply
[+] [-] z3t4|6 years ago|reply
[+] [-] aloer|6 years ago|reply
Online gaming?
Collaborative tools?
[+] [-] bobwaycott|6 years ago|reply
Online gaming can be a good option—if building games is your thing. Personally, I find game-based demos/exercises with certain tools to be a bit too far removed from my daily work to help certain concepts really take root.
[+] [-] kerkeslager|6 years ago|reply
[+] [-] chrisbro|6 years ago|reply
https://en.wikipedia.org/wiki/RabbitMQ
[+] [-] elbrujohalcon|6 years ago|reply
[+] [-] truth_seeker|6 years ago|reply
[+] [-] fastbmk|6 years ago|reply
1. Erlang/Elixir are less than 1% of programming languages market.
2. Erlang/Elixir popularity is trending down.
[+] [-] therockhead|6 years ago|reply
[+] [-] wprapido|6 years ago|reply