I still recall a CS jobs fair at my college ~8 years ago, where some company representative took a very bad liking to Erlang on my CV (I took a "Programming Languages" course where Erlang was one of ~6 languages we chewed through. s/o venkat!)
He kept asking "why use this over Java at my work X". He seemed to have the idea I wanted to join his company and rewrite their web administrative tools used by 5 people in Erlang. Not sure why he got so annoyed, and didn't ask me about anything other than a language he thought was silly!
In one CS job application in college I wrote in the list of skills "can write good prose". By this I meant both that I knew English, which was not obvious at that time and place, and that I can structure text logically, e.g. for documentation.
The interviewer kept mocking it and asking what did I mean by prose and whether I can also write in rhymes if needed, and ended up not making an offer. Twenty years later, good writing is still the most important skill in my career and the one that I rely on most often.
Seems to be a general knee-jerk reaction from typical CS-majors and other Java/C++ "bros" around Erlang. A more mature reflection could easily see lots of potential niches for Erlang, with the big caveat from a business perspective of the relative difficulty in finding devs. I had the pleasure to encounter Erlang professionally many many moons agos, in it's originally intended role within telecoms switching, and was nothing but impressed by it.
The argument here might be less about Erlang and more about Maslow's hammer - i.e., if all you've got is a hammer then all you can see are nails.
Maybe experience with language enthusiasts? When a language has fewer users, the users that it has are more likely to be enthusiasts.
Reminds about me I was talking to a company at a booth. They were using Java and up comes this guy who started professing that that was all wrong and they should rewrite their whole stack in Lisp.
I guess you’ll become allergic to minority programming languages if that happens often enough.
From my experience a lot of Java guys have no idea about the wider world of languages out there. They can be total experts in Java and get very confused why you are not writing something in Java.
The vast majority of programmers don't read HN, reddit or really follow any industry news. They work in Java or C# which are entirely self sufficient with very comprehensive SDKs.
My father worked at Ericsson for a long time back then, and while he wasn't involved in any of the AXE-N or Erlang stuff, he did hear water-cooler talk about the ongoing political battles. So this is like 3 layers removed from the people who were actually involved in it all, but anyway:
There was a huge push to move on from the in-house PLEX to industry standard software engineering. Object orientation, component architecture, UML, design patterns, etc.; think of a software engineering kool-aid from that time period, they were drinking it. And C++ was the industrial strength vehicle to deliver all of this.
AXE-N wasn't just a project that flopped, it was the crown jewel of this giant effort to switch to the new "industry standard" way to write telephony software. So the failure was a giant cultural shock.
But even after the AXE-N failure, the phalanx behind it weren't done for, and apparently it was they who managed to push through a ban on Erlang, that embarrassing thorn in their side.
As for what happened later, I don't know. Anybody who knows what language Ericsson is currently using to program their exchanges?
I think all modern Ericsson mobile network base stations use Erlang (at least for the configuration part) so most likely just using your phone you come in contact with an Erlang system everyday (depending on what gear your ISP use, but Ericsson is one of the largest providers of telecom equipment).
There were and are good reasons not to use Erlang. It's much (like an order of magnitude) harder to find developers who know it and almost impossible to find developers who can use it well (e.g know what a gen_server is). Much Erlang code I saw at Ericsson were technically written in Erlang but very much resembled imperative style code you'd find in a Java project. Erlang is also not very good at interacting with the surrounding system which is pretty important for a SoC.
I left Ericsson many years ago, but I bet that they are still using ClearCase for version control, FrameMaker for documentation, and that there are still five managers for every developer. Pretty sure they are still drinking the kool-aid, trying to put AI in their exchanges glued together with TypeScript.
I love erlang so much. It's just such a bizarre combination of things that should't mix as well as they do. Pattern match as assignment still feels futuristic; I know they didn't invent that but it must have taken serious vision & confidence to commit to it. Plus agents as the core abstraction, immutability semantics.
It all feels very CS-theory elegant and then you get to OTP, the most grimy roughneck Système D ass standard library I've ever seen. But that has no actual problems anywhere in my experience other than that you'll go years not knowing something is in there bc you didn't guess what five letter abbreviation it would be under.
Plus the syntax is like what you would get if you described a programming language to someone who had never seen one. Code from another planet. Flawless.
For confused fellows: In the context of Erlang, OTP stands for "Open Telecom Platform."
The key components of OTP include:
OTP Behaviors: Behaviors are generic programming frameworks that define the structure and behavior of a particular type of process. OTP provides several predefined behaviors, such as gen_server, gen_fsm, and gen_event, which encapsulate common patterns of communication and state handling.
Supervision Trees: OTP introduces a hierarchical structure called a supervision tree, which allows for the supervision and management of processes. Supervision trees help handle process failures and enable the system to automatically restart failed processes or take appropriate actions in response to failures.
Application Framework: OTP provides an application framework that facilitates the organization and configuration of Erlang applications. The framework allows developers to define dependencies between applications, manage application startup and shutdown, and handle system-wide configuration.
OTP Design Principles: OTP promotes the use of certain design principles, such as "Let It Crash" and "Fail Fast," which encourage building fault-tolerant systems that can recover from failures quickly and gracefully.
you'd think that an article about how facebook bought the erlang-implemented whatsapp because facebook messenger was less popular would bother to mention that facebook messenger used bob ippolito's mochi media's mochiweb, an erlang web server, to handle the comet connections that allowed facebook messenger to deliver asynchronous notifications to web browsers
that is, facebook messenger was also to a significant extent written in erlang, though i don't have any insight into how much of the internal facebook messenger system was erlang-based (i imagine probably not that much)
also why does the article say
> In 2022, it received a notable update that speeded up the execution of Erlang programs significantly through the introduction of a ‘just in time’ compiler.
and not mention that the hipe jit compiler was added in 02000, 22 years earlier, with x86 support added the next year? there have literally been jit compilers for erlang since last millennium. https://www.researchgate.net/publication/225129672_The_HiPEx...
There was a Commercial Uses of Functional Programming workshop talk back in 2009 where the engineers behind Facebook Chat talked about their use of Erlang.
>Then suddenly, in 1998, Erlang was banned within the Ericsson. The reason? To avoid the costs of development of a language that was unique to Ericsson, and instead to access the shared effort being put into the development of other languages.
Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS).
> Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS).
AFAIU, nothing to do with FOSS, which wasn't really on the radar for a somewhat conservative telecom company at the time. It was about custom in-house stuff (PLEX, Erlang, etc.) vs. "industry standard" (C++, OO, UML, etc.).
Motorola Labs had a similar-ish internal programming language around this time. It also died because devs didn’t want to use an internally-developed language. The important things were hiring, training, support, external tools, etc. Being a “good” language was at the bottom of the list. They went with C++. Even though I worked on that PL I knew they were right not to bet a giant business on a bunch of researchers.
I didn't know him personally, but I loved how he came across to me as both a cantankerous 'old' man AND an infectiously 'young' enthusiast in his talks.
>Like WhatsApp, Facebook Messenger had used Erlang and ejabberd but moved away from it after finding it difficult to recruit engineers.
Since WhatsApp handled much more users with just 35 engineers, I would have thought it wouldn't be so much of a deal to recruit some Erlang engineers. They didn't need to recruit thousands of engineers.
Facebook Messenger moved to a C++ server, built like the rest of Facebook C++ servers.
The story I heard was they were looking for people who already knew Erlang and were having trouble finding them, so they rewrote the thing in a language that they could hire for.
From my recollection, when I was at WhatsApp, we only hired two server engineers who had used Erlang before. I was considered knowledgable about Erlang, because I vaguely remembered seeing the Slashdot post when Ericsson open sourced it. Pretty much we hired smart people and gave them an Erlang book. As a result, I don't think we did OTP applications properly, and we might have missed out on a couple other things, and our code might not be very idiomatic always, but I think we did ok.
But, beyond hiring, Erlang is a weird fit inside Facebook's datacenter infrastructure. The Messenger server team was (and I believe still is) very small, and rewriting into something with less impedence mismatch for a small team to deal with probably made sense. For WhatsApp, we spent quite a lot of effort making our software fit at Facebook, and we had to give up FreeBSD and from what I've heard, hotloading, too.
I think the assumption Sun's Java could become something that eclipsed most languages including Erlang is over, as the prototype distributed JVMs just became another navel-gazing exercise.
The Erlang/Elixir OTP has design constraints like any other solution, but has survived 4+ business cycles. One can only hope Julia ends up with something similar, but more user friendly.
Distributed systems is hard, and training obstinate people is like trying to teach a Raccoon TCL. If a design really needs Erlang/Elixir, than there are few alternatives that won't degenerate into a polyglot dumpster-fire.
In practice, hot code reloading and mnesia are pretty cool concepts but very far from trivial. The vm and OTP provide mechanisms to work with such advanced concepts, but it still requires an intense amount of discipline and process to execute effectively.
Consider hot code reloading to be like changing the tyres on a moving car, the pieces and tools are there to make it possible, but in almost every situation it's more sensible to stop and change. Backwards compatibility for all API changes, code correctness and deployment methodology need to be fixed before you can start a hot rollout. That's not even to consider a flexible application architecture that will be compatible with such changes in functionality (OTP does indeed ensure this is compatible on the basic process level, but there is no magic bullet around high level architecture).
For given hard requirements, e.g. upgrading the application without dropping TCP connections - this can be a killer feature that removes considerable development effort, but usually it's a very cool thing that you don't need.
That being said, hot code reloading on the individual module level I have found to be a useful and dirty trick to change the behaviour of running applications (add log statements / telemetry where they weren't before to diagnose production issues). This still has all the caveats and danger of the above, but is more limited in scope.
These build on the facilities you're quoting: Phoenix uses hot reload to enable smooth local development, Phoenix pub/sub builds on OTP distribution, Docker builds use OTP releases, ETS tables are everywhere...
Mnesia is the only major piece that I agree is left out, for good reasons (it's fine software but really not built for cloud deployments).
Aside from the main article, I find it very convincing marketing strategy to announce a continuation of the subject in next chapter that will be paid. I could already see the quality of the authors and I feel like I want to pay now to read more. I'm all in for this kind of marketing.
[+] [-] hoten|2 years ago|reply
He kept asking "why use this over Java at my work X". He seemed to have the idea I wanted to join his company and rewrite their web administrative tools used by 5 people in Erlang. Not sure why he got so annoyed, and didn't ask me about anything other than a language he thought was silly!
[+] [-] throw_pm23|2 years ago|reply
The interviewer kept mocking it and asking what did I mean by prose and whether I can also write in rhymes if needed, and ended up not making an offer. Twenty years later, good writing is still the most important skill in my career and the one that I rely on most often.
[+] [-] Simon_O_Rourke|2 years ago|reply
The argument here might be less about Erlang and more about Maslow's hammer - i.e., if all you've got is a hammer then all you can see are nails.
[+] [-] microtonal|2 years ago|reply
Reminds about me I was talking to a company at a booth. They were using Java and up comes this guy who started professing that that was all wrong and they should rewrite their whole stack in Lisp.
I guess you’ll become allergic to minority programming languages if that happens often enough.
[+] [-] rapsey|2 years ago|reply
The vast majority of programmers don't read HN, reddit or really follow any industry news. They work in Java or C# which are entirely self sufficient with very comprehensive SDKs.
[+] [-] MattPalmer1086|2 years ago|reply
[+] [-] vbezhenar|2 years ago|reply
It's a real problem. So after few bad encounters businesses might prefer to hire boring people.
[+] [-] ricardobayes|2 years ago|reply
[+] [-] kramerger|2 years ago|reply
To be fair, this is exactly how I would react if Haskell is the main language on someones CV.
Disclaimer: Im okay with Erlang but hate Haskell and ML derivatives
[+] [-] bombolo|2 years ago|reply
That's what people do with rust! Not erlang!
[+] [-] jabl|2 years ago|reply
There was a huge push to move on from the in-house PLEX to industry standard software engineering. Object orientation, component architecture, UML, design patterns, etc.; think of a software engineering kool-aid from that time period, they were drinking it. And C++ was the industrial strength vehicle to deliver all of this.
AXE-N wasn't just a project that flopped, it was the crown jewel of this giant effort to switch to the new "industry standard" way to write telephony software. So the failure was a giant cultural shock.
But even after the AXE-N failure, the phalanx behind it weren't done for, and apparently it was they who managed to push through a ban on Erlang, that embarrassing thorn in their side.
As for what happened later, I don't know. Anybody who knows what language Ericsson is currently using to program their exchanges?
[+] [-] swinglock|2 years ago|reply
Open sourcing Erlang and it finding some success outside, meant it's no longer an in-house only language.
[+] [-] eproxus|2 years ago|reply
[+] [-] ojl|2 years ago|reply
[+] [-] bjourne|2 years ago|reply
I left Ericsson many years ago, but I bet that they are still using ClearCase for version control, FrameMaker for documentation, and that there are still five managers for every developer. Pretty sure they are still drinking the kool-aid, trying to put AI in their exchanges glued together with TypeScript.
[+] [-] jimmySixDOF|2 years ago|reply
https://news.ycombinator.com/item?id=36546954
[+] [-] giraffe_lady|2 years ago|reply
It all feels very CS-theory elegant and then you get to OTP, the most grimy roughneck Système D ass standard library I've ever seen. But that has no actual problems anywhere in my experience other than that you'll go years not knowing something is in there bc you didn't guess what five letter abbreviation it would be under.
Plus the syntax is like what you would get if you described a programming language to someone who had never seen one. Code from another planet. Flawless.
[+] [-] bruce343434|2 years ago|reply
The key components of OTP include:
[+] [-] kragen|2 years ago|reply
that is, facebook messenger was also to a significant extent written in erlang, though i don't have any insight into how much of the internal facebook messenger system was erlang-based (i imagine probably not that much)
also why does the article say
> In 2022, it received a notable update that speeded up the execution of Erlang programs significantly through the introduction of a ‘just in time’ compiler.
and not mention that the hipe jit compiler was added in 02000, 22 years earlier, with x86 support added the next year? there have literally been jit compilers for erlang since last millennium. https://www.researchgate.net/publication/225129672_The_HiPEx...
[+] [-] foldr|2 years ago|reply
It’s notable because previous JITs were not very successful at increasing performance:
https://www.erlang.org/blog/the-road-to-the-jit/
[+] [-] avsm|2 years ago|reply
http://cufp.org/2009/functional-programming-facebook.html
The abstract covers the pros and cons behind using Erlang back then pretty well.
[+] [-] klelatti|2 years ago|reply
The article does say that that Facebook Messenger used Erlang.
I've amended the comment on the jit compiler to say 'updated jit compiler' - my mistake.
[+] [-] troupo|2 years ago|reply
[+] [-] coldtea|2 years ago|reply
Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS).
[+] [-] jabl|2 years ago|reply
AFAIU, nothing to do with FOSS, which wasn't really on the radar for a somewhat conservative telecom company at the time. It was about custom in-house stuff (PLEX, Erlang, etc.) vs. "industry standard" (C++, OO, UML, etc.).
See also my other comment https://news.ycombinator.com/item?id=36570765
[+] [-] silverlake|2 years ago|reply
[+] [-] qxfys|2 years ago|reply
[+] [-] mercer|2 years ago|reply
[+] [-] DeathArrow|2 years ago|reply
Since WhatsApp handled much more users with just 35 engineers, I would have thought it wouldn't be so much of a deal to recruit some Erlang engineers. They didn't need to recruit thousands of engineers.
What platform have Facebook moved Messenger to?
[+] [-] toast0|2 years ago|reply
The story I heard was they were looking for people who already knew Erlang and were having trouble finding them, so they rewrote the thing in a language that they could hire for.
From my recollection, when I was at WhatsApp, we only hired two server engineers who had used Erlang before. I was considered knowledgable about Erlang, because I vaguely remembered seeing the Slashdot post when Ericsson open sourced it. Pretty much we hired smart people and gave them an Erlang book. As a result, I don't think we did OTP applications properly, and we might have missed out on a couple other things, and our code might not be very idiomatic always, but I think we did ok.
But, beyond hiring, Erlang is a weird fit inside Facebook's datacenter infrastructure. The Messenger server team was (and I believe still is) very small, and rewriting into something with less impedence mismatch for a small team to deal with probably made sense. For WhatsApp, we spent quite a lot of effort making our software fit at Facebook, and we had to give up FreeBSD and from what I've heard, hotloading, too.
[+] [-] Joel_Mckay|2 years ago|reply
The Erlang/Elixir OTP has design constraints like any other solution, but has survived 4+ business cycles. One can only hope Julia ends up with something similar, but more user friendly.
Distributed systems is hard, and training obstinate people is like trying to teach a Raccoon TCL. If a design really needs Erlang/Elixir, than there are few alternatives that won't degenerate into a polyglot dumpster-fire.
Happy computing =)
[+] [-] amadeuspagel|2 years ago|reply
> OTP or the ‘Open Telecoms Platform’ provided capabilities such as ‘hot code reloading’, databases (he ‘Mnesia’ package) and many other facilities.
And yet most erlangvm users use it with phoenix as a better rails, deploying with git and docker, attaching a postgres database.
[+] [-] rkachowski|2 years ago|reply
Consider hot code reloading to be like changing the tyres on a moving car, the pieces and tools are there to make it possible, but in almost every situation it's more sensible to stop and change. Backwards compatibility for all API changes, code correctness and deployment methodology need to be fixed before you can start a hot rollout. That's not even to consider a flexible application architecture that will be compatible with such changes in functionality (OTP does indeed ensure this is compatible on the basic process level, but there is no magic bullet around high level architecture).
For given hard requirements, e.g. upgrading the application without dropping TCP connections - this can be a killer feature that removes considerable development effort, but usually it's a very cool thing that you don't need.
That being said, hot code reloading on the individual module level I have found to be a useful and dirty trick to change the behaviour of running applications (add log statements / telemetry where they weren't before to diagnose production issues). This still has all the caveats and danger of the above, but is more limited in scope.
[+] [-] ramchip|2 years ago|reply
Mnesia is the only major piece that I agree is left out, for good reasons (it's fine software but really not built for cloud deployments).
[+] [-] tru1ock|2 years ago|reply
[+] [-] barkerja|2 years ago|reply
[+] [-] jarek83|2 years ago|reply
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] amelius|2 years ago|reply
[+] [-] Rovanion|2 years ago|reply