One of the authors here. This website is still a work in progress.
It's an ongoing effort to gather all the resources that will help you use Erlang in a business. The booksite is divided in three sections focusing particularly on Erlang/OTP’s higher level concepts in the current open source ecosystem, how to use it in production (while setting up a pipeline for continuous development and delivery), and how to build a team when you’re starting from scratch.
We're working mostly linearly in each section, but they're not all progressing at the same pace. We have nothing yet on "building a team" although our outlines are all planned; the "hard things to get right" chapter was supposed to come in later but we saw enough questions about these topics to rush it out.
We might release things from each section as they are ready and we see fit. In the meanwhile, we're always appreciating feedback and people checking in to see what's new in terms of content.
One of the things I've learned from many attempts to document internal processes at organizations, is that often the authors are least equipped to explain things.
They know too much, they often don't recognize how many pieces of assumed information they're injecting into their so-called explanations. You often get circular explanations in addition to circular reasoning and it can be off-putting.
A better use of time is to get someone who is just started to exhibit competence to write the docs. They won't assume the jargon, they can't assume deep knowledge, and because they're living right at the pain point they can be especially sympathetic. And if what they explain is dead wrong, this is your chance to correct a horrible misunderstanding before they get too far along. Pointing out the 'false friends' in a system early on can be pretty powerful. Educationally or as an opportunity for improving the system.
In other words, I'd encourage you to invite new people to contribute. Just keep an eye on what they do and be ready to jump in with corrections.
I think they deserve more thorough coverage because 1) you really need them in a lot of applications, and 2) they're not in OTP, so picking one (especially if you're not really an Erlang expert) and employing it in the right way are a bit trickier.
A simple example of an application that probably needs a circuit breaker is a web application that can keep running even if the database is not available, even if all it does is return an error message to the browser.
Just wanted to thank you and Tristan for the great resource. It's much appreciated!
The cheatsheet about what the processes do when they trap exits/are linked/are port of OTP is awesome. I never remember that very well and was looking for a handy reference to have. Now I have it.
Docker and Kubernetes references are great as those are often involved in today's development and operations, and everyone is probably wondering what's the best way to combine those with Erlang.
What are the arguments for adopting Erlang instead of Elixir for new projects in 2019? I understand there is a lot of existing Erlang code out there, but it has a reputation for being less approachable than something like Elixir (syntax and tooling).
A section on "stuff that's not in OTP, but many people end up using".
For instance, poolboy comes to mind, alongside the circuit breakers I mentioned in the other comment.
lager for logging.
At one point, there was a lot of stuff in the basho github repo that seemed pretty widely used.
It's been a bit since I've used Erlang so maybe there are some others that come to mind as "you're likely to see this included in a production Erlang system".
You could and probably should promote this to the Elixir community too. Especially the OTP parts are very relevant fundamental knowledge for any Elixir dev.
This might be on your radar already, but the rendering is broken in Firefox. I'm using version 60.8.0esr, which is what I get with "apt-get install firefox".
When I open the link I see the table of contents on the left and a bunch of empty white space. Scrolling down, the content eventually comes into view, but scrolls on top of the table of contents, making it impossible to click.
I'm blocking the google fonts site, so maybe that's part of the problem.
There seems to be a slight encoding issue with external URLs. In the dependencies section under "Private Hex Mirrors", there is a link to mini_repo which results in a 404. It seems like the underscore in mini_repo is encoded to %5f which github doesn't seem to like?
This is amazing, thanks for all of your hard work! It's very encouraging to see continual effort put into the Erlang ecosystem. I haven't used Erlang in around 5 years so this looks right up my alley for getting up to date with the latest best practices.
As an aside, it's been a little bittersweet seeing all of the attention and momentum going into Elixir. Yes it's great for BEAM and Erlang certainly benefits from those contributions as well. But there's just something so beautiful about Erlang, it deserves more attention than it gets. Thanks for keeping the dream alive!
As an Elixir dev for coming up on four (four! Jeez where has the time gone?) years, I’ve yet to really delve into Erlang. I can read the syntax well enough to figure out why my Hackney request is doing odd things with my HTTP headers etc..., but that’s about it. To all Erlangers out there, what advantages/niceties/powers do you have in Erlang that make you reach for the language over Elixir?
Mostly I just like it better. But otherwise in my day-to-day off the top of my head:
Less “magic” (fewer macros that may change semantics, straightforward config handling), focus on declarativeness, no variable re-binding, more powerful test framework (common_test), seamless Dialyzer support, better logging framework for structured logging, simpler/minimal syntax, personal preferences for tooling (until recently, direct support for releases was lacking in Elixir), and force of habit.
I've noticed there's a section in there about deploying with Docker. I know many other development communities swear by Docker, but a number of the OTP/BEAM devs I've spoken to seem unsure about it.
I also tend to believe Docker makes little sense, where OTP releases are the alternative.
What's the opinion here? This question only pertains to OTP development.
It's just a matter of using the right tool for the right job. There's a lot of things Docker can do that an OTP application already handles (and in a way that's better-integrated with your application). Same with an orchestration system (e.g. Kubernetes).
You can use Docker to provide a common/consistent base on which you'd build the nodes running your OTP applications. If you have zero non-OTP dependencies, then this is overkill (you'd be better off deploying directly to some server, be it a VM or bare-metal, and managing configuration/deployment on that level), but most real-world applications do have other Dockerizable dependencies (databases come to mind, though there's ongoing debate on whether or not putting Postgres in a Docker container is actually smart). Also, using Docker to abstract the runtime environment for BEAM does help considerably with deployment to various PaaS platforms (e.g. Heroku, Bluemix, Elastic Beanstalk, etc.).
Personally, I lean toward a couple big servers dedicated to BEAM instances (running all the OTP apps), and a couple big servers dedicated to some SQL database (usually Postgres). Whether you maintain those servers as Docker containers or EC2 instances or DO droplets or physical servers is an implementation detail that doesn't really matter that much as long as it works and it's easy for you (or your DevOps / sysadmin teams) to maintain.
On that note: I don't think it makes sense to have a bunch of little containerized microservices each running a separate BEAM and a separate set of OTP applications. OTP already takes care of that sort of orchestration, and OTP applications already are (or can be) microservices. Just throw 'em all into a big server and let BEAM's scheduler and OTP's supervision model do their jobs.
Hi, one of the authors here, the Docker chapter and soon to come Kubernetes chapter hopefully shed some light on this very question. They are not alternatives, docker and k8s complement Erlang.
It’s difficult to deploy Erlang apps in Erlang-style using Docker. The style I speak of is zero downtime deploys where the VM is never turned off, but is updated with new software as a living organism.
You can run BEAM apps in a container the same way you’d run any other app. It’s possible. But BEAM is less like a PHP/Python/Ruby etc interpreter and more like an abstraction for the OS (some might even consider it to be one). When you’re in Erlang you think in Erlang and the runtime abstracts things like processes, threads, and network partitions away... so you tend to want a big beast BEAM instance (or a few big beasts) rather than a lot of tiny ones (as one might scale up a horizontally scaled app like one on Heroku dynos).
It can be done. It’s just kinda going against the grain.
> I also tend to believe Docker makes little sense, where OTP releases are the alternative.
Docker makes a lot of sense, actually, because it allows you to run integration tests easily and ship Erlang releases in the same way as everything else. There are exceptions, of course. As for OTP releases, they are a good way intermediate step before packaging code to RPMs, but upgrades should be done the usual way: via blue-green deployment. Don't even argue with me :-) I happened to support a huge system that uses OTP upgrades, and the amount of complexity and tricky bugs, that live code loading introduces, is immense. No one is smart enough to do it :-)
> If you expect failure to happen on an external service, do not make its presence a guarantee of your system. We’re dealing with the real world here, and failure of external dependencies is always an option.
What is the use case for erlang given the current container technology and cloud providers? I mean many problems that erlang solved (and beautifully did it) can be dealt with docker and kubernetes
In a nutshell, docker and k8s are to OTP what region failover is to k8s tech. They operate on different layers of abstraction and impact distinct components.
OTP allows handling partial failures WITHIN an instance, something K8s can’t help with.
I agree that some of the problems that OTP solves can now be taken over by Docker/K8s if you modify your architecture a bit. I.e. if you only have a few processes running on BEAM it's likely you could replace them by a few containers with a supervisor that restarts containers with errors, though you would have to start thinking about how they communicate (HTTP?) and some other things. It wouldn't scale to the same extent either (you don't want 1000s of containers on a single node) and Erlang provides some other benefits as well (e.g. single interface to manage all your processes, consistency if all your services are in Erlang, efficient threading). I would say that the main use case is when the model of using 1000s+ of processes helps you build an easy to understand architecture, which can be in messaging, streaming, gaming, or other applications with many simultaneous users/clients.
[+] [-] mononcqc|6 years ago|reply
It's an ongoing effort to gather all the resources that will help you use Erlang in a business. The booksite is divided in three sections focusing particularly on Erlang/OTP’s higher level concepts in the current open source ecosystem, how to use it in production (while setting up a pipeline for continuous development and delivery), and how to build a team when you’re starting from scratch.
We're working mostly linearly in each section, but they're not all progressing at the same pace. We have nothing yet on "building a team" although our outlines are all planned; the "hard things to get right" chapter was supposed to come in later but we saw enough questions about these topics to rush it out.
We might release things from each section as they are ready and we see fit. In the meanwhile, we're always appreciating feedback and people checking in to see what's new in terms of content.
[+] [-] hinkley|6 years ago|reply
They know too much, they often don't recognize how many pieces of assumed information they're injecting into their so-called explanations. You often get circular explanations in addition to circular reasoning and it can be off-putting.
A better use of time is to get someone who is just started to exhibit competence to write the docs. They won't assume the jargon, they can't assume deep knowledge, and because they're living right at the pain point they can be especially sympathetic. And if what they explain is dead wrong, this is your chance to correct a horrible misunderstanding before they get too far along. Pointing out the 'false friends' in a system early on can be pretty powerful. Educationally or as an opportunity for improving the system.
In other words, I'd encourage you to invite new people to contribute. Just keep an eye on what they do and be ready to jump in with corrections.
[+] [-] davidw|6 years ago|reply
I think they deserve more thorough coverage because 1) you really need them in a lot of applications, and 2) they're not in OTP, so picking one (especially if you're not really an Erlang expert) and employing it in the right way are a bit trickier.
A simple example of an application that probably needs a circuit breaker is a web application that can keep running even if the database is not available, even if all it does is return an error message to the browser.
[+] [-] rdtsc|6 years ago|reply
The cheatsheet about what the processes do when they trap exits/are linked/are port of OTP is awesome. I never remember that very well and was looking for a handy reference to have. Now I have it.
Docker and Kubernetes references are great as those are often involved in today's development and operations, and everyone is probably wondering what's the best way to combine those with Erlang.
[+] [-] Driky|6 years ago|reply
[+] [-] cpeterso|6 years ago|reply
[+] [-] davidw|6 years ago|reply
A section on "stuff that's not in OTP, but many people end up using".
For instance, poolboy comes to mind, alongside the circuit breakers I mentioned in the other comment.
lager for logging.
At one point, there was a lot of stuff in the basho github repo that seemed pretty widely used.
It's been a bit since I've used Erlang so maybe there are some others that come to mind as "you're likely to see this included in a production Erlang system".
[+] [-] verttii|6 years ago|reply
[+] [-] jlarocco|6 years ago|reply
When I open the link I see the table of contents on the left and a bunch of empty white space. Scrolling down, the content eventually comes into view, but scrolls on top of the table of contents, making it impossible to click.
I'm blocking the google fonts site, so maybe that's part of the problem.
[+] [-] _randyr|6 years ago|reply
[+] [-] dxhdr|6 years ago|reply
As an aside, it's been a little bittersweet seeing all of the attention and momentum going into Elixir. Yes it's great for BEAM and Erlang certainly benefits from those contributions as well. But there's just something so beautiful about Erlang, it deserves more attention than it gets. Thanks for keeping the dream alive!
[+] [-] shijie|6 years ago|reply
[+] [-] mononcqc|6 years ago|reply
Less “magic” (fewer macros that may change semantics, straightforward config handling), focus on declarativeness, no variable re-binding, more powerful test framework (common_test), seamless Dialyzer support, better logging framework for structured logging, simpler/minimal syntax, personal preferences for tooling (until recently, direct support for releases was lacking in Elixir), and force of habit.
[+] [-] dynamite-ready|6 years ago|reply
I also tend to believe Docker makes little sense, where OTP releases are the alternative.
What's the opinion here? This question only pertains to OTP development.
[+] [-] yellowapple|6 years ago|reply
You can use Docker to provide a common/consistent base on which you'd build the nodes running your OTP applications. If you have zero non-OTP dependencies, then this is overkill (you'd be better off deploying directly to some server, be it a VM or bare-metal, and managing configuration/deployment on that level), but most real-world applications do have other Dockerizable dependencies (databases come to mind, though there's ongoing debate on whether or not putting Postgres in a Docker container is actually smart). Also, using Docker to abstract the runtime environment for BEAM does help considerably with deployment to various PaaS platforms (e.g. Heroku, Bluemix, Elastic Beanstalk, etc.).
Personally, I lean toward a couple big servers dedicated to BEAM instances (running all the OTP apps), and a couple big servers dedicated to some SQL database (usually Postgres). Whether you maintain those servers as Docker containers or EC2 instances or DO droplets or physical servers is an implementation detail that doesn't really matter that much as long as it works and it's easy for you (or your DevOps / sysadmin teams) to maintain.
On that note: I don't think it makes sense to have a bunch of little containerized microservices each running a separate BEAM and a separate set of OTP applications. OTP already takes care of that sort of orchestration, and OTP applications already are (or can be) microservices. Just throw 'em all into a big server and let BEAM's scheduler and OTP's supervision model do their jobs.
[+] [-] kungfooguru|6 years ago|reply
Jose Valim, creator of Elixir, also wrote about this recently http://blog.plataformatec.com.br/2019/10/kubernetes-and-the-...?
[+] [-] whalesalad|6 years ago|reply
You can run BEAM apps in a container the same way you’d run any other app. It’s possible. But BEAM is less like a PHP/Python/Ruby etc interpreter and more like an abstraction for the OS (some might even consider it to be one). When you’re in Erlang you think in Erlang and the runtime abstracts things like processes, threads, and network partitions away... so you tend to want a big beast BEAM instance (or a few big beasts) rather than a lot of tiny ones (as one might scale up a horizontally scaled app like one on Heroku dynos).
It can be done. It’s just kinda going against the grain.
[+] [-] h91wka|6 years ago|reply
Docker makes a lot of sense, actually, because it allows you to run integration tests easily and ship Erlang releases in the same way as everything else. There are exceptions, of course. As for OTP releases, they are a good way intermediate step before packaging code to RPMs, but upgrades should be done the usual way: via blue-green deployment. Don't even argue with me :-) I happened to support a huge system that uses OTP upgrades, and the amount of complexity and tricky bugs, that live code loading introduces, is immense. No one is smart enough to do it :-)
[+] [-] sb8244|6 years ago|reply
> If you expect failure to happen on an external service, do not make its presence a guarantee of your system. We’re dealing with the real world here, and failure of external dependencies is always an option.
[+] [-] btbuildem|6 years ago|reply
[+] [-] rsrsrs86|6 years ago|reply
[+] [-] mononcqc|6 years ago|reply
In a nutshell, docker and k8s are to OTP what region failover is to k8s tech. They operate on different layers of abstraction and impact distinct components.
OTP allows handling partial failures WITHIN an instance, something K8s can’t help with.
[+] [-] samvher|6 years ago|reply