In my experience, simple concurrency is easy. Confused concurrency is exceptionally hard.
I think about this in the same way I think about C++. C++ is quite nice, if your teams limit themselves to a reasonable subset of the language. But it can become an absolute monster of cumbersome complexity, if these guardrails go away.
Concurrency feels the same. If you take a certain view on concurrency, which is "do it as simply as possible, don't break certain rules, don't try to be too clever," I find it fairly straightforward to reason about.
At the same time, 9 out of 10 of the hardest things I've debugged have always been nasty, awful concurrency bugs. The reason it was so hard? The concurrency was complicated, interwoven, assumption-laden, and absolutely impossible for anyone to reason about. Implicit contracts between calls required entire whiteboards of non-trivial state to understand, and even then, with all that work, it was a needle in a haystack.
I guess I would just say: It doesn't have to be hard. It's only hard if you don't respect it from the beginning.
I recently wrote some of the type of code that you would be upset to read, so maybe someone can sympathize with this a bit.
The task at hand wasn’t easy at all, it required both:
1. Making numerous network requests (some endpoints support batching, so do that as much as possible)
2. The entire main loop needs to be as fast as reasonably possible
The only real way to satisfy both, given that serially executing the network activity would waste tons of time, was some fairly tricky concurrent logic. I ended up using a few queue structures to collect day before making batch network calls, and stitching together all of the results at the end (block on all work threads, collect, report, then start next iteration)
Every step along the way made sense, every addition of complexity felt justified. And the first person to come to me asking how the fuck it got so complicated is frustrated, so my knee jerk reaction is to get frustrated back saying well how else could we satisfy complex requirements with a complex solution.
We should always be challenging ourselves and others to keep solutions simple, but remember that the simplest solution may itself still be complex.
One needs to pick the concurrency model and design up front. Document it before writing a single line of code. Then stick to the plan. It takes discipline, but it's not difficult.
I mostly write in C with pthreads and as long as one is willing to apply discipline to the task, there's no reason to fear concurrency.
If one starts adding a thread here and a mutex there and some shared structures over there without any prior plan then yes, absolutely, that leads to a world of hurt and a deep hole that is very difficult to get out of. So, don't do that.
> "do it as simply as possible, don't break certain rules, don't try to be too clever,"
This has to be a universal rule of programming. It's equally true for Ruby and Perl, which are extremely effective languages, but devolve quickly because of their uncanny ability to power cleverness. It's the whole premise of Go, by being boring by design.
Complex is easy, simple is hard, but it seems mostly because we fail to be able to resist the allure of cleverness.
You just sum up the entire raison d'être of reactjs and the stores ecosystem around it: force all the team to "do it as simply as possible, don't break certain rules, don't try to be too clever".
It's not that react, redux and cie are something new or disruptive. It just enforces a set of design rules that turns a lot of concurrent inputs and outputs into a linear workflow.
This is incredibly vague. There is no "set of rules" that just magically works without thought.
There is just SOOOO much that can go wrong. But one general rule crystallizes itself pretty quickly:
DO NOT EVER MUTATE STATE!
I.e. when you write concurrent code that builds on state mutation, you are opening Pandorra's box. If you want concurrency that people can understand and maintain, write immutable, functionally pure code (same data in, same result out, no exceptions). That is relatively easy to police and enforce, while still flexible enough to solve most use cases. It may however, not be the most performant.
I've written about it elsewhere, but I think that concurrency is easier if you really stick to a methodology.
Fork-join is the simplest methodology, and supported in a very wide number of languages. In fact, its so stupidly easy, you probably are going to over-think it the first time you use it.
1. Fork when you need concurrency (aka: speed)
2. Join when you need consistency
That's it. Don't write mutexes, don't write barriers. Just join. That's all you need: if you ever need "synchronized" access, do it in the "single" or "master" thread, when all other threads are joined.
You _cannot_ have a race-condition if you only have one thread. Joins ensure that only one thread is running. What else do you want?
Fork-join parallelism has relatively low utilization. Its not the fastest approach, but its the "easiest" approach in my eyes.
------------
The #1 issue with parallelism books, in my eyes, is that we start with mutexes and threads, which are a very hard way to do parallelism.
I argue instead: beginners should start with easy (but inefficient) forms of parallelism, and then work their way towards complicated tools as performance becomes more-and-more of a consideration.
and then there was a follow up to it saying the measurements were wrong IIRC, and then a further article showing, more or less that getting your own mutex replacement correct is so tricky that for 99.99% of programmers, just don't bother.
I would suggest a change in wording. Concurrency is not hard, its asynchronous updates that are hard. That's why digital logic circuits use a clock, and it's also why mixing logic into event driven user interfaces eventually gets out of hand.
The best way to make concurrency easier is to avoid it in the first place. This requires you to think about how correct your data really needs to be.
For example, a bank balance. Most people would assume that bank balances must always be correct. But this isn't really true. Back in the days when people wrote checks, your bank balance was almost never accurate -- it was on you to keep a local register and know your balance, and if you shared the account with someone else, to reconcile with them frequently. ATMs also use eventual consistency. When you take out $300 from the ATM, it attempts to update the balance, but it will let you take the money out if the bank is unavailable. Because the bank is willing to risk $300 that you aren't overdrawn. In most cases the "eventual" part is nearly instantaneous so you don't notice, but it doesn't actually have to be.
So the number one way to make concurrency easy is to avoid it altogether.
One of the unfortunate consequences of Go "making concurrency easy" is that many more concurrent programs are now being written by amateurs, with predictable results. This feels like a Hickey-an "simple vs. easy" dichotomy -- what would a language look like that tried to make concurrency simple, rather than easy?
Delayed syncing is not the same as "avoiding concurrency altogether" or having data be 'less correct'. Those aren't even simplifications for relaxing the latency of synchronization, they aren't correct descriptions at all.
I know this is an excerpt, but there are a lot more reasons that concurrency is hard. One of the biggest ones is that your concurrency can look like it works, until it doesn't. Made worse by the fact that real-world concurrency can be hard to test locally.
I will have to agree with this. Until I started working with Erlang/Elixir, I would probably have agreed with the OP article instead. Or at least its title, for reading about concurrency problems has become rather obsolete (for me).
For years, I certainly made my own share of concurrency horrors, in everything from C and Python to Java and JS. Eventually I (sort of) got it right in each, mostly by building up experience in approaching it the right way. But it always remained a source of easy to make mistakes (and sometimes near-impossible to solve complexities, with larger systems).
However, since working with Erlang/Elixir, I sometimes wonder what the F#$& I was doing all that time. Many ideas were certainly a bit challenging to wrap my head around at first, but once they clicked there (kind of) was no way back.
This will probably sounds arrogant, and it's okay with me if I get down-voted because of it. But the more I work with Erlang/Elixir (and a few others in the FP field), the more it becomes clear to me how much the languages I used to program in (and their typical challenges), now most of all look like a (rather sick) joke to me. Again, I know this might sound pretentious, if not offensive. But it is sincerely what I feel. In my defense: I'm not some kind of hipster, still wet behind the ears. I've been programming for nearly 3 decades and in well over a dozen different language. I'm one of those kids that were ripping apart assembly code of "protected" games and desktop software, and hacked around on micro-controllers, during the nineties.
While Erlang/Elixir may have shortcomings too (although I personally have not found any), it is one of the best things I've encountered for a long, long time. I'm also charmed by ClojureScript, but for totally different reasons. However, I do like it for when I'm forced to build something more complex for browser/front-end. But that's a whole different story, and not much relevant to this concurrency "issue".
Concurrency is hard because we invented a bunch of theoretical machinery (semaphores, mutexes, condition variables,...) and embedded it in languages such that most of the programs one can express are bad: racey or deadlock prone. Compare with modern sequential languages which have near-perfect denotational semantics: anything which compiles means something.
If you want practical guidance, my best is to avoid all of the concurrency constructs that are taught in schools. Use log-structured single-writer multiple reader, wait-free, processes and shared memory.
Problem of getting the concurrency right is very much identical to the problem of getting the system design right.
In both cases it takes a lot of hard work to learn how to not produce "dead-borns" or "genetically pre-disposed".
In both cases the course of "least resistance" (using intuition instead of logical reasoning) used by many as the default will most likely fail you. To develop an intuition of what is easy and what is hard with both system design and concurrency design is a matter of years of hard work of trial and error.
The actual question is why the complexity is so counterintuitive. I think this is to do with the Dunning-Kruger effect (the less you know the more confident you are).
In both cases the solution seems to be to keep beginners away from this and to have the same consistent, peer reviewed, approach across the system, and a process of improving it in a structured way. And of cause prefer higher level abstractions -> computation graph, pipeline, event loop, etc. over just hacking an ad-hoc Frankenstein out of low level primitives (which seem totally "simpler" if you ask a beginner).
Modern, sane and productive languages usually come with strong abstractions for concurrent usage, e.g. atomic reference wrappers, thread safe wrappers (AtomicInteger) or special methods (Interlocked.increment) for numbers and concurrent collections.
With them and a basic understanding of single system concurrency, single system concurrency is and always will be trivial.
They are low level abstractions and they don't save from the hardest to debug and fix problems - race conditions and inconsistent states, especially in distributed applications.
No mainstream languages give you a solid high level abstraction to build on. Some frameworks, kind of, do: ASIO in C++, AKKA Typed Actors and Reactive Streams and JDK CompletableFuture in JVM world, etc.
For all running threads, imagine that every operation can interleave in any possible way. Since the permutations grows in factorial time, the only way to manage concurrency is through well-defined interaction points with no more than a few interesting parts of execution. Are there only 4 possible sequences of significant instructions? Cool. Almost any more and it gets hard to reason about because there are so many paths that aren't obviously enumerable.
A simple way to avoid the problems highlighted in the article is to use immutable data-structures.
If you need to update that across threads, am atom with CAS logic will usually be sufficient.
If you like it erlang style: an agent can serialize changes nicely.
If you need coordinated changes, use software transactions.
Of course: next we'll introduce deadlocks with DB transactions. Fun!
I think Go adds even more complexity to concurrency by pushing the CSP model. CSP is hard for many problems where classic shared-memory using locks and atomics are well defined (think of classic concurrency programming textbooks).
Go provides all of that, you have sync.Mutex and sync.Cond, and you are of course free to use them. People do use them in practice, precisely because some code is easier to write that way.
Saying that Go "adds even more complexity to concurrency" doesn't scan. It lets you write code using whichever model you prefer--CSP or mutexes.
It is definitely true that CSP is hard for many problems... but locks and shared memory is also hard for many problems. There is no silver bullet, so you need both.
In theory I like channels, in practice I find them challenging to program with. The fact that only the sender can safely close a channel is frequently frustrating.
[+] [-] mcqueenjordan|5 years ago|reply
In my experience, simple concurrency is easy. Confused concurrency is exceptionally hard.
I think about this in the same way I think about C++. C++ is quite nice, if your teams limit themselves to a reasonable subset of the language. But it can become an absolute monster of cumbersome complexity, if these guardrails go away.
Concurrency feels the same. If you take a certain view on concurrency, which is "do it as simply as possible, don't break certain rules, don't try to be too clever," I find it fairly straightforward to reason about.
At the same time, 9 out of 10 of the hardest things I've debugged have always been nasty, awful concurrency bugs. The reason it was so hard? The concurrency was complicated, interwoven, assumption-laden, and absolutely impossible for anyone to reason about. Implicit contracts between calls required entire whiteboards of non-trivial state to understand, and even then, with all that work, it was a needle in a haystack.
I guess I would just say: It doesn't have to be hard. It's only hard if you don't respect it from the beginning.
[+] [-] corytheboyd|5 years ago|reply
The task at hand wasn’t easy at all, it required both:
1. Making numerous network requests (some endpoints support batching, so do that as much as possible) 2. The entire main loop needs to be as fast as reasonably possible
The only real way to satisfy both, given that serially executing the network activity would waste tons of time, was some fairly tricky concurrent logic. I ended up using a few queue structures to collect day before making batch network calls, and stitching together all of the results at the end (block on all work threads, collect, report, then start next iteration)
Every step along the way made sense, every addition of complexity felt justified. And the first person to come to me asking how the fuck it got so complicated is frustrated, so my knee jerk reaction is to get frustrated back saying well how else could we satisfy complex requirements with a complex solution.
We should always be challenging ourselves and others to keep solutions simple, but remember that the simplest solution may itself still be complex.
[+] [-] jjav|5 years ago|reply
One needs to pick the concurrency model and design up front. Document it before writing a single line of code. Then stick to the plan. It takes discipline, but it's not difficult.
I mostly write in C with pthreads and as long as one is willing to apply discipline to the task, there's no reason to fear concurrency.
If one starts adding a thread here and a mutex there and some shared structures over there without any prior plan then yes, absolutely, that leads to a world of hurt and a deep hole that is very difficult to get out of. So, don't do that.
[+] [-] lloeki|5 years ago|reply
This has to be a universal rule of programming. It's equally true for Ruby and Perl, which are extremely effective languages, but devolve quickly because of their uncanny ability to power cleverness. It's the whole premise of Go, by being boring by design.
Complex is easy, simple is hard, but it seems mostly because we fail to be able to resist the allure of cleverness.
[+] [-] xfer|5 years ago|reply
[+] [-] BiteCode_dev|5 years ago|reply
It's not that react, redux and cie are something new or disruptive. It just enforces a set of design rules that turns a lot of concurrent inputs and outputs into a linear workflow.
[+] [-] joubert|5 years ago|reply
I’m not sure I know what you mean by that.
> If you take a certain view on concurrency, which is "do it as simply as possible, don't break certain rules, don't try to be too clever,"
Is there a list of those rules?
[+] [-] marta_morena_28|5 years ago|reply
There is just SOOOO much that can go wrong. But one general rule crystallizes itself pretty quickly:
DO NOT EVER MUTATE STATE!
I.e. when you write concurrent code that builds on state mutation, you are opening Pandorra's box. If you want concurrency that people can understand and maintain, write immutable, functionally pure code (same data in, same result out, no exceptions). That is relatively easy to police and enforce, while still flexible enough to solve most use cases. It may however, not be the most performant.
[+] [-] dragontamer|5 years ago|reply
Fork-join is the simplest methodology, and supported in a very wide number of languages. In fact, its so stupidly easy, you probably are going to over-think it the first time you use it.
1. Fork when you need concurrency (aka: speed)
2. Join when you need consistency
That's it. Don't write mutexes, don't write barriers. Just join. That's all you need: if you ever need "synchronized" access, do it in the "single" or "master" thread, when all other threads are joined.
You _cannot_ have a race-condition if you only have one thread. Joins ensure that only one thread is running. What else do you want?
Fork-join parallelism has relatively low utilization. Its not the fastest approach, but its the "easiest" approach in my eyes.
------------
The #1 issue with parallelism books, in my eyes, is that we start with mutexes and threads, which are a very hard way to do parallelism.
I argue instead: beginners should start with easy (but inefficient) forms of parallelism, and then work their way towards complicated tools as performance becomes more-and-more of a consideration.
[+] [-] DrBazza|5 years ago|reply
and then there was a follow up to it saying the measurements were wrong IIRC, and then a further article showing, more or less that getting your own mutex replacement correct is so tricky that for 99.99% of programmers, just don't bother.
[+] [-] phkahler|5 years ago|reply
[+] [-] bob1029|5 years ago|reply
If everyone is working in parallel, but on independent tasks, there are no issues.
[+] [-] jedberg|5 years ago|reply
For example, a bank balance. Most people would assume that bank balances must always be correct. But this isn't really true. Back in the days when people wrote checks, your bank balance was almost never accurate -- it was on you to keep a local register and know your balance, and if you shared the account with someone else, to reconcile with them frequently. ATMs also use eventual consistency. When you take out $300 from the ATM, it attempts to update the balance, but it will let you take the money out if the bank is unavailable. Because the bank is willing to risk $300 that you aren't overdrawn. In most cases the "eventual" part is nearly instantaneous so you don't notice, but it doesn't actually have to be.
So the number one way to make concurrency easy is to avoid it altogether.
[+] [-] nemo1618|5 years ago|reply
[+] [-] CyberDildonics|5 years ago|reply
[+] [-] PaulBGD_|5 years ago|reply
[+] [-] mianos|5 years ago|reply
[+] [-] anko|5 years ago|reply
having lightweight isolated processes and no (or little) shared mutable state matters a lot. :)
[+] [-] elmo2you|5 years ago|reply
For years, I certainly made my own share of concurrency horrors, in everything from C and Python to Java and JS. Eventually I (sort of) got it right in each, mostly by building up experience in approaching it the right way. But it always remained a source of easy to make mistakes (and sometimes near-impossible to solve complexities, with larger systems).
However, since working with Erlang/Elixir, I sometimes wonder what the F#$& I was doing all that time. Many ideas were certainly a bit challenging to wrap my head around at first, but once they clicked there (kind of) was no way back.
This will probably sounds arrogant, and it's okay with me if I get down-voted because of it. But the more I work with Erlang/Elixir (and a few others in the FP field), the more it becomes clear to me how much the languages I used to program in (and their typical challenges), now most of all look like a (rather sick) joke to me. Again, I know this might sound pretentious, if not offensive. But it is sincerely what I feel. In my defense: I'm not some kind of hipster, still wet behind the ears. I've been programming for nearly 3 decades and in well over a dozen different language. I'm one of those kids that were ripping apart assembly code of "protected" games and desktop software, and hacked around on micro-controllers, during the nineties.
While Erlang/Elixir may have shortcomings too (although I personally have not found any), it is one of the best things I've encountered for a long, long time. I'm also charmed by ClojureScript, but for totally different reasons. However, I do like it for when I'm forced to build something more complex for browser/front-end. But that's a whole different story, and not much relevant to this concurrency "issue".
[+] [-] scott_meyer|5 years ago|reply
If you want practical guidance, my best is to avoid all of the concurrency constructs that are taught in schools. Use log-structured single-writer multiple reader, wait-free, processes and shared memory.
[+] [-] mrbald|5 years ago|reply
In both cases it takes a lot of hard work to learn how to not produce "dead-borns" or "genetically pre-disposed".
In both cases the course of "least resistance" (using intuition instead of logical reasoning) used by many as the default will most likely fail you. To develop an intuition of what is easy and what is hard with both system design and concurrency design is a matter of years of hard work of trial and error.
The actual question is why the complexity is so counterintuitive. I think this is to do with the Dunning-Kruger effect (the less you know the more confident you are).
In both cases the solution seems to be to keep beginners away from this and to have the same consistent, peer reviewed, approach across the system, and a process of improving it in a structured way. And of cause prefer higher level abstractions -> computation graph, pipeline, event loop, etc. over just hacking an ad-hoc Frankenstein out of low level primitives (which seem totally "simpler" if you ask a beginner).
[+] [-] Traubenfuchs|5 years ago|reply
With them and a basic understanding of single system concurrency, single system concurrency is and always will be trivial.
[+] [-] mrbald|5 years ago|reply
No mainstream languages give you a solid high level abstraction to build on. Some frameworks, kind of, do: ASIO in C++, AKKA Typed Actors and Reactive Streams and JDK CompletableFuture in JVM world, etc.
[+] [-] anonacct38|5 years ago|reply
Shared mutable data concurrency is hard because it removes ability to locally reason about code.
can now fail.[+] [-] dragontamer|5 years ago|reply
[+] [-] sillysaurusx|5 years ago|reply
It has a delightful idea: to ensure the current thread isn’t preempted, set a global “lock” variable to true.
The scheduler checks “lock”. If it’s true, the current thread is added to the front of the threads list. Otherwise it’s added to the back.
Then the scheduler runs the first thread in the threads list.
So by setting “lock” to true, you are essentially instructing the scheduler never to switch threads.
[+] [-] hedora|5 years ago|reply
[+] [-] dehrmann|5 years ago|reply
[+] [-] beders|5 years ago|reply
If you like it erlang style: an agent can serialize changes nicely.
If you need coordinated changes, use software transactions.
Of course: next we'll introduce deadlocks with DB transactions. Fun!
[+] [-] galaxyLogic|5 years ago|reply
[+] [-] amelius|5 years ago|reply
https://www.bbc.com/news/science-environment-24645100
[+] [-] diehunde|5 years ago|reply
[+] [-] klodolph|5 years ago|reply
Saying that Go "adds even more complexity to concurrency" doesn't scan. It lets you write code using whichever model you prefer--CSP or mutexes.
It is definitely true that CSP is hard for many problems... but locks and shared memory is also hard for many problems. There is no silver bullet, so you need both.
[+] [-] initplus|5 years ago|reply
[+] [-] chrisseaton|5 years ago|reply