top | item 16705941

Proposal: Non-cooperative goroutine preemption

150 points| mseepgood | 8 years ago |github.com | reply

71 comments

order
[+] tinco|8 years ago|reply
The way I understand this, and that I think some other people in this thread misunderstand, is that this is not a change to the Go language in any way.

The way Go works is that you can spawn green threads, and the Go runtime will magically run them concurrently, in such a way that you do not have to worry about many problems usually associated with concurrent programming.

This proposal seeks to modify the way the "magically" part works. Right now, it suggests that what Go does is to 'cooperatively' preempt at function prologues. This change would instead allow the runtime to preempt at a choice of "safe points" the compiler suggests throughout your Go code.

Note that this is a little bit in the gray area between cooperative and non-cooperative. It's not like the runtime will just randomly preempt, it still requires cooperation from the code to have these safe points. But maybe I'm misunderstanding the details.

Anyway, not much changes for the Go programmer. The Go programmer can expect his code to be a little more predictable, at the cost of having his code be a little less performant, this performance cost I think will not be significant, if not fully compensated by the increased parallelism. All the promises the Go runtime makes towards the programmer will still hold.

[+] masklinn|8 years ago|reply
> This proposal seeks to modify the way the "magically" part works. Right now, it suggests that what Go does is to 'cooperatively' preempt at function prologues. This change would instead allow the runtime to preempt at a choice of "safe points" the compiler suggests throughout your Go code.

That's how it already works, it's just that safepoints only get introduced at function prologues. This is an issue as CPU-heavy functions without any non-inlined function calls mean very few safe points which is problematic for GC and timely scheduling and may even lock up the entire system.

Previous attempts were made to add loop preemption but there are issues with that approach.

The approach in TFA is to make safe points "opt-out", all Go code would be considered safe by default with some "unsafe" regions being excluded. As the author states, this would make Go preemptive rather than cooperative ("I propose that we implement fully non-cooperative preemption").

> Anyway, not much changes for the Go programmer. The Go programmer can expect his code to be a little more predictable

Go programmers can expect their code to be a little less predictable, not more: currently your code is guaranteed to run uninterrupted between two function calls, that will not be the case anymore.

In theory it should not make any difference, in practice it will probably uncover concurrency bug you're currently shielded from by the runtime's behaviour (though likely not that many as you can already have multiple routines running in parallel).

[+] mseepgood|8 years ago|reply
> at the cost of having his code be a little less performant

Where did you read this? The proposal states: "This approach will solve the problem of delayed preemption with zero run-time overhead."

[+] zzzcpan|8 years ago|reply
The proposal is to turn all go code into safe points.
[+] chimeracoder|8 years ago|reply
> Right now, it suggests that what Go does is to 'cooperatively' preempt at function prologues. This change would instead allow the runtime to preempt at a choice of "safe points" the compiler suggests throughout your Go code.

This is incorrect. Go has had preemptive scheduling (instead of cooperative) since at least 1.2 (possibly 1.1). The proposal here is to make it preempt at arbitrary points, not just at certain points.

The points at which it currently can preempt is far greater than simply function prologues, which is why full preemption has very little impact on the end user (it already appears to preempt at essentially arbitrary points).

[+] hinkley|8 years ago|reply
This reminds me of the safe point mechanism in IBMs J9 JVM to make GC cheaper by shortening the mark phase.
[+] anonacct37|8 years ago|reply
I'm really excited about this both for the improved long tail latency as well as (what I believe is) a new way of handling this problem.

I'm not aware of another M:N threading model that handles preemption this way with signals. Erlang accomplishes it with reduction budgets and reduced performance.

[+] PDoyle|8 years ago|reply
Pre-emptive runtimes are hard. Inserting yield points judiciously in loops isn't that incredibly difficult, and neither is reducing the cost of a yield point. It's far harder to make sure every machine instruction in every part of the runtime is prepared to be interrupted and have all it assumptions violated by the time it resumes.
[+] bradfitz|8 years ago|reply
> Inserting yield points judiciously in loops isn't that incredibly difficult, and neither is reducing the cost of a yield point.

The Go team has done both over the past few years but we haven't been happy enough with the performance. Hence Austin's proposal.

[+] zzzcpan|8 years ago|reply
Imagine how much simpler and faster a runtime could be if every function was split into a series of non interruptible blocks. Sure, a bit tricky with loops, you'd have to cap them to make sure they are not hogging the system and you don't introduce too much overhead. But small price to pay for not wasting years of man-hours on silly low level stuff and barely achieving usable results.
[+] stefanchrobot|8 years ago|reply
I'd love to see this implemented, especially that I already enjoy this feature a lot in Elixir/Erlang/BEAM.
[+] dmytrish|8 years ago|reply
The problem with that approach in Go is that BEAM embraces the shared-nothing approach, and the Go runtime is quite promiscuous about sharing mutable data. I am afraid that introducing non-deterministic preemptive stops will just make Go concurrency a mess similar to the mutex/conditionals mess they've been trying to avoid.
[+] nostalgeek|8 years ago|reply
interesting, is the current situation one of the reason Go debugging experience is so poor? Delve debugger is constantly crashing for me, making step debugging extremely hard, especially when debugging tests. If there is one thing that is inferior to Java and C# with Go it's definitely the debugging experience.
[+] arghwhat|8 years ago|reply
Not at all. Non-cooperative goroutines would make it much harder to implement a debugger.

Delve crashing is just related to Delve being buggy. Single-stepping is done at the instruction level, and the task of a Go debugger mostly relates to mapping instructions to source code, and understanding the runtime.

You can debug Go with gdb, but without Go runtime comprehension, all you will see is a bunch of worker threads executing Go runtime code, rather than the individual goroutines of your process.

[+] anonacct37|8 years ago|reply
I've actually never run into a problem debugging go binaries with gdb. Which is weird because everyone's always told me it doesn't work. I'm sure there are edge cases but over the last 5 years or so my experience has been pretty good.

emacs + gud mode is a great experience for go debugging. I've even used it to debug low-level issues like plugin loading and verification.

[+] krylon|8 years ago|reply
Can someone make an informed guess on how much that would complicate the implementation? And/or how much of a performance impact it would have (for better or for worse)?
[+] arghwhat|8 years ago|reply
So, preemptive green-threads? This seems like a counter-productive and highly complicated re-implementation of what the OS thread model does, but inside the Go runtime.

I would very much like for goroutines to remain cooperative. It is much simpler, naturally more performant, and quite easy to reason about. A local implementation of preemptive execution seems like a very high price to pay to only handle a few corner-cases.

That is not to say that preemptive scheduling does not have their place: They make a lot of sense at an OS level where independent processes are scheduled, but little sense as an internal green-thread implementation.

[+] jerf|8 years ago|reply
Why do you think cooperative goroutines are easier to reason about? In Node, you can say with a straight face that once you start executing a bit of code, you can tell that that code will execute without pre-emption until a fairly obvious point where it stops running. In Go, the runtime inserts pre-emption points basically whenever it feels like it; yes, there are some pragmatic rules about where they may be, but even if you know the rules you can't count on them without intensely studying whether or not certain functions got inlined or not or things like that, and you can have no confidence about what other versions of Go (or other implementations) may do.

From what I can see, you could theoretically "reason" about your code if you compile it with the optimizer output on, study it carefully, and understand it intimately, but it would be a dangerously fragile line of reasoning subject to change without notice.

And I say all this ignoring the fact that whatever you are reasoning about is almost certainly already a bug according to the spec and something the race detector will scream about. I would not feel confident running a program that has known race conditions in it, but that you feel like you've reasoned via intimate knowledge of the runtime that they can't actually happen. I'd much rather see you just write the code to be concurrently correct anyhow, and then I can do things like "upgrade to the next version of Go" or "cross-compile to another architecture that may do things that violate your fragile reasoning" without stark terror.

[+] mseepgood|8 years ago|reply
> So, preemptive green-threads? This seems like a counter-productive and highly complicated re-implementation of what the OS thread model does, but inside the Go runtime.

You can't easily spawn 10.000 OS threads.

> I would very much like for goroutines to remain cooperative. It is much simpler, naturally more performant, and quite easy to reason about.

Cooperative is less simple to reason about, as the proposal explains. Preemption fully delivers what the developer already expects of goroutines.

[+] chrisseaton|8 years ago|reply
> This seems like a counter-productive and highly complicated re-implementation of what the OS thread model does, but inside the Go runtime.

But I think you still get advantages such as being able to manage stacks yourself, which is what allows for many more coroutines than native threads.

[+] dozzie|8 years ago|reply
> So, preemptive green-threads? This seems like a counter-productive and highly complicated re-implementation of what the OS thread model does, but inside the Go runtime.

It would be indeed a reimplementation of what OS does, but note that such a reimplementation is the very reason why quite straightforward Erlang code can easily handle hundreds of thousands active connections on modest hardware.

[+] weberc2|8 years ago|reply
> naturally more performant

Is this true? The proposal seems to indicate that preemption is more performant? Or at least it says "no runtime overhead", which is presumably in contrast to the current implementation.

[+] alex7o|8 years ago|reply
They should just make it possible to add safeties manually
[+] icholy|8 years ago|reply
isn't that what runtime.Gosched does?