Graceful Shutdown in Go: Practical Patterns

[zdc1]

I've been bitten by the surprising amount of time it takes for Kubernetes to update loadbalancer target IPs in some configurations. For me, 90% of the graceful shutdown battle was just ensuring that traffic was actually being drained before pod termination.

Adding a global preStop hook with a 15 second sleep did wonders for our HTTP 503 rates. This creates time between when the loadbalancer deregistration gets kicked off, and when SIGTERM is actually passed to the application, which in turn simplifies a lot of the application-side handling.

[rdsubhas]

Yes. Prestop sleep is the magic SLO solution for high quality rolling deployments.

IMHO, there are two things that kubernetes could improve on:

1. Pods should be removed from Endoints _before_ initiating the shutdown sequence. Like the termination grace, there should be an option for termination delay. 2. PDB should allow an option for recreation _before_ eviction.

[LazyMans]

We just realized this was a problem too

[evil-olive]

another factor to consider is that if you have a typical Prometheus `/metrics` endpoint that gets scraped every N seconds, there's a period in between the "final" scrape and the actual process exit where any recorded metrics won't get propagated. this may give you a false impression about whether there are any errors occurring during the shutdown sequence.

it's also possible, if you're not careful, to lose the last few seconds of logs from when your service is shutting down. for example, if you write to a log file that is watched by a sidecar process such as Promtail or Vector, and on startup the service truncates and starts writing to that same path, you've got a race condition that can cause you to lose logs from the shutdown.

[tmpz22]

Is it me or are observability stacks kind of ridiculous. Logs, metrics, and traces, each with their own databases, sidecars, visualization stacks. Language-specific integration libraries written by whoever felt like it. MASSIVE cloud bills.

Then after you go through all that effort most of the data is utterly ignored and rarely are the business insights much better then the trailer park version ssh'ing into a box and greping a log file to find the error output.

Like we put so much effort into this ecosystem but I don't think it has paid us back with any significant increase in uptime, performance, or ergonomics.

[utrack]

Jfyi, I'm doing exactly this (and more) in a platform library; it covers the issues I've encountered during the last 8+ years I've been working with Go highload apps. During this time developing/improving the platform and rolling was a hobby of mine in every company :)

It (will) cover the stuff like "sync the logs"/"wait for ingresses to catch up with the liveness handler"/etc.

https://github.com/utrack/caisson-go/blob/main/caiapp/caiapp...

https://github.com/utrack/caisson-go/tree/main/closer

The docs are sparse and some things aren't covered yet; however I'm planning to do the first release once I'm back from a holiday.

In the end, this will be a meta-platform (carefully crafted building blocks), and a reference platform library, covering a typical k8s/otel/grpc+http infrastructure.

[RainyDayTmrw]

I never understood why Prometheus and related use a "pull" model for data, when most things use a "push" model.

[PrayagS]

> another factor to consider is that if you have a typical Prometheus `/metrics` endpoint that gets scraped every N seconds, there's a period in between the "final" scrape and the actual process exit where any recorded metrics won't get propagated. this may give you a false impression about whether there are any errors occurring during the shutdown sequence.

Have you come across any convenient solution for this? If my scrape interval is 15 seconds, I don't exactly have 30 seconds to record two scrapes.

This behavior has sort of been the reason why our services still use statsd since the push-based model doesn't see this problem.

[karel-3d]

one tiny thing I see quite often: people think that if you do `log.Fatal`, it will still run things in `defer`. It won't!

    package main
    
    import (
     "fmt"
     "log"
    )
    
    func main() {
     defer fmt.Println("in defer")
    
     log.Fatal("fatal")
    }

this just runs "fatal"... because log.Fatal calls os.Exit, and that closes everything immediately.

    package main
    
    import (
     "fmt"
     "log"
    )
    
    func main() {
     defer fmt.Println("in defer")
    
     panic("fatal")
    }

This shows both `fatal` and `in defer`

[wbl]

If a distribute system relies on clients gracefully exiting to work the system will eventually break badly.

[Rhapso]

And i believe that so much that I don't even consider graceful shutdown in design. Components should be able to safely (and even frequently) hard-crash and so long as a critical percentage of the system is WAI then it shouldn't meaningfully impact the overall system.

The only way to make sure a system can handle components hard crashing, is if hard crashing is a normal thing that happens all the time.

All glory to the chaos monkey!

[ikiris]

There's a big gap between graceful shutdown to be nice to clients / workflows, and clients relying on it to work.

[smcleod]

Way back when, in physical land - I used STONITH for that! https://smcleod.net/2015/07/delayed-serial-stonith/

[XorNot]

There's valid reasons to want the typical exit not to look like a catastrophic one even if that's a recoverable situation.

That my application went down from sig int makes a big difference compared to kill.

Blue-Green migrations for example require a graceful exit behavior.

[eknkc]

Yeah. However, I do not need to pull the plug to shut things down even if the software was designed to tolerate that.

In a second thought though, maybe I do. That might be the only way to ensure the assumption is true. Like the Netflix's chaos monkey thing a couple years ago.

[icedchai]

Relying on graceful exit and supporting it are two different things. You want to support it so you can stop serving clients without giving them nasty 5xx errors.

[Thaxll]

No one said that.

[fpoling]

I was hoping the article describe how to perform the application restart without dropping a single incoming connections when a new service instance receives the listening socket from the old instance.

It is relatively straightforward to implement under systemd. And nginx has been supporting that for over 20 years. Sadly Kuberenets and Docker have no support for that assuming it is done in load balancer or the reverse proxy.

[joaohaas]

You're probably looking for Cloudflare's tableflip: https://github.com/cloudflare/tableflip

[giancarlostoro]

I had a coworker that would always say, if your program cannot cleanly handle ctrl c and a few other commands to close it, then its written poorly.

[danhau]

Your coworker is correct.

[amelius]

Ctrl-C is reserved for copy into the clipboard ... Stopping the program instead is highly counter-intuitive and will result in angry users.

[gchamonlive]

This is one of the things I think Elixir is really smart in handling. I'm not very experienced in it, but it seems to me that having your processes designed around tiny VM processes that are meant to panic, quit and get respawned eliminates the need to have to intentionally create graceful shutdown routines, because this is already embedded in the application architecture.

[cle]

How does that eliminate the need for the graceful shutdown the author discusses?

[eberkund]

I created a small library for handling graceful shutdowns in my projects: https://github.com/eberkund/graceful

I find that I typically have a few services that I need to start-up and sometimes they have different mechanisms for start-up and shutdown. Sometimes you need to instantiate an object first, sometimes you have a context you want to cancel, other times you have a "Stop" method to call.

I designed the library to help my consolidate this all in one place with a unified API.

[mariusor]

Haha, I had the exact same idea, though my API looks a bit less elegant. Maybe it's because it allows the caller to set-up multiple signals to handle and in which way to do it.

https://pkg.go.dev/git.sr.ht/~mariusor/wrapper#example-Regis...

[pseidemann]

I did something similar as well: https://github.com/pseidemann/finish

[deathanatos]

> After updating the readiness probe to indicate the pod is no longer ready, wait a few seconds to give the system time to stop sending new requests.

> The exact wait time depends on your readiness probe configuration

A terminating pod is not ready by definition. The service will also mark the endpoint as terminating (and as not ready). This occurs on the transition into Terminating; you don't have to fail a readiness check to cause it.

(I don't know about the ordering of the SIGTERM & the various updates to the objects such as Pod.status or the endpoint slice; there might be a small window after SIGTERM where you could still get a connection, but it isn't the large "until we fail a readiness check" TFA implies.)

(And as someone who manages clusters, honestly that infintesimal window probably doesn't matter. Just stop accepting new connections, gracefully close existing ones, and terminate reasonably fast. But I feel like half of the apps I work with fall into either a bucket of "handle SIGTERM & take forever to terminate" or "fail to handle SIGTERM (and take forever to terminate)".

[cientifico]

We've adopted Google Wire for some projects at JustWatch, and it's been a game changer. It's surprisingly under the radar, but it helped us eliminate messy shutdown logic in Kubernetes. Wire forces clean dependency injection, so now everything shuts down in order instead... well who knows :-D

https://go.dev/blog/wire https://github.com/google/wire

[Savageman]

I wish it would talk about liveness too, I've see several times apps that use the same endpoint for liveness/readiness but it feels wrong.

[liampulles]

I tend to use a waitgroup plus context pattern. Any internal service which needs to wind down for shutdown gets a context which it can listen to in a goroutine to start shutting down, and a waitgroup to indicate that it is finished shutting down.

Then the main app goroutine can close the context when it wants to shutdown, and block on the waitgroup until everything is closed.

[mariusor]

If you look at the article, it presents some additional niceties, like having middleware that is aware of the shutdown - though they didn't detail exactly how the WithCancellation() function is doing that.

So if you send a SIG-INT/-TERM signal to the server there's a delay to clean up resources, during which the new requests get served a response that doesn't try to access them and fail in unexpected ways, but a configurable "not in service" error.

[gitroom]

honestly i always end up wrestling with logs and shutdowns too, nothing ever feels simple - feels like every setup needs its own pile of band aids