top | item 18912522

Canary analysis: Lessons learned and best practices from Google and Waze

113 points| Daviey | 7 years ago |cloud.google.com | reply

20 comments

order
[+] not_kurt_godel|7 years ago|reply
There are 2 gaps here, at least as they relate to my understanding of what "canaries" are (based on experience):

1. Requiring manual validation of canary results is antithetical to CI/CD principles. If you can't trust your canaries enough to automatically promote or block based on results, doing manual validation is just a band-aid that will have to be ripped off painfully eventually.

2. Canaries should run continuously against all deployment stages, not just as a one-time approval process in your pipeline. Good canaries give visibility into baseline metrics at all times, not just when validating a new application version.

Overall I would say this guide aligns more with what I would term as a non-CI/CD load-test approval workflow rather than a "canary".

[+] mrtrustor|7 years ago|reply
Author here.

1. Yes. This is what the article explains in the "Monitor your new pipeline before you trust it" section. This is more of a "how to get started", and when you just created your canary config, you probably don't want to trust it just yet to push to production.

2. I'm not sure I understand here. What version of your application is your canary running if you're not using it to validate a new version? The same as the baseline? But then, what are you using it for?

[+] joatmon-snoo|7 years ago|reply
For clarification, "canary" in this article refers to rolling out a new release to some small subset of production traffic, a la canary release (eg https://martinfowler.com/bliki/CanaryRelease.html).

At least a few other people in the comments are saying that in their experience, "canaries" are black-box monitoring programs that simulate critical user journeys. This is not what this article is discussing.

[+] btmiller|7 years ago|reply
> Spinnaker is an open-source, continuous delivery system built by Netflix and Google

Politely correct me if I'm wrong...isn't Spinnaker originally a Netflix system, having nothing to do with Google? Unless perhaps the author is alluding to open source contributions by Google after the tool went open source?

[+] svachalek|7 years ago|reply
It is originally a Netflix creation but Google has been a contributor since it went open source. In particular, the canary analysis features were co-developed with Google. (I was a contributor on the Netflix side.)
[+] jedberg|7 years ago|reply
It's just a question of where you draw the line. Google made a lot of contributions after it went open source, particularly with the canary analysis.

But I mean, I evaluated the initial design docs, so did I build it too? :)

[+] mrtrustor|7 years ago|reply
Author here. As mentioned in the other comments, Google is a very active contributor to Spinnaker. Today, Netflix and Google are the 2 companies contributing the most to the project, hence this sentence.