There are 2 gaps here, at least as they relate to my understanding of what "canaries" are (based on experience):
1. Requiring manual validation of canary results is antithetical to CI/CD principles. If you can't trust your canaries enough to automatically promote or block based on results, doing manual validation is just a band-aid that will have to be ripped off painfully eventually.
2. Canaries should run continuously against all deployment stages, not just as a one-time approval process in your pipeline. Good canaries give visibility into baseline metrics at all times, not just when validating a new application version.
Overall I would say this guide aligns more with what I would term as a non-CI/CD load-test approval workflow rather than a "canary".
1. Yes. This is what the article explains in the "Monitor your new pipeline before you trust it" section. This is more of a "how to get started", and when you just created your canary config, you probably don't want to trust it just yet to push to production.
2. I'm not sure I understand here. What version of your application is your canary running if you're not using it to validate a new version? The same as the baseline? But then, what are you using it for?
For clarification, "canary" in this article refers to rolling out a new release to some small subset of production traffic, a la canary release (eg https://martinfowler.com/bliki/CanaryRelease.html).
At least a few other people in the comments are saying that in their experience, "canaries" are black-box monitoring programs that simulate critical user journeys. This is not what this article is discussing.
> Spinnaker is an open-source, continuous delivery system built by Netflix and Google
Politely correct me if I'm wrong...isn't Spinnaker originally a Netflix system, having nothing to do with Google? Unless perhaps the author is alluding to open source contributions by Google after the tool went open source?
It is originally a Netflix creation but Google has been a contributor since it went open source. In particular, the canary analysis features were co-developed with Google. (I was a contributor on the Netflix side.)
Author here. As mentioned in the other comments, Google is a very active contributor to Spinnaker. Today, Netflix and Google are the 2 companies contributing the most to the project, hence this sentence.
[+] [-] not_kurt_godel|7 years ago|reply
1. Requiring manual validation of canary results is antithetical to CI/CD principles. If you can't trust your canaries enough to automatically promote or block based on results, doing manual validation is just a band-aid that will have to be ripped off painfully eventually.
2. Canaries should run continuously against all deployment stages, not just as a one-time approval process in your pipeline. Good canaries give visibility into baseline metrics at all times, not just when validating a new application version.
Overall I would say this guide aligns more with what I would term as a non-CI/CD load-test approval workflow rather than a "canary".
[+] [-] mrtrustor|7 years ago|reply
1. Yes. This is what the article explains in the "Monitor your new pipeline before you trust it" section. This is more of a "how to get started", and when you just created your canary config, you probably don't want to trust it just yet to push to production.
2. I'm not sure I understand here. What version of your application is your canary running if you're not using it to validate a new version? The same as the baseline? But then, what are you using it for?
[+] [-] joatmon-snoo|7 years ago|reply
At least a few other people in the comments are saying that in their experience, "canaries" are black-box monitoring programs that simulate critical user journeys. This is not what this article is discussing.
[+] [-] btmiller|7 years ago|reply
Politely correct me if I'm wrong...isn't Spinnaker originally a Netflix system, having nothing to do with Google? Unless perhaps the author is alluding to open source contributions by Google after the tool went open source?
[+] [-] svachalek|7 years ago|reply
[+] [-] jedberg|7 years ago|reply
But I mean, I evaluated the initial design docs, so did I build it too? :)
[+] [-] mrtrustor|7 years ago|reply