[tbranyen]

I'm talking about running week or month long tests with control and multi test cells containing new functionality, configuration, or code to determine the viability of a single or combination of changes by analyzing statistical output driven by p-value and pre-determined target metrics.

These types of experiments are extremely valuable in uncovering hard-to-find bugs, assuming you have sufficient logging and confidence around your metrics. They let you know a problem exists and roughly where it is in the product. From there you can drill down and investigate your source code until the discrepancy is found.

[EricMausler]

This makes sense to me. Not the kind of AB testing I had in mind, but fair point. I was thinking more about decision making processes, not operations troubleshooting.

[ratww]

Yes, I know this and I agree. I understand they use the same techniques and terminologies, but this is just not the kinda thing the article is criticizing.