[cdavid]

There are cases where A/B testing does not make sense (not enough users to measure anything sensible, etc.). But if the A/B test results were inconclusive, assuming they were done correctly, then what was the point of launching the underlying feature ?

As for the HIPPO pushing for an A/B test because of lack of confidence, all I can say is that we had very different experiences, because I've almost always seen the opposite, be it in marketing, search/recommendation, etc.

[simonw]

"not enough users to measure anything sensible" is definitely a big part of it: even for large established companies there are still plenty of less than idler used features that don't have enough activity for that to make sense.

A former employer had developed a strong culture of A/B testing to the point that everyone felt pressure to apply it to every problem.