[Jemaclus]

I think there's a pretty big difference between QA (letting bugs go by) and A/B testing, and your post appears to me to be conflating the two. I would argue that you are better off spending your time QAing a feature that you have high confidence is positive ROI, than spending weeks waiting for an A/B test to reach stat sig.

I don't disagree with your statement, I just think you are addressing a different problem from A/B testing and statistical significance.