[scottfr]

Disagree on using Bayesian statistics. Frequentist statistics are perfect for A/B testing.

There are so many strong biases people have about different parts about UI/UX. One of the significant benefits of A/B testing is that it lets you move ahead as a team and make decisions even when there are strongly differing opinions on your team. In these cases you can just "A/B test" and let the data decide.

But if you are using Bayesian approaches you'll transition those internal arguments to what the prior should be and it will be harder to get alignment based on the data.

[eru]

Not necessarily.

You can present your Bayesian approaches in such a way that it's almost independent of the prior. Your output will be 'this experiment should shift your odds-ratio by so-and-so-many logits in this or that direction' instead of an absolute probability.

[JHonaker]

You have to make almost the exact same choices when you rely on fequentist tools. The main difference is they’re pre-made during the development of the tool, so you don’t get insight into what they are without studying the theory behind the test.

[grega5]

We can agree to disagree. My claim is actually quite the reverse. For A/B testing specifically, Bayes is much better suited to address the practical questions you would usually have when running A/B experiments. See my response to AlexeyMK below.

[miksumiksu]

Fixing dysfunctional decision making by delegating it to "data", what could go wrong? Might as well flip a coin and save the money.