[e10v_me]

I thought about multiple comparison corrections. Here what my thoughts were:

1. Experiments with 3 or more variants are quite rare in my practice. I usually try to avoid them.

2. In my opinion, the Bonferroni correction is just wrong. It's too pessimistic. There are better methods though.

3. The choice of alpha is subjective. Why use a precise smart method to adjust a subjective parameter? Just choose another subjective alpha, a smaller one :)

But I can change my opinion if I see a good argument.

[cschmidt]

If you work for a large website (as I used to), they probably run hundreds of tests a week across various groups. So false positives are a real problem, and often you don't see the gain suggested by the A/B when rolling it out.

I agree that Bonferroni is often too pessimistic. If you Bonferroni correct you'll usually find nothing is significant. And I take your point that you could adjust the $\alpha$. But then of course, you can make things significant or not as you like by the choice.

False Discover Rate is less conservative, and I have used it successfully in the past.

People have strong incentives to find significant results that can be rolled out, so you don't want that person choosing $\alpha$. They will also be peaking at the results every day of a weekly test, and wanting to roll it out if it bumps into significance. I just mention this because the most useful A/B libraries are ones that are resistant to human nature. PM's will talk about things being "almost significant" at 0.2 everywhere I've worked.

[e10v_me]

Thank you for explanation and for drawing a vivid picture) I will add FWER and FDR to the roadmap. Which specific controlling procedures do you find the most useful on practice?

I'm considering the following: - FWER: Holm–Bonferroni, Hochberg's step-up. - FDR: Benjamini–Hochberg, Benjamini–Yekutieli.