[kdamica]

Hard disagree with this. Unlike medical experiments, the cost of being wrong startup experiments is very low: you thought there was a small effect and there was none. It’s usually just a matter of pushing one variant vs another and moving on.

There are certainly scenarios where more rigor is appropriate, but usually those come from trying to figure out why you’re seeing a certain effect and how that should affect your overall company strategy.

My advice for startups is to run lots of experiments, do bad statistics, and know that you’re going to have some false positives so that you don’t take every result as gospel.

[bravesoul2]

The danger I think is less the numbers but what are you measuring makes sense. E.g. sure your A beats B in click through rate. But if the person then thinks fuck I was duped and closes the browser then that's no good.

[lemmsjid]

Huh, I agree with your last sentence but think the author did a good job of explaining that the cost of layout experimentation in a startup can grow over time if the results of the experiments are overstated. The first question for a startup should always be: is this work worth doing in the first place? Tinkering with layout can be a tempting but fruitless rabbit hole. Even if it doesn’t tie up resources it can lead to a false sense of progress and get product thinking stuck in local maximae.