False premises common in anti-A/B-testing arguments

[ehutch79]

Some of these fall premises, are actually admitted to be true in the article... wtf?

Also, a lot of the quotes are not so much against a/b test, but people who claim a/b testing is a panacea, and are recommending the wrong tools to the wrong people. (ie tools based around static content to people with entirely dynamically generated content)

basically the biggest problem with a/b testing is that some of it's most noisy proponents have no idea what they're talking about. (this is true of a lot of things)

[dylanrw]

The biggest problem I have with A/B or Split testing is that it makes it easy for inexperienced people to assume there are multiple right answers to every problem and that they all warrant 'discussion' or testing... Most often this is not the case, and people who jump straight to A/B everything are the sort who don't really have/trust any experience and would rather spin wheels and throw spaghetti at the wall.

That said, sometimes there are two (or more) options that warrant testing (say a marketing strategy) and then it works great.

Test strategy, don't test everything, don't test design principles.

[adw]

The biggest problem I have with not A/B testing is that it breeds a culture where people assume they really know what's going on.

Common sense is often only common sense in retrospect!

[varelse]

I've always thought the problem with A/B testing is that it's mostly hill-climbing optimization which is likely to hit and get stuck at a local optimum.

[wikwocket]

A/B testing doesn't necessarily mean "test two subtle shades of the same thing." For a great example, see the Highrise A/B test: http://37signals.com/svn/posts/2991-behind-the-scenes-ab-tes...

This is a good example that disproves many of the false premises mentioned in the OP, as the clear winner of the A/B tests was a bold dynamic design with lots of character, that was completely different from the original designs.

[noelwelsh]

You could A/B test two completely different business models if you had the resources to do so. Your imagination and your bank account are the meaningful limits on what you can test, not the technology.

[zaguios]

He addresses exactly that point with False Premise #2. It really depends on how you use A/B testing, if you only do it to test small incremental changes then yes, you're correct, but there's nothing stopping you from testing two totally different features/designs with A/B testing.

[ojilles]

Only if you keep testing variations on the same thing. There's nothing about A/B testing that says (or drives you toward) doing that, per se.

[greglindahl]

This would be 100x more useful if it was "Avoiding common pitfalls of A/B testing" -- since the author admits that poor usage of A/B testing makes most of the premises true.

[jfarmer]

His point is that these outcomes aren't inherent in the process of A/B testing, not that they aren't consequences of certain types of A/B testing.