Great article. Designing a/b systems always seems (relatively) simple at the start, but in my experience there are 1,000 things you don't think of until you have massive amounts of worthless results. Add this to the list of things to watch out for.
My experience is the opposite. Spend time simplifying your thinking, and it stays simple. But it is very, very easy to start overthinking things and then you go down a rabbit hole.
Consider this for an example. If you're testing per session behavior, then you can just use a session cookie. If you're testing logged in behavior, you can use the login id. You've just covered most of the things you want to test.
When you start worrying about cross-device both logged in and not, then you have a world of pain. So treat it as an identity problem, throw away all of the users you find questionable, and work with that. And yes, this is a pain, which is why you do it as seldom as you can!
[+] [-] jasondavis|10 years ago|reply
http://drjasondavis.com/blog/2013/09/12/eight-ways-youve-mis...
Can't tell you how many bad a/b tests we ran at Etsy until we figured things out ;)
[+] [-] danmccorm|10 years ago|reply
[+] [-] btilly|10 years ago|reply
Consider this for an example. If you're testing per session behavior, then you can just use a session cookie. If you're testing logged in behavior, you can use the login id. You've just covered most of the things you want to test.
When you start worrying about cross-device both logged in and not, then you have a world of pain. So treat it as an identity problem, throw away all of the users you find questionable, and work with that. And yes, this is a pain, which is why you do it as seldom as you can!