> The minimum threshold odds [to stop showing a variation] are calculated by 40% / number enabled variations
Uh... How did you choose those cutoffs? Looks like you have a significant chance of making the wrong choice.
Also:
> Once enough data is collected to start making conclusions (1000 sends per variation)
You should check out the Bayesian solution to the Multi-Armed Bandit problem. It's very close to what you are doing, but makes decisions much faster than you do because it isn't deciding to turn off a variation, merely to scale it down.
Thanks a lot for the feedback. This is a first pass implementation, but I agree that more thought should be put into the cutoff threshold, specifically for when there are only initially 2 (or maybe 3) variations.
With email optimization in particular be warned that people will respond to changes simply because it is different. But then acclimatize.
What that means is that the changed version tends to win the test, but then may or may not perform well. The flip side of this is that if you have a choice, have multiple variations on the same email that you rotate between so that people don't get too used to your emails.
Nice seeing GA applied to more things. People often do not realize how many problems with feedback mechanism can be solved with GA framework. Here is small tutorial that got me started with GAs long long time ago: http://www.ai-junkie.com/ga/intro/gat1.html
Neural network tutorial of the same site is pretty cool too.
dlss|12 years ago
Uh... How did you choose those cutoffs? Looks like you have a significant chance of making the wrong choice.
Also:
> Once enough data is collected to start making conclusions (1000 sends per variation)
You should check out the Bayesian solution to the Multi-Armed Bandit problem. It's very close to what you are doing, but makes decisions much faster than you do because it isn't deciding to turn off a variation, merely to scale it down.
erickerr|12 years ago
erickerr|12 years ago
We considered a weighted decision approach but 1) were turned off by posts like http://visualwebsiteoptimizer.com/split-testing-blog/multi-a... and 2) wanted to keep moving parts to a minimum for V1.
Any thoughts?
btilly|12 years ago
What that means is that the changed version tends to win the test, but then may or may not perform well. The flip side of this is that if you have a choice, have multiple variations on the same email that you rotate between so that people don't get too used to your emails.
TrainedMonkey|12 years ago
Neural network tutorial of the same site is pretty cool too.