Awesome work! Will look at replacing some hand rolled stuff internally. One gap I see with a lot of A/B testing analysis is that it only solves for conversion. While conversion is a great metric for many tests, in my experience, revenue is often the metric that matters most. Whether it's traditional eCommerce or selling tiered subscriptions, a lot of testing is geared towards 1) getting the customer to buy and 2) getting them into a more expensive product or plan. In the subscription scenario, some sort of customer life time value model is even better. I don't pretend to know all the math, but the calcs I've seen focused on revenue (AOV * conversion) need order level data (as opposed to aggregate) so it's not as easy to solve generically.
jessicaraygun|12 years ago