I feel like that example is missing some context - if signups did increase then their experiment was successful - we aren’t here to make pretty pages, we’re here to make money.
The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users. Most A/B tests focus on the former, very few on the latter. We had a free plan, and most users who signed up never made a single API request. So, assuming that the increase in signups is driving more business is just foolhardy.
> The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users.
Okay? The A/B test sought to measure which of two options A and B led to more signups.
> So, assuming that the increase in signups is driving more business is just foolhardy.
Your "A/B test enthusiast" was not testing for or trying to prove a causal relationship between increased signups and more business.
If he made the claim separately, then that is the context that is missing from now multiple comments.
You can always track signup/paying-users ratio. Purpose of landing/pricing page is to get the users to sign-up. Unless some dark pattern or misinformation is used to confuse users into sign-up, more users is a positive thing.
shubhamjain|1 year ago
lelanthran|1 year ago
That doesn't sound like a signup problem; what was the goal behind the free plan? Drive more paying users? Raise company profile? Lock in more users?
albedoa|1 year ago
Okay? The A/B test sought to measure which of two options A and B led to more signups.
> So, assuming that the increase in signups is driving more business is just foolhardy.
Your "A/B test enthusiast" was not testing for or trying to prove a causal relationship between increased signups and more business.
If he made the claim separately, then that is the context that is missing from now multiple comments.
blackoil|1 year ago