chashmataklu's comments

chashmataklu | 1 year ago | on: Five ways to reduce variance in A/B testing

Pretty sure most don't. Most A/B Test SAAS vendors cater to lightweight clickstream optimization, which is why they don't have features like Stratified Sampling. Internal systems are lightyears ahead of most SAAS vendors.

chashmataklu | 1 year ago | on: Five ways to reduce variance in A/B testing

TBH depends a lot on the business you're experimenting with and who you're optimizing for. If you're Lime Bike, you don't want to skew results because of a Doordasher who's on a bike for the whole day because their car is broken.

If you're a retailer or a gaming company, you probably care about your "whales" who'd get winsorized out. Depends on whether you're trying to move topline - or trying to move the "typical".