[RRL]

Back in 2015 I was pitching Optimizely internally. We already had an in house barebones A/B testing tool. Once I got over that hurdle our CTO let fly how his only experience using Optimizely had been a time when Optimizely went down and took his site down with it. Optimizely eventually brought out their head of product, and an army of senior engineers to convince him things were good now. We signed an annual contract, but Optimizely was never added to any page that was revenue critical/dependent on speed. Optimizely got their huge contract, and in the end marketing and design got their lightweight A/B testing tool, but we never realized the white paper level metrics jumps because it was never used on any page that would actually have material impact.