Appreciate the candid comments and opinions here. I'll break it down and go over them -
1. Having access to data is not the problem we're after (most companies have the first-party data in-house). The key challenges are around having a platform that fundamentally separates strings (copies) from code, and lets you update them effortlessly, based on inferences from that data. So, I am not sure I understand why this is a Google/Meta product?
2. UX is not the value add from this product - agree with you on it (even if it appeared to be the emphasis). The ability to make scientific edits without re-deployments and accelerating continuous iterations based on user feedback, is what we are going after.
3. Curious why you think A/B test results are fictional? Getting stat sig results is probably the surest way to conclude results. Perhaps there's a different angle you are talking about here?
4. RE: don't A/B test at all. Given the number of users that get exposed to every change a consumer company as large as Twitter makes, not testing can be disastrous, which brings up another great point - Large companies are struggling to use all the (generic) gen AI content today, because it needs to be performance tested before it can be placed in front of millions of users, and that's not scalable today.
5. You may be alluding to another good point - copy is as much as art, as it is science, and writing it well takes context, quality, and expertise. That's something we hold a strong opinion on, and we don't see this or any other tool eliminating that expertise. The goal is very much to streamline and augment those workflows.
doctorpangloss|2 years ago
Which audiences am I optimizing copy for? Where do they come from? Some Google, Meta TikTok or Apple owned channel right?
Google has indexed every website. Meta has every ad. Can't they just tell me what copy to use? Why don't they? I mean, they know! They know what copy works best, for pretty much everything. They can sort by clickthrough rate, revenue due to the purchase data they have, they have everything! You talk about SMB - they know every SMB! They know your margin and your COGS and whatever because they in aggregate they observe rational spending where all the cost is eaten by marketing; they know your potential market, etc. They know all this. They don't need to run tests. They can look at very recent, weeks old, historic data, and they have way more than enough samples to answer these questions to more or less the same degree of certainty and scientific rigor that any SMB doing it themselves as a first party can do.
I mean if they wanted to, they could run the A/B tests for you! Google could "just" serve a different web page with slightly altered copy. And see if more people "click" or "convert" or whatever. They have better technology, 1,000,000x more data... Why don't they do this? You wouldn't even need UX. It could just happen, you would check a box, and they would do this.
> fundamentally separates strings (copies) from code... and lets you update them effortlessly... The ability to make scientific edits without re-deployments and accelerating continuous iterations based on user feedback...
You keep talking about UX for developers and product managers. These are UX things. It doesn't actually matter. The existence or non-existence of what you're talking about doesn't correlate to higher or lower conversions, it isn't a scientific opinion on the practice of optimization, it is just a bunch of UX patterns to achieve it, but it could be achieved in many ways, perhaps with even better UX. Like in the example I gave, where Google "just" does this for you, which is the best UX because there is no UX, you don't need to separate strings from code, and you don't need to update them, because you don't need to do anything. Google could just do this. They own the channel, they see everything, they have the technology.
So why don't Google and Meta and Apple offer an automatic optimization product? You ought to have an opinion, it can't just be, "I don't know." I mean the sort of obvious answer is that "optimization doesn't really work" instead of "three paragraphs of bullshit."
> Curious why you think A/B test results are fictional? Getting stat sig results is probably the surest way to conclude results. Perhaps there's a different angle you are talking about here.
Well one reason I am very confident they are fictional is because the people who own the channels for a decade haven't offered a tool to do this.
I mean maybe they will. Maybe it was a technology problem, but I don't believe so. You could have Markov Chained your way through 5 word long taglines and whatever. They didn't need to way for generative AI to create valid test strings for people's websites. Indeed they could just let you copy the best performing taglines they see in their systems. Why. Don't. They?!
> Given the number of users that get exposed to every change a consumer company as large as Twitter makes
Another POV is that every change they made was bad. They thought they were a product organization, and they are really a backend engineering organization, where the best decisions are all based on first principles or executives' opinions, not on some unknowable measurement about audiences.