Besides the novelty factor (which others have mentioned), his 'paint' version is also much simpler, and therefor much easier to understand at a quick glance.
- Large text
- Not very much text
- A single image
- A single visual flow (top to bottom)
- A concise color palette (greens and black)
Compare to the 'standard' ad, where:
- Text is too small
- Too much text
- Too many images competing for attention
- Muddled visual flow
Even more importantly, a high CTR with lower conversion rate would drive up your costs while lowering your sales.
In this case the first ad is very clearly for a game with a very clear target market. If I am in to racing games, I may click it and I may buy it.
The second ad could appeal to people strickly on novelty which means you may get a lot of customers who do not care about the product.
While the results of this post are very interesting and the conclusion to test everything is still good, we should look at the conversion rate to see which ad is actually more effective.
I think another big factor is probably a white background vs. a black background. Bill Watterson said he avoided making strips with dark backgrounds too often because the human eye is drawn away from dark to light.
I would like to see an A/B test of the same basic artwork on a black background vs. the white one.
Good lesson on the value of in-market testing. That said, would love to see some theory and analysis about why the MS Paint ad outperformed the standard ad. My hunch is that the first ad -- while it obviously looks a lot more professional -- looks like every other banner ad on the internet. It reeks of ad-ness, and it may set off some psychological barrier to receptivity amongst viewers precisely because their brains have been trained to filter out ads. (Banner blindness, as one of the other posters has pointed out).
Conversely, the MS Paint ad is, if nothing else, novel. It looks pretty different from most display ads out there. It catches the brain's attention, rather than being caught in the brain's passive ad-filtering heuristics. This may be, if nothing else, a story about attention and awareness.
The novelness lies in the lack of polish. People who work in advertising like ads, and well-polished ads are more pleasing to their eyes.
To the rest of us, advertisements are used as something we don't want to see, but are willing to see, in order to see something we DO want to see. Many of us can pick out the disparity between the polished, targeted ads and the less-focused content it intersperses.
The MS Paint ad was much more readable, IMO. Too many adds try to cram too much into their restricted screen real estate, with the result that they're much less clear.
Factor in that most ads are competing with others alongside (and they all look the same); also as others have mentioned, we've adapted to filter out 'obvious' ads.
It might also have something to do with giving the impression that the game is going to be something novel, quirky or even bizarre, as opposed to another polished but bland EA title.
I think you're absolutely right. I would wager that if every ad on the internet was changed to a five-minute MS Paint scrawl, click-through rates would rapidly go back to where they were originally, if not drop even further.
Interesting, but the author is missing a critical point: the conversion after clicks. When I see the first ad, I clearly understand immediately that it's for a car game. I'm not likely to click, but if I click, I know what I want, and I'm very likely to actually download/install/play the game.
When I see the second ad, I wonder what it is, so yes, I'm more likely to click on it. But I'm not interested in car games. So I'm also much more likely to press the back button as soon as I understand what I'm being sold.
How does the math play out in the end and which ad is betetr? There's no way to tell from that data.
Hey alain94040, I didn't have time to test for CVR :( It would mean split testing demogrpahics/bids and possibly even landers. The point that everyone should take home is just to test every idea that comes to mind :)
We find with domains with landing pages you can have thousands of visits per day that result in no revenue if the reason for the visit (domain typed in on browser bar) is not related to what the person is actually looking for.
Both ads advertise Free, and it's clear that it's a car game in both ads... Also you need to consider the fact that higher CTR means higher relevancy score and potentially lower costs if you're paying by the click.
I think the comments in this thread about ctr vs actual conversions are a bit silly.
Pof's target audience here is "people who like video games" and they're trying to get people to play a free online game. They don't really have anything to teach people in verticals where the consumer is making a measurable commitment. You do not need to be very persuasive to "convert" with that crowd.
Almost all marketing advice more specific than "the customer is always right" and "sex sells" etc. is dependent on the product and the audience.
What this is is a great ad. It's really funny (The speed lines especially. This car does not look very fast, but the speed lines show it's got a lot of heart), and it understands its audience. I think it has broader application than many are giving it credit for, it probably would work with any sincere product (i.e. selling something other than "one simple rule") targeted at younger people that does not solve a "serious" problem. Once you are proposing that people spend more than $40-$50 you are pretty smarmy if you are trying to push them into an impulse purchase
No one should decide on a health insurer based on a MS Paint ad, no matter how hip, nostalgic, and casual they are. On the other hand, you can make a lot of money selling funny ads to people and I think that's what's going on here.
Users haven't seen an ad like this one a thousand times before. Some might not even think this is an ad. Some will be curious enough to actually click.
The MS Paint banner ad is not inherently more effective. If every banner ad on the internet was hand-drawn in MS Paint in 5 minutes, the joke would quickly grow old, and the CTR would vanish.
He also looked only at CTR and not CPA. The sort of people who click (due to the novelty) may not be the sort of people he can convert. His overall CPA may well be lower with the other ad.
I agree on the relevance of the novelty factor. It also came to my mind that this kind of childish MS Paint graphics are becoming somehow pop-culture, a meta-internet-meme if you want. Yesterday I spent half an hour on Christopher Poole's http://canv.as/ (just because I ain't brave enough to go to 4chan) and noticed a lot of these graphics; after some time lurking, I started considering them kinda cool. It's like... well, a brand.
The "test everything" mantra sounds good, but in practice, you generally have only so much data you can afford (in impressions per day, or whatever), and when your CTRs are often 0.1% or lower, you need quite a lot of data to get narrow confidence intervals around your CTRs. Using the basic binomial model, if you have two test conditions, one of which actually does 20-25% better than the other, (say, 0.11% versus 0.09%), your confidence intervals will keep overlapping until you have OOM 1M impressions. This is all just to say that running a whole lot of tests can quickly become expensive an impractical.
While testing some radical, weird treatments can give you valuable perspective, or shed light on the assumptions you've been making, testing every idea is rarely feasible. I would not, for instance, guess that that the author should test different versions of the second ad with colors or number of exclamation points changed.
As someone who has spent $100,000 on advertising over the last few years, at least $10,000 of which was on the Plenty of Fish platform, you're very wrong.
You only need about 1,000-10,000 impressions to get an idea of how a creative performs. Often less. As you get more and more used to each particular advertising platform, you also get a feel for how an ad is performing.
In my business, a difference of 0.02% CTR could mean the difference between earning 30% ROI and 50% ROI - the words "test everything!" mean everything to me and my results.
I agree with your point, however I would like to highlight your use of confidence intervals:
"If two statistics have non-overlapping confidence intervals, they are necessarily significantly different but if they have overlapping confidence intervals, it is not necessarily true that they are not significantly different."
In particular, it doesn't scale well at all with the number of choices (the familiar curse of dimensionality). You can reasonably test two variants, but if your design could plausibly vary along, say, 10 axes (not uncommon), you're going to have trouble collecting sufficient data to cover the whole 10-dimensional theoretical design space. So data-driven design can usually only be applied to a small part of the design space, typically testing a small number of alternatives.
The problem is, that this only works because it's so completely different from all the other ads and because of that reason escapes some of the banner blindness.
We've had similar results when we modified our logo and added mistakes. For example a rectangle Google AdSense Banner with a mirrored version of our Logo or just some crazy saturation affect applied on top of it actually got excellent CTRs. Even though it was kind of unnerving to have all my friends tell me about the mistakes that were in our ads...
Actually advertisers have known this since a long time. Ugly wins more attention. More attention = higher conversion.
The first job of ads is winning attention. You have to fight against all the clutter and stand out. You can do so by a variety of tactics. Use human faces. Use cute looking models. Use ugly fonts or clipart. Use mouse pointers. Use fluorescent colors. Use dashed coupon type borders.
You need to know however that while your conversion rates will increase, the number of complaints you receive will increase too.
This is actually well known among savvy internet marketers. And it's why those tips for belly/whitening/wrinkles "invented by a local mom" always look far from professional.
It works because:
- It stands out from the content
- It defeats ad-blindness
- It's not expected, so it makes you curious
If this style becomes the norm, users will learn to mentally filter ads made in Paint as well.
FIRST: tylerrooney, thanks for posting my blog on Hacker News! Much appreciated. Judging from the comments, we've got some super knowledgeable people here.
So to clarify, this was simply a CTR case study, I know there's another side of the coin for CVR but that would have taken more time and funds than I would have been willing to allocate :(
But hey, open invitation the community here: If you want to submit a 310x110 ad for the purpose of testing against the same demographic that I did, feel free to email it to me: [email protected]. I'll run it for a few days and I'll let you know how well your ad did :) And to make it worth your while, highest CTR ad (from Hacker News members ONLY) gets $100 credit to advertise on https://ads.pof.com. AND you can use your affiliate links so if you make some coin, it's yours to keep (Put it towards your Diablo 3 pre-purchase, yeah? lol) Just grab the direct link for Need For Speed World from your favorite network and send it over. And hell, if your ad beats mine, I'll post it on the POF blog, with your permission of course.
End date for this little challenge.... April 30th?
This is hilarious, but it doesn't take into account one of the major factors of online advertising, and that is branding, or brand awareness. While the official EA ad may not get as many clickthroughs, what it will get is subconcious eyeballs, and given enough impressions of the same ad in various formats, same EA logo, car, game title and branding, you don't need people to click on the ad to start to recognize that there is a new Need for Speed game available. It's similar to flooding the airwaves with a particular TV ad. The goal is awareness, which over time, can lead to a purchase, whether the person decides at some point to purchase online or offline. Maybe they're at GameStop and EA has placed an in-store display with the same EA Need for Speed branding. The person may have forgotten about the game, but walking in to this GameStop, they're memory is refreshed of it by being previously exposed to it through digital or television.
So in short, what's more important? Immediate click-through satisfaction or building real brand recognition that can show greater returns over time, mostly in ways not calculable?
I work with many MBA colleagues and whenever they encounter an ugly, but effective (ie. converting) ad, they can never drop their egos to accept that such an ad can work. They just feel they cannot "stoop that low" to adopt these techniques.
They would often blindly push for "simplicity", "sparse text", "nice picture", and when these ads go out to market, they get absolutely crushed.
One of the tragedies of a big wealthy company is that marketers can continually go out with these crappy creatives that don't sell and there really is no big consequence. It is often written off as a "learning opportunity".
Whereas if you look at the ads of people whose lifeblood depends on selling their product, they may not be the most attractive ads, but the ads that persist over time tend to be effective (ie. they sell). These guys need to eat, and they can only afford to make stupid ineffective ads for so long before they starve! So there are definitely some practical lessons that can be learned from them. They often knowingly or unknowingly follow the principles of advertising legend David Ogilvy.
Their ads tend to hit on direct marketing best practices:
- headline states in plain language what the product does (ie. no MBA jargon/buzz words)
- headline also hooks the reader to read a bit more
- it is clear who the product is for
- copy combats any objections in reader's mind
- no distractions that divert reader from clicking the "Join Now" button
- contains customer testimonials reader can relate to (ie. social proof)
- gives reader enough information on the page to make a decision (ie. none of this sparse text BS if it doesn't make sense)
Is it just me or did anyone else find the first ad terrible and the second one quite good? I think I have been in the game too long.
One of my favorite ads of all time was the stick drawn fat girl with the secret to losing weight. I am sure the person behind that ad has a serious bank account.
I love this kind of social experiment. There are a few theories as to why this resulted in a higher CTR. The obvious ones to me are:
1) "Look at the pretty picture!!"
2) "What the hell? EA is allowing this ad? Did they make it? I gotta see where this goes!"
3) Alternate to "Banner blindness", as @jonnathanson pointed out.
In 1 & 2, I feel like the higher CTR wouldn't matter because people are acting on curiosity of the implementation rather than the product. Once the outcome was revealed, and the banner is seen as no more than a trick, I'd be willing to bet that the orders or pre-orders of the game (in this case, playing for free) stayed roughly the same as if using the other banner. Just a hunch.
If you guys think this ad is on to something, you really should see the ads put up in the SomethingAwful Forums (general subforum: http://forums.somethingawful.com/forumdisplay.php?forumid=1 ). The interesting thing is that many (I'd estimate more than half) of the ads are put up by forum members themselves, making for a range of very informal, hilarious, parodic, often obscene ads that play to every stereotype ironically and knowingly, sometimes made shittily with MSPaint.
And where do those ads lead? Usually to a forum thread where members are playing/raiding an MMO together, or discussing a topic of great interest, or selling a service like painting portraits, web hosting, or resume editing (and in one or two cases, to a discussion of a particularly zeal-inspiring anime series).
This is definitely interesting. I know I'd be more inclined to click something that looks like that ridiculous paint drawing. But this bit bothered me:
Results? 0.049% CTR vs. 0.137 CTR
I hope I'm not the only one confused by this, but the lack of a percent symbol on the 0.137 means he went from 0.049% to 13%, an improvement of almost 300x.
Is this a typo? Did he mean to say he increased his CTR to 0.137% or did he actually increase his CTR by 300x?
Edit: Why is this being downvoted? It's a legitimate question.
[+] [-] ender7|14 years ago|reply
[+] [-] div|14 years ago|reply
At first glance, his simple ad looks like part of the content, the fancy ad just looks like an ad and is filtered out subconsciously.
Definitely a factor to take into account when you're targeting gamers I'd think.
[+] [-] wtvanhest|14 years ago|reply
Even more importantly, a high CTR with lower conversion rate would drive up your costs while lowering your sales.
In this case the first ad is very clearly for a game with a very clear target market. If I am in to racing games, I may click it and I may buy it.
The second ad could appeal to people strickly on novelty which means you may get a lot of customers who do not care about the product.
While the results of this post are very interesting and the conclusion to test everything is still good, we should look at the conversion rate to see which ad is actually more effective.
[+] [-] seldo|14 years ago|reply
I would like to see an A/B test of the same basic artwork on a black background vs. the white one.
[+] [-] jpeterson|14 years ago|reply
[+] [-] btilly|14 years ago|reply
People love that word.
[+] [-] dmvaldman|14 years ago|reply
[+] [-] jonnathanson|14 years ago|reply
Conversely, the MS Paint ad is, if nothing else, novel. It looks pretty different from most display ads out there. It catches the brain's attention, rather than being caught in the brain's passive ad-filtering heuristics. This may be, if nothing else, a story about attention and awareness.
[+] [-] stephengillie|14 years ago|reply
To the rest of us, advertisements are used as something we don't want to see, but are willing to see, in order to see something we DO want to see. Many of us can pick out the disparity between the polished, targeted ads and the less-focused content it intersperses.
[+] [-] screwt|14 years ago|reply
Factor in that most ads are competing with others alongside (and they all look the same); also as others have mentioned, we've adapted to filter out 'obvious' ads.
[+] [-] NameNickHN|14 years ago|reply
[+] [-] kronusaturn|14 years ago|reply
[+] [-] mikeash|14 years ago|reply
[+] [-] rhizome|14 years ago|reply
[+] [-] alain94040|14 years ago|reply
When I see the second ad, I wonder what it is, so yes, I'm more likely to click on it. But I'm not interested in car games. So I'm also much more likely to press the back button as soon as I understand what I'm being sold.
How does the math play out in the end and which ad is betetr? There's no way to tell from that data.
[+] [-] benpof|14 years ago|reply
[+] [-] larrys|14 years ago|reply
We find with domains with landing pages you can have thousands of visits per day that result in no revenue if the reason for the visit (domain typed in on browser bar) is not related to what the person is actually looking for.
[+] [-] jczhang|14 years ago|reply
[+] [-] bgilroy26|14 years ago|reply
Pof's target audience here is "people who like video games" and they're trying to get people to play a free online game. They don't really have anything to teach people in verticals where the consumer is making a measurable commitment. You do not need to be very persuasive to "convert" with that crowd.
Almost all marketing advice more specific than "the customer is always right" and "sex sells" etc. is dependent on the product and the audience.
What this is is a great ad. It's really funny (The speed lines especially. This car does not look very fast, but the speed lines show it's got a lot of heart), and it understands its audience. I think it has broader application than many are giving it credit for, it probably would work with any sincere product (i.e. selling something other than "one simple rule") targeted at younger people that does not solve a "serious" problem. Once you are proposing that people spend more than $40-$50 you are pretty smarmy if you are trying to push them into an impulse purchase
No one should decide on a health insurer based on a MS Paint ad, no matter how hip, nostalgic, and casual they are. On the other hand, you can make a lot of money selling funny ads to people and I think that's what's going on here.
[+] [-] klbarry|14 years ago|reply
1) Ethos (your perceived character) is the most important, with regards to pathos (emotion) and logos (logic)
2) People make judgments by comparison/anchoring.
3) People process information best from stories.
4) People are foremost interested in things that affect them.
5) Breaking patterns gets attention.
6) People look to other people's decisions when making decisions.
7) People will believe things more easily that fit their pre-existent mindset. The converse is also true.
8) People handle one idea at a time best.
9) People want more choices, but are happier with fewer.
10) People decide first, then rationalize - If people are stuck with something, they will like it more over time.
11) Experience is memory, the last part of the experience is weighted heavily.
[+] [-] joblessjunkie|14 years ago|reply
Users haven't seen an ad like this one a thousand times before. Some might not even think this is an ad. Some will be curious enough to actually click.
The MS Paint banner ad is not inherently more effective. If every banner ad on the internet was hand-drawn in MS Paint in 5 minutes, the joke would quickly grow old, and the CTR would vanish.
[+] [-] rlpb|14 years ago|reply
[+] [-] gghh|14 years ago|reply
[+] [-] abeppu|14 years ago|reply
While testing some radical, weird treatments can give you valuable perspective, or shed light on the assumptions you've been making, testing every idea is rarely feasible. I would not, for instance, guess that that the author should test different versions of the second ad with colors or number of exclamation points changed.
[+] [-] Cor|14 years ago|reply
You only need about 1,000-10,000 impressions to get an idea of how a creative performs. Often less. As you get more and more used to each particular advertising platform, you also get a feel for how an ad is performing.
In my business, a difference of 0.02% CTR could mean the difference between earning 30% ROI and 50% ROI - the words "test everything!" mean everything to me and my results.
[+] [-] michaelbarton|14 years ago|reply
"If two statistics have non-overlapping confidence intervals, they are necessarily significantly different but if they have overlapping confidence intervals, it is not necessarily true that they are not significantly different."
http://www.cscu.cornell.edu/news/statnews/stnews73.pdf
[+] [-] _delirium|14 years ago|reply
[+] [-] lucasvo|14 years ago|reply
We've had similar results when we modified our logo and added mistakes. For example a rectangle Google AdSense Banner with a mirrored version of our Logo or just some crazy saturation affect applied on top of it actually got excellent CTRs. Even though it was kind of unnerving to have all my friends tell me about the mistakes that were in our ads...
[+] [-] ankeshk|14 years ago|reply
The first job of ads is winning attention. You have to fight against all the clutter and stand out. You can do so by a variety of tactics. Use human faces. Use cute looking models. Use ugly fonts or clipart. Use mouse pointers. Use fluorescent colors. Use dashed coupon type borders.
You need to know however that while your conversion rates will increase, the number of complaints you receive will increase too.
[+] [-] corin_|14 years ago|reply
[+] [-] patton01|14 years ago|reply
[+] [-] acangiano|14 years ago|reply
It works because:
- It stands out from the content
- It defeats ad-blindness
- It's not expected, so it makes you curious
If this style becomes the norm, users will learn to mentally filter ads made in Paint as well.
[+] [-] felipemnoa|14 years ago|reply
That is the first flag that it is a scam.
[+] [-] eob|14 years ago|reply
The second ad says "I'm a human being, probably with a sense of humor"
[+] [-] benpof|14 years ago|reply
So to clarify, this was simply a CTR case study, I know there's another side of the coin for CVR but that would have taken more time and funds than I would have been willing to allocate :(
But hey, open invitation the community here: If you want to submit a 310x110 ad for the purpose of testing against the same demographic that I did, feel free to email it to me: [email protected]. I'll run it for a few days and I'll let you know how well your ad did :) And to make it worth your while, highest CTR ad (from Hacker News members ONLY) gets $100 credit to advertise on https://ads.pof.com. AND you can use your affiliate links so if you make some coin, it's yours to keep (Put it towards your Diablo 3 pre-purchase, yeah? lol) Just grab the direct link for Need For Speed World from your favorite network and send it over. And hell, if your ad beats mine, I'll post it on the POF blog, with your permission of course.
End date for this little challenge.... April 30th?
[+] [-] hkuo|14 years ago|reply
So in short, what's more important? Immediate click-through satisfaction or building real brand recognition that can show greater returns over time, mostly in ways not calculable?
[+] [-] valhallarecords|14 years ago|reply
They would often blindly push for "simplicity", "sparse text", "nice picture", and when these ads go out to market, they get absolutely crushed.
One of the tragedies of a big wealthy company is that marketers can continually go out with these crappy creatives that don't sell and there really is no big consequence. It is often written off as a "learning opportunity".
Whereas if you look at the ads of people whose lifeblood depends on selling their product, they may not be the most attractive ads, but the ads that persist over time tend to be effective (ie. they sell). These guys need to eat, and they can only afford to make stupid ineffective ads for so long before they starve! So there are definitely some practical lessons that can be learned from them. They often knowingly or unknowingly follow the principles of advertising legend David Ogilvy.
Their ads tend to hit on direct marketing best practices:
- headline states in plain language what the product does (ie. no MBA jargon/buzz words)
- headline also hooks the reader to read a bit more
- it is clear who the product is for
- copy combats any objections in reader's mind
- no distractions that divert reader from clicking the "Join Now" button
- contains customer testimonials reader can relate to (ie. social proof)
- gives reader enough information on the page to make a decision (ie. none of this sparse text BS if it doesn't make sense)
[+] [-] nosse|14 years ago|reply
[+] [-] jcampbell1|14 years ago|reply
One of my favorite ads of all time was the stick drawn fat girl with the secret to losing weight. I am sure the person behind that ad has a serious bank account.
[+] [-] tomkin|14 years ago|reply
1) "Look at the pretty picture!!"
2) "What the hell? EA is allowing this ad? Did they make it? I gotta see where this goes!"
3) Alternate to "Banner blindness", as @jonnathanson pointed out.
In 1 & 2, I feel like the higher CTR wouldn't matter because people are acting on curiosity of the implementation rather than the product. Once the outcome was revealed, and the banner is seen as no more than a trick, I'd be willing to bet that the orders or pre-orders of the game (in this case, playing for free) stayed roughly the same as if using the other banner. Just a hunch.
[+] [-] creamyhorror|14 years ago|reply
And where do those ads lead? Usually to a forum thread where members are playing/raiding an MMO together, or discussing a topic of great interest, or selling a service like painting portraits, web hosting, or resume editing (and in one or two cases, to a discussion of a particularly zeal-inspiring anime series).
Some purely parodic examples can be found here:
http://acapella.harmony-central.com/showthread.php?2332472-A...
And here is the current roster of ads, though many are from external advertisers and hence less funny:
http://forums.somethingawful.com/adlist.php
(Note that "goons" is the moniker for forum members, and many ads target them specifically with "goon discounts" and so on.)
Marketers with an attitude and something of a free hand could take a leaf from them.
[+] [-] chops|14 years ago|reply
Results? 0.049% CTR vs. 0.137 CTR
I hope I'm not the only one confused by this, but the lack of a percent symbol on the 0.137 means he went from 0.049% to 13%, an improvement of almost 300x.
Is this a typo? Did he mean to say he increased his CTR to 0.137% or did he actually increase his CTR by 300x?
Edit: Why is this being downvoted? It's a legitimate question.
[+] [-] jdkramar|14 years ago|reply
[+] [-] Cor|14 years ago|reply
People are more likely to trust something that looks amateur since it feels more like a recommendation than an advert.
Mr.Green, a well renowned CPA Marketer/Blogger, wrote a brilliant article about this. You can find it here - http://www.mrgreen.am/affiliate-marketing/the-ugly-truth/
[+] [-] benpof|14 years ago|reply
Some good lessons to be learned from that post.
[+] [-] Shpigford|14 years ago|reply
With the second it's very much "Haha, I wonder what this stupid ad goes to?" and then you just abandon it.
CTR means nothing without a conversion of some sort on the other side.