When I worked in adtech, I proposed a simple solution to ad fraud on our platform - assume any UA coming from an IP range owned by a cloud provider was a bot and don't show them ads, don't count their impressions.
Worst case, people using a VPN hosted on EC2 might not see an ad, but I was certain they'd cope. Best case, higher quality traffic increases yields.
The business team vetoed that, it would adversely impact their numbers in the current quarter, and well, a significant proportion of their total comp was bonuses tied to those numbers. Also, the shareholders seemed to be of the mindset that has infected cryptocurrency speculators where no matter what, the spice must flow, wait, I mean, the number go up.
Then there were all the "bonuses" and "incentives" negotiated with big advertisers or publishers. We just referred to them as kickbacks, because they totally are, but labeling something in code as kickbackPercentage was verboten. Just in case.
Advertising has always been corrupt. Bots just corrupt it a little further.
I made an ad fraud startup which ran into the exact same problem. With video ads, advertisers will pay 10x more to be in a “user initiated” video player (the user clicked or there was a reasonable expectation there’d be a video on that page - YouTube, camera icon on preceding link, etc).
But more often then not, those ads are running in an autoplay context (sticky video players in the bottom corner, large players in the footer, etc).
I built a mechanism which could tell the difference (viewport, fixed positioning, fingerprint elements on the page, provide a screenshot of the offended page with your ad on it, and various other signals).
We ran tests with advertisers, agencies, demand side platforms (dsps), server side platforms (ssps). We’d compare their existing fraud numbers (very low) with our reports (80% fraud).
Within each company, we’d have a champion saying “this is going to change everything. Quality supply, better performance metrics, etc etc”. Then we’d get introduced to a team whose compensation was driven by “percentage of spend” - every deal died at this stage.
My opinion: if your company is using online advertising tied to conversions (someone buying something, or potentially signing up for something) then you’ll be fine as long as it’s economical. But if your ads are for brand awareness, be extremely careful running on any website which isn’t in the top 10 sites on the internet.
> I proposed a simple solution to ad fraud on our platform - assume any UA coming from an IP range owned by a cloud provider was a bot
Ad fraud is an adversarial game, and you can't 'solve' it with simple things like this. You'll temporarily cause a lot of pain for the bot operators (yay!) but then they'll adjust and start sending the traffic from botnets (hacked consumer devices). Which is already what they do when trying to defraud advertisers on networks that take fraud seriously (which it sounds like your company didn't).
Currently working in ad tech. Your solution wouldn’t work today because so many current and upcoming privacy solutions being rolled out across platforms are run through a VPN or VPN-like service, and many with IP rotation. Many of the bots are using these services so that their IP appears as coming from a legitimate privacy service like Cloudflare.
The privacy service providers don’t have good incentives to attempt to identify which users are bots because it would undermine their privacy claims. Tough situation.
We do think we have a solution to this but it’s unlikely to last in a privacy arms race. There’s really only one logical place this ends with the ad model surviving, and it’s not pretty.
12 year adtech veteran here. Ad fraud is a business problem, not a tech problem.
All the major solutions are well known but not implemented because of the massive effect it would have on business for everyone down the chain for advertisers.
The entire industry meets every year. They can figure this out and fix most big problems in a few hours, but incentives are not aligned for that to happen.
Advertising is corrupt, but much of business (and public perception) is corrupt in a much similar way. What you are seeing is a lack of integrity shared by business leaders who don't ask for rigor when their minions report higher rates of "engagement". There's too much of an incentive for the people in charge to be satisfied with someone telling them that something is "up", and too much incentive for those coming up with the numbers to not perform due diligence. I've seen this multiple times firsthand.
Our number of subscribers is through the roof! Mr. Corneroffice will be thrilled! Wait as sec... 90% of them are people we gave away free trials to. Ahh... fuck it. Mr. Corneroffice isn't gonna know or care, and neither will the investors. We're BigCo! Everything we say is considered official by most people anyway. After all, Mr. Corneroffice told us "fail fast, fail early" and "fake it 'til you make it". If he secretly knew I was including the free subscribers, he'd be proud! But he can't know because then I might not get that sweet, sweet bonus. Besides, it's not a lie, right?
>When I worked in adtech, I proposed a simple solution to ad fraud on our platform - assume any UA coming from an IP range owned by a cloud provider was a bot and don't show them ads, don't count their impressions.
Wait, is that not automatic on every single Ad network out there? WTF?
It is wonderful that after cable TV, where a few thousand of individuals' viewing habits determined who watched what, we now have an absolute stellar system where everything can be measured, yet, nobody wants to see the real numbers. :D
> The business team vetoed that, it would adversely impact their numbers in the current quarter, and well, a significant proportion of their total comp was bonuses tied to those numbers
Furthermore they were not spending their own money to buy those ads. Shareholders money is (almost) for free. In my experience companies run by the founders with little or no investors are always more careful about the money they spend.
> Advertising has always been corrupt. Bots just corrupt it a little further.
I had friends in college who did door to door sales for a major cable company. At the beginning of every week they had to stop by the office to pick up a stack of flyers and at the end of every week they had to mark off the houses they visited on a map. They got paid per flyer distributed and every flyer was distributed directly into the trash can.
They got paid, no one got harassed by door to door sales people, and the cable company got their marked off maps. Everyone was happy.
When I worked at a company that got like almost 100% of their revenue from ads, we got a very serious warning from an ad agency to stop serving our ads to bots. They gave us like 7 days to fix our traffic or we'd get removed, effectively killing our company. Scary time to be an engineer that morning! That was like 6-7 years ago, though! It's kinda crazy that things have changed this much.
I’ve worked for a company that used an Azure hosted internet gateway/firewall that all their sites connected through using an MPLS VPN. So there are definitely some non-not users with traffic coming from there. But yeah, happy enough to not see ads when piping traffic through those ranges!
So how about starting by not actually blocking the bots, but publishing two numbers, the original calc including bots, and the adjusted with bots removed. Try to estimate said number for competitors as well. Then after some years try to move to using mainly the second set of numbers.
"assume any UA coming from an IP range owned by a cloud provider was a bot and don't show them ads"
As someone who is trying to get into google ads for their startup, what's the best way to do this ? Is there a nice Google Ads 101 other than Google's own tutorials ?
> it would adversely impact their numbers in the current quarter, and well, a significant proportion of their total comp was bonuses tied to those numbers
So much about how terrible things are in general can be tied back to this one simple concept.
If you are paying any bill for ads, make sure you only pay for conversions. Ie. unless the customer actually buys your product, then don't pay anything. Eg. instead of paying $0.002 per view of your ad, or $0.10 per click of your ad, pay $6 per person who clicks your ad and then buys your product.
That way, ad fraud doesn't affect you. Plenty of ad providers allow such an option. Internally, they convert your payments per conversion into payments per click and then payments per impression, but importantly this shifts the incentive to get rid of bots to them, not you - you no longer care. The provider also has an incentive to send you customers who will actually buy your product, and to send the bots and web scrapers to their other clients.
A few years ago, I tried to help out a locally owned gym do an experiment with Google Ad-Words. The place small storefront in a run-down strip mall straight out of the 1960s, but it had very good equipment for people that were in the know and it was 24 hours.
The owner was a nice guy but he relied entirely on word-of-mouth and his technical skills extended to making expletive laced, 128-point-font signs reminding people to put equipment away in Word. I got the sense the business was not going well after a protracted battle with a former business partner, where the business partner just wanted to liquidate the assets and call it a day. He managed to buy the business partner out and continued to operate the place---sans a few treadmills. I decided to try and help him out by volunteering my time and some money to see if a small, targeted ad buy from Google could help him out. It sure beat finding another place like this.
The dashboard made it look very targeted geographically. But, when I investigated where they were all coming from it was 95% international and from affiliates listings from very low quality sites with no information on them. A month and $300 later, I discontinued this experiment because the traffic was entirely bot driven.
From that experience, I've come to the conclusion that online Ad Tech is 99% a scam.
> Yet the digital advertising industry has grown so accustomed to working with inflated numbers that few are willing to unmask the fake clicks
That is a funny reason for why this phenomenon might have grown so big in the first place. The people in the ad-buying company who are in the best position to 'smell' when the numbers are off are also the ones most likely to individually benefit from inflated numbers (KPIs, number go up,...).
A principal-agent problem at its finest, that unfortunately might drag some white-collar workers into being at least morally complicit in cyber crime.
From my definition of white collar (office worker sitting on chair in front of computer all day), and blue collar (factory/shop/traveling/working with hands not computer), I would have thought white collar workers were the ones most likely capable of committing cyber crimes.
If you ask me, we should simply delete the entire advertisement sector—influencers included—and start from scratch. The amount of wasted resources (i.e. attention, money, electricity, bandwidth, time, smart people) is just insane! The benefits I see are mostly lot's of jobs and means for the biggest businesses to sell more stuff. That can hardly be worth it.
I know I'm dreaming and it will never happen, but still... let me dream.
The digital economy would doubtless take a hard hit if we knew the true scale of synthetic engagement out there. It's just easier not to look too hard. Reminds me of a joke:
"Doctor, my brother has lost his mind. He think he's a chicken."
"Have you tried telling him he's not a chicken?"
"No. You see, the thing is, we really need the eggs..."
> No. You see, the thing is, we really need the eggs.
Isn't that the opposite of the problem here? If the bots actually generated sales, the adbuyers would likely be quite happy. This seems more like the Emperor's New Clothes, if you're looking for a metaphor.
Must be a European joke. In the States I'm guessing it costs more to see a doctor than just go buy eggs.
And, yes, i know my euro-taxes essentialy pay for my doc, but a week in hospital that cost me €13.50, I'll live with. As well as a year's worth of post-care and meds....€0.00.
> “After all, who cares if big brands waste their money showing ads to bots?”
I'm not so sure this affects only big brands.
I've experienced that in march 2020. I ran ads in a very popular platform and was fairly happy about the results -- modest, but steady.
When the pandemics hit, my target-market shifted their focus away from my sort of product, and such shift happened almost overnight. Since I used to get a reasonable amount of real visitors, I thought "well, I believe the clicks on my ads will decrease sharply now".
Guess what? I kept getting the same amount of clicks on average, and my budget of course kept burning at the same rate. The only difference was that the "visits" started to last only a few seconds, the pages were not scrolled anymore, and the "visitors" never sent an inquiry.
> The case started when Uber pulled all online advertising and discovered barely any drop in app installs or sales.
Uber is a bad example I think. Not only they've been in the news so many times advertising is probably irellevant, there is an actual "natural" ceiling for demand for Uber in an area.
They may have to compete with ads in places where they have competition, but not overall.
> The technology to detect and block bots already exists
Yeah, right. When I close the site because i can't be arsed to look for palm trees in 120x80 photos, I'm a bot.
Honestly, I'd much rather they show their ads to bots than to humans. Hopefully, the day will come when every human is using a competent adblocker, but the ad industries numbers don't go down thanks to the bots, and people can get paid for content that everyone enjoys for free.
“Victimless crime” - I think not. Most affected are small businesses and startups with precious marketing dollars that can no longer afford to run an App Install campaign that reaches real users instead of bots. The big dogs can brute force their way through this. I’d venture to say that Ad fraud is a moat in itself.
If they're getting decent ROI on their ad spend, what does it matter? TV ads were normally shown to empty rooms while their occupants went to make themselves a drink. Still worth buying.
It's amazing how much compute time - both hardware and humans - is dedicated to figuring out how to show which ads to people.. and it turns out that many of the "views" aren't even people.
That's not my problem. Advertisers are part of the problem, they take in huge sums of money and feed the Google+fb monopoly, without ever knowing how their money is wasted, they "trust" them. There is nothing more opaque than google, the company that controls both the supply and demand of search ads. If you willingly participate in that market,i m sorry, not sorry
I am not an advertiser or anything, but businesses dont "trust" them. The moment businesses see an alternate option with more transparency and somewhat similar results - They will jump ship. Although we are way far from anything like that coming up (on that scale)
Its not trust so much as monopoly access to the digital ad market. Google has so much prime inventory (search results + youtube) that it's impossible to avoid it if you want any kind of scale for your campaigns.
I worked for a print ad business many years ago. They were basically running the same business model with the paper. Ads were placed in glossy and cheap/free pos magazines (cheap traffic) that mostly landed in the trash after collecting dust on the shelf. The bots consisted of distributors claiming high quality distribution while throwing the paper right into the trash. Everyone knew and hardly anyone cared because incentives were mostly aligned (mostly big brands).
I was hired to help with the transition to digital. Guess how that worked out. Next I tried running my own online marketing business. I thought not deceiving customers and actually providing real value was something clients would pay for. Some did, but competing with all the fraud became to tiresome eventually. Not going to work in this industry every again unless there are alternatives (includes scrubbing toilets)
Companies should be doing testing to check the efficiency of their ad spending. If they're happy with the Spend/Sales ratio then it is irrelevant whether they pay $x for 1000 impressions->10 real views->1 sale or $x for 10 impressions->10 real views->1 sale.
If they are not doing their homework, that is on them.
And frankly, 90%+ of advertising is a blight on the environment whether it's online or broadcast or physical so who cares.
Years ago our company was running an ad campaign on social media. Curious on the results I spent time looking into a randomly selected pool of the “users” that had been clicking on our ads. It quickly became clear that at least half (if not a lot more) of the clicks were from what appeared to be fake users or bots. We shut the whole ad spend there off immediately.
I think fraud is ubiquitous in advertising and often the publishers are to blame. Frequently on mobile devices the ads cover the content and you wind up clicking on the ad when you are trying to hide it.
Also I think it is not so accidental that mobile web sites have so many layout shifts because each layout shift is a chance you click on the wrong link and ka-ching!
[+] [-] EdwardDiego|3 years ago|reply
Worst case, people using a VPN hosted on EC2 might not see an ad, but I was certain they'd cope. Best case, higher quality traffic increases yields.
The business team vetoed that, it would adversely impact their numbers in the current quarter, and well, a significant proportion of their total comp was bonuses tied to those numbers. Also, the shareholders seemed to be of the mindset that has infected cryptocurrency speculators where no matter what, the spice must flow, wait, I mean, the number go up.
Then there were all the "bonuses" and "incentives" negotiated with big advertisers or publishers. We just referred to them as kickbacks, because they totally are, but labeling something in code as kickbackPercentage was verboten. Just in case.
Advertising has always been corrupt. Bots just corrupt it a little further.
[+] [-] tomhallett|3 years ago|reply
But more often then not, those ads are running in an autoplay context (sticky video players in the bottom corner, large players in the footer, etc).
I built a mechanism which could tell the difference (viewport, fixed positioning, fingerprint elements on the page, provide a screenshot of the offended page with your ad on it, and various other signals).
We ran tests with advertisers, agencies, demand side platforms (dsps), server side platforms (ssps). We’d compare their existing fraud numbers (very low) with our reports (80% fraud).
Within each company, we’d have a champion saying “this is going to change everything. Quality supply, better performance metrics, etc etc”. Then we’d get introduced to a team whose compensation was driven by “percentage of spend” - every deal died at this stage.
My opinion: if your company is using online advertising tied to conversions (someone buying something, or potentially signing up for something) then you’ll be fine as long as it’s economical. But if your ads are for brand awareness, be extremely careful running on any website which isn’t in the top 10 sites on the internet.
[+] [-] jefftk|3 years ago|reply
Ad fraud is an adversarial game, and you can't 'solve' it with simple things like this. You'll temporarily cause a lot of pain for the bot operators (yay!) but then they'll adjust and start sending the traffic from botnets (hacked consumer devices). Which is already what they do when trying to defraud advertisers on networks that take fraud seriously (which it sounds like your company didn't).
(Disclosure: I used to work on ads at Google)
[+] [-] IggleSniggle|3 years ago|reply
The privacy service providers don’t have good incentives to attempt to identify which users are bots because it would undermine their privacy claims. Tough situation.
We do think we have a solution to this but it’s unlikely to last in a privacy arms race. There’s really only one logical place this ends with the ad model surviving, and it’s not pretty.
[+] [-] manigandham|3 years ago|reply
All the major solutions are well known but not implemented because of the massive effect it would have on business for everyone down the chain for advertisers.
The entire industry meets every year. They can figure this out and fix most big problems in a few hours, but incentives are not aligned for that to happen.
[+] [-] ravenstine|3 years ago|reply
Our number of subscribers is through the roof! Mr. Corneroffice will be thrilled! Wait as sec... 90% of them are people we gave away free trials to. Ahh... fuck it. Mr. Corneroffice isn't gonna know or care, and neither will the investors. We're BigCo! Everything we say is considered official by most people anyway. After all, Mr. Corneroffice told us "fail fast, fail early" and "fake it 'til you make it". If he secretly knew I was including the free subscribers, he'd be proud! But he can't know because then I might not get that sweet, sweet bonus. Besides, it's not a lie, right?
[+] [-] mikeodds|3 years ago|reply
[+] [-] ClumsyPilot|3 years ago|reply
But every time I propose that market without rules at all leads to corriluption and inefficiency, people get up in arms
[+] [-] nonasktell|3 years ago|reply
Wait, is that not automatic on every single Ad network out there? WTF?
How long ago was that?
[+] [-] lofaszvanitt|3 years ago|reply
...
[+] [-] pmontra|3 years ago|reply
Furthermore they were not spending their own money to buy those ads. Shareholders money is (almost) for free. In my experience companies run by the founders with little or no investors are always more careful about the money they spend.
[+] [-] bosie|3 years ago|reply
your firm's higher ups received bonsues/incentives from publishers and/or big advertisers directly?
[+] [-] elliekelly|3 years ago|reply
I had friends in college who did door to door sales for a major cable company. At the beginning of every week they had to stop by the office to pick up a stack of flyers and at the end of every week they had to mark off the houses they visited on a map. They got paid per flyer distributed and every flyer was distributed directly into the trash can.
They got paid, no one got harassed by door to door sales people, and the cable company got their marked off maps. Everyone was happy.
[+] [-] kazinator|3 years ago|reply
Wee, I'm tunneling by browser session through IP address on a cloud provider so I don't have to see ads!
Meanwhile, ads are shown to my cracked residential router which has been enlisted into a botnet.
[+] [-] lovehashbrowns|3 years ago|reply
[+] [-] stephen_g|3 years ago|reply
[+] [-] sharperguy|3 years ago|reply
[+] [-] willis936|3 years ago|reply
[+] [-] codegeek|3 years ago|reply
As someone who is trying to get into google ads for their startup, what's the best way to do this ? Is there a nice Google Ads 101 other than Google's own tutorials ?
[+] [-] wnevets|3 years ago|reply
So much about how terrible things are in general can be tied back to this one simple concept.
[+] [-] dekken_|3 years ago|reply
[+] [-] winddude|3 years ago|reply
[+] [-] londons_explore|3 years ago|reply
That way, ad fraud doesn't affect you. Plenty of ad providers allow such an option. Internally, they convert your payments per conversion into payments per click and then payments per impression, but importantly this shifts the incentive to get rid of bots to them, not you - you no longer care. The provider also has an incentive to send you customers who will actually buy your product, and to send the bots and web scrapers to their other clients.
[+] [-] ok123456|3 years ago|reply
The owner was a nice guy but he relied entirely on word-of-mouth and his technical skills extended to making expletive laced, 128-point-font signs reminding people to put equipment away in Word. I got the sense the business was not going well after a protracted battle with a former business partner, where the business partner just wanted to liquidate the assets and call it a day. He managed to buy the business partner out and continued to operate the place---sans a few treadmills. I decided to try and help him out by volunteering my time and some money to see if a small, targeted ad buy from Google could help him out. It sure beat finding another place like this.
The dashboard made it look very targeted geographically. But, when I investigated where they were all coming from it was 95% international and from affiliates listings from very low quality sites with no information on them. A month and $300 later, I discontinued this experiment because the traffic was entirely bot driven.
From that experience, I've come to the conclusion that online Ad Tech is 99% a scam.
[+] [-] ezoe|3 years ago|reply
[+] [-] c7b|3 years ago|reply
That is a funny reason for why this phenomenon might have grown so big in the first place. The people in the ad-buying company who are in the best position to 'smell' when the numbers are off are also the ones most likely to individually benefit from inflated numbers (KPIs, number go up,...).
A principal-agent problem at its finest, that unfortunately might drag some white-collar workers into being at least morally complicit in cyber crime.
[+] [-] lotsofpulp|3 years ago|reply
[+] [-] civilized|3 years ago|reply
[+] [-] oldstrangers|3 years ago|reply
[+] [-] emaro|3 years ago|reply
I know I'm dreaming and it will never happen, but still... let me dream.
[+] [-] Oarch|3 years ago|reply
"Doctor, my brother has lost his mind. He think he's a chicken." "Have you tried telling him he's not a chicken?" "No. You see, the thing is, we really need the eggs..."
[+] [-] c7b|3 years ago|reply
Isn't that the opposite of the problem here? If the bots actually generated sales, the adbuyers would likely be quite happy. This seems more like the Emperor's New Clothes, if you're looking for a metaphor.
[+] [-] mountainb|3 years ago|reply
[+] [-] InCityDreams|3 years ago|reply
And, yes, i know my euro-taxes essentialy pay for my doc, but a week in hospital that cost me €13.50, I'll live with. As well as a year's worth of post-care and meds....€0.00.
Eggs are getting a bit pricey now, though.
[+] [-] myth2018|3 years ago|reply
I'm not so sure this affects only big brands.
I've experienced that in march 2020. I ran ads in a very popular platform and was fairly happy about the results -- modest, but steady.
When the pandemics hit, my target-market shifted their focus away from my sort of product, and such shift happened almost overnight. Since I used to get a reasonable amount of real visitors, I thought "well, I believe the clicks on my ads will decrease sharply now".
Guess what? I kept getting the same amount of clicks on average, and my budget of course kept burning at the same rate. The only difference was that the "visits" started to last only a few seconds, the pages were not scrolled anymore, and the "visitors" never sent an inquiry.
[+] [-] nottorp|3 years ago|reply
Uber is a bad example I think. Not only they've been in the news so many times advertising is probably irellevant, there is an actual "natural" ceiling for demand for Uber in an area.
They may have to compete with ads in places where they have competition, but not overall.
> The technology to detect and block bots already exists
Yeah, right. When I close the site because i can't be arsed to look for palm trees in 120x80 photos, I'm a bot.
[+] [-] NoGravitas|3 years ago|reply
[+] [-] dkobia|3 years ago|reply
[+] [-] throw827474737|3 years ago|reply
[+] [-] robertlagrant|3 years ago|reply
[+] [-] caseysoftware|3 years ago|reply
[+] [-] seydor|3 years ago|reply
[+] [-] iKevinShah|3 years ago|reply
[+] [-] manigandham|3 years ago|reply
[+] [-] iamacyborg|3 years ago|reply
https://www.mql.fm/002-60-million-60-billion-ad-fraud-questi...
[+] [-] jrm4|3 years ago|reply
Ah, yes, "advertising," that fine institution representing the best the world has to offer in purity and honesty...
[+] [-] kkkrist|3 years ago|reply
I was hired to help with the transition to digital. Guess how that worked out. Next I tried running my own online marketing business. I thought not deceiving customers and actually providing real value was something clients would pay for. Some did, but competing with all the fraud became to tiresome eventually. Not going to work in this industry every again unless there are alternatives (includes scrubbing toilets)
[+] [-] LatteLazy|3 years ago|reply
If they are not doing their homework, that is on them.
And frankly, 90%+ of advertising is a blight on the environment whether it's online or broadcast or physical so who cares.
[+] [-] JCM9|3 years ago|reply
[+] [-] PaulHoule|3 years ago|reply
Also I think it is not so accidental that mobile web sites have so many layout shifts because each layout shift is a chance you click on the wrong link and ka-ching!