My god, the quote from that "white hat" SEO guy just defies belief:
> There will certainly be webmasters out there who will strip you down to the bone asking for money in exchange of link removals. These are the most soulless snake oil salesmen on earth
To say that about webmasters, already victims of years of abusive SEO spamming, when they then refuse to help an abusive site clean up its own mess for free .. I have no words. Could anyone possibly be more of a self-interested, myopic, egocentric prick?
My rock-bottom opinion of pretty much anyone who has anything to do with active SEO is re-confirmed for the thousandth time.
"To say that about webmasters, already victims of years of abusive SEO spamming"
I suspect this comment was aimed at webmasters not who were the victims of spam, but those that profited from it. It's those who run directory sites and article farms that would accept cash to add your link, and now that people want the links removed, they're double dipping. And these people are indeed scummy.
I can understand someone getting frustrated. This forum may have great information, but it is also causing a lot of harm, possibly leading people to dangerous sites. Is it the owners responsibility to do something about that? I think so.
Just as you have lots of machines that have been compromised and are part of a botnet.
So what if the site requesting the link take downs was a victim of someone else spamming links out pointing at their site? It's not a white/black situation, there are many ways those links could come about and there are legitimate SEOs out there requesting they be taken down.
Blaming google for seo spam is not productive - spammers are legion and in constant battle to game google's ranking system, whatever it currently is. This won't change as long as google is used to search for things.
There is a solution to his particular problem: better forum software.
I don't want to trivialize the problem of writing spam-resistant forum software but it's not such an insurmontable problem (this forum being a proof positive for that).
For the reference: I've been running a fairly popular forum (http://forums.fofou.org/sumatrapdf) for several years, using forum software that I wrote.
I occasionally get human spam i.e. someone writes a post with the only purpose of linking to some other website. I just hellban them after I see the post in my rss reader.
And I didn't even write any special anti-spam code (other than hell-banning), because I'm lazy. I can easily come up with simple ideas e.g. putting all posts that contain links in moderation queue.
To reiterate: his problems were caused by a crappy forum software that didn't do much to protect from seo spam.
I don't really know how current best off-the-shelf forums fare in this respect.
I would rather not spend my time maintaining my own forum software so I have high hopes for http://www.discourse.org. I'm sure StackOverflow had plenty of spam problems so discourse people should understand the problem.
This really only works if you have a very small site. No offense, but Alexa's rank for fofou.org is over a million. No spammer is going to spend time writing custom logic to break your captcha. I bet you don't even need a captcha -- just having custom forum software is enough to stop all automated spam.
If your site was in the top 1,000, you'd have a different experience since the spammers would be adapting to every one of your exotic captchas no matter how many times you change it.
I'd attribute it to your custom software. Break phpBB and you can infect thousands of sites. Break your custom software you get to infect 1 site.
I ran a forum using SimpleMachines. Turned on all the spam fighting stuff. Didn't matter. Bots or Mechanical Turks made 10-30 new accounts a day. I tried moderating the accounts but who wants to spend time every day to delete the [email protected] account.
He's not blaming google for the original blog spam, he's blaming google for the deluge of emails requesting link removal. Since they are directly encouraging these emails by holding out the hope of rehabilitation they are responsible.
Short of evidence that the blog spam was malicious and unaffiliated with the site, there should be no rehabilitation. If the site has valuable content the owner can move it to a new domain and try again. This time without being a scumbag.
I think better software would help but I also don't think Discourse is that software. I'm sure they understand spam but they don't seem to understand how humans communicate.
One way to combat this problem is to deny new users the ability to post links by replacing the link text with [removed]. HN has a karma system with thresholds that must be passed in order to get additional privileges (like down-voting). If you have a site that assigns karma to users (even if it's a secret number) then you can set a threshold for allowing links to pass through. It's not perfect but it's better than letting brand new accounts post links, IMO.
Another solution is to hide all links when a page is viewed by a user who isn't logged in.
I saw the SEO post he's referring to (it was one of the articles in SEOMoz's top 10 monthly email). Site owners don't realize the amount of work and headache they cause forum communities when they contract out SEO work without an understanding of what that work entails. Or they do and just don't care.
Google's Penguin update didn't deter the spammers, either. Here we are nearly a year later and I'm still cleaning out accounts created en masse by XRumer or other bots.
I can't put words in the mouth of the SEO quoted, but (unless you know he was one of the people making removal requests against your site) I suspect he is not talking about you or sites like yours.
I should add that I don't agree with the rhetoric btw, but I think he is targeting a different kind of webmaster.
I think he is referring to webmasters who sold links (knowingly outside the guidelines) for years. They would previously have instantly removed the links if someone stopped paying.
As soon as Google stepped up their game and removing those links was important, those same webmasters started charging to take down the same links.
I would personally point to the irony of this (google creating a market that enriches people who have been abusing their system for years) rather than calling it immoral. Ymmv.
Hope that helps clarify some things and I hope I'm not distorting the guy's real meaning.
I'm wondering about the graph on Gary's site: what type of business' fortune is linked this tightly to Google's rankings? Is it a particular category, or are there several categories of businesses that suffer from (or leverage) this? That's a pretty frightening situation to be in, IMO.
So this "SEO" who helping his clients to "clean up their profiles" doesn't even know how to disavow links [1]. Well, it's not surprising that somebody who hired spammers once would hire another idiot later.
Well Google requires you to contact the website owners and do everything you can to get the links removed BEFORE you submit a list of disavow links. Google are supposed to check that you really did remove a large part of the links before disavowing.
This is a great post that I'm sure rings true with many 'a webmaster. Time is money. There's no two ways about it. If you want a webmaster to modify their site in your favor, you should be willing to compensate them for their time in doing so.
What finally worked was a combo of: custom javascript on the signup form, disallowing new users from posting links in posts or signatures (and auto-banning ones who continually made attempts), insta-banning users who filled out non-linkedin links in the "linked-in link" profile field. We still get spam, but now it's maybe one post a week, instead of 10-20 a day, so it's at a level that moderators/user reports can handle it.
Rel=no follow discourages spam due to PR flow, but won't stop it. It still has an influence, however diminished.
Take an hour or two and write a script that selects all users with less than 100 posts and iterates through their posts removing links. The integrity of actual users is maintained while 99% of spam links eliminated. You spend a total of 2 hours, but don't have to deal with the emails any more - which could save many more hours.
Does this constitute a long-term solution to the problem for active communities? That is, default to rel='nofollow' for an untrusted class of users, which presumably includes all the spammers?
I'd be tempted to ask for money as well, but I feel like at the very least, you should ask for an apology. They may not have known that their SEO agency was using shady tactics, but since times changed, and it's obvious that they put bad links out there, it seems like an apology would be more than a token gesture.
The problem here is the work. The solution is to write a program. Have the white hat SEO people write a program that spiders the site with admin privileges and removes offending posts. It should come with a "dry run" mode that lets you spot check it. When they get it right, you can run it for them. It's a win-win: your forums get cleaned up and they did the work.
It's not really a win-win. Any time he spends on this is a loss from his point of view since the SEO's client already ruined his forum. It's some chutzpah to get annoyed when your victim refuses to help you hide your misdeeds.
If your business relies on SEO or is affected by Google search rank in a critical way, it may not be a viable business long-term.
Yes, that means I'm talking about a lot of online businesses.
The intelligent thing to do is sell a product or service that has value on its own and neither relies on SEO nor is it likely to be blames by other sites or companies for lowering their SEO.
I feel bad for these folks, but if you are planning on starting a business that doesn't really provide much value on its own that is identifyable outside of the roach motel of SEO, then you are headed into the ocean in a dingy with a small outboard motor, imo.
Link removal requests can also be malicious. A blackhat seo will check his competitors domain to see if there's a spf record, if they're signing their mail, and if there's a catch-all (simply by checking if random mail is accepted). If there's neither they get a list of backlinks, from public web-crawls or sites like ahref, and request these links be taken down by sending mail with spoofed email addresses.
This is sad, sad, sad. I wonder how much awesome communities died that way. I stopped visiting Orkut when communities there got overran with spam. Also the same apply to some Usenet groups and Google Groups I used to like.
The solution is obvious -- write a routine that automatically goes through the entire forum database and disables all the links -- leave the names, but rewrite the links so they're just text, not hyperlinks. Sort of like:
The above deletes the "http:// prefix, but leaves the original destination name, in case anyone wants to object that their post has been edited after the fact. So technically, it's no longer a link back to the originating site, but it's otherwise unchanged.
There must be a programmatic solution to this problem.
Do some outbound links have value on this forum? If not, then you could remove all links, or remove the "link" part of the link (change @href to text).
If some outbound links have value you need to identify those, and it's more complex, but a Bayesian analysis of posts should be able to score posts on their "spaminess" and remove the links on only the most spam-like comments.
There may be some false-positive doing this, but since no information is actually removed (only the links, not the content) it should be quite ok.
Random links on forums already have pretty low value. I would assume that forum spam has come down now that this is the case.
As the article states, websites are looking the clean up the spammy inbound links as google has threatened to deindex sites that don't make an effort to do this.
There's a difference between buying links, which Google actively started prohibiting at a certain point, and the automated forum spam you are mentioning. The latter is done using software like XRumer, and has always been penalized by Google. The former wasn't prohibited in the beginning, when link deals were often mutual agreements, most of the time involving a traffic component as well.
The snake oil salesmen that are mentioned, are the ones who actively participated in the scheme by selling links and making a buck, and are trying to make another quick buck now that the rules have changed.
Moreover, some links aren't even paid at all, but just look manipulative. For example, if you developed a wordpress theme, and your link is in the footer of tons of blogs, you might get penalized for manipulating the anchor text of your links in a non-natural way. In those legitimate situations, webmasters do have a moral obligation to cooperate.
I don't think anyone would think that of an honest entrepreneur being spammed to death by link spamming software.
On a side note, there are plenty of forums on the web that have survived the spam wave, if it were core to your business, you could have protected yourself.
Wow, so Google penalizes a site if a link to that site shows up in spammy pages? That seems like a new business model for black-hat SEOers: "hey, nice site you have there - it'd be a shame if links to it started appearing all over my spammy network - $$$ will make sure that doesn't happen." Really search engines should just give zero weight rather than negative weight to links from spammy sites.
Here is a question for those who know: is there anything forum regulars can do when we see spam posts beyond letting mods know about it?
I was active in a now dead forum that would get hit once or twice a week. The mods would clean it up with in a half a day but until then those posts would just sit. Does the Google web crawler consider words in replies to the spam post? I used to reply occasionally with words like: scam, fraud, got ripped off. I have always wondered if I was wasting my time. A few times I checked the link to see if it went some where legitimate (Google the base url), if so, I then searched the site for an informational web form. If there was one, left a message that their SEO company was using sleazy methods with a link to the forum post.
Google ought to make a code phrase that forum users can use to red flag spam posts. Though some of the posts were for Japanese and Chinese sites. The spam might not have been meant for Google but other search engines...
[+] [-] venus|13 years ago|reply
> There will certainly be webmasters out there who will strip you down to the bone asking for money in exchange of link removals. These are the most soulless snake oil salesmen on earth
To say that about webmasters, already victims of years of abusive SEO spamming, when they then refuse to help an abusive site clean up its own mess for free .. I have no words. Could anyone possibly be more of a self-interested, myopic, egocentric prick?
My rock-bottom opinion of pretty much anyone who has anything to do with active SEO is re-confirmed for the thousandth time.
[+] [-] natrius|13 years ago|reply
[+] [-] SquareWheel|13 years ago|reply
[+] [-] megablast|13 years ago|reply
Just as you have lots of machines that have been compromised and are part of a botnet.
[+] [-] dchuk|13 years ago|reply
[+] [-] halcyondaze|13 years ago|reply
[+] [-] taf2|13 years ago|reply
[+] [-] kkowalczyk|13 years ago|reply
There is a solution to his particular problem: better forum software.
I don't want to trivialize the problem of writing spam-resistant forum software but it's not such an insurmontable problem (this forum being a proof positive for that).
For the reference: I've been running a fairly popular forum (http://forums.fofou.org/sumatrapdf) for several years, using forum software that I wrote.
I don't even require the user to log in, I don't require moderation for posts and yet I had zero automatic spam (I attribute this to my unusual captcha http://blog.kowalczyk.info/article/16fw/Best-captcha-is-exot...).
I occasionally get human spam i.e. someone writes a post with the only purpose of linking to some other website. I just hellban them after I see the post in my rss reader.
And I didn't even write any special anti-spam code (other than hell-banning), because I'm lazy. I can easily come up with simple ideas e.g. putting all posts that contain links in moderation queue.
To reiterate: his problems were caused by a crappy forum software that didn't do much to protect from seo spam.
I don't really know how current best off-the-shelf forums fare in this respect.
I would rather not spend my time maintaining my own forum software so I have high hopes for http://www.discourse.org. I'm sure StackOverflow had plenty of spam problems so discourse people should understand the problem.
[+] [-] Fizzer|13 years ago|reply
If your site was in the top 1,000, you'd have a different experience since the spammers would be adapting to every one of your exotic captchas no matter how many times you change it.
[+] [-] greggman|13 years ago|reply
I ran a forum using SimpleMachines. Turned on all the spam fighting stuff. Didn't matter. Bots or Mechanical Turks made 10-30 new accounts a day. I tried moderating the accounts but who wants to spend time every day to delete the [email protected] account.
I gave up and shut it down.
[+] [-] bradleyjg|13 years ago|reply
Short of evidence that the blog spam was malicious and unaffiliated with the site, there should be no rehabilitation. If the site has valuable content the owner can move it to a new domain and try again. This time without being a scumbag.
[+] [-] krichman|13 years ago|reply
[+] [-] coderdude|13 years ago|reply
Another solution is to hide all links when a page is viewed by a user who isn't logged in.
[+] [-] a5seo|13 years ago|reply
point external links through a redirect like /out?to=<url> and put /out in you robots.txt
change your url structure and simply 404 the pages with spammy links
[+] [-] markdown|13 years ago|reply
No, this is almost as bad as what medium.com does with all its links (rewriting them to point inwards towards redirects for tracking purposes).
A web link should be respected.
[+] [-] mutagen|13 years ago|reply
Google's Penguin update didn't deter the spammers, either. Here we are nearly a year later and I'm still cleaning out accounts created en masse by XRumer or other bots.
[+] [-] will_critchlow|13 years ago|reply
I should add that I don't agree with the rhetoric btw, but I think he is targeting a different kind of webmaster.
I think he is referring to webmasters who sold links (knowingly outside the guidelines) for years. They would previously have instantly removed the links if someone stopped paying.
As soon as Google stepped up their game and removing those links was important, those same webmasters started charging to take down the same links.
I would personally point to the irony of this (google creating a market that enriches people who have been abusing their system for years) rather than calling it immoral. Ymmv.
Hope that helps clarify some things and I hope I'm not distorting the guy's real meaning.
(written on my phone. Please excuse typos).
[+] [-] dhimes|13 years ago|reply
[+] [-] 8ig8|13 years ago|reply
Don't charge for link removal; charge for priority link removal. Same day service: $1,000 per link. One week: $500. Etc.
Free link removal: First-come, first-serve at your own leisurely pace.
Sell the old forum to someone else and let them handle the requests.
BTW, PocoMail was a godsend back in the day. Thanks.
[1]:http://news.ycombinator.com/item?id=4994246
[+] [-] slaven|13 years ago|reply
[+] [-] nkuttler|13 years ago|reply
[1] http://www.youtube.com/watch?v=393nmCYFRtA and https://www.google.com/webmasters/tools/disavow-links-main (login required)
[+] [-] JimWestergren|13 years ago|reply
[+] [-] phasevar|13 years ago|reply
[+] [-] jedireza|13 years ago|reply
Then with an auto-reply (or 'link policy' page) inform 3rd party sites that the link-juice (good or bad) is no longer flowing.
[+] [-] kalleboo|13 years ago|reply
What finally worked was a combo of: custom javascript on the signup form, disallowing new users from posting links in posts or signatures (and auto-banning ones who continually made attempts), insta-banning users who filled out non-linkedin links in the "linked-in link" profile field. We still get spam, but now it's maybe one post a week, instead of 10-20 a day, so it's at a level that moderators/user reports can handle it.
[+] [-] tronalddump|13 years ago|reply
Take an hour or two and write a script that selects all users with less than 100 posts and iterates through their posts removing links. The integrity of actual users is maintained while 99% of spam links eliminated. You spend a total of 2 hours, but don't have to deal with the emails any more - which could save many more hours.
[+] [-] Scramblejams|13 years ago|reply
[+] [-] codezero|13 years ago|reply
[+] [-] unknown|13 years ago|reply
[deleted]
[+] [-] javajosh|13 years ago|reply
[+] [-] bradleyjg|13 years ago|reply
[+] [-] hakaaaaak|13 years ago|reply
Yes, that means I'm talking about a lot of online businesses.
The intelligent thing to do is sell a product or service that has value on its own and neither relies on SEO nor is it likely to be blames by other sites or companies for lowering their SEO.
I feel bad for these folks, but if you are planning on starting a business that doesn't really provide much value on its own that is identifyable outside of the roach motel of SEO, then you are headed into the ocean in a dingy with a small outboard motor, imo.
[+] [-] arn|13 years ago|reply
[+] [-] Father|13 years ago|reply
[+] [-] speeder|13 years ago|reply
[+] [-] lutusp|13 years ago|reply
s!http://!!g
The above deletes the "http:// prefix, but leaves the original destination name, in case anyone wants to object that their post has been edited after the fact. So technically, it's no longer a link back to the originating site, but it's otherwise unchanged.
No human intervention required. Problem solved.
[+] [-] bambax|13 years ago|reply
Do some outbound links have value on this forum? If not, then you could remove all links, or remove the "link" part of the link (change @href to text).
If some outbound links have value you need to identify those, and it's more complex, but a Bayesian analysis of posts should be able to score posts on their "spaminess" and remove the links on only the most spam-like comments.
There may be some false-positive doing this, but since no information is actually removed (only the links, not the content) it should be quite ok.
[+] [-] robryan|13 years ago|reply
As the article states, websites are looking the clean up the spammy inbound links as google has threatened to deindex sites that don't make an effort to do this.
[+] [-] rizz0|13 years ago|reply
The snake oil salesmen that are mentioned, are the ones who actively participated in the scheme by selling links and making a buck, and are trying to make another quick buck now that the rules have changed.
Moreover, some links aren't even paid at all, but just look manipulative. For example, if you developed a wordpress theme, and your link is in the footer of tons of blogs, you might get penalized for manipulating the anchor text of your links in a non-natural way. In those legitimate situations, webmasters do have a moral obligation to cooperate.
I don't think anyone would think that of an honest entrepreneur being spammed to death by link spamming software.
On a side note, there are plenty of forums on the web that have survived the spam wave, if it were core to your business, you could have protected yourself.
[+] [-] ajenner|13 years ago|reply
[+] [-] Isofarro|13 years ago|reply
Good way to clue Google in to this network, and then drop it from the index.
"Really search engines should just give zero weight rather than negative weight to links from spammy sites"
That's what was happening before Penguin/Panda - and it wasn't working. There's no risk to link spamming. Now there is.
[+] [-] BashiBazouk|13 years ago|reply
I was active in a now dead forum that would get hit once or twice a week. The mods would clean it up with in a half a day but until then those posts would just sit. Does the Google web crawler consider words in replies to the spam post? I used to reply occasionally with words like: scam, fraud, got ripped off. I have always wondered if I was wasting my time. A few times I checked the link to see if it went some where legitimate (Google the base url), if so, I then searched the site for an informational web form. If there was one, left a message that their SEO company was using sleazy methods with a link to the forum post.
Google ought to make a code phrase that forum users can use to red flag spam posts. Though some of the posts were for Japanese and Chinese sites. The spam might not have been meant for Google but other search engines...