Google Cans Comment Spam

fthiess writes "Comment spam is in many ways even more annoying than regular email spam, since you generally have to do more than just hit the delete button to get rid of it. Its defining characteristic is that spammers abuse websites where the public can add content (blogs, wikis, forums, and even top referrer lists) to increase their own ranking in search engines. It seems, however, that the days of content spam are numbered: today Google announced that, in partnership with MSN Search and Yahoo!, that they have implemented a way to block content spam." (More below.)

"Briefly, you just change your blogging/wiki/forum/etc. software so that any hyperlinks in publicly-contributed text have a new rel=nofollow attribute added to any anchor tags. Google, MSN, and Yahoo! will now no longer index any such links, so the motive for content spamming disappears. Especially hopeful is the fact that a slew of makers of blogging software, including Six Apart, have announced they are supporting the new attribute."

Cooperation is a good thing

It's nice to see Google, MSN, and Yahoo cooperating on this effort.

and of course...

Don't forget to put that attribute in your track-back links either :)

Simon.

It's one way...

It certainly will help filtering some of the spam sites out of Google rank and so on, but the links will still be there in blog comments, bulletin boards, etc. The Googlebot will not follow the links, but human readers won't see the NOFOLLOW tag - and they'll click. It means that moderators still have manual work to do.

Re:It's one way...

It is not a solution meant to change the content on a website (that would be tantamount to censorship). It only changes how the search engines handle the links (note: the supporters/developers of such a standard are search engine companies).

The best question raised in this post [slashdot.org] is if such a tag is standards acceptable.

Re:It's one way...

Censorship? Are you saying that's a bad thing.

How's that old saying go? Your right to free speech ends at my rights to kick you in the nuts when you spam me.

Something like that, anyway...

Re:It's one way...

The comment spam is mostly used to get a better searchengine ranking. A blog which uses this attibute on link tags is far less interesting to comment spammers, so chances are the moderaters have to delete less spam.

Re:It's one way...

Eventually, this might reduce the amount of comment spam.

But somehow I don't think spammers really care if a blog uses this system or not. It's probably easier to just spam all blogs than to figure out which are useless. Just like email spammers don't care much if an address is valid or not.

Some people think that adding spam filters to an email account reduces the spam sent, while it only reduces the amount of spam received. This solution does neither.

However, all efforts to fight spam should be welcomed and supported. Despite my pessimism, it will be interesting to see how it turns out.

Use CSS

It is pretty easy to make rel="nofollow" visible to normal users too in modern web browsers using CSS. You could use something like this:

a[rel="nofollow"]:before {
content: url(an-image-representing-nofollow-links.png);
}

That will display the given image before any links marked as nofollow.

Is the result valid HTML/XHTML?

Does HTML/XHTML allow "rel" attributes on links? And if so, is "nofollow" an allowed value for that tag?

Re:Is the result valid HTML/XHTML?

Yes and yes.

From the W3C:

Links in HTML documents - The A element [w3.org]:

rel = link-types [CI]
This attribute describes the relationship from the current document to the anchor specified by the href attribute. The value of this attribute is a space-separated list of link types.

Basic HTML data types - Link types [w3.org]:

Authors may use the following recognized link types, listed here with their conventional interpretations. In the DTD, %LinkTypes refers to a space-separated list of link types. White space characters are not permitted within link types.

These link types are case-insensitive, i.e., "Alternate" has the same meaning as "alternate".

User agents, search engines, etc. may interpret these link types in a variety of ways. For example, user agents may provide access to linked documents through a navigation bar.

...

Authors may wish to define additional link types not described in this specification. If they do so, they should use a profile to cite the conventions used to define the link types. Please see the profile attribute of the HEAD element for more details.

Re:Is the result valid HTML/XHTML?

Authors may wish to define additional link types not described in this specification. If they do so, they should use a profile to cite the conventions used to define the link types. Please see the profile [w3.org] attribute of the HEAD element for more details.

I think this last paragraph is important. "nofollow" is not on the official list of link-types [w3.org]. If blog authors wish to use this attribute in anchor elements, they need to define it properly (or at least properly reference a definition).

Remember back in the 90's when Netscape and MS were breaking standards right and left so that their browsers would have an edge on the competition? That was the wrong way to do it, and it created the mess we're in now with sloppy HTML spewed all over the web and designers unable to use compliant designs because the most popular browser doesn't even try to support standards (an example here [meyerweb.com]). Google is doing this the right way. They went back and read the HTML specification to see if it was already capable of doing what they needed. It does? Great! Let's utilize the standard!

Granted, HTML these days has a much better design than it did in the pre-4.0 specifications. Back when Netscape and MS were at each other's throats the document format was actually incapable of doing a lot of things that designers wanted to do on the web. But HTML is a very mature format these days.

Now if only...

Slashdot could implement something like this, it would make article comments meaningful again.

Re:Now if only...

Slashdot could implement something like this, it would make article comments meaningful again.

They could even selectively add or omit it based on the comment's moderation. Include the nofollow tag by default, but if a comment with a link in it is moderated highly, remove the tag so search engines can use it. Sounds like the best of both worlds..

Re:Could work the other way too ;)

RTFA. Slashdot could modify slashcode to automatically add the attribute to all links posted in comments. Comment spammers can't do anything about it, so they'll move away to other sites.

No normal links (i.e. not in visitor contributed content) should have the attribute. So slashdot will still be full of normal links; only the links in the comments will have the attribute.

Miserable Failure?

But what will happen then with the miserable failure [google.com] and weapons of mass destruction [blueyonder.co.uk]? Can't anyone efficiently bomb google anymore?

Band aid

I'm not really into blogging so I don't know how big of a problem this is. I get some spam in my guestbook, which I promptly remove. The spam iteself is what's really irritaing, not the potential "elevating" of the spamvertised site in search-engines, where I've never personally run across one that I can remember.

Am I correct in assuming that these sites pops up and down relatively often? Maybe it'd be possible to use temporal component to the rating. Say if the link points to a site which was just registered two days ago, it's given a very very low weight, and then you ramp up as time goes by. As spam gets deleted from blogs and guestbooks, time would work against these spammers. Or? I dunno.

New denial of service attack...

Hmmm...if a malicious program adds the tag to links served by a compromised html server, you could have an interesting and different sort of denial of service attack, although it would be slow to take effect.

It's a pagerank question, not indexing

Google, MSN, and Yahoo! will now no longer index any such links
Not quite. What happens is, that the link wont add anything to the site in question. As you probably all know, most search engines rank pages by incoming links - it's not just google. By adding this tag, the incoming link wont count.
I think this is a great idea. It will probably break the w3c compliance, but hey - anything to piss off a spammer.

Useful links

Forums and Blogs often contain very useful links. What about them? What about all those sites that are *only* linked to from blogs and forums, and that actually are great and useful sites?

Re:Useful links

Links in the main body of the blog post will be fine. Blogs of course, have high page rank because bloggers comment on each other's blogs. This tag may have a side effect of generally reducing the page rank of blogs.

As for useful links in comments; if they're really good sites, people are bound to blog about them more generally. And my poor blog gets few enough hits that it will be no problem for me to manually edit genuine comments to remove nofollow tags.

Opportunity for Firefox (plugin)

Why not modify Firefox (or provide a plugin) that allows such links to be grayed out or otherwise marked specially?

Actually, are there any plugins already in existence that modify the appearance of a link based on a regexp match?

Re:Opportunity for Firefox (plugin)

Why not modify Firefox (or provide a plugin) that allows such links to be grayed out or otherwise marked specially?

Actually, are there any plugins already in existence that modify the appearance of a link based on a regexp match?

Let me introduce you to the wondeful world of userContent.css [mozilla.org].

Something like this should work:

a[rel="nofollow"] {
text-decoration: line-through ! important;
border-bottom: dotted thin gray ! important;
color: gray ! important
}

Re:Opportunity for Firefox (plugin)

Actually, you probably want to use

a[rel~="nofollow"]

instead of just an equals. There are other defined values for the rel attribute and you don't want to have your CSS miss this just because it contains more than just the "nofollow" tag.

But what if I need to search

for ch3aP Can.adi n v31g.r a?

Wikipedia

Wikipedia already implemented this feature. See here [wikipedia.org].

Just Remove The Sites

This is not a solution as far as I'm concerned.

Why stop the indexing of relative links from blogs to make google's life easier?
99% of the links posted in comments are relavent and would be beneficial to index. Why stop this for the 1% of jackasses out there?

The domains contained in the links from blogspam are well known, and there are plenty of blacklists out there. Why doesn't googleyahoomsn just remove these sites from its database? Its such an easy solution. I believe they already do this in some circumstances for link trading systems whose only goal is to get higher pagerank.

This solves only 1/2 of the problem

While this will prevent spammers from bumping up their sites' Page Rank (probably their primary motivation for comment spam anyway), it doesn't prevent their bots from spamming targeted blogs etc. in the first place. That is still best handled by the blog software providers.

For example, WordPress has a variety of different plugins [wordpress.org] for handling comment spam. The best one I've seen renders a series of characters graphically (a la TicketBastard) which the user (a human, of course) has to type into a text field on the comment form before their comment is accepted. Blogs implementing this type of mechanism typically have spam coming from bots drop down to zero.

Re:This solves only 1/2 of the problem

The best one I've seen renders a series of characters graphically (a la TicketBastard) which the user (a human, of course) has to type into a text field on the comment form before their comment is accepted.

Sure, that's great for humans using a graphical browser, with images turned on, and 20/20 vision. But that doesn't cover all internet users. What about text browsers? What about screen readers?

This is the age of internet accessibility folks, and it's exactly why I refuse to use Captcha [captcha.net] tests on my own blog [h4xx0r.co.uk] - instead, I currently filter all comments and trackbacks through wp-spamassassin [ioerror.us]. Haven't had a single problem yet, although it's early days.

The rel="nofollow" trick sounds promising for killing off the PageRank cheats, but it won't stop humans clicking the links...

This is open to severe abuse

There are a lot of people out there who understand the PageRank system, and complain that if they add outgoing links on their site then their previous PageRank will be "leaked" to other sites, rather than their own internal pages.

Well, luckily Google has now released a way for people to link to each other without leaking PageRank. Yes, the nofollow relation. So, now everyone can link to each other, and no-one gets any benefit out of it whatsoever.

This tag is not a bad idea, but I think the good things it could stamp out weren't considered anywhere near as much as the few bad things it can stamp out..

Much better solution - spamassassin

Why doesn't Google use spamassassin or another spam filtering tool? One based on Bayesian analysis - to determine if a page is just spam, and give it a lower score.
This can be done whether it is linked in a blog or not, and will improve the overall quality of the search database.

- Erwin

!!! MAKE $$$ FA$T !!!!

Hey! This is the first time that I can comment spam and have it not modded off topic!

That Won't Help for the Biggest Thorns in My Side

Whenever I'm searching for technical information, a couple of sites always come up that are useless to me. They have a question/answer format, questions are left in the clear for search engines, while the answers require registration. What I need is a way to filter those sites out from my searches, so that they simply don't show up in any result set. Hmm might be a good excuse to play with writing Firefox plug-ins... :-)

Re:A gift to Microsoft

Well, since MSN Search seems to apply the same policy as Google it would do them no good either.

Re:A gift to Microsoft

You are assuming that Microsoft talks to Microsoft. With so many divisions and levels of middle management. It is possible for something like this to happen. Because they know they are competing with google but they never got the memo that the MSN Search is blocking these sites.

Re:A gift to Microsoft

For the largest company in the world having the flattest organizational structure can still be big. I am sure most employees don't report to Balmer or Gates. I don't know how flat it is. But say for There is MSN Search team belongs to the MSN team who reports to Gates. Then there is the Front Page team that works for the Office Team that belongs to the software development team which reports to Gates. For a company that size the structure is very flat. But still there are middle management involved and Information may not spread across.

Re:A gift to Microsoft

Oh the horrors. And what if I created a site with 5 million links to my competitor's website, then stuck it behind a firewall on my corporate intranet so google couldn't search it at all?! Just think of how much damage I'd be doing to them, with all of those unindexed links!

Re:A gift to Microsoft

Are people against Microsoft beacause of their business practices, their product, or just because they are a large company who did extremely well

All of the above to a certian extent.

Sysadmin Geeks who have to clean up the messes left by shoddy Microsoft products, day after day, hate their products because they make extra work for us. We hate Outlook, IE, and IIS because their penchant for spreading worms and viruses. We hate service packs which break more than they fix. We hate Frontpage because of the non-standard, blecherous, broken HTML it spews forth. We hate the general lackadasical attitude Microsoft has about security and quality in general.

Libertarian-minded geeks hate Microsoft for their flagrant disregard for the law and the courts. We hate them for the way they blatantly infringe on other company's patents and lawyer their way out of it. We hate the way they bankrupt or buy out anyone making a product which actually competes with them. We hate the way they use puppet companies (SCO, BSA) as hired thugs to bully other companies on their behalf.

Anti-corporate geeks hate Microsoft because it's a prime example of corporate greed run amok and of the dangers of unfettered capitalism.