Microsoft Tracks Down Mass Fake Web Pages

An anonymous reader writes "According to an article on New York Times, Microsoft researchers have discovered tens of thousands of junk Web pages, created only to lure search-engine users to advertisements. While most of us have run across them from time to time, the company researchers have found the pages are deliberately generated in vast numbers by a small group of shadowy operators. By following the money trail, Microsoft researchers were able to track the flow from big-name advertisers to search engine spammers. Many use Google's blogspot.com to set up spam doorway pages. 'The practice has proved to be a vexing problem for the major search companies, which struggle to prevent both spammers and companies specializing in improving legitimate clients' Web traffic -- a field known as search-engine optimization -- from undermining their page-ranking systems. Surprisingly, the researchers noted that the vast bulk of the junk listings was created from just two Web hosting companies and that as many as 68 percent of the advertisements sampled were placed by just three advertising syndicators.' The report is available at Microsoft Strider Search Ranger project page."

The easy way

They could have saved a lot of time and money by just visiting forums like DigitalPoint. These doorways and other spammy sites are for sale every day. It's no secret.

Re:The easy way

Well it's about time someone tracked down these spammers. I can't count how many times I was searching for porn on the internet and got an ad page. The nerve of some companies.

Re:The easiest way

Quick, somebody make a few thousand clones of this report.

another ripoff

Man. This Microsoft project is just a ripoff of Google's Gandalf Search Wizard project...

Re:another ripoff

The report is available at Microsoft Strider Search Ranger project page.

Man. This Microsoft project is just a ripoff of Google's Gandalf Search Wizard project...

Yeah, but let's not forget that even before that was AOL's Smeagol Browser Gollum project ...

Re:

The report is available at Microsoft Strider Search Ranger project page.

Man. This Microsoft project is just a ripoff of Google's Gandalf Search Wizard project...

Yeah, but let's not forget that even before that was AOL's Smeagol Browser Gollum project

Whe

Re:

Register.com among the Businesses, Melbourne IT to the Australians; Tucows I was in my youth that is forgotten, in the South ENom, in the North GoDaddy, to the East I go not...

Why?

Is it really cheaper to use Page Ranking companies instead of just well, PAYING for an advertisement on Google or MSN or something?

Re:Why?

Is it really cheaper to use Page Ranking companies instead of just well, PAYING for an advertisement on Google or MSN or something?

Yes, or they wouldn't do it.

Re:Why?

The average return on investment on Search Engine Optimisation (generally: increasing your search position on specific keywords relevant to your business) can be about 10x more than the return on keyword purchasing, which can cost 0.30c - several dollars. Every click costs money.

Once you've optimised to your keywords in "natural search" e.g. *free* results, then your investment keeps paying (you need to maintain positions of course, but this is lower cost, especially if you're in a niche) whereas in paid advertising you have to keep giving money to Google and, in competitive industries, your cost per click will be subject to significant inflation...

Re:

Sometimes businesses do stuff that doesn't work out -- they go bankrupt everyday.

Re:

it may not be cheaper, but it may be more effective. search engines generally identify results that were purchased, and i'm sure a user is less like to click on it if they see that. the clients of these companies are buying their way into the results witho

Re:Why?

It is also more effective. How many times do you click on ads? Now how many times do you click on search results? 'nough said...

"time to time"?

While most of us have run across them from time to time...

Time to time? For mee it seems like more than 50% when I scan the search results. Maybe less, maybe more, but certainly more than "time to time". For many of my searches, I may not find anything truly relevant until the second and third page. People have learned how to play Google to the point where more and more Windows Live is starting to give better results (scary!).

Re:

Maybe the best thing to do is to automatically skip to the 2nd page of results and write off the first page as search engine spam.

Re:"time to time"?

I agree. I run a small business out of Nigeria that helps people in unfortunate situations recover lost money, and we rely on upfront investments from Americans. We always promise a good cut of the money to our American investors. This search engine spam really puts the hurt on my business, too.

Re:

I have never seen results that bad. You must be searching for porn, where spam is to be expected.

I beg your pardon... "Erotica" is a perfictly legitimate subject.

Ironically

they harvested most of their results from Google.

Nice work

There's actually some pretty decent research here. The site cloning report is a good read.

http://research.microsoft.com/SearchRanger/Spam_At tack_by_Website_Clones.htm [microsoft.com]

The cloning of popular blogs as been a scourge for a while now, both for manipulating search engines and good old fashioned advertising - using someone else's content to draw visitors in

Nice work

Thanks for a informative post. Beats the typical whiny M$ iz S4T4|\| crap.

Google does keep up, but quietly- anecdotally, last week I was searching for a certain spec ARM9 dev board (the VULCAN-Lite) with USD also as a search term and all kinds of fake keyw

Re:

You are 100% correct that Google does help clean up it's searches. I do about 100 web searches a day to learn stuff, every time I come across spammy results I send Google a note. I think it's working, because the next week when I want to learn more on a to

Then Microsoft realized...

It's coming from inside the building!!!

How does this help them?

PageRank is designed to be resistant to exactly this sort of attack. The amount of Google karma you get is proportional to the karma of the pages that link to you. Creating lots of pages with no karma that link to you therefore shouldn't do you any good

Re:How does this help them?

It works because you don't realize the size of this thing. They're talking about millions of fake pages here, lots of them pointing at other fake pages to raise their pagerank so they can in turn point at yet more pages. You would think Google would have someone seeking these kind of sites out and applying a discount on their domain though (although when that happens the spammers just move on anyway).

Re:

Er, that sounds like the old saw "we lose a penny on each one sold, but we make it up in volume".

If there's only so much karma going into your pages, there's only so much karma they have to give, no matter how huge it is. A trillion pages pointing at my p

Re:

Presumably some of these trillion pages have a karma greater than or equal to epsilon.

The scummiest part of it all is that some of the pages in question will be on domains that someone let expire and someone else immediately snatched up. They get their Pag

Re:

Indeed, and this has happened to me.

Re:

Happened to Andrew Koenig (or maybe his publisher Addison-Wesley) -- The website for Accelerated C++ (http://www.acceleratecpp.com ) was either hijacked or expired, and snapped up by some lowlife for about 2 months before it got fixed.

Re:

DAMN! That's http://www.accelerated cpp.com [acceleratedcpp.com].

Re:

Every page has to start with some small, intrinsic amount of karma, otherwise there would be none to pass around. By creating enough bogus pages, you can aggregate some amount of link karma to bestow on the site of your choosing. In principle, I guess t

Re:

Every page has to start with some small, intrinsic amount of karma, otherwise there would be none to pass around.

There has to be a "root set", but that root set doesn't have to consist of all pages. There's some evidence that it includes all top-level pag

Re:

Creating lots of pages with no karma that link to you therefore shouldn't do you any good at all

That's not how it works. You assume it's a zero-sum game, but it's not. Every page gets some weight even if no one links to it. It's small, but it's positive. W

Re:

Every page gets some weight even if no one links to it. It's small, but it's positive.

That's not the impression I'm under - I thought that most pages were not part of Google's "root set". See my reply here:

http://slashdot.org/comments.pl?sid=227331&th [slashdot.org]

Re:

That's not the impression I'm under - I thought that most pages were not part of Google's "root set"

I understand that you have such an impression, but that's a wrong impression. Every page gets a non-zero weight by default. If you think about it you will s

Re:

Since the answer is a closely guarded secret within Google, it's always fun to be contradicted by someone speaking in authoritative tone of voice who knows as little about this as I do :-)

You're mistaken about your argument against, in any case; PageRank i

Re:

who knows as little about this as I do

How do you know that?

I can tell you that it does not have the property you assign to it

The delay I mentioned is due to links being made, not links being discovered. Think about some small community of scientists making

Re:

Oooh, hints of dark and secret knowledge! Those are always very impressive.

The delay I mentioned is due to links being made, not links being discovered. Think about some small community of scientists making an almost closed cluster of sites about their ni

Re:

our site is actually working with one of these companies (on the receiving end of the paycheck, though). they want to put "ads" on our site that link to other sites. they dont care at all what the ads look like or where they are on the page, but just that

And?

Ok. Forgive me if MS just discovering this makes me think they just entered 2002. That crap is _not_ new folks.

On the other hand, what idiot spouts off about two hosting companies being responsible without naming them? Seriously. This isn't Fark, you can't get kicked off for calling some asshole out.

Re:And?

... but you can get sued for libel if you're wrong.

Re:

but the best bit: Phillip Rosenthal, chief technology officer of one of the companies, ISPrime, an Internet services company based in New York, said the activity had been traced to a single customer and violated the company's acceptable-use policy. He said

And in other news...

Microsoft researchers have discovered tens of thousands of junk Web pages, created only to lure search-engine users to advertisements.

In other news, Microsoft researchers have discovered that the sky is blue and that water is wet.

Re:

Microsoft researchers have discovered that the sky is blue

I live in London, you insensetive clod!

Obligatory Bill Hicks

Obligatory Bill Hicks...

If you work in advertising, kill yourself.
--Bill Hicks - Another Dead Hero

Bad neighborhoods

Google is already developing methods to deal with clusters of these fakes. Usually they're scraping web directories and databases. I've seen a lot of this lately, searching for dental hygiene schools for my girlfriend. Usually they're linking to each other, even if they're huge clusters. Legit SEO guys (yes, there are consultants who actually try to get your site linked legitimately and by hand) call these areas "bad neighborhoods". Whatever Google's doing, though, clearly isn't enough, and a lot of these guys are using adsense to make money. Martinibuster's [martinibuster.net] got a few good links on the subject.

A few years ago...

...a friend of mine figured he could get great Google listings by autogenerating trashy link farm pages, he had the top 1000 porn search terms all cunningly mispelled, ie "Brittney Spares" and hundreds of thousands of static pages all linking into each other across a bunch of subdomains. For about a year we reckoned he had some stupid percentage of all porn listings in Google, and in that time he made around $1,000,000 from banner clicks. Eventually Google caught onto it and blocked his sites enmass, but he'd made enough to buy some property by then.

Microsoftie wearing a white hat?

I just finished reading how much the Strider group at M$ has accomplished and how, and it is rather amazing. They lifted the covers off of typo-domain squatters exploiting Google's programs, a progressive honeypot setup that detects which levels of XP are attackable by different mal-ware attacks (up to and including reporting zero-day exploits if the latest "patch hardened" machine is exploited], and now this project. Even better, they are publishing the "how", and any OS (AKA Mac OS or any of the Linux distros) could benefit by using similar approaches on even more machines.

So -- from an admitted open source advocate -- here's a rare kudo to the giant in Redmond for keeping a "white hat" and his group -- and letting them work.

Re:

I agree. Whatever else you say about MS, and there's lots to say, they seem to have given their security researchers a lot of freedom and because of their size and power have the resources and brainpower to tackle these problems in pretty cool ways. The

is this research reliable

I read the research paper a couple days ago after reading about it in the NY Times. Seeing how this research is Microsoft funded and implicates Google, claiming they're syndicators are in cooperation with the spammers, one has to question researcher bias.

Firefox is good.

Firefox has an extension called customizegoogle [customizegoogle.com]. It adds a 'filter' option to a google results page. Allows one to filter out the sneaky pages that hi-jack your search query.