The Man Behind Google's Ranking Algorithm

nbauman writes "New York Times interview with Amit Singhal, who is in charge of Google's ranking algorithm. They use 200 "signals" and "classifiers," of which PageRank is only one. "Freshness" defines how many recently changed pages appear in a result. They assumed old pages were better, but when they first introduced Google Finance, the algorithm couldn't find it because it was too new. Some topics are "hot". "When there is a blackout in New York, the first articles appear in 15 minutes; we get queries in two seconds," said Singhal. Classifiers infer information about the type of search, whether it is a product to buy, a place, company or person. One classifier identifies people who aren't famous. Another identifies brand names. A final check encourages "diversity" in the results, for example, a manufacturer's page, a blog review, and a comparison shopping site."

Hrm, and all this time I though it was...

Pigeon Rank?

Re:Hrm, and all this time I though it was...

parent is not offtopic - http://www.google.com/technology/pigeonrank.html [google.com]

apple vs Apple

The formulas can tell that people who type "apples" are likely to be thinking about fruit, while those who type "Apple" are mulling computers or iPods.

Well the results for both "apple" and "Apple" are identical for me (apple computer dominated), with the exception of the text in the ads on the right hand side (which are both for apple computers). Maybe they are doing other stuff (Linux users prefer computers over fruit?).

Does anyone see anything different when they search for "apple" versus "Apple"?

Re:apple vs Apple

No, but I DO see the difference between 'appleS' and 'apple', just as the text you're quoting mentions.

Amit Singhal ...

... is not to be confused with Amit Singh [kernelthread.com], who also works at Google and has authored an excellent book on Mac OS X Mac OS X Internals [osxbook.com].

...only one?

> They use 200 "signals" and "classifiers," of which PageRank is only one.

How many did they expect PageRank to be? In the words of someone immortal, "There can be only one.".

North America Centric

I wish I could give google.ca a signal to return pages from North America.

I'll search for a product and the first page of results will all be *.co.uk results.

Not much use to that. Makes me think on how to rephrase the search, which is good.

Feature Request

My ongoing gripe with Google is the number of times when the first page is filled with shopping sites, "review" pages, and click through pages that exist only to grab you onto the way to where you really want to go.

I would love a switch, or even a subscription, that would allow me to filter these usually useless types of pages and instead show me pages with real content.

Re:Feature Request

This probably does what you want.

http://www.givemebackmygoogle.com/ [givemebackmygoogle.com]

It just negates a whole lot of affliate sites.

This is part of the query it feeds to Google.

-inurl:(kelkoo|bizrate|pixmania|dealtime|pricerunn er|dooyoo|pricegrabber|pricewatch|resellerratings| ebay|shopbot|comparestoreprices|ciao|unbeatable|sh opping|epinions|nextag|buy|bestwebbuys)

Re:Feature Request

Firefox extension: http://www.customizegoogle.com/ [customizegoogle.com] lets you filter out URLs from the results (plus dozens of other useful things).

You can filter out Wikipedia mirrors (using that extension) with the list here: http://meta.wikimedia.org/wiki/Mirror_filter [wikimedia.org]

Many other things are goo(gle)d

Pagerank is the source of all wisdom in google... but there is so much more... Like string searching & matching algos, file searching.. you name it.. Just the other day I was searching for books about Google's algorithms... I found zero interesting stuff.. They keep their algorithms secret and out of the public domain... (like they should..). we praise Pagerank, but if we knew what other stuff is there, we would all be members of Church of Google (http://www.thechurchofgoogle.org/) :P

Re:Many other things are goo(gle)d

Could it not simply be that they're not keeping it under wraps to avoid sneaky webmasters manipulating their sites, but to prevent competitors gaining an edge?

How does it work

It is rather simple (I am an insider).

Google breaks pages in words. Then, for evey word it keeps a set which contains all the pages (by hash ID) that contain that word. A set is a data structure with O(1) lookup.

When you search for "linux+kernel" google just does the set union operation on the two sets.

Now a "word" is not just a word. In google sees that many people use the combination linux+kernel, a new word is created, the linux+kernel word and it has a set of all the pages that contain it. So when you search for linux+kernel+ppp we find the union of the linux+kernel set and the "ppp" set.

So every time you search, you make it better for google to create new words. And this is part of the power of this search engine. A new search engine will need some time to gather that empirical data.

Of course, there are ranks of sets. For example, for the word "ppp" there are, say, two sets. The pages of high rank that contain the word ppp, and the pages of low rank. When you search for ppp+chap, first you get the set union of the high rank sets of the two words, etc.

Now page rank has several criteria. Here are some:
well ranked site/domain, linked by well ranked page, document contains relevant words, search term is in the title or url, page rank not lowered by google emploee (level 1), page rank increased, etc.

It is not very difficult actually.

(posting AC for a reason).

Now I understand

Search over the last few years has moved from Give me what I typed to Give me what I want, says Mr. Singhal

So this is why all my results are links to lesbian porn regardless of what I search for.

Googling Uncommon Characters and Exact Phrases

One of the most annoying things about google for me is how it interprets queries with strange characters common to almost all programming languages. A google search for "ruby <<" returns no results related to the ruby append operator. A Simple search for "<<", by itself returns ZERO results.

Re:Googling Uncommon Characters and Exact Phrases

One of the most annoying things about google for me is how it interprets queries with strange characters common to almost all programming languages. A google search for "ruby <<" returns no results related to the ruby append operator. A Simple search for "<<", by itself returns ZERO results.

Yes, well you see that's a problem common to most search systems. Non-alphanumeric characters tend to be reserved for search logic. It would indeed be nice if there was a way to force literals into the search terms, but for now we just have to make do the way we always have: search for ruby append [google.com] instead, or (if you don't know what it's called) search for ruby string operators [google.com] and find out.

Re:Googling Uncommon Characters and Exact Phrases

Yes. Try to find information on the web about the language "C+@". It's real, and it was developed at Bell Labs some years ago back in the Plan 9 era, but it's unsearchable.

One search feature

that has been lost was the "NEAR" keyword that AltaVista used earlier. I found it rather useful.

This could allow for a better search result when using for example "APPLE NEAR MACINTOSH" or "APPLE NEAR BEATLES"

Ho hum... Times changes and not always for the better...

Toileat seat

Does the algorithm account for the toilet seat's positon?

Google is human too

One interesting thing about the article was the down-to-earth lack of abstraction in the problems described, such as the teak patio palo alto problem. Other search engines brag about their web-filtered-by-humans approach, as opposed to the "cold" algorithmic approach of Google. But it turns out Google is pretty human too, only with higher ambitions of creating generalizations from the human observations.

if only...

If only they could solve googlebombing on news.google.com by bloggers with right wing agendas. The left wing agendas seem to be gone already, for some reason.

The most annoying thing about Google's results...

I find it extremely annoying the google indexes blogs.
Blogs are read only by bloggers and the press, and present absolutely no interest to normal people (including me). Currently, because of google's idiotic blog fetish, I have to eliminate 50% of the results just based on URLs, hoping that I won't stumble upon someone's personal ramblings. Blogs became popular only due to google's absolutely unexplainable love to blog content, and sticking it into perfectly normal search results, it's like searching in a world-wide-Myspace now.
The most amazing thing is when Google puts blog search results above the source of the story, to which the blogs are linking in the first place. I'm just waiting for this fad to die out like podcasting did. Unfortunatly, google controls the popularity blogging so it won't die out naturally, google at least has to stop indexing them... or put a "show/hide blog results" checkbox...

Page rank is only a part of the story

I'd like to know how they transform their queries before running them against the index. I.e. how they decide whether they should throw out the "stop" words (most prepositions, some verbs, some nouns) or keep them, whether they should throw in an alternative spelling or synonym, whether they should throw in a semantically related word or two to increase recall (this is evident when you search for something and get related words highlighted in the results), when to stem and when not to stem.

Those are the things that keep them ahead. Page rank is pretty much solved by now, which is why this dude is allowed to talk about it even at this level of detail.

WRT page rank it'd be interesting to know how they train the classifiers and individual classifier weights. The problem is that human experiments are extremely expensive for this stuff.

Re:Page rank is only a part of the story

Read the article, it gives a pretty clear picture of what's going on if you're a little familiar with classification ideas, eg bagging, boosting etc. Don't read further if you're familiar with those terms.

A classifier is a black box which takes some data as input, and computes one or more scores. The simplest example is a binary classifier, say for spam. You feed some data (eg an email) and you get a score back. If it's a big score say, then the classifier thinks it's spam, and if it's a small score it's not spam. More generally, a classifier could give three scores to represent spam, work, home, and you could pick the best score to get the best choice.

So you should really think of a classifier as a little program that does one thing really well, and only one thing. For example, you can build a small classifier that looks if the input text is english or russian. That's all it does.

Now imagine you have 100 engineers, and each engineer has a specialty, and each builds a really small classifier to do one thing well. The logic of each classifier is black boxed, so from the outside it's just a component, kind of like a lego brick. What happens when you feed the output of one lego brick to the input of another lego brick?

Say you have three classifiers: english spam recognizer, russian spam recognizer, english/russian identifier. You build a harness which uses the english/russian identifier first, and then depending on the output your program connects the english spam recognizer or the russian spam recognizer.

Now imagine a huge network with some classifiers in parallel and some classifiers in series. At the top there's the query words, and they travel through the network. One of the classifiers might trigger word completion (ie bio -> biography as in the article), another might toggle the "fresh" flag, or the "wikipedia" flag etc. In the end, your output is a complicated query string which goes looking for the web pages.

The key idea now is to tweak the choice thresholds. To do that, there's no theory. You have to have a set of standard queries with a list of the outputs the algorithm must show. Let's say you have 10,000 of these queries. You run each query through the machine, and you get a yes/no answer for each one, and you try to modify the weights so that you get a good number of correct queries.

Of course you want to speed things up as much as possible, you can use mathematical tricks to find the best weights, you don't need to go get the actual pages if your output is a query string you just compare the query string with the expected query string etc, but that would be depend on your classifiers, the scheme used to evaluate the test results, and how good your engineers are.

The point is that there's no magic ingredient, it's all ad-hoc. Edison tried a hundreds of different materials for the filament in his lightbulb. Google is doing the same thing according to the article. What matters for this kind of approach is a huge dataset (ie bigger than any competitors') and a large number of engineers (not just to build enough components, but to deprive its competitors of manpower). The exact details of the classifier components aren't too important if you have a comprehensive way of combining them.

Re:I'm familiar with all this stuff

Good question. I agree with you that the article doesn't say anything valuable in this respect :(

When you say that your system is limited by human involvement, I presume you mean that implementing new features can have serious impact on the overall design (and therefore on testing procedures)? Feel free to not answer if you can't.

One thing I found interesting in the article is that Google's system sounds like it scales well. It reminded me of antispam architectures like Brightmail's (if memory serves), which have large numbers of simple heuristics which are chosen by an evolutionary algorithm. The point is that new heuristics can be added trivially without changing the architecture. I think their system used 10,000 when they described it a few years ago at an MIT spam conference. Adjustments were done nightly by monitoring spam honeypots.

I'd love to see better competition in the search engine space. I hope you succeed at improving your tech.

Break through!

From TFA:
>>A search-engine tweak gave more weight to pages with phrases like "French Revolution" rather than pages that simply had both words.

So, now search engines are giving more importance to connected words rather than scattered words. How refreshing!

The Man Behind Google's Ranking Algorithm

Come now, everyone knows there's no man behind Google's page rank. It's handled entirely by an army of birds.

http://www.google.com/technology/pigeonrank.html [google.com]

"Millions Of Black Boxes"?

Not sure about this:

"Google rarely allows outsiders to visit the unit, and it has been cautious about allowing Mr. Singhal to speak with the news media about the magical, mathematical brew inside the millions of black boxes that power its search engine."

I could see tens of thousands, maybe hundreds of thousands, but millions?

do no evil?

"But last year, Mr. Singhal started to worry that Google's balance was off. When the company introduced its new stock quotation service, a search for "Google Finance" couldn't find it. After monitoring similar problems, he assembled a team of three engineers to figure out what to do about them."

But then they changed the algorithm and now Google Finance site is at the top.

The most informative line in the article...

One of the New Yorkers munched on cake.

Old google data

I find it frustrating when i am searching for free market data, often available in the form of press releases or summaries of whitepapers. Things such as the size of a particular software or appliance market.

When i search Google usually gives me information from 2001, 2002, 2003 and it is hard to tell it i want only data from 2006/2007. The problem is that the sites that end up in the search constantly refresh the ads and links around their old stories which makes google think its fresh.

This was not a big problem when most of the internet content was no more than a year old. This problem will get worse unless Google is smarter about recognising that the core content of a page (magazine story, whitepaper etc) was written in 2002 and it is now out of date and should be further down the list than something written in 2007.

Re:Google...

In Soviet Russia, they shoot idiots why don't realize this joke is dead.

Re:Google sucks.

Google Search is a primitive tool used by fanboys "Googling" for pictures of Natalie Portman.

Ha! Shows what you know. The only pics I search for are of a tall drink of Texas water named Patricia Vonne and of Cowboy Neal in his homemade Hulk costume. Who knew the Hulk wore a tri-corner hat & rainbow wrestling boots?

Re:Algorithm?

So how do you call the "thing" that you use to impement a heuristic?