The Problem of Search Engines and "Sekrit" Data

Nos. writes: "CNet is reporting that not only Google but other search engines are finding password and credit card numbers while doing its indexing. An interesting quote from the article by Google: 'We define public as anything placed on the public Internet and not blocked to search engines in any way. The primary burden falls to the people who are incorrectly exposing this information. But at the same time, we're certainly aware of the problem, and our development team is exploring different solutions behind the scenes.'" As the article outlines, this has been a problem for a long time -- and with no easy solution in sight.

Oh Yeah?

[Knunov]

"...search engines are finding password and credit card numbers while doing its indexing."

This is very serious. Could you please post the exact search engines are query strings so I can make sure my information isn't there?

Knunov

Re:Oh Yeah?

[Karma 50]

Just search for your credit card number.

By the way, does google have that realtime display of what people are searching for?

Google exploit patch for Apache

[Anarchofascist]

% cd /var/www
% cat > robots.txt
User-agent: *
Disallow: /
^D
%

The Problem of Search Engines and "Sekrit" Data

[NTSwerver]

Please change the title of this article to:

The Problem Incompetent System Administrators

If data is 'sekrit'/sensitive/confidential - don't put it on the web. It's as simple as that. If that data is available on the web, search engines can't be blamed for finding it.

I've got a solution!

[CraigoFL]

Every web server should have a file in their root directory called "secret.xml" or somesuch. This file could list all the publicly-accessible URLs that have all the "secret" data such as credit card numbers, root passwords, and private keys. Search engines could parse this file and then NOT include those URLs in their search results!

Brilliant, huh? ;-)

On second thought, maybe I shouldn't post this... some PHB might actually think it's a good idea.

Google exploit patch 0.2 for Apache

[Anarchofascist]

Oops! Version 0.2 already:

% cat > /var/www/html/robots.txt
User-agent: *
Disallow: /
^D
%

Hell, No.

[tomblackwell]

You should be writing that type of data on the backs of envelopes and leaving them scattered around your living room...

Re:Insert foot in mouth....

[simong]

Not necessarily, they are chief executives after all.

Re:Oh Yeah?

[4of12]

Yeah!

I just typed in my credit card number and found 15 hits on web sites involving videos of hot young goats.

Oh, for regular expression searching in Google

[EnglishTim]

I could be a rich man...

(Not, of course that I'd ever do anything like that...)

Searching with regular expressions would be cool, though...

Must... blame... someone....

[JMZero]

INetPub means "INetPublic" not "INetPubrobably a great place to put my credit card numbers".

Why are stupid people not to blame for anything anymore?

Re:A symptom of poor programming...

[Brainless]

I manage a Cold Fusion web server that we allow clients to post their own websites to. Recently, their programmer accidentally made a link to the admin section. Google found that link and proceeded into the admin secion and indexed all the "delete item" links as well. I found it quite amusing when they asked to see a copy of the logs complaining the website was hacked and I discovered GoogleBot deleted every single database entry for them.

Business Model

[Alomex]

A while back there was a thread here about the weakness of the revenue model for search engines. Maybe we have found the answer, think about all the revenue that Google could generate with this data!

Anybody knows when Google is going public?

Blaming Google for this...

[night_flyer]

Is like blaming the Highway department for speeders...

Re:Stopping Google won't stop the problem...

[mobiGeek]

but Google undoubtedly uses techniques beyond that of the casual browser

Uhh...no.

HTTP is an extremely basic protocol. Google's bots simply do a series of GET requests.

It would be possible that Google's bots have a database of username/passwords for given sites, but the more likely scenario is that they have stumbled across another way to get the "protected" information:

• a link which contains a username and/or password
  /protected/show_article.pl?username=foo&passwo rd=bar&num=1
• a link to the pages which by-passes the protection scheme
  /no_one_can_find_this_cause_Im_3l33t/article1.html
• someone else posted the information elsewhere, and this is what is actually crawled

I ran robots for nearly 2 years and was harassed by many a Webmuster who could prove that my robots had hacked their site. They'd show me protected or secret data. It typically took 3 to 5 minutes to find the problem...usually the muster was the problem themself.

HERE'S A NOTE OF WARNING TO WEBMASTERS:
Black text links on black backgrounds in really small fonts are NOT secure.

Maybe I should get this posted to BugTraq...or would MS come after me??

Re:A symptom of poor programming...

[Anonymous Coward]

First thing I do when bored and surfing (porn sites) is whenever I see a new link with a non-standard index page (anything other than index.html) I chop it off and see if I can get a listing. Since most porn sites run on Apache, and Apache by default does not disable directory listings, and since most porn sites are designed on Windows (can you say index.htm?), and since the default Apache index page is index.html, this leads to a great deal of free fun.

Re:Nice work, Legion303.

[well_jung]

"Trees cause more pollution than automobiles do." --Ronald Reagan '81