The Problem of Search Engines and "Sekrit" Data 411
Nos. writes: "CNet is reporting that not only Google but other search engines are finding password and credit card numbers while doing its indexing. An interesting quote from the article by Google: 'We define public as anything placed on the public Internet and not blocked to search engines in any way. The primary burden falls to the people who are incorrectly exposing this information. But at the same time, we're certainly aware of the problem, and our development team is exploring different solutions behind the scenes.'" As the article outlines, this has been a problem for a long time -- and with no easy solution in sight.
Tangential Google Question (Score:5, Interesting)
If I want to find lyrics to a song, the site that has them will often be down, but the cache will still have them in there.. Why is what google is doing 'okay' but what the origional site not okay? Or do they just leave google alone?
Google shouldn't lift a finger (Score:2, Interesting)
Insert foot in mouth.... (Score:2, Interesting)
"Webmasters should know how to protect their files before they even start writing a Web site," wrote James Reno, chief executive of Amelia, Ohio-based ByteHosting Internet Services. "Standard Apache Password Protection handles most of the search engine problems--search engines can't crack it. Pretty much all that it does is use standard HTTP/1.0 Basic Authentication and checks the username based on the password stored in a MySQL Database."
And chief executives of a hosting company should know how Basic Authentication works before hosting web sites...
Crewd
Re:A symptom of poor programming... (Score:5, Interesting)
"Index of
"Index of
"Index of
"Index of
"Index of
Re:Tangential Google Question (Score:3, Interesting)
Given that they do have (for now) some sort of immunity, it opens a loophole for publishing illegal data. Simply set up your site with all of Metallica's lyrics / guitar scores (all 5 of them, heh). Submit it for indexing to Google, but don't otherwise attract attention to the site. When you see the spider hit, take it offline. Now the data is available to anyone who searches for it on Google, but you're not liable for anything. The process could be repeated to update the cache.
robots.txt (Score:2, Interesting)
From my web logs, I see that a lot of HTTP bots don't care crap about /robots.txt. Another thing which happens is that they read robots.txt only once and cache it forever in the lifetime of accessing that site, and do not use a newer robots.txt when it's available. It'd be useful to update what a bot knows of a site's /robots.txt from time to time.
HTTP bot writers should adhere to using information in /robots.txt and restricting their access accordingly. In a lot of occasions, webmasters may setup /robots.txt to actually help stop bots from feeding on junk information which they don't require.. or things which change regularly and need not be recorded.
Many crawlers ignore robots.txt (Score:3, Interesting)
Sure enough. (Score:3, Interesting)
At any rate--scary it is.
standing naked in front of the window (Score:3, Interesting)
this guy's just looking for free hype for his book. if that's the kind of advice he offers, he's doing more harm than good.
Re:Tangential Google Question (Score:2, Interesting)
Re:A symptom of poor programming... (Score:5, Interesting)
-Legion
Re:Stopping Google won't stop the problem... (Score:3, Interesting)
Re:Stopping Google won't stop the problem... (Score:2, Interesting)
Web servers could ship configured to not AutoIndex, only allow specific file types (.jpeg,
Of course, putting something in public that you don't want someone to see is just plain stupid, but apparently we need to make stupid people feel like they're allowed on the 'net.
MicroSoft Passport Credit Card # avaliable (Score:2, Interesting)
script for extracting credit card numbers from
the Passport database. Scary. Dont buy anything
through it until they fix it.
Blissful ignorance backfires again. (Score:3, Interesting)
Google's comment was:
"The primary burden falls to the people who are incorrectly exposing this information."
This is where they should have stopped. Those who find their credit card information in a search engine will learn a lesson and use services that actually take care of their customers' security and privacy. Google shouldn't have to clean up incompetent people's mess.
In the long run, these things can only lead to the ignorant (wannabe?) players in the market slowly dying because they don't know what they are doing.
I personally hope someone gets a taste of reality here, and that only the serious players survive. The MCSE crowd may finally learn that there's more to it than blind trust in their own (lacking) ability.
Different file types make my day (Score:3, Interesting)
I'll never forget the day I first saw a .pdf in Google search result. Not that long ago I saw my first .ps.gz in a search result. I mean, how dope is that!? They're ungzipping the file, and then parsing the postscript! Soon they'll start uniso-ing images, untarring files, unrpming packages, .... You'll be able to search for text and have it found inside the README in an rpm in a Red Hat ISO.
Can't wait until images.google.com starts doing OCR on the pix they index...
Re:A symptom of poor programming... (Score:2, Interesting)
It's a silly mistake, I don't have a clue as to how google came accross the link. Like with anything new it's going to take some time before this becomes "common sense" and people do not put this information on public servers.
- subsolar
P.S. It's possible to generate a url that when clicked by somebody behind a linksys router to enable remote administration if you know the password. I've turned it in to linksys but gotten nothing but silence from them.
Password search (Score:3, Interesting)
filetype:htpasswd htpasswd
Scary how many
-- Azaroth
DMCA (Score:2, Interesting)
Not much worse than some "commercial-grade" encryption...
Maybe somebody should consider suing Google under the DMCA. I haven't studied the DMCA with enough detail to be sure of this (and much less studied law, for that matter), but i guess Google is easily guilty of the following "crimes" against modern society:
- linking to decryption algorithms
- linking to reverse enginnering tools
- linking to passwords that could be used to circumvent somebody's copyright.
- storing and distributing all the above (with google's cache)
As I understand current legislation, Google should not even have the right to define what is public or not like they're trying to do. Even the safe-harbour provisions do not immunize them from having to remove unlawful content.
Such a lawsuit would make for an interesting debate, and with a bit of luck could get us all rid of this stupid law.
C.
Re:A symptom of poor programming... (Score:1, Interesting)
Re:Tangential Google Question (Score:2, Interesting)
Oooh.. that's a particularly good one.. kinda like getting high-bandwidth web service FOC, if you build your site URLs to ride along the google cache instead of your own... (gears cranking)..
Re:What did they expect? (Score:1, Interesting)
Not really, more a problem that incompetent administrators don't know, don't care, or don't think it's their problem; the stupidest of users don't know what they shouldn't do, and unscrupulous folk take advantage of that.
The existing systems and models work, it's just that badly run sites don't use them correctly, if at all. Most users have the basic knowledge (or concern/fear) not to post credit card numbers in a public location online or to an untrusted site, but some stupid ones will (perhaps deservedly) pay for their mistakes and others may get screwed over by badly secured 'trusted' sites or convincing but spurious sites.
At any other time in the past few years, this would not ordinarily be a societal problem. Sure, a few peoples' passwords and credit card numbers will leak out.... But now, this is a national security problem, because we are being attacked by a foreign force who might abuse leaked passwords to access critical systems and cause chaos in this country. President Bush and his staff are very concerned about a cyberwar, because it can be waged without physically having Arabs in the States to commit the terrorism. That is very dangerous indeed.
This is hardly now a concern related to regular people publishing on the web. It IS a concern about security, but sensitive information should be protected by a competent admin with appropriate controls. Passwords to truly critical systems are protected both through online security methods and physical requirements; you aren't going to find them published for all to read on the family page of the guy with access to the button, nor in any cache of his online activities and postings.
There is reason for concern there, but the solution to that concern is to make sure that the appropriate procedures are followed. As I say, it's completely irrelevant to 'standards' of publishing on the web or who can put up a homepage - only (if anything) to the competence and security awareness of those running the servers.
There are probably millions of "here's my cat, I've joined 500 webrings, I like icecream, here's some annoying MIDI music that a button put on the page for me, pleeeease sign my guestbook" pages on Geocities and its like (Homestead is a really bad example) but I wouldn't call them a threat to National Security.
Anything that might be called such a threat should not even be stored in an unprotected computer, let alone online. If anything the main problem might be having sensitive information on a private computer, where a cable ISP has discouraged or failed to mention appropriate firewalls (as is often the case). But for seriously sensitive stuff, this situation would never be permitted.
I'm not sure what the solution is, but a good first step is for companies to raise the barrier to entry to publishing web pages. Geocities and Angelfire should force users to demonstrate their competence before uploading their first page.
Why? Crap page design results in something people don't want to look at. All it does is waste space on Geocities' servers, you never have to see it if you don't want to look for it. The only possible gripes are if
a) they publish sensitive information on there - which only rebounds on them if it's CC numbers or passwords. An adult with access to truly sensitive information would be bound by employment/secrecy clauses in their contracts so are hardly likely to 'accidentally' reveal a government secret.
or b) they clog up the search engine results with lots of crappy listings. Which is true, but good search engines take into account how popular the page is, which tends to push the crappy pages down to the end of the listings.
Perhaps requiring an A+ certification number would help? And Microsoft should take away the parts of FrontPage that allow users to generate documents without writing in HTML. That would help ease the problem, I reckon.
No it wouldn't, because the problem is not with page designing abilities, or with posting sensitive information on the web. If someone puts on there homepage "My password is: " they take responsibility for that. If they were to put "The Nuclear Launch Code is:" they would probably be shot at dawn, but that's not going to happen with the people trusted with such information.
If the host has a password list that is world readable, then there's a serious security problem. If anything the A+ certification should be required for those hosting, not creating, the pages.
In conclusion - if everybody does their part to help solve this problem and stop information leakage, we will be a safer, more secure society without giving up any more civil liberties.
Probably, but the point is that the large-scale sensitive material you reference is secured by appropriate technology, contract agreements and competent staff; but for personally sensitive material, apart from not being at fault for doing anything incredibly stupid, the people we entrust with such information must be competent and concerned. It's not about the codes for war, it's not about the abilities of Frontpage users, the actual problem lies somewhere in the middle.
(now using Frontpage to display the codes for war on the US government homepage, there might be a problem).
Re:Tangential Google Question (Score:2, Interesting)
FileName: MyFile
Size: FileSize
Encode in Base64, rot13 it, and then call it protected under DMCA, bonus points.
Of course, your web server would only accept connections from the google spiders, and you'd effectively have a free file distribution service. Not saying this would actually work, but I think there's a chance it'd work.