Github Kills Search After Hundreds of Private Keys Exposed 176
mask.of.sanity writes "Github has killed its search function to safeguard users who were caught out storing keys and passwords in public repositories. 'Users found that quite a large number of users who had added private keys to their repositories and then pushed the files up to GitHub. Searching on id_rsa, a file which contains the private key for SSH logins, returned over 600 results. Projects had live configuration files from cloud services such as Amazon Web Services and Azure with the encryption keys still included. Configuration and private key files are intended to be kept secret, since if it falls into wrong hands, that person can impersonate the user (or at least, the user's machine) and easily connect to that remote machine.' Search links popped up throughout Twitter pointing to stored keys, including what was reportedly account credentials for the Google Chrome source code repository. The keys can still be found using search engines, so check your repos."
At least... (Score:5, Funny)
they've been seen by 'many eye balls'.
That's good right?
Re: (Score:1)
I don't know about that. I, for one, don't want my eyeballs seeing any of you guys expose your private "keys"
Re:At least... (Score:5, Funny)
Re: (Score:2)
I don't know about that. I, for one, don't want my eyeballs seeing any of you guys expose your private "keys"
Especially Goldmember's private key.
Re: (Score:2)
neither do I, but seeing the locks would be something I'd be interested in
Re: (Score:2)
You guys are making a real hash of this thread.
This is why developers are not sysadmins (Score:3, Insightful)
This is why developers are not sysadmins.
These kinds of repositories need to learn that and not let these folks do this sort of thing. If would be simple to use a regex to filter out the posting of these sorts of files. Maybe Devs should even be charged a couple dollars to get a decent review of these things.
Re:This is why developers are not sysadmins (Score:5, Insightful)
No. This is actually completely absurd. A developer that cannot grasp the concept that private keys have to be kept private, cannot be trusted to do anything but screw up the most basic security provisions when writing code.
They should get a kick in the ass, such as three months without any sort of commit privileges, and mandatory code review for an year. THAT should be enough to make it stick, and impress on them the real gravity of their failure. Otherwise, they will just chalk it up as "an annoyance done by those uninteresting people who should learn to code before they go pestering code-gods".
Re:This is why developers are not sysadmins (Score:5, Interesting)
Sysadmins should also know how to code. Nothing better than showing them their screwup and the solution to it.
Plus, since all sysadmins, real ones anyway, are already competent in several scripting languages it is not that hard a skill to add if all you need to do is be better than bottom of the barrel programmers.
Re: (Score:2)
I can't remember the last time a developer had a workable, secure solution to my problems.
There's a reason you hear, 'fix your own code' a lot more than 'fix your servers' in a development environment.
Re:This is why developers are not sysadmins (Score:4, Interesting)
I dunno about that here. Ever since they rolled out Sophos Full Disk Encryption on every desktop and server here, it's contributed more to downtime than any virus/malware ever has. I think literally every person in this office has had to have their machine completely rebuilt after it got corrupted somehow, and that includes our testing servers as well.
All I can say is, thank god our production servers are out of our company's control. They haven't had any issues, but then again, they also don't have Sophos malware on them either.
Re: (Score:2)
This is likely out of the hands of your sysadmins. Servers should not be running this POS, EOS. But you know, then some CIO reads about it in a magazine somewhere....
Anyway, my condolences to your build farm.
Re: (Score:2)
I'm just glad they don't advertise "business solution" software in Skymall.
Sadly, they do. Not many, but just enough to drive your IT dept batty.
Here is an example:
http://www.skymall.com/shopping/detail.htm?pid=204348403&c=102961963 [skymall.com]
Re: (Score:2)
No. This is actually completely absurd. A developer that cannot grasp the concept that private keys have to be kept private, cannot be trusted to do anything but screw up the most basic security provisions when writing code.
They grasp the concept just fine. It isn't that they don't understand, it's that they don't see it as their problem.
Re: (Score:1)
Re: (Score:2)
Could they not supply a .gitignore ?
Either way simple enough to have a find script run, before you make it public. Basically on every commit turn off public access, run your clean script, then turn it back on. If this causes errors, that seems better than this.
I admit at work we mostly use a combination of CVS and a 2x4 to hit developers with to avoid these issues while still having a nice simple repository.
Re:This is why developers are not sysadmins (Score:5, Insightful)
(Yes, there is also a nice ~/.ssh/config file, so that you also know which locks those key fits...)
Re:This is why developers are not sysadmins (Score:4, Insightful)
I've seen several people comment that they have their home directory config files under version control. If you're using git for that, it's a fairly simple next step to then "backup" the repo to github.
"It's only config files; nobody would be interested in those."
Re: (Score:2)
There's absolutely nothing wrong with that. My question is why they're storing their home dir on a *public* git repo...
Re: (Score:3)
Duh... it's so they don't have to remember their passwords to get their configuration and RSA keys back!
Re: (Score:2)
There's absolutely nothing wrong with that. My question is why they're storing their home dir on a *public* git repo...
Because it's only free if it's a public repository.
Re: (Score:2)
your link has only .pub files, though. I see no private keys here.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Yo can launch a filter like program from an event before the commit is saved on the remote repository, and stop with an error, this force the careless developer to amend their commits
Re: (Score:2)
If we could pair this with some sort of clue bat strike via IP that would be best.
Re: (Score:2)
If we could pair this with some sort of clue bat strike via IP that would be best.
I hear a guy is working on a IP-based face-stabbing machine.
Re: (Score:2)
For new screwups the soloution would be to just reject the push and let the developer sort it out.
For existing screwups it's not so easy. One of the characteristics of hash-based dvcs systems like git is that they make it REALLY painful to change history. You could generate new commits and blacklist the old ones but doing so would tip off all users of the repository that something was up and those users would still have their copies of the original commits.
Re: (Score:2)
Or just leave the keys and let them learn their lessons the hard way.
Re: (Score:2)
How are other people endangered if I leave my private key in a public place?
Re: (Score:2)
Re: (Score:1)
This is why developers shouting "give me full access now" should always be denied - there is a totally different mindset between developers and admins (or DevOps) when it comes to protecting things like SSH keys.
Both groups have similar (or certainly overlapping) technical skill sets, but have very different motivations.
ObXKCD: http://xkcd.com/705/
Re: (Score:3)
The best system I have seen is not allow devs any access to production environments. All pushes done by sysadmin and dev boxes identical to production.
This demonstrates a critical flaw... (Score:2)
Re: (Score:3)
Re: (Score:2)
The first thing you learn is that your private SSH keys are sacrosanct. Most developers seems to just go through a howto on how to generate a SSH key and don't think about anything after that. They're probably all using node.js or something.........
Followed by going through the git howto that tells them to .
git init
git add
git commit -m "Initial Commit"
Re: (Score:3)
This is why key management should be part of the operating system, and every piece of software that doesn't use those APIs should be suspect.
It's simply too big a subject to expect everyone who is in danger of falling prey to something similar (everyone who uses a computer) to manage on their own. If you know where every individual piece of software you run stores every single key, you are a very, very rare person. You're also probably mistaken.
Even if we started down the path, it would take a long time, th
Re: (Score:2)
Those people should just be fired. This is pretty much par for the course for the average new developer.
Re:This is why developers are not sysadmins (Score:4, Insightful)
Developers are the best sysadmins you can have, since they're actually somewhat competent.
It's just that they've got better things to do and are paid more.
Developers are normally careless sysadmins. Sysadmins are usually inept programmers. Someone that really can do both well is a great asset.
Good sysadmins get paid about the same as good developers. At least that's my experience.
Lol (Score:1)
'nuff said.
Deserving (Score:1)
Developers (using the term loosely) deserve whatever ill comes from checking in private keys. Public repo or otherwise
Re:Deserving (Score:5, Insightful)
Exactly, GitHub shouldn't disable a site feature to protect the stupid.
Re: (Score:2)
Re: (Score:3)
I have to say I agree. We need to stop coddling people who do mindblowingly stupid things.
Let them post their private keys. Let them learn the hard way what happens.
Same with people who steady a scalding cup of coffee by putting it between their legs, while driving no less.
I don't even WANT to know what prompted the warning label on a toilet brush that stated, "Not to be used for personal hygiene".
Bring back lawn darts, kinder eggs and buckyballs, remove the warning labels from everything, and add a law t
Re: (Score:2)
I'd say oblivious, and deserves a -5, not a +5.
The developers aren't the only people at risk here. Anyone who checks out the development tree is at risk. And just having the source code doesn't automatically protect you. You aren't very likely to go over the code with a fine toothed comb. And even if you did, it would be quite easy to miss an inserted "mal-feature". (I can't really call it a bug, since I'm thinking of code that was doing what it was intended to do, just not what *you* intended it to do
Search engines (Score:5, Informative)
site:github.com inurl:id_dsa
Idiots...
Works on sourceforge too... (Score:3)
Re:Search engines (Score:5, Funny)
Stop, Google will shutdown search because of that
Re: (Score:3)
Re: (Score:2)
intitle:index.of
Interesting, it still finds lots of music. Weird that the MAFIAA hasn't discovered this yet, and sent them lawyer's love letters...
On a hunch, I tried my own search terms too, an I was quite surprised to notice that inurl:cfm inurl:page_id has several vulnerable sites on its second page [google.com]...
Re: (Score:2)
Re:Search engines (Score:5, Interesting)
Heck, Google disabled searching number ranges after some enterprising folks used them to harvest credit card numbers - doing searches for numbers between 4000000000000000 and 5999999999999999 which will get back lists of credit cards (Visa/MC) that Google indexed because someone put the list up.
Re: (Score:2)
Mikko Hypponen recently posted an example of such a search query:
https://twitter.com/mikko/status/287615660115243009 [twitter.com]
(Don't worry; the "instaban" only affects that query; subsequent searches will still work.)
Proof... (Score:2)
...that even (supposedly) smart people can be stupid.
Re: (Score:3)
Re: (Score:2)
I thought most of those lived in or near Redmond, WA and Redwood City, CA.
Re: (Score:2)
Re: (Score:2)
In a environment where idiots write code, you will never see a coder calling "idiot" to another.
Been there, saw that.
(I got fired, by the wat - I wasn't idiot enough!)
Re: (Score:2)
I wasn't idiot enough!
Maybe you were...
It depends. In some jurisdictions, being fired gives you rights to extra compensation, so it may be better to be fired rather than resigning...
Moreover, being fired gets you unemployment benefit, whereas resigning doesn't.
So the smart move may indeed be to show how smart you are, and cause yourself being fired. Of course, it all depends on the circumstances, as being fired rather than resigning dimishes your chances of finding a job elsewhere.
Re: (Score:2)
It was a smart move.
I ended up in a job with a lesser payment (probably because of the laid off), but 6 months later when a LOT of guys were fired (the company decided to shrink the department - somewhat I had envisioned by my previous experiences in this field), I already was employed, with my bills being paid, while a couple dozen of professionals (a very few of them so good or better than me) were struggling for a new position in the market.
I'm not a profet, I didn't knew for sure if the department will
WTF? (Score:2)
Re: (Score:1)
Well, they're also protecting you (potentially) from the actions of idiots.
I just saw this, sort of (Score:5, Interesting)
Re: (Score:2)
Same thing on computers' case labels like at work.
Re: (Score:2)
Those aren't actual working keys, though most of the time. Usually on machines from the big guys, they're nonworking keys - because the real activation key is built into the BIOS. For earlier (pre-Vista) versions of Windows, they would require manual act
Re: (Score:2)
overreaction? (Score:4, Insightful)
Re:overreaction? (Score:4, Insightful)
Because some of these might be test keys or place holders. If the key is not valid on any system and is just test data, it should not be a big deal to post publicly.
Re: (Score:2)
Example: the keys for Vagrant [github.com]. Vagrant is a system for managing virtual machines for development purposes. The ssh keys are used to facilitate passwordless login. They aren't typically exposed to the outside world, and they are clearly labelled as insecure.
Re: (Score:2)
That thing should really generate new ones on install.
That way there are no keys to expose to the world that anyone would know.
Re: (Score:2)
One of the things people do with it is build base boxes, which are preconfigured virtual machines, and share those base boxes for other people to build upon. In order to do so, the people who receive these base boxes need the private keys they are configured with.
You could distribute the private keys with the base boxes, I suppose, but then you are stuck sharing multiple files instead of just one, and you can't install a base box by running one single command with a URL argument any more. It increases
Re: (Score:2)
I think you've misunderstood. A base box is a virtual machine. A digital file. They are run during development. They aren't production servers that other people access, and they aren't physical hardware.
Re: (Score:2)
People will use them in production, it will happen.
Best to have the machine regenerate keys on first boot and make the user go get them.
Re: (Score:2)
I think the summary is wrong. status.github.com [github.com] seems to indicate that github's search cluster died, not that they took it down. More likely is that there was a flood of search requests for private keys at the same time and the search cluster buckled.
Re: (Score:2)
Some projects deal with SSH keys and include them for testing purposes: https://github.com/trolldbois/sslsnoop/blob/master/test/id_dsa-1.key [github.com]
Stupid people... (Score:4, Insightful)
These stupid people should be had their accounts suspended.
People should be accountable for their actions, and these idiots are potentially compromising third party data security!
ICO didn't fined Sony for the information leak on that Anonymous attack? Why in hell GITHUB user's should be less accountable for things THEY ARE FSCKING COMMITING in their accounts?
Re: (Score:2)
You put a automated test in the same basket as an idiot that commit private keys to God knows what, and tells ME to think?=P
Re: (Score:2)
Why would they have their accounts suspended?
Why should, then, GITHUB shutdown a useful feature (the search!) in order to prevent data mining over that keys?
Pretty stupid or not, GITHUB is hurting everybody because of them.
So, or GITHUB are pretty stupid bastard themselves, or some of these keys can cause some serious havoc. I stand what I said : identify and suspend the idiots instead of fscking everybody!
github did not take search down because of this (Score:2, Informative)
According to their twitter and status pages, the search is currently unoperational due to problems with their search cluster. They recently released changes to their search including, I believe, a move to ElasticSearch. The linked article says as much, too, so yet another fail in a slashdot summary.
Nothing has changed... (Score:5, Funny)
Looks like these grad students have all growned up and uploading it all to the cloud.
Re: (Score:3)
When people did stuff like that in my sysadmin classes we were encouraged to teach them a lesson. Far better to edit their login script to log them right back out than delete their homedir contents, or change their path so they got other versions of common programs. Probably the meanest was to make it so instead of calling the work submission script it called rm on whatever they were trying to submit as their classwork.
Re: (Score:3)
A subtler prank that I pulled on a friend who left himself logged in to one of the public undergrad labs (where there was the risk that an actual asshole would delete your stuff, send email as you, or something similarly cruel) was to add "echo 'sleep 1' >> .cshrc" to the end of his .cshrc before logging him out. I chuckled to myself, and then forgot about it.
A week later, when it was 5 minutes before a submission deadline and he was yelling at the terminal to finish logging in (since it was taking 2-
Re: (Score:3)
I hate that attitude though. I told a friend once that I didn't use authentication with X windows because no one is ever going to bother to interfere and I didn't care if anyone was snooping. So he went out and decided to pop up random pictures on my screen and post messages until I relented. So I wasted a lot of time learning all about authentication and configuring it correctly, not to keep out adversaries but to keep out friends...
Re:Nothing has changed... (Score:5, Funny)
Yeah ... I was "that guy". The first time I installed Linux in 2000, I was annoyed that I needed "permission" to write to a directory outside of my home directory. I was coming from a Windows world, after all.
I solved this "problem" by chmod 777 the entire filesystem. Hah. Problem solved. Needless to say, I couldn't start the machine back up again. I'm guessing it killed itself from the shear embarrassment. After that, I decided it may be in my best interest to read the manual.
I'll do that one of these days :)
Re: (Score:2)
It's probably obvious and I'm just being stupid, but I can't think what you could possibly break by setting all perms to 777. Yeah, you'll mark a bunch of non-executable files as executable, but nothing should be trying to execute them anyway. There may be a few files (like /etc/passwd|shadow) which some components might refuse to use if they're world-readable, I suppose...
Any idea what broke?
BTW, my similar story: I purchased a NeXT machine in 1991. It came with a 110 MB hard drive, which wasn't a lot
Re: (Score:2)
Anything with the sticky, setuid, or setgid bits set.
Re: (Score:2)
chmod'ing 777 kills the setuserid-bit.
This means that programs like "su" or "ping" or "passwd" break.
Also, some important daemons probably check whether their required files are world-writable and refuse continuing...
Re: (Score:2)
chmod'ing 777 kills the setuserid-bit.
Duh. I knew it was something obvious. Thanks.
Re:Nothing has changed... (Score:4, Interesting)
Someone in my class installed a game in the officially-public network share. He was writing an AI for it, for a project. Other students found it, and played it.
It had taken a lot of hacking to get the game to run on Linux, and he was annoyed other students had played it without putting in that effort. So, he altered the 'start.sh' script to generate an ssh key, add the public part to the user's authorized_hosts file, and move the private key somewhere obscure.
He then got bored with the AI project.
Some time later, while helping in a tutorial, I was showing a student how to set up an SSH key. The authorized_keys file already contained about 20 entries. The AI guy was sitting at the next computer, and told me what he'd done (I knew him quite well, but he hadn't told me what he'd done until now). He found over 200 private keys in the obscure place. He deleted them, chown -R go-rwx'd the game, and we thought that was the end of it...
About a year later, Debian had that OpenSSL bug. The sysadmins ran a script across everyone's authorized_keys file, and removed any entries from keys generated by Debian OpenSSL. The email ended (I still have it):
By the way: some of you have FAR TOO MANY authorized_keys ENTRIES
and we seriously recommend that you radically shrink these down.
As I said, we recommend kerberos tickets or ssh-agent instead!
...so I don't think they knew how they got there.
Re: (Score:2)
This is common even in other walks of life. I see a few of my colleagues routinely bringing their passports in their brief ases they lug to work. They constantly open it to retrieve reading materials, music players, etc in the bu
Lol (Score:2)
"Oh sh**..."
Not so many (Score:4, Insightful)
Hundreds of keys from a million accounts; less than one in a thousand developers screwed up. Call a doctor at once! Then ask him about outliers in large populations.
Re: (Score:2)
Well, you have to consider quality over quantity. There may only be less than one in a thousand developers who screwed up but what if the keys that were exposed belong to super-important servers such as those that control Google Chrome source code or some other big project?
Re: (Score:2)
A big mess to clean up (Score:5, Informative)
Not quite. They're already out there. The keys are still in the revision history. People have forked and cloned it.
Hopefully the developers who created these keys know that besides removing them from the repo, the keys can no longer be used. They must be removed from every .ssh/authorized_keys file, every service like Github that uses them for deploying code, etc.
Re: (Score:2)
Github is officially denying [github.com] that the search feature being killed has anything to do with the exposure of keys. They also have a link on the same page to information on how to purge keys from your repository. (Make of that what you will.)
Proof Github took search down? (Score:2)
This doesn't suggest github took anything down on purpose: https://status.github.com/messages [github.com].
Seems to me they were just experiencing some technical difficulties from all the people sharing those search links and having a laugh at the stupids...
I skimmed over the github site and didn't find anything that would suggest otherwise at least.
Of course I didn't read the articles because they seem badly misinformed and confuse private keys with passwords.
Hard Coding does its lethal work (Score:2)
Re: (Score:2)
Re: (Score:2)