Bots Scanning GitHub To Steal Amazon EC2 Keys 119
New submitter juniq writes: As one developer found out, posting your Amazon keys to GitHub on accident can be a costly mistake if they are not revoked immediately.
"When I woke up the next morning, I had four emails from Amazon AWS and a missed phone call from Amazon AWS. Something about 140 servers running on my AWS account. What? How? I only had S3 keys on my GitHub and they where gone within 5 minutes! Turns out through the S3 API you can actually spin up EC2 instances, and my key had been spotted by a bot that continually searches GitHub for API keys. Amazon AWS customer support informed me this happens a lot recently; hackers have created an algorithm that searches GitHub 24 hours per day for API keys. Once it finds one it spins up max instances of EC2 servers to farm itself bitcoins."
"When I woke up the next morning, I had four emails from Amazon AWS and a missed phone call from Amazon AWS. Something about 140 servers running on my AWS account. What? How? I only had S3 keys on my GitHub and they where gone within 5 minutes! Turns out through the S3 API you can actually spin up EC2 instances, and my key had been spotted by a bot that continually searches GitHub for API keys. Amazon AWS customer support informed me this happens a lot recently; hackers have created an algorithm that searches GitHub 24 hours per day for API keys. Once it finds one it spins up max instances of EC2 servers to farm itself bitcoins."
Now where have I heard this before... (Score:5, Insightful)
Oh right: http://developers.slashdot.org... [slashdot.org]
Re:Now where have I heard this before... (Score:5, Funny)
There ought to be bots scanning /. to weed out duplicates.
Re: (Score:1)
Slashdot de-dupe..
Re: (Score:3)
This was also covered at Defcon 22... and not just for AWS keys: http://youtu.be/nBR7Kru6JX0?t=... [youtu.be]
Re: (Score:2)
I imagine it would be rather trivial for Git-Hub and others to scan uploads before they go "live" and either censor keys or ask "are you sure you want to do this"
The Stupidity of Human Greed (Score:1)
People....
Re: (Score:2, Insightful)
How? They got $2000 worth of cpu time from amazon. If they'd waited the guy would have surely thought to change the damn API key the next morning (surely no one is dumb enough, to think just removing the stuff after the fact will help - I guess they also don't bother getting new credit cards when their wallet gets lost and returned by a nice stranger that very same day).
Re: (Score:2)
I assume the idea is that you make more money stealing $1 many times from more people over a year than you do trying to steal all of it from all of it at once.
Re: (Score:3)
Re: (Score:3, Insightful)
I assume the idea is that you make more money stealing $1 many times from more people over a year than you do trying to steal all of it from all of it at once.
Wal-Mart!
Re:The Stupidity of Human Greed (Score:5, Insightful)
It works in real life all the time, actually - companies do this quite routinely. Jack up your bill by $2 and they can rack up millions over the year, and it doesn't matter if it's a contract or not because how much are you going to spend trying to recovery $24/year? If it's a cellphone contract, the max they're going charge you is $48 more over the contract's 2 year lifespan. You going to sue them over that? Now repeat that for a million subscribers and that's an extra $2M a month in free profits.
Maybe you can jack it up by $5 ($60/year) because it's still too low to bother.
It's why they made class-action lawsuits, because someone stealing $48M/year would get sued/arrested/etc if it was against one person, but against 1M people? Worthless.
As for this, well, given it's still "free money", even at Bitcoin's deflated value of around $350 or so, it's still free money. Who cares about efficiency or anything when you can steal CPU cycles like that - just scan github checkins for the key, then use the APIs to automatically create sessions and all that and rack up the bitcoins. Even the github scanner doesn't have to be owned by the user - they probably stole some guy's EC2 credential and are using one of his instances for it unbeknownst to the user. Free money!
Re: (Score:2)
Not even that.
raise the price by 1% and it goes right to your bottom line. Most people won't even notice that the $99 item costs $99.99. It is how gas companies do it.
Re: (Score:2)
Doesn't matter if their nuclear road flare gets their instances shutdown before a single shitcoin is mined. Given the speed of CPU hashing, even 1000 instances would take days to amount to anything. (the fastest dedicated miner does 6TH/s, and it would take a week to generate 0.5BTC -- worth about $150)
Re: (Score:2)
but the bot operator isn't paying for it, why do they care how efficient it is? if it takes a week worth of cpu time to generate $150 that is still $150 of pure profit for them. at the same time they can probably also sell cycles on the instances they hijacked to spammers or some other group
Re: (Score:2)
Well, you do realize that AWS DOES offer instances for GPU calculations. Still much slower than dedicated hardware, but basically it's free (as in stolen).
Summary without technobabble (Score:1)
Guy stores his password online, is surprised when he gets got.
Re: (Score:2)
More like use a fricking passphrase at least to protect your private key and use some kind of agent to save you from typing that passphrase again and again.
I sometimes use passphraseless private key when I control where it is stored but never store a passphraseless private key on line.
Be aware of key loggers and other means to get you passphrase once your private key is stored online also.
Wait! Should I understand they aren't using PKA?
Sorry then, shame on me.
Re:Summary without technobabble (Score:5, Informative)
More like don't upload your PRIVATE keys to a PUBLIC repository.
Re: (Score:3)
I entirely agree but for some, namely the ones who still use symmetric keys, this has become an old school thought.
In Canada, the government bought 30 millions certificates for all its citizens in oder to authenticate for government on line services for a buck a piece. Total: 30,000,000$
I would have been glad to provide it to them for 10,000$ and guess what? All privaye keys were kept centrally ;-) Us, old school guys just couldn't believe it.
The big thinkers/marketing guys decided that it was just to comp
Re: (Score:2)
In the end, education and instant knowledge is needed.
Re: (Score:3)
The big thinkers/marketing guys decided that it was just to complicated for citizens to manage and keep their secret key in a secure location
It is. Do not underestimate the ignorance of the common user, especially one who just wants to use their computer. Now if the government had charged $5 and sent a USB key with the certificate on it, maybe the end-user would take more care of it as they understand physical keys in a way that they don't with electronic versions.
Look at how many times you have to use the
Re: (Score:2)
I agree it is currently. It is funny although what a little education could do but most of the times, educated people are less easy to profit from. Therefore, marketing guys will rarely suggest educating people as a solution.
Re: (Score:2)
>Thirdly, "bought" 30 million certs?
Oh and yes, that's why we were both laughing our hearts out and calling shenanigan at the same time. As I wrote in my OP, I would have been glad to generate those certs for them for 10,000$ instead of the 30,000,000$ they spent. But hey, a buck a piece for certs is a great deal, isn't it?
The usb key solution was suggested as well but the conclusion was that dumb users would lose their usb keys and that it would become too costly to manage.
In the end, we seem to be doom
Re: (Score:2)
http://www.isacc.ca/isacc/_doc... [isacc.ca]
Re: (Score:1)
Quarantine or bust.
Re: (Score:2)
Re: (Score:1)
I remember a couple years ago someone accidentally pushed his .bash_history and it turned out to have a big log of him watching child porn movies. Pretty sure it was a Slashdot story but I really don't want to go searching for that.
Re: (Score:2)
.bash_history eavesdropping is indeed kidiporn...
Re: (Score:2)
ASCII porn in bash is evil man! At least he was not using rc.
Talk about being on the wrong side of .zsh_history (Score:2)
That was too funny not to search for; it obviously got taken down [github.com] from github, but someone posted the (very NSFW) .zsh_history [bit.ly] on a paste site when the story broke in 2012. If you search for lines starting with "mplayer", you can see how they're clustered into several obvious jerkoff sessions... ROTFL! The file names were so sick they made my eyes pop open.. I had to push them back into their sockets!
This wins the prize for the worst commit I've ever seen. There's a lesson for pedophiles here: echo ".zsh_hi
Re: (Score:2)
Run a WAF, log suspicious packets at the IP level, then you will discover a big underworld...
Lesson: don't use root AWS API keys (Score:5, Interesting)
AWS strongly discourages the uses of root API keys, as they give bad guys who find them the "keys to the kingdom". Why should the credentials for one's S3 account also work for creating EC2 instances?
Amazon provides extensive control over access credentials through IAM, so one can create (for example) an S3-specific user with limited privileges and generate API keys for that user. If they get compromised, the bad guy has limited access: they might be able to add new files to S3, which is bad, but it's less bad than them spinning up hundreds of servers for nefarious purposes, deleting all your files, etc.
Judicious user of IAM can also reduce user errors: I use Amazon Glacier for backing up certain critical files (e.g. wedding photos, baby photos, copies of wills, passports, etc.). I created an "upload, view, and restore/download" user for Glacier that explicitly does not have the "delete" permission enabled. I have a second IAM user with "view and delete" permissions. API keys for both users are stored in FastGlacier, with the "delete" user credentials stored encrypted so I need to enter a password to switch to that user. The user without delete permissions is the default user and the credentials are not stored with a password. This way I can do the standard backup/restore functions needed while working with backups but significantly reduce the possibility of my accidentally deleting backed-up files if I fat-finger the wrong key.
Re:Lesson: don't use root AWS API keys (Score:5, Insightful)
they could save themselves a lot of trouble by making a web UI for IAM that isn't shit, and actually documenting the system somewhere
Re:Lesson: don't use root AWS API keys (Score:4, Informative)
This may be anon but mark it up... very insightful. Yes it is important to create role based accounts/permissions for AWS through IAM, but abso-fucking-lutely the IAM UI and the process for modifying permissions/authorization is positively fucking shit. Garbage. Crap. A cankerous sore on the taint of the internet. It's not very good. And the documentation for it is so poor to the point of being virtually non-existent. I did use it, and on an AWS project I worked on made sure we had specific accounts created permissions limited depending on the roles required. But it is basically hacking at the configuration until you get it right because it is the only real way to figure it out. Not what you want when it involves security. It is too easy to fuck up and leave yourself vulnerable... never mind the people who take one look at it and say WTF? and not set up security at all and just use the root access key/secret key. Saying that this is something that only experts should use so it's OK to be cryptic is not the answer... if only security experts should set up accounts Amazon would likely lose 75% of their business (i.e. a lot of it).
--theshowmecanuck ... posting anon because not only did I want to mod parent up, but hopefully someone from amazon reads /. and will help get their shit together on this.
Re: (Score:2)
As someone that just started to use AWS where I work, I feel you are being far to complementary, a first year computer grad could come up with a better interface. The API's they provide aren't a whole lot better either.
Re: (Score:1)
Surprise! After using IAM for a few years now, it is as crappy and poorly thought at as the UI itself is. You'll find it doesn't do what you need to and there is no way to do a lot of important stuff properly because the means for limiting scope does not exist or so wide in scope as to be useless.
Amazon doesn't know what they are doing, there is not any single thing to fix here, AWS is broken top to bottom in every way and will be forever. Its that way by design.
Re: (Score:3, Insightful)
Amazon provides extensive control over access credentials through IAM, so one can create (for example) an S3-specific user with limited privileges and generate API keys for that user. If they get compromised, the bad guy has limited access: they might be able to add new files to S3, which is bad, but it's less bad than them spinning up hundreds of servers for nefarious purposes, deleting all your files, etc.
Agreed, but when you sign up as a new user, Amazon presents you with the root key right there. Someone signing up to test things out is likely to just copy and paste that key wherever it's needed to get off and running. I did that when I was playing with ceph, fortunately I was evaluating locally and didn't leak the key. I think a better option would be for Amazon to give you *no* API key by default, requiring you to make one with the necessary permissions and at least have a cursory look at the options.
Re: (Score:3)
And I recently found out about their IAM roles, which means that an EC2 instance can have it's own keys, that are automatically rotated, and available to any AWS-SDK you're running on the machine (or fetchable on a local IP). This is safer than passing keys, root or IAM user ones.
HACKERS ARE SCANNING 24/7 (Score:5, Funny)
HACKERS, I TELL YOU
Re: (Score:1)
Sounds like multiple failures (Score:5, Insightful)
Re:Sounds like multiple failures (Score:5, Insightful)
People make mistakes every day of their lives. We always have, we always will. It's how we learn what not to do. It's just that almost all mistakes are harmless... except on the internet. There it's like living in a minefield. Make a bad step and boom. It's not a question of amateur hour, it's a question of being human.
Re: (Score:1)
Just fucking use PKI instead of stupid API keys. Are their API keys PKI or not. See my post above:
http://slashdot.org/comments.p... [slashdot.org]
then, since 1977:
https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:2)
Easy enough to make this mistake, and not realising it.
Develop something that needs Amazon S3 access, and put it on GitHub. It's easy enough to forget about removing your keys before doing a git commit, putting them on GitHub.
Next time maybe you do remember to do so; possibly not realising your first mistake. The keys remain available in previous versions of your software, and you'll never see this old version until you happen to do a rollback to exactly that revision. Rollbacks don't happen too often; to t
Re: (Score:1)
By removing the key, I assume he rewrote the history, and not just removed it in a further commit.
Re: Sounds like multiple failures (Score:2)
Even if you rewrite history, the dangling commits are still in github and can be accessed - you have to clone the repo and delete the original.
Which isn't something you'd expect unless you understand how git works.
Re: (Score:1)
They'd need to know the SHA-1 to find it though.
Re: (Score:3)
No. It is a big fucking problem of Amazon that IAM S3 keys can be used outside S3. A BIG FUCKING problem! A major security incident.
They, Amazon, need to sort their shit out. If I have IAM keys that are S3 read only on a certain bucket, then I EXPECT that it is read only on that bucket. If somebody has those keys, then all I want them to do is to read from that bucket. Not start EC2 instances, or change my Route53 records, or anything else.
This is Amazon's fault. No two way about it.
If somebody got th
Re: (Score:2)
Okay, so you might not find it intuitive that S3 keys can lead to new EC2 instances - I wouldn't have guessed that either, even though I've used both repeatedly.
Actually, S3 keys either can or can't be used to launch EC2 instances, depending on user configuration. Using IAM, you can grant AWS keys only as many rights as are necessary, so if your program only needs to write to a certain S3 bucket, you should grant access to the key only to write to that bucket. Then, that key couldn't be used for any other purpose (like launching EC2 instances).
Also, IAM roles can be used to avoid using AWS keys in your code/config files at all if your code will be running within EC
Bots are not "algorithms" (Score:1, Interesting)
Anyway, I'm getting sick of hearing the word "algorithms" used as it seems to be in the movies a lot lately.
Re: (Score:2)
I once came up with an algorithm that wasn't very good, so I just commanded the computer to "Enhance!", and it got much better.
I guess i am old (Score:5, Interesting)
As a complete side note: I hate when people like the author don't know the difference between 'where' and 'were'....fuck, no wonder he was easy fodder
Re: I guess i am old (Score:2)
Re: (Score:2)
'I posted my credentials openly on the internet and am now shocked that I have been taken advatage of'... no way!
If you're going to ding someone on grammar I'd suggest you at least spell correctly while doing it.
Re: (Score:2)
Do you think maybe there is a fundamental difference between a missed keypress and having gross dyslexia?
Re: (Score:2)
Yep. Not checking your own spelling while attacking someones grammar on slashdot is stupid.
Dyslexia, on the other hand, does not appear to be correlated to intelligence.
Re:I guess i am old (Score:5, Insightful)
I guess i am too old to understand how loose people treat the internet these days. 'I posted my credentials openly on the internet and am now shocked that I have been taken advatage of'... no way! You shared the keys to your kingdom and someone abused it?? Shocking.
He wasn't surprised they were abused. He's surprised what he thought was keys to S3 unlocks the entire kingdom. Like you lose a card for electronic bus tickets and you think the worst they can do is ride the bus for free. Except the card doubles as a "secure ID" and they order thousands of dollars worth of goods from cooperating merchants, which you didn't even know was possible. Sure posting them on Github was ignorant and entirely his fault, but if you try to assess the risk then it's probability * impact and I think he just got a big surprise when it came to impact. I think you're being too hard on him.
Re:I guess i am old (Score:4, Informative)
No, he was surprised that what he *thought* were keys to S3 unlocked the whole kingdom. In reality, the keys he was using were root credentials, and were always intended to unlock the whole kingdom.
Re: (Score:2)
No, he was surprised that what he *thought* were keys to S3 unlocked the whole kingdom. In reality, the keys he was using were root credentials, and were always intended to unlock the whole kingdom.
This is entirely Amazons fault they should improve their crappy credential system. The is point that everyone makes mistakes so root keys are a really bad idea. Especially since you generate them when you are in a "just get this shit working" mode, not "lets design a super secure key management system" mode.
Re: (Score:2)
He's surprised what he thought was keys to S3 unlocks the entire kingdom.
Why? Is there even such a thing as an "S3 key"? I've been using AWS for a long time, and I've never seen one of these (unless you count the goofball time-limited key pre-signing thing, and those keys can't be used for any purpose outside of S3).
There are, of course, AWS keys, and those keys can be granted privileges. If you grant an AWS key full access, then yes, it can be used for any API call. But really that's bad practice. If your application needs only access to a specific S3 bucket, you'd create an AW
Re: (Score:2)
That's English.
How do you say the "were" part of "werewolf"? You say it just like "where". Or like "wear".
How do you pronounce "read", or "lead"? Is that "reed" or "read"? Is that "red" or "read"?
Some people can remember the rules, but not all of the exceptions. It's far worse if it's not even their native language and they didn't have this drilled in from early childhood. The fact that we write everything down with 26 letters is quite handy. Not so much the fact that there is no unique mapping of symbols t
Dumb (Score:1)
Dumbass.
searches 24 hours per day? (Score:1)
Their algorithm will crash on March 8, a 23-hour day.
Re: (Score:3)
With his money refunded he probably didn't file a police report. Which means Amazon doesn't have to deal with nosy investigators.
Re: (Score:2)
Another young idiot (Score:1)
Why the f*ck would you post your S3 keys on github anyway? I sincerely hope you are not looking for sympathy because you won't find any here (nor do you deserve any).
God-damn "social media" generation that feels they need to share everything. Good riddance.
Horrible design on part of Amazon (Score:5, Insightful)
When you sign up for a developer account, you should be asked how much you plan to spend per month. $2375/day would not be a common option for an individual. Given proliferation of free 15GB storage accounts, a very low end developer account with no credit card is not a crazy option. People will learn the API and use it in future, but neither them nor hackers will have enough quota to run a production site. This is just like limited data cell plans where a single buggy app can run up crazy charges. Good that they refunded money, but fundamental structural problem must be fixed.
people mess up. Bank calls before approving $10,00 (Score:2)
People mess up. An API key will occasionally get into a file that's in a source tree. It is an API - an Application Programming Interface, so programs will have that key. Program source gets posted to github. Shit happens.
My bank knows shit happens and I could get my card stolen, so when I make a large purchase which appears unusual my cell phone rings - the bank's security company calling to verify the purchase. That's the bank doing a good job.
People should watch where they're walking. That doesn't
Re: (Score:2)
He did exactly that and .gitignore was accidentally lost.
Service providers are responsible for 90% of work by designing solutions secure by default and in depth. It should take multiple explicit steps and dismissed warnings for an inexperienced developer to incur a catastrophic financial liability.
Re: (Score:3)
It reminds me of the stories of teenagers getting themselves a website and then hosting some video that went viral overnight... they wake up to a $16k bill.
Who is stupid enough to hardcode keys? (Score:2)
Really, leaving your money on a park-bench somewhere exposes it to be stolen...
Re: (Score:3)
I've lived all over the U.S.A. for decades, near plenty of stupid people, and I have never heard anyone use the phrase "on accident" in my life. That's the type of error you see in technical manuals from Asian countries. People do say "on purpose", which can easily confuse people who learn English as a second language into saying "on accident" instead of "by accident".
To spot Americans with two-digit IQs on the Internet, look for two unforgivable homonym goofs: confusion between "they're", "their", and "the
Re: (Score:2)
Let us see... are you:
You get the idea.
What makes you so sure the guy is from Israel?
Give the man some slack (Score:5, Insightful)
Surely it is not that unreasonable to (1) realize that those keys will be scraped within 5 minutes after uploading to an obscure project, and (2) not realize that an S3 key in a free trial subscription wouldn't allow racking up $2375 in EC charges within 10 hours?
Re: (Score:1)
In his defense this is exactly what you can expect from Amazon so this "trial" is *perfect* - AWS is awful, is full of traps, and does all kinds of extremely expensive unpredictable things. Then when it happens to you and it isn't a PR disaster for you they go "its your fault" - in fact everything is your fault, you can see it in the docs, where they will tell you stuff like "this doesn't work but if that is a problem its because you don't know how to do your job" where "this doesn't work" is a perfectly re
Give the man some slack (Score:2)
The mistake he made was not understanding the tools he was using. Apparently neither do you.
(1) The key could have been scraped at any time once it was pushed, because you can't actually "delete[d] all traces from GitHub" (some ways are more thorough than others, but nothing is foolproof with Google wandering the earth). He needed to revoke his keys immediately.
(2) There is no such thing as an "S3 key". There are only AWS API keys, which potentially have access to every service that you have enabled (plu
Re: (Score:3)
I assume that you also blame the subprime borrowers for signing a contract that they didn't fully understand without putting most of the blame on the banks that knew damn well what they were doing?
The fact that one person can be blamed for a mistake due to lack of experience does not mean that there is not someone else (i.e., Amazon and the people
Re: (Score:1)
The mistake he made was not understanding the tools he was using. Apparently neither do you.
(1) The key could have been scraped at any time once it was pushed, because you can't actually "delete[d] all traces from GitHub" (some ways are more thorough than others, but nothing is foolproof with Google wandering the earth). He needed to revoke his keys immediately.
You do realize that you actually can modify a git repository to delete all traces of a file you previously comitted, don't you? If not, check out http://git-scm.com/book/en/v2/Git-Tools-Rewriting-History
Per usual, you had to repost a horrible story. (Score:1)
This type of problem has been reported many times before, with much more knowledgable writeups.
The original poster is so naive that they didn't even bother to read enough Amazon documentation to realize there is no such thing as an "S3 key" - API access by AWS keys is limited only by the IAM profile of the key (and my guess is that the OPs keys were unrestricted). They also apparently didn't realize how version control systems work, otherwise they would have known that deleting the key from a revision does
5 min? Actually those keys are STILL on GitHub (Score:2)
In addition to the various other oversights already mentioned, OP doesn't seem to understand Git (or perhaps SCMs in general) given that those (now revoked) keys are still [github.com] on [github.com] GitHub -- there was no need for a bot to be all that quick.
Although I wouldn't blame OP for any single one of these oversights -- nobody's perfect -- it's fair to say that it took a number of different oversights / misunderstandings on OP's part for this to become a real problem.
Keys in plain sight (Score:2)
Re: (Score:2)
You almost spelled Rivest right:
https://en.wikipedia.org/wiki/... [wikipedia.org]
Keep on the good work!