Predicting Malicious Web Attacks 82
KentuckyFC writes "Recommendation systems attempt to guess what books, movies, or news people are likely to be interested in. Companies such as Amazon, Google, and Netflix have developed algorithms to mine vast databases looking for correlations that they then use to recommend new items. Now a team of computer scientists has used some of the same filtering techniques to predict the origin of malicious Web attacks so that they can be blacklisted in advance. The team mined a database of hundreds of millions of security logs looking for correlations between victims. The correlations were then used to produce a predictive blacklist of potential attackers. The team says its algorithm is up to 70 per cent more successful at predicting the origin of attacks than current state-of-the-art predictive blacklisting."
What we really need. (Score:1, Offtopic)
Hiro Protagonist. And his sword. And his undefeatedness-nous.
Re:What we really need. (Score:-1, Flamebait)
No doubt useful (Score:4, Insightful)
But this is still treating the symptom as opposed to the core problem, which is poor security in OS and app design.
Microsoft is starting to come around on this to an extent (not running as administrator), but shouldn't we be more concerned about true security?
Re:No doubt useful (Score:4, Insightful)
Re:No doubt useful (Score:2)
Perhaps I should included my title in my post. No doubt this useful given the current situation. But we wouldn't be in this position so much if we had well designed systems in place from day 1.
I do think this is interesting how we can use massive data sets to predict and map trends so much quicker. But I'd rather not have to worry about them in the first place.
Re:No doubt useful (Score:3, Insightful)
Re:No doubt useful (Score:2)
Unix was designed from day 1 with the notion that it is a multi-user system that needs serious integrated security. Windows was designed for a home PC with a single user. It wasn't designed with the notion that it would be on the internet, or need much in the way of security.
It isn't to say that we couldn't have forseen security concerns to design it correctly in the first place. Most *nix systems were always designed this way. Windows opted not to follow that model.
Choosing Security vs. Dancing Pigs vs. Unix (Score:2)
Bruce Schneier [schneierfacts.com] says that give a choice between security and dancing pigs on your computer, people will take the dancing pigs every time.
When Windows came out, it was perfectly secure - there's only one user in the universe, and she's allowed to do whatever she wants. ("Format C: "? Sure!).
Unfortunately, while Unix was designed from the beginning for security, it didn't always _stay_ designed for security, and some of the things that were done for security had serious tradeoffs. Networking was usually the worst, certainly from TCP/IP's beginnings in 4.2BSD, but also other protocols and other applications had problems, and you're not secure unless everything's secured in some way.
(Back when I was a newbie learning security, RTM's father used at least the last three of those methods to crack into my accounts :-)
Re:Choosing Security vs. Dancing Pigs vs. Unix (Score:2)
In truth, all you need to do is read the Art of War, and you'll know that implementing proper Windows permissions couldn't possibly the the answer to security. You'll also realize that collaborative filtering couldn't possibly the the answer to security either.
The only answer is to be one step ahead of the attackers, and to think up what they're going to throw at you next.
(That's not to say that proper Windows permissions don't help, and that collaborative filtering doesn't help, but security is war, and the white hats need to keep trying to win. Just because you have a certain security measure doesn't mean you're secure.)
Re:No doubt useful (Score:5, Insightful)
but shouldn't we be more concerned about true security?
What is "true security" against the main threat of the modern era: social engineering? How does your operating system protect you from from responding to that e-mail you've just received from your long lost uncle in Nigeria? How do you protect a user that will click on the user account control pop-up as many times as is required to install that cool "weather forecasting" program that sits in his task tray?
Or were you referring to "true security" in the context of firearms, expendable redshirts and moats filled with laser wielding sharks? ;)
Re:No doubt useful (Score:2)
Don't underestimate sharks with friggen laser beams!
I agree that Social Engineering is likely the number one threat in many cases.
UAC is security theater in that people are trained to simply click allow, absolving Microsoft of responsibility.
What I mean by true security is sandboxing and accountability. Look at Chrome's design, in that a browser window (process) has limited access to data on your HDD.
Users in an enterprise environment frankly shouldn't have access to install software at all. And the more I think about it, I wonder if not only thin-client remote terminals are the way to go for the future, but temporary kiosk sessions as well.
Lastly, a really good file system from a security standpoint should not only have an access time, but log the user who accessed it at that time.
Education is the best weapon to combat social engineering (and it isn't that hard to tell people NEVER give out your password), but a well designed system certainly helps.
Re:No doubt useful (Score:2)
Users in an enterprise environment frankly shouldn't have access to install software at all.
Which leads us to the true security question/issue. The only truly secure system is one users don't have access to. In any other environment, where people are trying to get work done. a completely locked down environment can impede the business. The end goal, whether the security types like it or not, isn't a secure environment. It's to make money or reach some other objective. Security is relevant in that it supports your progress towards that objective. The economic reality is that there is tension between complete security, which keeps you from losing money, and productivity, which is how you make money.
In my company's environment, we have a pretty good focus on security, and things are generally pretty locked down. But we have classes of users that benefit from less locked down environments, because the IT guys don't know how to install something from source, for example, and can't be bothered (or, more charitably, are kept too busy) to step out of the MCSE box to learn. Fortunately, we have been able to work things out so that some of us enjoy a bit more freedom than others.
Comment removed (Score:3, Interesting)
Re:No doubt useful (Score:2)
As long as I can get the user to visit my site and load up my malware I can spew spam, I can DDOS, etc.
Not true. If neither the user nor app have admin/root access, and you're using a secure browser (say, Chrome) then your malicious web site can't do squat. The biggest hole here right now would be that plugins aren't fully sandboxed, and Acrobat has a serious vulnerability every other week. But that is partially why I keep recommending to businesses to use Foxit as opposed to Acrobat.
Comment removed (Score:2)
Re:No doubt useful (Score:2)
Every browser uses javascript and plugins. Saying that Chrome isn't secure because of these things is silly.
Chrome places each process in a jail and prohibits access to the HDD to make changes to your system.
Will there be Acrobat exploits that can also be accessed via Chome? Yes, until Chrome figures out how to fully sandbox plugins, but Google said they are working with plugin vendors to make them play nice within Chrome's security concepts. Chrome is still more secure than IE and Firefox, not just because it is new, but because of how it is designed.
Webkit itself has been around in usage for years. But Google's use of Webkit is more secure than Safari, Konqueror, Arora, etc. because of its security model.
Re:No doubt useful (Score:2)
sers in an enterprise environment frankly shouldn't have access to install software at all.
Unfortunately it's rarely that simple. I've worked in two "enterprise" environments in my IT career. One (my current job) makes this fairly easy to implement -- most of our operations run around web based database apps and Office. Very easy to lock users into restricted accounts.
The other enterprise I worked for was an insurance agency. The insurance industry has so much legacy software that restricting users to non-admin accounts is not possible unless you are willing to sacrifice needed functionality. Many of these legacy apps come directly from the insurance companies that you do business with and there is no alternative. You either use them or you don't write business with that particular company.
I eventually had to settle for imaging our workstations and restoring them from the image whenever the user managed to fuck them up. Not the ideal solution but it was the best I could do in that situation.
Re:No doubt useful (Score:2)
Both of the two enterprise environments I've worked in have used proprietary legacy apps that "need" admin rights.
Most of the time, all the app really needs is write access to a certain folder. However, in the rare instance that the process truly does need administrator access, I make the app/process into a Windows service that starts automatically at login with System level access. The user doesn't have admin access, and other apps don't. That one app is elevated.
Re:No doubt useful (Score:3, Interesting)
What is "true security" against the main threat of the modern era: social engineering?
Social engineering will always be a problem but there is a simple fix. Restrict the user on damage they can do on their own given the worse case circumstances and you will also end up with the same prevention of malware in the process.
Speaking of which... Why does a web page ever need to communicate with the OS to make file changes to the OS? Why?! Why I ask?!
This is a flawed premise and will solve 99% of the problems we face with internet security.
The OS must sandbox the browser and its add-ons between it and the OS.
In fact... Why stop there... The OS must be sandboxes between it and the user.
Basically true security is basically given the users and OS like the iPhone and patting them on the back and say "have a nice day".
"But I want to use my legacy apps?" they say...
"Well I want a pony!" you reply "But you'll just have to deal with a limited OS because we can't have nice things because they keep installing viruses on their machine!"
How do you protect a user that will click on the user account control pop-up as many times as is required to install that cool "weather forecasting" program that sits in his task tray?
Require the "weather forecasting" app to submit an approval to a central repository like the iPhone.
See where I am leading you...
Seriously... In the future the average user will put up with an OS like the iPhone and they'll be happy because it just works or appears to and the admins of the world will be happy because people aren't screwing things up with bot nets.
Win7 and IE8 might be a big step in that direction but we'll have to see.
Re:No doubt useful (Score:2)
Re:No doubt useful (Score:2)
Re:No doubt useful (Score:2)
I think there is a company willing to do that. They're in Redmond, Washington.
Re:No doubt useful (Score:2)
Re:No doubt useful (Score:2)
" "Well I want a pony!" you reply "But you'll just have to deal with a limited OS because we can't have nice things because they keep installing viruses on their machine!" "
Thank you, you've just made Jonathan Zittrain's point exactly [futureoftheinternet.org].
Except he thinks this is a bad development and can still be changed.
Re:No doubt useful (Score:1)
Require the "weather forecasting" app to submit an approval to a central repository like the iPhone.
See where I am leading you...
Yes, and it is a bad idea. Secure the OS by securing the OS, not by adding in a random trusted third-party that will probably make mistakes anyway (maybe we should call that "security by authority"?). Sandbox applications so they only have access to the files and services they need, perhaps with permissions like "safe" network access which is capped or can only access one server or port or has to display the bandwidth used on screen and be advertized as a possibly dangerous high-network usage application (ex. for a p2p app). Google's Android has a per-application permissions system where users are told which permissions an application is requesting on install. App Armor allows for simple sandboxing on Linux. IE8's sandbox is a definite good step in the right direction.
With App Armor there have been suggestions of "generic profiles" like web browser, game, p2p program, etc. which would have less strict limitations than a program-specific profile but still limit what the application can do while presenting the limitations in a way the user can understand.
One way to handle anything like network access limitations I mentioned above might be to create a separate virtual network adapter for every application.
Let legacy applications live in virtualized environments if necessary. There is no reason to not let an application run just because it is old, although paying some amount of emulation penalty is reasonable and unlikely to be an issue.
Re:No doubt useful (Score:0)
The lasers are mounted to the sharks, technically, the sharks are wielding them.
We leave that to the bad tempered sea bass.
What the military means by "Secure a computer" (Score:2)
I used to work with an ex-Navy guy - our lab became much neater once he joined us, and more secure as well. But different organizations have much different concepts of what it means to "secure a computer" -
A testament to my everlasting love. (Score:-1, Troll)
Eliza Dorbenbrod I want to fuck you like a blunderbuss.
Re:No doubt useful (Score:4, Insightful)
"True security" is a fantasy. No such thing exists, nor will it ever.
We should be concerned with balancing risk reduction with its cost. We should not be concerned with your silly fantasy.
Re:No doubt useful (Score:2)
Designing the OS to be secure as opposed to chasing people attacking vulnerabilities left by design in the OS is silly?
Re:No doubt useful (Score:0)
"Designing the OS to be secure" is the problem here, how do you define a "secure OS?"
imagine, for example, if every door in your house had a lock and key (bathroom, bedroom, etc.) that you should lock when you leave the room, and unlock before you can enter.
most people, after a certain amount of time will just say "to hell with it" and leave all the doors open. the functionality remains in place for those who wish to always lock and unlock the doors. this would be a secure OS. but enforcing everyone, all the time, to lock/unlock all the doors, is essentially what you are saying and this, as GP as indicated, is indeed a fantasy.
Re:No doubt useful (Score:0)
Speak for yourself. People call you insightful--but I see no reason I should have to accept most of society's ...incredibly high risk tolerance when it comes to computing.
I've got a browser. I know it's a problem. I can't fix it. If I want to surf the web I need cookies, javascript, flash. They're all gaping huge problems. If you claim they aren't, you unconditionally do not know what you're talking about. You may claim the benefits outweigh the risk--but at that point, the claim is synonymous with "I volunteer to permit total strangers full access to my system in order to the surf the web"--and there's nothing wrong with people rejecting that conclusion and taking measure to enforce it.
So my browser instead runs in a vmware image--and when that image shuts down *everything* on the VM goes back to the way it was before. It isn't perfectly safe (there's non-theoretical VM escapes), but it's good enough.
You can talk about "balance" all you want--but most people will trade *anything* for cost savings. I'd rather own my machine--than let some guy in china use it--thanks.
Re:No doubt useful (Score:2)
I truly agree, bad OS design, some lack of security based on too much money it would cost, and not enough people really taking security seriously, there will always be that one person with a password equal to a dogs name or their birthday!!!
Re:No doubt useful (Score:0)
Yeah and did you notice they said 70% more effective than the leading techniques. I wonder how effective that actually is. Without any knowledge of the existing algorithms it could be complete garbage.
Minority Report (Score:2, Interesting)
What about false positives? Can they be held responsible for blacklisting an innocent site?
Re:Minority Report (Score:2)
You can't be held responsible for blacklisting sites right now, what makes this any different from any other Blacklist?
If you want to get traffic to/from the site then Whitelist it.
Now they say their approach is 70% better than existing Predictive Blacklist technology, well how good is that, 70% better than horrible false positives and annoyed customers is not enough. Throwing darts at the DNS listings is also not optimal, so how good is this new technique.
BTW Amazon and Netflix recommend crap to me I don't want all the time.
Re:Minority Report (Score:1)
Most people don't understand the internet. I'm not sure how this blacklisting thing would be deployed, but your average person is just going to accept it and move on. Now, what if it blacklists something like Amazon or eBay? Would Amazon be able to sue someone over lost revenue because all the Joe I. Pod's out there stopped visiting their site all of a sudden? Especially if it was just a false positive.
As an afterthought, is there really a distinction between a false positive and a deterred attack? (Think Minority Report again, if you know your future you can avoid it.) Say you predict an attack and it doesn't happen, did you foil the attempt - the guys behind it gave up because everyone saw it coming - or was it just a false positive?
Re:Minority Report (Score:1)
Hello, I am Anh Le, the second author of the work.
First, investigating the false positive is not the main focus of our work. We did our analysis on the log entries generated by the intrusion detection systems (IDS) deployed at various sites. Granted that there are false positives in the dataset, these false positives, however, are from the IDSs because of, for example, bad signatures and configuration errors. This is itself an area of active research.
Furthermore, the entries included in the blacklist appeared at least one time in the past. In other word, they are flagged as attackers at least once. Hence, they are not really innocent although, again, it's very possible that some of them are false positives.
Link to our paper: http://arxiv.org/abs/0908.2007 [arxiv.org]
Finally a use for this technology (Score:3, Funny)
There's finally a use for this collaborative filtering technology.
Re:Finally a use for this technology (Score:2)
Well, according to TFA it's not quite ready for prime time.
Re:Finally a use for this technology (Score:1)
Re:Finally a use for this technology (Score:2)
If it doesn't, they won't have much of a website left after today.
Let me get this straight... (Score:0)
I have a facebook account.
Facebook gets cracked.
Consequently, Best Buy blocks my access to their ecommerce website because, as a facebook user, I'm potentially a cracker?
I'm so confused.
predictive blacklist of potential attackers... (Score:0)
Referer: slashdot.org
Oops. There goes another server. (No, TFA isn't slashdotted. Yet.)
Re:predictive blacklist of potential attackers... (Score:2)
You may joke about it, but I wrote a "slashdotted" snort rule for a web development and hosting company.
Did I read this right... (Score:5, Interesting)
recommendation systems may soon be providing you not only with books and movie tips but a happier surfing experience too
I am a little weary of making my surfing experience happier by allowing the system to do my thinking for me. Just think, "clippy" for the browser.
Re:Did I read this right... (Score:1)
weary (tired of) or wary (nervous about)?
Re:Did I read this right... (Score:1)
Re:Did I read this right... (Score:2)
Re:Did I read this right... (Score:0)
>"clippy" for the browser
*Jumps out of Windows into Mac-random feline*
Re:Did I read this right... (Score:2)
sarcasm on
Yeah, and I'm pissed that I can't get Clippy working on Ubuntu!! The little dog in the search box too!! What, when you start to go geeky you can't have pets anymore? It's just WRONG, I tell you!! I'm going to send some hate mail to Canonical, and find out what the deal is. This just pisses me off!!
sarcasm off
Seriously - all those user agents and stuff should have been a tipoff. A corporation that offers cartoonish characters as part of a "serious" operating system can't be trusted with security.
Re:Did I read this right... (Score:2)
recommendation systems may soon be providing you not only with books and movie tips but a happier surfing experience too
I am a little weary of making my surfing experience happier by allowing the system to do my thinking for me. Just think, "clippy" for the browser.
The article doesn't seem to say how it will be implemented, but I would assume it would be some server-side app that generates firewall (and possibly spam filter) rules.
Hoory Preemptive Blacklisting! (Score:1)
Umm... (Score:2, Funny)
Re:Umm... (Score:2)
That, or go Skynet. The ideal way to stop all web attacks would be to bring down the internet itself. I so hope these guys did their homework [xkcd.com].
the new 404 (Score:3, Funny)
"People..." (Score:5, Funny)
"People who attacked this site ALSO attacked..."
Re:"People..." (Score:3, Funny)
"Was this review helpful? Yes or no"
Re:"People..." (Score:1)
Amazon should patent "1-click attacking"
Re:"People..." (Score:3, Informative)
Amazon should patent "1-click attacking"
Ptech [google.com] already has it patented!
Re:"People..." (Score:2)
We used to get targetted many, many moons ago by people searching Google for "phpBB version x.y.z". If you want to predict web attacks, Google says:
Results 1 - 20 of about 80,600,000 for "phpbb version x.y.z"
Hrmm (Score:0)
Companies such as Amazon, Google, and Netflix have developed algorithms to mine vast databases looking for correlations that they then use to recommend new items.
I swear, if I see a "coorelationisnotcausation" tag by you slashbags, I'm turning in my AC card.
Yes, it's an insightful comment when used properly.
Please do not use it here, just because you saw the word correlation.
Re:Hrmm (Score:2)
a "coorelationisnotcausation" tag
Thanks, I knew that there was a perfect tag for this story! Marking it as such allows two benefits which I can easily define:
The Article is obviously a fake (Score:4, Insightful)
Or greatly exaggerated...
"The team mined a database of hundreds of millions of security logs"
Nobody actually keeps security logs, certainly not hundreds of millions of somebodies.
The kind of people that DO keep security logs probably wouldn't hand them over either.
I call shenanigans
Re:The Article is obviously a fake (Score:2)
Obviously they have developed hacking technology to break open all these systems to get at their logs to determain if they have been hacked. Well they will be blacklisting themselves later this afternoon.
Re:The Article is obviously a fake (Score:3, Insightful)
Fixed:
Now it makes more sense, and is quite believable, no?
Re:The Article is obviously a fake (Score:1)
Stoopid (Score:0)
Heard the one about the hordes of people who will deliberately attempt to get public computers and corporate networks blacklisted?
Neither have these morons!
Next week, how to prevent network attacks by DOS'ing yourself.
False positives (Score:2)
False positives, here we come...
Meatware needed (Score:5, Insightful)
This sounds great, but only if it requires human intervention to implement the block. I used to work in a NOC, and we would have loved to throw up a warning on the big screens that an attack is 80% likely from the following netblocks in the next N hours. That way we would have a strategy developed for defending before it even started and would be able to minimize downtime.
On the other hand, if you make this automatic you're going to piss off a lot of people very quickly because it's going to be wrong more often than you want.
Re:Meatware needed (Score:2, Insightful)
Exactly. Because even if it's true, and it's 70% more accurate... I've yet to see a predictive system that's even remotely accurate. It may predict say... 50% of the sources of an ongoing attack (assuming a collaborative effort to determine when attacks are happening, and that you're not the first one hit), but that's far from enough to prevent a DDoS attack. And if you "accidentally" block... Say Canada (which I've seen before), then that's a LOT of costumers you just pissed off, but hey... Doesn't matter, that DDoS attack would have blocked access anyway, so how would they notice ;)
Re:Meatware needed (Score:0)
And if you "accidentally" block... Say Canada (which I've seen before), then that's a LOT of costumers you just pissed off...
I can attest to this. The number of repertory companies and historical reenactment groups in Toronto alone is staggering.
False positives (Score:2)
Re:False positives (Score:1)
Hello, I am Anh Le, the second author of the work.
I responded to the concern about false positives in one of the replies above. In brief, investigating the false positives is not the main focus of our work, and it is an area of active research in the intrusion detection system community.
Link to our paper: http://arxiv.org/abs/0908.2007 [arxiv.org] [arxiv.org]
ummmm (Score:-1, Offtopic)
...Nerds
The Minority Report (Score:1)
Okay if your going to do this anyway, here let me gaze into my crystal ball. Blacklist China, North Korea, and major parts of Russia.
save your money (Score:0)
Predictorator (Score:2)
Calculate the annoyance factor
If site is shitty, + .1
If site has a "clever" name, such as bit.ly, +.1
If site's name has become widely used as a verb or other part of speech, +.1
+ unique users in the last 24 hours / 100,000,000
Calculate the monetary factor
If site sells something, +.05
If site makes revenue through ads, +.05
If site is partnered or associated with a megacorp like a bank, ms/google, etc., +.1
+ dollars lost per minute of downtime (based on the last 24 hours) / 1,000,000
Calculate the brought it upon themselves factor
If site pissed off 4chan, +.2 * number offenses / time (in weeks) since last offense
If site pissed off other nerds, +.1 / time (in weeks) since last offense
Annoyance factor + Monetary factor + Brought it upon themselves factor = attack risk.
If attack risk >= 1, attack is imminent.
Otherwise, the attack risk is the probability of an attack occurring within the next 24 hours.
Never assume anything (Score:1)
Sounds vageule familiar... (Score:2)
They *guess* that you may be guilty before it happens and blacklist you.
Does it also predict its own false positives? (Score:2)
Great, so it can "predict" IP or site origins of malicious attacks, but can it also predict its own inevitable false positives? If so, how is it better than a DNSBL or other blacklist, except that it can make money for its owners without requiring constant updating and the requisite human labor?
I'd hate to use an IP or own a site that it happened to incorrectly "predict" as the source of an impending-but-as-yet-not-real attack. They might as well compile a Minority Report against me. How would that be any better for me, as an innocent victim, than having my IP/site unfairly blacklisted by SORBS/Spamhaus/Spamcop?
sidreporter? (Score:2)