Forgot your password?
typodupeerror
Security Networking

Predicting Malicious Web Attacks 82

Posted by kdawson
from the you-will-begin-ddosing-me-in-three-two-one dept.
KentuckyFC writes "Recommendation systems attempt to guess what books, movies, or news people are likely to be interested in. Companies such as Amazon, Google, and Netflix have developed algorithms to mine vast databases looking for correlations that they then use to recommend new items. Now a team of computer scientists has used some of the same filtering techniques to predict the origin of malicious Web attacks so that they can be blacklisted in advance. The team mined a database of hundreds of millions of security logs looking for correlations between victims. The correlations were then used to produce a predictive blacklist of potential attackers. The team says its algorithm is up to 70 per cent more successful at predicting the origin of attacks than current state-of-the-art predictive blacklisting."
This discussion has been archived. No new comments can be posted.

Predicting Malicious Web Attacks

Comments Filter:
  • Hiro Protagonist. And his sword. And his undefeatedness-nous.

  • No doubt useful (Score:4, Insightful)

    by Enderandrew (866215) <enderandrew@gm[ ].com ['ail' in gap]> on Tuesday August 18, 2009 @03:16PM (#29110727) Homepage Journal

    But this is still treating the symptom as opposed to the core problem, which is poor security in OS and app design.

    Microsoft is starting to come around on this to an extent (not running as administrator), but shouldn't we be more concerned about true security?

    • Re:No doubt useful (Score:4, Insightful)

      by dyingtolive (1393037) <brad...arnett@@@notforhire...org> on Tuesday August 18, 2009 @03:19PM (#29110797)
      Why do both have to be mutually exclusive? Why can't the problem be approached from both sides by different groups whose skillsets are appropriate for what they're doing?
      • Perhaps I should included my title in my post. No doubt this useful given the current situation. But we wouldn't be in this position so much if we had well designed systems in place from day 1.

        I do think this is interesting how we can use massive data sets to predict and map trends so much quicker. But I'd rather not have to worry about them in the first place.

        • Re: (Score:3, Insightful)

          by dave562 (969951)
          I think the underlying issue has come from the fact that people have been more focused on making computers do what they want them to do, and not focused on making them do it securely. It's great to sit on the sidelines and talk about how it should have been done better/smarter/more securely in the first place. That perspective does not take into account the reality that computers are relatively new and new functionality comes out almost every day. To consider another aspect of security, we've been living
          • Unix was designed from day 1 with the notion that it is a multi-user system that needs serious integrated security. Windows was designed for a home PC with a single user. It wasn't designed with the notion that it would be on the internet, or need much in the way of security.

            It isn't to say that we couldn't have forseen security concerns to design it correctly in the first place. Most *nix systems were always designed this way. Windows opted not to follow that model.

            • Bruce Schneier [schneierfacts.com] says that give a choice between security and dancing pigs on your computer, people will take the dancing pigs every time.

              When Windows came out, it was perfectly secure - there's only one user in the universe, and she's allowed to do whatever she wants. ("Format C: "? Sure!).

              Unfortunately, while Unix was designed from the beginning for security, it didn't always _stay_ designed for security, and some of the things that were done for security had serious tradeoffs. Networking was usually th

              • by kabloom (755503)

                In truth, all you need to do is read the Art of War, and you'll know that implementing proper Windows permissions couldn't possibly the the answer to security. You'll also realize that collaborative filtering couldn't possibly the the answer to security either.

                The only answer is to be one step ahead of the attackers, and to think up what they're going to throw at you next.

                (That's not to say that proper Windows permissions don't help, and that collaborative filtering doesn't help, but security is war, and th

    • Re:No doubt useful (Score:5, Insightful)

      by Shakrai (717556) on Tuesday August 18, 2009 @03:23PM (#29110855) Journal

      but shouldn't we be more concerned about true security?

      What is "true security" against the main threat of the modern era: social engineering? How does your operating system protect you from from responding to that e-mail you've just received from your long lost uncle in Nigeria? How do you protect a user that will click on the user account control pop-up as many times as is required to install that cool "weather forecasting" program that sits in his task tray?

      Or were you referring to "true security" in the context of firearms, expendable redshirts and moats filled with laser wielding sharks? ;)

      • Don't underestimate sharks with friggen laser beams!

        I agree that Social Engineering is likely the number one threat in many cases.

        UAC is security theater in that people are trained to simply click allow, absolving Microsoft of responsibility.

        What I mean by true security is sandboxing and accountability. Look at Chrome's design, in that a browser window (process) has limited access to data on your HDD.

        Users in an enterprise environment frankly shouldn't have access to install software at all. And the more I

        • Users in an enterprise environment frankly shouldn't have access to install software at all.

          Which leads us to the true security question/issue. The only truly secure system is one users don't have access to. In any other environment, where people are trying to get work done. a completely locked down environment can impede the business. The end goal, whether the security types like it or not, isn't a secure environment. It's to make money or reach some other objective. Security is relevant in that it sup

        • Re: (Score:3, Interesting)

          by hairyfeet (841228)

          And if you just take the PCs away from the silly users and lock them away in safes they'll be 100% secure! Seriously sandboxing is a bandaid on a bullet wound, and is as much bullshit as "as long as they can't get root its okay". Well, no its not. If I have control over your network connection why would I give a shit if it is sandboxed or not? As long as I can get the user to visit my site and load up my malware I can spew spam, I can DDOS, etc. Just like if you get a hold of the local user account you can

          • As long as I can get the user to visit my site and load up my malware I can spew spam, I can DDOS, etc.

            Not true. If neither the user nor app have admin/root access, and you're using a secure browser (say, Chrome) then your malicious web site can't do squat. The biggest hole here right now would be that plugins aren't fully sandboxed, and Acrobat has a serious vulnerability every other week. But that is partially why I keep recommending to businesses to use Foxit as opposed to Acrobat.

            • by hairyfeet (841228)

              Uhhh... I take it you haven't read this [zdnet.com] yet, have you? The reason Chrome has been a "secure browser" so far is the SAME reason I would argue that Linux hasn't been seeing viruses: It just ain't a big enough target yet to waste serious brain power on. But as it grows more popular (I can remember a time when FF never had hardly any problems either) the amount of exploits WILL go up, because as I said (and you yourself pointed out with your post) sandboxes are bandaids on bullet wounds.

              With so many legit reaso

              • Every browser uses javascript and plugins. Saying that Chrome isn't secure because of these things is silly.

                Chrome places each process in a jail and prohibits access to the HDD to make changes to your system.

                Will there be Acrobat exploits that can also be accessed via Chome? Yes, until Chrome figures out how to fully sandbox plugins, but Google said they are working with plugin vendors to make them play nice within Chrome's security concepts. Chrome is still more secure than IE and Firefox, not just because

        • by Shakrai (717556)

          sers in an enterprise environment frankly shouldn't have access to install software at all.

          Unfortunately it's rarely that simple. I've worked in two "enterprise" environments in my IT career. One (my current job) makes this fairly easy to implement -- most of our operations run around web based database apps and Office. Very easy to lock users into restricted accounts.

          The other enterprise I worked for was an insurance agency. The insurance industry has so much legacy software that restricting users to non-admin accounts is not possible unless you are willing to sacrifice needed functionality

          • Both of the two enterprise environments I've worked in have used proprietary legacy apps that "need" admin rights.

            Most of the time, all the app really needs is write access to a certain folder. However, in the rare instance that the process truly does need administrator access, I make the app/process into a Windows service that starts automatically at login with System level access. The user doesn't have admin access, and other apps don't. That one app is elevated.

      • Re: (Score:3, Interesting)

        by vertinox (846076)

        What is "true security" against the main threat of the modern era: social engineering?

        Social engineering will always be a problem but there is a simple fix. Restrict the user on damage they can do on their own given the worse case circumstances and you will also end up with the same prevention of malware in the process.

        Speaking of which... Why does a web page ever need to communicate with the OS to make file changes to the OS? Why?! Why I ask?!

        This is a flawed premise and will solve 99% of the problems we f

        • Yes, because we all want one company controlling what apps we can install.
          • by dave562 (969951)
            Some of us would be happy if one company would give us a central repository that we could manage for our own networks. Software whitelisting isn't exactly a new concept.
        • by lennier (44736)

          " "Well I want a pony!" you reply "But you'll just have to deal with a limited OS because we can't have nice things because they keep installing viruses on their machine!" "

          Thank you, you've just made Jonathan Zittrain's point exactly [futureoftheinternet.org].

          Except he thinks this is a bad development and can still be changed.

        • by AnyoneEB (574727)

          Require the "weather forecasting" app to submit an approval to a central repository like the iPhone.

          See where I am leading you...

          Yes, and it is a bad idea. Secure the OS by securing the OS, not by adding in a random trusted third-party that will probably make mistakes anyway (maybe we should call that "security by authority"?). Sandbox applications so they only have access to the files and services they need, perhaps with permissions like "safe" network access which is capped or can only access one server or port or has to display the bandwidth used on screen and be advertized as a possibly dangerous high-network usage application (e

      • I used to work with an ex-Navy guy - our lab became much neater once he joined us, and more secure as well. But different organizations have much different concepts of what it means to "secure a computer" -

        • The Army sends out computer technicians to look at log files.
        • The Navy ties the computer down with ropes and netting to keep it from bouncing around in rough seas, and does whatever it takes to keep the computer room water tight.
        • The Marines send a squad of guys with automatic weapons to make sure no
    • Re:No doubt useful (Score:4, Insightful)

      by Lord Ender (156273) on Tuesday August 18, 2009 @03:59PM (#29111387) Homepage

      "True security" is a fantasy. No such thing exists, nor will it ever.

      We should be concerned with balancing risk reduction with its cost. We should not be concerned with your silly fantasy.

      • Designing the OS to be secure as opposed to chasing people attacking vulnerabilities left by design in the OS is silly?

    • I truly agree, bad OS design, some lack of security based on too much money it would cost, and not enough people really taking security seriously, there will always be that one person with a password equal to a dogs name or their birthday!!!

  • Minority Report (Score:2, Interesting)

    Wonderful. It's Minority Report for the internet.
    What about false positives? Can they be held responsible for blacklisting an innocent site?
    • You can't be held responsible for blacklisting sites right now, what makes this any different from any other Blacklist?

      If you want to get traffic to/from the site then Whitelist it.

      Now they say their approach is 70% better than existing Predictive Blacklist technology, well how good is that, 70% better than horrible false positives and annoyed customers is not enough. Throwing darts at the DNS listings is also not optimal, so how good is this new technique.

      BTW Amazon and Netflix recommend crap to me I don't

      • Well, think about this scenario.
        Most people don't understand the internet. I'm not sure how this blacklisting thing would be deployed, but your average person is just going to accept it and move on. Now, what if it blacklists something like Amazon or eBay? Would Amazon be able to sue someone over lost revenue because all the Joe I. Pod's out there stopped visiting their site all of a sudden? Especially if it was just a false positive.
        As an afterthought, is there really a distinction between a false positi
    • by anhml (1621039)

      Hello, I am Anh Le, the second author of the work.

      First, investigating the false positive is not the main focus of our work. We did our analysis on the log entries generated by the intrusion detection systems (IDS) deployed at various sites. Granted that there are false positives in the dataset, these false positives, however, are from the IDSs because of, for example, bad signatures and configuration errors. This is itself an area of active research.

      Furthermore, the entries included in the blacklist appear

  • by kabloom (755503) on Tuesday August 18, 2009 @03:19PM (#29110795) Homepage

    There's finally a use for this collaborative filtering technology.

    • by mcgrew (92797) *

      Well, according to TFA it's not quite ready for prime time.

      There are some potential problems to iron out. For example. the team isn't quite sure how to handle the constantly changing pattern of malicious attacks and malicious attackers may soon find that it's not too hard to fool recommendation systems if you try hard enough.

    • by n9891q (863780)
      I'm not so sure. How much gee-whiz collaborative filtering whiz-bang technology does one need to predict that a mention in Slashdot will produce an attack and outage? I bet their research shows Slashdotting in the top-10 attacks.
  • by bigredradio (631970) on Tuesday August 18, 2009 @03:24PM (#29110883) Homepage Journal

    recommendation systems may soon be providing you not only with books and movie tips but a happier surfing experience too

    I am a little weary of making my surfing experience happier by allowing the system to do my thinking for me. Just think, "clippy" for the browser.

    • by natehoy (1608657)

      weary (tired of) or wary (nervous about)?

    • sarcasm on

      Yeah, and I'm pissed that I can't get Clippy working on Ubuntu!! The little dog in the search box too!! What, when you start to go geeky you can't have pets anymore? It's just WRONG, I tell you!! I'm going to send some hate mail to Canonical, and find out what the deal is. This just pisses me off!!

      sarcasm off

      Seriously - all those user agents and stuff should have been a tipoff. A corporation that offers cartoonish characters as part of a "serious" operating system can't be trusted with secur

    • by sorak (246725)

      recommendation systems may soon be providing you not only with books and movie tips but a happier surfing experience too

      I am a little weary of making my surfing experience happier by allowing the system to do my thinking for me. Just think, "clippy" for the browser.

      The article doesn't seem to say how it will be implemented, but I would assume it would be some server-side app that generates firewall (and possibly spam filter) rules.

  • Great idea! Protect us from the presumed dangers of the internet! After all, such terms as "presumed innocence" are overrated and outdated terms anyway...
  • Umm... (Score:2, Funny)

    ... wouldn't blocking people's access in advance considered an attack in and of itself? So the service should simply block itself off and be done with it.
    • by Eudial (590661)

      That, or go Skynet. The ideal way to stop all web attacks would be to bring down the internet itself. I so hope these guys did their homework [xkcd.com].

  • the new 404 (Score:3, Funny)

    by FudRucker (866063) on Tuesday August 18, 2009 @03:27PM (#29110931)
    Were sorry but you have been labeled an Internet Terrorist, your search for "PC + Game + Cheats" is a flagged keyword.
  • "People..." (Score:5, Funny)

    by natehoy (1608657) on Tuesday August 18, 2009 @03:30PM (#29110971) Journal

    "People who attacked this site ALSO attacked..."

    • Re: (Score:3, Funny)

      "Was this review helpful? Yes or no"

      • by natehoy (1608657)

        Amazon should patent "1-click attacking"

        • Re: (Score:3, Informative)

          by megamerican (1073936)

          Amazon should patent "1-click attacking"

          Ptech [google.com] already has it patented!

          • by Inda (580031)
            Not sure I get your Ptech link but it reminded me of something.

            We used to get targetted many, many moons ago by people searching Google for "phpBB version x.y.z". If you want to predict web attacks, Google says:

            Results 1 - 20 of about 80,600,000 for "phpbb version x.y.z"
  • by Tekfactory (937086) on Tuesday August 18, 2009 @03:37PM (#29111085) Homepage

    Or greatly exaggerated...

    "The team mined a database of hundreds of millions of security logs"

    Nobody actually keeps security logs, certainly not hundreds of millions of somebodies.

    The kind of people that DO keep security logs probably wouldn't hand them over either.

    I call shenanigans

    • Obviously they have developed hacking technology to break open all these systems to get at their logs to determain if they have been hacked. Well they will be blacklisting themselves later this afternoon.

    • Re: (Score:3, Insightful)

      by Red Flayer (890720)
      Yes, they worded that poorly.

      Fixed:

      The team mined a database of hundreds of millions of security log entries

      Now it makes more sense, and is quite believable, no?

    • by anhml (1621039)
      Hello, I am Anh Le, the second author of the work. We analyzed the corpus of security logs that were collected by Dshield.org: "Dshield is a repository of firewall and intrusion detection logs collected at hundreds of different networks all over Internet. The participating networks contribute their logs, which are then converted into a common format that includes the following fields: time stamp, contributor ID, source IP address, destination IP address, source port number, destination port number, and pro
  • False positives, here we come...

  • Meatware needed (Score:5, Insightful)

    by pheared (446683) <kevin@ph[ ]ed.net ['ear' in gap]> on Tuesday August 18, 2009 @03:45PM (#29111209) Homepage

    This sounds great, but only if it requires human intervention to implement the block. I used to work in a NOC, and we would have loved to throw up a warning on the big screens that an attack is 80% likely from the following netblocks in the next N hours. That way we would have a strategy developed for defending before it even started and would be able to minimize downtime.

    On the other hand, if you make this automatic you're going to piss off a lot of people very quickly because it's going to be wrong more often than you want.

    • Re: (Score:2, Insightful)

      by twisteddk (201366)

      Exactly. Because even if it's true, and it's 70% more accurate... I've yet to see a predictive system that's even remotely accurate. It may predict say... 50% of the sources of an ongoing attack (assuming a collaborative effort to determine when attacks are happening, and that you're not the first one hit), but that's far from enough to prevent a DDoS attack. And if you "accidentally" block... Say Canada (which I've seen before), then that's a LOT of costumers you just pissed off, but hey... Doesn't matter,

  • What about the people who are blacklisted unfairly? If the false positives are 1%, a huge number of servers will be blocked. This is the same problem with lie detectors and drug testing -- innocents get snared in the net. You need a way to confirm the positive, and not just blacklist based solely on this algorithm.
    • by anhml (1621039)

      Hello, I am Anh Le, the second author of the work.
      I responded to the concern about false positives in one of the replies above. In brief, investigating the false positives is not the main focus of our work, and it is an area of active research in the intrusion detection system community.
      Link to our paper: http://arxiv.org/abs/0908.2007 [arxiv.org] [arxiv.org]

  • Didn't anybody watch this? there have been other story lines along this genre, and it never works out, never, they always get the wrong person and it's used for evil.

    Okay if your going to do this anyway, here let me gaze into my crystal ball. Blacklist China, North Korea, and major parts of Russia.
  • Calculate the annoyance factor

    If site is shitty, + .1
    If site has a "clever" name, such as bit.ly, +.1
    If site's name has become widely used as a verb or other part of speech, +.1
    + unique users in the last 24 hours / 100,000,000

    Calculate the monetary factor

    If site sells something, +.05
    If site makes revenue through ads, +.05
    If site is partnered or associated with a megacorp like a bank, ms/google, etc., +.1
    + dollars lost per minute of downtime (based on the last 24 hours) / 1,000,000

    Calculate the brought it up

  • that is right, never assume anything. Assumption has caused more wars, fights, and upset in society than anything else. "Assuming something is the Weapon of Mass Destruction".
  • Sounds a lot like Minority Report [imdb.com].

    They *guess* that you may be guilty before it happens and blacklist you.

  • Great, so it can "predict" IP or site origins of malicious attacks, but can it also predict its own inevitable false positives? If so, how is it better than a DNSBL or other blacklist, except that it can make money for its owners without requiring constant updating and the requisite human labor?

    I'd hate to use an IP or own a site that it happened to incorrectly "predict" as the source of an impending-but-as-yet-not-real attack. They might as well compile a Minority Report against me. How would that be an

  • sidreporter [emergingthreats.net] could be used to gather such security logs more or less respecting privacy.

Recent research has tended to show that the Abominable No-Man is being replaced by the Prohibitive Procrastinator. -- C.N. Parkinson

Working...