Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Security

Network Intrusion Detection Systems Fail to Impress 215

Posted by michael from the got-root dept.
TheBongPipe writes "I'm reading a nice test here about 7 commercial IDSs. Who won the prize? Nobody..." They also looked at Snort, but found that all the products generated way too many false alarms.
This discussion has been archived. No new comments can be posted.

Network Intrusion Detection Systems Fail to Impress More

Network Intrusion Detection Systems Fail to Impress

Comments Filter:

  • What does everyone use (Score:1, Interesting)

    by mAIsE ( 548 )
    What IDS do slashdot users use ?

  • Those systems are rather difficult to design at times. I think it's odd that 7 systems didn't work to their full-potential.

  • False Alarms (Score:2, Insightful)

    by qurob ( 543434 )
    They also looked at Snort, but found that all the products generated way too many false alarms.

    Too many false alarms isn't necessarily a bad thing. In intrusion detection you'd rather take the false positives, than the alternative.

    The rate of false fire alarms, and false burgular alarms is VERY high compared to the actual number of real emergencies.

    • Car Alarms (Score:5, Insightful)

      by Codex The Sloth ( 93427 ) on Thursday July 11, 2002 @01:10PM (#3865137)
      Like Car Alarms, if it goes off all the time, people will just ignore it -- At some point, the noise drowns out the signal.

      You would hope that the increase in false positives decreases the number of false negatives but that isn't necessarily true either.

      • Re:Car Alarms (Score:2, Interesting)

        by Jucius Maximus ( 229128 )
        "Like Car Alarms, if it goes off all the time, people will just ignore it -- At some point, the noise drowns out the signal."

        I have been to a certain obscure country visiting my relatives where a common joke is this: A car alarm sounding means another dork can't figure out how to enter their car.

        The only flaw with this analogy is that car alarms are supposed to alarm people (i.e. draw their attention) who have no personal interest in the safety of your car to pay attention, and alarm the crook that they have been detected so they will hopefully run away without stealing the car.

        I am no security professional, but I would expect that such intrusion detection software does not give the cracker any warning that they have been detected by a real person or security system, so they have no reason to leave the system alone. It also does not give the IP of the potential threat to everyone else on the network so that the threat can be DDOS'd.

      • Re:Car Alarms (Score:3, Funny)

        by Asprin ( 545477 )
        Like Car Alarms, if it goes off all the time, people will just ignore it -- At some point, the noise drowns out the signal.

        Yup, yup, I *know* what you mean!

        I've got RAID array in my office that's part of the main production file server and there's this alarm that's been going off for, like, 16 MONTHS on the thing. Don't worry, it's not important - it's only a fan in the back of the drive tray that gets stuck sometimes, then it works itself loose and everything goes back to norm@#&$%@#$89d sifsd00JE{PGJE....

    • The problem with false alarms (Score:5, Insightful)

      by why-is-it ( 318134 ) on Thursday July 11, 2002 @01:35PM (#3865293) Homepage Journal
      Too many false alarms isn't necessarily a bad thing. In intrusion detection you'd rather take the false positives, than the alternative.

      Spoken like someone who does not carry the IDS support pager at nights and on week-ends!

      The problem with too many false IDS alarms is that the staff tend to treat it like the boy who cried wolf. After awhile, you disregard the pages or treat them with less consideration because the last n pages have all been false alarms.

      I think that IDS is important, but if there are too many false IDS alerts, it becomes difficult to put up with. Because they are strictly reactive systems, it is improbable that there will ever be a perfect IDS that never raises false alarms, but clearly there is a lot of work to do. I am surprised that Snort did so poorly, since it really is a nice system, but it takes a long time to build up a good set of heuristics...

      The rate of false fire alarms, and false burgular alarms is VERY high compared to the actual number of real emergencies.

      That's right. And in my area, if the police department are called out to the same location for three false burglar alarms in one year, they will not respond to any subsequent alarms automatically. And the fire department charges a fine of $300.00 per incident if they receive more than three false fire alarm calls to the same location in one year. Why? Because, as you said, the number of false alarms is much higher than the number of actual emergencies. The false alarms cost time and money and if all the resources are busy dealing with false alarms, there is nobody left to help when a genuine emergency occurs.
      • If you're sure their false, why don't you tune your ruleset?
      • I am surprised that Snort did so poorly, since it really is a nice system, but it takes a long time to build up a good set of heuristics...

        I completely agree with this statement, but can't understand why all the "false alarm" complaints. When you build an IDS, you need to start with all the alarms turned on then identify which non-intrusion signatures cause false alarms, then TAKE THEM OUT. This is the first rule of IDS's. Any site which explains how to install snort will tell you that you need to customize the config for your situation. Another thing that gives snort the edge(besides speed and cost) is that its open source, so the real TCP/IP hackers out there will always be tinkering to make it better and more efficient at catching true intrusions.

    • Homer: Now, here's my "Everything's O.K."alarm!
      [Homer flips a switch on the device, and it begins to emit a high pitched, incredibly loud beep. The rest of the Simpsons cover their ears as Homer speaks up]
      Homer: This will sound every three seconds, unless something isn't okay!
      Marge: Turn it off, Homer!
      Homer: It can't be turned off! [alarm fizzles out] But it, uh, does break easily.
      -- "The Wizard of Evergreen Terrace"

      This sounds about as useful.

  • Snort and GUI (Score:4, Insightful)

    by fabiolrs ( 536338 ) on Thursday July 11, 2002 @01:08PM (#3865120) Homepage
    I had a nice experience using snort.

    Come on, reading the article I saw the guy said a Snort disadvantage is not having a GUI. What kind of technical user this guy is? :/ Point and click is not always the best solution...

    • Snort GUI... (Score:4, Interesting)

      by Midnight Ryder ( 116189 ) <midryder.midnightryder@com> on Thursday July 11, 2002 @01:44PM (#3865354) Homepage
      Funny part is, you can take your pick of UI's for snort, on just about any platform (I run snort on WinNT on one network, and snort on Linux on another. And I've got a GUI for both of 'em ;-)
    • We use Snort on several boxes as sensors, reporting to a central database server. It's a sweet setup that's tedious, but not difficult to maintain. The database (ACID) [cmu.edu] is easy to use, and works as a PHB pacifier.

      Obviously we didn't get the whole thing up and going in a day, and we still spend time updating/tweaking signatures; but it wasn't rocket science.
    • Command line and text is not always the best solution either. There is nothing inherently inferior to using a GUI in and of itself.

  • False Positives... (Score:4, Funny)

    by Liora ( 565268 ) on Thursday July 11, 2002 @01:09PM (#3865126) Journal
    Like a pregnancy test, I think the false positives are preferable to sitting around thinking you're safe.
    • Take 1000 pregnancy tests and tell me which one is the accurate one. If you can't be sure you're pregnant in the first place, how can you even hope to figure that problem out?

      That's basically the reason it 'failed to impress.'
      • Much like a pregnancy test, you're probably going to find out the truth eventually.... Those early indicators just give you a clue as to what to do next. More security measures and planning for the future. Unless of course you wanted to be hacked ;).
        • Heh yeah, but that takes time. Unfortunately, I'd have to drag out the metaphor to silly extremes to make that point. Instead let me say this: When I have a problem with the mailserver (Exchange), for example, it's a pain in the ass to get to the right point in the log just to troubleshoot one problem. The reason for that is the event log logs allll kindsa silly stuff. Not all errors are indicated in red either.

          If I had an intrusion detection package that made a similar kinda log, it's easy to imagine that it'd constantly fill with new log events. I wouldn't know what to look for until the damage had been done.

          Get what I mean?

    • Re:False Positives... (Score:5, Insightful)

      by sam_handelman ( 519767 ) <samuel...handelman@@@gmail...com> on Thursday July 11, 2002 @01:28PM (#3865264) Journal
      Well, that's true, unless

      ALERT: Intrusion attempt detected! User Liora mistyped his password!

      the rate of false-positives is high enough that

      ALERT: Account Liora has recieved login attempts from three different IPs in the past 12 hours!

      you stop paying attention; unlike with AIDS testing (which has a very high false positive rate) the user is simply likely to ignore the system even if real threats occur

      ALERT: 1,262 attempts to login as user "root" in past 7 seconds!

      So it becomes desirable to lower the false positive rate to a manageable level, WHATEVER the rate of false negatives is, because otherwise you won't actually catch anything.

      The purpose of the AIDS test is to assure you that you don't have AIDS. False negatives are unacceptable, false positives can be dealt with.

      The purpose of the IDS is not to prevent intrusions - that would be nice, but it's not going to happen. The purpose of the IDS is to identify the (coloquially) hackers, so that you can retaliate against them / deter them, before they get you too many times, or get too many different people once. To do this, you need a deterence-level set of positives which is small enough (and true positive rich enough) for you to actually act on them.

      Oh, and because I get off on it when people with agree with me: this is no substitute for real, human-level, security measures. Someone who expects any system of this kind to protect them from lousy sysadmin decisions deserves the rusty metal sodomy they will no doubt recieve.
      • The purpose of the IDS is not to prevent intrusions...

        This is so true, I wish the PHBs would get this! Detection and prevention are two different things, it is like comparing a pregnancy test to a condom.

        You should have at least used the later, but to be sure, use the former as well.

    • Re:False Positives... (Score:2, Informative)

      by techwolf ( 26278 )
      If you're a Network or System Administrator, you should KNOW you're not safe.

      You SHOULD be testing your systems constantly.
      You SHOULD be installing new patches.
      You SHOULD be subscribed to CERT style mailings.

      You SHOULD NOT think you are safe because you're small. Security though obscurity is the biggest false sense if security I've seen. Former employees, especially the guy you replaced are a pretty large threat.

      For beginners out there, here are some places to start... (Some of these are OLD links, but still contain some useful information and yes, they're Linux oriented.):
      Beginners Guide to Armoring Linux [enteract.com]
      Linux Security Guide [nic.com]
      Nessus [nessus.org]
      Traditional HOWTO [tldp.org]

  • Is not exactly an accurate science; either one has to deal with false alarms, and probably more than a pleasant number of them for the sake of paranoia, or simply be more sure of their systems.

    Informal poll who among us uses an IDS, and why? I've always figured them for a way to be lazy, but that perspective is likely not shared. I'd love to see some other opinions!

  • Always compare with a placebo (Score:5, Funny)

    by Anonymous Coward on Thursday July 11, 2002 @01:09PM (#3865129)
    Compare with my program that suddenly displays "!!! RED ALERT !!!" at random.

  • Need more detail (Score:4, Insightful)

    by seosamh ( 158550 ) on Thursday July 11, 2002 @01:10PM (#3865131)
    It'd be nice to have some more detail on their results. The chart on the page shows Snort detected all the attacks listed in the chart except the SYN flood. And the footnote on that entry says Snort was down because of "configuration error."

    Gee, whose fault is that?

    • Re:Need more detail (Score:2, Insightful)

      by Lothsahn ( 221388 )
      The problem with snort is that it doesn't have a GUI. Most likely, it was harder than the other systems to set up. Although it's not directly Snort's fault that it wasn't configured properly, perhaps it is Snort's fault in that it is too confusing to setup.

      Remember--there are significantly more computers than talented sysadmins out there, which means that tools such as these must be made to not only work for experienced and knowledgable admins as well as "n00b" admins.

      I think it's at least partly Snort's fault. Make it easier to use, so that configuration errors are less common.

  • A False Alarm is still an Alarm (Score:5, Insightful)

    by pheph ( 234655 ) on Thursday July 11, 2002 @01:10PM (#3865136) Homepage
    I worked in a NOC for some time and found that while false alarms generally take away from the impact of real alarms, they still alarm you that something not quite right is going on in your network.

    They also go on to mention all ask too much of their users in terms of time and expertise to be described as security must-haves. IDSs are not screen-savers. Those who are setting up an IDS better have a good understanding of how they work and how to configure these applications. Point-and-click doesn't really apply to something this involved.

    ... As for stability, I think this report is correct, the only IDS I've used that didn't crash consistanty was snort (with ACID)

    • "As for stability, I think this report is correct, the only IDS I've used that didn't crash consistanty was snort (with ACID)"

      I've run NAI's IDS (the one that came bundled with PGP 7.1) for a year on Win2k, and it hasn't crashed yet. It does come up with false positives, especially if you configure it to be "sensitive", but once they occur, you can determine whether or not you want to continue to listen for them. It consistently tagged something the Mac's on our LAN were doing as a "fraggle" attack, so I turned off "fraggle" detection.
      Not a perfect solution, but soooo much better than nothing.
    • ... As for stability, I think this report is correct, the only IDS I've used that didn't crash consistanty was snort (with ACID)

      I can vouch for this also. About six months ago I setup a snort logging to mysql box at work to monitor our class-C and have had zero problems. It does take a while to tweak and prune things to eliminate all the portscan misdetects (any/any is certainly not advised) and such, but overall it's performed sweet. The price is right certainly, comparing it to those listed in the article. As far as console analysis, you simply cannot do without ACID as the parent said. Head to CERT and download a copy. My only complaint is that the db lookups perform a little sluggishly on the p2-233. :-)
  • The problem with many of the statistics from this test is that the management software was considered equal to the actual IDS machines. The "uptime" was actually garnered from the management software staying connected for the entire period. Given all of the complaints about java consoles being sluggish, I can only wonder at what the console machine was...

    From reports of the test the "wu-ftpd" exploit used wasn't an actual exploit, but was only a replay of one of the signatures of a decade old exploit. Since not all of the systems use signature detection, and since the "exploit" didn't actually exploit anything (*gasp*) some of the IDS systems didn't pick it up.
  • It didn't crash the entire time.

    According to the chart, it also didn't detect code red worm, SYN flood, or wu-ftp exploit.

    Was this box even operating correctly?

    • Re:Lancope Stealthwatch M100 (Score:1, Informative)

      by Anonymous Coward
      Yes, the box was working.. it is an entirely different type of IDS. apples to oranges type of thing.. IMHO, anyoner who tried to deploy it to replace snort/iss/dragon is an idiot. It does work well to COMPLIMENT a tradional IDS, but not to replace.

      It functions by analysis of traffic flow.. system x communicating to system y on ports a,b,c. if the pattern changes beyond a threshold, an alarm.

  • $20,000 for crap (Score:3, Interesting)

    by Rupert ( 28001 ) on Thursday July 11, 2002 @01:16PM (#3865178) Homepage Journal
    It amazes me that people will pay $20,000 for a product that regularly crashes, doesn't detect all intrusions, and can only be kept up by constant, expensive intervention from the vendor, when for $20,000 less you can have a similar product that doesn't crash, detects just as many intrusions (though not all of them) and can be maintained either by the vendor, or by anyone else with the wit to understand it.

    IDS are complex systems. Anyone pretending they have a packaged solution should rot in jail.

  • False Alarms get annoying for real admins (Score:3, Interesting)

    by Wingchild ( 212447 ) <brian.kern@gmail.com> on Thursday July 11, 2002 @01:16PM (#3865179)
    I recall a user we had on our network who thought it'd be cute to install BlackIce on his box, to better secure it. Nevermind the fact that I, and the rest of the admins at my company, had firewalls in place and had never had an intrusion on our network.

    Imagine the fun the first time we try to deploy an antivirus package to his desktop just to be blocked for -- are you sitting down? -- an attempted NetBIOS intrusion.

    After the second time we tried to deploy (and failed) BlackIce locked down the system so that it couldn't be accessed across the network by any other workstation, despite our having adminsitrative rights. That was cute.

    Just throwing up a little real world example of how annoying these false alarms can be.

    • Re:False Alarms get annoying for real admins (Score:1, Informative)

      by Anonymous Coward
      RE:
      had firewalls in place and had never had an intrusion on our network.

      HAHAHAHAHA. seriously. HAHAHA. This is called the hard-candy security princple.. Hard and crunchy on the outside, soft and chewy inside. Perimiter firewalls are insiufficient for medium-large businesses, and generally small ones as well. Got VPN connections? Allow ANY traffic in to a DMZ or internal network?

      As for BlackICE.. 'NetBIOS intrusion' isnt even an attack type. What happened was probably that the user blocked the netbios PORT via the firewalling and it showed as a 'NetBIOS port probe' with the reason=firewalled. Also, BlackICE does not disable ALL network activity. By IP address or port, yes. Not by network or everything. That would be stupid. See http://www.networkice.com/advice for BlackICE attack details.

    • Re:False Alarms get annoying for real admins (Score:1, Interesting)

      by Anonymous Coward
      Yup, and I change the admin password on all workstations every 60 days, PLUS remove all the damned domain admin hooks in the local system users hive.

      Keep your damned NOC hands out of my Fricking PC's. I'll deploy my OWN updates than you. (I am always at least 2 weeks ahead of corperate on every update, and I have NEVER had a virus origionate or propagate from my offices.. the last 2 Virii propagated from the NOC computers...)

      The local admin is who is in charge... you remote dweebs can keep your damned fingers out of my stuff.
    • So the IDS did alarm you to a misconfiguration in your network that was preventing you from correctly administering your machines the way you intended. Isn't this a good thing? Didn't it alert you to behaviour on yoru network that you didn't approve of? Isn't that the point?

    • What a wonderful idea! (Score:5, Interesting)

      by Subcarrier ( 262294 ) on Thursday July 11, 2002 @01:47PM (#3865369)
      I recall a user we had on our network who thought it'd be cute to install BlackIce on his box, to better secure it. Nevermind the fact that I, and the rest of the admins at my company, had firewalls in place and had never had an intrusion on our network.

      I hate to tell you this but, at this day and age when everything is being outsourced, some users feel they need to protect their machines against the "IT support". ;-)

    • Sounds like you need to lock your users down a little more; there's your primary security problem.

    • Where I work they try to mandate anti-virus software running on all the PCs. Only problem? The damn anti-virus software locks up my PC almost daily.

      The really shitty part is that just when I get the anti-virus off my system, IT (in their infinite wisdom) pushes an "update" onto my system.. sending me back into blue-screen land.

      I finally installed BlackIce on my system and set it up to deliberately block the fuckers. Yeah, IT gets pissed and thinks I'm stupid or something, but at least my computer doesn't consistently bluescreen anymore.

  • Info for commercial client (Score:3, Informative)

    by lamj ( 153635 ) <jasonlam&flashmail,com> on Thursday July 11, 2002 @01:16PM (#3865182)
    This article came from the point of view of a normal administrator trying to also manage security. It is mostly based on the assumption that you use the default ruleset (there's no mention of what ruleset is put to use).

    Nowadays you really have to be selective about what ruleset you use, logging too much isn't a good thing. This is part of the reason you need a qualified Intrusion analyst who have the expertise to determine which ruleset is useful and which isn't.

    The worst thing that can happen (which does happen quite often) is after paying for the expensive distributed sensor IDS system, the logs are never processed or read by anyone.

    As stated by the article, an IDS is suppose to log anomalies, that is any abnormal behaviour. But anomalies is only useful if you have a technical guy capable of analysing the traffic. In fact, I would rather have a faulty IDS system that misses packets than to have a good IDS system and all logs go down the drain at the end of the day.

    • That's the beauty of Snort. If you use Snort as your IDS you have saved $20,000 that you can then spend on the services of someone that knows how to set Snort up. 20K buys a fair amount of consulting, and from the article it sounds like you are going to need expensive help no matter what IDS you use. It also appears that Snort is at least as robust and useful as the competition, so you might as well go for the least expensive option.

  • From the article:

    But Opus One's servers run OpenVMS, not Windows. Even though it is trivially easy to figure out what operating system a Web server uses, not one of the IDSs did so. Instead, they collectively generated literally millions of alarms about attacks that never happened.

    That's an unrealistic expectation to place on an IDS, from the start. You get an IDS to log attack attempts first, not the attacks themselves - if the attacks are known (have signatures) your machines should be protected against them in the first place.

  • Prevention (Score:2, Interesting)

    by idfrsr ( 560314 )
    Prevention is another way to help secure a network, rather than simply detection.

    CycSecure [cyc.com] from Cycorp [cyc.com] the makers of OpenCyc [opencyc.org], the AI reasoning system, helps prevent attacks by using an AI engine to simulate attacks on your network to identify problems.
    It's worth looking into.

  • So, anything out there besides Snort? I just installed 1.8.7 on my Linux machine and was (un)pleasantly surprised to find that my (un)favorite Snort feature was brought back: random mysterious death of the snort daemon, with no logging or other diagnostic. But only in daemon mode, mind you, making the problem fun to debug.

    Luckily it doesn't do it on FreeBSD which is where I really need it running, but it is really frustrating and doesn't instill a lot of confidence. Grumble grumble, bitch, moan, etc.

    • So, anything out there besides Snort? I just installed 1.8.7 on my Linux machine and was (un)pleasantly surprised to find that my (un)favorite Snort feature was brought back: random mysterious death of the snort daemon...
      I will have to say I have Snort running on a Linux 2.4 host with full event logging and have yet to see it crash a single time.
    • I originally fielded snort (1.7x) on a home-built k7 server running slack 7 / kernel 2.2.13, in that setting it would segfault when local traffic began to push the 100 mbit level.

      More recently I've run short 1.8x on a dual P3 w/ ECC Ram & hardware RAID for a year now, first on kernel 2.2.19, and more recently 2.4.17.

      Not a single failure in 12+ months of either snort or the kernel. I suspect that the combination of a marginal backplane and IDE disk, coupled with the older kerenel probably caused the first machine's problems.

  • This review was poorly done (Score:5, Informative)

    by Anonymous Coward on Thursday July 11, 2002 @01:19PM (#3865201)
    This review wasnt done very well. There was a lot of discussion on the Security Focus Focus-IDS list. Robert graham, main craeted of the BlackICE engine (and the guy who wrote altivore) summed it up nicely in this posting (text below): http://online.securityfocus.com/archive/96/279595. Also, the entire thread can be found at: http://online.securityfocus.com/archive/96/280125/ 2002-07-08/2002-07-14/1

    Actually, most of his posts tend to have interesting (and qualified) views on IDS> sure he is biased (a vendor) but his commentary is usually thought out and not vendor-ish.

    > From: Andrew Plato [mailto:aplato@anitian.com]
    > In-Reply-To:
    > >http://www.nwfusion.com/techinsider/2002/0624secu rity1.html
    > Next time they should do RealSecure on one of my Win2k
    > appliances.

    No.

    While it is true that the reviewer found a bug with the Nokia platform that
    doesn't exist on Windows or Solaris, there wasn't anything especially wrong
    with the platform.

    The issue is that the reviewer was hostile towards IDSs. A customer wants
    his product to work, so when they don't, they will keep calling tech support
    until it does. Reviewers want the products not to work, so they will
    construct the nature of the test in order to make sure this happens. The
    reviewer, in this case, never called ISS; the first we heard about him was
    at the end of this review, not at the first crash of the Nokia box.

    RealSecure has a unique feature called "audit" events. These are supposed to
    trigger on normal traffic, such as every HTTP GET request. These are useful
    either to create audit trails, or as "anomaly detection": turn on all
    audits, then turn off those that trigger normally on your network.

    This reviewer turned on audit events, which flooded the console. The setup
    that Nokia provided them (256-megs of RAM and a database limited to
    2-gigabytes) is perfectly reasonable for the network they had, but not if
    all audits were turned on. (The Nokia bug we fixed was related to the fact
    that it didn't have enough memory to handle the event load). The reviewer
    complained about an overload of false-positives and the box crashing, but
    this was because the reviewer drove the product to the point where this
    happened.

    In truth, it isn't always obvious which of our events are "Audits" and which
    ones are "Attacks"; this is an issue fixed in 7.0 of our product. I doubt
    this would have made a difference in the review: 7.0 has a lot more audits,
    allowing reviewers to overload the product even more if they desire.

    Imagine a review of automobiles, where a reviewer grabs a Ford Explorer and
    starts complaining that it still crashes, even with the Firestone tires
    fixed. One might ask if the there is a problem with the Ford, but one might
    also ask if the reviewer intentionally drove the car until it crashed. Next
    time you are driving down the freeway, violently jerk the steering wheel all
    the way to the right. If you survive, you'll understand what I mean.

    I'm not saying the review is wrong. As the reviewer said, he learned a lot
    about IDS during the process of reviewing these products. If you, too, don't
    know much about IDS but are planning to install one, you will likely get the
    same experience: being overwhelmed with alerts that are "false-positives",
    and a general sense that the product isn't working. The first few months of
    running the IDS are likely to be particularly frustrating. I suggest (a)
    working with a consultant to tune the system, (b) working with the vendor's
    support in order to get suggestions from them, (c) learning more about the
    system. You are going to do (c) anyway: after a few months, you are going to
    have learned a heck of a lot more about hacking and defense then you ever
    dreamed possible. Read the review: take it with a grain of salt knowing the
    reviewer wanted all the products to fail, but realize that this likely to be
    your experience the first few months after installing the product, you are
    likely to be overwhelmed with events and unlikely to be impressed during the
    first few months of ownership.

    Robert Graham
    Chief Architect
    Internet Security Systems

    • Well, as the person who got to keep calling the vendors (with some it was more than once per day for multiple days) I can tell you we >did talk to the vendors. We had better support than the average user since we were writing a review. We effectively had an unlimited support contract, as reviews normally do. Nobody involved was "anti-IDS". The fact the fellow from ISS didn't know we were doing the review is a problem between him and Nokia. "Reviewers want the product not to work" is not true, at least not in this case.

  • Who are these people? (Score:3, Interesting)

    by gmhowell ( 26755 ) <gmhowell@gmail.com> on Thursday July 11, 2002 @01:20PM (#3865207) Homepage Journal
    Just read the article. A bit poorly written. What were the IDS run on? Why no analysis of Snort? I'll say that I find Snort way over my head, but that's because I haven't RTFM enough. Why would one want a GUI on a server? (one of the points they marked it down for). Why did it crash? I've NEVER had a linux box crash. NEVER. I've also very, very rarely had a program freeze up enough to require a kill -9 (other than Netscape Navigator and some other buggy stuff. Not stuff like exim, apache, etc.) As a matter of fact, scroll down, and it seems that the downtime was due to their problem, not Snort (footnote at bottom of uptime table).

    There are complaints about false-positives. I've played with Snort and there are ways to decrease the alarms put up. For example, a certain number of bum packets in a certain length of time. Not each and every packet.

    Looking at the info at the bottom of the article, the authors should know what they are doing. But given the misrepresentations and inaccuracies releative to Snort, why should I believe their testing of non-Free software was any better?

    Maybe it was eWeek or some similar publication about six or nine months ago did a similar check. The article was much longer and more in depth. They were also more appreciative of the programs out there. Now, some will say "just to appease their advertisers". Well... Maybe. But if that is the case, why did Snort get their nod as the best?

  • NIDS will not catch everything (Score:3, Insightful)

    by lamj ( 153635 ) <jasonlam&flashmail,com> on Thursday July 11, 2002 @01:23PM (#3865232)
    I would also like to remind everyone having pride in their own IDS that NIDS will never catch every single attack. (At least for the next little while)

    Signature based detection is only good if the attack utilize abnormal or unique traffic to exploit the vulnerability. It will not pick out attacks that uses normal common traffic (for obvious reasons).

    IDS evasion techniques are also heavily worked on, plus all application level evasive techniques (eg. sidestep). We can just never be totally dependent on the NIDS for telling us intrusion has occured. It works for most attacks but will fail for some.
    • In the past year and a half strides have been made in building anomoly-based detection systems that do not necessarily suffer the weakness of rule lag that signature-based systems do. These systems go about the process a little bit more intelligently by reporting on traffic outside the "norm."

      The catch with such a system is that you have to be very careful about measuring what your "norm" is. If you capture a profile on a very noisy network, then a lot of potentially dangerous traffic could go unreported.

      As with most things in security and system administration, your solution will only be as good as the person or persons who design, implement and support a system. If you don't have a trained analyst evaluating and tweaking your IDS solution, you're in trouble. There's currently no such thing as a true IDS appliance.

      -buffy

  • Snort with ACID and MySQL (Score:5, Informative)

    by agrounds ( 227704 ) on Thursday July 11, 2002 @01:24PM (#3865238)
    Not having a GUI?!?
    I've been running Snort for some time now, and love it! I'm using MySQL logging with ACID and ADODB under Apache for a front end. You just can't get any easier than fill-in-the-blanks SQL querys and intuitive packet layouts. Obviously, they want a strictly out-of-the-box product, and aren't willing to invest any time to make a solid IDS.

    As to the false positives, I can concur that in the beginning it was daunting seeing the flood of alerts, but in time, you figure out what is normal and what is not. A little restructure, or a few rule overrides, or rewritten rules, and it's seamless. All it takes is time. This is akin to bitching that your fresh *nix install doesn't have everything just the way you want it, with all your custom apps and modules. You can easily reduce the number of snort alerts by passing the command option as:
    snort -D -o -i eth2 -c /etc/snort/snort.conf
    This (the -o) changes the rules order to Pass:Alert:Log killing home network normal activity before alert processing. It helps immensely!
  • In the same vein this article http://www.gigaweb.com/mktg/man_sec_mon/cpane2.asp [gigaweb.com] compares various managed security services, which also offer and utilize some of the various IDS systems you've mentioned.

  • IDS only as good as the person reading it (Score:3, Interesting)

    by fruey ( 563914 ) on Thursday July 11, 2002 @01:25PM (#3865247) Homepage Journal
    IDS systems generally automate a number of things that hackers do all the time; possible exploits are often false positives but the sysadmin should be able to see a false positive a mile off.

    I have used Snort and Qualys (the high priced commercial outsourced IDS) and both give false positives quite frequently. However, proving they are false positives is part of the skill of a good human sysadmin. This is why IDSes will never replace a good sysadmin. He or she should be able to see the report and say without any shadow of doubt in his speech that any particular exploit shown by the IDS is a false postive or not.

    This still means that each IDS has its good points; but why anyone would pay a lot for a system that cannot, by definition, be any better than an up to date Snort and human reading of the report, and knowing your network inside out. Those who buy into big commercial IDSes clearly are investing in software when they should be investing in people, training those people, and understanding those people. Too many middle managers think their sysadmin speaks a language they will never learn, and therefore need these things to understand. But a good sysadmin should try hard to find ways to communicate with them, and can if need be annotate a nice little Snort report and be done with it.

  • great testing (Score:3, Insightful)

    by Guttata ( 35478 ) on Thursday July 11, 2002 @01:26PM (#3865251)
    If you look at the table, snort looks like it was doing great, except that it somehow missed the SYN attack. So, based off that chart, none of the IDS corrected detected all of the attacks... however, you read on a bit further, and..

    Snort was off the air at the time of the attack because of misconfiguration on our part.

    I don't have a lot of confidence in their results.
    • Hey they were managing these other IDSs at the same time. And from the way they make this sound, their goal was to discover a commercial solution, and the open source snort IDS was merely around for reference. Sure they didn't state this anywhere in the article, but reading through it, they seem to make the common act of using the open source software as the "cheap, but working" reference, which many people do. Hell Microsoft fakes this tactic when boasting IIS...
  • because they make their IDS too complex # cat shoot_em_up_bad_guy_ids.conf #
  • It's definitely true that this is one of the most notable weaknesses of intrusion detection systems as they exist now. I work in a financial institution where upper management has finally made a sensible decision and devoted a full-time person (me) to network security but that's not the case in many smaller organizations. The vast majority of (external) intrusion attempts are from script kiddies that use pre-fab tools and put forth little effort to conceal their actions. In my opinion, this is justification for most networks to run in a "low paranoia" mode. This would get rid of excessive false-positives and the noise created by Joe Kiddie and his 10,000 buddies who are out there constantly port scanning class A subnets.

  • No GUI for Snort? Acid! (Score:5, Informative)

    by stere0 ( 526823 ) <slashdotmail@stC ... minus physicist> on Thursday July 11, 2002 @01:47PM (#3865366) Homepage
    The author doesn't mention ACID [cmu.edu], a very good and useful interface to Snort (or at least I haven't seen it). Since he also complains about the lack of GUI (Puh-leese, an IDS is not for interns!), I suppose he hasn't heard of it. Quoting the website:

    The Analysis Console for Intrusion Databases (ACID) is a PHP-based analysis engine to search and process a database of security events generated by various IDSes, firewalls, and network monitoring tools. The features currently include:

    • Query-builder and search interface for finding alerts matching on alert meta information (e.g. signature, detection time) as well as the underlying network evidence (e.g. source/destination address, ports, payload, or flags).

    • Packet viewer (decoder) will graphically display the layer-3 and layer-4 packet information of logged alerts

    • Alert management by providing constructs to logically group alerts to create incidents (alert groups), deleting the handled alerts or false positives, exporting to email for collaboration, or archiving of alerts to transfer them between alert databases.

    • Chart and statistics generation based on time, sensor, signature, protocol, IP address, TCP/UDP ports, or classification
    ACID has the ability to analyze a wide variety of events which are post-processed into its database. Tools exist for the following formats:
    • using Snort (www.snort.org [snort.org])
    • Snort alerts
    • tcpdump binary logs
    using logsnorter ( www.snort.org/downloads/logsnorter-0.2.tar.gz [snort.org])
    • Cisco PIX
    • ipchains
    • iptables
    • ipfw

  • Just say no. (Score:1, Informative)

    by Anonymous Coward
    IDS's are just plain silly. Do the following:

    1. Turn off unused services, and disable suid software. If this is not possible, then don't use the application.
    2. Take steps to minimize compromise of at-risk services. Use chroot, aplication specific uid/gid.
    3. Leran how to apply packet filters to ingress/egress points.
    4. Apply patches for applications and O/S
    5. Stop going to expensive, lame 'security' seminars.

    fini!

    IDS will someday approach NMS systems in uselessness
  • IDS Systems are only as good as the people (i.e. admins) using them... ged68
  • ... you may as well go with Snort, which is free. All but the $2,500 Nokia are $12,500-$25,000. Excellent article.
  • IDS systems need to be tuned! Don't have any NT machines on that subnet? Turn off all of the NT related signatures! Get tons of false alarms on a particular alert which isn't applicable? Turn it off! It's a matter of risk assessment. Are you more likely to miss something important because of this alert which goes off all the time and has a low probability of being legitimately triggered? Turn it off! You won't catch everything this way but the goal is to at least catch SOMETHING that you would not have if you didn't have the IDS!
  • IDSes are NOT meant to work out of the box. Snort's FAQ specifically states that you should disable rules for things you don't need! By default, it includes a lot of stuff. Luckily, the rules are neatly organized into files, so you can comment them out, and stop getting warnings you don't want! Likewise, using Snort without Acid is well... not very common. Yet, there is no mention of Acid in this article. I can only imagine that the rest of this article is flawed due to the reviewer's lack of knowledge.

  • Clueless reporters (Score:4, Insightful)

    by Python ( 1141 ) on Thursday July 11, 2002 @01:59PM (#3865444)
    After reading this report, I'm left with the impression that the reporters that wrote it had unrealistic expectations for the products they deployed, and little knowledge about either how to configure, tune or properly setup the products they were testing. But then, they admit that everything they are complaining about isn't such a big deal and so on. In short, a totally sensationalized waste of time. Skip it if you haven't read it already.

    To wit, we've used all of the products they complain endlessly about, and all I can say is RTFM. All of the problems they encounter are either configuration problems or worse, PEBKAC.

    If you want to really learn about IDS, and you don't have the budget to buy a commercial IDS, download a copy of snort [snort.org] and learn for yourself. This report strikes as the type of complaing you get from an IT customer that wants to buy a product, turn it on, never configure it and expect it to magically work.


    By now, readers with security expertise probably are asking why we didn't tune the IDSs to reduce the chatter and improve our chances of seeing real attacks. The short answer is that we did, or at least we tried to (emphasis added). Including setup time, this project stretched along three months; and during that period we worked on these systems almost every day.

    Wow! What a revelation! You mean you have to know what you're doing and it actually takes time to configure these powerful tools?! In a word, DUH. IDS'es must be tuned. IT products must be configured properly. These things take time, sometimes a lot of time. The core of their complaints revolve around their inability to do either of these things well. Given that lots of people manage to do this effectively everyday and have been for years and years, we're left to conclude that these reporters were not up to the task. And here it is:


    Don't expect IDSs to be plug-and-play devices.
    These folks actually expected NIDS to be plug-and-play, and thats what they seem upset about. NIDS are powerful sniffers, they need to be tuned, they need to be configured and yes, this IS an ongoing process - but they are not plug-and-play devices.

    Futhermore, all of IT is an ongoing process. A big, circular, ongoing process that requires competent personnel to manage, maintain, tune, test, patch, configure , deploy and yes, spend TIME on. Anyone that expects to be able to deploy close to a dozen different IDS products as plug and play devices into a production network in 90 days with questionable expertise is fooling themselves.

    To be effective, they require a lot of tuning, and a fair amount of security expertise (emphasis added). They'll also require willingness to spend a lot of time sifting through reports, at least until the configuration is tuned properly. Even then, IDSs will require constant updating as new attacks appear. IDSs can be lifesavers and invaluable educational tools - but only for those with a lot of patience and a willingness to learn.
    And then they say as much. Again, this report is total waste of time. Its overly sensationalized and stems from a lack of expertise on the products in question. Skip it, download snort or buy one of the commerical products, take a class, read a book and learn for yourself. You won't learn much from this report that common sense wouldn't have told you already.

    • Let's see, I'm going to spend 5 digits up to protect way upwards of 7 digits in information. Therefore I should insert the CD, click the mouse, and that's all the expertise I need.

      So how come alarm companies exist? According to this logic, everyone should send a co-op to Home Depot and have them install the company alarm system. Maybe it'll take a day.

      Where have I seen this attitude before? Oh yeah, the guys who turned on their new computer, plugged it in, and called it their web server. The ones that scan my boxes' port 80. The ones that are owned by Code Red, Nimda, and Klez. The "network administrators" with an MSCX certificate on the wall. The Windoze users.
    • IMO this is the most insightful comment about this article that I have read. You are exactly on point.

      IDS is not a simple technology and anyone who expects to filter and analyze WAN traffic with the click of a mouse should scurry away with their MCSE between their legs. IDS takes tuning. Snort was originally written with the intent that its users would write their own rules to adapt to their own environments. (Apologies to Marty if I am not 100% accurate here.) Instead, so many excellent rules have been written and distributed that the work has been done already for most of us and the project has grown stable and accurate enough to go commercial - and compete impressively.

      IDS is a science and an art, not a prepackaged app that you can stick a label on: "good", "fair", "sucks!". YMMV according to the time and research you invest in making the product work to its full potential.

  • I can't imagine using an IDS in this fashion (Score:2, Insightful)

    by Anonymous Coward
    Was this test run outside of a firewall/filter? Because it would seem like the IDS is best used behind a full strength firewall in the first place; that is, most of the attacks should never reach the IDS anyway. We use Snort to monitor our networks, and under proper configuration the number of alerts is manageable / predictable under normal circumstances.

  • Why is Snort the clear winner? Because it's the only one that doesn't cost anything. If none of them work as well as they should, at least with Snort you aren't blowing money on the software :)
  • I am the Director of Managed Security for a company on Hawaii, and we rovide managed Security Services to various companies around the state based on Snort. Snort truly is a very good IDS, and if configured properly it will generate few if any false positive alerts. Most of the reason that people say bad things about a product is due to their own lack of experience in setting it up.

  • IP address vs domain name (Score:3, Insightful)

    by chill ( 34294 ) on Thursday July 11, 2002 @02:14PM (#3865548) Journal
    The author of the article complained that the majority of the systems reported results by IP and not domain name.

    This statement alone gives me reason to doubt their abilities. Combine this large amounts of traffic they have with a reverse-DNS lookup for each and you would have crippled your DNS servers.

    This is configurable in Snort, and mentioned in detail in the Snort docs.

    A good solution is to either dedicate a DNS server to the IDS box, or use a script/utility to do reverse-lookups on the items you are interested in. Not live, but when a human is looking at things.

    They are right about one regard -- IDS configuration, monitoring and maintenance isn't for non-professionals. You *need* to know what you are doing.
  • You can't simply plug these things in and expect them to work perfectly. If you don't know what you are doing with a high-powered IDS then you don't have any business using or judging them.

    You need to take quite a while (based on your network) and OPTIMIZE your rules for a product like SNORT so that you are getting alerted to the types of things you want to know about while minimizing false positives.

    It is pretty obvious that the tester didn't do that, and as a result he had nothing but bad things to say.

    Let's let an experienced Snort user configure his conf files and then run the test again. I think you may find that the results are different.

  • Topology (Score:2, Insightful)

    by HappyPhunBall ( 587625 )
    This article would have benefited greatly from a diagram or two. IDS behind the firewall? IDS in front of the firewall? Possibly they mentioned it, but I failed to find reference in the article. Myself, I prefer to keep the IDS behind the firewall as I only care about packets that get through. How do /.'ers deploy?

  • What?! (Score:2, Interesting)

    by shftleft ( 261411 )
    2. Offline because of configuration error.

    Gee, I wonder if they should learn how to configure Snort before they test it.
  • I suppose this is a good time to plug my university's project, STAT. STAT is an open sourced IDS framework. It allows you to monitor arbitrary events and take arbitrary actions based on them. It's possible to extend the field of STAT's vision by writing extensions to STAT in the STATL language. It's also trivial to write responses to known exploits.

    You can find more info about STAT at
    http://www.cs.ucsb.edu/~rsg/STAT/

    STAT already has 2 extensions, NetSTAT and USTAT that watch for common network and unix-level exploits. Other projects include making java-level IDS's and mobile agent IDS's. It's a great project and it blows everything else out of the water. If you're dissatisfied with IDS's as they are, check out STAT.
  • They also looked at Snort, but found that all the products generated way too many false alarms.

    Curses, foiled again! If it weren't for that pesky "not too many false alarms" requirement, I'd be able to create terrific security software. I'm picturing a system that generates a "WARNING: NETWORK SECURITY BREACH" message every five minutes, rain or shine. Keeps the sysadmins on their toes, and foils all network intruders who aren't fast enough to be in and out in five minutes.

  • I just finished attending a week-long training session on Dragon from Enterasys Networks, and it is by far the best NIDS I have ever used. I've worked with several systems (Cisco, Snort, Intruder Alert, etc.) but Dragon is the first one I've used that really doesn't seem to miss a beat. I'd be interested to see how it stacks up against the others that didnt' impress the authors of this article.

  • Fine tuning... is the key to a succesfull IDS (Score:3, Informative)

    by Erik_ ( 183203 ) on Thursday July 11, 2002 @02:54PM (#3865800)
    But Opus One's servers run OpenVMS, not Windows. Even though it is trivially easy to figure out what operating system a Web server uses, not one of the IDSs did so. Instead, they collectively generated literally millions of alarms about attacks that never happened.
    If you know your web servers are running on OpenVMS, why do the even bother to log IIS directed attacks ? An IDS needs fine tuning to only log what is needed, or more likely what is NOT needed. And every so often, you need to check the logs and fine-tune the rules, and update the rules for new signatures.
    To most security specialist, this is common sense...
  • What we need are systems that don't just report intruders, but locate them. Enough information needs to be collected to get a conviction.

    We'll probably see this: Your local intrusion detection system will connect to some big service, probably run by Verisign, which can query account info for major ISPs, SS7 phone call data, logs of recent packets, and related info, and then relays it to the National Infrastructure Protection Center, which will pick a few script kiddies every day and send out big guys in suits to bring them in. About once a month, a big, televised raid. A few months of that, and the attack rate will go way down.

    A few thousand honeypots tied to such a system will put a big dent in the distributed attacks, too.

  • A Great front end for Snort - PureSecure (Score:2, Informative)

    by Anonymous Coward
    PureSecure is a great front end to Snort. Check it out at http://demarc.com [demarc.com]

    Things I like about it:

    installation script is truly a marvel (installs snort, mysql, apache, perl modules)

    Login screen/authentication

    Big Brother like monitoring

    File integrity checking

    IDS using Snort sensors

    free to use for non-commercial use

    no I don't work for them I just like the software.

  • Where's the frigging methodology? (Score:3, Informative)

    by Joseph_ShawII ( 257564 ) on Thursday July 11, 2002 @05:21PM (#3867039)

    I don't trust any "real world" shootout that doesn't show how the IDS were plugged into the network, how they determined an attack, and other such key points. You can't just say "we plugged it in and nothing worked." IDS are much more complicated than that. How and where were they plugged into the network fabric? Were they using switch port mirroring or passive ethernet taps at the uplinks? How do they know these attacks happened without initiating them themselves? That last one is the biggest single problem with "real world" testing. Unless you're launching the attacks yourself you do not know, and unless I missed it, they were relying on attacks to just happen out of the blue.

    Now, they do raise some important issues with the backend storage of events and the need for clarity with the false positives and false negatives, but many of these can be dealt with by implementation of a real-time security console that does some form of event correlation from multiple security devices that says "The IDS sees this as a problem, the firewall sees it as a problem, and the target sees it as a problem. It's probably a problem. RED ALERT!" It's a much more intelligent way of dealing with events than just forwarding each one to a pager.

    We've always said security is a process which must be maintained and firewalls/IDS systems are not a panacea to network security. As someone who's been responsible for a large scale IDS roll-out at Enron Broadband Services, where we were ISS' single largest customer for RealSecure before everything went to hell, I feel confident saying that Network IDS is a very useful tool, provided you keep it out of the hands of people who have absolutely no clue what they're doing with it, like the three gentlemen who are responsible for this article.

    Joseph

  • Alerts != Alarms (Score:2, Informative)

    by Anonymous Coward
    There is a misconception among IDS noobs that alerts are like alarms: alerts generate alarms. An IDS generating 1000 alerts should not mean that an admin would receive 1000 alarms/pages.

    For example as an IDS admin I want to see alerts for failed telnet attempts internally. If it's 5 within one minute directed at one host, it's not a problem, probably just human error. If it's 1000 per minute, then I want to see an alarm, and get paged. The "reviewers" of the IDS products would have understood this, and taken it into consideration had they ever deployed an IDS in a production environment.

    Alerts are not a bad thing even if they are false positives. False positive alarms are a bad thing, especially when they wake you up at 3AM. Again it comes back to tuning.

    I can however verify what they experienced with ISS. At my last company we had ISS come out and install, configure and tune Real Secure. I can tell you from first hand experience, that the ISS products suck. I ended up installing Snort in order to keep the ISS products honest. My experience was that ISS had a nasty habit of dropping packets. Snort had no problem keeping up with our frational DS3 (30 Mb).

    More recently I also had the priveledge of seeing Real Secure crash repeatedly on a Nokia 530 (installed and configured by ISS engineers) during an IDS pilot. We kicked ISS to the curb and are now in the process of installing Snort with support for ACID, and MySQL.

    IMO the reviewers were idiots. They obviously didn't spend much time with any of the products. Which is unfortunate, because some of the good products got lumped in with some of the bad ones, which were all failed together because some reviewers obviously didn't RTFMs.

    FYI there is also a really pretty GUI for Snort:

    www.demarc.org

  • If people are expecting security-in-a-box from an IDS, of course it's not going to live up to their expectations.

    An IDS is nothing more than something to alert you to any abnormal conditions. It's a tool to help filter out the noise and show you what you want to know.
  • None of the IDS systems they named "detect" intrusions. NO IDS system on the market really detects intrusions. They either look for known signatures of various exploits, probes, what have you, and report on them, or they do some form of anomoly detection based on a "baseline" for the network they're observing.

    Neither of them is flawless OR a complete solution.

    NONE of them are going to be perfect out of the box. It takes skill and experience to know what's important and what's not. None of these IDS systems are going to catch the guy doing a slow map of your IP space. ALL of them will false positive on some things, and miss others completely.

    It takes a human at the other end to look and see and decide what's a threat and what's not.

    We won't go into the lack of information on how they configured each one of these IDS systems. I use snort at home on my LAN, but have worked with NFR and Cisco's Netranger - and each has it's advantages and disadvantages. If you're SERIOUS, you're combining something from a commercial heavy duty IDS, with Snort, with dumps from all your syslogs, some kind of host based IDS, and putting together the individual pieces to see what's happening. Then you might be able to detect a skilled intruder.

    It's NOT for the faint of heart, the clueless, or, it seems, the media pundits.

Slashdot Top Deals

To the systems programmer, users and applications serve only to provide a test load.

Close