Forgot your password?
typodupeerror
Security IT

What to Do When Your Security is Breached 177

Posted by ScuttleMonkey
from the set-fire-to-the-servers-and-run dept.
ancientribe writes "When you've got a full-blown security breach on your hands, what do you do? If you've been smart, you'll already have a computer security incident response team — and a plan — in place. But many companies are too resource-strapped to have a full-blown, fully-tested incident response strategy. DarkReading has some tips on what to do — and what not to do."
This discussion has been archived. No new comments can be posted.

What to Do When Your Security is Breached

Comments Filter:
  • by Anonymous Coward on Monday March 26, 2007 @05:59PM (#18493915)
    When your security is breached by a handful of thugs you must immediately run out and attack a random neighbor's house.
  • The problem is (Score:5, Insightful)

    by Rosco P. Coltrane (209368) on Monday March 26, 2007 @06:00PM (#18493935)
    many IT managers decide to purchase Microsoft so when something happens, well, "we couldn't go wrong with Microsoft" or "it's Microsoft, not us". Unfortunately, that's the extent of their plan, after pulling the network cable, i.e. cover their asses.
    • Re:The problem is (Score:4, Insightful)

      by Archangel Michael (180766) on Monday March 26, 2007 @06:10PM (#18494063) Journal
      Bingo.

      I would further add, that they chose Microsoft because Microsoft promises lower TCO through lowered administrative (geek) needs.

      I suppose that most Microsoft shops wouldn't even know if they were breached, because most breaches don't actually desctroy data, they just steal it.

      • Re:The problem is (Score:5, Interesting)

        by cptgrudge (177113) <cptgrudge@gmaSLA ... com minus distro> on Monday March 26, 2007 @06:46PM (#18494495) Journal

        I suppose that most Microsoft shops wouldn't even know if they were breached, because most breaches don't actually desctroy data, they just steal it.

        It's so much worse than that.

        Back in my younger days at a summer tech job for a US school district, I found that an NT4 SQL server had been compromised a group of people. They were based out of France, I think, from what I could tell from the IP addresses, and had actually set themselves up quite nicely, with organized file structure and their own IRC and FTP server running on it. They were using it as a repository to store files and a few French movies. After I told the sysadmin in place at the time about it, I was stunned when he said, "Well, are they hurting anything?"

        After some persuasion on my part, he rebuilt the server. Three times. After it kept getting hacked by the same people.

        • Re: (Score:2, Insightful)

          by Anonymous Coward

          Back in my younger days at a summer tech job for a US school district, I found that an NT4 SQL server had been compromised a group of people. They were based out of France, I think, from what I could tell from the IP addresses, and had actually set themselves up quite nicely, with organized file structure and their own IRC and FTP server running on it. They were using it as a repository to store files and a few French movies. After I told the sysadmin in place at the time about it, I was stunned when he sai

          • Seems to me the OS made it too easy to change the defaults to something less secure. At the very least, you should have to read a man page to figure out how to turn off your security so thoroughly. Or is it just that the OS was insecure by default?
            • Re:The problem is (Score:4, Interesting)

              by sumdumass (711423) on Tuesday March 27, 2007 @08:39AM (#18499897) Journal
              It was probably neither.

              I know some IRC groups were the members get their company servers to provide dumps and bots. And of course non one ever knows it is going on.

              So I'm going to guess that if they went through the trouble of hacking it three more times, it was probably an inside job to some extent.
              • by cptgrudge (177113)

                So I'm going to guess that if they went through the trouble of hacking it three more times, it was probably an inside job to some extent.

                I captured the traffic with a network sniffer. Specifically, the IRC traffic sent in cleartext, and they were all chatting in French. So unless they were local students routing traffic through French IP addresses and all speaking in French, I kinda assumed they were in France.

                But the incompetent admin is probably more to blame. This all happened when there was no

      • by canuck57 (662392)

        I would further add, that they chose Microsoft because Microsoft promises lower TCO through lowered administrative (geek) needs.

        In defense of Microsoft (I usually bash them) I will say the OS does have many features that are rarely deployed that can dramatically improve security. But here in is the problem, you can take someone from McDonald's on Monday and be a senior administrator by Friday and not even know these features exist. Because you work for $25K less per year, management loves this. Which is

  • by Anonymous Coward
    Complain! Call the help desk!
  • by klenwell (960296) <klenwell&gmail,com> on Monday March 26, 2007 @06:02PM (#18493961) Homepage Journal
    But since ours is a relatively small company, we went with the open-source Thai fighters.
  • by had3l (814482)
    Run from side to side?
  • my plan (Score:5, Funny)

    by trybywrench (584843) on Monday March 26, 2007 @06:04PM (#18493987)
    Kent Brockman: So, professor, would you say it's time for everyone to panic?
    Professor: Yes I would, Kent.
  • by Anonymous Coward on Monday March 26, 2007 @06:04PM (#18493989)
    what to do if you burn your hand:

    1. first, remove your hand from the burning stove.
    2. use ice to cool your hand
    3. seek medical attention.

    wow. Thanks. I never would have figured any of that out on my own.
    • by Kandenshi (832555) on Monday March 26, 2007 @06:22PM (#18494207)
      eh? Your steps are a bit off :P Don't use ice to cool a burn, you're likely to cause further damage. Just use running cold water to cool things down. I'd also suggest tossing a bit of sterile gauze over it too, if things are more than mildly bad.

      "To treat a minor burn, run cool water over the area of the burn or soak it in a cool water bath (not ice water). Keep the area submerged for at least 5 minutes."
      http://www.nlm.nih.gov/medlineplus/ency/presentati ons/100213_1.htm [nih.gov]

      "Flush the burn with cool running water or apply cold- water compresses (a wet towel or handkerchief) until the pain lessens. Do not use ice or ice water, which can cause more damage to the tissues."
      http://www.personalmd.com/healthtopics/crs/burn1.h tm [personalmd.com]

      *emphasis mine*
      • by StikyPad (445176)
        It's worth mentioning that those are idiot-proof guides, for people who need to read about what to do when burned. There's nothing wrong with using ice as long as you avoid prolonged exposure. Since the definition of "prolonged" is inveresely proportional to tissue damage, it's simpler to just tell people not to do it, especially from a liability standpoint.
        • by sumdumass (711423)
          It would likely depend on how bad the burn is and were it is at and a couple of other things.

          I used to work the ovens and the char broiler when doing my restaurant tour coming out of high school. This is just personal observation but a lot of times when I would get burnt, the only the best thing I could do is just put some gauze around it to try and shield the heat of the broiler and ovens from it. When I used ice or running water it would blister and I would have two more stages of dealing with it. One we
      • No, No,

        The voice of user experience has clearly told me that you should use Dry Ice to cool off your hand. This way, you don't have to worry about infections caused by something in the water...
    • You forgot

      4. ???
      5. Profit!

    • by vux984 (928602) on Monday March 26, 2007 @06:35PM (#18494359)
      Based on the other (correct) replies about not using ice to cool your burnt hand we can conclude 2 things:

      1) You apparently shouldn't rely on what you 'figured out on your own'.
      2) In addition to getting a plan for a security breach you should also look at getting some help with your first aid plan too.

      • by StikyPad (445176)
        Actually, there's nothing wrong with using ice, as long as you don't leave it on there long enough for your skin to freeze. Again, common sense.
    • Ice will just make it worse. [stjoehospital.com]

      Only thing worse than a hollow article is a wrong one.

    • by Moraelin (679338) on Monday March 26, 2007 @07:28PM (#18494955) Journal
      I don't know, their approach seems kinda... dangerous to me, but maybe that just shows that they're the big security gurus and I'm just a lowly coder. Maybe I can learn something from them. Or maybe they're talking out the ass, I dunno.

      For starters the advice to wait until the whole team is assembled, including the accountants, lawyers, etc, then holding meetings to determine your strategy, etc, before even unplugging the damn thing... dunno, it seems to me bordering on criminal. Yes, you don't want to let one lone cowboy handle it from end to end, but a trained admin could at the very least be able to unplug the computer from the network and isolate the damage before it goes any worse. Or know enough to decide if it has to be unplugged. But if he thinks it is, it should be step #1 not IIRC step #4 after you're done holding your meetings and informing the employees and having PR draft the vaguely worded announcement that tries to make it sound unimportant to your customers.

      Waiting for the designated accountant, and the designated lawyer, and the HR guy, and God knows who else to arrive at the middle of the night and hold their meeting while a breach is in progress and someone is downloading your productive database, seems to me dumb to the extreme. To reuse your example, it's like saying you should keep your hand in the stove until you talked to your lawyer and your doctor and a designated family member, make sure you have a strategy, and only then pull the hand out. By that time, it could be burned to a crisp.

      I mean, by the elder Gods, especially when you include such non-techies... surely you've seen these guys when they have to give you a spec for a program. If you wait for them to hold a meeting on such technical issues as "are we in aggreement that we need to unplug the server?", at least one goes into responsibility avoidance mode and refuses to be remembered as the one who took any decision, at least one goes into alpha-dog-pissing-on-everything-to-mark-his-territ ory mode, etc. It's a meeting that could well take hours without going anywhere.

      Frankly. I'd rather just trust the "cowboy" admin to know his job well enough, and know whether he needs to unplug the servers because of a serious breach, or just let it be if it's just a DDOS, while the non-techies deal with their own domain of competence. There is _nothing_ a non-techie can add that's meaningful to that kind of an inherently techie decision. Just like you don't have the admins tell the company lawyers what to do, have the decency to not have the admin hang around and wait for the lawyers to tell him what to do. It's not only a better use of the admins' time, it's also a better use of the lawyers' time, who could be doing something that's a better use of _their_ skills in that time.

      I'll aggree, though, that the advice at step 1 seems to be dangerously content free. It's something which, although it may sound otherwise, actually noone ever actually did as such. Even if one "cowboy" admin did offer to contain the incident, it's not like someone let him deal with the _whole_ affair, including the HR, legal and financial aspects. Which is the domains they mention that you need on that team. More likely the "cowboy" just dealt with the servers, while the lawyer did his own job, the HR guy did his own, etc. I don't think (m)any people let the admin draft the press release too, for example. So the whole "don't let one 'cowboy' deal with it all" advice is basically like saying "don't try to fly on a broomstick off a bridge": you weren't actually planning to do that anyway, and it's not really giving you any insight you didn't already have.

      Finally, I don't know, maybe I'm just paranoid by trade, but the whole thing looks more like PR and a bit of an IT-for-PHBs magazine than anything actually serious about security or IT. It reads like little more than an advertisment for the three companies they mention, with a bit of a scare theme to make you contact them ASAP, than anything else. I'm also a tad cir
      • by Bender0x7D1 (536254) on Monday March 26, 2007 @08:24PM (#18495509)

        Depending on what you want to accomplish, pulling the plug or the network cable isn't something you want to do. If you want to catch the people who did it, instead of just minimize the damage, you need to approach this from a forensics POV. If you power-off the system, you lose everything that is stored in memory, which may be the only location where an important email, webpage or IP address is stored. Without this information it may not be possible to track-down the attacker. Yes, if they are communicating directly with the machine, you can get this info from a router or even the ISP but, if they are using some sort of anonymizer, you can't. Also, the rootkit (or whatever) may have a self-destruct built-in; can't communicate for 3 minutes, delete and overwrite everything. This would mean pulling the network cable will destroy any important information on your system. You might have backups for your data, but you don't for the attacker's information.

        Another important consideration is that powering down the system may prevent any information that's gathered from being admissible in court (U.S. jurisdiction). For example, can you guarantee that the email address on the disk is the attackers email, or is it from an email sent or received, or something else. Since you didn't shutdown properly, you may not be able to claim that the address is really attacker124@gmail.com, but might be attacker123, or attacker224, etc. - meaning no warrant and no charges. There are devices out there that you can plug into a USB port that will attempt to copy everything from RAM just so you have a complete record - then you can pull the plug, since that will prevent the hard drive from being written to. This preserves the information and it can be used as evidence. Whatever you do, don't do a normal shutdown.

        So, a reason you might want to wait for your lawyers and HR people is to determine if you need to worry about prosecution, or just make the problem go away. If they compromised an old desktop, or the web server in your DMZ, you might decide that it isn't worth it to pursue a conviction - lawyer's call - they know how expensive/difficult it will be. If the system holds personal information, the HR guy may need to help make the call. Ex. - Do you have to report a breach to all of your customers? Just employees? No reporting required, it isn't the info designated under the laws and/or regulations. Now, if it is a development server, you might want to leave it live if you suspect corporate espionage. You can bring in the feds and let them assess the situation. You might also want to buy time to work with you ISP to trace the attack. You actions should be done based on what the server contains and its value - which is why you have the CIO or CEO in the room.

        Now, a lot of this may not apply to your situation, but it isn't a black and white issue. There are a lot of things to consider. If you want some good information, I would recommend any of Brian Carrier's work - papers and his book. I have read a couple of his papers and they were really good and, while I haven't read his book, it has been recommended to me by others.

        • 1. I was assuming a serious breach. If it's a development server, frankly, it has no excuse to have any real data on it, or be accessible from the internet at all. So it would hardly qualify as a security breach, or be possible to breach.

          2. Depending on the zone where a server is, leaving it happily running without isolating it, can be an invitation for the problem to magnify. E.g., for most companies there is more than one server in a DMZ: if one is pwned, it can be used as a proxy to attack the others. E.
          • I was referring to a software development server instead of a web development server, but we can discuss any generic, internal server that has important information. I agree that they shouldn't be accessible from the Internet, but there are always ways in - if there were cost effective ways to 100% prevent it, everyone would use them. Maybe someone's home PC got compromised and the attacker can come in on a VPN - nothing "internal" is compromised, but everything is accesible.

            A qualified admin MAY be able

            • by Moraelin (679338)
              1. I wasn't talking about formatting and reinstalling. I was talking about pulling the network cable. There's a difference. Wiping away all evidence _is_ stupid, but containing the attack isn't.

              2. No offense, but your examples illustrate precisely what scares me.

              So basically a compromised server, where there's a breach in progress, should be left untouched because the boss's powerpoint presentation in on it? I.e., actual confidential data may be leaked, the problem can magnify, the company can open itself t
              • 1. It may have a self-destruct. A common tactic is to wipe/overwrite the drive after an attack. So, if it loses communication, it might assume it was discovered and kill your data.

                So, which is more important: Blowing a presentation or leaking private company information? Depends on what the presentation means and what the information is. The presentation may be backed-up but what if the server has special hardware or software on it? How do you have backups of that? If you have a single license for so

  • First thing to do is to pull the plug, and stop any further damage. After you're not connected to the Net, THEN you can figure out what happened and how to fix it
    • I'm not sure if you meant the RJ45 or the AC plug.

      In some cases, you may NOT want to pull the plug.

      Sometimes proper forensic evaluation requires both plugs remain attached until the experts are done.

      As the article said though, sometimes you have to balance continuing harm with the need to preserve the crime scene.
      • Well after running 'netstat -pav' its reasonably safe to pull the R45 plug since you have a record of any connections incoming.
        After that logs of stuff like 'ps aux' and syslog along with a backup of the hard drives allows you to pull the AC plug.
      • by hurfy (735314)
        Assuming, of course you are big enough for any experts to give a damn.

        But that was a given I suppose since we are assembling a team :(
        I would like to know what us cowboys should be doing....

        Preserve what? No one is gonna care who stole what from us. Hell, someone stole a few grand worth of actual merchendise and we had the who and the where and noone gave a damn then. Even if we decided to spend all our money to find out who...then what? Odds are they are offshore anyways and noone could do anything even if
        • Re: (Score:2, Interesting)

          by eli pabst (948845)
          Preserve what? No one is gonna care who stole what from us.

          You can preserve the evidence of how you got owned, like the means of entry, how privilege elevation was performed, what was done on the system. It's not uncommon for crackers to upload a binary, execute it so that it's running in memory and then delete the binary file, so if the bash_history was wiped you may never find any evidence it was even there unless you looked at the system while it was running. Figuring out how you were compromised ma
    • For the most-part (ie where you're a company, and investigators from the police, FBI or insurance company will be involved), you're dead right. Just kill the power.

      After that: DON'T TOUCH ANYTHING.

      You'll preserve everything on the hard-drive exactly as-is (this server IS logging everything, right?) without any shut-down scripts or anything else. Then the drives can be imaged in such a way as to be permissible as evidence. Sadly it's often the case in serious breaches that well-meaning, talented and curiou

  • by Todd Knarr (15451) on Monday March 26, 2007 @06:09PM (#18494061) Homepage

    I'd note that even if your company has a response plan, you may find it either completely useless or so general that it doesn't provide any help. Look at the article's point #1: it's almost nothing but "If $X, you may need $Y.". And it's far from complete. That's going to be a flaw in any security response plan: it's likely to not address the actual problem you face. Problems that you've thought of tend to get caught earlier before they turn into full-blown incidents, it's the ones nobody thought of that are most likely to bite you badly and it's exactly those that a plan won't cover. About the only part of the plan that'll be guaranteed to be useful is the part explaining what parts of the system are responsible for what and how to lock them down to preserve the evidence while you figure out where the breach is and what you need to do next. Beyond that you're into a twisty maze of little possibilities, all almost but not quite completely unlike each other, and what you need most isn't a plan but someone with enough Clue to analyze the situation and formulate a plan to fit it on the fly.

  • Switch to a paper only office, and an air-tube network.
  • by davidwr (791652) on Monday March 26, 2007 @06:11PM (#18494085) Homepage Journal
    All businesses should have contingency plans for all disasters.

    For most disasters, whether it's an IT disaster, a natural disaster, a non-natural physical disaster like a fire, a real or frivolous patent lawsuit, employee or company malfeasance, or what not, you need a plan.

    For "terminal" disasters, like a nuclear blast that kills all employees and destroys all company assets, folding up shop may be the right business plan. For small businesses, extreme disasters like car wreck that kills all the employees might also be terminal in a slightly less catastrophic way. In these cases, at least you can plan to sell your business or its assets to another entity, so your customers have continuity.

    Basically, divide your disasters into categories, and plan and insure accordingly:
    0) end of the world, big asteroid or global thermonuclear war
    1) major catastrophe, we are dead, forget about the customer, nuclear detonation event
    2) end of the company, save the customer, Enron
    3) end of the management team, save the company, MCI
    4) we can recover from this but it's gonna hurt a lot, Vonage(?)
    5) it's a flesh wound, CEO dies of heart attack
    6) mosquito bite, SCO sues IBM
    7) what? something happened? I didn't even notice, {if I had an example it would be #6}
    • When we get nuked I think its ok to just screw the business and work on saving your own ass.
    • "For "terminal" disasters, like a nuclear blast ... at least you can plan to sell your business or its assets to another entity, so your customers have continuity."
      Im gonna go out on a limb here and say if I have to deal with a nuclear blast, my customers are going to pretty low there on the list of things that im worried about the continuity of...

    • by toadlife (301863)
      For most companies, data breaches [attrition.org] usually fall into the #7 slot.
    • by Tom (822)

      All businesses should have contingency plans for all disasters.
      When it comes to IT security, the most popular contingency plan is "pretend it didn't happen".

      More often than not, applied internally as well as externally.
      • by sumdumass (711423)
        Well, you know, when the IDS is a series of letters for excessive bandwidth usage and RIAA take down notices, you gotta be willing to ignore some stuff.
  • Outsource (Score:3, Insightful)

    by DogDude (805747) on Monday March 26, 2007 @06:12PM (#18494101) Homepage
    If you're working for a company too small for a "Security response team", and chances are, you are, then you've got to consider outsourcing. If a security breach happened, then obviously you don't have the expertise in house to handle security in house, and you're just putting out fires after they happen. It's time to start looking to outsource whatever it was that was broken. In this day and age, unless you're doing something very, very custom, there's really little value to having in house web serving, email, etc.
  • You'll use this [darkreading.com] link. "Print buttons" are your friend, unless you really like 2 pages of content being spread over 10 pages.

    • the article is only 2, not 10 pages long to begin with.
      • I found that out when I read it.

        Still, it's pointless to have people who are "supposed to be tech smart" here posting news sites and aggregators that have 5-10 pages of stuff that 1 page would suffice.
  • Clearly (Score:5, Funny)

    by eviloverlordx (99809) on Monday March 26, 2007 @06:13PM (#18494111)
    The appropriate response is to shoot the lieutenant responsible for security. Then promote another ambitious, yet expendable underling to his/her place. Come on - this is Evil Overlord 101-level stuff.
  • by FirstTimeCaller (521493) on Monday March 26, 2007 @06:13PM (#18494113)

    It's been a long time (thankfully) since I've had to deal with this. But I'd echo the article about disconnecting from the net to eliminate further attacks. Then I'd remove the drive and save it for forensics -- replacements are cheap (I'm assuming a small business doesn't have expensive RAID setups). Assume that everything has been compromised and restore from a backup prior to the intrusion (hopefully you can tell when that was).

    Oh, and keep your clocks synchronized. This will help if you need to trace intrusions across systems.

  • Don't panic! (Score:5, Insightful)

    by mandelbr0t (1015855) on Monday March 26, 2007 @06:18PM (#18494163) Journal
    I've dealt with a couple security breaches in the past. It's never easy, and there's always that feeling of being violated as well. The important thing is to not lose your head about it, or you'll make mistakes that could lead to another or worse breach.

    First, find out the extent of the breach. Analyze your log files. Find out what time it happened. Find out who was logged in at the time, and find out any log messages from any system services that can help you figure out what the problem was. If you can't figure out what the scope of the breach was with a high level of confidence, then you have to assume the worst: the entire network is compromised.

    Second, salvage what you can. Again, be very careful about doing this. Hopefully you have a backup somewhere which would allow you to avoid or shorten this step as much as possible. In essence, do what you have to do to the compromised machine to avoid losing work, but always be concious of the fact that the machine is compromised, and may be transmitting or recording keylogs or other sensitive information. If possible, disconnect the compromised machines from the Internet and isolate it from the rest of your LAN.

    Third, plan for the future. How would this breach be avoided in the future? Was it an OS problem? If so, then maybe you need to install OpenBSD instead. Was it a problem with a particular package you were using? Choose a different package. Can you configure your firewall or server to prevent or limit the abuse that caused the problem in the first place (e.g. fail2ban to deal with SSH phishing attacks) or install monitoring software to alert you of a problem (e.g. an IDS like Snort)? Do your users need further training? Does your password policy allow weak passwords? Etc.

    Finally, take a deep breath. Unless you've been totally negligent in your job, there wasn't much you could do to prevent it. Don't worry about the fact that you don't have enough to go to the police; most Network Administrators don't have the hardware, training or certification to present evidence in a courtroom anyway. If you can go to the cops, then bully for you! Make that black-hat asshole pay!
  • by FMota91 (1050752) on Monday March 26, 2007 @06:19PM (#18494179)
    Windows XP: What's security?
    Windows Vista: This wouldn't happen to me anyway, I'm the Most Secure OS (tm)!
    Mac OS X: I never get any viruses!
    GNU/Linux: Me neither!
    Windows Vista User Access Control: You are entering a conversation with flaming probability 89%. Cancel or Allow?
    Windows Vista: [to Vista UAC] Allow. [to the others] That's because nobody uses you!
    GNU/Linux: Oh yeah...
    Mac OS X: That's because only elite people use Mac OS X. Because you're not worth them.
    GNU/Linux: Wait! Windows Vista, you lie! Lot's of people from all around the world use me! In fact, they even improve me! That's because we believe that...
    Mac OS X and Windows Vista: [at the same time] Shut up Linux.
    Windows Vista: [to Mac OS X] But anyway, even if there were a "Security Breach", it's not like they'd be able to mess anything up!
    Mac OS X: That's because it's impossible to do anything in Vista.
    Windows Vista User Access Control: [to Vista] You are coming to a sad realization... Cancel or Allow?

    NB: the views or opinions expressed by any of the characters do not necessarily resemble the views or opinions of the author.
    • OpenBSD (Score:4, Funny)

      by davidwr (791652) on Monday March 26, 2007 @06:28PM (#18494263) Homepage Journal
      OpenBSD: [walks into room, looks around, walks out, shaking his head not understanding why everyone can't be as secure as he is]
  • Got done.... (Score:5, Informative)

    by Creepy Crawler (680178) on Monday March 26, 2007 @06:24PM (#18494225)
    I got done reading this, and it's pretty dumb.

    "If you're a big company, you already have a security team. If not, hire one." DOH!

    That smacks me of the same kind of response from slashdot about legal advice... "Im being sued by the RIAA, should I ignore it?"

    Still, why not gander around and see what the the real security experts and such say about such matters:

    The Coroners Toolkit [fish2.com] Tools for Unix

    Nagios detection suite [nagios.org]

    Honeypots for 'sticking hackers' [honeynet.org]

    And there's the wonderful tools in the Linux kernel for bridges and such that can be made to monitor data as if there was no computer there at all. Also, PF in FreeBSD can route and filter based on much more criteria than Linux netfilter can (like via OS).

    You should have a secure layout of your network along with a respectable sensor network. The Sensornet should be separate from the general network.

    If you already work in IT, these things should be obvious, as it is the similar measures required for data recovery on non-hack problems.
  • "external consultants or forensics experts -- should be selected prior to an event, experts say."

    What a shocker...
  • by Anonymous Coward
    It boggles me that so many people come up with so many "solutions" yet hardly anyone comes up with the really important step to take: you backup your data, wipe the HD clean and re-install your OS. No matter what you use; be it Linux, Solaris, BSD.
  • I am curious to how many people actually go the next step to get the bad guy caught and how successful they are with it? It seems like its a tough battle to get the identity of the person behind an IP.
    • by mandelbr0t (1015855) on Monday March 26, 2007 @06:48PM (#18494517) Journal
      I've considered it, but there's a lot of barriers. First, you need enough evidence for a subpoena. That means that the chain of custody has to be preserved, and the crime scene needs to be secured by the police. Usually that means giving the compromised machines, relevant logs from monitoring equipment, etc. over to Law Enforcement for an indeterminate amount of time. I know I can't live without my servers for that long.

      You need to get the subpoena to identify the person behind the attack. That assumes that your evidence actually points to a specific suspect. Unless your attacker was a complete moron, or your network logs are incredibly voluminous, that's not very likely. Once the subpoena is served and you've got your suspect and laid charges, you need to present evidence. That requires an expert witness. If you're lucky, YOU are the expert witness, but there's training and certification involved in that process. Otherwise, you get to hire an expert witness, and that won't be cheap. Your opponent will probably hire an opposing expert, just to confuse everybody.

      Overall, I'd say that chances of success are incredibly low. Legal fees will be very high, and you have to turn over a fair chunk of your network assets to Law Enforcement. Basically, if you aren't really, really sure that you've got your man, it's really not worth the time and effort to find out who it was. That effort is much better spent allowing you to sleep at night knowing that people aren't getting in, IMO.
      • First, you need enough evidence for a subpoena.

        No, first you need to figure out a way to make the RIAA/MPAA think your attacker is hosting vast amounts of pirated music and movies. Let *them* get the subpoenas and harass your attacker with frivolous lawsuits...

  • by Q-Branch (554342)
    Just patch a socket. Problem solved. I learned that watching 24.
  • by thewils (463314) on Monday March 26, 2007 @06:40PM (#18494421) Journal
    It was an open FTP server. Some kind soul put about 14Gb of movies on one of our servers, then we noticed the hole (mainly because of the space) and shut down access to that server.

    So in our case the response was:

    1. Stop access.
    2. Buy beer and popcorn
    3. Watch movies.

    • Re: (Score:3, Funny)

      by GiovanniZero (1006365)
      ah, your response is eerily similar to cops with drug raids.
    • Let's assess your response step by step.

      1. Assemble an incident response team.
      Gather the buddies round the terminal, see what we got here.

      2. Assess the initial damage and the risk for more.
      You measured the damage, all 14GB of it. In assessing the risk for more of this damage, you noted that no ftp write access had been tried in a while, concluding that the risk was relatively low.

      3. Develop a notification plan.
      You sent an email-to-all that there's going to be a movie night, cancel your dates, postpone dinne
    • If you're not ultra concerned about the server, you can have some fun. One technique is replace the executables with similarly sized programs which do something slightly malicious, such as alter their internet settings to take them offline until they figure out how to fix it. Or corrupt a couple bytes in the file with a hex editor. Usually it seems like people just run irc bots on compromised systems. What they don't realize is that this gives you a method of determining channel passwords, bot passwords
  • Just close your eyes, count to ten, then start shouting "Serenity now" over and over again until the problem passes you by. :)
  • and kiss your ass goodbye!
  • Lift off and nuke the site from orbit.

    It's the only way to be sure.

  • Easy... (Score:5, Funny)

    by andreMA (643885) on Monday March 26, 2007 @07:10PM (#18494755)
    When in confusion
    or in doubt
    Run in circles
    scream and shout.

    And yeah, pull the ethernet cables out.
  • easy... (Score:2, Insightful)

    by trouser (149900)
    Burn the place to the ground, kill everyone, start again.
  • My version: (Score:5, Informative)

    by Alex Belits (437) * on Monday March 26, 2007 @09:27PM (#18496095) Homepage
    1. Keep the suits and incompetent people the hell out!

    Once a compromise happened, there is no time to listen to lawyers or marketing executives. If they have anything to say, they would write a document where they list all recommendations they can care about -- for example, how "This site is pwn3d" web page is supposed to look like, whether it is a good thing to send all users a letter "please cancel your credit card", or what information can be released to authorities. If they didn't do that already, let them write those things while sysadmins are working.

    This, of course, means that if there is only one sysadmin competent enough to investigate and fix the problem, then he would have to work on it alone.

    2. Shut it down and investigate changes made by the attackers.

    Before doing any investigation or recovery, shut the compromised and potentially compromised devices down. No malicious code should remain running. Whatever services should remain, must run in the minimal mode on separate hardware. For example, keep email running on a newly installed box. All investigation should be done in a clean environment -- drives moved to dedicated "clean" machines, or original servers booted from clean images (CD, PXE, replacement drives) on a private subnet, not accessible to anyone but people involved in incident response. Make full images of compromised hosts' storage whenever possible.

    Backups are your friend. IDS logs are, too, but make sure that your IDS isn't compromised, and actually recorded something meaningful.

    3. Don't worry about the person who originated the the attack.

    Find its results and, if possible, method. Likely there will be at least one person within the company (malicious or more likely negligent) and at least one outside. Screw them both, they don't have access to your network anymore because it's off.

    4. Immediately restore known-clean backups, perform audit on potentially compromised data and update the systems.

    Backup is "known-clean" if investigation shown that it is from a state before the attack and does not contain vulnerable versions of software or compromised authentication information that allowed attack to happen. Usually some data has to be restored from a compromised system because it's more recent than backup (or because you are an idiot and forgot to back it up). Audits are supposed to be painful. Once data is in place, update software and configuration. Erase all compromised authentication keys and tokens.

    5. Document the process.

    I mean, technical details.

    6. Tell everyone that they are screwed.

    Explain to every office drone that they are going to get new passwords. They won't like it, so keep your LART ready.

    Oh, btw:
    http://abelits.livejournal.com/30214.html [livejournal.com]
    http://abelits.livejournal.com/30681.html [livejournal.com]
    http://abelits.livejournal.com/30872.html [livejournal.com]
    • by dkleinsc (563838)
      You forgot a step just before number 6: Get your resume ready

      Remember, when it hits the fan, it's going to be propelled down the hierarchy, even if the real problem was that some executive wanted to cut security funds.
  • story (Score:5, Interesting)

    by 18769 (720646) on Monday March 26, 2007 @10:22PM (#18496507)
    I'm just a grad student, and one day, I installed something (I think it might've been an nfs server) without firewalling it (I did some sort of thing which had the deamon reject connections from outside my subnet). I was hacked. Funny thing is, they went straight from my machine to my roommate's, an old 486 which was also a webserver. From my roommate's machine, the hacker served a rootkit (cleverly named "..." in the root html directory).

    Enter the FBI, who showed up in my roomate's lab asking about his computer (amoung other things). Picture yourself a grad student answering his lab door to find men in suits (an uncommon experience) who say they're part of the FBI (also uncommon), and mean it (still less common). After some questions, it was hesitantly established that my roomate was not the hacker serving root kits from his home computer.

    From there, the FBI (with our permission) bugged our appartment. They put a "tap" in our appartment, which consistend of a special switch and a *very* loud windows machine that sat on our internet connection listening for hacker activity. The installation of the tap involved 7 FBI agents, none of which new nearly as much as my roomate about networking (that the broadcast ping couldn't get through their special switch with the word "tap" on it was a real mystery). Neadless to say, I didn't fool around with bittorrent or the like durring that time.

    After a month or two, they caught the hacker (who was sweedish, apparently), and eventaully prosecuted him successfully.

    Point is: sometimes it is useful to not reinstall immediately when hacked -- it can result in a good story :)

  • by BlueTrin (683373) on Tuesday March 27, 2007 @03:09AM (#18498363) Homepage Journal
    Create an account on this website [monster.com] ?
  • #1 advise (Score:5, Insightful)

    by Tom (822) on Tuesday March 27, 2007 @03:49AM (#18498523) Homepage Journal
    Disclaimer: I've actually written the security policy in place at my company, and I used to be the guy responsible for security before my last career move.

    My advise to sysadmins who notice a breach is this:
    Take your hands off the damn keyboard. Don't do anything unless you are 150% certain that you can see all possible consequences of your action. Call the IRT if you have one.
    If there's nobody to call and you have to act right now, pull the power plug on the machine, then call the experts. Don't power the machine up again under any circumstances. If you want to look at the harddrive, make a copy first and mount it read-only in a different machine.

    Why? Because back in the days when I was, err... looking around inside machines not my own, one of the things everyone I knew did was put in some scripts, tools, something, that'd wipe the logs or even the machine if my shell gets killed or the machine shut down or rebooted.

    TFA assumes that you learn of the incident long after it has happened. Many incidents in real life are being noticed while they are going on, no matter if it's a remote access or your machine running an FTP server that wasn't there last month. That FTP server is almost certainly patched, and one of the things it might do is destroy evidence if you kill it. There might be an invisible process watching it to wipe evidence if you kill -9 it. Heck, /sbin/kill could've been replaced by a trojan and not do what you expect it to do. Even /sbin/init is suspect. Your kernel, boot record, on some machines even your BIOS has possibly been manipulated.
  • Mostly old news (Score:2, Informative)

    by neilbaby (53319)
    At the risk of tooting my own horn, I blogged [bea.com] about similar material about a week before the Dark Reading publication. My blog focused more on the PR foul-ups that companies tend to make and ways to prevent those foul-ups rather than the technical response. It was based off of a recent Google vulnerability that got publicly posted as ?revenge? by the vulnerability discover who was unhappy about having not gotten enough credit for previously reported Google vulnerabilities. Neil Smithline BEA WebLogic Secur
  • As the guy who wrote this story, just wanted to say thanks to all posters for some excellent discussion. Most of the criticism has been both valid and useful, and we'll try to keep some of these comments in mind for future stories. I also offer a special note of thanks to those who offered extra insight -- I'm the first to concede that a short story like this doesn't cover all the angles on a complex subject like this. Also a really big thanks to those who flamed the critics on the story's behalf.:) If you
    • by Alex Belits (437) *
      The fact that a bunch of people on a message boards both supported and opposed you does not make you any less wrong and unqualified to give advice about incident response.

news: gotcha

Working...