Forgot your password?
typodupeerror
Security IT

Cleaning Up the Mess After a Major Hack Attack 100

Posted by Soulskill
from the cut-the-lines dept.
Hugh Pickens writes "Kevin Mandia has spent his entire career cleaning up problems much like the recent breach at Stratfor where Anonymous defaced Stratfor's Web site, published over 50,000 of its customers' credit card numbers online and have threatened to release a trove of 3.3 million e-mails, putting Stratfor is in the position of trying to recover from a potentially devastating attack without knowing whether the worst is over. Mandia, who has responded to breaches, extortion attacks and economic espionage campaigns at 22 companies in the Fortune 100 in the last two years and has told Congress that if an advanced attacker targets your company then a breach is inevitable (PDF), calls the first hour he spends with companies 'upchuck hour' as he asks for firewall logs, web logs, and emails to quickly determine the 'fingerprint' of the intrusion and its scope. The first thing a forensics team will do is try to get the hackers off the company's network, which entails simultaneously plugging any security holes, removing any back doors into the company's network that the intruders might have installed, and changing all the company's passwords. 'This is something most people fail at. It's like removing cancer. You have to remove it all at once. If you only remove the cancer in your leg, but you have it in your arm, you might as well have not had the operation on your leg.' In the case of Stratfor, hackers have taken to Twitter to announce that they plan to release more Stratfor data over the next several days, offering a ray of hope — experts say the most dangerous breaches are the quiet ones that leave no trace."
This discussion has been archived. No new comments can be posted.

Cleaning Up the Mess After a Major Hack Attack

Comments Filter:
  • by dbIII (701233) on Wednesday January 04, 2012 @09:22AM (#38584062)
    Clean installs on everything, new passwords, and don't trust anything executable that has been on the compromised machines anywhere near the time it was hacked.
    It's not a huge deal here anyway - because this lot have a high profile everyone forgets how small they really are. Your local newspaper probably has a bigger operation and a hell of a lot more subscribers.
    • by Anonymous Coward on Wednesday January 04, 2012 @09:30AM (#38584100)

      Imagine that you have 1000 employees. Every workstation, every server, every switch, every usb-stick, every external drive could hold the seed to restoring hacker control on your network. You'd have to wipe all of them before allowing them to reconnect to the network.

      Then, a week from now, someone asks IT for a file from the off-line backups, and your network is owned again.

      • by dbIII (701233)
        These people do not have anywhere near 1000 employees. Even if they did that just makes it easier because you can justify putting more people into action, bringing in as many people from the outside as needed and getting the job done.

        Then, a week from now, someone asks IT for a file from the off-line backups, and your network is owned again.

        I really thought putting the advice to prevent this in the title would be obvious enough. Maybe bold and in capitals with a BLINK tag might help next time :(

      • by Anonymous Coward on Wednesday January 04, 2012 @09:51AM (#38584198)
        I did work for a Fortune 100 company. We had Disaster Recovery Plans which involve that exact sort of thing. We rehearsed it once for an entire market we operated in, it took about 11 staff and 12 hours to do. We did it during the night during a weekend to reduce impact. Many of the systems were still operating during the rehearsal. We did phones, servers, workstations, restoration of images/backups, phones, network infrastructure, most HSMs (Excluding CAs) etc. As for viruses coming back from at-rest data backups, well, we virus scan them before it's used and nearly all of them are digitally signed (so tampering after the fact is harder) but I can't think of what else you can do. We can load our own heuristics and signatures onto our distributed IDSs though, so if we did find any type malware on our system, we can identify it and add it to the IDS and it would be picked up when/if it was on offline backups and when/if it's restored. The biggest weaknesses we identified was network bottlenecks, outdated documentation and outdated client software to handle the procedures.
      • by Lumpy (12016) on Wednesday January 04, 2012 @09:51AM (#38584210) Homepage

        Not a problem here. we simply re store the workstation boot image from the creation CD and run all the updates on it.
        Thumb drives, not a problem, thumb drives dont work here.

        as for switches, I can update ios on every switch in 60 seconds. not a hard thing to do.

        as for the "backups" problem. I have yet to see a hacker that can infect a machine using an odf file, I'm not backing up ANY executables.

        Honestly I can do a complete wipe and restore in under 5 days for a company that has 1000 employees and 20 servers. IF the IT department was set up and run by competent people.

        If it's a typical cluster-turd... far far longer.

        • by Xugumad (39311)

          > as for the "backups" problem. I have yet to see a hacker that can infect a machine using an odf file

          http://www.openoffice.org/security/bulletin.html [openoffice.org]

          Although I'll admit, never seen any of those in a real use-case.

        • by fatphil (181876)
          "I have yet to see a hacker that can infect a machine using an odf file"

          Have you considered the possibility that you have insufficient experience in the field?

          "I'm not backing up ANY executables."

          What about the executable components that can be embedded in the ODF files you are so happily backing up? Are you deliberately not backing up emails? If so, your backups are useless. If you are backing up emails, then you cannot be sure you're not backing up executables.

          Your whole stance looks like you have no unde
          • by extra88 (1003)

            Your whole stance looks like you have no understanding of the problems that can be faced.

            Why assume the worst? More likely he wasn't inclined to go into that level of detail here.

            If he's already going so far as to prevent the use of USB flash drives isn't it likely that email attachments are handled in a similarly aggressive manner (e.g. executables automatically removed, remaining attachments quarantined, etc.)? Workstation backups needn't include email; email belongs on email servers local copies are just a cache.

            • by fatphil (181876)
              Why assume the worst? Because this was a scenario pertaining to security, and your security has been proved to be insufficient a priori. That's why. Big forehead-slapping Duh!

              I love your assumption that the email servers haven't been compromised. That's a great one to save time in the restoration effort, I agree, but please don't waste your time applying for an IT role anywhere near where I work.
              • by Lumpy (12016)

                I guarantee my email servers are not compromised, they are sitting OFF in a storage room for the past 3 years. we switched to Gmail for Exchange hosting 3 years ago to get rid of having to manage the huge turd that Exchange server is.

        • by wvmarle (1070040) on Wednesday January 04, 2012 @11:56AM (#38585454)

          Honestly I can do a complete wipe and restore in under 5 days for a company that has 1000 employees and 20 servers.

          That's not too bad. But of course any machine that's not been wiped and restored can not be allowed on the network. And for the employees that means up to five days of not being able to do much. That's a long time to wait.

        • as for switches, I can update ios on every switch in 60 seconds. not a hard thing to do.

          But how do you know the update was actually applied, and it wasn't rogue firmware falsely telling you it succeeded?

      • by Xugumad (39311) on Wednesday January 04, 2012 @10:20AM (#38584420)

        > Imagine that you have 1000 employees. Every workstation, every server, every switch, every usb-stick, every external drive could hold the seed to restoring hacker control on your network. You'd have to wipe all of them before allowing them to reconnect to the network.

        I wish people would remember this when they claim company's estimates of damage from a cracked system are excessive. You can bring an entire company to a standstill for an extended period of time by requiring (unless as a customer you're just fine with them taking risks with your data?) multiple critical systems to be isolated and rebuilt from scratch at the same time, even if there's no clear damage done, because you have no other way of verifying they're clear.

        In a high security environment, destroying the physical machines to be sure (tampered firmware, stuff hidden in bad blocks on the hard drive, or who knows what else) is probably a sensible move.

      • by pclminion (145572)
        I assume that by "wiping" you include re-flashing all the firmware (including BIOS), in all of your devices including printers, routers, etc. It ain't as easy as wiping a bunch of drives. In fact, probably cheaper to throw all the equipment right in the recycle truck.
    • by Arrogant-Bastard (141720) on Wednesday January 04, 2012 @09:55AM (#38584240)
      This also includes clean installs on employee portable systems (laptops, PDAs, tablets, phones) as well as anything they have at home that can connect to the corporate network.

      Of course, this will never happen.

      Then it's time to go through all backup media and sanitize it, since of course a potential future restore could re-initiate the breach.

      Of course, this will never happen.

      Meanwhile, forensic work needs to be done to figure out what the vector(s) was/were for this incident. It's not enough to just identify and deal with those, however; they need to be studied in context in order to achieve an understanding of what additional, latent vectors exist that could be used.

      Of course, this will never happen.

      And then it's time for a very pointed session with a copy of Marcus Ranum's "Six Dumbest Ideas in Computer Security", because chances are pretty high that this organization used all six.

      Of course, this -- especially this -- will never happen.
    • by Demonoid-Penguin (1669014) on Wednesday January 04, 2012 @10:08AM (#38584322) Homepage

      Clean installs on everything, new passwords, and don't trust anything executable that has been on the compromised machines anywhere near the time it was hacked. It's not a huge deal here anyway - because this lot have a high profile everyone forgets how small they really are. Your local newspaper probably has a bigger operation and a hell of a lot more subscribers.

      I seriously doubt your local newspaper has more money involved - or any local newspaper. Maybe some of the national broadsheets - but that's a moot point.

      Cleanups aren't complicated - but fixes are - they just sound simple. And most commonly people seem to believe they are the same thing - I contend that they're not.

      In my experience these things happen again and again to the same companies (though the majority put a lot of effort into keeping it secret). Not the same dog each time, but definitely the same leg action.

      I've done a bit of due diligence on companies, listened in on workers at lunch, chatted to ex-staff, and hired investigators - and I've found few that are as clean as presented - it's like buying a pub where the bartenders or staff don't dip into the till, or regulars (and staff) have never dealt in drugs (rare as hen's teeth).

      I'm not talking about defending against attackers - and I don't dispute that a determined, well resourced, intelligent attack will always succeed if time permits (it's like robbing armoured cash vans really - or so I've heard). I'm talking about the things that make it easy for attackers - I believe that if you raise the bar enough - all the hurdlers don't get better - just a few of them (and when you're robbed you're robbed, so number of occurrences is important)

      What interests me is why there's always talk of plugging gaps and fixing procedures - but never any mention of fixing the primary problem. The primary problem being institutional psychology. Like storing your beer on the nature strip it having it stolen (surprise - people want your beer). Then "cleaning up" by making sure all liquor is secured inside the premises, and "fixing" the problem by telling people to store their beer in the fridge and lecturing them on physical security. It overlooks the possibility that only an untrustworthy idiot would put beer on the nature strip in the first place - and even if they don't put it on the nature strip again they will probably lose a house key, or leave a window open.

      • A. i don't know if that sort of stupidity can be "cured" (even with vigorous application of the stick of knowledge)*1.
      • B. I strongly suspect the problem starts at the top (board of directors) - but I'll allow for the possibility the shareholders (or the institutional representatives that vote on their behalf) play a part in the process.*2

      *1 I don't believe lazy, stupid staff change if you send them to motivation and inspiration seminars either, certainly I've seen no evidence to support it.

      I'm working on a theory that dumb travels downward - I call it "The Argument from Moron Motion"

      • by dbIII (701233)

        I seriously doubt your local newspaper has more money involved

        Which means more resources then doesn't it?
        Then again, how much was a subscription with these guys? Remember not all the compromised cards are current accounts, so it adds up to something a lot smaller than the trumpet they are blowing. It's beginning to look like a small group that didn't have the resources to get a temporary hosted web presence going by now instead of a larger group that didn't have the competence to get something going.

        • I seriously doubt your local newspaper has more money involved

          Which means more resources then doesn't it? Then again, how much was a subscription with these guys? Remember not all the compromised cards are current accounts, so it adds up to something a lot smaller than the trumpet they are blowing. It's beginning to look like a small group that didn't have the resources to get a temporary hosted web presence going by now instead of a larger group that didn't have the competence to get something going.

          I meant to try and find out something about their holding and earnings - but forgot. A cursory search didn't show much I could confirm. Apparently they have 20 full-time staff, mostly just analysts, they are a major customer of Media Monitors (which takes a few bucks) - the estimate I got was that they probably spend close to 7 figures per annum there. I'd guess they'd be using the other big 6 - all of which are more expensive - then there's the collectors and clippers I don't know about (probably heaps).

          • by dbIII (701233)
            It's good to see somebody has actually thought about it and looked around instead of assuming thousands of employees with desktop PCs like another poster above.
            I also should have written "capital city newspaper", which is what I really meant, instead of "local newspaper". Some people heard the "shadow CIA" line from some PR somewhere and assumed that this clipping service was a huge organisation of thousands instead of something smaller than places a lot of the readers would be working in.
            • I also should have written "capital city newspaper", which is what I really meant, instead of "local newspaper".

              I was pretty impressed with what so few people did with so much contradictory, nebulous, information. If you're not familiar with their work try google a "site:" search for a list of their (formerly) public releases, then "cache:" to read one.

              I asked around to get a few opinions on my guesstimates for mid-western newspapers (I already has some on the few Australian equivalents), it was worth the trouble if only to feed a few red herrings to a gossipy journalist "friend" (May-te!). Right now he's probably at

      • Given the usage, it has to be a term that's fairly commonly known somewhere but I've never seen it.

        • by gknoy (899301)

          http://en.wiktionary.org/wiki/nature_strip [wiktionary.org]

          Australian term, meaning "An area of grass beside a roadway, possibly with a few trees or shrubs, lying in between the footpath part and the roadway proper". Basically: Fools store their beer in their front yard, and complain when people take them for free.

          • http://en.wiktionary.org/wiki/nature_strip [wiktionary.org]

            Australian term, meaning "An area of grass beside a roadway, possibly with a few trees or shrubs, lying in between the footpath part and the roadway proper". Basically: Fools store their beer in their front yard, and complain when people take them for free.

            Pretty much - except that your nature strip, unlike your front yard, is not your property - it belongs to the local council (for all intensive purposes) though you maintain it. It's where you put your garbage, and thing you don't want (like old furniture) for public collection.

            Though people will sometimes complain if things on the nature strip are taken by the public (like out of their rented rubbish skip) the complaint has no legal standing (much like the bullshit legal sounding jargon people append to the

      • by mcgrew (92797) *

        it's like buying a pub where the bartenders or staff don't dip into the till

        Man, that's some 20th century thinking there. I don't know of a single tavern that doesn't have a camera pointed at the cash register.

        • it's like buying a pub where the bartenders or staff don't dip into the till

          Man, that's some 20th century thinking there. I don't know of a single tavern that doesn't have a camera pointed at the cash register.

          Like that makes any difference. Your faith in technology and your own opinion are both misplaced, and have led you to ignore reality. Next you'll be declaring no convenience store staff fiddle the till because of the security camera and pickpockets died with Dickens.

  • And, as usual... (Score:5, Insightful)

    by AngryDeuce (2205124) on Wednesday January 04, 2012 @09:28AM (#38584086)
    A bunch of people that had nothing to do with the breach will more than likely end up losing their jobs over it (often the same people that warn about these vulnerabilities beforehand), while the retards that caused the breach, either through their ineptitude or refusal to spend money on proper security, walk away unharmed.
  • In all seriousness, there really needs to be a court recognized standard for IT security due diligence. There are too many organizations doing their own thing or using "compensating controls" that only work in some auditors dream world.
    • I'm working on that right now - I'm on the SBA Information Security Task Force - determining what really are the best practices out there. It's an all but impossible task.

  • by Anonymous Coward on Wednesday January 04, 2012 @09:33AM (#38584116)

    I'm curious though. In the PDF Kevin Mandia states that 90% of private enterprises don't know their networks have been compromised until the government (DoD, etc) tell them. So, how does the government know that these companies are compromised ?
    I mean, apart from seeing spammy emails coming out, or in the case of the spooks, them seeing information on another system somewhere that's obviously been "stolen" from a US bank or something, how would they know ?
    What sort of things would have to happen for a company to get a "Hey, you have bad guys all over your network" visit from the government guys ?

    • by Lumpy (12016) on Wednesday January 04, 2012 @09:55AM (#38584236) Homepage

      If the DoD sees an attack from pepsi.com, IT's pretty easy from that point to figure out that pepsi.com is compromised. Even an MCSE can figure that one out.

      • Re: (Score:2, Insightful)

        by Anonymous Coward

        You miss the point.

        If someone has access to a corporate network, and is smart, they're not going to blow it by using that companies internet facing machines to start running portscans on DoD machines, well, not unless they are script-kiddy stupid.

        So, the target network is breached surreptitiously and information is quietly pilfered....al la corporate espionage........how's the DoD ever going to know ?

        They should say "The government tells 90% of the small subset that do something stupid like launch DDoS atta

    • by httptech (5553) on Wednesday January 04, 2012 @11:15AM (#38584978) Homepage

      It's pretty simple - the attackers install backdoor trojans which phone home to various command-and-control (C2) servers. In some cases when the USG identifies a high-value (i.e. involved in corporate and/or government espionage) C2 in the U.S. they get a warrant to monitor all network traffic to and from that host at the upstream. Once you have netflow or pcap data you can pretty easily tell who the compromised companies are when you see their corporate firewall IP hitting the C2 at regular intervals.

      Private-sector researchers do this as well sometimes, but you need cooperation from the upstream. Or in some cases, the attackers are sloppy enough to leave behind publicly-accessible server logs ala Shady RAT.

    • by datastew (529152)

      In the second paragraph on page 4, Mr. Mandia says that over 90% of the private enterprises don't know their networks have been compromised until they learned from the FBI, DoD, or some other third party (emphasis added). However, in the last paragraph of page 5, he says that over 90% of the breaches his company responds to are first detected by the government.

      The "some other third party" phrase seems more realistic to me unless he deals mostly with government agencies themselves.

  • I agree with the sentence "the most dangerous breaches are the quiet ones" the reason is you don't know you got hacked and you don't know what they got. When a hack is quite the hacker can come and go as they pleases and instead of getting in and getting out the hacker has time to explore and make more exploits and holes for themselves to get in and out on. The best and most dangerous hackers are the ones you don't know about or can't stop from getting in and out of your network. I remember my security i
  • Anonymous published 50,000 credit card numbers online... and just whose interest is this attack in?... O_o
  • by jellomizer (103300) on Wednesday January 04, 2012 @10:14AM (#38584366)
    That is why I think we really need to stop encouraging and supporting these criminal hackers and put more consolidated effort into finding them and stopping them.
    What they are doing is about the same as saying. I don't like the rich so I will steel from the poor who has to pay him.

    Ha Ha we will laugh at the company who didn't fix all their security patches in time and didn't block done that Zero Day vulnerability. Or in real life terms. It is the companies fault for not operating their business in an impenetrable fortress.
    • by KiloByte (825081) on Wednesday January 04, 2012 @10:48AM (#38584670)

      Uhm no, mere vandals need to be cherished and promoted; those who work for the Chinese govt won't tell you something is amiss.

      It is the companies' fault for not following basic security practices, especially if what they take taxpayers' money for is "intelligence".

    • That is why I think we really need to stop encouraging and supporting these criminal hackers and put more consolidated effort into finding them and stopping them. What they are doing is about the same as saying. I don't like the rich so I will steel from the poor who has to pay him.

      It's not even that complicated. There's no sophisticated motive behind Anonymous other than simple vandalism. You see the behavior in small children who like to knock things down just to see them fall. People are looking for po

  • by JRHelgeson (576325) on Wednesday January 04, 2012 @10:52AM (#38584714) Homepage Journal

    Like Kevin Mandia, I too clean up these messes professionally. Cleaning these things up starts with the data gathering and analysis, virus scans, offline analysis - and more that are not mentioned.

    The MOST important thing that ANY admin should know is that the true professional hackers do not use rootkits. They will use exploits to gain their foothold, but rather than install a rootkit, they will install remote network admin utilities, such as Dameware NT utilities (old), or more recently I've seen LabTech Software.

    From www.labtechsoftware.com
    IT Systems Management Software providing a leading remote monitoring and management (RMM) solution for Managed Service Providers (MSP) and IT...

    This software is great for Managed Service Providers - it also is a dream come true for cyber-criminals as it provides a backdoor into networks using signed code that will not appear on any antivirus, anti-malware or anti-rootkit scan. It can sit dormant for years, get backed up, and restored. Even if you do run anti-virus scans on your backups prior to restoring them - as one commenter stated above - it would be of no use.

    So, when I am gathering the data dump, what I do is look for ALL network management tools, and I have created scripts that search for these.
            *****
            Google this: C:\WINDOWS\LTSVC\LTSVC.exe Hijackthis
            You will find examples of people who have run Hijackthis on their computer and posted the log online - the common complaint is that they keep getting reinfected and cannot figure out how. They've run {insert virus tools here} a number of times and cannot figure it out. They usually resort to reinstalling the OS.
            *****
    Anyhow - gathering up all the logs from every device on the network, linking how they went from machine-to-machine, enumerating lists of installed software on each machine, and also performing offline analysis of drives, searching for any file/directory modifications based upon time stamp. It is FAR more involved, but it is the only way to enumerate the intrusion.

    Removal must be done all at once. Either cut the network access of all the devices, then remove, or write a custom removal script and schedule it as a task to have everything be done at precisely the same moment.

    I then have custom IDS signatures that look for any unauthorized Remote Management & Monitoring software.

    • Re: (Score:1, Troll)

      by fatphil (181876)
      "I then have custom IDS signatures that look for any unauthorized Remote Management & Monitoring software."

      Is enumerating only a subset of badness better than, or worse than, attempting to enumerate all badness? There might be an answer at the end of a google search for "enumerating badness"...
      • I am quite familiar with "enumerating badness".
        This is only done as part of a clean-up effort.
        If management tools are running where they should not be, I want to know about it.
        "Enumerating badness" is precisely what is required when you are hunting down an intrusion. It is not the best policy to take when defending one.

        The overarching lesson I've learned in all these years is that a secure network is a well managed network. If you do not actively manage your network - there are plenty of criminals that wou

    • Just out of curiosity, do you have to travel a lot with your job. I do Security Engineering now and have done IDS and Log monitoring in the past and was thinking that I would enjoy incident handling, but the thing that has kept me out of it was the 100% on call, get on a flight now to fly who knows where.
      • No, I have not flown out to respond for many years now. I do travel in state and neighboring states quite a bit, but for the most part there are enough good people in each market with sufficient capabilities.

    • This is very outdated knowledge unfortunately and I think the example is no longer relevant, and has been obsoleted. I'm also a security auditor, and Dameware NT utilities was a common installation on Windows NT, 2000 and some 2003 servers that were compromised.

      Unfortunately, Dameware NT utilities requires an open port on the firewall. Before Windows 2003's adoption, most servers had a public IP and were using no firewall or a software firewall. Thus, someone could exploit a Windows 2000 machine and th

      • Midnight_Falcon - did you not notice that I put the word (old) AFTER Dameware NT? It is less common now, but did the issue just go away? No, they have updated their software.

        The point I wish to make, and have done, is that many hackers do not leave rootkits behind. They simply set themselves up as rogue network administrators within your network.

        • I'd agree that some hackers don't leave rootkits and instead prefer to setup legitimate network access, use service accounts to get into the directory (LDAP/AD etc). Also, most remote access software has been changed/modified so that it's harder to use in an exploitative way -- look at Citrix GotoAssist or logmein -- hard to install those surreptitiously, or at least maintain them from there. That's why it's becoming less and less common to use legitimate software as an attack vector, along with lack of s
          • I'm not talking about hackers that run botnets - yes, they use rootkits. Never at any point have I stated that rootkits are obsolete or no longer used. What I am saying, and what I have said quite clearly, is that some criminals that want to obtain and maintain access to a corporate network are using remote network admin software. So, be on the lookout for it. That is all.

    • by dbIII (701233)

      They usually resort to reinstalling the OS.

      With respect, shouldn't that be the first step? The system is no longer a known quantity and has been under the control of somebody else so nothing at all can be trusted at all without examining it from outside of that system.
      Of course if it's a compost heap of poorly documented interdependant things some of which no install media or configuration details are available it is very tempting to just try to find what has been broken, fix it, and trust that the collect

      • With respect, shouldn't that be the first step?

        The first step of moving on, yes. I would agree - but due to many factors it is not practical for many users.

        We're like Doctors in many respects - we can make all the recommendations we want, but the patient is going to do whatever they are going to do.

        And referring to trustworthy backups - when the remote management software has been in place for x number of months, and it has been backed up, restoring the machine while doing virus scans profits you nothing i

  • by Anonymous Coward

    Its 2012, and only one goddamn place on the internet has a paywall, and thats the one story slashdot links to. It was bullshit when you did it in 1998, and its bullshit now. (and apparently, it will be bullshit when you are still doing it in 2022)

  • They could by all means be that. Or it could be that the "3 million +" emails that are being used as a threat are only the loud part of the breach, by the same logic therefore, the less dangerous part.

Possessions increase to fill the space available for their storage. -- Ryan

Working...