Cleaning Up the Mess After a Major Hack Attack 100
Hugh Pickens writes "Kevin Mandia has spent his entire career cleaning up problems much like the recent breach at Stratfor where Anonymous defaced Stratfor's Web site, published over 50,000 of its customers' credit card numbers online and have threatened to release a trove of 3.3 million e-mails, putting Stratfor is in the position of trying to recover from a potentially devastating attack without knowing whether the worst is over. Mandia, who has responded to breaches, extortion attacks and economic espionage campaigns at 22 companies in the Fortune 100 in the last two years and has told Congress that if an advanced attacker targets your company then a breach is inevitable (PDF), calls the first hour he spends with companies 'upchuck hour' as he asks for firewall logs, web logs, and emails to quickly determine the 'fingerprint' of the intrusion and its scope. The first thing a forensics team will do is try to get the hackers off the company's network, which entails simultaneously plugging any security holes, removing any back doors into the company's network that the intruders might have installed, and changing all the company's passwords. 'This is something most people fail at. It's like removing cancer. You have to remove it all at once. If you only remove the cancer in your leg, but you have it in your arm, you might as well have not had the operation on your leg.' In the case of Stratfor, hackers have taken to Twitter to announce that they plan to release more Stratfor data over the next several days, offering a ray of hope — experts say the most dangerous breaches are the quiet ones that leave no trace."
Re:FIrst Post (Score:4, Interesting)
experts say the most dangerous breaches are the quiet ones that leave no trace.
You would not have known.
In fact, security experts would like that to be your last thought before you go to sleep at night, and your first thought when you wake up, and uppermost in your mind when they pad your bill with zeroes.
Clean up? Start fresh (Score:3, Insightful)
It's not a huge deal here anyway - because this lot have a high profile everyone forgets how small they really are. Your local newspaper probably has a bigger operation and a hell of a lot more subscribers.
Re:Clean up? Start fresh (Score:4, Informative)
Imagine that you have 1000 employees. Every workstation, every server, every switch, every usb-stick, every external drive could hold the seed to restoring hacker control on your network. You'd have to wipe all of them before allowing them to reconnect to the network.
Then, a week from now, someone asks IT for a file from the off-line backups, and your network is owned again.
Re: (Score:2)
I really thought putting the advice to prevent this in the title would be obvious enough. Maybe bold and in capitals with a BLINK tag might help next time :(
Re:Clean up? Start fresh (Score:5, Interesting)
Re:Clean up? Start fresh (Score:5, Insightful)
Not a problem here. we simply re store the workstation boot image from the creation CD and run all the updates on it.
Thumb drives, not a problem, thumb drives dont work here.
as for switches, I can update ios on every switch in 60 seconds. not a hard thing to do.
as for the "backups" problem. I have yet to see a hacker that can infect a machine using an odf file, I'm not backing up ANY executables.
Honestly I can do a complete wipe and restore in under 5 days for a company that has 1000 employees and 20 servers. IF the IT department was set up and run by competent people.
If it's a typical cluster-turd... far far longer.
Re: (Score:2)
> as for the "backups" problem. I have yet to see a hacker that can infect a machine using an odf file
http://www.openoffice.org/security/bulletin.html [openoffice.org]
Although I'll admit, never seen any of those in a real use-case.
Re: (Score:2)
Have you considered the possibility that you have insufficient experience in the field?
"I'm not backing up ANY executables."
What about the executable components that can be embedded in the ODF files you are so happily backing up? Are you deliberately not backing up emails? If so, your backups are useless. If you are backing up emails, then you cannot be sure you're not backing up executables.
Your whole stance looks like you have no unde
Re: (Score:3)
Your whole stance looks like you have no understanding of the problems that can be faced.
Why assume the worst? More likely he wasn't inclined to go into that level of detail here.
If he's already going so far as to prevent the use of USB flash drives isn't it likely that email attachments are handled in a similarly aggressive manner (e.g. executables automatically removed, remaining attachments quarantined, etc.)? Workstation backups needn't include email; email belongs on email servers local copies are just a cache.
Re: (Score:1)
I love your assumption that the email servers haven't been compromised. That's a great one to save time in the restoration effort, I agree, but please don't waste your time applying for an IT role anywhere near where I work.
Re: (Score:1)
Re: (Score:2)
I guarantee my email servers are not compromised, they are sitting OFF in a storage room for the past 3 years. we switched to Gmail for Exchange hosting 3 years ago to get rid of having to manage the huge turd that Exchange server is.
Re:Clean up? Start fresh (Score:5, Insightful)
Honestly I can do a complete wipe and restore in under 5 days for a company that has 1000 employees and 20 servers.
That's not too bad. But of course any machine that's not been wiped and restored can not be allowed on the network. And for the employees that means up to five days of not being able to do much. That's a long time to wait.
Re: (Score:2)
Hand out a custom (or even standard) version of this for emergencies with an instruction sheet tucked into the CD cover.
Your taxes paid for it:
http://www.spi.dod.mil/lipose.htm [dod.mil]
Re: (Score:1)
as for switches, I can update ios on every switch in 60 seconds. not a hard thing to do.
But how do you know the update was actually applied, and it wasn't rogue firmware falsely telling you it succeeded?
Re:Clean up? Start fresh (Score:4, Insightful)
> Imagine that you have 1000 employees. Every workstation, every server, every switch, every usb-stick, every external drive could hold the seed to restoring hacker control on your network. You'd have to wipe all of them before allowing them to reconnect to the network.
I wish people would remember this when they claim company's estimates of damage from a cracked system are excessive. You can bring an entire company to a standstill for an extended period of time by requiring (unless as a customer you're just fine with them taking risks with your data?) multiple critical systems to be isolated and rebuilt from scratch at the same time, even if there's no clear damage done, because you have no other way of verifying they're clear.
In a high security environment, destroying the physical machines to be sure (tampered firmware, stuff hidden in bad blocks on the hard drive, or who knows what else) is probably a sensible move.
Re: (Score:2)
Re:Clean up? Start fresh (Score:5, Interesting)
Of course, this will never happen.
Then it's time to go through all backup media and sanitize it, since of course a potential future restore could re-initiate the breach.
Of course, this will never happen.
Meanwhile, forensic work needs to be done to figure out what the vector(s) was/were for this incident. It's not enough to just identify and deal with those, however; they need to be studied in context in order to achieve an understanding of what additional, latent vectors exist that could be used.
Of course, this will never happen.
And then it's time for a very pointed session with a copy of Marcus Ranum's "Six Dumbest Ideas in Computer Security", because chances are pretty high that this organization used all six.
Of course, this -- especially this -- will never happen.
Re:Clean up? Start fresh (Score:5, Insightful)
The trick is to recognize that you cannot make these mistakes. Period. No matter who you have to run over, who you have to piss off, who you have to overrule, who you have to upset, no matter what. You have to be, and yes I am, an arrogant bastard. Because the moment you compromise, you're doomed. We've seen it over and over and over and over again, we're seeing it again today, we'll see it again tomorrow. Every single data breach incident I've ever read about included at least one of those six mistakes, and most of them included several. Yet incompetent, weak-willed IT people insist on making them because "we've always done it this way" or "that can't work!" or "but it would break..." or for a thousand other reasons...none of which matter. (What good is having a spiffy computing environment if it's not secure?)
The problem isn't that we don't know what to do. We do. The problem is lack of will to do it.
Re:Clean up? Start fresh (Score:4, Insightful)
Clean installs on everything, new passwords, and don't trust anything executable that has been on the compromised machines anywhere near the time it was hacked. It's not a huge deal here anyway - because this lot have a high profile everyone forgets how small they really are. Your local newspaper probably has a bigger operation and a hell of a lot more subscribers.
I seriously doubt your local newspaper has more money involved - or any local newspaper. Maybe some of the national broadsheets - but that's a moot point.
Cleanups aren't complicated - but fixes are - they just sound simple. And most commonly people seem to believe they are the same thing - I contend that they're not.
In my experience these things happen again and again to the same companies (though the majority put a lot of effort into keeping it secret). Not the same dog each time, but definitely the same leg action.
I've done a bit of due diligence on companies, listened in on workers at lunch, chatted to ex-staff, and hired investigators - and I've found few that are as clean as presented - it's like buying a pub where the bartenders or staff don't dip into the till, or regulars (and staff) have never dealt in drugs (rare as hen's teeth).
I'm not talking about defending against attackers - and I don't dispute that a determined, well resourced, intelligent attack will always succeed if time permits (it's like robbing armoured cash vans really - or so I've heard). I'm talking about the things that make it easy for attackers - I believe that if you raise the bar enough - all the hurdlers don't get better - just a few of them (and when you're robbed you're robbed, so number of occurrences is important)
What interests me is why there's always talk of plugging gaps and fixing procedures - but never any mention of fixing the primary problem. The primary problem being institutional psychology. Like storing your beer on the nature strip it having it stolen (surprise - people want your beer). Then "cleaning up" by making sure all liquor is secured inside the premises, and "fixing" the problem by telling people to store their beer in the fridge and lecturing them on physical security. It overlooks the possibility that only an untrustworthy idiot would put beer on the nature strip in the first place - and even if they don't put it on the nature strip again they will probably lose a house key, or leave a window open.
*1 I don't believe lazy, stupid staff change if you send them to motivation and inspiration seminars either, certainly I've seen no evidence to support it.
I'm working on a theory that dumb travels downward - I call it "The Argument from Moron Motion"
Re: (Score:2)
Which means more resources then doesn't it?
Then again, how much was a subscription with these guys? Remember not all the compromised cards are current accounts, so it adds up to something a lot smaller than the trumpet they are blowing. It's beginning to look like a small group that didn't have the resources to get a temporary hosted web presence going by now instead of a larger group that didn't have the competence to get something going.
Re: (Score:2)
Which means more resources then doesn't it? Then again, how much was a subscription with these guys? Remember not all the compromised cards are current accounts, so it adds up to something a lot smaller than the trumpet they are blowing. It's beginning to look like a small group that didn't have the resources to get a temporary hosted web presence going by now instead of a larger group that didn't have the competence to get something going.
I meant to try and find out something about their holding and earnings - but forgot. A cursory search didn't show much I could confirm. Apparently they have 20 full-time staff, mostly just analysts, they are a major customer of Media Monitors (which takes a few bucks) - the estimate I got was that they probably spend close to 7 figures per annum there. I'd guess they'd be using the other big 6 - all of which are more expensive - then there's the collectors and clippers I don't know about (probably heaps).
Re: (Score:2)
I also should have written "capital city newspaper", which is what I really meant, instead of "local newspaper". Some people heard the "shadow CIA" line from some PR somewhere and assumed that this clipping service was a huge organisation of thousands instead of something smaller than places a lot of the readers would be working in.
Re: (Score:2)
I also should have written "capital city newspaper", which is what I really meant, instead of "local newspaper".
I was pretty impressed with what so few people did with so much contradictory, nebulous, information. If you're not familiar with their work try google a "site:" search for a list of their (formerly) public releases, then "cache:" to read one.
I asked around to get a few opinions on my guesstimates for mid-western newspapers (I already has some on the few Australian equivalents), it was worth the trouble if only to feed a few red herrings to a gossipy journalist "friend" (May-te!). Right now he's probably at
Nature strip? What's that? (Score:2)
Given the usage, it has to be a term that's fairly commonly known somewhere but I've never seen it.
Re: (Score:2)
http://en.wiktionary.org/wiki/nature_strip [wiktionary.org]
Australian term, meaning "An area of grass beside a roadway, possibly with a few trees or shrubs, lying in between the footpath part and the roadway proper". Basically: Fools store their beer in their front yard, and complain when people take them for free.
Re: (Score:2)
http://en.wiktionary.org/wiki/nature_strip [wiktionary.org]
Australian term, meaning "An area of grass beside a roadway, possibly with a few trees or shrubs, lying in between the footpath part and the roadway proper". Basically: Fools store their beer in their front yard, and complain when people take them for free.
Pretty much - except that your nature strip, unlike your front yard, is not your property - it belongs to the local council (for all intensive purposes) though you maintain it. It's where you put your garbage, and thing you don't want (like old furniture) for public collection.
Though people will sometimes complain if things on the nature strip are taken by the public (like out of their rented rubbish skip) the complaint has no legal standing (much like the bullshit legal sounding jargon people append to the
Re: (Score:3)
it's like buying a pub where the bartenders or staff don't dip into the till
Man, that's some 20th century thinking there. I don't know of a single tavern that doesn't have a camera pointed at the cash register.
Re: (Score:2)
it's like buying a pub where the bartenders or staff don't dip into the till
Man, that's some 20th century thinking there. I don't know of a single tavern that doesn't have a camera pointed at the cash register.
Like that makes any difference. Your faith in technology and your own opinion are both misplaced, and have led you to ignore reality. Next you'll be declaring no convenience store staff fiddle the till because of the security camera and pickpockets died with Dickens.
Re: (Score:2)
And, as usual... (Score:5, Insightful)
Re: (Score:3)
Re: (Score:1)
Re:And, as usual... (Score:5, Insightful)
Which is why I send the email 10 times with a receipt request. Boss is too stupid to turn off that feature, and I also get a reply from him saying, "PLEASE STOP EMAILING ME THIS!"
Never EVER trust your boss. he will burn you to save his own butt.
Re:And, as usual... (Score:5, Insightful)
Any job that requires a CYA email archive is not worth having.
Re: (Score:2)
Food, clothing, and shelter are nice to have.
Also, a CYA archive may protect you from one idiot in an otherwise good job.
Save it all.
Call Kevin Mandia for your security needs (Score:2)
Re: (Score:2)
I'm working on that right now - I'm on the SBA Information Security Task Force - determining what really are the best practices out there. It's an all but impossible task.
Uh huh. Yeah, sure. (Score:3)
"I wanted to be less sarcastic and cynical this year."
Tell us another one.
Government warnings?? (Score:4, Interesting)
I'm curious though. In the PDF Kevin Mandia states that 90% of private enterprises don't know their networks have been compromised until the government (DoD, etc) tell them. So, how does the government know that these companies are compromised ?
I mean, apart from seeing spammy emails coming out, or in the case of the spooks, them seeing information on another system somewhere that's obviously been "stolen" from a US bank or something, how would they know ?
What sort of things would have to happen for a company to get a "Hey, you have bad guys all over your network" visit from the government guys ?
Re:Government warnings?? (Score:4, Funny)
If the DoD sees an attack from pepsi.com, IT's pretty easy from that point to figure out that pepsi.com is compromised. Even an MCSE can figure that one out.
Re: (Score:2, Insightful)
You miss the point.
If someone has access to a corporate network, and is smart, they're not going to blow it by using that companies internet facing machines to start running portscans on DoD machines, well, not unless they are script-kiddy stupid.
So, the target network is breached surreptitiously and information is quietly pilfered....al la corporate espionage........how's the DoD ever going to know ?
They should say "The government tells 90% of the small subset that do something stupid like launch DDoS atta
Re:Government warnings?? (Score:4, Informative)
It's pretty simple - the attackers install backdoor trojans which phone home to various command-and-control (C2) servers. In some cases when the USG identifies a high-value (i.e. involved in corporate and/or government espionage) C2 in the U.S. they get a warrant to monitor all network traffic to and from that host at the upstream. Once you have netflow or pcap data you can pretty easily tell who the compromised companies are when you see their corporate firewall IP hitting the C2 at regular intervals.
Private-sector researchers do this as well sometimes, but you need cooperation from the upstream. Or in some cases, the attackers are sloppy enough to leave behind publicly-accessible server logs ala Shady RAT.
Re: (Score:2)
In the second paragraph on page 4, Mr. Mandia says that over 90% of the private enterprises don't know their networks have been compromised until they learned from the FBI, DoD, or some other third party (emphasis added). However, in the last paragraph of page 5, he says that over 90% of the breaches his company responds to are first detected by the government.
The "some other third party" phrase seems more realistic to me unless he deals mostly with government agencies themselves.
Paywall. (Score:1)
http://www.nytimes.com/2011/12/30/technology/hacker-attacks-like-stratfors-require-fast-response.html [nytimes.com]
quite breaches (Score:2)
Good job, guys... (Score:2)
More effort into tracking and capturing hackers. (Score:3)
What they are doing is about the same as saying. I don't like the rich so I will steel from the poor who has to pay him.
Ha Ha we will laugh at the company who didn't fix all their security patches in time and didn't block done that Zero Day vulnerability. Or in real life terms. It is the companies fault for not operating their business in an impenetrable fortress.
Re:More effort into tracking and capturing hackers (Score:4, Insightful)
Uhm no, mere vandals need to be cherished and promoted; those who work for the Chinese govt won't tell you something is amiss.
It is the companies' fault for not following basic security practices, especially if what they take taxpayers' money for is "intelligence".
Re: (Score:1)
It's not even that complicated. There's no sophisticated motive behind Anonymous other than simple vandalism. You see the behavior in small children who like to knock things down just to see them fall. People are looking for po
Serious Hackers don't leave viruses/rootkits. (Score:5, Interesting)
Like Kevin Mandia, I too clean up these messes professionally. Cleaning these things up starts with the data gathering and analysis, virus scans, offline analysis - and more that are not mentioned.
The MOST important thing that ANY admin should know is that the true professional hackers do not use rootkits. They will use exploits to gain their foothold, but rather than install a rootkit, they will install remote network admin utilities, such as Dameware NT utilities (old), or more recently I've seen LabTech Software.
This software is great for Managed Service Providers - it also is a dream come true for cyber-criminals as it provides a backdoor into networks using signed code that will not appear on any antivirus, anti-malware or anti-rootkit scan. It can sit dormant for years, get backed up, and restored. Even if you do run anti-virus scans on your backups prior to restoring them - as one commenter stated above - it would be of no use.
So, when I am gathering the data dump, what I do is look for ALL network management tools, and I have created scripts that search for these.
*****
Google this: C:\WINDOWS\LTSVC\LTSVC.exe Hijackthis
You will find examples of people who have run Hijackthis on their computer and posted the log online - the common complaint is that they keep getting reinfected and cannot figure out how. They've run {insert virus tools here} a number of times and cannot figure it out. They usually resort to reinstalling the OS.
*****
Anyhow - gathering up all the logs from every device on the network, linking how they went from machine-to-machine, enumerating lists of installed software on each machine, and also performing offline analysis of drives, searching for any file/directory modifications based upon time stamp. It is FAR more involved, but it is the only way to enumerate the intrusion.
Removal must be done all at once. Either cut the network access of all the devices, then remove, or write a custom removal script and schedule it as a task to have everything be done at precisely the same moment.
I then have custom IDS signatures that look for any unauthorized Remote Management & Monitoring software.
Re: (Score:1, Troll)
Is enumerating only a subset of badness better than, or worse than, attempting to enumerate all badness? There might be an answer at the end of a google search for "enumerating badness"...
Re: (Score:3)
I am quite familiar with "enumerating badness".
This is only done as part of a clean-up effort.
If management tools are running where they should not be, I want to know about it.
"Enumerating badness" is precisely what is required when you are hunting down an intrusion. It is not the best policy to take when defending one.
The overarching lesson I've learned in all these years is that a secure network is a well managed network. If you do not actively manage your network - there are plenty of criminals that wou
Re: (Score:1)
Re: (Score:2)
No, I have not flown out to respond for many years now. I do travel in state and neighboring states quite a bit, but for the most part there are enough good people in each market with sufficient capabilities.
Re: (Score:1)
Unfortunately, Dameware NT utilities requires an open port on the firewall. Before Windows 2003's adoption, most servers had a public IP and were using no firewall or a software firewall. Thus, someone could exploit a Windows 2000 machine and th
Re: (Score:3)
Midnight_Falcon - did you not notice that I put the word (old) AFTER Dameware NT? It is less common now, but did the issue just go away? No, they have updated their software.
The point I wish to make, and have done, is that many hackers do not leave rootkits behind. They simply set themselves up as rogue network administrators within your network.
Re: (Score:2)
Re: (Score:2)
I'm not talking about hackers that run botnets - yes, they use rootkits. Never at any point have I stated that rootkits are obsolete or no longer used. What I am saying, and what I have said quite clearly, is that some criminals that want to obtain and maintain access to a corporate network are using remote network admin software. So, be on the lookout for it. That is all.
Re: (Score:2)
With respect, shouldn't that be the first step? The system is no longer a known quantity and has been under the control of somebody else so nothing at all can be trusted at all without examining it from outside of that system.
Of course if it's a compost heap of poorly documented interdependant things some of which no install media or configuration details are available it is very tempting to just try to find what has been broken, fix it, and trust that the collect
Re: (Score:2)
The first step of moving on, yes. I would agree - but due to many factors it is not practical for many users.
We're like Doctors in many respects - we can make all the recommendations we want, but the patient is going to do whatever they are going to do.
And referring to trustworthy backups - when the remote management software has been in place for x number of months, and it has been backed up, restoring the machine while doing virus scans profits you nothing i
NYT paid link (Score:1)
Its 2012, and only one goddamn place on the internet has a paywall, and thats the one story slashdot links to. It was bullshit when you did it in 1998, and its bullshit now. (and apparently, it will be bullshit when you are still doing it in 2022)
Re: (Score:2)
Linked:
http://www.nytimes.com/2011/12/30/technology/hacker-attacks-like-stratfors-require-fast-response.html?_r=1 [nytimes.com]
The most dangerous breaches are the quiet ones (Score:1)