What to Do When Your Security is Breached 177
ancientribe writes "When you've got a full-blown security breach on your hands, what do you do? If you've been smart, you'll already have a computer security incident response team — and a plan — in place. But many companies are too resource-strapped to have a full-blown, fully-tested incident response strategy. DarkReading has some tips on what to do — and what not to do."
Do what the government does. (Score:4, Funny)
Re: (Score:2)
The problem is (Score:5, Insightful)
Re:The problem is (Score:4, Insightful)
I would further add, that they chose Microsoft because Microsoft promises lower TCO through lowered administrative (geek) needs.
I suppose that most Microsoft shops wouldn't even know if they were breached, because most breaches don't actually desctroy data, they just steal it.
Re:The problem is (Score:5, Interesting)
I suppose that most Microsoft shops wouldn't even know if they were breached, because most breaches don't actually desctroy data, they just steal it.
It's so much worse than that.
Back in my younger days at a summer tech job for a US school district, I found that an NT4 SQL server had been compromised a group of people. They were based out of France, I think, from what I could tell from the IP addresses, and had actually set themselves up quite nicely, with organized file structure and their own IRC and FTP server running on it. They were using it as a repository to store files and a few French movies. After I told the sysadmin in place at the time about it, I was stunned when he said, "Well, are they hurting anything?"
After some persuasion on my part, he rebuilt the server. Three times. After it kept getting hacked by the same people.
Re: (Score:2, Insightful)
Re: (Score:2)
Re:The problem is (Score:4, Interesting)
I know some IRC groups were the members get their company servers to provide dumps and bots. And of course non one ever knows it is going on.
So I'm going to guess that if they went through the trouble of hacking it three more times, it was probably an inside job to some extent.
Re: (Score:2)
So I'm going to guess that if they went through the trouble of hacking it three more times, it was probably an inside job to some extent.
I captured the traffic with a network sniffer. Specifically, the IRC traffic sent in cleartext, and they were all chatting in French. So unless they were local students routing traffic through French IP addresses and all speaking in French, I kinda assumed they were in France.
But the incompetent admin is probably more to blame. This all happened when there was no
Comment removed (Score:4, Funny)
Re: (Score:2)
In defense of Microsoft (I usually bash them) I will say the OS does have many features that are rarely deployed that can dramatically improve security. But here in is the problem, you can take someone from McDonald's on Monday and be a senior administrator by Friday and not even know these features exist. Because you work for $25K less per year, management loves this. Which is
What to Do When Your Security is Breached? (Score:2, Funny)
Dispatch the Tie Fighters (Score:5, Funny)
How about... (Score:2, Funny)
Re: (Score:3, Funny)
Then reverse the polarity FTW!
my plan (Score:5, Funny)
Professor: Yes I would, Kent.
I love these content-free articles (Score:5, Funny)
1. first, remove your hand from the burning stove.
2. use ice to cool your hand
3. seek medical attention.
wow. Thanks. I never would have figured any of that out on my own.
Re:I love these content-free articles (Score:5, Funny)
"To treat a minor burn, run cool water over the area of the burn or soak it in a cool water bath (not ice water). Keep the area submerged for at least 5 minutes."
http://www.nlm.nih.gov/medlineplus/ency/presentat
"Flush the burn with cool running water or apply cold- water compresses (a wet towel or handkerchief) until the pain lessens. Do not use ice or ice water, which can cause more damage to the tissues."
http://www.personalmd.com/healthtopics/crs/burn1.
*emphasis mine*
Re: (Score:2)
Re: (Score:2)
I used to work the ovens and the char broiler when doing my restaurant tour coming out of high school. This is just personal observation but a lot of times when I would get burnt, the only the best thing I could do is just put some gauze around it to try and shield the heat of the broiler and ovens from it. When I used ice or running water it would blister and I would have two more stages of dealing with it. One we
Re: (Score:2)
The voice of user experience has clearly told me that you should use Dry Ice to cool off your hand. This way, you don't have to worry about infections caused by something in the water...
Re: (Score:2)
1) If sterile water is unavailable, will non-sterile water work well enough ?
Yes, for some definition of well enough. It's better than not cooling the wound. But in all cases the burn should be treated with some kind of antiseptic before beeing dressed. Any third degree burn, or second degree of significant size should be treated by a medical professional. If in doubt get help.
2) Is non-sterile gauze cheaper than sterile gauze, & if so, where can it be purchased ?
Yes it is. It is commonly only us
Re: (Score:2)
You forgot
4. ???
5. Profit!
Re:I love these content-free articles (Score:5, Insightful)
1) You apparently shouldn't rely on what you 'figured out on your own'.
2) In addition to getting a plan for a security breach you should also look at getting some help with your first aid plan too.
Re: (Score:2)
Running water, not ice (Score:2)
Only thing worse than a hollow article is a wrong one.
I dunno, it's sorta... news to me (Score:5, Interesting)
For starters the advice to wait until the whole team is assembled, including the accountants, lawyers, etc, then holding meetings to determine your strategy, etc, before even unplugging the damn thing... dunno, it seems to me bordering on criminal. Yes, you don't want to let one lone cowboy handle it from end to end, but a trained admin could at the very least be able to unplug the computer from the network and isolate the damage before it goes any worse. Or know enough to decide if it has to be unplugged. But if he thinks it is, it should be step #1 not IIRC step #4 after you're done holding your meetings and informing the employees and having PR draft the vaguely worded announcement that tries to make it sound unimportant to your customers.
Waiting for the designated accountant, and the designated lawyer, and the HR guy, and God knows who else to arrive at the middle of the night and hold their meeting while a breach is in progress and someone is downloading your productive database, seems to me dumb to the extreme. To reuse your example, it's like saying you should keep your hand in the stove until you talked to your lawyer and your doctor and a designated family member, make sure you have a strategy, and only then pull the hand out. By that time, it could be burned to a crisp.
I mean, by the elder Gods, especially when you include such non-techies... surely you've seen these guys when they have to give you a spec for a program. If you wait for them to hold a meeting on such technical issues as "are we in aggreement that we need to unplug the server?", at least one goes into responsibility avoidance mode and refuses to be remembered as the one who took any decision, at least one goes into alpha-dog-pissing-on-everything-to-mark-his-territ ory mode, etc. It's a meeting that could well take hours without going anywhere.
Frankly. I'd rather just trust the "cowboy" admin to know his job well enough, and know whether he needs to unplug the servers because of a serious breach, or just let it be if it's just a DDOS, while the non-techies deal with their own domain of competence. There is _nothing_ a non-techie can add that's meaningful to that kind of an inherently techie decision. Just like you don't have the admins tell the company lawyers what to do, have the decency to not have the admin hang around and wait for the lawyers to tell him what to do. It's not only a better use of the admins' time, it's also a better use of the lawyers' time, who could be doing something that's a better use of _their_ skills in that time.
I'll aggree, though, that the advice at step 1 seems to be dangerously content free. It's something which, although it may sound otherwise, actually noone ever actually did as such. Even if one "cowboy" admin did offer to contain the incident, it's not like someone let him deal with the _whole_ affair, including the HR, legal and financial aspects. Which is the domains they mention that you need on that team. More likely the "cowboy" just dealt with the servers, while the lawyer did his own job, the HR guy did his own, etc. I don't think (m)any people let the admin draft the press release too, for example. So the whole "don't let one 'cowboy' deal with it all" advice is basically like saying "don't try to fly on a broomstick off a bridge": you weren't actually planning to do that anyway, and it's not really giving you any insight you didn't already have.
Finally, I don't know, maybe I'm just paranoid by trade, but the whole thing looks more like PR and a bit of an IT-for-PHBs magazine than anything actually serious about security or IT. It reads like little more than an advertisment for the three companies they mention, with a bit of a scare theme to make you contact them ASAP, than anything else. I'm also a tad cir
Re:I dunno, it's sorta... news to me (Score:5, Insightful)
Depending on what you want to accomplish, pulling the plug or the network cable isn't something you want to do. If you want to catch the people who did it, instead of just minimize the damage, you need to approach this from a forensics POV. If you power-off the system, you lose everything that is stored in memory, which may be the only location where an important email, webpage or IP address is stored. Without this information it may not be possible to track-down the attacker. Yes, if they are communicating directly with the machine, you can get this info from a router or even the ISP but, if they are using some sort of anonymizer, you can't. Also, the rootkit (or whatever) may have a self-destruct built-in; can't communicate for 3 minutes, delete and overwrite everything. This would mean pulling the network cable will destroy any important information on your system. You might have backups for your data, but you don't for the attacker's information.
Another important consideration is that powering down the system may prevent any information that's gathered from being admissible in court (U.S. jurisdiction). For example, can you guarantee that the email address on the disk is the attackers email, or is it from an email sent or received, or something else. Since you didn't shutdown properly, you may not be able to claim that the address is really attacker124@gmail.com, but might be attacker123, or attacker224, etc. - meaning no warrant and no charges. There are devices out there that you can plug into a USB port that will attempt to copy everything from RAM just so you have a complete record - then you can pull the plug, since that will prevent the hard drive from being written to. This preserves the information and it can be used as evidence. Whatever you do, don't do a normal shutdown.
So, a reason you might want to wait for your lawyers and HR people is to determine if you need to worry about prosecution, or just make the problem go away. If they compromised an old desktop, or the web server in your DMZ, you might decide that it isn't worth it to pursue a conviction - lawyer's call - they know how expensive/difficult it will be. If the system holds personal information, the HR guy may need to help make the call. Ex. - Do you have to report a breach to all of your customers? Just employees? No reporting required, it isn't the info designated under the laws and/or regulations. Now, if it is a development server, you might want to leave it live if you suspect corporate espionage. You can bring in the feds and let them assess the situation. You might also want to buy time to work with you ISP to trace the attack. You actions should be done based on what the server contains and its value - which is why you have the CIO or CEO in the room.
Now, a lot of this may not apply to your situation, but it isn't a black and white issue. There are a lot of things to consider. If you want some good information, I would recommend any of Brian Carrier's work - papers and his book. I have read a couple of his papers and they were really good and, while I haven't read his book, it has been recommended to me by others.
I was assuming a serious breach (Score:2)
2. Depending on the zone where a server is, leaving it happily running without isolating it, can be an invitation for the problem to magnify. E.g., for most companies there is more than one server in a DMZ: if one is pwned, it can be used as a proxy to attack the others. E.
Re: (Score:2)
I was referring to a software development server instead of a web development server, but we can discuss any generic, internal server that has important information. I agree that they shouldn't be accessible from the Internet, but there are always ways in - if there were cost effective ways to 100% prevent it, everyone would use them. Maybe someone's home PC got compromised and the attacker can come in on a VPN - nothing "internal" is compromised, but everything is accesible.
A qualified admin MAY be able
Re: (Score:2)
2. No offense, but your examples illustrate precisely what scares me.
So basically a compromised server, where there's a breach in progress, should be left untouched because the boss's powerpoint presentation in on it? I.e., actual confidential data may be leaked, the problem can magnify, the company can open itself t
Re: (Score:2)
1. It may have a self-destruct. A common tactic is to wipe/overwrite the drive after an attack. So, if it loses communication, it might assume it was discovered and kill your data.
So, which is more important: Blowing a presentation or leaking private company information? Depends on what the presentation means and what the information is. The presentation may be backed-up but what if the server has special hardware or software on it? How do you have backups of that? If you have a single license for so
Re: (Score:2)
I'm guessing you work either in a highly techno-centric company (on the scale of Yahoo, Google, Amazon, etc) or in a market sector that is so obscenely profitable that the higher-ups and/or the stockholders don't scream bloody murder every time someone from IT wants to spend money.
Either way, I call BS. Your scenario is a pipedream.
Pull the plug!! (Score:1)
Sometimes yes sometimes no (Score:2, Interesting)
In some cases, you may NOT want to pull the plug.
Sometimes proper forensic evaluation requires both plugs remain attached until the experts are done.
As the article said though, sometimes you have to balance continuing harm with the need to preserve the crime scene.
Re: (Score:2)
After that logs of stuff like 'ps aux' and syslog along with a backup of the hard drives allows you to pull the AC plug.
not necessarily (Score:4, Insightful)
Re: (Score:3, Interesting)
If you are big enough to have an Incident Response Team worth talking about (ie, more than the single IT guy), you should have seperate security analysis/reporting ability beyond what the box will report.
Re: (Score:2)
But that was a given I suppose since we are assembling a team
I would like to know what us cowboys should be doing....
Preserve what? No one is gonna care who stole what from us. Hell, someone stole a few grand worth of actual merchendise and we had the who and the where and noone gave a damn then. Even if we decided to spend all our money to find out who...then what? Odds are they are offshore anyways and noone could do anything even if
Re: (Score:2, Interesting)
You can preserve the evidence of how you got owned, like the means of entry, how privilege elevation was performed, what was done on the system. It's not uncommon for crackers to upload a binary, execute it so that it's running in memory and then delete the binary file, so if the bash_history was wiped you may never find any evidence it was even there unless you looked at the system while it was running. Figuring out how you were compromised ma
Re: (Score:2)
After that: DON'T TOUCH ANYTHING.
You'll preserve everything on the hard-drive exactly as-is (this server IS logging everything, right?) without any shut-down scripts or anything else. Then the drives can be imaged in such a way as to be permissible as evidence. Sadly it's often the case in serious breaches that well-meaning, talented and curiou
A plan may not apply (Score:4, Insightful)
I'd note that even if your company has a response plan, you may find it either completely useless or so general that it doesn't provide any help. Look at the article's point #1: it's almost nothing but "If $X, you may need $Y.". And it's far from complete. That's going to be a flaw in any security response plan: it's likely to not address the actual problem you face. Problems that you've thought of tend to get caught earlier before they turn into full-blown incidents, it's the ones nobody thought of that are most likely to bite you badly and it's exactly those that a plan won't cover. About the only part of the plan that'll be guaranteed to be useful is the part explaining what parts of the system are responsible for what and how to lock them down to preserve the evidence while you figure out where the breach is and what you need to do next. Beyond that you're into a twisty maze of little possibilities, all almost but not quite completely unlike each other, and what you need most isn't a plan but someone with enough Clue to analyze the situation and formulate a plan to fit it on the fly.
easy (Score:1)
First you have to file a (Score:2)
I'm a stickler for paper work.
Re: (Score:2)
part of a larger contingency plan (Score:5, Funny)
For most disasters, whether it's an IT disaster, a natural disaster, a non-natural physical disaster like a fire, a real or frivolous patent lawsuit, employee or company malfeasance, or what not, you need a plan.
For "terminal" disasters, like a nuclear blast that kills all employees and destroys all company assets, folding up shop may be the right business plan. For small businesses, extreme disasters like car wreck that kills all the employees might also be terminal in a slightly less catastrophic way. In these cases, at least you can plan to sell your business or its assets to another entity, so your customers have continuity.
Basically, divide your disasters into categories, and plan and insure accordingly:
0) end of the world, big asteroid or global thermonuclear war
1) major catastrophe, we are dead, forget about the customer, nuclear detonation event
2) end of the company, save the customer, Enron
3) end of the management team, save the company, MCI
4) we can recover from this but it's gonna hurt a lot, Vonage(?)
5) it's a flesh wound, CEO dies of heart attack
6) mosquito bite, SCO sues IBM
7) what? something happened? I didn't even notice, {if I had an example it would be #6}
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
It's all the mid-level bureaucrats who live at HQ who will be gone which will be problematic.
Re: (Score:2)
Re: (Score:2)
More often than not, applied internally as well as externally.
Re: (Score:2)
Outsource (Score:3, Insightful)
Re: (Score:2)
If you're smart... (Score:2)
Except that (Score:2)
Re: (Score:2)
Still, it's pointless to have people who are "supposed to be tech smart" here posting news sites and aggregators that have 5-10 pages of stuff that 1 page would suffice.
Clearly (Score:5, Funny)
Insightful or Funny You Chose (Score:2)
Disconnect and reinstall... (Score:4, Insightful)
It's been a long time (thankfully) since I've had to deal with this. But I'd echo the article about disconnecting from the net to eliminate further attacks. Then I'd remove the drive and save it for forensics -- replacements are cheap (I'm assuming a small business doesn't have expensive RAID setups). Assume that everything has been compromised and restore from a backup prior to the intrusion (hopefully you can tell when that was).
Oh, and keep your clocks synchronized. This will help if you need to trace intrusions across systems.
Congratulations, you just killed your forensics (Score:1)
Let's assume the bad guys never stored any forensically useful stuff on disk in clear text. Peter Gutmann [auckland.ac.nz] has a few things to say about recovering useful information from RAM chips.
The question for the real world is:
Is it worth going this far just to catch the bad guys?
Don't panic! (Score:5, Insightful)
First, find out the extent of the breach. Analyze your log files. Find out what time it happened. Find out who was logged in at the time, and find out any log messages from any system services that can help you figure out what the problem was. If you can't figure out what the scope of the breach was with a high level of confidence, then you have to assume the worst: the entire network is compromised.
Second, salvage what you can. Again, be very careful about doing this. Hopefully you have a backup somewhere which would allow you to avoid or shorten this step as much as possible. In essence, do what you have to do to the compromised machine to avoid losing work, but always be concious of the fact that the machine is compromised, and may be transmitting or recording keylogs or other sensitive information. If possible, disconnect the compromised machines from the Internet and isolate it from the rest of your LAN.
Third, plan for the future. How would this breach be avoided in the future? Was it an OS problem? If so, then maybe you need to install OpenBSD instead. Was it a problem with a particular package you were using? Choose a different package. Can you configure your firewall or server to prevent or limit the abuse that caused the problem in the first place (e.g. fail2ban to deal with SSH phishing attacks) or install monitoring software to alert you of a problem (e.g. an IDS like Snort)? Do your users need further training? Does your password policy allow weak passwords? Etc.
Finally, take a deep breath. Unless you've been totally negligent in your job, there wasn't much you could do to prevent it. Don't worry about the fact that you don't have enough to go to the police; most Network Administrators don't have the hardware, training or certification to present evidence in a courtroom anyway. If you can go to the cops, then bully for you! Make that black-hat asshole pay!
Script of comments to come... (Score:5, Funny)
Windows Vista: This wouldn't happen to me anyway, I'm the Most Secure OS (tm)!
Mac OS X: I never get any viruses!
GNU/Linux: Me neither!
Windows Vista User Access Control: You are entering a conversation with flaming probability 89%. Cancel or Allow?
Windows Vista: [to Vista UAC] Allow. [to the others] That's because nobody uses you!
GNU/Linux: Oh yeah...
Mac OS X: That's because only elite people use Mac OS X. Because you're not worth them.
GNU/Linux: Wait! Windows Vista, you lie! Lot's of people from all around the world use me! In fact, they even improve me! That's because we believe that...
Mac OS X and Windows Vista: [at the same time] Shut up Linux.
Windows Vista: [to Mac OS X] But anyway, even if there were a "Security Breach", it's not like they'd be able to mess anything up!
Mac OS X: That's because it's impossible to do anything in Vista.
Windows Vista User Access Control: [to Vista] You are coming to a sad realization... Cancel or Allow?
NB: the views or opinions expressed by any of the characters do not necessarily resemble the views or opinions of the author.
OpenBSD (Score:4, Funny)
Re: (Score:2, Funny)
OpenBSD: [angry] I'm not Linux you freak! Why is everyone always mixing us up?! [leaves room in tantrum]
Re:OpenBSD (Score:5, Funny)
Mac OS X: No no, that was OS/2 that died. Remember? You got his kidneys.
Secrecy is not the way to ensure security. (Score:2)
NetWare is proprietary (perhaps built on some FLOSS, but with proprietary software mixed in) so you can't completely know how well the implementation you're running follows a design you favor. No proprietary software's security can be verified as completely as FLOSS. It's never wise to conclude that any proprietary software is "secure by default".
Re: (Score:2)
And that is an ahistorical non sequitur. Neither my age nor the first distribution date of NetWare matter. The freedoms to inspect, share, modify, and run the software anytime for any reason are necessary freedoms in and of themselves, particularly for assessing security and reaching the conclusion that you did about NetWare. So long as you use proprietary software, you can't fully assess the security of your software.
Incidentally, according to Wikipedia [wikipedia.org], "the first versions of NetWare were designed" i
Got done.... (Score:5, Informative)
"If you're a big company, you already have a security team. If not, hire one." DOH!
That smacks me of the same kind of response from slashdot about legal advice... "Im being sued by the RIAA, should I ignore it?"
Still, why not gander around and see what the the real security experts and such say about such matters:
The Coroners Toolkit [fish2.com] Tools for Unix
Nagios detection suite [nagios.org]
Honeypots for 'sticking hackers' [honeynet.org]
And there's the wonderful tools in the Linux kernel for bridges and such that can be made to monitor data as if there was no computer there at all. Also, PF in FreeBSD can route and filter based on much more criteria than Linux netfilter can (like via OS).
You should have a secure layout of your network along with a respectable sensor network. The Sensornet should be separate from the general network.
If you already work in IT, these things should be obvious, as it is the similar measures required for data recovery on non-hack problems.
Re: (Score:2)
Ok. I have a Windows network, a Linux network and a MacOS network. I can prevent machines from migrating networks. If they attempt to, they will be isolated via rather nasty tools (arp corruption tools).
Also, it is NOT rather easy to fake network signatures from consistent data streams. It's easy to fake a NMAP scan though.
The key is I can segregate networks and I have the technological means to do so without me
Experts say hire experts (Score:2)
What a shocker...
Huh? Reinstall ofcourse! (Score:2, Insightful)
Re: (Score:2)
Re:Anyone ever followup with law enforcement agenc (Score:5, Insightful)
You need to get the subpoena to identify the person behind the attack. That assumes that your evidence actually points to a specific suspect. Unless your attacker was a complete moron, or your network logs are incredibly voluminous, that's not very likely. Once the subpoena is served and you've got your suspect and laid charges, you need to present evidence. That requires an expert witness. If you're lucky, YOU are the expert witness, but there's training and certification involved in that process. Otherwise, you get to hire an expert witness, and that won't be cheap. Your opponent will probably hire an opposing expert, just to confuse everybody.
Overall, I'd say that chances of success are incredibly low. Legal fees will be very high, and you have to turn over a fair chunk of your network assets to Law Enforcement. Basically, if you aren't really, really sure that you've got your man, it's really not worth the time and effort to find out who it was. That effort is much better spent allowing you to sleep at night knowing that people aren't getting in, IMO.
Re: (Score:2)
No, first you need to figure out a way to make the RIAA/MPAA think your attacker is hosting vast amounts of pirated music and movies. Let *them* get the subpoenas and harass your attacker with frivolous lawsuits...
Patch a socket (Score:2, Funny)
We had a security breach once (Score:5, Funny)
So in our case the response was:
1. Stop access.
2. Buy beer and popcorn
3. Watch movies.
Re: (Score:3, Funny)
So you did the right thing! (Score:3, Funny)
1. Assemble an incident response team.
Gather the buddies round the terminal, see what we got here.
2. Assess the initial damage and the risk for more.
You measured the damage, all 14GB of it. In assessing the risk for more of this damage, you noted that no ftp write access had been tried in a while, concluding that the risk was relatively low.
3. Develop a notification plan.
You sent an email-to-all that there's going to be a movie night, cancel your dates, postpone dinne
You can have more fun than that (Score:2)
"Serenity now!" (Score:2)
Put your head between your legs (Score:2)
Game over, man (Score:2)
It's the only way to be sure.
Easy... (Score:5, Funny)
or in doubt
Run in circles
scream and shout.
And yeah, pull the ethernet cables out.
easy... (Score:2, Insightful)
My version: (Score:5, Informative)
Once a compromise happened, there is no time to listen to lawyers or marketing executives. If they have anything to say, they would write a document where they list all recommendations they can care about -- for example, how "This site is pwn3d" web page is supposed to look like, whether it is a good thing to send all users a letter "please cancel your credit card", or what information can be released to authorities. If they didn't do that already, let them write those things while sysadmins are working.
This, of course, means that if there is only one sysadmin competent enough to investigate and fix the problem, then he would have to work on it alone.
2. Shut it down and investigate changes made by the attackers.
Before doing any investigation or recovery, shut the compromised and potentially compromised devices down. No malicious code should remain running. Whatever services should remain, must run in the minimal mode on separate hardware. For example, keep email running on a newly installed box. All investigation should be done in a clean environment -- drives moved to dedicated "clean" machines, or original servers booted from clean images (CD, PXE, replacement drives) on a private subnet, not accessible to anyone but people involved in incident response. Make full images of compromised hosts' storage whenever possible.
Backups are your friend. IDS logs are, too, but make sure that your IDS isn't compromised, and actually recorded something meaningful.
3. Don't worry about the person who originated the the attack.
Find its results and, if possible, method. Likely there will be at least one person within the company (malicious or more likely negligent) and at least one outside. Screw them both, they don't have access to your network anymore because it's off.
4. Immediately restore known-clean backups, perform audit on potentially compromised data and update the systems.
Backup is "known-clean" if investigation shown that it is from a state before the attack and does not contain vulnerable versions of software or compromised authentication information that allowed attack to happen. Usually some data has to be restored from a compromised system because it's more recent than backup (or because you are an idiot and forgot to back it up). Audits are supposed to be painful. Once data is in place, update software and configuration. Erase all compromised authentication keys and tokens.
5. Document the process.
I mean, technical details.
6. Tell everyone that they are screwed.
Explain to every office drone that they are going to get new passwords. They won't like it, so keep your LART ready.
Oh, btw:
http://abelits.livejournal.com/30214.html [livejournal.com]
http://abelits.livejournal.com/30681.html [livejournal.com]
http://abelits.livejournal.com/30872.html [livejournal.com]
Re: (Score:2)
Remember, when it hits the fan, it's going to be propelled down the hierarchy, even if the real problem was that some executive wanted to cut security funds.
Re: (Score:2)
I can't imagine a system that would actually preserve that without truly obscene cost. Also unless finding the source of attack is much more important than recovery (what is almost never the case in real life), it's usually worthless. The only exception to this is compromised virtual environment running on a clean physical host.
and chancing your swap file being over written/purged, many root kits will remove themselves when the system is rebooted to avoid d
Re: (Score:2)
This data is worthless.
and anything being run at the console give the option to save the current sessions to a disk(we don't know where the hacker was doing their dirty work from).
What console? What sessions? Compromised desktops are not worth messing with, and servers don't do that.
Much of this can easily be collected and wr
Re: (Score:2)
story (Score:5, Interesting)
Enter the FBI, who showed up in my roomate's lab asking about his computer (amoung other things). Picture yourself a grad student answering his lab door to find men in suits (an uncommon experience) who say they're part of the FBI (also uncommon), and mean it (still less common). After some questions, it was hesitantly established that my roomate was not the hacker serving root kits from his home computer.
From there, the FBI (with our permission) bugged our appartment. They put a "tap" in our appartment, which consistend of a special switch and a *very* loud windows machine that sat on our internet connection listening for hacker activity. The installation of the tap involved 7 FBI agents, none of which new nearly as much as my roomate about networking (that the broadcast ping couldn't get through their special switch with the word "tap" on it was a real mystery). Neadless to say, I didn't fool around with bittorrent or the like durring that time.
After a month or two, they caught the hacker (who was sweedish, apparently), and eventaully prosecuted him successfully.
Point is: sometimes it is useful to not reinstall immediately when hacked -- it can result in a good story :)
What to do next ? (Score:3, Funny)
#1 advise (Score:5, Insightful)
My advise to sysadmins who notice a breach is this:
Take your hands off the damn keyboard. Don't do anything unless you are 150% certain that you can see all possible consequences of your action. Call the IRT if you have one.
If there's nobody to call and you have to act right now, pull the power plug on the machine, then call the experts. Don't power the machine up again under any circumstances. If you want to look at the harddrive, make a copy first and mount it read-only in a different machine.
Why? Because back in the days when I was, err... looking around inside machines not my own, one of the things everyone I knew did was put in some scripts, tools, something, that'd wipe the logs or even the machine if my shell gets killed or the machine shut down or rebooted.
TFA assumes that you learn of the incident long after it has happened. Many incidents in real life are being noticed while they are going on, no matter if it's a remote access or your machine running an FTP server that wasn't there last month. That FTP server is almost certainly patched, and one of the things it might do is destroy evidence if you kill it. There might be an invisible process watching it to wipe evidence if you kill -9 it. Heck,
Mostly old news (Score:2, Informative)
Story's author writes back (Score:2, Interesting)
Re: (Score:2)
Re: (Score:1)
Re:Well...I'll give you some help (Score:2, Funny)
A good start, but... (Score:4, Insightful)