Forensics On a Cracked Linux Server 219
This blog entry is the step-by-step process that one administrator followed to figure out what was going on with a cracked Linux server. It's quite interesting to me, since I have had the exact same problem (a misbehaving ls -h command) on a development server quite a while back. As it turns out, my server was cracked, maybe with the same tool, and this analysis is much more thorough than the one I was able to do at the time. If you've ever wondered how to diagnose a Linux server that has been hijacked, this short article is a good starting point.
Forensics (Score:5, Insightful)
How did he get access and On tools (Score:5, Insightful)
* An exploit unknown to the public.
* A user accessing this server from an already compromised host. The attacker could then sniff the the password.
On tools...it's important to note that in forensics on a Linux box, your friends are ethereal (for watching packets on open connections), netstat (to see what's listening), and strace (shows you what UNIX API calls a running process makes, which gives you very good idea about what's going on.)
Other tools: nmap may be useful for seeing what's going on with 62.101.251.166 and 83.18.74.235. The service detection options, in particular. Always do this on a sandboxed host. Something running in a VM might be useful in this regard.
Anyway, nice article. This is almost exactly how I proceeded when one of my own servers was hacked a few years ago.
Re:Forensics (Score:2, Insightful)
Re:This is not forensics (Score:2, Insightful)
Re:Forensics (Score:5, Insightful)
It's unfortunate that this cracker made such an elementary mistake, it would have been interesting to see more advanced techniques in detecting rootkits. However, his analysis of the rootkit itself does provide some good information as to what a typical rootkit will generally do (replace core binaries, hide itself, use innocuous-looking names, etc).
Re:Yeah obvious FUD article (Score:1, Insightful)
Replace 'LINUX' with another version of Unix (the name of which will be withheld to protect the innocent). Some years ago, I ventured out onto the shop floor where I worked and encountered a terminal logged on to a critical production server. Nobody responsible seemed to be around. Typing 'whoami' returned 'root'. I promptly called the IT department's computing security group, informed them of the problem and hung around to see who showed up. After about 15 minutes with neither the original user or IT security appearing, I just logged the system off and left.
Who knows what damage could have been done to that system before I arrived?
Re:This is not forensics (Score:3, Insightful)
The definition of the word forensics is, "The use of science and technology to investigate and establish facts in criminal or civil courts of law." The original poster's argument is correct. This was not forensics. It was an analysis.
Re:Yeah obvious FUD article (Score:3, Insightful)
Don't be blinded by your religion.
Does rtkhunter... (Score:4, Insightful)
If you think that rtkhunter will protect you from a Linux kernel module rootkit your completely delusional. NOTHING will _reliably_ locate a LKM rootkit. That's the point of it.
Think about it. Rtkhunter relies on the ability of the kernel to accurately indicate files sizes, file names, and running proccesses as well as a bunch of other little detail things that normal rootkit makers tend to get wrong. When that kernel is subverted and controlled by it's new owner to give rtkunter, as well as other processes (such as your bash shell) false information about the system then those things are completely worthless.
It's the same as virus scanning on Linux (or any other system). Once the attacker gets root access then they have access to the kernel. Once they have access to the kernel they can use the kernel against you to hide what they are doing. Since userspace runs on top of the kernel then any sort of activity can be hidden by making the kernel lie to anything running in userspace.
This includes logging daemons, rootkit detection software, administrators, virus detection, rpm checksums, or anything else that people use to give themselves a FALSE sense of security.
There are two ways to reliably detect a rooted machine.
The first way is to use a network-based Intrusion Detection System (IDS). One of the best ones is commercially supported open source application called Snort. These guys can be hooked up to networks in a passive and completely undetectable way and are used to monitor traffic. They will alert administrators to any unusual network activity.
Network based IDS can be fooled, but as a administrator your at least operating on the same playing feild since your own software isn't used against you.
The second, and more reliable way, is to use a checksum-style IDS. MD5deep, AIDE, or Tripwire are 3 very good examples of this.
However how people use these things are completely worthless. If you keep the checksums and run the checksum software on the same machine as the one your trying to detect, then it's not good. Since they rely on the kernel any kernel-level rootkit can defeat them and the attacker can edit and substitute incorrect checksums.
In order for stuff like AIDE to be usefull it needs to be ran from read-only media and from a different operating system then the one your checking. (for example booted up in a knoppix cdrom, or a removable disk in a dedicated unconnected-to-any-network 'Tripwire' machine)
Both forms of IDS are very expensive and difficult to correctly use. Virtual machines make this stuff somewhat easier, but it's still much better to have dedicated machines for these things.
rtkhunter is nice if it's job is to make you feel good. If it's job is to make sure your machine is secure then it's shit. (no offense to the rtkhunter authors, I am sure they understand it's role and effectiveness.. to bad their users don't tend to) It's only good for kiddies that don't know better and if your being owned by kiddies then you have bigger problems.
selinux? (Score:5, Insightful)
I've only performed one Ubuntu install and most of my experience is with Red Hat and Fedora linux distros. Fedora installs selinux with a targeted policy enforcing by default which I think is a good thing. I had an experimental Fedora web server with PHPbb installed which was comprimised via the PHPbb application but looking through the log files it appeared that selinux had thwarted attempts to root the box or setup a zombie to connect to an irc server.
Other than the mistake of an outdated PHPbb application I also made the mistake of allowing execution of code in
It's easy to blame M$ for this. (Score:1, Insightful)
If you read the article though, you get to the conclusion:
The compromised could be caused by: An exploit unknown to the public. [or] A user accessing this server from an already compromised host. The attacker could then sniff the the password.
I'll give you two guesses at which one of these methods were used by the script kiddie who's mistakes are so well spelled out in the rest of the write up.
Security is only as strong as your weakest link. If you use an OS with a one in four chance of compromise [slashdot.org], you have a one in four chance of giving away all of your passwords.
Some Elemental Precautions (Score:3, Insightful)
1. Change the ssh port to something other than 22;
2. Use different root passwords on each machine;
3. Use selinux to block connections from IP addresses you do not control and to ports you don't want the machine connected to (like 6667);
4. If possible route all packets through a bridged machine which you can then use to monitor activities... be especially wary of IRC connections;
5. If you have email users set them up as nologin or
6. If you use ftp do not allow anonymous logins or, if you must allow connections, do not allow anonymous uploads;
7. Configure syslog so that it logs to several locations; and,
8. Use access lists on the routers to limit connections both in and out (including the new ssh port);
Crackers often forget to change lsof (list open files) and that utility can often be used (or reinstalled) to determine if a machine has been cracked and where the nasty bits are hidden.