Forensics On a Cracked Linux Server 219
This blog entry is the step-by-step process that one administrator followed to figure out what was going on with a cracked Linux server. It's quite interesting to me, since I have had the exact same problem (a misbehaving ls -h command) on a development server quite a while back. As it turns out, my server was cracked, maybe with the same tool, and this analysis is much more thorough than the one I was able to do at the time. If you've ever wondered how to diagnose a Linux server that has been hijacked, this short article is a good starting point.
This is not forensics (Score:5, Informative)
rkhunter anyone? (Score:4, Informative)
A SANS reference (Score:2, Informative)
Mirror (Score:4, Informative)
http://64.233.183.104/search?q=cache:TyrHbOqUhLgJ
Further discussion... (Score:5, Informative)
Slashdotted, Mirrordot cache (Score:5, Informative)
Re:How did he get access and On tools (Score:5, Informative)
I have seen a number of rootkits for Linux as far back as 97-98 which were considerably more advanced. It was a bit of an arms race between the admins (including me) and the guys who were breaking in. By the end the best rootkits could:
1. Load a whole hidden fs with tools into a ramdisk or hidden area on the filesystem not visible using normal tools.
2. Hide all sockets, processes and files belonging to the rootkit completely. You simply could no longer see them using netstat, ps and other similar tools.
3. Monitor network driver state for the promisc flag and "scrub" backdoor traffic out of it so it is no longer visible using tcpdump and ethereal.
4. Adjust memory totals and df so that you do not see them. This was also the only way we found to catch it. Try to allocate 95% of the remaining free memory and see the system oops magestically.
5. Doctor logs so that you could not notice anything.
6. The rootkit itself handled all connections via something that looked like ssh. I never managed to figure out how it loaded. One of the executables in the system loaded at startup was backdoored. Probably sendmail or one of the other daemons it could not do without.
7. The rootkit managed to masq changed files completely. Tripwire and md5sums were reporting all OK while executables were being changed.
That was a the tech level in 97. I would expect 10 years later a good rootkit to be even better. Looking at the blog post I can only laugh.
If you suspect a system is cracked:
1. Take it offline and take the disks out. Analyse the system completely offline looking at the disk from another system mounted as ro (on SCSI discs use the RO jumper). Never ever even try to start it. Nowdays knoppix is a great help. Most importantly - do not fsck systems before mounting as the rootkit may hide in orphaned areas which fsck will fix.
2. If you are monitoring traffic, monitor it on a switch span port or create yourself a simple multiple interface box which serves as a firewalling bridge (so you can hijack the more interesting bits and alter them). Lex Book PCs are a good choice as they can run either Linux or BSD and are as portable as a laptop. A recent Via with 2 Ethernet ports is also a good choice as it can handle up to 1GB of traffic across as a bridge.
Re:selinux? (Score:3, Informative)
Re:How did he get access and On tools (Score:5, Informative)
1. Take it offline and take the disks out.
And I've been told don't use the 'shutodwn' command--instead, pull the power plug out of the wall. A rootkit could include a cleanup routine that gets run at shutdown time.
Re:How did he get access and On tools (Score:3, Informative)
Re:selinux? (Score:4, Informative)
Re:Further discussion... (Score:4, Informative)
http://www.spectrum.ieee.org/jul07/5280 [ieee.org] for those interested.