Slashdot Log In
Forensics On a Cracked Linux Server
Journal written by Noryungi (70322) and posted by
kdawson
on Fri Aug 24, 2007 12:33 PM
from the hmmm-ls-looks-funny dept.
from the hmmm-ls-looks-funny dept.
This blog entry is the step-by-step process that one administrator followed to figure out what was going on with a cracked Linux server. It's quite interesting to me, since I have had the exact same problem (a misbehaving ls -h command) on a development server quite a while back. As it turns out, my server was cracked, maybe with the same tool, and this analysis is much more thorough than the one I was able to do at the time. If you've ever wondered how to diagnose a Linux server that has been hijacked, this short article is a good starting point.
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Story is FUD from a M$ shill (Score:2, Funny)
Yeah obvious FUD article (Score:5, Funny)
The bottom line is that a LINUX SERVER CAN'T BE CRACKED.
Maybe this admin got his login info phished by Nigerian scammers, I don't know. The guy probably is wondering why his Ebay account has a bunch of negative feedback and his MySpace is all jacked up and hasn't put 2 and 2 together with that time he responsed to that clever email asking for the triple whammy of MySpace/Ebay/root on your servers so that you could clear the money transfer.
That or he didn't have his updates turned on and had an outdated BIND. And its not like BIND means Linux is unsecure.
Even not that the idea that Linux is crackable is laughable and not worht front page at digg let alone slashdot. You don;t see Technorait or Bruce Perens' site posting garbage like this ever so why slashdot editors can't see thru it i dont kno.
Re: (Score:3, Insightful)
Don't be blinded by your religion.
Re:Yeah obvious FUD article (Score:5, Funny)
Parent
Re:Yeah obvious FUD article (Score:5, Funny)
ASCII art is lame
If you really want to blast them
Then try a haiku
So in my rage, I wrote this (and used the code layout):
Today I posted
Today I looked like an ass
It is Friday, beer
Parent
Re: (Score:3, Interesting)
Re:Yeah obvious FUD article (Score:5, Funny)
Parent
Forensics (Score:5, Insightful)
Re: (Score:2, Insightful)
Re:Forensics (Score:5, Insightful)
It's unfortunate that this cracker made such an elementary mistake, it would have been interesting to see more advanced techniques in detecting rootkits. However, his analysis of the rootkit itself does provide some good information as to what a typical rootkit will generally do (replace core binaries, hide itself, use innocuous-looking names, etc).
Parent
Re:Forensics (Score:5, Funny)
It makes for an interesting read
Anonymous in case the admin actually reads slashdot.
Parent
Re:Forensics (Score:5, Funny)
Parent
How did he get access and On tools (Score:5, Insightful)
* An exploit unknown to the public.
* A user accessing this server from an already compromised host. The attacker could then sniff the the password.
On tools...it's important to note that in forensics on a Linux box, your friends are ethereal (for watching packets on open connections), netstat (to see what's listening), and strace (shows you what UNIX API calls a running process makes, which gives you very good idea about what's going on.)
Other tools: nmap may be useful for seeing what's going on with 62.101.251.166 and 83.18.74.235. The service detection options, in particular. Always do this on a sandboxed host. Something running in a VM might be useful in this regard.
Anyway, nice article. This is almost exactly how I proceeded when one of my own servers was hacked a few years ago.
Further discussion... (Score:5, Informative)
Parent
Re:Further discussion... (Score:5, Interesting)
For example, one of Vodafone Greece's first reactions to finding that some of their switching systems had been rootkitted was to remove the offending software. This removal was one of the main contributing factors to the authorities having no chance to ever find the group that had compromised the system, that along with a couple of other screwups led to Vodafone getting fined a pretty hefty sum.
http://en.wikipedia.org/wiki/Greek_telephone_tapp
IEEE Spectrum had a recent article that had MUCH better information than Wikipedia though, I don't have it with me at the moment unfortunately.
Parent
Re:Further discussion... (Score:4, Informative)
http://www.spectrum.ieee.org/jul07/5280 [ieee.org] for those interested.
Parent
Re:How did he get access and On tools (Score:5, Funny)
Clearly, we as sysadmins should rethink the long-standing policy of setting all root passwords to either love, secret, sex, or god. Perhaps we should at least add another password to the list, like "unhackable" or something truly secure like that.
Parent
Re: (Score:2)
1. We already know that it was meant to be running Apache. Perhaps there was some PHP application which wasn't very secure? Even so, if that were the case then the exploit they used must have been fairly convoluted because it probably wouldn't have got them root access immediately.
2. We don't know what other services were supposed to be running, how/if they were firewalled and secured. SSH, for instance, is only as secure as the weakest password on
Re:How did he get access and On tools (Score:5, Informative)
I have seen a number of rootkits for Linux as far back as 97-98 which were considerably more advanced. It was a bit of an arms race between the admins (including me) and the guys who were breaking in. By the end the best rootkits could:
1. Load a whole hidden fs with tools into a ramdisk or hidden area on the filesystem not visible using normal tools.
2. Hide all sockets, processes and files belonging to the rootkit completely. You simply could no longer see them using netstat, ps and other similar tools.
3. Monitor network driver state for the promisc flag and "scrub" backdoor traffic out of it so it is no longer visible using tcpdump and ethereal.
4. Adjust memory totals and df so that you do not see them. This was also the only way we found to catch it. Try to allocate 95% of the remaining free memory and see the system oops magestically.
5. Doctor logs so that you could not notice anything.
6. The rootkit itself handled all connections via something that looked like ssh. I never managed to figure out how it loaded. One of the executables in the system loaded at startup was backdoored. Probably sendmail or one of the other daemons it could not do without.
7. The rootkit managed to masq changed files completely. Tripwire and md5sums were reporting all OK while executables were being changed.
That was a the tech level in 97. I would expect 10 years later a good rootkit to be even better. Looking at the blog post I can only laugh.
If you suspect a system is cracked:
1. Take it offline and take the disks out. Analyse the system completely offline looking at the disk from another system mounted as ro (on SCSI discs use the RO jumper). Never ever even try to start it. Nowdays knoppix is a great help. Most importantly - do not fsck systems before mounting as the rootkit may hide in orphaned areas which fsck will fix.
2. If you are monitoring traffic, monitor it on a switch span port or create yourself a simple multiple interface box which serves as a firewalling bridge (so you can hijack the more interesting bits and alter them). Lex Book PCs are a good choice as they can run either Linux or BSD and are as portable as a laptop. A recent Via with 2 Ethernet ports is also a good choice as it can handle up to 1GB of traffic across as a bridge.
Parent
Re:How did he get access and On tools (Score:5, Informative)
1. Take it offline and take the disks out.
And I've been told don't use the 'shutodwn' command--instead, pull the power plug out of the wall. A rootkit could include a cleanup routine that gets run at shutdown time.
Parent
Re: (Score:3, Informative)
Virtual machines rule... (Score:3, Interesting)
If I did care, I could either suspend the virtual machine or make a snapshot of it.
Virtual machines are cool
Re: (Score:3, Interesting)
The IRC-bot creator
Looks as if there was another way... (Score:4, Funny)
sPh
This is not forensics (Score:5, Informative)
Re: (Score:2, Insightful)
Re: (Score:3, Insightful)
The definition of the word forensics is, "The use of science and technology to investigate and establish facts in criminal or civil courts of law." The original poster's argument is correct. This was not forensics. It was an analysis.
Re: (Score:2)
rkhunter anyone? (Score:4, Informative)
Does rtkhunter... (Score:4, Insightful)
If you think that rtkhunter will protect you from a Linux kernel module rootkit your completely delusional. NOTHING will _reliably_ locate a LKM rootkit. That's the point of it.
Think about it. Rtkhunter relies on the ability of the kernel to accurately indicate files sizes, file names, and running proccesses as well as a bunch of other little detail things that normal rootkit makers tend to get wrong. When that kernel is subverted and controlled by it's new owner to give rtkunter, as well as other processes (such as your bash shell) false information about the system then those things are completely worthless.
It's the same as virus scanning on Linux (or any other system). Once the attacker gets root access then they have access to the kernel. Once they have access to the kernel they can use the kernel against you to hide what they are doing. Since userspace runs on top of the kernel then any sort of activity can be hidden by making the kernel lie to anything running in userspace.
This includes logging daemons, rootkit detection software, administrators, virus detection, rpm checksums, or anything else that people use to give themselves a FALSE sense of security.
There are two ways to reliably detect a rooted machine.
The first way is to use a network-based Intrusion Detection System (IDS). One of the best ones is commercially supported open source application called Snort. These guys can be hooked up to networks in a passive and completely undetectable way and are used to monitor traffic. They will alert administrators to any unusual network activity.
Network based IDS can be fooled, but as a administrator your at least operating on the same playing feild since your own software isn't used against you.
The second, and more reliable way, is to use a checksum-style IDS. MD5deep, AIDE, or Tripwire are 3 very good examples of this.
However how people use these things are completely worthless. If you keep the checksums and run the checksum software on the same machine as the one your trying to detect, then it's not good. Since they rely on the kernel any kernel-level rootkit can defeat them and the attacker can edit and substitute incorrect checksums.
In order for stuff like AIDE to be usefull it needs to be ran from read-only media and from a different operating system then the one your checking. (for example booted up in a knoppix cdrom, or a removable disk in a dedicated unconnected-to-any-network 'Tripwire' machine)
Both forms of IDS are very expensive and difficult to correctly use. Virtual machines make this stuff somewhat easier, but it's still much better to have dedicated machines for these things.
rtkhunter is nice if it's job is to make you feel good. If it's job is to make sure your machine is secure then it's shit. (no offense to the rtkhunter authors, I am sure they understand it's role and effectiveness.. to bad their users don't tend to) It's only good for kiddies that don't know better and if your being owned by kiddies then you have bigger problems.
Parent
They got the webserver too (Score:2)
A SANS reference (Score:2, Informative)
Can't read TFA (Score:2)
Time to put my tinfoil hat back on.
I had to do this once. (Score:4, Funny)
Re: (Score:3, Funny)
Mirror (Score:4, Informative)
http://64.233.183.104/search?q=cache:TyrHbOqUhLgJ
Meta-cracking (Score:5, Funny)
1. Infect Linux server of some guy with a blog.
2. Guy blogs about how he dealt with said infection.
3. Blog posting gets linked to on Slashdot.
4. Millions of computers attempt to access the blog, hence bringing down the server.
Don't you see? We've a socially engineered botnet!
(And please, for the love of all that is sacred and funny, don't reply to this and add steps for "???" and "Profit". It's just tired and completely not funny. And the clever little variation on that theme you're thinking about posting right now isn't funny either.)
Re:Meta-cracking (Score:5, Funny)
2. ????
3. Profit!
Parent
Wish I would have known... (Score:2)
Slashdotted, Mirrordot cache (Score:5, Informative)
Raise your hand (Score:5, Funny)
Re:Raise your hand (Score:5, Funny)
'ls' is not recognized as an internal or external command,
operable program or batch file.
Oh noes!
Parent
selinux? (Score:5, Insightful)
I've only performed one Ubuntu install and most of my experience is with Red Hat and Fedora linux distros. Fedora installs selinux with a targeted policy enforcing by default which I think is a good thing. I had an experimental Fedora web server with PHPbb installed which was comprimised via the PHPbb application but looking through the log files it appeared that selinux had thwarted attempts to root the box or setup a zombie to connect to an irc server.
Other than the mistake of an outdated PHPbb application I also made the mistake of allowing execution of code in
Re: (Score:3, Informative)
Re:selinux? (Score:4, Informative)
Parent
That's it, I'm switching to Windows (Score:4, Funny)
Who needs clever hacks? (Score:3, Interesting)
It's pretty appalling. We would need an army of sysadmins--an army which is currently employed already--to really do something about it. Most of what we see are primitive script kiddie hacks, but guess what--that's good enough, and rarely are the perpetrators hunted down.
Who knows what the more sophisticated hackers are up to!
Some Elemental Precautions (Score:3, Insightful)
1. Change the ssh port to something other than 22;
2. Use different root passwords on each machine;
3. Use selinux to block connections from IP addresses you do not control and to ports you don't want the machine connected to (like 6667);
4. If possible route all packets through a bridged machine which you can then use to monitor activities... be especially wary of IRC connections;
5. If you have email users set them up as nologin or
6. If you use ftp do not allow anonymous logins or, if you must allow connections, do not allow anonymous uploads;
7. Configure syslog so that it logs to several locations; and,
8. Use access lists on the routers to limit connections both in and out (including the new ssh port);
Crackers often forget to change lsof (list open files) and that utility can often be used (or reinstalled) to determine if a machine has been cracked and where the nasty bits are hidden.
*Bourne* Shell? (Score:5, Funny)
I knew it! Jason Bourne was involved in this!
Parent
Re:Ssshhh.... Secrets Revealed... (Score:5, Funny)
The 220,000 or so members of the Slashdot Members Who Post Authoritative Statements On The Inner Workings Of Microsoft To Support Their Arguments warmly welcomes you to the club.
Parent