Forensics On a Cracked Linux Server 219
This blog entry is the step-by-step process that one administrator followed to figure out what was going on with a cracked Linux server. It's quite interesting to me, since I have had the exact same problem (a misbehaving ls -h command) on a development server quite a while back. As it turns out, my server was cracked, maybe with the same tool, and this analysis is much more thorough than the one I was able to do at the time. If you've ever wondered how to diagnose a Linux server that has been hijacked, this short article is a good starting point.
Story is FUD from a M$ shill (Score:2, Funny)
Re: (Score:2)
Yeah obvious FUD article (Score:5, Funny)
The bottom line is that a LINUX SERVER CAN'T BE CRACKED.
Maybe this admin got his login info phished by Nigerian scammers, I don't know. The guy probably is wondering why his Ebay account has a bunch of negative feedback and his MySpace is all jacked up and hasn't put 2 and 2 together with that time he responsed to that clever email asking for the triple whammy of MySpace/Ebay/root on your servers so that you could clear the money transfer.
That or he didn't have his updates turned on and had an outdated BIND. And its not like BIND means Linux is unsecure.
Even not that the idea that Linux is crackable is laughable and not worht front page at digg let alone slashdot. You don;t see Technorait or Bruce Perens' site posting garbage like this ever so why slashdot editors can't see thru it i dont kno.
Re: (Score:3, Insightful)
Don't be blinded by your religion.
Re:Yeah obvious FUD article (Score:5, Funny)
Re:Yeah obvious FUD article (Score:5, Funny)
ASCII art is lame
If you really want to blast them
Then try a haiku
So in my rage, I wrote this (and used the code layout):
Today I posted
Today I looked like an ass
It is Friday, beer
Re: (Score:2)
I am clearly no poet;
it seemed rather odd.
Re: (Score:2)
Re: (Score:3, Interesting)
Re: (Score:2)
Re: (Score:2)
Re:Yeah obvious FUD article (Score:5, Funny)
Re: (Score:2)
Re: (Score:2)
Well said. It's like two teams of kids at Easter, one team wearing black hats, the other white. Let the kids loose in a field full of Easter eggs. Most people like the white-hat team better, as they don't pick up their eggs and throw them at other people. However, that doesn't mean the less-liked team will find fewer eggs, nor refrain from throwing them. I'll have to quote Scotty again:
"The more they overthink the plumbing, the easier it is to stop up the drain..."
As for the poster's belief that any ph
Re: (Score:2)
1. is a good enough sysadmin to secure his box
2. doesn't need "his friend" to check out his box if it doesn't do what it's supposed to.
I work for a hosting company with several thousand servers (client's hire a real or virtual server and have root access) and based on what I see on a dialy basis, I can tell you the following:
1. Linux servers do get compromised.
2. The vast majority of compromises on Linux servers are application level.
3. Root compromises tend to be cause by:
Forensics (Score:5, Insightful)
Re: (Score:2, Insightful)
Re:Forensics (Score:5, Insightful)
It's unfortunate that this cracker made such an elementary mistake, it would have been interesting to see more advanced techniques in detecting rootkits. However, his analysis of the rootkit itself does provide some good information as to what a typical rootkit will generally do (replace core binaries, hide itself, use innocuous-looking names, etc).
Re:Forensics (Score:5, Funny)
It makes for an interesting read
Anonymous in case the admin actually reads slashdot.
Re:Forensics (Score:5, Funny)
Re: (Score:2, Interesting)
Re: (Score:2, Funny)
Are you too stupid to know what redundant means? I guess you are.
Hey mod you're an dumbass.
Wait, "dumbass moderator" see, THAT is redundant.
How did he get access and On tools (Score:5, Insightful)
* An exploit unknown to the public.
* A user accessing this server from an already compromised host. The attacker could then sniff the the password.
On tools...it's important to note that in forensics on a Linux box, your friends are ethereal (for watching packets on open connections), netstat (to see what's listening), and strace (shows you what UNIX API calls a running process makes, which gives you very good idea about what's going on.)
Other tools: nmap may be useful for seeing what's going on with 62.101.251.166 and 83.18.74.235. The service detection options, in particular. Always do this on a sandboxed host. Something running in a VM might be useful in this regard.
Anyway, nice article. This is almost exactly how I proceeded when one of my own servers was hacked a few years ago.
Further discussion... (Score:5, Informative)
Re:Further discussion... (Score:5, Interesting)
For example, one of Vodafone Greece's first reactions to finding that some of their switching systems had been rootkitted was to remove the offending software. This removal was one of the main contributing factors to the authorities having no chance to ever find the group that had compromised the system, that along with a couple of other screwups led to Vodafone getting fined a pretty hefty sum.
http://en.wikipedia.org/wiki/Greek_telephone_tapp
IEEE Spectrum had a recent article that had MUCH better information than Wikipedia though, I don't have it with me at the moment unfortunately.
Re:Further discussion... (Score:4, Informative)
http://www.spectrum.ieee.org/jul07/5280 [ieee.org] for those interested.
Re: (Score:2)
Do I shut down & clean, thereby protecting my clients, data, shareholders and above-all my ass?
Or do I play amateur cop / responsable citizen (depends on your point of view), and try and sniff and smoke the bastards out?
Tough call.
Having said that, some of my clients are massive multinationals, (like Vodafone), and they seem more preoccupied with cutting costs than taking this kind of threat seriously. Whilst a local entity - to take your example, Greece - could not necessarily
Re:How did he get access and On tools (Score:5, Funny)
Clearly, we as sysadmins should rethink the long-standing policy of setting all root passwords to either love, secret, sex, or god. Perhaps we should at least add another password to the list, like "unhackable" or something truly secure like that.
Re: (Score:2)
Re: (Score:2)
1. We already know that it was meant to be running Apache. Perhaps there was some PHP application which wasn't very secure? Even so, if that were the case then the exploit they used must have been fairly convoluted because it probably wouldn't have got them root access immediately.
2. We don't know what other services were supposed to be running, how/if they were firewalled and secured. SSH, for instance, is only as secure as the weakest password on
Re: (Score:2)
http://all.net/dtk/download.html [all.net]
Re: (Score:2)
I don't even trust Putty just because it *is* windows.
Re:How did he get access and On tools (Score:5, Informative)
I have seen a number of rootkits for Linux as far back as 97-98 which were considerably more advanced. It was a bit of an arms race between the admins (including me) and the guys who were breaking in. By the end the best rootkits could:
1. Load a whole hidden fs with tools into a ramdisk or hidden area on the filesystem not visible using normal tools.
2. Hide all sockets, processes and files belonging to the rootkit completely. You simply could no longer see them using netstat, ps and other similar tools.
3. Monitor network driver state for the promisc flag and "scrub" backdoor traffic out of it so it is no longer visible using tcpdump and ethereal.
4. Adjust memory totals and df so that you do not see them. This was also the only way we found to catch it. Try to allocate 95% of the remaining free memory and see the system oops magestically.
5. Doctor logs so that you could not notice anything.
6. The rootkit itself handled all connections via something that looked like ssh. I never managed to figure out how it loaded. One of the executables in the system loaded at startup was backdoored. Probably sendmail or one of the other daemons it could not do without.
7. The rootkit managed to masq changed files completely. Tripwire and md5sums were reporting all OK while executables were being changed.
That was a the tech level in 97. I would expect 10 years later a good rootkit to be even better. Looking at the blog post I can only laugh.
If you suspect a system is cracked:
1. Take it offline and take the disks out. Analyse the system completely offline looking at the disk from another system mounted as ro (on SCSI discs use the RO jumper). Never ever even try to start it. Nowdays knoppix is a great help. Most importantly - do not fsck systems before mounting as the rootkit may hide in orphaned areas which fsck will fix.
2. If you are monitoring traffic, monitor it on a switch span port or create yourself a simple multiple interface box which serves as a firewalling bridge (so you can hijack the more interesting bits and alter them). Lex Book PCs are a good choice as they can run either Linux or BSD and are as portable as a laptop. A recent Via with 2 Ethernet ports is also a good choice as it can handle up to 1GB of traffic across as a bridge.
Re:How did he get access and On tools (Score:5, Informative)
1. Take it offline and take the disks out.
And I've been told don't use the 'shutodwn' command--instead, pull the power plug out of the wall. A rootkit could include a cleanup routine that gets run at shutdown time.
Re: (Score:3, Informative)
Re: (Score:2)
Re: (Score:2)
A former co-worker used to turn her G5 off every day by pressing the front button. I saw her do this once and said (very nicely) "You know it's better to shut down from the menu, right?" and she answered "Yeah, I know you're not supposed to do that, but it's faster." She had been doing that nightly (or maybe just weekly) for a couple years.
On a G5 (and, indeed, most PCs anf Macs <6-7 years old) pressing the power button should result in a clean shutdown.
Virtual machines rule... (Score:3, Interesting)
If I did care, I could either suspend the virtual machine or make a snapshot of it.
Virtual machines are cool
Re: (Score:2)
Re: (Score:3, Interesting)
The IRC-bot creator
Re: (Score:2)
Re: (Score:2)
But by the way, do not begin to pretend that most installations of Jabber are any better administered than most installations of IRC. Plain text passwords stored on the server is just an amazingly bad idea: it's almost as stupid as Subversion keeping your user passwords in plain taext in your home
Re: (Score:2)
As far as Jabber vs IRC vs the rest of the IM I agree they all suck and they can all be used for zombie control. You can write a BOT that logs in on yahoo, AIM or anything else you like. I used to have a Yahoo Messenger BOT that talked to a MON alert system and pinged me when something went apeshit in the network (you could also get network status and such). Writing it was quite trivial, unfo
Re: (Score:2)
This article was junk, as you point out the state of play 10 years ago was already way ahead of this.
I always assumed lack of progress was caused by the kiddies discovering that a large install base of Windows machines was more profitable than the odd *nix machine that took a bit of work to get into.
feh.
Re: (Score:2)
We found a couple of backdoors now and then none of which was particularly fancy. For example sendmail had an extra command added which executed a shell, etc. So I suspect that he loaded the rootkit straight into memory over the network after accessing the compromised machine through the backdoor. As a result it was never present for forensics.
The most unpleasant bit was that he nuked the machine at the slightest su
Re: (Score:2)
There are some things that can be done to prevent dictionary attacks from working. Or at least from working enough that they would succeed. there used to be an email program you could run (monkey business or something like that.) that would just return hits on anything tried other then the actual accounts. It was designed to make harvesting addresses as useless as mailing to the dictionary itself. I don't see why so
Looks as if there was another way... (Score:4, Funny)
sPh
This is not forensics (Score:5, Informative)
Re: (Score:2, Insightful)
Re: (Score:3, Insightful)
The definition of the word forensics is, "The use of science and technology to investigate and establish facts in criminal or civil courts of law." The original poster's argument is correct. This was not forensics. It was an analysis.
Re: (Score:2)
1. The art or study of formal debate; argumentation.
The 1913 PD Websters gives:
"Belonging to courts of judicature or to public discussion and debate; used in legal proceedings, or in public discussions; argumentative; rhetorical; as, forensic eloquence or disputes."
One of the paper dictionaries I have here says: "1. belonging to, used in, or suitable to courts of judicature or to public discussion and debate."
(Emphasis mine.)
By that, I think this became f
Forensics has an established meaning in security (Score:2)
That was what was meant. You can argue semantics and definitions all you like, but anyone
'forensics' also means 'public speaking' (Score:2)
The funny thing is, even the definition you tried to apply does not fit. The term 'forensics', when used in the context of 'an argumentative exercise' mean
Re: (Score:2)
Re: (Score:2)
Isn't there other data that could be lost if you shut the machine down?
rkhunter anyone? (Score:4, Informative)
Re: (Score:2)
Does rtkhunter... (Score:4, Insightful)
If you think that rtkhunter will protect you from a Linux kernel module rootkit your completely delusional. NOTHING will _reliably_ locate a LKM rootkit. That's the point of it.
Think about it. Rtkhunter relies on the ability of the kernel to accurately indicate files sizes, file names, and running proccesses as well as a bunch of other little detail things that normal rootkit makers tend to get wrong. When that kernel is subverted and controlled by it's new owner to give rtkunter, as well as other processes (such as your bash shell) false information about the system then those things are completely worthless.
It's the same as virus scanning on Linux (or any other system). Once the attacker gets root access then they have access to the kernel. Once they have access to the kernel they can use the kernel against you to hide what they are doing. Since userspace runs on top of the kernel then any sort of activity can be hidden by making the kernel lie to anything running in userspace.
This includes logging daemons, rootkit detection software, administrators, virus detection, rpm checksums, or anything else that people use to give themselves a FALSE sense of security.
There are two ways to reliably detect a rooted machine.
The first way is to use a network-based Intrusion Detection System (IDS). One of the best ones is commercially supported open source application called Snort. These guys can be hooked up to networks in a passive and completely undetectable way and are used to monitor traffic. They will alert administrators to any unusual network activity.
Network based IDS can be fooled, but as a administrator your at least operating on the same playing feild since your own software isn't used against you.
The second, and more reliable way, is to use a checksum-style IDS. MD5deep, AIDE, or Tripwire are 3 very good examples of this.
However how people use these things are completely worthless. If you keep the checksums and run the checksum software on the same machine as the one your trying to detect, then it's not good. Since they rely on the kernel any kernel-level rootkit can defeat them and the attacker can edit and substitute incorrect checksums.
In order for stuff like AIDE to be usefull it needs to be ran from read-only media and from a different operating system then the one your checking. (for example booted up in a knoppix cdrom, or a removable disk in a dedicated unconnected-to-any-network 'Tripwire' machine)
Both forms of IDS are very expensive and difficult to correctly use. Virtual machines make this stuff somewhat easier, but it's still much better to have dedicated machines for these things.
rtkhunter is nice if it's job is to make you feel good. If it's job is to make sure your machine is secure then it's shit. (no offense to the rtkhunter authors, I am sure they understand it's role and effectiveness.. to bad their users don't tend to) It's only good for kiddies that don't know better and if your being owned by kiddies then you have bigger problems.
Re: (Score:2)
There's an interesting third approach, used by Sysinternals's (now part of MS) RootkitRevealer for Windows.
Basically, enumerate all the files on the system using the usual OS APIs. Then, scan the entire raw disk, and enumerate all the files on the system by manually interpreting the directory structures stored on-disk. Any files whose directory entries exist on-disk, but don't show up in the OS's API (with a few standard system exceptions) are being hidden from the OS API layer by a rootkit.
It's certain
Re: (Score:2)
First, there's the administrator permission level, which is supposed to be the Windows equivalent to wheel (and Administrator being the Windows equivalent to root.)
Then, there's LocalSystem, which has higher permissions than Administrator. (And, Administrator is necessarily slightly lower than *nix root.)
Of course, there's still exploits against the LocalSystem account, but it's not as easy to get to as Administrator or an equivalent
Re: (Score:2)
They got the webserver too (Score:2)
A SANS reference (Score:2, Informative)
Can't read TFA (Score:2)
Time to put my tinfoil hat back on.
I had to do this once. (Score:4, Funny)
Re: (Score:3, Funny)
Re: (Score:2)
Re: (Score:2)
Mirror (Score:4, Informative)
http://64.233.183.104/search?q=cache:TyrHbOqUhLgJ
Meta-cracking (Score:5, Funny)
1. Infect Linux server of some guy with a blog.
2. Guy blogs about how he dealt with said infection.
3. Blog posting gets linked to on Slashdot.
4. Millions of computers attempt to access the blog, hence bringing down the server.
Don't you see? We've a socially engineered botnet!
(And please, for the love of all that is sacred and funny, don't reply to this and add steps for "???" and "Profit". It's just tired and completely not funny. And the clever little variation on that theme you're thinking about posting right now isn't funny either.)
Re:Meta-cracking (Score:5, Funny)
2. ????
3. Profit!
Wish I would have known... (Score:2)
Slashdotted, Mirrordot cache (Score:5, Informative)
Raise your hand (Score:5, Funny)
Re:Raise your hand (Score:5, Funny)
'ls' is not recognized as an internal or external command,
operable program or batch file.
Oh noes!
Re: (Score:2)
Re: (Score:2)
dir
Re: (Score:2)
C:\Documents and Settings\user>ls -h
'ls' is not recognized as an internal or external command, operable program or batch file.
C:\Documents and Settings\user>
Crap! I'm screwed! Someone hacked my system!
Re: (Score:2)
Re: (Score:2)
selinux? (Score:5, Insightful)
I've only performed one Ubuntu install and most of my experience is with Red Hat and Fedora linux distros. Fedora installs selinux with a targeted policy enforcing by default which I think is a good thing. I had an experimental Fedora web server with PHPbb installed which was comprimised via the PHPbb application but looking through the log files it appeared that selinux had thwarted attempts to root the box or setup a zombie to connect to an irc server.
Other than the mistake of an outdated PHPbb application I also made the mistake of allowing execution of code in
Re: (Score:2)
Re: (Score:2)
Just pick up the nearest telephone and state your problem; the issue will be patched in the next release.
Re: (Score:3, Informative)
Re:selinux? (Score:4, Informative)
Re: (Score:2)
Ineffective rootkit (Score:2)
In fact, the whole crack of the server seems to be pretty amateurish. Still, even if the analysis was not very good, it is interesting article.
That's it, I'm switching to Windows (Score:4, Funny)
Cracked spell checker? (Score:2)
Questions? Call 1-800-no-spell
Who needs clever hacks? (Score:3, Interesting)
It's pretty appalling. We would need an army of sysadmins--an army which is currently employed already--to really do something about it. Most of what we see are primitive script kiddie hacks, but guess what--that's good enough, and rarely are the perpetrators hunted down.
Who knows what the more sophisticated hackers are up to!
Re: (Score:2)
Re: (Score:2, Interesting)
It's more about 2 screwed up business models (If you look at it from a technical point of view).
They want cheap servers with bandwidth, buy cheap servers and buy shitloads of bandwidth. Offer them for really cheap prices ( 10,000 Servers. They may have five or six people on a shift for maintaining these systems. These guys are responsible for patch management and backup/restore, plus they have to physically replace the systems which crash (Usuall
Some Elemental Precautions (Score:3, Insightful)
1. Change the ssh port to something other than 22;
2. Use different root passwords on each machine;
3. Use selinux to block connections from IP addresses you do not control and to ports you don't want the machine connected to (like 6667);
4. If possible route all packets through a bridged machine which you can then use to monitor activities... be especially wary of IRC connections;
5. If you have email users set them up as nologin or
6. If you use ftp do not allow anonymous logins or, if you must allow connections, do not allow anonymous uploads;
7. Configure syslog so that it logs to several locations; and,
8. Use access lists on the routers to limit connections both in and out (including the new ssh port);
Crackers often forget to change lsof (list open files) and that utility can often be used (or reinstalled) to determine if a machine has been cracked and where the nasty bits are hidden.
*Bourne* Shell? (Score:5, Funny)
I knew it! Jason Bourne was involved in this!
Re: (Score:2)
. O <-You O <- Jason Bourne
./|\
./ \ / \
Whoosh!
Re: (Score:2)
I ask because everyone seems to be looking very closely at the initial OS distro, and almost any server that's been put into useful production has been tweaked in some way from the official packages. Stuff gets compiled from source. Custom stuff gets coded. Packages get installed out of third-party reposit
Re: (Score:2)
Another option for a perfectly secure box that wasn't mentioned in TFA is that the friend could have run a Trojan that opened the initial hole.
Re: (Score:2)
Re: (Score:2)
Suuuure you are.
Re:Ssshhh.... Secrets Revealed... (Score:5, Funny)
The 220,000 or so members of the Slashdot Members Who Post Authoritative Statements On The Inner Workings Of Microsoft To Support Their Arguments warmly welcomes you to the club.