Mystery Malware Affecting Linux/Apache Web Servers 437
lisah writes "Reports are beginning to surface that some Web servers running Linux and Apache are unwittingly infecting thousands of computers, exploiting vulnerabilities in QuickTime, Yahoo! Messenger, and Windows. One way to tell if your machine is infected is if you're unable to create a directory name beginning with a numeral. Since details are still sketchy, the best advice right now is to take proactive steps to secure your servers. 'We asked the Apache Software Foundation if it had any advice on how to detect the rootkit or cleanse a server when it's found. According to Mark Cox of the Apache security team, "Whilst details are thin as to how the attackers gained root access to the compromised servers, we currently have no evidence that this is due to an unfixed vulnerability in the Apache HTTP Server." We sent a similar query to Red Hat, the largest vendor of Linux, but all its security team could tell us was that "At this point in time we have not had access to any affected machines and therefore cannot give guidance on which tools would reliably detect the rootkit."'"
Should have used IIS (Score:5, Funny)
Re:Should have used IIS (Score:4, Funny)
Re:Should have used IIS (Score:4, Funny)
Re:Should have used IIS (Score:5, Informative)
In all seriousness though, IIS 6 has a pretty darn good security track record; seemingly better than Apache 2's, even if it is blasphemy for me to say it. I've previously decried the use of raw vulnerability statistics to make comparative claims about different products' security [slashdot.org], but I think the fact that such a widely-deployed product as IIS 6 has been found to have only a single remote access vulnerability in the last four years [secunia.com] really speaks for itself.
I mean, I'm just a Unix guy who's never had much use for a Windows web server, but that's my $0.02...
LOLserver? (Score:5, Funny)
Re:LOLserver? (Score:5, Funny)
Re: (Score:3, Funny)
Re:LOLserver? (Score:4, Funny)
Re: (Score:2, Funny)
Re:Is Idiocracy coming true? (Score:5, Funny)
Re:Nimda Code Red Chunked Encoding...... (Score:2, Funny)
Re:Should have used *BSD (Score:2, Insightful)
Something's fishy! (Score:5, Funny)
Re:Something's fishy! (Score:5, Informative)
Re: (Score:2)
Re:Something's fishy! (Score:5, Funny)
Re:Something's fishy! (Score:5, Funny)
and all sorts of weird stuff's started happening in the server room
Re:Something's fishy! (Score:4, Funny)
Now I google it and I see it's from a dumbass TV show. I'm pissed off.
Hummm, no ahah ?! (Score:2, Interesting)
Re:Hummm, no ahah ?! (Score:4, Funny)
press release?? (Score:2, Insightful)
Yawn.
Re: (Score:2)
Yawn.
Interesting.
Someone posts a story about compromised Apache servers and all it rates from the Geek is a yawn.
Am I safe? (Score:2, Funny)
Does this rootkit work on a hardened Gentoo install with no LKM support on SPARC64? :P
-:sigma.SB
Re:Am I safe? (Score:5, Funny)
Maybe; they're still compiling it.
Re:Am I safe? (Score:5, Funny)
Re:Am I safe? (Score:5, Funny)
mkdir 1 (Score:5, Insightful)
Yes, also if you can run your tummy while patting your head you aren't infected also.
I think.... this crazy idea is the virus!
Re: (Score:2)
Well... (Score:2, Funny)
Re: (Score:2)
Uh oh. I have no idea how to run my tummy. Crap, I must be infected!
Re: (Score:2, Funny)
Re: (Score:2)
If you crap, then you DO know how to run your tummy!
Re: (Score:2)
Yes, also if you can run your tummy while patting your head you aren't infected also.
I heard that if you can spread your fingers and your hand covers your entire face, your server is infected.
Re: (Score:2, Informative)
I can see thousand of people trying to make numeric directories
I just mkdir'd a numeric directory then remembered I run OpenBSD on my net-facing servers.
Re:mkdir 1 Un-cross keys, avoid the Lahar... (Score:2)
Run your tummy makes me think of being run over, or loosing a hot bowel of a lahar surmounting, umm, surpassing even Mt. Pinatubo.
Re: (Score:2)
This isn't always the case in older variants of the rootkit. To be certain your server isn't compromised, it's best to sniff packets for a brief 3-5 minute period. You can do this using the command below:
tcpdump -nAs 2048 src port 80 | grep "[a-zA-Z]\{5\}\.js'"
That's another way to check apparently.
Re: (Score:3, Informative)
Note the / at the beginning of the grep regex.
Re: (Score:2)
$ mkdir 1
mkdir: cannot create directory `1': File exists
Read it careful people... (Score:3, Informative)
Re:Read it careful people... (Score:5, Insightful)
Re: (Score:2, Insightful)
Can't be malware (Score:3, Funny)
Isn't that always the case with FOSS. If it was for Microsoft then it would be _real_ malware....
Re:Can't be malware (Score:4, Insightful)
*sniff sniff* Is something burning?
Re: (Score:3, Insightful)
The tough question is what is the malware that is infecting the servers themselves. There have been reports of this for weeks now, and apparently it may go back months (see eg. http://www.channelregister.co.uk/2008/01/16/mysterious_web_infection_continues/ [channelregister.co.uk]), and AFAICS:
a) no one knows the initial attack vector (on the _servers_)
b) the malware (on the _servers_) seems to be difficult to detect
c) and no one seems to know how to remove it successfully either - some have suggeste
What are the common factors? (Score:5, Insightful)
To figure out what the compromise vector is, it's probably going to be necessary to figure out what the compromised servers have in common -- and how that differs from uncompromised servers. (Keeping in mind that currently-uncompromised servers may have the same vulnerability, and that attackers or their software just may not have gotten to them yet.)
I'd suggest enumerating factors such as OS, OS version, remote access methods (ssh, ftp, etc.), Apache versions, Apache modules, add-ons like CPanel, network/ASN, and so on -- anything could be a culprit at this point.
And that includes things that have nothing to do with Linux or Apache: for example, it's possible that the attackers acquired root passwords by infecting Windows systems used by administrators -- then just waited for them to initiate ssh sessions to their servers. It'd probably be best to leave all possibilities open and consider them equally likely until evidence starts accumulating in favor of/against them. (In re-reading that last statement, I suppose it sounds a bit trite. I'm just trying to discourage premature conclusions that anything is at fault until somebody can produce evidence to support saying so.)
Re:What are the common factors? (Score:5, Insightful)
Re: (Score:3, Insightful)
That (use of data harvested from ssh attacks) is entirely possible. Some of those attacks had to successful against some hosts.
So maybe one possible line of investigation would be to see if any hosts which defended against ssh dictionary attacks (say, by throttling back or denying connections from hosts that made too many ssh tries) were compromised. (I suppose it'll also be necessary to assess the strength of their root passwords; sufficiently weak ones might not require a concerted ssh attack to be com
Re:Fail2ban (Score:5, Informative)
Re:What are the common factors? (Score:4, Informative)
Other info as of last week:
Various discussions:
http://www.webhostingtalk.com/showthread.php?t=651748 [webhostingtalk.com]
(useful discussion starts on page 3 or so)
http://www.theregister.co.uk/2008/01/11/mysterious_web_infection/ [theregister.co.uk]
(describes the inability of ScanSafe to work out what's happening)
Trend have a piece on their blog:
http://blog.trendmicro.com/e-commerce-sites-invaded/ [trendmicro.com]
SANS/ISC
http://isc.sans.org/diary.php?storyid=3834&rss [sans.org]
Re: (Score:3, Informative)
This is why I treat all local root exploits as seriously as remote root exploits. All it takes is one buggy PHP script and then the attacker can try local root vulns.
The register's older writeup on this ... (Score:5, Informative)
my $.02 of course
ssh + bad password (Score:5, Informative)
* Don't allow root to ssh into your machine.
* Disable ssh1.
* Limit sudoers.
* Have good passwords.
* ???
* PROFIT!!
Seems like a formula everyone should know.
Re: (Score:2, Informative)
* Disable password authentication in SSH -- require key-based authentication
Re: (Score:3, Informative)
Attackers can trampoline onto other machines in the network if they share the same key. If you're going to do then be careful about which machines can freely contact each other, and use separate keys for each server.
Re:ssh + bad password (Score:4, Informative)
I've had admins on my network simply copy both pub & private ssh keys from server to server (they're in the same directory). They leave the private keys on the machine and forget or don't know what they've done. An attacker with root on that machine can then su into the account and access other machines.
Re: (Score:3, Insightful)
In addition, if they really might render the machine useless, they likely shouldn't have it on the 'net.
Re: (Score:3, Interesting)
No, it is not. On *BSD family of Operating Systems root can only login on the local console anyway.
If you screw something up badly, you ssh in as yourself first, and then perform `su' — something, that only members of the wheel-group (gid 0) are allowed to do.
My FreeBSD machines all run a crude log-watcher [virtual-estates.com], which blocks-out machines, from where root- and similar logins are attempted, i
Re: (Score:3, Interesting)
I was most surprised when I found that Redhat (Our cooperate Linux of Choice) appears to allow this as the default. Certainly, The Debian box i use as a home server never used to allow that, however, checking i see that since I upgraded from Woody, it does allow remote SSH as root. Thats worrying.
Well have to fix that.
Re: (Score:3, Interesting)
Allow me to insert one step before ???
* Follow-up on your SSH logs. If you see a phishing attack, do something about it!
That something could be:
- Report the IP to the owner of the netblock who can be found at ARIN [arin.net]. All netblock owners must have an IP-admin address or an abuse address. Unfortunately, my experience is that most of these go to /dev/null. There are those who actually have responsible NOC staff, and they will act on your complaint if you send them a copy of the relevant logs.
- Block furth
Re:ssh + bad password (Score:5, Interesting)
Re: (Score:3, Informative)
Re:ssh + bad password (Score:5, Informative)
I'll see your good point, and raise you a pf.conf snippet:
That's how you can block non-massively-distributed password dictionary attacks on the BSDs, anyway. Sadly, the fact that OpenBSD's firewall can perform this task on its own means that we probably won't see this feature worked into OpenSSH itself any time soon -- so on Linux you'll need a third-party script such as DenyHosts [sourceforge.net], as others have already pointed out.
(And yeah, unlike this PF configuration, DenyHosts lets you synchronize your table with a sort of universal blacklist of blocked hosts, so some might choose to run it on BSD anyway. It sounds like a good idea on paper, but boy does it suck when your home IP address keeps inexplicably winding up on the blacklist due to what turns out to be a single site's massively misconfigured server.)
But I think the most important lesson to be learned here, assuming that this thing does turn out to be an ssh attack, is that allowing single-factor, password-based administrative logins to a highly connected host is never a good idea. If you have the luxury of complete control over the site and its users (or are simply a highly empowered BOFH), disable password-based logins entirely and force the use of ssh public keys:
As a concession, if you want to ensure access without having to carry around an encryption key on a USB dongle, on Linux you can use PAM and libpam-opie to set up secondary access using a dual-factor combination of an S/Key one-time password and your regular login password (S/Key [wikipedia.org] is like Steve Gibson's much-trumpeted "Perfect Paper Passwords [grc.com]" system, which is ingenious in its own right, except that S/Key is designed so that you don't need to keep your secret key stored unencrypted on the very server you're worried about protecting):
With the above configuration you can still log in seamlessly using your ssh private key. But if you get stuck somewhere without access to your private key, you just pull your S/Key passwords list out of your pocket and enter the next password in the sequence, as prompted, followed by your login password. This PAM configuration has the nice property that if you enter the correct S/Key password but then an incorrect Unix password, you will be asked for the next one-time password in the sequence before you can continue: so unless your attacker is exceptionally good at plaintext attacks on large cryptographic hashes, a successful brute-force attack becomes impossible.
Wow, this post got a lot longer than I wanted it to... I'm, um, going out to get some fresh air or something.
Re: (Score:3, Interesting)
I know "security through obscurity" isn't really secure, but it has entirely eliminated attempts to crack the root password on all the servers I run.
Re: (Score:3, Funny)
Which is hardly an advantage on Linux because everybody can su to root. We have RMS to thank for that one. Apparently the GNU way is fairer to the users.
More details are available... (Score:4, Informative)
http://blog.trendmicro.com/e-commerce-sites-invaded/ [trendmicro.com]
If you happen to have one of these compromised systems, I am sure that Trend would like to talk to you about it...
I'm not sure I buy it (Score:5, Insightful)
I would bet the path of the TCP/IP packets route through compromised providers who have an injection strategy. Remember a few months ago how IPSs were injecting their own java script and ads into the pages of other sites?
http://ars.userfriendly.org/cartoons/?id=20070703 [userfriendly.org]
This is the most likely scenario I can think of.
Re: (Score:3, Interesting)
\begin{snarky}
I'm surprised some of these "admins" can find their servers, let alone moderately well hidden rootkits.
\end{snarky}
Many system administrators do not have a deep background in *nix security. If they can install a Linux box, they're apparently qualified. There are many admins who are extremely competent in security matters, but I have not seen anything coming from those people. (Perhaps they weren't infected?) So, I have not heard (read) of anything fr
Re: (Score:2, Funny)
Re:Ubuntu as well? (Score:4, Funny)
Re:Ubuntu as well? (Score:5, Funny)
Re:Ubuntu as well? (Score:4, Insightful)
so basically its most likely they used the traditional means of gaining access (not through holes, but merely through bad personal security practices regarding passwords and password management). And it only affects windows clients. So how is this problem not your typical someone cracked your machine? Oh wait, I smell Microsoft FUD... ewwwwww
Re:Ubuntu as well? (Score:5, Insightful)
Re:Ubuntu as well? (Score:4, Insightful)
Other than using and safeguarding secure root passwords, not much can be done at this time to be proactive in preventing servers from being compromised,
Turn off root's log in and get rid of cPanel and similar programs as well. I understand the need for an easy to use remote admin tool (as much as I'd love people to actually learn the shell), but can't we do better than a web-based program for this stuff?
Re:Ubuntu as well? (Score:4, Interesting)
GUIs provide metaphors for users, they have no place in administration.
</grumpyOldFart>
Re: (Score:3, Funny)
One great unknown thus far is how the servers come to be infected.
So as long as it defends your precious *nix community, and lays potential blame at the door of MS, it is perfectly acceptable practice to make accusatory conclusions with no evidence or proof. This kind of MS bashing just makes the *nix community look like desperate hypocrites, and only furthers my resolve to continue supporting the MS platform for another 15 years of satisfied usage.
Why can't you just accept the fact that everyone knows that every platform is vulnerable to some extent, and probably 90
Re:Ubuntu as well? (Score:5, Insightful)
If the current thinking is indeed that the Linux servers were inappropriately accessed through stolen passwords, how is that a security flaw of Linux or Apache? Like he asked, how is using a legitimate password equal to cracking the server?
On the other hand, turning Windows clients into bots *IS* an example of that software's (and QuickTime's and Yahoo! Messenger's) insecurity and vulnerability!
Re: (Score:3, Insightful)
If the current thinking is indeed that the Linux servers were inappropriately accessed through stolen passwords, how is that a security flaw of Linux or Apache? Like he asked, how is using a legitimate password equal to cracking the server?
On the other hand, turning Windows clients into bots *IS* an example of that software's (and QuickTime's and Yahoo! Messenger's) insecurity and vulnerability!
Using a legitimate password is not equal to cracking the server. But it must be made to look so because the PR firms the M$ movement uses must cast aspersions on Apache and Linux so as to draw attention away from the actual insecure and vulnerable system. Most PHBs never read past the headlines, so this is major spin for the M$ party.
Re: (Score:3, Insightful)
From TFA - "All reports thus far say the compromised servers are running Linux and Apache."
"And it only affects windows clients. So how is this problem not your typical someone cracked your machine? Oh wait, I smell Microsoft FUD"
Are you really that illiterate? Just an FYI - Microsoft doesn't make Apache OR Linux. If compromised severs are being used, it is certainly not the type that "only affects windows clients". Duh. Only here would such blatant anti-MS bullshit get modded "Insightful". I took a way more "insightful" shit this morning.
So, Mac planet is not alone discrediting every single security alert as "FUD" :)
It seems there are other people who sees a story validated by 4 different, independent security companies as FUD. Apache is planets number 1 webserver and Linux is number 1 Webserver OS. What else would a blackhat supported by mafia would target? It is not like "I am proving Linux is unsecure", it is "I have purchased a previously unknown compromised account list and I am using it to infect millions of MS Windows users running
Re: (Score:3, Funny)
Well I am sure the 3% of the population that don't fit into either category are relieved as hell.
Re: (Score:3, Insightful)
In general, these systems probably don't even give users shell access, and even then, password cracking is probably out of the picture. And brute forcing a password over ssh? thats probably troublesome too, it would be hard to get over about 50 attempts/second.
On the other hand, there are a whole bunch of local vulnerabilities that can be exploited, after a system is compermised. In some cases, a weak php include vulnerability could potentially allow the ap
Re: (Score:3, Informative)
Re:Ubuntu as well? (Score:4, Insightful)
It's possible to install software on a Linux webserver that exploits vulnerabilities in Windows clients. This is news?
Here's a shocker: it's possible to exploit Windows boxes with services hosted on a Commodore64.
Windows has more malware packages than legitimate software packages. They've really solved that ease of installation problem.
Re:Funny (Score:5, Insightful)
Problems with IIS were as a result of vulns in the application and/or Windows operating system - totally different problem.
Would you blame a lock company if the user left his keys in the lock?
Ed
Re:Funny (Score:5, Insightful)
In other words, they have no idea how the servers were compromised. Because they can't find out how, they're guessing it was a root password that was stolen. In other words, its still just as likely a flaw in some software.
Re:Funny (Score:5, Insightful)
A pretty good guess, otherwise we could expect to see millions of Apache web servers compromised (there are over 75 million Apache web servers in active service) and anticipate a much greater number of Windows clients infected.
The significance of this story is not that Windows clients are the target, the significance is that the infecting agent is originating from Apache/Linux servers.
Ed
Re: (Score:2, Interesting)
I agree with your reasoning on the significance of the story.
Re:Funny (Score:5, Funny)
Depends. How good is my lawyer?
Re:Funny (Score:4, Funny)
I think their class restricts them to Lawful Evil; should they change alignment, they et disbarred. So, none, at a guess.
Re: (Score:2, Interesting)
Please let me know what the last critical security flaw for IIS was. I'd love to know.
Also, let me know how many critical security flaws there have been for Apache in the last year or so.
Thanks!
Re: (Score:2)
Re: (Score:3, Informative)
I think it's funny that Apache is affected by the same drama that affected IIS all those years ago.
Except IIS had security hole after security hole.
There's been no such security hole found in apache yet. So I'd wait before making comparisons to IIS.
Re: (Score:3, Insightful)
Re: (Score:2, Insightful)
Re: (Score:2, Funny)
Re: (Score:3, Informative)
Now then, which admin is first for choosing bad passwords?
Re:Software sucks. (Score:5, Insightful)
1) If the market really wanted extensive 'software liability' then we'd already have it. Customers would demand it, suppliers would figure out how much it would cost to provide it, and prices would sort themselves out. Turns out the prices go WAY up, and customers (most of them) don't want to pay them.
2) What happens to Linux in a world with mandatory software liability? Who is liable? The company providing install and support? The volunteer contributor who wrote that line of code? The project maintainer who accepted the patch?
Re:Software sucks. (Score:5, Funny)
Re:Software sucks. (Score:5, Insightful)
Even then, there is no ability to develop your skills because you spend 99% of your time learning new environments.
Software is HUGELY complex these days and it takes a log of study, knowledge, and skill to be any good at it. Companies don't want to hear that. They want to increase productivity by "KLOC." (Un)fortunately, there is a lot of "art" and "creativity" in software development and without well defined product specs, rigid test plans, and quality assurance which adds delays and cost to a project you won't get better code.
Standard business upside potential vs downside risk. Upside potential: first to market, profit!!! Downside risk: blame some hacker.
Re: (Score:2)
GOD DAMNIT! How am I becoming old?
Comment removed (Score:5, Informative)