Mystery Malware Affecting Linux/Apache Web Servers

lisah writes "Reports are beginning to surface that some Web servers running Linux and Apache are unwittingly infecting thousands of computers, exploiting vulnerabilities in QuickTime, Yahoo! Messenger, and Windows. One way to tell if your machine is infected is if you're unable to create a directory name beginning with a numeral. Since details are still sketchy, the best advice right now is to take proactive steps to secure your servers. 'We asked the Apache Software Foundation if it had any advice on how to detect the rootkit or cleanse a server when it's found. According to Mark Cox of the Apache security team, "Whilst details are thin as to how the attackers gained root access to the compromised servers, we currently have no evidence that this is due to an unfixed vulnerability in the Apache HTTP Server." We sent a similar query to Red Hat, the largest vendor of Linux, but all its security team could tell us was that "At this point in time we have not had access to any affected machines and therefore cannot give guidance on which tools would reliably detect the rootkit."'"

Re:Something's fishy!

[JeepFanatic]

If you run those values through a hex to ascii converter you get SKYNET

Re:Funny

[Vellmont]

I think it's funny that Apache is affected by the same drama that affected IIS all those years ago.

Except IIS had security hole after security hole.

There's been no such security hole found in apache yet. So I'd wait before making comparisons to IIS.

Read it careful people...

[cbart387]

The servers are linux (because of an access issue. The computers being hurt by this are windows. At least that's how I read the article (see quote from article below).

According to a press release issued earlier this month by Finjan, a security research firm, compromised Web servers are infecting thousands of visitors daily with malware that turns their Windows machines into unwitting bots to do the bidding of an as yet unidentified criminal organization. Security firms ScanSafe and SecureWorks have since added their own takes on the situation, though with varying estimates on the number of sites affected. All reports thus far say the compromised servers are running Linux and Apache.

The register's older writeup on this ...

[chris.dag]

The Register has been on this for a while and although the story is older it is better written and has more interesting details: http://www.channelregister.co.uk/2008/01/16/mysterious_web_infection_continues/ [channelregister.co.uk]

my $.02 of course

ssh + bad password

[Panaflex]

I see this type of attack all the time, the fact that someone automated it and gave it a zombie machine is not surprising.

* Don't allow root to ssh into your machine.
* Disable ssh1.
* Limit sudoers.
* Have good passwords.
* ???
* PROFIT!!

Seems like a formula everyone should know.

Re:mkdir 1

[grub]

I can see thousand of people trying to make numeric directories :)

I just mkdir'd a numeric directory then remembered I run OpenBSD on my net-facing servers. :P

Re:ssh + bad password

[whoever57]

* Don't allow root to ssh into your machine.

Dangerous if you don't have easy physical access to your machine. It is possible to screw up a machine in such a way that a normal user cannot log in, but root can. It is better to:

* Disable password authentication in SSH -- require key-based authentication

Re:ssh + bad password

[Panaflex]

That's a good idea - but be careful!

Attackers can trampoline onto other machines in the network if they share the same key. If you're going to do then be careful about which machines can freely contact each other, and use separate keys for each server.

Re:Ubuntu as well?

[Anonymous Coward]

And the malware infects Microsoft clients, genius.Don't get me wrong, I think this is a big deal, but his point is that unless you are running Windows OR have an Apache webserver this doesn't effect you. Linux desktop users are not effected.

More details are available...

[Anonymous Coward]

... though a solution has not been yet:

http://blog.trendmicro.com/e-commerce-sites-invaded/ [trendmicro.com]

If you happen to have one of these compromised systems, I am sure that Trend would like to talk to you about it...

Re:Software sucks.

[0racle]

I agree. The people who made this problem possible should be sued and held accountable.

Now then, which admin is first for choosing bad passwords?

Re:mkdir 1

[padyer]

That regex is b0rken. It matches any file of at LEAST 5 chars that ends in .js. The story at cPanel says that it is exactly 5 random chars and then the .js. Should be changed to

tcpdump -nAs 2048 src port 80 | grep "/[a-zA-Z]\{5\}\.js'"

Note the / at the beginning of the grep regex.

Re:ssh + bad password

[MrDoh1]

I've found that Denyhosts is a nice tool to take care of securing SSH and blocking hosts of incorrect SSH attempts.

Re:ssh + bad password

[Archiviste]

You should also have some process that completely blocks ssh login attempts from a given IP after so many failed login attempts instead of letting the hi-jacker poll your machine for as long as he wishes.

Not quite "after so many failed login attempts", but useful nonetheless: DenyHosts [sourceforge.net]

Re:ssh + bad password

[Crimsonjade]

I personally use http://www.fail2ban.org/wiki/index.php/Main_Page [fail2ban.org]

Re:What are the common factors?

[imipak]

Apparently it's not Cpanel.

Other info as of last week:

Various discussions:
http://www.webhostingtalk.com/showthread.php?t=651748 [webhostingtalk.com]
(useful discussion starts on page 3 or so)
http://www.theregister.co.uk/2008/01/11/mysterious_web_infection/ [theregister.co.uk]
(describes the inability of ScanSafe to work out what's happening)
Trend have a piece on their blog:
http://blog.trendmicro.com/e-commerce-sites-invaded/ [trendmicro.com]
SANS/ISC
http://isc.sans.org/diary.php?storyid=3834&rss [sans.org]

Re:Fail2ban

[whoever57]

Fail2ban is another nice way to deal with these brute force attacks.

You can use fail2ban, but SSH can be protected very nicely with Netfilter/IPTABLES -- just limit the rate of new connections to something like 3 per 3 minutes for each host and this will slow down any dictionary attack to the point that it is very unlikely to be successful.

Re:Ubuntu as well?

[Isauq]

Right, right, they're running your typical LAMP stack. You know, like most of the internet. Statistically speaking, if you have a site, you more than likely have a site served by Apache on Linux. In truth, I've heard of very few servers that receive significant traffic that DON'T run Apache and even fewer that don't run Linux. As the internet is based around open international standards, there's no reason a Linux-based server couldn't serve packets containing harmful windows executable code. Your first point is a non-issue.

As to your second point, it's only natural that windows machines be targeted. More critical security vulnerabilities as part of the base operating system that is almost certainly being run as root ("administrator" if you've never used *nix) means greater capability for general chaos. Alternatively, more useful machines for ye olde botnette. One problem with targeting Linux machines is the Unix permissions model that would create a situation where even if someone were to find a hole by which they could access the system, they would still need to find a method by which to elevate their user privileges to root so that they could accomplish more than manipulation of the user's home directory. This leads to the second problem- security flaws in a *nix system are almost certainly related to the software installed on them rather than the Linux kernel itself, making it a roulette game whether your particular method of attack is even present to be exploited.

In a system that has been systematically secured by experts from all callings for years on end, it becomes, with each patch level, more and more likely for the human equation's unreliability to be the single greatest point of failure. Being fallible, people resort to insecure data practices for their own convenience, out of laziness, from a lackadaisical attitude, or out of habit; thus creating a situation where the likelihood of a partial or full breach rapidly approaches one. This is a well known point of failure, and is even counted on, at some level, with a sane backup policy and data redundancy.

What's more, while rootkits and their dangers are very real, one cannot say that it is a vulnerability of a system that someone in possession of what is assumed to be a secure superuser password can install software on that system. Were you to steal the keys to a car, you certainly wouldn't find it strange be able modify the engine of said vehicle- after all, with keys you can unlock the door and with a minimum of effort pop the hood and go to work.

So yes, my dear Coward, grandparent was correct- this is somewhat more elegant than we're used to seeing, but it is most certainly presented in a way thst prompts one to think that there is something "wrong" with Apache or Linux.

Re:ssh + bad password

[Niten]

I'll see your good point, and raise you a pf.conf snippet:

### MACROS AND TABLES SECTION
table <wan_bruteforce> persist

### PACKET FILTERING SECTION
block in quick on $if_wan inet from <wan_bruteforce>
# ...
pass in on $if_wan inet proto tcp from any to ($if_wan) \
port ssh flags S/SFRA synproxy state \
(max-src-conn-rate 3/30, overload <wan_bruteforce> flush global)

That's how you can block non-massively-distributed password dictionary attacks on the BSDs, anyway. Sadly, the fact that OpenBSD's firewall can perform this task on its own means that we probably won't see this feature worked into OpenSSH itself any time soon -- so on Linux you'll need a third-party script such as DenyHosts [sourceforge.net], as others have already pointed out.

(And yeah, unlike this PF configuration, DenyHosts lets you synchronize your table with a sort of universal blacklist of blocked hosts, so some might choose to run it on BSD anyway. It sounds like a good idea on paper, but boy does it suck when your home IP address keeps inexplicably winding up on the blacklist due to what turns out to be a single site's massively misconfigured server.)

But I think the most important lesson to be learned here, assuming that this thing does turn out to be an ssh attack, is that allowing single-factor, password-based administrative logins to a highly connected host is never a good idea. If you have the luxury of complete control over the site and its users (or are simply a highly empowered BOFH), disable password-based logins entirely and force the use of ssh public keys:

# /etc/ssh/sshd_config
PubkeyAuthentication yes
ChallengeResponseAuthentication no
PasswordAuthentication no
KerberosAuthentication no
GSSApiAuthentication no
UsePAM no

As a concession, if you want to ensure access without having to carry around an encryption key on a USB dongle, on Linux you can use PAM and libpam-opie to set up secondary access using a dual-factor combination of an S/Key one-time password and your regular login password (S/Key [wikipedia.org] is like Steve Gibson's much-trumpeted "Perfect Paper Passwords [grc.com]" system, which is ingenious in its own right, except that S/Key is designed so that you don't need to keep your secret key stored unencrypted on the very server you're worried about protecting):

# /etc/ssh/sshd_config
PubkeyAuthentication yes
ChallengeResponseAuthentication yes
PasswordAuthentication no
KerberosAuthentication no
GSSApiAuthentication no
UsePAM yes

# /etc/pam.d/ssh
auth requisite pam_opie.so
auth required pam_unix.so nullok_secure

With the above configuration you can still log in seamlessly using your ssh private key. But if you get stuck somewhere without access to your private key, you just pull your S/Key passwords list out of your pocket and enter the next password in the sequence, as prompted, followed by your login password. This PAM configuration has the nice property that if you enter the correct S/Key password but then an incorrect Unix password, you will be asked for the next one-time password in the sequence before you can continue: so unless your attacker is exceptionally good at plaintext attacks on large cryptographic hashes, a successful brute-force attack becomes impossible.

Wow, this post got a lot longer than I wanted it to... I'm, um, going out to get some fresh air or something.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:What are the common factors?

[Alioth]

It's quite possibly due to buggy PHP scripts. I've seen it before; what happens is the attacker goes for some unpatched vulnerability in PHPnuke, PHPbb or similar software. This gets them non-root access. They use this to 'wget some-hack.c' to the /tmp directory, build this hack then execute this to exploit a local root exploit.

This is why I treat all local root exploits as seriously as remote root exploits. All it takes is one buggy PHP script and then the attacker can try local root vulns.

Re:Ubuntu as well?

[hostyle]

This is old news. Its caused by a rootkit: http://www.cpanel.net/security/notes/random_js_toolkit.html [cpanel.net]

Re:Should have used IIS

[Niten]

In all seriousness though, IIS 6 has a pretty darn good security track record; seemingly better than Apache 2's, even if it is blasphemy for me to say it. I've previously decried the use of raw vulnerability statistics to make comparative claims about different products' security [slashdot.org], but I think the fact that such a widely-deployed product as IIS 6 has been found to have only a single remote access vulnerability in the last four years [secunia.com] really speaks for itself.

I mean, I'm just a Unix guy who's never had much use for a Windows web server, but that's my $0.02...

Identified about 5 months ago

[fmavituna]

I identified this rootkit in a system about 5 months ago and slightly documented some behaviours of it (I think only behaviour I've missed was numerical directory thingy). Related blog post 25.08.2007 - http://ferruh.mavituna.com/makale/exploit-paketleri/ [mavituna.com] ).

There is one more thing to add, it modify all valid HTTP responses, add .js after body tag in all interfaces. There was one article that mentioned most of the compromised servers based UK, it was same for me. And considering it's been about 5 months, I assume UK websites were prime target in the start.

Re:Ubuntu as well?

[budgenator]

What that means is that probably every server in the data center had the same root password and somebody leaked it or sold it. We had a server that was managed by command dental system, and every system they sold had one of 5 root passwords which quickly became common knowledge in the industry.

Re:ssh + bad password

[Panaflex]

Sorry, I wasn't clear:

I've had admins on my network simply copy both pub & private ssh keys from server to server (they're in the same directory). They leave the private keys on the machine and forget or don't know what they've done. An attacker with root on that machine can then su into the account and access other machines.

Re:Funny

[Niten]

Except IIS had security hole after security hole.

That's a lie [secunia.com]. I mean, ten years ago, maybe; but IIS today is pretty damn secure by anybody's standards.

Where are all these vulnerabilities that you insist exist in IIS, from any time during the last five years? OSS FUD doesn't smell any better than Microsoft FUD.

Re:The register's older writeup on this ...

[Mediamon]

Some additional reports from earlier this week and previous...

http://blog.trendmicro.com/e-commerce-sites-invaded/ [trendmicro.com]

http://www.scmagazineus.com/Attack-injects-malicious-JavaScript-into-e-commerce-sites/article/104206 [scmagazineus.com]

http://www.theregister.co.uk/2008/01/11/mysterious_web_infection/ [theregister.co.uk]

http://www.cpanel.net/security/notes/random_js_toolkit.html [cpanel.net]

http://isc.sans.org/diary.html?date=2008-01-18 [sans.org]

http://isc.sans.org/diary.html?date=2008-01-14 [sans.org]

http://www.webhostingtalk.com/showthread.php?p=4902045 [webhostingtalk.com]