Cracked Linux Boxes Used to Wield Windows Botnets

m-stone writes "Online auction house eBay recently did a threat assessment to better understand the forces ranging against them. The company is keeping the fine details under wraps, but the biggest source of danger for the company is apparently botnets. You're never going to guess who was running them. '[Dave Cullinane, eBay's chief information and security officer] noticed an unusual trend when taking down phishing sites. 'The vast majority of the threats we saw were rootkitted Linux boxes, which was rather startling. We expected Microsoft boxes,' he said. Rootkit software covers the tracks of the attackers and can be extremely difficult to detect. According to Cullinane, none of the Linux operators whose machines had been compromised were even aware they'd been infected. Because Linux is highly reliable and a great platform for running server software, Linux machines are desired by phishers, who set up fake websites, hoping to lure victims into disclosing their passwords."

Confirmed

[mccalli (323026)]

I've noticed a large increase in attempts to crack my co-lo Linux servers recently, and it must be said that two got through (shared site, some customers running old content management apps and the kits hit). When we watched the behaviour of the cracked box, it was connecting back to...I think undernet.org or similar?...and sending controls via IRC. Plus doing a spot of spamming of its own bat.

Our set-up is that we have a host OS install doing nothing but running VMware Server and then any real stuff gets done in a VM, so this was easy for us to recover from quickly via VM snapshotting. But still, it's a trend that's noticeably on the increase.

Cheers,
Ian

Re:Confirmed

[mdeslaur (530851)]

Even phishers like Linux better than Windows!

Happens to sites that hosts others too...

[Shivetya (243324)]

Nothing like getting a stupidly high bandwidth bill to find out your hosting server has been hacked. Its even better when you have to fight them to prove its their fault for being hacked and not yours for being cohosted by them!

and yes they are running Linux... they apparently didn't cover all their bases and were caught by more than one known exploit and some default settings.

Just because its Linux does not make it secure, you actually have to use it correctly.

Brute Force Attacks

[superbrose (1030148)]

I'm sure pretty much everybody who is running a Linux server (or any server as a matter of fact), especially with services like SSH enabled, is currently subject to brute force attacks.

When I looked at my auth log I noticed a huge amount of brute force attacks for all my servers, so I installed denyhosts [sourceforge.net], which seems to work fine.

I guess the problem is also that in many distributions SSH servers are configured to allow root logins, and if nobody looks at the log files these go totally unnoticed.

Re:Brute Force Attacks

[Russell Coker (125579)]

Run your sshd on a port other than 22. Most attackers only scan the well-known ports. Running your sshd on a different port removes a lot of the noise from your logs and allows you to concentrate on the real issues.

The "Host" sections in the /etc/ssh/ssh_config file allows you to specify which port to use for each host you connect to (so you don't need to type "-p 1234" every time you connect).

Re:Brute Force Attacks

[jwo7777777 (100313)]

Guns don't kill people, catchphrases kill people.

Re:Confirmed

[Bert64 (520050)]

This is nothing new, crackers have always preferred unix machines for a number of reasons. A few years ago many crackers wouldn't even bother trying to own windows machines.
You never see many people who compromise a windows machine and manually set up anything on it, windows machines are typically mass hacked and used as throwaway systems, for spamming or dossing (once a large flood of dos or spam comes from a system, it very quickly gets noticed and the system usually gets shut down). The hassle of using windows remotely (half assed command line interface etc), lack of default tools and typical low uptimes/stability discourage them being used interactively or for any kind of non-throwaway uses.

Conversely, unix machines are typically more stable, and have a far more flexible interface that's more geared up to remote cli usage. Installing something like an IRC server to collect malware is often much easier, and there's usually package management which can be used to easily install any external libraries or additional tools that might be required. There are also typically standard server apps installed and ready to use (ftpd, apache, rcp, tftp etc) which can be used to host malware, for easy download to other compromised machines (most systems have ftp/rcp/tftp clients by default, even windows).

Crackers will often turn a compromised unix machine into their "home", and keep a set of tools/exploits in a hidden directory, and use the machine for manual probing, testing of new tools and launching of other attacks, but they will rarely use windows systems for anything other than dossing/spamming or defacing a website if it hosts one.

Re:Confirmed

[mccalli (323026)]

How are they trying? My logs show lots of attempts at phpbb, etc. vulnerabilities.

phpbb, Drupal and PHPNuke attempts mostly. Plus old sshd vulnerabilities, though we're up to date there and nothing got through.

Cheers,
Ian
(oh yeah, and first post! Only took a mere eight years or so...)

Re:Confirmed

[jackharrer (972403)]

I've seen the same. Actually my server has been offline for last few days as it became compromised and I don't have time to sort it out.
I got like thousands of bruteforce attacks on ftp plus some on phpBB.
I also noticed few weeks ago that when they couldn't break in they just DDosed it.

It looks like it's getting serious, especially if you're server is registered with some DNS name, not just IP.

Re:Confirmed

[mccalli (323026)]

Fortunately, each user has their own account, so i can easily check which user owns any malicious processes or files that appear on the system.

May want to be careful about that assumption. A lot of these things go out under the apache user and the mails via the [e-mail] account.

Look for tell-tale things like apache processes running when you're an apache2-only site (they're disguised processes that are really something else, obviously). Do an ls -al in all the home directories, look for directories whose name is just a space character, check /tmp isn't mounted executable...that kind of thing.

Cheers,
Ian

Re:Confirmed

[TheRaven64 (641858)]

Having a different filesystem makes sense for /tmp, because you do not care particularly about integrity. You can mount it async and disable journalling, because you don't expect data there to persist beyond a reboot. Having it as a separate mount (which can be achieved via remount) also allows you to disable support for setuid binaries on it, or even just disable execution. That's why it's always been fairly standard practice for it to be a separate slice. The same applies to /var, which typically wants to be write-optimised, while /usr wants to be read-optimised.

Re:Speaking as a Bot...

[rtb61 (674572)]

More likely the prefer Linux, because after going to all the time and effort of creating a botnet you don't want some other cracking asshat hijacking your botnet.

With windows of course those poor hard working crackers and continually having to rebuild their botnet as other crackers pilfer their bots as readily as they orginally gained, 24/7 no rest for the wicked.

So winbots while easy to gain are nearly impossible to keep because of course they are just so slutty, they are anybodies ;).

Maybe ....

[Chrisq (894406)]

Maybe the slashdot host had to pause to update a botnet;-)

true

[by Anonymous Coward]

I work in security and this is consistent with my experience.

A fair amount of it, I'm sorry to say, is due to the perception that Linux boxes are much more secure than Windows and therefore don't need (a) up-to-date patches (b) proper security reviews of any app code (which these days usually means web apps) (c) defence in depth (block outbound connections from your web server, except for a hole poked in tcp|udp/53 to/from your DNS server if needed (d) proper security monitoring. Review your firewall logs! Run an external syslogNG box! use netflow, nagios, ntop etc -- can you account for all the packet flows from the machine? If you have time to spare, look into Snort.

Thus proving Linux is not as secure as touted.

[by Anonymous Coward]

Get the facts. [microsoft.com]

Remote ease-of-use

[SnowZero (92219)]

This really doesn't suprise me. With tools like ssh and shells installed by default, Linux is just plain easier to use remotely. Linux machines would also tend to stay up and online, whereas (predominantly Windows) desktops are often shut off when not in use. So, Linux makes the best "control console" for a botnet. The "army" should still be made up of Windows desktop machines, due to their large numbers.

Good News & Bad News

[eldavojohn (898314)]

It's the double edged sword of software popularity.

Linux is becoming so respected and desired as an operating system for servers that phishers & hackers are slowly turning their attention towards it being profitable.

I think this will be the true test for Linux to prove that it can beat Windows in all departments.

I actually see this as good news although I must confess that when I get home I'm going to check & double check the configurations on the ports on my router and all my Linux boxes. When toying with app servers & apache, I have noticed tons of port scanners probing my Linux boxes. I paid them no mind although now ... perhaps I should.

Re:Good News & Bad News

[morgan_greywolf (835522)]

Yes. You should Here's what I do. (I guess you could say these are some security tips for those running Linux boxes at home and leaving them up on the Net):

• Run a hardware NAT firewall/router. Any ol' Linksys, Dlink or Netgear thang will do. Just remember it's not the be all and end all to security problems.
• Open as few ports as absolutely possible. I have nothing open on my router except port 22 and BitTorrent, and I don't leave BitTorrent running all the time
• Check your logs at least once a day. Look for any suspicious signs -- missing log entries, ssh connects you weren't expecting, services running that you don't normally have running, NICs going into promiscuous mode unexpectedly, excessive mail being pumped through any MTAs, etc.
• When running OpenSSH, I disallow password authentication. This prevents problems with users due to the use of stupid passwords. My sshd only accepts a valid RSA key exchange as acceptable authorization.
• Regularly update and run rootkit checkers. These are not be all end all, but they help spot obvious rootkits
• Make cron jobs that regularly scan your system for unusual permissions -- world writeable, binaries that are setuid, etc. and for suspicious files. There are programs and scripts that will do this for you. STFW or check with your distro.
• Perform MD5 checking on your files and executables, espcially.
• Regularly check your /etc/passwd and /etc/group files for new or unusual entries.
• Don't run NIS -- it's inherently insecure. You should be using OpenLDAP if you need directory authorization on your network.

Windows vs. Linux

[derian_cf (1167901)]

I work for a fairly well known dedicated server provider. If I had to give a rough estimate, I'd say we're 40% Windows and 60% Linux environments. Not surprisingly, the number of boxes that get hacked (rooted entirely or not) is about equal between the two, however the purpose for which they're hacked is generally quite different. 80% of the hacked Linux boxes are used for UDP floods, things like that. Also IRC bots. Interestingly enough, in my 6 months working there, I don't believe I've ever seen a Windows box used for phishing. They're always used for FTP servers hosting movies/music/programs and/or IRC servers doing the same thing.

Interesting to note

[thegnu (557446)]

I think it's interesting to note that while we get submerged in a barrage of Windows trolls, that the hackers hack one or a few Linux boxes and use them to control the hundred or more Windows boxes they've hacked.

Still looks bad for Windows. Plus, here's betting they're servers, and not home computers behind a plain old linksys router.
-Nathan

The Money Quote

[The New Andy (873493)]

eBay recently did an in-depth analysis of its threat situation, and while the company is not releasing the results of this analysis, it did uncover a huge number of hacked, botnet computers, said Dave Cullinane, eBay's chief information and security officer, speaking at a Microsoft-sponsored security symposium at Santa Clara University.

I'm not denying that Linux boxes can be (and are) hacked, but the circumstances for this particular quote seem a little shady. It seems a little irresponsible (on the part of the submitter) to not mention the money trail. And it seems a little strange not to release the results... what are they afraid of?

I've seen a few of these

[wizman (116087)]

The company I work for performs emergency Linux support services. We get a lot of calls from peoples boxes who are attacked. I've seen at least two eBay/PayPal phishing sites recently. In both cases, it had nothing at all to do with Linux itself.

Case #1: Customer running a web server had vulnerable PHP applications (I believe it was an outdated WordPress). Someone was able to use this vulnerability to wget a few php scripts and bury them under some subfolders.

Case #2: Customer had a non-root account with a weak password. This account was in the "root" group, giving it write access to a number of system files. Cracker was able to brute force the password quite easily, make a directory called eBay under /var/www/html, and stick some php code in there.

In both cases, the php scripts were logging username and password guesses into a text file. The text file was within the same web root, allowing the cracker to easily grab the latest passwords over http instead of needing to re-crack. Also, in both cases, there were at least a dozen usernames and passwords in the text files.

The lesson: Keep your web apps up to date, use strong passwords, and don't add anyone to the root group.

double standard

[nomadic (141991)]

Because Linux is highly reliable and a great platform for running server software, Linux machines are desired by phishers

So when phishers target windows servers, it's because windows has horrible security, but when they target linux servers, it's because linux is just awesome?

Root not needed to host phishing scams

[rjamestaylor (117847)]

I am a supporting system administrator for Linux/UNIX servers at a large hosting
company. I have come across many Linux servers that are compromised and being
used to host phishing scams, spamware, IRC servers, etc. Rarely, however, do I
see a "root'ed" server -- that is, a server on which an unauthorized
person or program has gained root privileges illicitly. In fact, having root
access is not necessary to host web content, send mail or provide other
Internet-facing services.

All that is needed is the privilege to put content served by the web server in
place. That could be a script for server-side execution, page or fragment for
browser- (client-) side execution, etc. If you can upload to the web content
(DocumentRoot or include) directories and the web server automatically servers
that content, you, too, can host a phishing scam or illicit media for download.

If a directory in the DocumentRoot tree on a web server can be written to by the
web server (the apache or nobody system account) then it is easy to inject one's
illicit content on that server. OS is irrelevant at that point. In fact, if a
web server has world- or apache-writable directories in the web content area the
OS *must* allow any web client to upload whatever they desire to that server.
It is the responsibility of the owner of the server to restrict who gets to
upload what content to his/her server.

I try to explain to web designers that granting write access to the
apache/nobody user is BAD, but often I hear back: "Ya, but, I can't make
the script work without opening the permissions." Usually, this is done on
PHP Content Management System portal sites that allow content to be uploaded
directly from the web browser by arbitrary users. There is a little bit of
effort required to make doing this difficult -- and it can be tricky to get
right -- but forcing the script to work by removing world/apache write
privileges is EASY:

$ sudo chmod -R 777 /var/www/html

Ugh. Then, when that same customer is complaining that, "Hey! I've been
hacked!" I respond, "no, you haven't. You been compromised. You
allowed *anyone* to upload *anything* to your server and set apache to
automatically server that content. You were trusting *everyone* on the Internet
to behave. Your trust was broken and now your server is distributing phishing
scams/malware/kidde porn/spam."

If you ever think you need to "open up" permissions so your PHP script
will "run right" you either need a different PHP script or help making
the script run "safely." It's harder than chmod'ing 777 but it's
definitely worth doing.

One server I worked on had a lazy owner who allowed apache full write and
execute access to his web content directories. He would not upgrade his PHP
scripts to patched versions that plugged well-publicized holes. After repeated
warnings I received a frantic call from him that his server was
"hacked" and running a banking phishing scam. I checked the weblogs
and found that 20,000 people had clicked the phishing scam links from their
webmail inbox and retrieved the malware-ladden web pages with Internet Explorer
-- meaning many of these people were sending their data right to the
Russian/terrorist criminals for funding their illicit operations. The customer
asked that I call the FBI to "find out who is responsible" and I said
I didn't need to make that call to find out: he was responsible.

That customer is now fully-turned around and is complying with the necessary
steps to ensure that his server is not used for illicit purposes any longer.

Root was never required for these compromises. Just poor administration.

Some comments on rootkits

[ajs318 (655362)]

There's a particularly nasty rootkit out there which overwrites certain system programs (such as ls, ps, netstat, md5sum and a few others) with modified versions, then does a chattr to stop you overwriting them (though lsattr is left alone). And while attempting to clean up a machine so infected, I've seen Perl scripts changing the value of $0. This means even if you've got a "clean" ps around (like a copy of busybox in your own non-root home directory ..... you do have a non-root login, don't you?), it will report the "wrong" thing. Another clue that this rootkit is installed, is that (at least on Debian and Slackware) coloured directory listings don't work properly, and invoking ls generates a non-fatal error message. (The "special" ls must be based on an older version.)

The www-data (Debian / Ubuntu) or apache (Fedora) user should not be running any process other than apache2 or httpd. If you see something like "accepting connections", that's a sign that someone could be running something nasty.

In general, watch for world-writable directories (they list with a green background in Debian) because that's one of the first steps in cracking a box ..... install a script in a user's home directory, then persuade it to run. Beware of badly-written PHP scripts which don't chmod uploaded files to make them non-executable (turning off short open tags is also surprisingly effective). And what you think might be a DDoS (repeated attempts to retrieve mail on nonexistent accounts via POP3) might actually be a password-guesser. Block the /24 with an iptables rule at once. Note, if you aren't within walking distance of your co-lo, make your first firewall rule
iptables -I INPUT 1 -s 10.20.30.40/32 -j ACCEPT
(replace 10.20.30.40/32 by a subnet specifier which will always contain your own IP address -- get this from your broadband company -- and just to make you all jealous, my one ends in /32 because my IP is static) and never, ever use -I INPUT 1; use -I INPUT 2 or -A INPUT instead. It's too easy to block yourself out with an injudiciously-applied rule (and I do live within walking distance of my co-lo). If you see a process running that looks suspicious, leave it running long enough to examine its /proc entry before applying kill -9. Give users who don't need shell access a "shell" of /bin/true or /usr/games/fortune -o; but be sure to include whatever "shell" you gave them in /etc/shells -- otherwise they will not be able to use FTP. (If they don't have any web space on your server, just e-mail, then use /bin/false and don't put that in /etc/shells. That will make it harder to use an ftpd-based exploit.)

Note that the binaries in this rootkit are 32-bit ..... so running 64-bit Debian (which has *no* 32-bit libraries) will break them. Personally, I'd like to see a patch that will make Perl give a segmentation fault if any script tries to alter $0. In fact, I'd like to see a kernel patch that will break any binary that was not compiled locally.