SkiifGeek writes "Sophos claims that they are detecting 6,000 new sites daily that have been compromised to serve malware to unsuspecting site visitors, with 80% of site owners not aware that they have been compromised — though this figure is probably on the low side. With increasingly vocal arguments being put forward by security experts criticizing the performance and capability of site validation tools (though many of these experts offer their own tools and services for similar capabilities), and rising levels of blended attacks, perhaps it is time you reviewed the security of your site and what might be hiding in infrequently used directories."
Everytime I read about a new form of server malware, I try to check a LAMP server that I run. So far I've come up clean but I've hardly done a full inspection. Anyone know of a good way to scan a set up? Sophos says that they are detecting thousands of new sites - how are they scanning them?
"Everytime I read about a new form of server malware, I try to check a LAMP server that I run. So far I've come up clean but I've hardly done a full inspection"
The only way to be sure is to run it from a CD image and reboot nightly..
"Anyone know of a good way to scan a set up? Sophos says that they are detecting thousands of new sites - how are they scanning them?"
They're not, they just need a little boost to the stock price, please buy our PRODUC~1..:)
I thought about this myself. One possible solution that I considered would be to maintain a local list of files on your server and their CRC/Hash values. A script on the server would scan all the files and output a similar list than you could then check against your local copy and would quickly identify any new or changed files. This could be set to a cron job to do periodic scans or just initiate a manual scan whenever.
Might not be the best solution but it should be easy to implement. Larger sites can do incremental scans. It would be harder to detect corruption of databases, though, unless you know what to look for or have a concrete way of validating the contents. =Smidge=
For this, you'd want to use something like Tripwire or AIDE. It's been used for years, and will detect changes to files. You're right that it won't help you detect that somebody has managed to insert a chunk of javascript or PHP in your insecure mySQL/PHP web app, though. Perhaps a combination of Snort, Ntop (if it wasn't shit), a "hardened" PHP binary and config, and log monitoring would alert you in the case of an attack.
The problem is that there's a lot of badly written or out of date software out there t
I thought about this myself. One possible solution that I considered would be to maintain a local list of files on your server and their CRC/Hash values. A script on the server would scan all the files and output a similar list than you could then check against your local copy and would quickly identify any new or changed files. This could be set to a cron job to do periodic scans or just initiate a manual scan whenever.
Congratulations! You have just described Tripwire [sourceforge.net].
A quick and easy way to check all your files is to use md5deep, which will scan directories recursively and generate all the md5s into a single file which you could compare against a baseline:
what you would have to do is set up a script to take a local mirror of the website that has every authorized file, in it authorized form with their appropriate hashes and compare it to what is on the website via FTP and report any additions, deletions or changed files. This would tell you that the site proper is what you uploaded and unchanged. Other problems seems to be that some big server farms have been rooted which is probably out of your control, that why I wouldn't trust a script running on the websi
Radmind: http://radmind.org/ [radmind.org]. Radmind's is designed for this purpose exactly. It's a tripwire with the ability to roll back changes, or capture them and store them for deployment to other systems.
I don't have a great answer but I may have found a step in the right direction. Check out Tiger at http://www.nongnu.org/tiger/ [nongnu.org]. It's a Debian scanner that checks for common signs that someone has pwned your system. I'll warn you now that I haven't tried it yet but might do so in the next few weeks to see how it operates. It doesn't check specifically for any malware but does check for signs that someone has altered your system for remote control. Like I said, not a great solution, but another tool in the t
You first need a file integrity checker. AFICK (my favorite) or similar will do a run, on whatever period of time you set a cronjob and conf file at. You then get an email to you listing what files have changed over that period of time.
Also, keep an eye on how big your maillog files are - if they suddenly grow by some exponent, you've been turned into a spam server (or a newsletter went out - five seconds of peeking at the live output should tell you which).
In addition to the other tools mentionned by/.ers, there are 2 root-kit checking tools that are worth mentioning: - chkrootkit [chkrootkit.org] - rkhunter [rootkit.nl]
They are scripts that scan the system for known root kits, weird behaviours and hidden files in unusual places. They can both be used to scan an offline system (booted from a live-cd and the system mounted under some directory), and a live online system (they check the system for suspicious behaviour that may reveal a root-kit trying to hide it self - for example the "ps" co
I use tripwire to build hashes for all files, and store them "off-machine". The hashes are compared against a baseline, and any differences are highlighted. Disk use in monitored, and (not yet found) anomalies are investigated. Logs are examined, and ssh dictionary attacks are dropped. The server does NOT run database -- only (pure static) apache. Scripts are NOT run on this machine. Certain things are directed off to https, on another machine, with user/password authentication. That, in turn, actually talks
But I thought all these sites were validated and certified and IP imdemnified, else what was the point of paying huge wads of dosh to all the lawyers, oh wait, now I get it..:)
Perhaps the time has come to harden the "common stacks" so certain switches are off.
For example, once you set up your web site, "lock it" so if there are any changes to files or directories that shouldn't change, the site will break in a non-harmful way rather than be compromised.
If and when these files need updating, the "unlock" process should be done using a tool independent of the main web-server process, perhaps by using a different web-server process running on a different port or even a process on a different computer that validates the request then passes it on to the main web server.
Even better. When a server is compromised, it will burst into flames, burning down the entire data center it's hosted in. You know, just in case the virus spread.
Easier. For a LAMP stack, here's what's needed: Everything on the Web server should be mounted read-only, preferrably from a machine behind a firewall. A firewall sits behind that machine and your inside network. The only way to write to the file system should be from behind the firewall. Any temporary files that need to be created for download or parsing or whatever, where read/write is necessary, should only be done from a RAM disk. Reboot the server nightly.
Actual different machines with actual different firewalls are good for hosted solutions and IT departments that know what they are doing, but they are too complicated for non-geek do-it-yourself mom-and-pop-businessman/home-user solutions.
However, a stack that puts a virtual or other hardened subsystem to hide the non-read-only files and databases behind in an easy-to-use form should be doable.
For example, once you set up your web site, "lock it" so if there are any changes to files or directories that shouldn't change, the site will break in a non-harmful way rather than be compromised.
If it's not supposed to change at all, just issue chattr +i on it to make it immutable. Then it won't change, even w/ system root permissions. Just remember to unset the flag any time that you do want to change something ("chattr -i").
Err, you do realize I wrote "If it's not going to change" up there, right?
chattr is not meant to be any sort of cure-all - far from it. But for files (and even directories) in the chroot jail that don't change very often (if at all), like logo images, certain site-structural files and scripts, etc? It works just fine.
For instance, a typical Wiki or any sort of CMS rig-up has lots of CGI and/or PHP files that you wouldn't expect to see modified until you either apply a patch or customize them yourself. W
I would just run a perl script that does a regex on the access logs for anything that does not match the files that should be delivered to clients. Put the perl script in a cron job and let it run. Also do an MD5 hash on those files regularly and check for any changes to static files. And use very strong root passwords, don't let root account login remotely, and use ssh keys with no interactive logins.
I would just run a perl script that does a regex on the access logs for anything that does not match the files that should be delivered to clients. Put the perl script in a cron job and let it run.
Sounds like LogCheck. I'm using that; it provides a decent organized daily summary of all access to the server.
Also do an MD5 hash on those files regularly and check for any changes to static files.
Know any pre-written scripts/software that handle this? The trouble is, you'd have to secure the hashes as well, which gets tricky, plus it doesn't offer any help for regularly changing content or database contents (where a content hack like secretly added JS would probably be inserted). But still, better security for non-changing files seems like a good idea; the chattr +i suggestion above se
Okay, say someone's site is served by an ISP. The ISP gives the site owner a shell account and manages the LAMP infrastructure. The shell account is likely a virtualized instance, meant to limit the damage that each little site can do to the hosted infrastructure, not to limit the damage that the host does to little sites or their visitors. How can the site owner "check their own site" in such a case? Virtualization itself is a sort of rootkit conceptually, so how can the virtualized account check for malicious rootkits in its own instance or in the greater infrastructure?
That would be a sump move. If it was IP addresses then once an IP address was re-assigned to a good host you still wouldn't see their website. You have no way of removing IP addresses from your list.
If it was domains names the same problem would apply but after domains cleaned infected files.
I say all of this because I was a victim of stupid block lists when I got a new IP and tried to send email out on it. It was blocked because of the previous owner and getting removed from most lists was non-obvio
If I run FF and keep it patched, am I safe? If I did get compromised, what would the symptoms be?
I tend to think that keeping my OS patched keeps me pretty safe, but there's always a delay after a new vulnerability is discovered before the patches come out (the zero day) and what concerns me is that if someone has a very large network of compromised web servers, they can roll out a zero day vulnerability to all of them and do a lot of damage.
As to symptoms, I think spyware used to be the big problem, and infected computers would have popups and such. But now I think that infected machines will be used primarily to send spam. Is that correct?
with 80% of site owners not aware that they have been compromised
Wait. So 20% of site owners know their site has been comprimised and they haven't done anything about it and are still serving up malware? Sounds to me like someones making up statistics.
I for one would like some description of how they're detecting these 6000 new sites per day. Also, what are they considering a website? Do they include bot systems that configured to listen on port 80 as part of the worm propagation and command/control? That's not really a website in my opinion, but it may be in theirs. It would be great if they published a list of the 42000 new websites they have discovered over the past 7 days, you know just to back up their claim. Wouldn't hurt to notify the owners of those sites that they've got a problem.
Absent more detail, I am calling shenanigans on this statistic, Sophos, and the Register. I am soooo sick of the FUD.
"The figure of 2 million new site compromises per year seems to be quite significant, but could be explained by virtual hosting servers with many sites on the one physical server being compromised, leading to the same vector affecting multiple sites (in some cases thousands of sites)."
Oh, and it would probably kill most/all of the bot nets.
No -- many of the bots nowadays lock down the PC themselves (once they're in...) to keep it "safe" from competing bots. They even actively remove other bots when they can manage it.
As for the idea, though... I think about that as well. Even if just to get onto computers that haven't been compromised by a really effective bot yet (as I mentioned above) would be a big step.
Alas, most of the people talented enough to write such a thing are probably either: * well-employed enough that they don't want to risk
How to Check a LAMP Server? (Score:4, Interesting)
Format every single day (Score:1)
hit it with a hammer (Score:2)
Re: (Score:1)
The only way to be sure is to run it from a CD image and reboot nightly
"Anyone know of a good way to scan a set up? Sophos says that they are detecting thousands of new sites - how are they scanning them?"
They're not, they just need a little boost to the stock price, please buy our PRODUC~1
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re:How to Check a LAMP Server? (Score:5, Informative)
Might not be the best solution but it should be easy to implement. Larger sites can do incremental scans. It would be harder to detect corruption of databases, though, unless you know what to look for or have a concrete way of validating the contents.
=Smidge=
Parent
Re: (Score:2, Informative)
You're right that it won't help you detect that somebody has managed to insert a chunk of javascript or PHP in your insecure mySQL/PHP web app, though. Perhaps a combination of Snort, Ntop (if it wasn't shit), a "hardened" PHP binary and config, and log monitoring would alert you in the case of an attack.
The problem is that there's a lot of badly written or out of date software out there t
Re: (Score:1)
I meant "vulnerable", but feel free to insert jokes about neurotic software here:
Re: (Score:1, Informative)
http://radmind.org/ [radmind.org]
Re: (Score:2, Informative)
Congratulations! You have just described Tripwire [sourceforge.net].
Re: (Score:1)
will scan directories recursively and generate all the md5s into
a single file which you could compare against a baseline:
http://md5deep.sourceforge.net/ [sourceforge.net]
Re: (Score:1)
AIDE - Advanced Intrusion Detection Environment
AIDE is a file integrity checker that supports regular expressions. Licensed with GPL.
www.cs.tut.fi/~rammer/aide.html
Re: (Score:2)
Radmind (Score:2, Informative)
Re: (Score:1)
AFICK. (Score:2)
Also, keep an eye on how big your maillog files are - if they suddenly grow by some exponent, you've been turned into a spam server (or a newsletter went out - five seconds of peeking at the live output should tell you which).
Also, you can keep an eye on the http access l
Additionnal malware detection tools (Score:2)
- chkrootkit [chkrootkit.org]
- rkhunter [rootkit.nl]
They are scripts that scan the system for known root kits, weird behaviours and hidden files in unusual places.
They can both be used to scan an offline system (booted from a live-cd and the system mounted under some directory),
and a live online system (they check the system for suspicious behaviour that may reveal a root-kit trying to hide it self - for example the "ps" co
Re: (Score:2)
The server does NOT run database -- only (pure static) apache. Scripts are NOT run on this machine. Certain things are directed off to https, on another machine, with user/password authentication. That, in turn, actually talks
Re: (Score:3, Informative)
validated certified imdemnified .. (Score:2)
Hmm, time to improve the common tools (Score:4, Interesting)
For example, once you set up your web site, "lock it" so if there are any changes to files or directories that shouldn't change, the site will break in a non-harmful way rather than be compromised.
If and when these files need updating, the "unlock" process should be done using a tool independent of the main web-server process, perhaps by using a different web-server process running on a different port or even a process on a different computer that validates the request then passes it on to the main web server.
Re: (Score:1, Funny)
Re: (Score:1)
Everything on the Web server should be mounted read-only, preferrably from a machine behind a firewall. A firewall sits behind that machine and your inside network. The only way to write to the file system should be from behind the firewall. Any temporary files that need to be created for download or parsing or whatever, where read/write is necessary, should only be done from a RAM disk. Reboot the server nightly.
The database server should also sit on the
too complicated for "simple" setups (Score:1)
However, a stack that puts a virtual or other hardened subsystem to hide the non-read-only files and databases behind in an easy-to-use form should be doable.
Re: (Score:2)
For example, once you set up your web site, "lock it" so if there are any changes to files or directories that shouldn't change, the site will break in a non-harmful way rather than be compromised.
If it's not supposed to change at all, just issue chattr +i on it to make it immutable. Then it won't change, even w/ system root permissions. Just remember to unset the flag any time that you do want to change something ("chattr -i").
Re: (Score:2)
chattr is not meant to be any sort of cure-all - far from it. But for files (and even directories) in the chroot jail that don't change very often (if at all), like logo images, certain site-structural files and scripts, etc? It works just fine.
For instance, a typical Wiki or any sort of CMS rig-up has lots of CGI and/or PHP files that you wouldn't expect to see modified until you either apply a patch or customize them yourself. W
Just check your access logs (Score:1, Informative)
my 2 cents...
Re: (Score:2)
I would just run a perl script that does a regex on the access logs for anything that does not match the files that should be delivered to clients. Put the perl script in a cron job and let it run.
Sounds like LogCheck. I'm using that; it provides a decent organized daily summary of all access to the server.
Also do an MD5 hash on those files regularly and check for any changes to static files.
Know any pre-written scripts/software that handle this? The trouble is, you'd have to secure the hashes as well, which gets tricky, plus it doesn't offer any help for regularly changing content or database contents (where a content hack like secretly added JS would probably be inserted). But still, better security for non-changing files seems like a good idea; the chattr +i suggestion above se
Re: (Score:2)
virtualized rootkits (Score:3, Interesting)
Re: (Score:1)
Completely useless. (Score:4, Insightful)
I would love to put that list in my squid blocking file to protect my users.
Re: (Score:2)
If it was domains names the same problem would apply but after domains cleaned infected files.
I say all of this because I was a victim of stupid block lists when I got a new IP and tried to send email out on it. It was blocked because of the previous owner and getting removed from most lists was non-obvio
Re: (Score:2)
6000 sites? (Score:1, Interesting)
what does this look like from the client? (Score:4, Interesting)
I tend to think that keeping my OS patched keeps me pretty safe, but there's always a delay after a new vulnerability is discovered before the patches come out (the zero day) and what concerns me is that if someone has a very large network of compromised web servers, they can roll out a zero day vulnerability to all of them and do a lot of damage.
As to symptoms, I think spyware used to be the big problem, and infected computers would have popups and such. But now I think that infected machines will be used primarily to send spam. Is that correct?
Re: (Score:2)
What I wanna know is ... (Score:3, Interesting)
Imagine all the useful things we could do for the world if we all had access to this distributed computing power.
Re: (Score:2)
Re: (Score:2)
Well, I think you might be a bit late with that.
But think of the good things that could be done with a free and open implementation.
OTOH, it's been more than 25 years since the first true distributed OS was announced, and the idea hasn't exactly taken the world by storm.
80% (Score:1, Interesting)
with 80% of site owners not aware that they have been compromised
Wait. So 20% of site owners know their site has been comprimised and they haven't done anything about it and are still serving up malware? Sounds to me like someones making up statistics.
Yes... (Score:3, Interesting)
Somebody should warn... (Score:2, Funny)
Vendor FUD or Real? (Score:4, Interesting)
Absent more detail, I am calling shenanigans on this statistic, Sophos, and the Register. I am soooo sick of the FUD.
Harumph!
You misspelled Natalie (Score:1)
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
Oh, and it would probably kill most/all of the bot nets.
No -- many of the bots nowadays lock down the PC themselves (once they're in...) to keep it "safe" from competing bots. They even actively remove other bots when they can manage it.
As for the idea, though... I think about that as well. Even if just to get onto computers that haven't been compromised by a really effective bot yet (as I mentioned above) would be a big step.
Alas, most of the people talented enough to write such a thing are probably either:
* well-employed enough that they don't want to risk