Slashdot Log In
Spam Filtering For Small/Medium Business?
Posted by
kdawson
on Sun May 11, 2008 06:58 AM
from the dumpster-diving dept.
from the dumpster-diving dept.
or_is_it writes "The company I work for has been growing dramatically and I've been charged with the task of being the gatekeeper for our GFI Spam filters. This involves manually inspecting the subject line/to/from for all caught messages in each filter rule folder. For a company of about 50 people, in one day the number of spam messages can exceed 2,000. Neglect it for a day and you end up with quite a task on your hands. I've made the rules lax enough so important messages can go through, along with a few stray spams, for which I get bitched at. Tighten the rules up and then maybe an important time-sensitive email never gets to its intended recipient, and I get bitched at. Manually reading through all those subject lines is supposed to prevent that, but I'm only human and genuine messages can easily get overlooked. How do larger organizations deal with the spam issue? I can't imagine having one centralized person manually inspecting everyone's junk-mail header is the optimal solution. Purchasing a different commercial mail filter product is a possibility, but I'd like to hear some anecdotal evidence before jumping ship."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Client-based? (Score:5, Informative)
Combined effort is necessary (Score:4, Informative)
dnsbl/enhdnsbl is enabled for zen.spamhaus.org, bl.spamcop.net, combined.njabl.org, list.dsbl.org, dnsbl-1.uceprotect.net, dnsbl-2.uceprotect.net, dnsbl-3.uceprotect.net and sbl-xbl.spamhaus.org. With all these enabled there are very few spam messages falling through.
Adding to this I am using Mozilla Thunderbird which has a very good intelligent junk mail filter. The only disadvantage is that the junk mail filter has to learn what's junk or not.
The use of dnsbl/enhdnsbl also does bounce back to the sender with a reasonable message for the cases where a message is denied so the sender shall be informed about any messages that are denied. Of course - it isn't fool-proof, but it works for me.
Parent
Re: (Score:3, Informative)
Re:Combined effort is necessary (Score:5, Informative)
Do you generate a bounce, or do you reject with a 500 error and a proper message at spam time? You should not generate a bounce to remote mail. Ever. This is the cause of e-mail backscatter and is a significant problem. Always reject at SMTP time with a 500 error.
Parent
Re:Combined effort is necessary (Score:4, Informative)
From experience: you only need Spamhaus Zen and SpamCop for connection checking.
If you parse DATA before you accept it, you should incorporate URIBL.COM it's very good, and helps catch Yahoo and Gmail spam (which will get past Spamhaus and Spamcop all the time) because it scans bodies for naughty links
dsbl.org is REDUNDANT -- incorporated in Spamhaus Xen.
Spamhaus SBL-XBL -- incorporated in Spamhaus Xen.
NJABL.org is dead and a mirror of the CBL, I believe (-- incorporated in Spamhaus Xen also)
Never send bounce notices for spam. What notices leave your server are likely going to forged From: addresses....
Parent
Re:Client-based? (Score:5, Interesting)
I do it without a spam filter of any kind. I have only two technique.
First, simple rule-based filters throw clients and friends into their own folders by from: line alone. That covers everyone I know in advance.
The second set of rules simply looks for my full name, my company name, my e-mail signature, my telephone number, or my mailing address. These into the "it's damn likely a legitimate e-mail" folder. This folder gets about 2 spam e-mails per week.
The remaining I simply run through, in outlook express of all clients. Sorting wins the day. The greatest trick? Sort by the to: field. It doesn't take long to see that 75 messages went to moocow@mydomain.com, 75sevens@mydomain.com, or some other horribly malformed address to that doesn't exist. Sorting by subject does similar things -- like give you "70% off . .
Your spam has very simple patterns to look for. Sort by them, click the first, shift-click the last, and hit delete.
Last year, I was contracted by Viagra's H.R. department to do some quick work, I made it through unscathed.
Parent
Re: (Score:3, Interesting)
If I mail a letter to you, and you don't like the return address on the back of the envelope, you can do with it whatever you please, but it's not my responsibility to ensure that you'll open your mail. It is my responsibilty to deliver your mail. If you don't like the colour of the envelope, that's your problem.
I have a lot of clients
You just can't win. (Score:3, Interesting)
We run into the EXACT same issue you're running into.
The dilemma is if we don't tighten the spam filter enough, we'll get complaints from employees (who are not shy about sending EVERY LAST PIECE OF SPAM THEY GET to us.)
However, if they tighten the filter too much, then important emails that may seem spam-like begin to get blocked, and we get just as much heat for that.
The answer - do your best to block what spam you can, and if you get complai
Re:Client-based? (Score:4, Interesting)
Only solution that I know works is my own: Postfix with amavisd-new, spamassassin, clamav, postgrey, along with FuzzyOCR on smaller installs, though setting that up on a separate system to filter through might cover a large organization. Don't forget to include things like Spamhaus' Zen list, any of the *.countries.dk.net blocklists to filter out any geographical areas from which you don't expect legitimate mail, and also helo filtering--if the connecting mail server can't say helo/ehlo with something that resolves in DNS, it can just bugger right off.
Tell your boss that expecting not to lose email with spam filters in place is unreasonable, and that tasking one human to eyeball all the rejects is a serious misapplication of time and money.
Best of all, you should educate your boss to realize that email is not a reliable messaging system. There are far too many points of failure that could cause a message to be lost, most of them being outside of your own or your company's control. There exist many better ways to send time-sensitive material, like fax, overnight mail, and telephone calls. If a severe amount of money is to be lost because an email didn't make it on time or made it not at all, then the message should have been sent over a more reliable medium in addition to being emailed.
Only the severely clueless would rely on a system like the one you have set up. You have to allow for a certain failure rate in any system. That's a basic principle of quality control methods that have been in use for decades.
Parent
Barracuda SPAM filter (Score:4, Informative)
Re: (Score:3, Interesting)
Re: (Score:2)
Re: (Score:2)
Re:Barracuda SPAM filter (Score:4, Interesting)
Additionally I have a serious problem with the backscatter they cause. They should reject mail at SMTP time and not bounce them.
But Barracuda support is very very good. Very responsive and timely and overall a good people orgaization which can make the difference for wanting to deal with them.
Parent
Re: (Score:3, Informative)
Email systems that do not do this, yet do not send a bounce message "break" email. Possible to get a false positive and block a legit email with no error message back to sender. This is never a desired operation. If the message get a spam designation and th
Re:Barracuda SPAM filter (Score:5, Interesting)
Consider as well that the Barracuda appliances consist of (a) an open-source operating system (b) an open-source MTA (c) an open-source web server (d) an open-source spam scanner (e) an open-source virus scanner (f) other pieces of open-source software and (g) use community-mintained DNSBLs and RHSBLs. This is all held together with proprietary (closed-source) code, mostly for the purpose of providing a poorly-designed GUI interface. Any competent email system administrator should be able to create their own near-equivalent in an afternoon; it's not difficult. Such homebrewed creations have repeatedly been shown to vastly outperform Barracudas on multiple metrics, including cost, scalability, customization, security, and perhaps most importantly -- adaptability to new spammer techniques. (Barracuda is years behind the times and falling further back.)
It's very tempting to "just buy an appliance" and consider the problem solved, but it doesn't work. There's no substitute for expertise -- and given that much of that expertise is available for free, for the asking, on lists such as spam-l and spamtools and so on, it's difficult to understand why anyone would choose not to avail themselves of it.
Parent
Re: (Score:3, Funny)
Re: (Score:3, Insightful)
dajones70 (Score:2, Informative)
Re: (Score:3, Informative)
http://www.mailscanner.info/ [mailscanner.info]
Our organisation runs 5 Linux Servers around the UK for mail services and they are all using MailScanner + Postfix + SpamAssassin + ClamAV + Bitdefender.
Great installation instructions (all-but bitdefender) here: http://www.hughesjr.com/content/view/14/ [hughesjr.com]
The mailing list for MailScanner is very well supported by the users and the devs.
email != IM (Score:4, Insightful)
When will users learn...
Email is not instant messaging - with bad greylisting / random connection reset / busy server, you can get >=2 hours delay. And it's normal.
Re:email != IM (Score:5, Insightful)
email is ubiquitous and easy. 99.5% of the time, it's nearly instantaneous. Should I really have to get an IM account on google, yahoo, aim, microsoft, etc.... so I can deal with time-critical messages? And, for that matter, should everybody else?
Parent
Re:email != IM (Score:4, Interesting)
Parent
Force keywords in the subject line (Score:4, Interesting)
Set up an automated filter whereby anything that doesn't have the keyword in the subject gets dumped into a spam box to be sorted later. If the senders do the right thing, it assures their emails will be directed to the correct person.
This is just one example of active spam filtering as opposed to the passive spam filtering used in IT today.
Re: (Score:2)
Power to the people :) (Score:5, Insightful)
1) Set up the system to put junk mails in a folder the user can see
2) Train the end user to check their junk mails
3) Show the user how to set the spam triggers high or low and what the implications are
If user says they're too busy/important, advise them that due to your workload, their email box will be added to the "manually checked list" which gets done once per week. Point out the impact of losing a time-critical email wrongly flagged.
Most times they do it themselves. For those who are dead set on having someone else do it, hire a temp or arrange for an office junior to do it.
If you're in IT, you have better & more important things to do than check for real mail in a junk mail box...
Re: (Score:2)
Up until 2006 (I retired) I ran a in-house mail server (well, in-basement, actually) with about 250 users; when the SPAM started hitting the 200+ mark per day I figured the bandwidth savings alone would be a good reason to stop it as much as possible at the server.
I used ORBS, blocked all of asia-pacific net, and ran ASSP (Anti-Spam SMTP Proxy). After around 5 days of training I had SPAM down to maybe 3-5 a day per mailbox; I never could beat that number.
Nothing's perfect... (Score:3, Insightful)
I like the way spamassassin [apache.org] works - it can provide a rating for each message, which provides a mechanism for users to set the bar to their own preference, instead of having a single setting for the entire organization.
I'm not talking about using individual configurations for spamassassin, it's not realistic to expect most users to be able to deal with all the gory detail of spam filters.
Rather, spamassassin can set a header to indicate its confidence that a message is spam:It adds an asterisk for each "point" of spam score. Users should be able to create an email filter which picks off suspected spam and puts it into a separate folder based on a header like that. Maybe drop all 10+ messages centrally, and let users tweak a local filter to their liking, depending on whether they prefer false positives or negatives.
I use spamassassin as an example only because that's what I use. There are no doubt others which can provide something similar which users could filter on.
Commercial Services (Score:2)
Postini (Score:3, Informative)
For the record, Google purchased Postini in the not to distant past.
Frontbridge Spamshark (Score:4, Interesting)
I used to work for a mining company you've heard of. Our department had responsibility for managing the email vendor, who used Spamshark to filter spam coming into the organisation. From my limited knowledge of the setup, Spamshark does basic blacklisting etc. but also does selective blacklisting on specific IPs when an email is flagged by a user. So Alice flags a message as spam, Spamshark figures out the message id, grabs the IP address it came from (it knows because it previously handled the email), and then blacklists that IP for a certain amount of time. Now this internal blacklist is then shared to all the other customers who use Spamshark, so they are now protected too; resulting in a 5 nines hit rate on spam.
Like I said we just handled vendor relations, and the above description might not be totally accurate, but this is what I gathered when we dealt with them. I also remember getting about 10 complaints of spam a month for an organisation with 10's of thousands of email addresses - so it was very effective.
Re: (Score:2, Informative)
And more false posistives than you would actually like to have. I've been at the business end of one of Frontbridge's blacklists. One of the domains I admin got blacklisted a full three weeks after the hosting company screwed up and let phishers set up a paypal scam site as the "test1" user to live for all of 22 hours. Three weeks later, one of
Time to ditch GFI (Score:2)
OpenBSD spamd (Score:5, Informative)
Subscription based anti-spam solution (Score:2)
We're using Astaro Mail Security (www.astaro.com), which works great. Spam is down to a minimum, and it delivers much better results than open source solution I had in place before that.
FYI: I receive about 300 spam mes
ESVA all day long (Score:3, Informative)
Inside, there is greylisting and MailScanner. Within MailScanner, there is SpamAssassin, some RBL, ClamAV and all sorts of things.
For my organization, I find that in addition to everything else "stock" I can safely filter out all countries but the U.S. since we don't do business outside of our state, let alone our country... so it's safe to assume that anything from outside the US will be spam.
It is extremely effective. I have helped to get the VM set up in environments with multiple domains and it works very well too.
One problem with it is that it is rapidly aging. The user community has made some effort to get the VM up to date in some ways, but the 2.0 version as far as anyone can tell is still in discussion and planning. The project creator and leader is a one-man-show and he seems to have a life outside of this project for some reason. The user community is frantic to get something to replace the aging 1.7.1.5 machine we all use as the reference point for our installs.
This is largely a known-solved problem (Score:5, Informative)
Meanwhile, here is some general guidance. First, do not waste your money on commercial products -- they're expensive, poorly-maintained, and in many cases (e.g. Barracuda) actually make the spam problem worse via backscatter. (There are now several thousand Barracudas on a communally-maintained blacklist, making it obvious to everyone working in this field that Barracuda is completely incompetent.) Second, do invest your money and time in open-source solutions: it is easy for anyone who possesses baseline competence in mail to craft their own, superior spam handling system using postfix or sendmail or another open-source MTA, DNSBLs, RHSBLs, judicious configuration, and other tools such as rbldnsd, mimedefang, SpamAssassin, ClamAV, and so on. Third, a little googling will reveal near-cookbook procedures for combining these pieces of software together into a useful system; which cookbook procedure is appropriate for you depends on your environment -- which brings me to the fourth point, which is that you need to perform log analysis in order to understand your particular mix of spam/not-spam. Everyone's is different, which is why one-size-fits-all solutions usually fail. Only after you have some clue about the size and shape of your problem will you be able to determine which approach(es) are likely to minimize both false negatives (FN) and false positives (FP).
As an aside, one set of highly effective anti-spam tactics involves enforcing RFC requirements that have been in place for many years: for example, all mail servers must have rDNS; that rDNS must resolve to a host which in turn resolves back to the IP; the domain of the host must exist; the host must HELO as a valid FQDN or bracketed-quad IP; the envelope-sender's domain must exist; the host must not HELO as you; the host must wait for the SMTP greeting before HELO'ing; the host must handle a multi-line SMTP greeting; the MX records for the host must point to valid IP space; and so on. Enforcement of these requirements yields differing rates of spam control (which is again why log analysis is crucial) but has the very valuable property that it can be done at low computational and bandwidth cost. Substantial experience with these suggests that enabling them and augmenting them with a few DNSBLs (especially the Spamhaus Zen zone) is enough to deal with the overwhelming majority of the spam problem at most sites, reducing what's left to a much smaller issue to be dealt with.
Re: (Score:2)
sender IP (Score:2)
Kind of obvious (Score:2)
With sendmail, use mimedefang, spamassassin, and milter-greylist (actually that last can be implemented yourself in mimedefang, I just never had the time).
The nice thing about this solution is that it does not require you to pay some third party a huge amount of money each month, while doing exactly what they do (actually better), and it is fully customizable to fit into your environment (wan
that's actually a good solution (Score:4, Interesting)
Actually, that strikes me as a good solution; it's certainly better than having other employees dealing with spam as part of their daily routine and losing 30 minutes/day for everybody in the company. And by centralizing it, you have the ability to pick the tools to make your work more efficient, as opposed to having 50 employees each fiddle with their own spam filters.
ASSP (Score:2)
3 Steps (Score:4, Interesting)
1. Do what you can on the server. I like to use SpamAssassin to add spam scores to beginning of subject lines, so they sort by score in my inbox (I use "/*_SCORE(0)_*/"). I also automatically delete anything over a score of 11, since the highest I've ever seen a legitimate email score has been "10.something". Realistically, anything above an 8 is the sender's fault and they need to do something about it and anything above an 11 you can safely blame the sender (you won't be the only spam filter deleting their emails).
2. Provide the tools on the client. ThunderBird's "spam marker" is a must, and because it learns from what you mark, you aren't just marking them in vain. Also, to deal with spam in real-time, instead of using the junk folder, I like using the "delete junk!" button from the "Buttons!" add-on. Incoming junk gets marked and marked as read, and after marking the spam the filter missed, I hit "delete junk". Very easy and quick. Pre-configure Thunderbird for everyone.
3. Educate and support. If you have 1 and 2 in place, then make sure everyone knows what you are doing and why you chose to do it. Write a short manual or something. Educate them about their tools. They also need to know NOT to publish their addresses.
The idea is to make spam highly visible, and to make it *quick and easy* to deal with. Knowing you've facilitated these two goals should be enough to impress your employer and earn the respect you deserve from everyone you serve
I spent a few days migrating 100,000 emails from Windows Mail, because it was horrible. Thunderbird is a godsend and the add-ons make all the difference. If there is something you dislike or want, chances are someone made an add-on for it.
btw 2000 messages is *not* a lot of spam. It will get far worse with time.
Well, I can tell you about my decision. YMMV. (Score:3, Interesting)
You've basically got three options:
1. Go for a completely outsourced service.
Pros: It's someone else's problem to look after.
Cons: A company of 50 staff will never be terribly important to such a service provider. Unless they provide an extremely good control panel and logs, sooner or later someone's going to ask where an email is and your answer is going to be "er... let me get back to you on that.... er... I don't know".
2. Go for an appliance - either in the form of a prebuilt lump of tin like the Barracuda system mentioned elsewhere or in the form of a precooked Linux installation which is literally just a matter of "insert CD, boot, tell it what it's IP address is and what domain it's providing email for".
Pros: Dead easy to set up. Most also provide a nice web-based UI.
Cons: The decent ones are almost universally commercial and you have to pay licensing fees on a per-active-email-address basis, which can get very expensive - particularly when the vendor won't tell you how their system decides how many email addresses are regularly active and the first you know that you're exceeding the license is when suddenly all the spam filtering is disabled.
If you look closely, expect to find that many of them are architected around a number of single points of failure. And in the real world, nobody is likely to check a web-based UI on the offchance that they find an email misclassified as spam sat there.
3. Roll your own. If you take this route, I can strongly recommend rolling it around an existing framework rather than following a bunch of complicated instructions to configure Postfix that you have to re-learn every time anything needs tweaking. This is the route I took, and I based it around MailScanner. MailScanner provides a framework for plugging in spam and virus filters and allows you to divide spam according to its score. Delete high scoring spam, let low scoring spam through with a note in the subject line that it's suspected spam and let non-spam straight through.
Pros: You get to keep a close eye on all the configuration, can keep close track of the logs and respond quickly to any issues. Your users can easily set up filters for spam (for that matter, so can you) and their "potential-spam" where misclassified mail may wind up is in their email client rather than a separate web-based system.
Cons: You need to become intimately familiar with every aspect of your email system in order to manage it effectively. I would argue that any self-respecting sysadmin should be intimately familiar with his email system anyway, but YMMV.
Re:Despite other issues (Score:4, Insightful)
It's really not that good.
Parent
Re: (Score:3, Interesting)
As for the "false positives", only the most dubious of mailing lists seems to get caught (I still regularly check my spam just in case) and when I report them as "not spam", they
My REAL problem with Gmail/Google Apps (Score:3, Insightful)
I run email for several of my domains through Google Apps for Your Domain - essentially, Gmail. On my largest account, I get several hundred legit emails and 200-1000 spam messages each day. The problem isn't Gmail's filtering of this - it's actually damn good, with maybe 2-3 false negatives a week and maybe one false positive. Better than almost anything else I've seen.
The problem is that Gmail gives me NO options - as a user or domain administrator - to sift through the spam
ASSP is your answer (Score:3, Informative)
In a company of about 75 email accounts it has blocked 4 million spams in a little over a year.
The false negative rate is so low it might as well be zero, and the false positive rate as well.
It uses among
Re:SpamAssassin (Score:4, Insightful)
When set-up with good rules and RBLs it blocks at least 99% spam with very low false positives (I've never had a false positive).
Send anything tagged as spam to another account such as spam@domain (I do this) then you can manually check for false positives to further reduce the chance of losing legit email. (or if a user complains that an email they expected never arrived)
Parent
Re: (Score:2)
Last spam filtering i used was turning up false positives too often, although its been a while since i botherd with an automated system, i just relly on social engeniering now (dont give out my email to strangers, and use a webmail(yahoo as i had one lying about) for any signups.