Spam Filtering For Small/Medium Business? 453
or_is_it writes "The company I work for has been growing dramatically and I've been charged with the task of being the gatekeeper for our GFI Spam filters. This involves manually inspecting the subject line/to/from for all caught messages in each filter rule folder. For a company of about 50 people, in one day the number of spam messages can exceed 2,000. Neglect it for a day and you end up with quite a task on your hands. I've made the rules lax enough so important messages can go through, along with a few stray spams, for which I get bitched at. Tighten the rules up and then maybe an important time-sensitive email never gets to its intended recipient, and I get bitched at. Manually reading through all those subject lines is supposed to prevent that, but I'm only human and genuine messages can easily get overlooked. How do larger organizations deal with the spam issue? I can't imagine having one centralized person manually inspecting everyone's junk-mail header is the optimal solution. Purchasing a different commercial mail filter product is a possibility, but I'd like to hear some anecdotal evidence before jumping ship."
Force keywords in the subject line (Score:4, Interesting)
Set up an automated filter whereby anything that doesn't have the keyword in the subject gets dumped into a spam box to be sorted later. If the senders do the right thing, it assures their emails will be directed to the correct person.
This is just one example of active spam filtering as opposed to the passive spam filtering used in IT today.
Frontbridge Spamshark (Score:4, Interesting)
I used to work for a mining company you've heard of. Our department had responsibility for managing the email vendor, who used Spamshark to filter spam coming into the organisation. From my limited knowledge of the setup, Spamshark does basic blacklisting etc. but also does selective blacklisting on specific IPs when an email is flagged by a user. So Alice flags a message as spam, Spamshark figures out the message id, grabs the IP address it came from (it knows because it previously handled the email), and then blacklists that IP for a certain amount of time. Now this internal blacklist is then shared to all the other customers who use Spamshark, so they are now protected too; resulting in a 5 nines hit rate on spam.
Like I said we just handled vendor relations, and the above description might not be totally accurate, but this is what I gathered when we dealt with them. I also remember getting about 10 complaints of spam a month for an organisation with 10's of thousands of email addresses - so it was very effective.
Re:Barracuda SPAM filter (Score:3, Interesting)
that's actually a good solution (Score:4, Interesting)
Actually, that strikes me as a good solution; it's certainly better than having other employees dealing with spam as part of their daily routine and losing 30 minutes/day for everybody in the company. And by centralizing it, you have the ability to pick the tools to make your work more efficient, as opposed to having 50 employees each fiddle with their own spam filters.
3 Steps (Score:4, Interesting)
1. Do what you can on the server. I like to use SpamAssassin to add spam scores to beginning of subject lines, so they sort by score in my inbox (I use "/*_SCORE(0)_*/"). I also automatically delete anything over a score of 11, since the highest I've ever seen a legitimate email score has been "10.something". Realistically, anything above an 8 is the sender's fault and they need to do something about it and anything above an 11 you can safely blame the sender (you won't be the only spam filter deleting their emails).
2. Provide the tools on the client. ThunderBird's "spam marker" is a must, and because it learns from what you mark, you aren't just marking them in vain. Also, to deal with spam in real-time, instead of using the junk folder, I like using the "delete junk!" button from the "Buttons!" add-on. Incoming junk gets marked and marked as read, and after marking the spam the filter missed, I hit "delete junk". Very easy and quick. Pre-configure Thunderbird for everyone.
3. Educate and support. If you have 1 and 2 in place, then make sure everyone knows what you are doing and why you chose to do it. Write a short manual or something. Educate them about their tools. They also need to know NOT to publish their addresses.
The idea is to make spam highly visible, and to make it *quick and easy* to deal with. Knowing you've facilitated these two goals should be enough to impress your employer and earn the respect you deserve from everyone you serve
I spent a few days migrating 100,000 emails from Windows Mail, because it was horrible. Thunderbird is a godsend and the add-ons make all the difference. If there is something you dislike or want, chances are someone made an add-on for it.
btw 2000 messages is *not* a lot of spam. It will get far worse with time.
Re:Despite other issues (Score:3, Interesting)
As for the "false positives", only the most dubious of mailing lists seems to get caught (I still regularly check my spam just in case) and when I report them as "not spam", they never get mistaken for spam again, so I can't really complain either.
I'm not disagreeing with you, I'm simply just curious as to what makes it bad? Have I just been fortunate enough to not have any major problems or is there something that it should (or shouldn't) do when it comes to corporate use?
Re:email != IM (Score:4, Interesting)
Well, if you want an *anecdote*... (Score:1, Interesting)
Re:Barracuda SPAM filter (Score:4, Interesting)
Additionally I have a serious problem with the backscatter they cause. They should reject mail at SMTP time and not bounce them.
But Barracuda support is very very good. Very responsive and timely and overall a good people orgaization which can make the difference for wanting to deal with them.
Well, I can tell you about my decision. YMMV. (Score:3, Interesting)
You've basically got three options:
1. Go for a completely outsourced service.
Pros: It's someone else's problem to look after.
Cons: A company of 50 staff will never be terribly important to such a service provider. Unless they provide an extremely good control panel and logs, sooner or later someone's going to ask where an email is and your answer is going to be "er... let me get back to you on that.... er... I don't know".
2. Go for an appliance - either in the form of a prebuilt lump of tin like the Barracuda system mentioned elsewhere or in the form of a precooked Linux installation which is literally just a matter of "insert CD, boot, tell it what it's IP address is and what domain it's providing email for".
Pros: Dead easy to set up. Most also provide a nice web-based UI.
Cons: The decent ones are almost universally commercial and you have to pay licensing fees on a per-active-email-address basis, which can get very expensive - particularly when the vendor won't tell you how their system decides how many email addresses are regularly active and the first you know that you're exceeding the license is when suddenly all the spam filtering is disabled.
If you look closely, expect to find that many of them are architected around a number of single points of failure. And in the real world, nobody is likely to check a web-based UI on the offchance that they find an email misclassified as spam sat there.
3. Roll your own. If you take this route, I can strongly recommend rolling it around an existing framework rather than following a bunch of complicated instructions to configure Postfix that you have to re-learn every time anything needs tweaking. This is the route I took, and I based it around MailScanner. MailScanner provides a framework for plugging in spam and virus filters and allows you to divide spam according to its score. Delete high scoring spam, let low scoring spam through with a note in the subject line that it's suspected spam and let non-spam straight through.
Pros: You get to keep a close eye on all the configuration, can keep close track of the logs and respond quickly to any issues. Your users can easily set up filters for spam (for that matter, so can you) and their "potential-spam" where misclassified mail may wind up is in their email client rather than a separate web-based system.
Cons: You need to become intimately familiar with every aspect of your email system in order to manage it effectively. I would argue that any self-respecting sysadmin should be intimately familiar with his email system anyway, but YMMV.
Re:Barracuda SPAM filter (Score:5, Interesting)
Consider as well that the Barracuda appliances consist of (a) an open-source operating system (b) an open-source MTA (c) an open-source web server (d) an open-source spam scanner (e) an open-source virus scanner (f) other pieces of open-source software and (g) use community-mintained DNSBLs and RHSBLs. This is all held together with proprietary (closed-source) code, mostly for the purpose of providing a poorly-designed GUI interface. Any competent email system administrator should be able to create their own near-equivalent in an afternoon; it's not difficult. Such homebrewed creations have repeatedly been shown to vastly outperform Barracudas on multiple metrics, including cost, scalability, customization, security, and perhaps most importantly -- adaptability to new spammer techniques. (Barracuda is years behind the times and falling further back.)
It's very tempting to "just buy an appliance" and consider the problem solved, but it doesn't work. There's no substitute for expertise -- and given that much of that expertise is available for free, for the asking, on lists such as spam-l and spamtools and so on, it's difficult to understand why anyone would choose not to avail themselves of it.
Re:Despite other issues (Score:1, Interesting)
It's really not that good.
1) It is a hell of a lot easier to maintain for me (sysadmin)
2) It comes with a webmail interface.
3) I don't have to have redundant mail servers
4) Even our marketing guy can set it up. It's that easy
5) I get a lot of spam on my personal account, it filters it like a charm. I never get false positives and only once or twice will a spam message get through to my inbox.
6) It is free (as in beer) for businesses with fewer than 50 employees
That is why GMail is a viable alternative for small business.
An industry endorsement for Postini (Score:2, Interesting)
As you may know, it used to be that Postini was considered, by those of us in the anti-spam industry, something of a black hole, and not a service we would recommend.
However, having been in touch with their executive team in recent years, I had inside knowledge as to how that was changing - how they *wanted* that to change.
Recently, we decided to take our own spam filtering outside, to let someone else's servers do the heavy lifting. We tried several solutions, and finally, almost in desparation, I gave the 'ok' for us to try Postini (which of course is now owned by Google, but the exec team is still in place).
Let me tell you that we were *extremely* pleasantly surprised - the service really has been *very* good, it was relatively easy to set up (you do need to be familiar with how to set up your MX records, etc., but if you are already adminning a server, you should already be fairly comfortable with that).
The price is good, and the end user UI is excellent in that it's pretty easy for an end user to understand how to scan their "spam folder", how to get something delivered out of the spam folder, how to whitelist a sender, etc..
Honestly, it's one of the easiest-to-use of the offsite systems out there - and one bonus is that it gets the user support *off* internal admins.
And, the false positive rate is low, as is the false negative rate - which really is the bottom line test for spam filtering services.
We have a formal review for our corporate blog (http://www.TheInternetPatrol.com/) in the works, but in the meantime consider this an endorsement of Postini from the Institute for Spam and Internet Public Policy (http://www.isipp.com/)
Anne
Anne P. Mitchell, Esq
CEO/President
Institute for Spam and Internet Public Policy
Professor of Law, Lincoln Law School of SJ
Author, "The Email Deliverability Handbook"
Re:Power to the people :) (Score:2, Interesting)
2)
3)
Given the dictionary definition for "junk", this is not an unreasonable mistake.
- RG>
Re:Client-based? (Score:5, Interesting)
I do it without a spam filter of any kind. I have only two technique.
First, simple rule-based filters throw clients and friends into their own folders by from: line alone. That covers everyone I know in advance.
The second set of rules simply looks for my full name, my company name, my e-mail signature, my telephone number, or my mailing address. These into the "it's damn likely a legitimate e-mail" folder. This folder gets about 2 spam e-mails per week.
The remaining I simply run through, in outlook express of all clients. Sorting wins the day. The greatest trick? Sort by the to: field. It doesn't take long to see that 75 messages went to moocow@mydomain.com, 75sevens@mydomain.com, or some other horribly malformed address to that doesn't exist. Sorting by subject does similar things -- like give you "70% off . .
Your spam has very simple patterns to look for. Sort by them, click the first, shift-click the last, and hit delete.
Last year, I was contracted by Viagra's H.R. department to do some quick work, I made it through unscathed.
Re:Client-based? (Score:4, Interesting)
Only solution that I know works is my own: Postfix with amavisd-new, spamassassin, clamav, postgrey, along with FuzzyOCR on smaller installs, though setting that up on a separate system to filter through might cover a large organization. Don't forget to include things like Spamhaus' Zen list, any of the *.countries.dk.net blocklists to filter out any geographical areas from which you don't expect legitimate mail, and also helo filtering--if the connecting mail server can't say helo/ehlo with something that resolves in DNS, it can just bugger right off.
Tell your boss that expecting not to lose email with spam filters in place is unreasonable, and that tasking one human to eyeball all the rejects is a serious misapplication of time and money.
Best of all, you should educate your boss to realize that email is not a reliable messaging system. There are far too many points of failure that could cause a message to be lost, most of them being outside of your own or your company's control. There exist many better ways to send time-sensitive material, like fax, overnight mail, and telephone calls. If a severe amount of money is to be lost because an email didn't make it on time or made it not at all, then the message should have been sent over a more reliable medium in addition to being emailed.
Only the severely clueless would rely on a system like the one you have set up. You have to allow for a certain failure rate in any system. That's a basic principle of quality control methods that have been in use for decades.
You just can't win. (Score:3, Interesting)
We run into the EXACT same issue you're running into.
The dilemma is if we don't tighten the spam filter enough, we'll get complaints from employees (who are not shy about sending EVERY LAST PIECE OF SPAM THEY GET to us.)
However, if they tighten the filter too much, then important emails that may seem spam-like begin to get blocked, and we get just as much heat for that.
The answer - do your best to block what spam you can, and if you get complaints about some spam slipping through, tell them to delete it. We'll often add that we're working with the spam filter vendor to try and resolve the issue, but it's not that easily resolved.
And no - we don't go through each message looking for spam - it's not practical due to the number of employees we have. We DO give them the power block spam from specific addresses on their own, though. The benefit of this is the email is sent to a junk mail folder they can still access, which is useful should something legitimate end up there.)
googleapps is selective (Score:1, Interesting)
searching on google did find this Google Apps Administrator Help [google.com] page. So it looks like because the
Google has one of the best spam blockers in the business, and it's integrated into Google Apps. Spam is purged every 30 days. We have built in virus checking, and we enforce checking of documents before allowing a user to download any message. Most computer viruses are contained in executable files, so standard virus detectors scan messages for executable files that appear to be viruses. Google blocks viruses in the most direct possible way: by not allowing users to receive executable files (such as files ending in
Re:Client-based? (Score:3, Interesting)
If I mail a letter to you, and you don't like the return address on the back of the envelope, you can do with it whatever you please, but it's not my responsibility to ensure that you'll open your mail. It is my responsibilty to deliver your mail. If you don't like the colour of the envelope, that's your problem.
I have a lot of clients who routinely call me saying that one of their messages bounced from some server that says their message won't be delivered for one reason or another. The server received it, and then chose to request that I reconfigure mine. That's just not going to happen. I'm not going to reconfigure my server because another server admin wants me to make his life easier. My server is configured for my reliability, not his ease.
As for my being a spam processing machine, it's actually a lot easier when you get a lot more spam. If I received only 1'000 per day, it would be difficult. But by the time you cross 2'000 spam messages per day, it becomes a lot easier. And by 4'000, it's just funny.
For example, you may find it hard to tell if an e-mail is real by the subject. But if you've received the same subject three times, at three different addresses, it's spam. So when you sort 1'000 messages, how many are duplicate subjects? When I sort four thousand, there are loads of duplicates.
So I've actually got a bunch of extra addresses that I use loosely enough to be spammed thoroughly. It adds to the bulk, and makes sorting easier.