Revolutionary Spam Firewall Developed 507
psy writes "physorg has a story on a new spam firewall developed at The University of Queensland.
The new technology is the only true spam firewall in existence, according to co-developer Matthew Sullivan.
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through," said Mr Sullivan.
"In addition, our technology is accurate and fast. We recently completed a successful trial of a key layer of the spam firewall and it processed the emails at 90 messages per second, misclassifying only one out of 25,000 emails."
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
Spelling (Score:5, Funny)
If you can't spell correctly, then I don't want your v1agr4.
Re:Spelling (Score:5, Informative)
Will your algorithm do it with polynomial complexity
Re:Spelling (Score:4, Informative)
Actually, the number is 1,300,925,111,156,286,160,896. He missed a couple of possibilities and had to update the page.
Re:Spelling (Score:5, Informative)
They've updated... (Score:3, Funny)
Please don't tell me I'm the only one who finds it ironic that the number of different ways to spell it comes out as sextillions...
Re:Spelling (Score:5, Funny)
( Read More... [slashdot.org] | 2 [slashdot.org] of 1274 [slashdot.org] comments | it.slashdot.org [slashdot.org] )
Re:Spelling (Score:5, Interesting)
On the contrary (Score:3, Interesting)
You should re-run your study, and correlate against average IQ before and after...
Re:Spelling (Score:3, Insightful)
Re:Spelling (Score:3, Insightful)
Your typical Bayesian filter works on the message source, not the output of an HTML renderer. "viag<!--xyz-->ra" gets dumped into the spammy-word list along with "v1agr4" and other annoyances, so after the first one sneaks through and is manually classified, the rest are blocked.
Re:Spelling (Score:5, Insightful)
You can make a tokenizer that not only treas a word written like this: 't.r.i.c.k.y', as the word 'tricky', but also as a "pseudoword" like 'trick:dottedword.' So the "bayesian part" of the filter would see these two words: 'tricky' and 'trick:dottedword.'
And there is of course loads of information that can be extracted from the headers of the mail.
Re:Spelling (Score:5, Insightful)
Re:Spelling (Score:3, Insightful)
It's already happened when I sent an email to a client warning about a porn dialer. The repeated mention of porn got my message spam-trapped.
What's needed is a filter that checks these words & spellings in context-but that's far more difficult than the simplistic spell checker that's proposed.
but cant it (Score:2, Funny)
Re:but cant it (Score:4, Funny)
Not the first; not revolutionary (Score:5, Informative)
Damn fine spam firewalls, too, I might add. They handle around 115 messages per second, and can run up to eight filtering steps (including Bayesian analysis, which is similarly efficient to SVM, which the one in the article uses). Plus Barracuda's can do virus scanning.
I'm not sure how this is revolutionary.
Ciphertrust, too... (Score:5, Informative)
Re:Ciphertrust, too... (Score:4, Funny)
Re:Not the first; not revolutionary (Score:5, Insightful)
Re:Not the first; not revolutionary (Score:5, Insightful)
Isn't "revolutionary" just a marketing term for any stupid new product?
Re:Not the first; not revolutionary (Score:5, Insightful)
Re:Not the first; not revolutionary (Score:5, Insightful)
Re:Not the first; not revolutionary (Score:5, Informative)
I'm using Postfix at home and it's got some nifty features to allow you to do this sort of thing. You can write a simple SMTP server that listens on some port of 127.0.0.1 and configure postfix to send the mail though that. Your server scans the E-Mail and sends a reject or accept message back to postfix, which sends it on to the remote MTA. Your SMTP server then feeds the mail into another postfix server which listens on an odd port of 127.0.0.1 and doesn't have the restrictions that your publically accessable postix server does. There are packages available for all sorts of scanning based on this ability. Since you reject the message at MTA time, you don't have to bother with sending a bounce message, either.
Re:Not the first; not revolutionary (Score:3, Informative)
Re:Not the first; not revolutionary (Score:5, Informative)
From here I run accepted emails through AMaViS / SpamAssassin / ClamAV / Sophos Sweep (I have yet had Sophos catch a virus that ClamAV did not detect.. though ClamAV caught two that Sophos did not..) and will not deliver (but notify postmaster) of spams over a set value (ie 8), deliver spam between 5-8 tagged and items under a certain value get passed without tagging. Viruses are always blocked and reported.
Overall this has reduced unwanted email significantly. On networks of 40-60 users, between 35-50% of email is rejected at the SMTP level, about another 10% or so is quarantined (either viruses/spam), another 10% or so is tagged but delivered and the rest is legit.
I have yet had any compliants of false positives (granted there is a risk that they do not know) but have had a lot of priase for reduction in spam levels. I am not aware of any viruses penetrating.
Check out http://jimsun.linxnet.com/misc/postfix-anti-UCE.t
Re:Not the first; not revolutionary (Score:4, Insightful)
Re:Not the first; not revolutionary (Score:3, Interesting)
For example, here is a list of messages that we completely discarded yesterday (in other words, they were dumped before we even bothered invoking our spamass or antivirus routines):
Re:Not the first; not revolutionary (Score:3, Funny)
Is this the next nerd measuring stick?
Nerd #1: I overclocked my spam firewall, i'm getting 119 MPS now!
Nerd #2: Sweet! My mom promised I'd get a new spam firewall accelerator card for Christmas, I'll pwn your 119 MPS!
In fact, it's a step backwards! (Score:3, Interesting)
In fact, you can do all this with free software as well. It's just that the free software was freaking out on us, and requiring way too much handholding. We were losing email, and having huge delays.
The Barracuda (which we found through a
Re:Not the first; not revolutionary (Score:3, Informative)
Somebody tried to delete their backup files, which had $s appended. There were no backup files, so DWIM thought that somehow they'd mistakenly hit the $ key just after pressing *, and in fact meant to delete everything on the disk. And no, heaven forbid that it confirmed this a
Re:Not the first; not revolutionary (Score:3, Insightful)
Sourcecode? (Score:2, Insightful)
Re:Sourcecode? (Score:2, Funny)
Re:Sourcecode? (Score:3, Funny)
Make your chair happier by not sitting on it. Let it sit on you occasionally.
Support Vector Machine (SVM) (Score:2, Insightful)
1/25000 (Score:2, Insightful)
Re:1/25000 (Score:5, Insightful)
Although this is a great new technology, for a business setting, I don't know if even missing one e-mail is acceptable...
That's what everybody says but what's the other option? Letting all the SPAM come in? Do you really think that fed-up employee who gets hundreds of SPAMs a day is really going to do a better job of just mashing down the delete key then a SPAM filter with a 1/25000 error rate?
Of course I doubt this technology would perform that well but the point still stands -- if you don't have a computer flagging them then chances are you have a human flagging them. Who do you trust more?
Re:1/25000 (Score:5, Interesting)
-Adam
Re:1/25000 (Score:3, Insightful)
Re:1/25000 (Score:3, Insightful)
Assuming you give them multiple avenues to contact you, then they simply aren't that interested if they only send one email and drop it after that. Now, I can certainly see trying to make the email system as hardened as possible to prevent any missed email, but the idea that youre going to lose out on some huge sale because of one email being dropped is silly. The grandparent is correct. If you're at all serious in your business, im
Re:1/25000 (Score:5, Interesting)
Re:1/25000 (Score:2)
Re:1/25000 (Score:4, Insightful)
Well... how much money would it take to have the staff necessary to do the filtering manually (at a better rate - even humans are fallible), and how much would the potential business loss cost you? Assuming that the business was very profitable, and that the senders wouldn't call or send a follow-up email of course.
Re:1/25000 (Score:5, Interesting)
It is difficult. We're swatting away a million of the damn things a week and still our users complain. They also complain when we get false positives. And when, next week, we turn on the system that lets them see what we have blocked that was addressed to them, they'll complain too.
I think the one solution they would find acceptable is for me to personally read every one of those million messages and mark it as good or bad. I hope our VP doens't read slashdot....
Re:1/25000 (Score:3, Insightful)
- Some people that want no spam and can accept losing real email.
- Some people that want as little spam as possible without losing any real email.
This is what I like to call "normal."
Re:1/25000 (Score:2, Insightful)
Re:1/25000 (Score:2)
Re:1/25000 (Score:5, Insightful)
1) The e-mail is vitally important and your business will be seriously damaged by its failed delivery.
2) The e-mail was somewhat important, but not something large enough to materially change your revenue/profits.
If the first is the case, you probably shouldn't be using e-mail in the first place and/or whoever sent it is probably going to follow up with a FedEx or phone call.
In the case of number 2 (ha ha, number two), you've saved so much time not having to wade through spam that the losses are negated.
And human error is better? (Score:3, Insightful)
Since the firewall functions based upon code rather than emotion and intuition, the firewall's error rate is going to look better
Re:1/25000 (Score:5, Insightful)
That's why I use SpamAssassin - it does a good job, and is no worse at making false positives than I am. If I'm just as liable to make a false positive than an automatic filter, I'm better off saving my time.
Re:1/25000 (Score:4, Informative)
At our company, current just over 50% of all inbound email is detected as spam. Thus more than 50% of all our inbound email is spam, and the true figure (allowing for the false negatives which slip through) is probably in excess of 60% (and rising)
With a failure rate of 1 in 25,000, AND assuming that means a false positive rather than a false negative, then for our company taking into acount the volume of spam we receive it means 1 email in > 55,000 is wrongly identified.
I can assure you that our business is capable of coping with 1 missed email in > 55,000.
We certainly do not to business-threatening-essential transactions via insecure, non-guaranteeded publicly-transported email, and nor shoudl your business!
Re:1/25000 (Score:4, Interesting)
It is in principle possible to produce a reliable email system, but only if a receipt is returned to the sender when the recipient actually reads the mail, not when it arrives at his ISP for example.
Sadly some businesses do rely implicitly on things that usually, but not always, work, such as mobile phones, pagers, and text messaging. It may have been the same with pigeons, a predator might get the bird! Businesses should set up foolproof systems if they want to do well, a quick phone call to confirm receipt of critical items, for example. The occasional email, even now, takes many hours or even several days to arrive, there is no guarantee whatsoever of time of arrival, but again some seem to think it is "instant", because it very often is. Managers should be aware of these issues, sadly some are not.
But I hope this anti-spam firewall is a brilliant success, and that if it has minor shortcomings there will be satisfactory work-arounds. I am sick of spam, but the ultimate answer must be to ensure that it does not pay, i.e. that the probability of being caught multiplied by the fine greatly exceeds the potential profit. That requires legislation worldwide and some conceptually simple additions to existing mail servers, with care taken to protect the privacy of normal users. Given the political will, and some competent leaders (not Dubya or B. Liar, for a start) it should be easy.
Revolutionary (Score:2, Funny)
Re:Revolutionary (Score:2)
Not a firewall (Score:4, Informative)
Re:Not a firewall (Score:5, Funny)
Ahh, so *that's* what our system administrator is called..
I'll stick to 'Mordac' though.
Fetchmail? (Score:4, Insightful)
Fetchmail + SpamAssassin?
What am I missing here?
Doesn't save B/W: you need to run in INSIDE your network.
Don't care how fast it is: It's a dedicated server.
1/25,000 failure rate with no false positives: OK, that's good. But still not amazing.
How are their servers?
Deployment (Score:3, Interesting)
Yes... (Score:2, Funny)
Uh yeah, OK... (Score:5, Insightful)
Heck, why not do it with a million emails? Makes better headlines that way.
I don't see how this is any different to SpamAssassin (the term 'Mail Firewall' is pure marketing bullshit. It's a spam filter. Get over it.) except I bet it costs a hell of a lot more...
Re:Uh yeah, OK... (Score:3, Insightful)
Take of the rose-tinted spectacles.
Have a look at some of the recent MS or SCO research. *real* researchers give ther results they're paid to give, and don't give a damn about methods.
This a press release (presumably.. definately reads like one). Most of the 'facts' in it were probably dreamed up on the spur of the moment because they sounded good. Assuming they really ran the 25,000 email test then it's almost certain they reached the conclu
Re:Uh yeah, OK... (Score:4, Insightful)
Someone posted a non-slashdotted link. They've formed a company and are after funding - hence this press release. TBH Slashdot should stop giving these people airspace.
This is *not* science it's a corporate press release. If they had the integrity you ascribe to them (which really doesn't exist - everyone has an agenda, whether it's to get published or, in this case, to get money) then they'd never have allowed it to go out with claims like this is 'new' and 'revolutionary' which are quite obviously total bullshit.
And no, it's still not a firewall. I do exactly the same with postfix and spamassassin and that's not a firewall either. It's a mail filter.
What happens to the 1 mis-classified email? (Score:5, Interesting)
Mirror ?? (Score:2)
Useless (Score:3, Insightful)
I'd rather get 5 extra spam if it meant I also recieved every real email.
Re:Useless (Score:2, Insightful)
Re:Useless (Score:2)
I'll arrange your extra spam, sir.
My favorite line: (Score:5, Funny)
Maybe they should be working on a Slashdot-Firewall. Damn, I really should get back to work.
Oh, and since the linked article got
http://www.uq.edu.au/news/index.phtml?article=583
Spin doctors (Score:4, Insightful)
Heh. Does anyone else see that as a good way to downplay false positives?
"Oh, good point, Computer. That email from my boss actually was spam. I didn't realize that until you mentioned it."
Re:Spin doctors (Score:4, Insightful)
Advertising story (Score:2)
Spam firewall? I want a hard drive firewall (Score:4, Interesting)
What I want to see is a software hard drive "firewall." If you're not sure what I mean, think of what a product like zone alarm does when spyware.exe tries to access the internet on your pc. It pops up a window saying "do you want to allow this program..." Now, why can't we have the same thing for hard drive access? So, I download fungame.exe, and when I go to run it, my "firewall" tells me fungame.exe is trying to write to fifteen different directories to install different spyware products. It could only give a popup on the first time a program tries to write to a given directory, and have an option to not show any new notices for this program, to limit the annoyance factor. I think this would be a great tool to help lessen spyware/trojan problems. If the program interacted with spybot or a similar product, it could even automatically prevent writing of files that are known to be adware. Is there anything like this out there? Anyone who would be willing to help make it?
Re:Spam firewall? I want a hard drive firewall (Score:3, Insightful)
The guy is not asking for a sandbox. He is asking for the ability to give or deny individual processes write-access to the hard drive. That's something quite different from a sandbox.
I would also be interested is software that does this.
For those who belive this .. (Score:2)
For those who belive this software actually can do this well in real-life environment, I have this bridge that might interest you ...
Question (Score:2)
This One Goes to Eleven (Score:2)
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through"
Slashdotting has made it impossible to check for more meaning in the article, so can anyone tell me what the difference is supposed to be here. How does stopping mail and then allowing non-spam through differer from a spam filter? It sounds like pretty much what the qmail/spamassassin boxes I've set up as mail gateways do.
filter out email and junk words. (Score:2)
one bad thing about all the misspellings is that the spam poetry project got messed up..
As a self-appointed representative of ... (Score:5, Funny)
Big deal (Score:3, Funny)
Here's how it probably works (Score:5, Interesting)
The idea is that the mail server keeps a whitelist of "allowed" addresses which are always accepted. If a mail comes from an address which is not known, the mail server will reply with a "server unavailable, try later" error message. All real mail servers will try to send the message a little later (I don't know the exact time, but it's probably less than an hour. Someone else might know better).
The second time the remote mail server tries to connect, the server accepts the mail and adds the address to the whitelist.
However, mass mailers for spam don't do this but simply go on to the next address in the list if this happens. This way the spam message is filtered out.
Note that this method doesn't require any analysis of the actual content of the messgae, nor does it involve any manual actions from neither the sender nor the receiever. Currently it's porbably the best spam blocking method that exists.
Re:Here's how it probably works (Score:4, Informative)
That's how spamd [openbsd.org] works, and yes, it works tremendously well. I used to get 300 spam messages daily. I receive now one or two every week.
Re:Here's how it probably works (Score:3, Insightful)
Until the spammers catch on and start to resend their requests. This seems like a stop-gap solution.
Re:Here's how it probably works (Score:3, Insightful)
It is, but it's a GOOD stop-gap. In order to resend the bounced greylisted message, you'd have to be resending ALL soft bounced messages the number of which, assuming you're sending millions of emails a day, is not insignificant.
It makes the cost of doing business higher for spammers, which ideally cuts down on their profits, making spamming less attractive.
Re:Here's how it probably works (Score:4, Informative)
Our experience with greylisting has been (1) an 90%+ reduction in passed-through email (with no complaints from users about lost mail (yet)), (2) a dramatic decrease in server load because SpamAssassin doesn't see the message until after it gets past greylisting, and (3) people rediscover how useful email is once you get all of the crap out of their inbox.
Marketing Guy: What's the worst that could happen?
Dilbert: Our beta product could turn into an evil robot that annihilates the galaxy.
Re:Here's how it probably works (Score:3, Informative)
And for those that say this is a stop gap and won't be effective for very long, they are wrong.
The whole idea is to increase the cost to the spammer of sending out millions of emails. By greylisting they have to resend the same message at least twice, possibly multiple times, since they don't know how long the delay is.
On top of that if you combine greylisting with an RBL
I hope they don't reject my e-mail (Score:5, Funny)
amidoacetic platymyoid granomerite nonacceptant dorsoposteriad uninclined unshocked zibet intercity lornness
Re:I hope they don't reject my e-mail (Score:3, Funny)
-Adam
What's the problem? (Score:3, Funny)
Not to mention my enormous, permanently erect p3N1s.
Just say NO to spam-blocking!
Why filter at firewall layer? (Score:4, Insightful)
For example, Mail Avenger [mailavenger.org] allows you to filter spam based on network characteristics like SYN fingerprints and routes. It even integrates with the kernel firewall to filter out aggressive spammers and mail bombers. However, because it runs as an ordinary user-level process, it also has much more flexibility, for example allowing individual users to set different policies on different email addresses. What can a spam "firewall" do that you can't do with a system like Mail Avenger.
One Revolutionary anti-spam firewall right here! (Score:3, Interesting)
Once the relay determines a message is spam, it rejects and drops the message before it is transferred to the 'real' mail server. End users never even know the message was there...
We set up two of these about 6 months ago and eradicated most of our spam problems. (some still get through, on the order of 5 - 10 false negatives on a mailserver handling about 3k messages per day.)
The what where now? (Score:5, Funny)
False Positives (Score:3, Insightful)
They are celebrating false positives?
Won't work. (Score:3, Funny)
Fenley's torment.
Article slashdotted, but skeptical of the blurb (Score:3, Insightful)
All these phrasings automatically trigger my B.S. filter. Or should I say firewall.
Vapor (Score:3, Interesting)
Someone has figured out how to build a "spam firewall" that is different from everything out there. Yeah right. No details to tell us exactly how it is different.
My guess is that they took a software based product using baysien filters and some other common anti-spam filtering technology and packaged it in hardware. Won't really improve the function of the machine but could possibly help with performance (process mail faster).
I won't believe it is anything else until I actually see it. Unfortunately, I don't think that will happen anytime soon.
Revolutionary Mail Firewall? (Score:3, Informative)
Hell, there's even a product called the Mail Firewall [borderware.com] that pops up if you google for mail firewall [google.com].
Old news (Score:3, Informative)
Heuristic analysis - detects and blocks spam by various email characteristics
Black lists - checks if the sending server is in RBL (Realtime Blackhole List), dial-up or open-relay servers
DNS verification - checks if the sender is using a valid mail server
Keyword blocking - blocks spam according to keywords in subject and body
Anti-spoofing - blocks email masquerading as coming from within the organization - a common spam technique
Cookies/web beacons - blocks email cookies which help spammers identify the recipient as a "live" email
Header verifier - inspects various header signatures and blocks spam
Textual analysis - categorizes spam according to textual content like mortgages, pornography, dental care, etc
Spam signatures - an auto-updating spam database allows detection and blocking of spam according to smart signatures
Spam URL filtering - blocks email with links to spam sources and sponsors
Spam image filtering - blocks email containing spam associated images
Auto-updating database - local or remote spam blocking database based on thousands of Spam collecting bots and web crawlers
http://www.esafe.com/esafe/anti-spam.asp [esafe.com]eSafe
Re:One solution to spam (Score:3, Interesting)
Not new. Nobody ever sends the replies. Mailing lists automatically ban users who run it (I know I do... if they didn't want email they shouldn't have frikkin registered, so I grant them their wish and ban them.).
people not considering their mail important enough
Well if you don't consider my email important enough to read it before assuming it's spam, I don't see why I should continue the conversation.... Sucks for you if I just sent you a job offer..
Re:One solution to spam (Score:4, Interesting)
Solution (Score:4, Funny)