Revolutionary Spam Firewall Developed 507
psy writes "physorg has a story on a new spam firewall developed at The University of Queensland.
The new technology is the only true spam firewall in existence, according to co-developer Matthew Sullivan.
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through," said Mr Sullivan.
"In addition, our technology is accurate and fast. We recently completed a successful trial of a key layer of the spam firewall and it processed the emails at 90 messages per second, misclassifying only one out of 25,000 emails."
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
Deployment (Score:3, Interesting)
What happens to the 1 mis-classified email? (Score:5, Interesting)
Re:1/25000 (Score:5, Interesting)
-Adam
Re:1/25000 (Score:5, Interesting)
Spam firewall? I want a hard drive firewall (Score:4, Interesting)
What I want to see is a software hard drive "firewall." If you're not sure what I mean, think of what a product like zone alarm does when spyware.exe tries to access the internet on your pc. It pops up a window saying "do you want to allow this program..." Now, why can't we have the same thing for hard drive access? So, I download fungame.exe, and when I go to run it, my "firewall" tells me fungame.exe is trying to write to fifteen different directories to install different spyware products. It could only give a popup on the first time a program tries to write to a given directory, and have an option to not show any new notices for this program, to limit the annoyance factor. I think this would be a great tool to help lessen spyware/trojan problems. If the program interacted with spybot or a similar product, it could even automatically prevent writing of files that are known to be adware. Is there anything like this out there? Anyone who would be willing to help make it?
Here's how it probably works (Score:5, Interesting)
The idea is that the mail server keeps a whitelist of "allowed" addresses which are always accepted. If a mail comes from an address which is not known, the mail server will reply with a "server unavailable, try later" error message. All real mail servers will try to send the message a little later (I don't know the exact time, but it's probably less than an hour. Someone else might know better).
The second time the remote mail server tries to connect, the server accepts the mail and adds the address to the whitelist.
However, mass mailers for spam don't do this but simply go on to the next address in the list if this happens. This way the spam message is filtered out.
Note that this method doesn't require any analysis of the actual content of the messgae, nor does it involve any manual actions from neither the sender nor the receiever. Currently it's porbably the best spam blocking method that exists.
Re:Not the first; not revolutionary (Score:1, Interesting)
Re:Not the first; not revolutionary (Score:2, Interesting)
Plugged it in and set it up. My users actually asked me why the spam has stopped, since they don't understand the concept of filtering messages out tagged with *****SPAM***** in their email client. I must say I was really impressed. Until I had to send it back.
Re:One solution to spam (Score:3, Interesting)
Not new. Nobody ever sends the replies. Mailing lists automatically ban users who run it (I know I do... if they didn't want email they shouldn't have frikkin registered, so I grant them their wish and ban them.).
people not considering their mail important enough
Well if you don't consider my email important enough to read it before assuming it's spam, I don't see why I should continue the conversation.... Sucks for you if I just sent you a job offer..
Re:One solution to spam (Score:4, Interesting)
One Revolutionary anti-spam firewall right here! (Score:3, Interesting)
Once the relay determines a message is spam, it rejects and drops the message before it is transferred to the 'real' mail server. End users never even know the message was there...
We set up two of these about 6 months ago and eradicated most of our spam problems. (some still get through, on the order of 5 - 10 false negatives on a mailserver handling about 3k messages per day.)
Re:Not the first; not revolutionary (Score:1, Interesting)
In fact, it's a step backwards! (Score:3, Interesting)
In fact, you can do all this with free software as well. It's just that the free software was freaking out on us, and requiring way too much handholding. We were losing email, and having huge delays.
The Barracuda (which we found through a
Is it perfect? No? But most of my complaints are niceties in the GUI, so it's still well ahead of where we were before trying to maintain things ourselves.
This may be a new, rockin' way to detect spam, but if so, they need to pitch it better. They're focusing on the wrong things, IMO. I have an enterprise to run, and marketing jive doesn't cut it.
[1] It's a dessert wax and a floor topping!
Re:1/25000 (Score:5, Interesting)
It is difficult. We're swatting away a million of the damn things a week and still our users complain. They also complain when we get false positives. And when, next week, we turn on the system that lets them see what we have blocked that was addressed to them, they'll complain too.
I think the one solution they would find acceptable is for me to personally read every one of those million messages and mark it as good or bad. I hope our VP doens't read slashdot....
Vapor (Score:3, Interesting)
Someone has figured out how to build a "spam firewall" that is different from everything out there. Yeah right. No details to tell us exactly how it is different.
My guess is that they took a software based product using baysien filters and some other common anti-spam filtering technology and packaged it in hardware. Won't really improve the function of the machine but could possibly help with performance (process mail faster).
I won't believe it is anything else until I actually see it. Unfortunately, I don't think that will happen anytime soon.
Re:Here's how it probably works (Score:2, Interesting)
As mentionned in other posts, he's describing http://greylisting.org/ [greylisting.org]. Even if spammers adapt their software, the beauty of the system is that by the time the message is resent, it's probably already in a distributed spam database, so spamassassin will give it a higher score than if it had been accepted the first time around.
Re:Spelling (Score:5, Interesting)
In case of slashdotting.. (Score:1, Interesting)
The email spam nightmare could be halted in cyberspace by a groundbreaking firewall developed at The University of Queensland.
The new technology is the only true spam firewall in existence, according to co-developer Matthew Sullivan.
"Existing anti-spam software filters out spam whereas ours puts up a firewall, stopping all email traffic and only allowing real mail through," said Mr Sullivan.
"In addition, our technology is accurate and fast. We recently completed a successful trial of a key layer of the spam firewall and it processed the emails at 90 messages per second, misclassifying only one out of 25,000 emails."
"It turned out that the software was even better than us, picking up spam we'd incorrectly classified as legitimate emails."
A Specialist Systems Programmer at The University of Queensland, Mr Sullivan worked on the spam firewall concept largely in his spare time, only coming together this year to work on the project with Guy Di Mattina, a recent UQ Engineering honours graduate, and Dr Kevin Gates, a UQ mathematics lecturer.
Pivotal to the trio's spam firewall is the unique method of using a Support Vector Machine (SVM) to categorise emails. The only anti-spam software that analyses emails as a whole picture, rather than based solely on components such as key words or phrases, said Mr Sullivan.
"Using a SVM, we can train our spam firewall to accurately recognise legitimate emails to the extent that it can tell the difference between a pharmaceutical bulletin on Viagra and someone trying to sell Viagra," he said.
UQ's main commercialisation company, UniQuest, has formed a start-up company based on the technology and is seeking investment to take the spam firewall to market.
UniQuest Managing Director, David Henderson said the global cost of spam was estimated by the Radicati Group in 2003 to be $20.5 billion or $49 per user mailbox.
"With spam escalating and companies losing valuable employee time to deleting spam, UniQuest hopes to get this revolutionary spam firewall technology on the market quickly but it just depends on the level of funding we receive," said Mr Henderson.
Source: University of Queensland
Spam problem gets blown out of proportion.. (Score:1, Interesting)
As an endnote, after a few months of my account cripple with spam, I went through and followed the unsubcribe links for each mail that I got. Deleted them and repeated until after about 2 weeks I was no longer receiving spam(aside from the university's student announements which I considered to be the worst perpetrator of spam in existence).
I guess my point is that for the most part repeat spam shouldn't be a problem because it can be stopped. Now learning how to not subject yourself to new spam is a valuable thing. And way more useful than some filter/firewall bloat.
Re:1/25000 (Score:4, Interesting)
It is in principle possible to produce a reliable email system, but only if a receipt is returned to the sender when the recipient actually reads the mail, not when it arrives at his ISP for example.
Sadly some businesses do rely implicitly on things that usually, but not always, work, such as mobile phones, pagers, and text messaging. It may have been the same with pigeons, a predator might get the bird! Businesses should set up foolproof systems if they want to do well, a quick phone call to confirm receipt of critical items, for example. The occasional email, even now, takes many hours or even several days to arrive, there is no guarantee whatsoever of time of arrival, but again some seem to think it is "instant", because it very often is. Managers should be aware of these issues, sadly some are not.
But I hope this anti-spam firewall is a brilliant success, and that if it has minor shortcomings there will be satisfactory work-arounds. I am sick of spam, but the ultimate answer must be to ensure that it does not pay, i.e. that the probability of being caught multiplied by the fine greatly exceeds the potential profit. That requires legislation worldwide and some conceptually simple additions to existing mail servers, with care taken to protect the privacy of normal users. Given the political will, and some competent leaders (not Dubya or B. Liar, for a start) it should be easy.
Re:Not the first; not revolutionary (Score:3, Interesting)
For example, here is a list of messages that we completely discarded yesterday (in other words, they were dumped before we even bothered invoking our spamass or antivirus routines):
Barricuda tried to sell to us, but quite frankly, our existing system built on sendmail/mimedefang/spamass is working great. Barricuda admitted that they couldn't beat what we are doing ourselves.
Some things can't be accurately filtered (Score:2, Interesting)
1) I get a spam "from" you and forward it to you with a note saying "did you send this." You want to get this type of email. Since you might get such a message from anyone at any time, traditional "is he in my mailing list" filters aren't suitable.
2) I'm a spammer and malware writer, and I write a virus that sends mail from my victim's machine that looks identical to #1. Even though the message is malware-free, you definately do NOT want this message.
No human recipient can tell the two apart, by looking ONLY at the received email.
Of course, no computer can identify "friend or foe" by simply looking at the message either.
So, if you are looking for the perfect filter, it doesn't exist.
If you are looking for a filter that's better than a person, I recommend Yahoo for web-based mail and a number of good solutions for your own system.
In the above scenario, there are solutions. One requires analyzing multiple copies of the message to spot patterns, something big houses like AOL and Yahoo can do but small shops that may only get 1 copy of the message cannot. You can also use RBL lists that track zombied machines, but that won't trigger if the machine in question isn't RBL'd yet. Delay-try-again-later tactics like those mentioned elsewhere in this thread can help here, but are ruinous if you want legitimate complaints ASAP. "Man in the loop" solutions like sending a confirmation message might help, but many people ignore such requests.
Re:Spelling = easy (Score:2, Interesting)
Yes:
Almost forgot:
4.
5. Profit
Eh, never mind...
I did this myself (Score:2, Interesting)
On the contrary (Score:3, Interesting)
You should re-run your study, and correlate against average IQ before and after...