The Next Step in Fighting Spam: Greylisting 481
Evan Harris writes "I've just published a paper on a new and unique spam blocking method called "Greylisting". The best thing about it other than achieving better than 97% effectiveness in blocking spam, is that it practically eliminates the main problem of other solutions: the false-positive. There's even source code for an example implementation written as a perl filter for sendmail, along with instructions for installing, so you can get up and running quickly."
Bayesian Filtering (Score:3, Interesting)
How does the effectiveness of Greylisting compare with what others are seeing with existing techniques (such as Bayesian filtering)? Is it a false positives problem, such as digests and opt-in mailing lists getting incorrectly tagged as spam?
Re:security through obscurity, again? (Score:5, Interesting)
There are people out there who pay $10000 in startup costs, and then make $2000/week for spamming. The $10000 gets them software written by knowledgable internet security experts. This software finds any and every way to anonymify the email spam, and finds lists of people to spam.
As long as knowledgable internet security experts are getting paid good cash to enable spammers, and SMTP doesn't change, spam will only continue to get worse. There needs to be a fundamental change in SMTP protocols. It oughta take the spammers about 2 days to fix their MTA bug to get around greylisting.
My spam filtering (Score:1, Interesting)
I think not (Score:5, Interesting)
Re:can't believe their numbers (Score:3, Interesting)
How about Habeas' haiku method? (Score:4, Interesting)
Re:1 false positive is not acceptable. (Score:5, Interesting)
At USENIX '03 there was a paper presented on artificial intelligence techniques for spam detection. I can't provide a link since only USENIX members can download the paper (at this point, at least). I was a coauthor of that paper.
One of the things we've discovered in our research is that some classes of filters (most notably, the one I have been developing along with a few other individuals) are actually more effective at correctly classifying email than humans are. That is to say, you can train the learning algorithm on mostly-correctly-classified data, then re-run it over the training data, and almost miraculously, it discovers all kinds of email in the training set that was incorrectly classified.
I.e., this filter has discovered mail that I myself incorrectly thought was spam. It's scary, because there's a lot of it.
To assume that a human will always be 100% accurate at classifying their own email isn't just arrogant, it's plain wrong. Newer filters that will be introduced in the near future might possibly be more accurate than you, a frail human, could ever be.
Re:your first mistake (Score:2, Interesting)
Re:Questions (Score:1, Interesting)
clever hack for WHOIS contact addresses (Score:5, Interesting)
Why spam me? (Score:1, Interesting)
Bogofilter does pretty well for a client filter (Score:4, Interesting)
I'm currently using Bogofilter [sourceforge.net] (and looking into CRM114 [sourceforge.net]) and getting better than 99% accuracy (about 1 in 200 false negatives at the moment) and very very few false positives (maybe 2 in 5000 messages).
Of course these are MUA level filters (and yes, I know, I've already "paid" with bandwidth to download the spam) - however since the proposed "greylister" would have to be installed as the MTA at major ISPs (as the authors note) I'm not convinced that is more likely to get widespread adoption than the various sorts of adaptive client-based filtering now available, particularly as it requires a database to back the method up.
As far as I am concerned the major factor in a spam filter should be zero false positives - personally I don't mind reviewing one or two spams a week but I get really annoyed if I were to lose a real message (note the two false positives I have sent to date with bogofilter contained forwarded sales pitches along with a message).
Re:your first mistake (Score:5, Interesting)
Not to mention that if this is used in conjunction with other collaborative tools (i.e. RBL, checksums), by the time that the spamming MTA can return its IP address will have been submitted to MAPS/etc. and the contents of the message will have been submitted to Razor/Pyzor/DCC.
I think that this greylisting idea will be pretty hard to beat by Joe spammer. Since the game of spam detection is pretty much an arms race, slowing him down will probably be enough to turn the battle in your favour.
The Ironic Spam Solution (Score:2, Interesting)
I don't get any spam using this approach, because the spammers don't send big messages. And if *everyone* ignored small messages, spammers would have to close up shop because they could not afford to send millions of big messages.
(This is a joke. But you could do this at the SMTP level, by automatically replying to any sender who is not on your personal whitelist with the response: "Hey you, if you're real, send me back a HUGE reply!" And the SMTP server could cheerfully delete the last 99% of the first-time oversized email you get. I should write my own anti-spam paper and get mad Slashdot cred. Nah, I'm too lazy.)
Re:here are the stats (Score:3, Interesting)
You'll notice that the US is the #1 country Top 3 are:
- The United States, with over 80,000 open relays
- Korea and Japan pretty much tied at +15,000 each
- Japan, at just under 10,000
That's more than everyone else combined!E-Mail Secrets (Score:2, Interesting)
How about DSPAM ? (Score:3, Interesting)
I know this is happening as some complete bastard seems to be doing this using my domain as a "From:" address (well, [random-word]@schmerg.com), meaning that I'm been getting about 30 or 40 bounce messages a day for the last 2 or 3 months now. And although the odd sending IP is repeated, mostly they're all from different IP addresses. And of course I'm getting perfectly valid looking bounce messages from perfectly reasonable companies (and only a couple of abusive replies so far).
Now the problem is that the email is being uploaded to thier (non-open-relay) ISP's mailserver that will retry properly, and anyone else looking at the IP address will see a perfectly reasonable IP (the spammer seems to gave infected a lot of AT+T customers, ComCast customers, etc.). So short of blocking spam on subject, this spam is harder to prevent in the first place.
I've semi-automated a process to report the infected machines (that provoked a bounce message) to the appropriate ISP, and seem to havign some success in getting the ISPs to contact their customers, but I think this new form of spamming using a distributed attack will be particularly hard to block.
Anyone with a great idea (or who knows more about this scheme, or the identity of the twat behind it) I'd love to hear from you...
--
T
Spammers don't care about defeating the top 5%. (Score:3, Interesting)
These are widespread practices and yet spammers don't care about spending the effort to reach that 5% who so adamantly don't want to be reached.
Unless greymail is used by more than a quarter of *all* email readers, history has shown that the spammers just won't care.
That's a good point (Score:3, Interesting)
Limit Criminal Penalties (Score:2, Interesting)
It says that Mississippi tried to outlaw "flag burning", and the law was struck down as unconstitutional. So the Mississippi legislature responded by limiting the maximum penalty for assaulting a person who is engaged in flag burning to a $25 fine.
This sounds like a fine idea on how to handle the spam problem.
Create a law that states that the maximum penalty for physically assaulting a spammer is a $100 fine. I know more than a few people who'd be willing to pony up and take a whack at them.
Though this law would probably be far more satisfying than sensible.
Re:security through obscurity, again? (Score:5, Interesting)
I doubt it. I would assume the spam software would have a timeout, and I doubt it's ten minutes. If they want to hit-and-run and aren't even willing to make a second delivery attempt when an error code is returned, I doubt they're going to wait 10 minutes. I'm sure that within 30 seconds or less they'll consider it a dead connection and hang up.
Problem is, I used to have my sendmail HANG UP in real-time on an incoming connection as soon as it realized a message was spam. I.e., the incoming message was filtered in the DATA phase and if it was spam I hung up immediately. It worked great and it felt good, but there were many spam programs that took the disconnection as some kind of TCP/IP failure and immediatelty tried again. So I had one day where a single message was attempted to be delivered about 30,000 times as the spammer connected, I hung up, spammer software said "Oops, let me try again!" About one delivery attempt every second or so.
I'd be willing to bet if you put a 10 minute timeout in sendmail you'll see lots of spammer software disconnecting sooner and just trying again. It takes more of their resources, but takes more of yours, too.
Re:security through obscurity, again? (Score:2, Interesting)
If they are sending 250,000,000 emails a day, then that must be all they CAN (or think they can - amounts to the same thing) send or they would be sending more. If they have to send everything twice, then they have just dropped the number of emails to 125,000,000 - and cut their income from their activity in half.
Another possibility is that they double the size of their email farm and the width of their pipeline - but that also takes $$$, time, and resourses - my computer requires electricity or it refuses to work.
I am for anything that hits spammers in the pocketbook.
Re:security through obscurity, again? (Score:4, Interesting)
Yeah, spammers are so clever. Well, the fact is if for every one of these "smart" (yeah, right) spammers who has the help of a network consultant that will work around greylisting there are 5 dumbasses who don't (and I think I'm being generous there), then if I greylist I'd think over 80% of my spam problem would be eliminated. What's wrong with that? What's to "get"? Looking through headers I see the same bulk mailers used over the years, probably passed around as warez in spammer circles.
My Anti-Spam Idea (Score:3, Interesting)
I've submitted it to Slashdot. They rejected it. Tell me what you think. I'd like reactive approaches to get discussed a bit more. If you do too, submit this to Slashdot
Even if something like this could work... (Score:2, Interesting)
Listservs now well served by greylisting (Score:2, Interesting)
Well, the first e-mail to a news subscriber is often the e-mail required to confirm subscription. No reply, and the subscriber is plonked.
Sounds suboptimal to me.
So, you whitelist the listserv machines... until one of them has to change IP addresses. Whoops! No umpteen bazillions of e-mail messages go no where.
I'm sure that listserv admin would find the idea suboptimal about this time.
Nice idea, but No Slack, as far as I can see.