The Next Step in Fighting Spam: Greylisting 481
Evan Harris writes "I've just published a paper on a new and unique spam blocking method called "Greylisting". The best thing about it other than achieving better than 97% effectiveness in blocking spam, is that it practically eliminates the main problem of other solutions: the false-positive. There's even source code for an example implementation written as a perl filter for sendmail, along with instructions for installing, so you can get up and running quickly."
your first mistake (Score:4, Insightful)
You have just rendered Greylisting pretty useless by making it open source. Spammers are much smarter than you think and what you have basically done is shown them what they need to do in order to get around Greylisting. That's just my take on the issue, maybe I'm wrong but I doubt it.
security through obscurity, again? (Score:5, Insightful)
Re:security through obscurity, again? (Score:5, Interesting)
There are people out there who pay $10000 in startup costs, and then make $2000/week for spamming. The $10000 gets them software written by knowledgable internet security experts. This software finds any and every way to anonymify the email spam, and finds lists of people to spam.
As long as knowledgable internet security experts are getting paid good cash to enable spammers, and SMTP doesn't change, spam will only continue to get worse. There needs to be a fundamental change in SMTP protocols. It oughta take the spammers about 2 days to fix their MTA bug to get around greylisting.
Re:security through obscurity, again? (Score:4, Insightful)
The way to get around this, of course, being that you send each email twice. In other words, run through your database, then run through your database. Same IP addy, same sender, same recipient. As far as the MTA's concerned, it's retrying. Boom.
Re:security through obscurity, again? (Score:4, Insightful)
From the article: If we have never seen this triplet before, then refuse this delivery and any others that may come within a certain period of time with a temporary failure. (emphasis addded)
Later in the article it goes into much more detail about the delay, how long to delay if the triplet has not been seen before, life time of the whitelist, etc.
It also talks about configuring the times - they mention the default delay is 1 hour, but that their records suggest that 1 minute would have caught 99% of the same spam messages - "The data collected during testing showed that more than 99% of the mail that was blocked with the tested setting of 1 hour would still have been blocked with a delay setting of only 1 minute. At that point, having a larger initial delay will definitely help, as it gives time for other blocking methods to act. For this reason, it is suggested that at least a one hour delay value be kept as a default, since spammers will start adapting as soon as this method becomes known and starts being used. (again, emphasis added)
RTFA!
Re:security through obscurity, again? (Score:5, Insightful)
There is no magical waiting period or re-try period that cannot be trivially coded around. And, with good money on the line, will be trivially coded around.
You don't get it. Really smart people are getting paid a whole lot of money to make programs to exploit every possible crack in the way we send email. There is no general rule to spammers, except that it is a lot of money and they are very clever. Little bandaids are not going to stop this one - there needs to be a much more fundamental change. And I am not talking about laws against spam - I am talking about changes in the protocols we use to send email.
Re:security through obscurity, again? (Score:4, Interesting)
Yeah, spammers are so clever. Well, the fact is if for every one of these "smart" (yeah, right) spammers who has the help of a network consultant that will work around greylisting there are 5 dumbasses who don't (and I think I'm being generous there), then if I greylist I'd think over 80% of my spam problem would be eliminated. What's wrong with that? What's to "get"? Looking through headers I see the same bulk mailers used over the years, probably passed around as warez in spammer circles.
Re:security through obscurity, again? (Score:3, Informative)
Well, the fact is if for every one of these "smart" (yeah, right) spammers who has the help of a network consultant that will work around greylisting there are 5 dumbasses who don't
This does fuck all when your one spamking is responsible for 80% of the SPAM (by volume.
Re:security through obscurity, again? (Score:5, Interesting)
I doubt it. I would assume the spam software would have a timeout, and I doubt it's ten minutes. If they want to hit-and-run and aren't even willing to make a second delivery attempt when an error code is returned, I doubt they're going to wait 10 minutes. I'm sure that within 30 seconds or less they'll consider it a dead connection and hang up.
Problem is, I used to have my sendmail HANG UP in real-time on an incoming connection as soon as it realized a message was spam. I.e., the incoming message was filtered in the DATA phase and if it was spam I hung up immediately. It worked great and it felt good, but there were many spam programs that took the disconnection as some kind of TCP/IP failure and immediatelty tried again. So I had one day where a single message was attempted to be delivered about 30,000 times as the spammer connected, I hung up, spammer software said "Oops, let me try again!" About one delivery attempt every second or so.
I'd be willing to bet if you put a 10 minute timeout in sendmail you'll see lots of spammer software disconnecting sooner and just trying again. It takes more of their resources, but takes more of yours, too.
Re:security through obscurity, again? (Score:3, Insightful)
Quoting from the paper:
Re:security through obscurity, again? (Score:4, Insightful)
I disagree. When you are sending 250,000,000 emails a day -- restarting that script IS a big deal. It would, in effect, make them have to do the entire thing twice. That's a pretty big hit on their resources.
That's a good point (Score:3, Interesting)
Re:your first mistake (Score:5, Insightful)
Not trolling at all - you have a legitimate (though perhaps misguided) problem with this method.
You have just rendered Greylisting pretty useless by making it open source. Spammers are much smarter than you think and what you have basically done is shown them what they need to do in order to get around Greylisting. That's just my take on the issue, maybe I'm wrong but I doubt it.
So, the spammers themselves will be of significant help in debugging and helping to fix the code so they can't circumvent it, won't they? OSS means anyone who finds how the greylist script is beaten can figure out a fix and post it. Sounds like the best thing to do IMHO.
Soko
Re:your first mistake (Score:5, Funny)
Comment removed (Score:5, Funny)
Re:your first mistake (Score:5, Funny)
Who do you think writes spamming software?
Re:your first mistake (Score:5, Funny)
That's what I like to see. Someone with strong opinions. Or maybe not.
I think not (Score:5, Interesting)
Re:I think not (Score:2)
Just because something is open source (or in this case, open idea) doesnt mean that it is rendered useless. There are plenty of open source programs involving cryptography as well as other sensitive subjects that work perfectly well.
In this case as others have mentioned, it would require some fixes to patch up the things that spammers figure out quickly, but I do believe that in the end it will be a stronger system that if it w
Secret algorithms vs. secret keys (Score:3, Informative)
Re:your first mistake (Score:4, Informative)
Legitimate people first time sending won't really mind the few day wait and most MTAs will try for upto a month.
Tom
Re:your first mistake (Score:5, Informative)
Read the paper. Spammers would figure it out eventually. What it buys is what they have to do to get around it.
It means they have to do retrys...that means spam runs take longer, especially since they have to run...then wait for a locally defined timeout, and run all those addresses again
AND they have to do it from the same IP.
This raises their bandwidth profile. It wastes their time... all in all... it raises their cost of doing buisness and cuts into their profit margins.
It means they will have to upgrade their tools again. It means they get headaches. And of course, the next step is to impliment spam traps that watch activity and see that a spammer is spamming, and promotes them to a blacklist before they can even retry. (oh gee 1000 new greylist triplets from 1 IP in under 5 mins? Set the timeouts for that IP to 12 hours)
-Steve
Re:your first mistake (Score:5, Interesting)
Not to mention that if this is used in conjunction with other collaborative tools (i.e. RBL, checksums), by the time that the spamming MTA can return its IP address will have been submitted to MAPS/etc. and the contents of the message will have been submitted to Razor/Pyzor/DCC.
I think that this greylisting idea will be pretty hard to beat by Joe spammer. Since the game of spam detection is pretty much an arms race, slowing him down will probably be enough to turn the battle in your favour.
Re:your first mistake (Score:2, Interesting)
Spammers don't care about defeating the top 5%. (Score:3, Interesting)
These are widespread practices and yet spammers don't care about spending the effort to reach that 5% who so adamantly don't want to be reached.
Unless greymail is used by more than a quarter of *all* email readers, history has shown that the spammers just won't care.
Questions (Score:2, Insightful)
Re:Questions (Score:4, Insightful)
Since most personal users are on dialup or dynamic IPs, unless the mail client can upload the whitelist in a trusted fashion (or the MTA remembers what users the client sent messages to!), this won't work.
Do any mail clients include whitelist-collection? Mail.app for OS X does collect all addresses you've sent to, but I've never seen any tool to upload it somewhere.
Re:Questions (Score:4, Informative)
Re:Questions (Score:2, Informative)
Much of my e-mail comes through within seconds - I'm not sure I want that delayed too much. Although, this delay is on the first matching triplet.
Server disk space requirements for major providers would climb considerably, I would expect. Legitimate mass-mail programs, and mailing list services would have a problem, tho.
The algorithm takes advantage of the lazyness of spammers, which is no
can't believe their numbers (Score:5, Informative)
Re:can't believe their numbers (Score:5, Informative)
Re:can't believe their numbers (Score:3, Interesting)
Poor use of statistics (Score:5, Insightful)
The article also doesn't measure the amount of spam coming through those relays. Even if there are only 10 open relays in the UK at any one time, it still might be possible for all of the spam to be coming through them.
Certainly, closing down open relays is a good thing, but lowering the percentage of open relays doesn't prove anything about the source of spam
Re:here are the stats (Score:3, Interesting)
You'll notice that the US is the #1 country Top 3 are:
- The United States, with over 80,000 open relays
- Korea and Japan pretty much tied at +15,000 each
- Japan, at just under 10,000
That's more than everyone else combined!Open Relays a smaller problem? Viruses instead? (Score:2, Informative)
This doesn't really say what's actually being used by spammers, but it's a sign of improvement. At the least, it narrows the pool of available relays. Continuing progress will increase the spam pressure on those remaining, which will i
In case of /.'ing (Score:4, Informative)
By Evan Harris
Copyright 2003, all rights reserved.
Introduction
This paper proposes a new and currently very effective method of enhancing the abilities of mail systems to limit the amount of spam that they recieve and deliver to their users. For the purposes of this paper, we will call this new method "Greylisting". The reason for choosing this name should become obvious as we progress.
Greylisting has been designed from the start to satisfy certain criteria:
1. Have minimal impact on users
2. Limit spammers ability to circumvent the blocking
3. Require minimal maintenance at both the user and administrator level
User-level spam blocking, while somewhat effective has a few key drawbacks that make its use in the continuing spam war undesirable. A few of these are:
1. It provides no notice to the senders of legitimate email that is falsely identified as spam.
2. It places most of the costs of processing the spam on the receivers side rather than the spammers side.
3. It provides no real disincentive to spammers to stop wasting our time and resources.
As a result, Greylisting is designed to be implemented at the MTA level, where we can cause the spammers the most amount of grief.
For the purposes of evaluating and testing Greylisting, an example implementation has been written of a filter that runs at the MTA (Message Transfer Agent) level. The source for this example implementation is available as a link below, and as other implementations or additional utility code become available, they will also be linked.
Greylisting has been tested on a few small scale mail hosts (less than 100 users, though with a fairly diverse set of senders from all over the world, and volumes over 10,000 email attempts a day), however it is designed to be scalable, as well as low impact to both administrators and users, and should be acceptable for use on a wide range of systems, including those of very large scale. Of course, performance issues are very dependent on implementation details.
The Greylisting method proposed in this paper is a complimentary method to other existing and yet-to-be-designed spam control systems, and is not intended as a replacement for those other methods. In fact, it is expected that spammers will eventually try to minimise the effectiveness of this method of blocking, and Greylisting is designed to limit options available to the spammer when attempting to do so.
The great thing about Greylisting is that the only methods of circumventing it will only make other spam control techniques just that much more effective (primarily DNS and other methods of blacklisting based on IP address) even after this adaptation by the spammers has occurred.
The Greylisting Method
High Level Overview
Greylisting got it's name because it is kind of a cross between black- and white-listing, with mostly automatic maintenance. A key element of the Greylisting method is this automatic maintenance.
The Greylisting method is very simple. It only looks at three pieces of information (which we will refer to as a "triplet" from now on) about any particular mail delivery attempt:
1. The IP address of the host attempting the delivery
2. The envelope sender address
3. The envelope recipient address
From this, we now have a unique triplet for identifying a mail "relationship". With this data, we simply follow a basic rule, which is:
If we have never seen this triplet before, then refuse this delivery and any others that may come within a certain period of time with a temporary failure.
Since SMTP is considered an unreliable transport, the possibility of temporary failures is built into the core spec (see RFC 821). As such, any well behaved message transfer agent (MTA) should attempt retries if given an appropriate temporary failure code for a delivery attempt (see below for discussion of issues concerning non-conforming MTA's)
Re:In case of /.'ing (Score:2)
Tempfailing is not new and unique (Score:5, Informative)
First I heard of it was from Landon Noll and Mel Pleasant. It is noted in brief as one of the techniques in this plan to end spam [templetons.com] (though their plan, which did include the triplets, is not laid out in full there.)
It is a worthwhile technique for a little while, and if spammers were rational, would be worthwhile for some time to come. But spammers are not rational, and already this technique is not as useful as would be hoped.
Do a Google Search for Tempfailing [google.com] especially in ASRG to see statistics etc.
I am not sure what the spam filter is (Score:2)
Re:I am not sure what the spam filter is (Score:2)
I don't understand what everyone's problem is regarding spam. It is a nonissue for me.
Maybe SpamAssassin is just so good that I don't notice how annoyed I might be otherwise.
Easy way to stop spam... (Score:3, Informative)
I haven't received 1 single spam in recent months from doing this!
Re:Easy way to stop spam... (Score:3, Informative)
You can, however, establish classes of emails. Most people don't like this however, because you have to check multiple accounts, and it really doesn't stop the spam.
In order to sign up to certain services / sites you need to provide a valid email address.
While that email address can be a secondary email, if anything important is going to come in on the email (such as domain information via network solutions) you will still want to use your rea
anyone@domain (Score:3, Informative)
Oddly enough, I've been totaly promiscuous with these throw-away email addresses, but I've only got one SPAM from a company I actualy bought stuff for. So far no one has sold any of the address, and all the websites I've posted too either arn't scanned or are prote
Easy for end-users, sure. (Score:5, Insightful)
Look at it this way: you can stop crank calls by unlisting your phone numbers. But you can't unlist the hospital, the ambulance service, the fire department, etc.
We're not all end-users. Some of us are the plumbers.
clever hack for WHOIS contact addresses (Score:5, Interesting)
1 false positive is not acceptable. (Score:3, Insightful)
"it practically eliminates the main problem of other solutions: the false-positive."
What does 'practically eliminates' mean? If it gives false positives at all, it is just as useless as all those 'other solutions'.
Re:1 false positive is not acceptable. (Score:5, Interesting)
At USENIX '03 there was a paper presented on artificial intelligence techniques for spam detection. I can't provide a link since only USENIX members can download the paper (at this point, at least). I was a coauthor of that paper.
One of the things we've discovered in our research is that some classes of filters (most notably, the one I have been developing along with a few other individuals) are actually more effective at correctly classifying email than humans are. That is to say, you can train the learning algorithm on mostly-correctly-classified data, then re-run it over the training data, and almost miraculously, it discovers all kinds of email in the training set that was incorrectly classified.
I.e., this filter has discovered mail that I myself incorrectly thought was spam. It's scary, because there's a lot of it.
To assume that a human will always be 100% accurate at classifying their own email isn't just arrogant, it's plain wrong. Newer filters that will be introduced in the near future might possibly be more accurate than you, a frail human, could ever be.
Re:1 false positive is not acceptable. (Score:4, Insightful)
I get these chain emails from my brother. They are always some funky scheme to get money that won't work. I'd love to just delete them...but if I do this, he tells my mom I don't answer his email.
She then laces into me like you would not believe...blah blah blah he's your brother and you should love him. I don't need that grief...so instead I respond with a "not interested, no cash right now." Keeps the family happy.
I could see it being more important than this, though. Your boss sends you direct mail HE received and appends a "Should we do this" to the bottom. Or, worse, your marketting team constructs a direct mailing that fails your spam filter (no comments from the peanut gallery...obviously this is a good thing to find out, but this is not the way to find it out). Missing that one email could make somebody VERY angry and put you in danger. I have had messages from my boss/CEO/etc go into my junk folder and found them when cleaning it out.
It is correct for the spam engine to label these as spam email. It would be incorrect for it to delete them before they got to you. And so I subscribe to the school of thought that a single false positive makes any spam filter absolutely worthless. It is very easy to delete a message that gets through the filter. It is impossible to resurrect a mailing you never even knew you got.
Re:Reference for that paper (Score:3, Informative)
Time critical (Score:5, Insightful)
This would be great for personal mail, but that's about it. ISPs would have the same problems with it because their business-class users most likely use the same servers as their consumer-class users.
Re:Time critical (Score:5, Informative)
Re:Time critical (Score:2)
That would be mostly fixed by only imposing the delay on mail received from networks listed on blocklists such as the SBL [spamhaus.org] or SPEWS [spews.org]. Blocklists are just databases of IP ranges, they can be used for non-blocking purposes. Hopefully most of your business contacts use decent ISPs that don't harbor spammers (and if not, the delay would be a nice incentive for them to switch to a decent ISP that is friendlier than outright blocking).
Re:Time critical (Score:2)
True, If not that the delay would be a nice reason for the company to lay off the administrator implementing the rules, for a even cheaper administrator.
Re:Time critical (Score:3, Insightful)
Besides, if you're using SMTP for time-critical things, you have a problem, as SMTP is NOT a guarenteed delivery system.
Re:Time critical (Score:3, Insightful)
Oh, I've seen it, and I warn people against it all the time.
But, hey, most companies are schizo when it comes to IT. You wouldn't let your accounts recieveable rely on random people who may or may not pass on your records unchanged and unread; so why do you trust your business communications to SMTP email?
Re:Time critical (Score:3, Insightful)
Private instant messaging, IMHO, is much better for "time-critical" communication. Of course, it depends on what type of data you are sending and w
Re:Time critical (Score:4, Insightful)
some guy telling you to BUY THIS NOW != time critical.
your wife telling you to BUY THIS NOW == time critical, and in theory, your wife == whitelisted (or blacklisted, depending on personal preference).
Re:Time critical (Score:2)
spam.....hrmmm (Score:5, Insightful)
isn't it time to change the specification (RFC) and possibly the manner in which our current system works? i haven't come up with anything yet, but surely there must be some sort of handshaking/secure type connection that could be used - - some sort of postage (free) that is encrypted into the mail, that states that it is genuine....kind of like the hologram on those windows cds...
i dunno. file this story under redundant.
How about Habeas' haiku method? (Score:4, Interesting)
Re:spam.....hrmmm (Score:2)
It wouldn't an end all solution. I envision multiple strategies working in tandem, such as the above public/private key, a registered email sender verification service (live Verisign.. *shudder*), and even completely unsecure (for those pe
I'm not sure about this... (Score:3, Insightful)
I thought it was generally understood that most spam was sent by abusing open relays, thus hiding it's origin. This could be wrong. However if it's not, those figures aren't appllicable. Nor is spam going to be diverted since an open relay is generally running a regular mta and will attempt a retry. For instance, if qmail were running on an open relay and was abused by a spammer it would try again and again with an increasing delay (calculated logarithmically if memory serves) between attempts. So the mail will still get through.
When you further consider that if a spammer hits an open relay and hammers your mailserver from it and all of the "triplet's" are new, you're increasing your traffic, because all of that mail will be attempted again.
Bayesian Filtering (Score:3, Interesting)
How does the effectiveness of Greylisting compare with what others are seeing with existing techniques (such as Bayesian filtering)? Is it a false positives problem, such as digests and opt-in mailing lists getting incorrectly tagged as spam?
Re:Bayesian Filtering (Score:2)
Re:Bayesian Filtering (Score:2)
Re:Bayesian Filtering (Score:4, Insightful)
If people start adopting anti-spam technologies we would reduce the return spammers get from sending spam. Reduce this enough and the spamming business will no longer be profitable.
POPFile is great. I've also used SAProxy (http://saproxy.bloomba.com/) under windows and it works great too.
Again, the idea is not to eliminate all spam, but to reduce the return rate, and therefore the money made by spammers.
Re:Bayesian Filtering (Score:3)
Mod parent up!
There is a "Let's replace SMTP to stop spammers!" meme floating around. I haven't seen a single example of a new SMTP protocol that will actually stop spamming - they only make it marginally harder. Replace SMTP and you've gone to a whole lot of effort, and the spammers will still find a way to spam.
People shouldn't dismiss client-side filtering on the grounds that spammers are still wasting our resources. That's a temporary situation! Right now most people don't have good client-side filt
I have my own algorithm (Score:2, Insightful)
Any email with HTML in it, any email with
It never gets to my mail program.
I could also filter on subject lines containing any word whi isn't in thdictionary but since some of my friends don't spell too well...
Re:I have my own algorithm (Score:2, Insightful)
discussion about stopping spam, and someone in
the discussion says "Well, I filter out any
message with the words viagra or penis..."?
Does that get flagged as spam and discarded too?
Copy of spam logged? (Score:2, Insightful)
spammesilly@gt.rr.com (Score:2)
Not shit, mailto:spammesilly@gt.rr.com [mailto]
Send it to me baby!
I'm teaching my PC how to deal with it and I've been on a mission to sign up for every bullshit mailing list I can find, all the typical trash that pesters people to death.
Once I get this worked out I'll install it on my dad's PC and all my friends too. Then they can say bye-bye to spam.
I told my friend, "Your shit works because my shit is always broke!".. In other words, I am the guinna pig for everyone else. I test it then they
RFC 3514 (Score:5, Funny)
Filtering out spam (Score:2)
Published a paper? (Score:4, Informative)
For others looking for a solution, try POPFile [sourceforge.net]. Open source, cross platform, gives me 96% accuracy.
One more thing: "practically eliminates" is not the same as "eliminates".
Re:Published a paper? (Score:2)
--Dan
Re:Published a paper? (Score:5, Insightful)
Yes, I realize that for "serious" science still expect things to be published in peer reviewed journals, but in most cases I can't help but think that getting the article out there would be more useful. Sure, peer review is important, and somewhere to look for some kind of verification of the value of a paper is useful. But I much prefer the Research Index [researchindex.com] way, where I can get a good indication of the value of a paper by looking at how many people have cited a paper and WHO have cited a paper.
Anyway, pretending that putting up a document on a website is somehow less publishing a paper than having it printed in a journal, is just plain elitist. You should propably be a bit more critical to papers that are published that you don't know have been through a proper review, especially if you're not a domain expert yourself, but being aware of the source is something that you always need to be.
Re:Published a paper? (Score:3, Insightful)
Waiting for Article Title (Score:5, Funny)
In many countries SPAM is illegal and... (Score:2)
Re:In many countries SPAM is illegal and... (Score:2)
Wouldn't it be nice if (Score:2, Informative)
I wish your plan would work but I just don't think it will.
Plus the spammers can get their viagra at wholesale cost!
Delaying email by one hour! (Score:5, Insightful)
I'm wondering how I'm going to explain that to a new customer over the phone who says "I'll just email that file right now so we can go over it together".
Re:Delaying email by one hour! (Score:5, Insightful)
I think changing that perception of e-mail as near instant will be incredibly hard. And if you succeed it will just move even more traffic over to the IM networks and cause spamming of IM networks to escalate instead.
Re:Delaying email by one hour! (Score:4, Insightful)
Even if FTP were a solution, it does nothing to answer a new customer who says "I just heard about you and I'm excited about your products. Wanted to call and ask you some questions. I sent an email about 10 minutes ago with an outline of the project we're doing were you guys could really help out, have you had a chance to look it over yet".
There's a limitless number of these important common customer relationship scenarios, where the expectation of all parties involved is that email is delivered in under 1 minute and typically 5-10 seconds. And there are an infinite number of scenarios other than sales and customer service/relations where people quite reasonably expect email to be delivered in seconds.
Focusing on using FTP isn't just the wrong answer, it's not even an answer at all to the problem of email delivery taking an order of magnitude longer than users expect and depend upon.
But as others have pointed out, most users don't have access to FTP servers to receive files. Most corporate firewalls would prohibit users from setting up a FTP server. I would guess that almost any employee behind a corporate firewall wanting to somehow receive a file from a new customer via FTP who attempted to ask a sysadmin would get the answer "just have them send it as an attachment". FTP is simply not a viable protocol for customers and salespeople (or most others) to use to pass files back and forth.
Aside from not solving the unacceptable delay and the inappropriateness of using FTP, there is the problem of bad attitude. Specifically:
Where did "new customer" turn into "user". The word "user" in this context is often spoken in the tone of an overworked, grumpy sysadmin who's personal view of his priorities are decoupled from the larger organization's mission (usually taking care of customers, selling products, operating efficiently, and so on).
In this particular example, what is important is that the new customer whats to talk with someone about solving his problems. That someone is me, and I want to impress him, sell him something that will truely meet his needs, and hopefully turn him from "new customer" into "repeat customer" or even "loyal customer". THAT is what is important, and getting the customer's file quickly and easily with minimal hassle is merely a tool that enables the truely important work to happen.
Not having the email for 1 hour means I'll either have to call him back in an hour, while he probably calls some competitors and shops around. Often times people will buy from the first friendly, knowledgable person who goes to some effort to help them.... searching until they find that person/company. Delaying response to a new customer by 1 hours would put me at a competitive disadvantage.
Or we'll have to proceed without it (FTP is not an option), leading to frustration as he explains material that would have been much better delivered as a file. Maybe it would go ok, maybe not. But it's starting the whole process "on the wrong foot".
Then again, if your business is being a grumpy sysadmin where you have (captive) "users" rather than "customers", maybe delaying new email conversations is a big advantage which is not offset by any impact in "responsiveness" because it's already intentionally low.
Re:Delaying email by one hour! (Score:3, Insightful)
Just because themajority of people does something incorrectly doesn't mean it's suddenly the correct way to do it.
One good point about this proposal (Score:5, Insightful)
The real solution (Score:5, Funny)
Bogofilter does pretty well for a client filter (Score:4, Interesting)
I'm currently using Bogofilter [sourceforge.net] (and looking into CRM114 [sourceforge.net]) and getting better than 99% accuracy (about 1 in 200 false negatives at the moment) and very very few false positives (maybe 2 in 5000 messages).
Of course these are MUA level filters (and yes, I know, I've already "paid" with bandwidth to download the spam) - however since the proposed "greylister" would have to be installed as the MTA at major ISPs (as the authors note) I'm not convinced that is more likely to get widespread adoption than the various sorts of adaptive client-based filtering now available, particularly as it requires a database to back the method up.
As far as I am concerned the major factor in a spam filter should be zero false positives - personally I don't mind reviewing one or two spams a week but I get really annoyed if I were to lose a real message (note the two false positives I have sent to date with bogofilter contained forwarded sales pitches along with a message).
co-evolution (Score:4, Insightful)
Spam guards and spam co-evolve. Since greylisting is easy to get around by spammers, if it becomes widespread, spammers will take measures to avoid it, and the net result will be a lot of extra traffic.
In fact, the impact of this kind of system on mail could be pretty bad if widely adopted: large amounts of mail may end up being held up in delivering servers, and "informative" messages sent by helpful mail systems (about "temporary failures") may end up creating more junk mail than they avoid.
97%? not impressive. It's POPfile for me (Score:4, Informative)
I use it experimentally for general mail classification (business/personal/a variety of mailing lists etc., all in all 7 buckets) on my home machine, and it works fine in these conditions too, although the accuracy is a bit lower (around 95%).
missed the point (Score:4, Informative)
I've seen some comments that say (paraphrasing) "For real SPAM filtering use <POPFile|Spamassassin|...>", but these missed the point (or perhaps didn't read the paper?). This method is a "first-pass" filter, getting rid of e-mails for which no redelivery attempt will be made. The second-pass filter should still be implemented for everything that gets through the first pass. From the paper:
"The Greylisting method proposed in this paper is a complimentary method to other existing and yet-to-be-designed spam control systems, and is not intended as a replacement for those other methods. In fact, it is expected that spammers will eventually try to minimise the effectiveness of this method of blocking, and Greylisting is designed to limit options available to the spammer when attempting to do so."
EOL SMTP (Score:3, Funny)
What? Where is it? Oh, I'm still working on it. You can send and receive, but buddy lists are not implemented yet.
How about DSPAM ? (Score:3, Interesting)
I know this is happening as some complete bastard seems to be doing this using my domain as a "From:" address (well, [random-word]@schmerg.com), meaning that I'm been getting about 30 or 40 bounce messages a day for the last 2 or 3 months now. And although the odd sending IP is repeated, mostly they're all from different IP addresses. And of course I'm getting perfectly valid looking bounce messages from perfectly reasonable companies (and only a couple of abusive replies so far).
Now the problem is that the email is being uploaded to thier (non-open-relay) ISP's mailserver that will retry properly, and anyone else looking at the IP address will see a perfectly reasonable IP (the spammer seems to gave infected a lot of AT+T customers, ComCast customers, etc.). So short of blocking spam on subject, this spam is harder to prevent in the first place.
I've semi-automated a process to report the infected machines (that provoked a bounce message) to the appropriate ISP, and seem to havign some success in getting the ISPs to contact their customers, but I think this new form of spamming using a distributed attack will be particularly hard to block.
Anyone with a great idea (or who knows more about this scheme, or the identity of the twat behind it) I'd love to hear from you...
--
T
SpamAssassin (Score:4, Insightful)
SpamAssassin can also be used at the MTA-level, and while this tool might be an interesting test to integrate with SA, its claims that other systems cannot feed back to the sender that their mail has been blocked is flat-out wrong.
Most people do not do this because you are almost certainly getting this mail through a relay, and that relay is going to get the SMTP temporary error and try to send a warning to the user who sent it. Spammers regularly slam my home mail server by using my address as the "From" in an entire batch of spam. It's pretty seriously annoying to get that deluge of junk, and it's not really necessary. If your spam system just identifies spam and lets the user (or sysadmin) decide how to deal with it based on how "spamish" it is, you get a much more reasonable behavior.
I junk thousands of pieces of spam every week, and I *never* junk valid mail. Yes, I do have some spam in my inbox. Most of it is tagged as potential spam, and I delete that after cursory inspection of the from addresses. Some of it is missed, and the overhead that I suffer having to identify that myself is amazingly low compared to not being able to read my mail prior to SA.
Check out SA. The latest version is pretty impressive, and if this "new" technique (I don't think the idea of tracking connection quality is very new, it's certainly done in SA to some extent) turns out to be useful... well SA works on much the same principal as Perl: There's More Than One Way To Do It. Bayes, Blacklists, Whitelists, Obfuscation detection, Checksum trackers, you name it, SA uses it. None of these techniques gets to say "this is spam", they all just get to poke a message in the direction of being spam or non-spam. This leads to something far more reliable than any one techniqe.
HOW SPAMMERS WILL DEFEAT THIS: (Score:4, Informative)
However, here is how the spammers will adapt:
MailBlast 2.0 will send each mail TWICE. The first time to get the triplet on the greylist, the second time to actually send it. Or a little more sophisticated, it will run in one-hour blocks, and at the end of the hour, re-run the previous hours emails. No queue or other real-SMTP server functionality necessary.
Wild idea, maybe ISP' should do something at (Score:3, Insightful)
Like, say, putting a maximum limit on the number and size of e-mails that can be sent out a day.
Gather studies on how many people send 1 to n e-mails a day, and how many people send out e-mails of 1 byte to n bytes in size.
My guess is, it's a pretty distorted curve, with maybe a few thousand people -- of all those online -- sending out millions of e-mails a day. The maximum most "normal" people will send a day is probably 100 (and that's a large over-estimate).
My Anti-Spam Idea (Score:3, Interesting)
I've submitted it to Slashdot. They rejected it. Tell me what you think. I'd like reactive approaches to get discussed a bit more. If you do too, submit this to Slashdot
Anti-Spam Techniques: Honeypot spam detection! (Score:5, Informative)
The problem with the greylist method is it still slows down mail service, and potentially more than the relay blacklist features. The objective here is that end-user/networks should not be penalized in the fight against spam. We already waste too many resources, and according to my latest mail server stats, more than 65% of our inbound mail is UCE. I'm fed up with more than half my e-mail bandwidth being crap my users didn't request so more resource allocation on a local level in the fight against spam is counterproductive!
Here's a very clever, much more practical method I cound recently.
A company is Canada has set up what it calls SORBS [sorbs.net]: Spam and Open Relay Blocking System.
What's different from their blacklist is that they maintain "honeypots" strategically located around the Internet. These are servers they specifically set up as inbound mail relays, but never for legitimate purposes. If the servers get [select] mail activity, it's assumed to not be legitimate and it flags the source as a potential spammer... it makes a lot of sense. You create a domain name, but don't promote it in any legitimate manner, and/or you seed spam lists with these e-mail addresses and then let the spammers send to your key systems around the internet and *bam*, they're identified in real time, and then added to a blacklist.
I really like this idea. Like any other system, it has the potential for abuse but the beauty is the identity of the honeypot systems is kept secret, so it's very difficult for anyone other than spammers to exploit the network.
Re:My spam filtering (Score:3, Funny)
Re:Sarxpam (Score:3, Funny)