Novel Method for Universal Email Authentication

MKaplan writes "Most spam is sent using spoofed domains. Email authentication schemes such as SPF attempt to foil spoofing by having domain administrators publish a list of their approved outgoing mail servers. SPF is sharply limited by incomplete domain participation and failure to authenticate forwarded email. A paper describes a novel method to rapidly generate a near-perfect global SPF database independent of the participation of domain administrators. A single email from an unauthenticated domain is bounced and then resent — this previously unauthenticated domain and the server listed in the return path of the resent bounce are entered into a globally accessible database. All future emails sent from this domain via this server will be authenticated after checking this new database. Mechanisms to authenticate forwarded email and to nullify subversion of this anti-spam system are also described."

Greylisting?

[mmcuh (1088773)]

Isn't this the same thing as greylisting [wikipedia.org]?

Re:Greylisting?

[by Anonymous Coward]

No, not at all. If you don't want to read the article, just keep guessing how it works, and we'll let you know if you are getting warm.

Re:

[flydpnkrtn (114575)]

You've been here a while, hmm? Next you'll be talking about Soviet Russia

Not exactly. I think.

[khasim (1285)]

He's talking about "bouncing" messages ... but I cannot tell if he means resending an accepted message or denying it at SMTP time.

Then he talks about having people install software:

Auto-Resend software will ensure that almost no one will see or be required to manually respond to the email seen in Figure 2. Auto-Resend software is a simple onetime update for webmail systems, email clients, and local mail servers.

Yeah, installing new software is a great solution.

That's the problem.

[khasim (1285)]

He does not CLEARLY explain what he is intending.

I believe he means denying at SMTP time, so the sender will try again after X minutes.

Which is kind of like greylisting. The FIRST problem is that the spammers have adapted to this and retry.

The SECOND problem with this is he's saying:

Unique sub-addresses are dispatched in the 'From' field with routine outgoing email. [e-mail] may send RIAuser^[e-mail] to one individual and RIAuser^[e-mail] to another individual.

Huh? So this is also about SENDING email?

Now if you could bounce the message, it would just go back to the original IP, so I don't see why that would help either though.

And it doesn't address the issue of "fast flux" where the domains are "legit" in that they exist and point to the IP address of the sending machine ... for a few minutes.

So he's talking about "bouncing" messages ... installing new software ... and altering the "From:" addresses on stuff YOU send ...

No fucking way is this going to work.

Re:

[SCHecklerX (229973)]

Which is kind of like greylisting. The FIRST problem is that the spammers have adapted to this and retry.

This is exactly why greylisting is effective. It pushes the cost of spamming back on the spammers. Now they have to have a semi-legitimate mail relay, vs. fire and forget. If everyone greylisted, then the spammer's mail queues would be huge.

Of course, all bets are off with zombies that start using legitimate SMTP servers, but there are solutions to that already in place:

1. Many ISPs volunteer their list

So?

[khasim (1285)]

This is exactly why greylisting is effective. It pushes the cost of spamming back on the spammers. Now they have to have a semi-legitimate mail relay, vs. fire and forget. If everyone greylisted, then the spammer's mail queues would be huge.

So? They don't care. They have, effectively, limitless bandwidth and limitless processor power.

Greylisting WAS effective ... before so many people adopted it. Now it only catches the dumbest spammers.

The only place this fails is if the spammers as part of their owning of zombie hosts begin to check for the proper SMTP server to relay through and configure accordingly. Admittedly, this is not too difficult to do, but they aren't doing it yet.

No. It fails when they implement (as they have) a process to resend any temp rejections after X minutes.

Greylisting had THREE features:
#1. It could temp reject spam and if the spammer never tried again ... success.

#2. It could temp reject spam and if the spammer randomized the "From:" username/domain ... success.

#3. It could temp reject spam and if the IP addresses was listed in a blacklist within the temp reject time frame ... success.

Now all that is left is #3. It costs the spammers NOTHING to upgrade the zombies. And if they get the spam through, the spammer wins.

Now, the zombie can appear MORE legit than a lot of the real mail servers out there.

Greylisting is effective for me

[baileydau (1037622)]

Do you have any data on the exact date or extent to which greylisting became ineffective?

I don't know about the GP, but for me greylisting is very effective. I have a personal domain for my wife and myself. I have a catchall mail address.

Here are some stats for part of last week:

Start Date 23/09/07 04:02
End Date 28/09/07 17:00
                5.54 days

Total spam: 4624
Spam blocked with greylisting: 4478 (96.8%)

spam via backup MX: 69 (1.5%)
spam retried (got past greylisting): 77 (1.7%)

Total through to end user: 146
Identified as spam (SpamAssassin): 123 (84.2%)

backup MX marked as spam: 50 (72.5%)
direct marked as spam: 72 (93.5%)

Total to end user not marked as spam: 23 (0.5%)

NB. Up until about a month ago, ~25% of SPAM came via my backup MX, which doesn't have greylisting. I don't know why it dropped, but I'm happy it did.

Re:

[afabbro (33948)]

He does not CLEARLY explain what he is intending.
The SECOND problem with this is he's saying:
Huh? So this is also about SENDING email?

Ah, I'd wondered where Robert McElwaine had gone...

Re:

[FlyveHest (105693)]

I believe he means denying at SMTP time, so the sender will try again after X minutes.

Which is kind of like greylisting. The FIRST problem is that the spammers have adapted to this and retry.

Huh? When I take a look at how many mails are bounced on all my domains, thanks to greylisting, each day, and hold it against how much spam actually enters my mailbox, i'd say they haven't adapted at all.

When you are sending millions of mails, retrying is far, far more expensive than just ignoring it.

Random 450s

[flyingfsck (986395)]

Instead of greylisting, I have experimented with a system where my SMTP receiver would send '450 try again later' messages randomly to incoming connection attempts. It actually works. Since the vast majority of incoming mail is spam, a 50% rejection rate reduces spam by almost 50% since most spammers will not retry, while legitimate mailers will.

In the end, I settled for an even simpler approach, where I just throttle the receiver dramatically, so that it takes about 10 seconds to receive a message. Most

Re:

[Bogtha (906264)]

It seems to be greylisting, except instead of rejecting the message during delivery and relying on standard SMTP features, he wants to accept the message, send a bounce, have the other party install software to automatically re-send the message upon receipt of the bounce, and then add the sender's mail server to a whitelist the second time the email comes through. Awful idea for all different kinds of reasons.

Re:Greylisting?

[tacocat (527354)]

I don't know, I didn't get that far. The article and the concept is bullshit.

The 'From' field is the keystone of their identification process. Well, I got news for you if you bothered to read the RFC. 'From' does not have to represent the real sender. I can forge it up all I want into anything I want and you can't tell. I didn't get past section 3 where this is before I determined the rest isn't worth reading.

Once again we have another company trying to come up the next Big Thing and they don't know what the hell they are talking about. SPF is cute -- but relies too much on people setting it up and correctly. I suppose you could pay a service to act as a third party validator, but that's turning into a boondoggle too.

I don't think bouncing email at valid senders is going to win any friends.

Perhaps there is a way to do it successfully and with great accuracy. I would love to say I'm working on it. But quite frankly, if I do figure it out I probably won't mention to anyone since I really don't want the legal hassle of trying to defend my idea against someone else's billions. I can block spam. I can block spam to the tune of 99+%. The rest is trivial. I was even surprised to hear them say 94% was the average. Perhaps people would be better off if they stopped using SpamAssassin.

Sorry, my opinion is that statistical filtering is more than sufficient if it's managed well. I think few people are willing to do the work required of them to make them spam free. Kind of like locking the door to keep out the crooks.

Re:Greylisting?

[MightyMartian (840721)]

How many times have we heard the "this will fix Spam real good" claim? First it was "close those open relays, ye bastards", and lo, that worked for about a week. Then it was "Well, we'll just keep these black lists, and that'll fix things", until of course the complexity of maintaining such lists and the harsh consequences for any poor bastard who somehow found himself the victim of a false positive tried to get himself off said lists. Then there was "We'll just tarpit consumer IPs based upon some nifty string-matching" and the matching "we'll check reverse IPs, and if they don't match, fuck ya!" which of course buggered up all those poor guys using their cable and DSL connections to run small personal mail servers, or anyone with a retarded or miserable provider who refused to alter reverse DNS entries. Then there was "Hey, you don't have an MX record for that IP, so down the shitter ye go!", which nailed anyone who might be sending from sort of a proxy, and didn't want their actual mail servers advertised as such so that they didn't become victims of joe jobs and distributed dictionary attacks. Then there came greylisting, which actually worked for a while, but seriously screwed with "immediate delivery" that all those in the post UUCP world had become accustomed to with email, not to mention the smart spammers learning from the trick and just retrying. SPF was then heralded as the end-all and be-all, but of course has its own problems (particularly with message forwarding, which requires rewriting the header), not to mention that everyone came into compliance with neutral records, so at least the big guys wouldn't jettison mail from their server due to lack of an SPF record.

At the end of the day, you're right. Statistical filtering, with the careful use of all of the above solutions (though I think whitelists/blacklists are as bad as the problem they attempt to solve) is the only way to reliably filter spam. You're never going to catch it all, but the ISP I worked at was catching, by my estimate, about 90% to 95%, which meant that a guy getting about fifty spam a day was down to three or four, and in many cases less than that. It does mean work, there's no solution that doesn't require monitoring, management and tweaking, because the spammers are smart bastards who learn the tricks as fast we can come up with them.

That's already implemented with Spamcop

[no-body (127863)]

Mail servers are authenticated by Spamcop and forward spam automatically to Spamcop which adds it to their database. When using reject_rbl_client bl.spamcop.net SPAM is blocked.
Works like a charm!

Re:

[no-body (127863)]

&& that's IP based, not domain name based, so the SPAM originating IP is known and can be blocked

Re:

[John Hasler (414242)]

My ISP uses it. It frequently bounces my Debian mail. I'm moving my mail to Newsguy where I can turn the damn RBLs off and filter my mail myself.

Fails to account for SMTP farms...

[pathological liar (659969)]

So what happens when you receive an email from a big site like Sympatico, Hotmail, or any number of other places that have farms of SMTP servers, where your message isn't guaranteed to be resent from the same IP?

This also requires users to install software to use effectively, and features CAPTCHAs which are a usability nightmare and not nearly as impregnable as the author thinks.

All that effort instead of just adding a TXT record to their domains.

Re:Fails to account for SMTP farms...

[bennomatic (691188)]

So what happens when you receive an email from a big site like Sympatico, Hotmail, or any number of other places that have farms of SMTP servers, where your message isn't guaranteed to be resent from the same IP?

And OKing the receipt of any address at a domain from such an infrastructure seems less than ideal. I mean, if I send out all my email for "[e-mail]" from Hotmail's SMTP servers, I'm not sure I want that to automatically give the go-ahead so that anyone can send spam from "[e-mail]" and "[e-mail]", etc..., from those domains.

SPF, as I understand it, has some contexts in which it works well. But it doesn't cut with fine-enough a blade as far as I'm concerned. Automating the process so that I (if I haven't set up SPF records) could allow spammers to use my domain with more authority by responding to an automated message just doesn't sound like a good idea. I think this opens up the door for a lot more spam if people believe in it.

If it went a step further and tried to authenticate each time a unique USER@DOMAIN pair sent an email via a particular host, I could see that being useful. The protocol could be extended such that even the SMTP farms could conceivably use something to say, "if authorized at one of my servers, an email should be authorized at all of my servers". But it's a lot of work to get there, and the size of such a universal database would be ridiculous, and it seems that for there to be a single-source host for such a thing, there would have to be a lot of cooperation between some major corp^H^H^H^H sources of funding.

Re:

[MightyMartian (840721)]

Let's just try to imagine the resources required for this sort of a setup in the case of a distributed dictionary attack. The ISP I used to work at, which was small and had about a thousand email addresses, was, on average, getting nailed with about 500,000 such attacks per day (and with some days being double that or more). In fact, it got so bad that the crappy IMail server I was forced to use because it ran under Windows would actually become non-responsive. Putting in two old Pentium-233s with Linux

FUSSP

[Just some bastard (1113513)]

Basically this guy is proposing an automated whitelist (for domains without SPF records) via a local database. At least I think what the paper is about, I gave up reading it earlier. It lacks a concise summary, doesn't read like a well researched paper and the diagrams don't even display without javascript.

The author may be an anti-spam kook [rhyolite.com] but the paper is so badly written I can't be bothered identifying which.

Cue form response

[bperkins (12056)]

I'm surprised I don't see it.

Re:Cue form response

[Epsillon (608775)]

Your post advocates a

(*) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
(*) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
(*) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
(*) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
(*) Open relays in foreign countries
(*) Features in MTA software that can be disabled, such as MDNs
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
(*) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
(*) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
(*) Dishonesty on the part of spammers themselves
(*) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:
(*) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
(*) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(*) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!

I didn't spend too much time looking through the options, so go easy if I got it wrong. Will that do?

oldie but goodie.

[djdavetrouble (442175)]

As soon as I saw a spam Item, I just skipped forward to this good old reliable post.
Funny because its true !
I really love this one...

No, I didn't RTFA..

[by Anonymous Coward]

...but this had to be posted.

Your post advocates a

(X) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
(X) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
(X) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
(X) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

(X) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
(X) Blacklists suck
(X) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
(X) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(X) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!

Re:

[MarsDefenseMinister (738128)]

That's the point of SPF. It detects when the header is fabricated, and the virus detector bounce messages can just be dumped.

Most of the million spams a month I see are from Norton or some other idiot company telling me that the message I sent to them contained a virus. No, if they'd just had the data (but they already do! I use SPF) they could see that my server has one valid mailing IP address, and it's not on a Korean DSL line.

Re:No, I didn't RTFA..

[ScrewMaster (602015)]

Which just continues to show that all sophisticated security systems can and will be defeated by morons. There is no force on the planet more powerful than human stupidity.

Re:

[nuzak (959558)]

Your reply indicates an attitude of:

[ ] "My approach is immune from all criticisms"
[X] "Doing SOMETHING is better than nothing!"
[ ] Willfull ignorance of founded criticism.

Yes, it's a worn out joke (and yes, the form is a JOKE, it applies to ALL current antispam approaches). Yes, moderators are stupid. You must be new here.

Re:

[Matt Perry (793115)]

Your reply indicates an attitude of ... "Doing SOMETHING is better than nothing!"

That's somewhat true. You have to form a hypothesis and see where it leads if you are going to solve a problem. I doubt that anyone is going to come out of the woodwork and say "I have the solution" and be correct. We have to try many different things to see what works and what doesn't. Dismissing things out of hand isn't going to help find a solution to the spam problem.

Re:

[plover (150551)]

What his joke is offering is the insight that every easy route (and most moderately complex routes) to blocking spam has already been tried and failed. Every "new" whiz-bang spam filtering idea these days is merely a rehash or mashup of previous filtering ideas that still retain the problems that plagued the original ideas. The method described in this paper is not novel -- it's a complex mashup of whitelists, CAPTCHAs, Bayesian filters, and new mail client software, each of which has been tried and has t

Major flaw in methodology

[Todd Knarr (15451)]

The proposed scheme ignores one thing: the majority of bounce messages today are false bounces caused by spammer joe-jobs, therefore they themselves get flagged as spam and deleted/ignored. In addition, it also increases the annoyance of greylist authentication schemes, since a spammer forging my address in the From field will cause every host participating in this scheme to send me a verification e-mail for a message I didn't send which I'll have to deal with. The proposed scheme makes a very fundamental mistake: assuming that you can trust the sender's address in a message to be the true sender's address. You can do that only after you've determined the message is authentic and not spam, at which point you don't need this scheme anymore.

Re:Major flaw in methodology

[Dan B. (20610)]

Not so, most of the backscatter is sent to [e-mail] which is either quietly dropped if you have smart filters that look for mailer-daemon@ etc as the sender, or passed to your 'no one by that name' catch all mailbox. Some mail systems will in fact be terribly misconfigured for backscatter, but then how is that different from what we have today?

The worst email storm I got was when some spammer decided to use my domain as the sender of all his junk and send all hi junk twice. I do have SPF entries in my DNS so ANYTHING that would encourage others to actually USE this system is a GOOD THING.

Now if there were just a few simple packages available that would give us the one-click (tm) ability to add SPF filtering to Sendmail/Postfix/Qmail/etc, and MS Exchange 5.5/2000, then I would guess that 50% or more of the domain spoofing spam would cease. That can only be good, as I only get UCE from real domains that I can't check for authenticity, from spammers who bother to follow RFCs and send twice after postgrey (greylist filtering) blocks them first time around.

I'm not sure you understand what SPF solves

[Degrees (220395)]

SPF only solves the problem of SpammerS sending mail to MailserverB with a forged header to make the message look like it came from MailserverA. The assumption is that UserB might open the message if it says it comes from UserA.

SPF causes MailserverB look up DNS data for the email domain for MailserverA, and compare it's SPF published IP addresses with the IP address of the incoming email connection from SpammerS. If the two don't match, then MailserverB hangs up on SpammerS with a 566 eat-shit-and-die er

The BIG issue

[Skiron (735617)]

Is MS windows boxes that are comprised and doing this - you can see this where the spam mails get 'chinese whispered from one box to another and end up incoherent (to say the least).

Any ISP should/could get suspicious of thousands of mails sent from one 'home user' source at anytime. But when you have thousands of 'users' doing the same thing, it gets lost in the noise.

One simple solution is:

if account == home user & running MS
      if mails sent > 10 per minute
          block it
      fi
fi

etc.

Very easy.

Re:

[crossmr (957846)]

I have a friend who works for a large ISP here in town and they do something like that but the thresholds are much higher. He told me a story about a woman who had been blocked multiple times but refused to clean the viruses off her computer but would call and bitch that she couldn't send any e-mail. I guess each time you trip the system and get blocked its a longer block. The last time she had called in he said it looked like she'd been blocked at least a dozen times based on the length of that block.

Participation in SPF

[by Anonymous Coward]

"SPF is sharply limited by incomplete domain participation"

That's not a big problem. 99% of non-participating domains fit in default SPF record "a/24 mx/24 ptr -all", we use it in qmail for few years. Together with Spamassassin it results in 99,8% antispam accuracy (warning: one big exception is yahoo.com, you should use domainkeys or add ptr:yahoo.com to default spf rule)

Check out existing discussion...

[enbody (472304)]

A Google search revealed this intelligent discussion of the scheme.
http://www1.ietf.org/mail-archive/web/asrg/current/msg12403.html [ietf.org]

"Office Live" link

[hey (83763)]

Whats with the "Office Live" link at the bottom of the article?!
Suggests a Microsoft-owned site.

Mis-read

[johnw (3725)]

Am I alone in having read that as "Novell Method for Universal EMail Authentication"? Might have been more interesting.

email has already been replaced

[Chapter80 (926879)]

The spam problems of email are causing people to migrate to trusted systems.

As I stood at a kiosk at a trade show this week, and waded through my spam-filled email on a few services (work email, hotmail, and gmail), the young woman at the kiosk next to me accessed her myspace and facebook accounts and responded to friends only.

She turned and said that only old people use email. And she was a VENDOR at the conference.... Things that make you go hmmmmmmmm......

Re:

[adrianmonk (890071)]

The spam problems of email are causing people to migrate to trusted systems.

As I stood at a kiosk at a trade show this week, and waded through my spam-filled email on a few services (work email, hotmail, and gmail), the young woman at the kiosk next to me accessed her myspace and facebook accounts and responded to friends only.

If you think myspace users don't get spam through myspace, you apparently haven't ever used myspace. And if you think myspace handles the spam that does exist well, you real

Bounces Won't Work

[maz2331 (1104901)]

Many if not most mail servers now drop messages to invalid recipients at SMTP time and don't send bounces any more. I've had to implement this on every mail server I set up to keep the mail queues from backing up to several thousand messages to invalid "bounce" addresses.

It would work if bounce messages were still sent.

Not SPF, and similar to what I use...

[argent (18001)]

This is just an additional layer over automatic whitelisting of addresses using tagged responses.

Some years ago I set up for my family a pretty simple set of procmail rules and scripts that bounced messages that hadn't otherwise been classified as spam or been whitelisted with requests that they be resent with a certain keyword in the subject line. For example:

"Hello, you just sent me the following message. Could you send me the message again with the word 'leisure' in the subject line? You can reply to this message if you like, just be sure to add 'leisure' to the subject line."

Over a period of several years the only spam that's gotten through this has been from a 419er.

The advantage of a subject line token like this is that you can tell people the token to use, or put the token in the subject line when you send the message so it's usually there when the recipient replies.

Whether you take the resulting message and whitelist the sender address, or some other information in the header that you consider reasonable, that's up to you. It's not really the same thing as the SPF database, though, even if you choose to make the same kind of information the key you use for whitelisting. The point of SPF is that it's supposed to be authoritative for the organizations involved, and doesn't include things like "I sent something with my work address from Earthlink and now you're accepting mail from my work domain through Earthlink's servers".

And using this to whitelist the sender rather than their whole domain gives you a lot finer control.

If it walks like a duck...

[jumperboy (1054800)]

This is clearly Challenge/Response with automated whitelisting. The following Wikipedia entry addresses every facet of this system:

http://en.wikipedia.org/wiki/Challenge-response_spam_filtering [wikipedia.org]

My current approach

[eric76 (679787)]

I've been using greylisting. For me, it really hasn't become less effective, but I have noticed that the mix of the spam has changed dramastically.

I'm getting ready to switch to two methods.

First, on one specific account that has become inundated with spam (probably because it is on just about every web page with registered IANA port assignments), I'm in the process of switching it over to the point where it will only accept unencrypted e-mail from a select list of whitelisted sources. If someone is not on that list of whitelisted sources, they are going to have to encrypt the e-mail using my public PGP key for the e-mail to be delivered.

Second, our mail server has something in the range of 100 to 200 users. I am generated thousands of additional e-mail addresses and aliasing them on the server to a single account. Those thousands of new e-mail addresses, initially 8,192 e-mail addresses, will be listed on various web pages for the spammers to harvest.

As e-mail starts to be delivered to those addresses, I will opt-out of all the e-mail so that they know the e-mail address is real and gets read. Once the spam reaches a certain level, I will then start blacklisting every incoming server delivering e-mail to one of those 8,192 addresses.

The length of time on the blacklist will vary. No IP address will be removed until a reverse DNS lookup for it exists.

If the reverse DNS lookup gives any idea that it may be a dialup, dhcp, or anything else that makes it look like it is probably a home computer (e.g. dialup-10-1-1-99.example.com), the IP address will be blocked for a month or more.

If the reverse DNS indicates that it is an smtp server (e.g. mta09.example.com), it will be blacklisted for maybe 24 or 48 hours.

Anything else will be blacklisted for one to two weeks. If additional e-mails arrive from a blacklisted IP address, the clock will start over.

I figure that with 8,192 spamtrap addresses and 100-200 user addresses, most spam zombies will be far more likely to hit the spamtrap addresses first where they may be automagically blacklisted.

So many reasons it doesn't work

[Jay L (74152)]

This scheme seems every bit as awful as those "Hi! Before anyone e-mails me the first time, I make them go through these steps" filters

- It causes backscatter
- It doesn't work with mail from mailing lists
- It's not accessible

Additionally:
- It doesn't work well with sites that have many MTAs (requires one bounce/CAPTCHA per MTA)
- It doesn't work well with an SMTP server that sends for many domains (requires one bounce per MTA per outgoing domain)
- It merely confirms that "this server can send mail for domain X". If you've got a spambot and can determine your user's domain name (e.g. comcast.com), this won't stop anything at all.

The author brushes off concerns with bold (well, italic now) statements like:

Resend software is a simple onetime update for webmail systems, email clients, and local mail servers...Universal Distribution of Auto-Resend Software is a Surprisingly Simple Thing to Achieve

Hah! A simple one-time update for all servers and clients everywhere! Granted, RIA doesn't depend on that update happening, but it's clear even the author thinks it'd be a pain without auto-resend.

There is little disincentive to implement Auto-Resend software as it is a one-time upgrade that remains dormant until needed.

There is a huge disincentive; looking up a user's mailbox to see if he did, indeed, send the message you claim he sent is a ridiculously expensive operation, if it's even possible at the server level. It could also lead to a privacy leak if done wrong; people could forge RIA bounces to probe outgoing mail flows.

At best, it potentially doubles the volume of outgoing mail, which deepens queues, requires more disk space, etc. etc.

I'm guessing the author is unfamiliar with high-volume mail sites - the very ones he wants to implement this scheme first.

Suspicious Domains Will Be Neutralized By CAPTCHA Encoded Sub-addresses

Great. So now e-mail that's "suspicious" requires intervention from a sighted human, and all his "auto-resend" silver bullets are used up. He does imagine yet another client change that will "nicely reformat" a CAPTCHA. Yeah, right. Oh, and now he's e-mailing me graphics on my Blackberry.

In general, he seems to imagine that he personally runs the One True RIA list, and we all trust his determinations of what is and isn't "suspicious", with reputation scores, rate limiting, etc. That is, of course, ridiculous; the original MAPS RBL has splintered and grown to the point where there are over 200 DNSBLs available.

He talks about automatically e-mailing users that he has "detected" are running zombies. Right, because that's a good idea and isn't spam.

Domains commonly associated with phishing (e.g. Paypal.com, Citibank.com)

As if there's a way to create a comprehensive, or even useful, list of "domains commonly associated with phishing".

with the passage of time it will become difficult for spammers to purport that all of their spam is sent via increasingly obsolete or esoteric brands of software.
Of course it won't. I still get spam from "The Bat!". Before, he forgot about the big guys; now he's forgetting about the long tail. Spammers can make up any number of X-Mailer names.

Re:

[nuzak (959558)]

I didn't read the entire paper, but after looking over the pretty pictures it looks like the sending party has to resend the message? That will only happen 50% of the time.

It could be greylisting, where the resend will be automatic. From the sender's point of view, there was just a delay. It's hard to say -- the article is not terribly well-written. The author's name is familiar, so googling on it turns up some other papers:

http://home.nyc.rr.com/spamsolution/UniversalAuthentication.htm [rr.com]

some discussion ca

Re:Still barking up the wrong f'ing tree...

[SCHecklerX (229973)]

I dunno. I've been pretty spam-free for the past several years using mimedefang, milter-greylist, and spamassassin.

The key is to reject the obvious nonsense before invoking your cpu-intensive analysis. I reject on the order of 90+% of everything that my mail server sees (even more at the last place I worked where they were using the same system). False positives on my home mail server are near 0. The ones that are mistakenly flagged, are simply flagged as spam, so I still see them, they weren't rejected or discarded. More at work got through, but that is because we have to be more conservative due to not having a good way to do bayesian filtering for individuals (I left before I had the time to run that project with the internal mail admins).

1. Implement Greylisting. Spammers don't retry
2. Reject if sending server is in zen.spamhaus.org or list.dsbl.org
3. Reject if helo is not a FQDN or IP Address
4. Reject if envelope sender claims to be an address from your domain (obviously our real users get through)
5. Reject if helo claims to be your own mail server
6. Reject if helo is an ip address from RFC1918 (again, short circuit on your own routing)

Then call spamassassin on anything that is left (SA will increase/decreas scores based again on RBLs that we don't outright reject, SPF records, etc):

1. use sa-update daily both with standard spamassassin rule updates, and, more importantly, the stuff at saupdates.openprotect.com
2. if you are able, create a way to easily train your bayes on false positives and stuff that wasn't rated high enough. I do this with specific courier IMAP folders that get checked once an hour
3. Tune your sa rules to taste. I had to decrease some things (lots of friends use yahoo mail), and increase others (Stock image spam. Ugh).

Re:

[Bogtha (906264)]

I'm frankly rather baffled at the lengths that people will go to in order to try to {filter / reject / stop transmission of} spam. We've already seen for years that such efforts are futile, because the same spammers will just adapt and find a way to pump out their crap anyways.

I receive approximately one spam email every 45 seconds. Constantly. Without spam filtering, I would go to bed with an empty inbox and wake up to 500 spam emails. Spam filtering, far from being futile, is the only thing that

Re:Still barking up the wrong f'ing tree...

[Anne Thwacks (531696)]

we need an economic solution

Nope. We need a solution involving cruise missiles though bedroom windows late at night.

We need Spam Assasin Ninjas clad in impregable black carbon-fibre capes with the knives of cutting edge technology and the deadly intent of artificial intelligence enhanced mania.

We need mountains of spammer bodies piled high on the forefront of technological .

We need chain gangs of spammers publicly televised chanting "The Only Good Spammer is a dead Spammer" to the sound of hammers hitting rocks.

IN Summary: Cruel and inhuman tortue is not enough for these guys