Novel Method for Universal Email Authentication

MKaplan writes "Most spam is sent using spoofed domains. Email authentication schemes such as SPF attempt to foil spoofing by having domain administrators publish a list of their approved outgoing mail servers. SPF is sharply limited by incomplete domain participation and failure to authenticate forwarded email. A paper describes a novel method to rapidly generate a near-perfect global SPF database independent of the participation of domain administrators. A single email from an unauthenticated domain is bounced and then resent — this previously unauthenticated domain and the server listed in the return path of the resent bounce are entered into a globally accessible database. All future emails sent from this domain via this server will be authenticated after checking this new database. Mechanisms to authenticate forwarded email and to nullify subversion of this anti-spam system are also described."

That's already implemented with Spamcop

[no-body]

Mail servers are authenticated by Spamcop and forward spam automatically to Spamcop which adds it to their database. When using reject_rbl_client bl.spamcop.net SPAM is blocked.
Works like a charm!

Re:That's already implemented with Spamcop

[no-body]

&& that's IP based, not domain name based, so the SPAM originating IP is known and can be blocked

That's the problem.

[khasim]

He does not CLEARLY explain what he is intending.

I believe he means denying at SMTP time, so the sender will try again after X minutes.

Which is kind of like greylisting. The FIRST problem is that the spammers have adapted to this and retry.

The SECOND problem with this is he's saying:

Unique sub-addresses are dispatched in the 'From' field with routine outgoing email. [e-mail] may send RIAuser^[e-mail] to one individual and RIAuser^[e-mail] to another individual.

Huh? So this is also about SENDING email?

Now if you could bounce the message, it would just go back to the original IP, so I don't see why that would help either though.

And it doesn't address the issue of "fast flux" where the domains are "legit" in that they exist and point to the IP address of the sending machine ... for a few minutes.

So he's talking about "bouncing" messages ... installing new software ... and altering the "From:" addresses on stuff YOU send ...

No fucking way is this going to work.

Participation in SPF

[Anonymous Coward]

"SPF is sharply limited by incomplete domain participation"

That's not a big problem. 99% of non-participating domains fit in default SPF record "a/24 mx/24 ptr -all", we use it in qmail for few years. Together with Spamassassin it results in 99,8% antispam accuracy (warning: one big exception is yahoo.com, you should use domainkeys or add ptr:yahoo.com to default spf rule)

Re:The BIG issue

[crossmr]

I have a friend who works for a large ISP here in town and they do something like that but the thresholds are much higher. He told me a story about a woman who had been blocked multiple times but refused to clean the viruses off her computer but would call and bitch that she couldn't send any e-mail. I guess each time you trip the system and get blocked its a longer block. The last time she had called in he said it looked like she'd been blocked at least a dozen times based on the length of that block.

Re:Major flaw in methodology

[Dan B.]

Not so, most of the backscatter is sent to [e-mail] which is either quietly dropped if you have smart filters that look for mailer-daemon@ etc as the sender, or passed to your 'no one by that name' catch all mailbox. Some mail systems will in fact be terribly misconfigured for backscatter, but then how is that different from what we have today?

The worst email storm I got was when some spammer decided to use my domain as the sender of all his junk and send all hi junk twice. I do have SPF entries in my DNS so ANYTHING that would encourage others to actually USE this system is a GOOD THING.

Now if there were just a few simple packages available that would give us the one-click (tm) ability to add SPF filtering to Sendmail/Postfix/Qmail/etc, and MS Exchange 5.5/2000, then I would guess that 50% or more of the domain spoofing spam would cease. That can only be good, as I only get UCE from real domains that I can't check for authenticity, from spammers who bother to follow RFCs and send twice after postgrey (greylist filtering) blocks them first time around.

Check out existing discussion...

[enbody]

A Google search revealed this intelligent discussion of the scheme.
http://www1.ietf.org/mail-archive/web/asrg/current/msg12403.html [ietf.org]

I'm not sure you understand what SPF solves

[Degrees]

SPF only solves the problem of SpammerS sending mail to MailserverB with a forged header to make the message look like it came from MailserverA. The assumption is that UserB might open the message if it says it comes from UserA.

SPF causes MailserverB look up DNS data for the email domain for MailserverA, and compare it's SPF published IP addresses with the IP address of the incoming email connection from SpammerS. If the two don't match, then MailserverB hangs up on SpammerS with a 566 eat-shit-and-die error code.

That's all SPF does: eliminate impersonation.

For that, it's a great idea.

If you think that SPF is going to eliminate all spam, then you have misplaced hopes. Don't throw out SPF just because it is a piece of the solution instead of the whole solution.

The problem you describe is solved with SURBLs.

IMO, people should use both.

Re:Still barking up the wrong f'ing tree...

[SCHecklerX]

I dunno. I've been pretty spam-free for the past several years using mimedefang, milter-greylist, and spamassassin.

The key is to reject the obvious nonsense before invoking your cpu-intensive analysis. I reject on the order of 90+% of everything that my mail server sees (even more at the last place I worked where they were using the same system). False positives on my home mail server are near 0. The ones that are mistakenly flagged, are simply flagged as spam, so I still see them, they weren't rejected or discarded. More at work got through, but that is because we have to be more conservative due to not having a good way to do bayesian filtering for individuals (I left before I had the time to run that project with the internal mail admins).

1. Implement Greylisting. Spammers don't retry
2. Reject if sending server is in zen.spamhaus.org or list.dsbl.org
3. Reject if helo is not a FQDN or IP Address
4. Reject if envelope sender claims to be an address from your domain (obviously our real users get through)
5. Reject if helo claims to be your own mail server
6. Reject if helo is an ip address from RFC1918 (again, short circuit on your own routing)

Then call spamassassin on anything that is left (SA will increase/decreas scores based again on RBLs that we don't outright reject, SPF records, etc):

1. use sa-update daily both with standard spamassassin rule updates, and, more importantly, the stuff at saupdates.openprotect.com
2. if you are able, create a way to easily train your bayes on false positives and stuff that wasn't rated high enough. I do this with specific courier IMAP folders that get checked once an hour
3. Tune your sa rules to taste. I had to decrease some things (lots of friends use yahoo mail), and increase others (Stock image spam. Ugh).

If it walks like a duck...

[jumperboy]

This is clearly Challenge/Response with automated whitelisting. The following Wikipedia entry addresses every facet of this system:

http://en.wikipedia.org/wiki/Challenge-response_spam_filtering [wikipedia.org]

Re:Fails to account for SMTP farms...

[Anonymous Coward]

SPF is a limited tool, designed to reduce forgery in a lightweight way for domains willing to keep track of where they send email from, and for MTA's willing to drop the connection if the "FROM" line used to identify where the bounce message might go, not the "From:" line, turns out to be not match the allowed IP addresses for that record.

It's that simple. It's lightweight, saves time for servers, and helps make sure that all email can be tracked back more usefully to a hostname that takes responsibility for email. It's failing for several reasons:

1: It breaks forwarding, unless the systems doing forwarding do some interesting software changes to keep track of the original "FROM" line and re-write it to send the email, and pass back any bounce messages. This isn't already implemented in most MTA's and takes some work.

2: Microsoft poisoned the well by attaching their "SenderID" to it, then completely screwing up the ISO standards submissions by refusing to release their SenderID related XML patents in a way that anyone but a Microsoft bed-partner could use them.

Thus, SPF is effectively dead until these fundamentally political issues are resolved.