Earthlink Deploying Challenge-Response Anti-Spam System

deliasee writes "The Washington Post reports that Earthlink is preparing to offer new spam filter technology that requires sender authentication. AOL is still concerned that such technologies will put too much burden on consumers." The day after it's deployed, every legitimate mailing list on the planet will get challenges from all the Earthlink subscribers...

Nice moves

[hendridm]

I was hoping more ISPs would adopt the challenge-response system, like MailBlocks [mailblocks.com], previously featured [slashdot.org] on Slashdot. Way to go Earthlink! If I was interested in dialup, this would be a big selling point for me. I'm still waiting for a service that offers the challenge-response feature of MailBlocks but allows me to forward to my existing provider. I mean, a 12MB inbox is pretty lame. There are free providers [fastmail.fm] that can give me that much space...

Intrusive and Easily Fooled

[Templar]

Ok, so I create a new hotmail account -- [e-mail]. I send one test message and respond to the challenge, authorizing all future email from my address, then I close the account, use the address as my return address, and spam away.

Then, I give the address to all my fellow spammers and we use it until it dies. Then we make a new one.

Gee, that was tough.

How about mandatory authentication instead? Or even better, program all routers to only allow properly signed outgoing packets. Spam and hackers disappear overnight.

Michael's comment

[Rev.LoveJoy]

This is true, but perhaps it illustrates an opportunity for developers of mailing list software more than it exposes a flaw in Earthlink's plan to thwart spam?

As a network admin, many of the remote users I support (sales reps, on-the-road types) use Earthlink dial-up while travelling. At times, some of the program's that Earthlink has used to stop people from using their services to spam have make my job harder. However, I do not begrudge Eartlink for these inconviences, at least they, as a major ISP, are doing *something* about this problem.

My two cents,
-- RLJ

Response...

[Duncan3]

And in day 2, spammers automate the responses.

Results:
1. Spammers get free AUTOMATED account verification.
2. The load on the email system doubles.

Conclusion:
Nice "solution" dumbass.

Needs to be 'hard' in some way

[Ed Avis]

Of course it is no good if the spammers can set up automated systems to respond to the challenge. There are only two ways around this:

- Make the challenge 'AI-complete', that is, to give a correct answer you must be a thinking human being and not a computer. But then how can the other end check that the answer is correct? Having humans generate a fixed number of questions and provide sample answers also isn't going to work, since spammers will learn the correct answers. You need a way to generate an unlimited number of questions and to mark the answers automatically, and clearly this can't be done if the questions are intended to be too hard for a computer.

- Make the response computationally burdensome, so a computer can do it but only at the cost of some CPU power (so large bulk mailings would be impractical). This is what Hash Cash [cypherspace.org] and similar systems suggest.

It looks like Earthlink's system will rely on sending pictures you have to look at. Apart from the practical problems of clogging the wires with image files, I worry about OCR potential. The examples of this stuff I've seen on Yahoo, where you have to type in a number shown in a partially 'obscured' image, wouldn't have been difficult to develop OCR software for if you were so minded.

There's also the question of the spammer taking the challenge and sending it out to some other user. That user, by now used to replying to challenges from Earthlink and other addresses, will respond to the question and send the correct answer back to the spammer. D'oh!

Re:How do two people with C/R communicate?

[IIEFreeMan]

> How do two people with challenge and response communicate?
> If the challenge always gets thrugh, then the spammer will just issue cahllenges as spam.
> If they don't get through, then you would have a nasty mail loop.

In TMDA [tmda.net] (a challenge response system in python) at least, when you send a email to somebody, they don't get a challenge when they answer. It's logical because if you send him an email, you know he will not spam you :)
So i assume earthlink system will act the same.

Re:Correction

[Ed Avis]

Spammers seem to be sending a whole bunch of crap from my address ([e-mail]) even now. At least, I keep seeing what appear to be genuine delivery failure notifications of Russian spam sent from my address. Any system which trusts individual email addresses, without relying on some real authentication such as PGP signatures, is broken.

A simple rule is: Headers can be forged. Don't trust anything in the headers for antispam purposes. This includes the sender and recipient.

Re:How do two people with C/R communicate?

[Garion911]

One idea: Any emails you send out, the recpt is automaticly added to the "ok, let through" list.

Re:Now the spammers get address validation for fre

[PerlGuru]

I don't know about earthlink but ticketmaster's sys uses random different patterns obscuring the text. As for the text, the fonts they use vary, size varies, lines are not straight, and most of the fonts look like they are hand written (with even a single letter appearing differently in the same image)

I'd guess there system is pretty effective.

I assume

[ceswiedler]

I assume that the challenge-response is intended for messages already tagged as potential spam. In other words, low-scoring messages (spam-wise) wouldn't get the challenge. I certainly wouldn't expect a perfectly not-spam message to require the CR. Earthlink's (and other) spam-rating systems are pretty good, I think using it for the 'grey-area' emails would work well. And block the obvious spam without hesitation.

One question: shouldn't it be REALLY OBVIOUS to ISPs what is spam and what isn't? It seems that if a nearly-identical message gets sent to a large enough percentage of their users, it's clearly spam. Is this hard to do? Are spammers clever enough to distribute emails to avoid this?

Relative speed

[SunPin]

Way to go Earthlink! If I was interested in dialup, this would be a big selling point for me.

Earthlink offers DSL and cable. I'm using it right now.

I am definitely in favor of a little pain up front in increased traffic from challenge-response to get the spam boys off the net.

I suspect that when the spammers stop sucking up so much bandwidth, net speeds will increase for everyone--including dial up users.

Remember when 14.4K was fast? So do I. And I think with a correction in the system, it can be a decent speed.

Challenge Response Works great

[Anonymous Coward]

I've been using ASK (http://www.paganini.net/ask) which is an Open Source PHP based Challenge-Response system. It has a "Whitelist" which allows you to add approved senders and listserves as you can have either a From or a To address. It works so well because virtually all spammers use phony email addresses. Until spammers use valid email addresses, this type of system will continue to work. If they start using valid email addresses, then they can be dealt with in other ways.

Filtering instead of Blocking

[thehun101]

It would be useful if the system could be used to filter instead of block, at least for the first few months. Perhaps, if there is not response to a challenge after 72 hours, and email could be redirected to a 'Spam' or 'Bulk' filder.
This way, If I get monthly newsletters from donotreply@... and I want to keep getting it, i can approve that email. After about 3 months of this type of filtering and I would probably have approved everything I want to receive. Then, I could turn it back to blocking instead of filtering.

-the Hun

Adaptive teergrubing anyone?

[Tackhead]

Instead of challenge-response (putting the burden onto the end user), why not put the burden on the inbound mailserver?

A residential broadband customer mailing through his ISP's mail server is whitelisted (most stuff from that server is nonspam). An rr.com luzer with an open proxy is tarpitted into oblivion (everything else in 24.0.0.0/8 is spam). Yes, Joe Linux running (non-relaying) Sendmail on his Linux box is also tarpitted, but he's not trying to send a million mails a day. So he's not hurtin'.

I can see a scaling problem in that you'd have to run some sort of adaptive filtering process on the receiving end, which might be prohibitive CPU-wise. OTOH, if you only scanned 1% of all inbound mails for "spamminess", you'd still rapidly figure out that for a US ISP, 24.0.0.0/8 is an ocean of spam with a few islands of real email, and 200.0.0.0/7 is a shitstorm of spam. You don't need to analyze every inbound mail - you only need a statistically-valid sampling of the inbound mail queue to figure out which netblocks are teh sux0r.

Having it be adaptive would be cool - because a South American ISP (which probably has less of a problem with 200.0.0.0/7 than, say, Earthlink does, because they have legitimate users emailing each other from within those netblocks). So an ISP in .mx would end up with a different set of teergrubing weights. They might end up letting most of 200.0.0.0/7 in, only tarpitting the worst /24s, and teergrubing all 24.0.0.0/8 because so few of their users get anything but spam from rr.com netblocks.

Think of it as combining the best part of SPEWS (naughty netblocks are noticed semi-automatically), without as much collateral damage (if you're an ISP, a 10 second delay to anyone emailing one of your customers from a naughty netblock will never be noticed, but it'll *kill* some dirtball trying to spam to 10000 of your users through an open proxy.)

Re:How do two people with C/R communicate?

[1729]

You can't have an automated challenge/response system, because that defeats the point.

That's not true. There is an approach where you show a "proof of computational effort"; that is, your computer spends 10 or so seconds computing the response to a challenge. Here's a paper [microsoft.com] on the subject.

Re:Nice moves

[Anonymous Coward]

Mailblocks is a piece of junk, I had nothing but trouble dealing with them. And I wonder how Earthlink got out of being including in the numerous lawsuits Mailblocks are filing based on their patent covering challenge/response.

I think Mailblocks is the perfect example of a company the /. crowd would hate: rich guy comes into the market late, drops a bundle of money, scoops up some shaky patents, and tries to sue/shutdown their competitors (some of which have been in business for over a year longer than them).

By the way, Spam Arrest [spamarrest.com], one of Mailblocks' competitors (and currently being sued by Mailblocks) has no disk quota. (10 MB max message size though). Their enterprise product will forward messages through to your smtp server. A little pricey, but good for businesses.

Matador [mailfrontier.com] also does challenge-response (in addition to filtering) but runs on your desktop, if you are into that kind of thing.

Yes, I've done a lot of research into the anti-spam products!

I used to use this...

[All Names Have Been]

I was using this until I realized I was spending more time enabling/disabling the C/R system or screwing with the whitelist that I was dealing with SPAM. Everytime I wanted to sign up for some mailing list (it it coming from company.com or parentcompany.com or ???) or a user would sign up for some service that sent an email automatically, which, of course, would never appear, causing complaints and yet another trip to vi to modify the whitelist.

Don't even get me started on all those damn email card companies - lots of missing Easter cards because dumbassonlinecards.com wasn't in the whitelist and again, noone is going to send confirmation mails from an automated system.

The whole thing got dumped. Back to SpamAssassin, which causes far fewer headaches. Fortunately, this Earthlink deal is an opt-in system. I couldn't stand to use it myself and I bet few customers will live with this long-term.

Um, the blind?

[cnoocy]

So does this mean that if you're blind, you don't get to send mail to C/R users? Another hurdle for blind users is just what the net needs.

micro payments

[goombah99]

Challenge response is going to be effective but intrusive since a human must read the challenge and reply. this will suck when I sent the family newsletter to 40 friends I havent written to in a couple years and get 40 fresh challenges because my presence on their whitelist had expired. likewise even for automated things I sign up for like like slashdot updates or t rowe price stock reports

I'd like to suggest a way this could all be done automatically, so transparently your an AOL grandma could do it, and almost non-intrusively. Like the lessig-style stamp, all users would be charged say 0.01 cents to send ME an e-mail. but I would automatically refund this payment if either 1) the sender was in my addressbook/whitelist or 2) I did not file the e-mail in my junk mailbox.

what is needed is some sort of distributed postal service to handle the actual micropayments. And this is the main problem--how to collect these. I think the least intrusive method is that when you get an e-mail account you put down a pre-payment, lets say $10 on account at the postal service. when you send messages that are welcome your account is not depleted. when you send messages that aren't it slowly drains.

the cost of the postal service ditributed servers could probably be paid for by
1) the charges for unwanted e-mail
2) interest on the deposits on account.
thus people would be willing to set up these servers.

the final missing ingredient is a centralized server that coordinated the actual postal servers. all this would be would be like a DNS that told all of the remote servers the names of the other ones so they could communicate account info.

the transactions themselves would be in number about twice as the number of e-mails handled (one to the post office from the first ISP to receive the mail to validate the payment code in the header, one from to the postal service me to authorize refund/no refund), and the accounting message size very small.

Perhaps this is a rotten idea. its main benefits are 1) its not intrusive and is nearly transparent 2) it pays for itself 3) requires changes only at the browser level.

I does not stop spam from showing up in my inbox, but makes it very expensive to mass mail.

flame on! or suggests problems and their solution.

Calling all perl wizards and poor college kids!

[MattGWU]

Perl gurus, start your editors!
How many lines will it take to write a script to automatically reply to challanges? As long as the messages have predictable structure, you should be able to write a parser to pick out the word or picture they want, then throw it back.

College kids: Are you bored, broke, and of weak moral fiber? You too can make money while sitting on your ass by replying to email challanges for the princely sum of 3 cents per message! Combine the first suggestion with the second, and you've got yourself a money machine.

It's great to see an ISP take some decisive steps, but this scheme has weaknesses. Interesting to see how it goes. Despite the concerns, I'm cautiously optimistic.
As a twist, it would be interesting to see how that anti-spam vs. spam lawsuit with the copyrighted haiku goes (don't recall the parties names, but it's gotten coverage here). Maybe something similar could be combined with the challange-response system to make it illegal to respond to the challange under false pretenses. Raises a few slippery-slope legal issues that if you're going to touch, you might as well criminalize spam outright (which would be fine, of course).

Re:How do two people with C/R communicate?

[platypus]

And what happens if ReplyTo != From ?

Re:Too drastic?

[letxa2000]

I don't know why more people/ISPs aren't using this. This system seems to be the most effective because it doesn't have silly little measures

I agree. It's so simple yet so effective. It really makes me wonder why people invest time and money in silly, less-friendly and potentially less-effective solutions such as C/R.

it seems to rate the spam based on its content, which no spammer can get around.

They're starting to try. When they start breaking up words so that "cock" is "c.o.c.k" they're making an effort to avoid filters, but also are addressing the Bayesian filters since that will normally get broken up into 4 tokens, one for each letter. Of course, if they do it enough then a single token "c" might actually become a commomn characteristic of spam for that user.

Anyway, Bayesian works great now. I think spammers will evolve to deal with it, but all that is necessary is to implement new token-identifying logic in the Bayesian filter... the Bayesian approach itself is very solid.

It's a hell of a lot faster to do than actually placing calls to people and talking to them, and people

I agree. I suspect you will see spammers actually analyzing the C/R responses. If it's something the software has seen before and is capable of responding automatically, it will. Those that it can't will be forwarded to someone to quickly deal with it. If some of the megaspammers make as much as they supposedly do, hiring a teenage kid at $6/hr to spend the day answering C/R responses is not a huge investment.

Bayesian Filter + Challenge Response

[juggler314]

A number of folks have pointed out how this really doesn't work so well in a real world situation. This is pretty much true, there are myriad problems. What can work fantastically is a two tiered approach though: 1) Use a Bayesian filter to sort your mail however you want (for simplicity lets just say spam/not spam). 2) Forward all filtered mail marked as spam to your CR prog of choice - this chunk of mail should already be confirmed in the high 90%'s to be spam - the few false positives should get caught. The reason this works so well is that the Bayesian filter approach is pretty solid, but there's always a worry of a few important false positives sifting through. This gets rid of those. If you really want to go balls-out you could make use of a service such as spamgourmet.com for ordering processes. Whenever you order something where you are expecting some automoted return mail that might hit the Bayesian filter AND also not respond to the CR use one of the self destruct e-mails. You should never get more than 5 or so e-mails from an order anyway. You can then just filter everything from your bogus self destruct e-mails into a generic "orders" folder.

Precedence: Bulk

[Euphonious Coward]

All they need to do to handle legitimate mailing lists, at least at first, is to challenge only mail that is not explicitly labeled with "Precedence: bulk". Legitimate mailing lists carry that label, but spam never does.

Once the spammers are obliged to label their stuff "bulk", half the battle is won. Then they start collecting a "white list" of legitimate mailing list sources, and label every bulk message not on it as "suspected spam" and dump it in a separate folder.

Re:Too drastic?

[BlackHawk-666]

TMDA utlises shortlived email addresses for this purpose. It will create an email alias that anyone can send to...but just for x (5 for example) days. Give this to the company as you sign up and you will receive their confirmations. You can either leave it like that and then 5 days later they can't spam you, or whitelist them and give them your permanent address.

See "Guarded Email" paper

[dwheeler]

For more details on a challenge-response system, see my paper on "Guarded Email" at: http://www.dwheeler/guarded-email [www.dwheeler].

Guarded email completely deals with some of the problems noted in these comments:

1. How do you receive challenges? Yes - if you SEND a message to someone, then you can set things up to automatically RECEIVE messages from that someone.
2. Can blind people send email? Yes - the challenge should be human-readable, but not computer-processable. That's easy.
3. Can you prevent loops? Yes - you have to think about it, but there are simple loop-prevention techniques so that EVERYONE can use these kinds of systems.

Could help slow some worms, viruses. . .

[GeorgieBoy]

. . .as long as people aren't getting them from their buddies. Even so, if emails are scanned for viruses/worms in attachments before they get to the user, there can be more wins than just stopping spam.

Re:Adaptive teergrubing anyone?

[milo_Gwalthny]

Take a look at the front page article in the WSJ today... about one of Earthlink's most virulent spammers. He used 300+ dial-up accounts, set up with fraudulent/stolen billing info and was sending (they say) 1 million+ spams per day. Took them like a year and a John Doe lawsuit to finally figure out who he was and stop him. Interestingly, one of the ways they were tracking his accounts was by which passwords he used (he tended to use just a few for all of his accounts)--thought he would catch on to that.

Great article, wish I could post a link. To your point... wouldn't this guy have been automatically whitelisted?

Re:Too drastic?

[letxa2000]

This is moot anyway since spammers don't actually provide return email addresses.

Oh, I'm sure they'd start using actual return addresses... at yahoo, hotmail, etc. As long as they (the free email accounts) last long enough to collect some challenges that's all they need. Even if the accounts are closed by hotmail you can still send email "from" that account.

C/R doesn't even have a chance of working large-scale while there are free email providers such as Yahoo.

And even if it does, as someone else said, you just start sending spam with email addresses that have a high chance of being whitelisted. [e-mail], [e-mail]. So now instead of sending 1 spam to each user they'll send the same message 100+ different times from different addresses that they have concluded are more likely to be whitelisted in the hopes that one of them actually is whitelisted.

At best, C/R doubles spam traffic by generating a C/R request for each spam sent--now instead of just getting bounces sent to some poor innocent victim, the innocent victim will get bounces plus thousands of C/R requests. At worst, spammers will take the brute-force approach mentioned above of sending hundreds of copies of the same spam to every user using different "common" whitelisted email addresses. Either way the spam problem arguably gets worse, not better.

Blindness

[druske]

If the challenge is based on an image ("please respond with the fuzzy word in the subject line" or somesuch), where does that leave vision impaired email users? How do they respond to a challenge to get their email delivered?

One problem with this system.

[illumin8]

Did anyone notice that in order to workaround automated systems that need to send legitimate email, such as Amazon when you buy something, or mailing lists you subscribe to, they give you a second email address that will not be protected by Challenge/Response?

I can see this being a big problem. In my experience, people only get spam if they have done one of several things:

1. Published their email address on a web page to be picked up by harvesters.
2. Given their email address to an online retailer that sells it.
3. Signed up for some spyware scam where they again give their email address to someone that will add it to a spam list.
4. Opened a Hotmail account, which, it seems is automatically sold to all the various spam providers.

In almost all of these cases, the act that caused spam to be received was the user giving out their email address to a non-trustworthy source.

How is having a second email address that people will just type into any webpage that promises free porn and bypasses Challenge/Response going to curb the spam problem? I give this system only 1-2 months before spam is back at it's initial volume, just using the new email address instead of the old.

You need to also educate users about the problems of giving their email address out to unreputable places on the net. A lot of users don't correlate their spam problem with the fact that they typed their email address into some website to get a free porno password the night before.

Re:Too drastic?

[corz]

Almost a year after Paul Graham's "A Plan For Spam" Bayesian is still the easiest system to develop as well as the easiest for the user to use. It is extremely effective (99.5%+) with very few false positives and doesn't require any additional effort for the sender and only requires that the user report false positives and false negatives--and that is mostly only needed at the beginning. Once it is initially tuned it's not necessary to do much of anything--it just keeps learning and working.

Personally I use a combination of SpamAssassin's bayesian abilities along with TMDA, a challenge/response system. I only require confirmation for messages that SpamAssassin identifies as being over my threshold of 5. In my .tmda/filters/incoming file I have the following rule:

pipe "/usr/bin/spamc -c" ok

That means that if SpamAssassin says its clean, then no confirmation is required and TMDA delivers the message to my inbox.

Simple, effective, the best of both worlds.

Re:Relative speed

[batobin]

As a web host AND web designer, I can say that larger web pages aren't the fault of poor design. Page sizes are simply larger these days. Take for example loading this thread at +2 or +3. It would take minutes to load on a 14.4. Is that the fault of large images? Of inefficient code? Nope.

I have a feeling if you saw pages designed for 14.4 today, you'd be deeply disappointed.

Re:Nice moves

[phat_joe23]

It drives network traffic as well up to the sky.

Hardly. If you're on Earthlink and decide to opt-in for this, it simply means that everybody you know has to send you one extra email once.

And that every time you get spammed from a new address (read: constantly), the system fires off another confirmation email from you. It effectively doubles the number of network connections spam generates. /joe

Re:How about another approach...

[grishnav]

I run a legitimate e-mail server for my family, but cannot afford an SSL certificate for it. I instead use a self-signed one.

If self-signed certificates would be allowed, then spammers would make their own. So that can't be allowed.

If they are prompted, as you suggested earlier, it would inevitably lead to people who just ignore invalid ones, because they are sick of being prompted. My little mail server gets creamed.

Nice idea, but unless you get Verisign to give away free certs, I can't see it working.

An alternative solution?

[NanoProf]

A fundamental problem of Spam is that the sender of an email cannot be identified and verified with 100% accuracy, so it is difficulty to filter 100% effectively. However, there is one and only one part of an incoming message that must of necessity be accurate- the To: address. So use the To: address to identify the sender! Publish your public address: "[e-mail]". Any email to foo generates a reply "Thanks for the note. Mr. Foo loves you so much that he's generated a special personal email address just for you to use: '[e-mail]'. Please use this address in the future- sorry but you'll need to resend the message just sent to this new address. Don't ever give out this secial address to any else, because if Mr. Foo begins to receive spam on this To: address, he will automatically filter all future messages to foo_RANDOMSTRING straight to the trash." Every sender gets a unique RANDOMSTRING, so you can filter on the To: address. It's similar to throw-away email addresses, but coupled to a public address that triggers auto-generation of new RANDOMSTRING addresses. The sender has the inconvenience of adding [e-mail] to their address book. Also, spammers can read the auto-reply and then add foo_RANDOMSTRING to their spam list, but this could be made difficult by putting it in a distorted gif image. The email client would also need to be configured to set Reply-To: correctly on folowups. One nice thing is that for user-requested bot-generated emails, one can simply give them a new RANDOMSTRING-based email address right off in the registration form or whatever. The ever-expanding number of [e-mail] addresses adds to the overall load on the servers, but is that handle-able (nasty things could happen if your inbox got Dos'd)? In such a world, people would get used to pinging new people with just a short message to obtain their personalized RANDOMSTRING address. Kind of a weird system but maybe it's interesting to think about?

Re:How do two people with C/R communicate?

[inbox]

Hrm... I think that yes, in fact, you do get 9000000 challenges from everybody on the list. The sender's e-mail address is not whitelisted at the Earthlink mail server, it is whitelisted at each e-mail account.

Otherwise, a spammer just sends one message from an address, responds to the challenge, and then spams away.

Or am I misunderstanding it?

Re:You can do this yourself.

[lobotomy]

Or better yet, what happens when a confirmation message is sent to confirm your confirmation message? Is there any looping message detection built in? Maybe if both sides are using the same program, but this could be disasterous if two users have different challange-response systems that don't know about each other.