Using Email Networks as P2P Spam Filters

Oscar Boykin writes "New Scientist is running a story on using the social network in email as a P2P network. The idea is that email networks have structure that is conducive to a type of search called percolation search . This means email clients could query the social network of email users to filter spam. This story is based on a preprint available."

Secure?

[geomon (78680)]

The authors propose that their system have access to inbound and outbound contacts. For trusted email accounts, that might work. But what about email accounts that people may want to creat to sheild their identity (political dissidents, whistleblowers). They would have to live outside of the spam protection network and would, I assume, be seed accounts for spammers.

Am I missing something in this analysis?

Re:Secure?

[seoYak (883591)]

I don't think that i'll trade my privacy for a reduction in spam.

Re:Secure?

[by Anonymous Coward]

"Those who would trade privacy for a reduction in spam deserve neither." Benjamin Franklin.

Re:Secure?

[Mad_Rain (674268)]

Wouldn't the whistleblowers and political dissedents just have to email and be read by a few trustable sources to become trusted themselves? I guess the hard part would be to get past the filters of those first few, but it would be possible after that.

Re:Secure?

[geomon (78680)]

I guess the hard part would be to get past the filters of those first few, but it would be possible after that.

True. I guess my concern would have been whether their proposed system could be mined for information regarding frequency of connection between two emailers.

Re:Secure?partially...

[spectrokid (660550)]

You could collect email adresses in a hashed form, just like passwords are stored on a server. You would be able to check if the sender is in the list, but not be able to "un-hash" the list back into real adresses. The way to get around it would be for spammers to attach their sender adresses to these funny mails people do like to forward to their friends.

Re:Secure?

[rescendent (870007)]

Also you would not be able to be emailed by people who you haven't already approved the email address; would they have to phone you first?

For example:
People who change email address (Gmail, dropping a spammed email)
People legitimately contacting you (Old friend, people wanting to know more about your website etc.)
etc.

It would be like setting your telephone to only accept certain phone numbers and scrapping the phonebook. Bad for people, terrible for business.

Though I suppose spam is worse because

Nice...but not necessary

[PenguinBoyDave (806137)]

Since switching to Thunderbird, I get nearly no spam...maybe one or two per day. I like fancy stuff, but when simple works, go with it!

Re:Nice...but not necessary

[winkydink (650484)]

One or two per day out of how many? 3? 5? 1000?

Re:Nice...but not necessary

[Dukael_Mikakis (686324)]

I use gmail, which does an excellent job at filtering spam.

But I think this could even be a step back. Like the parent says, I think most informed people have solved the issue of filtering spam pretty effectively (Thunderbird, Yahoo, Gmail, Bayesian filters, etc.) and so we don't generally *see* much spam.

The *REAL* problem with spam is traffic and network pollution. Spam wastes a ridiculous amount of bandwidth and (through spyware) hijacks our systems' cycles to do something that is (with filters) ultimately to no end. This seemingly won't solve the bandwidth consumption issue and might worsen the problem by polling all your friends over the network and then using your personal cycles to scan said email against all the known spam on your friends' computers.

People forget that the true detriment of spam these days is the traffic it causes, not cluttering your inbox (if you're smart).

Spam degrades the reliability of the medium

[erice (13380)]

People forget that the true detriment of spam these days is the traffic it causes, not cluttering your inbox (if you're smart)

You've got to be kidding. Spam is text, or very nearly so (HTML). Unless you are using floppies and a 2400bps modem, the bandwidth/storage costs are irrelevent.

What is relevent is that it forces people to either use spam filters that randomly throw good messages away or they miss good messages becuase they can't be seen among all that spam. In either case, the loss rate goes fr

2004 spam for one user costs $0.55

[erice (13380)]

For people that have to pay the costs associated with building out network capacity and maintaining it, spam's bulk is indeed the main problem.

No, actually it isn't. I run my own mail server. I keep all mail, including spam. I get something like 200 spams/day. All spam for 2004 amounts to a bit more than 100MB. At the somewhat inflated price of $0.50/GB, that is about 5 cents to store all the spam for 2004. You may quibble over the exact number but you would be hard pressed to come up with storage c

Re:Nice...but not necessary

[geoffspear (692508)]

That's a nice theory, but it seems more likely that the more effective spam filtering gets, the more spam will be sent. If it takes 100x more messages to get the same results, the spammers will just send 100x more messages. And they'll need to turn even more machines into zombies to do it.

Re:Nice...but not necessary

[nokiator (781573)]

I am not keeping my hopes high on this. The response rate for the regular snail mail is about 1%, that is 99% of the junk mail people receive basically goes to trash. Still, corporate America is willing to spend more than $46 billion on direct mail marketing every year. Considering that postal junk mail is a lot more expensive to send than e-mail spam, a 0.1 or even 0.01% response rate for e-mail spam may be sufficient to sustain the current spam rate, if not continued growth.

Re:Nice...but not necessary

[Dukael_Mikakis (686324)]

I would agree but the incremental cost of sending spam is basically free ... especially if people let spyware hijack their computers to do the heavy lifting.

Re:Nice...but not necessary

[psetzer (714543)]

Better filtering has to be on the network level, so that individuals don't have to opt in. The people who know how to set up effective filtering won't likely buy anything from spamvertisers in the first place, meaning that, while it makes those peoples' lives easier, it doesn't affect the spammers' bottom line much.

Re:Nice...but not necessary

[fireduck (197000)]

I've been using thunderbird since 0.7 or so, and I'm not entirely in love with it's spam filtering. I've had it turned on since the feature was introduced, so it's had a long time to learn what is spam (95% of my email) and what's not (the other 5%). My most recent download of my earthlink email (which i've just about given up on), had about 60 messages, only 1 of which wasn't spam. But of the 59 spam, 17 weren't marked as spam (despite me junking very similar emails a few days earlier). Perhaps the spa

Re:Nice...but not necessary

[PenguinBoyDave (806137)]

1.0.2 has a *junk* filtering system. When you turn it on it will go through your inbox and mark what it THINKS is junk. It them moves the *junk* to the junk folder where you can review it. If you find something that is not junk, you can click "not junk" and move it back to the inbox. Once I "trained" it for about a week, I have never found anything I wanted in the junk folder, and like I said, I only get about one or two spam messages in my inbox per day, compared to the 150+ that end up in the junk fol

Re:Nice...but not necessary

[conteXXt (249905)]

glad you told me. I was happily using it.

gentoo implimentation of courier-imap in ssl mode.

I have had no problems with it.

NOTE: 3 users, all me. NOT in production environs

Any suggestions for a better replacement?

Re:Nice...but not necessary

[Linux_ho (205887)]

I'm using Thunderbird with IMAP over SSL and it works great, I've had no problems at all. What server software are you using?

Great...

[yotto (590067)]

...Now the RIAA's going to sue me for getting spam.

Potential for harm

[davidwr (791652)]

Imagine the potential for harm if I infiltrated a social network and then identified my enemies as spammers, either deliberately or because I or the software agent I use was somehow tricked into doing so.

Social network-based spam-detection is a part of, not a total, solution, and its limits need to be recognized.

Re:Potential for harm

[Dukael_Mikakis (686324)]

This isn't spam-detection on a user basis (e.g. a network with "these email addresses are trustworthy and these email addresses are spammers") it's spam detection based on a spam database.

The potentials for abuse that I see are if you don't keep a spam database at all, and so you will not flag any query as spam (even if it clearly is), or if you try to keep an "Anti-Spam" database (I dunno, a database of legitimate emails. Only problems with these abuses are if you have no database it shouldn't matter be

Isn't this basically how Razor works?

[forevermore (582201)]

Granted, I just skimmed the article, but isn't this exactly how Razor works? (simplified) Communities of people flag messages, senders, etc. as spam, and the mail server (or in my case, spamassassin) compares the messages to the community spam archive for matches before delivery.

Re:Isn't this basically how Razor works?

[HermanAB (661181)]

Razor, Razor2, Pyzor, DCC...

Isn't this how Yahoo works

[CrazyJim1 (809850)]

You click a multi-user message as marked as spam, then it turns into spam for everyone else too.

Reduces to a standard spam filter

[tdvaughan (582870)]

According to the article the method works by asking its network of email users if they've seen the spam before:

Similar software on each computer that receives the query would then check the message against its own spam database, and so on, until a match is found, or the message is deemed original.

So it can't deal with spam that includes a unique random ID and would tag emails from a mailing list as spam. Once more: nice try, but it won't work in the real world.

Re:Reduces to a standard spam filter

[p2sam (139950)]

see dcc, razor, and pyzor for details. but there is such a thing as fuzzy checksum that hashes two similar input to the same output.

Ob

[lheal (86013)]

In Korea, only old people get P2P spam.

Actually, I think we should find a way to attach the same stigma to spam customers that we do to the spammers. Why do spam customers not have to go to jail? They're as much the problem as the spammers.

I can see something like having all the spam customers' names published online, so you google for "spam" and "lheal" and up pops my list of purchases. The other spammers then get a very clean list of people to spam. Over time, the net would be segregated into those who like spam and those who don't.

Yeah, unworkable idea, but so are all the others.

Yet Another Insane Proposal

[abb3w (696381)]

I think we should find a way to attach the same stigma to spam customers that we do to the spammers.

Nah... Congress can't make stupidity illegal; they'd lose too many votes. The universe, not being elected, can... but tends to be in favor of capital punishment as a way of preventing repeated behavior.

An utterly illegal and unethical solution would be to start up a V1AGRA spam outfit, and taint the supply so that one pill in twenty was actually a disguised lethal dose of cyanide. This would cut into dem

Re:Ob

[bill_kress (99356)]

I used to keep a list of spam customers and refuse to patronize them. I still avoid them when I can, but have totally lost the ability to keep the list in my head. In my old age, about all I remember for sure is I'll never buy an x-10 camera thingy.

Hmmm...

[creimer (824291)]

When you thought it was safe to use email again...

Re:Hmmm...

[creimer (824291)]

wow u noticed

YahooMail, GMail and Hotmail Do This Already

[osewa77 (603622)]

What strikes me is that the idea of "pooling information" isn't really new. When one yahoo-mail/HotMail/Gmail user marks a particular mailing as spam, it affects the likelihood that the same email would be marked as spam for other yahoo users. So, the idea of "pooling information about spam" (from article) is already in use! However, it would be nice to create explicit protocols to allow such data (what mailings I have marked as spam) to be made public so that people using other email providers or their own mail servers can share in this pool of knowledge. Of course, the big three email providers (yahoo mail, hotmail, and gmail) will be foolish to make this information public: the spam filtering is one thing that makes a yahoo/gmail account more attractive to potential users! Good idea in theory, but bad business prospects. To add insult to injury, there is no way for the researchers to profit from the arrangement.

Re:YahooMail, GMail and Hotmail Do This Already

[lheal (86013)]

... (yahoo mail, hotmail, and gmail) will be foolish to make this information public: the spam filtering is one thing that makes a yahoo/gmail account more attractive ...

Don't reject that idea so quickly, as I think you're on to something. The protocol would encapsulate the information that "[e-mail] marked this message as spam". What the email provider does with that information is something else.

Not only that, but Google and Yahoo! could team up against Hotmail, or AOL, or whoever. Maybe

would that really be good?

[overbom (461949)]

If I were a spammer:

I'd change an email client to respond with any message from certain folks I don't like to report all of their messages as spam to poison the social network. a couple of clients out there saying "yup, I've already got a message like that here, and my user marked it as spam".

think globally, act locally, right?

Re:would that really be good?

[taustin (171655)]

I'd send myself a bunch of spam complaints, mark them as spam, and see if the abuse desk at my ISP was using this filter system.

Not a particularly new idea...

[Otto (17870)]

This isn't a new idea... except that they propose to integrate it into the mail client and have everybody you've ever sent mail to or received mail from be a potential contact, weighted by frequency that you email them. That's a bit new, but not as effective as it seems.

For one thing, it would block mailing list messages, which are messages that you probably do share with your contacts.

For another, it does not consider that most spam has random keywords seeding into every copy sent, so those would have to be ignored somehow, which introduces a fuzzy match algorithim, which means the possibility of false matches exists, and since you're asking others (probably all using the same algorithim against their databases) you have increased the chances of a false match being found.

In any case, collaborative networks already exist in a better form. Users mark messages as spam when they get them, a flag is created and sent to some central place that all users check against for matches. The algorithim for fuzzy matching resides in one place and is only used as an indicator in spam assassin in any case, not as the sole indicator..

Large scale systems like Google's GMail can use people flagging messages as spam to filter similar enough messages from other users, sort of thing. I'm pretty sure they do something like this, in fact, as my GMail account has *never* made a mistake in it's spam detection.

And so forth. There's better ways than relying on a random query of your contacts to see what they think.

Mmmm, buzzwordie

[cloudmaster (10662)]

Sorry, I can't read the article. There were too many buzzwords in the post.

spam filters should reduce network load

[sPaKr (116314)]

Skipping past the security issues. One of the goals of spam filters should be reducing network load not increasing it. If we have to send our spam to several differnt peers to be scored this would compound the network load problems. Mostly this is a bad idea(tm) from the get go. I think the only thing that will really stop spam is to force something like pgp(gpg) signatures on all mail. Here's hoping the new national ID cards will have public certs encoded on them. It would be cool if someone would step in and get PKI working for the rest of us. Also we should drag the boddies of spammers through major cities behind a horse, while allowing victums to beat the spammer with large sticks like golf clubs.

Re:spam filters should reduce network load

[AnotherBlackHat (265897)]

One of the goals of spam filters should be reducing network load not increasing it. If we have to send our spam to several differnt peers to be scored this would compound the network load problems.

All other things being equal, reducing network load is better than not reducing it.

But all other things are NOT equal.
If it's a tradeoff between reducing the load on the human (by reducing the amount of spam they must deal with) and reducing the load on the network, I'll pick "reducing the load on the hum

Re:spam filters should reduce network load

[sPaKr (116314)]

You would have a point if this was the only solution to the spam problem. Fortunately it isn't.

Probably easy to bypass

[Anonymous Writer (746272)]

Similar software on each computer that receives the query would then check the message against its own spam database, and so on, until a match is found, or the message is deemed original.

Spammers can ranodomly generate content for their spam to bypass this. Have the actual spam message text as a JPEG image followed by random, gibberish text in both the same background and foreground colour so it is invisible. If the system looks for messages that are identical by comparing the text, then spam messag

Sounds Like SpamNet

[MBCook (132727)]

That sounds like Cloudmark SpamNet (I think that was what it was called). I used it a few years ago when it was in beta and it worked great. The idea is people marked mail they got as spam if it was. When they did that, a hash of the message (or title, or something like that) was sent to their server. When your mail came in, it was hashed and checked to see if it was spam. It was VERY accurate. It had only one problem:

Cloudmark.

I signed up for the free beta and was told that it would be free forever (they were going to charge businesses, IIRC). Then they chagned their mind but said that early adpoters/beta users would get it free for life. Then it left beta and they offered me a $5 discout (one time) for their subscription service (or some other pointless trinket offer like that). As far as I'm concerned they ripped me off.

That set me off trying other things, and I eventually found POPFile, which I use to this day (great software). I've posted this to Slashdot before (a long time ago). Some nice guy from a anti-spam company gave me a code for a free version of their product to be nice (I never used it, I had found something by then and didn't feel like switching again).

The point of all this is that it is a nice method that really works. If there was an open source project that did the same thing, I would use it. Untill then, I've got a solution that works fine.

But this isn't new (if I'm right about what it is, the article is down).

Standard Form Letter

[Golthur (754920)]

Your post advocates a

(X) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
(X) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
(X) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
(X) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
(X) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
(X) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
(X) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( ) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
(X) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(X) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!

Re:Standard Form Letter

[Linux_ho (205887)]

I'll add my own here:

(X) Similar to DCC and Razor, but far less bandwidth efficient than either

You should also have checked:

(X) Users of email will not put up with it
(X) Requires immediate total cooperation from everybody at once

Bigger problem...

[Not_Wiggins (686627)]

What is one person's spam is another person's desired mail. I'm not talking about advertising, either. For example, I know for a fact that there are a lot of people out there that "knee-jerk" react to service messages from their bank, credit card, whatever... stuff they even signed up for that they mark as spam. Since I want to get my "your payment has posted" email, do I want to rely on the network of people around me that signed up for the same thing with the same company and report it as spam because they're too lazy to just unsubscribe?

I have no friends..

[Harald Paulsen (621759)]

..you insensitive clod!

missing the point

[cahiha (873942)]

Social filtering of email is already widely implemented. It's not implemented at the level of the end user email client, but at the level of email servers, which compare messages to users and blackhole addresses.

A P2P approach and querying of other people's address books has huge privacy and compatibility problems without any obvious advantages.

Re:Wondering if this works for mailinglists

[geoffspear (692508)]

If you're sending messages to email addresses that didn't actually subscribe then yes, you're a spammer and you should be blocked.

A well-designed opt-in list won't have any fake addresses on it (although it may have messages to invalid addresses bounce is once-valid accounts stop working), because anyone with half a brain designing an opt-in list would require the addresses it's mailing to be validated by the recipients of the messages before sending them anything.