Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
×
Spam Security IT

Using Email Networks as P2P Spam Filters 108

Oscar Boykin writes "New Scientist is running a story on using the social network in email as a P2P network. The idea is that email networks have structure that is conducive to a type of search called percolation search . This means email clients could query the social network of email users to filter spam. This story is based on a preprint available."
This discussion has been archived. No new comments can be posted.

Using Email Networks as P2P Spam Filters

Comments Filter:
  • Secure? (Score:5, Interesting)

    by geomon ( 78680 ) on Thursday May 12, 2005 @03:03PM (#12512150) Homepage Journal
    The authors propose that their system have access to inbound and outbound contacts. For trusted email accounts, that might work. But what about email accounts that people may want to creat to sheild their identity (political dissidents, whistleblowers). They would have to live outside of the spam protection network and would, I assume, be seed accounts for spammers.

    Am I missing something in this analysis?
    • Re:Secure? (Score:5, Interesting)

      by seoYak ( 883591 ) on Thursday May 12, 2005 @03:08PM (#12512207)
      I don't think that i'll trade my privacy for a reduction in spam.
    • Wouldn't the whistleblowers and political dissedents just have to email and be read by a few trustable sources to become trusted themselves? I guess the hard part would be to get past the filters of those first few, but it would be possible after that.
      • I guess the hard part would be to get past the filters of those first few, but it would be possible after that.

        True. I guess my concern would have been whether their proposed system could be mined for information regarding frequency of connection between two emailers.
    • by spectrokid ( 660550 ) on Thursday May 12, 2005 @03:26PM (#12512441) Homepage
      You could collect email adresses in a hashed form, just like passwords are stored on a server. You would be able to check if the sender is in the list, but not be able to "un-hash" the list back into real adresses. The way to get around it would be for spammers to attach their sender adresses to these funny mails people do like to forward to their friends.
    • Re:Secure? (Score:2, Interesting)

      by rescendent ( 870007 )

      Also you would not be able to be emailed by people who you haven't already approved the email address; would they have to phone you first?

      For example:
      People who change email address (Gmail, dropping a spammed email)
      People legitimately contacting you (Old friend, people wanting to know more about your website etc.)
      etc.

      It would be like setting your telephone to only accept certain phone numbers and scrapping the phonebook. Bad for people, terrible for business.

      Though I suppose spam is worse because

  • by PenguinBoyDave ( 806137 ) <david AT davidmeyer DOT org> on Thursday May 12, 2005 @03:03PM (#12512156)
    Since switching to Thunderbird, I get nearly no spam...maybe one or two per day. I like fancy stuff, but when simple works, go with it!
    • One or two per day out of how many? 3? 5? 1000?
    • and for webmail, shinyfeet [shinyfeet.com] has the best spam filtering I have experienced. fancy or not, it certainly works (and with their unlimited setup, you'd need something that works)
    • by Dukael_Mikakis ( 686324 ) <andrewfoerster AT gmail DOT com> on Thursday May 12, 2005 @03:15PM (#12512322)
      I use gmail, which does an excellent job at filtering spam.

      But I think this could even be a step back. Like the parent says, I think most informed people have solved the issue of filtering spam pretty effectively (Thunderbird, Yahoo, Gmail, Bayesian filters, etc.) and so we don't generally *see* much spam.

      The *REAL* problem with spam is traffic and network pollution. Spam wastes a ridiculous amount of bandwidth and (through spyware) hijacks our systems' cycles to do something that is (with filters) ultimately to no end. This seemingly won't solve the bandwidth consumption issue and might worsen the problem by polling all your friends over the network and then using your personal cycles to scan said email against all the known spam on your friends' computers.

      People forget that the true detriment of spam these days is the traffic it causes, not cluttering your inbox (if you're smart).
      • by Anonymous Coward
        I agree with the main point of your post, but wanted to address something you said.

        I use gmail, which does an excellent job at filtering spam.

        I see this stated here on /. a lot. While gmail seem to be improving in this regard, I haven't had the experience of excellence.

        A couple months ago the majority of my spam was actually legitimate email from my mailing lists. As of this moment, I don't see any legitimate mail in my Spam folder.

        However, about 20% of the actual spam I get ends up in my Inb
      • The Real problem of bandwith usage will go away by itself when enuf people filter properly. If sent spam is not seen by anybody it will cease to be effective and will become worthless to advertisers.
        if nobody pays for spam, nobody sends spam.

        • That's a nice theory, but it seems more likely that the more effective spam filtering gets, the more spam will be sent. If it takes 100x more messages to get the same results, the spammers will just send 100x more messages. And they'll need to turn even more machines into zombies to do it.
          • If all they do is send more spam that the filters trap, then all that will happen is spam traps will be more full. Winning such an intractable arms race, where the defense scales so much better than the offense, requires the attacker to get much smarter. While evolutionary selection might suggest they will both get smarter, in some predator/prey equilibrium, the economics suggest the defense has the advantage. It's cheaper in the aggregate to defend from multiple spam attacks with cheaply distributed filter
        • I am not keeping my hopes high on this. The response rate for the regular snail mail is about 1%, that is 99% of the junk mail people receive basically goes to trash. Still, corporate America is willing to spend more than $46 billion on direct mail marketing every year. Considering that postal junk mail is a lot more expensive to send than e-mail spam, a 0.1 or even 0.01% response rate for e-mail spam may be sufficient to sustain the current spam rate, if not continued growth.
        • I would agree but the incremental cost of sending spam is basically free ... especially if people let spyware hijack their computers to do the heavy lifting.
        • Better filtering has to be on the network level, so that individuals don't have to opt in. The people who know how to set up effective filtering won't likely buy anything from spamvertisers in the first place, meaning that, while it makes those peoples' lives easier, it doesn't affect the spammers' bottom line much.
      • People forget that the true detriment of spam these days is the traffic it causes, not cluttering your inbox (if you're smart)

        You've got to be kidding. Spam is text, or very nearly so (HTML). Unless you are using floppies and a 2400bps modem, the bandwidth/storage costs are irrelevent.

        What is relevent is that it forces people to either use spam filters that randomly throw good messages away or they miss good messages becuase they can't be seen among all that spam. In either case, the loss rate goes fr
    • How does Thunderbird provide any protection? When I was using it it didn't seem to have any spam filters.
      • It does have a built-in bayesian filter for a long time now. When were you using it?
      • 1.0.2 has a *junk* filtering system. When you turn it on it will go through your inbox and mark what it THINKS is junk. It them moves the *junk* to the junk folder where you can review it. If you find something that is not junk, you can click "not junk" and move it back to the inbox. Once I "trained" it for about a week, I have never found anything I wanted in the junk folder, and like I said, I only get about one or two spam messages in my inbox per day, compared to the 150+ that end up in the junk fol
    • I've been using thunderbird since 0.7 or so, and I'm not entirely in love with it's spam filtering. I've had it turned on since the feature was introduced, so it's had a long time to learn what is spam (95% of my email) and what's not (the other 5%). My most recent download of my earthlink email (which i've just about given up on), had about 60 messages, only 1 of which wasn't spam. But of the 59 spam, 17 weren't marked as spam (despite me junking very similar emails a few days earlier). Perhaps the spa
  • Great... (Score:5, Funny)

    by yotto ( 590067 ) on Thursday May 12, 2005 @03:06PM (#12512185) Homepage
    ...Now the RIAA's going to sue me for getting spam.
  • Potential for harm (Score:4, Insightful)

    by davidwr ( 791652 ) on Thursday May 12, 2005 @03:07PM (#12512202) Homepage Journal
    Imagine the potential for harm if I infiltrated a social network and then identified my enemies as spammers, either deliberately or because I or the software agent I use was somehow tricked into doing so.

    Social network-based spam-detection is a part of, not a total, solution, and its limits need to be recognized.
    • This isn't spam-detection on a user basis (e.g. a network with "these email addresses are trustworthy and these email addresses are spammers") it's spam detection based on a spam database.

      The potentials for abuse that I see are if you don't keep a spam database at all, and so you will not flag any query as spam (even if it clearly is), or if you try to keep an "Anti-Spam" database (I dunno, a database of legitimate emails. Only problems with these abuses are if you have no database it shouldn't matter be
    • Yes this is a very dangerous tool in the hands of a spammer or just about anyone with a little brains. But just about any communication tool anti-spammers create to be used on a wide public basis, can and will be used by spammers against those who created it and use it.
  • We do a mailing once to twice a week and there are hundreds of wrong/fake mailaddresses on it. We do not filter this, the mailserver dumps the returns. However now we are already marked sometimes as spam, while we craft the messages in such a way that at least spamassassin does not mark it as spam. It is a user subscription list, certainly not spam. This kind of solution will make this situation worse.
    • by geoffspear ( 692508 ) * on Thursday May 12, 2005 @03:52PM (#12512723) Homepage
      If you're sending messages to email addresses that didn't actually subscribe then yes, you're a spammer and you should be blocked.

      A well-designed opt-in list won't have any fake addresses on it (although it may have messages to invalid addresses bounce is once-valid accounts stop working), because anyone with half a brain designing an opt-in list would require the addresses it's mailing to be validated by the recipients of the messages before sending them anything.

    • It sounds like you want people to think you are a "responsible" sender of email, but then you admit the following:
      • Your mail server throws away bounce messages, rather than routing them to a human being who can remove the bad addresses from your list. Every bounce message you ignore is essentially stolen bandwidth and CPU on somebody else's server- if you were truly a responsible mailer, you would take steps to minimize your impact on others' systems.
      • You have to "craft the messages" to get around spam
      • Taking texts out of context is your hobby I guess, anyway a reply:
        If you get enough trash back because of users, the nicest way is to let the mailserver handle it. A CPU can do the dumping a lot faster than a person can lookup an account, and take the person of the mailinglist

        The spamassassin side of the story: We do not like to send out a plain text message, but nice HTML formatted messages. We take care that this requested e-mail is not mistaken for spam by already routing it through a filter to preven
        • If you get enough trash back because of users, the nicest way is to let the mailserver handle it. A CPU can do the dumping a lot faster than a person can lookup an account, and take the person of the mailinglist

          Again you state that your mail server is "dumping" the bounces as they come in. Does your definition of "dumping" mean it's removing the bad email addresses from your list?

          If you are removing the bad addresses when bounces come in, whether manually or using an automatic process, then you need

  • by forevermore ( 582201 ) on Thursday May 12, 2005 @03:10PM (#12512243) Homepage
    Granted, I just skimmed the article, but isn't this exactly how Razor works? (simplified) Communities of people flag messages, senders, etc. as spam, and the mail server (or in my case, spamassassin) compares the messages to the community spam archive for matches before delivery.
  • by CrazyJim1 ( 809850 ) on Thursday May 12, 2005 @03:11PM (#12512263) Journal
    You click a multi-user message as marked as spam, then it turns into spam for everyone else too.
  • by tdvaughan ( 582870 ) on Thursday May 12, 2005 @03:13PM (#12512285) Homepage
    According to the article the method works by asking its network of email users if they've seen the spam before:
    Similar software on each computer that receives the query would then check the message against its own spam database, and so on, until a match is found, or the message is deemed original.

    So it can't deal with spam that includes a unique random ID and would tag emails from a mailing list as spam. Once more: nice try, but it won't work in the real world.
  • Ob (Score:5, Interesting)

    by lheal ( 86013 ) <lheal1999NO@SPAMyahoo.com> on Thursday May 12, 2005 @03:13PM (#12512286) Journal
    In Korea, only old people get P2P spam.

    Actually, I think we should find a way to attach the same stigma to spam customers that we do to the spammers. Why do spam customers not have to go to jail? They're as much the problem as the spammers.

    I can see something like having all the spam customers' names published online, so you google for "spam" and "lheal" and up pops my list of purchases. The other spammers then get a very clean list of people to spam. Over time, the net would be segregated into those who like spam and those who don't.

    Yeah, unworkable idea, but so are all the others.
    • I think we should find a way to attach the same stigma to spam customers that we do to the spammers.

      Nah... Congress can't make stupidity illegal; they'd lose too many votes. The universe, not being elected, can... but tends to be in favor of capital punishment as a way of preventing repeated behavior.

      An utterly illegal and unethical solution would be to start up a V1AGRA spam outfit, and taint the supply so that one pill in twenty was actually a disguised lethal dose of cyanide. This would cut into dem

    • I used to keep a list of spam customers and refuse to patronize them. I still avoid them when I can, but have totally lost the ability to keep the list in my head. In my old age, about all I remember for sure is I'll never buy an x-10 camera thingy.
  • Hmmm... (Score:3, Funny)

    by __aaclcg7560 ( 824291 ) on Thursday May 12, 2005 @03:15PM (#12512311)
    When you thought it was safe to use email again...
  • What strikes me is that the idea of "pooling information" isn't really new. When one yahoo-mail/HotMail/Gmail user marks a particular mailing as spam, it affects the likelihood that the same email would be marked as spam for other yahoo users. So, the idea of "pooling information about spam" (from article) is already in use! However, it would be nice to create explicit protocols to allow such data (what mailings I have marked as spam) to be made public so that people using other email providers or their own mail servers can share in this pool of knowledge. Of course, the big three email providers (yahoo mail, hotmail, and gmail) will be foolish to make this information public: the spam filtering is one thing that makes a yahoo/gmail account more attractive to potential users! Good idea in theory, but bad business prospects. To add insult to injury, there is no way for the researchers to profit from the arrangement.
    • ... (yahoo mail, hotmail, and gmail) will be foolish to make this information public: the spam filtering is one thing that makes a yahoo/gmail account more attractive ...

      Don't reject that idea so quickly, as I think you're on to something. The protocol would encapsulate the information that "userx@foo.com marked this message as spam". What the email provider does with that information is something else.

      Not only that, but Google and Yahoo! could team up against Hotmail, or AOL, or whoever. Maybe

  • This just in, Spammers now use virtual social engineering to clog new P2P network spam filtering, Inboxes bulging with "extra inches"
  • by overbom ( 461949 ) <overbom AT yahoo DOT com> on Thursday May 12, 2005 @03:21PM (#12512382)
    If I were a spammer:

    I'd change an email client to respond with any message from certain folks I don't like to report all of their messages as spam to poison the social network. a couple of clients out there saying "yup, I've already got a message like that here, and my user marked it as spam".

    think globally, act locally, right?
  • by Otto ( 17870 ) on Thursday May 12, 2005 @03:23PM (#12512403) Homepage Journal
    This isn't a new idea... except that they propose to integrate it into the mail client and have everybody you've ever sent mail to or received mail from be a potential contact, weighted by frequency that you email them. That's a bit new, but not as effective as it seems.

    For one thing, it would block mailing list messages, which are messages that you probably do share with your contacts.

    For another, it does not consider that most spam has random keywords seeding into every copy sent, so those would have to be ignored somehow, which introduces a fuzzy match algorithim, which means the possibility of false matches exists, and since you're asking others (probably all using the same algorithim against their databases) you have increased the chances of a false match being found.

    In any case, collaborative networks already exist in a better form. Users mark messages as spam when they get them, a flag is created and sent to some central place that all users check against for matches. The algorithim for fuzzy matching resides in one place and is only used as an indicator in spam assassin in any case, not as the sole indicator..

    Large scale systems like Google's GMail can use people flagging messages as spam to filter similar enough messages from other users, sort of thing. I'm pretty sure they do something like this, in fact, as my GMail account has *never* made a mistake in it's spam detection.

    And so forth. There's better ways than relying on a random query of your contacts to see what they think.
  • by cloudmaster ( 10662 ) on Thursday May 12, 2005 @03:24PM (#12512417) Homepage Journal
    Sorry, I can't read the article. There were too many buzzwords in the post.
  • I've already had this idea for years, seriously, I almost made it my graduation project.

    Create a distributed spam filter that fingerprints incoming mail based on a number of criteria, have the user mark spam with a certain 'undesiredness factor', blacklist email fingerprinted as spam and propagate this information to other people using the same system... This way it should be possible to create some kind of 'network' that classifies email much more reliable than a simple content filter or address blacklist
  • by sPaKr ( 116314 ) on Thursday May 12, 2005 @03:26PM (#12512442)
    Skipping past the security issues. One of the goals of spam filters should be reducing network load not increasing it. If we have to send our spam to several differnt peers to be scored this would compound the network load problems. Mostly this is a bad idea(tm) from the get go. I think the only thing that will really stop spam is to force something like pgp(gpg) signatures on all mail. Here's hoping the new national ID cards will have public certs encoded on them. It would be cool if someone would step in and get PKI working for the rest of us. Also we should drag the boddies of spammers through major cities behind a horse, while allowing victums to beat the spammer with large sticks like golf clubs.
  • Similar software on each computer that receives the query would then check the message against its own spam database, and so on, until a match is found, or the message is deemed original.

    Spammers can ranodomly generate content for their spam to bypass this. Have the actual spam message text as a JPEG image followed by random, gibberish text in both the same background and foreground colour so it is invisible. If the system looks for messages that are identical by comparing the text, then spam messag

    • Generating random text, and attaching a jpeg doesn't work for two reasons.

      First, the random text doesn't look like legitimate e-mail, so it will be completely ignored by most spam filters. That leaves a jpeg attachment and bogus headers, which out to look pretty 'spammy' to a mail filter. How often do total strangers send you legitimate email that contains nothing but a JPEG and text that doesn't have any 'important' words in common with your other mail?

      Second, if you're like me, then your mail client i
  • Sounds Like SpamNet (Score:3, Informative)

    by MBCook ( 132727 ) <foobarsoft@foobarsoft.com> on Thursday May 12, 2005 @03:36PM (#12512550) Homepage
    That sounds like Cloudmark SpamNet (I think that was what it was called). I used it a few years ago when it was in beta and it worked great. The idea is people marked mail they got as spam if it was. When they did that, a hash of the message (or title, or something like that) was sent to their server. When your mail came in, it was hashed and checked to see if it was spam. It was VERY accurate. It had only one problem:

    Cloudmark.

    I signed up for the free beta and was told that it would be free forever (they were going to charge businesses, IIRC). Then they chagned their mind but said that early adpoters/beta users would get it free for life. Then it left beta and they offered me a $5 discout (one time) for their subscription service (or some other pointless trinket offer like that). As far as I'm concerned they ripped me off.

    That set me off trying other things, and I eventually found POPFile, which I use to this day (great software). I've posted this to Slashdot before (a long time ago). Some nice guy from a anti-spam company gave me a code for a free version of their product to be nice (I never used it, I had found something by then and didn't feel like switching again).

    The point of all this is that it is a nice method that really works. If there was an open source project that did the same thing, I would use it. Untill then, I've got a solution that works fine.

    But this isn't new (if I'm right about what it is, the article is down).

  • by Golthur ( 754920 ) on Thursday May 12, 2005 @03:56PM (#12512777)
    Your post advocates a

    (X) technical ( ) legislative ( ) market-based ( ) vigilante

    approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

    ( ) Spammers can easily use it to harvest email addresses
    (X) Mailing lists and other legitimate email uses would be affected
    ( ) No one will be able to find the guy or collect the money
    ( ) It is defenseless against brute force attacks
    (X) It will stop spam for two weeks and then we'll be stuck with it
    ( ) Users of email will not put up with it
    ( ) Microsoft will not put up with it
    ( ) The police will not put up with it
    ( ) Requires too much cooperation from spammers
    ( ) Requires immediate total cooperation from everybody at once
    ( ) Many email users cannot afford to lose business or alienate potential employers
    ( ) Spammers don't care about invalid addresses in their lists
    (X) Anyone could anonymously destroy anyone else's career or business

    Specifically, your plan fails to account for

    ( ) Laws expressly prohibiting it
    ( ) Lack of centrally controlling authority for email
    ( ) Open relays in foreign countries
    ( ) Ease of searching tiny alphanumeric address space of all email addresses
    ( ) Asshats
    ( ) Jurisdictional problems
    ( ) Unpopularity of weird new taxes
    ( ) Public reluctance to accept weird new forms of money
    ( ) Huge existing software investment in SMTP
    ( ) Susceptibility of protocols other than SMTP to attack
    ( ) Willingness of users to install OS patches received by email
    ( ) Armies of worm riddled broadband-connected Windows boxes
    (X) Eternal arms race involved in all filtering approaches
    ( ) Extreme profitability of spam
    (X) Joe jobs and/or identity theft
    ( ) Technically illiterate politicians
    ( ) Extreme stupidity on the part of people who do business with spammers
    ( ) Dishonesty on the part of spammers themselves
    (X) Bandwidth costs that are unaffected by client filtering
    ( ) Outlook

    and the following philosophical objections may also apply:

    ( ) Ideas similar to yours are easy to come up with, yet none have ever
    been shown practical
    ( ) Any scheme based on opt-out is unacceptable
    ( ) SMTP headers should not be the subject of legislation
    ( ) Blacklists suck
    ( ) Whitelists suck
    ( ) We should be able to talk about Viagra without being censored
    ( ) Countermeasures should not involve wire fraud or credit card fraud
    ( ) Countermeasures should not involve sabotage of public networks
    ( ) Countermeasures must work if phased in gradually
    ( ) Sending email should be free
    ( ) Why should we have to trust you and your servers?
    ( ) Incompatiblity with open source or open source licenses
    (X) Feel-good measures do nothing to solve the problem
    ( ) Temporary/one-time email addresses are cumbersome
    ( ) I don't want the government reading my email
    ( ) Killing them that way is not slow and painful enough

    Furthermore, this is what I think about you:

    (X) Sorry dude, but I don't think it would work.
    ( ) This is a stupid idea, and you're a stupid person for suggesting it.
    ( ) Nice try, assh0le! I'm going to find out where you live and burn your
    house down!
    • I'll add my own here:

      (X) Similar to DCC and Razor, but far less bandwidth efficient than either

      You should also have checked:

      (X) Users of email will not put up with it
      (X) Requires immediate total cooperation from everybody at once
      • Yeah, that's what you get when you try to fill out the form and have it posted before someone else does :)

        I didn't know if users of email would put up with it or not, so I didn't check that one.

        I definitely should have added my own option for consuming tons of extra bandwidth per spam, though - this thing would make the existing bandwidth use of spam look like a raindrop compared to the ocean...
  • Bigger problem... (Score:3, Insightful)

    by Not_Wiggins ( 686627 ) on Thursday May 12, 2005 @04:05PM (#12512898) Journal
    What is one person's spam is another person's desired mail. I'm not talking about advertising, either. For example, I know for a fact that there are a lot of people out there that "knee-jerk" react to service messages from their bank, credit card, whatever... stuff they even signed up for that they mark as spam. Since I want to get my "your payment has posted" email, do I want to rely on the network of people around me that signed up for the same thing with the same company and report it as spam because they're too lazy to just unsubscribe?
  • I have been using Spamihilator for a while now, with the DCC plug-in activated, that checks a fuzzy check-sum of the message with servers that hold a list of other users who use this filter. I have found that it does block a number of newsletters that large numbers of people receive, however a simple list of newsletter definitions do a good job of preventing this problem. I just put this filter with a DNSBL filter that checks Spamcop and other blacklists, and a learning filter, with no spam reaching my inb
  • ..you insensitive clod!
  • In the last few months, as gmail's customer base has grown, their spam capturing capabilities have reached about 99.5% with a 0% false positive rate. And I get about 100 spams per day. It has been weeks since Gmail last falsely identified an incoming spam for me.

    This type of searching (i.e efficiently searching through a long-tailed distribution) my contacts and archived mail is probbaly just one part of the equation - only about 25% of my email is from other gmail users. But nearly all of my legit email i
  • Here is something I thought you spamassassin users would find useful. Teaching the bayesian filter is difficult when just starting a new mail server, so here is a couple years of archived spam mail for you. Run these through 'sa-learn' and you'll have reduced your spam quite a bit: Spam Archive [www.em.ca]
  • Social filtering of email is already widely implemented. It's not implemented at the level of the end user email client, but at the level of email servers, which compare messages to users and blackhole addresses.

    A P2P approach and querying of other people's address books has huge privacy and compatibility problems without any obvious advantages.
  • Yet another fake eBay site.... (>_<);

    I entered a bogus but properly formatted CC# but it appeared to reject it. Oh well. Enjoy the relevant information and use it to avoid being duped....

    The phish was sent from 80.247.227.76 in France through a redirect page at href=http://projekt.ig-immobilien.com/signin.html in Germany to the phish site itself at:

    href=http://61.190.66.139/ws/index2.php?MfcISAP I Co mmand=SignInFPP

    in China via the phish email link:

    href=https://signin.ebay.com/ws/eBayISAPI.dll?S

Saliva causes cancer, but only if swallowed in small amounts over a long period of time. -- George Carlin

Working...