SpamNet: Razor for the Masses

UCRowerG writes "From CNET News on Yahoo!: "Conceived by Napster co-founder Jordan Ritter and open-source developer Vipul Ved Prakash, the company is touting the benefits of democracy, networking and collaboration in the war against unscrupulous e-mail marketers." " Since Prakesh is responsible for Razor, hopefully there will be Linux support as well, but once again I gotta throw my props at Spamassassin which catches over a hundred spam for me each day.

Here's the URL...

[EnglishTim]

http://www.cloudmark.com/ [cloudmark.com]

... because the guy who posted this obviously couldn't be bothered....

Re:Signatures?

[torpor]

Its way more complicated than that. Just read the "whats new" page for a good summary:

http://razor.sourceforge.net/docs/whatsnew.html

I'm frankly quite happy to see Razor come to fruition.

I had exactly the same idea for how to do this (with distributed signature databases) in '93 when I started a well known ISP. The plan was to offer spam-killing as a second-tier service to offer our customers, but alas: at the time, it was considered by management (read: VC) more profitable to allow open spam relays to our subscribers than it was to try to get subscribers to pay for a service like this, so the implementation details went nowhere.

Excellent to see it come to light in the form of working code, OSS style.

Spamassassin over Spambouncer

[waldoj]

I've run both Spamassassin and Spambouncer. For the curious, I prefer Spamassassin, and here's why.

I was very impressed with Spambouncer. It was the first spam-heuristic system that I'd used (previously, I'd relied solely on MAPS, ORBS, ORDB, RBL, etc.), and I was very impressed. I found that it rejected a lot of legitimate mail until I grepped my "Sent Items" folder, extracted every "To" field and made that my white list. (The assumption being that if I've e-mailed somebody, I don't mind hearing from them.) That worked very well, and I was happy with Spamassassin. The odd piece of spam would get through, and I still had 1:100 legitimate messages get put in my spam folder. But it made my life much simpler.

Then I tried Spamassassin. The big reason was because I wanted to take part in Razor and know that I was a part of a collaborative process. Also, Spambouncer hadn't been updated in months, which struck me as odd. But I also just wanted to try something different. I found that Spamassassin was better. Not in a way that made Spambouncer look bad, it was just clear that Spamassassin was a superior product. For example, Spamassassin provides a complete scoring in the headers, so you know exactly what criteria caused the message to be block. And I never had to set up a whitelist -- it just works. I still get that tiny little bit of spam that gets through, no more or less than with Spambouncer, but that's really not a complaint. It's very, very rare that a legitimate piece of mail gets caught up in the system. Best of all, the nonexistent addresses on my system that spammers have somehow discovered ([e-mail], [e-mail]) can be forwarded via my aliases table to Spamassassin's (Or is it Razor's? I forget.) server to be automatically added to their honeypot collection.

I'll stick with Spamassassin, I think. It appears to be the most mature, stable, simple, straightforward spam filtering product available today. For those looking to set up server-side spam filtering, I highly recommend it.

-Waldo Jaquith

Have Your Cake and Eat It Too

[pjrc]

I've been using Razor with Spamassassin for many months. All you need to do it install the razor package (and the various perl modules it wants), and then add a line like this in your .spamassassin/user_prefs file:

score RAZOR_CHECK 5.0

I've also got the other "network tests" enables (blacklists), but I assign them low scores since they have a lot of false positives.

Using spamassassin with razor and the blacklists really works. My spam file has 836 spams automatically filtered between March 1 to today, June 19. Of those 836 messages, 511 have the RAZOR_CHECK string in the "X-Spam-Status" line that spamassassin adds to the header.

Not too bad, considering Razor uses a rigid message digest that fails if the spammer adds any "random" content to the messages. Saddly, it seems like that's becoming more common. Rumor has it that Razor is someday going to use "fuzzy" matches with one of two algorithms that somehow accomplish such a feat. Anyone know when/if this is supposed to happen??

Re:Signatures?

[ftobin]

Excellent to see it come to light in the form of working code, OSS style.

Only the client of Razor is OSS. The author has explicitily stated that the server will not be released under an Open Source license. This is exactly why I'm implementing Pyzor [sourceforge.net], which is a razor-like system but where both the client and server are released openly under the GPL.

Re:Alanis would love this.

[thesolo]

Or if you want to keep using the IE engine for compatibility

Translation: If you want to keep seeing poorly-coded sites remain poorly-coded sites, use the IE engine.

The more people that use Mozilla, the more web designers have to code for standards compliancy. Using IE or IE's engine will only result in a monopoly on the web.

Spamcop uses the 'collaborative method' too.

[Chmarr]

I use Spamcop to filter my incomming mail at the MTA level, and I've been exceedingly happy with it. Apart from one or two that 'slip through', the only spam I receive nowdays comes through MTAs I have no control over.

Quick brief on how it works. There are two portions:

- Reporting tool, that allows you to forward spam to SpamCop for analysis. This will pick apart the headers and body, find out where the spam originated from (even if it's gone through legitimate relays and aliasing systems, such as mailing lists), and will send complaints to the relevant owners of the IP block owners, MTAs and web sites. It does a VERY good job of figuring out who's responsible.

- Blocking tool that uses a RBL-style blocking list, which lists IP addresses of spam originators. If enough spam gets reported within 24 hours, the IP sending the spam gets added to the list. You can use this to block addresses where spam has originated from so you dont even receive the spam. People get their IP addresses unblocked only if spam stops being sent from that IP.

The system is very good. It relies on you and others reporting spam to SpamCop in a very workable collaberative effort.

http://spamcop.net/ [spamcop.net]

Alternative to SpamAssassin that Uses Razor

[al3x]

Check out Blackhole [groovy.org] by the Groovy Organization, which integrates really well with Qmail but will work with just about any MTA. I found SpamAssassin's documentation to be mediocre at best, and had a helluva time getting it operable. Blackhole worked right off the bat for my Qmail/Courier IMAP/OpenBSD 3.0 setup, and can use Razor amongst other filtering methods. The software is constantly updated, and the developer plesant and responsive. Give it a try!

Pyzor for an open source *server*.

[Moderation abuser]

Pyzor works in a very similar way to Razor, but the client and server are open sourced. The Razor *server* is not open sourced.

http://pyzor.sourceforge.net/

Oh, BTW, Spamassasin *uses* Razor.

Spam filtering +Virus scanning = MailScanner

[M1m3R]

I've got MailScanner...
http://www.sng.ecs.soton.ac.uk/mai lscanner/
...run ning on my personal server. It's a "wrapper" for SpamAssassin and a number of server-side virus scanners. Pretty cool tool.

Re:Hey Ximian!

[Azog]

Yes, it would be AWESOME if Evolution just had a checkbox in the mail preferences dialog where you could turn on SpamAssassin or other filtering programs.

However, I have SpamAssassin working with Evolution now. It was kind of a hassle to set up but it works... here's the overview:

- get and install SpamAssassin, test that it works by piping a good email and a spam email through it
- check that my fetchmail works, write a .fetchmailrc file
- check that my procmail works, write a .procmailrc file
- disable the regular pop mailboxes in Evolution
- add a new "local delivery" mailbox to Evolution
- wrote a tiny script I called "getmail" that does "fetchmail -m procmail" and make sure that it gets my email from the POP3 server correctly
- added the getmail to my crontab to run every 5 minutes
- added a filter rule to Evolution: if specific header X-Spam-Flag = YES, drop the email in my Spam folder

and that was it. :-/ No more spam! But you can see why having this integrated into Evolution would be nice.

- - - - my .fetchmailrc - - - -
poll mail.arnor.net
user "slashmail" password "secret" is user "thoffman" here
- - - - my .procmailrc - - - -
:0fw
| spamassassin -P
- - - - my "~/bin/getmail" script
#!/bin/bash
/usr/bin/fetchmail -m /usr/bin/procmail >> ~/log/fetchmail
- - - - - my crontab - - - - -
*/5 * * * * /home/thoffman/bin/getmail
- - - - -

Re:Nilsimsa's popularity will be its own demise

[Paul Wright]

There are faster search algorithms than the obvious linear search. The creator of nilsimsa suggested one, and I implemented it to see how fast it was. I found it would be workable [dynip.com] for a DCC-like system holding on to 2 million digest codes at a time, handling about 3 million messages a day. The server would need about half a gig of physical memory to retain the entire database in memory. Since a Razor system only holds spam digests rather than the digests of all mail, I imagine it'd be OK for that, too.

I BSD-ified my example code so if anyone wants to use it, feel free.

Achilles heel

[pizza_milkshake]

In an interview, Prakash and company CEO Karl Jacob said Cloudmark's software solves the problem of identifying spam and quickly updating e-mail filters by harnessing the intelligence of the Web community at large.

Ad that is why it will never work.