Spam Catchers Block Latest Crypto-Gram 246
An anonymous reader writes "Bruce Schneier sent out a note about SpamAssassin and possibly other spam filters blocking his excellent Crypto-Gram newsletter. Fortunately you can get it here (early no less!)." Schneier's email reads, in part "Tomorrow I will be sending out the February CRYPTO-GRAM, as I do on the 15th of every month. In the process of creating this month's Crypto-Gram, I discovered that SpamAssassin thinks that this issue is spam, probably because of certain links and descriptions of scams in the text. I have anecdotal evidence that other spam filters block Crypto-Gram as well. ... I'd apologize for the inconvenience, but I'm not sure what I could do to make it less so -- I don't intend to alter my content to accommodate spam filters."
Hopefully SpamAssassin didn't (Score:5, Funny)
um, i could be terribly wrong here (Score:4, Interesting)
In principle, yes, in practice, no. (Score:2, Insightful)
Re:In principle, yes, in practice, no. (Score:2)
Re:In principle, yes, in practice, no. (Score:4, Insightful)
You are right that it would not be encryption in the sense that it doesn't protect privacy of the message (indeed, in order to read the message, you only need Bruce's public key, which is indeed, uhmm, public...).
However, it would still fulfull the goal of evading spamassassin, because, as far as I know, spam assassin is not yet smart enough to figure out that the message has been "encrypted" with Bruce's private key, and to fetch the public key from the Bruce's webserver to decrypt it.
But then again, rot13 would probably be enough to evade spamassassin too... as long as you don't mispell inventive as ivntenive that is...
Re:In principle, yes, in practice, no. (Score:2)
Re:In principle, yes, in practice, no. (Score:3, Informative)
Re:In principle, yes, in practice, no. (Score:2)
Just FYI -- it's quite possible with RSA but not with DSA. DSA can be used for digital signatures only, not for general encryption.
Re:um, i could be terribly wrong here (Score:3, Informative)
See, when you PGP encrypt some text, it is only possible to encrypt it to one person (one public key). That's just how it works, it's inherent in the encryption methods used; however, PGP and GPG get around this by duplicating the entire message for each public key that it is encrypted to.
My point is that if you had a mailing list with 1000 subscribers, and you wanted to encrypt it, you'd basically be increasing the size of the encrypted message 1000-fold, because you need 1000 copies of the message, each encrypted to a given recipient. Obviously, this isn't feasable...
What they could do, though, is sign the messages. I know SpamAssassin, at least, reduces a message's spam score if there is a PGP signature attached to it.
However, if you were just trying to obscure the contents of the mail from the spam filter but not the user, you could just gzip the message and make it an attachment. I don't know how well that would go over with the spam filter, but at least it wouldn't find your m/blow.*job/s in the message
Re:um, i could be terribly wrong here (Score:2)
My point is that if you had a mailing list with 1000 subscribers, and you wanted to encrypt it, you'd basically be increasing the size of the encrypted message 1000-fold, because you need 1000 copies of the message, each encrypted to a given recipient. Obviously, this isn't feasable...
Actually it's trivial. You encrypt with a private key then anyone who has the public key can decrypt it.
This is how PGP/GPG signing of a message works. You have a checksum encrypted with a private key, when you receive the message the software attempts to decrypt the checksum then compare it with what it has calculated the checksum to be. If the decryption fails the message isn't from the claimed source, if the checksum fails it has been altered.
Re:um, i could be terribly wrong here (Score:3, Informative)
Incorrect. When PGP or GnuPG encrypts a message with a public key, they really just encrypt the message with a symmetric cypher and a sufficiently long, random key. Then they encrypt the key with the public key. (The reason for this is that public key cryptography is much, much slower than symmetric key stuff.) So for sending to multiple recipients, all that needs to be added is some additional header data for each recipient.
-rw-r--r-- 1 phil phil 212358 2003-02-16 13:01 original-rw-r--r-- 1 phil phil 90343 2003-02-16 13:02 one-recipient.gpg
-rw-r--r-- 1 phil phil 90893 2003-02-16 13:04 three-recipients.gpg
A better solution would still be to encrypt the message with a particular public key for which the private key was widely available. Encrypting the message with Bruce Schneier's private key makes sense cryptographically, but I don't believe PGP and GnuPG support that sort of behavior.
--Phil (Far too much of a crypto geek)
Re:um, i could be terribly wrong here (Score:2)
So, you're saying Bruce Schneier's private key is widely available? ;-)
Re:um, i could be terribly wrong here (Score:2)
Re:um, i could be terribly wrong here (Score:2)
Really? A lot of HTML spam used to encrypt itself and then have a little piece of javascript that decrypted it embeded in the email. This was quite easy to block, since all a spam-blocker had to do was spot emails not containing any words of your favourite language. I'm not sure if SA does this, but I wouldn't be surprised if it blocks all encrypted messages.
Seems like it worked fine.... (Score:5, Funny)
Re:Seems like it worked fine.... (Score:2, Interesting)
White List (Score:5, Insightful)
filtering software I've ever run, including spamassasin (which I like a great
deal) has a whitelist option. If you're running some kind of filtering
software, it behooves you to keep an eye on what it's blocking, hence, I am
sure that people are aware of it and have adjusted their software accordingly.
SealBeater
So let's send spam as Bruce Schneier (Score:3, Interesting)
Re:White List (Score:2)
Whitelist (Score:5, Interesting)
But you don't always want to get everything people send you (everybody has those people who send you things they think are funny but you just can't stand). So there should be levels of "friendship" in the whitelist, so that some senders can be considered dubious (their mail shouldn't be deleted like spam, but perhaps placed in a different "Uninteresting" folder).
Re:Whitelist (Score:3, Interesting)
I, for one, would love to see a feature like this in a mail program!
Actually, I'd like to participate in the development of an existing open source email app if someone could recommend one. Java based would be nice.
Re:Whitelist (Score:2)
SpamAssassin already does something like this. First, it comes with a set of whitelisted addresses, like ebay.com, etc. It's quite possible that Cryptogram will be included in that list next time around.
But better than that, SA has something called autowhitelisting that keeps track of the average spam score for people who sucessfully get mail delivered to you (through the filter). This means that a good friend, who's mails are normally fine, can send you a spam-ish mail that gets through, or if your friends are borderline, like above, they may only get a few points for being a friend and a spam-ish mail will trigger the filter.
Re:Whitelist (Score:2)
Java could easily handle 100+ emails a minute on a decent PC. Java is not dogslow. If you think it is, you haven't tried it in a looong time. The JIT compiler makes it run extremely fast, actually. What's slow is the startup time which is unimportant unless you're writing Java CGI's which nobody would do anyway for a high load app.
Classification - that's just filters. Cross-referencing sources on the 'net - Java's networking support is great.
Now memory - you got me there. Java does use a lot of memory compared to a native app, but these days with RAM prices so low I'm not too concerned about memory.
Re:open source java mail client (Score:2)
Sweet! Just what I wanted. Thank you very much.
Awesome sig OFFTOPIC, I know (Score:2)
It reminds me of another good one:
Arrogance is compensation for a lack of intelligence.
that I think was the same person, but I can't seem
to find either of those quotes.
Re:Awesome sig OFFTOPIC, I know (Score:2)
I do not know who invented the quote you mentioned (but I'm pretty sure it wasn't the same person... ie "not me"
And by the way: if anyone can find a classical source that actually invented my sig, I would be interested in knowing it. As far as I know, I invented it, but if someone else came up with it first, I plagiarised it completely unknowingly. If I didn't really come up with it, I would like to give credit where credit is due.
Re:Whitelist (Score:3, Informative)
Well, in terms of Spamassassin, you could create rules which subtracts a particular number of points from the spam score of any particular message, rather than letting it through automatically, which gives it a better chance to go through if it's a pretty un-spam-like content.
Re:Whitelist, header matchups and viruses (Score:3, Insightful)
We often see viruses and spam being send with spoofed sender address, and some spammers are clever enough to even use sender addresses from the same domain, which would be more likely to be on the whitelist. It would be nice to combine the whitelist with signature checking, if you know the senders public key, you simply filter anything unsigned.
SpamAssassinAssassin (Score:5, Funny)
I'm sure Paul Graham will be glad to write it in lisp.
Or, of course, we could just do what the obvious solution is: get in a P.O. Box, send out spam for herbal viagra and penis enlargement, and when you get the checks in the mail HUNT THE CUSTOMERS DOWN AND KILL THEM.
It's simple, really.
This is a non-issue.... (Score:5, Interesting)
I do tech support for a webhosting company and people call us every day complaining about their spam but as soon as we offer blocking software based on lists, etc all we get is complaints that some more-valuable-than-gold email is going to get lost and ruin their entire business.
This is a simple choice and people have to learn they can't have their cake and eat it too.
Re:This is a non-issue.... (Score:5, Insightful)
Huh. It occurs to me that it seems like some spam filters might pass a turing test if the only output is their spam judgment. Wow. The future is now, dude.
Re:This is a non-issue.... (Score:4, Insightful)
Except if it's done upstream from you, perhaps even without your knowledge (eg a few months ago it was found that Mac.com was aggressively filtering, with a lot of false positives).
The problem with filters (Score:5, Insightful)
obPlug: This is why I created Trustic [trustic.com].
Re:The problem with blocking IP adresses is... (Score:3, Interesting)
So blocking untrusted servers doesn't make email unreliable? I find that very hard to believe. Considering that most of the time it is Net blocks that are blocked, not just individual IP addresses.
blocking IP addresses is also open to abuse... If I had a grudge against an ISP, I could fake some SPAM headers and send it to any of the IP blockers. Maybe send several copies from different accounts. Getting an IP listed is usually easier than getting it removed, so in the mean time many legitimate emails are being blocked...
I believe you have to attack the root of the problem, and that is stopping the SPAM at the origin. This is probably the more difficult approach, but it is the only one that will avoid dropping legitimate mail.
Re:The problem with filters (Score:3, Interesting)
Re:The problem with filters (Score:2, Informative)
I have filters for all my mailing lists and so forth in my
The problem with content filtering (Score:5, Insightful)
This is exactly the problem with most content filtering approaches.
It is very hard to discern the difference between talk about sex, spam, viruses, etc and talk from sex, spam, viruses, etc. Newsletter authors go as far as writing "v*rus" and "sl*mmer" so that pitiful content filtering blocks don't trash them.
It gets even worse for email lists that use inline text ads. The ads alone would constitute spam, but they're nestled within several paragraphs of high-quality discussion.
The problem is that content filtering approaches usually only analyze the "spamminess" of a piece. They usually don't analyze the "goodness" of a piece. So if I put "hot teens go crazy for debt-free viagra while earning $$$ from home" in the middle of some fine Shakespeare, that will get flagged as spam.
The new "bayesian" approaches are finally dealing with this problem -- something can look an awful lot like spam, but it will be saved if it looks even more like legitimate email.
In this case, spam doesn't generally run for 21 pages with words like "cryptography," and "full disclosure."
Re:The problem with content filtering (Score:2, Funny)
Nor would you be wrong to insert that, since that's roughly the Cliff's Notes reduction of several Shakespeare plays.
Re:The problem with content filtering (Score:5, Funny)
eMerchant of Venice. Act I Scene IV, right?
Re:The problem with content filtering (Score:4, Insightful)
The problem with that is that if you score mail by the percentage of spam, rather than the absolute amount, the obvious response by spammers is to ADD 21 pages cribbed from a crypto newsletter to the end of their penis-enlarging spam. Maybe even fake the headers to make it look like it came from a respected source.
Maximum size of spam (Score:2, Interesting)
I have
Const maxspamsize = 42695
in my spam filter - I've only receive one piece of spam larger than than in the last 12 months (a giant promotion for a Korean trade show). It speeds up my spam filter processing and lets large newsletters (with false triggers like this) through without a problem.
Re:Maximum size of spam (Score:2)
Yes, it works now. But if your criterion were widely used, spammers would just bulk up their spam. (Now they often add some gibberish to the end, or even within HTML tags, so that it's not normally visible, and breaks up trigger words.)
Re:Maximum size of spam (Score:3, Interesting)
My point remains valid. Because there is a direct cost to the spammer to adapt.
If they bulk up their spam that's going to slow them down, increase their costs (even if bandwidth costs aren't going to be passed back to them now, the more they use, the more visible they become). They become more visible.
Or they continue on their way. The reality is that they concentrate on the easy targets - you and I will never purchase their services so people taking this approach aren't really in their target audience anyway. I know this is (surprisingly) less true than one might think. Spammers do work to overcome basic obstacles, but that adds more costs and time - they don't work hard to avoid tar pits, because there are so few of them.
So I still see it as a win...large emails are very unlikely to be spam. If that changes, well so be it, but that will hurt the spammers. In the meantime I reap the benefit of fewer false positives and faster spam filtering.
Final comment - over the last six months I've seen spam get slightly larger (from about 32k peak size to about 45k peak size). But I haven't been analysing for any trends - just the outliers.
Re:The problem with content filtering (Score:3, Insightful)
Re:The problem with content filtering (Score:3, Informative)
Not for long. Filtering software such as SpamAssassin is now being used at the server level to recognize junk email for thousands of clients.
For example, the University of Colorado at Boulder [colorado.edu] now uses SpamAssassin to scan all incoming student email. This means SpamAssassin handles the spam filtering needs of a student population of 30,000. There is no doubt that as the spam problem increases, filtering solutions will begin to appear at the ISP level.
Re:The problem with content filtering (Score:3, Interesting)
Re:The problem with content filtering (Score:2)
In this case, spam doesn't generally run for 21 pages with words like "cryptography," and "full disclosure."
Well, as a matter of fact, my bayesian filter marked the message spam, when I test-sent the html-file as an attachment to myself.
Re:The problem with content filtering (Score:2)
when I test-sent the html-file as an attachment to myself
That should explain it, although I could be wrong. If your filter doesn't look inside of attachments (which I think is the norm,) it could very well look like spam. If it does look inside of attachments, I think it's time to review the mail you trained your "spam" on. Many others, even those using SpamAssassin with its default threshold, have mentioned that the CryptoGram gets through.
Re:The problem with content filtering (Score:2)
The three spammiest words were "million" "trust" and "reports."
Re:The problem with content filtering (Score:2)
So I just code all incoming files with embedded HTML as spam.
And how exactly would you know? After all if I send you HTML mail you will never see it.
About 50% of my non-spam personal email is HTML. Of course the statistics might be off as I did help write HTML.
you just blocked MSN email (Score:2)
So I just code all incoming files with embedded HTML as spam.
It was reported just a few days ago that MSN emails now have ONLY an HTML mime part -- no more plaintext. Just like spam. Sure, it's fun to say things like "well, I don't, and wouldn't, have any friends on MSN", but that's just being juvenile, isn't it?
Re:you just blocked MSN email (Score:2)
Whenver someone sends me html-only email, I inform them that I can't read their email. If they refuse to send text, what can I do? They're the ones sending non-standard email formats.
SPEWS (Score:3, Insightful)
http://www.antispews.org/ the SPEWS fansite (not!)
Personally I see less problem with client-side blocking, as there is less chance that any 2 people would use exactly the same combination of blocklisting/priorities/etc. Plus, programs like Spamassassin use quite a lot of processing power, so large mail servers (eg. for an ISP) would need significant additional resources to handle this. Thus it is best to move such individualized and resource-intensive applications to the client-side anyway.
YMMV.
Re:SPEWS (Score:2)
Unless he sends actual spam, or is on an ISP that supports spamming, he wouldn't end up on the SPEWS list in the first place.
Re:SPEWS (Score:4, Insightful)
And this is why the SPEWS blocklist is so effective and so good. If he were on it, then that would mean that he and/or his network fell into one of the following categories:
Because spam causes abuse to email servers, even when the mail is refused either for reasons of an IP based blocklist, or for content filtering ... abuse in the form of higher costs for the server operators and recipients ... the proper goal is to get the spammer not just blocked from being able to get mail into your mailbox, but fully disconnected from the internet to prevent these kinds of costly abuses in the future. And since only the ISP hosting them can actually disconnect them, it will be the job of that ISP to do so. Most ISPs will when they realize the situation. A few ISPs refuse to, and that's when it comes time to put pressure on the ISP by expanding the blocking of the ISP's network, forcing them to consider that their legitimate customers will be leaving if they do not disconnect the spammer. SPEWS gradually expands listings so that the point where the ISP finally understands this can be reached with the minimum of so called collateral damage (which is not really, because these are customers who are paying money to an ISP which harbors spammers, so they share in the guilt).
Bruce Schneier's mail server happens to not be listed by SPEWS. So it can be said that he is not a spammer, is not running an ISP that harbors spammers, and is not using an ISP that harbors spammers. That is a good thing and shows that SPEWS not only works, but works better than content based filtering.
Content based filtering also is a direct violation of the principles of the US First Amendment right to free speech (although the actual amendment only applies to restrictions imposed by the government and does not apply to private businesses in most cases, if not all). Infringement of free speech happens when the decision is based on what the content is. When restrictions are not affected by the content, then such restrictions are considered fair since any content can be passed when the behaviour that evoked the restrictions is not done. And the whole spam issue is about behaviour, not content. The bad behaviour is the act of inappropriately choosing multiple recipients for sending the message ... e.g. unsolicited bulk email (UBE).
Of course on your own mail server you have a right to use whatever methods you deem appropriate based on how you want to balance your costs, the quality of your service to your customers, and how much cost you want to pass on to your customers. Obviously you have to be in contractual agreement (possibly implied) with your customers about what methods are chosen. If you only offer one kind of service and your customer does not want that kind, by being properly aware of what you do offer, they can go elsewhere. Or you can offer a diversity of services the customer can choose from (e.g. a customer control panel to control the methods of spam filtering for their email accounts). So the choice of what method to use to block spam is strictly a relationship between a provider and its own customer.
In the case of a network owned by a business only to serve that business function, then it's simply the commercial version of "my server, my rules".
Re:SPEWS (Score:2)
Is a spammer
Is an ISP harboring a spammer (or an upstream ISP thereof)
Is a customer of an ISP harboring a spammer
/dev/null would accomplish THAT goal). The point is to allow genuine communication. That means a perfect antispam would allow 100% of "useful" communication (whatever you define "useful" to be) and deny 100% of everything else. Blocking "customers of ISPs" goes directly against that: purposefully denying non-spam traffic is a broken concept. Blacklisters tend to justify such behavior as "zero tolerance," and "putting pressure on ISPs," but I think attacking innocent bystanders is extremely offensive, ineffective and just plain wrong.
uh, this is exactly why things like blacklists *are* broken. There are plenty of spammers not on any blacklist, so don't think of (!blacklisted) as equal to (whitelisted). Also, (blacklisted) != (spammer) as well, since alot of these list ops don't care about false positives or collateral damage.
Secondly, consider your "is a customer of an isp harboring a spammer" rule. The point of antispam efforts is not to block out all spam. (redirect all email to
So what if your favorite blacklist decides to stuff the entire 64.*.*.* IP address range? you will cut a lot of spam but suffer enormous collateral damage. Find a spammer, block the spammer. but don't bomb his whole neighborhood "to prove a point."
Re:SPEWS (Score:2)
No anti-spam method is perfect. It is unlikely any will ever be. Don't expect some clever new spam to be blocked until the blockers get more clever. It's a game of one-upsmanship.
Go back and read my original post. These are not false positives or collateral damage. They are intended. When the customers of an ISP are listed and blocked in order to pressure the ISP to stop its support of the abuses by spammers, that is not an error, not a mistake, not a false positive, or collateral damage. It is in fact intended and for the described purpose.
In war, we speak of collateral damage as the UNintended targets of things such as bombs. Blocking customers of ISPs is not that. It is more like trade sanctions. The trade sanctions don't work in Iraq because most of the people cannot switch to living in a different country. But most customers of an ISP that doesn't get the clue, can switch to another ISP. And it has accomplished the intent in many cases.
You're missing a goal. The other goal is to keep the communications cost effective. Consider that an onslaught of spam, even though it is not going to be delivered for whatever reason, can overload a mail server, possibly even crashing it, and deny other communications. Spam attacks can deny the timeliness of communications. There won't be any form of 100% perfect useful communications until every spammer is gone. That goal cannot ever be realized given human nature, but we can get very close by making sure that ISPs deal with the issues of spam that they should be doing. Once they are doing that, then we'll at least have 99.9999% usefulness.
We obviously disagree. In my opinion, what you call attack is nothing more than a boycott. And remember that it is the recipient mail system operator making that decision to use SPEWS or some other blocklist. If they believed as you do, they would not use it (I presume you do not).
First of all, picking that specific address range is stupid. SPEWS will not expand a listing to an unrelated ISP. The 64.0.0.0/8 block is broken up into many allocations by ARIN. But maybe you can use the 12.0.0.0/8 network instead, since it is allocated entirely to one ISP.
It's not bombing a neighborhood. If enough people were to use SPEWS, then the ISP would eventually realize that they will lose more money by legitimate businesses leaving than they get from spammers. Bombing is lasting damage that has to be rebuilt over. Blocking an ISP is fixed by a very simple action of disconnecting the offending spammers.
I'm sure you would be quite pissed off if your mail bounced because your ISP was listed in SPEWS. But consider that the operator of the mail server used by the party you tried to send the mail to is equally pissed off at your ISP for letting one of its customers continue to attack his server. Actually, it is more likely he will be even more pissed off, because the costs in terms of resources consumed and wasted at the recipient server exceeds the money the ISP makes from the spammers, the money the spammers make for themselves, and the cost to the customers to switch ISP, combined.
Re:SPEWS (Score:3, Insightful)
Or he might be
A customer of UUNet which spews has listed because it disagrees with some of the content they host
NOBODY with a brain is using SPEWS anymore. Listing the largest commercial internet supplier in the US was simply idiotic. And it was done for completely illegitimate reasons.
The whole blacklist concept boils down to vigilante tactics, use threats to keep people in line. The problem being that the people who run the lists tend to turn into self-important little tinpot dictators after a short time.
Content based filtering also is a direct violation of the principles of the US First Amendment right to free speech
Unture, with the exception of Limabaugh whose judgment in Nixon is opinionated nonsense the Federal courts have all rulled that the junk fax laws are constitutional.
Re:SPEWS (Score:2, Interesting)
UUNet has become one of the worst ISPs around due to their harboring of large numbers of spammers. And they do absolutely nothing to respond to complaints reported to them. They just let the spammers keep spamming.
There are completely legitimate reasons for blocking UUNet. It's the spam. You may be confusing SPEWS with some small-time renegade blocklist.
As soon as I see SPEWS operators "turn into self-important little tinpot dictators" I'll certainly stop using it. But I have not seen it happen. Feel free to point out any specifics if you are aware of them.
If anything, it is the very act of harboring spammers that is a vigilante tactic. Given that the costs of transmitting email are heavily slanted to the recipient end when spam is involved (because the spammers use special software to send email that scale up more effectively than ordinary MTA software), such a tactic could be in active use by some ISPs to drive up the costs for others (their competition).
Read my statement again, this time carefully. I said it is a violation of the principles. I did not say it is a violation of the Constitution and/or First Amendment itself (see the way the clause is written). Since the Constitution places restrictions on the government, it is the government that is the one that has to be sure not to restrict speech based on its content. You and I are free to do so within the context of our property rights and those of others. While it would be wrong for me to go delete your messages (that would be violating your property rights), you could certainly delete them yourself if you choose to. But I do fully believe in the principles the US Constitution was based on, and I practice my life that way. Thus, I do not use content based filtering. That's my choice.
Re:SPEWS (Score:2)
Since the SPEWS maintainers refuse to answer any correspondence whatsoever there is no way you can possibly know what criteria they are applying. They state that their criteria for listing UUNET is content hosted by a UUNET customer. Of course SPEWS could be lying.
The realtime blacklists are simply not transparent.
Comparing UUNET to vigilantes is simple sophistry.
Re:SPEWS (Score:2)
Oh SPEWS and its ilk have gone further than that. Several of the Blacklists have blacklisted whole countries. In particular China and Korea.
I doubt many people blocked due to a single spammer are going to think "oh, well, I may not be able to send e-mail to my most important client - but at least while I'm losing thousands of dollars, I know I'm helping to fight spam!" Most, I'd bet, would just call up the offending receiver and complain that they're getting bounce messages when they try and send e-mail and that the receivers should fix their mail servers as soon as possible.
This happened to us as we are customers of UUNET which SPEWS listed because they don't like some of the content they host. Switching ISPs was never considered, we simply used the backup feed to send out an email to all the mailing lists we host stating that we had been blacklisted, have no intention of changing feeds and people who wanted to participate in those working groups could fix their mail servers pronto.
Re:SPEWS (Score:2)
I'm just going to make this point: blocking non-spammers will only hurt SPEWS in the long-run, as its current effectiveness is based on the majority of people using it. As people find that they either have to turn to some other source than SPEWS or accept that occasionally people they must communicate with cannot send them email without a whitelist, they will not think "yay, we're helping to eliminate spammers!" and instead think "Goddamned broken SPEWS thing accidently blocking valid people - remove it!"
Eventually, SPEWS will have caused enough problems that no one (except a small core of vigilantes on their own servers) will use it. And then it will have no power over the vast majority of spammers. SPEWS would be more effective if it only blocked spammers and many people used it - it could help make spam an ineffective method of contacting people. Force the cost of spamming up, not the cost of happening to use an ISP that hosts a spammer. All you're going to do is create things like this [antispews.org], and force people against you. SPEWS does not help the cause by being a vigilante and trying to force people who have no buisness with the spammer to take action against their ISP.
Besides - which is easier: getting the ISP to drop the spammers acount, or telling people who want to communicate with you to stop using SPEWS? The path of least resistance is likely to be followed... which may include "ok, we'll just ignore the bozos using SPEWS."
Re:SPEWS (Score:2, Informative)
Heh, this antispews.org money-making scam [google.com] is a rather funny one. Strangely enough the Hostway Corporation started hosting the site three days after t3marketing lost [ilaw.com.au] their lawsuit against Joe McNicol [ilaw.com.au]. The Hostway Corporation is behind the t3marketing and many other "direct marketing" buggers. So it's no wonder that they are listed in SPEWS and using every possible way - sue spamfighters, spread FUD, etc. - to help them to continue poison our mailboxes.
That being said, I'm not sure if the SPEWS way of doing things is such a good idea but the antispews.org site is still run by spammers and should be treated as such.
Re:SPEWS (Score:2)
Clearly you're not on dial-up. By the time I've downloaded 100 spams, half the annoyance is over.
economics (Score:2)
Some of us aren't so lucky. The rest of us actually need eyeballs on our newsletters [borisfx.com] and try to test our content through filters before sending it out. I am consistantly amazed at the little things that flag my newsletter as spam.
Can someone run it through SpamAssassin? (Score:2, Informative)
When you run SpamAssassin in test mode, it tells you what rules got hit. You can also look at the headers in "Spam-Tagged" email to see what rules got hit. I looked for "Spam Testing" pages on the 'net, but had no luck.
Could someone run the Crypto newsletter through SA to find out what cased its evaluation?
As an aside, Counterpane could have done this to find out what the problem was, too. Not that they should have to, but they could have.
Re:Can someone run it through SpamAssassin? (Score:2)
as a result, it'll look different than someone subscribed to the list, since spamassassin does rely a bit on the headers, not just the text:
Oh, and as a side note, when i tried to paste this, unedited, slashdot spat the following at me:
Lameness filter encountered. Post aborted!
Reason: Please use fewer 'junk' characters.
the irony is thick. as is, of course, the irony that i could completely bypass it by formatting my message as "code". Now, back to the show:
X-Spam-Status: Yes, hits=5.1 required=5.0
tests=BALANCE_FOR_LONG_20K,BALANCE_FOR_LONG_40K,D
FROM_MISSING,MISSING_HEADERS,NORMAL_HTTP_TO_IP,OP
SPAM_PHRASE_01_02,SUBJ_MISSING,SUPERLONG_LINE,US_
US_DOLLARS_4
version=2.44
X-Spam-Flag: YES
X-Spam-Level: *****
X-Spam-Checker-Version: SpamAssassin 2.44 (1.115.2.24-2003-01-30-exp)
SPAM: This mail is probably spam. The original message has been altered
SPAM: so you can recognise or block similar unwanted mail in future.
SPAM: See http://spamassassin.org/tag/ for more details.
SPAM:
SPAM: Content analysis details: (5.10 hits, 5 required)
SPAM: FROM_MISSING (-0.0 points) Missing From: header
SPAM: DATE_MISSING (0.8 points) Missing Date: header
SPAM: SUBJ_MISSING (0.3 points) Subject: is empty or missing
SPAM: OPT_IN (1.5 points) BODY: Talks about opting in
SPAM: US_DOLLARS_4 (0.4 points) BODY: Nigerian scam key phrase ($NNN.N m/USDNNN.N m/US$NN.N m)
SPAM: US_DOLLARS_2 (0.1 points) BODY: Nigerian scam key phrase ($NNN.N m/USDNNN.N m/US$NN.N m)
SPAM: BALANCE_FOR_LONG_20K (-0.7 points) BODY: Message text is over 20K in size
SPAM: BALANCE_FOR_LONG_40K (-0.1 points) BODY: Message text is over 40K in size
SPAM: SPAM_PHRASE_01_02 (0.5 points) BODY: Spam phrases score is 01 to 02 (low)
SPAM: [score: 1]
SPAM: SUPERLONG_LINE (0.0 points) BODY: Contains a line >=199 characters long
SPAM: NORMAL_HTTP_TO_IP (1.3 points) URI: Uses a dotted-decimal IP address in URL
SPAM: MISSING_HEADERS (1.0 points) Missing To: header
hope this helps,
gleam
Re:Can someone run it through SpamAssassin? (Score:3, Informative)
SPAM: FROM_MISSING (-0.0 points) Missing From: header
SPAM: DATE_MISSING (0.8 points) Missing Date: header
SPAM: SUBJ_MISSING (0.3 points) Subject: is empty or missing
SPAM: MISSING_HEADERS (1.0 points) Missing To: header
See this posting [slashdot.org] for one with the headers, which shows that SpamAssassin doesn't tag it as spam anyway.
Re:Can someone run it through SpamAssassin? (Score:2)
but it looked like it would have made it to my inbox with no problems.
-gleam
Re:Can someone run it through SpamAssassin? (Score:2)
Go read the DNSSEC mailing list, there has been a considerable amount of discussion about OPTIN.
Or read any of the privacy mailing lists where the term opt in is used in the exact same context
The big problem with developing a SPAM solution is that nobody wants to hear any solutions, start describing something and they will interrupt your first sentence to tell you their idea. Then when you explain that the idea is not new and has severe drawbacks they assert that it works for them so it should be good enough for anyone.
Same thing happened to IACR newsletter in 1991 (Score:2)
Re:Same thing happened to IACR newsletter in 1991 (Score:2)
I somehow doubt it, Canter and Segal didn't send out their mass mailling till 1993.
One of the IACR board members was a crypto expert with the unfortunate name of Don Beaver. And there were some references to "hardcore bits" and LaTex. It was all too much for the filters.
I think you are describing a censorship filter rather than a spam filter there.
A possible solution to the spam problem... (Score:5, Interesting)
The first inclination one has would be to suggest that everyone close their open relays. But this depends on people doing the right thing all the time, and has proven ineffective.
Fortunately, there's another way.
Right now, everyone who receives mail has to listen to everyone who tries to connect. The problem is how do you separate the wheat from the chaff?
The solution is to take advantage of the information SMTP and TCP/IP give you when a connection is established. The fact that you're receiving a connection gives you the address of the sender. And during an SMTP transaction, one of the SMTP commands (the MAIL FROM command) gives you the domain of the email's sender, e.g. "MAIL FROM slashdot@sysexperts.com".
When you're sending email to someone else, you do so by looking up the MX records for their domain, which tells you which systems are responsible for receiving email for that domain. This gives us a possible answer to the spam problem.
Suppose instead of blindly accepting email from everyone, you were to take the domain given to you by the MAIL FROM command, look up the MXes for that domain, and reject the email connection if the IP address of the sender doesn't match one of the domain's MXes?
Now, suddenly, you would end up rejecting email sent from every unauthorized relay, because the owner of the domain can make any system that is allowed to send email on behalf of his domain into an MX (and, if he doesn't want that system to be used for delivering email, then he simply makes such systems the lowest priority MXes in the list and blocks outside port 25 connections to them ... something he's probably doing anyway).
Suddenly, the only systems that spammers can send email from are systems that they legitimately control and that are defined as MXes for a domain they control. Suddenly, spammers have to set up and maintain their own domains and their own boxes. The costs have just become a lot higher, which will get rid of most of the spammers.
And suddenly, blocking spam becomes orders of magnitude easier -- you only have to deal with spammers who have decided to pay the (now much higher) price for sending spam and who cannot use someone else's system to do their dirty work without permission.
Re: (Score:2, Interesting)
Re:A possible solution to the spam problem... (Score:2)
Correct. We are one of those. We have our own mail server (rented rack in Dallas) but use Speakeasy SMTP servers (Our SDSL provider located in NY) for outgoing opt-in newsletters. We don't HAVE to do it this way, but for technical reasons, its faster and easier for us. And it doesn't count toward our bandwidth on the rack.
Another server we have, on another SDSL line was blacklisted because it was SDSL, effectively shuting down all mail services. The server has been up on that IP for years, and averaged 10 to 50 emails per day, so we surely were not spammers. Our SMTP server are not open relays. It was blacklisted purely because it was on SDSL ip ranges.
please no (Score:5, Informative)
My primary mailbox is with a small, local ISP. I can't buy broadband from them, so I get my connectivity via cablemodem. I do have a mailbox in the cablemodem company domain -- that's the one I give out when I expect abuse. (I do it this way because I expect to be dealing with that ISP long after the cable vendor has either ceased to exist or has treated me badly enough that I left.)
So I want my outbound mail to appear to have come from the ISP. Setting Reply-To is usually adequate, but not always -- when a human is looking for the address, they could easily grab the wrong one. And it creates potential confusion I don't want to create. So I set my from address to name@isp.com.
I can't relay through the ISP's relays, because I'm outside of their IP range. (If they did some form of authenticated SMTP, such as SMTP-after-POP, they could let me.) And the cable vendor's mail relays won't send mail out with some other domain name on it. So I send everything out directly, no relays.
If you look at many headers, I suspect you'll find that I'm not the only one forging my From: address for legit reasons. The presence of the X-Authentication-Warning header some MTAs add correlates fairly weakly with spam. (Some details of it -- e.g. no valid reverse DNS for the sending machine's IP -- could be useful indicators.)
Agreed (needed even more at university) (Score:2)
At university this is even more necessary, since my university blocks port 25 at most of their routers - the only exception is that anyone in the university can connect to a "server" (actually a load-balancing cluster) which acts as a central relay. This means it's impossible to send mail unless it's either tunnelled in some way (not an option for me, my web host charges extra for ssh), or through this relay server. The relay accepts mail with any faked From address, on the basis that some people (including some departments) need this functionality, and if someone spams through it, they have it logged and know who to blame.
(Before you ask whether my uni gives me an e-mail address: yes it does, but I do game modifications, and I don't want to use my uni address for that. Also, my domain is more permanent than an address that disappears when I graduate)
Isn't that the entire problem? (Score:2)
In other words, you're sending a mail that appears to come from a ISP address, without ever being in contact with that ISP. Assuming there are rouge boxes out there, how could you possibly fix it without breaking your setup? I think the solution is fair - if you want to send with a @domain address, you must authenticate with the @domain servers somehow. If your ISP doesn't offer it now, I'm pretty sure they would if there was real demand for this feature. And if your tell your ISP that this will help reduce spam, they'll like it. They're not more fond of spam than you are (except those ISP profiting more from having spammers than getting spam, but they're few)
Kjella
Re:please no (Score:2)
Example: Mozilla Mail - I have three different IMAP servers that I talk to (Work, Personal, and my ISP's). When I reply to a mail from any of those, Mozilla correctly changes my "From:" field to the appropriate setting so the emails will come back to whichever account I was sending from.
However, with Mozilla, you can only specify ONE outgoing SMTP server, which in my case is my local machine. This would mean almost all of outgoing emails would get rejected, as the reverse lookup of my Mail-From: header wouldn't match my local IP.
Plus, it's not uncommon for large ISPs or organizations to have a different set (IP block) of outgoing SMTP vs. incoming SMTP, to distribute load.
Re:A possible solution to the spam problem... (Score:2)
Why Block? (Score:2)
What I personally don't understand is why people would trust a program enough to trash a message just because the program itself thinks it was Spam. I get about 150 messages per day; maybe 80% of it is spam (7 years with the same email address will do that), but I use SA's default behavior, which is to rewrite the subject and provide a report. I see that familiar "****SPAM****", and by the time I expunge, I've casually scanned the subject lines of the tagged messages (false positives are extremely rare, but possible, as this article is a prime example of).
SpamAssassin? not really (Score:3, Informative)
X-Spam-Status: No, hits=2.3 required=5.0
tests=BALANCE_FOR_LONG_20K,BALANCE_FOR_LONG_40K,
PAM_PHRASE_01_02,SUBJECT_MONTH,SUBJECT_MONTH_2,
US_DOLLARS_2,US_DOLLARS_4
version=2.43
X-Spam-Level: **
Now that's not really tagged as spam...
Re:SpamAssassin? not really (Score:3, Informative)
Anybody running 2.20 deserves every false positive they get.
meganet (Score:5, Interesting)
Crazy talk ... that's user error (Score:2, Informative)
I use procmail with SpamAssassin in this manner:
It only takes a little bit of thought and minimal configuration to keep your mail from incorrectly being flagged as SPAM. For me, using this method has led to zero (0) false positives on messages from known sources, for two years. Every once in a while a SPAM message sneaks into my INBOX (a couple a year), but then I submit it to a SPAM database used in SA's checks (like Razor), or adjust any particularly annoying rules' scores, and it doesn't make a repeat appearance for me.
If your find that any particular newsletter is being treated as SPAM by your mail filters, there's probably a very simple way for you to make sure it isn't filtered out. Use the tools you have wisely, and you won't be disappointed.
Funny until it's something trully important... (Score:2, Funny)
A Simple question... (Score:4, Insightful)
For each mailing list I subscribe to, I use a special address suffix just for that list, that bypasses all of my spam checks (including SpamAssassin ), and just goes right into the mailbox that I use for that mailing list.
No problems with false positives, and it saves me the overhead or running SpamAssassin on every incoming message from a busy list.
it just seems like common sense, no one should have a problem with SpamAssassin misclassifying incoming newsletters if they just think about how they organize their email.
initial analysis for Bruce (Score:5, Informative)
Aside from the spot-on comments that people have made regarding adding a whitelist entry Crypto-Gram (an obvious candidate for whitelisting if there ever was one, given that it frequently discusses spam, scams, and probably even includes text straight out of some spams), here is my initial analysis and response to him.
Oh, first one other comment: SpamAssassin does not block content. SpamAssassin only flags probable spam. What the site or user does with that flag is their own business. Some mail administrators misuse SpamAssassin to block email, but we do not recommend blocking email. Really.
------
[...] One false positive (or a related set of false positives) is not really a statistically useful sample size. To get to a high rate of filtering, most filters do have some false positives. You can get fewer false positives with customization of one form or another (personalized Bayes training, whitelists, rules, automatic learning algorithms). Our goal (everyone's goal, I think) is to get the best ratio of false positives to false negatives. It's a difficult balance sometimes and some legitimate content has a harder time.
On to the data:
I checked your newsletter with two versions of SpamAssassin: the current stable version (2.44) and the very-soon-to-be-released development version (2.50).
A score of 5.0 is the default threshold to be flagged as spam.
In SA 2.44, your mail receives a score of 3.20 (2.40 as I received it, but I believe the score would be about 3.20 for most people). That's on the high side, but has bit to go before being flagged as spam. The score is the same with network tests (DNS blacklist tests and Razor).
In SA 2.50, your message would probably receive a score of 1.90 without network tests and 1.00 with network tests. Note that the test scores may change a bit before the final release of 2.50, but those are better scores, more what we like to see for non-spam content. They would be even lower when using Bayes (part of SA 2.50). Those lower scores are not unexpected because... well, 2.50 is better. :-)
Based on these results, it's not clear to me why yesterday's newsletter was flagged as spam. Some possibilities:
Can you give me more information about the false positive that you experienced or was reported to you?
Thanks.
Dan
------
If I find out more of interest before the thread is closed to comments, I'll try to post a follow-up to my post.
Re:initial analysis for Bruce (Score:2)
Not a problem.
bcl
My Results (Score:2)
Headers added by spamassassin:
It still sounds like I should upgrade to 2.5 when it comes out, sounds like some very nice features. Keep up the good work.
Bad news, it's in Razor (Score:3, Informative)
I just got the email today and it failed. I'm running 2.44 from Debian and haven't yet looked at tweaking any of the rules.
Here's the verbose banner that SA put on my copy:
It looks like some dumbass has entered it into Razor. Unfortunately, some people (and yes I did this originally) had their procmail setup to enter an email into razor if it is deemed "spam" by SA or something else. Those 3.9 points are what puts it over the threshold.
Just shows that... (Score:3, Interesting)
What is needed is a foolproof way of saying "I want this, please send it to me" and then being able to reject it safly without needing the other party to do it for you. For example:
I send a message to cryto-gram, including a key. This key can then be used to send it to me, and I accept it (key in combination with who send it and so on, I am sure someone with even more experience can figure out a fool proof way). Good stuff. But then I realise that I don't want this anymore, and I simply remove the acceptance of this key in my own software (and send a message that I don't want it anymore, no harm being nice to the nice), and it will be filtered away.
Or something along those lines, I can asure you that I haven't fixed up a foolproof and perfect system yet
Re:Just shows that... (Score:2)
This can be done via TMDA [tmda.net], a whitelist-centric anti-spam package. You can create sender-based addresses so that the originating org has a direct pipe to you inbox, but that anyone else trying to use the address will have to confirm their identity first. You can remove that direct pipe at your leisure.
Sure, Bruce has nothing better to do... (Score:2)
You want the newsletter, then it's up to you to make sure you can receive it. I think there's pretty near zero chance that Bruce is going to waste his time jumping through hoops for your benefit and your benefit alone. Well, ok, I won't speak for Bruce, but speaking as a Debian developer, if you send me a question or request for help, and my response bounces because I'm not on your whitelist, I'm simply going to delete your question/request, and will probably add your name to my killfile, just so I don't have to deal with that crap in the future.
Ancient Procmail Secret... (Score:2, Informative)
..."Ancient Gurus srb and guenther say, 'Sort your mailing lists to the folders before you filter your spam.'"
Crypto-Gram isn't the only mailing list that gets hit by misunderstandings - all automatic mail handling is always confused about automailers and mailing lists. And even due to usability factors, it makes sense to sort mailing lists to folders anyway, and use a client that supports multiple specific folders.
Re:Ancient Procmail Secret... (Score:2)
This works well, except that sometimes the mailing lists can be spammed too (eg lists which don't require subscribtion to post).
White list (Score:2)
False alarm? (Score:3, Interesting)
Note that SpamAssassin isn't on my whitelist or anything like that -- it just worked.
False alarm?
The future of e-mail (Score:3, Insightful)
It has some other advantages too:
I learned this from the electronic greeting industry. Similar to Usenet 2 [usenet2.org] and Internet Mail 2000 [cr.yp.to], messages semaphores will become the future of e-mail. People will create web content as easy as they create e-mail messages now and semaphore the recipients (using IM or email) to look at their content. Recipients who are interested will click on the URL in the semaphore. Recipients who want mail from Bruce, will open it. Bruce might even (G)PG(P)-sign the announcement notice so that spammers can't pretend to be him.
Then again, why should Bruce have to mail anyone at all? If his newsletter is so good, his readers will bookmark his page and read it every now and then, just like I do with DaemonNews or ArsTechnica.
The Internet is evolving, and Bruce is whining along the way. Mass-mailed newsletters are going the way of the dino-WAIS-server (just like FTP [slashdot.org]
-ez
Re:Free Speech (Score:2)
How can we vote him out? Thy wouldn't let the votes be counted when it came to voting a president in.
The US First Amendment only applies to restrictions imposed by the government.
And in any case will be suspended in the next few weeks, after all they have already suspended due process and habeas corpus so it is only a matter of time before the first ammendment goes as well.
One of the problems with the death penalty is that its popularity with the voters means that opponents are considered unelectable. So you end up with politicians like Clinton who know that it does not work but will pay lip service to it to get elected, or you get psychopaths like Bush.