Paul Graham Describes Dangers of Spam Blacklists 611
CRoby writes "Paul Graham posted an essay describing the danger and corruption of the main spammer blacklists today. It discusses MAPS and the SBL, the blacklist created to try to alleviate the abuses of MAPS, and suggests (maybe) another blacklist's creation."
Not like people get all radical about it... (Score:5, Interesting)
Oh, ok. Nothing like over reacting a bit.
today? (Score:0, Interesting)
today? Articles linked are from 2000 and 2002!
I don't know how many times you can use the word "vigilante" in one article
Pay and you get removed (Score:5, Interesting)
Interesting: The company won't say who they are. [admins.ws] They say this was approved by local authorities, but this is bullshit. Local authorities can not brake federal law in Germany.
Re:Pure and simple... (Score:1, Interesting)
The only reason we actually fixed the problem was because the boss couldn't get his email on the road (the server had crashed again). Incidentally, I was the only one available to actually do the fix, and I did it with Linux/qmail and an old box over the weekend. $0 spent.
Maybe if we had been blacklisted to the point of not being able to send any email, they would have paid more attention. Instead most of our mail was still going through, so we were allowed to be a menace to the net.
Guideline, not a rule (Score:5, Interesting)
If you're sending a ton of mail, i.e., spam, little of it gets through. If you're only sending one or two messages, ie, likely legit mail, it goes through just fine.
Combined with more specific stuff further back (bayes, et. al), it's been quite effective at reducing the amount of spam sent, and the amount of mail that gets scanned.
The problem isn't blacklists, its how people use them.
Re:today? (Score:3, Interesting)
You're right. The correct words are 'overreacting assholes'.
Most RBLs are run by assholes who have no concept of how to properly manage something as complex as a RBL.
And no, I've never been blocked by one and I weight RBL positives very low.
"Power-hungry weenies" (Score:5, Interesting)
Interestingly enough, the owner of the acme.com domain who was recently featured in a story due to his getting more than a million spam mails (well, attempts to send spam) a day, agrees:
(from http://www.acme.com/mail_filtering/shame_frameset. html [acme.com])
Re:Not like people get all radical about it... (Score:5, Interesting)
What's the alternative? Having some centralized, international spam cop whose job it is to clean up every ISP on the planet? If ISPs get a completely free pass on spam and don't have to care whether their subscribers are abusing other people or not, where is their incentive to prevent the abuse? The way you avoid the tragedy of the commons is by getting people to see their individual stake in the issue.
Certainly the quote that you're pointing out isn't the most diplomatic or effective way of putting it, and I doubt this kind of thinking is behind that quote - it probably is the knee-jerk reaction that you're identifying it for. Still, the idea might have some merit.
Re:Definitely a bad idea... (Score:2, Interesting)
We ended up by deciding to temporarily block mail from servers on certain blacklists (Spamhaus and Spamhaus XBL), sending a message back to the sender which allows them to release the mail. We also use SpamCop, but in a looser way; only if the mail comes from a SpamCop listed server and fails certain other tests do we, again temporarily, quarantine it. Otherwise we mark it as Spam, pass it through, and ask the recipient to tell us if it was Spam so that we can block it next time.
In either case the original sender, presuming it's a real person, has the ability to release the mail. (Of course we check all released mail, and if it's Spam the sender goes on our own permanent blacklist!).
I'm all too aware that this has the potential to add more useless mail to the system, but in practice most of these relase messages never even leave our server because the original came from a non-valid address. And it does work pretty well.
These, and other, rules allow us to block most of the Spam, which amounts to about 2/3 of all the mail we receive. And I've had a lot of compliments from the end users, so they appreciate what we're doing.
The moral is you can't trust the blacklists absolutely, but they have a very useful advisory role to play.
Re:Definitely a bad idea... (Score:2, Interesting)
If you are from a blacklist and your message has lots of chick-scratch in it or other spammer tricks and it generally looks like a piece of spam, it's more likely to be caught and blocked.
But using the SBL alone and giving it the final decision over accepting mail is just giving it way too much power.
What a clusterfuck (Score:4, Interesting)
By allowing the abuse it's outcome becomes a certainty. We're going to have to bite the bullet and dump open SMTP. And I think we're going to have to do this quickly. The levels of SPAM continue to rise. I often see ten to twenty times as many spam connections on my mail servers than legitimate connections, and this is a constant, flowing, amount of SPAM 24/7. Even with RBLs, spamassassin, etc, SPAM still gets through. The solution will not be found with another bandaid. It's time to dump SMTP and move to something that demands cryptographic authentication for users and hosts before allowing the transport session to complete. --M
Sorry... (Score:2, Interesting)
As an admin of a small mailserver hosting a handful of private domains I'm a very happy user of various DNS blacklists. I use some blacklists to reject ALL e-mail from countries like Korea & China due to the constant flood of spam from those countries. I also use other blacklists in conjunction with SpamAssassin to more accurately deal with spam. If you don't like the way I manage my mailserver then tough! I probably don't want e-mail from you anyway. If you have a LEGITIMATE problem with being blacklisted then e-mail me another way (like from gmail, hotmail, etc) and I'll consider whitelisting you. I've also got a few specific mailservers whitelisted exactly because I was asked (nicely!) to do so.
Bottom line - my server, my rules.
paulgraham.com is blocked (Score:1, Interesting)
66.163.161.45/32
Now do a DNS lookup on paulgraham.com: 66.163.161.45
The problem is that yahoo can host multiple sites on the same IP and the blacklists cant differentiate. The problem is the lack of granularity not, as Mr. Graham writes, an abuse of power by the SBL people
Re:Definitely a bad idea... (Score:4, Interesting)
Anyway, they shouldn't be blocking entire blocks of IPs. That doesn't even make sense. What does one guy on one IP out of hundreds or thousands who spammed for most of a day before he got caught have to do with my server which has run clean and reliable and secure and in good faith (including SPF and everything else) for the better part of a decade?
As Paul Graham already stated, this is just a strongarm tactic to harass as many innocent parties as possible. There's no other explanation for it. Are two spammers really worth denying tens of thousands of (in the case of Paul Graham) Yahoo customers?
There are bad-actors; rogue hosts. It's pretty clear when you're dealing with one who isn't. And if you were quick to put people on the SBL list, then take them down just as quickly. It is unacceptable that it took three weeks after the incident for them to finally remove them from the list.
I have admined companies like this. (Score:1, Interesting)
I once worked for a hosting company that hosted spam servers "on the side". As an admin it was a constant battle with the blacklists
Management never understood this (or rather, they understood it very well, the spammers paid $20X the hosting of regular servers...), indeed, they started a second company just to host the bad servers.
I was ordered to lie constantly, and to shift IP's around etc, to make it harder for the black lists to get us. IMO, I think that the blacklists should have taken out the WHOLE hosting company.
While 99% of my customers were legit, and I worked hard to keep spammers off of our "normal" list, I knew that we were hosting spammers on purpose. In fact, part of the reason I was let go was that I complained that doing this was immoral, and that it risked our hosting business as a whole.
So, if they blocked your whole hosting company, I would suspect that the hosting company was playing games like this.
(As an aside, when I was let go from that job I was estatic. Indeed my co-workers wished that they could be "let go" too. In the end, the turnover at that company was about 120% a year...)
Re:Calling a spade a spade (Score:3, Interesting)
The point is still a good one. Is it morally reprehensible to target innocents for the purposes of shaping institutions of power? Is this not fundamentally the definition of terrorism? If you agree on both counts, then MAPS is an opt-in terrorist network dedicated to the destruction of spammers.
Re:Abuse my hind end (Score:3, Interesting)
If you suggest I move... that's rediciulous. Let's all just up and move to a different town each time a spammer comes by. Sure. Maybe if you're Bill Gates.
It is NOT easy to change ISPs, nor is it necessarily even possible. Oh, it's my fault for living here. Well excuse me - get the hell off your high horse. It's people like you making e-mail unuseable.
Re:Definitely a bad idea... (Score:4, Interesting)
You hit the nail right on the head. In fact, a fly on the wall related to me the entire conversation from the morning they decided to set this thing up:
Person 1: I'm bored this morning, how 'bout you?
Person 2: Yeah, me too, dewd. Let's start harassing as many innocent parties as we can!
Person 1: Yeah, dewd! That'd be way wicked cool!
Blame the spammers' money and the greed of the ISPs. It used to be quite common for a spammer to run under his pink contract from an IP address until people got fed up and blocked that specific IP. Certain ISPs would then assign the spammer a new IP address knowingly full well what they were doing with the explicit intent of allowing that spammer to bypass the blocklists from people who were obviously and explicitly taking steps to avoid the spam. Unfortunately as it turned out truly innocent customers were being assigned a dirty IP address that had been previously sullied by a spammer. The moment their email server came online they were already blocked because of what had happened there before. Talk about unfair.
The spam-friendly ISPs forced the blacklisting of IP blocks: there was simply no other way to filter out the spam coming from those netblocks. Other users of that hosting service may be inconvenienced, but the system admin's right to take steps to prevent spam from gumming up the works of HIS OWN NETWORK outweights the right of anybody else to expect email originating from the same IP address used to send out three trillion ads for vgiara the week before to be received with open arms.
Does this catch innocent people in the crossfire? Unfortunately, yes. But with 4,228,250,625 possible IP addresses those who maintain the blacklists can't be expected to personally review each and every email asking to be whitelisted and spend time and effort determining who is telling the truth and who is following spam rule #1.
If widget.qqq has your domain blacklisted then your beef is with the admin of widget.qqq. Period. End of story. Beg him to whitelist you. Buy him a pizza. Send him some free (as in beer) beer. Serenade him at three in the morning. Send three billion statements of character witness. But his network, his gate, his key, his rules on granting admission.
Let's look at this another way: If I am throwing a party and, on the advice of my friend who told me that people who wear Mickey Mouse shirts are boring, I deny admission to people wearing Mickey Mouse shirts from whom will you beg entry and who shall be called nasty names for listening to somebody else?
Of course, that's the solution, isn't it? We must ban any and all people from publishing an opinion regarding the statistical probability that an email from a given IP address is spam.
You know why they do that? (Score:3, Interesting)
Mine *doesn't* host spammers, and I'm in a contract. I can't pressure them to stop hosting spammers if they don't host any.
I stopped using RBLs/MAPS/SPEWS years ago and have never looked back. Even more interesting is that the volume of spam *did not* increase, but the complaints about being bounced/not getting through decreased.
Distributed List (Score:3, Interesting)
Suppose a "distributed" blacklist were created. I could blacklist the whole Internet, but I'd be the only one, so it wouldn't mean a thing. On the other hand, if 75,000 people have blacklisted an IP, there might be something there.
It needn't be totally distributed, I don't think. A community-run site, where, whenever you get obvious spam, you post the originating IP, could work. You'd post it, and that IP would have, say, 10 "points." The rating would "decay" by one point a day, so a site listed, but that went clean, would quickly leave the list: in ten days, each rating would be down to zero.
You could then simply query the site for a given IP, and it'd return the "points" a site had. This also allows you a lot more customizability: if you were obsessed with blocking all potential spam, you could block anything with more than 5 points. If you wanted to be careful, you might set it to, say, 1000 points.
Unless the people running the site keeping track of the ratings begin blatantly making up ratings, this idea means that a blacklist is much less immune to being "bad." And it allows IPs to "fade" out of the list over time.
Re:You know why they do that? (Score:3, Interesting)
That's the biggest problem with RBLs... you have *no* way of knowing how effective they are. Since mail gets blocked at the server, you can't tell how many false positives or true positives there are.
How much spam are you blocking? How much legit mail are you blocking? You have no way of knowing.
Randomly denying 6 out of every 10 emails delivered would probably be just as effective as using an RBL.
Speaking of blacklists (Score:3, Interesting)
I find myself doing for example a lookup of ad.marketingscum.com followed by a whois lookup of the IP address. If I find that they own a larger network like
NetRange: 216.73.80.0 - 216.73.95.255
CIDR: 216.73.80.0/20
NetName: DOUBLECLICK-NET
I enter the complete network into my blacklist. Are there any realtime blacklists for this purpose? This would be quite useful, wouldn't it?
Re:Definitely a bad idea... (Score:3, Interesting)
Re:Load of FUD by Paul Graham, competitor to Spamh (Score:2, Interesting)
Re:Wrong (Score:1, Interesting)
Bzzt. You get the award for not RTFA today. The SBL added Yahoo, because out of tens of thousands of sites they host, two were accused of spamming.
Yahoo represents a little more than "one or two people".
Additionally, you missed the point he was making about blacklists in general, which is that they start out rejecting spam... and then the guys who run it go on a power trip and start blocking out whoever they feel like.
He didn't say not to block spam hosts; he said that when they blacklist NON-spam sites by the truckload in order to pressure an ISP, they are specifically targetting innocent users in order to carry out their agenda.
Re:Pure and simple... (Score:3, Interesting)
The reason for the block? All Charter IP addresses have been put into a "residential" blocklist by one RBL nut that decided such a list was a good idea. Everyone knows that you should have to buy a T1 to send email. This is because people who really need to send email have the budget to pay 800$/mo for it, apparently. Unfortunately, Juno and NetZero both seem to agree.
Re:So what (Score:3, Interesting)
Okay, but I question how you can actually know how much the RBL is costing you.
Millions and millions of rejected messages versus the occasional manual intervention. It's a pretty easy judgement. I can even figure an average spam message size, multiply by the number received, compare that to my ham traffic, weight it against the cost of running my mail service and produce a dollars and cents figure of what RBLs save me (and that's before I factor in the costs associated with users having to deal with those spams if they were delivered). If I'm rejecting two thirds of all delivery attempts at the front door, I don't need to have mail systems that are three times the size and three times the cost.
If an employee sends an email asking for product information from Companies A, B, C, and D, but only gets answers from C and D, is he going to call you up assuming there's a problem or is he going to assume A and B aren't interested?
You seem to be conflating the case where I am using RBLs and the case where someone else is. If my employee attempts to send an email to a system that has us on their blocklist, my employee gets a non-delivery report from my system, advising him that the message was not delivered, including a transcript of the SMTP dialogue ("552 We don't like people with a "K" in their name"). Typically, he would then contact me and ask what was up, and I then deal with it in whatever way is appropriate. In the case where somebody elses employee tries to send to us, and we reject because of a RBL listing, that remote person gets a non-delivery report from their own system, and it is for the remote admin to deal with it as appropriate. I can only take responsibility for my own systems, I can't be postmaster for everybody else.
Shorts are no place for a hamster.