The New Yorker On Spam

aqk notes an article in the Aug. 6th New Yorker surveying the spam problem up-to-date. The New Yorker may not be exactly the MSM, but it is pretty influential. The author got only one fact wrong that I noticed: Canter and Siegel's seminal spam was propagated through Usenet and not email. Still, it's a good look at the history of spam and the scale of the problem today. The amount of spam that "spam king" Robert Alan Soloway, indicted under the CAN-SPAM Act, is accused of sending over a period of four years is now pumped out about every 30 seconds, around the clock, around the world.

Need More Exposure to Ideas and Methods

The New Yorker may not be exactly the MSM, but it is pretty influential.

What were you hoping? That if it were mainstream media (MSM), it would bring to light the problems of spam? That it would influence the businessmen & marketers that make the spam?

This article is a great short history on spam but no new information was presented to me here (and judging from the summary neither did it shed light on anything new to you).

I laugh at either of these hopes because the average person already deals with spam daily (my relatives began reaching out for me on ways to censor that from my younger cousins years ago) and we have a different mindset than businessmen & marketers.

The article mentions the epic article [paulgraham.com] by Paul Graham entitled "A Plan for Spam." It may look long and arduous but I heavily recommend you read that. I will not forget reading that article nor will Slashdot [slashdot.org]. I think it helps more for the "mainstream media" to publish things like this for their readers.

Yes, it has code in it. Yes, it requires a bit of a priori knowledge in some places (pun intended). But, you know, a lot of times the best stuff comes from outsiders and I personally think that newspapers should develop a 'tech section' where they can throw off the mittens & grade school knowledge that need to be on in order to handle your average reader. I know many newspapers have entire sections devoted to sports--sometimes even just one particular sport if it's in season! I've seen many newspapers have 'articles/ads' for new automobiles, why not new technology? I know Popular Mechanics is ... well, popular for lack of a better word so why aren't newspapers picking up on this and printing more tech-heavy articles? I guess all I can do is bitch about it because I don't have the same mindset as the people trying to sell the news.

Which brings me back to an important point, you're not going to change anyone's mind. Everyone knows about it and if you think that Wallstreet businessmen are going to pick up the New Yorker & their jaw will drop when they read this article, you're sadly mistaken. If you think marketers will read this and say "My God, I need to start thinking about what I'm doing to the networks of the world," you're deluding yourself.

What we need is an article that causes people to seriously ask themselves how we can keep e-mail free and uncensored while at the same time stopping spam. When I was asked by my aunt, they were concerned for their daughter using the internet and opening a spam message to see a guy with his legs split around a phallic-looking cactus in an ad for Viagra. I showed them how to use Thunderbird instead of Outlook Express and how to turn on junk mail filter. I also pointed out how vulnerable you leave yourself to spam if you print your e-mail in plain text on the internet. They never had a problem with it again.

So while this article is informational, it does nothing practical for the reader. I realize--and I think a lot of people will agree with me--that the best way to stop spam is to stop clicking on it and show others how to do the same. The 0.001% response will dry up and spammers will drop off. Articles on how to configure yourself to spot spam would probably be the best thing mainstream media could print--sure would have helped my relatives!

Re:Need More Exposure to Ideas and Methods

I know Popular Mechanics is ... well, popular for lack of a better word so why aren't newspapers picking up on this and printing more tech-heavy articles? I

Because such articles don't sell advertising. Popular Mechanics, Popular Science, Scientific American, etc., can sell ads because they have nothing but tech-heavy, jargon-laiden articles, and so the advertisers know exactly who they are targeting.

Newspapers are general-purpose publications, written for the widest audience possible. It's hard enough for them to sell ads these days without having to have specialized sections for the tech reader.

That being said, newspapers should be trying to innovate, because if they don't, well...it's the death knoll for newspapers.

Re:Need More Exposure to Ideas and Methods

A "death knoll"? Are the newspapers going to roll down it and hit their heads?

It'll be hard to change minds.

Which brings me back to an important point, you're not going to change anyone's mind.

I'm in the middle of starting up a small business and was talking to someone about marketing. This individual (Not an in-duh-vidual - a Ph.D.) suggested that I send out mass emails. I told him that I can't do that because I'll be a spammer and my ISP will yank my account. He then mentioned that they're are ways to mask my origins. I said if I get caught doing that, I'll be in even more trouble. Besides, I DON'T want to be a spammer.

My point? Spamming has become so standard and everyday that people don't even give it a second look now and just consider it an annoyance at worst. The only people who really care are those of us in IT.

Re:

I hope your friend's Ph.D isn't in a computer science related field. It seems logical that an acceptance of this would infect the rest of the world, though. Many businesses have enjoyed moderate success by sending out "mass-mail" through the USPS for yea

Re:Need More Exposure to Ideas and Methods

So while this article is informational, it does nothing practical for the reader. I realize--and I think a lot of people will agree with me--that the best way to stop spam is to stop clicking on it and show others how to do the same.

This is definitely a start in the right direction, but it's not the whole story. I'm convinced that a massive part of the problem is that there's a widespread belief that spammers make millions of dollars.

No doubt, a very few do. A very few have mansions and island retreats in the Bahamas. But these people are like the Michael Jordans of spammers, the people who have spent an incredible amount of time and effort into honing their spamming skills not just into an art, but a lucrative profession.

The problem is that most spammers aren't the Michael Jordans of spam. They're just people who have heard that spammers make millions of dollars, and they want in on that action. They go out and download the latest scripts and fire off a few million e-mails. No one responds. So they fire off a few million more. After enough times, someone will respond, and they've made $20 bucks. Flush with the thought of new mansions, they fire off millions more. Whoops, that $20 was charged to a stolen card, so they're back to zero.

The point is that the world has changed. Back in the day, there was a lot of money to be made from spam. Now, though, you have a very few scummy individuals who have made massive amounts of money. You have thousands of scummy individuals who think they can do they same thing, but fail miserably. It doesn't matter, though, all you need are the few who do make millions to keep the perception alive that spam = TONS of money, and you'll have people lining up to do it.

What need to happen is that they need to stop focusing so much on the spam "kings" and go after the regular guys who send it out. The people without the million-dollar houses. The people who think that it doesn't hurt anything to fire off a few million e-mails to try to sell some Vigara (yes, I misspelled it deliberately). The press need to cover those stories too. (They really need to cover them more.) People stop seeing Bill the multi-millionaire spam king and start seeing Ted the worthless loser who was so desperate that he thought he could make a million dollars by sending spam.

It's not enough to make spam unprofitable. People have to know it's unprofitable, and that when caught, they'll end up in jail for nothing.

Re:Need More Exposure to Ideas and Methods

Let's step back from spam a second. If prostitution is the world's oldest profession, being a con artist is a close second. Before spam, these people were jumping out in front of cars to collect a paycheck, enticing people through telephone calls into shady business transactions, and so on. Spam is only a new form of an old trade. These people are always going to feed off the ignorant, the naive, the bleeding hearts, and the foolish. You will never regulate this kind of predatory behavior out of existence. All of us make bad choices. Some of those bad choices involve being the con artist, and some of those bad choices involve letting ourselves be duped. You can't stop this, you can only hope to contain it. That being said, the most effective approaches to spam are going to be those that assume the existence of the problem going forward - i.e. we can not stop nor get rid of spam - and manage it effectively while educating people against the tricks of the trade. I think spam is largely an overblown issue, that most competent sysadmins have tool sets that manage it very well, and that the average user is much more educated then us slashdotters assume. To put it briefly - spam is an overblown issue that just gives the government an excuse to get their grubby hands on our tubes. In Soviet Russia, the internet surfs you!

Re:

This is definitely a start in the right direction, but it's not the whole story. I'm convinced that a massive part of the problem is that there's a widespread belief that spammers make millions of dollars.

No doubt, a very few do. A very few have mansions and island retreats in the Bahamas. But these people are like the Michael Jordans of spammers, the people who have spent an incredible amount of time and effort into honing their spamming skills not just into an art, but a lucrative profession.

Replace "spammers" with "drug dealers" and the statement is still true.

In fact, I think many, if not most, illegal activities are driven by the same motivation. It's a lottery; people rationally know that their chances of 'winning big' (being the multi-mi

Re:

Yes, it requires a bit of a priori knowledge in some places (pun intended).

Umm, what pun?

Incidentally, the New Yorker is one of the most prestigious magazines in the world (albeit prestige derived much more from its past authorial and editorial quality th

Proper verification of senders

Spam wouldn't be such a problem if we had proper verification of senders. Whether that's through some central identity or whatever. I realize this idea is extremely unpopular and is not in the spirit of the original Internet, but heck, if you had to essentially have an ID that verified who you were and if you sent out spam, you'd lose it, how much less spam would there be?

Re:

It's not just that this idea is unpopular, in my mind it is untenable. The nature of a decentralized system such as the internet is such that it, by its nature, resists control. So you want to require every internet user to have an ID for verification pu

Re:

That will never happen. Never.

Reason? Simple. Who would immediately lose their "internet rights"? Clueless people with spam sending trojans. The same people that pretty much everyone who earns money through the internet loves. ISPs love them, because they

Re:

If you ask me, nothing less than the total removal of all spamming should be acceptable. Filtering doesn't get it because you still have all of this spam and zombies wasting terabits of bandwidth. And you have spammers pwning all these clueless idiots' W

Re:Proper verification of senders

If you ask me, nothing less than the total removal of all spamming should be acceptable. Filtering doesn't get it because you still have all of this spam and zombies wasting terabits of bandwidth. And you have spammers pwning all these clueless idiots' Windows boxes, turning them into spam zombies. Remove the ability to send spam without screwing yourself, and you'll solve all of the spam-related problems. Filtering is like putting makeup on a facial wound. Removing the ability to send spam in the first place is the cure.

Sounds great... where do I sign up? but seriously. There are a number of problems with "Removing the ability to send spam in the first place"
1) What exactly is spam? -- Some people would say that spam is any e-mail they don't want. Others will say any e-mail they didn't ask for. yet others point to the dictionary and say "unsolicited usually commercial e-mail sent to a large number of addresses"
This brings up the first problem... if we go with the last (and most technical) of those definitions, all a spammer has to do is start to "properly" personalize the messages (for some value of personalize)
If we go with the first, how can you check on the sending end if the recipient wants it?
if we go with the second, what about when I want to send e-mail to a friend I've lost touch with? he didn't ask for the e-mail, therefore my message is "spam"

Even if we, as the GP suggested, impose a technical restriction on e-mail such that it has to be authenticated as to who it's from, all that does is make the filtering easier. What is going to prevent the dedicated spammer from "registering" a new identity? where would everybody's identities be registered? would you trust a centralized registry of "registered senders"? for some reason i think not.

I've heard suggestions of using a "web-of-trust" method of "registering identities", but even with that idea, you're going to end up with many separate webs. and bog help you if you want to send e-mail between the webs, you'd be effectively unknown, and thus declared "SPAM".

All to often the way it seems with technology is that we put band-aids on everything. Endless patch-Tuesdays, etc. and that when a new system is proposed and agreed upon it (a) takes forever to get off the ground ... IPv6 anyone?, or (b) is fored to be backwards compatible, which defeats the purpose.

Maybe I'll have to think of an algorithm to dynamically and auto-magically create a positive, and negative web-of-trust, both for senders and for servers... but that's more for another time

"Web of trust" won't work.

Check out TFA. They even mention Hotmail.

Hotmail is one of the world's largest providers of e-mail service, with two hundred and eighty-five million registered accounts in more than two hundred countries. "We filter them all, and that takes huge amounts of computer processing power and Internet bandwidth, and it requires us to work constantly to keep the numbers from getting worse," Scarrow said. "We do this to minimize the impact on our customers, but it's a hell of a job."

Yet about half the spam that gets through my system comes from HotMail and GMail.

And let's not forget the cute ads that Microsoft appends to outgoing Hotmail messages. So, someone sends spam through Hotmail, which ends up with the ad attached ... and it gets reported as spam ... so when a legitimate message comes through from Hotmail it also has the ad and so it gets flagged as spam by SpamAssassin.

That's great. The spam gets through and the legitimate messages are blocked. Maybe Microsoft could have put a bit more thought into their process? No? Getting the ads out is too important?

Here's a thought. How about Microsoft and Google throttle the outbound connections on their servers? One message every 5 seconds? And take an account off-line AND ALL ITS PENDING MESSAGES if they get a complaint? Google has smart people. I'm sure they could work out an automatic arrangement with the larger anti-spam sites.

The only "web of trust" you can really trust is your own white list.

I'd rather focus on the opposite. Identifying ranges that are 99.9%+ likely to be spammers. Like most of the home accounts on Comcast and Verizon and such.

No. It comes from their servers.

You mean appears to come from HotMail and GMail.

Nope. They can't fake the IP address if you don't have pipelining turned on. It's coming from their IP's.

Of course, very little stops spammers from also opening a ton of "legitimate" free accounts and spamming using those, but they can (sometimes) get tracked and shut down...

That's the problem. You cannot "trust" Hotmail or GMail because they ARE used by spammers.

And there is no technological reason why they could not address that issue. They know that spammers will open accounts with them. Yet they take no steps to mitigate that. Even limiting the outbound emails from each account would help. And having an automated process for reporting and blocking spam from them would pretty much solve the rest of the problem with them.

Re:Proper verification of senders

Ask your friends to stop using subjects like:

"You will be able to penetrate deeper"

"15% discount automatically on BOTH watches!"

At least in gmail they are still around, and gmail will let you search for them easily. I am more worried about my university bouncing legit email as spam and I never see it... No way to find those.

Address book doesn't help?

Have you put both the "From" and the "Reply-To" addresses on the mailing list into your Gmail address book? I've found that seems to keep mail in my Inbox instead of in Spam. (I think it's only the "From" that matters, but you can put both in there to be s

Not MSM?

How is the New Yorker *NOT* MSM? it's practically on every newsstand from here to Moscow (and yes, I have traveled that far, and seen it there) It's lasted for over 80 years so far and has won more than just a handful of nationally and internationally recognized awards for journalism.

If anything the New Yorker is a good way to reach people that might not be quite as technically proficient or knowledgeable.

Re:

MSM ?

The three letter acronym MSM can refer to:

Maastricht School of Management, in Maastricht, the Netherlands

Metal-semiconductor-metal junction.

Miami Sound Machine

Men who have sex with men

Million Skirted Men, a movement advocating men's right to wear skirts.

Suck

I run a mail server for our 5 person business. I left at 8pm last night and got in at 6:45am this morning. During that time, 191 messages where blocked due to the content of the headers. 1,799 connection attempts where rejected due to being on rbls or part of my block list(182,910 entries). 351 pieces of spam still got through that and got caught by the filter which I went through by hand to verify that none of it was valid for users. I just finished going through everyones inboxes to make sure nothing got through. Wanna know how many valid pieces of mail for all 5 users? 17. 17 out of the potential 2,341 attempted mail deliveries within a 11 hour time span.

Just because your inbox doesn't have a lot of spam doesn't mean someone out there isn't making sure you see it that way.

How much does spam cost?

No kidding. I admin a medium sized ISP. We have 8 (soon to be 9) distributed servers dedicated to email.

3 load balanced e-mail filtering appliances, at the Internet facing edge. (Basically, BSD boxes running postfix, spamassasin, clamav, policyd, DCC checks, RBL and a few other checkers and daemons I'm forgetting.) They get about 90% of our spam.

2 load balanced postfix boxes, running policyd on our outgoing mail, they will greylist any naughty customers with a zombie that have sent to much. Also, they do inbound user verification with LDAP, if spam has BCCed an invalid recipient or two, reject. Add another layer of anti-virus on the way to the customers. This catches another 8-9%. I'm guessing around 1% gets through.

1 DCC server, because we exceeded the threshold for being able to use free DCC long ago. (I'll admit it's a bit under used.)

1 MTA running exim for the hosted domains. This has spamassain, and a few other services, supplementary to everything in front of it. I'd say it gets most of the rest for those with hosted domains.

1 big bad 8x processor pop server that runs webmail and pop for the customers. It does no spam checking, because it could never handle the load, just stores what we think is not spam for the customers, around 25,000 accounts.

By comparison, we need one (1) production, not counting backups, provisioning server. It handles minor things like DHCP for 15,000 customers.

Now you have an idea on what your ISP spends its money and resources on. There is no small industry selling you solutions to fight the SPAM.

They try to send, but don't really succeed

The amount of spam that "spam king" Robert Alan Soloway, indicted under the CAN-SPAM Act, is accused of sending over a period of four years is now pumped out about every 30 seconds, around the clock, around the world.

Well, they're trying to send a lot, but with a proper setup at and around your mail server, you will not be seeing much of it anyway.

Simple greylisting helps a lot, supplemented with greytrapping-generated blacklists (with 24 hour expiry) it's even fun to watch. The last 2-3 percent that actually makes it through to be seen by content filtering gets converted back to free electrons.

I've had a series of blog entries over at bsdly.blogspot.com [blogspot.com] about this and the conclusion is clear - with a competent system administrator, Spam is a solved problem [blogspot.com] (Links to other refs inside, follow links).

Re:

I don't think an ever-increasing percentage of my bandwidth and CPU time going into spam filtering counts as a "solved" problem.

How much time should I spend on becoming the hypothetical "competent" sysadmin? How much should I have to pay someone else to d

Re:

You know, I actually do use greylisting. And a lot of other techniques, too.

They all add up, and they really do require a lot of extra hardware.

Do you have any clue what percentage of the bandwidth I pay for is going to the initial TCP packets from hosts