Forty Percent of All Email is Spam

PCOL writes "There's an interesting article on spam in today's Washington Post which includes an inside look at AOL's spam control center in Northern Virginia. The story reports that roughly 40 percent of all e-mail traffic in the US is now spam, up from 8 percent in late 2001 and nearly doubling in the past six months; that AOL's spam filters now block 1 billion messages a day; and that spam will cost U.S. organizations more than $10 billion this year from lost productivity and the equipment, software and manpower needed to combat the problem."

My tests shows

[brakk]

90% of my email is spam

Accuracy

[NitroPye]

I wondder how accurate the AOL spam filter is. If some people are accidentaly getting their emails blocked or others not getting emails delivered. Does anyone know on which principal the AOL filter works. Is it just a bunch of email addresses known to be spammers or is it some kind of guessing filter that has certain words and phrases coined as spam.

Optimistic

[Rosonowski]

I think this is a bit optimistic. I get 300 peices of email a day, and I'm lucky if more then 50 are legitimate mail.

Does not surprise me

[nenolod]

I'd say more like 60% though. However, i'd also say that 40% of idiots make up statistics to prove their point, and 90% of people know that.

Anyway, I get about 1800 messages a day, total. Messages are ran through procmail and a complex spam filtering perl script that I wrote for myself. about 600-700 messages are blocked per day, therefore being more than 40%.

I'd also state that most SMB popups are SPAM.

Mod parent as FUNNY

[Anonymous Coward]

"call your congressperson and have them pass an anti-spam bill. that's the only way to solve this problem"

Yeah right. Just like the new telemarketing bill that has loopholes for some of the worst telemarketers.

Take this with a grain of salt

[mrhandstand]

The srticle states that 40% of Internet traffic is Spam. And where does this statistic comec from? From Brightmail...a vendor of anti-spam software. Remember...liars, damn liars, and statisticians

Losing a figurative war on spam

[Nonac]

Aside from the AOL spam control center, most of the spam prevention discussed in this email is aimed at trying to stop the sender through legislation and black lists. Legislation will never work, and black lists are marginal.

The answer to this shortcoming in the current email infrastructure is redesigning email protocols to allow spam to be stopped as it is sent.

I don't have the answer, but something that forces the sender to verify that the recipient will accept the message before it is relayed will be a start. I also like the idea that came from Microsoft recently of forcing the sender to pay the recipient a small amount of money.

The problem with bayesian filters is that they filter too much spam. The more people that use bayesian filters, the more messages the spammers will have to send to get through. Because it is almost free to send messages, they will continue to increase the number of messages they send until it gets to a point that email infrastructure can't handle it anymore.

Re:Accuracy

[bheerssen]

A follow up question: how much spam gets past their filters and do they use a standard deviation accordingly to arrive at those numbers? It is conceivable that the actual figure is higher.

What say you "just hit delete" crowd?

[walt-sjc]

Citing "Freedom of speach", the first ammendment, etc, there still seems to be an ignorant crowd that thinks that we shouldn't have any legal means to curb spam. They still think technology can solve a social problem. As ISPs put increasingly invasive filters on email servers, legit email gets lost. When 99% of all email is spam, will you STILL think it's ok? When ISP's raise your internet fees due to spam, will you still defend its legality? When you are on the road paying $.50 / minute downloading spam for half an hour, even though your local filter blocks it from your view will you still be happy?

There are people who want to re-invent the email protocol to solve the problem. Yeah, doing something technological can help the FUTURE, but what are we going to do for the 5 years it takes to develop, implement, and deploy this new technology?

Think about it.

Antisemitic post

[Anonymous Coward]

Kind of funny when a post that attacks someone for "fear" includes a bit of anti-semitism midway through it.

"Propaganda's m'friend But I calls it "fact""

Propaganda is in fact typically a meaningless term. Quite often what is labelled "propaganda" is factual, but it is labelled "propaganda" for the sole reason that the opponent does not agree with the facts presented and would rather see them censored or otherwise dismissed.

isn't it ironic???

[Botchka]

that the biggest purveyor of filling my postal mail box with crap that I haven't signed up for or asked for (ie: cd's and cd holders that are worthless), is now fighting spam. Give me a break! How about they stop mailing those stupid #@%@$%^& cd's and filling the landfills with garbage that doesn't degrade. They are hypocrites!

Go after the businesses who pay spammers

[kalislashdot]

You know it's a funny thing because businesses like and hate spam. They like it because it brings in money and they hate it because they have to spend money on spam filters and lost work time.

Here is a possible solution. Spammers cover their tracks. Well instead of trying to go after spammers go after the business that use them. Those businesses MUST be traceable because they include ways to buy their product. If we must make a law, which would only work in the US, it should say "You can't hire a spammer to send your mail". Then when www.pacificmeds.com sends me a spam for "save money on prescription drugs" they can be fined.

Go after the source, not the person who fills the need. Once the need is squashed by the law spam will reduce greatly.

Re:What say you "just hit delete" crowd?

[ErikZ]

"Yeah, doing something technological can help the FUTURE, but what are we going to do for the 5 years it takes to develop, implement, and deploy this new technology?"

Probably the same thing we would do if we didn't develop the tech. Just sit there and delete spam.

I hate spam (Don't we all)

[SirLantos]

{Complaint}It the past 6 months are so I have been recieving about 200% more spam. I get to work in the morning and delete 90% of my e-mail becasue its spam. Out of every 200-300 e-mails I recieve, I actual only care about 10-20 of them, the rest is spam.{/Complaint}

The problem is that nobody can find a reasonable solution. Here are some examples of common solutions:
1."Make spam illegal out right."
Problem: OK, this is a bit extreme. Even if you did manage to do that, companies from outside the US or companies/people can hide where the e-mails are coming from, good luck catching them.

2."Charge for e-mails."
Problem: The people that want that are the post office folks. I seriously doubt anybody would sit back and allow this. Just thinking about pisses me off.

3."Find the people that send spam and destroy them."
Problem: OK, this is my personal favorite. But, the goverment already made that illegal. It's like the saying goes: "Some people are alive simply because it is illegal to kill them." BTW, all of you peeps out there that are going to yell at me for suggesting something like that: RELAX, IT WAS A JOKE!!! Have a sense of humor for goodness sake.

That's just my opinion,
SirLantos

Re:Spam Control

[Qzukk]

I'm no software designer, but surely we could find some concept for migrating off of SMTP and POP and to a better, more secure protocol

Sure. Just like we convinced everyone to close off their open relays. Not going to work.

Re:Spam Control

[Ravensign]

I agree with this principle.

At what % do we look around and say, its time for a new protocol with spam avoidance built in?

50, 60, 75?

Old fashion torch burning mob

[boy_afraid]

What we need is an old fashion torch burning mob running around and storming these Kings of Spam to roast them alive. I'm sure we would have NO objections from anybody. I hate the strange hints that my penis is too small. Is it? Is there something someone is not telling me? I don't think it's too small. I think it's normal.

WTF is up with these printer ink mail. Do they think I'm running an illegal printing shop and I need all this ink because I'm running 24x7? I should be printing fake 20 dollar bills and but more ink!

Accountability Void

[ipmcc]

If ISPs could find some way to limit each accounts number of outgoing messages, or charge per outgoing message over, say, 500 messages a day, this would probably be much less of a problem.

At the core of this problem is the Accountability Void, and the temptation that carries with it. When you look at the lengths that (some) ISPs and watchdogs go to block (much to libertarian chagrin) kiddie porn and other potentially offensive material, its clear that solving the spam problem is NOT about technical feasibility. If there was impetus there would be a solution. The problem is that the ISP can say "we dont send it, we dont receive it, its not our problem," the spammer can say "I send it, but I use fake accounts that get closed in 6 hours, so I don't have to take responsibility for it" and, for the most part, the receiver says "I received this, but theres really not much I can do about it." I describe this phenomenon as an "Accountability Void." No one is responsible for spam.

Until there is an accountability structure in place, either legislative, technical, or economic, spam will go on. One of these days, AOL or some other "big enough" player is going to do something that will "change everything" like demand digital signatures, or some other method that fills the accountability void and spam will cease to be a problem.

The spam is hidden, not gone

[phorm]

The problem is, you are still getting spam. The filter may block you from seeing most of it, and it may stop you from getting tags with linked images, etc... but it's still coming in.
You, and your ISP, are paying for the bandwidth it uses. And if you ever had to travel and get email by dialup/cellphone... you can expect that you'll notice spam simply by the large delays it takes you to download email.

Client-side filters only mask the problem... it's like having an air-freshener and big fan in a public washroom.... the stink is still lingering in the background.

Re:Spammunition

[walt-sjc]

Is your spam problem GONE or is it simply hidden from view? You and your ISP have alreay paid the cost of that spam. The cost to you seems minimal, but to a large ISP it is HUGE. When your ISP raises it's rates due to the volume of spam that you do not see yet still receive, will you still be happy with your filter as "The Solution" to spam?

Don't get me wrong, I have been filtering spam for years. Filters can minimize the impact of the spam problem, but they do nothing to solve it.

Re:What say you "just hit delete" crowd?

[Ed Avis]

'Using email intelligently' consists of having multiple email addresses and trying to keep them secret? WTF?

Re:Technological solutions will be easiest

[Anonymous Coward]

The easiest way to transfer the cost to the sender is to change the protocol so that mail is stored on the sending server, and that only a message header with a link to the message body is stored on the receiving server.

This way, spammers will either a) need their own server (which can unambiguously identified and blocked), or b) the spammer's ISP will have to store the millions of message bodies and handle all the generated traffic. This will give a good incentive to the spammer's ISP for getting rid of the spammer.

Also, this would get rid of fake originating server identification in headers, as a fake header would lead to no message body :-) .

For this to work, it is important that the message subject be considered a part of the message body, not its header. Otherwise, spammers will try to put their message in the subject!

This would be a simple yet very effective solution.

Re:What say you "just hit delete" crowd?

[Christianfreak]

Your rant doesn't make a whole lot of sense. I don't think the problem is that people think that SPAM shouldn't be regulated, (okay maybe a tiny minority), its not regulated because there is no way to do so. I see very little SPAM that doesn't have forged headers or that didn't come through an open relay.

We don't need new laws. The SPAM is already illegal. You can't enforce a NO SPAM list because a) spammers are difficult to track anyway and b) even if they weren't there is nothing finacially or otherwise preventing them from re-routing their SPAM through international servers.

That said I think a lot of the filtering software misses the point. Its not as difficult to find the owners of open relays. I really think that we should go after ISPs that knowingly or not have open relays. Easier to track than the spammer himself and if you get the open relays you stop a whole lot of spam right away.

As for overseas sites, maybe thats where we need treaties and insentives for foriegn governments to crack down on said open relays (I know it will never happen). In the meantime that's where filtering is a good idea.

Re:What say you "just hit delete" crowd?

[Anonymous Coward]

"the great unwashed AOL users will whine until their gov't wastes more of my tax money."

As oppose to the spammers wasting your money?

The money it takes to move that spam around comes out of your pocket and out of your employer's pocket. Free speech is "free;" it doesn't cost the person or the listener anything. Spam is NOT free speech because it is costing YOU resources, even indirectly.

And though you are keen to point out that you don't get very much spam (I don't either, by the same methods you employ), my dept has to support the infrastructure that is impacted the most by spam. Money to cover this comes out of my employers pocket. I work for a medical center which means that it's money (albeit small in the grand scheme of things) that is taken _away_ from healthcare.

Holding ideals such as freedom of speech is noble enough, but you've got to be careful about applying your views to the real world, lest you bind yourself into a situation (like this) that ends up screwing you over.

Re:Sounds about right to me

[Chanc_Gorkon]

With one exception......viruses. Filter these suckers out at the server. This will cut your opps I clicked on something I should not have syndrome.

Only for AOL?

[www.sorehands.com]

The article stated the figure came from Brightmail not AOL.

If it was AOL or Verizon, then I would think that the numbers would be skewed as they have sued spammers and those spammers have agreed not to send spam on those networks.

Grasshopper, remember the two rules of spammers.

1. Spammers lie.

2. If a spammer says anything, see rule 1.

Re:I thought about it, and you know what?

[Wntrmute]

Truth in advertising laws.

Restrictions on how/when/where some businesses can advertise. (Tobacco/Alcohol)

Nike v. Kasky [motherjones.com]

It's not as clear-cut as you make it sound.

Compare it to the real world:

[Fritz Benwalla]

About 18 percent of the traffic carried by the US Postal Service is bulk mailing, but USPS studies say that postal employees spend 25 percent of their time sorting it. All a waste? Keep in mind that the DMA asserts the $50 billion was raised as a result of bulk mailings by charities.

I'd be interested in knowing what the total load on our economy is from the two forms, inluding manpower, network load, inconvenience etc. My suspicion is that the hyperventilation over spams growth is driving up the percieved cost, especially when you consider the cheapness of bandwidth, and that spam control is an automation battle leaving the real expensive resource, humans, to design the filters and clean up what they miss.

"The spammers are evil folks," Evil? Like Hitler evil?

Opportunists, yes. Using mildly unethical means to further themselves in business venture, often. But I wonder how many people who are apoplectic about the "evilness" of spammers cheat on their wives, cheat on their taxes, park in handicapped zones, etc. . .All no more evil than faking a return address, and certainly no less.

-----

Why 40% does not seem unrealistic.

[DaemonSD]

A lot of people here are saying that more than 40% of their email is spam and that the figure quoted is somehow wrong. A lot of people here also fail to take into consideration that the 40% figure is very likely an approximation or an average and is not valid for every single user on the internet. Being computer literate, having a website, posting on different websites and other internet activities contribute to more spam because of email harvesting. Sure, you and I get more spam than the average Joe, my spam is more like 80% of all emails received, but do not forget about all the people that are on AOL and have only given their email to their family relatives. Granted, they will receive some spam too, but surely not as much as the rest of us.

Re:now i get spam

[You're All Wrong]

Nice innit?

However did you notice in the article it said:
"nearly doubling in the past six months, according to Brightmail Inc., a major vendor of anti-spam software."

So I'm not 100% sure the stats can be believed - it's in their interest to tell you it's all doom and gloom. It's even in their interest to have you spammed, but that of course would be conspiracy theory central...

YAW.

Re:There's really a workable solution to spam...

[Fritz Benwalla]

Maybe I'm not getting this. I don't know what you do for a living, but let's say you're a web designer.

I get your name from Bob, a former client, and I want you to design my web site. I send you e-mail to that effect and you never see it because I'm not on the white list? If that's the case it would be unworkable for 90% of business e-mail, since most of what you're trying to do is make contact with prospects you've never met before.

Sorry if I misunderstood, but it seems as though for business e-mail accounts I'd spend more time manageing my white list that deleting spam.

-----

Re:What say you "just hit delete" crowd?

[Azghoul]

Apparently you're vehement, but I do not agree that it requires legislation. When you drive into a downtown at night to see a show, and you park your car, do you leave it unlocked? You do things ALL the time to prevent abuse against you, why should using better/different email processes be any different??

It /is/ easy and simple, "friend". I've had many accounts over the years and i DON'T get spam unless i make a mistake (like I noted in my first post) or I have a separate account just for junk.

Explain to me how, "friend", that I can get away with not having anywhere near the spam problem you do, even though I've been online for about as long as it's possible to have been.

Sorry you're so vehement against it. It's a techical problem, there are technical solutions.

Re:What say you "just hit delete" crowd?

[amcguinn]

If we go the way I described, you have two options

1. Just carry on. Your emails from strangers will be buried in ever-larger piles of spam, but they mean a lot to you so it's worth receiving them.
2. Use a web-based "comment submission" system on your web site instead of listing an email address.

Number 2 might not be available to you today if you use a basic static-content-only web hosting service, but it is likely to become more widely available as the need for it grows. In theory it is also capable of being spammed, but it is easier to protect than an email address.

As to whether legislation will work; I'm leaving that question to others. I have my doubts, though.

Remove the Filters

[jetsetscoot]

Is 40% what the user sees or what hits the ISP?

What if for one day - 24 hours - everyone who is running a spam filter at any level simply took the filters down. Show the users what the real flood of junk looks like. I bet the hue and cry would provoke real efforts - legal or technical - to solve the problem once and for all.

I find myself thinking; what's all the fuss about, I only actually see a half dozen spam messages a day in my Hotmail and POP accounts. But I know that for every piece I see there are untold dozens being blocked by filters. Filters merely hide the scope of the problem from the end users, but ISP's still have to deal with the bandwidth.

Take down the filters for a day and let everyone see the real scope of the horror that is spam

-Jetset

- I can't hear the forest for all the falling trees-

Re:Accuracy

[corbettw]

Umm, what AOL is doing is right and proper. Is your host the MX record for a domain? No? Then noone should be accepting mail from it. Can your host be authenticated with reverse IP look-ups, crosschecked with MX? No? Then, again, noone should be getting your mail. (All except your own ISP, that is.)

This might be inconvient for you, but this system exists as a deterent to spammers. Don't like it? Get your own IP addresses for home use or host your own domain somewhere (that's what I do).

Re:isn't it ironic???

[hkmwbz]

Except they paid to send you stuff through snail mail. Spammers basically use other people's bandwidth and disk space to send out their crap. Hypocrites? Not at all.

Re:False Positives?

[Anonymous Coward]

And AOL doesn't have 40 million users - they have 40 million SNs (screen names). If they counted only one user per account instead of SNs, their claim would be substantially smaller.

Re:100%-ish effective spam-prevention technique

[gdr]

This works until one of your friends enters your email address into a form on the web (say to send you a electronic birthday card) and it gets added to a spammers list.

It's also possible that a spammer could harvest email addresses using a Outlook virus that infected one of your friends or anyone who has been sent an email that has your email address in the header (or body for that matter).

I don't know if these sort of viruses are common but if they're not now they could be in the future.

Having multiple email addresses is a good idea but, unfortunately, not a perfect solution. Once your "safe" email address is in the hands of a spammer they can pass it on to other spammers and it can become unusable quite quickly.

Re:Maybe that's the way to go...

[dpille]

...with economical impacts in the billions might attract enough federal attention to get some standardized laws...

Maybe I'm too cynical, but I'd expect to see a tax cut to benefit the wealthiest spammers instead of anything that would help the common email recipient.