Spam Hits 95% of All Email

An anonymous reader writes "Commtouch released its Email Threats Trend Report based on the automated analysis of billions of email messages weekly. The report examines the appearance of new kinds of attachment spamsuch as PDF spam and Excel spam together with the decline of image spam, as well as the growing threat of innocent appearing spam containing links to malicious web sites. Image spam declined to a level of less than 5% of all spam, down from 30% in the first quarter of 2007; also, image pump-and-dump spam has all but disappeared, with pornographic images taking its place."

Re:Summary only link

[speaker of the truth]

Considering this is the land of the goatse posts and I've never heard of commtech before, how do I know this isn't a virus PDF?

doubtful

[jsldub]

I highly doubt that, "All Email"?

Did they track private networks? Encrypted Email?

That's not an unrealistic number

[SaDan]

I work at an ISP and we do SPAM detection and elimination at our border routers. We scan both incoming and outgoing email, and will auto blacklist our own internal IPs if we detect SPAM.

The highest two-week percentage of rejected incoming email that I've seen broke 97% a few months ago. It's normally between 90% and 95%.

It's loads of fun dealing with this crap.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:SPAM @ 95%?!

[Nimey]

Gmail's spam filters have definitely improved. When I first got my account (in '03?) I foolishly posted something to Usenet via DejaGoogle (required my @gmail.com account) and the spam just started rolling in. I still get lots of spam, but almost all of it is properly routed to the spam folder, and thanks to the CustomizeGoogle extension, I don't even see the spam count.

OK, another data point

[CustomDesigned]

Checking my mail stats, since 4 am this morning, I've received 51985 emails, 51909 of which were filtered as spam. That's 99%. Checking the bandwidth monitor, the spam has consumed a steady 100Kbit/s since 4 am, despite being mostly blocked in SMTP envelope via SPF and reputation (SPF blocks forgeries, reputation blocks spammers with the balls to use their own domain).

Re:call me a cynic, but

[Snocone]

Can you imagine any other form of communication that was 95% inefficient?

Flirting.

Let us pick some text randomly off a googled link and exercise our imagination.

"First for Emailing - UK's only Emailing Academy

We are offering you two free e-courses value $45 each. One is our new success emailing communication programme and the other is our popular lifestyle coaching programme

SUCCESS EMAILING Communication Tips - series of 4 communication tips modules. Designed to get you connecting and interacting more easily and effectively plus monthly success emailing newsletter with tips, quotes and news..."

When there is a large industry which advertises itself in terms like that instead of the original [flirtzone.com] then perhaps there would be a point to be made that email communications are unusually inefficient. In the meantime, well, sure looks to me like anyone who has ever interacted with the opposite sex should have no problem imagining a form of communication in which 5% efficiency would be a striking -- well nigh unbelievable actually -- increase, and somehow that communication medium has not died out in several millions of years.

*looks around* Ah .... neee-ver mind.

So where's the invisible hand?

[suv4x4]

Since most slashdotters are libertarians for some reason (and I could argue even I am to some degree) my question is: where's the technological efficient solution to this.

We've seen some "free market" solutions which basically required that you pay a fee to every mail provider so they don't trash your email. And this didn't particularly help spam either.

I come to the conclusion that spam as an issue is one of two things, or both of those things:

1) Not that big of a problem (hard to believe if you are a mail provider / ISP yourself)

2) Impossible to solve by means of free market solutions, and requires cooperation and standardization of new technology.

Point 2 is hard to happen since every little startup that comes with a mini solution, trumpet it on their own and hence they are only a nuissance to deal with in the big picture (due to lack of a single standard, it's impossible to have clients which make the process of whitelisting easier and even half automatic).

Here are couple of solution which would get us half-there, but are only quarter-implemented right now:

1) Whitelist SMTP servers by talking back to the supposed mail of origin and comparing IP-s. The SMTP may return list of IP-s this host responds from. This is then cached and used for further authentication on this domain. It *may* lead to DoS if many hosts do a first-time check simultaneously, but it's unlikely (and less problematic, given we're eliminating 95% of bad emails this way).

2) Test-for-human-intelligence in your first email to a new email. Such as, I don't know, some sort of CAPTCHA you fill-in? Once this is done, communication can proceed without further tests between those two emails. The receiver still has the option to block you, lest you employ a mechanical turk.

Those solutions are boring, they're incomplete in a way, they introduce hassle, but if we *all* agree on those, they can be made less of a hassle, and still not lose their efficacy.

That would require the likes of AOL, Hotmail, Gmail and so on free mail providers to cooperate with the likes of Microsoft, Apple, Linux devs and so on, to implement this on both the clients and servers.

Right now, I could see Hotmail cooperating with Microsoft (.. wink, wink.. :P ), but that's where it ends.

Why we can't stop spam with our current techniques

[damn_registrars]

We can't stop it because we aren't addressing the real problem. Spam is an economic problem. People send out spam because they make money off of it. And they will therefore continue to send out spam as long as they make money off of it.

If you want to stop spam, you have to remove the economic incentive. To do that, you need to cut off the co-conspirators that are allowing the spamvertised domains to be established and hosted. If you can either prevent them from getting a cut off the action, or punish them severely for taking their cut, then you can stop spam.

Until then, if all we do is try to filter spam out, we'll just continue to see the costs of inaction. Beyond that, we're ignoring the fact that filtering has real costs, as well. Filtering doesn't prevent the spam from traversing the internet, and furthermore it requires human time to update as the spammers change their tactics.

Email is dead, long live Email

[kthejoker]

As email asymptotically reachs 100% spam, we will have essentially created a mechanism whose sole goal is to deliver us undesired ads and scams. Talking about spam detectors and blockers and blacklists is irrelevant. Why devote all of this energy to ensure that maybe 5, 10, or 20 people can contact you or your business a day? Or even 20,000, which only highlights the issue that separating spam from valid emails is just bad juju. Simply put, there is no solution to asynchronous communication that is not too tedious or too restrictive. We'd be a lot better off if we blew up all the email servers, and put all of the energy and cost savings into developing encrypted telepathy. You think I'm kidding.

Only a few more percentage points to go...

[s_p_oneil]

...before it reaches the level of spam I get in the mailbox in front of my house. I swear, if we want to save the trees, we need to start by arresting the people putting all those unwanted 20-100 page sales catalogs in everyone's mailbox every day.

There is already a solution for 90% of the problem

[Anonymous Coward]

SPF for the moment, transitioning to DKIM as quickly as possible.

Unfortunately, the largest sources of the problem (comcast, rr, attbi, etc.) believe they derive income from NOT stopping viruses, worms, and spam, and they aren't held accountable for damage caused by their greed. So their misinformed and incompetent staff are not going to implement any fixes.

If you haven't implemented SPF, and aren't seriously studying DKIM, you should not be a mail service provider. A person shouldn't expect a plumber's apprentice to perform a colonoscopy properly, after all.

But armchair libertarians often forget that laissez-faire capitalism only functions properly when all customers are perfectly informed. Most people never heard of standards-based anti-spoofing technologies and do not understand how preventing spoofing impacts spam management; so they cannot make the informed choices that would allow "the invisible hand of the marketplace" to become an iron fist crushing the incompetent service providers.

Re:Greylisting to the rescue! (or not)

[Anti-Trend]

I knew somebody would bring up greylisting. :) During the business day[1], I work for a company that produces several widely-used anti-spam appliances and a service-based filter as well. We see about 2,000 networks a week, and get a pretty good feel for spam trends and countermeasure effectiveness. I can say with all honesty that in my experiences, greylisting hurts more than it helps for most organizations.
 
Basically, greylisting is putting an email transaction on hold to see if the sender will retry. The idea is that if the sender is illigitimate, they won't bother resending. However, spammers have been onto this method for as long as it's existed, much moreso lately. All they have to do is take greylisted hosts and move them to the end of their script for later processing. The second time around, the spam gets through anyway. Even with its meager benefits, most organizations want email to come through as quickly as possible, and greylisting delays email by its very nature. It's also much less effective than existing technology that won't hinder most legitimate mail like DNSBL and/or SPF, spamwords+OCR (for image spam), and blocking on unknown recipients.
 
  To summate, if greylisting makes you happy, then don't let me dissuade you from using it. it does indeed stop some spam. But please don't give the false impression that it's a magic bullet; most of the complaints we receive are from clients who've enabled greylisting and can't figure out why their mail is delayed.

[1] I am also a consultant to another firm who hosts manged email with spam filtering. Due to the complaints above, we have also disabled greylisting there. It was only effective at stopping about 5% of spam reliably, but a delay is put on all mail that isn't otherwise whitelisted. There are plenty of other methods which are both more effective and don't slow down the mailflow or tie up much resources on the MTA.