Forty Percent of All Email is Spam 625
PCOL writes "There's an interesting article on spam in today's Washington Post which includes an inside look at AOL's spam control center in Northern Virginia. The story reports that roughly 40 percent of all e-mail traffic in the US is now spam, up from 8 percent in late 2001 and nearly doubling in the past six months; that AOL's spam filters now block 1 billion messages a day; and that spam will cost U.S. organizations more than $10 billion this year from lost productivity and the equipment, software and manpower needed to combat the problem."
now i get spam (Score:3, Interesting)
Maybe that's the way to go... (Score:2, Interesting)
Sure, we'll still have to worry about foreign sources, but I'm sure the U.N. will be happy to help with this issue.
Sounds about right to me (Score:5, Interesting)
550 Spammer Go Away!
What is spam? (Score:3, Interesting)
Spam Control (Score:4, Interesting)
I'm no software designer, but surely we could find some concept for migrating off of SMTP and POP and to a better, more secure protocol.
Other thoughts?
-cheezus_es_lard
Technological solutions will be easiest (Score:5, Interesting)
A system like Hash Cash [cypherspace.org] could solve the problem. The most popular free mail clients could start including hash-cash postage with each sent message, and then in a couple of years' time start to drop incoming messages that don't have postage paid. AOL could include hash cash in their mail client easily. *Easily*. That spam-detection centre they run is not cheap. Even Microsoft would add hash cash to Outlook, Outlook Express and Hotmail, since it's another encouragement to upgrade to a new Outlook release (which of course requires a new Windows version).
Getting the whole world to upgrade its mail clients is a hard task, but getting every government in the world to pass anti-spam laws and enforce them is much harder. Goodness knows it's bad enough trying to get _one_ legislature to take a sane view on anything technology-related.
Ratio is higher here (Score:3, Interesting)
A simple solution is replacing the broken SMTP with something that requires authentication and doesnt give you the ability to modify the headers unless you run the server. If the spammers have to use real email addresses or had a real way of tracking them easily attached to every email, they would stop.
Just like how cockroaches scatter when you turn on the lights.
Psychological profile of spammers (Score:3, Interesting)
My theory: most spammers are the cyber equivalent of "flashers" - sexual deviants who derive thrill from shocking unsuspecting citizens. I believe that the products offered are largely irrelevant. It is the shock value which motivates the spammer. Perhaps they could be prosecuted under similar sex crimes laws that allow us to go after the "flasher".
Re:Spam Control (Score:5, Interesting)
It's not a technical issue (ignoring open relays, which can already be fixed without changing any protocols).
The fundamental issue is that one of the most important uses of email is to let anybody, anywhere email you, with no hassle. Of course, spammers take advantage of that.
What's needed is accountability. Give someone internet or smtp access? Make sure you have a way of billing them for any spam they send, and put it in big letters when they sign up.
40 percent by number or by size ? (Score:5, Interesting)
So, when they say 40%, is that by number of messages or total size?
more like 60-70% (Score:5, Interesting)
Blocking spam before it gets to our main mail server has extended the life of our mail server indefinately. The less we have to spend on hardware, the more time and energy we can spend on building quality of service for our customers. That keeps the customers happy, and keeps the business people doubly happy, since they don't lose customers and don't have to buy new hardware every year for a mail system.
Re:Sounds about right to me (Score:3, Interesting)
A friend of mine is a sysadmin at Vanderbilt University in TN. He said they can only place spam filters on client machines, and that no filtering is allowed on the receiving server whatsoever. I asked him why, and he said they believed it was unjust to assume that any message was unwanted by the users, that it was their choice alone to decide what was spam and what was not.. Pretty insane if you asked me.
Re:Accuracy (Score:3, Interesting)
Not terribly. Several years ago, after I first got broadband, I set up my own mail server because my ISP's was constantly going down. I've run it since then with no trouble.
Several weeks ago, I started getting bounces on mail I sent to AOL addresses. Turns out AOL uses lists of IP addresses that are known to belong to ISPs but not be their mail servers and refuses connections from them.
Their attitude is that I have no business running my own mail server, that I should use my ISP's instead (gee, maybe if my ISP's didn't suck I would). So, yes, I can say that at least a few of those 1 billion are legitimate mail.
What the article didn't say (Score:5, Interesting)
is that by data volume or by quantity? (Score:2, Interesting)
Comment removed (Score:5, Interesting)
BrightMail (Score:4, Interesting)
They monitor a LOT of mail boxes...many customers plus many created mailboxes for spam. If a message hits a number of mailboxes in a short time span that message is forwarded to their NOC. A person looks at it and decides if it's spam. If so they tag it as spam before sending it to other customers that receive it.
It works very well. We now block almost all of the spam we receive and have not had ONE single false positive.
Re:Sounds about right to me (Score:2, Interesting)
Re:My tests shows (Score:5, Interesting)
Since the logs were cycled on Sunday morning, there have been 8332 messages, 5824 of which were spam, for a percentage of 69.89%.
This number has increased substantially over the last 3 weeks. This time last month we were below 50%.
Re:Go after the businesses who pay spammers (Score:3, Interesting)
Started working for new company under contract. Help the bossman w/ his spam. Make him do it legitimately by unconfirming all lists and sending reconfirmation notices. Result: 60% reconfirm (including people who had reported us for spamming before). Now we have nice, clean lists and the reply-to/return-path headers are actually LEGIT! Imagine that... an honest bulk mailer. Too bad our rep is already soured. We even have people who are afraid to click on the unconfirm links for fear of being added to another list.
I'm thinking of writing an (anonymous) article for
-- S
Re:Technological solutions will be easiest (Score:3, Interesting)
For example suppose the standard postage amount is a problem which typically requires five seconds of CPU time on modern systems. Then no proxy even if it were taken over by crackers could send out more than one spam every five seconds. This is a greatly reduced rate of spam and probably low enough to make spamming not worth the effort.
What am I missing (Score:2, Interesting)
Type 1- Legitimate headers. No problem you've got someone to harass to remove you from the list. You can look up the domain name contact the admin and generally make their lives difficult. And if all else fails simply block everything from that domain.
Type 2- Forged headers, can't even send a bounce message back no real options for tracing short of contacting the isp in charge of the ip address.
Type 1 doesn't seem to be a problem. Type 2 is where most of my spam seems to come from. It seems that the simple solution would be when
sendmail/qmail whatever is receiving the message and gets the reply-to address it should pause and see if it exists. If it doesn't just leave the connection open and if they are bulk spamming
the server it's coming through will quickly have issues when it has 20,000 hanging connections. When a user pops/imaps to check their mail have the pop server see if the reply-to exists, if they don't dump it to dev null. It would seem that this would keep emails trackable. For it to get to the user the user would have the ability to get back to a person.
So my question becomes, what's the hole in this kind of answer? It seems simple enough. Am i missing something?
And yes i know my spelling is horible...
Re:What say you "just hit delete" crowd? (Score:2, Interesting)
Fair questions.
Let's look at the future: Currently, people are willing to accept email from unknown senders. If the volume of spam continues to increase as you plausibly predict, that is where the system will break. I assume that well-known people already read emails only from whitelisted senders, and that if I send email to, say, Tony Blair or Linus Torvalds it will not get read. As unwanted mail increases, the number of people doing whitelist-only filtering will increase too. Note that this can be done almost perfectly with existing protocols & software, and the only changes that will become necessary will be to prevent forged From: lines, which would not be too big a hole for spammers in any case.
That is what is at stake, therefore: our ability to communcate by email with people we have not established a relationship with. That would be an actual loss, but is it worth legislating for?
Bear in mind that it is not 'given' that there must be a legislative solution, any more than that there must be a technical solution. Both technical and legal solutions run into choppy waters when attempting to separate spam from non-spam.
It is possible that email will slowly die, and be replaced by something else - you can imagine instant messaging expanding into non-instant messaging too, but with authenticated senders and enforced whitelists. SMTP email would become like Usenet, swamped in useless messages, its functions of old taken over by different media.
There's really a workable solution to spam... (Score:2, Interesting)
On my Cingular phone, I have the capability of setting up a simple "Reject if not in list" filter, this weeds out anyone I don't know and anyone I don't want calling me on my cellphone.
On my mail filter I have whitelisting, if you're not on the whitelist, I don't see your e-mails ever. No need to holistic filtering techniques, RBL's, or anything else... if you're not pre-approved to contact me you eat a bounced e-mail.
Now that simple filtering method should cover all end-users, home accounts, and the like. The only accounts that should now be able to receive spam are your group and management accounts. root@, webmaster@, sales@, etc.. cannot readily be blocked this way unless you're looking to minimize your customer and user base (which would be fine on some days...
However, that is one place legislation can take care of business.... Any UBE\SPAM\Junk to management addresses should be punishable by large fines, perhaps some caning, beatings, etc.. as your local human rights limits allow =)
And for those that want to receive spam there is always the opt-in by not using whitelisting.
Your personal whitelist will just be something else you can carry with you like your checkbook or USB drive/smart card...go into an internet cafe, stick in your USB dongle, check your e-mail. Web based e-mail could keep your whitelists in their database, but I see this as a security hole since yahoo or whomever could add themselves to your whitelist as they want.
Re:40% ...? (Score:2, Interesting)
Hah! I got ya beat! I just got email from an account that I couldn't access for almost eight months and there were 7,018 messages in it! Of which 4 were keepers and the rest pure crap. What boggled me was that the account supposedly had a 2 meg limit that the admins never imposed and just let it grow and grow.
Re:Too many problems (Score:3, Interesting)
Also, note that if payment for messages (whether real cash or hash cash) becomes widely adopted, spam will stop because there won't be any money in it any longer. So the problem of costs to the ISP is also dealt with.
Of course it is possible for ISPs to configure their mail servers to check hash postage on each message and drop them if it's not valid. This would save the storage costs of spam. And if a particular other host always sends messages with bad postage you could stop accepting connections from that host. But all this is optional: I feel a postage system has the best chance of getting started if it is adopted from the bottom up by mail user agents rather than ISPs' mail servers. Both is better though.
I don't think that hash cash works by having a problem sent from the recipient to the sender which the sender must then generate the answer to. Rather, you have a one-way function where it is hard to generate the answer but easy to check that the answer is correct. The 'problem' includes the recipient's email address and the message content - so you cannot reuse the same postage for two messages.
The recipient just has to look at the message body, the To: header and the postage, and verify that the postage is a correct answer (which can be done quickly).
Comment removed (Score:3, Interesting)
Re:I thought about it, and you know what? (Score:4, Interesting)
Seriously, what gave you that idea? Are corporations citizens? Do you think they have the right to vote? Does the second amendment apply to them? Does a sufficiently old corporation have the right to run for president, if it was founded in this country?
My impulse is to think that was an incredibly asinine statement, but I do not claim to be an expert on constitutional law. In fact, "mildly informed" is putting it too strongly. So educate me, back up the claim that "Corporate speech and individual speech are equally protected under the First Amendment."
Re:What say you "just hit delete" crowd? (Score:3, Interesting)
It's not that easy (or simple), friend. It's not just about giving your email address away to trusted folks, nor is it about placing your email in places that can be trolled from the web.
About 99.9% of my mail I get is spam. I receive about 2000 emails a day to my personal account and if I'm lucky, 3 of them are legit (I now mostly communicate with my family and friends via IM).
I run my own email server (and have since '93). The problem is that my system is constantly being dictionary-attacked for addresses. No matter where you hide (or don't hide) your account names, having some fucker scan every known name in the universe against your system WILL get it. And once one spammer gets it, they all do. (You do know they trade lists, right?).
As it stands now there is no good way of preventing dictionary scanning. Yes you can make it hard (and I do by catching more than 2 User Unknowns, IP firewalling off the address that started the scan and sending back 1MB of /dev/random data from sendmail as a response message), but when someone wants to scan you, they will; even if they have to do it one address-per-envelope at a time.
My son (who is 11) receives close to 300 spams a day (because he has his first name as his account name [as my entire family does on my system*]), 200 of which would make Solomon blush [hey! click here to see girls get fucked by turtles]. I, of course, filter HIS mail by hand (he pops from an account that I forward mail to).
I *want* legislation; badly. I want it to be illegal to forge headers. Since my state (North Carolina) *HAS* anti-spam laws already, it would be really nice to be able to enact them. But because of the forging, it's next to impossible to do unless I quit my day-job just to parse headers and track down companies so that I can take them to small claims court.
I can't do that. And I don't have the time. And nor should I be required to. So what's the answer: unfortunately, legislation.
SPAM is a plague of locust for the 'net. I equate it to kids who crack/cheat on multiplayer games and make them unplayable by everyone else. SPAM has absolutely ruined the usefulness of email.
* By using full first names as accounts (nicholas@blah.com) it's easy to guess my accounts. I should NOT however, be forced to use anything else because of the abuse of the system by lowlifes who are too lazy to get a real job to make money.
Yeah, I'm vehement about this. Check out other posts by me here to see.
What's the point? (Score:3, Interesting)
First, a fundamental problem: There IS NO COMMUNICATION between your mail client and a sender. Therefore, you have no way of submitting the hash problem TO the sender, he can only return an answer. Therefore, if this even happens, it HAS to be server-based. Re-read the site you quoted, nowhere do they talk about mail clients. There's a reason.
I wasn't thinking of the cost to the SMTP server but of the human cost of spam - wasted time in deleting it and the fact that people are turned off email altogether because of it. This, IMHO, is a much more serious problem than wasted bandwidth.
What, you think bandwidth pays for itself? So eventually your ISP costs go up, not so good. Besides, it's easier to stop spam at the choke point (server) than trying to track it down later. And for people paying to d/l spam on, say, a mobile device, having to d/l it IS the problem.
Also, note that if payment for messages (whether real cash or hash cash) becomes widely adopted, spam will stop because there won't be any money in it any longer. So the problem of costs to the ISP is also dealt with.
Yes, but GETTING it widely adopted is the big problem here. You have to mandate it, probably, and it's easier to get webmasters to switch than, say, my mom, who has no idea what a mail client is. And, for ISP's, the problem is in the voluntary-adoption period. Who takes the hit first? Who starts off with this, when it will increase CPU load even for the sender, while all the spammers are still out there? And how will you get wide-scale participation? It's all well and good to talk about this stuff, but there has to be some method of implementation, where you get from here to total adoption. And voluntary adoption wouldn't work, actually, because the sender's client probably won't understand what the receiving server wants when it asks for the hash, unless they also upgraded to the hash deal. So, in the voluntary phase, do you drop these emails? Do you let them through, defeating the point?
The recipient just has to look at the message body, the To: header and the postage, and verify that the postage is a correct answer (which can be done quickly).
I can look at the header and the body NOW and tell it's spam. Really, I didn't think it was ACTUALLY president Mugabe trying to send me money when I got that email. If you have to d/l the message, look at the message, and look at the header, then there is no advantage over the status quo.
Whitelists! (Score:3, Interesting)
Re:What's the point? (Score:4, Interesting)
I could be wrong on this but having looked at the hash cash site I think that no communication from receiver to sender is necessary. The problem is based on the message body and the recipient name. The sender knows these at the beginning.
The costs to ISPs in the short term will be no worse than at present. In the long term costs to ISPs will fall as spam traffic declines.
You are right that adoption is a problem but that is no reason not to start now. Of the 10% of messages I get that are not spam, almost all are from relatively knowledgeable people who can upgrade to the latest version of Pine or whatever to get hash postage. For other users, it just needs AOL or Microsoft to put out a new release, which as likely as not will be an automatic update. Attaching postage to your message increases CPU load, but only for a few seconds per message sent, and even that can happen in the background.
The advantage over the status quo is that legitimateness of a message can be checked *automatically*. That is the point, you don't have to have your time wasted by checking and deleting spam, this job can be done by the computer. Children do not have to look at pornographic messages, etc etc. Saving time for humans, not computers, is the most important thing. Though like I said, in the long term making spam uneconomical will reduce the load on ISPs as well.
And unlike Bayesian filtering there is no way around it, the message has to cost a few seconds of CPU time or else the postage will not be valid. (Assuming the hash function is cryptographically secure in the sense there is no easy way to get either partial or total collisions with a given hash value.)
Re:SPAM Report (Score:1, Interesting)
Try this on for size:
(various countries).blackholes.us if you like
(spews|spamhaus).relays.osirusoft.com
(rel
(proxies|formmail).re
I used to reject with 5xx errors on things like the DSBL, but have moved them down to 4xx series. That gives the lusers a chance to fix their situation and have the mail "magically" arrive if they find a clue later on.
It also gives me the ability to manually whitelist things that look valid if that doesn't work.
Re:Technological solutions will be easiest (Score:3, Interesting)
You are right that mailing lists would be a problem, but most non-technical users don't subscribe to mailing lists surely? They use web discussion forums or whatever. I don't see customer notifications as a problem, surely each customer doesn't get more than three or four notifications each month and that is certainly manageable. Sending out huge numbers of messages to _all_ your customers isn't feasible, and that is the point.
In a perfect world we would have real cash payments for mail (IMHO); one cent per message or something like that, with the possibility to waive payment for known senders. But that is hard to implement so hash cash is a compromise solution. In any case you have to compare the disadvantages of a hashcash-based system with the current spam-ridden Internet mail system, unless you have an alternative to propose.
Re:Technological solutions will be easiest (Score:3, Interesting)
All this depends on the existence of open relay servers which take messages and compute the postage for them, presumably to support legacy email clients which don't add postage for themselves, and moreover are misconfigured to accept incoming messages from anywhere. Presumably these servers would not be any more numerous than open SMTP relays are now.
You're right that mailing lists are a problem. Such addresses would have to be explicitly whitelisted by their subscribers - or maybe if you tell your mail program 'I am subscribed to misc-discuss@goatse.cx' then it would accept messages which had valid postage for that mailing list address as well as those with valid postage for your own address.
For systems like AOL there is no extra load on the server because the postage can be added at client machines - you see the hourglass for a few seconds after pressing 'send', or more likely, the postage is computed in the background while the message is in the outbox. At least, this is how I think it is intended to work: the Hash Cash site doesn't say specifically whether postage should be computed at the client or on the mail server. But IMHO doing it end-to-end is better.
How the hell do you people get all this spam? (Score:1, Interesting)
Do all the
And the best way to do this (Score:3, Interesting)
Let's face it; if we focus solely on the spammers themselves, we'll have little luck reducing the flow.
But if the court system allow people to sue the companies that contracted out for spam, a few hefty verdicts might cause corporations to think otherwise.
The cost of spam from an ISP point of view (Score:3, Interesting)
Our abuse department is manned by one person 365 days a year, a bunch of scripts, a largish database integrated with our customers database, and lots of red tape. This person calls our customers when they are the source of spam or other non UCE conforming use of our network (including running an open-relay). He explains the situation politely and asks the customer to conform to the policy written in the contract. If the customer does not comply after the first warning, he must look for another ISP to do business with, for we send him an official letter (with official receipt acknowledgement)each time we interact with him.
All in all, given our company size, a bit over 1% of our costs are burnt by our abuse department. Needless to say, we relay these costs to our customers, as do most of our competitors.
This is only half of the cost of spam from our point of view. Our mail servers farm is sized in order to perform well even with 40% of the mail being spam. These are larger human and hardware costs associated with spam as well (though more diluted and thus difficult to pinpoint).
Spam costs people and companies a lot of money, we feel the need for the Internet mail system to be reengineered in order for the cost of sending email to become high enough so that spammers don't get away with their offense.
The Brightmail report is not a big surprise.
Forward it to your Congressman (Score:1, Interesting)
If billions are being wasted in time and equipment on this problem then that's a lot more than the DMA is paying them!
2 rules to remember:
1. Taking money from a group to ignore what they are doing isn't politics, it's a protection racket.
2. When you take an oath of office and then place your self above your constituents that is treason.
" Treason - A betrayal of trust or confidence."
The best investment you could make is "donate" a few thousand dollars to a Congressman. Depending on your industry you could get millions back and a loophole so you have to pay little or no taxes!!
Re:False Positives? (Score:3, Interesting)
I remember one instance not too long ago where AOL even admitted that address had been forged and they were blocking incorrectly, but they couldn't figure out how to unblock manually. This was straight from an AOL represenative's mouth.
Telemarketing and Spam (Score:1, Interesting)
Why could the same principle not be given to spam? Of course this would not apply to spammers from other countries, but it is a start. Just because it is not a total solution does not mean that it shouldn't be done.