Has Microsoft 'Solved' Spam?

MsWillow writes to tell us the Seattle PI is running a story looking back at Bill Gates promise to have the spam problem "solved" in two years. Well, it looks like time is up, and the verdict is -- an emphatic "maybe". From the article: "Microsoft says it sees things differently. To "solve" the problem for consumers in the short run doesn't require eliminating spam entirely, said Ryan Hamlin, the general manager who oversees the company's anti-spam programs. Rather, he said, the idea is to contain it to the point that its impact on in-boxes is minor. In that way, Hamlin said, Gates' prediction has come true for people using the right tactics and advanced filtering technology."

Same way they solved Virii

Give me a break, I very distinctly remember Microsoft saying that with the advent of protected mode operating systems that virii would become a thing of the past. Hmmm, do I even need to say any more?

Re:Same way they solved Virii

[Brian is writing graffiti on the palace wall. The Centurion catches him in the act]
Centurion: What's this, then? "Romanes eunt domus"? People called Romanes, they go, the house?
Brian: It says, "Romans go home. "
Centurion: No it doesn't ! What's the latin for "Roman"? Come on, come on !
Brian: Er, "Romanus" !
Centurion: Vocative plural of "Romanus" is?
Brian: Er, er, "Romani" !
Centurion: [Writes "Romani" over Brian's graffiti] "Eunt"? What is "eunt"? Conjugate the verb, "to go" !
Brian: Er, "Ire". Er, "eo", "is", "it", "imus", "itis", "eunt".
Centurion: So, "eunt" is...?
Brian: Third person plural present indicative, "they go".
Centurion: But, "Romans, go home" is an order. So you must use...?
[He twists Brian's ear]
Brian: Aaagh ! The imperative !
Centurion: Which is...?
Brian: Aaaagh ! Er, er, "i" !
Centurion: How many Romans?
Brian: Aaaaagh ! Plural, plural, er, "ite" !
Centurion: [Writes "ite"] "Domus"? Nominative? "Go home" is motion towards, isn't it?
Brian: Dative !
[the Centurion holds a sword to his throat]
Brian: Aaagh ! Not the dative, not the dative ! Er, er, accusative, "Domum" !
Centurion: But "Domus" takes the locative, which is...?
Brian: Er, "Domum" !
Centurion: [Writes "Domum"] Understand? Now, write it out a hundred times.
Brian: Yes sir. Thank you, sir. Hail Caesar, sir.
Centurion: Hail Caesar ! And if it's not done by sunrise, I'll cut your balls off.

Re:Same way they solved Virii

We're also speaking English, not French. So we don't need some committee to tell us which words we can and can't use. Virii makes reasonable sense, sounds cool, and is immediately understood.

You're saying we should all use the poncy variant "virii" for viruses because you prefer it. Are you sure you're not French?

Re:Same way they solved Virii

You just wrote : A burn inside hell, boy of monkey.

Don't use translation tools.

Re:Same way they solved Virii

It's almost as bad as that non-sensical word: irregardless.
Oh come on now. For all intensive purposes it means the same as regardless.

Ugh. now I feel dirty.

In short...

Microsoft has solved spam by ... erm... recommending all the strategies that people were already using before Microsoft set out to solve spam. A hearty thank you to Uncle Bill, then.

Re:In short...

You missed one. Microsoft solved spam by ... redefining "solved."

Embrace and extend

Embrace: solve Pronunciation Key (slv, sôlv)
v. solved, solving, solves
v. tr.

1. To find a solution to.
2. To work out a correct solution to (a problem).

Extend: 3. Not actually find a solution to. See half measure, plagarism.

I knew it! Microsoft is behind OpenSource!

SpamAssassin [apache.org] & Thunderbird [mozilla.com] heuristic learning, have been keeping my inbox 99,7% spamfree for the last 2 years.

Stupid as i am, i never realized that i have Microsoft to thank for it.

When you fail,

try, try again.

Or you can move the goalpost in the middle of the game. That's easier.

Eliminating spam means eliminating spam!

Horse before the cart

You solve spam when it stops being sent, not when you stop recieving it.

These technologies wont work until they are nearly 100% effective. If even a few messages slip through to some users, some people will buy things from spam ads. Which is all the economic incentive a spammer needs. So all they do is hide the problem, not really solve it.

Bandwidth is still being wasted.

Michael

A Plan for Spam

I scoff at Bill Gates' "efforts" to reduce spam. What has he done precisely?

Probably just deferred the responsibility to one of his underlings. Aside from that, he talks about crazy methods such as deciding how much money the sender has to pay you before you open the e-mail [cbsnews.com].

Gates has plenty [microsoft.com] of articles [microsoft.com] which detail how much he hates spam. Anyone can sit down and write this, but Gates gets the high exposure interviews with the Wall Street Journal and the AP.

Gates is all talk. If you want to read some articles from some very interesting people, check out A Plan for Spam [paulgraham.com] by Paul Graham. It talks about simple ways to write Bayesian spam filters and does a very good job at describing how they work. Another valuable member of the anti-spam community is Jonathon Zdziarski [nuclearelephant.com] who has written many books about how to actually get rid of spam. You can also read the Slashdot interview [slashdot.org] with him.

My Hotmail Inbox

My Hotmail Inbox averages about 2 spams a week. However, my "junk mail" occasionally has a legitimate email dropped in there too. However all things considered, 2 spams a week in my Inbox isn't that bad.

So, yeah, Microsoft may have "solved" spam .. but their solution has rounding errors.

Re:My Hotmail Inbox

My Hotmail Inbox averages about 2 spams a week. However, my "junk mail" occasionally has a legitimate email dropped in there too. However all things considered, 2 spams a week in my Inbox isn't that bad.

That's not "solving" spam, that's masking it. My company uses RBLs at the external mail gateways to try and control the flow of spam into our network. 80% (200,000 of 250,000 daily messages) is directly blocked via this method... that bandwidth is still being used, but we halt the flood of the e-mail to our internal mail servers before it can be a burden to our users.

Of the mail that does get through, another 20% is still spam that didn't get blocked by an RBL so it has to pass through another anti-spam gateway (spamassassin) that does analysis and tagging of the message before passing it on to the internal mail server. Of the mail that gets through, roughly 5-10% is probably mismarked as not being spam when it is. That ends up being a shitload of mail that still gets through into a user's inbox that they have to review and delete. Spread that across thousands of users and you have a very real problem.

What we really need are vigilantes to go out and kill the spammers. We have their names and their addresses on the ROKSO list. Kill those 200 spammers and it'll prove a powerful lesson to the remaining ones that haven't popped up on the radar yet. People need to learn that if they spam they will die. Without that threat I'm afraid spam will only become an ever-increasing problem until there will come a point where e-mail is a completely useless medium to use for communications without redesigning the protocol.

So, anyone got an ex-con brother who doesn't care whether he lands back in prison or not? ;-)

/joking of course, please don't kill anyone... just break their hands.

Re:close as i get

I get tons of spam in gmail but it all goes to the spam folder. I don't even remember the last time I had spam/phish mails on my gmail inbox.

Hotmail's Spam Filter is TOO Good

I run an Internet business. I hate when people write me from a hotmail address because there are low odds that my even *replying* to their email will get through their filter. Every once in awhile I'll run into this situation...

Customer with a hotmail address emails me with a question.

I hit reply and give them my answer

A few days later they write me again asking why I haven't responded.

I reply again. They don't get my response. They then get pissed and I lose the sale.

The problem is that Hotmail errs on the side of filtering out too much when you can't even reply to a hotmail user. And many people don't even bother to check their "spam" folders.

I'm no computer engineer, but I would think that merely replying to an email should make it through a spam filter 100% of the time. It's amazing that a company like Microsoft can't hire engineers competent enough to figure that out.

Well as a computer engineer

How would you know that an email is a reply?

I am to unpopular to get a lot of spam but the few I get on my gmail account all seem to be beginning with "Re:" clearly seeking to trick me into believing it is a reply.

Of course you could check the headers but these could easily be faked. In seen spams in the past that got through where I had real trouble figuring out where the fuck they came from. Some I even seemed to have sent myself.

The only real way to check it would be for hotmail to keep a track record of everyone you send mail to, add them to your adress book and then let those emails bypass your spam filters.

Silly Hotmail for not doing that. OH wait, they do! When you send an email via hotmail you are asked wether you want to add that person to your contact list. Most people don't bother.

My tip to you? Make it very clear that if they contact you via hotmail it may be filtered. Also check why you are being spam filtered. Is it based on your hostname or is the content of your email to spammy?

I know your pain, I dealt with it myself although in my case I am not depended on hotmail users so simply don't care that much. It is a lot of extra work but that is the cost of spam. No spam, no spam filters. It is something people often forget, it is not just the bandwidth cost and the time wasted sorting through spam but also the fact the real emails get lost in the mess. But don't worry, Bill Gates promised he would solve it. Has he ever lied before?

Re:Well as a computer engineer

How would you know that an email is a reply?

You're a computer engineer and you don't know about the "In-Reply-To" smtp header?

I don't know whether I'm being Informative or Flamebait here...

Justin.

Re:Well as a computer engineer

How would you know that an email is a reply?

Using the In-Reply-To: header flag, perhaps? It uses the unique Message-Id. That's how threading works (in good MUAs - Thunderbird has it's own very very strange message threading). Save the message-id for outgoing e-mails, for each user. When a message is received, match the In-Reply-To header against the list of Message-Ids. If it's there, whitelist.

Easy.

Paul Graham

I thought that Paul Graham and some other folks, solved this problem with Bayesian filtering.

Paul Graham has a famous essay, A Plan For Spam: http://www.paulgraham.com/spam.html [paulgraham.com]

I thought spam was dead...

There is this site called Slashdot [slashdot.org] that reported this just 10 days ago...

That's an easy one...

No.

But, to their credit, that is an extremely hard problem to solve. In many other areas of software engineering, where you "solve" a problem once, the solution is much easier because it is just a technical limitation to be overcome. Spam is different, however, because you're fighting against other people all who have strong financial incentives to defeat your system.

I'd still say "don't promise what you can't deliver", though. As some critics have pointed out [blogspot.com], failure to do that just may be a systemic problem at Microsoft right now. Hopefully there will be some internal accountability for this one.

Lies, Damn Lies, and Marketing

It's amazing to me how adept markedrones have become in spinning reality to fit their needs.

Spam still chokes mail gateways and causes everyone who uses email a hassle. You still can't advertize your email address. Upwards of 90% of the mail that reaches my mail server is spam, usually. Mail filters have been there for more than two years, though they've gotten better as spam has gotten better.

Spam volume has leveled off, but that's mostly because the system is already saturated.

If Microsoft really wanted to do something about spam, they'd fix the bugs and unthinkable design decisions that has allowed their software to be taken over and used to send it.

Microsoft? More likely everyone else.

Even if we've managed to keep spam to a minimum, and we've changed the word "eliminate" somehow to mean "reduce", can anyone honestly say we have Microsoft to thank for all this?

Oh, and that prediction I made 5 years ago about reducing telemarketers' phone calls? You can all thank me now.

Outlook 2003's spam filter has solved it for me

Actually Microsoft has done far more than anybody else in helping me with Spam. The spam filter for Outlook 2003 is very good and Office Update regularly provides updates to the filter that bring it up to date with some of the latest major sources/types to look for. I set it up a level in how aggressive it is, which has resulted in a false positive or two every now and again, and I have not seen any spam in my inbox in some time.

Don't knock MS on spam until you see Outlook 2003's spam filter. The question becomes if they have the technology that they do in Outlook then why can't the incorporate it into hotmail as well? I would ask the same question about Exchange but I guess they figure most people using an Exchange server are doing it with Outlook.

What to do with SPAM when you get it

Other initiatives by the company include efforts to teach consumers about what to do with spam when they do receive it.

Here [pitt.edu] is an idea:

THREE BEAN SALAD w/SPAM!

7-oz can SPAM, cubed 1/2"
1/3 cup choppd onion
16-oz can cut green beans, drained 1/3 cup sugar
1/3 cup cooking oil
16-oz can yellow wax beans, drained
1/3 cup cider vinegar
1/4 tsp pepper
16-oz can kidney beans, drained
1 tbsp stone ground mustard

In medium bowl combine SPAM, green beans, wax beans, kidney beans and onion. In small bowl combine remaining ingredients; pour over SPAM mixture. Stir gently, mixing thoroughly. Cover; refrigerate 2 to 3 hours or until serving time. Yield: 6 servings.

Business plan

Step 1: Make outrageous promise
Step 2: Make sure the media pick it up and spread it around
Step 3: Do nothing
Step 4: Redefine what you meant 2 years on
Step 5: Profit!

A bit more complicated than the underwear gnomes' business plan, but much more profitable.

Supply and Demand?

Microsoft and Google and Symantec are not the warriors on the spam battle front. They can do nothing to properly reduce the costs of fighting spam (the costs that the end user doesn't see but definitely pays for). The warriors are us, geeks and techies who know the real solution.

Spam continues to be produced because it is generating income. I like to don my black hat and look at the spam forums and see that there still are people making boatloads of money for little investment. Investing US$10,000 in a spam campaign has net some people US$50,000 in a few months!

Why does spam generate income? Users continue to click. I have e-mail relationships with people all over the world on a daily basis, and it really blows my mind how some very bright people seem to be Internet morons. I honestly believe that the great majority of the world's Internet users have no idea how to properly browse or read e-mail.

Turning off images is a huge step in the right direction (I had already told many people to turn them off if the e-mail programmed allowed it). What other things have you told your friends or family to do to prevent the dreaded "my computer is so slow" phone call? How many times have you EVER clicked spam? The ratio is the answer to the question: teach others proper Internet usage techniques.

How Does Microsoft Change a Light Bulb?

They don't. Instead they define dark as the new standard.

And you thought it was a joke... receiving spam is now the Microsoft definition of being spam-free!

Spam is not 'solved' by filtering

Spam is often seen by companies as a method to make another profit. They come up with a box or a product that usually should be able to weed out any spam, and YOU, the customer and/or enterprise, should pony up some cash. This is not "solving spam", this is only getting rid of one of the symptoms of spam, leaving the problem relatively untouched. Messagelabs will continue to report that, how much is it these days?, percent of all e-mail traffic is either spam or virus-infected.

The Microsofts (and Ciscos, etc...) of this world probably think that once e-mail spam stops reaching peoples inboxes, the incentive for spammers to spam will vanish, and with it, the problem of spam. WRONG.

Marketing and salesforces all over the world have somehow gotten it into their heads that they have some God-given right to pester and harass consumers anytime, anyplace to beat them over the head with whatever they have around that should make you empty your pockets. And e-mail has been a relatively cheap way for them to harass us. But if that won't last, they will find newer, even more intrusive ways to get into our wallets^H^H^H^H^H^H^H hearts. Texting my mobile phone, calling me with product advertisements, harassing me while I'm shopping for groceries, Inserting picture-in-picture commercials during television, etc, etc, etc... I could go on for hours about how evil everything involving marketing and sales is, but hey, we all know that don't we?

My point is: Spam is not solved by either filtering messages, or making unsollicited commercial e-mail impossible. If Microsoft really wants to enhance the quality of my life, make sure I can for instance enjoy a half hour of television without being constantly interrupted by commercials, and keep those salesdroids away from my favorite supermarket, and away from my phone. Thank you.

What happened to the "math equation" solution?

I thought Gates' solution was to have SMTP senders solve a simple math equation from each mail item they wished to post to a server, thus causing spammers a massive slowdown.

To the best of my knowledge this solution is not in practice and Microsoft is using Bayesian filtering which way predates Bill's promise.

Wrong, wrong, wrong

Hormel [spam.com] is really the ONLY company that can legitimately do something about the problem of SPAM®

Is this even something that microsoft *can* do?

Several others have mentioned that spam will be "solved" once the sending of it has been stopped. I am not sure that Microsoft could ever solve spam in this sense (or any company, for that matter). I don't deny that MS could make great inroads on the problem based purely on their numbers, but when other operating systems, other filters, other mail programs, etc. exist, Microsoft couldn't possibly be responsible for these.

This is not to say they are not responsible for their corner of the world, but the best they can do is fix their SMTP holes, include spam filtering software in all of their software/webware products, and if they are feeling useful, develop a clear and documented solution that could used on other systems/programs.

However asking MS to "solve the problem" is a bit much, even if they did overextend the claim originally.

Microsoft spams me

Not only has Microsoft not stopped other companies sending out spam, they continue to send me spam themselves. I have an open issue with TrustE relating to the Small Business newsletter that Microsoft has been sending me for many months. Every attempt to unsubscribe is met with complete failure. Even complaining to TrustE back in November, and reiterating the complaint two or three more times, has so far only resulted in form letter responses from Microsoft that are completely unhelpful.

In the past, though not for this issue, I have sent unsubscribe requests to Microsoft by registered mail and THOSE were ignored as well.

How can me possibly expect Microsoft to solve the spam problem if they themselves resort to spamming users and refusing unsubscription requests?

got worse in hotmail

If they have "solved spam" they haven't implemented in hotmail yet. I notice the amount spam increasing to be increasing and to be getting through to the "filtered" mail.

I observe this to be cyclic. Hotmail makes an improvement or some spam king gets busted, then it goes done. But it always comes back to above its previous highs once they learn invasion and new spam-asshole fills the void.

If only spammers were competing against Microsoft

...they'd be eliminated by now.

The only decline in spam I've ever had, was caused by using open source spamfilters, blacklists and other stuff no Microsoft-employee has ever touched.

If anything, the main reason spam is still here is because it's just as easy to turn a Windows box into a zombie now as it was two years ago.

Irony

It's ironic that in setting out to 'solve' spam, Microsoft all but destroyed the momentum around SPF [openspf.org], fracturing it into several different, incompatible implementations.

Re:Irony

It's ironic that in setting out to 'solve' spam, Microsoft all but destroyed the momentum around SPF

I am now seeing SPF records for fully 1/3 of incoming external email on my medium-sized company's mailserver. Of course I also greylist [ee.ethz.ch] (which virtually eliminates the crap fom zombie PCs), but of the mail that makes it though the filters, the percent using SPF is slowly but surely climbing.

Do you know of some evidence that shows that SPF adoption is slowing?

MSN/Hotmail routinely ignores abuse compaints

MSN/Hotmail is well known for ignoring abuse complaints. I get a huge quantity of spam originating from Hotmail's servers, mostly 419 scams. More than half the time I report it it gets sent back because "it doesn't reference a hotmail user." All mails travel through hotmail servers, if you report spam to the MSN address they actually frequently reject the mail because they run a content filter which detects it as spam! See this discussion [spamcop.net] for more info. I ended up finding an address that got me a live person once, and after some bitching they took care of one account. I ended up writing a letter to the FTC (these aren't just spam emails, they're scams) expressing my concern with the lax attitude towards the abuse of hotmail's own system.

Sorry Bill, if you want to be tough on spam, start with your own company. It doesn't seem to care about the rest of the internet. If Hotmail cleans up its act, I'll start believing your sincerity in the fight against spam.

There's only one way to solve spam.

Well, two. Same basic idea, but attacking it from two different sides:

1) Execute all spammers.
2) Execute all the imbeciles who buy from them.

Spam is a human problem, not a technology problem. Think of it as the black market, only even sleazier.

Impact on in-boxes is minor - Other solutions

I agree with Microsoft on this. I have been using http://pobox.com/ [pobox.com] for some time now and the results are dramatic. With their filters I can log in and view messages that were rejected and those that are held for review, and have the option of releasing false-negatives and putting them on my whitelist. I still get 5 or 6 spams a day but I can handle this easily. The rejects are in the thousands sometimes. This all happens before the email gets to my email account. Pobox.com is a forwarding service. Mail for me goes there and then is sent to wherever I wish (up to 3 redirects).

Any program that can make the impact minimal is IMHO - as the article says - the ojbective. I can deal with some junk mail, I just don't want to spend any significant time cleaning it all up. What pobox.com does not get, gmail usually picks it up and places it in my spam folder. Nice. If Microsoft can do this then I think they are on the right track.

Another problem that would be solved by uPayments.

Imagine if you got paid ten cents per email delivered to your mailbox. You come in in the morning to find a hundred spams waiting for you. Sweet! They just paid for your morning quadruple-mocha-latte and a king sized muffin. And none of your friends or customers would blink twice about paying a dime to send you a message.

Heck, I could live on my spam-account proceeds.

There's a lot of Internet problems that would be solved by this kind of automatic micropayment system. If Itunes has taught us anything, it's that if you set the price right, it will be low enough that people won't think twice about using the system legitimately, but high enough to add up to significant money in aggregate.

For example newspapers -- real newpapers (which I define by having journalistic shoe leather on the ground in your city) are dying because they don't have a practical way to pay for real journalism. Which is why they are increasingly cutting back on journalism and filling out the space with opinion -- syndicated at that. To subscribe to the paper for a year, the cost is enough that you have to think about it, predict what your probable future interest in the paper is. If your browser could be configured to send the paper a dime per page read up to a set daily limit, you'd probably spend several times the newspaper's asking subscription price per year without ever thinking of it.

What *they* have done

By doing absolutely FUCK ALL about spam in the past two years, rather than aggressively trying to protect its consumers (I use 'consumer' in the force-fed sense), Microsoft have solved their problem (if not ours) by leading their dumb users into accepting spam (if bill gates couldn't fix it, there's nothing that can be done) - their attitude has changed from the questioning human spirit of resistance "this is ludicrous why should I have to put up with this - something must be done!! " to the quite british depressive "ohhh (sigh) spam, it's like rain i just put up with it and make myself a cup of tea". you shouldn't! ATTITUDE!! it's such dumb passive majorities that allow atrocities to be committed. that leaves our majority to fight HARD to make the system better for everyone.

K9 spam blocker rules

I have to say, since someone here pointed me to K9, and since installing it and spending a couple of weeks "training it", I almost don't notice spam any more. It's awesome. I must get over 100 spam emails a day (easily), but I can't remember the last time one got through (or the last time a legitimate email got snagged).

If you haven't tried K9, and you aren't happy with your current spam solution, give it a try...

Sender Policy Framework

Sender policy framework [openspf.org] is a system to prevent fake sender address in emails. it works [openspf.org] by checking the claimed sender domain, in the email, against a TXT record in the DNS system. The TXT record contains information of ip's or hostnames, allowed to send email on behalf of the domain in question.
If the email have a faked sender address it can be bounced or labeled suspicious.

This works amazingly well, and stops all faked sender emails before it's accepted in the server. Effectivly blocking virus and spam sent with forged addresses. Non exsisting domains are allready blocked in the mail servers so if everyone owning a domain was to implement [openspf.org] this. It would make me a very happy person. Ofcouse spammers can still send email from domains under their own control, but those go into online blacklists [google.com] fairly quickly

Unfortunatly it does not have the widest accept yet, but growing all the time. After hotmail implemented it in their DNS records, spam is at an all time low around here. Not getting a single spam email from faked hotmail addresses in ages.
And only 6 months ago I had a dedicated "sent from hotmail" folder since it was 99% likly to be spam anyway...

sepski