Looping E-mails Beat The Net Down

Staili writes "Singapore-based women's magazine caused problems when it forwarded its mails to a large list of recipients, mainly mailing lists. In addition to [e-mail], some help and subscribe lists were included; the type of addresses that tend to send out an automatic reply confirming receipt. And the loop was ready." I'm sure anyone who's messed with mail enough has accidentally created a loop or two in their day, but this is really slimey.

Wouldn't it be funny...

[taliver]

If a "solved" problem like email actually brought the net down... for a while. How do you get patches for a sendmail program without using the internet?

Why was the header stripped...

[zubernerd]

My question is: Is it normal for a server to strip the headers from e-mails...
FROM THE ARTICLE: ["At savoixmagazine.com the mail headers were cut so it was almost impossible to find out where the mail originated from," said Drahtmuller. The everyday analogy is a letter stripped of its envelope that had the original return address printed on it, repackaged in a new envelope with a different return address, and forwarded on. "Usually mail loops like this are not possible with Unix systems because they always maintain the headers," he added.]
I'm not a e-mail expert, but why where those headers missing? (I did not see any reason given in the article.)

Haven't we all done this?

[Charles Dodgeson]

I don't think that there is an email admin around who hasn't managed to be part of such a loop. It is remarkably hard to put together systems which will interact correctly with all of the other ways that other systems might be broken.

And for anyone who thinks that email is a "solved" problem, should read my rant about broken autoresponders [goldmark.org]. (which is not about loops, but does cover how "solved" things can be broken).

happened at my school once...

[jeffy124]

back when i was a freshman in college someone managed to assemble an email list of all the students/faculty/staff. It was first used by someone outside the school to spam the entire campus, with all the addresses in the To and Cc fields, making the list available to anyone who received it. So someone attempted to sell their Chem Eng books, and you can picture the hell that broke out.

Quickly the list became nothing but people hitting reply-all and saying "knock it off!" and "get me off the list!" Of course, all those emails and addresses in the emails meant trouble for the mail server, causing mail to get delivered multiple times and DOS'ing normal mail.

It got so bad that I had about 100 emails in a five minute span at one point. It took a Dean's sending out an email to an announcements list pointing out school policy on mass mailings to stop it.

Thankfully, everyone from those trying to sell stuff to those saying "quit it!" all had to write a 500-word essay about why what they did was wrong.

E-Mail Database

[yintercept]

I am actually surprised by the number of times people send out email not knowing who will receive it or the number of people in their CC list. Most email clients don't let the end user see how much damage they have done. The goal of a developer is to give the users the power to get their job done, but so often you find people are clueless on what the power is or how to use it.

Personally, I would like to see email merge with databases. With a good relational DB, it is easy to show users what's gone through the pipe and how many emails your company has sent to a client, etc.. You can integrate the email into your CRM, etc. You can also place constraints on the system that can prevent this type of mailing list abuse that generates so much unwanted garbage.

Working with pure email clients (sendmail, exchange, whatever) seems to be like trying to fit a round cat through a square hole. [rgreetings.com]

oh yeah, i created a loop

[legLess]

I once inherited a smallish network (70 nodes) that was using an NT box as a web gateway and mail server. It was running something called Xtramail, which is a truly bloody horrible piece of software. While I was trying to figure out how to gracefully get rid of this box (a 486 on ISDN), one of the users wanted to create a mailing list.

Ok, no problem. Read the docs, slurp this list, check these buttons, viola. One of the cute little checkboxes was "Only allow owner to send list mail." Duh - I checked it. The guy sent his email (only about 200 list members) and we went home.

I came in the next morning to 20,000 emails just in the queue. That fucker sent our tens of thousands of emails overnight, because the send restrict wasn't working. There were a couple dead addresses on the list, and they of course bounced - and Xtramail politely returned those bounces to the entire list. Wash, rinse, repeat. If that place had had a real server and a real 'net connection, it could have sent millions of emails in that time. As it was, many people on the list were (quite justifiably) pissed.

So I called up whoever owned Xtramail at that time (Artisoft at that time, but a different company now - can you say, "hot potato?") and had a slightly polite shit fit. The guy flat-out refused to acknowledge it was a problem, until I made him go through the same steps on his local copy.

Crickets.

"Uh, looks like that option isn't working. I'll have to file a bug report." Then I spent another 45 minutes trying to get accounting to refund the $200 I'd given them for the support call.

They never did fix the bug, but I gave up my plans to have a graceful transition. I pulled that POS out the same day and installed another little NT mailer, quite a nice one, until I replaced the whole thing with a qmail FreeBSD box.

No moral to the story, really ('cept I should have been more paranoid, and tested the list more). But I bet more than a few readers have had that quick "oh shit" feeling as they saw the queue filling up.

Mail loops, and circle jerks

[Peter H.S.]

Mail loops, and mail circle jerks can be a nasty experience;
This is my tiny war story.

I do some volunteer work as a sysadmin in our local Internet club (300 apartments sharing a 2Mbit, soon-to-be 4Mbit line). When we started however, we only had a 512Kbit line, so in our wisdom we configured our MTA (Qmail) to bounce mails above 20Mbyte in size. We also thought it would be a good idea, to use our inet-feed provider as a backup mail relay, so in case our servers were down, mail would queue up there, ready to be delivered, when we went online again.

But one of our users had set up his Outlook mail account on his work, to forward all mail to his mail account at our network. So far so good, but then, just before leaving work one day, he mailed his home account at our network a 300Mbyte attachment (splitted up in 10 30Mbytes parts).

This is what happenend then;
Qmail recieved each attachment, but didn't bounce them, until the entire mail was recieved.
To my knowlegde, Qmail then appended the right RFC error message header to the mail, and bounced it, headers, attachement and all.
But the mailserver on the other end (MS Exchange) didn't respect that, but instead it forwarded the bounced mail to our server again, while rewriting the headers and subject.
The two mailservers now bombarded each other with 30Mbyte mails, and since we had the slimmer pipe, we were losing the battle. (I believe that this scenario is called a "mail circle jerk").

It took some time, to straighten things out. Oh, and we discovered that the back up mail relay really did work, since it kicked into action, when we brought our mailserver up, and promptly tried to deliever a +Gigabyte of bounced and resend mail.

Lessons to be learned; mail qoutas can have nasty sideeffects, backup mail releays can be a double-edged sword when things turns nasty. Automatic forwarding is can be very nasty indeed. And finally; How does your MTA or MUA forwarding rules react to a RFC error messages?

This was predicted a loooonnnggg time ago.

[Ace905]

Wow, it's always rewarding to find one of your theories proven.

I actually wrote an article for the hacker community on this exact problem about a year ago. I hosted the article at myhometechie.com - which is my own web site. I also submitted the article to hackcanada, and 2600.com - which is the authority for hacking issues.

Well, despite my long trek and obvious dedication to showing up at 2600's conference, H2K, in July of 2000 - they didn't feel like printing my article ; and I definately did not want to test my theory.

Oh well. Their loss. This could have been averted if the problem of looping auto-repliers had become common knowledge.

You can find my article here [myhometechie.com].

down boy, down!

[Erris]

Why not do it on Tursday morning, or some other time that allows the nitwit that made the change to fix it immediately when the customer calls in a trouble ticket? (Because all the skilled telco employees were "downsized", and only the retards are left?)

Downsizing can make anyone look retarded. When there are not enough people to do the work, the work does not get done.

Downsizing is only half the problem anyway. There are whole industries where the average age of engineers and craftsmen is around 50. Those companies have not hired waves of new people for 20 years or so, and fired many of those that were lucky enough to get on. Think that 60 year old overworked survivor really cares about training sucessors? Nope, they are looking for a package and will give the job to you the way they got it, learn as you burn. Many great mistakes will be repeated. I believe that this really boils down to a single factor. Does the person in question really give a shit about the consequences of his or her actions?

You are entitled to your opinion. Most normal people quit jobs where things are starting to fail. The lucky ones find good alternatives. The loyal ones get stuck with a job that much more difficult. How many years of your life are you willing to give up to hopeless causes? Everyone knows the general rules. Some are lucky enough to put the big changes off as good practice, sometimes the law, demands.

I feel awful for people who do real work at the telcos. Change sucks, and they are getting plenty of it. Imagine starting your career there before deregulation. Off you whent to serve the regulated monopoly and the public. You accepted low salaries in exchange for stability and pride of serving one of the best and cheapest telco services in the world. You also put up with the more inane political nonsense and tried to just do your job.