ISP Recovers in 72 Hours After Leveling by Tornado

aldheorte writes "Amazing story of how an ISP in Jackson, TN, whose main facility was completely leveled by a tornado, recovered in 72 hours. The story is a great recounting of how they executed their disaster recovery plan, what they found they had left out of that plan, data recovery from destroyed hard drives, and perhaps the best argument ever for offsite backups. (Not affiliated with the ISP in question)"

Users need their porn!

[Trigun]

Now that that's out of the way, it never ceases to amaze me how many companies have little to no severe disaster recovery plans, and how a little bit of ingenuity(sp?) can go a long way in a company.
Times of crisis and how one deals with them are the mark of successful businesses/employees/people. I don't think that we could recover so quickly should a disaster of that size hit my job, but it'd be fun to try.

Nice work!

[Tebriel]

This is what happens when people make intelligent plans and the modify them as they see other plans work or fail. I'm glad to see that this was a work in progress rather than some arcane plan in a binder somewhere that no one ever looked at.

Fire...

[Shut the fuck up!]

...is a good enough argument for off site backups. If you don't have them, your backup plan is not enough.

Re:Nice work!

[blackp]

One of the problems with a plan in a binder somewhere, is that the tornado would have probably taken out the binder as well.

Re:Fire...

[Stargoat]

Everyone should have off-site backups. It's not very expensive (>100 dollars for tapes). It's not very hard (drive tapes to site). It's not difficult to get the backups if you need them (drive to site with tapes). It just makes sense.

so...

[2MuchC0ffeeMan]

let me get this straight, all the houses around the isp have no power, no phone... but they still need to get online?

Cool, but could be better

[MicroBerto]

While that's awesome, I still think that small businesses and big ones should both have offsite tape backups. Even if this means the owner brings back and forth a case of tapes to his home once a week or so. That alone would have saved much of this trouble.

Then I've seen the other end of the spectrum - a 6 Billion dollar corporation's world HQ IT center... wow. They have disaster recovery sessions and planning like I never would have imagined. Very cool facility, but it has to be like that. Some day if they get burned, it's all over.

Truly stunning

[dbarclay10]

What amazes me isn't that these people were able to restore service to their customers in 72 hours. They used standard systems administration techniques. BGP was specifically mentioned.

No, what amazes me is that this is news. The IT industry is so full of idiots and morons and MCSEs that taking basic precautions earns you a six-figure salary and news coverage. These folks didn't even have off-site backups, it was luck that they were able to resume business operations (ie: billing) so soon.

Moral of the story? When automobile manufacturers start getting press coverage for doing a great job because unlike their competition, they install brakes in their vehicles, you know that the top-tier IT managers and executives have switched industries.

72 hours thats pretty bad

[silas_moeckel]

OK I just may be jaded I work in a secor that thinks 5 minutes is earth shattering ammounts of downtime. 72 hours would ahve me everybody that works for me and some C level guys fired at the companies I work for. First things first what did they do wrong backups stored on site this is page 2 of a disaster recovery howto backup need to be stored onsite and remote, they also need to be verified as functional (yes I am that manager that insists that servers be restored and checked for functionality on the backup hardware during a work window) From the story it wasent even client data as much as it was there billing DB and other office information. When will people learn that information makes a lot of businesses and needs to be protected a nominal cost to do proper backups and house them remotly even if it's in a bank vault a few towns over perferably the other coast. Satalite uplinks can provide decent ammounts of bandwith in a pinch though the latency is horid.

Re:Fire...

[Zathrus]

Everyone should have off-site backups. It's not very expensive (>100 dollars for tapes)

Er, for how much data? For your personal computer, maybe (but the tape drive will cost you considerably more than that $100), but I don't think you're going to back up a few hundred gigs of business data on ~$100 of tapes. And I suspect you meant 100... although if the latter then you're almost certainly correct!

It's not very hard (drive tapes to site). It's not difficult to get the backups if you need them (drive to site with tapes)

If your offsite backup is within convienent driving distance then odds are it's not far enough offsite. A flood, tornado, hurricane, earthquake, or other large scale natural disaster could conceivably destroy both your onsite and offsite backups if they're within a few miles. The flipside is that the further the distance the more the inconvienence on an ongoing basis and the more likely you are to stop doing backups.

There's far more to be considered here, but I'm not the DR expert (my wife is... seriously). It does make sense to have offsite backups, but you have to have some sense about those too.

Re:72 hours thats pretty bad

[Xerithane]

I think I speak for everybody when I say, "Uh, what?"

Re:Amazing is an innapropriate adjective

[venom600]

Wow! This is exactly the reason that systems administrators generally dislike most members of their development group. Your attitude does not do very much to endeer us 'cable monkeys' and 'PHB's to you.

"IT people", who give a shit about logs and backups and think plugging a PC and monitor into a powerbar is "computer science"

If you think this is all that is involved in running a remotely large and reliable network, you are sadly mistaken my friend. A lot of thought, planning and testing goes into most corporate network infrastructures.....kinda like software development.

"Computer Science" is a very broad term that encompasses much more than just 'programming'.

make sure off-site is far enough away

[DiveX]

Many companies in the World Trade Center thought that off-site backup meant the other building.

Re:Compare and contrast...

[FattMattP]

A couple of friends of mine were badly burned because the web hosting company they were using lost all their data

It sounds like your friends got badly burned because they didn't back up their data, not because of their ISP. Always back up your data. That goes doubly so if your data is stored on someone else's computer.

Re:Amazing is an innapropriate adjective

[HardCase]

Actually, the hour delay was because of lazy people who kick their network cables out of the wall, then insist that a technician hold their hand to plug it in. It doesn't take an hour to find the problem...in fact, if you listened to the nice help desk man, he would have asked you to look for the end of the plug lying on the carpet. Instead, you wasted 15 minutes of his time explaining to him that you're a programmer who just doesn't care.

What takes an hour is that the technician has to take care of the other 20 people who can't be bothered to plug a cable back into the wall on their own.

Oh, and, of course, the tech also has to take care of real work - like fixing the programmer's machine after he installs the latest Webshots and Gator software.

Me: "It took our technican an hour to get all of the malware off of Stratjakt's computer that he downloaded from the Internet."

CTO: "Didn't he read the email that I sent out every month for the last six months telling the employees not to install non-work-related software?"

Me: "Well, I asked him about that...he said that he was a programmer and just doesn't care."

CTO: "He's fired."

Oh, and, incidentally, when your self-administering software becomes proficient enough to keep your big foot from wrapping around the network cable and yanking it out of the wall, then I'd say you really had something worthwhile. At this point, though, I have my doubts.

Re: so...

[snake_dad]

Yes, ofcourse you are right. We all know that ISPs only have customers immediately next to the company building. Damn those CAT5 cable length limitations...

Re:Amazing is an innapropriate adjective

[sloth jr]

IT is about handling the shit storm that happens when the software that YOU write fucks up in the colossal way that it does.

Keep up the good work.

sloth jr

Re:Screw remote backup....

[Anonymous Coward]

As an Architect, even building a below-ground bunker might not protect you from the full force of Nature with a capitol 'N'.

I'm in California, and as such, we design buildings to take a certain scale of earthquake or less; not because clients are cheap, but because above a certain point all bets are off, no matter what kind of building you've built! At some point the force of Nature you're dealing with is so staggering that no amount of preparation or work can give you a guaranteed resistance.

I doubt many buildings could take a direct hit from a tornado; and even if they could that's not saying that everything that's not the building (i.e. all that fancy computer equipment and nice people inside) wouldn't be sucked out and sent to OZ in a minute...

What about practicing your disaster recovery?

[sllim]

The company I work for practices disaster recovery once a year on all our major systems.

In the article the writer was talking about how much work it was to migrate the T1 connections, and how they hadn't forseen that. That is exactly the sort of thing that a practice disaster recovery uncovers.

If you want the model from the place I work it is simple enough:

1. Run the disaster recovery during a 24 hour period
2. Pat yourself on the back for what worked.
3. Ignore what doesn't work.
4. Repeat next year.

Of course next year gets a new step:
3.5 Act surprised that stuff didn't work.

Re:Poor tech support

[koa]

Actually.. I ran a technical support department for a small ISP for a couple years.

It amazing how accurate you are in reguards to customer viewpoint on downtime.

After having done it myself, I actually have MUCH more respect for technicul support engineers/supervisors becuase within reason most "downtime" is fixed even before the customer knows about it (i.e. small blips in service).

And the majority of people who purchase an ISP's services have absolutely no idea what it takes to respond to an outtage.

Not good enough

[vasqzr]

When you go to a DRP seminar, they make the claim that the majority of business that are knocked out for longer than 48 hours go out of business within 1 year.

Re:Amazing is an innapropriate adjective

[Artifex]

When I told him the his HD was dead, he looked at me with shock, as he explained that the last months worth of his so valuable work was on his disk. I asked him if he backed it up anywhere. He said no. He then asked me if we backed it up. I said no, we don't do that for local drives.

This is really sad, and the company could have fired him for being incompetent. He basically destroyed their intellectual property through negligence, wasting all the money they invested in his project, which was almost certainly more than just his salary for that time period.

If a truck driver gets a load and forgets to check his own tie-downs, and as a result loses the load before reaching his destination, whose fault is it?

Besides, as supreme programmer, he should be motivated to work sometimes from home in the middle of the night, and have backups there :)