So You've Lost a $38 Billion File

smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."

Sounds more like a $200,000 file

[Anonymous Coward]

But maybe that's just me, someone who opted not to work in government after studying political science.

Redo the work?

[Anonymous Coward]

For that kind of money, I'd probably just send the HD to data recovery specialists.

$38 billion?

[pipatron]

How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?

Re:Tapes?

[teflaime]

Most ENTERPRISES still have tape at some level as part of a comprehensive disaster recovery plan. Tape is easy to offsite, fairly reliable overall and still have comprehensive support available in all platforms. Most INDIVIDUALS don't do backups at all.

A thought that crosses my mind...

[physicsboy500]

If you're going to back up a file (actually a set of files) that is worth that much, wouldn't it be smart to go a bit further than keeping a backup copy on magnetic media?! Maybe in more than one place too?!

Re:Redo the work?

[omeomi]

For that kind of money, I'd probably just send the HD to data recovery specialists.

Well, this is the government. They probably didn't have a budget for data recovery, but they did have a budget for scanning documents...the actual dollar amounts of each probably matter very little ;-)

Re:$38 billion?

[__aapbzv4610]

The value is based on what the data represents, not the material labor in re-scanning them. loose example, I could spen $30 on painting supplies and create a $100,000 masterpiece (well not by me, but, anyway...)

Backups are the devil

[Maxo-Texas]

Because no one ever restores them regularly to test them.

I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.

And the backup tape was unreadable. The next good backup was 17 days older.

After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.

You should REGULARLY test your backups.
You should have LOTS of backups.

Re:Tapes?

[Detritus]

Tape works great if you are willing to spend big bucks for top-class hardware. Unfortunately, most people try to get by with the cheap stuff, which is very unreliable. Try to explain to a manager why you need a $50K tape system to backup a $10K server. Computers have gotten very cheap, high-quality tape transports haven't.

Re:Tapes?

[daeg]

Hm. Tapes with a proven shelf life of many, many years, or DVDs where a single scratch can render 4GB of data worthless. I wonder which enterprises (or governments) should chose?

Damn!

[Penguinisto]

That has got to rank up there w/ the all-time worst 'oh-shit' moments in that poor bastard's career (if it still exists). I wonder what the sysadmin was thinking, storing data on the same partition as the OS. No sane production environment rig that I know of would (or at least should) have that. It may be a Windows thing, but on most servers I've dinked with, the OS sat on a pair of RAID disks by itself, and all the data sat on the monster pile of disks on their own logical RAID drive (at least RAID 5... 5+0 w/ a hot spare preferred).

That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?

/P

This should be...

[Anonymous Coward]

This should actually be a best-practice kind of situation really. Accidents happen. Both hard disks got formatted. The tapes were shit. BUT, we had a paper trail. It was painstaking, but it was recoverable. EXCELLENT! This is exactly why banks have paper records of everything, and why they pay a LOT of money to have them properly stored.

That said... the excerpt is a bit misleading. The data was worth $38 billion. They didn't lose $38 billion. They managed to get it back in shape for $200,000, which is not pennies, but probably well worth the effort.

Re:Tapes?

[Red Flayer]

Well, the summary states that the files were rescanned at a cost of $200,000 -- so it sure sounds like the hard copies were preserved.

Re:Tapes?

[Itninja]

Easy to offsite???

I work for an IT organization and we pay a company called Iron Mountain $100's monthly to schlep our boxes and boxes of backup tapes to their offsite storage facility.

And remember there is a difference between making 'backups' (store my important files somewhere else so I can get them in case of a system failure) and preparing for 'disaster recovery' (store everyones files somewhere else so we can rebuild the entire infrastructure in case the building burns to the ground).

Alaskan Pipeline

[Doc Ruby]

I bet it cost a lot less than $200K to bribe the government officials (probably with a few bottles of wine) not to check whether they were protecting their $38B investment with more than $45K worth of IT staff.

Re:I could be a douche and say it has never happen

[Overzeetop]

What happens when you fuck up that big?

If you the CEO of a major corporation, you get a severance check that looks like a phone number. If you're the IT plebe, you get to look for a new job.

Re:Data recovery?

[mwvdlee]

I'm just assuming the harddisks were secure erased, considering that is what pretty much every govenment in the world does when formatting harddisks.

Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.

Someone is trying to cover their ass

[Unique2]

Primary disk: Accidently deleted.
Backup disk: Accidently formatted.
Tape: Unreadable.

What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?

Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.

DVDs are a joke.

[mosch]

An LTO Ultrium 3 tape holds 400GB uncompressed, and you can buy libraries that hold hundreds or thousands of these tapes (and dozens of drives).

Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.

The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.

Re:Tapes?

[Anonymous Coward]

in such case, you simply use a decent backup tool with barcoded tapes.

Re:Time for...

[jadavis]

It was actually only worth $200k, since that's the amount of money it took to recover from the problem.

The fact that it was related to an account worth $38B is scary, but not the actual cost.

Re:Tapes?

[segfaultcoredump]

HA is not the same as DR

I can have a simple HA cluster that involves two nodes attached to a single disk array, all sitting in the same rack somewhere. Take a guess what happens when the power for the building goes out?

HA is nice, but will do nothing for you in the event of a disaster.

You can structure your site so that you get both, but doing so requires a lot more work (stretched clusters and SAN's spread over miles) and you have to be careful that you dont trash your performance while you are at it. (real time replication over distance involves latency, and you have to be careful about what that will do to your app)

Re:Tapes?

[Miseph]

;rolleyes:

There really isn't anything "Insightful" about pointing out a grammar error. Making personal insults isn't either.

C'mon mods, this is just embarrassing.

Re:Time for...

[FunkyELF]

I don't. Please inform us.

Every time this happens to me I search around and find a bunch of tools that will only get you 5 files or will only get you files under 20kb or some other stupid restrictions.

Re:Tapes?

[evilviper]

you _can_ render the data on a DVD unreadable/unaccessible because of a scratch.

No, you _CANT_.

For that, this "scratch" has to be a massive gash, which goes more than halfway through the depth of the plastic disc. Additionally, it has to span the entire radius of the disk just to make it "difficult" to recover a significant portion of the data.

Their decay increases greatly at increased temperatures and, expecially, under UV from the sun.

Your backup solution should NOT involve throwing a bunch of bare DVD-Rs on the dash board of your car.

Any backup solution involves climate control, and light-proof packaging. Your tapes would crumble in no time, otherwise.

Re:Tapes?

[TheLink]

"The tapes are about $100, which is close to par with hard drives now (400 GB native), but in the not too distant future, LTO4 will be out, which doubles the capacity."

Yeah, but when LTO4 is out you can't use them without buying a new _expensive_ LTO4 drive.

Whereas in the not too distant future when new hard drives with double the capacity are out, you can still use them with your existing computers (as long as they still support SATA).

Basically HDDs = media + drive, and they are about the same price as tapes on a per GB basis if not cheaper. Multiple HDDs have better bandwidth than multiple tapes with one tape drive.

And I've heard horror stories where backup tapes can only be read by the same drive the backups were made on.

When you factor all that in, tape isn't that great, it's still better in some areas, but it should be cheaper for all its disadvantages.

Re:$38 Billion is a big incentive for fraud

[wyverspur]

Here's a thought....Why the hell is critical information stored on a smegging desktop/laptop? IMAO they got what they deserved.

NO critical information like that should ever be anywhere except on a properly backed up network drive, preferably with a document versioning system to track changes.

As I work for a government institution, I keep no less than three copies of data, one of which is offsite

(Just my 2cents)

Re:Time for...

[Azrael43]

I've found PhotoRec [1] to be excellent for recovering, for example, photos deleted from a memory card (friend's mistake, not mine). It will also recover other most other file formats and appears to handle various filesystems. There is another program called TestDisk available at the same site which is meant to fix corrupted filesystems. -Azrael- [1] http://www.cgsecurity.org/wiki/PhotoRec [cgsecurity.org]

Re:Tapes?

[Achromatic1978]

No, I will not call back a tape from two weeks ago because you made a mistake and saved over your Excel spreadsheet the wrong way, or because you managed to delete something because you're sloppy with a mouse.

Yes you will, or should, when the user's manager gets involved. If you would like to think that because a user trashes a month's accounts, that you can wave some magic hand and say "Yes, I know that data is in pristine condition on last week's backup, but no, you're not getting it just because Waldo over there is as dumb as dogshit to have trashed it in the first place", you either work in some kind of hell hole out of Dilbert (think "I am Mordac, the Preventer of Information Services"), or have been interpreting your job description far too literally. (We wonder why so many people have so much distate for certain IT people.) However,

Should the user's department be charged the cost that your DR service bills for the retrieval of said tape? Absolutely.

Should this retrieval work be prioritized accordingly within your task list? Absolutely.