So You've Lost a $38 Billion File

smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."

Time for...

Seppuku [wikipedia.org]?

Re:Time for...

Thanks for alleviating my ignorance [wikipedia.org].

Re:Time for...

I once deleted a file worth $38 billion. I was afraid my boss was going to fire me, but he shook my hand and laughed "Fire you?!! We just invested $38 billion in your education! We can't fire you after that kind of investment." And true enough, I never deleted a file worth that much again.

Redo the work?

For that kind of money, I'd probably just send the HD to data recovery specialists.

Re:Redo the work?

For that kind of money, I'd probably just send the HD to data recovery specialists.

Well, this is the government. They probably didn't have a budget for data recovery, but they did have a budget for scanning documents...the actual dollar amounts of each probably matter very little ;-)

$38 billion?

How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?

I'll do it!

So the information is still available in 300 boxes and it would cost about $200,000 to scan and recreate the $38 billion file again?
I'll do it for $1 billion.

maxtor?

As their IT consultant I stand by my use of Maxtor drives.

Backups are the devil

Because no one ever restores them regularly to test them.

I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.

And the backup tape was unreadable. The next good backup was 17 days older.

After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.

You should REGULARLY test your backups.
You should have LOTS of backups.

Damn!

That has got to rank up there w/ the all-time worst 'oh-shit' moments in that poor bastard's career (if it still exists). I wonder what the sysadmin was thinking, storing data on the same partition as the OS. No sane production environment rig that I know of would (or at least should) have that. It may be a Windows thing, but on most servers I've dinked with, the OS sat on a pair of RAID disks by itself, and all the data sat on the monster pile of disks on their own logical RAID drive (at least RAID 5... 5+0 w/ a hot spare preferred).

That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?

/P

Data recovery?

With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.

The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.

We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.

Re:Data recovery?

I'm just assuming the harddisks were secure erased, considering that is what pretty much every govenment in the world does when formatting harddisks.

Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.

I lost 75 trillion dollars!

I launched notepad and wrote 75 trillion dollars and saved the file. And, because I was feeling very extravagent, I deleted the file. I am rich. I can afford to lose 75 trillion dollars without batting and eyelid and am man enough to brag the info to the whole world.

Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.

Someone is trying to cover their ass

Primary disk: Accidently deleted.
Backup disk: Accidently formatted.
Tape: Unreadable.

What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?

Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.

Re:Tapes?

Yea, tape is pretty common. DVD burners simply aren't rated for backups as some burned DVDs don't have a very long shelf life. Now sounds like some screwed up in purchasing cheap tapes as well. Oh no.

BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.

Re:Tapes?

Well, the summary states that the files were rescanned at a cost of $200,000 -- so it sure sounds like the hard copies were preserved.

Re:Tapes?

Not even close. We use LTO2 tapes and keep them offsite for 18 months. We've run several test recoveries on tapes > 12 months old with success. Some of our tapes have been in circulation for about 24 - 30 months now and are still writing without difficulty. For restoration purposes, the actual media is rarely the problem. Changes in encryption passwords (with a poorly documented history), files in use, and lost/orphaned files are the most common reasons for restoration failure.

Re:Tapes?

Really? For what volume of data? For people with 100s of GB of transactional data, tape robots are pretty much the only option, or you'll be spending your whole day swapping DVDs. OTOH, it sounds like this was relatively static data (since it could be re-entered from paper), so maybe a DVD version would have been an appropriate measure as well. There's also a lesson here that you should frequently do test restores from backup tapes.

Re:Tapes?

In a backup system where you're taking full backups once a week, even with no data churn at all, you end up backing 1/7th of your estate every night. Starts getting a real challenge at about the 15Tb mark, and becomes a whole new adventure in pain when you're talking 100-200Tb.

And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.

But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.

Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.

Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and ... well, no one's going to spend that kind of sum of money (millions) trivially.

Re:Tapes?

Hm. Tapes with a proven shelf life of many, many years, or DVDs where a single scratch can render 4GB of data worthless. I wonder which enterprises (or governments) should chose?

DVDs are a joke.

An LTO Ultrium 3 tape holds 400GB uncompressed, and you can buy libraries that hold hundreds or thousands of these tapes (and dozens of drives).

Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.

The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.

Re:Tapes?

Easy to offsite???

I work for an IT organization and we pay a company called Iron Mountain $100's monthly to schlep our boxes and boxes of backup tapes to their offsite storage facility.

And remember there is a difference between making 'backups' (store my important files somewhere else so I can get them in case of a system failure) and preparing for 'disaster recovery' (store everyones files somewhere else so we can rebuild the entire infrastructure in case the building burns to the ground).

Re:I could be a douche and say it has never happen

What happens when you fuck up that big?

If you the CEO of a major corporation, you get a severance check that looks like a phone number. If you're the IT plebe, you get to look for a new job.

Re:Actual Cost??

1) Write "200 000 000 000 USD" on the paper.
2) Type what's on the paper into a .txt file
3) Save the file
4) Delete the file
5) Empty the recycle bin
6) Recreate file by retyping data from the paper
7) Post the story on the /. how big financial disaster you've made, and how you've saved your ass

Re:And this is why...

And this is why... ...print will never be dead.

Right, because:

Nobody has ever thrown away papers that were actually needed...

Paper is an inexpensive and compact way to store terabytes of information...

Paper is trivially easy to instantly duplicate on a large scale...

Paper is trivially easy to haul off-site and store...

People constantly generate diffs between the most recently archived paper copy, and all work they have done every day since. They don't just make undocumented changes, willy-nilly, requiring just as much effort to backup daily changes as it is to backup full copies of everything...

No question, paper is superior. The data retention problems we always hear about are in every way caused by digital storage methods, and have nothing to do with the policies and people running the organizations...

(No I will not pay for any damaged caused by this post overloading your sarcasm meter.)