Slashdot Log In
So You've Lost a $38 Billion File
Posted by
Zonk
on Tue Mar 20, 2007 09:58 AM
from the makes-you-feel-better-about-yourself dept.
from the makes-you-feel-better-about-yourself dept.
smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Time for... (Score:5, Funny)
Re:Time for... (Score:5, Funny)
Parent
Re:Time for... (Score:5, Funny)
Parent
Re:Time for... (Score:5, Insightful)
The fact that it was related to an account worth $38B is scary, but not the actual cost.
Parent
Re:Time for... (Score:5, Interesting)
Parent
$38 Billion is a big incentive for fraud (Score:5, Interesting)
Parent
Re:$38 Billion is a big incentive for fraud (Score:5, Funny)
I trust by the grace of almighty god you are in good health this fine and beautiful day. I was a data entry clerk for the Alaska Department of Revenue, Prior to being fired, I secured access to a hidden fund worth $38,000,000,000 (THIRTY-EIGHT BILLION DOLLARS).
If I ever tried to utilise this fund in my name, the funds would risk being confiscated by the government, so I would like you assistance to find a trustworth foreign assistant who can invest these funds.
This proposal is 100% risk free, and I can offer you a 10% fee for your help.....
Parent
Re:$38 Billion is a big incentive for fraud (Score:4, Interesting)
Anyway, what DIDN'T shock me about this story is that after formatting the main disk, the tech immediately (and blissfully) formatted the backup as well. I've seen stuff like that happen like ten times. ("Oh, well, after I replaced the drive, I figured I should replace the backup tapes too, so we could have a fresh start, so I threw them out." or "I figured I should make a backup right away, so I over-wrote the good backup with the new, bad, data.") I don't want to blame the victim, but sometimes it's like the data wants to be destroyed at that point. My favorite was when someone added a second drive to an important source control server to do nightly drive to drive back-ups. Then, they stopped doing tape backups nightly and switched to weekly. Then, they forgot they disconnected a fan during the HDD installation (or it was accidently disconnected -- it remains a debated point), then the server fried itself and the drives. Then everyone lost a day of work rebuilding the source archive based on their local data. Good times.
Parent
Re:Data Recovery options? (Score:5, Interesting)
Google was my friend. Shortly I learned more about backup superblocks, how to run "mkfs.ext3 -n" to do a dummy mkfs and find out where my backup superblocks are, and "fsck.ext3 -b nnn" to repair the filesystem using the backup superblock.
I was back running in less than an hour, including google time. Repairing an accidental mkswap on top of ext3 is actually one of the easier things to fix.
On the other hand, having a system and procedures that made it possible to kill regular and backup data that way, and storing unconfirmed tapes, is clearly not a good idea. Whenever I burn a CD/DVD, I take the few extra minutes and verify it right away. If the backup tape was only a few months old, odds are it was improperly written, as opposed to degraded. They should check their other backup tapes.
Parent
Redo the work? (Score:5, Insightful)
Re:Redo the work? (Score:5, Insightful)
Well, this is the government. They probably didn't have a budget for data recovery, but they did have a budget for scanning documents...the actual dollar amounts of each probably matter very little
Parent
Re:Redo the work? (Score:4, Funny)
Parent
Re:Redo the work? (Score:4, Interesting)
A law firm handed me a computer that wouldn't boot. On it was some pictures taken concerning a wrongful death case. It turns out that the pictures were a hindsight and in the middle of fixing it, The task turned from getting the computer to run to getting the pictures from the drive. The drive was failing and was larger then the 137gig 28bit LBA limits. But we didn't know this because it was never booted and XP pre SP1 did not enable 48bit addressing by default. And even after SP1, if you didn't update your ATAPI driver to x.1135 or later, it wouldn't be enabled by default even if you have the ability. So connecting it to another computer made it worse. Eventually the fault in the drive which was a crashed head, made it impossible for us to recover past the boot sector running traditional recovery software. The data recovery specialist were able to get around everything we added to the problem as well as the problem itself and retrieved better then 98% of everything on the drive. I think one file was bad but we weren't concerned with it at all.
Long story short, the pictures showed someone's negligence in a wrongful death case and once they were presented or added to the evidence pile, the defendant's insurance company settled for 2 mill. The lawsuit was for more then that so you could probably guess what it could have been worth. The firms cut was in the area of 40% from what I understand. So it was worth 40% of 2 mill to them. $2500 seems like a little amount in comparison.
Parent
The Senator (Score:3, Funny)
Senator Ted Stevens remarked that they should have sent it in an Internet, apparently tubes are much more reliable than tape.
Re: (Score:3, Funny)
And this is why... (Score:4, Interesting)
Re:And this is why... (Score:4, Funny)
Parent
Re:And this is why... (Score:5, Funny)
Right, because:
Nobody has ever thrown away papers that were actually needed...
Paper is an inexpensive and compact way to store terabytes of information...
Paper is trivially easy to instantly duplicate on a large scale...
Paper is trivially easy to haul off-site and store...
People constantly generate diffs between the most recently archived paper copy, and all work they have done every day since. They don't just make undocumented changes, willy-nilly, requiring just as much effort to backup daily changes as it is to backup full copies of everything...
No question, paper is superior. The data retention problems we always hear about are in every way caused by digital storage methods, and have nothing to do with the policies and people running the organizations...
(No I will not pay for any damaged caused by this post overloading your sarcasm meter.)
Parent
$38 billion? (Score:5, Insightful)
Re:$38 billion? (Score:4, Informative)
Parent
Re:$38 billion? (Score:4, Funny)
Parent
Re:$38 billion? (Score:4, Insightful)
Parent
I'll do it! (Score:5, Funny)
I'll do it for $1 billion.
maxtor? (Score:5, Funny)
Backups are the devil (Score:5, Insightful)
I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.
And the backup tape was unreadable. The next good backup was 17 days older.
After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.
You should REGULARLY test your backups.
You should have LOTS of backups.
inject + extract (Score:5, Informative)
After you run the backup, memove then restore that file, make sure it has the current date in it.
I've had that as a feature in my backup scripts for over 10 years...
Parent
I'd hate to be the tech support guy... (Score:4, Funny)
Damn! (Score:5, Insightful)
That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?
Alaskan Pipeline (Score:3, Insightful)
Data recovery? (Score:5, Interesting)
The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.
We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.
Re:Data recovery? (Score:5, Insightful)
Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.
Parent
Not to bad (Score:4, Informative)
I know that many companies would not be able to recover information lost in that manner.
I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.
I lost 75 trillion dollars! (Score:5, Funny)
Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.
Someone is trying to cover their ass (Score:5, Insightful)
Backup disk: Accidently formatted.
Tape: Unreadable.
What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?
Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.
Re:Someone is trying to cover their ass (Score:5, Informative)
Having worked on backups in an Alaskan company run very similarly to the department in question, I think it sounds reasonable. I was a consultant at the time, and I pointed out that the backups have never been tested. It was on the weekly report. It was on the weekly report for about a year. Many people making much more than you make saw that the backups have never been tested. Then there was a crash. It turned out that the backups, set up long before I got there, were set in a tape library. There were 5 tapes and a cleaning tape. The backups would backup server 1 onto tape 1. Then, server 2 onto tape on - set to overwrite. The least important server in the room was last on the backup list, and it was the last to issue the command every night to backup onto tape 1 - set to overwrite. So the email was gone forever. Somehow, the consulting company I work for that pointed out for over a year that backups weren't tested and may not work was to blame for not fixing what was broken long before we were brought in. So I find the description of events quite plausible.
Parent
Vista (Score:4, Funny)
Perspective (Score:5, Funny)
Re:Tapes? (Score:5, Informative)
BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.
Parent
Re:Tapes? (Score:5, Insightful)
Parent
Re:Tapes? (Score:5, Interesting)
Parent
Re:Tapes? (Score:4, Insightful)
Parent
Re:Tapes? (Score:5, Insightful)
I work for an IT organization and we pay a company called Iron Mountain $100's monthly to schlep our boxes and boxes of backup tapes to their offsite storage facility.
And remember there is a difference between making 'backups' (store my important files somewhere else so I can get them in case of a system failure) and preparing for 'disaster recovery' (store everyones files somewhere else so we can rebuild the entire infrastructure in case the building burns to the ground).
Parent
Re:Tapes? (Score:5, Informative)
Parent
Re:Tapes? (Score:5, Informative)
And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.
But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.
Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.
Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and ... well, no one's going to spend that kind of sum of money (millions) trivially.
Parent
Re: (Score:3, Insightful)
Re:Tapes? (Score:5, Insightful)
Parent
DVDs are a joke. (Score:5, Insightful)
Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.
The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.
Parent
Re:I could be a douche and say it has never happen (Score:5, Insightful)
Parent
Re:Actual Cost?? (Score:4, Informative)
The site is slashdotted, so I can't read TFA, but my guess would be the information isn't actually "worth" $38B. It just represents an accounted amount of $38B.
The actual value of the data is what it would cost to replace it (or perhaps do without it) -- in this case, $200,000. Consider an analogy (20th-century, but illustrative): if you were to send a paper bank-check for $10,000 via a courier, the declared value for insurance would not be $10,000. It would be the cost of recovering from the loss of the check, which would be the stop-payment fee plus the cost of sending a new one.
Parent
Re:Actual Cost?? (Score:5, Funny)
2) Type what's on the paper into a
3) Save the file
4) Delete the file
5) Empty the recycle bin
6) Recreate file by retyping data from the paper
7) Post the story on the
Parent