So You've Lost a $38 Billion File 511
smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
Time for... (Score:5, Funny)
Re: (Score:2)
Re: (Score:3)
Geez guys! They can find the Pr0n you "deleted". I guess there needs to be more significant motivation than $38 Bil USD.
Re:Time for... (Score:5, Insightful)
The fact that it was related to an account worth $38B is scary, but not the actual cost.
Re:Time for... (Score:5, Interesting)
$38 Billion is a big incentive for fraud (Score:5, Interesting)
Re:$38 Billion is a big incentive for fraud (Score:5, Funny)
I trust by the grace of almighty god you are in good health this fine and beautiful day. I was a data entry clerk for the Alaska Department of Revenue, Prior to being fired, I secured access to a hidden fund worth $38,000,000,000 (THIRTY-EIGHT BILLION DOLLARS).
If I ever tried to utilise this fund in my name, the funds would risk being confiscated by the government, so I would like you assistance to find a trustworth foreign assistant who can invest these funds.
This proposal is 100% risk free, and I can offer you a 10% fee for your help.....
Re:$38 Billion is a big incentive for fraud (Score:4, Interesting)
Anyway, what DIDN'T shock me about this story is that after formatting the main disk, the tech immediately (and blissfully) formatted the backup as well. I've seen stuff like that happen like ten times. ("Oh, well, after I replaced the drive, I figured I should replace the backup tapes too, so we could have a fresh start, so I threw them out." or "I figured I should make a backup right away, so I over-wrote the good backup with the new, bad, data.") I don't want to blame the victim, but sometimes it's like the data wants to be destroyed at that point. My favorite was when someone added a second drive to an important source control server to do nightly drive to drive back-ups. Then, they stopped doing tape backups nightly and switched to weekly. Then, they forgot they disconnected a fan during the HDD installation (or it was accidently disconnected -- it remains a debated point), then the server fried itself and the drives. Then everyone lost a day of work rebuilding the source archive based on their local data. Good times.
Re: (Score:3, Funny)
Re: (Score:3, Funny)
they paid the neighbor kid $71,800 to run norton data recovery [google.com]? That was a waste.
Re: (Score:2)
Re:Time for... (Score:5, Funny)
Re:Time for... (Score:5, Funny)
Re:Data Recovery options? (Score:5, Interesting)
Google was my friend. Shortly I learned more about backup superblocks, how to run "mkfs.ext3 -n" to do a dummy mkfs and find out where my backup superblocks are, and "fsck.ext3 -b nnn" to repair the filesystem using the backup superblock.
I was back running in less than an hour, including google time. Repairing an accidental mkswap on top of ext3 is actually one of the easier things to fix.
On the other hand, having a system and procedures that made it possible to kill regular and backup data that way, and storing unconfirmed tapes, is clearly not a good idea. Whenever I burn a CD/DVD, I take the few extra minutes and verify it right away. If the backup tape was only a few months old, odds are it was improperly written, as opposed to degraded. They should check their other backup tapes.
Redo the work? (Score:5, Insightful)
Re:Redo the work? (Score:5, Insightful)
Well, this is the government. They probably didn't have a budget for data recovery, but they did have a budget for scanning documents...the actual dollar amounts of each probably matter very little
Re: (Score:2, Interesting)
Re: (Score:2)
I sent a drive away in the past to find 3 pictures and it cost somewhere around $2500 for then entire drive (they wouldn't do just the three pictures). The pictures were worth 40% of 2 mill to my customer. Quite a bit less then 38 billion but I still had
Re:Redo the work? (Score:4, Interesting)
A law firm handed me a computer that wouldn't boot. On it was some pictures taken concerning a wrongful death case. It turns out that the pictures were a hindsight and in the middle of fixing it, The task turned from getting the computer to run to getting the pictures from the drive. The drive was failing and was larger then the 137gig 28bit LBA limits. But we didn't know this because it was never booted and XP pre SP1 did not enable 48bit addressing by default. And even after SP1, if you didn't update your ATAPI driver to x.1135 or later, it wouldn't be enabled by default even if you have the ability. So connecting it to another computer made it worse. Eventually the fault in the drive which was a crashed head, made it impossible for us to recover past the boot sector running traditional recovery software. The data recovery specialist were able to get around everything we added to the problem as well as the problem itself and retrieved better then 98% of everything on the drive. I think one file was bad but we weren't concerned with it at all.
Long story short, the pictures showed someone's negligence in a wrongful death case and once they were presented or added to the evidence pile, the defendant's insurance company settled for 2 mill. The lawsuit was for more then that so you could probably guess what it could have been worth. The firms cut was in the area of 40% from what I understand. So it was worth 40% of 2 mill to them. $2500 seems like a little amount in comparison.
Re: (Score:3, Informative)
Re:Redo the work? (Score:4, Funny)
The Senator (Score:3, Funny)
Senator Ted Stevens remarked that they should have sent it in an Internet, apparently tubes are much more reliable than tape.
Re: (Score:3, Funny)
And this is why... (Score:4, Interesting)
Re:And this is why... (Score:4, Funny)
Re:And this is why... (Score:5, Funny)
Right, because:
Nobody has ever thrown away papers that were actually needed...
Paper is an inexpensive and compact way to store terabytes of information...
Paper is trivially easy to instantly duplicate on a large scale...
Paper is trivially easy to haul off-site and store...
People constantly generate diffs between the most recently archived paper copy, and all work they have done every day since. They don't just make undocumented changes, willy-nilly, requiring just as much effort to backup daily changes as it is to backup full copies of everything...
No question, paper is superior. The data retention problems we always hear about are in every way caused by digital storage methods, and have nothing to do with the policies and people running the organizations...
(No I will not pay for any damaged caused by this post overloading your sarcasm meter.)
$38 billion? (Score:5, Insightful)
Re:$38 billion? (Score:4, Informative)
Re:$38 billion? (Score:4, Funny)
Re:$38 billion? (Score:4, Insightful)
Re: (Score:2)
The whole thing is a joke... (Score:2, Interesting)
What, another hyperbole-filled, wildly inaccurate Slashdot post? Inconceivable.
Re: (Score:2)
The fund is worth $38 billion. Dividend payouts from the fund last year were over $650 million, about $1,107 per payout, roughly 600,000 payouts.
This fund has nothing to do with tax revenue or annual gross product, it comes from oil revenues over a period of decades. The point of the fund is so the gov't doesn't have to pay out of a current account, and so doesn't NEED to depend on annual tax revenues.
Re: (Score:3, Informative)
At the simplest level, it's saved up money from the oil boom the state had in the 70's that the permanent fund corporation invests, saves, and takes care to insure it's always going to be there. Once a year it calculates earnings, subtracts operating and inflations costs, and hands out the remainder to qualifying Alaska residents. Usually it's in the area of $1000, but can fluctuate quite a bit.
They p
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
I can spend $200,000 on labor producing a large pile of shit (for that kind of money, it'd be very large), but it wouldn't be worth shit (well, actually, that'd be exactly what it's worth, but you get the point).
Re: (Score:2)
Yeah, it's far too hard to read the HEADLINE and/or the SECOND SENTENCE of the article, which both explain EXACTLY where the figure comes from...
I'm sure it was much faster to post a comment to
Investment VS Opportune Cost (Score:2)
Your investmen
Re: (Score:2)
Oh, and the $60 billion lost to "web video piracy" as per
http://www.havocscope.com/Counterfeit/webvideos.h
Re:Easy... LOL (Score:2)
I'll do it! (Score:5, Funny)
I'll do it for $1 billion.
Re: (Score:2)
maxtor? (Score:5, Funny)
Backups are the devil (Score:5, Insightful)
I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.
And the backup tape was unreadable. The next good backup was 17 days older.
After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.
You should REGULARLY test your backups.
You should have LOTS of backups.
Re: (Score:2)
inject + extract (Score:5, Informative)
After you run the backup, memove then restore that file, make sure it has the current date in it.
I've had that as a feature in my backup scripts for over 10 years...
Re: (Score:2)
Re: (Score:3, Interesting)
The hardware; is extremely expensive. And the software ain't cheap (if you expect any degree of automation or features).
This extortion racket is precisely why most people don't do backups, and of the few that do do backups, they do not test them. (but you've spent the money - never really understood that).
I have memories of ten years of sob stories; guys who were calling in to tech support because they were about to lose their jobs because they were poor steward
Where to start... (Score:2)
Over the next few days, as the department, the division and consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data, it became obvious the worst-case scenario was at hand.
2) Always make sure the backup really works.
3) Better procedures are needed if a single tech can reformat both hard drives in the same session
4) Much better hardware and software are needed for data worth $38 billion.
5) Paper backups are a good last resort and as a check on data integrity
I'd hate to be the tech support guy... (Score:4, Funny)
Damn! (Score:5, Insightful)
That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?
Alaskan Pipeline (Score:3, Insightful)
Data recovery? (Score:5, Interesting)
The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.
We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.
Re:Data recovery? (Score:5, Insightful)
Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.
I could be a douche and say it has never happened (Score:2)
Re:I could be a douche and say it has never happen (Score:5, Insightful)
Not to bad (Score:4, Informative)
I know that many companies would not be able to recover information lost in that manner.
I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.
To err is human... (Score:2)
From , "nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone." [msn.com]
Really? Why is the that the oil money payouts or the military contract accounts are the only ones that ever get deleted? The IRS is using the same database that they've been using for the past fifty plus years, but they never seem to have that problem.
Remember kids... (Score:2)
If you aren't regulary testing your recovery capabilities, your nightly backups are masturbation. It may make you feel good for a bit, but it's not satisfying.
I lost 75 trillion dollars! (Score:5, Funny)
Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.
Someone is trying to cover their ass (Score:5, Insightful)
Backup disk: Accidently formatted.
Tape: Unreadable.
What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?
Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.
Re:Someone is trying to cover their ass (Score:5, Informative)
Having worked on backups in an Alaskan company run very similarly to the department in question, I think it sounds reasonable. I was a consultant at the time, and I pointed out that the backups have never been tested. It was on the weekly report. It was on the weekly report for about a year. Many people making much more than you make saw that the backups have never been tested. Then there was a crash. It turned out that the backups, set up long before I got there, were set in a tape library. There were 5 tapes and a cleaning tape. The backups would backup server 1 onto tape 1. Then, server 2 onto tape on - set to overwrite. The least important server in the room was last on the backup list, and it was the last to issue the command every night to backup onto tape 1 - set to overwrite. So the email was gone forever. Somehow, the consulting company I work for that pointed out for over a year that backups weren't tested and may not work was to blame for not fixing what was broken long before we were brought in. So I find the description of events quite plausible.
Vista (Score:4, Funny)
Perspective (Score:5, Funny)
Re:Tapes? (Score:5, Informative)
BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.
Re:Tapes? (Score:5, Insightful)
Re:Tapes? (Score:5, Interesting)
Re: (Score:3, Informative)
Re: (Score:2)
Re:Tapes? (Score:4, Insightful)
Re:Tapes? (Score:5, Insightful)
I work for an IT organization and we pay a company called Iron Mountain $100's monthly to schlep our boxes and boxes of backup tapes to their offsite storage facility.
And remember there is a difference between making 'backups' (store my important files somewhere else so I can get them in case of a system failure) and preparing for 'disaster recovery' (store everyones files somewhere else so we can rebuild the entire infrastructure in case the building burns to the ground).
Re: (Score:3, Insightful)
I can have a simple HA cluster that involves two nodes attached to a single disk array, all sitting in the same rack somewhere. Take a guess what happens when the power for the building goes out?
HA is nice, but will do nothing for you in the event of a disaster.
You can structure your site so that you get both, but doing so requires a lot more work (stretched clusters and SAN's spread over miles) and you have to be careful that you dont trash your performance while you are at it. (rea
Re: (Score:3, Interesting)
Re: (Score:3, Insightful)
Yes you will, or should, when the user's manager gets involved. If you would like to think that because a user trashes a month's accounts, that you can wave some magic hand and say "Yes, I know that data is in pristine condition on last week's backup, but no, you're not getting it just because Waldo
Re: (Score:2)
Of course, if you need to store terabytes upon terabytes of data, tape may still by the only option....
Re:Tapes? (Score:5, Informative)
Data Volume (Score:3, Informative)
Id skip on the DVD backup, sounds like a mistake waiting to happen. Backing this up to a network drive over Gig-E is still going to be a mess, but it should be a few hours of slacktime.. (yes in theory you could manage 240 gigs in roughly 35 minutes over gig-E, but you couldnt pull off enough seeks in that time via the hard drive (800k seeks * 8 ms/seek= 6400s ~= 106 minutes).
Storm
Re:Tapes? (Score:5, Informative)
And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.
But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.
Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.
Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and ... well, no one's going to spend that kind of sum of money (millions) trivially.
Re: (Score:3, Interesting)
As an IT auditor, I do ding IT shops when they don't do full system restores (which has the dual benefits of verifying that the techs are capable, and verifying that the media is readable). I'm going to be printing out this story and showing it to people who don't do full system restores... I get along fine with BOFHs, and I can sympathize with them about the burden of SOX, but while I'
Re: (Score:2)
They maynot of been maintaining their tapes, or they were using low quality equipment, or it was just one of those things.
For the illogical 'anti-tape' crowd, I will point uot that they did have a problem with the hard drive as well.
Re: (Score:3, Insightful)
Re: (Score:2)
Re:Tapes? (Score:5, Insightful)
Re: (Score:3, Insightful)
No, you _CANT_.
For that, this "scratch" has to be a massive gash, which goes more than halfway through the depth of the plastic disc. Additionally, it has to span the entire radius of the disk just to make it "difficult" to recover a significant portion of the data.
Your backup solution should NOT involve throwing a bunch of bare DVD-R
Re: (Score:2)
Re: (Score:2)
For home use, online backups are fine, but for even small businesses, volumes of data approaching 1TB are not uncommon.
re: tape backup (Score:2)
DVDs are a joke. (Score:5, Insightful)
Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.
The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.
Re: (Score:3, Informative)
Add robots, and it gets seriously expensive for large installations...
Re: (Score:3, Informative)
I guess it depends on what you're backing up...
Of all our clients, the smallest backup we've got is about 14 GB. That's too big for a single DVD-R, but it fits just fine on a DDS3 [wikipedia.org]. We can also easily automate a tape backup - just instruct a secretary or someone to swap the tape in the morning. Tapes are reasonably durable too...more rugged, in general, than a removable HDD.
Re: (Score:3, Insightful)
Yeah, but when LTO4 is out you can't use them without buying a new _expensive_ LTO4 drive.
Whereas in the not too distant future when new hard drives with double the capacity are out, you can still use them with your existing computers (as long as they still support SATA).
Basically HDDs = media + drive, and they are about the same price as tap
Re: (Score:2)
That implies that the original and the back up had been reformatted.
Re: (Score:2)
I've often been in the situation where I couldn't properly test the backup system because management decided to save money by not buying any spare hardware.
"Hey, we bought you a tape drive and a box of tapes, quit complaining!"
Re: (Score:2)
Note from the article (and the summary) the IT guy formatted both the original and the backup hard drives. Then there was the bad tape. So that's 2 sources. What surprises me is it wasn't on another tape somewhere.
Re: (Score:2)
Re:Actual Cost?? (Score:4, Informative)
The site is slashdotted, so I can't read TFA, but my guess would be the information isn't actually "worth" $38B. It just represents an accounted amount of $38B.
The actual value of the data is what it would cost to replace it (or perhaps do without it) -- in this case, $200,000. Consider an analogy (20th-century, but illustrative): if you were to send a paper bank-check for $10,000 via a courier, the declared value for insurance would not be $10,000. It would be the cost of recovering from the loss of the check, which would be the stop-payment fee plus the cost of sending a new one.
Re:Actual Cost?? (Score:5, Funny)
2) Type what's on the paper into a
3) Save the file
4) Delete the file
5) Empty the recycle bin
6) Recreate file by retyping data from the paper
7) Post the story on the
Re: (Score:2)
Re: (Score:3, Funny)