Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Data Storage The Almighty Buck IT

So You've Lost a $38 Billion File 511

Posted by Zonk from the makes-you-feel-better-about-yourself dept.
smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."
This discussion has been archived. No new comments can be posted.

So You've Lost a $38 Billion File More

So You've Lost a $38 Billion File

Comments Filter:

  • Time for... (Score:5, Funny)

    by Short Circuit ( 52384 ) * <mikemol@gmail.com> on Tuesday March 20, 2007 @10:59AM (#18414365) Homepage Journal
    Seppuku [wikipedia.org]?
    • He surely wouldn't want for a Second.
      • Data recovery.

        Geez guys! They can find the Pr0n you "deleted". I guess there needs to be more significant motivation than $38 Bil USD.

        • Re:Time for... (Score:5, Insightful)

          by jadavis ( 473492 ) on Tuesday March 20, 2007 @12:10PM (#18415899)
          It was actually only worth $200k, since that's the amount of money it took to recover from the problem.

          The fact that it was related to an account worth $38B is scary, but not the actual cost.

          • Re:Time for... (Score:5, Interesting)

            by nelsonal ( 549144 ) on Tuesday March 20, 2007 @12:35PM (#18416373) Journal
            It has to be worth more than $200k (or else the would have just written off the file). I agree that the info isn't worth the $38B though. The account would still be there even if those transactions weren't. That said I thought all the states learned the lesson of 9/11 of remote offsite backups especially in a state as geologically active as Alaska. Hope the warehouse with the paper isn't near the data center...

          • $38 Billion is a big incentive for fraud (Score:5, Interesting)

            by spun ( 1352 ) <loverevolutionary&yahoo,com> on Tuesday March 20, 2007 @12:39PM (#18416457) Journal
            Ah, but we don't know the actual cost. Thirty eight billion is a lot of money. Suppose I wanted to skim some of that money, but I knew that the documentation existed in paper and computerized form. Perhaps I know someone in the records department who can shuffle some papers, but then the computerized records won't match. Oops, now those records are gone and we have no choice but to scan in the documents that I have changed, now everything agrees and there is no record of where that extra million or ten went.

            • Re:$38 Billion is a big incentive for fraud (Score:5, Funny)

              by Anonymous Coward on Tuesday March 20, 2007 @01:29PM (#18417363)
              Dear deeply respected and trusted Sir,

                I trust by the grace of almighty god you are in good health this fine and beautiful day. I was a data entry clerk for the Alaska Department of Revenue, Prior to being fired, I secured access to a hidden fund worth $38,000,000,000 (THIRTY-EIGHT BILLION DOLLARS).

                If I ever tried to utilise this fund in my name, the funds would risk being confiscated by the government, so I would like you assistance to find a trustworth foreign assistant who can invest these funds.

                This proposal is 100% risk free, and I can offer you a 10% fee for your help.....

    • Re: (Score:2)

      by thc69 ( 98798 )
      No, time for Google [wikipedia.org] Seppuku [livejournal.com].

    • Re:Time for... (Score:5, Funny)

      by Rakshasa Taisab ( 244699 ) on Tuesday March 20, 2007 @11:26AM (#18414993) Homepage
      Thanks for alleviating my ignorance [wikipedia.org].

  • Redo the work? (Score:5, Insightful)

    by Anonymous Coward on Tuesday March 20, 2007 @11:02AM (#18414433)
    For that kind of money, I'd probably just send the HD to data recovery specialists.

    • Re:Redo the work? (Score:5, Insightful)

      by omeomi ( 675045 ) on Tuesday March 20, 2007 @11:08AM (#18414587) Homepage
      For that kind of money, I'd probably just send the HD to data recovery specialists.

      Well, this is the government. They probably didn't have a budget for data recovery, but they did have a budget for scanning documents...the actual dollar amounts of each probably matter very little ;-)

    • Re: (Score:2, Interesting)

      by bilbobob ( 1036984 )
      HD Recovery specialists? They could have could have bought something like GetBackData for NTFS and saved themselves $199921 in recovery costs. As far as I'm aware, reformatting your HD is one of the least successful methods of permanently destroying your data (even if you mean too).
    • Do you really think that is appropriate? I would have brought the recovery team here instead of sending it away. You know how important packages get lost in the mail. It would suck to lose this and have it be your last change at getting something out of it.

      I sent a drive away in the past to find 3 pictures and it cost somewhere around $2500 for then entire drive (they wouldn't do just the three pictures). The pictures were worth 40% of 2 mill to my customer. Quite a bit less then 38 billion but I still had

    • Re: (Score:3, Informative)

      by SydShamino ( 547793 )
      From the CNN article, they spent about $75,000 on data recovery specialists (who could not recover the data), then spent another $125,000 rescanning all the documents from paper.

  • The Senator (Score:3, Funny)

    by jeevesbond ( 1066726 ) on Tuesday March 20, 2007 @11:03AM (#18414437) Homepage

    Senator Ted Stevens remarked that they should have sent it in an Internet, apparently tubes are much more reliable than tape.

    • Re: (Score:3, Funny)

      by Donniedarkness ( 895066 )
      They DID send it in an internet, but we were clogging up the tubes with "youtubes" and "myspaces" that it didn't reach them until AFTER they finished copying it!

  • And this is why... (Score:4, Interesting)

    by bobetov ( 448774 ) on Tuesday March 20, 2007 @11:03AM (#18414439) Homepage
    ...print will never be dead.

    • Re:And this is why... (Score:5, Funny)

      by evilviper ( 135110 ) on Tuesday March 20, 2007 @11:34AM (#18415127) Journal

      And this is why... ...print will never be dead.

      Right, because:

      Nobody has ever thrown away papers that were actually needed...

      Paper is an inexpensive and compact way to store terabytes of information...

      Paper is trivially easy to instantly duplicate on a large scale...

      Paper is trivially easy to haul off-site and store...

      People constantly generate diffs between the most recently archived paper copy, and all work they have done every day since. They don't just make undocumented changes, willy-nilly, requiring just as much effort to backup daily changes as it is to backup full copies of everything...

      No question, paper is superior. The data retention problems we always hear about are in every way caused by digital storage methods, and have nothing to do with the policies and people running the organizations...

      (No I will not pay for any damaged caused by this post overloading your sarcasm meter.)

  • $38 billion? (Score:5, Insightful)

    by pipatron ( 966506 ) <pipatron@gmail.com> on Tuesday March 20, 2007 @11:03AM (#18414441) Homepage
    How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?

    • Re:$38 billion? (Score:4, Informative)

      by stoolpigeon ( 454276 ) * <bittercode@gmail> on Tuesday March 20, 2007 @11:06AM (#18414511) Homepage Journal
      right - the account is worth 38 billion - the file was apparently worth about 200 grand in labor. of course it didn't cost that much to make the first time, as it was done over a longer period without all the o.t.

    • Re:$38 billion? (Score:4, Funny)

      by 192939495969798999 ( 58312 ) <[info] [at] [devinmoore.com]> on Tuesday March 20, 2007 @11:07AM (#18414547) Homepage Journal
      The method for arriving at that figure was also tragically lost. A team of monkeys recreated the figure in 3 minutes with a number pad at a cost of $45.

    • Re:$38 billion? (Score:4, Insightful)

      by __aapbzv4610 ( 411560 ) on Tuesday March 20, 2007 @11:08AM (#18414591)
      The value is based on what the data represents, not the material labor in re-scanning them. loose example, I could spen $30 on painting supplies and create a $100,000 masterpiece (well not by me, but, anyway...)

      • Re: (Score:2)

        by mwvdlee ( 775178 )
        Presumably you could not recreate that same identical masterpiece if the original were destroyed. In this case they could.
    • No matter what they might have lost, it wasn't worth $38 billion. The state gross product of Alaska is only $33 billion, and their tax revenue will only be a fraction of that. And it's not like they lost any of that either, just some files.

      What, another hyperbole-filled, wildly inaccurate Slashdot post? Inconceivable.

      • Hmm. Did you RTFA?

        The fund is worth $38 billion. Dividend payouts from the fund last year were over $650 million, about $1,107 per payout, roughly 600,000 payouts.

        This fund has nothing to do with tax revenue or annual gross product, it comes from oil revenues over a period of decades. The point of the fund is so the gov't doesn't have to pay out of a current account, and so doesn't NEED to depend on annual tax revenues.

      • Re: (Score:3, Informative)

        by Xzzy ( 111297 )
        The Alaska Permanent Fund is not tax funded at all. Technically, it's not even part of the state government.

        At the simplest level, it's saved up money from the oil boom the state had in the 70's that the permanent fund corporation invests, saves, and takes care to insure it's always going to be there. Once a year it calculates earnings, subtracts operating and inflations costs, and hands out the remainder to qualifying Alaska residents. Usually it's in the area of $1000, but can fluctuate quite a bit.

        They p

    • Re: (Score:2)

      by mikael ( 484 )
      Because that is the sum of all the financial transactions that were legally documented. Suppose every pae of those 800,000 pages is a single tax return worth on average $5000 to the state. Then you get a rather large number: 800,000 pages * $5000 = $40,000,000,000
      • Yes, but as has been pointed out numerous times, the file itself is not worth $38 billion as was stated. If I write a check for $100 and it gets destroyed, I can just write you another check. That means the check itself isn't worth $100. It's only worth how much it costs to get that check printed and reissued.

    • Re: (Score:2)

      by mwvdlee ( 775178 )
      Actually, the value of those files is what somebody else would pay for it.

      I can spend $200,000 on labor producing a large pile of shit (for that kind of money, it'd be very large), but it wouldn't be worth shit (well, actually, that'd be exactly what it's worth, but you get the point).

    • How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?

      Yeah, it's far too hard to read the HEADLINE and/or the SECOND SENTENCE of the article, which both explain EXACTLY where the figure comes from...

      I'm sure it was much faster to post a comment to /. and check every few hours until someone to posts an answer.

    • Lets say you have a small software development company. You have 6 employees that cost you on average of $90k per year each (including taxes, 401k match, health care, salary, office space, etc...) Now lets say that it takes you 2 years to build your application. Over those two years your team spent 20% of the time in meetings, 15% of the time goofing off, 10% of the time debugging, 20% of the time designing/planning, 5% of the time training, and the rest (25%) actually making progress on code.

      Your investmen
    • File it right next to the $18 billion (or whatever this week'd ass-originated number is) in "losses" that the RIAA/MPAA attribute to piracy.

      Oh, and the $60 billion lost to "web video piracy" as per
      http://www.havocscope.com/Counterfeit/webvideos.ht m [havocscope.com]

  • I'll do it! (Score:5, Funny)

    by mwvdlee ( 775178 ) on Tuesday March 20, 2007 @11:06AM (#18414509) Homepage
    So the information is still available in 300 boxes and it would cost about $200,000 to scan and recreate the $38 billion file again?
    I'll do it for $1 billion.

  • maxtor? (Score:5, Funny)

    by ElephanTS ( 624421 ) on Tuesday March 20, 2007 @11:06AM (#18414525)
    As their IT consultant I stand by my use of Maxtor drives.

  • Backups are the devil (Score:5, Insightful)

    by Maxo-Texas ( 864189 ) on Tuesday March 20, 2007 @11:09AM (#18414621)
    Because no one ever restores them regularly to test them.

    I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.

    And the backup tape was unreadable. The next good backup was 17 days older.

    After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.

    You should REGULARLY test your backups.
    You should have LOTS of backups.

    • Re: (Score:2)

      by mwvdlee ( 775178 )
      Isn't it possible to have tapes automatically verify data? Sure, it may take a while longer for the process to finish, but atleast you'll know the process was actually worth the wait.
    • Like an old boss of mine once said, "You don't have a backup if you've never done a restore."

    • Re: (Score:3, Interesting)

      by jafac ( 1449 )
      I used to work for a backup software company.

      The hardware; is extremely expensive. And the software ain't cheap (if you expect any degree of automation or features).

      This extortion racket is precisely why most people don't do backups, and of the few that do do backups, they do not test them. (but you've spent the money - never really understood that).

      I have memories of ten years of sob stories; guys who were calling in to tech support because they were about to lose their jobs because they were poor steward
  • 1) No surprise reading this:

    Over the next few days, as the department, the division and consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data, it became obvious the worst-case scenario was at hand.

    2) Always make sure the backup really works.

    3) Better procedures are needed if a single tech can reformat both hard drives in the same session

    4) Much better hardware and software are needed for data worth $38 billion.

    5) Paper backups are a good last resort and as a check on data integrity

  • I'd hate to be the tech support guy... (Score:4, Funny)

    by condour75 ( 452029 ) on Tuesday March 20, 2007 @11:11AM (#18414669) Homepage
    And it's not in the recycle bin? Ok, let's not panic. Click start, go to find, choose files and folders...

  • Damn! (Score:5, Insightful)

    by Penguinisto ( 415985 ) on Tuesday March 20, 2007 @11:12AM (#18414681) Journal
    That has got to rank up there w/ the all-time worst 'oh-shit' moments in that poor bastard's career (if it still exists). I wonder what the sysadmin was thinking, storing data on the same partition as the OS. No sane production environment rig that I know of would (or at least should) have that. It may be a Windows thing, but on most servers I've dinked with, the OS sat on a pair of RAID disks by itself, and all the data sat on the monster pile of disks on their own logical RAID drive (at least RAID 5... 5+0 w/ a hot spare preferred).

    That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?

    /P

  • Alaskan Pipeline (Score:3, Insightful)

    by Doc Ruby ( 173196 ) on Tuesday March 20, 2007 @11:13AM (#18414723) Homepage Journal
    I bet it cost a lot less than $200K to bribe the government officials (probably with a few bottles of wine) not to check whether they were protecting their $38B investment with more than $45K worth of IT staff.

  • Data recovery? (Score:5, Interesting)

    by evilviper ( 135110 ) on Tuesday March 20, 2007 @11:16AM (#18414777) Journal
    With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.

    The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.

    We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.

    • Re:Data recovery? (Score:5, Insightful)

      by mwvdlee ( 775178 ) on Tuesday March 20, 2007 @11:29AM (#18415061) Homepage
      I'm just assuming the harddisks were secure erased, considering that is what pretty much every govenment in the world does when formatting harddisks.

      Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.
  • But I'll admit that about 6 years ago, when moving from one old laptop to another, I accidentally erased all of the data from my main and my backup Jaz disks before ensuring that every piece of data from 6th grade through my first year of college was backed up onto new media. It wasn't that I didn't have a good strategy for backup: it was adequate for my needs at the time. It's not that I didn't know I shouldn't format disks with important stuff on them: I figured I had transferred it already. At age 18,

  • Not to bad (Score:4, Informative)

    by geekoid ( 135745 ) <dadinportlandNO@SPAMyahoo.com> on Tuesday March 20, 2007 @11:17AM (#18414797) Homepage Journal
    That they cuold get recovery for only 200K.

    I know that many companies would not be able to recover information lost in that manner.

    I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.

  • To err is human, to really f-up requires root.

    From , "nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone." [msn.com]

    Really? Why is the that the oil money payouts or the military contract accounts are the only ones that ever get deleted? The IRS is using the same database that they've been using for the past fifty plus years, but they never seem to have that problem.
  • It's not about the backup, it's about the restore.

    If you aren't regulary testing your recovery capabilities, your nightly backups are masturbation. It may make you feel good for a bit, but it's not satisfying.

  • I lost 75 trillion dollars! (Score:5, Funny)

    by 140Mandak262Jamuna ( 970587 ) on Tuesday March 20, 2007 @11:24AM (#18414957) Journal
    I launched notepad and wrote 75 trillion dollars and saved the file. And, because I was feeling very extravagent, I deleted the file. I am rich. I can afford to lose 75 trillion dollars without batting and eyelid and am man enough to brag the info to the whole world.

    Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.

  • Someone is trying to cover their ass (Score:5, Insightful)

    by Unique2 ( 325687 ) on Tuesday March 20, 2007 @11:39AM (#18415215)
    Primary disk: Accidently deleted.
    Backup disk: Accidently formatted.
    Tape: Unreadable.

    What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?

    Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.

    • Re:Someone is trying to cover their ass (Score:5, Informative)

      by AK Marc ( 707885 ) on Tuesday March 20, 2007 @01:01PM (#18416849)
      It sounds more like the reality is that none of these ever existed and someone got caught-out.

      Having worked on backups in an Alaskan company run very similarly to the department in question, I think it sounds reasonable. I was a consultant at the time, and I pointed out that the backups have never been tested. It was on the weekly report. It was on the weekly report for about a year. Many people making much more than you make saw that the backups have never been tested. Then there was a crash. It turned out that the backups, set up long before I got there, were set in a tape library. There were 5 tapes and a cleaning tape. The backups would backup server 1 onto tape 1. Then, server 2 onto tape on - set to overwrite. The least important server in the room was last on the backup list, and it was the last to issue the command every night to backup onto tape 1 - set to overwrite. So the email was gone forever. Somehow, the consulting company I work for that pointed out for over a year that backups weren't tested and may not work was to blame for not fixing what was broken long before we were brought in. So I find the description of events quite plausible.

  • Vista (Score:4, Funny)

    by Joe The Dragon ( 967727 ) on Tuesday March 20, 2007 @12:03PM (#18415761)
    They may of been trying to do a clean install of vista and it some how took out the back up disk and the same time as the main disk. And they where using dell systems.

  • Perspective (Score:5, Funny)

    by rlp ( 11898 ) on Tuesday March 20, 2007 @01:06PM (#18416947)
    $38 billion is a lot of money. To put that in perspective, for $38 billion, Alaska could build over fifty bridges to nowhere.

Slashdot Top Deals

One man's constant is another man's variable. -- A.J. Perlis

Close