So You've Lost a $38 Billion File

smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."

Time for...

[Short Circuit]

Seppuku [wikipedia.org]?

Re:

[voice_of_all_reason]

He surely wouldn't want for a Second.

Re:

[Philip K Dickhead]

Data recovery.

Geez guys! They can find the Pr0n you "deleted". I guess there needs to be more significant motivation than $38 Bil USD.

Re:Time for...

[jadavis]

It was actually only worth $200k, since that's the amount of money it took to recover from the problem.

The fact that it was related to an account worth $38B is scary, but not the actual cost.

Re:Time for...

[nelsonal]

It has to be worth more than $200k (or else the would have just written off the file). I agree that the info isn't worth the $38B though. The account would still be there even if those transactions weren't. That said I thought all the states learned the lesson of 9/11 of remote offsite backups especially in a state as geologically active as Alaska. Hope the warehouse with the paper isn't near the data center...

$38 Billion is a big incentive for fraud

[spun]

Ah, but we don't know the actual cost. Thirty eight billion is a lot of money. Suppose I wanted to skim some of that money, but I knew that the documentation existed in paper and computerized form. Perhaps I know someone in the records department who can shuffle some papers, but then the computerized records won't match. Oops, now those records are gone and we have no choice but to scan in the documents that I have changed, now everything agrees and there is no record of where that extra million or ten went.

Re:$38 Billion is a big incentive for fraud

[Anonymous Coward]

Dear deeply respected and trusted Sir,

  I trust by the grace of almighty god you are in good health this fine and beautiful day. I was a data entry clerk for the Alaska Department of Revenue, Prior to being fired, I secured access to a hidden fund worth $38,000,000,000 (THIRTY-EIGHT BILLION DOLLARS).

  If I ever tried to utilise this fund in my name, the funds would risk being confiscated by the government, so I would like you assistance to find a trustworth foreign assistant who can invest these funds.

  This proposal is 100% risk free, and I can offer you a 10% fee for your help.....

Re:$38 Billion is a big incentive for fraud

[iocat]

They hired Dell and Microsoft to do the recover. Why not drivesavers [drivesavers.com] or some other reputable firm that doesn't just run Norton but looks at the physical bits on the disc.

Anyway, what DIDN'T shock me about this story is that after formatting the main disk, the tech immediately (and blissfully) formatted the backup as well. I've seen stuff like that happen like ten times. ("Oh, well, after I replaced the drive, I figured I should replace the backup tapes too, so we could have a fresh start, so I threw them out." or "I figured I should make a backup right away, so I over-wrote the good backup with the new, bad, data.") I don't want to blame the victim, but sometimes it's like the data wants to be destroyed at that point. My favorite was when someone added a second drive to an important source control server to do nightly drive to drive back-ups. Then, they stopped doing tape backups nightly and switched to weekly. Then, they forgot they disconnected a fan during the HDD installation (or it was accidently disconnected -- it remains a debated point), then the server fried itself and the drives. Then everyone lost a day of work rebuilding the source archive based on their local data. Good times.

Re:

[HTTP Error 403 403.9]

If you must go with the car analogy, it's more like losing your key and saying "zoinks! I just lost my $85,000 car key!"

It's actually Velma's car and she said, "Jinkies! I'm locked out of the Mystery Machine!"

Re:

[iamhassi]

"they tried to recover and it cost them $71,800."

they paid the neighbor kid $71,800 to run norton data recovery [google.com]? That was a waste.

Re:

[thc69]

No, time for Google [wikipedia.org] Seppuku [livejournal.com].

Re:Time for...

[Rakshasa Taisab]

Thanks for alleviating my ignorance [wikipedia.org].

Re:Time for...

[Anonymous Coward]

I once deleted a file worth $38 billion. I was afraid my boss was going to fire me, but he shook my hand and laughed "Fire you?!! We just invested $38 billion in your education! We can't fire you after that kind of investment." And true enough, I never deleted a file worth that much again.

Re:Data Recovery options?

[dpilot]

About a month or 2 back, a slip of the fingers turned my root filesystem into a linux swap partition.

Google was my friend. Shortly I learned more about backup superblocks, how to run "mkfs.ext3 -n" to do a dummy mkfs and find out where my backup superblocks are, and "fsck.ext3 -b nnn" to repair the filesystem using the backup superblock.

I was back running in less than an hour, including google time. Repairing an accidental mkswap on top of ext3 is actually one of the easier things to fix.

On the other hand, having a system and procedures that made it possible to kill regular and backup data that way, and storing unconfirmed tapes, is clearly not a good idea. Whenever I burn a CD/DVD, I take the few extra minutes and verify it right away. If the backup tape was only a few months old, odds are it was improperly written, as opposed to degraded. They should check their other backup tapes.

Redo the work?

[Anonymous Coward]

For that kind of money, I'd probably just send the HD to data recovery specialists.

Re:Redo the work?

[omeomi]

For that kind of money, I'd probably just send the HD to data recovery specialists.

Well, this is the government. They probably didn't have a budget for data recovery, but they did have a budget for scanning documents...the actual dollar amounts of each probably matter very little ;-)

Re:

[bilbobob]

HD Recovery specialists? They could have could have bought something like GetBackData for NTFS and saved themselves $199921 in recovery costs. As far as I'm aware, reformatting your HD is one of the least successful methods of permanently destroying your data (even if you mean too).

Re:

[sumdumass]

Do you really think that is appropriate? I would have brought the recovery team here instead of sending it away. You know how important packages get lost in the mail. It would suck to lose this and have it be your last change at getting something out of it.

I sent a drive away in the past to find 3 pictures and it cost somewhere around $2500 for then entire drive (they wouldn't do just the three pictures). The pictures were worth 40% of 2 mill to my customer. Quite a bit less then 38 billion but I still had

Re:Redo the work?

[sumdumass]

Nah, Nothing like that. It isn't really exciting either. I cannot really give any names or anything. But I can generically describe what happened.

A law firm handed me a computer that wouldn't boot. On it was some pictures taken concerning a wrongful death case. It turns out that the pictures were a hindsight and in the middle of fixing it, The task turned from getting the computer to run to getting the pictures from the drive. The drive was failing and was larger then the 137gig 28bit LBA limits. But we didn't know this because it was never booted and XP pre SP1 did not enable 48bit addressing by default. And even after SP1, if you didn't update your ATAPI driver to x.1135 or later, it wouldn't be enabled by default even if you have the ability. So connecting it to another computer made it worse. Eventually the fault in the drive which was a crashed head, made it impossible for us to recover past the boot sector running traditional recovery software. The data recovery specialist were able to get around everything we added to the problem as well as the problem itself and retrieved better then 98% of everything on the drive. I think one file was bad but we weren't concerned with it at all.

Long story short, the pictures showed someone's negligence in a wrongful death case and once they were presented or added to the evidence pile, the defendant's insurance company settled for 2 mill. The lawsuit was for more then that so you could probably guess what it could have been worth. The firms cut was in the area of 40% from what I understand. So it was worth 40% of 2 mill to them. $2500 seems like a little amount in comparison.

Re:

[SydShamino]

From the CNN article, they spent about $75,000 on data recovery specialists (who could not recover the data), then spent another $125,000 rescanning all the documents from paper.

Re:Redo the work?

[Solder Fumes]

No kidding...if "Mary" from Dell "canna fiss eet, you drag to reee-cyc bin yesss? reboot? all gone, have nice day now" then who are we mortals to argue?

The Senator

[jeevesbond]

Senator Ted Stevens remarked that they should have sent it in an Internet, apparently tubes are much more reliable than tape.

Re:

[Donniedarkness]

They DID send it in an internet, but we were clogging up the tubes with "youtubes" and "myspaces" that it didn't reach them until AFTER they finished copying it!

And this is why...

[bobetov]

...print will never be dead.

Re:And this is why...

[imboboage0]

http://static.flickr.com/2/3704348_8ceda601f9.jpg [flickr.com]

Re:And this is why...

[evilviper]

And this is why... ...print will never be dead.

Right, because:

Nobody has ever thrown away papers that were actually needed...

Paper is an inexpensive and compact way to store terabytes of information...

Paper is trivially easy to instantly duplicate on a large scale...

Paper is trivially easy to haul off-site and store...

People constantly generate diffs between the most recently archived paper copy, and all work they have done every day since. They don't just make undocumented changes, willy-nilly, requiring just as much effort to backup daily changes as it is to backup full copies of everything...

No question, paper is superior. The data retention problems we always hear about are in every way caused by digital storage methods, and have nothing to do with the policies and people running the organizations...

(No I will not pay for any damaged caused by this post overloading your sarcasm meter.)

$38 billion?

[pipatron]

How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?

Re:$38 billion?

[stoolpigeon]

right - the account is worth 38 billion - the file was apparently worth about 200 grand in labor. of course it didn't cost that much to make the first time, as it was done over a longer period without all the o.t.

Re:$38 billion?

[192939495969798999]

The method for arriving at that figure was also tragically lost. A team of monkeys recreated the figure in 3 minutes with a number pad at a cost of $45.

Re:$38 billion?

[__aapbzv4610]

The value is based on what the data represents, not the material labor in re-scanning them. loose example, I could spen $30 on painting supplies and create a $100,000 masterpiece (well not by me, but, anyway...)

Re:

[mwvdlee]

Presumably you could not recreate that same identical masterpiece if the original were destroyed. In this case they could.

The whole thing is a joke...

[Rocketship Underpant]

No matter what they might have lost, it wasn't worth $38 billion. The state gross product of Alaska is only $33 billion, and their tax revenue will only be a fraction of that. And it's not like they lost any of that either, just some files.

What, another hyperbole-filled, wildly inaccurate Slashdot post? Inconceivable.

Re:

[Red Flayer]

Hmm. Did you RTFA?

The fund is worth $38 billion. Dividend payouts from the fund last year were over $650 million, about $1,107 per payout, roughly 600,000 payouts.

This fund has nothing to do with tax revenue or annual gross product, it comes from oil revenues over a period of decades. The point of the fund is so the gov't doesn't have to pay out of a current account, and so doesn't NEED to depend on annual tax revenues.

Re:

[Xzzy]

The Alaska Permanent Fund is not tax funded at all. Technically, it's not even part of the state government.

At the simplest level, it's saved up money from the oil boom the state had in the 70's that the permanent fund corporation invests, saves, and takes care to insure it's always going to be there. Once a year it calculates earnings, subtracts operating and inflations costs, and hands out the remainder to qualifying Alaska residents. Usually it's in the area of $1000, but can fluctuate quite a bit.

They p

Re:

[mikael]

Because that is the sum of all the financial transactions that were legally documented. Suppose every pae of those 800,000 pages is a single tax return worth on average $5000 to the state. Then you get a rather large number: 800,000 pages * $5000 = $40,000,000,000

Re:

[illegalcortex]

Yes, but as has been pointed out numerous times, the file itself is not worth $38 billion as was stated. If I write a check for $100 and it gets destroyed, I can just write you another check. That means the check itself isn't worth $100. It's only worth how much it costs to get that check printed and reissued.

Re:

[mwvdlee]

Actually, the value of those files is what somebody else would pay for it.

I can spend $200,000 on labor producing a large pile of shit (for that kind of money, it'd be very large), but it wouldn't be worth shit (well, actually, that'd be exactly what it's worth, but you get the point).

Re:

[evilviper]

How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?

Yeah, it's far too hard to read the HEADLINE and/or the SECOND SENTENCE of the article, which both explain EXACTLY where the figure comes from...

I'm sure it was much faster to post a comment to /. and check every few hours until someone to posts an answer.

Investment VS Opportune Cost

[RingDev]

Lets say you have a small software development company. You have 6 employees that cost you on average of $90k per year each (including taxes, 401k match, health care, salary, office space, etc...) Now lets say that it takes you 2 years to build your application. Over those two years your team spent 20% of the time in meetings, 15% of the time goofing off, 10% of the time debugging, 20% of the time designing/planning, 5% of the time training, and the rest (25%) actually making progress on code.

Your investmen

Re:

[Mateo_LeFou]

File it right next to the $18 billion (or whatever this week'd ass-originated number is) in "losses" that the RIAA/MPAA attribute to piracy.

Oh, and the $60 billion lost to "web video piracy" as per
http://www.havocscope.com/Counterfeit/webvideos.ht m [havocscope.com]

Re:Easy... LOL

[Maxo-Texas]

And, I have ot say those numbers were smelly. Not sure where they pulled them from.. but...

I'll do it!

[mwvdlee]

So the information is still available in 300 boxes and it would cost about $200,000 to scan and recreate the $38 billion file again?
I'll do it for $1 billion.

Re:

[Misch]

Halliburton? Didn't we tell you to get off slashdot?

maxtor?

[ElephanTS]

As their IT consultant I stand by my use of Maxtor drives.

Backups are the devil

[Maxo-Texas]

Because no one ever restores them regularly to test them.

I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.

And the backup tape was unreadable. The next good backup was 17 days older.

After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.

You should REGULARLY test your backups.
You should have LOTS of backups.

Re:

[mwvdlee]

Isn't it possible to have tapes automatically verify data? Sure, it may take a while longer for the process to finish, but atleast you'll know the process was actually worth the wait.

inject + extract

[mengel]

Every night before backups add a file to partitions being backed up (like '.backuptst.txt') with the date in it.

After you run the backup, memove then restore that file, make sure it has the current date in it.

I've had that as a feature in my backup scripts for over 10 years...

Re:

[Hoi Polloi]

Like an old boss of mine once said, "You don't have a backup if you've never done a restore."

Re:

[jafac]

I used to work for a backup software company.

The hardware; is extremely expensive. And the software ain't cheap (if you expect any degree of automation or features).

This extortion racket is precisely why most people don't do backups, and of the few that do do backups, they do not test them. (but you've spent the money - never really understood that).

I have memories of ten years of sob stories; guys who were calling in to tech support because they were about to lose their jobs because they were poor steward

Where to start...

[dtjohnson]

1) No surprise reading this:

Over the next few days, as the department, the division and consultants from Microsoft Corp. and Dell Inc. labored to retrieve the data, it became obvious the worst-case scenario was at hand.

2) Always make sure the backup really works.

3) Better procedures are needed if a single tech can reformat both hard drives in the same session

4) Much better hardware and software are needed for data worth $38 billion.

5) Paper backups are a good last resort and as a check on data integrity

I'd hate to be the tech support guy...

[condour75]

And it's not in the recycle bin? Ok, let's not panic. Click start, go to find, choose files and folders...

Damn!

[Penguinisto]

That has got to rank up there w/ the all-time worst 'oh-shit' moments in that poor bastard's career (if it still exists). I wonder what the sysadmin was thinking, storing data on the same partition as the OS. No sane production environment rig that I know of would (or at least should) have that. It may be a Windows thing, but on most servers I've dinked with, the OS sat on a pair of RAID disks by itself, and all the data sat on the monster pile of disks on their own logical RAID drive (at least RAID 5... 5+0 w/ a hot spare preferred).

That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?

/P

Alaskan Pipeline

[Doc Ruby]

I bet it cost a lot less than $200K to bribe the government officials (probably with a few bottles of wine) not to check whether they were protecting their $38B investment with more than $45K worth of IT staff.

Data recovery?

[evilviper]

With hard drives, data doesn't just go away. Sure, it may not be recoverable with simple "undelete" software, but data recovery experts will charge far less than $200,000 to pull important files off of a wiped hard drive.

The same goes for tapes. There is no mention in the article of why they were "unreadable" what level of damage there was to the data, etc.

We all make mistakes, but 3 layers of backup data storage all failing suggests a horrifically poor system in-place. Not JUST "very bad," that's hard to believe, without some massive natural disaster causing it.

Re:Data recovery?

[mwvdlee]

I'm just assuming the harddisks were secure erased, considering that is what pretty much every govenment in the world does when formatting harddisks.

Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.

I could be a douche and say it has never happened

[PrimeWaveZ]

But I'll admit that about 6 years ago, when moving from one old laptop to another, I accidentally erased all of the data from my main and my backup Jaz disks before ensuring that every piece of data from 6th grade through my first year of college was backed up onto new media. It wasn't that I didn't have a good strategy for backup: it was adequate for my needs at the time. It's not that I didn't know I shouldn't format disks with important stuff on them: I figured I had transferred it already. At age 18,

Re:I could be a douche and say it has never happen

[Overzeetop]

What happens when you fuck up that big?

If you the CEO of a major corporation, you get a severance check that looks like a phone number. If you're the IT plebe, you get to look for a new job.

Not to bad

[geekoid]

That they cuold get recovery for only 200K.

I know that many companies would not be able to recover information lost in that manner.

I worked for a company that had not had a back up, at ALL for 4 years. All there business was lectronic. If the system had crashed there company would die. I spent 6 mopnths trying to them to pay for a back up system. FInally the provided a tape drive thawas 5 years old and completly inadequate... I decided to go elsewhere.

To err is human...

[Slipgrid]

To err is human, to really f-up requires root.

From , "nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone." [msn.com]

Really? Why is the that the oil money payouts or the military contract accounts are the only ones that ever get deleted? The IRS is using the same database that they've been using for the past fifty plus years, but they never seem to have that problem.

Remember kids...

[foo fighter]

It's not about the backup, it's about the restore.

If you aren't regulary testing your recovery capabilities, your nightly backups are masturbation. It may make you feel good for a bit, but it's not satisfying.

I lost 75 trillion dollars!

[140Mandak262Jamuna]

I launched notepad and wrote 75 trillion dollars and saved the file. And, because I was feeling very extravagent, I deleted the file. I am rich. I can afford to lose 75 trillion dollars without batting and eyelid and am man enough to brag the info to the whole world.

Come on guys, it took only 200,000$ to create the data. It probably had records of payments totalling 38 billion dollars. But what they lost was 200,000$ not 38 billion dollars.

Someone is trying to cover their ass

[Unique2]

Primary disk: Accidently deleted.
Backup disk: Accidently formatted.
Tape: Unreadable.

What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?

Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.

Re:Someone is trying to cover their ass

[AK Marc]

It sounds more like the reality is that none of these ever existed and someone got caught-out.

Having worked on backups in an Alaskan company run very similarly to the department in question, I think it sounds reasonable. I was a consultant at the time, and I pointed out that the backups have never been tested. It was on the weekly report. It was on the weekly report for about a year. Many people making much more than you make saw that the backups have never been tested. Then there was a crash. It turned out that the backups, set up long before I got there, were set in a tape library. There were 5 tapes and a cleaning tape. The backups would backup server 1 onto tape 1. Then, server 2 onto tape on - set to overwrite. The least important server in the room was last on the backup list, and it was the last to issue the command every night to backup onto tape 1 - set to overwrite. So the email was gone forever. Somehow, the consulting company I work for that pointed out for over a year that backups weren't tested and may not work was to blame for not fixing what was broken long before we were brought in. So I find the description of events quite plausible.

Vista

[Joe The Dragon]

They may of been trying to do a clean install of vista and it some how took out the back up disk and the same time as the main disk. And they where using dell systems.

Perspective

[rlp]

$38 billion is a lot of money. To put that in perspective, for $38 billion, Alaska could build over fifty bridges to nowhere.

Re:Tapes?

[MindStalker]

Yea, tape is pretty common. DVD burners simply aren't rated for backups as some burned DVDs don't have a very long shelf life. Now sounds like some screwed up in purchasing cheap tapes as well. Oh no.

BTW article is silly, the file isn't worth $38 billion $200K at best because thats the cost of rescanning everything. Would be interesting to see an accounting record of how much recreating all the documents would cost had they not had a hard copy.

Re:Tapes?

[Red Flayer]

Well, the summary states that the files were rescanned at a cost of $200,000 -- so it sure sounds like the hard copies were preserved.

Re:Tapes?

[Philosinfinity]

Not even close. We use LTO2 tapes and keep them offsite for 18 months. We've run several test recoveries on tapes > 12 months old with success. Some of our tapes have been in circulation for about 24 - 30 months now and are still writing without difficulty. For restoration purposes, the actual media is rarely the problem. Changes in encryption passwords (with a poorly documented history), files in use, and lost/orphaned files are the most common reasons for restoration failure.

Re:

[Detritus]

7 tracks, 1/2 inch wide tape.

Re:

[stoolpigeon]

yeah they do - when it is more than a couple boxes in ma's basement.

Re:Tapes?

[teflaime]

Most ENTERPRISES still have tape at some level as part of a comprehensive disaster recovery plan. Tape is easy to offsite, fairly reliable overall and still have comprehensive support available in all platforms. Most INDIVIDUALS don't do backups at all.

Re:Tapes?

[Itninja]

Easy to offsite???

I work for an IT organization and we pay a company called Iron Mountain $100's monthly to schlep our boxes and boxes of backup tapes to their offsite storage facility.

And remember there is a difference between making 'backups' (store my important files somewhere else so I can get them in case of a system failure) and preparing for 'disaster recovery' (store everyones files somewhere else so we can rebuild the entire infrastructure in case the building burns to the ground).

Re:

[segfaultcoredump]

HA is not the same as DR

I can have a simple HA cluster that involves two nodes attached to a single disk array, all sitting in the same rack somewhere. Take a guess what happens when the power for the building goes out?

HA is nice, but will do nothing for you in the event of a disaster.

You can structure your site so that you get both, but doing so requires a lot more work (stretched clusters and SAN's spread over miles) and you have to be careful that you dont trash your performance while you are at it. (rea

Re:

[Philosinfinity]

4 Node (2 active and 2 passive) cluster attached to continuous replicating SANs. 2 nodes (1 active and 1 passive) & 1 SAN onsite, 2 nodes & the other SAN in a remote side, CoLo, or hot site. That's our basic design for critical applications. Active nodes provide network load balancing while the passive nodes allow us failover potential. Granted the cost for such a solution is extremely high, but in an enterprise environment where 3 days of downtime cost more than an entire year of housing and ba

Re:

[Achromatic1978]

No, I will not call back a tape from two weeks ago because you made a mistake and saved over your Excel spreadsheet the wrong way, or because you managed to delete something because you're sloppy with a mouse.

Yes you will, or should, when the user's manager gets involved. If you would like to think that because a user trashes a month's accounts, that you can wave some magic hand and say "Yes, I know that data is in pristine condition on last week's backup, but no, you're not getting it just because Waldo

Re:

[Itninja]

Indeed. For super-critical, but not super-massive, data like this especially. I use a couple of external 250GB hard drives to back up my data fast. Then I take one of the HD's home with me for off-site storage. The best part is, there's never a 'restore' period! Just plug the external into the PC and a 'removable device' appears in 'my computer'.

Of course, if you need to store terabytes upon terabytes of data, tape may still by the only option....

Re:Tapes?

[greginnj]

Really? For what volume of data? For people with 100s of GB of transactional data, tape robots are pretty much the only option, or you'll be spending your whole day swapping DVDs. OTOH, it sounds like this was relatively static data (since it could be re-entered from paper), so maybe a DVD version would have been an appropriate measure as well. There's also a lesson here that you should frequently do test restores from backup tapes.

Data Volume

[tempest69]

800,000 images * 300 kb /image (decent scan after compression) = 800k * 300kb = 800 MB * 300 = 2.4GB * 100= 240 Gigs..

Id skip on the DVD backup, sounds like a mistake waiting to happen. Backing this up to a network drive over Gig-E is still going to be a mess, but it should be a few hours of slacktime.. (yes in theory you could manage 240 gigs in roughly 35 minutes over gig-E, but you couldnt pull off enough seeks in that time via the hard drive (800k seeks * 8 ms/seek= 6400s ~= 106 minutes).

Storm

Re:Tapes?

[Sobrique]

In a backup system where you're taking full backups once a week, even with no data churn at all, you end up backing 1/7th of your estate every night. Starts getting a real challenge at about the 15Tb mark, and becomes a whole new adventure in pain when you're talking 100-200Tb.

And then of course, you have 'churn' to worry about. Now, my company does use disk as part of it's backup strategy. Backup to disk and snapshot copies are valuable.

But, well, if you're doing full backups weekly, incremental (or differential) daily, then you're in practice backing up 450% of your 'live' storage every month.

Even onto 'cheap' disk, that gets spendy _very_ fast. That's even before you consider the need to offsite your data for disaster recovery. Tape's still the only real viable way of doing that in bulk. Whilst you can replicate storage arrays, the hardware and bandwidth to do this is also horrifically expensive, especially if you're doing that 1-for-1.

Some people do. Where I work at the moment, 4 of everything is bought, and that includes storage. 1 for dev, one for test, one for production and one for DR. But this kind of thing, does not come cheap, and ... well, no one's going to spend that kind of sum of money (millions) trivially.

Re:

[greginnj]

A file restore is as different from a full system restore as an engine bench test is different from a full rocket launch.

As an IT auditor, I do ding IT shops when they don't do full system restores (which has the dual benefits of verifying that the techs are capable, and verifying that the media is readable). I'm going to be printing out this story and showing it to people who don't do full system restores... I get along fine with BOFHs, and I can sympathize with them about the burden of SOX, but while I'

Re:

[geekoid]

Tape works just fine, espcially for large amounts of data.
They maynot of been maintaining their tapes, or they were using low quality equipment, or it was just one of those things.

For the illogical 'anti-tape' crowd, I will point uot that they did have a problem with the hard drive as well.

Re:

[Detritus]

Tape works great if you are willing to spend big bucks for top-class hardware. Unfortunately, most people try to get by with the cheap stuff, which is very unreliable. Try to explain to a manager why you need a $50K tape system to backup a $10K server. Computers have gotten very cheap, high-quality tape transports haven't.

Re:

[varmittang]

Yeah, when you have a good 300+ gigs of data that needs to be backed up daily, me sitting there burning DVDs are not the way its going to get done.

Re:Tapes?

[daeg]

Hm. Tapes with a proven shelf life of many, many years, or DVDs where a single scratch can render 4GB of data worthless. I wonder which enterprises (or governments) should chose?

Re:

[evilviper]

you _can_ render the data on a DVD unreadable/unaccessible because of a scratch.

No, you _CANT_.

For that, this "scratch" has to be a massive gash, which goes more than halfway through the depth of the plastic disc. Additionally, it has to span the entire radius of the disk just to make it "difficult" to recover a significant portion of the data.

Their decay increases greatly at increased temperatures and, expecially, under UV from the sun.

Your backup solution should NOT involve throwing a bunch of bare DVD-R

Re:

[An dochasac]

Tapes are still widely used in the industry. It scares me how many companies may be choosing the convenience of DVD over the stability of tape. I've seen digital 8mm digital (exabyte) tapes survive unharmed 10 years in an attic where temperatures ranged from 15F-150F and yet I've seen CDs and DVDs without a scratch on them mysteriously stop working after less than 2 years. As for hard drives, they can be made part of a viable storage technology, but the drives themselves are notoriously unreliable. I'd

Re:

[Ryan Amos]

Yep. Hard drives and DVDs are backup media, tape is disaster recovery media. DVDs are not durable enough nor do they really hold enough (tape has much better density; 400GB per LTO3 tape compared to 9GB per DL-DVD.) Tape is still the undisputed king for long-term, high capacity backups.

For home use, online backups are fine, but for even small businesses, volumes of data approaching 1TB are not uncommon.

re: tape backup

[King_TJ]

Where I work, we finally phased out our tape backup drives last year in our server room. A product called the "CRU Dataport" is a very nice removable, hot-swap hard drive frame and carrier assembly. You just install their frame in an external 5.25" drive bay, and buy a set of carriers for it. Install suitable SATA drive drives in each carrier, and switch them out nightly just like backup tapes. Backup software like "Backup Exec" can still be used, but it will treat each cartridge just like it did the ba

DVDs are a joke.

[mosch]

An LTO Ultrium 3 tape holds 400GB uncompressed, and you can buy libraries that hold hundreds or thousands of these tapes (and dozens of drives).

Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.

The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.

Re:

[imsabbel]

You forget to mention that such a tape (just the tape) is also as expensive as a 250Gbyte HD.
Add robots, and it gets seriously expensive for large installations...

Re:

[Ephemeriis]

Just curious... do most people still use *tape* for backup? Personally, I use multiple hard drives and a DVD burner on a daily basis.

I guess it depends on what you're backing up...

Of all our clients, the smallest backup we've got is about 14 GB. That's too big for a single DVD-R, but it fits just fine on a DDS3 [wikipedia.org]. We can also easily automate a tape backup - just instruct a secretary or someone to swap the tape in the morning. Tapes are reasonably durable too...more rugged, in general, than a removable HDD.

Re:

[TheLink]

"The tapes are about $100, which is close to par with hard drives now (400 GB native), but in the not too distant future, LTO4 will be out, which doubles the capacity."

Yeah, but when LTO4 is out you can't use them without buying a new _expensive_ LTO4 drive.

Whereas in the not too distant future when new hard drives with double the capacity are out, you can still use them with your existing computers (as long as they still support SATA).

Basically HDDs = media + drive, and they are about the same price as tap

Re:

[geekoid]

"...e that you have also reformatted the back up hard drive."

That implies that the original and the back up had been reformatted.

Re:

[Detritus]

Another question, doesn't anyone test their backup systems?

I've often been in the situation where I couldn't properly test the backup system because management decided to save money by not buying any spare hardware.

"Hey, we bought you a tape drive and a box of tapes, quit complaining!"

Re:

[Volante3192]

Hard drives and backup tapes are both magnetic, and they technically did keep a backup on a format that wasn't magnetic: the original hard copies.

Note from the article (and the summary) the IT guy formatted both the original and the backup hard drives. Then there was the bad tape. So that's 2 sources. What surprises me is it wasn't on another tape somewhere.

Re:

[lymond01]

Well, let's say it's all of Alaska's tax income data for the 2006 year. Of course it's not impossible to recreate the data, but since it's basically electronic money, you could consider it "lost" if only temporarily.

Re:Actual Cost??

[ClickOnThis]

Err.... if it only cost $200,000 to replace the data, where the hell does the $38 Billion figure come from?

The site is slashdotted, so I can't read TFA, but my guess would be the information isn't actually "worth" $38B. It just represents an accounted amount of $38B.

The actual value of the data is what it would cost to replace it (or perhaps do without it) -- in this case, $200,000. Consider an analogy (20th-century, but illustrative): if you were to send a paper bank-check for $10,000 via a courier, the declared value for insurance would not be $10,000. It would be the cost of recovering from the loss of the check, which would be the stop-payment fee plus the cost of sending a new one.

Re:Actual Cost??

[saigon_from_europe]

1) Write "200 000 000 000 USD" on the paper.
2) Type what's on the paper into a .txt file
3) Save the file
4) Delete the file
5) Empty the recycle bin
6) Recreate file by retyping data from the paper
7) Post the story on the /. how big financial disaster you've made, and how you've saved your ass

Re:

[rubycodez]

that would have been a better ending to the story, "they then went back to the original documents in cardboard boxes, but alas their auditing firm used ex-Arthur Andersen employees and they had all been shredded."

Re:

[Detritus]

Please report to your nearest Microsoft customer reeducation camp.