One Way To Save Digital Archives From File Corruption

storagedude points out this article about one of the perils of digital storage, the author of which "says massive digital archives are threatened by simple bit errors that can render whole files useless. The article notes that analog pictures and film can degrade and still be usable; why can't the same be true of digital files? The solution proposed by the author: two headers and error correction code (ECC) in every file."

par files

[ionix5891]

include par2 files

What about the "block errors"?

[MathFox]

Most of the storage media in common use (disks, tapes, CD/DVD-R) already do use ECC at sector of block level and will fix "single bit" errors at firmware level transparently. What is more of an issue at application level are "missing block" errors; when the low-level ECC fails and the storage device signals "unreadable sector" and one or more blocks of data are lost.

Off course this can be fixed by "block redundancy" (like RAID does), "block recovery checksums" or old-fashioned backups.

Re:What files does a single bit error destroy?

[Jane Q. Public]

That's complete nonsense. Just for one example, if the bit is part of a numeric value, depending on where that bit is, it could make the number off anywhere from 1 to 2 BILLION or even a lot more, depending on the kind of representation being used.

Very, very old news....

[gweihir]

It has been done like that for decades. Look at what archival tape does or DVDisaster or modern HDDs.

Also, this does not solve the problem, it just defers it. Why is this news?

Also, Bittorrent

[NoPantsJim]

I remember reading a story of a guy who had to download a file from Apple that was over 4 gigabytes, and had to attempt it several times because each came back corrupted due to some problem with his internet. Eventually, he gave up and found the file on bit torrent, but realized if he saved it in the same location as the corrupted file, it would check the file and then overwrite it with the correct information. He was able to fix it in under an hour using bittorrent rather than trying to re-download the file while crossing his fingers and praying for no corruption.

I know it's not a perfect example, but just one way of looking at it.

Re:To much reinvention

[Anonymous Coward]

Or, for those that prefer a BSD to a SysV unix, it also works fine on FreeBSD.

Re:What about the "block errors"?

[tepples]

anyone know of the equivalent RAID model for things like tape?

Four tapes data, one tape PAR2.

Re:To much reinvention

[An dochasac]

"says massive digital archives are threatened by simple bit errors that can render whole files useless.
Isn't this what filesystem devs have been concentrating on for about 5 years now?

Not just 5 years. ZFS's CRC on every datablock and Raid Z (no raid hold) are innovative and obviously the next step in filesystem evolution. But attempts at redundancy aren't new. I'm surprised the article is discussing relatively low teck old hat ideas such as two filesystem headers. Even DOS's FAT used this raid0 type of brute force redundancy by having two FAT tables. The Commodore Amiga's Intuition filesystem did this better than Microsoft back in 1985 by having forward and backward links in every block which made it possible to repair block pointer damage by searching for a reference to the bad block in the preceding and following block.
And I suppose if ZFS doesn't catch on, 25 or 30 years from now Apple or Microsoft will finally come up with it and say, "Hey look what we invented!"

Re:Ecc?

[TheThiefMaster]

Asking for a definition of ecc [google.co.uk] turns it up, so it's obviously not that uncommon. And as we're talking about data corruption, it's the obvious one.

Most IT techs would recognise the term from "ECC Ram", which is ram that is capable of correcting bit errors and is often required by server motherboards.