Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Is it possible to encode data is such a way that it can self-recover from corruptions? (corruptions - for example, hard drive or usb drive with bad sectors)

I'm not looking for solutions involving backups, raid, or file system repair. I'm looking for solutions that employ some kind of self-healing process on raw data.

I'm also aware that I can checksum my data ahead of time and that this will help me know (in most cases) if corruption has occurred. For me this is only half the battle, I need the system to not only know that something has changed but be able to restore from the change itself.

I am aware of ZFS and its capabilities:

ZFS can heal the data if the storage pool has redundancy via ZFS mirroring or RAID. If the storage pool consists of a single disk, it is possible to provide such redundancy by specifying "copies=2" (or "copies=3"), which means that data will be stored twice (thrice) on the disk, effectively halving (or, for "copies=3", reducing to one third) the storage capacity of the disk.

But I am looking for somewhat different solutions/tools, for example something that allows me to protect data on a usb flash drive that is formatted FAT,FAT32, exFAT. Does such a thing exist?

share|improve this question
    
What kind of data are you trying to protect? Databases, documents, images, etc? Are you speaking in general, or is this for a specific application? –  Taegost Jul 17 '13 at 17:49
    
Data in general, no specific "kind" of data. –  dtmland Jul 17 '13 at 17:57
    
You can not protect data on media which does not offer parity or error recovery. Any scheme you deploy is subject to data corruption just as the data that the scheme is supposed to protect. –  Ярослав Рахматуллин Jul 18 '13 at 0:11
    
@ЯрославРахматуллин Initially i did not believe this to be possible, but using tools like those mentioned in this answers below you CAN protect data on media which <uses a file system that> does not offer parity or error recovery. –  dtmland Jul 18 '13 at 2:05
    
Yeah, so long as the errors are smaller than the maximum error correction those tools provide. If you have more than 16 corrupted bytes per 255 bytes, your data is gone. If you don't buy the cheapest flash sticks for data you care about, or if you don't store it on floppies, then the security provided by parity bit error correction schemes is more of a soothing balm for paranoia than protection against data loss. It's better than nothing, but it's not safe. –  Ярослав Рахматуллин Jul 18 '13 at 2:40

2 Answers 2

up vote 2 down vote accepted

No such tool exists to magically recover corrupted data unless you have sufficient parity/backup copies of the data in question. @grawity indicates tools like PAR2, QuickPar, and WinRAR can do what you want. To work properly those tools will build parity data into the archive so recovery will (hopefully) be possible if some data is corrupt. I am a long time WinRAR/7z user but I have never explored this functionality so I cannot comment on its effectiveness.

Most file systems have some measure of "self healing" like NTFS check disk, which is just as likely to corrupt your data as it is to fix it.

ZFS is not the only file system with what I would consider "true" self healing capabilities, but it is the most mature. Btrfs and ReFS also have "true" self healing capabilities. Even file systems with "true" self healing require parity data to heal.

I would avoid putting data you really care about on a flash drive if at all possible, not because it is flash, but because even if you have parity data it will still only be contained on one physical device. If you must use a flash drive make sure it uses high quality memory and avoid cosmic ray storms. Experiment with some of the tools @grawity mentioned, I know I will :)

share|improve this answer
2  
"Self-healing" is not the same as an ordinary disk check, which just tries to make metadata consistent. ZFS self-healing code uses only the copy of data that has the correct checksum. And this doesn't need to be done at filesystem level – the same (or similar) algorithms can be implemented by archivers like (Win)RAR or by specialized tools like PAR2/QuickPar. Many computers use RAM that has error correction (ECC) built in, for detecting magically flipped bits. –  grawity Jul 17 '13 at 18:23
1  
@grawity Yes, that is why I used "self healing" in quotes. NTFS check disk isn't really self healing, but it tries to fill that role. What it comes down to is if you don't have a good copy of the data there is no way to heal, so a self contained archive (WinRAR) cannot self heal, it can only tell you that it is corrupt. In fairness the same is true at the file system level because parity data is still required to "self heal". I say "arbitray file system" because some file systems have the correct tools built in (i.e. ZFS, Btrfs, ReFS). I have edited my answer to be more clear on these points. –  ubiquibacon Jul 17 '13 at 18:40
2  
WinRAR is able to generate additional "recovery volumes" that can recover from variable amounts of corruption or data loss; for example, split a large archive into 10 volumes + 1 recovery volume, and be able to regenerate a full missing volume. I don't know how advanced it is, however. For recovery of a single volume, simply XORing all of them would generate enough "recover information". But I'm sure there are much more advanced algorithms than just that. I know very little about them, although erasure coding and Reed-Solomon come to mind now. –  grawity Jul 17 '13 at 20:26
    
@grawity Thanks for the heads up on recover volumes. I will explore that functionality in WinRAR as soon as I get a chance. –  ubiquibacon Jul 17 '13 at 20:53

Thanks to an old slashdot post, I have found a solution/tools that will protect data on flash drives.

Shielding your files with Reed-Solomon codes

Looks like it uses data parity to heal corrupted/damaged sectors.

What follows is a simple description of a way I use to additionally "shield" my important files, so that even if some sectors hosting them are lost, I still end up salvaging everything.

The idea behind this process is error correcting codes, like for example the ubiquitous Reed-Solomon. With Reed-Solomon, parity bytes are used to protect a block of data from a specified maximum number of errors per block. In the tools described below, a block of 223 bytes is shielded with 32 bytes of parity. The original 223 bytes are then morphed into 255 "shielded" ones, and can be recovered even if 16 bytes from inside the "shielded" block turn to noise...

The "freeze" and "melt" scripts can be used to create files that can be deployed to a USB flash drive (or any medium for that matter).

We "shield" a file with the freeze.sh script, which is part of my package; we then melt.sh the frozen file, and verify (through md5sum) that the new generated file is exactly the same as the original one. We then proceed to deliberately destroy 64KB of the shielded file (that's a lot of consecutive sectors!), using dd to overwrite 127 sectors with zeros. We invoke melt.sh again, and we see that the new generated file (data3) has the same MD5 sum as the original one - it was recovered perfectly.

Fantastic!!!

share|improve this answer
    
awesome, someone has duplicated cp and rsync. o_O –  Sirex Jul 17 '13 at 21:29
    
@Sirex How so? How do those achieve the same result? –  dtmland Jul 17 '13 at 21:40
    
well, "shield" is copying the file, and "freeze" is fixing bad data with good. In fact (from what i can tell), this is essentially cp file1 file2; rm file1; cp file2 file1, no ? –  Sirex Jul 17 '13 at 23:45
1  
So how exactly is this solution better than a Parchive? –  Karan Jul 18 '13 at 2:33
1  
I've used PAR files and WinRAR's recovery records before and they've helped a lot, but this solution is new to me so I can't compare them without further research. –  Karan Jul 18 '13 at 2:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.