I do manual backups using glorified copy tools (Cobian, Synctoy) to an external drive, and recently I've run into a large batch of camera raw files which became silently corrupted on my main drive. So.. what are the solutions to detect such corruption in a home setup without RAID arrays or specialized backup software?

Since the corruption happened on my main drive, I was able to notice it when my catalogue software started throwing errors. If I had not manually noticed it, and the 'good' drive (backup or main) died, I would have run into a backup restore failure.

One solution I realize is checksumming all files, storing the hash along with the file in both original and backup locations. Then I would have to continually run tasks across both the original and the backup copies that compare these checksums. I've tried using corz "Checksum" - and gave up soon after seeing it would take an ungodly amount of time to hash near 10 terabytes of data. I don't see how it's possible to run this continuously.

Are there solutions that incorporate corruption detection in a better manner? I do not mind manually correcting the issues as they arise.

  • I tried using a file diff program (windiff) but this obviously takes a lot longer than checksumming does, and furthermore requires all drives to be connected while comparing.. – user2421873 Aug 20 '15 at 3:31

If you want to proactively detect the corruption, then you need a dedicated task that runs continuously in the background, re-computes checksums and compares them to previously stored values. There's no way around this, because bitrot is spontaneous random phenomenon. Furthermore, the common form of it is a single bit flips and that cannot be detected in any other way except for combing through all stored data and verifying its consistency.

On the other hand, if you wanted to recover from corruption, then there are options. For example, you can use FreeNAS as your storage facility - it uses zfs, which can be set up with redundancy, which in turn will allow for recovery of corrupted blocks (that are detected automatically upon accessing them through the use of mandatory checksumming).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.