-2

Something that really bothers me is silent data corruption. Over the years, I experienced it many times that suddenly you want to open some old pictures just to scratch your head and wonder wether there wasn't another photo in that folder. I store all my project files and pictures in my dropbox and have additionally an external RAID 1 disk (WDMyBookMirror) where I backup these files to. About a year ago I had upgraded from an older MyBook Mirror to a new one. The old one was still lying around with the old data, and I decided to run a comparison with FileMerge of the folder trees of the old disk with the current state of my new Backup disk that is in use. Turns out about 10 files have disappeared in this one year or so from my new disk. Random photos and videos. If I hadn't run the comparison, I would never have noticed.

This gets me to the question. How can I store my data to be really sure that I'm not losing any files without noticing?

Apparently the RAID 1 didn't safe me from the silent data corruption. Neither did the NTFS file system on it, and a cloud solution such as dropbox would probably just sync the erroneous file changes into the cloud, corrupting/losing the data in other places too. Creating regular backups of the backup also won't inform me about pictures silently lost or corrupted.

So, what to do? I read a bit about ReFS and ZFS. Is that the way to go?

I don't mind paying for redundancy - but what's the way to store my data so I can be SURE that silent data corruption can't happen?

  • 1
    Store it in multiple places, on multiple forms of media. Tape, cloud, disk, both hot-disk and cold-disk. – music2myear Apr 3 '19 at 21:52
  • That won't protect me of silent data corruption though. Sure, the data might still be somewhere - but unless I specifically search for a file or try to use it, I won't know that the file corrupted (or depending on the data, I might not even know that it's corrupt when I'm using it) – user1282931 Apr 4 '19 at 7:34
  • This topic is better served by open discussion because there is not a single concrete way to accomplish what OP is asking. Perhaps a forum, rather than a QA site, that is dedicated to storage, may be the place to delve into the details of this problem. – music2myear Apr 4 '19 at 16:36
2

The ZFS file system is designed to be highly resistant to silent data corruption. Using sophisticated checksum techniques including block-chaining and storing checksums of data in media locations that are removed from the data storage locations themselves, plus the ability to store multiple copies of metadata ("ditto blocks"), ZFS can not only detect silent corruption, but when the ZFS pool has sufficient redundancy, ZFS can repair the corruption. In this sense, ZFS is described as a "self-healing" file system.

Even for single-disk storage systems, ZFS has significant advantages over many traditional filesystems. Briefly, from Oracle's ZFS docs:

The benefits of storing multiple copies of ZFS user data are as follows:

  • Improves data retention by enabling recovery from unrecoverable block read faults, such as media faults (commonly known as bit rot) for all ZFS configurations.

  • Provides data protection, even when only a single disk is available.

  • Enables you to select data protection policies on a per-file system basis, beyond the capabilities of the storage pool.

Many other aspects about ZFS can be found in the related documentation at that link.

That said, no system can ever be perfect. But ZFS is a strong tool to have available.

| improve this answer | |
  • Do you happen to know the pros and cons (especially against ReFS)? – user1282931 Apr 4 '19 at 7:31
0

For stability, sapphire optical disks are available, though expensive. Barring that, create multiple Zip files on different servers. Since Zip has internal checksums, you'd know immediately if an archive is damaged.

Digital technology is based on accommodating the error rate through redundancy, checksums, and other means. Even theoretically perfect media will still be subject to errors due to cosmic rays, electrical noise, etc.

| improve this answer | |
  • Interesting Idea with the ZIP files, but I still want to be able to easily access and use all my files without having to unzip them first. I understand that data errors can always happen - what bothers me more is that they go unnoticed most of the time – user1282931 Apr 4 '19 at 7:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.